A Python library for automating interaction with websites.
.. image:: /assets/mechanical-soup-logo.png :alt: MechanicalSoup. A Python library for automating website interaction.
MechanicalSoup was created by
M Hickford, who was a fond user of the
Mechanizelibrary. Unfortunately, Mechanize was
incompatible with Python 3 until 2019__ and its development stalled for several years. MechanicalSoup provides a similar API, built on Python giants
Requests__ (for HTTP sessions) and
BeautifulSoup__ (for document navigation). Since 2017 it is a project actively maintained by a small team including
|Latest Version| |Supported Versions|
PyPy3 is also supported (and tested against).
Download and install the latest released version from
pip install MechanicalSoup
Download and install the development version from
pip install git+https://github.com/MechanicalSoup/MechanicalSoup
Installing from source (installs the version in the current working directory)::
python setup.py install
(In all cases, add
installcommand to install in the current user's home directory.)
The full documentation is available on https://mechanicalsoup.readthedocs.io/. You may want to jump directly to the
automatically generated API documentation__.
From__, code to get the results from a Qwant search:
.. code:: python
"""Example usage of MechanicalSoup to get the results from the Qwant search engine. """
import re import mechanicalsoup import html import urllib.parse
Connect to duckduckgo
browser = mechanicalsoup.StatefulBrowser(user_agent='MechanicalSoup') browser.open("https://lite.qwant.com/")
Fill-in the search form
browser.select_form('#search-form') browser["q"] = "MechanicalSoup" browser.submit_selected()
Display the results
for link in browser.page.select('.result a'): # Qwant shows redirection links, not the actual URL, so extract # the actual URL from the redirect link: href = link.attrs['href'] m = re.match(r"^/redirect/[^/]/(.)$", href) if m: href = urllib.parse.unquote(m.group(1)) print(link.text, '->', href)
More examples are available in__.
For an example with a more complex form (checkboxes, radio buttons and textareas), read__ and __.
|Build Status| |Coverage Status| |Requirements Status| |Documentation Status| |CII Best Practices| |LGTM Alerts| |LGTM Grade|
Instructions for building, testing and contributing to MechanicalSoup: see__.
.. |Latest Version| image:: https://img.shields.io/pypi/v/MechanicalSoup.svg :target: https://pypi.python.org/pypi/MechanicalSoup/ .. |Supported Versions| image:: https://img.shields.io/pypi/pyversions/mechanicalsoup.svg :target: https://pypi.python.org/pypi/MechanicalSoup/ .. |Build Status| image:: https://travis-ci.org/MechanicalSoup/MechanicalSoup.svg?branch=master :target: https://travis-ci.org/MechanicalSoup/MechanicalSoup .. |Coverage Status| image:: https://codecov.io/gh/MechanicalSoup/MechanicalSoup/branch/master/graph/badge.svg :target: https://codecov.io/gh/MechanicalSoup/MechanicalSoup .. |Requirements Status| image:: https://requires.io/github/MechanicalSoup/MechanicalSoup/requirements.svg?branch=master :target: https://requires.io/github/MechanicalSoup/MechanicalSoup/requirements/?branch=master .. |Documentation Status| image:: https://readthedocs.org/projects/mechanicalsoup/badge/?version=latest :target: https://mechanicalsoup.readthedocs.io/en/latest/?badge=latest .. |CII Best Practices| image:: https://bestpractices.coreinfrastructure.org/projects/1334/badge :target: https://bestpractices.coreinfrastructure.org/projects/1334 .. |Gitter Chat| image:: https://badges.gitter.im/MechanicalSoup/MechanicalSoup.svg :target: https://gitter.im/MechanicalSoup/Lobby .. |LGTM Alerts| image:: https://img.shields.io/lgtm/alerts/g/MechanicalSoup/MechanicalSoup.svg :target: https://lgtm.com/projects/g/MechanicalSoup/MechanicalSoup/ .. |LGTM Grade| image:: https://img.shields.io/lgtm/grade/python/g/MechanicalSoup/MechanicalSoup.svg :target: https://lgtm.com/projects/g/MechanicalSoup/MechanicalSoup/