MechanicalSoup

by MechanicalSoup

MechanicalSoup / MechanicalSoup

A Python library for automating interaction with websites.

3.6K Stars 329 Forks Last release: about 1 year ago (v0.12.0) MIT License 554 Commits 23 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

.. image:: /assets/mechanical-soup-logo.png :alt: MechanicalSoup. A Python library for automating website interaction.

Home page

https://mechanicalsoup.readthedocs.io/

Overview

A Python library for automating interaction with websites. MechanicalSoup automatically stores and sends cookies, follows redirects, and can follow links and submit forms. It doesn't do JavaScript.

MechanicalSoup was created by

M Hickford
, who was a fond user of the
Mechanize 
library. Unfortunately, Mechanize was
incompatible with Python 3 until 2019
__ and its development stalled for several years. MechanicalSoup provides a similar API, built on Python giants
Requests 
__ (for HTTP sessions) and
BeautifulSoup
__ (for document navigation). Since 2017 it is a project actively maintained by a small team including
@hemberger 
__ and
@moy
__.

|Gitter Chat|

Installation

|Latest Version| |Supported Versions|

PyPy and PyPy3 are also supported (and tested against).

Download and install the latest released version from

PyPI 
__::

pip install MechanicalSoup

Download and install the development version from

GitHub 
__::

pip install git+https://github.com/MechanicalSoup/MechanicalSoup

Installing from source (installs the version in the current working directory)::

python setup.py install

(In all cases, add

--user
to the
install
command to install in the current user's home directory.)

Documentation

The full documentation is available on https://mechanicalsoup.readthedocs.io/. You may want to jump directly to the

automatically generated API
documentation 
__.

Example

From

__, code to get the results from
a Qwant search:

.. code:: python

"""Example usage of MechanicalSoup to get the results from the Qwant
search engine.
"""

import re import mechanicalsoup import html import urllib.parse

Connect to duckduckgo

browser = mechanicalsoup.StatefulBrowser(user_agent='MechanicalSoup') browser.open("https://lite.qwant.com/")

Fill-in the search form

browser.select_form('#search-form') browser["q"] = "MechanicalSoup" browser.submit_selected()

Display the results

for link in browser.page.select('.result a'): # Qwant shows redirection links, not the actual URL, so extract # the actual URL from the redirect link: href = link.attrs['href'] m = re.match(r"^/redirect/[^/]/(.)$", href) if m: href = urllib.parse.unquote(m.group(1)) print(link.text, '->', href)

More examples are available in

__.

For an example with a more complex form (checkboxes, radio buttons and textareas), read

__
and 
__.

Development

|Build Status| |Coverage Status| |Requirements Status| |Documentation Status| |CII Best Practices| |LGTM Alerts| |LGTM Grade|

Instructions for building, testing and contributing to MechanicalSoup: see

__.

Common problems

Read the

FAQ
__.

.. |Latest Version| image:: https://img.shields.io/pypi/v/MechanicalSoup.svg :target: https://pypi.python.org/pypi/MechanicalSoup/ .. |Supported Versions| image:: https://img.shields.io/pypi/pyversions/mechanicalsoup.svg :target: https://pypi.python.org/pypi/MechanicalSoup/ .. |Build Status| image:: https://travis-ci.org/MechanicalSoup/MechanicalSoup.svg?branch=master :target: https://travis-ci.org/MechanicalSoup/MechanicalSoup .. |Coverage Status| image:: https://codecov.io/gh/MechanicalSoup/MechanicalSoup/branch/master/graph/badge.svg :target: https://codecov.io/gh/MechanicalSoup/MechanicalSoup .. |Requirements Status| image:: https://requires.io/github/MechanicalSoup/MechanicalSoup/requirements.svg?branch=master :target: https://requires.io/github/MechanicalSoup/MechanicalSoup/requirements/?branch=master .. |Documentation Status| image:: https://readthedocs.org/projects/mechanicalsoup/badge/?version=latest :target: https://mechanicalsoup.readthedocs.io/en/latest/?badge=latest .. |CII Best Practices| image:: https://bestpractices.coreinfrastructure.org/projects/1334/badge :target: https://bestpractices.coreinfrastructure.org/projects/1334 .. |Gitter Chat| image:: https://badges.gitter.im/MechanicalSoup/MechanicalSoup.svg :target: https://gitter.im/MechanicalSoup/Lobby .. |LGTM Alerts| image:: https://img.shields.io/lgtm/alerts/g/MechanicalSoup/MechanicalSoup.svg :target: https://lgtm.com/projects/g/MechanicalSoup/MechanicalSoup/ .. |LGTM Grade| image:: https://img.shields.io/lgtm/grade/python/g/MechanicalSoup/MechanicalSoup.svg :target: https://lgtm.com/projects/g/MechanicalSoup/MechanicalSoup/

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.