=========

Memorious

The solitary and lucid spectator of a multiform, instantaneous and almost intolerably precise world.

-- Funes the Memorious <http:>_,
Jorge Luis Borges

.. image:: https://github.com/alephdata/memorious/workflows/memorious/badge.svg

memorious

• Maintain an overview of a fleet of crawlers
• Schedule crawler execution in regular intervals
• Store execution information and error messages
• Distribute scraping tasks across multiple machines
• Make crawlers modular and simple tasks re-usable
• Get out of your way as much as possible

.. image:: docs/memorious-ui.png

Design

When writing a scraper, you often need to paginate through through an index page, then download an HTML page for each result and finally parse that page and insert or update a record in a database.

memorious

crawlers

stages

stage

crawlers

The basic steps of writing a Memorious crawler:

1. Make YAML crawler configuration file
2. Add different stages
3. Write code for stage operations (optional)
4. Test, rinse, repeat

Documentation

The documentation for Memorious is available at

memorious.readthedocs.io 

docs

To build the documentation, inside the

docs

make html

You'll find the resulting HTML files in /docs/_build/html.