by c4software

c4software / python-sitemap

Mini website crawler to make sitemap from a website.

233 Stars 92 Forks Last release: Not found GNU General Public License v3.0 137 Commits 0 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:


Simple script to crawl websites and create a sitemap.xml of all public link in it.

Warning : This script only works with Python3

Simple usage

>>> python --domain --output sitemap.xml

Advanced usage

Read a config file to set parameters: You can overide (or add for list) any parameters define in the config.json

>>> python --config config/config.json

Enable debug:

    $ python --domain --output sitemap.xml --debug

Enable verbose output:

  $ python --domain --output sitemap.xml --verbose

Enable Image Sitemap

More informations here

  $ python --domain --output sitemap.xml --images

Enable report for print summary of the crawl:

  $ python --domain --output sitemap.xml --report

Skip url (by extension) (skip pdf AND xml url):

  $ python --domain --output sitemap.xml --skipext pdf --skipext xml

Drop a part of an url via regexp :

  $ python --domain --output sitemap.xml --drop "id=[0-9]{5}"

Exclude url by filter a part of it :

  $ python --domain --output sitemap.xml --exclude "action=edit"

Read the robots.txt to ignore some url:

  $ python --domain --output sitemap.xml --parserobots

Human readable XML

$ python3 --domain --images --parserobots | xmllint --format -


$ python3 --domain --num-workers 4

with basic auth

You need to configure

in your

$ python3 --domain --auth

Output sitemap index file

Sitemaps with over 50,000 URLs should be split into an index file that points to sitemap files that each contain 50,000 URLs or fewer. Outputting as an index requires specifying an output file. An index will only be output if a crawl has more than 50,000 URLs:

$ python3 --domain --as-index --output sitemap.xml

Docker usage

Build the Docker image:

  $ docker build -t python-sitemap:latest .

Run with default domain :

  $ docker run -it python-sitemap

Run with custom domain :

  $ docker run -it python-sitemap --domain

Run with config file and output :

You need to configure config.json file before

  $ docker run -it -v `pwd`/config/:/config/ -v `pwd`:/home/python-sitemap/ python-sitemap --config config/config.json

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.