Need help with python-sitemap?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

c4software
243 Stars 97 Forks GNU General Public License v3.0 140 Commits 16 Opened issues

Description

Mini website crawler to make sitemap from a website.

Services available

!
?

Need anything else?

Contributors list

# 125,789
sitemap
Firebas...
Objecti...
C
77 commits
# 373,602
Python
sitemap
14 commits
# 410,938
sitemap
C#
HTML
python-...
7 commits
# 375,997
sitemap
C
Shell
weather...
3 commits
# 555,994
Python
sitemap
2 commits
# 108,030
pytorch
Tensorf...
CSS
multita...
1 commit
# 311,265
HTML
Shell
mifare
Android
1 commit
# 401,195
R
Shell
C++
bayesia...
1 commit
# 389,962
C
HTML
templat...
Gradle
1 commit
# 10,437
F#
Xamarin
Univers...
f-sharp
1 commit
# 613,878
Python
sitemap
1 commit

Python-Sitemap

Simple script to crawl websites and create a sitemap.xml of all public link in it.

Warning : This script only works with Python3

Simple usage

>>> python main.py --domain http://blog.lesite.us --output sitemap.xml

Advanced usage

Read a config file to set parameters: You can overide (or add for list) any parameters define in the config.json

>>> python main.py --config config/config.json

Enable debug:

    $ python main.py --domain https://blog.lesite.us --output sitemap.xml --debug

Enable verbose output:

  $ python main.py --domain https://blog.lesite.us --output sitemap.xml --verbose

Enable Image Sitemap

More informations here https://support.google.com/webmasters/answer/178636?hl=en

  $ python main.py --domain https://blog.lesite.us --output sitemap.xml --images

Enable report for print summary of the crawl:

  $ python main.py --domain https://blog.lesite.us --output sitemap.xml --report

Skip url (by extension) (skip pdf AND xml url):

  $ python main.py --domain https://blog.lesite.us --output sitemap.xml --skipext pdf --skipext xml

Drop a part of an url via regexp :

  $ python main.py --domain https://blog.lesite.us --output sitemap.xml --drop "id=[0-9]{5}"

Exclude url by filter a part of it :

  $ python main.py --domain https://blog.lesite.us --output sitemap.xml --exclude "action=edit"

Read the robots.txt to ignore some url:

  $ python main.py --domain https://blog.lesite.us --output sitemap.xml --parserobots

Human readable XML

$ python3 main.py --domain https://blog.lesite.us --images --parserobots | xmllint --format -

Multithreaded

$ python3 main.py --domain https://blog.lesite.us --num-workers 4

with basic auth

You need to configure

username
and
password
in your
config.py
before

$ python3 main.py --domain https://blog.lesite.us --auth

Output sitemap index file

Sitemaps with over 50,000 URLs should be split into an index file that points to sitemap files that each contain 50,000 URLs or fewer. Outputting as an index requires specifying an output file. An index will only be output if a crawl has more than 50,000 URLs:

$ python3 main.py --domain https://blog.lesite.us --as-index --output sitemap.xml

Docker usage

Build the Docker image:

  $ docker build -t python-sitemap:latest .

Run with default domain :

  $ docker run -it python-sitemap

Run with custom domain :

  $ docker run -it python-sitemap --domain https://www.graylog.fr

Run with config file and output :

You need to configure config.json file before

  $ docker run -it -v `pwd`/config/:/config/ -v `pwd`:/home/python-sitemap/ python-sitemap --config config/config.json

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.