Instagram Crawler

Below is what you can do with this program: - Get Instagram posts/profile/hashtag data without using Instagram API.

crawler.py

liker.py

This crawler could fail due to updates on instagram’s website. If you encounter any problems, please contact me.

Install

1. Make sure you have Chrome browser installed.
2. Download chromedriver and put it into bin folder:

   ./inscrawler/bin/chromedriver

3. Install Selenium:

   pip3 install -r requirements.txt

4. cp inscrawler/secret.py.dist inscrawler/secret.py

User Auth

1. Open

   inscrawler/secret.py

   file.
2. Change the

   username

   and

   password

   variables' value to the ones corresponding to your Instagram account.

   username = 'my_ig_username'
   password = '***********'
   

Crawler

Usage

positional arguments:
  mode
    options: [posts, posts_full, profile, hashtag]

optional arguments:
  -n NUMBER, --number NUMBER
                        number of returned posts
  -u USERNAME, --username USERNAME
                        instagram's username
  -t TAG, --tag TAG     instagram's tag name
  -o OUTPUT, --output OUTPUT
                        output file name(json format)
  --debug               see how the program automates the browser
  --fetch_comments      fetch comments
Turning on the flag might take forever to fetch data if there are too many commnets.
  --fetch_likes_plays   fetch like/play number
  --fetch_likers        fetch all likers
Instagram might have rate limit for fetching likers. Turning on the flag might take forever to fetch data if there are too many likes.
  --fetch_mentions      fetch users who are mentioned in the caption/comments (startwith @)
  --fetch_hashtags      fetch hashtags in the caption/comments (startwith #)
  --fetch_details       fetch username and photo caption
only available for "hashtag" search

Example

python crawler.py posts_full -u cal_foodie -n 100 -o ./output
python crawler.py posts_full -u cal_foodie -n 10 --fetch_likers --fetch_likes_plays
python crawler.py posts_full -u cal_foodie -n 10 --fetch_comments
python crawler.py profile -u cal_foodie -o ./output
python crawler.py hashtag -t taiwan -o ./output
python crawler.py hashtag -t taiwan -o ./output --fetch_details
python crawler.py posts -u cal_foodie -n 100 -o ./output # deprecated

1. Choose mode

   posts

   , you will get url, caption, first photo for each post; choose mode

   posts_full

   , you will get url, caption, all photos, time, comments, number of likes and views for each posts. Mode

   posts_full

   will take way longer than mode

   posts

   . [

   posts

   is deprecated. For the recent posts, there is no quick way to get the post caption]
2. Return default 100 hashtag posts(mode: hashtag) and all user's posts(mode: posts) if not specifying the number of post

   -n

   ,

   --number

   .
3. Print the result to the console if not specifying the output path of post

   -o

   ,

   --output

   .
4. It takes much longer to get data if the post number is over about 1000 since Instagram has set up the rate limit for data request.
5. Don't use this repo crawler Instagram if the user has more than 10000 posts.

The data format of

posts

The data format of

posts_full

Liker

Set up your username/password in

secret.py

Usage

positional arguments:
  tag

optional arguments:
  -n NUMBER, --number NUMBER (default 1000)
                        number of posts to like

Example

python liker.py foodie