awesome_crawl

by zhangslob

zhangslob / awesome_crawl

腾讯新闻、知乎话题、微博粉丝,Tumblr爬虫、斗鱼弹幕、妹子图爬虫、分布式设计等

231 Stars 96 Forks Last release: Not found 47 Commits 0 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

awesome_crawl Awesome


| 爬虫 | 说明 | 技术栈 | | ------------------------- | -------------- | -------------- | | 腾讯新闻 | 采集所有腾讯新闻的链接和新闻详情 | scrapy,mongo,redis | | 知乎话题 | 从话题广场出发,先采集子话题ID,再采集ID下所有问题 | scrapy,mongo,redis | | 微博粉丝 | 采集大V的所有粉丝 | scrapy,mongo,redis | | Tumblr爬虫 | 下载指定Tumblr博主的资源 | requests,concurrent | | 妹子图爬虫 | go语言下载妹子图 | goroutine,goquery | | 全站爬虫 | 分布式全站爬虫——以"搜狗电视剧"为例 | scrapy,redis,分布式 |

如果能帮助你,那就最好了。欢迎关注公众号:

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.