Github url


by code4craft

code4craft /webmagic

A scalable web crawler framework for Java.

9.2K Stars 3.8K Forks Last release: about 3 years ago (WebMagic-0.7.3) Apache License 2.0 1.1K Commits 23 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:


Readme in Chinese

Build Status

A scalable crawler framework. It covers the whole lifecycle of crawler: downloading, url management, content extraction and persistent. It can simplify the development of a specific crawler.


  • Simple core with high flexibility.
  • Simple API for html extracting.
  • Annotation with POJO to customize a crawler, no configuration.
  • Multi-thread and Distribution support.
  • Easy to be integrated.


Add dependencies to your pom.xml:


WebMagic use slf4j with slf4j-log4j12 implementation. If you customized your slf4j implementation, please exclude slf4j-log4j12.


Get Started:

First crawler:

Write a class implements PageProcessor. For example, I wrote a crawler of github repository infomation.

public class GithubRepoPageProcessor implements PageProcessor { private Site site =; @Override public void process(Page page) { page.addTargetRequests(page.getHtml().links().regex("(https://github\\.com/\\w+/\\w+)").all()); page.putField("author", page.getUrl().regex("https://github\\.com/(\\w+)/.\*").toString()); page.putField("name", page.getHtml().xpath("//h1[@class='public']/strong/a/text()").toString()); if (page.getResultItems().get("name")==null){ //skip this page page.setSkip(true); } page.putField("readme", page.getHtml().xpath("//div[@id='readme']/tidyText()")); } @Override public Site getSite() { return site; } public static void main(String[] args) { Spider.create(new GithubRepoPageProcessor()).addUrl("").thread(5).run(); } }
  • page.addTargetRequests(links)

Add urls for crawling.

You can also use annotation way:

@TargetUrl("\\w+/\\w+") @HelpUrl("\\w+") public class GithubRepo { @ExtractBy(value = "//h1[@class='public']/strong/a/text()", notNull = true) private String name; @ExtractByUrl("https://github\\.com/(\\w+)/.\*") private String author; @ExtractBy("//div[@id='readme']/tidyText()") private String readme; public static void main(String[] args) { OOSpider.create( , new ConsolePageModelPipeline(), GithubRepo.class) .addUrl("").thread(5).run(); } }

Docs and samples:


The architecture of webmagic (refered to Scrapy)


There are more examples in




Lisenced under Apache 2.0 lisence


To write webmagic, I refered to the projects below :

  • Scrapy*

A crawler framework in Python.

  • Spiderman*

Another crawler framework in Java.


QQ Group: 373225642 542327088

Related Project

A web console based on WebMagic for Spider configuration and management.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.