Java Web Crawler Libraries

java web-crawler
CodeKingPlusPlus picture CodeKingPlusPlus · Jul 1, 2012 · Viewed 38.1k times · Source

I wanted to make a Java based web crawler for an experiment. I heard that making a Web Crawler in Java was the way to go if this is your first time. However, I have two important questions.

  1. How will my program 'visit' or 'connect' to web pages? Please give a brief explanation. (I understand the basics of the layers of abstraction from the hardware up to the software, here I am interested in the Java abstractions)

  2. What libraries should I use? I would assume I need a library for connecting to web pages, a library for HTTP/HTTPS protocol, and a library for HTML parsing.

Answer

cuneytykaya picture cuneytykaya · Nov 18, 2012

Crawler4j is the best solution for you,

Crawler4j is an open source Java crawler which provides a simple interface for crawling the Web. You can setup a multi-threaded web crawler in 5 minutes!

Also visit. for more java based web crawler tools and brief explanation for each.

Related questions

How to get a web page's source code from Java

I just want to retrieve any web page's source code from Java. I found lots of solutions so far, but I couldn't find any code that works for all the links below: http://www.cumhuriyet.com.tr?hn=298710 http://www.…

Read More
Syntax error, insert "... VariableDeclaratorId" to complete FormalParameterList

I am facing some issues with this code: import edu.uci.ics.crawler4j.crawler.CrawlConfig; import edu.uci.ics.crawler4j.crawler.CrawlController; import edu.uci.ics.crawler4j.fetcher.PageFetcher; import edu.uci.ics.crawler4j.robotstxt.RobotstxtConfig; …

Read More
Selenium wait for Ajax content to load - universal approach

Is there a universal approach for Selenium to wait till all ajax content has loaded? (not tied to a specific website - so it works for every ajax website)

Read More