Java HTML Parsing

java html parsing web-scraping
Richard Walton picture Richard Walton · Oct 26, 2008 · Viewed 109k times · Source

I'm working on an app which scrapes data from a website and I was wondering how I should go about getting the data. Specifically I need data contained in a number of div tags which use a specific CSS class - Currently (for testing purposes) I'm just checking for 

div class = "classname"

in each line of HTML - This works, but I can't help but feel there is a better solution out there.

Is there any nice way where I could give a class a line of HTML and have some nice methods like:

boolean usesClass(String CSSClassname);
String getText();
String getLink();

Answer

rajsite picture rajsite · May 18, 2011

Another library that might be useful for HTML processing is jsoup. Jsoup tries to clean malformed HTML and allows html parsing in Java using jQuery like tag selector syntax.

http://jsoup.org/

Related questions

Which HTML Parser is the best?

I code a lot of parsers. Up until now, I was using HtmlUnit headless browser for parsing and browser automation. Now, I want to separate both the tasks. As 80% of my work involves just parsing, I want to use a …

Read More
Remove HTML tags from a String

Is there a good way to remove HTML from a Java string? A simple regex like replaceAll("\\<.*?>", "") will work, but things like &amp; wont be converted correctly and non-HTML between the two angle brackets will be removed (…

Read More
Parse JSON from HttpURLConnection object

I am doing basic http auth with the HttpURLConnection object in Java. URL urlUse = new URL(url); HttpURLConnection conn = null; conn = (HttpURLConnection) urlUse.openConnection(); conn.setRequestMethod("GET"); conn.setRequestProperty("Content-length", "0"); conn.setUseCaches(false); conn.setAllowUserInteraction(false); conn.setConnectTimeout(timeout); conn.…

Read More