Web Scraping with Scala

scala web-scraping libraries
Michael Tingley picture Michael Tingley · Feb 7, 2013 · Viewed 16.8k times · Source

Just wondering if anyone knows of a web-scraping library that takes advantage of Scala's succinct syntax. So far, I've found Chafe, but this seems poorly-documented and maintained. I'm wondering if anyone out there has done scraping with Scala and has advice. (I'm trying to integrate into an existing Scala framework rather than use a scraper written in, say, Python.)

Answer

Adam Gent picture Adam Gent · Feb 7, 2013

First there is a plethora of HTML scraping libs in JVM all you need to do is pimp one of them (pimp my library pattern).

The four I have used are:

  • HtmlUnit - Will emulate the browser and even run Javascript
  • Jericho - Preserves formatting and ideal if you want to edit the scraped HTML
  • NekoHtml
  • JSoup -- does not work with Scala. Might work

I have used Selenium but never for scraping. Scala has a wrapper around selenium.

I would recommend pimping an existing Java library over some half baked Scala lib.

Related questions

How to use java.String.format in Scala?

I am trying to use a .format method of a string. But if I place %1, %2, etc. in the string, java.util.UnknownFormatConversionException is thrown pointing to a confusing Java source code piece: private void checkText(String s) { int idx; // If …

Read More
How can I change column types in Spark SQL's DataFrame?

Suppose I'm doing something like: val df = sqlContext.load("com.databricks.spark.csv", Map("path" -> "cars.csv", "header" -> "true")) df.printSchema() root |-- year: string (nullable = true) |-- make: string (nullable = true) |-- model: string (nullable = …

Read More
Spark - load CSV file as DataFrame?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with df.registerTempTable("table_name") I have tried: scala> val df = sqlContext.load("hdfs:///csv/file/dir/file.csv") Error …

Read More