get out links from nutch

web-crawler nutch
surajz picture surajz · Sep 15, 2011 · Viewed 7.3k times · Source

I am using nutch 1.3 to crawl a website. I want to get a list of urls crawled, and urls originating from a page.

I get list of urls crawled using readdb command.

bin/nutch readdb crawl/crawldb -dump file

Is there a way to find out urls that are on a page by reading crawldb or linkdb ?

in the org.apache.nutch.parse.html.HtmlParser I see outlinks array, I am wondering if there is a quick way to access it from command line.

Answer

surajz picture surajz · Sep 20, 2011

From command line, you can see the outlinks by using readseg with -dump or -get option. For example,

bin/nutch readseg -dump crawl/segments/20110919084424/ outputdir2 -nocontent -nofetch - nogenerate -noparse -noparsetext

less outputdir2/dump

Related questions

Web Cralwer Algorithm: depth?

I'm working on a crawler and need to understand exactly what is meant by "link depth". Take nutch for example: http://wiki.apache.org/nutch/NutchTutorial depth indicates the link depth from the root page that should be crawled. So, …

Read More
How is an aggregator built?

Let's say I want to aggregate information related to a specific niche from many sources (could be travel, technology, or whatever). How would I do that? Have a spider/crawler who will crawl the web for finding the information I …

Read More
An alternative web crawler to Nutch

I'm trying to build a specialised search engine web site that indexes a limited number of web sites. The solution I came up with is: using Nutch as the web crawler, using Solr as the search engine, the front-end and …

Read More