Does solr do web crawling?

solr web-crawler
Murali Krishna Pinjala picture Murali Krishna Pinjala · Nov 23, 2009 · Viewed 27.8k times · Source

I am interested to do web crawling. I was looking at solr.

Does solr do web crawling, or what are the steps to do web crawling?

Answer

Jon picture Jon · Nov 23, 2009

Solr 5+ DOES in fact now do web crawling! http://lucene.apache.org/solr/

Older Solr versions do not do web crawling alone, as historically it's a search server that provides full text search capabilities. It builds on top of Lucene.

If you need to crawl web pages using another Solr project then you have a number of options including:

If you want to make use of the search facilities provided by Lucene or SOLR you'll need to build indexes from the web crawl results.

See this also:

Lucene crawler (it needs to build lucene index)

Related questions

Solr vs. ElasticSearch

What are the core architectural differences between these technologies? Also, what use cases are generally more appropriate for each?

Read More
ElasticSearch, Sphinx, Lucene, Solr, Xapian. Which fits for which usage?

I'm currently looking at other search methods rather than having a huge SQL query. I saw elasticsearch recently and played with whoosh (a Python implementation of a search engine). Can you give reasons for your choice(s)?

Read More
How to delete all data from solr and hbase

How do I delete all data from solr by command? We are using solr with lily and hbase. How can I delete data from both hbase and solr? http://lucene.apache.org/solr/4_10_0/tutorial.html#Deleting+Data

Read More