How To Extract Page URLs From Any Website in Bulk?

web-scraping screen-scraping scraper extractor
Junaid K. picture Junaid K. · Apr 26, 2014 · Viewed 9k times · Source

I'm looking for a Free solution/tool/software through which I can pull out all of the website's page URLs. Site has approx 992,000 pages so I need the URLs of all of them in excel sheet.

I'm using "site: mywebsite.com" and it gives me 992,000 results. I know I can make the max results per page 100 but that still doesn't make my life easier. Also google won't show any results over 1000. Tried to use the Google API but without any luck. Tried Sitemap Generators but they didn't work either.

Answer

seagulf picture seagulf · May 2, 2014

You can use a crawler tool to crawl the entire website and save the URLs visited. Free tools include:

Related questions

Web scraping with Python

I'd like to grab daily sunrise/sunset times from a web site. Is it possible to scrape web content with Python? what are the modules used? Is there any tutorial available?

Read More
What's the best way of scraping data from a website?

I need to extract contents from a website, but the application doesn’t provide any application programming interface or another mechanism to access that data programmatically. I found a useful third-party tool called Import.io that provides click and go …

Read More
Scrape web page contents

I am developing a project, for which I want to scrape the contents of a website in the background and get some limited content from that scraped website. For example, in my page I have "userid" and "password" fields, by …

Read More