Can I use WGET to generate a sitemap of a website given its URL?

php wget web-crawler bots
Salman A picture Salman A · Oct 16, 2010 · Viewed 16.6k times · Source

I need a script that can spider a website and return the list of all crawled pages in plain-text or similar format; which I will submit to search engines as sitemap. Can I use WGET to generate a sitemap of a website? Or is there a PHP script that can do the same?

Answer

Salman A picture Salman A · Jul 19, 2011 
wget --spider --recursive --no-verbose --output-file=wgetlog.txt http://somewebsite.com
sed -n "s@.\+ URL:\([^ ]\+\) .\+@\1@p" wgetlog.txt | sed "s@&@\&@" > sedlog.txt

This creates a file called sedlog.txt that contains all links found on the specified website. You can use PHP or a shell script to convert the text file sitemap into an XML sitemap. Tweak the parameters of the wget command (accept/reject/include/exclude) to get only the links you need.

Related questions

How to use wget in php?

I have this parameters to download a XML file: wget --http-user=user --http-password=pass http://www.example.com/file.xml How I have to use that in php to open this xml file?

Read More
Using WGET to run a cronjob PHP

I tried to do a cron and run a url every 5 mintues. I tried to use WGET however I dont want to download the files on the server, all I want is just to run it. This is what I …

Read More
Cron job creating empty file each time it runs

I have a php script I want to run every minute to see if there are draft news posts that need to be posted. I was using "wget" for the cron command in cPanel, but i noticed (after a couple …

Read More