How do you archive an entire website for offline viewing?

html web-crawler archive
jskunkle picture jskunkle · Feb 11, 2009 · Viewed 67.3k times · Source

We actually have burned static/archived copies of our asp.net websites for customers many times. We have used WebZip until now but we have had endless problems with crashes, downloaded pages not being re-linked correctly, etc.

We basically need an application that crawls and downloads static copies of everything on our asp.net website (pages, images, documents, css, etc) and then processes the downloaded pages so that they can be browsed locally without an internet connection (get rid of absolute urls in links, etc). The more idiot proof the better. This seems like a pretty common and (relatively) simple process but I have tried a few other applications and have been really unimpressed

Does anyone have archive software they would recommend? Does anyone have a really simple process they would share?

Answer

chuckg picture chuckg · Feb 11, 2009

You could use wget:

wget -m -k -K -E http://url/of/web/site

Related questions

how to extract links and titles from a .html page?

for my website, i'd like to add a new functionality. I would like user to be able to upload his bookmarks backup file (from any browser if possible) so I can upload it to their profile and they don't have …

Read More
Hide Email Address from Bots - Keep mailto:

tl;dr Hide email address from bots without using scripts and maintain mailto: functionality. Method must also support screen-readers. Summary Email obfuscation without using scripts or contact forms Email address needs to be completely visible to human viewers and maintain …

Read More
Is it possible for Scrapy to get plain text from raw HTML data?

For example: scrapy shell http://scrapy.org/ content = hxs.select('//*[@id="content"]').extract()[0] print content Then, I get the following raw HTML code: <div id="content"> <h2>Welcome to Scrapy</h2> <…

Read More