Using wget to recursively fetch a directory with arbitrary files in it

shell wget
jerodsanto picture jerodsanto · Nov 7, 2008 · Viewed 754.6k times · Source

I have a web directory where I store some config files. I'd like to use wget to pull those files down and maintain their current structure. For instance, the remote directory looks like:

http://mysite.com/configs/.vim/

.vim holds multiple files and directories. I want to replicate that on the client using wget. Can't seem to find the right combo of wget flags to get this done. Any ideas?

Answer

Jeremy Ruten picture Jeremy Ruten · Nov 7, 2008

You have to pass the -np/--no-parent option to wget (in addition to -r/--recursive, of course), otherwise it will follow the link in the directory index on my site to the parent directory. So the command would look like this:

wget --recursive --no-parent http://example.com/configs/.vim/

To avoid downloading the auto-generated index.html files, use the -R/--reject option:

wget -r -np -R "index.html*" http://example.com/configs/.vim/

Related questions

How to get the contents of a webpage in a shell variable?

In Linux how can I fetch an URL and get its contents in a variable in shell script?

Read More
Why wget ignores query string in the url?

I want to use wget to download the following 18 html files: http://www.ted.com/talks/quick-list?sort=date&order=desc&page=18 http://www.ted.com/talks/quick-list?sort=date&order=desc&page=17 ... http://www.ted.…

Read More
Emulating a browser to download a file?

There's an FLV file on the web that can be downloaded directly in Chrome. The file is a television program, published by CCTV (China Central Television). CCTV is a non-profit, state-owned broadcaster, financed by the Chinese tax payer, which allows …

Read More