How to get past the login page with Wget?

wget
Señor Reginold Francis picture Señor Reginold Francis · Aug 24, 2009 · Viewed 297k times · Source

I am trying to use Wget to download a page, but I cannot get past the login screen.

How do I send the username/password using post data on the login page and then download the actual page as an authenticated user?

Answer

jarnoan picture jarnoan · Sep 16, 2009

Based on the manual page:

# Log in to the server.  This only needs to be done once.
wget --save-cookies cookies.txt \
     --keep-session-cookies \
     --post-data 'user=foo&password=bar' \
     --delete-after \
     http://server.com/auth.php

# Now grab the page or pages we care about.
wget --load-cookies cookies.txt \
     http://server.com/interesting/article.php

Make sure the --post-data parameter is properly percent-encoded (especially ampersands!) or the request will probably fail. Also make sure that user and password are the correct keys; you can find out the correct keys by sleuthing the HTML of the login page (look into your browser’s “inspect element” feature and find the name attribute on the username and password fields).

Related questions

Using wget to recursively fetch a directory with arbitrary files in it

I have a web directory where I store some config files. I'd like to use wget to pull those files down and maintain their current structure. For instance, the remote directory looks like: http://mysite.com/configs/.vim/ .vim holds …

Read More
How to set proxy for wget?

I want to download something with wget using a proxy: HTTP Proxy: 127.0.0.1 Port: 8080 The proxy does not need username and password. How can I do this?

Read More
How to specify the location with wget?

I need files to be downloaded to /tmp/cron_test/. My wget code is wget --random-wait -r -p -nd -e robots=off -A".pdf" -U mozilla http://math.stanford.edu/undergrad/ So is there some parameter to specify the directory?

Read More