I am using LWP to download content from web pages, and I would like to limit the amount of time it waits for a page. This is accomplished in lwp like this:
my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->get($url);
And this works fine, except for whenever the timeout reaches its limit, it just dies and I can't continue on with the script! I'd really like to handle this timeout properly so that I can record that the url had a timeout and then move on to my next one. Does anyone know how to do this? Thanks!
LWP::Agent's get()
returns a HTTP::Response object that you can use for checking errors:
use LWP::Agent;
use HTTP::Status ();
my $ua = LWP::UserAgent->new;
$ua->timeout(10);
my $response = $ua->get($url);
if ($response->is_error) {
printf "[%d] %s\n", $response->code, $response->message;
# record the timeout
if ($response->code == HTTP::Status::HTTP_REQUEST_TIMEOUT) {
...
}
}
Btw, the better practice nowadays is to use Try::Tiny instead of eval {...}
. It gives you try {...} catch {...}
. and it resolves some problems with checking if $@
(see the background section in the Try::Tiny
documentation).