wget or curl from stdin

unix curl wget stdin xargs
maximdim picture maximdim · Jan 22, 2012 · Viewed 12.2k times · Source

I'd like to download a web pages while supplying URLs from stdin. Essentially one process continuously produces URLs to stdout/file and I want to pipe them to wget or curl. (Think about it as simple web crawler if you want).

This seems to work fine:

tail 1.log | wget -i - -O - -q

But when I use 'tail -f' and it doesn't work anymore (buffering or wget is waiting for EOF?):

tail -f 1.log | wget -i - -O - -q

Could anybody provide a solution using wget, curl or any other standard Unix tool? Ideally I don't won't want to restart wget in the loop, just keep it running downloading URLs as they come.

Answer

Kyle Jones picture Kyle Jones · Jan 22, 2012

What you need to use is xargs. E.g.

tail -f 1.log | xargs -n1 wget -O - -q

Related questions

How do I get cURL to not show the progress bar?

I'm trying to use cURL in a script and get it to not show the progress bar. I've tried the -s, -silent, -S, and -quiet options, but none of them work. Here's a typical command I've tried: curl -s http://…

Read More
How do I pipe or redirect the output of curl -v?

For some reason the output always gets printed to the terminal, regardless of whether I redirect it via 2> or > or |. Is there a way to get around this? Why is this happening?

Read More
CURL to pass SSL certifcate and password

I need to specify a certificate with CURL i tried with --cert option it is not working. Could you please let me know to specify the keystore and passpharse while invoking with curl?

Read More