downloading all the files in a directory with cURL

bash shell curl ftp sftp
Alex Gordon picture Alex Gordon · Aug 2, 2012 · Viewed 109k times · Source

I am using cURL to try to download all files in a certain directory.

here's what my list of files looks like:

enter image description here

I have tried to do in bash script: iiumlabs.[].csv.pgp and iiumlabs* and I guess curl is not big on wildcards.

curl -u login:pass ftp.myftpsite.com/iiumlabs* -O

question: how do i download this directory of files using cURL?

Answer

kkeller picture kkeller · Aug 2, 2012

If you're not bound to curl, you might want to use wget in recursive mode but restricting it to one level of recursion, try the following;

wget --no-verbose --no-parent --recursive --level=1\
--no-directories --user=login --password=pass ftp://ftp.myftpsite.com/
  • --no-parent : Do not ever ascend to the parent directory when retrieving recursively.
  • --level=depth : Specify recursion maximum depth level depth. The default maximum depth is five layers.
  • --no-directories : Do not create a hierarchy of directories when retrieving recursively.

Related questions

How to urlencode data for curl command?

I am trying to write a bash script for testing that takes a parameter and sends it through curl to web site. I need to url encode the value to make sure that special characters are processed properly. What is …

Read More
Save file to specific folder with curl command

In a shell script, I want to download a file from some URL and save it to a specific folder. What is the specific CLI flag I should use to download files to a specific folder with the curl command, …

Read More
Curl with multiline of JSON

Consider the curl command below, is it possible to allow newline in JSON (without the minify) and execute directly in bash (Mac/Ubuntu) curl -0 -v -X POST http://www.example.com/api/users \ -H "Expect:" \ -H 'Content-Type: text/json; …

Read More