downloading with curl using multiple connections

curl download file-transfer
user1030749 picture user1030749 · Dec 4, 2011 · Viewed 12.4k times · Source

I'm trying to tranfer a multi-gb file from one server to another; problem is RTT is 150ms+. Ive already tried using aria2 but its limited to 16 connections, lftp doesn't have any protection against stalled transfers.

I'm wondering if its possible to download one file with multiple connections using curl cli.

Answer

alijandro picture alijandro · Nov 25, 2016

It's possible. Fetch the total file size with -I option in curl.

Then you can fork many process in a shell, every curl connection with a different Content-Length header to download different part of the file.

After all the tasks finish, then merge all the download slices to a big file.

I have written a simple script and it's available here mcurl.sh, with -s option you can specify how many tasks you create to download the big file.

Related questions

How do I measure request and response times at once using cURL?

I have a web service that receives data in JSON format, processes the data, and then returns the result to the requester. I want to measure the request, response, and total time using cURL. My example request looks like: curl …

Read More
Download file using libcurl in C/C++

I am building an application (on windows using Dev-C++) and I want it to download a file. I am doing this using libcurl (I have already installed the source code using packman). I found a working example (http://siddhantahuja.wordpress.…

Read More
Downloading a large file using curl

I need to download remote file using curl. Here's the sample code I have: $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); $st = curl_exec($ch); $fd = fopen($tmp_name, 'w'); fwrite($fd, $st); …

Read More