What is the fastest way to transfer files over a network (FTP, HTTP, RSync, etc.)

Question

I'm trying to figure out the best way to transfer large amounts of data over a network between two systems. I am currently looking into either FTP, HTTP, or RSync, and I am wondering which one is the fastest. I've looked online for some answers and found the following sites:

• http://daniel.haxx.se/docs/ftp-vs-http.html
• http://www.isi.edu/lsam/publications/http-perf/

The problem is that these are old, and talk more about the theoretical differences between how the protocols communicate. I am more interested with actual benchmarks, that can say that for a specific setup, when transferring files of varying sizes one protocol is x% faster then the others.

Has anyone test these and posted the results somewhere?

Answer 1

Alright, so I setup the following test:

• Hardware: 2 desktops Intel Core Duo CPU @ 2.33GHz, with 4G of RAM.
• OS: Ubuntu 11.10 on both machines
• Network: 100Mb dedicated switch, both machines are connect to it.
• Software:
  • Python HTTP server (inspired by this).
  • Python FTP server (inspired by this).
  • Python HTTP client (inspired by this).
  • Python FTP client (inspired by this).

I uploaded the following groups of files to each server:

1. 1 100M file.
2. 10 10M files.
3. 100 1M files.
4. 1,000 100K files.
5. 10,000 10K files.

I got the following average results over multiple runs (numbers in seconds):

|-----------+---------+----------|
| File Size | FTP (s) | HTTP (s) |
|-----------+---------+----------|
|      100M |       8 |        9 |
|       10M |       8 |        9 |
|        1M |       8 |        9 |
|      100K |      14 |       12 |
|       10K |      46 |       41 |
|-----------+---------+----------| 

So, it seems that FTP is slightly faster in large files, and HTTP is a little faster in many small files. All in all, I think that they are comparable, and the server implementation is much more important then the protocol.