How to measure network performance (how to benchmark network protocol)
First, a bit of a background. There are many various comparisons of distributed version control systems (DVCS) which compare size of repository, or benchmark speed of operations. I haven't found any that would benchmark network performance of various DVCS, and various protocols used... beside measuring speed of operations (commands) involving network like 'clone', 'pull'/'fetch' or 'push'.
I'd like to know then how would you make such comparison; how to measure network performance of an application, or how to benchmark network protocol. I envision here among others also measuring dependence of performance on both bandwidth of network and latency (ping time) of network; some protocols sacrifice latency in the form of more round-trip exchanges (negotiation) to send minimal required final "pack".
I would prefer solutions involving only one computer, if possible. I'd like to see open source solutions, working on Linux. But I would also welcome more generic answers.
Preferred OS: Linux
Preferred languages: C, Perl, shell script
Possible measurements:
- total number of bytes transferred from server to client and from client to server in one session; this can also be used to measure overhead of protocol (bandwidth)
- number of round-trips (connections) in one transaction (latency)
- dependency of speed of network operation (time it takes to clone/pull/push) from network bandwidth, and from network latency (ping time)
How to make such measurements (such benchmarks)?
Added 02-06-2009:
A simplest benchmark (measurement) would be a network version of time command, i.e. command which run would give me number of bytes transferred, and number of round trips / network connections during execution of a given command.
Added 09-06-2009:
Example imaginary output for mentioned above solution of network version of time command could look like the following:
$ ntime git clone -q git://git.example.com/repo.git
...
bytes sent: nnn (nn kiB), bytes received: nnn (nn kiB), avg: nn.nn KB/s
nn reads, nn writes
Note that it is only an example output, detailing kind of information one might want to get.
Added 09-06-2009:
It looks like some of what I want can be acheived using dummynet, tool (originally) for testing networking protocols...
Answer
If I am understanding you correctly, you are basically interested in something like Linux 'strace' (Introduction) for network-specific system calls?
Possibly a combination of a profiler and a debugger, for network applications (i.e. 'ntrace'), providing a detailed analysis of various optional measurements?
Under Linux, the strace utility is largely based on functionality that is provided by the Linux kernel, namely the ptrace (process tracing) API:
- Process Tracing Using Ptrace
- Playing with ptrace - part I
- Playing with ptrace - part II
- System Call Tracing using ptrace
- Bytecode Injection into a Running Process using Ptrace()
- Better process control in Linux with an improved ptrace()
Using ptrace, it should be possible to obtain most of the data that you're interested in.
On Windows, you'll probably want to look into detours in order to intercept/redirect Winsock API calls for inspection/benchmarking purposes.
If you don't really need all that much low level information, you can probably also directly use strace (on linux) and only use it to trace certain system calls, for example consider the following line which would only trace calls to the open syscall (Using the additional -o FILE parameter, you can redirect all output to an output file):
strace -e trace=open -o results.log
By passing an additional -v flag to strace, you can increase its verbosity to get additional information (when working with SCMs like git that are composed of many smaller shell utilities and standalone tools, you'll probably also want to look into using the -f flag in order to also follow forked processes).
So, what you would be interested in, is all syscalls that are related to sockets, namely:
- accept
- bind
- connect
- getpeername
- getsockname
- getsockopt
- listen
- recv
- recvfrom
- send
- sendto
- setsockopt
- shutdown
- socket
- socketpair
(in the beginning, you'll probably only want to look into dealing with the send.../recv... calls, though)
To simplify this, you can also use "network" as parameter to trace, which will trace all network-related calls:
-e trace=network: Trace all the network related system calls.
So, a corresponding strace invocation could look like this:
strace -v -e trace=accept,bind,connect,getpeername,getsockname,getsockopt,listen,recv,recvfrom,send,sendto setsockopt,shutdown,socket,socketpair -o results.log -f git pull
When the program is finished running, you'll then mainly want to examine the log file to evaluate the data, this can then be easily achieved by using regular expressions.
For example, when running the following in a linux shell:
strace -v -o wget.log -e trace=connect,recv,recvfrom,send,sendto wget http://www.google.com
The resulting log file contains messages like these:
- recv(3, "HTTP/1.0 302 Found\r\nLocation: htt"..., 511, MSG_PEEK) = 511
- sendto(4, "\24\0\0\0\26\0\1\3^\206*J\0\0\0\0\0\0\0\0"..., 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
Looking at the man pages for these two system calls, it's obvious that 511 and respectively 20 are the number of bytes that are transferred. If you also need detailed timing information, you can pass the -T flag to strace:
-T -- print time spent in each syscall
In addition, you can get some statistics by passing the -c flag:
-c: Count time, calls, and errors for each system call and report a summary on program exit. On Linux, this attempts to show system time (CPU time spent running in the kernel) independent of wall clock time. If -c is used with -f or -F (below), only aggregate totals for all traced processes are kept.
If you also need to examine the actual data processed, you may want to look into the read/write specifiers:
-e read=set: Perform a full hexadecimal and ASCII dump of all the data read from file descriptors listed in the specified set. For example, to see all input activity on file descriptors 3 and 5 use -e read=3,5. Note that this is independent from the normal tracing of the read(2) system call which is controlled by the option -e trace=read. -e write=set: Perform a full hexadecimal and ASCII dump of all the data written to file descriptors listed in the specified set. For example, to see all output activity on file descriptors 3 and 5 use -e write=3,5. Note that this is independent from the normal tracing of the write(2) system call which is controlled by the option -e trace=write.
You can also customize the max length of strings:
-s strsize: Specify the maximum string size to print (the default is 32). Note that filenames are not considered strings and are always printed in full
Or have strings be dumped as hex:
-xx: Print all strings in hexadecimal string format.
So, using strace for much of this, seems like a good hybrid approach, because it is very easy to do, but still there's a good amount of low level information available, if you find that you need additional low level information, you may want to consider extending strace instead or filing corresponding feature requests with the strace project on sourceforge.
However, thinking some more about it, a less involved and more platform-agnostic way of implementing a fairly simple network traffic benchmark, would be to use some form of intermediate layer, in between the client and the actual server: a server that's basically metering, analyzing and redirecting the traffic to the real server.
Pretty much like a proxy server (e.g SOCKS), so that all traffic is tunneled through your analyzer, which can in turn accumulate statistics and other metrics.
A basic version of something like this could probably be easily put together just by using netcat and some shell scripts, more complex versions may however benefit from using perl or python instead.
For a python implementation of a SOCKS server, you may want to look into pysocks.
Also, there's of course twisted for python:
Twisted is an event-driven networking engine written in Python and licensed under the MIT license.
If you do need to have more low level information, you'll probably really want to look into intercepting system calls, though.
If you also need protocol-specific efficiency data, you might want to look into tcpdump.