tcpdump vs tcpflow (or "why isn't tcpdump ASCII packet data human readable?")
I have used both, and I conclude that I can read html data from webpages with tcpflow but cannot do so with tcpdump. The best I get is some ugly ASCII text with lots of period symbols.
My understanding is that tcpdump doesn't reassemble packets, whereas tcpflow does. But if that was the key difference, wouldn't the packet data from tcpdump still be human readable - just in smaller chunks? Is the problem that tcpdump is limited to ASCII and most network traffic is encoded in UTF-8?
I'm a rookie on network analysis/programming so forgive me if I'm missing something obvious.
Answer
To get that encrypted data one should use tcpdump with option tcpdump –A (capital a). It transfers text without any headers and is used mainly for web pages. Hence we get response page easily.
I think you are getting confused between an application layer and transport layer packet.
I do not know about tcpflow but tcpdump capture the whole packet (including header and all other stuff) not just the data.The html data which you are mentioning would be in the data part of a tcp/udp/icmp packet whichever you are using and so it needs you to understand the structure of tcp/udp/icmp packet as well ...
I capture this packet on my machine and HTML data is clearly visible , you need to write script to get it from the output with a knowledge of packet structure.
16:15:17.968325 IP 172.31.9.84.80 > 172.31.9.177.56559: P 2575928381:2575928808(427)
ack 65285447 win 1716 <nop,nop,timestamp 21031622 15165633>
0x0000: 4500 01df 66fd 4000 4006 66d8 ac1f 0954 E...f.@[email protected]
0x0010: ac1f 09b1 0050 dcef 9989 8c3d 03e4 2d47 ..... P.....=..-G
0x0020: 8018 06b4 1a2a 0000 0101 080a 0140 eac6 .....*.......@..
0x0030: 00e7 68c1 4854 5450 2f31 2e31 2032 3030 ..h.HTTP/1.1.200
0x0040: 204f 4b0d 0a44 6174 653a 2046 7269 2c20 .OK..Date:.Fri,.
0x0050: 3130 204a 756c 2032 3030 3920 3130 3a32 10.Jul.2009.10:2
0x0060: 303a 3136 2047 4d54 0d0a 5365 7276 6572 0:16.GMT..Server
0x0070: 3a20 4170 6163 6865 2f32 2e30 2e35 3220 :.Apache/2.0.52.
0x0080: 2852 6564 2048 6174 290d 0a4c 6173 742d (Red.Hat)..Last-
0x0090: 4d6f 6469 6669 6564 3a20 4672 692c 2031 Modified:.Fri,.1
0x00a0: 3020 4a75 6c20 3230 3039 2030 393a 3331 0.Jul.2009.09:31
0x00b0: 3a30 3420 474d 540d 0a45 5461 673a 2022 :04.GMT..ETag:."
0x00c0: 3266 6231 3333 2d37 372d 6131 3935 3436 2fb133-77-a19546
0x00d0: 3030 220d 0a41 6363 6570 742d 5261 6e67 00"..Accept-Rang
0x00e0: 6573 3a20 6279 7465 730d 0a43 6f6e 7465 es:.bytes..Conte
0x00f0: 6e74 2d4c 656e 6774 683a 2031 3139 0d0a nt-Length:.119..
0x0100: 4b65 6570 2d41 6c69 7665 3a20 7469 6d65 Keep-Alive:.time
0x0110: 6f75 743d 3135 2c20 6d61 783d 3130 300d out=15,.max=100.
0x0120: 0a43 6f6e 6e65 6374 696f 6e3a 204b 6565 .Connection:.Kee
0x0130: 702d 416c 6976 650d 0a43 6f6e 7465 6e74 p-Alive..Content
0x0140: 2d54 7970 653a 2074 6578 742f 6874 6d6c -Type:.text/html
0x0150: 3b20 6368 6172 7365 743d 4953 4f2d 3838 ;.charset=ISO-88
0x0160: 3539 2d31 0d0a 0d0a 3c68 746d 6c3e 0a0a 59-1....<html>..
0x0170: 3c62 6f64 793e 0a49 6e64 6961 6e20 696e <body>.Indian.in
0x0180: 7374 6974 7574 6520 6f66 2074 6563 686e stitute.of.techn
0x0190: 6f6c 6f67 7920 2e2e 2e2e 2e20 7468 6973 ology.......this
0x01a0: 2069 7320 7468 6520 7465 7374 2070 6167 .is.the.test.pag
0x01b0: 652e 0a3c 4120 6872 6566 3d22 6164 2e68 e..<A.href="ad.h
0x01c0: 746d 6c22 3e61 6263 643c 2f41 3e0a 0a3c tml">abcd</A>..<
0x01d0: 2f62 6f64 793e 0a3c 2f68 746d 6c3e 0a /body>.</html>.
The last 7-8 lines describe the html data.
use -s0 to capture whole frame and -X to print in above ASCII human readable format.
To get that encrypted data one should use TCPDUMP with option –A (capital a). It transfers text without any headers and is used mainly for web pages. Hence we get response page easily.
For eg:
I request index.html to 172.31.9.84 at port 80
Then I requested GET/index.html (an example page that contains only text “Indian institute of technology this is the test page”)
At this moment when I captured packets I got something as:
Request:
18:10:42.387426 IP 172.31.9.177.42943 > 172.31.9.84.80:
P 1:15(14) ack 1 win 46 <nop,nop,timestamp 6644101 353753635>
E..B.G@[email protected]^R.Mb.L.....ke.....
.ea....#GET /index.html
Response:
18:10:42.388127 IP 172.31.9.84.80 > 172.31.9.177.42943: P 1:53(52)
ack 15 win 1448 <nop,nop,timestamp 353758368 6644101>
E..h..@[email protected].. ..P..b.L.^R.[....d......
.....ea.<html>
<body>
Indian institute of technology this is the test page
</body>
</html>