"No route to host" when I do a telnet
I have two VM's in Azure with different public IP's:
10.10.1.9
10.10.1.6
When I do a telnet with following command from the server 10.10.1.6, I get the a error:
telnet 10.10.1.9 2181
Trying 10.10.1.9...
telnet: connect to address 10.10.1.9: No route to host
When I do a tcpdump in 10.10.1.9 side, I get the following log:
#tcpdump -i eth0 port 2181
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
07:55:35.530270 IP 10.10.1.6.55910 > 10.10.1.9.eforward: Flags [S], seq 1018543857, win 14600, options [mss 1418,sackOK,TS val 181360935 ecr 0,nop,wscale 7], length 0
Same time I also do a tcpdump on 10.10.1.6 side while i do a telnet from 10.10.1.6 to 10.10.1.9
tcpdump -i eth0 port 2181
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
07:55:57.970696 IP 10.10.1.6.55910 > 10.10.1.9.eforward: Flags [S], seq 1018543857, win 14600, options [mss 1460,sackOK,TS val 181360935 ecr 0,nop,wscale 7], length 0
**tcpdump on 10.10.1.9 with arp **
#tcpdump -i eth0 port 2181 or arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
08:00:18.356153 IP 10.10.1.6.55944 > 10.10.1.9.eforward: Flags [S], seq 3337054296, win 14600, options [mss 1418,sackOK,TS val 181643770 ecr 0,nop,wscale 7], length 0
08:00:42.294801 ARP, Request who-has 10.10.1.6 tell 10.10.1.9, length 28
08:00:42.295859 ARP, Reply 10.10.1.6 is-at 12:34:56:78:9a:bc (oui Unknown), length 28
tcpdump on 10.10.1.6
tcpdump -i eth0 port 2181 or arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
08:00:40.805565 IP 10.10.1.6.55944 > 10.10.1.9.eforward: Flags [S], seq 3337054296, win 14600, options [mss 1460,sackOK,TS val 181643770 ecr 0,nop,wscale 7], length 0
08:00:45.805204 ARP, Request who-has 10.10.1.9 tell 10.10.1.6, length 28
08:00:45.805721 ARP, Reply 10.10.1.9 is-at 12:34:56:78:9a:bc (oui Unknown), length 28
08:02:04.752283 ARP, Request who-has 10.10.1.9 tell 10.10.1.6, length 28
08:02:04.753141 ARP, Reply 10.10.1.9 is-at 12:34:56:78:9a:bc (oui Unknown), length 28
Sequence of run : First I ran tcpdumps on both 10.10.1.9 and 10.10.1.10 and then tried doing telnet from 10.10.1.10.
arp -a on 10.10.1.9
#arp -a
? (10.10.1.7) at 12:34:56:78:9a:bc [ether] on eth0
? (10.10.1.4) at 12:34:56:78:9a:bc [ether] on eth0
? (10.10.1.1) at 12:34:56:78:9a:bc [ether] on eth0
? (10.10.1.8) at 12:34:56:78:9a:bc [ether] on eth0
? (10.10.1.10) at <incomplete> on eth0
? (10.10.1.11) at 12:34:56:78:9a:bc [ether] on eth0
? (10.10.1.6) at 12:34:56:78:9a:bc [ether] on eth0
? (10.10.1.5) at 12:34:56:78:9a:bc [ether] on eth0
arp -a on 10.10.1.6
#arp -a
? (10.10.1.1) at 12:34:56:78:9a:bc [ether] on eth0
? (10.10.1.10) at <incomplete> on eth0
? (10.10.1.9) at 12:34:56:78:9a:bc [ether] on eth0
Thanks in advance.
Answer
The tcpdump in 10.10.1.9 says that it received a packet from 10.10.1.10. but could not reply back...as a result we get "No route to host" on 10.10.1.10 side.
You should get "No route to host" if there is, in fact, no route from 10.10.1.10 to 10.10.1.9, not just because a packet sent from 10.10.1.10 to 10.10.1.9 didn't get a reply. I.e., you should only get "No route to host" if 10.10.1.10 couldn't send a packet to 10.10.1.9 in the first place!
Now, perhaps the OS running on 10.10.1.10 is being stupid and returning EHOSTUNREACH ("No route to host") rather than, for example, ETIMEDOUT ("Operation timed out") if it never gets a SYN+ACK back from the initial SYN.
Or perhaps there was a route from 10.10.1.10 to 10.10.1.9 during the time the
23:46:30.003480 IP 10.10.1.10.42946 > 10.10.1.9.eforward: Flags [S], seq 2823099523, win 14600, options [mss 1418,sackOK,TS val 74982205 ecr 0,nop,wscale 7], length 0
packet was sent, but 10.10.1.9 wasn't able to, or decided not to, respond to that initial SYN with a SYN+ACK, and when 10.10.1.10 retransmitted the SYN, it was no longer able to send packets to 10.10.1.9, and reported "No route to host".
If this is reproducible, I would suggest running tcpdump on both hosts, to see more details as to what happened. I would suggest running a command such as
tcpdump -i eth0 port 2181 or arp
so that, for example, if the problem is that the ARP entry for the other host timed out on one of the hosts, and a subsequent attempt to re-ARP for the other host's MAC address failed, that will show up. (I'm assuming here that there's no router in between 10.10.1.10 and 10.10.1.9, so that "No route to host" really means "No ARP entry for host".)
(Another possibility is that there's some sort of "packet filter"/firewall in place on one or the other host, handling some ports differently from others, so that connecting to port 22 is possible but connecting to port 2181 isn't possible.)