Containers in a host "suddenly" loses connection to outside-world containers. However, some hosts were refreshed and suddenly we had the following situation:
Here's an example:
[root@pprdespap322 deploy]# ping ci.docker.company.net
PING pprdespap324.corp.company.net (10.137.55.22) 56(84) bytes of data.
64 bytes from pprdespap324.corp.company.net (10.137.55.22): icmp_seq=1 ttl=64 time=0.282 ms
64 bytes from pprdespap324.corp.company.net (10.137.55.22): icmp_seq=2 ttl=64 time=0.341 ms
^C
--- pprdespap324.corp.company.net ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.282/0.311/0.341/0.034 ms
Now, from the container itself, we cannot ping the same host:
[root@pprdespap322 deploy]# docker run -ti quay.io/coreos/registry ping ci.docker.company.net
WARNING: IPv4 forwarding is disabled. Networking will not work.
ping: unknown host ci.docker.company.net
The first time I saw this warning was in the initial versions of Docker... Having Docker 1.9.1 and 1.10.3, How to solve this problem?
I reviewed http://chrisgilmerproj.github.io/ubuntu/network/docker/2013/09/05/ipv4-forwarding-and-docker.html and it helped me solving the problem on the host.
I added the following to /etc/sysctl.conf:
net.ipv4.ip_forward=1
I then restarted the network service and validated the setting:
[root@pprdespap322 deploy]# systemctl restart network
[root@pprdespap322 deploy]# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
[root@pprdespap322 deploy]# docker run -ti quay.io/coreos/registry ping ci.docker.company.net
PING pprdespap324.corp.company.net (10.137.55.22) 56(84) bytes of data.
64 bytes from pprdespap324.corp.company.net (10.137.55.22): icmp_seq=1 ttl=63 time=0.329 ms
64 bytes from pprdespap324.corp.company.net (10.137.55.22): icmp_seq=2 ttl=63 time=0.306 ms
^C
--- pprdespap324.corp.company.net ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.306/0.317/0.329/0.021 ms
All containers now can communicate with outside world containers!