Docker's `docker0` device dies repeatedly (`inet addr` disappears)

Bosh picture Bosh · Mar 8, 2015 · Viewed 28.9k times · Source

I'm running Docker version 1.4.1, build 5bc2ff8 on Ubuntu 14.04. When I docker run any container, after a few minutes my docker0 bridge "dies", and the container stops being able to reach the network. Before the connection dies, running ifconfig reports a docker0 device with an inet addr like:

docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99  
          inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: xxxx::xxxx:xxxx:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          [... etc.]

But after the connection dies, ifconfig shows that the ipv4 address has gone away:

docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99 
          inet6 addr: xxxx::xxxx:xxxx:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8116 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15995 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:2444859 (2.4 MB)  TX bytes:17440729 (17.4 MB)

Restarting docker, e.g. with sudo service docker restart, brings the device back up -- but all my containers die and the problem starts over again. I can't reliably get anything to run for more than a few minutes at a time. Not long enough to even complete a docker build for most projects.

  1. What could be causing this?
  2. How can I diagnose it?
  3. What are some possible solutions?

Thanks!


Update: I can reliably trigger this docker0-dropping behavior simply by starting a container with docker run -t -i ubuntu /bin/bash, and then exiting with ctrl-d. When I do so, here's what I see in /var/log/syslog

myhost kernel:  docker0: port 1(veth80ddeaf) entered disabled state
myhost kernel:  device veth80ddeaf left promiscuous mode
myhost kernel:  docker0: port 1(veth80ddeaf) entered disabled state
'

myhost dhclient: Internet Systems Consortium DHCP Client 4.2.4
myhost dhclient: Copyright 2004-2012 Internet Systems Consortium.
myhost dhclient: All rights reserved.
myhost dhclient: For info, please visit https://www.isc.org/software/dhcp/
myhost dhclient: 
myhost dhclient: Listening on LPF/docker0/56:84:7a:fe:97:99
myhost dhclient: Sending on   LPF/docker0/56:84:7a:fe:97:99
myhost dhclient: Sending on   Socket/fallback
myhost kernel:  IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready

Update #2: The frequency of failure seems to depend on how long the container runs. For example:

docker run -i -t  ubuntu   sleep 0
--> `docker0` "survives" ~100% of the time

docker run -i -t  ubuntu   sleep 1
--> `docker0` survives ~80% of the time

docker run -i -t  ubuntu   sleep 5
--> `docker0` survives ~0% of the time

Answer

larsks picture larsks · Mar 9, 2015

How can I diagnose it?

When docker0 has an ip address, does it go away if you don't start any containers? If it persists indefinitely until you start a container, I would start by looking at the Docker logs as well as tailing the system logs when you start a container.

Does the ip address disappear at set intervals (e.g., every N minutes)? If so, I would look for logs from cron to see if some periodic task is responsible.

Are you running NetworkManager? Does disabling NetworkManager make the problem go away? I am running Docker on a system with NetworkManager without a problem, but I have no-auto-default=* set in my config, which may have an impact on this sort of thing.

Update

This is very suspicious:

myhost dhclient: Internet Systems Consortium DHCP Client 4.2.4
myhost dhclient: Copyright 2004-2012 Internet Systems Consortium.
myhost dhclient: All rights reserved.
myhost dhclient: For info, please visit https://www.isc.org/software/dhcp/
myhost dhclient: 
myhost dhclient: Listening on LPF/docker0/56:84:7a:fe:97:99
myhost dhclient: Sending on   LPF/docker0/56:84:7a:fe:97:99
myhost dhclient: Sending on   Socket/fallback

There should not be any dhclient process listening on docker0, and this is absolutely what is causing your ip address to disappear. If you are not explicitly running a dhcp client on this interface, this really suggests that NetworkManager is in fact trying to manage this interface. You said you disabled NetworkManager, but did you confirm that the process was stopped? What is the parent process of the dhclient that is listening on docker0? If you stop the dhclient process, does it get restarted? Does the problem go away?