Can't resolve hostnames between docker containers

Question

I have two containers created in separate compose files (done for application isolation -- each application may have multiple containers defined in the compose file such as a backing database).

These containers are linked via an external network named "common".

An example compose file would be:

version: '2'

services:

  rabbitmq:
    image: "rabbitmq:3-management"
    hostname: "rabbitmq"
    container_name: "rabbitmq"
    environment:
      RABBITMQ_ERLANG_COOKIE: "SWQOKODSQALRPCLNMEQG"
      RABBITMQ_DEFAULT_USER: "rabbitmq"
      RABBITMQ_DEFAULT_PASS: "rabbitmq"
      RABBITMQ_DEFAULT_VHOST: "/"
    ports:
      - "15672:15672"
      - "5672:5672"

networks:
  default:
    external:
      name: common

Docker versions:

root@server:~/# docker version
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Mon, 10 Oct 2016 21:38:17 +1300
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Mon, 10 Oct 2016 21:38:17 +1300
 OS/Arch:      linux/amd64
root@server:~/# docker-compose version
docker-compose version 1.8.1, build 878cff1
docker-py version: 1.10.3
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013

The common network was created using:

docker network create common

Then I bring containers up using:

docker-compose up -d

Inspecting the network I get:

root@server:~# docker network inspect b5e8f81a8ea0
[
    {
        "Name": "common",
        "Id": "b5e8f81a8ea063149298d2023be5740c8d971e0329a741abdafbac59fd882684",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.0.0/16"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "6137864e71161b417f0659b88b3e17538fb60277ca818e58b255d5cc17932c3c": {
                "Name": "db",
                "EndpointID": "5d9800dedcc22bdb8e08a1d04712df4e2f2846f5448e4ce888f164f7877ce5a4",
                "MacAddress": "02:42:c0:a8:00:03",
                "IPv4Address": "192.168.0.3/16",
                "IPv6Address": ""
            },
            "61b1288bf250196d02f3933c8bbde732fbcd24bdf3949c4f53b3b05aa87f3c7f": {
                "Name": "rabbitmq",
                "EndpointID": "484e3cc05e5a852150e6a7c429dc73d9ce4b097d4f53b07800c9998ef977565c",
                "MacAddress": "02:42:c0:a8:00:02",
                "IPv4Address": "192.168.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

Other containers have crashed because they can't resolve the 'rabbitmq' hostname. Inspecting the crashed containers shows they are on the same network:

root@server:~# docker inspect bae5214bf619
"NetworkSettings": {
    "Bridge": "",
    "SandboxID": "54a8b0833204522a75cf2e4195c5838b336ab82438aa53d9f1f38276eb9f6061",
    "HairpinMode": false,
    "LinkLocalIPv6Address": "",
    "LinkLocalIPv6PrefixLen": 0,
    "Ports": null,
    "SandboxKey": "/var/run/docker/netns/54a8b0833204",
    "SecondaryIPAddresses": null,
    "SecondaryIPv6Addresses": null,
    "EndpointID": "",
    "Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "IPAddress": "",
    "IPPrefixLen": 0,
    "IPv6Gateway": "",
    "MacAddress": "",
    "Networks": {
        "common": {
            "IPAMConfig": null,
            "Links": [
                "db"
            ],
            "Aliases": [
                "rulesengine",
                "bae5214bf619"
            ],
            "NetworkID": "b5e8f81a8ea063149298d2023be5740c8d971e0329a741abdafbac59fd882684",
            "EndpointID": "",
            "Gateway": "",
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "MacAddress": ""
        }
    }
}

If I login to a container that is not trying to access the rabbitmq container and hasn't crashed, the hostname does not resolve. But, the ip address of the rabbitmq container is reachable.

root@server:~/David.Deployments# docker exec -it 6137864e7116 bash
root@db:/# ping rabbitmq
ping: unknown host
root@db:/# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2): 56 data bytes
64 bytes from 192.168.0.2: icmp_seq=0 ttl=64 time=0.149 ms
64 bytes from 192.168.0.2: icmp_seq=1 ttl=64 time=0.043 ms

When signed into the rabbitmq container:

root@server:/# docker exec -it rabbitmq bash
root@rabbitmq:/# uname -n
rabbitmq
root@rabbitmq:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
445: eth0@if446: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:c0:a8:00:02 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.2/16 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:c0ff:fea8:2/64 scope link
       valid_lft forever preferred_lft forever
root@rabbitmq:/# cat /etc/hosts
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.0.2 rabbitmq

Are there any suggestions for how I can resolve this issue or things I should be checking?

Update 1

• I have tried adding external_links, but the problem persists and no entry is added to /etc/hosts for the external hostname 'rabbitmq'

Answer 1

Per the discussion here: https://github.com/docker/docker/issues/13381

poga's suggestion to restart docker worked for me: systemctl restart docker