Docker Swarm Overlay Network Not Working Between Nodes

Question

i am trying to connect my docker services together in docker swarm.

the network is made of 2 raspberry pi's.

i can create an overlay network called test-overlay and i can see that services on either raspberry pi node can connect to the network.

my problem:

i cannot link to services between nodes with the overlay network.

given the following configuration of nodes and services, service1 can use the address http://service2 to connect to service2. but it does NOT work for http://service3. however http://service3 is accessible from service4.

node1:
  - service1
  - service2
node2:
  - service3
  - service4

i am new to docker swarm and any help is appreciated.

inspecting overlay

i have run the command sudo docker inspect network test-overlay on both nodes.

on the master node this returns the following:

[
    {
        "Name": "test-overlay",
        "Id": "skxhz8sb3f82dhh9jt9t3j5yl",
        "Created": "2018-04-15T20:31:20.629719732Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "3acb436a0cc9a4d584d537edb1546988d334afa4793cc4fae4dd6ac9b48828ea": {
                "Name": "docker-registry.1.la1myuodpkq0x5h39pqo6lt7f",
                "EndpointID": "66887fb1f5f253c6cbec149aa51ab85168903fdd2290719f26d2bcd8d6c68dc8",
                "MacAddress": "02:42:0a:00:00:04",
                "IPv4Address": "10.0.0.4/24",
                "IPv6Address": ""
            },
            "786e1fee538f81fe41ccd082800c646a0e191b0fd912e5c15530e61c248e81ac": {
                "Name": "portainer.1.qyvvlcdqo5sewuku3eiykaplz",
                "EndpointID": "0d29e5452c208ed637ae2e7dcec026f39d2431e8e0e20765a9e0e6d6dfdc60ca",
                "MacAddress": "02:42:0a:00:00:15",
                "IPv4Address": "10.0.0.21/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4101"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "d049fc8f8ae1",
                "IP": "192.168.1.2"
            },
            {
                "Name": "6c0da128f308",
                "IP": "192.168.1.3"
            }
        ]
    }
]

on the worker node this returns the following:

[
    {
        "Name": "test-overlay",
        "Id": "skxhz8sb3f82dhh9jt9t3j5yl",
        "Created": "2018-04-20T14:04:57.870696195Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "4cb50161119e4b58a472e1b5c380c301bbb00a23fc99fc2e0712a8c4bde6d9d4": {
                "Name": "minio.1.fo2su2quv8herbmnxqfi3g8w2",
                "EndpointID": "3e85786304ed08f02c09b8e1ed6a153a3b4c2ef7afe503a1b0ca6cf341521645",
                "MacAddress": "02:42:0a:00:00:d6",
                "IPv4Address": "10.0.0.214/24",
                "IPv6Address": ""
            },
            "ce99b3788a4f9438e276e0f52a8f4d29fa09179e3e93b31b14f45339ce3c5315": {
                "Name": "load-balancer.1.j64h1eecsc05b7d397ejvedv3",
                "EndpointID": "3b7e73d27fe30151f2dc2a0ba8a5afc7f74fd283159a03a592be10e297f58d51",
                "MacAddress": "02:42:0a:00:00:d0",
                "IPv4Address": "10.0.0.208/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4101"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "d049fc8f8ae1",
                "IP": "192.168.1.2"
            },
            {
                "Name": "6c0da128f308",
                "IP": "192.168.1.3"
            }
        ]
    }
]

Answer 1

it seems this problem was because of the nodes being not being able to connect to each other on the required ports.

TCP port 2377 for cluster management communications
TCP and UDP port 7946 for communication among nodes
UDP port 4789 for overlay network traffic

before you open those ports.

a better and simpler solution is to use the docker image portainer/agent. like the documentation says,

The Portainer Agent is a workaround for a Docker API limitation when using the Docker API to manage a Docker environment.

https://portainer.readthedocs.io/en/stable/agent.html

i hope this helps anyone else experiencing this problem.