How to make Docker container see real user IP?

docker networking docker-networking jwilder-nginx-proxy firewalld
ololoepepe picture ololoepepe · Nov 28, 2017 · Viewed 8.7k times · Source

The problem

Inside nginx-proxy Docker container (more info below), I always see the same IP adress for every connection: 172.18.0.1 (which is the nginx-proxy network gateway). For example:

nginx.1    | www.my-site.tld 172.18.0.1 - - [28/Nov/2017:17:22:21 +0000] "GET /some/path HTTP/2.0" 200 46576 "https://www.my-site.tld/some/path" "Mozilla/5.0 (Linux; Android 4.4.2; PSP5507DUO Build/KVT49L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 YaBrowser/17.10.0.446.00 Mobile Safari/537.36"

No matter if I make requests from the external network, of from the local machine, or even from the server (host) itself, the IP address is always 172.18.0.1.

What I need is to see the real client IP.

Attempts to fix

I have googled a lot, and tried different solutions, but none worked for me. I tried the following:

  • disabling firewalld (nothing works at all in that case, of course, iptables are empty);
  • using --userland-proxy=false (no luck);
  • using --net=host (not a solution at all, and does not work, too).

Related links:

So, any ideas? Seems to be either iptables routing issues or internal Docker bug. Anyway, it is for sure related to Docker, because the following displays correct remote user IP:

[root@server]# nc -lv 12345

[user@remote-client]$ nc -vz MY.IP.ADDRESS.HERE 12345

Very weird!

Now I'm going to provide full information about the system, so please be patient. :)

Basic

I have Centos 7.4 and Docker installed:

# cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)

# uname -r
3.10.0-693.5.2.el7.x86_64

# docker version
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:41:23 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:49 2017
 OS/Arch:      linux/amd64
 Experimental: false

firewalld is used as a firewall.

Network configuration

Interfaces

There are three physical interfaces:

# ifconfig | grep -A 7 enp
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::1ad6:c7ff:fe00:fc98  prefixlen 64  scopeid 0x20<link>
        ether 18:d6:c7:00:fc:98  txqueuelen 1000  (Ethernet)
        RX packets 11530516  bytes 2220554890 (2.0 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 24237462  bytes 31702254967 (29.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::1ad6:c7ff:fe06:ef36  prefixlen 64  scopeid 0x20<link>
        ether 18:d6:c7:06:ef:36  txqueuelen 1000  (Ethernet)
        RX packets 948513  bytes 143294797 (136.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2371584  bytes 3207775040 (2.9 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 31.172.141.111  netmask 255.255.255.128  broadcast MY.EXTERNAL.IP.HERE
        inet6 fe80::127b:44ff:fe46:c2d1  prefixlen 64  scopeid 0x20<link>
        ether 10:7b:44:46:c2:d1  txqueuelen 1000  (Ethernet)
        RX packets 46772225  bytes 57416420859 (53.4 GiB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 28814037  bytes 10943786995 (10.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

And a virtual bridge (enp1s0 and enp2s0):

# ifconfig | grep -A 7 virbr
virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.1  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::1ad6:c7ff:fe00:fc98  prefixlen 64  scopeid 0x20<link>
        ether 18:d6:c7:00:fc:98  txqueuelen 1000  (Ethernet)
        RX packets 11542053  bytes 2042945874 (1.9 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 26586017  bytes 34919742970 (32.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Firewall zones

The enp3s0 interface is connected to the Internet (--zone=public), and virbr0 is connected to the internal local network (--zone=internal).

Forwarding and masquerading

IPv4 forwarding is enabled:

# cat /proc/sys/net/ipv4/ip_forward
1

and masquerading on public zone is configured with firewalld:

# firewall-cmd --zone=public --list-all
success

Docker configuration

Docker has default configuration.

Networking

I only created a network:

# docker network create -d bridge nginx-proxy

Here is it's configuration:

# docker network inspect nginx-proxy
[
    {
        "Name": "nginx-proxy",
        "Id": "192821446c9a5891fd1a7e240533cefb81ba0548033acb605eea805abec83505",
        "Created": "2017-11-24T14:25:10.687199435+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "1c563c33a45c9feabee4cb5ec3194464c66bfad1e233dedb797e269649067159": {
                "Name": "nginxproxy_nginx-proxy_1",
                "EndpointID": "683ceaced8eb416b485775734ca6828e9eced5ea6e3c7a76960c4eb807b521d9",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

Images and containers

I'm using jwilder/nginx-proxy image. Here is my docker-compose.yml:

version: '3'
services:
  nginx-proxy:
    image: jwilder/nginx-proxy:alpine
    labels:
      - 'com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy=true'
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
    ports:
      - '80:80'
      - '443:443'
    networks:
      nginx-proxy:
networks:
  nginx-proxy:
depends_on:
  - nginx-proxy

networks:
  nginx-proxy:
    external:
      name: nginx-proxy

As you can see, nginx-proxy container is listening on ports 80 and 443 at all interfaces.

# docker ps | grep nginx-proxy
1c563c33a45c        jwilder/nginx-proxy:alpine                        "/app/docker-entry..."   8 hours ago         Up 8 hours          0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   nginxproxy_nginx-proxy_1

iptables

Container-related rules

Here are iptables rules related to the container:

# iptables-save | grep -e br-192821446c9a -e 172.18.0 -e '\*'
*mangle
*security
*raw
*filter
-A FORWARD -o br-192821446c9a -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-192821446c9a -j DOCKER
-A FORWARD -i br-192821446c9a ! -o br-192821446c9a -j ACCEPT
-A FORWARD -i br-192821446c9a -o br-192821446c9a -j ACCEPT
-A DOCKER -d 172.18.0.2/32 ! -i br-192821446c9a -o br-192821446c9a -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 172.18.0.2/32 ! -i br-192821446c9a -o br-192821446c9a -p tcp -m tcp --dport 80 -j ACCEPT
*nat
-A POSTROUTING -s 172.18.0.0/16 ! -o br-192821446c9a -j MASQUERADE
-A POSTROUTING -s 172.18.0.2/32 -d 172.18.0.2/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 172.18.0.2/32 -d 172.18.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A DOCKER -i br-192821446c9a -j RETURN
-A DOCKER ! -i br-192821446c9a -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.18.0.2:443
-A DOCKER ! -i br-192821446c9a -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.18.0.2:80

Complete iptables rules list

And here is the complete output of iptables-save:

# iptables-save
# Generated by iptables-save v1.4.21 on Tue Nov 28 19:16:31 2017
*mangle
:PREROUTING ACCEPT [9063259:7480026035]
:INPUT ACCEPT [250719:30179859]
:FORWARD ACCEPT [8795875:7445096680]
:OUTPUT ACCEPT [1078071:1430559812]
:POSTROUTING ACCEPT [9842972:8867742437]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_internal - [0:0]
:PRE_internal_allow - [0:0]
:PRE_internal_deny - [0:0]
:PRE_internal_log - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -j POSTROUTING_direct
-A PREROUTING_ZONES -i enp3s0 -g PRE_public
-A PREROUTING_ZONES -i virbr0 -g PRE_internal
-A PREROUTING_ZONES -g PRE_public
-A PRE_internal -j PRE_internal_log
-A PRE_internal -j PRE_internal_deny
-A PRE_internal -j PRE_internal_allow
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Tue Nov 28 19:16:31 2017
# Generated by iptables-save v1.4.21 on Tue Nov 28 19:16:31 2017
*security
:INPUT ACCEPT [207361:25027156]
:FORWARD ACCEPT [8764800:7437165960]
:OUTPUT ACCEPT [1078077:1430561824]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
# Completed on Tue Nov 28 19:16:31 2017
# Generated by iptables-save v1.4.21 on Tue Nov 28 19:16:31 2017
*raw
:PREROUTING ACCEPT [9063267:7480026451]
:OUTPUT ACCEPT [1078080:1430562828]
:OUTPUT_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_internal - [0:0]
:PRE_internal_allow - [0:0]
:PRE_internal_deny - [0:0]
:PRE_internal_log - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A PREROUTING_ZONES -i enp3s0 -g PRE_public
-A PREROUTING_ZONES -i virbr0 -g PRE_internal
-A PREROUTING_ZONES -g PRE_public
-A PRE_internal -j PRE_internal_log
-A PRE_internal -j PRE_internal_deny
-A PRE_internal -j PRE_internal_allow
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Tue Nov 28 19:16:31 2017
# Generated by iptables-save v1.4.21 on Tue Nov 28 19:16:31 2017
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1078082:1430564354]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:FORWARD_IN_ZONES - [0:0]
:FORWARD_IN_ZONES_SOURCE - [0:0]
:FORWARD_OUT_ZONES - [0:0]
:FORWARD_OUT_ZONES_SOURCE - [0:0]
:FORWARD_direct - [0:0]
:FWDI_internal - [0:0]
:FWDI_internal_allow - [0:0]
:FWDI_internal_deny - [0:0]
:FWDI_internal_log - [0:0]
:FWDI_public - [0:0]
:FWDI_public_allow - [0:0]
:FWDI_public_deny - [0:0]
:FWDI_public_log - [0:0]
:FWDO_internal - [0:0]
:FWDO_internal_allow - [0:0]
:FWDO_internal_deny - [0:0]
:FWDO_internal_log - [0:0]
:FWDO_public - [0:0]
:FWDO_public_allow - [0:0]
:FWDO_public_deny - [0:0]
:FWDO_public_log - [0:0]
:INPUT_ZONES - [0:0]
:INPUT_ZONES_SOURCE - [0:0]
:INPUT_direct - [0:0]
:IN_internal - [0:0]
:IN_internal_allow - [0:0]
:IN_internal_deny - [0:0]
:IN_internal_log - [0:0]
:IN_public - [0:0]
:IN_public_allow - [0:0]
:IN_public_deny - [0:0]
:IN_public_log - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o br-828c38e3582b -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-828c38e3582b -j DOCKER
-A FORWARD -i br-828c38e3582b ! -o br-828c38e3582b -j ACCEPT
-A FORWARD -i br-828c38e3582b -o br-828c38e3582b -j ACCEPT
-A FORWARD -o br-7cbbdaf3d8fe -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-7cbbdaf3d8fe -j DOCKER
-A FORWARD -i br-7cbbdaf3d8fe ! -o br-7cbbdaf3d8fe -j ACCEPT
-A FORWARD -i br-7cbbdaf3d8fe -o br-7cbbdaf3d8fe -j ACCEPT
-A FORWARD -o br-92f8d769de0b -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-92f8d769de0b -j DOCKER
-A FORWARD -i br-92f8d769de0b ! -o br-92f8d769de0b -j ACCEPT
-A FORWARD -i br-92f8d769de0b -o br-92f8d769de0b -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-6a38b645b1c7 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-6a38b645b1c7 -j DOCKER
-A FORWARD -i br-6a38b645b1c7 ! -o br-6a38b645b1c7 -j ACCEPT
-A FORWARD -i br-6a38b645b1c7 -o br-6a38b645b1c7 -j ACCEPT
-A FORWARD -o br-2d90be830c58 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-2d90be830c58 -j DOCKER
-A FORWARD -i br-2d90be830c58 ! -o br-2d90be830c58 -j ACCEPT
-A FORWARD -i br-2d90be830c58 -o br-2d90be830c58 -j ACCEPT
-A FORWARD -o br-192821446c9a -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-192821446c9a -j DOCKER
-A FORWARD -i br-192821446c9a ! -o br-192821446c9a -j ACCEPT
-A FORWARD -i br-192821446c9a -o br-192821446c9a -j ACCEPT
-A FORWARD -o br-b61364bf1724 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-b61364bf1724 -j DOCKER
-A FORWARD -i br-b61364bf1724 ! -o br-b61364bf1724 -j ACCEPT
-A FORWARD -i br-b61364bf1724 -o br-b61364bf1724 -j ACCEPT
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -j OUTPUT_direct
-A DOCKER -d 172.18.0.2/32 ! -i br-192821446c9a -o br-192821446c9a -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 172.18.0.2/32 ! -i br-192821446c9a -o br-192821446c9a -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.20.0.2/32 ! -i br-2d90be830c58 -o br-2d90be830c58 -p tcp -m tcp --dport 1194 -j ACCEPT
-A DOCKER-ISOLATION -j RETURN
-A FORWARD_IN_ZONES -i enp3s0 -g FWDI_public
-A FORWARD_IN_ZONES -i virbr0 -g FWDI_internal
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o enp3s0 -g FWDO_public
-A FORWARD_OUT_ZONES -o virbr0 -g FWDO_internal
-A FORWARD_OUT_ZONES -g FWDO_public
-A FWDI_internal -j FWDI_internal_log
-A FWDI_internal -j FWDI_internal_deny
-A FWDI_internal -j FWDI_internal_allow
-A FWDI_internal -p icmp -j ACCEPT
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDO_internal -j FWDO_internal_log
-A FWDO_internal -j FWDO_internal_deny
-A FWDO_internal -j FWDO_internal_allow
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A FWDO_public_allow -m conntrack --ctstate NEW -j ACCEPT
-A INPUT_ZONES -i enp3s0 -g IN_public
-A INPUT_ZONES -i virbr0 -g IN_internal
-A INPUT_ZONES -g IN_public
-A IN_internal -j IN_internal_log
-A IN_internal -j IN_internal_deny
-A IN_internal -j IN_internal_allow
-A IN_internal -p icmp -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -d 224.0.0.251/32 -p udp -m udp --dport 5353 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p udp -m udp --dport 137 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p udp -m udp --dport 138 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 2049 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 20048 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p udp -m udp --dport 20048 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 111 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p udp -m udp --dport 111 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 139 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 445 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 58846 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 8112 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 53 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p udp -m udp --dport 53 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p udp -m udp --dport 67 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 80 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 443 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p udp -m udp --dport 1194 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 1195 -m conntrack --ctstate NEW -j ACCEPT
COMMIT
# Completed on Tue Nov 28 19:16:31 2017
# Generated by iptables-save v1.4.21 on Tue Nov 28 19:16:31 2017
*nat
:PREROUTING ACCEPT [118631:13941821]
:INPUT ACCEPT [6147:464301]
:OUTPUT ACCEPT [18836:1989867]
:POSTROUTING ACCEPT [2593:197858]
:DOCKER - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_ZONES - [0:0]
:POSTROUTING_ZONES_SOURCE - [0:0]
:POSTROUTING_direct - [0:0]
:POST_internal - [0:0]
:POST_internal_allow - [0:0]
:POST_internal_deny - [0:0]
:POST_internal_log - [0:0]
:POST_public - [0:0]
:POST_public_allow - [0:0]
:POST_public_deny - [0:0]
:POST_public_log - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_internal - [0:0]
:PRE_internal_allow - [0:0]
:PRE_internal_deny - [0:0]
:PRE_internal_log - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -j OUTPUT_direct
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.101.0.0/24 ! -o br-828c38e3582b -j MASQUERADE
-A POSTROUTING -s 172.102.0.0/24 ! -o br-7cbbdaf3d8fe -j MASQUERADE
-A POSTROUTING -s 172.101.0.0/24 ! -o br-92f8d769de0b -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.102.0.0/24 ! -o br-6a38b645b1c7 -j MASQUERADE
-A POSTROUTING -s 172.20.0.0/16 ! -o br-2d90be830c58 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o br-192821446c9a -j MASQUERADE
-A POSTROUTING -s 172.19.0.0/16 ! -o br-b61364bf1724 -j MASQUERADE
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -j POSTROUTING_ZONES_SOURCE
-A POSTROUTING -j POSTROUTING_ZONES
-A POSTROUTING -s 172.18.0.2/32 -d 172.18.0.2/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 172.18.0.2/32 -d 172.18.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.20.0.2/32 -d 172.20.0.2/32 -p tcp -m tcp --dport 1194 -j MASQUERADE
-A DOCKER -i br-828c38e3582b -j RETURN
-A DOCKER -i br-7cbbdaf3d8fe -j RETURN
-A DOCKER -i br-92f8d769de0b -j RETURN
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-6a38b645b1c7 -j RETURN
-A DOCKER -i br-2d90be830c58 -j RETURN
-A DOCKER -i br-192821446c9a -j RETURN
-A DOCKER -i br-b61364bf1724 -j RETURN
-A DOCKER ! -i br-192821446c9a -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.18.0.2:443
-A DOCKER ! -i br-192821446c9a -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.18.0.2:80
-A DOCKER ! -i br-2d90be830c58 -p tcp -m tcp --dport 1195 -j DNAT --to-destination 172.20.0.2:1194
-A POSTROUTING_ZONES -o enp3s0 -g POST_public
-A POSTROUTING_ZONES -o virbr0 -g POST_internal
-A POSTROUTING_ZONES -g POST_public
-A POST_internal -j POST_internal_log
-A POST_internal -j POST_internal_deny
-A POST_internal -j POST_internal_allow
-A POST_public -j POST_public_log
-A POST_public -j POST_public_deny
-A POST_public -j POST_public_allow
-A POST_public_allow ! -o lo -j MASQUERADE
-A PREROUTING_ZONES -i enp3s0 -g PRE_public
-A PREROUTING_ZONES -i virbr0 -g PRE_internal
-A PREROUTING_ZONES -g PRE_public
-A PRE_internal -j PRE_internal_log
-A PRE_internal -j PRE_internal_deny
-A PRE_internal -j PRE_internal_allow
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Tue Nov 28 19:16:31 2017

Answer

ololoepepe picture ololoepepe · Dec 20, 2017

Finally, answering my own question.

After some research I have found that it was IP masquerading that caused Docker to recognize client addresses incorrectly.

The cause

In my case, firewalld is configured to share internet connection with other machines in local network. This is accomplished by setting zone to public for the external interface (enp3s0) and to internal for the local interfaces (enp1s0, enp2s0), and by enabling IP masquerading for outgoing packages.

And that's it. Masquerading is also applied to packages coming from public network to docker networks, as if the package was going to (another) external interface.

Solution

As far as I understand all that iptables and Docker stuff, there is one acceptable workaround: add all virtual network interfaces created by Docker to the internal zone, so masquerading rule is not applied to packages going to the corresponding networks. For example:

# firewall-cmd --permanent --zone=internal --change-interface=br-192821446c9a

To make the solution persistent, it is required to create the corresponding script in the /etc/sysconfig/network-scripts directory. For example, for the br-192821446c9a interface, a file /etc/sysconfig/network-scripts/ifcfg-br-192821446c9a should be created, with the following content:

DEVICE=br-192821446c9a
TYPE=Bridge
BOOTPROTO=none
IPADDR=172.18.0.1
PREFIX=16
DEFROUTE=yes
IPV4_DNS_PRIORITY=100
NAME=br-192821446c9a
ONBOOT=no
ZONE=internal

Hope this answer will save somebody's time.

Related questions

Multiple docker containers with same container port connected to the same network

I am having an app that depends on multiple docker containers. I use docker compose so that all of them are in the same network for inter-container communication. But, two of my containers are listening to the same port 8080 inside …

Read More
Communication between multiple docker-compose projects

I have two separate docker-compose.yml files in two different folders: ~/front/docker-compose.yml ~/api/docker-compose.yml How can I make sure that a container in front can send requests to a container in api? I know that --default-gateway option …

Read More
Forward host port to docker container

Is it possible to have a Docker container access ports opened by the host? Concretely I have MongoDB and RabbitMQ running on the host and I'd like to run a process in a Docker container to listen to the queue …

Read More