Cannot connect to Go GRPC server running in local Docker container

Question

I have a go grpc service. I'm developing on a mac, sierra. When running a grpc client against the service locally, all is well, but when running same client against same service in the docker container I get this error:

transport: http2Client.notifyError got notified that the client transport was broken EOF.
FATA[0000] rpc error: code = Internal desc = transport is closing

this is my docker file:

FROM golang:1.7.5

RUN mkdir -p /go/src/github.com/foo/bar
WORKDIR /go/src/github.com/foo/bar

COPY . /go/src/github.com/foo/bar
# ONBUILD RUN go-wrapper download
RUN go install

ENTRYPOINT /go/bin/bar

EXPOSE 51672

my command to build the image:

docker build -t bar .

my command to launch the docker container:

docker run -p 51672:51672 --name bar-container bar

Other info:

• client program runs fine from within the docker container
• connecting to a regular rest endpoint works fine (http2, grpc related?)

• running the lsof command in OS X yields these results

  $lsof -i | grep 51672
  com.docke   984 oldDave   21u  IPv4 0x72779547e3a32c89      0t0  TCP *:51672 (LISTEN)
  com.docke   984 oldDave   22u  IPv6 0x72779547cc0fd161      0t0  TCP localhost:51672 (LISTEN)
  

• here's a snippet of my server code:

  server := &Server{}
  endpoint := "localhost:51672"
  lis, err := net.Listen("tcp", endpoint)
  if err != nil {
      log.Fatalf("failed to listen: %v", err)
  }
  
  s := grpc.NewServer(grpc.Creds(creds))
  
  pb.RegisterExpServiceServer(s, server)
  
  // Register reflection service on gRPC server.
  reflection.Register(s)
  
  log.Info("Starting Exp server: ", endpoint)
  
  if err := s.Serve(lis); err != nil {
      log.Fatalf("failed to serve: %v", err)
  }
  

Answer 1

When you specify a hostname or IP address​ to listen on (in this case localhost which resolves to 127.0.0.1), then your server will only listen on that IP address.

Listening on localhost isn't a problem when you are outside of a Docker container. If your server only listens on 127.0.0.1:51672, then your client can easily connect to it since the connection is also made from 127.0.0.1.

When you run your server inside a Docker container, it'll only listen on 127.0.0.1:51672 as before. The 127.0.0.1 is a local loopback address and it not accessible outside the container.

When you fire up the docker container with "-p 51672:51672", it'll forward traffic heading to 127.0.0.1:51672 to the container's IP address, which in my case is 172.17.0.2.

The container gets an IP addresses within the docker0 network interface (which you can see with the "ip addr ls" command)

So, when your traffic gets forwarded to the container on 172.17.0.2:51672, there's nothing listening there and the connection attempt fails.

The fix:

The problem is with the listen endpoint:

endpoint := "localhost:51672"

To fix your problem, change it to

endpoint := ":51672"

That'll make your server listen on all it container's IP addresses.

Additional info:

When you expose ports in a Docker container, Docker will create iptables rules to do the actual forwarding. See this. You can view these rules with:

iptables -n -L 
iptables -t nat -n -L