Connect docker-compose to external database

Question

I have a set up of 4 containers that need to talk to each other and two of those need to connect to an external database.

I started working with composer and link everything together.

The containers are able to talk with each other without many issues, however they can't connect to the external database.

The external DB is up and running and I can easily connect to it via shell.

The docker-compose file looks like this:

version: "3"

services:  

  bridge:
    # version => 2.1.4
    build: ./lora-gateway-bridge
    ports:
      - "1680/udp:1700/udp"
    links:
      - emqtt
      - redis
    environment:
      - MQTT_SERVER=tcp://emqtt:1883
    networks:
      - external
    restart: unless-stopped

  loraserver:
    # version => 0.16.1
    build: ./loraserver
    links:
      - redis
      - emqtt
      - lora-app-server
    environment:
      - NET_ID=010203
      - REDIS_URL=redis://redis:6379
      - DB_AUTOMIGRATE=true
      - POSTGRES_DSN=${SQL_STRING} ###<- connection string
      - BAND=EU_863_870
    ports:
      - "8000:8000"
    restart: unless-stopped

  lora-app-server:
    build: ./lora-app-server 
    # version => 0.8.0
    links:
      - emqtt
      - redis
    volumes:
      - "/opt/lora-app-server/certs:/opt/lora-app-server/certs"
    environment:
      - POSTGRES_DSN=${SQL_STRING} ### <- connection string
      - REDIS_URL=redis://redis:6379
      - NS_SERVER=loraserver:8000
      - MQTT_SERVER=tcp://emqtt:1883
    ports:
      - "8001:8001"
      - "443:8080"
    restart: unless-stopped

  redis:
    image: redis:3.0.7-alpine
    restart: unless-stopped

  emqtt:
    image: erlio/docker-vernemq:latest
    volumes:
      - ./emqttd/usernames/vmq.passwd:/etc/vernemq/vmq.passwd
    ports:
      - "1883:1883"
      - "18083:18083"
    restart: unless-stopped

It seems like they are unable to find the host where the database is running.

All the example that I see talk about a database inside the docker-compose, but I haven't quite grasp how to connect the container to an external service.

Answer 1

From your code I see that you need to connect to an external PostgreSQL server.

Networks

Being able to discover some resource in the network is related to which network is being used.

There is a set of network types that can be used, which simplify the setup, and there is also the option to create your own networks and add containers to them.

You have a number of types that you can choose from, the top has the most isolation possible:

• closed containers = you have only the loopback inside the container but no interactions with the container virtual network and neither with the host network
• bridged containers = your containers are connected through a default bridge network which is connected finally to the host network
• joined containers = your containers network is the same and no isolation is present at that level (), also has connection to the host network
• open containers = full access to the host network

The default type is bridge so you will have all containers using one default bridge network.

In docker-compose.yml you can choose a network type from network_mode

Because you haven't defined any network and haven't changed the network_mode, you get to use the default - bridge.

This means that your containers will join the default bridge network and every container will have access to each other and to the host network.

Therefore your problem does not reside with the container network. And you should check if PostgreSQL is accessible for remote connections. For example you can access PostgreSQL from localhost by default but you need to configure any other remote connection access rules.

You can configure your PostgreSQL instance by following this answer or this blog post.

Inspect networks

Following are some commands that might be useful in your scenario:

• list your available networks with: docker network ls
• inspect which container uses bridge network: docker network inspect --format "{{ json .Containers }}" bridge
• inspect container networks: docker inspect --format "{{ json .NetworkSettings.Networks }}" myContainer

Testing connection

In order to test the connection you can create a container that runs psql and tries to connect to your remote PostgreSQL server, thus isolating to a minimum environment to test your case.

Dockerfile can be:

FROM ubuntu
RUN apt-get update
RUN apt-get install -y postgresql-client
ENV PGPASSWORD myPassword
CMD psql --host=10.100.100.123 --port=5432 --username=postgres -c "SELECT 'SUCCESS !!!';"

Then you can build the image with: docker build -t test-connection .

And finally you can run the container with: docker run --rm test-connection:latest

If your connection succeeds then SUCCESS !!! will be printed.

Note: connecting with localhost as in CMD psql --host=localhost --port=5432 --username=postgres -c "SELECT 'SUCCESS !!!';" will not work as the localhost from within the container is the container itself and will be different than the main host. Therefore the address needs to be one that is discoverable.

Note: if you would start your container as a closed container using docker run --rm --net none test-connection:latest, there will be no other network interface than loopback and the connection will fail. Just to show how choosing a network may influence the outcome.