Prometheus - Target Connection refused

Question

I'm trying to get a Prometheus container to scrape metrics from cAdvisor.

This is my prometheus.yml:

global:
  scrape_interval: 10s
  evaluation_interval: 10s

scrape_configs:
 - job_name: "prometheus"
   static_configs:
     - targets: ["localhost:9090"]

 - job_name: "docker"
   static_configs:
     - targets: ['localhost:9323']

 - job_name: "cadvisor"
   scrape_interval: 5s
   static_configs:
     - targets: ['localhost:7070']
       labels:
         alias: "cadvisor"

And my docker-compose.yml:

version: "3.5"

services:
  app:
    container_name: app
    build: "./app"
    restart: always
    volumes:
      - ./app:/app
    depends_on:
      - db
    links:
      - db
    ports:
      - 8080:8080
  db:
    container_name: db
    image: postgres
    restart: always
    volumes:
      - ./data/postgres:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=articlesdb
      - POSTGRES_USER=dbuser 
      - POSTGRES_PASSWORD=dbpassword
    ports:
      - "5432:5432"
  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    #build:
    #  context: ./prometheus
    #  dockerfile: Dockerfile
    ports:
      - 9090:9090
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    depends_on:
      - cadvisor
  cadvisor:
    image: google/cadvisor:latest
    container_name: cadvisor
    ports:
      - 7070:8080
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    depends_on:
      - db

In the targets section Prometheus says that the connection to the cAdvisor container and Docker daemon is refused, while Prometheus' to itself works. I can reach cAdvisor at localhost:7070/metrics with either cURL or on my browser. What could be the issue, and how can I fix it?

Answer 1

In your docker-compose file you named the cAdvisor service ‘cadvisor’ so in the docker network it can be accessed via the DNS name cadvisor. Change your prometheus.yml static_config like this to scrape the service:

 - job_name: "cadvisor"
    scrape_interval: 5s
    static_configs:
      - targets: ['cadvisor:7070']
        labels:
          alias: "cadvisor"