Using prometheus custom metrics with SpringBoot

Question

I'm not sure it's a bug but I'm trying to implement this library in the past day without any results. I think maybe it's a conflict with the dependencies versions but I tried everything.

So I have a very simple spring boot project that serves the /test endpoint and I configure

1. Counter
2. Histogram

Now I have the prometheus metrics under /actuator/prometheus.

The desired result is that the metrics will contain the following additional metrics:

1. requests_test_total
2. requests_latency_seconds

I'm triggering the /test endpoint, but these metrics are not shown in the prometheus metrics.

Am I missing something?

These are my dependencies:

dependencies {
    compile('org.springframework.boot:spring-boot-starter-web')
    compile('org.springframework.boot:spring-boot-starter-actuator')

    compile('io.micrometer:micrometer-registry-prometheus')

    compile "io.prometheus:simpleclient:0.8.1"
    compile "io.prometheus:simpleclient_hotspot:0.8.1"
    compile "io.prometheus:simpleclient_httpserver:0.8.1"
    compile "io.prometheus:simpleclient_pushgateway:0.8.1"
    compile group: 'io.prometheus', name: 'simpleclient_spring_boot', version: '0.8.1'

    compile group: 'org.slf4j', name: 'slf4j-api', version: '1.7.30'
    compile group: 'ch.qos.logback', name: 'logback-classic', version: '1.2.3'
    compile group: 'ch.qos.logback.contrib', name: 'logback-json-classic', version: '0.1.5'
    compile group: 'ch.qos.logback.contrib', name: 'logback-jackson', version: '0.1.5'
    compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.9.3'
    compile group: 'net.logstash.logback', name: 'logstash-logback-encoder', version: '6.3'
}

And this is the endpoint class implementation:

package prometheus.prometheusclientexample;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import io.prometheus.client.Counter;
import io.prometheus.client.Histogram;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;


@RestController
public class SimpleEndpoint {
    private static final Logger logger = LoggerFactory.getLogger("MainLogger");
    static final Counter test_requests = Counter.build().name("requests_test_total").help("Total /test requests.").register();
    static final Histogram requestLatency = Histogram.build()
            .name("requests_latency_seconds").help("Request latency in seconds.").register();


    @RequestMapping("/test")
    public String test2() throws InterruptedException {
        Histogram.Timer requestTimer = requestLatency.startTimer();
        try {
            logger.info("Before");
            test_requests.inc();
            logger.info("This is a simple endpint");
        }
        finally {
            requestTimer.observeDuration();
        }
        return "OK NEW";
    }
}

/actuator/prometheus response :

# HELP jvm_gc_memory_allocated_bytes_total Incremented for an increase in the size of the young generation memory pool after one GC to before the next
# TYPE jvm_gc_memory_allocated_bytes_total counter
jvm_gc_memory_allocated_bytes_total 0.0
# HELP jvm_gc_max_data_size_bytes Max size of old generation memory pool
# TYPE jvm_gc_max_data_size_bytes gauge
jvm_gc_max_data_size_bytes 4.294967296E9
# HELP logback_events_total Number of error level events that made it to the logs
# TYPE logback_events_total counter
logback_events_total{level="warn",} 0.0
logback_events_total{level="debug",} 368.0
logback_events_total{level="error",} 0.0
logback_events_total{level="trace",} 0.0
logback_events_total{level="info",} 12.0
# HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management
# TYPE jvm_memory_max_bytes gauge
jvm_memory_max_bytes{area="heap",id="G1 Survivor Space",} -1.0
jvm_memory_max_bytes{area="heap",id="G1 Old Gen",} 4.294967296E9
jvm_memory_max_bytes{area="nonheap",id="Metaspace",} -1.0
jvm_memory_max_bytes{area="nonheap",id="CodeHeap 'non-nmethods'",} 6975488.0
jvm_memory_max_bytes{area="heap",id="G1 Eden Space",} -1.0
jvm_memory_max_bytes{area="nonheap",id="Compressed Class Space",} 1.073741824E9
jvm_memory_max_bytes{area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 2.44682752E8
# HELP tomcat_sessions_alive_max_seconds  
# TYPE tomcat_sessions_alive_max_seconds gauge
tomcat_sessions_alive_max_seconds 0.0
# HELP jvm_gc_live_data_size_bytes Size of old generation memory pool after a full GC
# TYPE jvm_gc_live_data_size_bytes gauge
jvm_gc_live_data_size_bytes 0.0
# HELP process_start_time_seconds Start time of the process since unix epoch.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.588860975956E9
# HELP tomcat_sessions_active_max_sessions  
# TYPE tomcat_sessions_active_max_sessions gauge
tomcat_sessions_active_max_sessions 0.0
# HELP tomcat_sessions_created_sessions_total  
# TYPE tomcat_sessions_created_sessions_total counter
tomcat_sessions_created_sessions_total 0.0
# HELP jvm_classes_unloaded_classes_total The total number of classes unloaded since the Java virtual machine has started execution
# TYPE jvm_classes_unloaded_classes_total counter
jvm_classes_unloaded_classes_total 0.0
# HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
# TYPE jvm_threads_live_threads gauge
jvm_threads_live_threads 24.0
# HELP process_files_max_files The maximum file descriptor count
# TYPE process_files_max_files gauge
process_files_max_files 10240.0
# HELP http_server_requests_seconds  
# TYPE http_server_requests_seconds summary
http_server_requests_seconds_count{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 1.0
http_server_requests_seconds_sum{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 0.144024648
http_server_requests_seconds_count{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/test",} 2.0
http_server_requests_seconds_sum{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/test",} 0.029707973
# HELP http_server_requests_seconds_max  
# TYPE http_server_requests_seconds_max gauge
http_server_requests_seconds_max{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 0.144024648
http_server_requests_seconds_max{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/test",} 0.022717086
# HELP jvm_memory_used_bytes The amount of used memory
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{area="heap",id="G1 Survivor Space",} 7340032.0
jvm_memory_used_bytes{area="heap",id="G1 Old Gen",} 4369520.0
jvm_memory_used_bytes{area="nonheap",id="Metaspace",} 4.1996312E7
jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'non-nmethods'",} 1204864.0
jvm_memory_used_bytes{area="heap",id="G1 Eden Space",} 7.9691776E7
jvm_memory_used_bytes{area="nonheap",id="Compressed Class Space",} 5303144.0
jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 6563328.0
# HELP process_uptime_seconds The uptime of the Java virtual machine
# TYPE process_uptime_seconds gauge
process_uptime_seconds 15.703
# HELP jvm_threads_peak_threads The peak live thread count since the Java virtual machine started or peak was reset
# TYPE jvm_threads_peak_threads gauge
jvm_threads_peak_threads 24.0
# HELP process_files_open_files The open file descriptor count
# TYPE process_files_open_files gauge
process_files_open_files 83.0
# HELP tomcat_sessions_expired_sessions_total  
# TYPE tomcat_sessions_expired_sessions_total counter
tomcat_sessions_expired_sessions_total 0.0
# HELP process_cpu_usage The "recent cpu usage" for the Java Virtual Machine process
# TYPE process_cpu_usage gauge
process_cpu_usage 0.002284847603487466
# HELP system_load_average_1m The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
# TYPE system_load_average_1m gauge
system_load_average_1m 11.57421875
# HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool
# TYPE jvm_buffer_memory_used_bytes gauge
jvm_buffer_memory_used_bytes{id="mapped",} 0.0
jvm_buffer_memory_used_bytes{id="direct",} 32768.0
# HELP jvm_buffer_total_capacity_bytes An estimate of the total capacity of the buffers in this pool
# TYPE jvm_buffer_total_capacity_bytes gauge
jvm_buffer_total_capacity_bytes{id="mapped",} 0.0
jvm_buffer_total_capacity_bytes{id="direct",} 32768.0
# HELP jvm_gc_memory_promoted_bytes_total Count of positive increases in the size of the old generation memory pool before GC to after GC
# TYPE jvm_gc_memory_promoted_bytes_total counter
jvm_gc_memory_promoted_bytes_total 0.0
# HELP jvm_memory_committed_bytes The amount of memory in bytes that is committed for the Java virtual machine to use
# TYPE jvm_memory_committed_bytes gauge
jvm_memory_committed_bytes{area="heap",id="G1 Survivor Space",} 7340032.0
jvm_memory_committed_bytes{area="heap",id="G1 Old Gen",} 1.90840832E8
jvm_memory_committed_bytes{area="nonheap",id="Metaspace",} 4.3646976E7
jvm_memory_committed_bytes{area="nonheap",id="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_committed_bytes{area="heap",id="G1 Eden Space",} 1.24780544E8
jvm_memory_committed_bytes{area="nonheap",id="Compressed Class Space",} 5898240.0
jvm_memory_committed_bytes{area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 6619136.0
# HELP jvm_threads_daemon_threads The current number of live daemon threads
# TYPE jvm_threads_daemon_threads gauge
jvm_threads_daemon_threads 20.0
# HELP jvm_buffer_count_buffers An estimate of the number of buffers in the pool
# TYPE jvm_buffer_count_buffers gauge
jvm_buffer_count_buffers{id="mapped",} 0.0
jvm_buffer_count_buffers{id="direct",} 4.0
# HELP jvm_threads_states_threads The current number of threads having NEW state
# TYPE jvm_threads_states_threads gauge
jvm_threads_states_threads{state="runnable",} 10.0
jvm_threads_states_threads{state="blocked",} 0.0
jvm_threads_states_threads{state="waiting",} 11.0
jvm_threads_states_threads{state="timed-waiting",} 3.0
jvm_threads_states_threads{state="new",} 0.0
jvm_threads_states_threads{state="terminated",} 0.0
# HELP tomcat_sessions_active_current_sessions  
# TYPE tomcat_sessions_active_current_sessions gauge
tomcat_sessions_active_current_sessions 0.0
# HELP system_cpu_count The number of processors available to the Java virtual machine
# TYPE system_cpu_count gauge
system_cpu_count 4.0
# HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
# TYPE jvm_classes_loaded_classes gauge
jvm_classes_loaded_classes 7680.0
# HELP system_cpu_usage The "recent cpu usage" for the whole system
# TYPE system_cpu_usage gauge
system_cpu_usage 0.5317742755465176
# HELP tomcat_sessions_rejected_sessions_total  
# TYPE tomcat_sessions_rejected_sessions_total counter
tomcat_sessions_rejected_sessions_total 0.0

Answer 1

if you use SpringBoot Micrometer Prometheus plugin I suggest to use the built-in MeterRegistry which collects and exports metrics in Prometheus format.
You avoid then the need of creating the counters/histograms (which you must do if use the io.prometheus Simpleclient library directly)

@Autowired
private MeterRegistry meterRegistry;

@GetMapping(value = "/main/test", produces = "text/plain")
public String test() {

    meterRegistry.counter("requests_test_total").increment();


    return "ok";
}

Enable the Prometheus endpoint in the application.properties

management.endpoints.web.exposure.include=health,info,loggers,prometheus