A Primer on Prometheus Metrics

Kumar Harsh  on October 14, 2021

Keeping an eye on metrics is one of the essential things that every developer should do for their applications. Prometheus is a tool that has gained a lot of popularity in the field of metrics. When you start with Prometheus, you may not recognize its potential at first because it takes time to explore the entire tool. But once you become acquainted with the features, you will find it highly useful.

Prometheus metrics libraries have been widely adopted by various monitoring systems such as InfluxDB, OpenTSDB, Graphite, and Scout APM, in addition to Prometheus users. Many CNCF projects now use the Prometheus metrics format to expose out-of-the-box metrics. They are also in Kubernetes core components like the API server, CoreDNS, and others. More Kubernetes monitoring information is available in the Prometheus guide. Many tools are pre-installed with support for this format, and they will automatically detect and scrape Prometheus metrics for you.

The Prometheus metrics format has become so popular that it has spun off into its own project, OpenMetrics, intending to make the metric format specification an industry standard.

Prometheus supports many programming languages like Golang, Java, Python, and Javascript. In this blog, we will be discussing how to use Prometheus metrics/Open metrics for Golang, Java, Python, and Javascript. You can use the following to navigate through the article.

What is Prometheus?

Prometheus is an open-source tool for time series database monitoring released by SoundCloud and is currently a community-driven project under the Cloud Native Computing Foundation. Prometheus has several client libraries, and they scrape data collected by the metric server. It is reliable, fast and secure too.

Some of the most critical features of Prometheus are:

Prometheus has many use cases, but it works very well with a dynamic recording of purely numeric time series. It is suited for both machine-centric monitoring as well as dynamic service-oriented architecture for monitoring. Where various companies are working on microservices, Prometheus supports multi-dimensional data collection and querying. There is no network storage dependency because each Prometheus server is standalone. Because of this, it works with broken infrastructure, too; you do not need to set up extensive infrastructure to use it.

Prometheus Metrics Format

Prometheus metrics format is the representation of data collected by Prometheus for your software. Metrics are vital for any software as they represent the performance of the software. Metrics representations help to see the data in a well-formatted manner and make it easily readable. 

These metrics provide the level of depth and hierarchy that you'll need to make the most of your measurements. To make it quick and easy to utilize your metrics, this model of metrics exposition advises that you should compute it ahead of time and save it under a new name if you want a different aggregate.

Metrics come in two broad formats: 

Let us start with dot-metrics. An example of dot-metrics is below:

production.server5.pod50.html.request.total
production.server5.pod50.html.request.error

These types of metrics provide the details about the hierarchy that you need to utilize the metrics fully. This metric format suggests calculating the metric upfront and storing it using a separate name. So in the above example, if we are interested in the requests metric across the application, we could rearrange it in the following ways:

production.service-nginx.html.request.total
production.service-nginx.html.request.error


But Prometheus does not use the dot-based metrics format; instead, it uses the series of labels or tags to represent the metrics. For example:

<metric name>{<label name>=<label value>, ...}


Metrics whose name is request_for_all, service equal to service, server equal to pod50, and environment equal to production can be used in the tag-based format.

request_for_all{service="service", server="pod50", env="production"}

It helps in representing multi-dimensional data where you can show any number of context-specific labels to any tag.

Suppose you have the metric requests_per_second, and it requires a lot of labels. In the tag-based, you bundle the labels then use them. Now you have n-dimensional data and can easily derive the graphs.

After seeing the two main formats in which we can represent the metrics, let’s see the format of metrics or Open Metrics. A metric is composed of several fields like: 

Prometheus Metric Types

There are four types of metrics available by Prometheus client libraries.

Counter

The counter generally refers to those values which only increased and can be reset to zero. The exact definition applies in Prometheus too. You can use counter metrics for total server requests, total database queries, tasks completed, or errors. All these metrics are monotonically increasing. The counter supports four client libraries, Java, Ruby, Go, and Python.

Gauge

A gauge is just a numerical value that can go up and down randomly.  Gauge is generally for those values that arbitrarily go up and down and can be counted, for example, temperature, current memory usage, real issues, and the number of active requests. It also supports by all four client libraries, Java, Go, Python, and Ruby.

Histogram

The histogram is a sample of observations like response time, request duration, or response size in customizable buckets. A histogram consists of four things and exposes multiple time series. The four components of the histogram are: 

A  histogram is for calculating various quantities. For example, using the histogram_quantile() function, you can calculate the quantile and aggregation of histograms. You can also use the histogram to calculate Apdex score. The histogram is always cumulative as it sums the previous value to the current one. It also supports Java, Go, Python, and Ruby client libraries.

Histograms are not only aggregatable, but they are also less expensive on the client because counters are quick to increment. So, why not utilize histograms all of the time? The short answer is that when using histograms, you must pre-select your buckets, and because of bucket cardinality, the expenses shift from the client to Prometheus itself. The default ten buckets cover a typical online service with latency ranging from milliseconds to seconds, although you may wish to tweak them occasionally. In this case, they’ve been modified to better track requests for PromQL, which have a two-minute default timeout.

Summary

The summary has almost the same structure as the histogram; it also samples observations like response time, request duration, or the response size. In addition, it also shows the sum of all the observed values, the total count of observations, and a configurable quantile over a sliding time window. A summary, similar to a histogram, has four components, as mentioned below:

All four client libraries are supported; you can check out the detailed documentation for Java, Go, Python, and Ruby.

Example of Prometheus Metrics

Prometheus metrics include various types of labels, as mentioned above. Those metrics are not strictly required and can have any number of metrics.

Hence a Prometheus metric can be as simple as:

http_requests 2

Or after including all of the above mentioned components:

http_requests_total{method="post",code="400"}  3   13950663630003


#HELP string in a metric represents the metric name and description of it. #TYPE string denoted the type of metric. If there is no type of metric, then it is said to be untyped. Everything else in a metric starting with # is considered a comment other than these two strings. You can use #HELP and #TYPE metadata lines to put in each metric. Hence an example metric with these strings is:

# HELP metric_name Description of the metric
# TYPE metric_name type
# Comment that Prometheus does not parse
http_requests_total{method="post",code="400"}  3   1395066363000

An advantage of this tag-based format for Prometheus metrics is that these labels can perform various operations like aggregation, scoping, segmentation, etc. 

Official Prometheus Metric Libraries

Prometheus provides many client libraries for its users to experience easy installation of the monitoring tool. But the top four libraries which Prometheus officially maintains are available for Golang, Java, Javascript, and Python. Let’s see the code instrumentation using each of the libraries one by one. 

You can check out the official list of code instrumentation libraries provided by Prometheus.

Golang

You can use the Golang library of Prometheus to instrument your Go application. For instrumenting your Go application, you have to install Prometheus, promauto, and promhttp using the following libraries.

go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promauto
go get github.com/prometheus/client_golang/prometheus/promhttp


For accessing the Prometheus metrics, you have to go to /metrics in the localhost.

Also, you can use prometheus/promhttp as a handler function for HTTP. The most basic code which will show metrics on http://localhost:9000/metrics is:

package main
import (
        "net/http"
        "github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
        http.Handle("/metrics", promhttp.Handler())
        http.ListenAndServe(":9000", nil)
}


For starting the application run:

go run main.go

Then for launching it run: 

curl http://localhost:2112/metrics


Additionally, you can add your application to generate custom metrics. In this example, we will expose a myapp_processed_ops_total counter which counts the number of operations that have been processed so far. It gets updated after 2 sec, and the counter gets updated by one. The complete code example is below:

package main
import (
        "net/http"
        "time"
        "github.com/prometheus/client_golang/prometheus"
        "github.com/prometheus/client_golang/prometheus/promauto"
        "github.com/prometheus/client_golang/prometheus/promhttp"
)

func recordMetrics() {
        go func() {
                for {
                        opsProcessed.Inc()
                        time.Sleep(2 * time.Second)
                }
        }()
}

var (
        opsProcessed = promauto.NewCounter(prometheus.CounterOpts{
                Name: "myapp_processed_ops_total",
                Help: "The total number of processed events",
        })
)

func main() {
        recordMetrics()
        http.Handle("/metrics", promhttp.Handler())
        http.ListenAndServe(":9000", nil)
}


Then run the application using:

go run main.go

For accessing the metrics collected by your application, go to http://localhost:9000/metrics.As a result, you will see the following output:

# HELP myapp_processed_ops_total The total number of processed events
# TYPE myapp_processed_ops_total counter
myapp_processed_ops_total 5


Other than the above features discussed, Prometheus Go client libraries expose many other metrics such as gauges, histograms, non-global registries, and functions for pushing metrics to Prometheus push getaways.

Java

Java client libraries are one of the most important libraries that Prometheus offers because Java is one of the most popular languages both for the frontend and backend. Prometheus Java client library provides many metrics such as histogram, counter, graph, summary, labels, etc.

Below we have provided a sample code using Java client libraries for collecting metrics like counter, gauge, histogram, summary.

import io.prometheus.client.Counter;
import io.prometheus.client.Gauge;
import io.prometheus.client.Histogram;
import io.prometheus.client.Summary;
import io.prometheus.client.exporter.HTTPServer;

import java.io.IOException;
import java.util.Random;

public class Main {

  private static double rand(double min, double max) {
      return min + (Math.random() * (max - min));
  }

  public static void main(String[] args) {
      Counter counter = Counter.build().namespace("java").name("my_counter").help("This is my counter").register();
      Gauge gauge = Gauge.build().namespace("java").name("my_gauge").help("This is my gauge").register();
      Histogram histogram = Histogram.build().namespace("java").name("my_histogram").help("This is my histogram").register();
      Summary summary = Summary.build().namespace("java").name("my_summary").help("This is my summary").register();

      Thread bgThread = new Thread(() -> {
          while (true) {
              try {
                  counter.inc(rand(0, 5));
                  gauge.set(rand(-5, 10));
                  histogram.observe(rand(0, 5));
                  summary.observe(rand(0, 5));


                  Thread.sleep(1000);
              } catch (InterruptedException e) {
                  e.printStackTrace();
              }
          }
      });
      bgThread.start();

      try {

          HTTPServer server = new HTTPServer(8080);
      } catch (IOException e) {
          e.printStackTrace();
      }
  }
}

For using it in Docker you just have to download, build and run.

$ git clone https://github.com/sysdiglabs/custom-metrics-examples
$ docker build custom-metrics-examples/prometheus/java -t prometheus-java
$ docker run -d --rm --name prometheus-java -p 8080:8080 -p 80:80 prometheus-java

For checking if it is working, go to localhost:

$ curl -v localhost
* Rebuilt URL to: localhost/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 08 Oct 2018 13:17:07 GMT
< Transfer-encoding: chunked
<
* Connection #0 to host localhost left intact


Now go to localhost:8080, and voila! You have your Prometheus dashboard.

Python

Prometheus supports Python 2 and 3 for its client libraries. First, you have to install the Prometheus client using this command:

pip install prometheus-client

Like Golang and Java client libraries, Python client libraries also provide metrics like histograms, counter, graphs, summary, labels, etc. Below is an example code for getting counter, gauge, histogram, and summary:

import prometheus_client as prom
import random
import time

req_summary = prom.Summary('python_my_req_example', 'Time spent processing a request')


@req_summary.time()
def process_request(t):
  time.sleep(t)


if __name__ == '__main__':

  counter = prom.Counter('python_my_counter', 'This is my counter')
  gauge = prom.Gauge('python_my_gauge', 'This is my gauge')
  histogram = prom.Histogram('python_my_histogram', 'This is my histogram')
  summary = prom.Summary('python_my_summary', 'This is my summary')
  prom.start_http_server(8080)

  while True:
      counter.inc(random.random())
      gauge.set(random.random() * 15 - 5)
      histogram.observe(random.random() * 10)
      summary.observe(random.random() * 10)
      process_request(random.random() * 5)

      time.sleep(1)

For very detailed documentation of how to use counter, graphs, gauge, summary, etc., using Python client libraries, you can check out the official repository of Prometheus Python client libraries.

For using it in Docker, run the following commands:

$ git clone https://github.com/sysdiglabs/custom-metrics-examples
$ docker build custom-metrics-examples/prometheus/python -t prometheus-python
$ docker run -d --rm --name prometheus-python -p 8080:8080 -p 80:80 prometheus-python

Now you can check whether the endpoint is working or not by going to localhost:8080

JavaScript

The Prometheus client for NodeJS provides metrics like graphs, summary, counter, gauges, etc. Here we have provided an example code using the NodeJS client’s library of Prometheus. 

const client = require('prom-client');
const express = require('express');
const server = express();
const register = new client.Registry();

// Probe every 5th second.
const intervalCollector = client.collectDefaultMetrics({prefix: 'node_', timeout: 5000, register});

const counter = new client.Counter({
  name: "node_my_counter",
  help: "This is my counter"
});

const gauge = new client.Gauge({
  name: "node_my_gauge",
  help: "This is my gauge"
});

const histogram = new client.Histogram({
  name: "node_my_histogram",
  help: "This is my histogram",
  buckets: [0.1, 5, 15, 50, 100, 500]
});

const summary = new client.Summary({
  name: "node_my_summary",
  help: "This is my summary",
  percentiles: [0.01, 0.05, 0.5, 0.9, 0.95, 0.99, 0.999]
});

register.registerMetric(counter);
register.registerMetric(gauge);
register.registerMetric(histogram);
register.registerMetric(summary);

const rand = (low, high) => Math.random() * (high - low) + low;

setInterval(() => {
  counter.inc(rand(0, 1));
  gauge.set(rand(0, 15));
  histogram.observe(rand(0,10));
  summary.observe(rand(0, 10));
}, 1000);

server.get('/metrics', (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(register.metrics());
});

console.log('Server listening to 8080, metrics exposed on /metrics endpoint');
server.listen(8080);

Prometheus Exporters

Exporters refer to the tools in Prometheus that help in exporting metrics. These are most useful when the code can not be instrumented with Prometheus in the raw state, for example, HAProxy, or Linux system stats. The Prometheus Github organization officially authorizes some third-party exporters, but many other exporters are still available outside the Prometheus organization. Prometheus has provided exporters in different categories; for example, in the Database category, there are database exporters, hardware-related exporters, etc. 

Several other categories of exporters are available like messaging, HTTP, Logging, APIs, Storage, Issue tracking, etc. For seeing the whole list of exporters, you can check out their official page.

Prometheus Is a Pull-Based Metrics System

Client libraries, often known as exporters, do not send metrics to Prometheus directly.

Instead, they offer an endpoint that returns a payload containing all available metrics in Prometheus format. Prometheus will use this endpoint to collect data at predetermined intervals. This endpoint must return a payload in a Prometheus-compatible format.

As a result, Prometheus pulls (or "scrapes") metrics as needed. This is known as a "pull-based system," and it's a suitable strategy for monitoring many servers and apps. It's preferable to have data arrive late than to lose data.

In this blog article, Prometheus discusses the challenges, benefits, and drawbacks of both pull and push systems. Without diving further into this topic, exposing an endpoint that emits metrics in a certain format is ideal for dealing with Prometheus. It doesn't rule out the possibility of Prometheus using a push-based method. However, you should only use this strategy when required.

Prometheus is a straightforward instrument, as evidenced by its user interface. Once you grasp the fundamentals of how and why things work the way they do, the rest will become clear. Prometheus was initially known for being a push-based system in general, and most of us assumed it was exclusively relevant for Kubernetes. On the other hand, Prometheus can collect metrics from a variety of sources, as this post demonstrates.

Recap and Closing Thoughts

Metrics are crucial for any application’s smooth functioning. For making the best use, metrics should be easy to interpret and understand. Prometheus metrics/Open metrics provide metrics for your application in various formats and many programming languages. You can generate your custom metrics using Java, Python, JavaScript, or NodeJs. Also, if you have to export your data to any other tool, the exporters of Prometheus are widely used for that purpose.

You can also use Prometheus for application performance and management purposes as it monitors the metrics and provides them in a readable form. However, ScoutAPM is one of the most modern APM tools of application performance and management. It has various features like memory bloat detection, N+1 queries, slow API response, etc. You can try it out free for 14 days without a credit card. Go and try out Scout APM now!

Happy coding!