Direct instrumentation with Prometheus client_golang

2021-08-14

5 min read

Instrumenting Applications with Prometheus client_golang

If you have an application with source code in your control, you can define metrics and add desired instrumentation inline to the code. Let’s look at the required basics and write a direct instrumented random number generator.

Interfaces involved in creating and exposing metrics

Metric interface

type Metric interface {
    Desc() *Desc
    Write(*dto.Metric) error
}

Source : github.com/prometheus/client_golang

A metric is a single value with it’s metadata (i.e. metric labels) exported to Prometheus. Counter, Gauge, Histogram and Summary implement the Metric interface.

Collector interface

type Collector interface {
    Describe(chan<- *Desc)
    Collect(chan<- Metric)
}

Source: github.com/prometheus/client_golang

Collectors are used by Prometheus to collect metrics. Anything that implements Collector interface is a collector. Some default collectors provided by Prometheus are Counter, Gauge, Histogram and Summary. These can be used for direct instrumentation of application.

Registerer interface

type Registerer interface {
    Register(Collector) error
    MustRegister(...Collector)

    Unregister(Collector) bool
}

Source: github.com/prometheus/client_golang

Each collector needs to register itself to a registry. You can use the default registry provided by Prometheus or create a custom registry. These registries implement the Registerer interface. Registries can unregister collectors if required.

Gatherer interface

type Gatherer interface {
    Gather() ([]*dto.MetricFamily, error)
}

Source: github.com/prometheus/client_golang

Metrics from all registered collectors are collected by a gatherer. These gatherers implement the Gatherer interface.

How to create metrics using these interfaces?

We start with a sample random number generator application that prints the generated number to standard output. Then, we instrument it to provide randomly generated numbers as metrics instead of printing to standard output.

package main

import (
    "fmt"
    "math/rand"
)

func main() {
    go func() {
        for {
            rnd := rand.Float64()
            fmt.Println(rnd)
        }
    }()
}

Choose a metric type for the metric to be exported
Choosing a metric type depends on the value being exported. If value only goes up we use Counter type.
If it can go up or down arbitrarily, we use Gauge type.
For example, a randomly generated number can go up or down arbitrarily. So, we choose Gauge type.
Create a gauge metric using the constructor for Gauge type
prometheus.NewGauge() constructs a metric of Gauge type based on the provided GaugeOpts. GaugeOpts are used to construct metric name and metadata.
For example, randomNumber is a metric of Gauge type. Metric name is random_number_generated and it has a constant label app: random_number_generator.
```
var (
    randomNumber = prometheus.NewGauge(prometheus.GaugeOpts{
        Namespace: "random",
        Subsystem: "number",
        Name:      "generated",
        Help:      "A randomly generated number",
        ConstLabels: prometheus.Labels{
            "app": "random_number_generator",
        },
    })
)
```
Register the collector to collect metrics
Since Counter, Gauge, Histogram and Summary metric types implement the Collector interface, we can register it for collection. We use the default prometheus registry for that.
For example, randomNumber collector is registered to default Prometheus registry using prometheus.MustRegister(). We do this inside init() as we want these to be registered as soon as application starts and all variables are defined.
```
func init() {
    prometheus.MustRegister(randomNumber)
}
```
Set metric value
Each metric type provides it’s own set of methods to set metric value. Gauge type provides a Set() method among others to set any arbitrary metric value.
For example, we set our metric value to the randomly generated number rnd.
```
rnd := rand.Float64()
randomNumber.Set(rnd)
```

This is what our instrumented application looks like at this point.

package main

import (
    "math/rand"

    "github.com/prometheus/client_golang/prometheus"
)

var (
    randomNumber = prometheus.NewGauge(prometheus.GaugeOpts{
        Namespace: "random",
        Subsystem: "number",
        Name:      "generated",
        Help:      "A randomly generated number",
        ConstLabels: prometheus.Labels{
            "app": "random_number_generator",
        },
    })
)

func init() {
    prometheus.MustRegister(randomNumber)
}

func main() {
    go func() {
        for {
            rnd := rand.Float64()
            randomNumber.Set(rnd)
        }
    }()
}

How to expose metrics?

Prometheus is pull based. It means that creating metrics is not enough. We need to a way to pull (i.e. scrape) them. We do so by exposing an http endpoint which returns the metrics in Prometheus exposition format.

Create a http handler for Gatherer
Since Gatherer interface that calls the Collect() method of all registered collectors, we need an http handler to for it. We use promhttp to create a http handler.
```
promhttp.HandlerFor(
    prometheus.DefaultGatherer,
    promhttp.HandlerOpts{},
)
```
Register handler to /metrics
Register the http handler to /metrics. This is the standard path to expose metrics.
```
http.Handle("/metrics", promhttp.HandlerFor(
    prometheus.DefaultGatherer,
    promhttp.HandlerOpts{},
))
```
Create a listener
Create a listener to receive http requests. Here, we listen at port 8080 in localhost.
```
http.ListenAndServe(":8080", nil)
```

Finally, this is what our instrumented application looks like.

package main

import (
    "math/rand"
    "net/http"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    randomNumber = prometheus.NewGauge(prometheus.GaugeOpts{
        Namespace: "random",
        Subsystem: "number",
        Name:      "generated",
        Help:      "A randomly generated number",
        ConstLabels: prometheus.Labels{
            "app": "random_number_generator",
        },
    })
)

func init() {
    prometheus.MustRegister(randomNumber)
}

func main() {
    go func() {
        for {
            rnd := rand.Float64()
            randomNumber.Set(rnd)
        }
    }()

    http.Handle("/metrics", promhttp.HandlerFor(
        prometheus.DefaultGatherer,
        promhttp.HandlerOpts{},
    ))

    http.ListenAndServe(":8080", nil)

}

Try it out

Run the application in one terminal and curl the http endpoint in another terminal to get the metrics output.

# Terminal 1
go run main.go

# Terminal 2
for i in {1..50};do curl -s localhost:8080/metrics|grep random_number_generated; sleep 3; echo "";done

Output:

# HELP random_number_generated A randomly generated number
# TYPE random_number_generated gauge
random_number_generated{app="random_number_generator"} 0.3726773052915178

# HELP random_number_generated A randomly generated number
# TYPE random_number_generated gauge
random_number_generated{app="random_number_generator"} 0.7400739387479879

# HELP random_number_generated A randomly generated number
# TYPE random_number_generated gauge
random_number_generated{app="random_number_generator"} 0.18367311516535434

# HELP random_number_generated A randomly generated number
# TYPE random_number_generated gauge
random_number_generated{app="random_number_generator"} 0.20768171508491354