In cases where instrumenting an application with Prometheus metrics directly is not feasible or when other metric formats needs to be converted to Prometheus exposition format, exporter can be used.
Example: node_exporter which has multiple custom collectors for hardware and OS metrics exposed by *NIX kernels.
Custom Collectors
Custom collectors are custom implementation of Collector interface unlike Counter, Gauge etc. They are ideal for converting existing data formats to Prometheus expostion format during collection.
Example: Node Collector and CPU Collector in node_exporter.
Custom collectors in Prometheus client_golang package
The Prometheus client_golang package provides some commonly used custom collectors such as Go collector and Process collector.
How to write an exporter for an application?
Let us assume that the math/rand golang library is a random number generator application that does not expose any Prometheus metrics.
We wish to expose randomly generated numbers as metrics but we can not change the source code. So, directly instrumenting it is not possible.
Instead, we will use custom collectors to convert it’s output to Prometheus exposition format. We will use the Prometheus client_golang package for this.
Follow the steps given below.
- Define a custom type which will implement the prometheus.Collector interface.
type randomNumber struct {
randomNumber *prometheus.Desc
}
randomNumber is a custom collector. It converts random number to Prometheus metrics.
- Create a descriptor used by Prometheus to store metric metadata.
var (
randomNumberDesc = prometheus.NewDesc(
prometheus.BuildFQName("random", "number", "generated"),
"A randomly generated number",
nil,
prometheus.Labels{
"app": "random_number_generator",
},
)
)
prometheus.NewDesc() takes metrics name, help text, variable and constant labels as arguments and creates a prometheus.Desc{}. This descriptor is used by prometheus to store immutable metrics metadata. The number of labels used defines the number of timeseries to be collected (more on this later).
- Implement prometheus.Collector interface
func (r randomNumber) Describe(ch chan<- *prometheus.Desc) {
ch <- r.randomNumber
}
func (r randomNumber) Collect(ch chan<- prometheus.Metric) {
ch <- prometheus.MustNewConstMetric(randomNumberDesc, prometheus.GaugeValue, rand.Float64())
}
Describe() sends all descriptors of metrics collected by randomNumber collector to the channel.
Collect() is called by Prometheus registry when collecting metrics. Metrics is sent via the provided channel.
Both these methods are required to satisfy the prometheus.Collector interface.
- Register custom collector to Prometheus registry.
func main() {
collector := randomNumber{
randomNumber: randomNumberDesc,
}
prometheus.MustRegister(collector)
}
collector is an instance of randomNumber collector registered to the default Prometheus registry.
- Expose metrics via HTTP
var (
addr = flag.String("listen-address", ":8080", "The address to listen on for HTTP requests.")
)
func main() {
flag.Parse()
http.Handle("/metrics", promhttp.HandlerFor(
prometheus.DefaultGatherer,
promhttp.HandlerOpts{},
))
log.Printf("Running server at %s\n", *addr)
log.Fatal(http.ListenAndServe(*addr, nil))
}
This is what complete code looks like,
package main
import (
"flag"
"log"
"math/rand"
"net/http"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
type randomNumber struct {
randomNumber *prometheus.Desc
}
var (
randomNumberDesc = prometheus.NewDesc(
prometheus.BuildFQName("random", "number", "generated"),
"A randomly generated number",
nil,
prometheus.Labels{
"app": "random_number_generator",
},
)
addr = flag.String("listen-address", ":8080", "The address to listen on for HTTP requests.")
)
func (r randomNumber) Describe(ch chan<- *prometheus.Desc) {
ch <- r.randomNumber
}
func (r randomNumber) Collect(ch chan<- prometheus.Metric) {
ch <- prometheus.MustNewConstMetric(randomNumberDesc, prometheus.GaugeValue, rand.Float64())
}
func main() {
flag.Parse()
collector := randomNumber{
randomNumber: randomNumberDesc,
}
prometheus.MustRegister(collector)
http.Handle("/metrics", promhttp.HandlerFor(
prometheus.DefaultGatherer,
promhttp.HandlerOpts{},
))
log.Printf("Running server at %s\n", *addr)
log.Fatal(http.ListenAndServe(*addr, nil))
}
Try it out
Run the exporter in one terminal and curl the http endpoint in another terminal to get the metrics output.
# Terminal 1
go run main.go
2022/02/25 14:28:03 Running server at :8080
# Terminal 2
for i in {1..10};do curl -s localhost:8080/metrics|grep random_number_generated; sleep 2; echo "";done
# HELP random_number_generated A randomly generated number
# TYPE random_number_generated gauge
random_number_generated{app="random_number_generator"} 0.6046602879796196
# HELP random_number_generated A randomly generated number
# TYPE random_number_generated gauge
random_number_generated{app="random_number_generator"} 0.9405090880450124
# HELP random_number_generated A randomly generated number
# TYPE random_number_generated gauge
random_number_generated{app="random_number_generator"} 0.6645600532184904
# HELP random_number_generated A randomly generated number
# TYPE random_number_generated gauge
random_number_generated{app="random_number_generator"} 0.4377141871869802
# HELP random_number_generated A randomly generated number
# TYPE random_number_generated gauge
random_number_generated{app="random_number_generator"} 0.4246374970712657
Recommendations for running exporter
- Deploy exporters along side the application to keep the architecture similar to direct instrumentation.
- Exporters should not perform scrapes on it’s own. Instead, metrics should be pulled when Prometheus scrapes targets based on it’s configuration.
- Failed scrape from collector should return 5xx errors.
