Prometheus is an Open-Source monitoring system and time-series database. In this post, I am going to dissect some of the Prometheus internals – especially, how Prometheus handles scraping other components for their metrics data.

This article assumes you have basic knowledge of both Prometheus and Go.

Run Groups

Let’s start with a quick detour about oklog/run which is used heavily within Prometheus. Oklog is a mechanism to manage goroutines.

An oklog goroutine will be started with the following syntax:

var g run.Group
g.Add(func() error {
	// Perform work
}, func(error) {
	// Cancel the goroutine
})

The first function will be executed as the goroutine. The second one will be executed when the first needs to be canceled. Whenever any of the goroutines in the group is stopped, all other goroutines will be stopped too.

This allows for a fine management of goroutines, as we can start multiple ones and know that if any crashes, we won’t enter a broken state, as the entire process will be restarted.

As an example, this group in cmd/prometheus/main.go (later refered to as main.go) handles SIGTERM signals, and exits whenever it receives one. As all goroutines are exited whenever any of them stops, this guarantees that a SIGTERM will stop the process fully.

Managers

Back to our area of interest, the scrapers. Prometheus has many structs it calls “managers”.

This post is looking into the scrape manager. However, there is a lot of cross-communication between all of them. So it is extremely interesting to have a basic idea what each of them does.

notifierManager
This manager receives alerts, and passes them through to the appropriate alertmanagers, effectively sending the notification for that alert.

discoveryManagerScrape and discoveryManagerNotify
Those managers, which share the same implementation handle updating scrape and alert targets when a service discovery provider detects a change.

scrapeManager
This manager handles scraping HTTP metric endpoints. We will see it in details later in this article.

ruleManager
This manager handles regularly looking at recording and alerting rules and calls the notifier manager when an alert is raised.

Config Reloading

In order to properly understand the scrape manager, we need to take another detour into config reloading. A lot of things in Prometheus revolve around config reloading, since that can happen from a SIGHUP, from an interaction in the web interface, and targets can even be added/removed by a service discovery provider

Every manager implements a variation of the ApplyConfig method, which is executed whenever a full configuration reload is required.

The scrape and notifier manager also listen for their respective discovery manager to send them updated target groups.
Hence, scrapeManager.Run takes a channel provided by the discovery manager as an argument.

This channel gets a message which contains all target groups whenever any of the providers updates its targets list.

For reference, each provider runs with a shared channel.
When that channel gets a message, the configuration is updated in memory and the triggerSend channel is triggered, effectively triggering a scrape sync.

Scraping Manager

A scrape synchronization will be caught by the scrape manager, which manages multiple scrape loops.

In this section, we are going to refer to the following simple configuration:

scrape_configs:
	- job_name: "file-discovery"
		file_sd_configs:
			- files: ['./targets.json']
	- job_name: "static"
		static_configs:
			- targets: ['localhost:9090']

Let’s go through this scrape manager in order of execution.
We first need to initialize it. The initialization process doesn’t do much outside of initializing the struct in memory.

ApplyConfig

Right after bootstrapping its time-series database, Prometheus will trigger a config reload, which will effectively be a load since it’s the very first one.

When that happens, the scrape manager’s ApplyConfig method will run.

This method loops through all scrape_configs sections of our configuration. In the example above, that will include our 2 configurations.
One is discovered by reading targets.json regularly and applying updates. The other is static and cannot change unless we have a full configuration reload.
The configuration will then be stored within the manager’s struct.

On a first configuration load, this is all it will do. Each scrape config will get a matching service discovery initialized which will fetch the initial configuration and pass it to the manager a bit later on.

Even the static configuration is technically a service discovery, even though it only triggers a configuration reload once, when it’s started.

On a configuration reload, ApplyConfig would also loop through every existing scrape pool and delete/reload the ones which have changed.

Run and targets reloading

Once the entire configuration as been loaded, Run will be executed.

The first thing Run does is execute reloader in another goroutine. It listens for requests to reload the targets configuration and process them.
That reload can be triggered by any service discovery provider.
As an example, the StaticProvider is a very simple one, since it only needs to run, and never has to update anything.

The reload method, which performs the actual job of reloading the targets will loop through all the target sets that were provided.

For each of those targets, it finds the appropriate pool, creating it if required, and triggers a pool sync.

One scrape pool will be created for each different job we have defined in our configuration. So in our case, we will get two: one for the file-based service discovery and one for our static config.

And that’s all the manager does, manage a pool of scrapers which will be able to regularly fetch the metrics endpoint of their targets.

The Scrape Pool

Creating a new scrape pool creates a new instance of the scrapePool struct.
The Sync method mentioned above loads the data.

When syncing, a list of target groups is provided, each of which can include multiple targets. Sync loops through them and turns them into an array of Target.

Once that array is setup, we move into the private sync method, which loops through each target again, deduplicates them, creates a new scrape loop, and runs it in a goroutine.

Scrape Loop

A scrape loop is an interface to run the infinite loop which will perform the scraping at the interval specified in the configuration. Each loop runs in a separate goroutine, and can be cleanly canceled at any moment if the configuration is reloaded, or the process is stopped.

It is provided with a targetScraper from the scrape pool.

The run method creates an infinite loop which on each iteration, calls the target scraper to fetch the data into a buffer, reads that buffer and parses the content into metrics data to be stored.

When called, the target scraper is the one actually making the HTTP request and reading the body.

At the end of the iteration, the loop waits for the configured polling interval before moving to the next iteration.

And “just like that”, we made an HTTP request to a metrics endpoint somewhere, read its content and analyzed it.

Of course, this is just scratching the surface.
Body parsing, storage and alert triggering bring their own set of complexity that we won’t cover in this article.

Conclusion

When I started reading and trying to understand the Prometheus code base, I found it quite tricky.
I think this is due to all the components communicating with each other, making it hard to understand one by itself, as it requires a lot of context around how the others are working.

Once I started understanding that context however, it really started making sense, and to see quite well architected.