Let's not monkey-patch instrumentation

Modern telemetry libraries allow easily configuring auto instrumentation, to automatically gather observability data about frameworks and libraries.

There are two main approaches to architecting those auto-instrumentation libraries. As middlewares/wrappers, or as monkey-patches. I believe middlewares are much better, here’s why.

Throughout this article, whenever talking about an instrumentation library in any language, I am always talking about the OpenTelemetry implementation for that language, and its contrib libraries.

Auto instrumentation is great

Auto instrumentation is an awesome thing, as it allows getting telemetry data for libraries in a much easier, and generic way than what someone would be able to do by themselves.

For example, if you’re writing a ruby application using the redis gem, you can use the redis auto-instrumentation package, which will automatically create spans whenever a redis query is executed.

The two ways auto-instrumentation can be built

There are two main ways auto-instrumentation packages can be architected. As monkey-patches, or as middlewares.

Monkey Patching

When architecting auto-instrumentation as a monkey-patch, you don’t have to do anything except add, and possibly initialize that plugin. It will then inject itself into the library to be used.

In node for example, the instrumentation package allows creating modules that will be self-pluggable, so that requiring the express auto-instrumentation becomes very easy:

const { registerInstrumentations } = require('@opentelemetry/instrumentation');

registerInstrumentations({
	instrumentations: [
		new ExpressInstrumentation()
	]
);

Other languages may do it differently, depending on the capabilities provided by that language. For example, java allows running an OpenTelemetry agent alongside a jar like this:

java -javaagent:opentelemetry-javaagent.jar -jar myapp.jar

Using this way, configuring auto-instrumentation plugins becomes very easy, as there is nothing else to think about regarding the way the codebase is structured.

Middlewares/wrappers

When architecting auto-instrumentation as a middleware, you have not only to add the plugin as a dependency on your project, but also to set it up explicitly. This is the case with the Go otelhttp package, which instruments the net/http library.

When making outgoing HTTP requests, each of them has to specify the proper Transport to get instrumentation. For example:

client := http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}
req, _ := http.NewRequest("GET", "https://dmathieu.com", nil)
res, _ := client.Do(req)
body, _ = io.ReadAll(res.Body)
fmt.Printf("%s", body)

Any outgoing HTTP request that doesn’t set this transport on its HTTP client will not have any instrumentation.

Using this way, configuring auto-instrumentation is a bit harder for legacy applications if they don’t allow easily extending the transports used to make HTTP requests.

Easier is not better

Despite what you may have felt like in my previous wording (except for the article’s title), I am vastly in favor of using auto-instrumentation as a middleware rather than as a monkey-patch.

Observability is not an afterthought

Auto-instrumentation as monkey patches treats observability as a sidecar. You don’t have to think about it until you go to production, and you need that data.
Well, I think it’s more than time we start treating observability as a first-class citizen, something that we look at from the start, and that we architect our apps around.

Psychologically speaking, injecting auto-instrumentation as a monkey patch, or as an external binary means you will always treat it as something optional that you can do without.

There is no such thing as magic

Coming from the ruby world, where monkey-patching can be very common, but also despised, I take a very bad eye at any library that does it. You need to have a very good reason to do so (aka, you don’t have any other choice).

Monkey-patching makes code a lot harder to read. Indeed, anybody can rewrite anything. So whenever you look for the behavior of a method, you can never be fully sure there isn’t something else overriding elsewhere.

Monkey patching also doesn’t usually play nice when other libraries try to monkey-patch as well. When that happens, it becomes very hard to know who runs first, and you may even entirely lose the entire behavior of one of the two libraries.

Modularity matters

Finally, monkey-patching assumes everybody wants to do things the same way, the one intended by the patch.
When making an HTTP request, you may not want to always use the provided instrumentation. Or you may want to add your own middleware on top of the one provided, to add your own behavior.

We can (sometimes) have both worlds

Of course, setting up auto-instrumentation in a legacy application may prove to be hard without any monkey-patching.
So it’s a good solution to start setting things up, before we can refactor anything problematic that prevents us from effectively using middlewares.

If an auto-instrumentation can be injected as a monkey-patch, we should also expect it to be injectable as a middleware, as is the case for example with the rack middleware.
Even though, unfortunately, the primary way to inject this auto-instrumentation is through monkey-patching, the middleware exists as a class which can be manually included, OpenTelemetry::Instrumentation::Rack::Middlewares::TracerMiddleware.

Some languages, such as Go do not allow monkey-patching. For those, the middleware solution is the only one.

Please give us middlewares, not monkey-patches

At the writing of this article, I’m not sure everybody will agree with my opinion, but I thought I had to share it anyway (if you made it to this part, you didn’t rage close your browser tab, so it may not be so bad).

My hope now is that we can move forward in the future towards being able to use both methods, rather than (in some libraries), being forced into using monkey-patching.