Zero allocations metrics with opentelemetry-go

In the past, Istio has suffered from performance issues from OpenCensus, which was used for metrics reporting. At extremes, we saw up to 20% of CPU spent just on incrementing various metrics. This was mitigated to some extent by batching metrics updates, optimizing OpenCensus itself, and caching parts of our OpenCensus usage.

At best, we got down to roughly 600ns and 3 allocations per metric update.

As OpenCensus is now deprecated, I have been looking into migration to OpenTelemetry - and hoping to avoid these issues this time around.

It turns out, with a bit of care, we can do much better!

Simple usage

A simple metrics update looks like this:

counter.Add(context.Background(), 1)

This is really fast - roughly 15ns and 0 allocations - meaning we can call it frequently without much concern for performance.

With labels

However, we usually need to add some labels (or "attributes" in OpenTelemetry terms), like so:

counter.Add(context.Background(), 1, api.WithAttributes(attribute.String("key", "value")))

This drops the performance quite a bit, back to our optimized OpenCensus levels.

Precomputed labels

However, if we know the labels ahead of time, we can pre-craft the options.

opts := []api.AddOption{api.WithAttributeSet(attribute.NewSet(attribute.String("key", "value")))}
counter.Add(context.Background(), 1, opts...)

With this we are back to 0 allocations, and only spending 50ns. Not bad!

Benchmarks

A full test can be shown below:

func BenchmarkRecord(b *testing.B) {
	provider := metric.NewMeterProvider()
	meter := provider.Meter("test")
	counter, err := meter.Float64Counter("foo", api.WithDescription("a simple counter"))
	if err != nil {
		log.Fatal(err)
	}
	b.Run("no labels", func(b *testing.B) {
		for n := 0; n < b.N; n++ {
			counter.Add(context.Background(), 1)
		}
	})
	b.Run("dynamic labels", func(b *testing.B) {
		for n := 0; n < b.N; n++ {
			counter.Add(context.Background(), 1, api.WithAttributes(attribute.String("key", "value")))
		}
	})
	b.Run("static labels", func(b *testing.B) {
		opts := []api.AddOption{api.WithAttributeSet(attribute.NewSet(attribute.String("key", "value")))}
		for n := 0; n < b.N; n++ {
			counter.Add(context.Background(), 1, opts...)
		}
	})
}

And results:

BenchmarkRecord
BenchmarkRecord/no_labels
BenchmarkRecord/no_labels-8             69666646                14.57 ns/op            0 B/op          0 allocs/op
BenchmarkRecord/dynamic_labels
BenchmarkRecord/dynamic_labels-8         1882459               575.1 ns/op           160 B/op          4 allocs/op
BenchmarkRecord/static_labels
BenchmarkRecord/static_labels-8         27076778                44.74 ns/op            0 B/op          0 allocs/op