While Istio makes it easy to secure your application traffic with mTLS with just a few steps, this doesn't universally work.
One application that comes up often is Prometheus. Securing Prometheus communications with Istio mTLS requires various workloads, each with pretty invasive tradeoffs.
Why is Prometheus special?
Most applications in Kubernetes call Services, generally by a DNS name like curl foo.bar.svc.cluster.local.
Prometheus, on the other hand, directly calls pods by their IP. Worse, it generally calls every single pod, making localized workarounds ineffective.
Istio doesn't handle traffic directly to pods well, breaking Prometheus from getting secured out-of-the-box by Istio.
In the end, most teams that want to secure their Prometheus metrics end up with bespoke solutions.
Securing Prometheus with Istio ambient
Istio ambient mode introduces a new data plane mode for Istio.
Among many other benefits, ambient brings the ability to handle traffic directly to pods.
What does this mean for Prometheus?
Secure mTLS just works:
$ istioctl install --set profile=ambient
$ kubectl label namespace application-namespace istio.io/dataplane-mode=ambient
$ kubectl label namespace prometheus-namespace istio.io/dataplane-mode=ambient
Just like that, all Prometheus metrics scrapes are end to end encrypted by Istio, without application changes required.
Even better, Istio now automatically provides us metrics about this traffic (and all other traffic):
sum(istio_tcp_sent_bytes_total{source_workload="prometheus"}) by (destination_workload, connection_security_policy)
{connection_security_policy="mutual_tls", destination_workload="a-v1"}
{connection_security_policy="mutual_tls", destination_workload="istio-egressgateway"}
{connection_security_policy="mutual_tls", destination_workload="productpage-v1"}
Here we can see this is Mutual TLS encrypted.
Thanks to Prometheus querying everything in the cluster, we can also stress-test our favorite visualization tools:
