It is a very common misconception that egress policies in Istio can be used for security purposes. This is not true. Despite repeatedly explaining this (and documenting it), I still often see people that do not believe it, and that they can just add one more check to lock things down.
In this post, I will show a variety of ways to bypass any possible check, and prove that these policies cannot be used as secure policies.
Note: this focuses on Istio, but most comments apply to any sidecar-based service mesh. These comments generally do not apply to node level approaches, like CNI's NetworkPolicy.
Inbound vs Outbound
The below diagram shows the two types of traffic in Istio, Inbound and Outbound:
Various configurations apply on the different paths. For example, all VirtualService routing applies to outbound traffic, while all AuthorizationPolicy applies to inbound traffic.
The sidecar runs in the same trust domain as the user application. As a result, the application can bypass its own sidecar. This means attempting to apply a policy on the outbound path and treating it as secure is flawed.
AuthorizationPolicy, which applies on inbound, is secure. While an application can bypass its own sidecar, it cannot bypass another Pod's sidecar, so the policy is guaranteed to be enforce. An attacker cannot bypass an inbound policy (or if they did, it would be a CVE).
Ways to bypass your sidecar
Below are is an incomplete list of ways an application can bypass its own sidecar, thus negating the ability to securely enforce egress policies.
- Do not run a sidecar
This is the simplest way. Just don't use a sidecar. This can be done by opting out of injection, among other approaches.
- Disable redirection setup
The sidecar works by capturing all traffic via iptables configured in the pod network namespace. This is controlled by a variety of configuration knobs. Some of these can be used to restrict the redirection to only certain traffic. This can be used to bypass the sidecar.
For example:
annotations:
traffic.sidecar.istio.io/excludeOutboundPorts: "80"
Would allow us to bypass the outbound sidecar for all traffic on port 80.
- Disable redirection after the fact
Similar to above, we can let the iptables to setup as usual, but then modify it later with our own calls to iptables. This does require NET_ADMIN, though.
- Customize the injected pod spec
Similar to just not using a sidecar, we can modify what we run as the sidecar.
This can be done with istioctl kube-inject and modifying the output by-hand, or using customized injection.
- Run our own proxy
Redirection works by sending all traffic to a port (15001), which is typically handled by the Istio sidecar proxy. However, we can simply run our own proxy on the same port, and let it do whatever we want (such as forwarding all traffic).
To avoid port conflicts, this could be done by starting faster than Istio or by just killing Istio's proxy and running ours.
Killing the existing proxy could be done with shareProcessNamespace: true and kill, or curl -X POST localhost:15000/quitquitquit.
- Don't use TCP
Istio only captures TCP traffic. Other traffic, such as UDP, will not go through the proxy at all.
- Don't use IPv4
Depending on the configuration, Istio may only configure redirection for IPv4 traffic. Using IPv6 traffic can bypass the proxy as well.
What should I do instead
As you can see, there is no way to feasibly enforce an application does not bypass its own sidecar. No matter what safeguards we attempt to add, there are ways to bypass them, and surely more than what is listed here.
So what can we do?
The easiest option is to just accept this, and not rely on it for security purposes.
If you do need to enforce egress policies in Istio, the only secure way is to use an Egress Gateway. This is secure because (and only if) we use NetworkPolicy in tandem, to ensure all traffic flows through the gateway. This differs from the sidecar, as the egress gateway is not in the same trust domain as the application.
What about REGISTRY_ONLY
The most common offender of this is REGISTRY_ONLY, which is often misconstrued as an egress firewall. Everything described above applies to REGISTRY_ONLY; do not use it for security purposes.