Agentgateway

While typical HTTP request routing happens based on the request headers/path, sometimes routing based on the body is useful. In particular, as AI use cases become more prevalent, the need to route based on the model field of the JSON request body is pretty handy to serve multiple models or route to different external LLMs. The Gateway API Inference Extension solution to this is to deploy an "external processor" that reads the body, and writes out a header; the header can then be matched as usual. An external processor is basically a service independent of the actual gateway proxy that processes the request. It does this by streaming the request headers/body between the proxy and the external processor. ...

I've spent the majority of my career building Istio's control plane, Istiod, with an emphasis on making it highly performant and scalable. And while it has come a very (very) long way, it's still a long way off from what a control plane could be. It's not alone. When I worked on building an open benchmark of Kubernetes Gateway control planes, I was surprised to find that no implementation met what I felt was a reasonable bar for a "highly scalable and performant control plane". ...