Body-Based Routing in Agentgateway

While typical HTTP request routing happens based on the request headers/path, sometimes routing based on the body is useful. In particular, as AI use cases become more prevalent, the need to route based on the model field of the JSON request body is pretty handy to serve multiple models or route to different external LLMs.

The Gateway API Inference Extension solution to this is to deploy an "external processor" that reads the body, and writes out a header; the header can then be matched as usual. An external processor is basically a service independent of the actual gateway proxy that processes the request. It does this by streaming the request headers/body between the proxy and the external processor.

The Body-Based Routing (BBR) processor is thousands of lines of code, and an additional component that needs to be deployed and maintained. Additionally, it enters the critical path of the request. Since I learned about this component, its bothered me quite a bit -- to have such a high cost to do so little work.

Can we do it natively?

When building Agentgateway, I wanted to do better. We had already made extensive use of CEL throughout the proxy, so supporting this was just a natural extension of that work, and fit in neatly. In the end, we are able to natively route based on the body directly in the proxy without any third party components:

apiVersion: gateway.kgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: bbr
spec:
  traffic:
    phase: PreRouting
    transformation:
      request:
        set:
        - name: X-Gateway-Model-Name
          value: 'json(request.body).model'

Results

Aside from the operational simplicity of this approach, the performance impacts are substantial as well. I compared Agentgateway (with native BBR) to Istio (with External Processor) BBR, and found the native approach to be ~4.5x better in throughput and latency.

Note: Istio was selected as a comparison as it is the only Open Source gateway with (documented) BBR support.