howardjohn's blog

The hidden cost of mpsc channels

Tue, 12 May 2026 00:00:00 +0000

Recently I have been spending a lot of time analyzing and optimizing memory usage in our Rust reverse-proxy, agentgateway. One thing that repeatedly came up was a surprisingly large amount of memory allocated to innocent-looking Tokio mpsc channels.

In my naive understanding, I would have assumed the following allocation pattern:

struct BigStruct {
    data: [u8; 1024],
}
fn main() {
  // Allocates ~1024 bytes
  let _ = tokio::sync::mpsc::channel::<BigStruct>(1);
  // Allocates ~1024*1024 bytes
  let _ = tokio::sync::mpsc::channel::<BigStruct>(1024);
}

However, in practice both of these are wrong: they each allocate 32kb!

Better Kubernetes CRD Docs

Mon, 11 May 2026 00:00:00 +0000

When trying to use any Kubernetes CRD (or even core types) I am pretty much always pulling up a reference doc of some sort to see what fields are available and what they do. However, I have pretty much always found the options for this to be pretty bad. To remedy this, I have been working on a new tool that automatically generates documentation for Kubernetes CRDs, and I wanted to share it here.

Envoy Quirks Part 2: Filter Chain Match

Fri, 17 Apr 2026 00:00:00 +0000

This is part two of a series.

Envoy Quirks Part 1: Clear Route Cache

Envoy Quirks Part 2: Filter Chain Match (this post)

One of Envoy's core features is, of course, its ability to match traffic and route it to the appropriate destination. This is done at two levels generally:

Filter Chain Matches define the top level matching of traffic, matching on attributes of the TCP and TLS handshake like port and SNI.
Route Matches define the matching of HTTP traffic, matching on attributes of the HTTP request like path and headers.

Filter chain matchers are notoriously tricky to get right, and have become a sort of rite of passage for Envoy users to learn how to use them correctly. Lets dig in.

Fast GitHub Actions with Blacksmith

Fri, 10 Apr 2026 00:00:00 +0000

After years of working on a project running our own testing infrastructure, I have been working on Agentgateway utilizing the free GitHub Actions runners.

While the free tier is quite generous, the performance is... not. As someone who has spent a lot of time optimizing build times, I found myself quite disappointed that after going through considerable lengths to optimize our workflows, our builds were still severely bottlenecked by the performance of the GitHub Actions runners.

Stop picking my Go version for me

Fri, 27 Mar 2026 00:00:00 +0000

The go.mod file contains a mandatory go <version number> directive. Since Go 1.21, when a change was introduced to make this include a full patch number (1.21.0 instead of 1.21), a number of projects have started using this wrong, hurting everyone.

The version is the minimum version your project can be compiled with. It is not the version you use to compile your project, but the minimum version that anyone can use to compile your project.

Exploring ListenerSets in Gateway API v1.5

Wed, 25 Mar 2026 00:00:00 +0000

With the retirement of ingress-nginx I've seen a lot of frustrations with Gateway API migrations due to differences in some of the resource models.

The common problem is that Ingress users are often running self-service models, where application teams fully own their ingress configuration, including TLS certificates.

This would look something like so:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: my-issuer
  name: app-a-routes
  namespace: app-a
spec:
  rules:
  - host: app-a.example.com
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: myservice
            port:
              number: 80
  tls:
  - hosts:
    - app-a.example.com
    secretName: myingress-cert

Combined with cert-manager a certificate would automatically be provisioned for app-a.example.com and linked up to the shared Nginx instance. DNS would then be handled with wildcard entries or external-dns.

Envoy Quirks Part 1: Clear Route Cache

Thu, 05 Mar 2026 00:00:00 +0000

This is part one of a series.

Envoy Quirks Part 1: Clear Route Cache (this post)

Envoy Quirks Part 2: Filter Chain Match

Over the years working with Envoy (via Istio), I've come across quite a few quirks and gotchas. I thought it would be fun to share some of them, and how to work around them. Many of these surprise even Envoy experts!

To start things off, lets talk about the "Clear route cache" option present on a number of Envoy filters.

Async DAGs in Bash

Wed, 04 Mar 2026 00:00:00 +0000

Recently I had a need to run a bunch of steps in a bash script, with interweaved dependencies for a CI pipeline.

While a simpler approach would do a linear flow like:

Setup Kubernetes cluster.
Deploy dependencies to Kubernetes.
Build image 1.
Build image 2.
Run tests.

could work, its slow; we don't take advantage that we could build our images while the cluster is setting up, for instance.

Essentially what we want is to be able to run a bunch of steps, declare their dependencies, and have them run as soon as their dependencies are met. This is essentially a DAG (directed acyclic graph) execution.

Interpreting near native speeds with CEL and Rust

Wed, 04 Mar 2026 00:00:00 +0000

When building out Agentgateway, we had a desire to introduce an embedded expression language to allow users to write custom logic to be evaluated at runtime.

This is tremendously useful for a variety of use cases, such as:

Extracting fields to log (request.headers["user-agent"]).
Evaluating authorization conditions (jwt.sub == "admin" || request.path == "/public").
Manipulating fields in requests/responses (x-llm-model: 'json(request.body).model').

and so on.

This provides a powerful way to allow users to customize behavior without needing custom compile-time extensions, external callouts, or complex YAML-based configuration.

Sharing Rust Build Cache

Wed, 18 Feb 2026 00:00:00 +0000

As I ramp up coding agent usage, I found myself wanting to share the Rust build cache across multiple copies of the same project (via git worktrees) to avoid multiple-minute cold builds. This was harder than expected, but I was able to get something working.

First, the failed attempts

Attempt 1: just copy the target directory

This seemed like a good idea, but took about 2 minutes which was almost as bad as doing a cold build in the first place. That won't fly.

Kubernetes v1.35 Pod Certificates won't help your service mesh

Mon, 22 Dec 2025 00:00:00 +0000

The new Pod Certificates feature in Kubernetes v1.35 promises to "drastically simplify service mesh architectures". It will not.

Body-Based Routing in Agentgateway

Wed, 26 Nov 2025 00:00:00 +0000

While typical HTTP request routing happens based on the request headers/path, sometimes routing based on the body is useful. In particular, as AI use cases become more prevalent, the need to route based on the model field of the JSON request body is pretty handy to serve multiple models or route to different external LLMs.

The Gateway API Inference Extension solution to this is to deploy an "external processor" that reads the body, and writes out a header; the header can then be matched as usual. An external processor is basically a service independent of the actual gateway proxy that processes the request. It does this by streaming the request headers/body between the proxy and the external processor.

Agentgateway at Home

Thu, 20 Nov 2025 00:00:00 +0000

How I run Agentgateway on my homelab setup.

CEL is a great language (you are using it wrong)

Wed, 08 Oct 2025 00:00:00 +0000

Common Expression Language (CEL) is a great little language for embedding users' custom logic into an application. However, typically when I discuss it, I get a response something along the lines of "AHHHH!! I HATE CEL!!!"

I, too, have been in that position. However, when building Agentgateway I decided to fully embrace CEL throughout the entire stack, and the results have been great. The problems with CEL are not about CEL itself, but in its usage -- and a lot of usages are poor, giving CEL a bad reputation.

How to build a scalable control plane

Tue, 16 Sep 2025 00:00:00 +0000

I've spent the majority of my career building Istio's control plane, Istiod, with an emphasis on making it highly performant and scalable. And while it has come a very (very) long way, it's still a long way off from what a control plane could be.

It's not alone. When I worked on building an open benchmark of Kubernetes Gateway control planes, I was surprised to find that no implementation met what I felt was a reasonable bar for a "highly scalable and performant control plane".

Rust vs Go

Tue, 16 Sep 2025 00:00:00 +0000

An often overlooked part of the Rust license is that within a year of usage, users are required to make a comparison to their (previously) favorite language. While I am a bit late, I am ready to pay my dues.

Given the overabundance of Rust vs X content, I'll try to cover only areas that haven't been discussed to death.

Lifetimes and Borrows

Did I say I wouldn't cover areas everyone has already talked about? I lied!

You should buy a faster CPU

Mon, 18 Aug 2025 00:00:00 +0000

In the past few years, CPUs have gotten really fast. Shockingly fast! Yet most people are stuck on previous generation mobile chips (whether by choice, or by their companies choice), at a huge detriment to their productivity.

Meanwhile, AI coding subscriptions like Cursor are all the rage these days. I'll skip the debate on exactly how useful these tools are, and focus on the pricing. Cursor is $480/year for the team plan (the cheapest corporate plan), and other providers are around the same, setting a clear price point: engineering productivity is worth at least $500/year.

Your AI workloads still need a service mesh

Tue, 05 Aug 2025 00:00:00 +0000

AI workloads introduce new requirements on networking infrastructure, but the same core requirements that service meshes solve not only remain, but are exacerbated. Your AI workloads still need a service mesh - they just need a better one

[Solo.io Blog] Gloo Mesh, The 100 Million Pod Mesh

Wed, 05 Feb 2025 00:00:00 +0000

Gloo Mesh’s ambient multi-cluster mode sets a new benchmark for scalability.

Exploring the new "go tool" support in Go 1.24

Mon, 20 Jan 2025 00:00:00 +0000

Go 1.24 introduces new support for "Tools", which allows easy consumption of tools (which are written in Go) as a dependency for a project. This could be anything from golangci-lint to protoc-gen-go.

In this post, I will cover usage and limitations.

Basic usage

Adding a tool to a project is nearly the same as a standard runtime dependency, with the additional -tool flag:

$ goimports # I don't have goimports yet!
zsh: command not found: goimports
$ go get -tool golang.org/x/tools/cmd/goimports
go: added golang.org/x/mod v0.22.0
go: added golang.org/x/sync v0.10.0
go: added golang.org/x/tools v0.29.0
$ go tool goimports --help
usage: goimports [flags] [path ...]

Once we add a tool, we can access it by go tool <name>.

Ztunnel through two lenses

Thu, 16 Jan 2025 00:00:00 +0000

Is it a per-node proxy? Is it a sidecar? The reality is somewhere in between

[Solo.io Blog] Introducing Ambient Multi-Cluster Mode to Gloo Mesh

Wed, 15 Jan 2025 00:00:00 +0000

Reimagine service mesh with Istio’s ambient mode—lightweight, efficient, and scalable. Gloo Mesh now extends this innovation to multi-cluster environments, delivering unmatched reliability, simplicity, and scale.

What happens when Ztunnel shuts down?

Fri, 20 Dec 2024 00:00:00 +0000

A deep dive into the termination behavior of Istio ambient mode and how it impacts your applications.

[AWS Blog] Transforming Istio into an enterprise-ready service mesh for Amazon ECS

Wed, 27 Nov 2024 00:00:00 +0000

How Istio ambient mode brings an enterprise-ready service mesh to Amazon ECS.

Scaling Ambient In Your Sleep

Fri, 15 Nov 2024 00:00:00 +0000

How ambient achieves massive scale without toil.

"Zero to Value" in two steps with Istio ambient mode

Tue, 05 Nov 2024 00:00:00 +0000

When we first started designing what eventually became Istio ambient mode, there were many directions we explored, both in terms of implementation, and what our goals were. What resonated most, though, was that we wanted to provide an incredibly easy onboarding story for a subset of functionality. This subset, ultimately, was getting Mutual TLS deployed for all service-to-service communication within a cluster. I talk a bit more about this here.

Since then, I think we have delivered on this promise... and gone even further! In this post, I wanted to highlight some of the areas that I think ambient helps deliver some serious value to users with minimal complexity.

Looking back on "Building Better Controllers" 2 years later

Sun, 03 Nov 2024 00:00:00 +0000

Over 2 years ago, I started working on some ideas to build better Kubernetes controllers. In this post, I wanted to give a bit of a retrospective on how things have gone since then.

Over the years working on Istio and other projects, I observed a number of major issues with controllers:

Most code was about error-prone event handling and state reconciliation, rather than business logic.
Most tests, in turn, were about the same.
This, in turn, made the code extremely complex, brittle, and often incorrect.
This complexity lead to user facing compromise: incorrectness and performance issues.

You might argue I should just write a better controller that is faster and without bugs. Maybe, but probably not.

[Cloud Native Now Blog] How Istio Ambient is Revolutionizing Cloud Connectivity

Fri, 01 Nov 2024 00:00:00 +0000

External Blog Post.

On-Demand Development Environments

Tue, 22 Oct 2024 00:00:00 +0000

Tools to create reproducible development environments are basically everywhere these days, from Development Containers to Nix wrappers to questionable Docker hacks. However, all of these (that I have found) have a common flaw that bothers me: they all require eagerly fetching the entire environment to get anything done.

This kills the premise of these environments providing any easy on-ramp for users when the first step is to download GBs of binaries. Across projects I work on, we have probably 5-10GB of dependencies, but its extremely unlikely a single developer will use more than a fraction of these at a time. Even for repeat contributors, updates to these are not always incremental (though some are), bringing continued pain as time goes on.

Inline (YAML) Langauge Injection in JetBrains IDEs

Fri, 27 Sep 2024 00:00:00 +0000

JetBrains IDEs (IntelliJ, GoLand, etc) have a nifty feature called Language Injection that lets you get full language features when a language is embedded within another. For example, a SQL query within a string within a Go file.

A few of these come out of the box, but they are pretty limited -- I only had some XML ones prior to enabling the Databases plugin which added a few SQL ones. Fortunately, there is the ability to add custom ones. Unfortunately, this is expressed in a proprietary language with, as far as I can tell, zero documentation.

[Solo.io Blog] Egress Gateways Made Easy

Mon, 19 Aug 2024 00:00:00 +0000

How Istio ambient mode greatly simplifies and improves egress gateway use cases.

[Solo.io Blog] HTTP Observability Without Compromises

Tue, 13 Aug 2024 00:00:00 +0000

How Gloo mesh brings HTTP observability to your mesh, without compromises.

I just want mTLS on Kubernetes

Mon, 12 Aug 2024 00:00:00 +0000

An overview of options to deploy mTLS on Kubernetes

NetworkPolicy: the wrong solution to the right problem

Fri, 09 Aug 2024 00:00:00 +0000

Core problems with the API make it a challenging to use in a secure, scalable manner.

Stop Trusting Your Nodes

Fri, 09 Aug 2024 00:00:00 +0000

Zero trust architectures should not treat nodes as highly privileged components.

The Past, Present, and Future of Istio Installation

Fri, 09 Aug 2024 00:00:00 +0000

Istio's installation has a long, winding, complex history, leading to an interesting current state.

In this post, I hope to explain some of the historical context of how we arrived to the current state, and where I think the project is going. This is all my personal perspective and memory of things that happened years ago, so there is likely some divergence from reality.

The Past

When I first started working on Istio in 2019, Istio 1.0 had just been released. The ecosystem was a pretty difference place back then.

Shipping Anything On Go Infrastructure

Fri, 05 Jul 2024 00:00:00 +0000

go run bash-script@latest

Service Mesh Adoption Curve

Mon, 24 Jun 2024 00:00:00 +0000

How we built a best-of-both-worlds experience with Istio ambient mode.

Everything you need to know about Istio installation

Fri, 07 Jun 2024 00:00:00 +0000

Installing Istio... hard could it be? A simple istioctl install is all you need... right?

Ambient and the SPOF Myth

Tue, 04 Jun 2024 00:00:00 +0000

Does Istio ambient introduce a SPOF? No.

Grafana Dashboard Development Workflow

Thu, 30 May 2024 00:00:00 +0000

How I iterate on Grafana dashboards

Podless Kubernetes

Tue, 07 May 2024 00:00:00 +0000

Sidecarless? Why not podless?

Testing a Kubernetes Networking Implementation Without Kubernetes

Thu, 25 Apr 2024 00:00:00 +0000

How Istio tests its networking proxy without Kubernetes, Docker, or root.

Securing Prometheus with Istio Ambient

Mon, 15 Apr 2024 00:00:00 +0000

tl;dr: it just works

Building a better Kubernetes Client

Sat, 23 Mar 2024 00:00:00 +0000

Like most other Kubernetes controllers in, Istio is written in Go and relies on the client-go library. While this provides an excellent low-level building block, usage in higher level code in Istio led to a variety of issues that led us to develop our own higher level, opinionated client for Istio.

This post covers the issues we faced and how we incrementally solved them.

Background knowledge

At a high level, client-go provides a few layers for interactions with the API server:

Advanced Helm Techniques

Tue, 27 Feb 2024 00:00:00 +0000

Rage bait for YAML templating haters

Opinionated Istio Feature Recommendations

Tue, 06 Feb 2024 00:00:00 +0000

Which features I recommend using, or not using, in Istio

Analyzing Go Binary Sizes

Sat, 06 Jan 2024 00:00:00 +0000

In Analyzing Go Build Times, I went over how to analyze and understand Go build times, and what factors impact build times. A close cousin to build times is build sizes.

Large binaries can lead to a variety of issues such as:

Generally, slower build times
Increased costs of storage
Increased costs and time to distribute
Increased memory usage at runtime (more on this in another article, hopefully)

So its generally nice to keep them small.

Truly Extensible Proxies

Fri, 13 Oct 2023 00:00:00 +0000

Exploring an extreme service mesh architecture to maximize extensibility.

Proxy Layers

Thu, 12 Oct 2023 00:00:00 +0000

The OSI model attempts to build a model for network communications, where increasingly high level layers are built upon lower layers. This is only slightly useful in practice, as the real world is not so simple.

In service mesh, generally discussion is reduced to L4 and L7, or TCP and HTTP. This oversimplifies the problem, leading to some confusion.

Thinking in terms of termination

Simply saying "HTTP" is not really clear about what is going on. Instead, I think its more useful to think about what layer we terminate.

Service Mesh Proxy Classes

Wed, 11 Oct 2023 00:00:00 +0000

When looking at service mesh (or even general networking) architectures, the basic idea is to send network traffic through some component, which handles various functionality. This could be authentication, authorization, encryption, observability, reliability, networking, etc. There are a few different classes of components that can do this, though:

Different types of proxy deployments

Native application enhancement. The application itself is compiled in with functionality. This could be something like gRPC (or, even more "meshy", gRPC with xDS), Finagle, Hystrix, etc. Even simply instrumenting your application with metrics and traces could be classified here.
"Sidecar", or running a proxy-per-application, is probably the most common service mesh deployment pattern today, used by Istio, Linkerd, and more.
Per-node proxy; like sidecar, but instead of per-application the proxy is per-node. Each node contains multiple unique workloads, so the proxy is multi-tenant.
Remote proxy. A completely standalone proxy deployment we send some traffic through. This could be correlated to one or many service(s), one or many workload(s), etc -- the correlation between proxies and other infrastructure components is flexible here.

Within each of these, there is 2 actors: a client and a server. This gives us 8 points to insert functionality. Presumably, all 8 will not be used at once -- but its possible. If we are willing to blur the lines a bit, even a traditional sidecar based service mesh utilizes 6 of these! The most rich "service mesh" functionality may exist in the sidecar, but the application itself has some functionality (even if its not terribly rich), and the node does as well (again, this may not be terribly rich -- kube-proxy, for example, has very minimal functionality). And the same is mirror on the client and server side.

User space isn't slow

Sun, 10 Sep 2023 00:00:00 +0000

In-kernel networking solutions, such as WireGuard, are not always faster than user space.

An optimal CI/CD system

Wed, 06 Sep 2023 00:00:00 +0000

Exploring an (unfortunately, hypothetical) CI/CD system for end to end tests on Kubernetes.

Istio Ambient is not a "Node Proxy"

Tue, 22 Aug 2023 00:00:00 +0000

The common messaging around Istio Ambient Mesh is that is a "node proxy."

For example, from The New Stack

... architecture that moves the proxy functionality from the pod-level to the node-level.

While this is technically accurate, it is misleading and really missing the point and benefits of Ambient.

A brief history of service mesh architectures

This skips quite a bit of information, but is close enough.

One of the earlier service meshes on the market was Linkerd 1 - not to be confused with Linkerd 2, which most people just call "Linkerd" today. Linkerd 1 was a per-node proxy that did all the service mesh functionality we know and love, at the node level.

Waypoint Proxies The Hard Way

Tue, 22 Aug 2023 00:00:00 +0000

How to achieve an architecture similar to "Waypoint Proxies" without ambient mesh, or even Istio.

Building a lot of docker images

Fri, 04 Aug 2023 00:00:00 +0000

Fully utilizing buildkit's potentional

GOMAXPROCS and GOMEMLIMIT in Kubernetes

Mon, 31 Jul 2023 00:00:00 +0000

How and why to easily these fields

Saying No In Open Source

Thu, 27 Jul 2023 00:00:00 +0000

As an open source maintainer, I am reviewing roughly 25 ideas per day - whether they are feature requests, design proposals, or pull requests. Inevitably, this leads to saying "No" quite a bit as well.

Usually, this is in a softer for like "No, not right now", "No, not in its current form", or "No, unless someone else approves", but the outcome is the same: the change is not accepted, and the emotional impact on the reviewer and contributor is similar.

Useful tools for Kubernetes

Wed, 12 Jul 2023 00:00:00 +0000

Most people using Kubernetes extensive have already defined alias k=kubectl and are using tools like kubectx. As someone really lazy though, I have found/developed a few less common tools to help work with Kubernetes efficiently.

`kubectl apply` from clipboard

This relies on zsh, and uses the zle to define a custom command.

function zle_apply {
  LBUFFER=" cat <<EOF | kubectl apply -f -
$(xclip -se c -o)
EOF"
  CURSOR=31
}
zle -N zle_apply; bindkey "^k" zle_apply

This defines a function and binds it to Ctrl+k.

Tracing shell scripts with OpenTelemetry

Tue, 11 Jul 2023 00:00:00 +0000

A little bit of OpenTelemetry, a lot of shell hackery

Zero allocations metrics with opentelemetry-go

Tue, 11 Jul 2023 00:00:00 +0000

In the past, Istio has suffered from performance issues from OpenCensus, which was used for metrics reporting. At extremes, we saw up to 20% of CPU spent just on incrementing various metrics. This was mitigated to some extent by batching metrics updates, optimizing OpenCensus itself, and caching parts of our OpenCensus usage.

At best, we got down to roughly 600ns and 3 allocations per metric update.

As OpenCensus is now deprecated, I have been looking into migration to OpenTelemetry - and hoping to avoid these issues this time around.

Analyzing Go Build Times

Sat, 24 Jun 2023 00:00:00 +0000

Go is often praised for its fast build times. While they are pretty quick, they are slow enough that I spend a lot of time waiting for them, enough that it prompted me to go down the rabbit hole of thoroughly analyzing them. This post covers all aspects of what makes Go builds fast or slow.

Throughout this blog, we will use Istio as an example of real-world codebase.

For reference on its size:

CRD Versioning

Wed, 07 Jun 2023 00:00:00 +0000

How versioning works in Kubernetes, especially with CustomResourceDefinitions, is a common source of confusion.

The documentation is pretty comprehensive but a bit complicated. This post aims to give a simple description of how versioning works and dispel some misunderstandings.

There is only one primary version

Consider a CRD with versions alpha and beta. A user can create and view either resource version.

Intuitively, they must be distinct things -- they are not.

LTS and Rolling Releases

Fri, 26 May 2023 00:00:00 +0000

Across the ecosystem, a variety of software support policies can be found, where "support" can mean, bug fixes, security patches, and sometimes technical support.

However, they can be roughly categorized into two types:

Rolling release: only latest release supported. At an extreme, this is the HEAD git commit.
LTS (Long Term Support): Support latest N versions. Sometimes specific versions have extended support. "Long" is relative: Kubernetes supports a version for around a year, while RHEL does for 10 years.

Enterprises love LTS

Historically, enterprise users have favored -- if not demanded -- LTS software.

Outbound sidecars are not secure enforcement points

Mon, 22 May 2023 00:00:00 +0000

It is a very common misconception that egress policies in Istio can be used for security purposes. This is not true. Despite repeatedly explaining this (and documenting it), I still often see people that do not believe it, and that they can just add one more check to lock things down.

In this post, I will show a variety of ways to bypass any possible check, and prove that these policies cannot be used as secure policies.

Ergonomic Map in Go

Tue, 31 Jan 2023 00:00:00 +0000

In many languages, on of the things I find myself doing is maping over a list to extract some field. For example, coverting a []Person to []Name.

Most languages these days have ways to do this pretty easily:

Kotlin:

people.map { it.Name }

JavaScript:

people.map(p => p.Name)

Rust:

people.map(|p| p.Name)

Scala:

people.map(_.Name)

With generics, Go finally can do this in a type safe manner:

Map(people, func(t Person) string {
  return t.Name
})

... but we immediately stand out amongst other languages as having ugly, verbose syntax.

Building Docker Images Fast (by not using Docker)

Mon, 23 Jan 2023 00:00:00 +0000

This post follows the journey Istio has taken over the years to optimize our docker image builds. While there is some useful tips to take out of this, this is not intended to be a guide on how a project should build images - the steps taken here go far past the needs of a typical project, optimizing exclusively for speed (and fun) regardless of the complexity to maintain.

For background, over the years Istio has consisted of ~10-20 docker images (many are for tests only) made up of ~10-15 Go binaries and various static files. We also have a few variants (debug and distroless) and architectures (amd64 and arm64). Aside from CI which is building thousands of these images daily, building images quickly is important for the inner development loop. While I try to run things locally where possible, in many cases each minor code change is built and loaded into a local Kubernetes cluster to more closely resemble a real world deployment. This makes image build time critical for efficient development

Kubernetes Operators

Mon, 19 Dec 2022 00:00:00 +0000

Kubernetes Operators are powerful tools when used right, pushing complex operations from human operators to code. This sounds great - and is in some cases - but often the tradeoffs operators expose are not taken into account, both by operator developers and users. This article goes over my take on when operators are useful and not, and what makes a good operator.

A basic installation operator

A common feature amongst almost all operators is the ability to deploy Kubernetes resources. For example, lets consider a hypothetical nginx operator.

Local Gateway

Fri, 12 Mar 2021 00:00:00 +0000

When deploying a proxy outside of Kubernetes, users typically reach for solutions like nginx or Traefik; it's pretty rare to see Kubernetes-native load balancers, like Istio, used outside of the Kubernetes context. While it is not documented anywhere on istio.io, it is completely possible to set up an Istio Gateway outside of Kubernetes - not just running out of the cluster, but not connected to any Kubernetes api-server whatsoever.

In this post, we will set up Istiod, and Istio ingress gateway, and a test application using docker-compose. The full configuration can be found in howardjohn/local-istio-gateway. Setting up a full mesh outside of Kubernetes is possible, but outside of the scope of this post.

Istio PR Approvers

Sat, 06 Mar 2021 00:00:00 +0000

The following graph shows PR approvals for istio/istio since 2020.

DevA 100 <-> DevB 50 indicates DevB approved 100 PRs from DevA, and DevA approved 50 PRs from DevB.

The data was collected from our policy bot data and graphed using flourish:

gcloud spanner databases execute-sql main --instance istio-policy-bot --project istio-testing --format=json --sql='select prr.Author as Approver, pr.Author as Author, count(*)
from PullRequestReviews as prr
join PullRequests as pr on pr.PullRequestNumber = prr.PullRequestNumber
where SubmittedAt > "2020-01-01T00:00:00.00Z" and prr.state = "APPROVED" and prr.RepoName = "istio"
group by Approver, Author' \
| jq '.rows[] | select((.[2]|tonumber) > 5) |  .[1] + "," + .[0] + "," + .[2]' -r

If the dynamic graph doesn't work a static version can be found here.

Talks

Mon, 01 Jan 0001 00:00:00 +0000

talks

howardjohn's blog

The hidden cost of mpsc channels

Better Kubernetes CRD Docs

Envoy Quirks Part 2: Filter Chain Match

Fast GitHub Actions with Blacksmith

Stop picking my Go version for me

Exploring ListenerSets in Gateway API v1.5

Envoy Quirks Part 1: Clear Route Cache

Async DAGs in Bash

Interpreting near native speeds with CEL and Rust

Sharing Rust Build Cache

Attempt 1: just copy the target directory

Kubernetes v1.35 Pod Certificates won't help your service mesh

Body-Based Routing in Agentgateway

Agentgateway at Home

CEL is a great language (you are using it wrong)

How to build a scalable control plane

Rust vs Go

Lifetimes and Borrows

You should buy a faster CPU

Your AI workloads still need a service mesh

[Solo.io Blog] Gloo Mesh, The 100 Million Pod Mesh

Exploring the new "go tool" support in Go 1.24

Basic usage

Ztunnel through two lenses

[Solo.io Blog] Introducing Ambient Multi-Cluster Mode to Gloo Mesh

What happens when Ztunnel shuts down?

[AWS Blog] Transforming Istio into an enterprise-ready service mesh for Amazon ECS

Scaling Ambient In Your Sleep

"Zero to Value" in two steps with Istio ambient mode

Looking back on "Building Better Controllers" 2 years later

[Cloud Native Now Blog] How Istio Ambient is Revolutionizing Cloud Connectivity

On-Demand Development Environments

Inline (YAML) Langauge Injection in JetBrains IDEs

[Solo.io Blog] Egress Gateways Made Easy

[Solo.io Blog] HTTP Observability Without Compromises

I just want mTLS on Kubernetes

NetworkPolicy: the wrong solution to the right problem

Stop Trusting Your Nodes

The Past, Present, and Future of Istio Installation

The Past

Shipping Anything On Go Infrastructure

Service Mesh Adoption Curve

Everything you need to know about Istio installation

Ambient and the SPOF Myth

Grafana Dashboard Development Workflow

Podless Kubernetes

Testing a Kubernetes Networking Implementation Without Kubernetes

Securing Prometheus with Istio Ambient

Building a better Kubernetes Client

Background knowledge

Advanced Helm Techniques

Opinionated Istio Feature Recommendations

Analyzing Go Binary Sizes

Truly Extensible Proxies

Proxy Layers

Thinking in terms of termination

Service Mesh Proxy Classes

User space isn't slow

An optimal CI/CD system

Istio Ambient is not a "Node Proxy"

A brief history of service mesh architectures

Waypoint Proxies The Hard Way

Building a lot of docker images

GOMAXPROCS and GOMEMLIMIT in Kubernetes

Saying No In Open Source

Useful tools for Kubernetes

kubectl apply from clipboard

Tracing shell scripts with OpenTelemetry

Zero allocations metrics with opentelemetry-go

Analyzing Go Build Times

CRD Versioning

There is only one primary version

LTS and Rolling Releases

Enterprises love LTS

Outbound sidecars are not secure enforcement points

Ergonomic Map in Go

Building Docker Images Fast (by not using Docker)

Kubernetes Operators

A basic installation operator

`kubectl apply` from clipboard