Cicd

After years of working on a project running our own testing infrastructure, I have been working on Agentgateway utilizing the free GitHub Actions runners. While the free tier is quite generous, the performance is... not. As someone who has spent a lot of time optimizing build times, I found myself quite disappointed that after going through considerable lengths to optimize our workflows, our builds were still severely bottlenecked by the performance of the GitHub Actions runners. ...

Recently I had a need to run a bunch of steps in a bash script, with interweaved dependencies for a CI pipeline. While a simpler approach would do a linear flow like: Setup Kubernetes cluster. Deploy dependencies to Kubernetes. Build image 1. Build image 2. Run tests. could work, its slow; we don't take advantage that we could build our images while the cluster is setting up, for instance. Essentially what we want is to be able to run a bunch of steps, declare their dependencies, and have them run as soon as their dependencies are met. This is essentially a DAG (directed acyclic graph) execution. ...

In Analyzing Go Build Times, I went over how to analyze and understand Go build times, and what factors impact build times. A close cousin to build times is build sizes. Large binaries can lead to a variety of issues such as: Generally, slower build times Increased costs of storage Increased costs and time to distribute Increased memory usage at runtime (more on this in another article, hopefully) So its generally nice to keep them small. ...

Exploring an (unfortunately, hypothetical) CI/CD system for end to end tests on Kubernetes.

Fully utilizing buildkit's potentional

A little bit of OpenTelemetry, a lot of shell hackery

Go is often praised for its fast build times. While they are pretty quick, they are slow enough that I spend a lot of time waiting for them, enough that it prompted me to go down the rabbit hole of thoroughly analyzing them. This post covers all aspects of what makes Go builds fast or slow. Throughout this blog, we will use Istio as an example of real-world codebase. For reference on its size: ...

This post follows the journey Istio has taken over the years to optimize our docker image builds. While there is some useful tips to take out of this, this is not intended to be a guide on how a project should build images - the steps taken here go far past the needs of a typical project, optimizing exclusively for speed (and fun) regardless of the complexity to maintain. For background, over the years Istio has consisted of ~10-20 docker images (many are for tests only) made up of ~10-15 Go binaries and various static files. We also have a few variants (debug and distroless) and architectures (amd64 and arm64). Aside from CI which is building thousands of these images daily, building images quickly is important for the inner development loop. While I try to run things locally where possible, in many cases each minor code change is built and loaded into a local Kubernetes cluster to more closely resemble a real world deployment. This makes image build time critical for efficient development ...