After years of working on a project running our own testing infrastructure, I have been working on Agentgateway utilizing the free GitHub Actions runners.
While the free tier is quite generous, the performance is... not. As someone who has spent a lot of time optimizing build times, I found myself quite disappointed that after going through considerable lengths to optimize our workflows, our builds were still severely bottlenecked by the performance of the GitHub Actions runners.
GitHub runner performance
For a long time, we were able to run our longest job, an E2E test that built a rust and go binary, bundled into a docker image, and deployed to a local Kubernetes cluster, in around 6-7 minutes. This was divided up around 1 minute for setup, 4-5 minutes to build, and 1-2 minutes to run the tests.
However, a few weeks ago the performance inexplicably dropped (without any changes to our code), and the same job jumped up to taking 10-11 minutes. This was the trigger for me to finally start looking into alternative runners.
Enter Blacksmith
Guided by this excellent comparison article we ended up looking into Blacksmith as an alternative to GitHub's hosted runners. After a quick test it was pretty clear I was never going to be able to go back: our E2E tests dropped down to 2-3 minutes! That is ~3-5x faster!
Additionally, with pressure taken off our tiny 10GB GitHub cache, we were able to turn caching on for a number of jobs that were previously disabled, speeding those up by 2x (these were non-Linux jobs that we chose to keep on GHA runners for now, though Blacksmith also supports Windows and MacOS runners).
While the cost of Blacksmith is pretty reasonable, its hard to beat "free" for an open source project. Fortunately, Blacksmith was kind enough to sponsor Agentgateway!
This sponsorship was not conditional on this blog post, I just like talking about fast CI pipelines.
If you are doing any significant amount of work on GitHub Actions, I highly recommend giving a runner that has reasonable CPUs: it's worth it!