Docker

Fully utilizing buildkit's potentional

This post follows the journey Istio has taken over the years to optimize our docker image builds. While there is some useful tips to take out of this, this is not intended to be a guide on how a project should build images - the steps taken here go far past the needs of a typical project, optimizing exclusively for speed (and fun) regardless of the complexity to maintain. For background, over the years Istio has consisted of ~10-20 docker images (many are for tests only) made up of ~10-15 Go binaries and various static files. We also have a few variants (debug and distroless) and architectures (amd64 and arm64). Aside from CI which is building thousands of these images daily, building images quickly is important for the inner development loop. While I try to run things locally where possible, in many cases each minor code change is built and loaded into a local Kubernetes cluster to more closely resemble a real world deployment. This makes image build time critical for efficient development ...