This post follows the journey Istio has taken over the years to optimize our docker image builds. While there is some useful tips to take out of this, this is not intended to be a guide on how a project should build images - the steps taken here go far past the needs of a typical project, optimizing exclusively for speed (and fun) regardless of the complexity to maintain.

For background, over the years Istio has consisted of ~10-20 docker images (many are for tests only) made up of ~10-15 Go binaries and various static files. We also have a few variants (debug and distroless) and architectures (amd64 and arm64). Aside from CI which is building thousands of these images daily, building images quickly is important for the inner development loop. While I try to run things locally where possible, in many cases each minor code change is built and loaded into a local Kubernetes cluster to more closely resemble a real world deployment. This makes image build time critical for efficient development

The beginning: naive docker building

Note: I don't think Istio actually ever did this, but its where most people start so it seems relevant to include it.

A basic Dockerfile for building a Go image can be found in 1,000s of "Getting Started With Docker" posts. An example from Docker:

FROM golang AS build
COPY go.mod ./
COPY go.sum ./
RUN go mod download
COPY *.go ./
RUN go build -o /app

FROM scratch
WORKDIR /
COPY --from=build /app /app
ENTRYPOINT ["/app"]

This uses a multi-stage build which is the bare minimum for making an acceptable image these days.

While this gets the job done and is common sight throughout projects, it's not very fast. We need to do 10 docker build and docker push commands in sequence to build the full project, and any minor change can invalidate the build caches.

Fortunately, we can do a lot better!

Building outside of docker

Rather than building everything inside of docker, we realized we could simply prepare all of our artifacts outside of docker, then copy them in during the build.

A very simplified version of a Make target would look like this:

docker.app1:
	go build ./app1
	cp Dockerfile.app1 app1 app1-dep docker-build/app1
	cd docker-build/app1
	docker build . -t app1
	docker push app1

This has a few benefits:

  • In a normal docker build, all inputs to the build need to be copied into the docker context. For a large repository, this can add substantial overhead.
  • Build caches can be persisted across different images, as well as with other builds (running locally, IDEs, etc).

While the consistent build environment of docker is lost, this wasn't a huge concern for us. Istio has its own build system that allows executing arbitrary commands in a docker environment, and Go binaries are fairly trivial to build anyways.

This represents where Istio started out. A full build took about 10 minutes on a powerful machine with this approach.

Collapsing build invocations

As one benefit of building outside of docker, we have a bit more flexibility in how we build. It turns out that:

$ go build /.app1 ./app2

Is nearly 2x faster than:

$ go build ./app1
$ go build ./app2

When you have 10 binaries, this shaves off a large amount of time on the build; initial testing shows a ~5x improvement.

Buildx

Docker buildx is the next generation docker builder. From my perspective, it is strictly better than using docker build, and often as simple as replacing with docker buildx build.

buildx allows far greater parallelization (more on this in the next section), advanced features, building and pushing together, a pretty CLI, and multi-architecture building support.

In most (all?) cases, images can be built and pushed faster simply by using buildx. However, it also has advanced functionality to push things further. For example, we can mount cache volumes during our build to re-use the Go build cache between builds, even when the layer is invalidated:

FROM golang AS base
COPY go.* ./
RUN --mount=type=cache,target=/go/pkg/mod \
	go mod download
RUN --mount=target=. \
	--mount=type=cache,target=/go/pkg/mod \
	--mount=type=cache,target=/root/.cache/go-build \
	go build -o /app .
# ...

Bake file

In my opinion, the best part of the new buildx system is the new bake command. This is seemingly pretty under-hyped, but extremely powerful. Essentially, it is Makefile for docker. A file defines the specification for all the images we may want to build, then docker buildx bake build img1 img2 ... img10 can build some or all of these targets.

For our use case, we didn't actually want to define targets like this, as it didn't fit well with our previous decisions. However, the killer feature is the parallelization. With bake, all images can be built and pushed in parallel.

We built a small wrapper to generate the Bake file definition on the fly, allowing us to have essentially the same developer experience we had with normal docker, but with parallelization. This led to a 10x improvement in build times.

Recap

At this point, our builds were in a fairly solid state. A single image could be built in pushed in ~10 seconds, allowing for a fairly tight development cycle. This was about 4s for the binaries to build and 6s for the docker image. However, the whole process still felt a bit wasteful.

Just to deploy a single image, we need to:

  • Build the Go binary.
  • Copy the Go binary (and some other small files) to our "staging" folder.
  • Copy the staging folder into the docker build context.
  • Copy each file from the context into its final location in the image.
  • Compress the entire image and send it (over localhost) to a local docker registry.
  • Copy and decompress (pull) the image from the local registry onto the Kubernetes node.

Thats a lot of copying!

Not satisfied with the current situation, I explored a variety of options for improvements, but fell short. We were going to have to re-invent the wheel.

Building images manually

Docker/OCI images are not that complicated. Ignoring a few details, an image essentially consists of a few "layers", which are essentially just tarballs. Docker adds a lot of infrastructure around constructing those in consistent environments, but we already threw all of that away and decided to just use it to copy files around.

On a modern machine, copying ~100mb doesn't take nearly 6s, but because of all the infrastructure that we weren't utilizing in docker, that is what we were experiencing. By sidestepping docker entirely, we can much more efficiently produce these images.

Enter, go-containerregistry, a fabulous library to deal with images directly. It also comes with an extremely handy CLI tool, crane.

Using this library, we were able to build a tool to efficient build a subset of Dockerfiles. Most notable, those without any RUN commands, although there are likely many other quirks; the goal of the tool was to build Istio images quickly, rather than building all images.

Build Spec

First, each image's Dockerfile was parsed and essentially executing as a dry-run. While in docker each command generates a new layer, we were content with a single layer. By following sequences of various COPY, ENV, and other commands, the Parse function can emit the final specification for the build:

type Args struct {
	// Image architecture.
	Arch string

	Env        map[string]string
	Labels     map[string]string
	User       string
	WorkDir    string
	Entrypoint []string
	Cmd        []string

	// Base image to use
	Base string

	// Files contains all files, mapping destination path -> source path
	Files map[string]string
	// FilesBase is the base path for absolute paths in Files
	FilesBase string
}

I thought about just hand-writing the specifications in this format instead of parsing Dockerfiles, but this allowed a single source of truth so that we could also build with the old docker approach. And it was a lot more fun.

Building

With the build spec in place, the build becomes pretty straightforward. We set a bunch of metadata on the image based on the parsed inputs (user, environment variables, command, etc) and assemble a tar based on all the files. Rather than many copies around for each file, we read each file into memory directly. When the target registry is localhost, we also disable compression to optimize things further.

With this in place, we simply use the library to assemble an image consisting of our configurations, the base image layer we selected, and our assembled layer, and push this to our destination registry(s).

All in all, this process is roughly 2x faster than buildx for building a single Istio image.

Future Improvements

  • Use sha256-simd; sha256 computation is a bottleneck on the build.
  • Remove the local registry middleman in the build process (Builder --> Local Registry --> Kubernetes node) by server the images directly from the builder as an ephemeral registry. Note: kind load or similar seem like good options at first, but the end up being slower than pulling.

What should I use?

Building images manually was inspired roughly 90% by "this sounds fun" and 10% practical purposes. I wouldn't recommend it for most cases.

If they meet your use cases, ko and apko are great.

For general purposes, buildx (and bake, if applicable) is generally good enough for most projects. I use this on most of my projects.

References

This project was made possible due to the fantastic work (and in many cases, help provided by) of the folks behind:

Thank you all!