For various purpose, I maintain quite a few of my own docker images. Sometimes because the upstream doesn't have their own, I don't like their version, they are not multi-architecture, I need multiple binaries in one image, etc - whatever the reason, I have a lot.

In doing this, I have found buildx's bake command invaluable. Since I think its not well-known and underused, I thought I would share some info.

For reference, here is the full file structure (with many services omitted):

.
├── build.sh
├── docker-bake.hcl
├── README.md
├── remote.hcl
├── shell
│   ├── bashrc
│   ├── Dockerfile
│   └── inputrc
└── service2
    └── Dockerfile

Buildx Bake

bake is, in my opinion, one of the nicest parts of the whole buildx setup. If you are ever building >1 image at a time, bake can probably do it faster - and avoid terrible scripts/Makefile/etc.

Here is my full bakefile to build all images:

variable "hubs" {
  default = ["localhost:5000"]
}

variable "platforms" {
  default = ["linux/amd64", "linux/arm64"]
}

images = [
  {
    name    = "shell"
    version = "v0.0.7"
  },
  {
    name         = "nettools"
    version      = "v0.0.6"
    dependencies = ["shell"]
  },
  # More images go here!
]

target "all" {
  # The secret sauce
  matrix = {
    item = images
  }
  name    = item.name
  context = item.name
  args    = {
    VERSION    = item.version
    VERSIONNUM = trimprefix(item.version, "v")
  }
  tags = [
    for x in setproduct(hubs, ["latest", item.version]) : join("/${item.name}:", x)
  ]
  # How dependencies work
  contexts  = {for x in lookup(item, "dependencies", []) : "howardjohn/${x}" => "target:${x}"}
  platforms = lookup(item, "platforms", platforms)
  # Obscure way to say "Push this to a registry"
  output    = ["type=registry"]
}

There are a few important bits here.

  • matrix is the crux of the whole thing, and how we avoid a ton of duplication. Normally, you would have to define a target for each image. Since very little changes between each image, this gets redundant fast. matrix is more or less a for loop to define targets.
    • inherits is fairly similar. I didn't use it here since it doesn't let me easily de-duplicate things that vary with each target (like the version).
  • contexts is another cool part. Often my images depend on each other. For example, above I have a shell image which sets up a nice base image. The nettools image has FROM howardjohn/shell. The issue with this is that if I am building both images at once, I may be referencing an old version (or the image may not even exist yet). contexts lets us override where we pull an image from - in this case, to the result of the shell target that we just built. Docker will handle the execution ordering for us.

And that's about it! With this, we can run docker buildx bake and we get all of our images.

I also like to keep a separate remote.hcl file I pass in addition (with -f) to push to a variety of additional remote registries.

Skipping builds

One problem with this is we rebuild everything every time. This is often cached, but still wasteful - and the cache is not permanent anyways. In addition, I intentionally use a floating latest tag and a specific version tag so that the tagged versions are immutable. Building every time would violate this, since Dockerfiles (or, at least mine) are not immutable.

To solve this, I wrap things in a small script that checks each image before building it. If the image already exists, its skipped entirely.

#!/bin/bash

WD=$(dirname "$0")
WD=$(cd "$WD"; pwd)

set -u

HUBS=""
TARGET="${TARGET:-all}"
DRY_RUN="${DRY_RUN:-0}"
FORCE="${FORCE:-0}"
REMOTE="${REMOTE:-0}"

while (( "$#" )); do
  case "$1" in
    -n|--dry-run)
      DRY_RUN=1
      shift
      ;;
    -f|--force)
      FORCE=1
      shift
      ;;
    -r|--remote)
      REMOTE=1
      shift
      ;;
    -t|--target)
      if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then
        TARGET=$2
        shift 2
      else
        echo "Error: Argument for $1 is missing" >&2
        exit 1
      fi
      ;;
    *) # unsupported flags
      echo "Error: Unsupported flag $1" >&2
      echo "  -n|--dry-run: do not build or push" >&2
      echo "  -f|--force: build even if there are no changes" >&2
      echo "  -r|--remote: push to remote" >&2
      echo "  -t|--target: (repeated) target to push" >&2
      exit 1
      ;;
  esac
done

EXTRA=""
if [[ "${REMOTE}" == 1 ]]; then
  EXTRA="-f remote.hcl"
fi

_green='\e[0;32m'
_yellow='\e[0;33m'
_clr='\e[0m'
function yellow() {
  echo -e "$_yellow"$*"$_clr"
}
function green() {
  echo -e "$_green"$*"$_clr"
}

# image_exists returns 1 if the image is missing, 0 if present
function image_exists() {
  if [[ "${FORCE}" == 1 ]]; then
    return 1
  else
    crane manifest "${1:?image name}" &> /dev/null
  fi
}

definition="$(docker-buildx bake all --print --progress=none -f docker-bake.hcl $EXTRA)"
needed=()
for target in $(<<<$definition jq -r '.group.all.targets[]'); do
  if [[ "${TARGET}" != "${target}" && "${TARGET}" != "all" ]]; then
    continue
  fi
  images=($(<<<$definition jq -r ".target[\"${target}\"].tags[]"))
  for image in ${images[@]}; do
    image_exists "$image" &
  done
  need=0
  for image in ${images[@]}; do
    wait -n # Wait for one tasks
    res=$?
    if [[ $res -ne 0 && $need -eq 0  ]]; then # Image is missing... we need to build it
      needed+=("$target")
      yellow "Building ${target}"
      need=1
      # let remaining complete so our next exit wait works
    fi
  done
  [[ $need -eq 0 ]] && green "Skipping ${target}"
done


if [[ ${#needed[@]} == 0 ]]; then
  yellow "No images to build"
  exit 0
fi

if [[ "${DRY_RUN}" == 1 ]]; then
  yellow "Skipping build due to dry run"
  docker-buildx bake ${needed[@]} --print --progress=none -f docker-bake.hcl $EXTRA
  exit 0
fi

docker-buildx bake ${needed[@]} -f docker-bake.hcl $EXTRA

This isn't perfect but works well enough.