On-Demand Development Environments

Tools to create reproducible development environments are basically everywhere these days, from Development Containers to Nix wrappers to questionable Docker hacks. However, all of these (that I have found) have a common flaw that bothers me: they all require eagerly fetching the entire environment to get anything done.

This kills the premise of these environments providing any easy on-ramp for users when the first step is to download GBs of binaries. Across projects I work on, we have probably 5-10GB of dependencies, but its extremely unlikely a single developer will use more than a fraction of these at a time. Even for repeat contributors, updates to these are not always incremental (though some are), bringing continued pain as time goes on.

What I have long sought is an environment where:

A variety of versioned dependencies are available to all developers, with minimal upfront configuration.
These binaries are fetched on demand when they are needed.

In this post, I'll cover some ideas I have to make this work.

The core: Nix

I don't particularly like Nix for reasons I won't get too far into here, but it does provide a fairly unique property: users can install a bunch of packages at once, including multiple versions of the same package, and they will never interfere with each other. In many ways, this is like having a container-per-package (which is slow and requires running everything in a container which doesn't have access to the environment like we want) or having everything be static binaries (not everything can be made static), but without the downsides.

Because Nix is so rough to use, there are a variety of wrappers around it. Flox, DevBox, and many more provide various degrees of abstraction over Nix. While none provide the experience we want, they do solve one critical problem: Nix makes it hard to reasonably pick a specific version of a binary to use. Many of these tools solve this by providing their own layer on top of Nix.

DevBox's has a very simple API to use, so I have utilized that, while leaving out the rest of DevBox to build a small, simple, bash script to build what we want:

$ devenv help
Usage: devenv <command>
Commands:
  search   - Search for a package
  show     - Show versions for a package
  install  - Add a package to the environment
  help     - Show this help message

The full source:

#!/bin/bash

set -eu

function curldevbox() {
  curl -sL "https://search.devbox.sh/v2/${1}" -H 'Accept: application/json'
}

function show_help() {
    echo "Usage: $0 <command>"
    echo "Commands:"
    echo "  search   - Search for a package"
    echo "  show     - Show versions for a package"
    echo "  install  - Add a package to the environment"
    echo "  help     - Show this help message"
}

function search() {
    curldevbox "search?q=${1:?package}" | jq '.results[] | .name + "\t" + .summary[0:70] + "\t" + .last_updated' -r | column -t -s $'\t'
}

function show() {
    curldevbox "pkg?name=${1:?package}" | jq '.releases[] | .version + "\t"+.last_updated' -r | column -t -s $'\t' | tac
}

function latest() {
    curldevbox "pkg?name=${1:?package}" | jq '.releases[0] | .version' -r
}

function resolve() {
    sys="$(curldevbox "resolve?name=${1:?package}&version=${2:-$(latest ${1})}" | jq '.systems."x86_64-linux"')"
    type="$(<<<"$sys" jq '."flake_installable".ref.type' -r)"
    owner="$(<<<"$sys" jq '."flake_installable".ref.owner' -r)"
    repo="$(<<<"$sys" jq '."flake_installable".ref.repo' -r)"
    rev="$(<<<"$sys" jq '."flake_installable".ref.rev' -r)"
    attr="$(<<<"$sys" jq '."flake_installable".attr_path' -r)"
    echo "${type}:${owner}/${repo}?rev=${rev}#${attr}"
}

function install() {
    binary="${1:?binary name}"
    version="${2:-$(latest ${1})}"
    flag="${3:-}"
    if [[ "${flag}" == "--unfree" ]]; then
    cat <<EOF > bin/${binary}
#!/usr/bin/env sh
# Runs '$binary@$version'
NIXPKGS_ALLOW_UNFREE=1 exec nix run --impure $(resolve "$binary" "$version") -- "\$@"
EOF
    else
    cat <<EOF > bin/${binary}
#!/usr/bin/env sh
# Runs '$binary@$version'
exec nix run $(resolve "$binary" "$version") -- "\$@"
EOF
    fi
    chmod +x bin/${binary}
    echo "Installed $binary@$version!"
}

# Check if an argument was provided
if [ $# -eq 0 ]; then
    echo "Error: No command provided"
    show_help
    exit 1
fi

case "$1" in
    "search")
        shift;
        search "$@"
        ;;
    "show")
        shift;
        show "$@"
        ;;
    "install")
        shift;
        install "$@"
        ;;
    "help")
        show_help
        ;;
    *)
        echo "Error: Unknown command '$1'"
        show_help
        exit 1
        ;;
esac

Now, say I want to run the legendary cowsay:

$ cowsay hello
zsh: command not found: cowsay
$ devenv install cowsay
Installed [email protected]!
$ cowsay hello
 _______
< hello >
 -------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Exactly what we want!

We can see we have the property we want as well: we only pay for what we use:

$ time devenv install nodejs
Installed [email protected]!

real    0.615
$ time devenv install rustc
Installed [email protected]!

real    0.580
$ time nodejs --version
v22.9.0

real    1.997
$ time rustc --version
rustc 1.81.0 (eeb90cda1 2024-09-04) (built from a source tarball)

real    14.417
$ time rustc --version
rustc 1.81.0 (eeb90cda1 2024-09-04) (built from a source tarball)

real    0.684

Here I install rustc and nodejs, and you can see there is only time spent the first time we execute the binary.

How it works

Once we use the DevBox API to resolve to a specific nixpkg reference, we use that to construct a nix run command. We then write a small shell script that execs that:

$ cat bin/python
#!/usr/bin/env sh
exec nix run github:NixOS/nixpkgs?rev=d4f247e89f6e10120f911e2e2d2254a050d0f732#python313 -- "$@"

With this, we can run this transparently.

Automatic setup

One missing piece here is we are just putting (references to) binaries under a bin/ -- but users would need to manually set PATH to use them ergonomically. While this is just a export PATH="./bin:$PATH" away, direnv makes this much nicer:

$ cat .envrc
PATH_add ./bin

With this simple file, direnv will automatically add bin/ to our path when we enter the directory, and remove it when we leave the directory. This enables us to have completely distinct environments for different projects, transparently.

The core: Nix#

How it works#

Automatic setup#

The core: Nix

How it works

Automatic setup