One thing I really enjoy about Go is its library and binary distribution system: a simple go install module@version away and you can quickly get an application... as long as its written in Go.

Its like curl | bash, but:

The problem is, I want to deploy my enterprise-grade shell script:

#!/bin/bash
echo "Hello world"

But I want all the goodies Go gives us for free.

It turns out, we can (ab)use Go's infrastructure to distribute more than just Go code!

Distributing shell scripts

Our goal is to make this work (spoiler: it works):

$ go run github.com/howardjohn/shgo/examples/hello-world@latest
Hello World

Go modules expect some Go code (source files, not compiled). The simplest approach we could do is just to do something like:

exec.Command("bash", "-c", "the script...")

There are a few problems here though. First, we are depending on a specific shell interpreter, rather than just accepting any arbitrary executable. It also can run into argument length limits (getconf ARG_MAX); this is extremely large on my machine, but still a concern.

Additionally, we are spawning a subprocess: our enterprise customers demand Hello World without an intermediary. It would be ideal to directly exec() without forking.

Getting a file

Our first problem if we want to execute something is we need a file to execute, but we just have some script in memory.

Fortunately, mmap can help us out here:


// MemFd takes a file name used (mostly for debugging), and the contents the file should contain, and returns the file descriptor.
func MemFd(name string, b []byte) (int, error) {
	fd, err := unix.MemfdCreate(name, 0)
	if err != nil {
		return 0, fmt.Errorf("MemfdCreate: %v", err)
	}

	err = unix.Ftruncate(fd, int64(len(b)))
	if err != nil {
		return 0, fmt.Errorf("Ftruncate: %v", err)
	}

	data, err := unix.Mmap(fd, 0, len(b), unix.PROT_READ|unix.PROT_WRITE, unix.MAP_SHARED)
	if err != nil {
		return 0, fmt.Errorf("Mmap: %v", err)
	}

	copy(data, b)

	err = unix.Munmap(data)
	if err != nil {
		return 0, fmt.Errorf("Munmap: %v", err)
	}

	return fd, nil
}

This gives us a file descriptor, but getting a file name is easy:

// MemFile takes a file name used (mostly for debugging), and the contents the file should contain, and returns the file name.
func MemFile(name string, b []byte) (string, error) {
	fd, err := MemFd(name, b)
	if err != nil {
		return "", err
	}
	// filepath to our newly created in-memory file descriptor
	fp := fmt.Sprintf("/proc/self/fd/%d", fd)
	return fp, nil
}

Finally, we can exec our file:


func Exec(name string, executable []byte) error {
	fp, err := MemFile(name, executable)
	if err != nil {
		return err
	}

	args := []string{name}
	env := os.Environ()
	return syscall.Exec(fp, args, env)
}

Putting it together

With all of this together, we can bundle our shell script into a Go process:

package main

import (
	_ "embed"
	"github.com/howardjohn/shgo"
)

//go:embed script.sh
var script []byte

func main() {
	if err := shgo.Exec("hello-world", script); err != nil {
		panic(err)
	}
}

And now we can do treat it just like a normal go program.

$ go run github.com/howardjohn/shgo/examples/hello-world@latest
Hello World
$ go install github.com/howardjohn/shgo/examples/hello-world@latest
$ hello-world
Hello World

We can version it as well, etc.

You probably shouldn't do this

If it wasn't clear at this point, this is (mostly) for fun -- you probably shouldn't do this. While there are some nice benefits, it does require your users:

  • To have Go installed
  • To run on Linux
  • To accept a ~2mb overhead on every script

If those are acceptable, I suppose its not to terrible... yet.

Bundling Docker

While Docker also provides a convenient way to distribute code, with many of the same benefits the go command has, in 2020 they didn't want to serve thousands of terabytes of Hello World for free anymore. Understandable. Fortunately, GitHub and Go module proxy haven't done the same yet, and allow 50 MB / 500 MB files respectively. Why not use them as our CDN instead?

Warning: unlike the shell script distribution above, this is definitely a bad idea. Aside from having little value (outside of being fun), this is more or less abusing two different services that kindly offer free hosting. Please don't abuse it.

Here is what we are after:

$ docker run hello-world # the uncool way to run a docker image
Hello from Docker!
$ go run github.com/howardjohn/shgo/examples/docker@latest # much better
Hello from Docker!

Using the same techniques as before, we can embed the docker image into our Go binary, then load that into docker and run it.

First we need to get the image into the binary. Docker can do this easily: docker pull hello-world; docker save hello-world | zstd > image.zst.

Next, we build our binary:

package main

import (
	_ "embed"
	"fmt"
	"github.com/howardjohn/shgo"
)

// Fetch with `docker pull hello-world; docker save hello-world | zstd > image.zst`
//go:embed image.zst
var image []byte

var loadScript = `#!/bin/bash
docker load -q -i %s > /dev/null
exec docker run --pull=never hello-world:latest
`

func main() {
	imageFile, err := shgo.MemFile("image", image)
	if err != nil {
		panic(err)
	}
	script := fmt.Sprintf(loadScript, imageFile)
	if err := shgo.Exec("docker", []byte(script)); err != nil {
		panic(err)
	}
}

That is it!

Dynamic Docker Images

Warning: we are moving beyond terrible ideas to really terrible ideas.

Now that we have gotten a taste for the "superior" way to distribute docker images, you may find it pretty tedious to have to manage a bunch of images-as-git-repos.

Inspired by kontain.me, we can do better!

What we want is to host a custom server such that we can run go run <domain>/<image name>@<image version>, and fetch a docker image on the fly. To do this, we need to understand how Go fetches code.

At a high level, it makes an HTTP request to the domain, and expects to be redirected to a version control system repository. Then it clones the repo, and caches it: after the first fetch, any future requests from any client should be served directly from the client.

With this in mind, we can run a server that will dynamically build a git repo based on the requested image name, pull down the docker image, and dynamically create a main.go like above with the docker image. Note that because of the Go module proxy, this should only happen once: after the initial load, everything is served from the module proxy.

The full implementation of this can be found here.

To get this working end to end, we need the Go module proxy to actually connect to our server. This would require putting this service on the public internet, which is probably a bad idea. Instead, I stopped short of that and just ran it locally (and also skipped setting up TLS).

Running locally shows off the concept, though:

$ GOSUMDB=off GOINSECURE=* GOPROXY=direct go run go.localhost/docker/hello-world@latest
Hello from Docker!
$ GOSUMDB=off GOINSECURE=* GOPROXY=direct go run go.localhost/docker/redis@latest
1:C 05 Jul 2024 21:45:39.435 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo