After dealing with clicking around in the Grafana UI and painstakingly exporting and import JSON over and over, I finally bit the bullet and committed to building Grafana dashboards "as code". Weeding through the immense set of options (that post doesn't even cover many of them!) is a bit overwhelming though, and most of the documentation for each is very... light.

With some trial and error I found a pretty good setup that mostly ticks all the boxes:

  • IDE support
  • Fast iteration (hot reloading)
  • Not a terrible language/experience to write in

Setup

First, we need a few tools:

go install github.com/google/go-jsonnet/cmd/jsonnet@latest
go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest
go install github.com/grafana/grizzly/cmd/grr@latest
  • Jsonnet is a tool to help compile a custom language into JSON. Grafana has libraries we will use that make this useable to build dashboards without doing everything ourselves.
  • jsonnet-bundler manages Jsonnet dependencies.
  • Grizzly helps watch + push our dashboards to Grafana.
  • Also install entr or equivalent with your preferred packaging tool.

I don't know why these couldn't all be one tool, but 🤷.

You'll also want the VS Code extension. I prefer JetBrains usually but the support seems a bit worse there.

Building the dashboard

I recommend just pulling from the example dashboard as a base, and tuning from there, to avoid needing to build everything from scratch.

The dashboard references a remote library provide by Grafana. Run jb install to fetch it (into vendor/).

Now we can actually build the dashboard: jsonnet -J vendor main.libsonnet. Here -J vendor points to where we should pull additional libraries from, and main.libsonnet is our entrypoint to our dashboards. This will spit out our raw dashboard JSON, which is not super useful for development.

Incremental development

For making incremental development sane we will want to actually visualize the changes we are making in real time. Grizzly, which we installed early, can help with that. It basically acts as a CLI interface to Grafana, allowing us to push and pull dashboards; we are interested in the "push" aspect.

Some basic setup to point it at our instance:

$ grr config create-context local
$ grr config set grafana.url http://localhost:3000

And now we can build and push (in a single step, so you don't need to run jsonnet directly) to our Grafana instance:

$ grr push -s main.libsonnet

Grizzly also has a watch command, but I couldn't get it to work. Instead, I use entr which generically watches for files changes and triggers a command:

$ find . | entr -s 'grr push -s main.libsonnet

Now on any file save, the dashboard is automatically pushed to Grafana (if it builds). Especially handy is that Grafana will live-reload the dashboard without any interaction required!

Why not using more Grizzly

Grizzly builds a custom API model based on Kubernetes object structure. For example:

apiVersion: grizzly.grafana.com/v1alpha1
kind: Dashboard
metadata:
  folder: sample
  name: prod-overview
spec:
  schemaVersion: 17
  ...

Many commands in grizzly rely on this format, though a few allow -s/--only-spec (fortunately, push, does) to avoid this. However, I didn't like this much because the format is custom only for grizzly, so locks into that tooling.

Additionally, most of the commands that require it didn't seem super useful to me. For example, grizzly has a serve command which seems to bundle a local Grafana instance. This didn't seem to live-reload, though, so it was less useful.

Overall, I decided to to avoid this. If you do, though, you can use the library to embed the dashboard into the wrapper type.