Saying No In Open Source

As an open source maintainer, I am reviewing roughly 25 ideas per day - whether they are feature requests, design proposals, or pull requests. Inevitably, this leads to saying "No" quite a bit as well. Usually, this is in a softer for like "No, not right now", "No, not in its current form", or "No, unless someone else approves", but the outcome is the same: the change is not accepted, and the emotional impact on the reviewer and contributor is similar. ...

July 27, 2023 · 5 min

Useful tools for Kubernetes

Most people using Kubernetes extensive have already defined alias k=kubectl and are using tools like kubectx. As someone really lazy though, I have found/developed a few less common tools to help work with Kubernetes efficiently. kubectl apply from clipboard This relies on zsh, and uses the zle to define a custom command. function zle_apply { LBUFFER=" cat <<EOF | kubectl apply -f - $(xclip -se c -o) EOF" CURSOR=31 } zle -N zle_apply; bindkey "^k" zle_apply This defines a function and binds it to Ctrl+k. ...

July 12, 2023 · 3 min

Tracing shell scripts with OpenTelemetry

A little bit of OpenTelemetry, a lot of shell hackery

July 11, 2023 · 6 min

Zero allocations metrics with opentelemetry-go

In the past, Istio has suffered from performance issues from OpenCensus, which was used for metrics reporting. At extremes, we saw up to 20% of CPU spent just on incrementing various metrics. This was mitigated to some extent by batching metrics updates, optimizing OpenCensus itself, and caching parts of our OpenCensus usage. At best, we got down to roughly 600ns and 3 allocations per metric update. As OpenCensus is now deprecated, I have been looking into migration to OpenTelemetry - and hoping to avoid these issues this time around. ...

July 11, 2023 · 2 min

Analyzing Go Build Times

Go is often praised for its fast build times. While they are pretty quick, they are slow enough that I spend a lot of time waiting for them, enough that it prompted me to go down the rabbit hole of thoroughly analyzing them. This post covers all aspects of what makes Go builds fast or slow. Throughout this blog, we will use Istio as an example of real-world codebase. For reference on its size: ...

 · June 24, 2023 · 26 min

CRD Versioning

How versioning works in Kubernetes, especially with CustomResourceDefinitions, is a common source of confusion. The documentation is pretty comprehensive but a bit complicated. This post aims to give a simple description of how versioning works and dispel some misunderstandings. There is only one primary version Consider a CRD with versions alpha and beta. A user can create and view either resource version. Intuitively, they must be distinct things -- they are not. ...

June 7, 2023 · 2 min

LTS and Rolling Releases

Across the ecosystem, a variety of software support policies can be found, where "support" can mean, bug fixes, security patches, and sometimes technical support. However, they can be roughly categorized into two types: Rolling release: only latest release supported. At an extreme, this is the HEAD git commit. LTS (Long Term Support): Support latest N versions. Sometimes specific versions have extended support. "Long" is relative: Kubernetes supports a version for around a year, while RHEL does for 10 years. Enterprises love LTS Historically, enterprise users have favored -- if not demanded -- LTS software. ...

May 26, 2023 · 8 min

Outbound sidecars are not secure enforcement points

It is a very common misconception that egress policies in Istio can be used for security purposes. This is not true. Despite repeatedly explaining this (and documenting it), I still often see people that do not believe it, and that they can just add one more check to lock things down. In this post, I will show a variety of ways to bypass any possible check, and prove that these policies cannot be used as secure policies. ...

May 22, 2023 · 4 min

Ergonomic Map in Go

In many languages, on of the things I find myself doing is maping over a list to extract some field. For example, coverting a []Person to []Name. Most languages these days have ways to do this pretty easily: Kotlin: people.map { it.Name } JavaScript: people.map(p => p.Name) Rust: people.map(|p| p.Name) Scala: people.map(_.Name) With generics, Go finally can do this in a type safe manner: Map(people, func(t Person) string { return t.Name }) ... but we immediately stand out amongst other languages as having ugly, verbose syntax. ...

January 31, 2023 · 5 min

Building Docker Images Fast (by not using Docker)

This post follows the journey Istio has taken over the years to optimize our docker image builds. While there is some useful tips to take out of this, this is not intended to be a guide on how a project should build images - the steps taken here go far past the needs of a typical project, optimizing exclusively for speed (and fun) regardless of the complexity to maintain. For background, over the years Istio has consisted of ~10-20 docker images (many are for tests only) made up of ~10-15 Go binaries and various static files. We also have a few variants (debug and distroless) and architectures (amd64 and arm64). Aside from CI which is building thousands of these images daily, building images quickly is important for the inner development loop. While I try to run things locally where possible, in many cases each minor code change is built and loaded into a local Kubernetes cluster to more closely resemble a real world deployment. This makes image build time critical for efficient development ...

January 23, 2023 · 8 min