In Analyzing Go Build Times, I went over how to analyze and understand Go build times, and what factors impact build times. A close cousin to build times is build sizes.

Large binaries can lead to a variety of issues such as:

  • Generally, slower build times
  • Increased costs of storage
  • Increased costs and time to distribute
  • Increased memory usage at runtime (more on this in another article, hopefully)

So its generally nice to keep them small.

Measuring build size

Measuring build size is not rocket science - just build it and check the size! I will build the Istio sidecar agent in these examples, as its a fairly large real world Go binary.

$ go build ./pilot/cmd/pilot-agent; du -sh pilot-agent
109M    pilot-agent
$ stat -c %s pilot-agent # If you prefer the exact size...

What is actually taking up space though? For that, we can reach to nm, which will "list symbols from object files".

With the right flags, we will find what symbols are taking up the most space:

$ nm -S --size-sort -t d pilot-agent | tail
0000000016675872 0000000000042060 T*Bootstrap).validate
0000000015690464 0000000000049950 T*HttpConnectionManager).validate
0000000024892736 0000000000052344 T
0000000016214464 0000000000055942 T*Cluster).validate
0000000021548512 0000000000058506 T
0000000082309760 0000000000065792 B runtime.trace
0000000053340256 0000000000072088 r runtime.itablink
0000000082375552 0000000000092792 B runtime.mheap_
0000000053225696 0000000000114540 r runtime.typelink
0000000053053056 0000000000168800 r runtime.findfunctab

Nice! The second column represents the size. Here our top 10 offenders appear to be some Envoy, Kubernetes, and CEL dependencies, along with core Go runtime components.

Go also provides a similar tool builtin, which gives the same results (but in the reverse order):

$ go tool nm -size -sort size pilot-agent | head
 3298680     168800 r runtime.findfunctab
 32c28e0     114540 r runtime.typelink
 4e8f380      92792 D runtime.mheap_
 32de860      72088 r runtime.itablink
 4e7f280      65792 D runtime.trace
 148cde0      58506 T
  f769c0      55942 T*Cluster).validate
 17bd540      52344 T
  ef6ae0      49950 T*HttpConnectionManager).validate
  fe7420      42060 T*Bootstrap).validate

goda weight is another handy tool (which we will use later as well), which gives similar info in a nicer form.

So if we sum up all of the symbols, surely it will match the binary size (109M)...?

$ nm -S --size-sort -t d pilot-agent | awk '{print $2}'| paste -sd+ - | bc | numfmt --to=iec-i --suffix=B


Analyzing symbols is an okay way to view whats in the binary roughly, but it shouldn't be thought of as a direct mapping to binary size (more on this later).

Making things smaller

CGO and static linking

Our current build is dynamically linked due to CGO usage:

$ file pilot-agent
pilot-agent: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked ... debug_info, not stripped

Are there improvements (or regressions) with static linking?

$ CGO_ENABLED=0 go build ./pilot/cmd/pilot-agent; du -sh pilot-agent
109M    pilot-agent

Nope, about the same. The file output does give us a clue of where to look next, though: debug_info, not stripped.

Stripping debug information

Go offers some control during the build of what is included in the binary:

	Omit the symbol table and debug information.
	Omit the DWARF symbol table.
$ CGO_ENABLED=0 go build -ldflags '-w' ./pilot/cmd/pilot-agent; du -sh pilot-agent
89M     pilot-agent
$ CGO_ENABLED=0 go build -ldflags '-s' ./pilot/cmd/pilot-agent; du -sh pilot-agent
75M     pilot-agent
$ CGO_ENABLED=0 go build -ldflags '-s -w' ./pilot/cmd/pilot-agent; du -sh pilot-agent
75M     pilot-agent

Some nice improvements here! Interestingly, it seems passing just -s is sufficient, despite most projects using -s -w.

Another point of interest is the improvements from removing symbols nearly exactly matches the total symbol size reported by nm. This makes sense; nm reports the symbol sizes, and -s strips the symbols.

Another possibility:

                remove all file system paths from the resulting executable.
                Instead of absolute file system paths, the recorded file names
                will begin either a module path@version (when using modules),
                or a plain import path (when using the standard library, or GOPATH).
$ CGO_ENABLED=0 go build -ldflags '-s' -trimpath ./pilot/cmd/pilot-agent; du -sh pilot-agent
75M     pilot-agent

This rounds to the same result, but does shave off 120KB total, so gives a modest improvement. It is more valuable as a means of reproducibility, though.

Surely these symbols provided some value though, so what are we loosing?

$ nm pilot-agent
nm: pilot-agent: no symbols

Well, that is one thing! The other notable loss is when attaching a debugger. However, much of the core functionality (panic stack traces, pprof, etc) remain fully functional (more on how this is possible, pclntab, later). As such, I think its generally a good idea to use these flags for production builds.

Digging deeper

At this point we know ~33M of the binary was symbols, but we still have 75M left. Where is that from?

bloaty is a nice tool to help inspect a binary.

$ bloaty pilot-agent
    FILE SIZE        VM SIZE
 --------------  --------------
  30.3%  32.9Mi  44.1%  32.9Mi    .text
  23.8%  25.8Mi  34.6%  25.8Mi    .gopclntab
  12.6%  13.8Mi  18.4%  13.8Mi    .rodata
   9.7%  10.6Mi   0.0%       0    .strtab
   7.4%  8.04Mi   0.0%       0    .debug_info
   5.0%  5.47Mi   0.0%       0    .debug_loc
   4.1%  4.45Mi   0.0%       0    .debug_line
   3.0%  3.25Mi   0.0%       0    .symtab
   1.5%  1.65Mi   0.0%       0    .debug_ranges
   1.1%  1.19Mi   1.6%  1.19Mi    .noptrdata
   1.0%  1.13Mi   0.0%       0    .debug_frame
   0.0%       0   0.4%   324Ki    .bss
   0.3%   323Ki   0.4%   323Ki    .data
   0.1%   111Ki   0.1%   111Ki    .typelink
   0.0%       0   0.1%  90.1Ki    .noptrbss
   0.1%  70.3Ki   0.1%  70.3Ki    .itablink
   0.0%  14.8Ki   0.0%  14.8Ki    .go.buildinfo
   0.0%  3.45Ki   0.0%       0    [Unmapped]
   0.0%  2.07Ki   0.0%  2.07Ki    [LOAD #2 [RX]]
   0.0%  1.44Ki   0.0%  1.44Ki    [ELF Section Headers]
   0.0%  1.18Ki   0.0%     845    [8 Others]
 100.0%   108Mi 100.0%  74.6Mi    TOTAL

So we have:

  • .text: contains the executable code of a program or application.
  • .rodata: contains constants, string literals, type names, etc.
  • .gopclntab: containers the Program Counter to Line Number Table.

We can extract these to inspect directly if we wish:

$ objcopy -O binary --only-section=.rodata pilot-agent /tmp/rodata

The format isn't exactly human readable, but its not entirely garbage.

Here is a short snippet, where we can clearly see some Kubernetes API fields:


Analyzing the remaining information

shotizam is a pretty neat tool to parse the rest of the binary. Unlike most other tools, it doesn't rely on the symbols (which we strip anyways!). Unfortunately, it hasn't been updated to accommodate some Go internals. A (partial) PR updates it though.

Lets give it a shot.

$ shotizam pilot-agent > /tmp/analysis
$ sqlite3
sqlite> .read /tmp/analysis
sqlite> select sum(size) from bin;
$ stat -c %s pilot-agent

Every byte is accounted for! This is great. Lets see what else we can get...

sqlite> select * from bin limit 10;

So we have Func, Pkg, What, and the Size... Lets see what types of space there are

sqlite> select What, sum(size) from bin group by What order by 2 desc;

Ah, so while we do add up to the exact bytes, some of that is marked as TODO. These are not "dark" or "non-useful" bytes, but rather just bytes the tool doesn't yet know how to categorize. On this binary, 80% is analyzed which is good enough for most cases.

34MB is text (which aligns with what we saw above as .text section of the binary), 9MB is function names (new info!), and then a variety of other categories make up the rest.

I believe TODO corresponds primarily to .rodata, text to .text, and the rest to .gopclntab.

We can also query by package or function name (or whatever else we want):

sqlite> select Pkg, sum(size) from bin where Pkg <> '' group by Pkg order by 2 desc limit 10;|3049990|1459406|1097645|857938|756050

sqlite> select Func, sum(size) from bin where Pkg <> '' group by Func order by 2 desc limit 10;|63368*Cluster).validate|61394|54903*HttpConnectionManager).validate|54772*Bootstrap).validate|46190*RouteAction).validate|43414|38303*Listener).validate|35136*JSONSchemaProps).Unmarshal|33583*JSONSchemaProps).Unmarshal|33556

So here we see Kubernetes is a big factor (as we have seen in the past, as are Envoy and CEL.

Reducing the binary size

Aside from our one trick of stripping debug symbols, there isn't any easy fixes to reducing the size of the binary other than reducing the amount of code compiled into the binary. Above, we saw how to analyze what is taking up space. But understanding how to remove it can be still be tricky.

For instance, in the above analysis we found using substantial space. Intuitively, I know this probably isn't actually used in pilot-agent, so its likely an inadvertent dependency. However, tracking down why its imported (possibly through a long chain of packages) can be tricky.

Again, goda can help. There are lots of handy tools within it, but I tend to use the tree reach(...) command. This shows the path from one package to another:

$ goda tree 'reach(./pilot/cmd/pilot-agent:all,' | hh cel-go
                  └ ~

Here we can see a fairly long series of imports before we finally import From there, we can do the refactoring work needed to prune it out of the dependency tree.

In this particular case, it was easier to mask the unexpected dependency behind a build flag; in many other cases its possible to simply refactor away dependencies.


  • -ldflags '-s -w' can help reduce binary sizes by stripping debug symbols; these are typically not required.
  • There aren't other (reliable) magic binary reduction methods; you'll need to understand what is in the binary and remove it.
  • go tool nm or other symbol analysis tools can give a rough idea of whats in a binary, but probably are not a reliable source of data given these symbols should be stripped anyways.
  • shotizam, on the other hand, analyzes parts of the binary beyond symbols, so is very useful to analyze the contents of a binary.
  • goda is a great tool to analyze dependencies.