gonative: Cross compiling Golang programs with native libraries

by Alan Shreve

Go has execellent support for cross compilation. There are scores of tutorials and helper tools on the internet with instructions on how to do it. This, in and of itself, is rather astounding because of how simple it is:

1 # set up cross compilation to windows_amd64
2 # you only need to do this once
3 cd $GOROOT/src
4 GOOS=windows GOARCH=amd64 ./make.bash --no-clean
5 
6 # whenever you want to cross compile
7 cd $YOUR_APP
8 GOOS=windows GOARCH=amd64 go build

There is a well-known limitation to this simplicty. If you want to use any C libraries, you’ll need to compile and link them with Cgo and from there you’re back to the mess that is cross-compiling C. Upon hearing this limitation you might say something totally reasonable like:

That’s not really a problem, there are tons of programs that don’t use Cgo.

And you’d be right, until I told you:

Some critical parts of the standard library use Cgo.

What exactly are those parts exactly? It turns out that if you cross-compile Go to Darwin or Linux your programs won’t use the system DNS resolver. They also can’t use the native host certificate store. They also can’t look up the user’s home directory, either. For some applications, that doesn’t matter so much. For ngrok, though, all three could matter, and two have already caused bugs.

A better solution

When I spoke at GopherCon, I told everyone that if they want to distribute production applications they should be compiling natively because of this limitation. I was wrong. It turns out that there’s a clever workaround for this that has been mentioned occasionally on the mailing list and which minux just reminded me of when he posted:

If the only cgo-dependent packages are in the standard library, then you do have workaround without setting up a full cross compilation environment.

Download the binary distribution for the target platform, extract the pkg/$GOOS_$GOARCH directory into your $GOROOT/pkg …

Which is quite clever and I want the simplicity of that build and deployment story. But instead of doing this as a one off, I decided to automate the entire process since I want you all to use it as well and because I’m going to need to do it again at every new release of Go.

gonative

gonative is a simple tool which creates a build of Go that can cross compile to all platforms while still using the Cgo-enabled versions of the stdlib packages. It’s a simple Go program of a few hundred lines and is available on github! If you’re the author of a Go program doing multi-platform distribution, please try it out and contribute:

github.com/inconshreveable/gonative

It’s really easy to install and run:

1     go get github.com/inconshreveable/gonative
2     gonative

I’ve tested targeting Windows/Linux/Darwin and they all work correctly!

About that portabililty thing . . .

One of the oft-touted advantages of Go programs is that they’re “dependency-free”. They don’t require a runtime or link against dynamic libraries which your users might not have installed on their systems.

So when I compiled my native-stdlib linux_386 test code and tried to run it on an amd64 machine, I was rather confused when I was confronted with this error message:

1     alan@inconshreveable:~$ ./test 
2     -bash: ./test: No such file or directory

minux’s help on the mailing list helped me understand what was going on. Let’s read the ELF header of this binary.

1     alan@inconshreveable:~$ readelf -l test
Elf file type is EXEC (Executable file)
Entry point 0x8066a50
There are 9 program headers, starting at offset 52

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  PHDR           0x000034 0x08048034 0x08048034 0x00120 0x00120 R E 0x1000
  INTERP         0x000bed 0x08048bed 0x08048bed 0x00013 0x00013 R   0x1
      [Requesting program interpreter: /lib/ld-linux.so.2]
  LOAD           0x000000 0x08048000 0x08048000 0x68e70 0x68e70 R E 0x1000
  LOAD           0x069000 0x080b1000 0x080b1000 0xc9403 0xc9403 R   0x1000
  LOAD           0x133000 0x0817b000 0x0817b000 0x10580 0x23554 RW  0x1000
  DYNAMIC        0x133080 0x0817b080 0x0817b080 0x00098 0x00098 RW  0x4
  TLS            0x000000 0x00000000 0x00000000 0x00000 0x00008 R   0x4
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x00000 RW  0x4
  LOOS+5041580   0x000000 0x00000000 0x00000000 0x00000 0x00000     0x4

See that INTERP section? The one that says:

[Requesting program interpreter: /lib/ld-linux.so.2]

/lib/ld-linux.so.2 is the 32-bit ELF interpreter that is required for programs that need to dynamically link libraries. My linux box didn’t have it installed, so the program was failing.

But wait, you said that Go programs doesn’t require a runtime or dynamically linked libraries!

And that’s totally true when you cross compile without native libraries. Those executables don’t call anything via C APIs. But when you link against native libraries, there are a few libraries you need to load dynamically. Let’s take a look at them:

alan@inconshreveable:~$ ldd test
        not a dynamic executable

What!? This hung me up for a while too, until I remembered seeing that DYNAMIC section in the ELF headers. It turns out that if you don’t have the 32-bit libraries on your system, ldd can’t analyze the binary properly. minux explains:

actually if you take a look at ldd’s source code you will find that it’s a script and just set some magic environment variable and then execute the program, essentially it’s asking the elf interpreter to dump needed shared library.

so if you don’t have the elf interpreter, the program will fail to execute and ldd mistake that as the program not being dynamically linked.

If we use readelf instead, we can see what’s going on:

alan@inconshreveable:~$ readelf -d test
Dynamic section at offset 0x133080 contains 19 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libpthread.so.0]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
more …

So now we can finally see that our natively compiled program now depends on the 32-bit elf interpreter so that it can dynamically load libpthread and libc.

A fun detour

Understanding executable formats and the guts of how Linux sets up your program to support dynamic linking is not an area I need to wander into frequently, so it was rather enjoyable to push my knowledge deeper into this area. For those of you who don’t wade there often either, I hope this was useful in understanding more about exactly how your programs are loaded and run and what you need to be aware of when you compile with native vs. non-native Go libraries. And for those of you who need to distribute Go programs that use native libs, try out gonative! I hope it makes things a little less painful.

Relevant mailing list thread

https://groups.google.com/forum/#!msg/golang-nuts/2XoGUvBalcw/ErSWiTlO17kJ