There are some emulators that will emulate the target hardware and the target OS. At that point, it is worth understanding the difference between virtualisation and emulation.

VirtualBox creates virtual machines on top of the current hardware: if you run Windows on an Intel x86₆₄ machine, you will be able to run GNU/Linux for x86₆₄ architecture and not for arm64.

An emulator will make a program believing it is running on a machine architecture is was build for: it will take the target architecture assembly code ans translate it on the fly into the underlying machine architecture instructions. The emulated OS will handle the memory allocated for the emulator by the host. It will also be an interface to the devices that need to be used by the emulated environment.

There are many examples of emulators. There are some emulators that will emulate the Nintendo or Sega consoles on your MS Windows personal computer. Another famous emulator is QEMU which we will use in this article.

Using an emulator comes with a performance penalty due to the additional computation to translate instructions into something understandable by the host machine and those instructions to translate comes from the emulated Operating System and the program your are running.

Fortunately, there are rooms for improvement for particular cases we are interested in in this article: we want to build binaries for different hardware but still for GNU/Linux. In that case, [QEMU][emu_site] provides another emulation mode called user-mode emulation: it will not emulate a full system, but only perform CPU instructions translation and capture the system calls. That way, those syscalls can be executed by the underlying kernel. No need anymore a full hardware emulation and no need to embed a full OS to run a program.

As an example, we will run busybox compiled for arm64 on an amd64 machine.

> docker pull arm64v8/alpine
> container=$(docker create arm64v8/alpine)
WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested
> docker cp ${container}:/bin/busybox .
> docker rm ${container}
> file ./busybox
busybox: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, stripped

Trying to run it will fail:

> uname -m -o
x86_64 GNU/Linux
> ./busybox ls
zsh: exec format error: ./busybox

Let's install the qemu-user-static package on ubuntu and test if this works:

> sudo apt install qemu-user-static
...
> qemu-aarch64-static busybox ls
qemu-aarch64-static: Could not open '/lib/ld-musl-aarch64.so.1': No such file or directory

The error message is not the same. As seen above busybox is not statically linked: the kernel will need to invoke the dynamic linker which is hard-coded in the .interp ELF sections:

> readelf --segments ./busybox
...
  INTERP         0x000200 0x0000000000000200 0x0000000000000200 0x00001a0x00001a R   0x1
        [Requesting program interpreter: /lib/ld-musl-aarch64.so.1]
...

We can retrieve the interpreter from the same OCI image and create a symlink in our /lib to the interpreter.

> sudo ln -s ./ld-musl-aarch64.so.1 /lib
> qemu-aarch64-static busybox ls
busybox
ls-musl-aarch64.so.1

Now it works. But there are two reasons why this is not convenient: - we have to pollute our system with many files here and there that do not belong to the machine architecture. To cope with this, we can use any technology based on chroot like OCI container. It is what we will do. - we have to explicitly call the emulator for each program we want to emulate: it will be a problem if we try to run a building script for instance.

Fortunately, this second point can be elegantly solved with the binfmt_misc Linux Kernel feature

To run a program, the shell will fork itself and call an exec function family which ends up into the system call execve which role is to really execute the program.

The Kernel will validate the executable file, determine which handler to use, replace the current in memory program (the forked one) with the one to execute, prepare virtual memory, a new stack for execution…

By default, there are a few handlers that we use everyday: - statically or dynamically linked ELF executables - script files like python, bash scripts

The kernel will generally look at the first bytes of an executable and try to find an handler for this magic sequence of bytes. For example, an ELF file starts with the information on the file type (magic sequence 0x7fELF), architecture registries size (32/64 bits), the endianess, the target architecture, the version of the ELF format…

If it is a statically linked program, the kernel can start the program directly because there are no shared libraries to load beforehand. Otherwise, it will transfer the execution responsibility to the dynamic linker in the .interp section.

If the first characters are the famous shebang #! character, the kernel will read the line until the end and invoke the interpreter.

The beauty of binfmt_misc is that you can register some new handlers in the kernel. An example given by the official documentation is that you can register Wine which acts as a thin layer to adapt Windows syscalls into Linux syscalls.

echo ':DOSWin:M::MZ::/usr/local/bin/wine:' > register

We could run the following command:

echo ':qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:' | sudo tee 

But this is automatically done when installing the package qemu-user-static package though System V.

> cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/libexec/qemu-binfmt/aarch64-binfmt-P
flags: POCF
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

In essence, what this tells is that any files starting with the magic sequence (with some flexibility provided by the mask bits set), will be run by qemu user-mode emulation. We can spot int the magic field, the value 0xb7 which is the code for ARM 64 architecture.

With this handler installed, we can transparently run the binary coming from the ARM64 version of Alpine Operating System.

> ./busybox ls
busybox
ls-musl-aarch64.so.1

As explained when we run busybox for arm64, we also had to take the standard C libraries implementation it was linked with and store it on your system. Generally, a program can have many dependencies and you don't want to pollute your system with binaries that are not adapted to your system.

We could store all the needed file into a dedicated directory and chroot into that to run your program. It is basically what containers technologies do (amongst other things).

Running a container is no more than:

• untar the image somewhere and chroot in that new root
• executing a program with special flags to handle namespaces, cgroups to make the program it is alone on a fresh machine. Behind the scene, the current process is fork=ed with a Linux enhanced version called =clone so that special flags are passed to the forked process and execve to execute the target process (the code is loaded, the stack is replaced by a new one…).

Hence, with qemu-user-static installed and the binfmt_misc handlers correctly configured, running the arm64v8/alpine we tool busybox from previously, should work transparently.

> docker run -it --rm arm64v8/alpine ls
WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested
bin    etc    lib    mnt    proc   run    srv    tmp    var
dev    home   media  opt    root   sbin   sys    usr

So far, we needed to install qemu on the host and register the binfmt_misc handler to automatically run the emulator. This is generally not a problem when hacking on your own machine. But what if the build must be done on a build machine which could be just a build node in Jenkins for example.

So either, all the nodes are installed with the qemu-user-static package. If it is not the case, you have some chances to be able to bypass this limitation by setting up everything is needed by a privileged container which would:

• open qemu-$arch-static executable at binary format registration time with the flag 'F'. This feature was added here
• register the binfmt_misc handlers on the host

There are several initiatives to do this:

• multiarch/qemu-user-static
• tonistiigi/binfmt

Once one of these container is run, any subsequent program execution that requires the emulator will be run although it cannot be found on the file system. The opened emulator will remain open until the binary format is removed. In the meanwhile, the emulator will remain available if chroot is used or if a program in run in a new mount namespace.