To run the tests of my i3 Go package, I use the following command:
go test -v go.i3wm.org/...
To run the tests of my i3 Go package on a different architecture, the only thing
I should need to change is to declare the architecture by setting
GOARCH=arm64 go test -v go.i3wm.org/...
“Easy!”, I hear you exclaim: “Just
apt install qemu, and you can transparently
emulate architectures”. But what if I want to run my tests on a native machine,
such as the various Debian porter
the rabbit hole we go…
On Plan 9, the cpu(1) command allows transparently using the CPU of dedicated compute servers. This has fascinated me for a long time, so I tried to replicate the functionality in Linux.
One of the key insights this project is built on is that
sshfs(1) can be used over
an already-authenticated channel, so you don’t need to do awkward reverse
port-forwardings or even allow the remote machine SSH access to your local
I learnt this trick from the 2014 boltblog post “Reverse SSHFS mounts (fs push)”.
The post uses
bidirectional wiring of stdin/stdout (as opposed to a unidirectional wiring like
in UNIX pipes).
Instead of clumsily running
dpipe in a separate window, I encapsulated the
necessary steps in a little Go program I call
cpu. The reverse sshfs principle
looks like this in Go:
sftp := exec.Command("/usr/lib/openssh/sftp-server") stdin, _ := sftp.StdinPipe() stdout, _ := sftp.StdoutPipe() session.Stdin = stdout session.Stdout = stdin sftp.Stderr = os.Stderr session.Stderr = os.Stderr const ( host = "" src = "/" mnt = "/mnt" ) session.Start(fmt.Sprintf("sshfs %s:%s %s -o slave", host, src, mnt)) sftp.Start()
Here’s how the tool looks in action:
Now that we have a tool which will make our local file system available on the
remote machine, let’s integrate it into our
go test invocation.
While we don’t want to modify the
go tool, we can easily teach our kernel how
to run aarch64 ELF binaries using
I modified the existing
/var/lib/binfmts/qemu-aarch64’s interpreter field to
/home/michael/go/bin/porterbox-aarch64, followed by
--enable qemu-aarch64 to have the kernel pick up the changes.
porterbox-aarch64 is a wrapper invoking
cpu like so:
cpu \ -host=rpi3 \ unshare \ --user \ --map-root-user \ --mount-proc \ --pid \ --fork \ /bindmount.sh \ \$PWD \ $PWD \ [email protected]
Because it’s subtle:
\$PWDrefers to the directory in which the reverse sshfs was mounted by
$PWDrefers to the working directory in which
[email protected]refers to the original command with which
bindmount is a small shell script preparing the bind mounts:
#!/bin/sh set -e remote="$1" shift wd="$1" shift # Ensure the executable (usually within /tmp) is available: exedir=$(dirname "$1") mkdir -p "$exedir" mount --rbind "$remote$exedir" "$exedir" # Ensure /home is available: mount --rbind "$remote/home" /home cd "$wd" "[email protected]"
This is what all of the above looks like in action:
Putting all of the above puzzle pieces together, we end up with the following picture:
go test ├ compile test program for GOARCH=arm64 └ exec test program (on host) └ binfmt_misc └ run porterbox-aarch64 └ cpu -host=rpi3 ├ reverse sshfs └ bindmount.sh └ unshare --user ├ bind /home, /tmp └ run test program (on target)
On the remote host, the following requirements need to be fulfilled:
apt install sshfs, which also activates the FUSE kernel module
sysctl -w kernel.unprivileged_userns_clone=1
If the tests require any additional dependencies (the tests in question require
i3), those need to be installed as well.
On Debian porter boxes, you can install the dependencies in an
session. Note that I wasn’t able to test
this yet, as porter boxes lacked all requirements at the time of writing.
Unfortunately, Debian’s Multi-Arch does not
yet include binaries. Otherwise, one might use it to help out with the
dependencies: one could overlay the local
/usr/bin/aarch64-linux-gnu/ on the
On first glance, this approach works as expected. Time will tell whether it’s useful in practice or just an interesting one-off exploration.
From a design perspective, there are a few open questions:
- Making available only
/homemight not be sufficient. But making available
/doesn’t work because
sshfsdoes not support device nodes such as
- Is there a way to implement this without unprivileged user namespaces (which are disabled by default on Linux)? Essentially, I think I’m asking for Plan 9’s union directories and namespaces.
- In similar spirit, can binfmt_misc be used per-process?
Regardless, if this setup stands the test of time, I’ll polish and publish the tools.