Testing Qt on Emulated Architectures Using QEMU User Mode

Have you ever wanted to test if your application works on ARM, but don't want to make an image and launch a real device, or wanted to test an architecture you don't even have hardware for? I have been following a set of instructions I have gathered from various online sources, and have been teaching others how to do the same, but I want to collect it all here and publish it here for you to follow as well. While writing this I will be following the instructions myself for an architecture I haven't done this for before.

The secret is of course QEMU, but to make testing easier I will set it up so that QEMU doesn't simulate an entire system, but runs foreign applications as if they were native inside my existing X11 desktop. This is possible through the qemu-user binaries, which can run statically linked executables or dynamically linked ones if you are in a properly set up chroot.

Setting up chroot

So how do you get this properly set up chroot? There are various ways depending on the Linux distro. I personally use Debian, and the instructions here will be almost identical to what is done on Ubuntu. If you use Suse or RedHat, I know you can find official guidelines on setting up a chroot for qemu.

First, get the dependencies (as root or sudo) :
apt-get install binfmt-support qemu qemu-user-static debootstrap


Most of these are just the tools needed, and binfmt-support is configuration glue what will make Linux launch ELF executables with non-native architectures using qemu, which is another piece of the magic puzzle.

For my example I am testing while writing this, I will be targeting the IBM mainframe architecture S/390x and will also need the cross-building tools for that as well. I can get all of those on Debian by just pulling in the right cross-building g++ (as root or sudo) :
apt-get install g++-s390x-linux-gnu


S/390x is interesting for me because Debian dropped official support for big endian PPC64, so I need a new way of testing that Qt works on 64bit big-endian platforms.

Now go to where you want your sysroots installed, and run debootstrap. Debootstrap is the Debian tool for downloading and installing a Debian image into a sub-directory
(as root or sudo) :
debootstrap stable --arch s390x stable debian-s390x http://deb.debian.org/debian/


Next prepare it for being a chroot system (as root or sudo) :
mount --rbind /dev debian-s390x/dev/
mount --rbind /sys debian-s390x/sys/
mount --rbind /proc debian-s390x/proc/


Enter the chroot (as root or sudo) :
chroot debian-s390x


If you run 'uname -a' here you will now get something like:
Linux hostname 4.x.x-x-amd64 #1 SMP Debian 4.x.x-x (2018-x-x) s390x GNU/Linux


In other words, a native AMD64 kernel with x390x userland! Every command you execute here will be using s390x binaries in the chroot, but using the native kernel for system calls. If you launch an X11 app, it will be using the X11-server running natively on your machine already.

Prepare chroot to be a sysroot

To build Qt, we also need all the Qt dependencies inside the chroot. In Debian we can do that using 'apt-get build-dep'.

Edit debian-s390x/etc/apt/sources.list and add a deb-src line, such as:
deb-src http://deb.debian.org/debian stable main


Install the Qt dependencies (inside chroot as root):
apt-get update
apt-get build-dep qtbase-opensource-src


Finally, for the setup we need to make sure it works a sysroot for cross-building by replacing absolute links to system libraries with relative ones that work both inside and outside the chroot (inside chroot as root):
apt-get install symlinks
cd /usr/lib/
symlink -rc .


Building Qt

We can now build Qt for this sysroot. If you want to test you own applications, you could also install an official prebuilt Qt version from Debian, and build against that.

Configuring Qt for cross-building to a platform like that looks like this on Debian, and only needs a few changes to work on other platforms:
$QTSRC/configure -prefix /opt/qt-s390x -release -force-debug-info \
-system-freetype -fontconfig -no-opengl \
-device linux-generic-g++ -sysroot /sysroots/debian-s390x \
-device-option CROSS_COMPILE=/usr/bin/s390x-linux-gnu-


You can remove the -no-opengl argument if you want to test OpenGL using the mesa software implementation, but otherwise I don't recommend using OpenGL inside the chroot.

If everything is setup correctly you don't need to specify '-system-freetype -fontconfig'. It is just there to catch failures in pkg-config, so we don't end up building a Qt version that requires manually bundling fonts.

In general cross-compiling with Qt always looks similar to the above, but note that if you are cross-building to ARM32, it looks a little bit different:
$QTSRC/configure -prefix /opt/qt-armhf -release -force-debug-info \
-system-freetype -fontconfig -no-opengl \
-device linux-arm-generic-g++ -sysroot /sysroots/debian-armhf \
-device-option DISTRO_OPTS=hard-float \
-device-option COMPILER_FLAGS='-march=armv7-a -mfpu=neon' \
-device-option CROSS_COMPILE=/usr/bin/arm-linux-gnueabihf-


Notice that ARM32 has a specific device name (linux-arm-generic-g++ instead of linux-generic-g++) and needs to have a floating-point mode set. Additionally, for older embedded architectures, it is generally a good idea to specify COMPILER_FLAGS to select a specific architecture. You need to pass compiler flags as a device-option, because the normal way of specifying QMAKE_CFLAGS and QMAKE_CXXFLAGS after configure options sets the host compiler flags, not the target compiler flags. If you need more complicated options, you should make your own qmake device target. You can always base it of linux-generic-g++ for good defaults.

With Qt configured like this, you can build it as usual and run make install to have the libraries installed into the /opt directory under the chroot.

Issue encountered; as I built qtbase, I hit a linking error:
qt5/qtbase/src/corelib/global/qrandom.cpp:143: error: undefined reference to 'getentropy'


The problem turns out to be that my cross-building tools have libc6 2.27 while the sysroot has libc6 2.24. The best thing to do would be using the same libc version on both sides. However, as they are fully compatible, except for newer features in one, I chose to just disable this feature in Qt. I accomplished this by calling the configure command again, this time with -no-feature-getentropy added.

With this change, everything I tried building built and installed without further issues.

Run a Qt application

To get some examples to test, go to the folder of the example in the build tree and run 'make install'. For instance, my standard example for widgets:
cd qtbase/examples/widgets/richtext/textedit
make install


Then (inside chroot as root, assuming you have created a testuser):
su - testuser
export DISPLAY=:0
export QT_XCB_NO_MITSHM=1
/opt/qt-s390x/examples/widgets/richtext/textedit/textedit -platform xcb


And it just works...
The -platform xcb argument is needed because the default for embedded builds is using the eglfs QPA. I am assuming you are developing on X11 and want the jaw-dropping experience of having a foreign architecture Qt application just launch like a normal Qt app.

In this setup, the applications running in QEMU will be much faster than under a full system emulation, because the kernel and the X11 server are still the native ones, and thus running at normal speed. However, for X11 it means sometimes having a few oddities, as there can be client-server mismatch in endianness. While it is supposed to be supported, it is such an edge case that it often has issues. This is the reason why QT_XCB_NO_MITSHM=1 is set. Otherwise, we would hit an issue with passing file descriptors between client and server, which is likely a bug in libxcb. If we were testing a little-endian architecture like arm64 or riscv instead, the environment variable would not be needed.

So that is it, quite a few steps, but most only done once. And as I wrote this, I followed the same instructions for a rather uncommon and not officially supported platform, and only hit two issues with easy work-arounds.