Benchmark of QtQuickCompiler on low-end embedded Linux

For application developers, performance optimization on embedded hardware is always a challenge. QtQuick provides a free performance boost through QtQuickCompiler, which improves performance simply by enabling an option without changing the source code.

Regarding QtQuickCompiler, the R&D team has already conducted performance benchmarks, such as:

• https://www.qt.io/blog/qt-6.6-and-6.7-make-qml-faster-than-ever-a-new-benchmark-and-analysis
• https://www.qt.io/blog/compiling-qml-to-c-a-4x-speedup

Nonetheless, I think those tests mainly concentrate on the QML language itself. As an application developer, my primary interest lies in gauging the potential improvements at the application level. Consequently, I chose the existing QtQuick Coffee Machine demo as the test subject and utilized the Colibri iMX6ULL hardware platform for testing.

Test Cases

I conducted tests using the following five cases:

1. qmlcachegen: Compiling with the default QtQuickCompiler qmlcachegen.
2. qmlsc-direct: Compiling with the QML script compiler in direct mode. (https://doc.qt.io/qt-6/qtqml-qml-script-compiler.html#direct-mode)
3. qmlsc-static: Compiling with the QML script compiler in static mode. (https://doc.qt.io/qt-6/qtqml-qml-script-compiler.html#static-mode)
4. qmltc: Compiling with the QML type compiler of QtQuickCompiler. (https://doc.qt.io/qt-6/qtqml-qml-type-compiler.html)
5. Off: Completely disabling QtQuickCompiler's qmlcachegen by setting the runtime environment variable QML_DISABLE_DISK_CACHE=true.

About qmltc

As of Qt 6.8, qmltc is still in the technical preview stage and has several limitations. For example, default QtQuickControls cannot be used directly in an application. (https://doc.qt.io/qt-6/qtqml-qml-type-compiler.html#known-limitations)

To address this, I created an experimental temporary solution using some CMake tricks: https://git.qt.io/se_japan_public/qtquickcontrolscompiler. This solution provides a wrapper function qt_add_qml_module_qmltc for qt_add_qml_module, which takes an additional QtQuickControls style parameter after the CMake target specification.

Example usage:

qt_add_qml_module_qmltc(apptestQtQuickControl Basic
    URI testQtQuickControl
    VERSION 1.0
    QML_FILES
         Main.qml
)

Since QtQuickControls styles are implemented in QML rather than C++, using qmltc for an application that includes a Button from the Basic style results in an error like this:

Error: Main.qml:10:5: Can't compile the QML base type "Button" to C++ because it lives in "QtQuick.Controls.Basic" instead of the current file's "untitled" QML module. [compiler]    Button {    ^^^^^^

Understanding this limitation, the solution involves feeding the required QML files to qmltc for compilation. The qt_add_qml_module_qmltc function's second parameter (e.g., Basic) automatically locates and includes the corresponding style’s QML files, binding them to the specified CMake target. This effectively makes components like Button available within the module. The required private libraries are also linked, though for now, only the Basic style is supported. 

Benchmark Setup

To conduct the benchmark, I added the following enhancements to the Coffee Machine demo:

1. A Backend Simulator to automate screen transitions.
2. A Benchmark Module to record FPS and other performance metrics.
3. A unit test to measure the QML engine’s load time.

Modified source code: https://git.qt.io/se_japan_public/qt5-qt6-benchmark/coffee-benchmark

The benchmark consists of two tests:

• Rendering Performance: The simulator automates screen transitions while recording FPS.
• Startup Performance: A unit test evaluates the QML engine’s load time.

Here are the benchmark results of 100 iterations.

Startup benchmark

FPS benchmark

Conclusion

The performance of qmltc is impressive, reducing QML load time by nearly half. However, as a reminder, qmltc is still in the technical preview stage, so it should be used cautiously in production projects.

The different qmlsc modes showed similar performance improvements.

In terms of FPS, qmltc also had the best numerical result, though all cases showed only minor improvements. This aligns with my expectations because the Coffee Machine demo does not extensively use StackView or Loader for dynamic QML loading. the current result of FPS is more likes that it has already limited by hardware constraints.

The Coffee Machine demo is relatively simple, and a more complex application may demonstrate even greater improvements.