Qt Quick WebGL release in Qt 5.12

One of the Qt 5.12 new features is Qt Quick WebGL platform plugin (also known as WebGL streaming). It was actually available as a technology preview from Qt 5.10 already, but starting with Qt 5.12 it is a released feature.

• TLDR
• Intro
• What is WebGL streaming
• How to use it
• Some demos
• Device Information
• Camera
• Use cases
• WebGL streaming vs actual web
• Licensing/pricing
• Conclusion

TLDR

$ ./your-qt-application -platform webgl:port=8998

Intro

If you missed previous blog-posts, here they are:

• Qt Quick WebGL Streaming;
• WebGL streaming in a Raspberry PI Zero W;
• Qt WebGL Streaming merged;
• Qt WebGL: Cinematic Experience.

There is also a good article by Jeff Tranter from ICS.

This post is intended to be a kind of an "unboxing" experience from the perspective of an average Qt "user" - I never tried Qt Quick WebGL streaming myself, neither did I participate in its development.

What is WebGL streaming

If you read past blog-posts, you can just skip this section. I would however recommend to at least read the documentation.

WebGL streaming is a QPA plugin that sends ("streams") OpenGL calls of your Qt Quick application over the network and in turn those are translated into WebGL calls and thus can be rendered at HTML5 Canvas. What it means in practice is that you can have an application running on a remote host and render its GUI in a local web-browser.

Here's how it looks schematically:

Here's also a video from KDE Akademy with a more detailed explanation by Jesus Fernandez.

But since I'm a simple Qt "user", I don't really care about any of that (it's all hidden from me anyway), and to me everything looks like this:

So I can have a Qt-based application running on some device and work with it from Safari on my iPad. Sounds alright.

Naturally, instead of a device (Raspberry Pi in this case) there could be a desktop computer "hosting" the application, but I think WebGL streaming will be used mostly on embedded platforms (see use cases section).

Let's now highlight a couple of points we've just learnt about the feature:

• The application itself does not run inside a web-browser. Web-browser only renders its GUI;
• So it is neither video-streaming, nor mirroring. It is about "decoupling" application's GUI and showing it in a web-browser;
• Since it's for OpenGL (ES) things only, WebGL streaming does not work with Widgets or any other non-OpenGL stuff.

In fact, if you try to launch some "non-compatible" Qt application using WebGL QPA, most likely you'll get the following error:

qt.qpa.webgl: WebGL QPA platform plugin: Raster surfaces are not supported

How to use it

You only need to install it:

...or, if you are not into installers, build Qt from sources as usual - no special configuration options needed. Actually, with earlier versions -opengl es2 option was required, but there is no need in that as Qt Quick Scene Graph can use ES subset even if there is a later version of OpenGL available.

Having installed Qt itself, build any Qt Quick application of yours and launch it with the following command line arguments:

$ ./your-qt-application -platform webgl

Yes, you don't need to make any modifications in your source code, it just works. Open the following address in your web-browser: 127.0.0.1:8080, where 127.0.0.1 is the IP address of the host running your application.

If you want to use a different port, you can specify it like that:

$ ./your-qt-application -platform webgl:port=8998

Needless to say, Qt WebGL is cross-platform, and it works equally fine on Linux, Mac OS and Windows. Although, there are some differences in launching applications:

Linux:

./your-qt-application -platform webgl:port=8998

Mac OS:

QT_QPA_PLATFORM=webgl:port=8998 ./your-qt-application.app/Contents/MacOS/your-qt-application

...because Qt version I have at the moment apparently ignores -platform option (which sounds like a bug that needs to be reported).

Windows:

your-qt-application.exe -platform webgl:port=8998

And of course you can do it in a cross-platform (duh) way via qputenv() in your main.cpp:

// ...
qputenv("QT_QPA_PLATFORM", "webgl:port=8998");
QGuiApplication app(argc, argv);
// ...

Speaking about web-browsers support, I tried several ones (except for the one with trident and his younger brother), and it worked in all of them, so it looks like WebGL is well supported in modern browsers nowadays. Although, I did experience a couple of occasional page reloads, and also Chrome on Android tablet even crashed once, so apparently not that well, but that's really outside the Qt's scope.

With regards to performance, the most "busy" time is during the initialization phase, when web-browser is receiving buffers, textures, glyphs, atlases and so on. After the first draw call, the bandwidth usage is pretty low. And by the way, since OpenGL ES calls are sent as binary data, it should be more "light-weight" than VNC. I am actually thinking about writing another blog-post to compare WebGL streaming and VNC in terms of network utilization.

Some demos

There is already a fair amount of demo videos in previous blog-posts, and here's also another nice compilation, so I decided to create a couple of my own.

Device Information

This one is a rather simple demo application. It gathers some information about the platform it is running on. For example, here's what it shows when I run it on my Mac:

Let's now run it on Raspberry Pi using WebGL streaming plugin, connect to it from a web-browser on the same Mac and ascertain that it no longer reports Mac OS as operating system and that platform is webgl now. There is one more thing to see here - pay attention to changing values of the "screen" resolution:

As you can see, when I resize the browser window, application (beside nicely adapting its layout) reports changed screen resolution values. This is because it takes canvas dimensions for the screen resolution. Let's check that in web-browser inspector:

By the way, we can take a look at user input events here as well:

So application does indeed run on Raspberry, and what I have in my browser is just its "streamed" GUI.

Camera

This demo is a bit more practical one - it's a camera controlled by robotic-ish arm which is mounted on a Raspberry Pi device:

The idea is to control the camera (its pan and tilt) with a Qt-based application running on the device, but to do that remotely from a web-browser on some tablet. And of course we would like to see what cameras is looking at (its viewfinder). And it also would be nice to make photos with the camera.

Here's a list of required hardware for such a setup:

• Pan-Tilt HAT;
• Raspberry Pi 3 Model B+. I could choose RPi Zero as well, but this HAT fits better on a "full-sized" RPi;
• Pibow 3B+ Coupe Royale;
• Raspberry Pi Camera Module V2.

And that's the manual I used to assemble it.

Now let's take a little detour (spoiler: I will be promoting Qt's commercial features). You might have noticed that for Device Information demo I used Boot to Qt image, saving myself quite some time and efforts with regards to building Qt-based application for Raspberry Pi and deploying it there.

But Boot to Qt is a commercial-only feature, and without it you'll have to go though some more steps setting up system environment. Here's what it takes with a regular Raspbian Stretch Lite image as an example:

• Since Qt Multimedia module is used, you need to make sure that GStreamer is installed in the system and you have correct plugins available;
• Get the latest Qt build (5.12) for Raspberry Pi. Either set a cross-compilation toolchain on your desktop or build it right on the device. Building Qt from sources directly on Raspberry Pi is actually a viable option (especially if you failed with cross-compilation toolchain), although compilation itself takes around 10 hours and requires some dances with increasing available swap size (1 GB of RAM is really not enough);
• Build V4L driver to make camera discoverable by GStreamer and thus Qt;
• Come up with a convenient way of building/deploying your applications on device.

Even though this list of steps is not too long, in practice it can take you up to several days before you get a working setup, whether with Boot to Qt image you get everything working out-of-the-box, and you can run your applications on the connected device right from the Qt Creator. But enough with the promotional part, let's get back to demo.

The GUI layout looks like the following:

Most of the space on the first tab is taken by the camera's viewfinder, which is implemented with VideoOutput and Camera itself:

Camera { id: camera }
VideoOutput {
    anchors.fill: parent
    fillMode: VideoOutput.PreserveAspectCrop
    source: camera
}

There are two sliders, horizontal and vertical - for controlling pan and tilt of the camera:

Slider {
    id: sliderTilt
    orientation: Qt.Vertical
    from: root.maxValue
    value: 0
    stepSize: 1
    to: -root.maxValue
    onPressedChanged: {
        if (!pressed)
        {
            backend.movePanTilt(basePath, sliderPan.value, sliderTilt.value)
        }
    }
}

Pan-Tilt HAT servos are interfaced via I2C, and due to the lack of time I went with fast and dirty solution - by using Pimoroni's Python library. At some point I would like to do it properly with C/C++, although it's really not the point of the demo.

There is also a button for taking photos using CameraCapture:

Button {
    scale: hovered ? (pressed ? 0.9 : 1.1) : 1
    background: Rectangle {
        color: "transparent"
    }
    Image {
        anchors.fill: parent
        source: "/img/camera.png"
    }
    onClicked: {
        camera.imageCapture.captureToLocation(basePath + "shots/" + getCurrentDateTime() + ".jpg");
    }
}

Second tab contains a list of taken photos:

...which is implemented by FolderListModel:

ListView {
    FolderListModel {
        folder: "file:" + basePath + "shots/"
        nameFilters: ["*.jpg"]
    }
    model: folderModel
    delegate: ItemDelegate {
        text: model.fileName
    }
}

Full application source code is available here.

Now let's see it in action. There are 3 spirits placed on my table, surrounding the camera, and I want to take photos of each. I built and ran the application on device with -platform webgl, and connected to it over Wi-Fi from Safari on my iPad:

[embed width="900"]https://youtu.be/7MhMZ3qMQNI[/embed]

As you can see, the plan worked out just fine: seeing camera's viewfinder, I can remotely control its position and take photos of the objects I'm interested in.

Use cases

Most obvious use case for WebGL streaming is the ability to have a decent GUI for some low-end device with limited computing power, without GPU, and quite often without any display at all. For instance, that is a common scenario for industrial automation domain, where you can have lots of headless devices installed all over the factory: they can be distributed over quite a significant area or even mounted in places with hazardous environment - being able to control/configure those remotely comes to be rather handy.

Reading discussion at Hacker News, I stumbled upon a "reverse" idea: what if it's the other way around, what if "device" is actually a very powerful server, and you work with it from your regular desktop. That way you can can perform some heavy calculations on the server while having GUI in your web-browser (which actually begs for an HTML-based frontend but more on this in the next section).

Another possible use-case is an anti-piracy measure. Let's say you want to protect your software from being "cracked" or "pirated". Obviously, if there is nothing running on the client, then there is nothing to crack as your users only have GUI rendered in their browsers, and the application itself is running on your server. Sounds interesting, but there are several drawbacks here:

• While WebGL streaming performs well in local network, using it over the internet will result in significant latency;
• Connection is not encrypted, so it is not secure;
• Currently only one connection at a time is supported (so only one user).

Overall, supporting only one connection at a time fairly reduces the number of possible use cases, and unfortunately it is unlikely that current implementation of the feature will improve in that regard, so it is more of a task for Qt 6. By the way, there is an idea to complement streaming with an ability of mirroring as in some cases having the latter is more important.

Speaking about mirroring, I would like to mention our recent webinar that we had together with Toradex. There you can see an interesting combination of WebGL streaming and Remote Objects, which allows you to implement mirroring functionality as of now already.

Another noticeable aspect of WebGL streaming is so-called "zero install" concept - you don't have to install/deploy anything on clients (desktops/tablets/smartphones/etc) as the only thing needed is just a web-browser. However, Qt for WebAssembly seems to be a bit more suitable for that purpose.

WebGL streaming vs actual web

Some of you might ask, what is the point of relying on WebGL streaming in the first place? Since it's all about web-browser, one can just take a regular web-server and create a web-application - result will be almost the same: backend is hosted on the remote device and HTML-based GUI is rendered in the web-browser.

That is a very good and fair question. I actually have some experience in web-development, so I asked this question myself. Let's try to answer it, hopefully without starting yet another holy war.

Indeed, in some cases it is enough just to have a simple REST API, especially if you only need to get some plain text data values. So it is likely that Qt-based application with WebGL streaming would be an overkill for such purpose.

However, in more sophisticated scenarios (for example, when you need to control some hardware) Qt-based application with WebGL-streamed GUI might fit better, because that way you'll get a powerful backend (C++/Qt), and I would also mention that creating a complex, appealing and performant frontend is (considerably) easier with Qt Quick rather than with HTML/CSS/JS, but this statement does look like a beginning of yet another holy war, so I'll keep that as my personal opinion.

And the last thing worth to mention here - if you already have a Qt-based application, then WebGL streaming is an obvious option, because it will cost you nothing to have a remote GUI for it.

Licensing/pricing

WebGL streaming plugin is available under commercial and Open Source licenses (but GPLv3 only). And for commercial customers it is included in both Application Development and Device Creation products with no additional charge.

Conclusion

So you're now able to use web-browser as a remote GUI client for your Qt Quick applications with no efforts - it only takes one command line parameter.

In terms of further development, I reckon the next thing to be expected is connection security/encryption, both for WebSocket and WebServer. WebSocket part should be pretty straightforward as QWebSocket already supports secure connection (wss://). And WebServer part, if you remember, from the very beginning was a temporary solution, and research on proper implementation (including support for HTTPS) is still ongoing.

Meanwhile, if you have any other feature-requests or maybe bugs to report, please use our tracker for that: http://bugreports.qt.io/ (choose QPA: WebGL component). Your feedback will help our product management team to shape the feature's roadmap.