Qt 6.5 released a new Multimedia backend using the FFmpeg library.
Starting with Qt 6.5.2, we enabled this feature in Qt for Android! Now, you might ask what does it mean for Qt for Android to have the new FFmpeg multimedia backend? What does it provide?
This blog post will try to answer these questions. If you have other questions, please ask in the comments.
Media playback using FFmpeg
A core functionality of the new multimedia backend is, of course, the playback of video files.
Using FFmpeg multimedia backend adds three main advantages for the playback of video files:
- Compatibility with different file formats - supports several different codecs and file formats, some unavailable before.
- Differences between devices - different devices support different video formats. Sometimes that depends on the vendor and the codecs provided. It is difficult for some video files to be played on all devices. Using FFmpeg helps by allowing software playback in case the hardware playback is impossible.
- Performance gain - faster to load, but still using hardware acceleration.
On top of this, FFmpeg is a cross-platform library. Using it in Qt for Android will bring the functionality known on other platforms.
Camera2 API
By developing a whole new media backend, we cleared up some technical debt on using the APIs supplied by Android. Previously, we used the Camera API, and we changed to support the Camera2 API.
The Camera API is very stable and works well for regular app development. But to support features available on the newest Android API levels, the Camera2 API is the way to go. It opens new possibilities by allowing much better control of the capture pipeline.
Development is ongoing to support both old and new camera options, so this is not a change that app developers can use today in Qt in its final form yet, rather than a stepping stone for new features in the future.
Media encoding using FFmpeg
After setting up a new pipeline with Camera2 API, we needed to connect the Camera API with FFmpeg encoding.
Media encoding on Android with FFmpeg was an "Achilles heel" since software encoding is not performant enough on mobile devices due to memory and processing constraints on most devices.
Earlier FFmpeg did not offer hardware encoding in Android. But that changed on FFmpeg 6.0 "Von Neumann" major release.
We looked closely into the development of the hardware acceleration feature in FFmpeg, and since its release, we contributed several patches to the FFmpeg media library.
The contributed patches enable AV1, VP9, and VP8 codec encoding using hardware acceleration. These patches are already merged into the master branch but have not been released yet. They will be available in future releases of FFmpeg and Qt.
