Serialization in and with Qt

In our first part of this series, we looked at how to set up messages, combine them, and reduce their overhead in the context of telemetry sensors.

This part focuses on the payload of messages and how to optimize them.

There are multiple methods to serialize an object with Qt. In part one, we used JSON. For this, all sensor information is stored in a QJsonObject and a QJsonDocument takes care to stream values into a QByteArray.

QJsonObject jobject;
jobject["SensorID"] = m_id;
jobject["AmbientTemperature"] = m_ambientTemperature;
jobject["ObjectTemperature"] = m_objectTemperature;
jobject["AccelerometerX"] = m_accelerometerX;
jobject["AccelerometerY"] = m_accelerometerY;
jobject["AccelerometerZ"] = m_accelerometerZ;
jobject["Altitude"] = m_altitude;
jobject["Light"] = m_light;
jobject["Humidity"] = m_humidity;
QJsonDocument doc( jobject );
 
return doc.toJson();

JSON has several advantages:

• Textual JSON is declarative, which makes it readable to humans
• The information is structured
• Exchanging generic information is easy
• JSON allows extending messages with additional values
• Many solutions exist to receive and parse JSON in cloud-based solutions

However, there are some limitations to this approach. First, creating a JSON message can be a heavy operation taking many cycles. The benchmark in part 2 of our examples repository highlights that serializing and de-serializing 10.000 messages takes around 263 ms. That might not read like a significant number per message, but in this context time equals energy. This can significantly impact a sensor which is designed to run for years without being charged.

Another aspect is that the payload for an MQTT message per sensor update is 346 bytes. Given that the sensor sends just eight doubles and one capped string, this can be a potentially huge overhead.

Inside the comments of my previous post, using QJsonDocument::Compact has been recommended, which reduces the payload size to 290 bytes in average.

So, how can we improve on this?

Remember I was referring to textual JSON before? As most of you know, there is also binary JSON, which might reduce readability, but all other aspects are still relevant. Most importantly, from our benchmarks we can see that a simple switch of doc.toJson() to doc.toBinaryData() will double the speed of the test, reducing the iteration of the benchmark to 125msecs.

Checking on the payload, the message size is now at 338 bytes, the difference is almost neglectable. However, this might change in different scenarios, for instance, if you add more strings inside a message.

Depending on the requirements and whether third-party solutions can be added to the project, other options are available.

In case the project resides “within the Qt world” and the whole flow of data is determined and not about to change, QDataStream is a viable option.

Adding support for this in the SensorInformation class requires two additional operators

QDataStream &operator<<(QDataStream &, const SensorInformation &);
QDataStream &operator>>(QDataStream &, SensorInformation &);

The implementation is straightforward as well. Below it is shown for the serialization:

QDataStream &operator<<(QDataStream &out, const SensorInformation &item)
{
    QDataStream::FloatingPointPrecision prev = out.floatingPointPrecision();
    out.setFloatingPointPrecision(QDataStream::DoublePrecision);
    out << item.m_id
        << item.m_ambientTemperature
        << item.m_objectTemperature
        << item.m_accelerometerX
        << item.m_accelerometerY
        << item.m_accelerometerZ
        << item.m_altitude
        << item.m_light
        << item.m_humidity;
    out.setFloatingPointPrecision(prev);
    return out;}

Consulting the benchmarks, using QDataStream resulted in only 26 msecs for this test case, which is close to 10 times faster to textual JSON. Furthermore, the average message size is only 84 bytes, compared to 290. Hence, if above limitations are acceptable, QDataStream is certainly a viable option.

If the project lets you add in further third-party components, one of the most prominent serialization solutions is Google’s Protocol Buffers (protobuf).

To add protobuf to our solution a couple of changes need to be done. First, protobuf uses an IDL to describe the structures of data or messages. The SensorInformation design is

syntax = "proto2";
package serialtest;
message Sensor {
    required string id = 1;
    required double ambientTemperature = 2;
    required double objectTemperature = 3;
    required double accelerometerX = 4;
    required double accelerometerY = 5;
    required double accelerometerZ = 6;
    required double altitude = 7;
    required double light = 8;
    required double humidity = 9;
}

To add protobuf’s code generator (protoc) to a qmake project, you must add an extra compiler step similar to this:

PROTO_FILE = sensor.proto
protoc.output = $${OUT_PWD}/${QMAKE_FILE_IN_BASE}.pb.cc
protoc.commands = $${PROTO_PATH}/bin/protoc -I=$$relative_path($${PWD}, $${OUT_PWD}) --cpp_out=. ${QMAKE_FILE_NAME}
protoc.variable_out = GENERATED_SOURCES
protoc.input = PROTO_FILE
QMAKE_EXTRA_COMPILERS += protoc

Next, to have a comparable benchmark in terms of object size, the generated struct is used as a member for a SensorInformationProto class, which inherits QObject, just like for the QDataStream and JSON example.

class SensorInformationProto : public QObject
{
    Q_OBJECT
    Q_PROPERTY(double ambientTemperature READ ambientTemperature WRITE setAmbientTemperature NOTIFY ambientTemperatureChanged)
[...]
public:
    SensorInformationProto(const std::string &pr);
[...]
     std::string serialize() const;
 [...]
private:
    serialtest::Sensor m_protoInfo;
};

The serialization function of protoInfo is generated by protoc, so the step to create the payload to be transmitted looks like this:

std::string SensorInformationProto::serialize() const
{
    std::string res;
    m_protoInfo.SerializeToString(&res);
    return res;
}

Note that compared to the previous solutions, protobuf uses std::string. This means you are losing capabilities of QString, unless the string is stored as a byte array (manual conversion is required). Then again, this will slow down the whole process due to parsing.

From a performance perspective, the benchmarks results look promising. The 10.000 items benchmark only takes 5 ms, with an average message size of 82 bytes.

As a summary, the following table visualizes the various approaches:

One promising alternative is CBOR, which is currently getting implemented by Thiago Macieira for Qt 5.12. However, as development is in progress it has been too early to be included in this post. From discussions on our mailing list, results are looking promising though, with a significant performance advantage over JSON, but with all its benefits.

We have seen various approaches to serialize data into the payload of an MQTT message. Those can be done purely within Qt, or with external solutions (like protobuf). Integration of external solutions into Qt is easy.

As a final disclaimer, I would like to highlight that those benchmarks are all based on the scenario of the sensor demo. The amount of data values per message is fairly small. If those structs are bigger in size, the results might differ and different approaches might lead to the better results.

In our next installment, we will be looking at message integration with DDS. For an overview of all the articles in our automation mini-series, please check out Lars' post.