QList changes in Qt 6
September 30, 2020 by Andrei Golubev | Comments
With Qt 6, changes are coming to many components. Containers are no exception. In this blog post I tried to capture the most significant changes to QList and related classes.
QVector and QList are unified
Previously, Qt featured very different implementations for these two containers: QVector being a natural and straightforward array-like container and QList being rather special in its implementation to nicely accommodate types defined and used by Qt itself. With Qt 6 updates to existing types, backed by what was done already in previous releases, it seemed that there is little to no benefit to have an implementation difference between the classes. Additionally, QVector proved to be a better choice in many cases.
Thus, in Qt 6, QVector and QList are unified and the model of QVector is used as the underlying implementation. What this means for more advanced users of the framework is that Qt 5 QList's extra level of indirection for generic types is now gone and elements are always directly stored in the allocated memory.
However, we decided that QList should be the real class, with implementation, while QVector should just become an alias to QList. The point of this is to:
- simplify porting efforts as many Qt APIs work with QList rather than QVector and we believe so does the user code, at least at the level of interaction with Qt
- make clear that QVector2D and friends are unrelated to QVector
- be in sync with QStringList and QByteArrayList naming convention
Fast prepend
While basing the new implementation of QList on QVector, we recognised one important use case: prior to Qt 6, QList had an amortised constant time of insertion at the beginning (a.k.a. prepend). To address this property, QList in Qt 6 supports optimised prepend as well and, to have more aligned container implementations, so do QString and QByteArray.
Unfortunately, everything comes at a price. In this case the price is iterator validity across container-modifying operations. Unlike QVector in Qt 5, QList (and hence also QVector) in Qt 6 allows less freedom in keeping the iterators valid between operations. As a rule of thumb, consider that any operation, that adds elements to, or removes them from, the container, invalidates the "remembered" iterators, even when the QList is not implicitly shared. More details and per-function exceptions to this rule can be found in the Qt 6 documentation snapshots, with the documentation constantly being updated to better reflect the current state of things.
QList may shrink on elements removal
Normally, QList manages its memory automatically. When growing, it allocates excess memory in advance to address further growing. Symmetrically, we wanted QList to reduce the occupied space when it is decreasing in size, especially when element removals leave large amounts of unused space. Shrinking to smaller capacity gives lower memory footprint, which is a nice property in certain use cases. This ability was a feature request already since Qt 5, however, for compatibility reasons, we could not update the code before as our users relied on the existing behaviour.
Convenient by default, such memory management mechanism can also become inefficient, for example, a long enough sequence of additions leads to repeated reallocation and extra element copying. Likewise, recurrent removals cause a lot of shrinking. In such cases, additions and removals incur costs that may be avoided.
QList::reserve() is a method that can be utilised to reduce the frequency of reallocations and hint the QList to preserve existing memory capacity as long as possible. When adding elements to the QList, reserving (and thus allocating) a known size upfront makes subsequent calls to growing functions (e.g. prepend, append, insert) boil down to the copying of the new data only. Similarly, a call to reserve() (e.g. with the current size as an input argument) will make the following removals avoid reducing the capacity.
In a way, reserve() allows to interfere into QList's memory management. To restore automatic behaviour, you should call QList::squeeze(). This will both tailor the allocated space exactly to the number of the elements stored and hint the QList that you no longer require current memory capacity to be something to hold on to. Note, however, that if your data is short-lived, calling squeeze(), which may reallocate the memory, can be costly. It may be easier (and faster) to just wait for the QList to be destroyed.
QList's memory is not limited by 2GiB
Before Qt 6, QList was limited to use at most 2GiB of memory. With a strong dominance of 64-bit architectures already today, this is a needless obstacle for users who wish to use more space for their task.
In Qt 6, the underlying size type of QList is changed to address this and it is possible to create QLists that allocate bigger amounts of memory (within the limits of what the system provides, of course). Consequently, all QList methods are changed as well to align with this and now use qsizetype, instead of int.
As a downside, users would most likely see their compiler warn about narrowing conversions. Consider the following example.
Perfectly correct code in Qt5:
void myFunc(QList<MyType> data) {
int size = data.size();
// ...
}
... would get a compiler warning in Qt 6 due to the (narrowing) conversion from qsizetype to int. A natural solution is to use the auto keyword instead of a specific type. This is also a remedy when you want to build against both Qt 5 and Qt 6.
Other smaller changes
We have also simplified QStringList and it is now an alias to QList<QString> as opposed to the Qt 5 version, where QStringList was a distinct class derived from QList<QString>. Careful readers may notice that QStringList used to have special methods not available in QList, for instance, QStringList::join(). These QStringList-specific methods are still available but are baked into QList<QString> similarly to how it is done for QByteArrayList.
As you may know, the QList implementation takes advantage of the additional traits of its element type, provided by a use of the Q_DECLARE_TYPEINFO() macro. Certain type traits allow to use faster algorithms to improve "out-of-the-box" performance. We have simplified this mechanism:
- Q_MOVABLE_TYPE and Q_RELOCATABLE_TYPE now mean the same thing. Since C++11 introduced a meaning for "movable" that is not quite exactly what Q_MOVABLE_TYPE meant (and means), we encourage client code to use Q_RELOCATABLE_TYPE instead
- trivially copyable and trivially destructible types no longer need to be marked as relocatable. They will be detected as such automatically
In any case, when in doubt about a type, you can always check and ensure specific traits at compile time. For example:
static_assert(QTypeInfo<MyType>::isRelocatable); // makes sure MyType is relocatable
You can read more on the subject in our documentation snapshots.
Do not hesitate to share your opinion, comment or suggest an improvement. Your feedback is what makes Qt better!
Blog Topics:
Comments
Subscribe to our newsletter
Subscribe Newsletter
Try Qt 6.9 Now!
Download the latest release here: www.qt.io/download.
Qt 6.9 is now available, with new features and improvements for application developers and device creators.
We're Hiring
Check out all our open positions here and follow us on Instagram to see what it's like to be #QtPeople.
Commenting for this post has ended.
In short: I like these changes very much! :)
Thanks! It is always nice to have some positive feedback :)
It is great to have a single QStringList being an alias to QList, if will prevent us from having the two different typeids in QVariant.
However, thought its good and rational to have >2Go container size, I'm a little sad to see the API becoming more complex : int size was very convenient. Using
auto
keyword will produce longer and longer build times, and disabling narrowing conversion warning is not a possibility in my business, so I guess we'll have to write casts everywhere.I'd rather continue using separate containers for large data than do this migration.
Qt5 container documentation was more readable than the current Qt6 alpha doc. Ex:
Qt6 :
QList::const_reference at(qsizetype i) const
Qt5 :
const T& at(int i) const
I wish that the C++ committee improves STL code readability, because
std::find(my_list.begin(), my_list.end(), my_var) != my_list.end()
is nowhere nearQLinkedList::contains(my_var)
.What about adding duplicate elements detection, listing and removal in QList ? (needs to know if it's sorted are not already)
Another problem with auto is that this will increase the probability of invisible integer overflow issues by hiding the type.
Even Qt's own style guide warns against using auto in cases like this.
Thanks for this article. You say that Qt 6's QList has maintained an optimised prepend operation, but the current Qt 6.3 documentation sates that " Inserting at the front or in the middle of a list can be quite slow, because it can lead to large numbers of items having to be moved by one position in memory.". And Prepending is O(n) while it was Amort. O(1) in Qt 5. https://doc.qt.io/qt-6/containers.html
Has the implementation changed since the time you wrote your post?
Is sizeof(QList) still the size of a pointer?
No, not anymore. Now, sizeof(QList) is equal to the size of 2 pointers + sizeof(qsizetype). qsizetype is a signed counterpart to size_t and on 64-bit architectures should be equal to the size of a pointer. So, in a nutshell, sizeof(QList) is equal to the size of 3 pointers. Thanks for pointing this out! I skipped this in the overview.
What is the benefit of this new layout? (I am using QVector of QVector of QVector etc... to make optimized copy-on-write kind of b-trees)
I would say that there's little benefit in the old layout rather than staying binary compatible. As our internal data structure is somewhat stable (hence no binary incompatibilities are expected), it seems easier to directly store everything in the class to eliminate needless indirection. This is what the standard library does as well and I guess for fair reasons. Also, removing additional pointer indirection simplifies the resulting machine code, which should give extra bits of performance. That said, we still support implicit sharing, just the implementation is slightly different now.
For your use case, I would recommend profiling to see if there's any regression. As an option, you could write a thin wrapper around QVector/QList that stores it as a pointer (which is, practically, what was done in Qt 5 for these containers) - this is, of course, not ideal, but at least you have a choice now and can experiment.
What is the underlying memory (de)allocation strategy when reducing a QList? I mean, does removing a single element causes a memory reallocation? (releasing “the big block” to allocate a slightly smaller one?). I believe not, but then what is the threshold?
QList grows exponentially (in powers of 2), the shrinking follows the same idea: when the size after removal is less than half of the allocated capacity, QList shrinks (which causes reallocation and excess memory release). However, this is an internal detail of the implementation, so please do not rely on it as on something bullet-proof. I specifically used opaque description so that people would not anticipate certain behaviour.
I understand this is an internal detail of the implementation, but I think the user documentation should at least get some glimpse or hint, that auto-squeezing is optimized. I don't see that in the user Qt 6 doc preview. "Removing an element" of a QList could lead at some point, to a new memory allocation,... this is not a “small thing”. I understand the purpose, you explained it, but having it as the default behavior is counter-intuitive. Hence some clear documentation will be needed.
Yes, I agree. Thanks for pointing out the documentation gap, this is something I should get to some time soon.
Hi there,
First thank you for the effort to keep updating and maintaining this awesome framework for so long.Some remarks:
You say that QVector implementation is the one that is going to be used in the new QList class ... ok if you want to. I think that i have never used QList directly. I always have used QVector.The part that lost me is the one in which QVector is going to be an alias to QList and that most APIs are going to be QList, and not the other way around?!?! Why??
Why the separation? Lists and vectors are separated data structures with different uses cases, vector will always solve most of your needs but list also have its uses cases.
Why kept maintaining separated containers from the ones of the STL, copy on write is a very good performance strategy but move semantics are here, I dont see the advantage of having to maintain a separated set of containers. Qt separated set of containers has been ones of the issues when learning Qt and is a hassle interface with other libraries that uses/expected STL containers. Could someone please help me to understand all of this?
Thanks again!!!!
Edit: For some reason I was thinking that QList was a linked list, my mistake, nevertheless i think that the fact that in Qt QList and QVector are the same it goes against STL semantics and cause confusion.
For the: "Good programmer is aware of the differences between Qt and STL containers and uses this knowledge to his benefit." That is gate-keeping at it's best.
I believe Tomasz explained first 2 questions pretty nicely.
Just to clarify once again, QList is not a linked list data structure. It is an array-like (contiguous memory) container akin to std::vector. Linked list (or simply list) data structure (not container) is called QLinkedList in Qt.
Indeed, C++ standard library containers are widely used nowadays. However, there are several reasons why we have our own containers in place of standard ones. From the top of my head:
In QList and some other classes, we use optimized algorithms that are not available in the standard library. For instance, some types, not being POD (plain old data) types, might still benefit from using things like memcpy (which is typically highly optimized) when you, as a user (not as a compiler), know extra properties about your types. Qt provides a way to tell about such properties. Additionally, certain containers may feature different implementations from the one provided by the standard, which also allows us to have performance or other benefits in certain/all cases.
Qt is a mature framework with its own user base. Simply changing one class to another is not a simple task nor for the Qt itself, nor for our users. Changes do not come cheap, they, more often that anyone would love to, involve API or behaviour changes which lead to new bugs and a lot of developer time being spent fixing regressions.
Qt5's QVector has been renamed to QList. Qt 6 QVector is a typedef for that "new" QList. Qt5 QList is gone (removed). In other words: QVector in Qt 6 is the same as in Qt 5 (well plus new APIs and fixes).
QList is simply "the default container". It is not a linked list.
1) I believe this was explained very well in the article and I completely agree with the logic of the explanation. 2) QList is not a linked list... if by lists you mean linked lists, well I have never used them, I believe this is useless type of container. 3) You cannot simply remove the COW based containers because of backward compatibility. You cannot simply replace QVector with std::vector in clients code, this would require complete redesign of many really large programs. Good programmer is aware of the differences between Qt and STL containers and uses this knowledge to his benefit.