Adapting Qt Jambi to a new minor version of Qt

Ever since Nokia announced that the official support for Qt Jambi would end after the 4.5 series, I've had a few people ask me how they can help adapt the Qt 4.6 changes to Qt Jambi. I thought this blog would be a good place to offer my experiences on the subject, so here's a short guide. I will use the current state of the Qt 4.6 branch as an example and try to focus on the different steps and considerations you will need to take to get a functioning Qt Jambi 4.6.0.

Please note that I won't actually make all the modifications necessary, but I'll try to outline what you need to do in order to get there yourself.

Prerequisites
In order to be able to do this work, you should be confident with both Java programming and C++ programming. You should also read up on JNI (no magic there, but it will help if you recognize the function calls.) Knowing a little bit of Qt Jambi and Qt, so that you understand the design principles behind the APIs, is also very useful. And you should read up on the Qt Jambi type system, and try to get familiar with the binding code generated by the Qt Jambi Generator.

There is a lot of complexity to manage in the binding code, so try to keep a cool head. Even if you don't understand each detail in the generated code, it is helpful to understand how a function declaration in C++ ends up as a functioning Java method once you have run the Qt Jambi Generator. Please read the article on the Generator in Qt Quarterly issue 29 [pdf]. This contains an overview of the bindings-layer in Qt Jambi and some reasoning behind the complexities within.

Getting the sources
First of all, you need to get the sources for both Qt and Qt Jambi, and you need to be able to build them. This is also covered elsewhere, so I won't spend much time on it here, but just link you in the right direction. What you need to do is:

1. Make a local clone of the Qt repository. The command to do this is listed in the header of the repository home page.
2. Build Qt (should be well-documented elsewhere.)
3. Make a clone of Qt Jambi in Gitorious so that you have somewhere to upload your work. This is done by making a user in Gitorious and hitting the "Clone this repository" link on the Qt Jambi repository.
4. Make a local clone of the repository you just made on your hard drive. The command to do this is listed on your repository's home page.
5. Build Qt Jambi. There are helpful instructions in the project wiki.

I myself have made a clone of the Qt Jambi repository called "eskils-clone" which contains all the changes made in the course of this blog.

Getting started
The first thing I always do when porting a new version of Qt to Java, is to run the Qt Jambi generator. This is part of the top level ant build, but sometimes it helps to run it manually so you can properly see all its output.

Example on Windows (assuming %JAMBIDIR% contains the location of your Qt Jambi clone):


cd %JAMBIDIR%generator
releasegenerator --output-directory=..


NOTE: Two handy command line arguments for the Qt Jambi Generator are --dummy and --diff. The former will run a dummy run, meaning that it will take all steps related to generating the files, except for actually writing the files to the disk. That way you can test the impact of your changes without actually committing anything to the generated files. The --diff argument causes the Generator to show you a detailed listing of the changes which will be made to each generated file.

The final output from the Generator is the number of files generated during the run. The number in parentheses is the number of files which were actually overwritten (because they had been modified since the previous run.)

In the current state, you will see lots and lots of warnings. Most of these are due to missing declarations in the type system for new APIs. The very minimal work we have to do when porting new API to Qt Jambi, is to go through all new types and identifying them in the type system. To help us in this work, the Qt Jambi Generator outputs a set of .log-files which contain potential issues.

1. mjb_rejected_classes.log: Lists all (implicitly or explicitly) rejected classes.
2. mjb_rejected_enums.log: Lists all rejected enum types.
3. mjb_nativepointer_api.log: Lists all functions that use QNativePointer as argument type or return type.
4. mjb_object_type_usage.log: Lists functions that are in danger of passing an object from C++ into Java that has limited life time.
5. mjb_reference_count_candidates.log: Lists functions that (based on their name and signature pattern) may retain references to objects owned by Java.

I'll look at each of them in order. It is useful to follow the order from the list above, as modifications made to fix some issues might create more entries in other .log-files. For instance, if a class has no type system entry, its functions will not be checked for e.g. native pointer API. Thus, when the class is added to the type system, you can get several more issues in the nativepointer API log.

Rejected classes
As a first step, we go through the "mjb_rejected_classes.log" file. This contains a list of "rejected classes". The rejected classes are the classes which were defined in the Qt headers, but which for some reason did not make it into the generated Java API. In the beginning of the file, there is a section called "Not in type system". This section contains the names of all classes which were rejected because they had no type system entry, the ones which were implicitly rejected. These are the ones we will focus our attention to, as all the other entries in the list have been rejected on purpose.

For each of the entries in the list, we need to identify two things: Which module it belongs to, and whether it is an object type, a value type or an interface type. Most of the time, this is easy, but tedious work. We go through the list of classes one-by-one, and make entries for them in the correct type system.

Each type system consists of three files: typesystem_<module>-common.xml, typesystem_<module>-java.xml and typesystem_<module>-java.java. When running the ant task, the first two will be merged into a single file called typesystem_<module>.xml. Make sure you edit the correct files, because if you edit the merged file, your changes will be overwritten the next time you run the ant build.

The reason the type system is split up this way, is that the "common" file should contain data which is not specific to any particular target language, and could thus be used for generating e.g. QtScript or Python bindings. Anything which is particular to Java and the JNI binding layer (like Java code injections etc.) needs to go into the Java-specific XML-file.The typesystem_<module>-java.java file contains snippets of Java code which are injected into the generated classes in Qt Jambi. It has been formatted to be viewable in a Java editor and get correct highlighting. This is simply a convenience to make maintaining the code snippets easier, as you can also put the code directly into the XML files.

1. The first class we see in the mjb_rejected_classes.log file is "QAbstractAnimation".
2. Looking at the documentation, we can tell that this class belongs to the "QtCore" module. It should thus be declared in the typesystem_core-common.xml file. We put all type declarations in the common file, since this information is useful to any language bindings.
3. Looking further at the documentation, we see that this class inherits QObject. That means it's definitely an object type, since our design principles do not allow value types to inherit from object types, and since QObjects cannot be interfaces in Qt.

Based on this reasoning, we make an entry in typesystem_core-common.xml as follows:

Merging the XML files and running the Qt Jambi Generator again, we'll see that the QAbstractAnimation entry is missing from the new mjb_rejected_classes.log file:


cd %JAMBIDIR%
ant generator.xmlmerge
cd generator
releasegenerator --output-directory=..


Remember to run the "generator.xmlmerge" ant task first, otherwise your changes will have no effect when running the Qt Jambi Generator. All types in QtKinetic and the QtStateMachine framework are object types, so I tagged them all as such.

To identify the meta type of a given class, it's useful to look at its contents and how it's used.

Lets do one more. In the list of rejected classes, we find the class "QByteArrayMatcher::Data". Looking at the documentation, we see that this class is not documented. This usually implies that the class is internal and should not be mapped to Qt Jambi. Since documentation is not always up-to-date prior to release, however, we can check the source code to be sure. In this case, the class is declared as private to QByteArrayMatcher, so we won't be able to access it from the binding layer, even if we wanted. In cases like these, we will explicitly reject the class:

In my clone I've added the type declarations for all new classes in the modules we currently support in Qt 4.6. For new modules (such as QtMultimedia) you would need to set up a type system for the module first, mimicking the approach taken in the other type systems.

Rejected enum types
The next step when all classes have been mapped, is to look at the mjb_rejected_enums.log file. The relevant section is "Not in type system" in this log file as well.

In the case of enum types, you will need to consider three things:

1. What module the type belongs to
2. Whether you want to map the type
3. Whether the type is a regular enum type or a flags type.

In the last case, the enum type will have an accompanying QFlags declaration, which needs to be specified as the flags type of the enum type.

Take for instance the QAbstractAnimation::DeletionPolicy enum type:

This is a regular enumeration and is not used as a flag type, so we simply declare it in the type system. A enum type such as QGraphicsEffect::ChangeFlag, however, is clearly a flags type, and must be declared as such:

I've done this for all new enum types.

Looking at the warnings
Reading the warnings produced by the Qt Jambi Generator can often give you useful hints about why the generated code might not compile. Therefore it's handy to go through all the warnings one-by-one, and either fix them or at least understand why they are happening before we move on. Some warnings alert you that a Java version of a given function has not been generated because either a return type or argument type was not recognized. This is e.g. the case with "skipping function 'QByteArray::QByteArray', unmatched parameter type 'Qt::Initialization'". The argument type "Qt::Initialization" is an internal enum type and was rejected when the enum types were mapped in the type system. Thus, the constructor expecting such an argument must also be internal, and should be rejected. We thus want to ignore this warning, and, indeed, all warnings regarding the type Qt::Initialization, so we add the following to the type system:

This will remove any warning regarding a function or field which has been skipped because it uses the internal type. The other warnings of the same type need to be considered individually.

Another warning which occurs alerts us of "private virtual functions". This is a sign of either a bug in the original API, or a hack in the original API which cannot be mapped into Java, where a virtual function is overridden as private in a subclass to avoid it being called from the outside in the cases where the object has been typed to the subclass. There is usually nothing you can do in such a case unless you can change the original API, so such warnings should usually be suppressed as well.

I've suppressed all warnings that we do not intend or need to fix.

A third warning which we can also suppress is regarding classes with equals operators but not qHash() functions related to them. These classes will be slow when used as key types for a hash table. If we think it makes sense to provide this functionality, we can implement a qHash() function for the classes (e.g. in "%JAMBIDIR%qtjambi_coreqtjambi_core_qhashes.h"). Otherwise we can suppress the warnings.

There is one warning concerning an "unsupported default value". This is a non-critical warning, as the only result is that the function in question will not have a default value for its argument in the Java version. It should be fixed when cleaning up the rest of the API, using the replace-default-expression tag, but we can leave it for now.

The remaining warnings are more severe. Lets look at them one-by-one.

Shadowing: QObject::children() const and QGraphicsObject::children() const; Java code will not compile
This warning informs us that the class QGraphicsObject uses function shadowing in C++, something which is not allowed in Java. In order for the code to compile, we either need to rename the Java version of the function to something which does not collide with function names in the superclasses, or we need to reject the function.

In this case, we can see from the code that the function declaration is an implementation detail to select the correct instance of a function in the multiple inheritance graph. Since we don't have this problem in Qt Jambi (QGraphicsItem::children() is already rejected by the type system since it is deprecated), we simply reject the function:

namespace 'com.trolltech.qt.core.QXmlStreamReader' for enum 'ReadElementTextBehaviour' is not declared
This warning message tells us that we have wrongfully placed the enum type "ReadElementTextBehavior" in the type system for QtCore, rather than QtXml where it belongs. Even though the QXmlStreamReader currently resides in QtCore in C++, it resides in the XML module in Qt Jambi. The reason for this is that the class was originally defined in QtXml in C++, but moved at a later time. This change was binary compatible in C++, but not in Java, so we could not move the class in Qt Jambi.

Therefore, we need to remember to place declarations directly related to this class in the XML type system instead. Fixing this error is simply a case of moving the enum-type declaration.

Duplicate enum values
In Qt Jambi, it is not possible for several enums to share a single value. When this happens, we have to reject all but one of the offending enums.

Sometimes the cause of the error is that an enum has been renamed in C++. Since there are no restrictions on the number of enums that can share the same integer value in C++, renaming will sometimes happen by introducing a new identifier and deprecating the old. This is the case e.g. for QWebSettings::WebAttribute, where the "LocalStorageDatabaseEnabled" enum has been renamed to "LocalStorageEnabled". In these cases, we must prefer the old identifier in Java to maintain binary compatibility. In such cases, the C++ and Java APIs will necessarily diverge:

Other times, several enums have intentionally been given the same value in C++ as part of the API. For instance, QWebHistory::HistoryStateVersion has a "DefaultHistoryVersion" which points to the current default among the other enums. In this case, we have to reject the DefaultHistoryVersion enum, and lose that part of the API.

A final problem is in the case of Qt::Modifier. In this case the Generator reports that all enums are set to zero. Looking at the code, we see that the enums in C++ are assigned to values in another enum type, and this is not currently supported by the Qt Jambi Generator. Until the Generator can be adapted to support this syntax, we have to reject the enum type Qt::Modifier from the type system. This will undoubtedly cause problems down the line, when new API begins using the enum type, but for now, we ignore it.

Class inherits from polymorphic class, but has no polymorphic id set
This will be the case when e.g. new events or style options are added to Qt. In the type system, we need to address how the binding layer can differentiate between different subclasses of a polymorphic type which is not a QObject. The expression that is generated for QEvent in particular, is given in the declaration in the type system:

This declaration tells us that QEvent is the base class of a polymorphic hierarchy of classes. The "%1" token is replaced by the object name in the generated code, and the given expression is evaluated at runtime to decide whether the object is of the given type. In this case, the object is an object of QEvent (and not a subclass) when the type() function returns QEvent::None.

We need to add similar expressions for the declarations of the new events. For instance, we add the following to the declaration of QTouchEvent:

In either of these three cases, a QEvent object should be cast to QTouchEvent.

Signature for function modification not found
This seems to be a bug in the Qt Jambi Generator, but it will have to be studied more closely. For the time being, we can ignore the warning.

Try compiling it
Then the time has come to try to compile Qt Jambi 4.6. At this point, we should have a pretty good start, since we've gone through all warnings and fixed the ones that would hinder compilation. However, there will most certainly be minor issues left that we need to fix before we are done.

The only way to find out, is to run the ant build and wait for the results. Then we need to go through each compile error and see what we can do to fix them. I'll list the errors I get here, and the solution I find for them.

Cannot open include file: 'foobar.h': No such file or directory
The compiler reports that it's unable to locate certain include files required by the generated code. This is because the classes have wrongfully been declared in the QtCore module in the type system, but are actually declared in QtGui in the C++ headers. Since the QtGui headers are not available when compiling the QtCore module, compilation fails. The fix is to move the declarations in question into the GUI type system.

cannot convert from 'QEasingCurve::EasingFunction' to 'double'
The compiler reports a couple of errors regarding conversion between QEasingCurve::EasingFunction and double. Looking at the documentation for the former, we see that it is a typedef for a function pointer. Since we do not support function pointer APIs in Qt Jambi, the Generator is confused, and we need to reject the functions that use this type in the type system.

Using QEasingCurve::EasingFunction in place of double, you would get a warning telling you the correct syntax of the signature in the eyes of the Qt Jambi Generator. Since it does not understand the function pointer type, it has assumed that EasingFunction is a typedef for "double", and thus we must use this in our signatures (unless we actually declare QEasingCurve::EasingType as a primitive type in the type system.)

In order to match the API in C++ as well as possible, we have to make sure we adapt a similar, hand-written API in Qt Jambi. This will be left for later. Until then, custom easing curves will not be supported by Qt Jambi.

Compile errors for QGenericMatrix
When compiling the QtGui module in the bindings, we see errors related to QGenericMatrix. Looking at the documentation for this, we see that it's a template class, and thus cannot be mapped to Java.

Rejecting it, however, will also reject all functions that depend on the class. For the time being, we accept the occlusion of this API from Qt Jambi. Later on, however, someone needs to make sure this API is properly mapped, e.g. by mapping the different specializations of QGenericMatrix to Qt Jambi instead, or perhaps by handwriting some of the functions.

function 'fooBar' already has a body
When compiling the QtGui module, we get several similar errors concerning functions that "already have bodies", i.e. they have been defined twice in the library. The problem in these cases is usually that the there is a const overload in the C++ API which otherwise has an identical signature. In Java, there is no such thing as a const specifier that alters the signature of a function, so we need to reject one of the overloads.

Another common cause of this error is that the function has overloads which take different types in C++, but which have been mapped to a common type in Qt Jambi. This can be the case, for instance, when two types both map into QNativePointer. An example of this problem is the pair of functions QPixmapCache::find(const QString &, QPixmap &) and QPixmapCache::find(const QString &, QPixmap *). QPixmap is a value type, so both signatures become QPixmapCache.find(String, QNativePointer) in Java. Since the latter overload provides no additional convenience in Qt Jambi, we reject it in the type system.

setAttributeArray(com.trolltech.qt.QNativePointer,com.trolltech.qt.QNativePointer) is already defined
Since the C++ code now compiles, we move on to the Java compilation. We see several errors indicating problems in QShaderProgram. Looking at the documentation, we see that the class has several overloads which take char* name as the first argument. At some point, each of these should be handled especially, using a java.lang.String for the name argument instead. However, for the moment, we only want to make the code compile, so we focus on the ones that break compilation.

Since the disambiguating argument in each of the overloads is a pointer to a value type, they have all got the ambiguous setAttributeArray(QNativePointer, QNativePointer, int) method. For now, we rename them all, with a comment saying that the API needs to be cleaned up.

Another similar error message in the same class is caused by the fact that Java does not support unsigned integer types (with the notable exception of char.) Therefore, both int and unsigned int will map to int in the Java code, which can cause ambiguities in overloads. In the case of QShaderProgram, we reject the unsigned integer version of the function.

We also reject a couple of overloads with two-dimensional arrays with a fixed size in the signature, since the Java does not support that. The result is that API is still usable, but not convenient, so it should be fixed before the API can be considered ready for use.

clone() is already defined in com.trolltech.qt.webkit.QWebElement
The final Java compile error is due to a bug in the Qt Jambi Generator. For classes where it's possible, Qt Jambi will automatically generate a clone() method to implement the Cloneable interface. If the class already has a function called "clone" with no arguments, this will of course cause problems. I've made a tiny fix to the Generator in this case, causing it to skip creating the clone() when one already exists. If the existing clone method does not have the correct signature, compilation will still fail, so the fix is not complete, but it will fix the problem for the time being.

Cannot find symbol : variable DefaultHistoryVersion
From earlier, we remember rejecting the enum "DefaultHistoryVersion". Apparently, this is used as a default value for an argument in a function, which causes the code to fail compiling. We need to remove the default value for now. A better solution for later might be to manually add DefaultHistoryVersion as a variable in the class, and have it point to the latest version. However, this would also require that we maintain the manually written code for each release to make sure the DefaultHistoryVersion is updated, so it might not be worth the trouble.

Tweaking the APIs
So Qt Jambi 4.6 finally compiles, but there is still much to be done. I've noted along the way that much of the convenience of the original API has been thrown away because of missing support in the Generator, or because of fundamental differences in the language. In many of these cases, we will have to try to adapt the intention of the API, and write code by hand to do so.

When this is done, it is also necessary to go through the remaining three log files. The native pointer API log gives us a list of methods which are currently difficult to use for users of our API, but probably functional. In these cases we need to take measures to introduce a more readable API. Some entries in this list are intentionally left unhandled, as they are instances of low-level API which needs the functionality of QNativePointer. Most of the API should be ported to use more easily approachable types, however.

The last two log files both give us the names of functions that the Generator thinks will potentially cause crashes with the current state of the type system. We need to look through these, read the code of the functions, and make decisions based on our best assumptions.

The API cannot be considered ready until all this has been taken care of, and there are no more warnings and no more entries in the log files which have not been intentionally left unhandled after close consideration.

The road ahead
I've already done a lot of the work necessary to make Qt Jambi 4.6 work and compile, but there is still a lot to be done. I haven't yet tested compilation on any platform other than Windows, and I haven't addressed any of the deficiencies in the automatically ported API as mentioned in the previous section.

In addition to this, several of the new examples and demos should be ported in order to test the new bindings and usability of the new API, and there's the question of the modules which have not yet been ported to Java.

UPDATE: If you wish to contribute to the port of Qt Jambi to 4.6, there is a repository dedicated to this. It contains the changes discussed in this blog. Send me a message on Gitorious if you want to be added to the qt-jambi-community group and have write access to the repository.

Good luck with Qt Jambi 4.6!