Some of you may have noticed a new committer to the Qt git repository. Someone who is tirelessly working night and day, but only seems to make merge commits. I am of course talking about Qt Continuous Integration System, who in addition to never getting any time off, never gets paid either.
This is because Qt Integration is a Python script, written by Simon and Paul, with invaluable assistance from Rohan and Jesper from the QA team. (And just to make sure: "invaluable" means "of immense value, so great that it cannot be measured", and not "so small that it cannot be measured".)
So what does this script do? First a little detour: If you are still reading this, you have probably been following the commit logs and seen commits of the type "Merge branch '4.6' of scm.dev.nokia.troll.no:qt/oslo-staging-2 into 4.6", but you may have wondered: What is a staging branch?
A staging branch is a git branch that is shared between a number of QtDF developers and is tested separately and then integrated into the public branch at intervals. The point of having staging branches is to catch build breakages and regressions before they make their way into the shared repository. Until now, the integration has been done manually, and there was no common baseline to compare against, so it was hard to compare the quality of branches: If there is a failure in one of two branches, is it introduced in the first, or fixed in the other?
Another problem we've had recently is that our test farm is now running on virtual machines, but most of the tests were written on real machines. Some tests end up exposing the difference, and suddenly turn unstable because of timing differences or other subtleties. We are fixing these tests of course, but in the meantime, this adds a whole new level of uncertainty to the interpretation of test results.
The Qt Integration script automates the process and solves the problems. The system works roughly like this:
1. Developers push changes into one of the integration branches.
2. Direct push access to Qt's 4.6 branch is _disabled_.
3. The script takes one integration branch after the other, merges the branch with Qt 4.6 and runs the resulting merge through the autotest system.
4. After a successful build of Qt and the execution of the tests, the script compares against the previous 4.6 results.
5. If there are no new regressions (*), the merged branch is pushed to 4.6
6. However, there is a "blacklist" of test functions that are known to be unstable, and those are allowed to regress without blocking the integration.
7. Then the next staging branch is tested. (goto point 3 :-))
(*) No new regressions means: Tests are failing in 4.6 are allowed to also fail in the integration branch. No other tests are allowed to fail, unless they are blacklisted
Note that at any point, 4.6 contains a version that has already passed the testing process.
What this means is that in theory, the Qt 4.6 branch will always build, and the number of autotest failures will steadily decline. Yes, I did say "in theory". The biggest limitation is that the system does not currently cover all the supported platforms. Currently we are doing Linux/X11, Linux/QWS, one Windows version, and one Mac version. We are going to add new platforms as we go along, but the system will probably never cover all the Tier 2 platforms. Any failures on the remaining platforms will be caught by the autotests running continuously on 4.6.
This means that we will still keep the 4.6-stable branch, just like before. The only difference is that it should now be even more stable. In theory 4.6-stable could go away in the future, if we could get the test cycle for all of our platforms to be fast enough. Also, please note that we are only running this on 4.6 at the moment. We will extend it to the master-going-to-become-4.7 branch in the not too distant future (but not this year).
Finally, I have to apologize for the lack of cool screenshots in this post. This is a script after all; just move along, nothing to see here...
OK, then, if you insist, here's one of the script running, taken just moments ago:
