If you are a contributor to Qt, you should know by now that your patches must not cause any regressions. The auto-tests in Qt (located in <QTDIR>/tests/auto) are there to ensure that. These tests are run automatically by the Qt Continuous Integration System before merging a staging branch into the mainline.
As a contributor, before submitting a merge request on Gitorious, you should always run locally the auto-tests related to what you are changing to simplify the reviewer's job. For instance, if you are patching something in QGraphicsView, you should compile and run individually 19 different auto-tests (the ones starting with 'qgraphics'). If you are used to that, you probably think that it is not the most pleasant and fast thing to do...
Well, let me introduce you Qt Autotester:
This handy little tool (that was used only internally until now) will help you in this process by making it a lot more simpler. Basically, you just select a set of auto-tests then the tool makes everything for you: it compiles, runs and shows you the results in a nice way.
- You can manage several versions of Qt, save a set of auto-tests in a profile and export the results.
- It is of course cross-platform.
- You can remotely run the auto-tests on a Symbian device if you are developing on Windows (it automatically does that if Qt is built for Symbian).
The source is available here: http://qt.gitorious.org/qt-labs/qt-autotester
In a separate branch, I rewrote the UI with QtQuick and the Scenegraph to make it look even more fancy!
This project is of course open to suggestions and contributions. Enjoy!
Cool, would be nice to see someone make it works with cmake :) (not me, I am infinitely too lazy)
How hard would it be to make this same program work with any set of unit tests? Say I want to use the same pretty interface for the unit tests on my own OSS project. Can I cherry-pick this program and fill it in with my list of auto-tests, or is it not ready for that yet?
I'm as curious as Nathan: can this be used for generic autotests?
@Nathan, @razvanpetru
Yes I thought about that, and it would not require a lot of modifications. It currently parses the auto-tests project file (auto.pro in tests/auto), so it could just take any .pro file as input instead. I will definitely look into that.
I wonder, how does this handle the GUI unit tests? Some months ago, I started work on my own test running tool (http://gitorious.org/qt-com...) as part of the community integration project I was working on at the time (http://blog.rburchell.com/2...), but one of the stumbling blocks I got stuck on at the time was that reliably creating and tearing down a test instance (dbus, GUI with no WM/styles, etc) was a lot of work.
I guess I'm going to have to look into this again sometime soon as I want to start revisiting the community integration stuff.
I'd also just like to add: great news! This is exactly the sort of thing I hope to see a lot more of after writing about the frustrations that many are feeling with trying to contribute to Qt (see http://blog.rburchell.com/2...).
@Robin
It is a bit problematic for GUI tests indeed... For now you can't do anything at the same time for these tests. I'm not sure what to do about that...
@Siddharth
Thank Shane and Liang for that :)
@Rohan
Like Bjørn Erik said, the idea is just to have some helper tool to automate and speed up things so I don't think your first point is relevant.
For your second point you are right, ideally the tool should query the list of tests from some server, but then it closes the door to running external auto-tests. Or then we need a different mode for Qt auto-tests and external auto-tests...
Is this blog entry linked from our contribution model help pages? :)
+2 kudo points for making it work on Symbian. Thank you.
I have a couple of suggestions:
First, the test running logic shouldn't be implemented in this tool, because then it can't be reused in automated systems. e.g. this tool probably runs the tests in a slightly different way than the Qt Continuous Integration system itself does - especially for the more complex things such as running of autotests on symbian devices.
Second, the parsing of test definitions from the .pro files probably shouldn't be done here, because it's more error prone than proper build system integration - this tool might have a different idea about what is/isn't a testcase than other systems which are trying to run the tests. The build system knows `make check' and ideally that's the one canonical way to determine what is and isn't a testcase.
@ Rohan
I believe Yoann wrote this nifty tool to make it easier to run Qt auto tests (tests/auto/*). His goal was clearly not to make a swiss army auto tester tool supporting every single use case without being good at anything. To put it simple; it is a replacement for the following workflow:
cd $class ./tst_$class cd .. cd $anotherClass ./tst_$anotherClass cd ..... it even supports running tests in parallel :-)
@Yoann Lopes
A cross platform solution is... yeah, going to be difficult, if possible at all. The solution I settled on (as I only run Linux) at the time, was to run an Xvfb instance via QProcess. I'm happy to help contribute to make that happen if it's something you'd like to see.
However...
I noticed that some tests don't seem to like Xvfb (unreliable failures), using Xephyr instead made tests pass, but meant that I had to have an annoying window hovering around. Never figured out why. :)
That's a really nice (and fancy looking) tool. Having a test suite for making sure that nothing is broken by your own changes is a must.
What I'd like to see is something like this tool integrated into QtCreator so that real TDD is possible. If you have tried it you will not write code without tests anymore. So no only generating a new class via the QtCreator templates but also a corresponding UnitTest and running it automatically after compile would be a dream.
This a more general question: how can QGraphicsSceneEvents be simulated in unit testing? QTest has support for QWidget events (like QTest::keyEvent) but not for QGraphicsWidgets.
@Alex
Just create the event and send it to the scene with QApplication::sendEvent().
@Yoann
Since events like QGraphicsSceneMouseEvent have no public constructor or property set methods I thought there might be established way to create and send them.
I agree 100% with Jens Stockhausen. TDD helpers in QtCreator are a must.
Yes, TDD in QtCreator would be really nice. Perhaps, the Test tool could be part of the qmake project file and be run automatically after building? so it would not opnly be integrated in QtCreator, but in all qmake build process...
Second the integration of some test-driven development features into QtCreator. If you compare use of QTestLib with QtCreator to, e.g., the JUnit tools available in Eclipse...
I would really like to see something comparable in QtCreator :-).