Concurrent Computer Corporation, founded in 1966 as Interdata, Inc., is a recognized leader in providing mission-critical, real-time computing solutions to the aerospace, government, industrial, and military market sectors.
One of Concurrent’s flagship products is NightStar™, a powerful, set of integrated tools for debugging and analyzing time-critical Linux applications. These tools have various GUI front-ends, such as the NightProbe data monitor and the NightTune performance tuner, that are written using C++/Qt 4. (NightProbe and NightTune are shown in the screenshots—which can be clicked to enlarge or shrink.)
NightStar tools run on all the most popular x86-based Linux platforms (both 32- and 64-bit), including Concurrent’s own RedHawk™ Linux® (a real-time Linux), Debian®, Fedora®, OpenSUSE®, Red Hat®, and Ubuntu®.
We were delighted to have the opportunity to discuss Concurrent’s use of Squish with Jeff Hollensen, their Chief Software Architect.
In February 2009, Concurrent encountered some problems with one of the NightStar tools when executing under heavy load—analyzing and controlling an application with hundreds of threads. They discovered that in some very rare cases the tool worked incorrectly, but they found it very difficult to consistently reproduce the problem. As a result, Concurrent decided that they needed to apply continual automated testing during product development. They hoped that such testing would help resolve the problem at hand or any other problems that might arise, and also help ensure long-term product stability.
Concurrent develops the NightStar GUIs using C++/Qt 4, so they needed to find compatible testing tools. They searched the Internet and one of the products they discovered was Squish.
They considered some of the tools they had researched and did an evaluation of Squish as the one that seemed the most promising. Jeff told us why Concurrent decided to adopt Squish:
Squish was the only full-featured solution which provided both interactive and batch testing, a well designed testing subsystem from which we can reuse components, and access to the underlying GUI toolkit used by our product in test scenarios.
Jeff also mentioned that:
The support we received from froglogic during our evaluation period was excellent which also helped in our decision.
Having licensed Squish, Concurrent then started to create a body of tests to help ensure the quality of their NightStar tools.
Concurrent tests a large number of Linux distributions as well as their own real-time RedHawk Linux which runs on a wide variety of hardware components. A critical feature of Concurrent’s testing process is regression testing, both to ensure that their products continually improve, and to prevent “old” problems from recurring. In practice, Concurrent creates a regression test for every reported problem, and these tests are added to their automated test facility which runs 24×7.
As Jeff made clear, manual testing is simply not sufficient for Concurrent’s needs:
With a relatively small staff, it would be impossible to continue to do significant development and still manually regression test our GUI tools while meeting the same quality goals.
One feature that Concurrent found especially useful is Squish’s ability to record GUI activities as an editable script (in Python, JavaScript, Perl, or Tcl, according to user preference). They often use this facility to create an initial test, and then edit the script to put in verification points and additional testing logic. Concurrent write their tests in Python, and were pleased to be able to use an open non-proprietary language for their test scripts.
Jeff told us that Concurrent also makes great use of the Object Map (a Squish feature that makes it possible to robustly identify application objects—such as controls—even in the face of application changes). And of course, they also use the Squish Spy when they want to inspect objects in running applications.
Perhaps the most challenging aspect of the NightStar testing is that the tools being tested are reporting on live data generated by the underlying operating system. This means that a large proportion of the data that the tools present to the user is unpredictable. One consequence of this for Concurrent is that application object identification is more challenging than it would be otherwise. For example, they cannot identify cells in a table by their contents since the contents will vary unpredictably from run to run.
Concurrent has been able to deal with the variability of the live data by using three techniques in combination. Firstly, they make the most of the testing API that Squish provides—for example, they access cells in tables by calling methods in Qt’s QAbstractItemModel API, something that is possible because of Squish’s tight integration with Qt. (Of course, the same applies to all the other GUI toolkits Squish supports.) Secondly, they write or extend their tests by adding additional logic, in their case using Python. Thirdly, they make use of Python libraries that provide access to the underlying system data that the NightStar tools are presenting, so that they can verify that what is being shown to the user matches the actual data being produced by the system.
To achieve continual testing, Concurrent developed their own test management system called AQUA (Automated QUality Assurance). AQUA is a web-based system that manages the selection of systems, the booting of a selected Linux distribution, the invocation of the test suite, and the analysis and logging of results using MySQL. Squish provides integrations for various standard test management systems, and is also easy to integrate with custom systems such as AQUA, through its use of simple open file formats and the provision of command line tools that are independent of the Squish IDE.
One Squish feature that Concurrent does not use is screenshot verification; this is because their graphs and other visual presentations vary with every run. However, they do make great use of Squish’s ability to take screenshots—they take one for every failure that occurs and store it in their AQUA system.
Having been a froglogic customer for about a year at the time of this writing, we asked Jeff how he would characterize the support he received. He said:
They have been very responsive and have helped us learn the tool as well as provided us with better ways to approach test challenges.
Jeff also told us that Squish had delivered all the benefits that he was hoping for, including shorter testing cycles and better test coverage and reliability. And perhaps most important of all for Concurrent, Squish has helped them improve and maintain product quality. In addition to all the obvious benefits, this has also demonstrably increased customer satisfaction.
froglogic’s team would like to thank Jeff for taking the time to share his and Concurrent’s experience with Squish, and we are looking forward to continuing this successful relationship.
