Weatherford Petroleum Consultants (WPC) provides consulting services in such areas as geology, geophysics, petrophysics, reservoir modelling, and production technology. They also develop the Sendra core flow simulator which simulates SCAL experiments.
WPC employs about 80 people in Norway, while its parent company, Weatherford, employs around 50,000 people world-wide. Sendra is used by more than two dozen oil companies, institutes, and universities around the world.
Although WPC develop a wide range of software, Sendra is the first product that they have used Squish with. The Sendra simulator was initially created in the early 1990s, but in 2009 a new version, “Sendra 2009”, brought the simulator into the twenty first century with a modern GUI written in C++/Qt 4. Sendra is developed, tested, and deployed on Windows.
We were very pleased to speak with Arent Arntzen, Sendra’s Scrum master (project manager). Arent has been doing software testing for about two years, and Squish is the first GUI testing tool he has used.
Arent told us that he wanted to use a GUI testing tool for Sendra to avoid having to create C++ unit test harnesses that to some extent would have had to reimplement much of the functionality handled by the GUI. With Squish, none of that is necessary since Squish automatically runs the entire application under test (AUT), so testing can be done against a live running application rather than only with individual classes in isolation. And, of course, Squish can also be used to access and exercise individual classes as well as the whole AUT.
WPC discovered Squish through a web search, and also met some of the Squish team at a trade show. They began by requesting an evaluation version to see if Squish could meet their needs, and after satisfying themselves of Squish’s capabilities—and finding that it was a far better match for their needs than a competing tool that they also evaluated—they bought their licenses.
For Sendra testing, WPC have a toolchain consisting of three tools: Squish, the QtTest module’s QTest class, and CruiseControl. The Sendra testers use CruiseControl to provide continuous integration and testing, including the management of their Squish-based automated regression tests. (Squish provides a specific integration for CruiseControl, as well as integrations for various other test management tools. Squish itself uses simple open file formats and provides command line tools that are independent of the Squish IDE which makes it easy to integrate Squish with just about any testing toolchain.) WPC use the QtTest module for some unit testing, but almost all of the Sendra testing is done by Squish, as Arent told us:
We did a long search for testing software and as far as I can see we are covering all the testing we can think of using Squish. The fewer tools we need the better—and so the potential coverage of Squish is one of its truly excellent features.
Arent also mentioned that he found Squish very easy to get started with, although it did take the Sendra testers a few weeks to really understand all the Squish concepts. And now, having learned Squish, they make use of almost all of its features on a regular basis.
Arent’s team write their tests in Python, and are very happy that they are able to use an open non-proprietary testing language for their tests.
At the time of this writing, WPC have developed over thirty Sendra test cases. They expect to have several hundred test cases by the time they have finished creating their complete set of test cases for the software as it stands today. And of course, this doesn’t include the additional tests that will be created to verify bug fixes and new features as part of Sendra’s on-going development and testing process.
One area that was initially a struggle for the Sendra testing team, concerned the robustness of their test scripts. They found that application changes could sometimes lead to test scripts breaking. Fortunately, by making use of Squish’s Object Map, problems that arise because of changes to application objects can be minimized, or sometimes eliminated entirely. Arent also mentioned that his team make extensive use the Squish Spy which shows application objects, including their names (useful for identifying them in manually written tests), and their properties at the time of spying (useful for inserting verification points, either using the Squish IDE or in manually written test code).
The Sendra testers do two separate sets of testing. One set of tests is short and fast to execute—these are designed to quickly verify that a binary ready for putting in their ftp download area is sound. The other set involves more involved and long-running tests and is used to provide deeper coverage and to keep track of the application’s performance as features are added and bugs fixed.
Arent told us that Squish had brought benefits in many areas, including, better test coverage, improved application quality, improved release predictability, and improved test reliability. He also said that using Squish had reduced the number of person hours required to perform testing, and has made it possible to have shorter testing cycles. A less tangible, but nontheless important benefit, is that Squish has improved the Sendra team’s peace of mind. This is because they add tests for fixed bugs, so their use of Squish is reducing the chances of regressions, or as Arent put it:
Bugs reported are squished forever so to speak!
The Sendra testers did not do a formal return on investment (ROI) study on Squish—the time savings, quality, and reliablity improvements that Squish helped them achieve, have already more than paid for the initial investment in licenses and learning time.
After the initial few weeks learning period, Sendra’s testers mastered the art of GUI application testing with Squish. Nonetheless, from time to time, as with any users of a technical product, they needed help. Here’s how Arent summarized Squish’s techical support:
Completely timely, precise, helpful, diplomatic, and in short: outstanding!
Squish has made a demonstrable contribution to making Senda into a better product—and to keeping it reliable, even in the face of bug fixes and the addition of new features.
froglogic’s team would like to thank Arent for taking the time to share his and his team and company’s experience with Squish, and we look forward to a continued successful relationship.