Perforce Software, founded in 1995, markets and supports Perforce, the Fast Software Configuration Management (SCM) system. Perforce is an award winning SCM system used to version and manage source code and all digital assets. Perforce SCM seamlessly handles distributed development and multi-platform environments and is used by more than 330,000 developers worldwide.
Recently, Perforce adopted Squish for GUI performance testing of their SCM system’s flagship GUI, the Perforce Visual Client (P4V). We discussed Perforce’s use of Squish with Tim Brazil, a Perforce Performance Lab engineer.
In mid-2009 Tim Brazil chose Squish to round out his testing arsenal and to help ensure that performance remains high as new features and bugfixes are applied to P4V. P4V, written in C++ and using the Qt GUI library, provides a graphical interface on Windows, Mac OS X, Linux, Solaris, and FreeBSD.
One feature in particular Tim liked was Squish’s support for multiple standard scripting languages for writing tests—these currently include JavaScript, Perl, Python, and Tcl. Using a standard scripting language avoids the need for testers to learn an obscure proprietary testing language, and means they can benefit from the large number of third party modules available for standard scripting languages.
Squish provides a complete GUI-based testing IDE capable of recording and playing back GUI tests, but for Tim, it was also Squish’s support for command line use that was particularly appealing.
“I prefer to work in a command-line environment that facilitates the use of scripting languages,” Tim explained. “It is evident that froglogic’s approach took engineers like myself into account when they designed the product. For example, features like the envars file, suite.conf file, squishserver, and squishrunner, allow me to design a fairly complex test environment with relatively minimal work.”
Tim has many years of experience in the testing field and is a strong proponent of using both automated and manual testing.
“I believe the best software testing solution is to use a good mix of automation and manual tools,”
said Tim.
“By utilizing the strengths of both, producing superior products can be achieved.”
“The benefits of automated testing include reliability, repeatability, comprehensiveness, and speed,”
Tim continues.
“Furthermore, automated tests can quickly and dependably navigate through test scenarios and are, therefore, more likely to uncover subtle timing problems.”
He pointed out that manual testing alone is often both slow and fallible, and that even with a comprehensive test plan at hand it can be difficult to reliably recreate the exact test actions and environment.
Tim uses Squish to test the performance of P4V, specifically on Windows Professional, Mac OS X, and SLED (SUSE Linux Enterprise Desktop). Squish is used to test the nightly build as well as the previous three P4V releases. A daily performance report is generated, identifying performance trends and pinpointing any possible areas of concern as P4V continually evolves.
Each test Tim uses is designed to work independently—if one test fails, it has no effect on subsequent tests. No third party tools are used; instead, a custom Perl script (runner.pl) runs the Squish tests. This script runs each set of Squish P4V tests for each version of P4V on each client machine, accumulating results into an XML file (see the screenshot below). Once the tests are complete, the XML results file is automatically parsed and converted into an HTML report for the test team’s review each morning.
We asked Tim what features of Squish he liked most, apart from the multiple scripting language and command-line tools support he’d already mentioned.
Squish’s Object Map came to mind:
“At previous companies I have used graphical test tools that were heavily invested in using coordinates to identify objects. This was a maintenance nightmare. Squish’s objects.map is unique and greatly facilitates test readability, robustness, and maintenance.”
Squish uniquely identifies application objects such as widgets, using the values of their properties. For every identified object, Squish also creates a corresponding symbolic name—the symbolic name is the one that is normally used inside test scripts. This means that if a developer changes one of an object’s properties, the test engineer only has to update the corresponding object property once in the Object Map to reflect the change. The symbolic name used for the object will continue to work in all the tests where it is used.
In some cases, an object property’s value may vary depending on the platform the application is being run on. Squish’s Object Map can accommodate such challenges since the properties used to identify an object can not only be matched for equality, but also using wildcard or regular expression matches—a feature that Tim has found to be particularly useful.
“One of the biggest challenges I’ve faced with automated GUI testing is the ability to maintain a robust test as the product changes and evolves over time,” Tim continues. “This is a problem because GUI application performance can vary on different operating systems. One particular issue we’ve had is being able to identify when a web page or GUI application object is ready to be tested, since load times vary. The wait*() functions Squish provides have been invaluable in helping me tune a test so that it provides consistent behavior when run on various platforms.”
In addition to Squish’s documented features, Tim has found Squish’s technical support team very helpful:
“I was impressed with froglogic’s support from the moment I started to evaluate Squish. Just because something isn’t stated in the documentation does not mean it cannot be done. Squish’s technical support team has quickly and efficiently provided solutions that work.”
Perforce’s Performance Lab depends on the reliability and repeatability of Squish tests to check application performance as well as behavior across multiple platforms. Squish’s usefulness and flexibility has allowed Perforce to rapidly adopt Squish as an integral part of their performance quality monitoring process. This has lead to time and cost savings compared with the previous manual testing, and at the same time ensured that tests are automatically and reliably repeated to ensure product quality.
Making a third party tool an integral part of P4V’s testing process is a significant commitment, but one that Perforce has chosen to make with Squish.
froglogic’s team would like to thank Tim for taking the time to share Perforce’s experience with Squish, and we are looking forward to a continued successful relationship.