The Qt Company, known worldwide for its cross-platform software framework, Qt, sat down with froglogic to discuss their use of the Squish GUI Tester to automate GUI tests of one of their premier development and design tools: Qt Creator. We interviewed Mr. Robert Löhning from The Qt Company to learn more about how his team is bringing high-quality, well-tested products to market through GUI test automation with froglogic’s tools.
Qt Creator is a cross-platform Integrated Development Environment (IDE) which enables developers to create applications on desktop, mobile and embedded platforms rapidly and fluidly. Qt Creator’s WYSIWYG advanced code editor allows for coding in multiple languages and includes features you’d expect from today’s IDEs, like code completion, syntax highlighting and code refactoring. It integrates with many popular Version Control Systems, features a Qt device emulator for testing and debugging, offers build and project management support built-in, and much more. Qt Creator also accepts contributions from the devoted community of Qt developers, making it a truly evolving, community-led project.
Mr. Löhning, a senior software engineer with a decade-long tenure at The Qt Company, oversees the Quality Assurance program of Qt Creator and is involved in QA practices for all of Qt. Having, in his own words, “a talent to break things and run into bugs,” Mr. Löhning became a natural fit for bringing automated testing into Qt Creator’s development efforts. We met with Mr. Löhning to talk more about how his team got started with Squish, their current testing workflows and what the future looks like for the team.
“Squish is the best possible solution for us…we didn’t see any other tool we could’ve chosen instead.”
In 2011, the team began looking into GUI test automation to supplement unit and manual testing already being conducted on Qt Creator. “Nothing was testing the tool from the user’s perspective…as a whole.” According to Mr. Löhning, the team had three requirements for a testing tool. First, because Qt Creator is available on Windows, Linux and macOS, it must support cross-platform testing. Second, the tool must allow blackbox GUI testing. That is, the tool should be able to hook into a production binary rather than needing access to production sources. And third, the tool should offer the possibility to extend its functionality through scripting. In other words, it should support library development for functionality that does not come out-of-the-box. Squish became the clear option for Qt Creator developers.
“There are cases where we don’t expect differences between platforms, and we can catch that with Squish.”
The team began their journey with Squish by streamlining the testing process of their release candidates. Mr. Löhning reported that the team began selecting manual tests conducted before each release, and turned them into cross-platform, automated tests with Squish. Some examples of tests the team was able to automate successfully include creating, building and running a project inside Qt Creator, testing for correct code completion, checking that the relevant context-sensitive help appears, and more. A list of these tests is available in The Qt Company’s public repository. Now, the team has several dozen Squish tests running continuously through a Jenkins CI server, relieving the burden of repetitive manual testing from Qt Creator’s development team.
Using Squish for GUI test automation has proven to be an effective regression detector: the team has reported nearly 200 “found by Squish” issues in their bug tracking system.
Mr. Löhning employs a multi-pronged approach in his testing. Not only is he using Squish to supplement manual and unit testing, but an ongoing project of his is the use of “fuzz testing.” Fuzz testing, or simply, fuzzing, is a method of software testing in which one inputs massive amounts of random data (or “fuzz”) into a program’s code in an effort to induce program crashes, freezes or memory access issues. Using Google’s fuzz testing infrastructure, OSS-Fuzz, Mr. Löhning reported finding defects in the code that were otherwise not apparent. Mr. Löhning’s multi-faceted testing efforts, combining GUI testing with Squish with manual, unit, fuzz and plugin tests, have secured Qt Creator’s place in the market as a premier IDE with exceptional quality.
Looking towards the future, Mr. Löhning and his team would like to extend their current arsenal of Squish tests by turning existing manual tests into automation candidates. He also mentioned wanting to utilize Squish for other Qt offerings, like Qt Design Studio, a UI design and development tool for rapid prototyping and development of complex User Interfaces. Lastly, because there are multiple pre-requisites to run Qt Creator, including a Qt installation, compiler and the appropriate toolchains, Mr. Löhning, together with froglogic engineers, will work towards benefiting from containerization. That is, having a suitable, ready-to-go environment in which to do the testing, such that other development engineers can contribute to, augment and help maintain the automated GUI tests created with Squish with ease.
We look forward to continued partnership with Mr. Löhning and his team, and The Qt Company as a froglogic partner, as we work to augment support for the diverse ecosystem of Qt-based applications and development tools.
