Multi-Platform Screenshot Verifications

As a software tester, you'll likely want to run your tests on as many platforms as possible. A single test suite which you can execute on all systems would be easiest to maintain. If you are using screenshot verification points (VPs), this may not work out-of-the-box. With screenshot VPs, which record a specific screen capture, they are likely to fail across multi-platform tests.

In this article, we'll show you how to handle challenges in multi-platform screenshot verification points.

Example: Designing Robust, Multi-Platform Tests with Screenshot VPs

We'll use the Address Book example application on Windows and Linux, using Squish for Qt. This application is packaged with all Squish installations. Taking a first look at the application's main window, it's clear the differences across platforms:

As a result, any screenshot comparison will not succeed out-of-the-box. Our approach will utilize some scripting and a set of screenshot VPs for each platform.

First we create our verification points in the Squish IDE, by putting a breakpoint on the script where we want to create the VP. Once we have created all the VPs, we use a folder structure, as in the below image, to store the VP files:

In this example, the screenshot verification VP_mainwindow simply compares a screenshot of the initial window from the Address Book. This looks as in the below image, on Windows systems:

While we placed the verification points in the test case resources here, you can also store them in the test suite resources or the Global Scripts folders.

For the scripting part, we are using the following code example:

import names
import os

def main():
    startApplication("addressbook")
    snooze(5)
    RunVerification("VP_mainwindow")

def RunVerification(name):
    directory = currentApplicationContext().osName
    test.vp(os.path.join(directory, name), "Verification point on {}".format(directory)) 

The RunVerification function decides from a Python function which folder the VP file is in that should be used. In our example, this is based on the operating system of the Application Under Test (AUT). An alternative would be a trial-and-error approach which tries all variants of a VP:

def RunVerification(name):
    testSettings.silentVerifications = True
    for root, dirs,_ in os.walk("verificationPoints",topdown=True):
        for directory in dirs:
            if test.vp(os.path.join(directory, name)):
                test.passes("Visual Verification {} passed on {}".format(name,directory))
                testSettings.silentVerifications = False
                return
    test.fail("Visual verification {} failed".format(name))
    testSettings.silentVerifications = False

This approach is more interesting for engineers working with different resolutions or color schemes, and who have no concrete way of determining which set of screenshot VPs should be used.