View Squish Test results of merge requests in GitLab

Introduction

It's very important to make automated GUI tests a part of a Continuous Integration (CI) process. Squish offers add-ons for many popular CI tools like Jenkins, Bamboo and TeamCity. The full list is here. If your tool is not on the list, it can be still integrated as Squish tests because we support a command-line interface for all of our tools. Moreover, Squish can generate JUnit reports, which can be parsed easily by many CI or Test Management tools.

GitLab

This article demonstrates how to run Squish tests on merge requests in GitLab. Squish will generate a JUnit report and we will configure a job in GitLab to use this report. Later GitLab will display a test summary on the merge request page. It contains information about newly failed tests, existing failures and resolved failures. Based on the above data, a decision about merge is made.

Runner Settings

The first step is to configure GitLab Runner. We will configure the following environment variables:

• SQUISH_DIR - Squish Installation directory
• SQUISH_LICENSEKEY_DIR - Squish license location
• SQUISH_SERVER_PORT - port at which squishserver process is started
• DISPLAY - display used to display an application GUI. We use VNC Server to provide a headless display

Job Configuration

Next step is to define a pipeline in .gitlab-ci.yml file. Pipelines are defined by specifying jobs that run in stages. In our example, we define job named "squish-tests1" which is run in a test stage.

squish-tests1:

    stage: test

    script:

    - echo DISPLAY=$DISPLAY

    - echo $SQUISH_DIR

    - echo "Starting squishserver..."

    - $SQUISH_DIR/bin/squishserver --port $SQUISH_SERVER_PORT 1>server.log 2>&1 &

    - sleep 5

    - echo "Register AUT..."

    - $SQUISH_DIR/bin/squishserver --port $SQUISH_SERVER_PORT --config addAUT addressbook /home/name/Squish/squish-for-qt-6.4.0/examples/qt/addressbook

    - $SQUISH_DIR/bin/squishrunner --port $SQUISH_SERVER_PORT --testsuite tests/suite_addressbook --reportgen junit,report.xml --reportgen stdout

    - echo "Stopping squishserver..."

    - $SQUISH_DIR/bin/squishserver --stop --port $SQUISH_SERVER_PORT &

    artifacts:

       reports:

          junit: report.xml

       paths:

         - server.log

The job performs the following actions:

1. Start squishserver and redirect its stdout and stderr output to a server.log file
2. Register the Application Under Test (AUT)
3. Call squishrunner to run the Test Suite and generate JUnit report and stdout report (so results are also printed to a screen)
4. Stop squishserver
5. Collect artifacts

Use Case

While working on a dev branch, we introduced a defect that is causing a failure in a test case. When committing a change, Squish Tests are executed.

Later in GitLab, we create a merge request to the master branch. On the Merge Request Page, a Test Summary shows a difference between test results on the master versus the merged branch.

In the picture above, a Test Summary shows that there is one new failed test. Based on this information, a decision about the merge can be made.