Concurrent Test Execution with Squish and Docker

Run your tests early, run your tests often. Easier said than done -- or is it? A Docker Swarm setup makes it easy to run all your tests in parallel, greatly reducing the time it takes to discover failures!

Running tests takes time. GUI tests in particular tend to require a lot of synchronization points, waiting for the application under test to catch up. This increases the runtime of the test, often resulting in test execution times in excess of five minutes -- for a single test case! With hundreds or thousands of test cases, it's clear that it's not viable to run the entire test suite every time: a single run of all test cases can easily take more than 12 hours!

And it's getting worse: user interfaces grow in complexity, exposing an ever increasing amount of functionality to the user. Thus, test suites grow in size, resulting in an ever increasing amount of testing necessary to measure the quality of the product. Sooner or later, it becomes clear that simply executing all of your test cases sequentially doesn't scale. If only there were a way to package the work into a neat package and then distribute the work over multiple workers…

Team Work: Shouldering The Work Jointly

Docker is a popular and free technology for packaging applications and all their dependencies into single files (called 'images') which can then be used to launch 'containers'. In a previous blog article, we showed how this can be used to package an application under test, Squish, as well as any relevant test scripts into a single image which then serves as the template for containers in which GUI tests are executed.

We can take this a step further: by using Docker Swarm. When using Docker in swarm mode, a group of Docker engines (typically running on different computers connected via a network) acts as a single unit. There are a number of aspects which make Docker Swarm attractive:

• It's part of the standard Docker installation -- no additional setup is required.
• It's straightforward to use.
• It automatically takes care of distributing tasks over the machines, making sure than none of them is overloaded.

Let's see how we can take advantage of that!

Getting Started

There are three important ingredients needed for a distributed GUI test automation setup using Docker Swarm:

1. You need some computers with Docker installed. Technically, a single machine will do but there won't be too much concurrency. It's a lot more fun if you have at least two machines, even if they are just virtual machines. Don't worry: you don't need to make this decision up front, growing a swarm by adding more machines is a matter of running a single command, as we will see later.
2. Your test suite should have no (or at least very few) dependencies between test cases. This means that the result of running a test should not depend on which tests have been run before. By removing any dependencies between test cases, you can run them in any order, skip arbitrary test cases -- or execute them one by one, on different computers. Have a look at our article on Finding & Fixing Dependencies Between Test Cases to learn more about the benefits of removing test case dependencies and how to do it.
3. A Docker image (or a set of images used in conjuction) is required which contains everything needed to run your tests. To get started with this, see our blog article on Squish GUI Testing With Docker Containers to learn more about how to create such a testing image based on the Squish GUI Tester tool.

Architecture

In general, the architecture of our Docker setup is straightforward:

• one computer acts as the 'manager' of the swarm, distributing the work
• other computers act as 'workers': doing the actual work. Typically, the manager computer is a worker, too -- i.e. it not only distributes work, it also takes care of it.

However, in order for the worker nodes of the swarm to do anything, they need to get hold of the testing image. This is made possible by using a shared Docker registry which acts as a hub for any Docker images to be used by the swarm. Before starting to dispatch test executions to the swarm, we will push the testing image to the registry such that all nodes use the exact same build of the application under test (and the exact same version of the test scripts!).

Finally, we need a place to collect test reports. This is a perfect use case for Squish Test Center, which acts as a central repository for all your test reports and provides a convenient web UI for reviewing and analyzing test reports. All worker nodes will push their test results straight to Squish Test Center.

A Private Docker Registry

Let's start by setting up a private Docker registry for holding (and sharing) the test image. Luckily, this is very easy: the logic required for hosting a registry is contained in a dedicated Docker image! We can just start a container based on that image.

Log into a computer which is reachable by all the machines you intend to make part of the swarm and run

$ docker run -d -p5000:5000 --restart always registry:latest

Unable to find image 'registry:latest' locally

latest: Pulling from library/registry

e95f33c60a64: Pull complete

4d7f2300f040: Pull complete

35a7b7da3905: Pull complete

d656466e1fe8: Pull complete

b6cb731e4f93: Pull complete

Digest: sha256:da946ca03fca0aade04a73aa94b54ff0dc614216bdd1d47585f97b4c1bdaa0e2

Status: Downloaded newer image for registry:latest

ddd5e8c1fb7815157fb9f9413f014b94e11bc1f9cf7dddb41a8267c2da64727f

This downloads the registry image and launches a container in the background (-d), making the registry service available on port 5000. In my case, I executed the command on a computer with the IP address 192.168.178.47, so the Docker registry is now available at 192.168.178.47:5000.

However, this registry is really just meant for local deployment -- it uses plain (unencrypted) HTTP communication. To make the Docker instances running on the worker machines actually use that registry, we will have to add it to a list of 'insecure registries'. To do so, log into each of the worker machines and make sure that the file /etc/docker/daemon.json contains the following:

{

  "insecure-registries" : ["192.168.178.47:5000"]

}

Of course, instead of 192.168.178.47, use the IP address of the computer on which you executed the above docker run command!

What's left to do is to put our testing image into our newly created registry. Our testing image, based on our first article about using Squish and Docker, is called addressbook-squish. Adding this image to the registry is a matter of performing two steps:

1. Tag the existing image, providing the host/port information by encoding it into the tag name:

   $ docker tag addressbook-squish 192.168.178.47:5000/addressbook-squish

2. Push the image to the repository:

   $ docker push 192.168.178.47:5000/addressbook-squish

Again, make sure to replace 192.168.178.47 with the IP address of the computer on which you launched the registry container!

Now that we have a central registry providing our testing image, let's create a swarm of Docker nodes to access the image.

Creating A Swarm

First, pick a machine which should act as the 'manager' of the swarm. The manager node needs to be reachable by all worker nodes, it's the machine which is responsible for dispatching work and inspecting the state of the swarm. For simplicity, we will pick the same computer for this task as the one which we chose for hosting the registry.

Putting a computer into swarm mode such that it acts as a manager is a matter of using the docker swarm command:

$ docker swarm init

Swarm initialized: current node (d0tcyemebrcgxjrq6xihz3zoe) is now a manager.
To add a worker to this swarm, run the following command:
    docker swarm join --token SWMTKN-1-04u5uytbvhf6j0vvosxgw00zk3x50mu44g1yhm60ttwmj3w4q2-btjn3sbqv4bxcv2jnps6qw1pt 192.168.178.47:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

Helpfully enough, the command also prints out what to do to make other machines join the swarm as workers! Let's follow the suggestion by connecting to two other machines (called swarmnode and swarmnode2) and running the above mentioned command:

$ ssh swarmnode docker swarm join --token SWMTKN-1-04u5uytbvhf6j0vvosxgw00zk3x50mu44g1yhm60ttwmj3w4q2-btjn3sbqv4bxcv2jnps6qw1pt 192.168.178.47:2377

This node joined a swarm as a worker.

$ ssh swarmnode2 docker swarm join --token SWMTKN-1-04u5uytbvhf6j0vvosxgw00zk3x50mu44g1yhm60ttwmj3w4q2-btjn3sbqv4bxcv2jnps6qw1pt 192.168.178.47:2377

This node joined a swarm as a worker.

And that's it! The three machines are now connected, acting as a single Docker swarm. We can inspect the state of the swarm using the docker node command. Note that this needs to be executed on the manager machine:

$ docker node ls

ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION

uvhrq0mnx4bgqn7tn6qgbkv1h     swarmnode           Ready               Active                                  20.10.3

n0kcmxqxllqx1h4t23sz85z7l     swarmnode2          Ready               Active                                  20.10.3

d0tcyemebrcgxjrq6xihz3zoe *   ubuntu              Ready               Active              Leader              19.03.6

We can see that there are three nodes in the swarm, all of them being ready for work. One of them (the one with the host name ubuntu) is the leader of the swarm.

Distributed Test Execution

Deploying tasks to a Docker swarm is done using the docker service command, which needs to be executed on the manager of the swarm. At the very least, it is given the name of an image (which should be accessible by all nodes in the swarm). The manager then takes care of picking an appropriate machine in the swarm (taking factors such as the system load into account) and launches a container on that machine. Unless otherwise specified, the manager will also make sure that the container is restarted automatically in case it terminates; this is useful for services which should always be available, e.g. databases or web servers.

Our use case is slightly different:

• We don't want the containers to restart. They should get started, run a test, and then terminate.
• The default command executed by our addressbook-squish simply causes a virtual display (Xvfb) to get started. It doesn't actually run any tests. See our previous blog article on the subject to learn more about this.

Luckily, these requirements are easily taken care of by passing additional arguments to docker service command. We will specify a different restart behavior (none) which ensures that the containers are stopped automatically. Also, we will specify a custom command to execute in the container. Due to our careful preparation of the addressbook-squish container, we know exactly where Squish is installed and where the test suites are stored. By taking advantage of this knowledge, which can craft a command line which executes just the test case we need:

$ docker service create \ --name=tst_adding \

  --restart-condition none \

  192.168.178.47:5000/addressbook-squish \

  /root/squish/bin/squishrunner --testsuite /tmp/suite_js --testcase tst_adding --local --reportgen testcenter,http://testcenter/project/addressbook?token=ABCDEF

There's a lot going on here, so let's break it down:

• docker service create is what's used to create a 'service' to be dispatched to the Docker swarm
• --name=tst_adding assigns the name tst_adding to the service. This will be useful later when inspecting the state of the swarm to see what all the nodes are working on.
• --restart-condition none changes the restarting behaviour of the service such that instead of restarting automatically, it just stops. This is exactly what we need: a 'fire and forget' container which runs for the duration of the test case and finally stops.
• 192.168.78.47:5000/addressbook-squish is the name of the image which should be used for creating a container. Note that this includes the address of our private Docker registry -- it's important that all worker nodes can access it.
• Finally, the command

  /root/squish/bin/squishrunner --testsuite /tmp/suite_js --testcase tst_adding --local --reportgen testcenter,http://testcenter/project/addressbook?token=ABCDEF

is executed in the newly-created container. It assumes that Squish is installed in /root/squish and that the test case to be executed is stored at /tmp/suite_js/tst_adding. It takes advantage of the command line interface of Squish, triggering a test run. The test report is sent to the Squish Test Center instance handling the URL http://testcenter/project/addressbook?token=ABCDEF.

Our suite_js test suite contains more than just one case though. When running the above command three times for different test cases (tst_general, tst_adding and tst_adding_data), three containers will be launched -- and, in our case, they get handled by different nodes in the swarm.

We can inspect the state of the nodes using the docker node ps command:

ID                  NAME                    IMAGE                                           NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS

gctzag2iqfzk         \_ tst_adding_data.1   192.168.178.47:5000/addressbook-squish:latest   swarmnode           Running             Running 11 seconds ago

06xn0zrx90ts        tst_general.1           192.168.178.47:5000/addressbook-squish:latest   ubuntu              Running             Running 18 seconds ago

yntt9w7x18zg         \_ tst_adding.1        192.168.178.47:5000/addressbook-squish:latest   swarmnode2          Shutdown            Complete 3 seconds ago                       

We can see that three test cases are executing concurrently -- in fact, one of them (tst_adding) finished just three seconds ago! The results can be viewed right in the Squish Test Center web interface.

Where To Go From Here

So there we have it -- after setting up a swarm, we can just throw additional test executions at it without concerning ourselves with the exact details of where the test is executed. Also, due to containerization, we can be sure that all test runs are reproducible and execute in isolation without affecting the rest of the system.

This is a great start, and there are a number of ways this could be extended:

• Of course, additional nodes could be added to the swarm. Simply run the docker swarm join command on a few more computers to increase the amount of computing power available to your test execution easily!
• Running all tests concurrently is great, but you can probably do better: maybe you don't need to run all your tests after a change to the applications source code! A tool like Squish Coco can be used to analyze source code differences and compute the test cases which are affected by a change. This allows reducing the number of tests to execute even further.
• Another, more sophisticated (albeit more complicated) system for orchestration of Docker containers is Kubernetes. Developed at Google, it's an all-in-one framework for distributed systems. It Introduces new terminology (pods, deployments, micro-services) and is arguably more difficult to get up and running than Docker Swarm. However, it supports a plethora of logging and monitoring services and is hugely popular among container orchestration tools.