How can DevOps help with testing

Tech blog

Azure DevOps offers a lot that is needed in a project. There is the possibility to manage documentation, agile boards such as Scrumboards, the Git repository and also the pipelines in Azure DevOps. As a test automation specialist, the test plan module is of course of interest to me.

In this article I want to describe

  • how test automation can be integrated into Azure DevOps,
  • how this can be started automatically,
  • how the test results can be presented clearly and
  • what the limits of Azure DevOps are. Because as much as it allows, everything is not offered.

1. Preparation

The test plan module offers the possibility of creating different configurations and configuration variables. Configuration variables are, for example, the operating system, the test environment or the browser used. These variables are combined into configurations. A configuration "Integration Environment - Windows 10 - Chrome" and another "Integration Environment - Windows 10 - Firefox" could be created.

2. Create the test cases with the test plan module

The test cases are then created in the test plan module. In order to structure these, they are stored in different test suites. There are three different types of test suites:

  • static: act like folders on the computer.
  • Requirement-based: Test cases that are stored in this folder are linked directly to the selected requirement.
  • Query-based: all test cases that correspond to a created filter are displayed here.

Test suites can contain test cases as well as additional test suites, so that any number of test suite levels can be created.

The test cases consist of several steps with the respective expected results. Single or multiple steps that are used over and over can be saved as split steps. An example are the steps for login. This means that only the “Login” step is inserted. If this changes, the step only needs to be adjusted once for all test cases.

The test suites are then linked to the configurations created previously. This is done in the test plan in the settings of the individual test suites.

3. Create the automated tests

The automated test cases are then created and checked into the Git repository. In this case, the C # language is ideal, as the automated test cases can then be linked to the test cases of the test plan - more on this in a moment. Which test framework is used is up to each and every one of them. In the case of C #, these would be MSTest, NUint and xUnit.

4. Linking the automated test cases

Azure DevOps offers the possibility to link the automated test cases with the test cases in the test plan. This creates a 1-to-1 relationship. This link has the advantage that the results of the test automation are displayed directly in Azure DevOps. If a test fails, the result is displayed in the test plan directly on the test case; these can then be easily reproduced manually by describing the test steps.

If a test case is automated in the test plan, it is set to the "Automated" status and the linked class and function are displayed.

It should be noted, however, that the link can only be established if C # is used as the language and Visual Studio for Windows as the IDEE. Alternatively, this can be done programmatically (see for example here)

5. Integration and execution of test automation in Azure DevOps

Once the test cases have been created and linked, they are pushed into the Azure DevOps Git module. In order to execute the test cases, they are triggered in an Azure pipeline. For this it is recommended to create the automation as a template (more here). In this way, the automation can be integrated in different pipelines. It can thus become part of the release pipeline and also be set up as a separate pipeline.

The required test suites and the respective configuration are specified so that the results are stored correctly. In this way, for example, it can later be understood that a certain test case did not work on environment A, but did work on environment B.

Variables such as usernames, passwords and endpoints are stored in the library in Azure DevOps. This means that the passwords are protected and all variables can be used in the various pipelines.

6. Analysis of the results

Once the test cases have been carried out, the results are displayed in different places.

On the one hand, the results can be seen in the pipeline under "Test". The results of the individual tests are listed there. If a test fails, the error message is also displayed. Logs are also attached as files.

The test plan offers most of the options for analyzing the test execution. The test results for the individual test cases are displayed for each linked configuration. This makes it possible to detect whether a test failed, for example in the Chrome browser, but worked in Firefox. A double click on the result shows the history of the test runs per test case and configuration.

The progress report is also in the test plan. This shows the history of positive and negative test executions as well as the number of tests that have not yet been executed. This can be shown filtered for the different test suites.

7. Lessons learned

Azure DevOps offers a lot. Both for test automators, who can easily integrate their automation in this way, and for test managers who want to quickly get an overview of the current status of automation. This can of course also be helpful for the team. However, there are also limitations.

As already described, the linking of automated test cases and the test plan only works with C # and Visual Studio for Windows. So far, I have not found that a link with other languages ​​or frameworks such as Cucumber is possible.

The reporting relates to the entire test case. You can only see whether test cases have failed, but not at which step. However, this can be particularly advantageous if a single step is carried out in many test cases and thus causes a whole series of tests to fail. Recognizing this is a little more complex.

As described, the progress report can be filtered according to the suites. However, this filter does not contain any suites that are on the first level, but only suites from the second level onwards. A "Login" suite is not displayed in the filter, but a "Login / correctLogin" suite is.

Logs for the individual tests are only appended if a test case fails. However, sometimes it can still be helpful to see the test logs, for example to find out whether the test is actually running correctly. To do this, however, a test must explicitly fail.

Easily define tests and quickly get an overview

Azure DevOps is a powerful tool. It offers the various roles of a project team (Dev, PO, Test, ...) numerous tools for their daily work. The test plan can be a helpful module, especially when it comes to the visibility and analysis of test automation. It offers - with restrictions - an easy and clear way of defining tests and making them visible in the project. Test managers and the entire team receive quick feedback on the current status of the automated tests. Since no programming knowledge is required for the module itself, other roles such as POs could also define the tests directly when creating the stories, which are then automated by test automakers.

Tweet Share Share on LinkedIn