Putting Test Case Specification directly in code

Recently we started to put test case specification of our integration and user acceptance tests directly in code. So before we write any tests, test cases are defined first and put in the code in the form of annotations. This includes specification for tests that will be automated as well as for tests that won’t. Also, we keep the description there even after the tests are written. This has many advantages, for one it provides more human-readable description for all tests. Even if your tests are written in English-friendly manner, there is usually more complicated data/mock setup that is not explained well in the code itself or it is scattered all over the codebase.

For developers and quality engineers, it is usually more convenient to work with the specification directly in code, without going back and forth to an external document. Other very positive feature is that we can easily generate reports. We can see how many test cases every feature has (we can easily tag test cases by features or other properties) and how many of them are already automated/implemented.

We keep high level descriptions of test cases in a test plan outside of the codebase and if there is a need for someone to go through detailed test cases without working with the repository, a report can be generated in any other format (and kept up to date by build server).

The only drawback might be that you need to keep the annotations and tests in sync, but you would have to do that anyway. Note that this might not be the best idea for teams where test cases are written by people who don’t know programming.

Tools

For JVM based projects, we use Arquillian Governor. You can read more about it in Arquillian Governor 1.0.0.Alpha1 Released. Basically it enables us to use test case specific Java annotations on test methods and generate reports from them.

@Test
@TestSpec(
    author = "John Smith",
    feature = "User accounts",
    test = "User can view other users",
    issue = "JIRA-565",
    prerequisites = "User Peter, User Marek",
    steps = "Log in as Peter, Navigate to the View users page",
    assertion = "Marek is listed on the screen",
    status = Status.AUTOMATED
)
public void userCanViewUsers() {
    // test implementation
}

We also write test case annotations in JavaScript, but we are not using any specific tool. At the moment, we just write custom annotations like this:

/**
 * @feature User accounts
 * @prerequisite User Peter
 * @prerequisite User Marek
 * @step Log in as Peter
 * @step Navigate to the View users page
 * @assert Marek is listed on the screen
 * @state manual
 */
"User can view other users": function () {
  console.log("* This test is not automated yet *");
}

We are going to implement some reporting on it in the future. There are already many comment parsers for JavaScript available so it shouldn’t take a lot of work.

Going further

Some people go even further and write test cases in business-readable DSLs that resemble human languages. One example of a testing framework that allows you to do that is Cucumber. Cucumber has many implementations for various languages, so it has the simple advantage of reusing knowledge of a particular DSL for writing any kind of test.

A test case executed by Cucumber can look like this:

Feature: Example feature
  As a user of cucumber.js
  I want to have documentation on cucumber
  So that I can concentrate on building awesome applications

  Scenario: Reading documentation
    Given I am on the Cucumber.js GitHub repository
    When I go to the README file
    Then I should see "Usage" as the page title

In my opinion, implementing rules for domain-specific language like this brings more complexity to writing tests and might not be worth the effort if all members of the team are engineers. It can be however very helpful when people writing test cases are not skilled in programming.

[Written when I was working as a Quality Engineer at Red Hat]

Last updated on 26.5.2015.