test automation – Navelpluis

The idea is to filter scenario execution based on the version of the software under test (SUT) using something like @version(v>= 1.0 && v<=2.1.1).

Unfortunately the current Gherkin parser & highlighting of IntelliJ’s IDEA doesn’t support the use of spaces between the parantheses, not even for string literals:

Not using spaces fixes the problem for now.

To apply the check, extend the implementation of the Before hook:

@Before
public void before(Scenario scenario) {
    scenario.getSourceTagNames().stream()
        .filter(tag -> tag.startsWith("@version"))
        .findFirst()
        .ifPresent(tag -> VersionComparison.checkVersion(tag, properties.getProperty("version")));

    // ...The rest of your code

Pick your own method for determining the version of the SUT. Comparing the version expression is done using EvalEx.

The VersionComparison class:

package util;

import com.ezylang.evalex.Expression;
import com.ezylang.evalex.data.EvaluationValue;
import org.junit.jupiter.api.Assumptions;

import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class VersionComparison {

    /*
     * Compares the version required for a scenario with the version of the NH installation.
     * A scenario can be tagged with '@version([EXPR])' where the expression CAN be:
     *
     * v [== | < | > | <= | >=] [MAJOR.MINOR.PATCH]
     *
     * For example:
     * 'v >= 1.6.0' or 'v < 1.6.0'
     *
     * Multiple checks can be combined:
     * 'v >=1.1.0 && v<1.4.0'
     *
     * The version numbers MUST consist of 3 elements.
     * For info on SemVer/Semantic Versioning see https://semver.org/spec/v2.0.0.html
     *
     * When the version doesn't match the scenario is skipped.
     */
    public static void checkVersion(String versionTag, String version) throws Exception {
        String expressionFromTag = prepareExpression(versionTag, version);
        Expression expression = new Expression(expressionFromTag);
        EvaluationValue result;

        try {
            result = expression.evaluate();
        } catch (Exception ex) {
            throw new Exception(
                String.format("Cannot evaluate expression '%s'", versionTag),
                ex
            );
        }

        // If the expression evaluates to 'False': the system under test does not have the correct version
        if(!result.getBooleanValue()) {
            Assumptions.assumeFalse(
                true,
                String.format(
                    "Incorrect version. Needed: '%s', actual: %s'",
                    versionTag, version
            ));
        }
    }

    private static String prepareExpression(String tag, String version) throws Exception {
        // Parse the tag to retrieve the expression
        // Matches everything between (..)
        Pattern tagPattern = Pattern.compile("\\((.*)\\)");
        Matcher tagMatcher = tagPattern.matcher(tag);

        if(!tagMatcher.find()){
            throw new Exception(String.format("Cannot find expression in tag '%s'", tag));
        }

        String expressionFromTag = tagMatcher.group(0);

        // Insert the version of the system under test.
        expressionFromTag = expressionFromTag.replaceAll("v",  version);

        // Replace _all_ SemVer versions.
        // Matches [MAJOR].[MINOR].[PATCH], for example `1.6.3` or `2.15.0`
        String regex = "\\b\\d+\\.\\d+\\.\\d+\\b";
        Pattern semVerPattern = Pattern.compile(regex);
        Matcher semVerMatch = semVerPattern.matcher(expressionFromTag);

        while (semVerMatch.find()) {
            String tmp = semVerMatch.group(0);
            // Conversion to a number is necessary, because the expression evaluator cannot
            // handle/does not support a SemVer number as data type.
            expressionFromTag = expressionFromTag.replaceAll(tmp, Long.toString(semVerToNumeric(tmp)));
        }

        return expressionFromTag;
    }

    /*
     * Converts a semVer string to a comparable number:
     * '1.6.1' becomes '1006001',
     * '1.12.2' becomes '1012002'.
     * Supports numbers up to 3 digits.
     */
    private static long semVerToNumeric(String semVer) {
        String[] parts = semVer.split("\\.");

        long major = Integer.parseInt(parts[0]);
        long minor = Integer.parseInt(parts[1]);
        long patch = Integer.parseInt(parts[2]);

        return (major * 1000000L) + (minor * 1000L) + patch;
    }
}

Important notes

All versions used, MUST be full SemVer numbers including all three parts: MAJOR.MINOR.PATCH, even if their value is zero.
The expression parsing MIGHT be too basic to handle complex expressions.

You cannot dynamically skip test scenarios using the oft suggested raising an AssumptionViolatedException or using Assume.assumeTrue(false). This still results in a failed build, exiting with code 1, which makes it unfit for use in a CI/CD pipeline.

What does work however, is using Assumptions.assumeTrue(false); Source. For example:

@Before
public void before(Scenario scenario) {
    Assumptions.assumeFalse(        scenario.getSourceTagNames().contains("@skipme")
    );}

Additional remarks:

Figuring this out took me way too much time. Maybe if I searched for ‘abort test’ instead of ‘conditionally skip scenario’, I would’ve figured it out sooner.
When implementing a Before("@sometag") and @Before function, both will be run for a scenario marked with ‘@sometag’.
Using the exception/Assume can work with a different runner: when running the scenario using JetBrain’s IntelliJ IDEA, the scenario will be nicely marked as ‘skipped’.

Of course, you can achieve the same by using tag filtering in your runner. The dynamic approach enables using your existing test set in a more flexible way. You can filter scenario execution based on product versions, web browser used and other (environment) parameters. All without having to keep or maintain multiple specific purpose run configurations.

Tag: test automation

Version Based Scenario Filtering

Dynamically Skipping Scenario