How to systematically generate test data

When adding new test functions or improving coverage of existing tests in Mbed TLS, a large number of test cases may need to be added. This can be time-consuming and error-prone when adding manually. Mbed TLS includes a Python framework which can be used to systematically generate test case data from input values. The framework is flexible, to support a variety of test functions, and uses inheritance to reduce code repetition in similar test functions.

This document outlines the components of the framework, and how these interact, with a step-by-step example showing how these components may be implemented.

Components of the framework

The framework uses classes derived from BaseTarget to define how test cases are generated, and the destination file for generated tests. A command line interface is used to generate test data from the framework, and write the data to the output files.

Each subclass of BaseTarget represents a set of test cases. The set of test cases of a subclass is the union of its own direct test cases, plus the test cases of its subclasses, and so on recursively. Subclasses normally either have subclasses or direct test cases but not both. Each individual test case is generated from one instance of a subclass.

Each class derived from BaseTarget can be categorized as one of the following:

File target class: generally a direct subclass of BaseTarget, uniquely representing a .data file: it’s the biggest set which makes sense as a unit. Each file can be separately generated by passing the file name as a TARGET argument to the script. This is a direct child of the root in the class hierarchy tree.
Test function class: a class representing a test function, which will generate test cases for the associated function. This is a leaf in the class hierarchy tree. It’s the smallest set that makes sense as a unit.
Abstract class: a class which defines common attributes and methods, and does not generate test cases directly. This is an internal node in the class hierarchy tree, representing sets of related test functions within a file.

Test data generation module

The base classes required to generate test data using the framework are defined in scripts/mbedtls_dev/test_data_generation.py. This module defines the BaseTarget class, the CLI entry point main(), and the TestGenerator class.

The TestGenerator class constructs a TARGET dictionary, containing an entry for each defined file target class, and defines methods for generating and writing test case data. main() uses this class to list which files can be generated, and to generate files. Target files can be passed as arguments to the script to generate only specified files; by default all will be generated. The --help argument can be used for more details.

The same entry point is also used in tests/scripts/generate_psa_tests.py, which does not use this test framework, and instead hard-codes the TARGET dictionary.

BaseTarget

BaseTarget defines the common attributes and methods required to generate test cases, and implements the recursive test generation method, generate_tests(). When called on a class X, generate_tests() will be called on all classes derived from X, which will generate test cases in classes where test_function is set. This method yields all test cases from X and its descendants.

BaseTarget is defined in scripts/mbedtls_dev/test_data_generation.py.

File target classes

These are child classes of BaseTarget, each representing a generated .data file. target_basename must be uniquely set in each of these classes, typically to "test_suite_xyz.generated" for a corresponding test_suite_xyz.function file. These classes are abstract, and may also implement other attributes or methods commonly used by test functions in the associated file. The script will call generate_tests() on each of these classes to generate the test cases for each “target_basename.data” file.

BignumTarget is an example of a file target class, defined in tests/scripts/generate_bignum_tests.py.

Test function classes

These are classes which represent a test function, and require implementation of all abstract methods of BaseTarget. The implementation of generate_function_tests() in the class will define how test cases are generated for the associated function. Each instance of the class represents a test case, and must be initialized with required input data.

BignumCmp is an example of a test function class, defined in tests/scripts/generate_bignum_tests.py.

Abstract classes

Abstract classes do not represent a test function, and are used to define common attributes and methods for its subclasses. These can be used to create a uniform structure for similar tests to conform to, which reduces repetition of code and variation in implementations.

BignumOperation is an example of an abstract class, defined in tests/scripts/generate_bignum_tests.py. This class defines common methods used for binary bignum operations, and provides a structure for the derived classes to use.

Adding new tests

There are two options for adding new tests:

Adding classes to an existing script, such as tests/scripts/generate_bignum_tests.py.
Creating a new script. You’ll need to update the build system as explained in “Adding the script to the build system”.

Creating a test script

This is only required if an existing script is not being extended. An example is included for each step, showing how to create a test generation script for test_suite_mpi.function, a basic equivalent of generate_bignum_tests.py.

Initial Python script

To create the script, import test_data_generation from mbedtls_dev and call test_data_generation.main() when the script runs. The scripts directory must be in the system path; for a script in tests/scripts, add it by importing scripts_path (see the example below).

For the example, create generate_bignum_ex_tests.py in tests/scripts/, containing:

#!/usr/bin/env python3
import sys

import scripts_path # pylint: disable=unused-import
from mbedtls_dev import test_data_generation

if __name__ == '__main__':
    # Pass command line arguments and a description for the script
    test_data_generation.main(sys.argv[1:], "Generate bignum tests.")

This script can now be run, although it will do nothing at this stage. Use the --help argument for usage details.

$ tests/scripts/generate_bignum_ex_tests.py

Generating a data file

To generate a data file from the script, you must define a subclass of BaseTarget, setting target_basename to the output file basename. This is a file target class. Generally target_basename is "test_suite_xyz.generated" for tests defined in test_suite_xyz.function. In the example, metaclass=ABCMeta is added to the class to explicitly indicate this is an abstract class. As Pylint does not recognize this as an abstract class, the abstract-method warning is disabled.

For the example, add the following class:

from abc import ABCMeta
...
class BignumTarget(test_data_generation.BaseTarget, metaclass=ABCMeta):
    #pylint: disable=abstract-method
    """Target for bignum test case generation."""
    # Using .ex in this example to avoid clash with `generate_bignum_tests.py`
    target_basename = "test_suite_mpi.ex.generated"

Running the script will now create test_suite_mpi.ex.generated.data, and running with --list will list the filename as a valid target.

Adding the script to the build system

This is not required for the example, but is included here for completeness. When adding a new script to the build system, the changes required include:

Adding the script to /tests/scripts/check-generated-files.sh.
Adding the script to /scripts/make_generated_files.bat.
Adding calls and targets in CMake and Make for the script and its generated files. The script will need to be called in the same places where generate_bignum_tests.py is currently called.
Ensuring the generated files are covered by the Git ignore list.

Note: for Mbed TLS 2.28 the requirements differ, the changes required are:

Adding the script to /tests/scripts/check-generated-files.sh.
Committing the generated files.

Generating test cases for a function

To add test cases for a function, a concrete class derived from BaseTarget must be added to the script. This class must implement all abstract methods and some attributes of BaseTarget. The minimum set of required attributes is listed in “Creating the function class”, and the abstract methods which must be implemented are described in “Implementing abstract BaseTarget methods. This class will generate test cases. This is a test function class.

As an example, test case generation is added for the test function mpi_add_mpi(). The added class will be derived from BignumTarget, and can be added either to tests/scripts/generate_bignum_tests.py or the example script described in the previous section. This example is simplified, and will only handle valid integer inputs. For a more complete implementation, see the BignumAdd class defined in tests/scripts/generate_bignum_tests.py.

Creating the function class

The class must derive from a file target class, and the following class attributes should be set:

count = 0: this resets the test case counting mechanism. Alternatively, to disable test case numbering in descriptions, set show_test_count = False.
test_function: the test function that the class will generate cases for.
test_name: a short descriptive name for the test, this can be the same as test_function or a more readable equivalent. This is used as the start of the description line.

For the example, add the following class:

class BignumAddExample(BignumTarget):
    """Test cases for bignum addition."""
    count = 0
    test_function = "mpi_add_mpi"
    test_name = "MPI add"

Class initialization

Each instance of the class will represent a different test case, each with unique inputs. To initialize an instance of the class, these inputs must be passed to the __init__() constructor method.

For mpi_add_mpi(), two input values are required, val_a and val_b. In this example, integers will be used as inputs and stored in the class instance. As int is a multi-precision type in Python, these can be used for bignum calculations.

For the example, add the following constructor to BignumAddExample:

    def __init__(self, val_a: int, val_b: int) -> None:
        self.val_a = val_a
        self.val_b = val_b

Implementing abstract BaseTarget methods

Test function classes must provide an implementation of all the abstract methods declared in BaseTarget.

For the example, the following imports will be required:

import itertools
from typing import Iterator, List
# After import scripts_path
from mbedtls_dev import test_case

The method arguments() returns the list of arguments passed to the test function. Arguments must have the syntax expected in .data files, with double quotes around arguments that are strings or hex data. In the example, we also use an auxillary method result() to calculate the expected output.

For the example, add the following methods to BignumAddExample:

    def arguments(self) -> List[str]:
        # Format input values as quoted hexadecimal strings, without leading 0x
        return [
            "\"{:x}\"".format(self.val_a),
            "\"{:x}\"".format(self.val_b),
            self.result()
        ]

    def result(self) -> str:
        # Return as a quoted hexadecimal string, without leading 0x
        return "\"{:x}\"".format(self.val_a + self.val_b)

The method generate_function_tests() handles the construction of class instances for each unique test case, and yields test_case.TestCase objects from each instance. This is a classmethod, which acts on the class, rather than a specific instance. In the example, a list of strings will be used as inputs, and a test case is generated for each possible combination. The standard module itertools is used for generation of unique input combinations.

For the example, add the following method to BignumAddExample:

    @classmethod
    def generate_function_tests(cls) -> Iterator[test_case.TestCase]:
        # Create a list of input values
        input_values = [0, 1, 2, 123]
        # combinations_with_replacement generates a list of all unique
        # combinations, including where inputs are the same i.e. (0, 0)
        for a_value, b_value in itertools.combinations_with_replacement(
            input_values, 2
        )
            # Initialize a class instance with these values
            class_instance = cls(a_value, b_value)
            # Yield the TestCase object from this instance
            yield class_instance.create_test_case()

Other BaseTarget attributes and methods

The method description() returns a description of the test case, in the format “test_name #count case_description” by default. case_description is an optional description for the specific test case, which may be explicitly passed when constructing an instance, or generated in the class. This can be used to add context, and in the example this will be generated by overriding the description() method.

For the example, add the following method to BignumAddExample:

    def description(self) -> str:
        if not self.case_description:
            self.case_description = "{} + {}".format(
                # hex() converts to a hex string with leading 0x
                hex(self.val_a),
                hex(self.val_b)
            )
        # Call description() in the parent class
        return super().description()

The attribute dependencies is a list of build macros, which specifies the required build config for the test case to run. This can be set as a class attribute if always required for the test function, or set conditionally in the constructor if dependent on the test case.

For example, the test function mbedtls_ecp_curve_info() takes a curve ID argument, in the format MBEDTLS_ECP_XXXX. Each test case will depend on the specified curve being enabled, and hence depend on the macro MBEDTLS_ECP_XXXX_ENABLED. This can be set in the constructor using the the following code:

    def __init__(self, curve_id: str) -> None:
        self.dependencies = ["{}_ENABLED".format(curve_id)]

Complete example test script

The full test script from the example snippets:

import itertools
import sys

from typing import Iterator, List

import scripts_path # pylint: disable=unused-import
from mbedtls_dev import test_case
from mbedtls_dev import test_data_generation


class BignumTarget(test_data_generation.BaseTarget, metaclass=ABCMeta):
    #pylint: disable=abstract-method):
    """Target for bignum test case generation."""
    # Using .ex in this example to avoid clash with `generate_bignum_tests.py`
    target_basename = 'test_suite_mpi.ex.generated'


class BignumAddExample(BignumTarget):
    """Test cases for bignum addition."""
    count = 0
    test_function = "mpi_add_mpi"
    test_name = "MPI add"

    def __init__(self, val_a: int, val_b: int) -> None:
        self.val_a = val_a
        self.val_b = val_b

    def arguments(self) -> List[str]:
        # Format values as quoted hexadecimal strings, without leading 0x
        return [
            "\"{:x}\"".format(self.val_a),
            "\"{:x}\"".format(self.val_b),
            self.result()
        ]

    def description(self) -> str:
        if not self.case_description:
            self.case_description = "{} + {}".format(
                # hex() converts to a hex string with leading 0x
                hex(self.val_a),
                hex(self.val_b)
            )
        # Call description() in the parent class
        return super().description()

    def result(self) -> str:
        # Return as a quoted hexadecimal string, without leading 0x
        return "\"{:x}\"".format(self.val_a + self.val_b)

    @classmethod
    def generate_function_tests(cls) -> Iterator[test_case.TestCase]:
        # Create a list of input values
        input_values = [0, 1, 2, 123]
        # combinations_with_replacement generates a list of all unique
        # combinations, including where inputs are the same i.e. ("0", "0")
        for a_value, b_value in itertools.combinations_with_replacement(
            input_values, 2
        ):
                # Initialize a class instance with these values
                class_instance = cls(a_value, b_value)
                # Yield the TestCase object from this instance
                yield class_instance.create_test_case()


if __name__ == '__main__':
    # Pass command line arguments and a description for the script
    test_data_generation.main(sys.argv[1:], "Generate bignum tests.")

Running this script will create 10 test cases in test_suite_mpi.ex.generated.data. The tests can then be generated and ran together with other suites by running make test. To run only these tests:

$ tests/scripts/generate_bignum_ex_tests.py
$ make tests/test_suite_mpi.ex.generated
$ cd tests
$ ./test_suite_mpi.ex.generated

Extending existing test classes

When adding test case generation for functions, there may be previously implemented classes which generate tests for similar functions. For example, the test function mpi_add_abs() is very similar to mpi_add_mpi(), but returns an absolute result. Test structure, input format, and useful test cases may also be common across test functions. For example, a variety of test functions for binary operations use two inputs A and B, and one output, and can be implemented similarly.

Variants

For variants of a test function, inheritance can be used to reduce repetition of code. For example, to implement the test function mpi_add_abs(), BignumAddAbs can inherit from the BignumAdd class. The necessary changes for implementing BignumAddAbs are then reduced:

count should be reset to 0 for the class.
test_function should be set to mpi_add_mpi_abs.
test_name should be set to a different name.
We take the same list of inputs as the parent class, but calculate the expected result differently.

For example, possible implementation of this class:

class BignumAddAbsExample(BignumAddExample):
    """Tests for absolute variant of bignum addition."""
    count = 0
    test_function = "mpi_add_abs"
    test_name = "MPI add (abs)"

    def result(self) -> str:
        return "\"{:x}\"".format(abs(self.int_a) + abs(self.int_b))

These tests will then be generated with the same input values as mpi_add_mpi, and use the same methods for generating descriptions and the list of arguments.

Using abstract classes

Abstract classes are used to define common methods and attributes for multiple test functions. This can reduce repetition of code, and provide a uniform structure when adding test generation for similar test functions. Defined in tests/scripts/generate_bignum_tests.py, BignumOperation implements common attributes and methods for testing of binary bignum operations. This includes defining the class constructor, arguments(), description() and test case generation methods. By deriving from this class, tests can be added for binary bignum operations with smaller and simpler classes. These may only require the class attributes to be set, and the result() method to be defined.

BignumAdd and BignumCmp are two examples in generate_bignum_tests.py which derive from BignumOperation.