Test Runner API

TestStatus

class TestStatus
Source Code:

TestStatus

PASSED

Attribute to mark test or test suite as passed.

FAILED

Attribute to mark test or test suite as failed.

SKIPPED

Attribute to mark test or test suite as skipped.

ERRORED

Attribute to mark test or test suite as errored. It indicates that an unexpected error has happened while testing the test suite.

TestCase

class TestCase
Source Code:

TestCase

name

Name of a test case.

Example:

test_configuration, embedded/test_mqtt_connection, etc.

status

Instance of TestStatus class.

Example:

TestStatus.PASSED

message

Message that explains the reason why a test failed. Available only when TestCase.status == TestStatus.FAILED.

stdout

Extra output when running a test case.

duration

Test case duration in seconds.

exception

Unexpected happened when running a test case. Should be an instance of the Python Exception class.

source

Instance of TestCaseSource (see below) that contains details of the source file and a test case line.

TestCaseSource

class TestCaseSource
Source Code:

TestCaseSource

file

Path of the source file where a test case is implemented. The relative path to a project folder is allowed.

line

Number of the line for a test case.

CustomTestRunner

CustomTestRunner class must inherit TestRunnerBase or the already implemented runners for supported Testing Frameworks. See the source code of already implemented runners.

class CustomTestRunner
Source Code:

TestRunnerBase,

setup(self)

Method called to prepare the test runner. This is called immediately before calling self.stage_building(). The default implementation does nothing.

teardown(self)

Method called immediately after the self.stage_testing() method has been called and the result recorded. This is called even if the test method raised an exception, so the implementation in subclasses may need to be particularly careful about checking the internal state. The default implementation does nothing.

stage_building(self)

Method called to build source code of a particular test.

stage_uploading(self)

Method called to upload a testing firmware to the target embedded device. This method is ignored on the Native development platform.

stage_testing(self)

Method called to run test cases and record test results via calling self.test_suite.add_case(TestCase(...)).

You can implement your own testing method depending on the target where tests are running. For example, connecting to the “BoardFarm/Remote CI Runner” via HTTT/TCP, gathering results, parsing them, and recoding via self.test_suite.add_case(TestCase(...)).

configure_build_env(self)

Method called to configure a SCons build environment in the testing mode. It is uesfult to provide extra runtime configuration for a testing framework (macros, includes, dynamic configs).

Example:

See Unity Test Runner.

on_testing_data_output(self, data)

Method called on receiving data when running tests at the stage_testing. data can be the complete test result output, line, or part of a line. When data are chunked, on_testing_data_output will be called multiple times until testing is completed.

Parameters:

data (str) -- testing output data

on_testing_line_output(self, line)

Method called on each line split from data received by on_testing_data_output method.

Parameters:

line (str) -- testing output line