Test Runner API

TestStatus 

class TestStatus

Source Code:: TestStatus

PASSED: Attribute to mark test or test suite as passed.

FAILED: Attribute to mark test or test suite as failed.

SKIPPED: Attribute to mark test or test suite as skipped.

ERRORED: Attribute to mark test or test suite as errored. It indicates that an unexpected error has happened while testing the test suite.

TestCase 

class TestCase

Source Code:: TestCase

name

Name of a test case.

Example:: test_configuration, embedded/test_mqtt_connection, etc.

status

Instance of TestStatus class.

Example:: TestStatus.PASSED

message: Message that explains the reason why a test failed. Available only when TestCase.status == TestStatus.FAILED.

stdout: Extra output when running a test case.

duration: Test case duration in seconds.

exception: Unexpected happened when running a test case. Should be an instance of the Python Exception class.

source: Instance of TestCaseSource (see below) that contains details of the source file and a test case line.

TestCaseSource 

class TestCaseSource

Source Code:: TestCaseSource

file: Path of the source file where a test case is implemented. The relative path to a project folder is allowed.

line: Number of the line for a test case.

CustomTestRunner 

CustomTestRunner class must inherit TestRunnerBase or the already implemented runners for supported Testing Frameworks. See the source code of already implemented runners.

class CustomTestRunner

Source Code:

TestRunnerBase,

setup(self): Method called to prepare the test runner. This is called immediately before calling self.stage_building(). The default implementation does nothing.

teardown(self): Method called immediately after the self.stage_testing() method has been called and the result recorded. This is called even if the test method raised an exception, so the implementation in subclasses may need to be particularly careful about checking the internal state. The default implementation does nothing.

stage_building(self): Method called to build source code of a particular test.

stage_uploading(self): Method called to upload a testing firmware to the target embedded device. This method is ignored on the Native development platform.

stage_testing(self)

Method called to run test cases and record test results via calling self.test_suite.add_case(TestCase(...)).

You can implement your own testing method depending on the target where tests are running. For example, connecting to the “BoardFarm/Remote CI Runner” via HTTT/TCP, gathering results, parsing them, and recoding via self.test_suite.add_case(TestCase(...)).

configure_build_env(self)

Method called to configure a SCons build environment in the testing mode. It is uesfult to provide extra runtime configuration for a testing framework (macros, includes, dynamic configs).

Example:: See Unity Test Runner.

on_testing_data_output(self, data)

Method called on receiving data when running tests at the stage_testing. data can be the complete test result output, line, or part of a line. When data are chunked, on_testing_data_output will be called multiple times until testing is completed.

Parameters:: data (str) -- testing output data

on_testing_line_output(self, line)

Method called on each line split from data received by on_testing_data_output method.

Parameters:: line (str) -- testing output line