Search documentation of common_test

# Event Handling

Copy Markdown View Source

## General

The operator of a `Common Test` system can receive event notifications continuously during a test run. For example, `Common Test` reports when a test case starts and stops, the current count of successful, failed, and skipped cases, and so on. This information can be used for different purposes such as logging progress and results in another format than HTML, saving statistics to a database for report generation, and test system supervision.

`Common Test` has a framework for event handling based on the OTP event manager concept and `gen_event` behavior. When the `Common Test` server starts, it spawns an event manager. During test execution the manager gets a notification from the server when something of potential interest happens. Any event handler plugged into the event manager can match on events of interest, take action, or pass the information on. The event handlers are Erlang modules implemented by the `Common Test` user according to the `gen_event` behavior (for details, see module `gen_event` and section `gen_event Behaviour` in OTP Design Principles in the System Documentation).

A `Common Test` server always starts an event manager. The server also plugs in a default event handler, which only purpose is to relay notifications to a globally registered `Common Test` Master event manager (if a `Common Test` Master server is running in the system). The `Common Test` Master also spawns an event manager at startup. Event handlers plugged into this manager receives the events from all the test nodes, plus information from the `Common Test` Master server.

User-specific event handlers can be plugged into a `Common Test` event manager, either by telling `Common Test` to install them before the test run (described later), or by adding the handlers dynamically during the test run using `gen_event:add_handler/3` or `gen_event:add_sup_handler/3`. In the latter scenario, the reference of the `Common Test` event manager is required. To get it, call `ct:get_event_mgr_ref/0` or (on the `Common Test` Master node) `ct_master:get_event_mgr_ref/0`.

## Use

Event handlers can be installed by an `event_handler` start flag (`ct_run`) or option `ct:run_test/1`, where the argument specifies the names of one or more event handler modules.

Example:

`$ ct_run -suite test/my_SUITE -event_handler handlers/my_evh1 handlers/my_evh2 -pa $PWD/handlers`

To pass start arguments to the event handler init function, use option `ct_run -event_handler_init` instead of `-event_handler`.

#### Note

All event handler modules must have `gen_event` behavior. These modules must be precompiled and their locations must be added explicitly to the Erlang code server search path (as in the previous example).

An event_handler tuple in argument `Opts` has the following definition (see `ct:run_test/1`):

    {event_handler,EventHandlers}

    EventHandlers = EH | [EH]
    EH = atom() | {atom(),InitArgs} | {[atom()],InitArgs}
    InitArgs = [term()]

In the following example, two event handlers for the `my_SUITE` test are installed:

    1> ct:run_test([{suite,"test/my_SUITE"},{event_handler,[my_evh1,{my_evh2,[node()]}]}]).

Event handler `my_evh1` is started with `[]` as argument to the init function. Event handler `my_evh2` is started with the name of the current node in the init argument list.

Event handlers can also be plugged in using one of the following test specification terms:

- `{event_handler, EventHandlers}`
- `{event_handler, EventHandlers, InitArgs}`
- `{event_handler, NodeRefs, EventHandlers}`
- `{event_handler, NodeRefs, EventHandlers, InitArgs}`

`EventHandlers` is a list of module names. Before a test session starts, the init function of each plugged in event handler is called (with the `InitArgs` list as argument or `[]` if no start arguments are specified).

To plug in a handler to the `Common Test` Master event manager, specify `master` as the node in `NodeRefs`.

To be able to match on events, the event handler module must include the header file `ct_event.hrl`. An event is a record with the following definition:

`#event{name, node, data}`

- `name` - Label (type) of the event.

- `node` - Name of the node that the event originated from (only relevant for `Common Test` Master event handlers).

- `data` - Specific for the event.

### General Events

The general events are as follows:

- `#event{name = start_logging, data = LogDir}` - `LogDir = string()`, top-level log directory for the test run.

  This event indicates that the logging process of `Common Test` has started successfully and is ready to receive I/O messages.

- `#event{name = stop_logging, data = []}` - This event indicates that the logging process of `Common Test` was shut down at the end of the test run.

- `#event{name = test_start, data = {StartTime,LogDir}}` - `StartTime = {date(),time()}`, test run start date and time.

  `LogDir = string()`, top-level log directory for the test run.

  This event indicates that `Common Test` has finished initial preparations and begins executing test cases.

- `#event{name = test_done, data = EndTime}` - `EndTime = {date(),time()}`, date and time the test run finished.

  This event indicates that the last test case has been executed and `Common Test` is shutting down.

- `#event{name = start_info, data = {Tests,Suites,Cases}}` - `Tests = integer()`, number of tests.

  `Suites = integer()`, total number of suites.

  `Cases = integer() | unknown`, total number of test cases.

  This event gives initial test run information that can be interpreted as: "This test run will execute `Tests` separate tests, in total containing `Cases` number of test cases, in `Suites` number of suites". However, if a test case group with a repeat property exists in any test, the total number of test cases cannot be calculated (unknown).

- `#event{name = tc_start, data = {Suite,FuncOrGroup}}` - `Suite = atom()`, name of the test suite.

  `FuncOrGroup = Func | {Conf,GroupName,GroupProperties}`

  `Func = atom()`, name of test case or configuration function.

  `Conf = init_per_group | end_per_group`, group configuration function.

  `GroupName = atom()`, name of the group.

  `GroupProperties = list()`, list of execution properties for the group.

  This event informs about the start of a test case, or a group configuration function. The event is sent also for `init_per_suite` and `end_per_suite`, but not for `init_per_testcase` and `end_per_testcase`. If a group configuration function starts, the group name and execution properties are also specified.

- `#event{name = tc_logfile, data = {{Suite,Func},LogFileName}}` - `Suite = atom()`, name of the test suite.

  `Func = atom()`, name of test case or configuration function.

  `LogFileName = string()`, full name of the test case log file.

  This event is sent at the start of each test case (and configuration function except `init/end_per_testcase`) and carries information about the full name (that is, the file name including the absolute directory path) of the current test case log file.

- `#event{name = tc_done, data = {Suite,FuncOrGroup,Result}}` - `Suite = atom()`, name of the suite.

  `FuncOrGroup = Func | {Conf,GroupName,GroupProperties}`

  `Func = atom()`, name of test case or configuration function.

  `Conf = init_per_group | end_per_group`, group configuration function.

  `GroupName = unknown | atom()`, name of the group (unknown if init- or end function times out).

  `GroupProperties = list()`, list of execution properties for the group.

  `Result = ok | {auto_skipped,SkipReason} | {skipped,SkipReason} | {failed,FailReason}`, the result.

  `SkipReason = {require_failed,RequireInfo} | {require_failed_in_suite0,RequireInfo} | {failed,{Suite,init_per_testcase,FailInfo}} | UserTerm`, why the case was skipped.

  `FailReason = {error,FailInfo} | {error,{RunTimeError,StackTrace}} | {timetrap_timeout,integer()} | {failed,{Suite,end_per_testcase,FailInfo}}`, reason for failure.

  `RequireInfo = {not_available,atom() | tuple()}`, why require failed.

  `FailInfo = {timetrap_timeout,integer()} | {RunTimeError,StackTrace} | UserTerm`, error details.

  `RunTimeError = term()`, a runtime error, for example, `badmatch` or `undef`.

  `StackTrace = list()`, list of function calls preceding a runtime error.

  `UserTerm = term()`, any data specified by user, or `exit/1` information.

  This event informs about the end of a test case or a configuration function (see event `tc_start` for details on element `FuncOrGroup`). With this event comes the final result of the function in question. It is possible to determine on the top level of `Result` if the function was successful, skipped (by the user), or if it failed.

  It is also possible to dig deeper and, for example, perform pattern matching on the various reasons for skipped or failed. Notice that `{'EXIT',Reason}` tuples are translated into `{error,Reason}`. Notice also that if a `{failed,{Suite,end_per_testcase,FailInfo}` result is received, the test case was successful, but `end_per_testcase` for the case failed.

- `#event{name = tc_auto_skip, data = {Suite,TestName,Reason}}` - `Suite = atom()`, the name of the suite.

  `TestName = init_per_suite | end_per_suite | {init_per_group,GroupName} | {end_per_group,GroupName} | {FuncName,GroupName} | FuncName`

  `FuncName = atom()`, the name of the test case or configuration function.

  `GroupName = atom()`, the name of the test case group.

  `Reason = {failed,FailReason} | {require_failed_in_suite0,RequireInfo}`, reason for auto-skipping `Func`.

  `FailReason = {Suite,ConfigFunc,FailInfo}} | {Suite,FailedCaseInSequence}`, reason for failure.

  `RequireInfo = {not_available,atom() | tuple()}`, why require failed.

  `ConfigFunc = init_per_suite | init_per_group`

  `FailInfo = {timetrap_timeout,integer()} | {RunTimeError,StackTrace} | bad_return | UserTerm`, error details.

  `FailedCaseInSequence = atom()`, the name of a case that failed in a sequence.

  `RunTimeError = term()`, a runtime error, for example `badmatch` or `undef`.

  `StackTrace = list()`, list of function calls preceding a runtime error.

  `UserTerm = term()`, any data specified by user, or `exit/1` information.

  This event is sent for every test case or configuration function that `Common Test` has skipped automatically because of either a failed `init_per_suite` or `init_per_group`, a failed `require` in `suite/0`, or a failed test case in a sequence. Notice that this event is never received as a result of a test case getting skipped because of `init_per_testcase` failing, as that information is carried with event `tc_done`. If a failed test case belongs to a test case group, the second data element is a tuple `{FuncName,GroupName}`, otherwise only the function name.

- `#event{name = tc_user_skip, data = {Suite,TestName,Comment}}` - `Suite = atom()`, the name of the suite.

  `TestName = init_per_suite | end_per_suite | {init_per_group,GroupName} | {end_per_group,GroupName} | {FuncName,GroupName} | FuncName`

  `FuncName = atom()`, the name of the test case or configuration function.

  `GroupName = atom()`, the name of the test case group.

  `Comment = string()`, why the test case was skipped.

  This event specifies that a test case was skipped by the user. It is only received if the skip is declared in a test specification. Otherwise, user skip information is received as a `{skipped,SkipReason}` result in event `tc_done` for the test case. If a skipped test case belongs to a test case group, the second data element is a tuple `{FuncName,GroupName}`, otherwise only the function name.

- `#event{name = test_stats, data = {Ok,Failed,Skipped}}` - `Ok = integer()`, current number of successful test cases.

  `Failed = integer()`, current number of failed test cases.

  `Skipped = {UserSkipped,AutoSkipped}`

  `UserSkipped = integer()`, current number of user-skipped test cases.

  `AutoSkipped = integer()`, current number of auto-skipped test cases.

  This is a statistics event with current count of successful, skipped, and failed test cases so far. This event is sent after the end of each test case, immediately following event `tc_done`.

### Internal Events

The internal events are as follows:

- `#event{name = start_make, data = Dir}` - `Dir = string()`, running make in this directory.

  This internal event says that `Common Test` starts compiling modules in directory `Dir`.

- `#event{name = finished_make, data = Dir}` - `Dir = string()`, finished running make in this directory.

  This internal event says that `Common Test` is finished compiling modules in directory `Dir`.

- `#event{name = start_write_file, data = FullNameFile}` - `FullNameFile = string(), full name of the file.`

  This internal event is used by the `Common Test` Master process to synchronize particular file operations.

- `#event{name = finished_write_file, data = FullNameFile}` - `FullNameFile = string(), full name of the file.`

  This internal event is used by the `Common Test` Master process to synchronize particular file operations.

### Notes

The events are also documented in `ct_event.erl`. This module can serve as an example of what an event handler for the `Common Test` event manager can look like.

#### Note

To ensure that printouts to `stdout` (or printouts made with `ct:log/2,3` or `ct:pal,2,3`) get written to the test case log file, and not to the `Common Test` framework log, you can synchronize with the `Common Test` server by matching on events `tc_start` and `tc_done`. In the period between these events, all I/O is directed to the test case log file. These events are sent synchronously to avoid potential timing problems (for example, that the test case log file is closed just before an I/O message from an external process gets through). Knowing this, you need to be careful that your `handle_event/2` callback function does not stall the test execution, possibly causing unexpected behavior as a result.

← Previous Page Using Common Test for Large-Scale Testing
Next Page → Dependencies between Test Cases and Suites