OpenTTCN/Developer corner/Running test cases and obtaining results
From OpenTTCN
Home | Developer's corner | Knowledge base | Working documents | Documentation | OpenTTCN IDE | Tutorials | Training | How do I | Frequently asked questions | Technical support |
Running test cases and obtaining results
In this article we will show how to automate execution of test campaigns with OpenTTCN Tester for TTCN-3 4.0beta2.2 and display campaign results through TTCN-3 control part using command-line tools.
We apply some simple heuristic rules to bring execution order and sequence of selected test cases, as well as conditional decisions made, closer to reasoning of a real human operator that runs test campaigns. This helps running campaigns offline and saves time of a human operator.
For this we develop a library of useful execution functions that can be called directly from the control part. We use OpenTTCN-specific test suite dynamic introspection capabilities to discover the list of test cases available for execution and then we run the test cases selectively.
Contents |
Library source code
In order to use examples in this article you need to create ControlPartLib.ttcn file somewhere in your system that you compile and import into the module containing control part of your selection.
You can obtain the source code of ControlPartLib.ttcn by following this link.
Library API
The two main functions in ControlPartLib are heuristicExecute() and printCampaignReport().
The heuristicExecute() function discovers the list of all test cases defined in the session using OpenTTCN-specific testcasenamelist() function and executes a subset of test cases from the list using OpenTTCN-specific extension of execute() command that accepts test case name as a variable or expression of charstring type rather than hard-coded test case instance call.
The printCampaignReport() function uses OpenTTCN-specific otGetUnixTime() to report a campaign report generation timestamp.
The rest of the library code is otherwise standard-compliant TTCN-3 code.
The heuristicExecute() function returns test campaign report in TestCampaignReport data structure that can be later used by printCampaignReport() to display test campaign results.
Here are signatures of heuristicExecute() and printCampaignReport():
/**
* Executes campaign of all test cases defined in the session that have names
* starting with tcNamePrefix using basic heuristic rules. Empty tcNamePrefix
* is the default and it means all test cases.
*
* Each test case is run repeatedly until pass verdict is produced, but at most
* maxRuns times and at least minRuns times in a row.
*
* Each test case duration is limited by maxCaseExecTime seconds.
*/
function heuristicExecute(
charstring tcNamePrefix := "",
float pauseBetweenCases := 0.0,
integer maxRuns := 5,
integer minRuns := 1,
float maxCaseExecTime := 20.0)
return TestCampaignReport { ... }
/**
* Prepares pretty-printed textual test campaign report and returns it as
* charstring
*/
function printCampaignReport(
TestCampaignReport tcr,
boolean printDetails := false,
integer tcNameColWidth := 50)
return charstring { ... }
Preparing control part
Now with all preparations made writing a control part is really easy. Here is an example:
module ctrl
{
import from ControlPartLib all;
control
{
var TestCampaignReport tcr;
// Select all L2 test cases exercising valid behaviour (BV),
// testing mobile station (MS), pause between test cases is
// 1 second, maximum number of attempts for each test case
// is 5, minimum number is 2.
tcr := heuristicExecute("TC_L2_MS_BV", 1.0, 5, 2);
var charstring tcrAsText := printCampaignReport(tcr,
true /* show also details part of test campaign report */);
log(tcrAsText);
}
}
Running control part
Now you can run the control part from the command line like this:
tester run L2 @control > campaign.log
Here is an example of output of log(tcrAsText) statement from the code snippet above that has some real-world background:
***********************************************************************************************************
*** TEST CAMPAIGN REPORT SUMMARY - Timestamp: 1264069347
--------------------------------------------------+----------+---------+------------------------+----------
Testcase | Wr.Cs.V* | Bt.Cs.V | Totals | Avg.Dur.
--------------------------------------------------+----------+---------+------------------------+----------
L2_MS.TC_L2_MS_BV_01 | error | pass | 1p 0i 0f 1e 0n | 2.4874
L2_MS.TC_L2_MS_BV_02 | fail | fail | 0p 0i 5f 0e 0n | 0.1538
L2_MS.TC_L2_MS_BV_03 | pass | pass | 2p 0i 0f 0e 0n | 0.1960
L2_MS.TC_L2_MS_BV_04 | pass | pass | 2p 0i 0f 0e 0n | 0.1990
L2_MS.TC_L2_MS_BV_05 | error | error | 0p 0i 0f 5e 0n | 20.0446
L2_MS.TC_L2_MS_BV_06 | error | error | 0p 0i 0f 5e 0n | 20.0506
L2_MS.TC_L2_MS_BV_07 | error | error | 0p 0i 0f 5e 0n | 4.3335
L2_MS.TC_L2_MS_BV_08 | error | fail | 0p 0i 2f 3e 0n | 18.2543
L2_MS.TC_L2_MS_BV_09 | error | error | 0p 0i 0f 5e 0n | 20.0480
--------------------------------------------------+----------+---------+------------------------+----------
* Wr.Cs.V = Worst case verdict; Bt.Cs.V = Best case verdict; Avg.Dur. = Average duration in secs
-----------------------------------------------------------------------------------------------------------
***********************************************************************************************************
*** TEST CAMPAIGN REPORT DETAILS - Timestamp: 1264069347
--------------------------------------------------+-------+---------+----------
Testcase | Run | Verdict | Duration
--------------------------------------------------+-------+---------+----------
L2_MS.TC_L2_MS_BV_01 | 00001 | error | 0.8390
| 00002 | pass | 4.1360
--------------------------------------------------+-------+---------+----------
L2_MS.TC_L2_MS_BV_02 | 00001 | fail | 0.2050
| 00002 | fail | 0.1430
| 00003 | fail | 0.1390
| 00004 | fail | 0.1370
| 00005 | fail | 0.1450
--------------------------------------------------+-------+---------+----------
L2_MS.TC_L2_MS_BV_03 | 00001 | pass | 0.1550
| 00002 | pass | 0.2370
--------------------------------------------------+-------+---------+----------
L2_MS.TC_L2_MS_BV_04 | 00001 | pass | 0.1980
| 00002 | pass | 0.2000
--------------------------------------------------+-------+---------+----------
L2_MS.TC_L2_MS_BV_05 | 00001 | error | 20.0470
| 00002 | error | 20.0450
| 00003 | error | 20.0429
| 00004 | error | 20.0450
| 00005 | error | 20.0429
--------------------------------------------------+-------+---------+----------
L2_MS.TC_L2_MS_BV_06 | 00001 | error | 20.0479
| 00002 | error | 20.0440
| 00003 | error | 20.0659
| 00004 | error | 20.0519
| 00005 | error | 20.0429
--------------------------------------------------+-------+---------+----------
L2_MS.TC_L2_MS_BV_07 | 00001 | error | 5.2709
| 00002 | error | 4.1009
| 00003 | error | 4.0979
| 00004 | error | 4.1009
| 00005 | error | 4.0970
--------------------------------------------------+-------+---------+----------
L2_MS.TC_L2_MS_BV_08 | 00001 | fail | 15.0830
| 00002 | error | 20.0440
| 00003 | fail | 16.0420
| 00004 | error | 20.0599
| 00005 | error | 20.0429
--------------------------------------------------+-------+---------+----------
L2_MS.TC_L2_MS_BV_09 | 00001 | error | 20.0530
| 00002 | error | 20.0440
| 00003 | error | 20.0459
| 00004 | error | 20.0539
| 00005 | error | 20.0429
--------------------------------------------------+-------+---------+----------
