OpenTTCN/How do I/Write correct test suites

From OpenTTCN

OpenTTCN DocZone   Home | Developer's corner | Knowledge base | Working documents | Documentation | OpenTTCN IDE | Tutorials | Training | How do I | Frequently asked questions | Technical support

How do I write solid, efficient, and portable test suites?

---

This page is primarily targeted to engineers writing test suites using TTCN-3 core language. It is useful generally for all engineers writing TTCN-3 test suites but it is especially useful for those who is still learning TTCN-3 and who wants to catch the "tricks of the trade".

Contents 1 Verdict recording 2 Use of timers 3 Ordering of timeout events 4 Test components 5 Ports 6 Connections and mappings 7 Take performance into account 8 Share your own design practices

Verdict recording

1. Recording pass verdicts in the preamble is strongly discouraged.
2. In the "receive any" branch of an alt statement, consider recording a fail verdict.
3. Record a fail verdict immediately when an error condition is recognized. Do this by placing the setverdict(fail); statement as the first statement in a statement block, especially before calling a postamble or any other function in that matter. The same applies to inconclusive (inconc) verdict.
4. Have a "log" statement explaining the reason of a fail or inconc verdict in plain text as the next line after the "setverdict" statement.
5. When programming a TTCN-3 adapter such as SUT adapter or platform adapter you shall call otReportError() function defined in <isl/TTCN3.h> immediately after you have noticed an error, or you suspect that an error has happened, or for any other reason when the adapter is not functioning properly. Calling this function causes a test case error.

Use of timers

1. Use of any timer.timeout is discouraged. Different standards and testing tools have different idea about what this construct does.
2. Limit maximum execution time that a "send-receive" statement sequence can take to avoid deadlocks caused by faulty test cases. Every time something is sent or called, start a supervising timer as well. This applies both to "send" and "call" statements.
3. Limit maximum execution time of a test case to avoid deadlocks caused by faulty test cases. For example, use execute(TC_MyTest, 10.0); statement in the TTCN-3 control part to limit the maximum time of TC_MyTest test case execution to 10 seconds.
4. Stop all running timers when you do not need them anymore. You do not need to stop expired timers.
5. Declare timers used in alternatives of defaults in the test component.
6. Start timer after the send operation if you want to measure timeout duration without taking the send operation duration into account. This applies to other operations as well.
7. Start timer before the send operation if you want to measure timeout duration and take the send operation duration into account. This applies to other operations as well.
8. Create a delay of 1 second by declaring a timer as timer TIMER_delay := 1.0; and by using a sequence of TIMER_delay.start; TIMER_delay.timeout; statements in your TTCN-3 code.

Ordering of timeout events

1. Place the timeout event as the last alternative of an alt statement if you are expecting only 0 to 1 responses to be handled in that alt statement without the help of the repeat statement.
2. Place the timeout event as the first alternative of an alt statement if you are expecting n messages per second to be handled in that alt statement with the help of the repeat statement.

Test components

1. Do not have unused ports in the test component that you are using. Define and use another test component type instead.
2. Always wait for termination of all test components that you have created and started dynamically using done operation. This is applicable if you are using parallel test components.
3. Use variables declared inside a test component mentioned in the runs on clause as a preferred way to communicate between functional entities whenever possible, rather than using formal and actual parameters of function calls. In this way you avoid overhead of passing parameters by value or by reference, what can improve performance of your test system.

Ports

1. Incoming messages stay in the input queue of a port forever (until the end of a test case) unless you process them. Use the -e option of the tester run command to visualize arrival of received messages that are appended to the end of the input queue of the port.
2. Always have the "receive any" alternative in your test case for each sequence of send and receive statements.
3. Use of inout all construct in port type definitions is discouraged in industrial-strength test suites. You may use this construct if you want to create a compact example or illustration of proof-of-concept. Note that this construct is deprecated in the recent editions of the TTCN-3 standard.

Connections and mappings

1. Do not map a connected port or vice versa.
2. Unmap all mapped ports when you do not need them anymore.
3. Disconnect all connected ports when you do not need them anymore.
4. Perform map and connect operations before starting a function as a behaviour of a parallel test component. This makes your dynamic test configuration more deterministic.

Take performance into account

1. Split a test case into preparation and execution phases:
   • Place the following to the preparation phase (preamble): creation of test components, start of test component behavior, map and connect operations, etc. Try to place complex operations to the preparation phase, if possible.
   • Place the following to the test execution phase (main body): send operations, timer operations, handling of receive and timeout events, etc. Try not to place complex operations in the middle of send and receive sequence.
2. Order alternatives of alt statements correctly:
   • Place valid expected behavior as the first alternative within the alt statement.
   • Place the "receive any" construct and timeout event handling as the last alternatives of an alt statement.

Share your own design practices

If you want to share your own TTCN-3 test suite design practices, please write us an e-mail to support@openttcn.fi. We will publish selected ones on our web pages and in the documentation. Please include a permission to publish your contribution by adding the following text to your e-mail:

I grant OpenTTCN Oy permission to use, modify, publish, and distribute this description of TTCN-3 design practice free of charge in publications created by OpenTTCN Oy.

If you want, we can acknowledge your submission in the publication (web page, documentation) with your name, company name, and URL of the website. We reserve the right to modify submissions to correct errors and provide additional information.