From OpenTTCN DocZone

OpenTTCN DocZone   Home | Developer's corner | Knowledge base | Working documents | Documentation | Tutorials | Training | How do I | Frequently asked questions | Technical support

Article's permanent URL

How do I fine-tune performance of OpenTTCN-based test system?

---

Introduction

The information on this page is primarily targeted to test system vendors and integrators building OpenTTCN Tester for TTCN-3 -based test systems who want to measure or improve the performance of their test systems.

This information is useful generally for all TTCN-3 test system builders but it is crucial for vendors building test systems having stringent timing requirements.

The overall performance of a TTCN-3 test system depends on multiple factors such as:

• hardware (speed of processor, size of processor L2 cache, speed and size of memory),
• operating system (Windows, Linux, etc),
• OpenTTCN Tester for TTCN-3 version used,
• type of OpenTTCN libraries used (language, internal communication mechanisms),
• network performance (loop-back driver performance, network hardware performance if distributed test system setup is used),
• various configuration settings,
• implementation of TTCN-3 coding and decoding (CD),
• implementation of TTCN-3 SUT adapter (SA),
• implementation of TTCN-3 Platform adapter (PA),
• etc.

Due to multiple factors relating to overall performance, OpenTTCN gives advice on how to improve performance of a test system, if necessary. A test system vendor or an integrator needs to perform empirical measurements to determine that any specific performance requirements of the current application are met. After these measurements are performed, test system vendor or integrator shall give application dependent hardware requirements to their customers if the test system is distributed as a software package.

High performance is already there

OpenTTCN Tester for TTCN-3 has taken performance of TTCN-3 execution into account in the original tool design by applying the following two principles:

• use of compiled languages during test execution
• separation of test execution into preparation and test operations phases

The following subsections discuss this in more detail.

Use of compiled languages

OpenTTCN Tester for TTCN-3 uses ANSI C and ANSI C++ language based components during test execution. Java language is not used during test execution for any purpose. Use of Java language is restricted to parsing of TTCN-3 Core Language and ASN.1 language modules. Java is used only during parsing stage to facilitate producing accurate and complete error messages if parsing errors are encountered.

Phases of test execution

Test execution is divided into two phases that have different performance characteristics: non-realtime preparation phase, and more realtime test operations phase. This allows execution of test operations in a more predictable manner and at faster speed than would be possible otherwise.

In some applications that have stringent timing requirements, the above may not be enough, but the test system vendor or the integrator needs to take the following performance improvement advice into account.

Possible performance improvements

Select the right hardware

1. Select powerful enough CPU. Consider factors such as processor speed and the size of L2 cache, etc. In applications requiring extreme performance consider using current dual-core processors such as Intel's "PentiumD/955 3.46GHz LGA775 FSB1066 2x2MB" or AMD's "Athlon 64 X2 4800+ Dual Core processor, socket 939". Contact CPU vendors for additional information.
2. Install enough memory. Consider factors such as memory access time, latencies, and the size of the memory, etc. Install the memory modules in 128 bit mode configuration if that allows faster memory access in your configuration. You may want to consider using ECC memory. Contact memory vendors for additional information.
3. Select correct motherboard. Consider factors such as chipsets, speed of used bus, number of processors supported, etc. In applications requiring extreme performance consider using motherboard supporting two or four dual-core processors. In some applications you may want to consider using separate add-on cards having their own processors for performing specific CPU intensive tasks. Contact motherboard vendors for additional information.
4. Use fast network interface cards. This information applies only to geographically distributed test systems. Consider factors such as performance of the network interface card and the drivers, etc. In applications requiring extreme performance or low jitter consider using Gigabit Ethernet or 10 Gigabit Ethernet network interface cards. You may want to consider setting up a dedicated Ethernet network for intra test system traffic.

Suspend other activity

1. Do not run other applications while time critical test campaign is running.
2. Do not perform resource consuming OpenTTCN operations while time critical test campaign is running. For example, do not use "importer3 load" command while running a test campaign.
3. Do not run second OpenTTCN test campaign while time critical test campaign is running.

Use Windows Task Manager to find out whether processor and memory usage are within reasonable limits during execution of the whole test campaign. Contact OpenTTCN engineers for details.

Configure OpenTTCN Tester properly

1. Create the test system so that TTCN-3 adapter processes are separated from the TTCN-3 test management GUI so that priorities can be set correctly. The general rule of thumb is that the test management GUI shall be run at a lower priority than test components and TTCN-3 adapters.
2. Use correct operating system priority for test components. Explore empirically and find a correct setting for TC_PRIORITY property that controls the priority of all test components within one testing session. If the test suite is created using constructs that cause "a busy loop", the TTCN-3 executable may hang the whole PC. Priority setting is currenly available only in the Windows version of OpenTTCN Tester for TTCN-3.

Configure logging and log filtering correctly

1. Direct log directly to a log file from the test component with -r (log-redirect) option to achieve better performance from logging. (Use of this option may increase performance by 100% compared to the default logging.)
2. Completely turn off logging with -q (quiet) option to achieve minimum performance effect from logging. This logging option still produces log events in case of fail, inconc, and error verdicts. (Use of this option may increase performance by 100% compared to the default logging.)
3. Selectively turn off logging with otSetTMFlags(2048) OpenTTCN specific pre-defined function to achieve minimum performance effect from logging. Turn on logging with otResetTMFlags(2048) OpenTTCN specific pre-defined function to enable logging when needed.
4. Use the -m (mismatch event production) option sparingly. Producing a mismatch report requires the TTCN-3 executable to go through two complete data structures instead of simply declaring a mismatch.
5. Use the -i (indent) option sparingly. Producing indented log output requires NxM log related function calls instead of N calls required otherwise. Here N=number of logged messages, and M=fields present in every logged message.

Configure adapters properly

1. Use correct operating system priority for the TTCN-3 SUT adapter and platform adapter. Explore empirically and find a correct priority setting for the adapters.
2. Turn off verbose debug output from the SDK in the production version of your adapter. Make sure you do not have call to otSetSAFlags(OT_SA_FLAG_VERBOSE) function defined in tri/tri_ext.h in your SUT adapter. Having this flag turned on during development of the test system is beneficial.
3. Turn off decoded value verification when codec has been tested properly in the production version of your adapter. Turn this off by calling otResetCDFlags(OT_CD_FLAG_VALIDATE_DECODED_VALUE) function defined in tci/tci_ext.h in your codec. Having this flag turned on during development of the test system is beneficial.

Codec performance improvements

Avoid O(n) algorithms in the codec

1. Register tciEncode encoding functions separately for each top-level type that participates in encoding. This is applicable to message-based, procedure-based, and external function encoding.
2. Register tciDecode decoding functions separately for each top-level type that participates in decoding. This is applicable to message-based (optional), procedure-based, and external function decoding.
3. Use hash table or STL map type data structure providing O(log n) algorithm to find the correct decoding function after the user-provided code has determined type of the received information. This is applicable to message-based decoding when no decoding hypothesis is used.

Application performance fine tuning

Divide test case to preparation and execution phases

1. Place to the preparation phase (preamble): creation of test components, start of test component behavior, map and connect operations, etc. Try to place complex operations to the preparation phase, if possible.
2. Place to the test execution phase (main body): send operations, timer operations, handling of receive and timeout events, etc. Try not to place complex operations in the middle of send and receive sequence.

Order alt statements correctly

1. Place valid expected behavior as the first alternative within the alt statement.
2. Place the "receive any" and timeout event handling as the two last alternatives of the alt statement.

Profile TTCN-3 code

1. Select one TTCN-3 test case of interest.
2. Turn performance statistics capture on by setting the property STAT_CAPTURE=ON and run the selected test case to collect the performance statistics. OpenTTCN engineers provide you additional information how to perform this action.
3. Supply the selected TTCN-3 test case and the above collected performance statistics files to OpenTTCN Ltd for analysis.
4. OpenTTCN engineers can profile the performance of a TTCN-3 executable and improve the performance, if necessary.