portugalfuturista/universalisos
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
universalisos/docs-extracted/development/test-framework-reference-manual.md
Fábio Coutada ae6144a1c5 feat(docs): extract all extractable PikeOS PDF manuals to markdown 
- Extract 37 of 45 PDFs under docs/ to docs-extracted/
- Preserve directory structure (apex, cdk, development, platform, etc.)
- Add docs-extracted/index.md with navigation table
- 8 PDFs were 0-byte/empty and could not be extracted
2026-07-06 23:07:19 +01:00

5309 lines
248 KiB
Markdown
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
title: "Test Framework Reference Manual"
source: "docs/development/test-framework-reference-manual.pdf"
category: "development"
pages: 126
extracted: "2026-07-06T23:05:35.500693"
---
# Test Framework Reference Manual
> Extracted from `docs/development/test-framework-reference-manual.pdf` (126 pages).
> Figures, diagrams, and tables may not render accurately in plain text.
PikeOS Test Framework
Reference Manual
Am Pfaffenstein 14, D-55270 Klein-Winternheim
Notice: The contents of this document are proprietary to
SYSGO AG and shall not be disclosed, disseminated,
copied, or used except for purposes expressly
authorized in writing by SYSGO AG.
PikeOS Test Framework Reference Manual
PikeOS D5.0, Document Version D5.0-3576
c 2005 – 2019 SYSGO GmbH
SYSGO GmbH Email: office@sysgo.com
Am Pfaffenstein 14
55270 Klein-Winternheim, Germany http://www.sysgo.com
All rights reserved.
PikeOS is a trademark of SYSGO GmbH. The designations used to identify other software or hardware products
in this publication may be trademarks of their manufacturers or sellers.
Contents
1 Basic Test Suite Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.1 Preparing a Test Suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2 Build-In Help Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3 Listing Test Cases and Test Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4 Building and Running Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4.1 Basic Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4.2 Partial Test Set Builds and Runs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.4.3 Manual Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.4.4 Specifying Labels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.4.5 Temporarily Changing PikeOS Project Configuration . . . . . . . . . . . . . . . . . . . . 11
1.5 Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.6 Cleaning Up the Test Suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.6.1 Removing Intermediate Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.6.2 Resetting to Initial State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.6.3 Reverting Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2 Overlays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1 Creating Overlays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1.1 The Directory Structure of an Overlay . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1.2 The Content of the Test Suite Specific Overlay . . . . . . . . . . . . . . . . . . . . . . . 14
2.1.3 The Content of the Test Suite Common Overlay . . . . . . . . . . . . . . . . . . . . . . . 14
2.1.4 The Content of the Generic Overlay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2 Using Overlays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3 Test Set Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.1 Creating Test Set Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.1.1 Naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.1.2 Directory Layout of a Test Set Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.1.3 The Test Case XML Description File V2.1 . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.1.3.1 The <objectives/> and <obj/> Elements . . . . . . . . . . . . . . . . . . . . . . 17
3.1.3.2 The <tags/> Element . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.1.3.3 The <description/> Element . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.1.3.4 The <equitable/> Element . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2 Using Test Set Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.3 Support for Multiple Test Set Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.4 Automatic Generation of Test Set Pool Variables from Trace-XML-files . . . . . . . . . . . . . . . 20
4 Verification Result Artifacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.1 Artifact Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.1.1 Building the Test Case Master Document . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.1.2 Building the Test Result Document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.1.3 Building the Structural Coverage Result Document . . . . . . . . . . . . . . . . . . . . . 22
4.2 Artifact Setup in XML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2.1 XML Document Fragments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
4 CONTENTS
4.2.2 Setup of the Test Case Master Document . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.2.2.1 The File doc/tc.xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.2.2.2 Configuring Label Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.2.2.3 Using description.xml Files . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.2.2.4 Additional Document Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.2.3 Setup of the Test Result Document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.2.3.1 The File doc/tr.xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.2.3.2 Additional Document Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.2.4 Setup of the Structural Coverage Result Document . . . . . . . . . . . . . . . . . . . . . 30
4.2.4.1 The File doc/sc.xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.2.4.2 Additional Document Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.3 Artifact Setup in TEX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.3.1 Managing Abbreviations and References with BibTeX . . . . . . . . . . . . . . . . . . . . 32
4.3.1.1 Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.3.1.2 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.4 Available Makefile Hooks for Document Creation . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.5 Changebars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5 Test Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.1 Application and Integration Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.1.1 Requirements on the Application Project . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.1.2 Requirements on the Integration Project . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.2 Special PikeOS Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
6 Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6.1 The Files in the Test Case Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6.1.1 The Test Case XML Description File . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6.1.1.1 The <rqdocs/> and <rqdoc/> Elements . . . . . . . . . . . . . . . . . . . . . . 40
6.1.1.2 The <rqlist/> and <rq/> Elements . . . . . . . . . . . . . . . . . . . . . . . . . 42
6.1.1.3 The <objectives/> and <obj/> Elements . . . . . . . . . . . . . . . . . . . . . . 42
6.1.1.4 The <testenvlist/> and <testenv/> Elements . . . . . . . . . . . . . . . . . . . . 43
6.1.1.5 The <tags/> Element . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6.1.1.6 The <description/> Element . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6.1.2 The Test Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
6.1.3 The symbol-global.*.txt Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.1.4 The t*_cflags.rc Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.1.5 The Makefile File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.2 General Rules for Writing Test Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.3 The Protocol Used by Master and Slave to Communicate . . . . . . . . . . . . . . . . . . . . . . 47
6.3.1 Three-Way Handshake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6.3.2 Filtering Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6.3.3 Running Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6.4 Interactive Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
7 Parameter Sets Defined by Equivalence Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
7.1 Defining Equivalence Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
7.2 Combining Equivalence Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
7.3 Choosing Representatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
7.3.1 Boundary Value Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.4 Extending Equivalence Classes by Logical Parameters . . . . . . . . . . . . . . . . . . . . . . . 53
7.5 Extending Equivalence Classes by Virtual Parameters . . . . . . . . . . . . . . . . . . . . . . . 53
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
CONTENTS 5
7.6 Using Equivalence Classes in Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
7.6.1 The Parameter List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
7.6.2 The Result and Objective List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
7.6.3 The for Attribute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
7.6.4 Including Header Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
8 Test Framework API Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
8.1 Common Master/Slave API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
8.1.1 Defines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
8.1.2 Enumerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
8.1.3 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
8.1.3.1 tfw_assertf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
8.1.3.2 tfw_coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
8.1.3.3 tfw_debugf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
8.1.3.4 tfw_dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
8.1.3.5 tfw_name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
8.1.3.6 tfw_strerror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
8.1.3.7 tfw_param . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
8.1.3.8 tfw_printf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
8.1.3.9 tfw_sendf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
8.1.3.10 tfw_sync . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
8.1.3.11 tfw_recvf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
8.1.3.12 tfw_tracef . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
8.1.3.13 tfw_wait . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
8.1.3.14 tfw_warnf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
8.1.3.15 tfw_reset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
8.2 Master Only API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
8.2.1 Structure Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
8.2.1.1 struct tfw_np_cb_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
8.2.2 Defines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
8.2.3 Function Type Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
8.2.3.1 tfw_np_char_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
8.2.3.2 tfw_np_line_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
8.2.4 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
8.2.4.1 tfw_bootmsg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
8.2.4.2 tfw_channel_id . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
8.2.4.3 tfw_channel_name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
8.2.4.4 tfw_channel_pause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
8.2.4.5 tfw_channel_read . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
8.2.4.6 tfw_channel_resume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
8.2.4.7 tfw_channel_write . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
8.2.4.8 tfw_connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
8.2.4.9 tfw_disconnect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
8.2.4.10 tfw_register . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
8.2.4.11 tfw_start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
8.2.4.12 tfw_join . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
8.2.4.13 tfw_tag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
6 CONTENTS
8.3 Slave Only API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
8.3.1 Structure Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
8.3.1.1 struct tfw_slave_cb_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
8.3.2 Defines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
8.3.3 Data Type Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
8.3.4 Function Type Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
8.3.4.1 tfw_slave_halt_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
8.3.4.2 tfw_slave_mutex_init_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
8.3.4.3 tfw_slave_mutex_destroy_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
8.3.4.4 tfw_slave_mutex_lock_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
8.3.4.5 tfw_slave_mutex_unlock_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
8.3.4.6 tfw_slave_coverage_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
8.3.4.7 tfw_slave_read_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
8.3.4.8 tfw_slave_reset_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
8.3.4.9 tfw_slave_write_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
8.3.5 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
8.3.5.1 tfw_coverage_beginf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
8.3.5.2 tfw_coverage_write . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
8.3.5.3 tfw_coverage_finish . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
8.3.5.4 tfw_noop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
8.3.5.5 tfw_init . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
8.3.5.6 tfw_secondary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
8.3.5.7 tfw_slave . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
8.3.5.8 tfw_read . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
8.3.5.9 tfw_write . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
8.4 User Interaction API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
8.4.1 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
8.4.1.1 tfw_interact_put_msgf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
8.4.1.2 tfw_interact_get_nl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
8.4.1.3 tfw_interact_get_uint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
8.4.1.4 tfw_interact_get_yn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
8.4.1.5 tfw_interact_get_cb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
9 Test Framework Tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
9.1 Global Only Tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
9.2 Normal Tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
10 Configuring Test Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
10.1 Target Hooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
10.1.1 The Variables Hook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
10.1.2 The Config Hook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
10.1.3 The Run Hook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
10.2 TCON Controlled Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
10.3 Unmanaged Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
A Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
A.1 A Test Case Using a Parameter Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
A.2 A Test Master Processing Boot Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
A.3 Testing a Communication Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
A.4 Sending Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
A.5 Mutexes in the Test Slave . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
CONTENTS 7
A.6 Ensure Master-Slave-Synchronisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
1 Basic Test Suite Usage
A test suite consists of test sets (containing test cases), and pre-defined configuration data used to compile and
run the test sets. In the following sections, it is described step-by-step how to prepare an existing test suite, how
to build and run test sets, and how to reset a test suite to the initial state it had before preparation.
All functionality of the test framework is exposed to the user by a single Makefile provided by the test suite. This
means that the user will usually call commands of the form:
[ts_example] $ make <command> OPTION1=<value1> OPTION2=<value2> ...
Usually the test framework is non-verbose while performing commands. By setting the special option VERBOSE
to yes the test framework can be instructed to be verbose about the commands being executed. Also various
commands will output additional log messages.
1.1 Preparing a Test Suite
Assume there is a test suite named ts_example and a shell is open in its base directory. At first a preparation
step is necessary. For this the command prepare is used:
[ts_example] $ make prepare
This command will retrieve the test framework from a predefined location and make it available to the test suite. If
the predefined location is not suitable it is possible to specify the location of the test framework using the option
TFW_DIR:
[ts_example] $ make prepare TFW_DIR=<tfw-directory>
The directory <tfw-directory> must contain the modules framework, common, and contrib of the test
framework. Further, the prepare command will create the directory sys/ in the test suite. Depending on the
configuration other directories might be created. All of this is done automatically by the prepare command.
The prepare command supports multiple additional options. See the detailed usage information of the prepare
command for the options provided by the test framework. It is output by the following command:
[ts_example] $ make help TFW_DIR=<tfw-directory> HELP=prepare
An important feature provided by the test frameworks prepare command is support for test suite overlays, which
can be used to extend or even replace parts of a test suite during test suite preparation. See section 2 for a
description of this concept.
The test suite itself might extend the prepare command with additional options, too. Refer to the test suite
documentation for details on this possible existing additional options.
1.2 Build-In Help Messages
The command help can be used to list all commands which are supported by the test framework installed:
[ts_example] $ make help
Further, the option HELP can be used to get detailed usage information for a specific command. For example, to
get detailed information on the help command the following command can be used:
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Listing Test Cases and Test Sets 9
[ts_example] $ make help HELP=help
The test framework supports various commands which depend on configuration data in the test suite. If this
configuration is not in place these commands are not shown in the output of the help command. This means the
output contains only the commands that will actually work.
1.3 Listing Test Cases and Test Sets
Two dedicated commands are available to list test cases (tclist) and test sets (setlist). See chapter 5 and 6
for a detailed description of test sets and test cases. For now it is only important to understand that a test case is
the most basic building block of testing. A test set is a certain set of test cases which are compiled into the same
binary. The test cases of a test suite can be listed using the following command:
[ts_example] $ make tclist
This will output all test cases of the test suite. The command supports various options that allow outputting only a
subset of the test suite’s test cases. For example it is possible to output only the test cases of a test set:
[ts_example] $ make tclist SET=<testsetname>
In a similar manner it is possible to output the names of all test sets of the test suite:
[ts_example] $ make setlist
Again it is possible to filter the output. For example the following command will output the test set a given test case
belongs to
[ts_example] $ make setlist TC=<testcasename>
For more information on the options available and their usage see the detailed usage information of tclist and
setlist.
1.4 Building and Running Tests
A test set is built by calling the build command and run by calling the run command. Both commands have the
same options, so they are discussed together.
1.4.1 Basic Usage
If no option is specified the build command builds all test sets for the default target board. But usually the user
specifies which test set to build and which target board shall be used:
[ts_example] $ make build SET=<testsetname> BOARD=<boardname>
Here it is important to understand that the only entity that can be built is the test set. The build command takes
care of the entire build process. It will build the test master and the test slave. For PikeOS test slaves this includes
cloning and building of e. g. application and integration projects. Output of the build command are binaries in
sys/bin/<testsetname>, which are used by the run command.
The same options are accepted by the run command:
[ts_example] $ make run SET=<testsetname> BOARD=<boardname>
The run command depends on the build command. If the test framework installation supports validation of XML
files the test cases XML files are validated before building. Both commands will output a log to the console and to
a file. The output of the build process is stored by default in a file in the test suites log/ directory.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
10 Basic Test Suite Usage
The output of the test run is stored in a file in the results/ directory. As file name the test set name with
the postfix .log respectively .res is used. The binaries and all intermediate files are stored within the sys/
directory. See section 1.6 for a description how to clean up.
1.4.2 Partial Test Set Builds and Runs
For development purposes the test framework supports building and running partial test sets. This means, it will
build a test set but the binary will only contain a subset of the test cases the test set consists of. This is done by
specifying e. g. the TC option:
[ts_example] $ make build TC="<testcasename1> <testcasename2>"
This will build the test set to which the test cases testcasename1 and testcasenme2 belong. But the binary
will only contain these two test cases. The other test cases of the test set are ignored. Of course, this makes
only sense for development. It depends on the test suite if partial test set builds are supported. If the test suite
does not support partial builds the build will usually fail with a linker error. If a partial test set build is run, it will
automatically run all compiled-in test cases. This is also what happens if the TC option is specified when calling
the run command:
[ts_example] $ make run TC="<testcasename1> <testcasename2>"
It is also possible to build the full test set, but execute only a part of it. For this the parameter SKIP_TO can be
used. Its value is the number of the test case to start at:
[ts_example] $ make run SET=<testsetname> SKIP_TO=5
This will build the specified test fully (i. e. no partial build, all test cases are included), but execution will start at the
fifth test case. The first four test case of the test set are skipped.
Another possibility is to use the command line options of the test master binary. It has an option that allows to
specify which test case of a test set shall be run. It can be set using the RUN_OPTS parameter:
[ts_example] $ make run SET=<testsetname> RUN_OPTS="-t tc_example_01"
Here the full test set is build, but by using the -t option the test master is instructed to run only the test case
tc_example_01. This feature is helpful in debugging single test cases of long running test sets that do not
support partial building.
The -t also supports running parts of a test case:
[ts_example] $ make run SET=<testsetname> RUN_OPTS="-t tc_example_01:3-8,10-15"
This will build the test set fully, but execute only sub test cases 3 to 8 and 10 to 15 of test case tc_exmaple_01.
In this case the result file generated by the test run will not contain objective results, but only sub results.
1.4.3 Manual Test Cases
A test case might be marked as "‘manual"’ (see section 6.1.1.4 for details). To include test cases marked as
manual the option INCLUDE_MANUAL_TC must be set to the value yes:
[ts_example] $ make run SET=<testsetname> INCLUDE_MANUAL_TC=yes
This will build and run the specified test set, including all test case marked as manual. Usually test cases are
marked as manual if they require user interaction.
For convenience test cases specified with the TC option are always included, no matter if marked as manual or
not.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Identification 11
1.4.4 Specifying Labels
The test framework supports the concept of labels. Labels are used to mark the association of test cases to a
certain test set. If they are used on such an association it is only applicable if the label is configured (see section
6.1.1.4 for how to configure labels). By using labels it is possible to disable test cases completely or to put them
into different test sets depending on configuration. Usually labels are configured statically in the testsuite.conf file.
But it is possible to overwrite this configuration using the LABEL option:
[ts_example] $ make build SET=<testsetname> LABEL="<Label1> <Label2>"
The value of the option LABEL is a white space separated list of the enabled labels. Note that the list of configured
labels is replaced completely if this option is used.
1.4.5 Temporarily Changing PikeOS Project Configuration
When building a test set it is possible to configure some aspects of the PikeOS projects used directly from the
command line. This feature is usually used for debugging purposes.
Supportedare the options APEX_OS_VARIANT, PIKEOS_COMPILATION_ASSERT, PIKEOS_COMPI-
LATION_LIBGCC, PIKEOS_COMPILATION_VERBOSE, PIKEOS_COMPILATION_WARNING_STRICT,
PIKEOS_COMPILATION_WERROR, PIKEOS_SMP, PIKEOS_TRACE, PSP_MEMSIZE, PSSW_COPY,
UK_BOOT_MESSAGE, and UK_LOG_LEVEL. Their values are passed through directly to the corresponding
option exposed in the PikeOS project configurator. This allows to select e. g. a multi-core kernel:
[ts_example] $ make build SET=<testsetname> PIKEOS_SMP=true
Please check the PikeOS documentation for a list of supported values.
1.5 Identification
The test framework outputs a short identification of itself if the command tfw_version is called. Additionally a
list of all installed test framework components is output.
[ts_example] $ make tfw_version
1.6 Cleaning Up the Test Suite
There are three different commands for clean-up: clean, distclean, and mrproper. These commands reliably
clean up test framework generated files. If test suite provided tools create or modify additional files, additional
steps might be necessary.
1.6.1 Removing Intermediate Files
The command clean can be used to remove all intermediate files in the test suite:
[ts_example] $ make clean
Usually intermediate files are stored within the sys/ directory. This command does neither remove build logs,
result logs, nor binaries.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
12 Basic Test Suite Usage
1.6.2 Resetting to Initial State
The command distclean can be used to remove all files that did not exist in the test suite after preparation:
[ts_example] $ make distclean
Additionally to the actions performed by the clean command this command will remove build logs, result logs,
and binaries, too.
1.6.3 Reverting Preparation
The command mrproper can be used to remove all created files. Additionally it undoes the effect of the prepare
command:
[ts_example] $ make mrproper
After this the test suite is in the same state as before the prepare command (see section 1.1) was executed.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
2 Overlays
With overlays it is possible to add new files and directories to a test suite or to replace existing files and directories.
Additionally it is possible to replace the content of the test frameworks contrib/ module and to make files
available used in the PDF document generation process. These features directly correspond to the three types of
overlays supported. These are:
• The test suite specific overlay, which supports creation or replacement of files and directories in the test
suites top level directory. The location of this overlay is configured using options named TE_OVER-
LAY_*_PATH.
• The test suites common overlay, which is used by the prepare command instead of the test frameworks
contrib/ module to populate the sys/ directory. The location of this overlay is configured using options
named TE_OVERLAY_*_COMMON_PATH.
• The generic overlay providing files for PDF document generation only. The location of this overlay is
configured using options named TE_*_COMMON_PATH.
The three types of overlays can be used independently of each other.
2.1 Creating Overlays
2.1.1 The Directory Structure of an Overlay
The general structure of an overlay is shown in Figure 1. In the example the overlay has the name “te_example”.
The name can be arbitrary but is usually prefixed with “te_”. The overlay contains a common/ directory. This
directory contains the generic overlay referenced to with the TE_*_COMMON_PATH. Additionally there are multiple
directories named overlays.*/. Each of them contains one or more test suite specific overlays and the test
suite common overlay for a certain architecture. In the example only the content of the directory overlays.$Y/ is
shown. In its sub-directory named common the test suite common overlay is stored. Further it contains directories
named like the test suites the overlay is for. These directories contain the test suite specific overlays.
te_example/
+ common/
+ overlays.$X/
+ overlays.$Y/
+ common/
+ ts_example/
+ ts_example2/
+ overlays.$Z/
Figure 1: The structure of an overlay.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
14 Overlays
2.1.2 The Content of the Test Suite Specific Overlay
The test suite specific overlay has a very simplest structure. Its contains files and directories that shall be copied
into the test suite. If the test suite contains already a file or directory with that name it is removed. Also this only
works in the top level directory of the test suite. This means, it is possible to replace the entire config/ directory
of the test suite, but not single files within it.
The most common usage of this type of overlay is replacing the directories config/ and verify/ of the test
suite.
2.1.3 The Content of the Test Suite Common Overlay
The content of this overlay is used as starting content of the sys/ directory in the test suite. Most importantly it
contains the seed of application and integration projects used by the test suite in its sub-directory seed/. It is also
possible to store target.* files in the top level directory of a test suite common overlay. The test framework will
use them unless a file with identical is also provided in the config/ directory of the test suite.
Sometimes this type of overlay is used to provide files which are need by multiple test suites of a project. This is
a supported use case.
2.1.4 The Content of the Generic Overlay
The content of this type of overlay is used for PDF document generation only. Also it is not made available directly
in the test suite but in a sub-directory of extern/. See section 4.3 for a description of which files are supported
in it.
2.2 Using Overlays
Overlays are applied to a test suite during preparation (see section 1.1). For this the location of the overlay to
use is passed as option to the prepare command. The test framework supports drawing the overlay from a TAR
archive, from CVS repository, and from a GIT repository.
To use an overlay from a TAR archive the following options must be passed to the prepare command:
TE_OVERLAY_TAR_FILE: This option specifies the path to the tar file containing the overlay. Usage of this option
is mandatory if an overlay from a TAR file is used.
TE_OVERLAY_TAR_PATH: This option is used to specify the location of the test suite specific overlay within the
tar file. This option is optional.
TE_OVERLAY_TAR_COMMON_PATH: This option is used to specify the location of the test suite common overlay
within the tar file. This option is optional.
TE_TAR_COMMON_PATH: This option specifies the location of the generic overlay within the tar file. This option
is optional.
For overlays from CVS the following options must be passed to the prepare command:
TE_OVERLAY_CVS_REPO: This option specifies the name of the CVS repository containing the overlay. It will
be passed directly to the cvs command and must contain everything needed by it (e. g. the user name) to
identify the repository. Usage of this option is mandatory if an overlay from a CVS repository is used.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Using Overlays 15
TE_OVERLAY_CVS_BRANCH: This option specifies the branch in the CVS repository. It is optional. If it is not
specified CVS HEAD is used.
TE_OVERLAY_CVS_PATH: This option is used to specify the location of the test suite specific overlay within the
CVS repository. This option is optional.
TE_OVERLAY_CVS_COMMON_PATH: This option is used to specify the location of the test suite common overlay
within the CVS repository. This option is optional.
TE_CVS_COMMON_PATH: This option specifies the location of the generic overlay within the CVS repository. This
option is optional.
For overlays from GIT the following options must be passed to the prepare command:
TE_OVERLAY_GIT_REPO: This option specifies the name of the GIT repository containing the overlay. Usage of
this option is mandatory if an overlay from a GIT repository is used.
TE_OVERLAY_GIT_BRANCH: This option specifies the branch in the GIT repository. It is optional. If it is not
specified the "master" branch is used.
TE_OVERLAY_GIT_PATH: This option is used to specify the location of the test suite specific overlay within the
GIT repository. This option is optional.
TE_OVERLAY_GIT_COMMON_PATH: This option is used to specify the location of the test suite common overlay
within the GIT repository. This option is optional.
TE_GIT_COMMON_PATH: This option specifies the location of the generic overlay within the GIT repository. This
option is optional.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
3 Test Set Pools
Test set pools enable the reuse of test sets in multiple test suites without error-prone manual copy-and-paste. A
test set pool is separate from the test suite, and provides additional test set to the test suite.
The following sections describe (1) the layout of the directory hierarchy to be used as a test set pool, (2) how to
work with test set and test cases from the test set pool, and (3) how to interface with a test suite.
3.1 Creating Test Set Pools
3.1.1 Naming conventions
Test set in a pool can be saved at arbitrary (at least two) directory levels. Valid directory names start with a letter
and do not contain dashes, dots or spaces.
3.1.2 Directory Layout of a Test Set Pool
A test set contains at least a sysconf.rc file. See Section 5 for a detailed explanation. A subdirectory tc/
contains the test cases comprising this test set.
testsetpool/
+ subdir1/
+ subdir2/
+ subsubdir2/
+ description.xml
+ int.conf
+ snip.xml
+ sysconf.rc
+ tc/
+ testcase_name_starting_with_a_letter_01/
+ tc.xml
+ tools/
3.1.3 The Test Case XML Description File V2.1
Test cases in the pool differ from traditional test cases (as described in Section 6) in the way that no requirements
are referenced. Therefore, the V2.1-format of the XML description is missing those parts. The test case description
comprises the following parts:
• Objectives of the test case
• A description part using a subset of HTML to explain:
Setup and constraints
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Using Test Set Pools 17
∗ Test setup (reference to a setup described in the Test Case Master document
∗ Additional constraints on the test environment
Test design
Test procedure
• Test input(s) (e.g. equivalence class definition)
Figure 2 shows a template for a test case description to be filled by the test case developer.
The XML document uses several elements. The root of the document is the <tc/> element. Its attributes are
described in section 6.1.1.
3.1.3.1 The <objectives/> and <obj/> Elements
The <objectives/> element contains a non-empty list of <obj/> elements. Each of them describes exactly one test
objective. The name of the test objective is given in the <obj/> elements id attribute. The test objective itself is
described in the text content of the <obj/> element.
3.1.3.2 The <tags/> Element
After the <objectives> element there may be an optional <tags> element. It can be used to configure different
aspects of the test framework. In the example the maximum time the test master waits for messages from the
slave is set to 30 seconds (default is 10) and the maximum number of non-protocol messages is set to 1000. See
section 9 for a full list of tags supported.
3.1.3.3 The <description/> Element
The format of the test case description is described in detail in section 6.1.1.6.
3.1.3.4 The <equitable/> Element
Equivalence classes are an optional part of the test case XML file described in section 7.
3.2 Using Test Set Pools
Test set pools are applied to a test suite during preparation (see section 1.1). There are four variants to specify
the source of a test set pool: a git repository, a cvs repository, a tar file, or a local directory.
To use a test set pool from a tar file, the following option is supported by the prepare command:
TFW_POOL_TAR_FILE: This option specifies the path to the tar file containing the test set pool.
To use a test set pool from a cvs repository, the following options are supported by the prepare command:
TFW_POOL_CVS_REPO: This option specifies the name of the CVS repository containing the test set pool. It will
be passed directly to the cvs command and must contain everything needed by it (e. g. the user name) to
identify the repository. Usage of this option is mandatory if a test set pool from a CVS repository is used.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
18 Test Set Pools
1 <?xml version="1.0" encoding="UTF-8"?>
2 <tc
3 xmlns="http://www.sysgo.com/xsd/tfw/tc-pool-2.1.xsd"
4 xmlns:tc="http://www.sysgo.com/xsd/tfw/tc-pool-2.1.xsd"
5 brief="<Brief description of the test case>"
6 id="testcase_name_starting_with_a_letter_01"
7 author="..."
8 rev="$Id: doc/tfwref/pool.tex 2019-05-09 14:23:47 +0200
e78edc5973358f48061a5f8cc192091b3a894f0c $"
9
10 >
11
12 <objectives>
13 <obj id="EXAMPLE1">description of objective EXAMPLE1</obj>
14 <obj id="EXAMPLE2">description of objective EXAMPLE2</obj>
15 </objectives>
16
17 <tags xmlns="http://www.sysgo.com/tfwtags">
18 <tag name="msg_nonproto" value="1000"/>
19 <tag name="msg_timeout" value="30"/>
20 </tags>
21
22 <tc:description xmlns="http://www.w3.org/1999/xhtml">
23 <h1>Test Setup</h1> <p>...</p>
24 <h1>Design</h1>
25 <h2>Preparation</h2> <p>...</p>
26 <h2>Test</h2> <p>...</p>
27 <h2>Cleanup</h2> <p>...</p>
28
29 <h1>Equivalence Classes</h1> <p>...</p>
30 <tc:description>
31
32 <equitable xmlns="http://www.sysgo.com/eq">
33 [...]
34 </equitable>
35
36 </tc>
Figure 2: Example for a test case description XML V2.1
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Support for Multiple Test Set Pools 19
TFW_POOL_CVS_MODULE: This option specify the CVS module to check out from the repository. This option is
mandatory if a test set pool from a CVS repository is used.
TFW_POOL_CVS_BRANCH: This option specifies the branch in the CVS repository. It is optional. If it is not
specified CVS HEAD is used.
To use a test set pool from a git repository, the following options are supported by the prepare command:
TFW_POOL_GIT_REPO: This option specifies the name of the GIT repository containing the test set pool. Usage
of this option is mandatory if an overlay from a GIT file is used.
TFW_POOL_GIT_BRANCH: This option specifies the branch in the GIT repository. It is optional. If it is not specified
the "master" branch is used.
To use a test set pool from a local directory, the following option is supported by the prepare command:
TFW_POOL_LOCAL_DIR: This option specifies the local directory containing the test set pool. Usage of this
option is mandatory if a test set pool from a local directory is used.
There are some common options for test set pools:
TFW_POOL_TARGET: The test set from the pool are checked out to ./pool/<pool-name>. The default pool
name is either the git repository name, the cvs module name, the basename of the tar file, or the lo-
cal directory name. This option can be used to explicitly specify a target directory, where to place
the test set relative to ./pool/. Setting this option to "arbitrary/path" results in a checkout to
"pool/arbitrary/path".
KEEP_VCS: By default the test set pool is checked out without version control files. Set this option, to be able to
modify and commit files to the source repository.
TFW_POOL_PATHS: By default the complete test set pool is checked out. This option can be specified, to limit the
checkout to specific paths of the pool. Multiple paths are separated by white space.
3.3 Support for Multiple Test Set Pools
A test suite can make use of multiple test set pools. This requires a Makefile, e.g. devel/Makefile.install
to specify the target specific variable values for each test set pool.
The variable TFW_POOLS specifies the target directories of the test set pools relative to the test suite directory.
E.g.
TFW_POOLS="pool/cvspool pool/gitpool"
For each of the test set pool targets, the corresponding options are specified as target-specific makefile variables.
E.g.
1 pool/cvspool: TFW_POOL_CVS_REPO :=
:ext:$(CVSUSER)@cvs.sysgo.com:/home/keeper/repos/00101-p4
2 pool/cvspool: TFW_POOL_TARGET := cvspool
3 pool/cvspool: TFW_POOL_CVS_MODULE := cert/tp
4
5 pool/gitpool: TFW_POOL_GIT_REPO := git@server:path/on/server/pool
6 pool/gitpool: TFW_POOL_TARGET := gitpool
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
20 Test Set Pools
It is important to specify TFW_POOL_TARGET to the same value as the target’s subdirectory in ./pool/, if the
pool repository has a different name.
3.4 Automatic Generation of Test Set Pool Variables from Trace-XML-files
Test suites specify the usage of test set from the pool via xml files in trace/ and verify/trace/ directories.
Those xml files must comply to the http://www.sysgo.com/xsd/tfw/rqtrace-2.1.xsd.
In those xml files, test cases coming from a test set pool are named by their dot-names, e.g.
cvspool.subdir2.subsubdir2-testcase_name_starting_with_a_letter_01
Upon generation of the test suite the rqtrace-files are parsed for test cases from test set pools, i.e. counting dots.
An appropriate Makefile is generated, which contains TFW_POOL_PATHS set for each identified test set.
pool/cvspool: TFW_POOL_PATHS += subdir2/subsubdir2
Different pools are distinguished by the name of the pool, as used in TFW_POOL_TARGET, which is also the first
directory part of the test set name.
This implies that the appropriate variables have to be set, where to get the test set pool from, by means of either
global options 3.2 or target specific makefile variables 3.3 as described in the corresponding sections above.
When using trace-XML-files, the test framework automatically removes unused test cases from the checkout pool.
To avoid removal of an otherwise unreferenced test case, append it to the variable TFW_POOL_TEST_CASE_LIST
as follows:
pool/cvspool: TFW_POOL_TEST_CASE_LIST += subdir2/subsubdir2/tc/tc_01
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
4 Verification Result Artifacts
The following sections describe how to create verification result artifacts, including how to modify the inputs.
4.1 Artifact Creation
This section describes how to create the artifacts if the test suite is already prepared for it. See Sections 4.2 and
4.3 for what must be set up before the commands described here are available.
4.1.1 Building the Test Case Master Document
The test case master document is the artifact containing the description of test approaches, test sets, test cases,
test tools, and everything else necessary to understand how the requirements under test are tested. For each
test case it lists which requirements are tested. It does neither contain test results nor target specific data. This
information is provided by the test result and the structural coverage result document. Usually, we refer to the test
case master document as the TC document.
To build the test case master document it is sufficient to call the tc command of the test framework:
[ts_example] $ make tc
This will create the TC document as HTML. It is also possible to create a PDF variant of the TC document. For
this the option PDF must be set:
[ts_example] $ make tc PDF=yes
With this command the HTML and the PDF variant of the TC document will be created. The test framework differs
between the “notrace”, the “specific”, and and the “generic” test case master document. The difference between
them is which test cases and test sets are described and if trace data is contained.
If the test suite uses either a test pool or manages trace data in the trace/ or verify/trace/ directory, only
the “notrace” document is supported. It contains no trace data and only test cases and test sets that are currently
enabled by labels (see section 6.1.1.4 for details on labels).
If the test suite neither uses a test pool nor manages trace data in the trace/ or verify/trace/ directory the
“specific” and “generic” documents are available, too. In this case the “specific” document is generated by default.
But the “notrace” document is still available by setting the option TYPE accordingly.
The “specific” document contains the same test cases and test sets as the “notrace” document. Additionally, it
also contains trace data. This means it explains which requirements are tested by objectives of the test cases. By
setting the option TYPE to generic the generic TC document can be created:
[ts_example] $ make tc PDF=yes TYPE=generic
It contains the description of all test cases and test sets that would be selected if all labels configured in the file
doc/labels.xml were enabled. I. e. this call is equivalent to
[ts_example] $ make tc PDF=yes LABEL="<list of labels from doc/labels.xml>"
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
22 Verification Result Artifacts
This feature is usually used if multiple test environments are used within a single project: the generic test case
master document lists all existing test cases and test sets and the test environment specific test result document
specifies which of them are applicable. By this a single TC document can be used for all test environments.
For development it is also possible to create a TC document containing only the description of specific test cases
and test sets using the options TC and SET when calling the tc command. For example
[ts_example] $ make tc PDF=yes TC=tc_example_01
will create a TC document containing only the description of the test case tc_example_01.
4.1.2 Building the Test Result Document
The test result document is the artifact containing the test results for the requirements under test. Except for
the introduction it is often completely auto-generated by the test framework. Usually, we refer to the test result
document as the "‘TR"’ document.
Before the test result document can be built test results must be available. Test results are taken from the result
log created by the test framework when a test set is run (see section 1.4). So let’s assume result logs for all test
sets of the test suite are available. Then the test result document can be built using the tr command:
[ts_example] $ make tr BOARD=<boardname>
Similar to the tc command this will create the HTML variant of the test result document. But there is one important
difference: When building the test result document the board must be specified (if it is different from the default
board). The reason for this is that the test environment must be known to decide upon unsupported requirements.
The option PDF can be used to create the PDF variant of the test result document, too:
[ts_example] $ make tr BOARD=<boardname> PDF=yes
After creation of the document it must be checked that the test results listed in the document are as expected.
Especially it must be checked that for all requirements with test result R_ERROR or R_FAIL a justification is given.
If a justification is missing the reason for this must be investigated. Basically there are four possibilities:
System under Test is broken The test suite detected a new, unknown error in the system under test. This must
be reported (e. g. by creating a problem report). Then a justification referring to the report is added to the
test result document.
Test suite is broken The source of the error is in the test suite. In this case it must be fixed. After this the
document creation process can be restarted.
Temporary target error There are many sources for temporary errors, which are neither related to the system
under test nor the test suite. This could be network errors like failing TFTP servers and unreachable license
servers. In this case the process must be redone starting at the point where the temporary error happened.
User error The user made an error in the process. A common error is a missing or wrong BOARD option. In this
case the process must be redone, starting with the faulty command.
4.1.3 Building the Structural Coverage Result Document
The structural coverage result document is the artifact containing detailed information upon the achieved structural
coverage. We refer to this document as the “SC” document. Structural coverage is measured using test cases.
The test results for these requirements are often added to an appendix of the structural coverage result document.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Artifact Setup in XML 23
They can be compared to the test results listed in the test result document. If the results are identical it can be
assumed that the instrumentation process didn’t introduce errors.
Before the structural coverage result document can be build result logs containing structural coverage data must be
available and a report must have been generated. Then the structural coverage result document can be generated
using the sc command:
[ts_example] $ make sc BOARD=<boardname> COV_TOOL=<coverage_tool>
This will generated the HTML variant of the structural coverage result document. Again the option BOARD is
necessary to decide upon unsupported requirements. The option COV_TOOL is used to specify the name of the
tool used for measuring structural coverage.
A PDF variant of the document can be created by setting the PDF option: sc command:
[ts_example] $ make sc PDF=yes BOARD=<boardname> COV_TOOL=<coverage_tool>
After creation of the document it must be checked that the content is as expected: Full structural coverage for the
system under test is needed. Any deviation must be justified.
Test results for all requirements must be identical to the ones listed in the test result document. Any deviation must
be qualified.
If the content of the document is not as expected the reason for this must be investigated. Possible reasons
include:
Justifications are not attached correctly to uncovered code Justifications are attached to uncovered code
using line numbers. Often even non-functional code changes affect the line numbers. In this case justi-
fications will be missing in the structural coverage document. To fix this the line numbers must be updated
and the report must be regenerated. After this a fixed structural coverage result document can be gener-
ated.
Requirement results are different from the ones listed in the TR document and no justification is given
Apart from the possible reasons listed in section 4.1.2 it can happen that the test cases did detect the code
changes introduced by instrumentation. A justification for this must be added to the document. Especially
it must explain what causes the deviation and why this is not a problem for measuring structural coverage.
Usually this is done in a separate section in the document.
Temporary target error There are many sources of temporary errors that can happen during testing, which are
neither related to the system under test nor the test suite. This could be network errors like failing TFTP
servers and unreachable license servers. In this case the process must be redone starting at the point
where the temporary error happened.
User error The user made an error in the process. A common error is to forget to specify the BOARD option. In
this case the process must be redone starting at the faulty command.
4.2 Artifact Setup in XML
This section describes how to setup creation of artifacts using XML files. This is used for the first step of document
creation.
4.2.1 XML Document Fragments
This section describes the building blocks of the documents. The source of all documents is maintained in XML
files. The test framework automatically creates various additional XML files based on e. g. current test results,
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
24 Verification Result Artifacts
test cases, and test environment. A mechanism to include these files into the document is provided. E. g. to
build the test case master document a file doc/tc.xml must exist. In Figure 3 on page 25 a minimal variant
of this file is shown. Using XSLT scripts it is converted to an HTML and a TeX document. The elements of the
http://www.sysgo.com/tcm namespace used by this document have the following semantics:
<tcm:approvals/> The element <tcm:approvals/> can be used to specify the names and the functions of the
persons which are going to approve the document. It’s content is a nonempty set of <tcm:signer/> elements.
This element may be used as document root.
<tcm:citations/> The purpose of the optional <tcm:citations/> element is to specify documents which shall always
be added to the list of referenced documents. Even, if they are not used explicitly by the document (in which
case they are added automatically). The individual documents are specified using child elements of type
<tcm:cite/>. This element may be used as document root.
<tcm:cite/> The <tcm:cite/> element has a single attribute named ’key’. It’s value is the BibTeX key of a document
that shall always be added to the list of referenced documents. This element is always the child of a
<tcm:citations/> element.
<tcm:entry/> A <tcm:entry/> element contains in its context a single entry of the revision history as text. This
element is always the child of a <tcm:revision/> element.
<tcm:file/> The element <tcm:file/> can be used to point to an XML file with any of the elements listed as “may
be used as document root” here as root element. The file will be interpreted as if the <tcm:file/> ele-
ment was replaced by the root element of the file pointed to. The file name must be specified relative
to the file that contains the <tcm:file/> element. So if doc/tc.xml contains a <tcm_file/> element refer-
ring ../sys/testsets.xml the file sys/testsets.xml is included. This element is always the child of a
<tcm:master_document/>, a <tcm:fragment_set/>, or a <tcm:group/> element.
<tcm:fragment/> Elements of type <tcm:fragment/> contain the actual text content. Allowed content are HTML
code and <tcm:latex/> elements only. This element may be used as document root.
<tcm:fragment_set/> The elements of type <tcm:fragment_set/> may contain elements of type
<tcm:approvals/>, <tcm:citations/>, <tcm:history/>, <tcm:file/>, <tcm:fragment/>, <tcm:group/>, and
<tcm:properties/>. This element may be used as document root.
<tcm:group/> The elements of type <tcm:group/> may contain elements of type <tcm:file/>, <tcm:group/>, and
<tcm:fragment/>. The effect of the <tcm:group/> element is that the level of every HTML heading defined
inside (either directly using <tcm:fragment/> or within a file included using <tcm:file/> is increased by one.
This element is always the child of a <tcm:master_document/>, a <tcm:fragment_set/>, or a <tcm:group/>
element.
<tcm:history/> The element <tcm:history/> is used to specify the revision history of the document. Its content
are elements of type <tcm:revision/>, which contain the revision data for a single revision. This element
may be used as document root.
<tcm:latex/> The element <tcm:latex/> can be used inside a <tcm:fragment/> to provide LaTeX code which is
copied verbatim to the TeX file. It is ignored completely for HTML output. In the example it is used to output
the bibliography and to mark the beginning of the appendix.
<tcm:master_document/> The <tcm:master_document/> element is the root element of the entire document.
It’s content may be elements of the types <tcm:approvals/>, <tcm:citations/>, <tcm:history/>, <tcm:file/>,
<tcm:fragment/>, <tcm:group/>, and <tcm:properties/>. All verification result documents are created using
these document fragments.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Artifact Setup in XML 25
1 <?xml version="1.0" encoding="UTF-8" ?>
2 <tcm:master_document
3 xmlns:tcm="http://www.sysgo.com/tcm"
4 xmlns="http://www.w3.org/1999/xhtml"
5 brief="Description of test suite [name]"
6 author="..." rev="$Id: doc/tfwref/tfwref.tex 2019-06-12 13:44:08 +0200
c5b3f0d1d51c8540f631eddf7ecfc0103e628c04 $"
7 >
8
9 <tcm:properties>
10 <tcm:prop key="Title" value="Title of the Document"/>
11 <tcm:prop key="ID" value="01234-5678-EXAMPLE-TC"/>
12 <tcm:prop key="Revision" value="20.0"/>
13 <tcm:prop key="Date" value="2018-05-04"/>
14 <tcm:prop key="State" value="App"/>
15 <tcm:prop key="Author" value="Jon Doe"/>
16 </tcm:properties>
17
18 <tcm:approvals>
19 <tcm:signer function="Author" name="Jon Doe"/>
20 ...
21 </tcm:approvals>
22
23 <tcm:history>
24 <tcm:revision revision="20.0" date="2017-11-13">
25 <tcm:entry>Initial revision</tcm:entry>
26 </tcm:revision>
27 </tcm:history>
28
29 <tcm:citations>
30 <tcm:cite key="RSTD,DSTD"/>
31 </tcm:citation>
32
33 <tcm:fragment>
34 <h1>Introduction</h1>...
35 <tcm:latex>\makeBib</tcm:latex>
36 </tcm:fragment>
37
38 <tcm:fragment>
39 <h1>Overview</h1>...
40 </tcm:fragment>
41
42 <tcm:group>
43 <tcm:file name="../sys/labels_available.xml"/>
44 <tcm:file name="integration-projects.xml"/>
45 </tcm:group>
46
47 <tcm:file name="../sys/tc_body.xml"/>
48
49 <tcm:fragment>
50 <tcm:latex>\appendix</tcm:latex>
51 </tcm:fragment>
52 <tcm:file name="../sys/tc2set.xml"/>
53
54 </tcm:master_document>
Figure 3: Minimal Content of doc/tc.xml
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
26 Verification Result Artifacts
<tcm:prop/> The <tcm:prop/> element is used to specify the value of a single document property shown on the
title page. It has the attributes ’key’ and ’value’. The following values for the ’key’ attribute are supported:
Author To specify the author of the document.
Date To specify the date of the document. The special value “today” is replaced with the current date.
ID To specify the document ID.
Revision To specify the revision of the document.
State To specify the approval state of the document.
Title To specify the title of the document.
This element is always the child of a <tcm:prop/> element.
<tcm:properties/> The <tcm:properties/> element is used to specify the document properties which will show up
on the title page of the PDF document generated. The individual properties are defined using <tcm:prop/>
child elements. These are the title, the ID, the revision and date, the state, and the author. If the value
attribute of the date property has the value “today” it is replaced with the current date.
<tcm:revision/> The element <tcm:revision/> specifies the revision history for a single revision. It has the
attributes “revision” and “date”. The “revision” attribute contains the name of the revision. In the “date”
attribute the date is given. Here the special value “today” is replaced with the current date. The actual
revision history entries are provided in subelements of type <tcm:entry/>. This element is always the child
of a <tcm:history/> element.
<tcm:signer/> The <tcm:signer/> element is used to specify name and function of a single signee. It has the
attributes “function” and “name”. The “function” attribute is used to specify the role of the signee. Supported
roles are Author, Functional Lead, Quality Assurance, Verifier, and Validator. The “name” attribute is used
to specify the full name. This element is always the child of a <tcm:approvals/> element.
4.2.2 Setup of the Test Case Master Document
The main entry point for the test case master document is the file doc/tc.xml. An example is shown in Figure
3. Usually the document explains after a short introduction the test approaches used. Then the description of the
individual test cases and test sets is given.
The following sub-sections describe the usual content of doc/tc.xml and how to modify it.
4.2.2.1 The File doc/tc.xml
The file doc/tc.xml usually defines four sections only. All other content is encapsulated inside <tcm:group/>
elements. By this it becomes a part of the previously defined section.
Introduction This is the first section of the document. It describes the purpose of the document and contains the
list of referenced documents.
Overview This section is used to give a short overview of the test suite and to explain the test approaches used.
In this section the file sys/labels_available.xml is included. It is automatically generated by the test
framework. It contains a short description (see section 4.2.2.2) of all test case labels available in the test
suite.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Artifact Setup in XML 27
1 <?xml version="1.0" encoding="US-ASCII"?>
2 <labels xmlns="http://www.sysgo.com/vv/labels">
3
4 <label name="disabled" show="no">
5 This label is used to disable test cases that are not used
6 anymore.
7 </label>
8
9 <label name="for-PikeOS">
10 This label is used to select test cases that are only
11 applicable if the target system is PikeOS based.
12 </label>
13
14 </labels>
Figure 4: Example for doc/labels.xml
In the example the file doc/integration-projects.xml is included, too. It is used to explain which
application and integration projects are used by the test suite. For PikeOS test suites this is a mandatory
part of the test case master document, which is needed to understand the test approaches.
none After this the file sys/tc_body.xml is included. It contains one or multiple sections describing all test
cases and test sets of the test suite. This file is automatically generated by the test framework.
Appendix The LaTeX command to switch to the appendix is emitted. After this the file sys/tc2set.xml is
included. It contains a table associating test cases with test sets.
4.2.2.2 Configuring Label Descriptions
For the test case master document a description of all labels in use is needed. The user provides this description
in the file doc/labels.xml (see Figure 4 for an example). For each label in use a <label/> element must exist in
this file. The description of the label is given as content of the <label/> element. The attribute show can be used
to exclude labels from the test case master document. For this it must be set to the value no.
The file doc/labels.xml is used by the test framework to generate the file sys/labels_available.xml (see
section 4.2.2.1).
4.2.2.3 Using description.xml Files
A common approach to assemble the test case master document is to use files named description.xml to
provide individual sections. E. g. there could be a single description.xml file for every test tool provided by
the test suite. Usually there are also description.xml files in the seed of application and integration projects.
All these files must be included manually in doc/tc.xml (either directly or indirectly) using <tcm:file/>. The only
exception to this rule are description.xml files in the test set directory.
The description.xml files in the test set directory are picked up automatically by the test framework. Refer-
ences to them are added to the file sys/testsets.xml. See Figure 5 for an example of such a description.xml
file. Usually it consists of a <tcm:fragment/> element only. It always contains a short description of the test set.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
28 Verification Result Artifacts
1 <?xml version="1.0" encoding="US-ASCII" ?>
2 <tcm:fragment
3 xmlns:tcm="http://www.sysgo.com/tcm"
4 xmlns="http://www.w3.org/1999/xhtml"
5 brief="" author="..." rev="$Id: doc/tfwref/tfwref.tex 2019-06-12 13:44:08 +0200
c5b3f0d1d51c8540f631eddf7ecfc0103e628c04 $"
6 >
7
8 <h1>PikeOS</h1>
9 <p>This test set contains test cases testing PikeOS. The tests are run within
10 a PSSW resource partition. For this the application project "app" and the
11 integration project "simple" are used.</p>
12
13 </tcm:fragment>
Figure 5: Example for the description.xml File of a Test Set with heading.
Depending on the type of TC document to be generated it either contains a level one headline containing the test
set name or not1 .
4.2.2.4 Additional Document Parts
The test suite might build additional document parts. To this end it must create new make file fragments defining
the necessary targets (see section 4.4 for details). The name of the targets must be added to the make variable
TFW_TC_HTML_DEPS.
4.2.3 Setup of the Test Result Document
The main entry point for the test result document is the file doc/tr.xml. The following sub-sections describe the
usual content of it and how to modify it.
4.2.3.1 The File doc/tr.xml
The content of the file doc/tr.xml (see Figure 6) is very generic. Usually it contains the following sections:
Introduction This section describes the purpose of the document. It lists the referenced documents. Also the file
sys/tr_labels_used.xml is included, which describes the labels enabled for the test run.
none The file sys/tr_body.xml is included. It contains one or more sections listing all test results.
Another section without a headline Finally the file includes sys/system.xml. This file is automatically cre-
ated by the test framework. It contains a description of the test environment (e. g. the PikeOS and test
framework version used). It contains already a headline. So it is included without headline and <tcm:group/>
element. To put it into the appendix the corresponding latex command is called before inclusion.
1
TC documents without trace data automatically add the headline.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Artifact Setup in XML 29
1 <?xml version="1.0" encoding="US-ASCII" ?>
2 <tcm:master_document
3 xmlns:tcm="http://www.sysgo.com/tcm"
4 xmlns="http://www.w3.org/1999/xhtml"
5 brief="test result document"
6 author="..."
7 rev="$Id: doc/tfwref/tfwref.tex 2019-06-12 13:44:08 +0200
c5b3f0d1d51c8540f631eddf7ecfc0103e628c04 $"
8 >
9
10 <tcm:fragment>
11 <h1>Introduction</h1>
12 <h2>Identification</h2> <p> ... </p>
13 <h2>Purpose of this Document</h2> <p> ... </p>
14 <tcm:latex>\makeBib</tcm:latex>
15 ...
16 </tcm:fragment>
17 <tcm:group>
18 <tcm:file name="../sys/tr_labels_used.xml"/>
19 </tcm:group>
20
21 <tcm:file name="../sys/tr_body.xml"/>
22
23 <tcm:fragment>
24 <tcm:latex>\appendix</tcm:latex>
25 </tcm:fragment>
26 <tcm:file name="../sys/system.xml"/>
27
28 </tcm:master_document>
Figure 6: Example for doc/tr.xml. Not shown are document properties, approvals, and other special elements.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
30 Verification Result Artifacts
4.2.3.2 Additional Document Parts
The test suite might build additional document parts. To this end it must create new make file fragments defining
the necessary targets (see section 4.4 for details). The name of the targets must be added to the make variable
TFW_TR_HTML_DEPS.
4.2.4 Setup of the Structural Coverage Result Document
The main entry point for the structural coverage result document is the file doc/sc.xml. The following sub-
sections describe the usual content of this file and how to modify it.
4.2.4.1 The File doc/sc.xml
Usually the file doc/sc.xml (see Figure 7 for an example) contains the following sections:
none The file sys/tr_labels_used.xml is included in a <tcm:group/> element without a previous headline.
When generating a PDF document this has the affect of attaching the content of the file included to the last
section defined in the TeX-part of the document.
Structural Coverage Results for . . . This section contains the actual structural coverage results. The <h1/>
element containing the headline sets the class attribute to the value landscape. This has the effect that in
the PDF document generated the entire section is formatted in landscape mode. Here it is suggested to
use this as the section contains many wide tables which will not fit nicely otherwise.
The actual structural coverage report is contained in the file sys/tr_coverage.xml, which is generated
by the test framework. Usually this file contains already justifications for all missing coverage. If not it is
suggested to add a sub-section “Expected Defects in Terms of Structural Coverage Results” as shown in
the example. If full coverage is achieved this section can be skipped.
Test Results for . . . This section contains the test results for the requirements of a single requirement document.
The important difference to equivalent section in the test result document is that this section contains test
results created with a modified (i. e. instrumented to measure structural coverage) software under test.
The actual results are provided by the test framework in a file named
sys/tr_tables_<shortcut>.xml, where <shortcut> is the abbreviation used for the require-
ment document within the test suite. If there are any expected deviations due to the instrumentation these
should be justified in a sub-section “Expected Defects Due to Structural Coverage Instrumentation” as
shown in the example.
If multiple requirement documents are under test this section is repeated for all of them.
none The file ends with including sys/system.xml. This file is automatically created by the test framework.
It contains a description of the test environment (e. g. the PikeOS and test framework version used). It
contains already a headline. So it is included without headline and <tcm:group/> element.
4.2.4.2 Additional Document Parts
The test suite might build additional document parts. To this end it must create new make file fragments defining
the necessary targets (see section 4.4 for details). The name of the targets must be added to the make variable
TFW_SC_HTML_DEPS.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Artifact Setup in XML 31
1 <?xml version="1.0" encoding="US-ASCII" ?>
2 <tcm:master_document
3 xmlns:tcm="http://www.sysgo.com/tcm"
4 xmlns="http://www.w3.org/1999/xhtml"
5 brief="Structural coverage results"
6 author="..." rev="$Id: doc/tfwref/tfwref.tex 2019-06-12 13:44:08 +0200
c5b3f0d1d51c8540f631eddf7ecfc0103e628c04 $"
7 >
8
9 <tcm:group>
10 <tcm:file name="../sys/tr_labels_used.xml"/>
11 </tcm:group>
12
13 <tcm:fragment>
14 <h1 class="landscape">Structural Coverage Results for [...]</h1>
15 <p>This section provides the structural coverage results for [...]</p>
16 </tcm:fragment>
17 <tcm:group>
18 <tcm:fragment>
19 <h1>Expected Defects in Terms of Structural Coverage Results</h1>
20 <p> ... </p>
21 </tcm:fragment>
22 <tcm:file name="../sys/tr_coverage.xml"/>
23 </tcm:group>
24
25 <tcm:fragment>
26 <h1>Test Results for [...]</h1>
27 <p>This section contains the results of the test cases executed on
28 the instrumented version of [...]. These results are not the results
29 for the certified version. The results for the certified version are in
30 the corresponding Test Results document.</p>
31 </tcm:fragment>
32 <tcm:group>
33 <tcm:fragment>
34 <h1>Expected Defects Due to Structural Coverage Instrumentation</h1>
35 <p>This section summarizes the deviations of test results compared to the
36 non-instrumented version of [...].</p>
37 </tcm:fragment>
38 <tcm:file name="../sys/tr_tables_<shortcut>.xml"/>
39 </tcm:group>
40
41 <tcm:file name="../sys/system.xml"/>
42
43 </tcm:master_document>
Figure 7: Example for doc/sc.xml
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
32 Verification Result Artifacts
1 @abbrev{abb:TFW,
2 short = {TFW},
3 description = {Test Framework}
4 }
Figure 8: Example for how to Store Abbreviation Records in the File abbreviations.bib.
4.3 Artifact Setup in TEX
This section describes the TEX parts of the document creation process. It uses the HTML file created from XML
as described in the previous section as input. It is converted to a TEX input file using a XSLT script provided by the
generic overlay (see section 2 for a description of overlays). After this the PDF will be generated using pdflatex
and bibtex.
4.3.1 Managing Abbreviations and References with BibTeX
The test framework has limited support for bibtex, the tool commonly used for management of references in the
TEX world. When creating the PDF document the test framework checks if references are used in the document.
If so, it will call bibtex. Additionally, it automatically adds \refdoc{} calls to the document for any <refdoc/>
element it finds in the test cases of the test suite. With this in place it is possible to add management of references
within the generic overlay.
For bibtex two types of input files are required:
*.bib The bib files contain the actual references in a formal description. These files are read by bibtex when
searching for references. Usually there are two files of this type: vvdocs.bib, which contains references
to documents and abbreviations.bib, which contains a list of commonly used abbreviation.
*.bst The bst files are bibtex styles. They specify how references should be formatted. To this end they
use a stack based programming language, which outputs TEX code. The style used for verification result
documents is named vvdocs.bst.
The BibTeX input files are usually stored in the generic overlay in doc/bib/.
4.3.1.1 Abbreviations
Abbreviations are stored in the file abbreviations.bib. A single record has the form as shown in Figure 8
This reference is used by adding \abbrev{abb:TFW} to the document. It will add the shortcut “TFW” with the
long description “Test Framework”. With \noabbrev{abb:TFW} it will add the shortcut to the table of shortcuts,
but won’t output it at the current position.
4.3.1.2 References
References are stored in the file vvdocs.bib. There are multiple records for different kinds of documents. See
Figure 9 for an example. The type certstd is used for certification standards, the type projdoc is used for
project documents, and the type intdoc is used for internal documents. If necessary a manual reference to a
document is added by adding e. g. the command \cite{SVVM} to the TEX input file. This will add the document
to the list of referenced documents and add a link to it at the current position. With \nocite{SVVM} a reference
is added without outputting any text at the current position.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Available Makefile Hooks for Document Creation 33
1 @certstd{DO-178B,
2 title = {ED-12B / DO-178B Software Considerations in Airborne
3 Systems and Equipment Certification},
4 version = {01/12/1992}
5 }
6
7 @projdoc{RSTD,
8 id = {00004-0003-RSTD},
9 title = {SYSGO Software Requirements Standard},
10 version = {01}
11 }
12
13 @intdoc{SVVM,
14 id = {00004-1501-SVVM},
15 title = {SYSGO Software Verification and Validation Manual},
16 version = {01}
17 }
18
19 @refdoc{HLRQ,
20 id = {00101-1000-HLRQ},
21 title = {PikeOS High Level Requirements},
22 version = {1.4 / 05}
23 }
Figure 9: Example for Document References in the File vvdocs.bib.
4.4 Available Makefile Hooks for Document Creation
In this sub-section it is described how to hook into the document build process to e. g. create additional document
fragments. This is done by creating makefile fragments adding dependencies to the relevant targets used by the
test framework. The final document is created in three steps. First a normalized XML document is created using
XML fragments as input. Then this XML document is converted to a TeX document. Finally the TeX document is
processed to generate a PDF document.
An example for how to create an additional XML fragment to be included by the test case master document is
given in Figure 10. The file created in the example is named $(SYS_DIR)/preformated.xml. The input
used is doc/input.c. The script tools/mk_preformated_xml.pl is called to perform the actual trans-
formation. The important point to make here is that the name of the target defined is added to the variable
1 ifeq ($(TFW_INSTALLED),yes)
2
3 TFW_TC_HTML_DEPS += $(SYS_DIR)/preformated.xml
4 $(SYS_DIR)/preformated.xml: doc/input.c
5 $(call INFO,mk_preformated_xml.pl,$@)
6 $(call FILE_C,$@)
7 $(_V)tools/mk_preformated_xml.pl $^ > $@
8
9 endif
Figure 10: Makefile fragment creating a document fragment for the test case master document.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
34 Verification Result Artifacts
TC document TR document SC document
XML → XML TFW_TC_HTML_DEPS TFW_TR_HTML_DEPS TFW_SC_HTML_DEPS
XML → TEX TFW_TC_TEX_DEPS TFW_TR_TEX_DEPS TFW_SC_TEX_DEPS
TEX → PDF TFW_TC_PDF_DEPS TFW_TR_PDF_DEPS TFW_SC_PDF_DEPS
Table 1: Hooks available for adding additional targets to the document build process.
TFW_TC_HTML_DEPS. By this the target is added as a precondition to the test framework internal target convert-
ing the XML fragments to a single HTML document. This means with this configuration the created file can be
included in doc/tc.xml using the statement <tcm:file name="../sys/preformatted.xml"/>
In a similar manner it is possible to add additional dependencies to the targets used for building the TeX and the
PDF file. Table 1 lists the variables available. When working with these variables please make sure
• to add new targets to the variables by using +=. Otherwise the entire document build process might fail.
• to use a make file fragment with a name that (if sorted alphabetically) is sorted before strings starting with
the sub-string “50_”.
4.5 Changebars
It is supported to add changebars to PDF documents. For this the normalized *.xml file of the previous released
PDF document is needed. The test suite must be stored in GIT. To generated e. g. a changebar for the TC
document the parameter TFW_CHANGEBAR_TC must be set to a blobish referencing the normalized tc.xml of
the previous PDF document2 .
2
This value can be taken from the dictionary of the previous released PDF using this command: pdftk tc.pdf dump_data | grep
-A1 DocumentSource
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
5 Test Sets
A test set is a set of test cases that are compiled into the same binary. So all test cases of a test set are executed
in the same environment. For PikeOS this means e. g. with the same setup of resource partitions and permissions.
A test set is created by creating a sysconf.rc file in a subdirectory of the test suites set/ directory. To create
the test set named example the following commands are sufficient:
[ts_example] $ mkdir -p set/example
[ts_example] $ touch set/example/sysconf.rc
The sysconf.rc file is the only mandatory file in the test set directory. See Figure 11 for an overview of the
optional files:
description.xml The content of the description.xml file is automatically added to the test case master docu-
ment. See section 4.2.2.3 for details.
[*.]snip.xml This file is used to modify the XML files of the application and integration project in a test set
specific manner. It is mostly useful for test cases testing the PSSW. The file can be either snip.xml
(applied on integration and application project by respective tags) or <project>.snip.xml where it is
then applied only to the respective project in the Custom Pool. For the latter case the supported projects
are: int, app, and $PARTITION_PROJECTS.
sysconf.rc This file is used to configure which projects shall be used by the test set. For
this the variables TC_APPLICATION_PROJECT, TC_INTEGRATION_PROJECT, TC_KER-
NEL_FUSION_PROJECT, TC_PSSW_FUSION_PROJECT, TC_PSP_PROJECT, TC_DEVELOP-
MENT_PROJECTS, TC_KDEV_PROJECTS, TC_LIBRARY_PROJECTS, and TC_SE_PROJECTS, are
used. See also section 5.1 and 5.2. Additionally, this file allows to set arbitrary environment variables which
will be available during test set build.
tfwtags.xml This file is used to configure the test framework, e. g. enabling debug support. Usually it is not
needed.
<projname>.sh This file is executed around cloning and configuring the project identified by "projname". It
works like <projname>.snip.xml except it offers bigger versatility.
<projname>.conf This file works like <projname>.sh except it contains commands to the configure script and
is executed at the time <projname> project is cloned.
Note that there are several scripts for adjusting the cloned projects. The order of execution is following: <pro-
jname>.sh, <projname>.snip.xml, <projname>.conf. To achieve best versatility, the <projname>.snip.xml
and <projname>.sh files use configuration tags or are called with parameters identifying the stage of project
cloning process. There are three stages recognized at the moment: "pre-script", "pre" and "post". "pre-
script" stage is executed before <projname>.conf is applied, "pre" is executed after <projname.conf> and
post is executed after the project is configured (i.e. configuration files are created). These parameters are
to be used either as configuration tags of the <projname>.snip.xml (see used namespace definition) or in
case of <projname.sh> it comes in a form of input argument (example <projname.sh> –pre-script). All files
are applied/called inside the project directory.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
36 Test Sets
$TS
+ Makefile
+ config/
+ doc/
+ set/
+ example/
+ description.xml
+ int.snip.xml
+ sysconf.rc
+ tfwtags.xml
+ tc/
+ tools/
+ verify/
Figure 11: The content of the set directory.
5.1 Application and Integration Projects
For PikeOS test suites application and integration projects are needed for the build process. Here the term
“application project” refers to the development project used for building the test slave binary. These projects are
provided to the test suite in form of a “seed”. The seed of a project is its (possible minimal) form, suitable as input
for pikeos-cloneproject. Upon make prepare the test framework already puts the seed of the application
project “app” and the integration project “simple” into the test suites sys/seed directory. The test suite may
provide the seed of additional projects in the directory config/seed. This directory can also be used to override
the test framework provided seed, too.
To use these projects the following content must be added to the test sets sysconf.rc file:
1 TC_APPLICATION_PROJECT=app
2 TC_INTEGRATION_PROJECT=simple
After cloning the PikeOS configurator creates a project.mk file in the project directory. It contains target architecture
specific settings. If necessary test suite provided Makefiles may include this file. For this its location is made
public in the environment variable ${APPLICATION_PROJECT_MK}. The application project is cloned early in the
build process. After this the variable is available during the entire build.
5.1.1 Requirements on the Application Project
To be suitable as an application project a project must provide a makefile that includes the file
$(TFW_SLAVE_SYSTEM_DIR)/app.mk. Additionally the file $(ODIR)/testcases.a must be added to the list of
files to be linked to the test slave binary. Usually this is done in a makefile.defs file using the code shown in
Figure 12.
Here order is important. The value of MODLIST must be set before inclusion of the file app.mk, as this file will
(depending on what is build) add more objects to MODLIST or override it. The files added by app.mk to MODLIST
are mandatory for the test slave application.
The C code entry point (see Figure 13) provided by the application project must include the header tfw.h and call
the functions tfw_init() (see section 8.3.5.5) and tfw_slave() (see section 8.3.5.7). after initialization. The macro
TFW_SLAVE_DELAY is passed by the build system and specifies how many milliseconds the test slave shall wait
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Special PikeOS Projects 37
1 APPLIST = slaveApp.elf
2 MODLIST = entry
3
4 include $(TFW_SLAVE_SYSTEM_DIR)/app.mk
5
6 slaveApp.elf: $(addprefix $(ODIR)/, $(addsuffix .o, $(MODLIST)))
7 slaveApp.elf: $(ODIR)/testcases.a
Figure 12: Example for a makefile.defs file. Important is the inclusion of app.mk, the usage of MODLIST and the order.
1 #include <tfw.h>
2 #include <p4.h>
3
4 extern int main(void)
5 {
6 tfw_slave_cb_t cb;
7
8 p4_sleep(P4_MSEC(TFW_SLAVE_DELAY));
9
10 cb = TFW_SLAVE_CB_INITIALIZER;
11 cb.tfw_slave_read = ...;
12 cb.tfw_slave_write = ...;
13
14 (void)tfw_init(&cb);
15
16 tfw_slave();
17 }
Figure 13: Minimal example for the entry point of an application project. It only shows the mandatory usage of
TFW_SLAVE_DELAY, the mandatory fields of the tfw_slave_cb_t data structure and the calls tfw_init() (see section 8.3.5.5)
and tfw_slave() (see section 8.3.5.7).
before startup. This is important if it is run in setups in which other components of the system need a long time for
boot up.
See also Section A.5 for an example of how to implement the mutex callbacks for tfw_init().
5.1.2 Requirements on the Integration Project
The integration project must be set up to include the file, which the application project installed into the custom
pool, in the boot image. This application must be started automatically upon system boot.
5.2 Special PikeOS Projects
During test set build it is possible to build additional PikeOS projects. Supported are kernel driver projects, system
extension projects, PSP projects, and normal development projects. If a kernel driver or a PSP project is build the
test framework performs automatically a kernel fusion. Similarly, if a system extension is build a PSSW fusion is
performed. For all these activities a test set build specific custom pool is used.
Which projects to build is controlled by setting any of the following variables in the test sets sysconf.rc file:
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
38 Test Sets
TC_DEVELOPMENT_PROJECTS This variable is used to specify the seed of additional development projects
to be build and installed into the custom pool. Its value is a white space separated list of seed names. The
specified projects are build and installed after the test case was build.
TC_LIBRARY_PROJECTS Similar to TC_DEVELOPMENT_PROJECTS this variable is used to specify devel-
opment projects to build and install. The only difference is that projects specified using this variable are
ensured to be build and installed first (i.e. before the test case and any other project).
TC_KDEV_PROJECTS This variable is used to specify a list of kernel driver projects to be build and installed.
Kernel fusion is performed automatically. Projects specified with this variable are build after TC_LI-
BRARY_PROJECTS are installed and before the test case is build.
TC_PSP_PROJECT This variable is used to specify the PSP to build and install. Kernel fusion is performed
automatically. The project specified with this variable are build after TC_LIBRARY_PROJECTS are installed
and before the test case is build.
TC_SE_PROJECTS This variable is used to specify a list of system extension projects to be build and in-
stalled. PSSW fusion is performed automatically. Projects specified with this variable are build after TC_LI-
BRARY_PROJECTS are installed and before the test case is build.
Normally the test framework detects using the configured board as input which fusion project must be used.
If a different fusion project shall be used the project to use can be configured using the variable TC_KER-
NEL_FUSION_PROJECT (for the kernel fusion project) and TC_PSSW_FUSION_PROJECT (for the PSSW fu-
sion project).
Except for TC_PSP_PROJECT3 all variables described in this section allow to use a custom name for the cloned
project. By default the basename of the seed is used. But it is possible to rename it by appending the new name
to the name of the seed, separated by a colon. For example:
1 TC_DEVELOPMENT_PROJECTS="tools/example.app:app1 tools/other.app:app2"
The names “app”, “psp”, and “int” are reserved for the application, the psp project, and the integration project.
3
PSP projects are automatically renamed to "psp".
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
6 Test Cases
A test case, the most atomic unit of testing, consists of
• a test case description,
• a test procedure, which has a slave and master part, and
• various files controlling the build process.
Test cases specific to a test suite are placed in the directory tc/. Furthermore, test cases from test set pools can
be used. Since the semantics of those both kinds of test cases differ slightly, this section will focus on (traditional)
test cases. The differences to test cases from test set pools are discussed in section 3.
6.1 The Files in the Test Case Directory
This section describes the various files in the test case directory, Figure 14 shows a simple example test
suite. In the example the test case is named tc_example_01. The test case description is stored in the file
tc_example_01.xml. See section 6.1.1 for a detailed description. The test case procedure is provided in the
files tc_example_01.master.c (the test master part) and tc_example_02.slave.c (the test slave part).
As a special case the test framework supports creation of the test case description and procedure files using a
definition file as input. This feature is not covered by this document.
The files tc_cflags.rc, tg_cflags.rc, and ts_cflags.rc are used to control the build process.
The all target of the file Makefile stored in the test case directory is automatically called during test case build.
It can be used to build additional object files.
6.1.1 The Test Case XML Description File
The test case description is stored in an XML file. It describes the implemented test and explains why the test
tests the given requirements. It is maintained together with the other files belonging to the test case.
The test case description contains the following pieces of information:
• References to the documents and their baseline specifying the requirements
• References to the requirements being tested, including the information whether the requirement is partly or
fully covered by this test case and if the test case covers normal range testing, robustness testing or both
• Objectives of the test case
• A description part using a subset of HTML-based attributes to explain:
Setup and constraints
∗ Test setup (reference to a setup described in the Test Case Master document)
∗ Additional constraints on the test environment
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
40 Test Cases
$TS
+ Makefile
+ config/
+ doc/
+ set/
+ tc/
+ example/
+ 01/
+ Makefile
+ symbol-global.master.txt
+ symbol-global.slave.txt
+ tc_cflags.rc
+ tc_example_01.def
+ tc_example_01.master.c
+ tc_example_01.slave.c
+ tc_example_01.xml
+ tg_cflags.rc
+ ts_cflags.rc
+ tools/
+ verify/
Figure 14: The content of the tc directory.
Test design
Test Procedure
Test input(s) (e.g. equivalence class definition)
Each test case shall be written from scratch using templates. Figure 15 shows a typical template for a test case
description to be filled by the test case developer.
The XML document uses a multiplicity of elements. The root of the document is the <tc/> element. It has the
following attributes:
author The three letter shortcut of the test case author’s name
brief A brief description of the test case. It will be used in the test case master document immediately after
the initial headline for the test case. It should describe in one or two short sentences what the test case is
about.
xmlns The namespace(s) used in the XML file.
id The name of the test case.
rev The CVS revision tag of the test case. Usually the value is $Id$.
6.1.1.1 The <rqdocs/> and <rqdoc/> Elements
The requirements under test are defined in requirement documents, which are usually kept in DOORS modules.
For all requirements under test the requirement document must be specified in the element <rqdoc/> including a
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
The Files in the Test Case Directory 41
1 <?xml version="1.0" encoding="US-ASCII"?>
2 <tc xmlns:tc="http://www.sysgo.com/tc" xmlns="http://www.sysgo.com/tc"
3 brief="<Brief description of the test case>"
4 id="tc_example_01" author="..." rev="$Id: doc/tfwref/tfwref.tex 2019-06-12 13:44:08
+0200 c5b3f0d1d51c8540f631eddf7ecfc0103e628c04 $"
5 >
6
7 <rqdocs>
8 <rqdoc name="01234-1000-EXPL-IF" baseline="1.0" prefix="01234-IF_EXPL-"/>
9 </rqdocs>
10
11 <rqlist>
12 <rq id="01234-IF_EXPL-1234" coverage="F" type="N" method="T"
13 obj="EXAMPLE1 EXAMPLE2"/>
14 </rqlist>
15
16 <objectives>
17 <obj id="EXAMPLE1">description of objective EXAMPLE1</obj>
18 <obj id="EXAMPLE2">description of objective EXAMPLE2</obj>
19 </objectives>
20
21 <tc:description xmlns="http://www.w3.org/1999/xhtml">
22 <h1>Test Setup</h1> <p>...</p>
23 <h1>Design</h1>
24 <h2>Preparation</h2> <p>...</p>
25 <h2>Test</h2> <p>...</p>
26 <h2>Cleanup</h2> <p>...</p>
27 <h1>Equivalence Classes</h1> <p>...</p>
28 </tc:description>
29 </tc>
Figure 15: Example for a Test Case XML Description File
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
42 Test Cases
definition of the attributes baseline and prefix of the document. All elements of the type <rqdoc/> are grouped in
a <rqdocs/> element. If the test case is not used for requirement based testing the <rqdocs/> element may be
omitted.
6.1.1.2 The <rqlist/> and <rq/> Elements
The tested requirements are specified within the element <rqlist/>. Each requirement itself is described by a
<rq/> element inside the <rqlist/> element. The <rq/> element describes in detail how the test case covers the
requirement using the following attributes:
coverage The coverage attribute is used to specify if the test case covers the requirement fully or not. Valid
values are:
F The requirement is fully covered by the test case.
P The requirement is partially covered by the test case. There will be at least one other test case covering
other aspects of the requirement.
Partially test cases must be added to the coverage map file (coverage.csv) to specify if the partial tests
add up to a complete test or not. Also the <rq/> element must have text content explaining which part of
the requirement is tested.
id The id attribute specifies the identifier of the requirement. This could be a DOORS object Id.
method The method attribute is used to specify the test method. Valid values are:
T “Testing”: The test procedure is executed on the target in an automated way.
I “Inspection”: A (possibly automated) inspection is performed to check the test objective.
A “Analysis”: An analysis is provided by the test case.
obj The attribute obj specifies a list of objective identifiers (separated by white space) used by the test case to
test the requirement.
type The type of the test is specified using the type attribute. Valid values are:
N Normal range testing
R Robustness testing
N+R Normal range and robustness testing
The <rq/> element may contain text content with a short description. In case of partial coverage the text content
of the <rq/> element must describe which part of the requirement is actually covered (optional) and which part is
not covered (mandatory).
It the test case does not test requirements the <rqlist/> element may be omitted.
6.1.1.3 The <objectives/> and <obj/> Elements
The <objectives/> element contains a non-empty list of <obj/> elements. Each of them describes exactly one test
objective. The name of the test objective is given in the <obj/> elements id attribute. The test objective itself is
described in the text content of the <obj/> element.
If the test case does not test objectives the <objectives/> element may be omitted.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
The Files in the Test Case Directory 43
1 <tags xmlns="http://www.sysgo.com/tfwtags">
2 <tag name="msg_nonproto" value="1000"/>
3 <tag name="msg_timeout" value="30"/>
4 <tag name="slave_reset" value="TFW_TAG_SLAVE_RESET_BEFORE_TESTCASE
5 | TFW_TAG_SLAVE_RESET_AFTER_SUBTESTCASE"/>
6 </tags>
Figure 16: Example for Test Framework Tags
6.1.1.4 The <testenvlist/> and <testenv/> Elements
The <testenvlist/> element contains a nonempty set of <testenv/> elements. Each of them specifies the name
of a test set the test case belongs to. Additionally it may specify constrains and set additional attributes. This
information is used by the test framework to deduce the relationship between test cases and test sets.
The attributes of the <testenv/> element are:
build If this optional attribute is set to the value manual, the test case is not built by default. This is useful if
the build process requires additional steps as e. g. user interaction. The test case is built only, if the test
framework is instructed to use manual test cases4 . Enabling this option requires a manual run as explained
below.
label The value of this optional attribute is a colon separated list of test case labels. It is used to put constrains
of test set membership of the test case. Test case labels are usually defined in the test suite configuration
(config/testsuite.conf). When calculating the test set memberships the test framework considers
only the <testenv/> elements without label attribute or with a label attribute containing at least one of the
enabled labels.
run If this optional attribute is set to the value manual, the test case is not run by default. This is useful for test
runs that require user interaction. The test case is run only, if the test framework is instructed to use manual
test cases.
set The name of the test set the test case belongs to.
6.1.1.5 The <tags/> Element
After the <testenvlist> element there may be an optional <tags> element. It is not taken from the usual name
space used in the test case xml file, but from the namespace named http://www.sysgo.com/tfwtags (see Figure 16
for an example).
It can be used to configure different aspects of the test framework. In the example the maximum time the test
master waits for messages from the slave is set to 30 seconds (default is 10) and the maximum number of non-
protocol messages is set to 1000. See section 9 for a full list of tags supported.
6.1.1.6 The <description/> Element
The last remaining element of the document is the <description/> element. It contains a textual description of the
test case using an HTML subset. To ease readability its features should be used carefully.
4
This is done by either using the argument INCLUDE_MANUAL_TC or by specifying the test case name using the TC. See section 1.4
for details.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
44 Test Cases
1 #include <tfw.h>
2 #include <tc_example_01.h>
3
4 void tc_example_01(void)
5 {
6 ...
7 }
Figure 17: API of Test Case Master and Slave Procedure Files
The following HTML elements are supported:
• <h1/>, <h2/>, and <h3/> (headings)
• <p/> (paragraphs)
• <dl/>, <dt/>, <dd/> (descriptive lists)
• <ul/>, <li/> (bullet lists)
• <ol/>, <li/> (ordered lists)
• <br/> (forced line feed)
• <pre/> (preformated output)
The easiest way to assure proper name spaces is to set the default name space for the <description/> element to
HTML, like show in Figure 15. In the template a list of HTML sections which should be part of the description is
given. The XML schema used to validate the XML file checks if the general layout is as shown in the example.
The content of the <description/> element should not exceed the limit of 80 characters per line. This is especially
important for the contents of the preformated text elements as these are copied verbatim to the output document.
There must be no <pre/> element with content more then 80 characters wide. Tabulators must not be used.
To ease the review process no unnecessary changes to the format of the file should be done. This leads to large
differences with little value.
6.1.2 The Test Procedures
The test master procedure is provided by the file tc_example_01.master.c. It is compiled using the host tool
chain and linked with the test master provided by the test framework. The test slave procedure is provided by the
file tc_example_01.slave.c. It is compiled using the CDK for the target and linked with the test framework
library for the slave.
Both files use the same API. They must include the file <tfw.h> and implement a function named like the test case,
i. e. tc_example_01() in the example. Also the test case specific header file (in the example <tc_example_01.h>)
must be included. See Figure 17 for an example. This function will be called by the test framework to perform the
actual test.
A single test case must implement a test slave procedure, a test master procedure, or both of them. The following
use cases occur:
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
The Files in the Test Case Directory 45
Test slave procedure only This is the most common case. In this case the test case provides a test slave only.
It is either directly linked with the object under test (e. g. if a library is tested) or it communicates with the
object under test using some API (e. g. if the test slave runs in a PikeOS partition and tests an external file
provider via its public API).
In this case the test framework generates a default test master procedure which simply starts the slave and
processes all messages it emits.
Test master procedure only This case can be used if it is not necessary to perform action on the target system,
e. g. if a test case performs user interactions only. Another example is a test case that checks that the boot
process of the target is aborting with a certain error message.
Internally the test framework will still build a binary for the target, even if no test case of the test set has a
test slave procedure. But it will respond with an assertion if the test master tries to execute the test slave
procedure.
Test slave and test master procedure This case is useful to test communication channels. For example if a
serial driver is to be tested. In this case the slave procedure will connect to the serial driver to be tested
and the test master procedure will open the serial interface of the host PC. Both are connected via a serial
line. Now the test master can stimulate the driver and the slave can use the drivers API to send data to the
master.
To start the test case the test framework will call the test master procedure. The test master procedure
is responsible to start the test slave procedure using the function tfw_start() (see section 8.2.4.11) and to
process the messages sent by the slave using tfw_join() (see section 8.2.4.12).
All symbols in the test case procedure files are made local to the test case. They are not available from other
test cases belonging to the same test set nor to any library linked to the test set. If global symbols are needed a
symbol-global.*.txt file (see section 6.1.3) must be used.
6.1.3 The symbol-global.*.txt Files
Per default all symbols in the test case procedure files are made local to the test case. They cannot be accessed
from the outside. This is a reasonable default. If additional symbols are needed they should always be provided
in form of a library linked to the test set. This allows to build test sets partially (i.e. using only a subset of the test
cases of the test set as described in section 1.4). In special use cases it might still be necessary do export global
symbols from the test case. For this the symbol-global.*.txt files shall be used.
Supported are the files symbol-global.slave.txt and symbol-global.master.txt in the test case
directory. If they exist they are expected to contain a list of symbol names (one per line). These symbols are made
globally available. The content of the file symbol-global.slave.txt is used for the test slave, the content of
the file symbol-global.master.txt is used for the test master5
6.1.4 The t*_cflags.rc Files
Using the files ts_cflags.rc, tg_cflags.rc and tc_cflags.rc it is possible to control the build process.
These files are sourced by a sub-shell of the one performing the build. They may set various environment variables
which are then picked up by the build process. It is not possible to set arbitrary variables. Only the ones the test
5
The file is feed directly to objcopy. The test master can be compiled for Windows. In this case it might be necessary to mangle names
in the same manner as done by Compilers for Windows (usually an additional underscore is added as prefix on 32 Bit Windows systems).
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
46 Test Cases
framework knows about will be picked up. The files are sourced in the order they are listed in the following
paragraph.
ts_cflags.rc This file is always sourced. It is used to specify “test suite specific” flags. The variables usually
set here are TS_CLFAGS, TSM_CFLAGS, TS_LDFLAGS, and TSM_LDFLAGS and TS_MAKE.
tg_cflags.rc This file is sources when building a test case belonging to the same test group. I. e. there is only
one such file for tc_example_01 and tc_example_02. It will be used for both of them. The variables usually
set here are TG_CFLAGS and TGM_CFLAGS.
tc_cflags.rc This file resides in the test case directory. It is sources for building a single test case only. The
available variables are TC_CFLAGS and TCM_CFLAGS.
The files are sourced multiple times during test case build. The current test framework implementation sources
them three times during test case build. But this behavior is not required.
The variables are used in the following ways:
for the test slave procedure When compiling the test slave procedure and related files the test framework auto-
matically adds the value of
${TS_CFLAGS} ${TG_CFLAGS} ${TC_CFLAGS}
as options to the compiler call. When linking the test slave binary the test framework automatically adds the
value of
${TS_LDFLAGS}
as option to the linker call.
for the test master procedure When compiling the test master procedure and related files the test framework
automatically adds the value of
${TSM_CFLAGS} ${TGM_CFLAGS} ${TCM_CFLAGS}
as options to the compiler call. When linking the test master binary the test framework automatically adds
the value of
${TSM_LDFLAGS}
as option to the linker call.
as interface for additional components At one point during the build process ${TS_MAKE} is interpreted as a
colon separated list of directories. The test framework will chdir to each of them and call
[] $ make all
This allows building additional components.
In more complex setups the variables are usually not set directly. Instead a function applying the necessary
changes is implemented in ts_cflags.rc. This functions can then be called in tg_cflags.rc. A common use case is
usage of a test library, which needs TS_MAKE to build it, TS_CFLAGS to add its directory to the include path, and
TS_LDFLAGS to pass it to the linker. In this case a function use_libfoo() can be used to easily enable the library
for a test set.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
General Rules for Writing Test Procedures 47
6.1.5 The Makefile File
If there is a file named Makefile in the test case directory the test framework will automatically chdir to it and
call
[] $ make all
The test framework puts no limit on this Makefile, but it should follow the following rules:
• It should neither create nor modify files in the test case directory. Instead all work should be done in the
directory ${TC_BUILD_DIR}. If it is implemented this way the test framework will take care of cleanup, too.
• If the purpose of Makefile is to build a library for a PikeOS test slave it should start with the instructions
include $(APPLICATION_PROJECT_MK) include $(PIKEOS_TARGET_FILES)/scripts/pikeos.mk
or similar. This will pull in the project.mk file of the application project and the necessary rules to build a
*.o from a *.c file.
6.2 General Rules for Writing Test Procedures
The test procedure is the functional part of a test case. It shall be designed and implemented carefully and in
a precise manner. The code shall be written in a maintainable and readable way, conforming to the applicable
coding standard and style. This will ease future extensions of the test case.
Most important is that the source code must match the description of the test procedure given in the test case
description. The general rule is that somebody, who knows the basics about the tested object but never saw the
test case before, shall at most need a few minutes to be sure, that the test description matches the implementation.
It might happen that some extra code is necessary, which is not mentioned in the description. This is fine, as
long as there is a comment describing what this extra code is used for. An example is code used to guard
implementation specific assumptions. For example if a null-terminated list of values provided by the tested system
is used, it should be checked using TFW_ASSERT() that none of these values is equal to 0. It is not necessary to
describe this detail (and what kind of list is implemented) in the test description.
The following criteria should be met during test case writing:
• Use comments when the purpose is not obvious
• Avoid complexity, rather split into more test cases
• The testing of the objectives should be independent (result of one objective should not affect the testing of
another)
• Design tests, so that both ways of a condition are evaluated (important for coverage)
6.3 The Protocol Used by Master and Slave to Communicate
Before the test slave and master procedures are explained, the protocol used to communicate is introduced shortly.
It is implemented completely by the test framework library. But basic knowledge of it is necessary to understand
the design of test procedures.
There are two kinds of messages: protocol messages and non-protocol messages. A protocol message has a
type, a certain length, a checksum, a sequence number and an optional payload. A non-protocol message is
created by the test frameworks protocol module if it receives octets on its communication channel which do not
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
48 Test Cases
belong to a protocol message. These characters are returned unchanged as the payload of an artificial non-
protocol message.
6.3.1 Three-Way Handshake
The protocol is always initiated by a three-way handshake. This is initiated by the slave periodically sending a
SYN message. The master answers with a SYNACK. Finally the slave finishes the handshake with an ACK. By
comparing the build Ids it is ensured that test master and test slave procedure match.
After a successful handshake normal messages may be used. The general rule is that the master is in control and
the slave only acts on orders. So after the handshake the slave is in the waiting-for-commands state. In this state
the master may send a command. This will be processed by the slave. It will generate (possibly multiple) answer
messages.
6.3.2 Filtering Output
After the handshake the master may choose to set a message filter in the test slave. This is used to limit the
amount of traffic on the serial line. Supported are disabling of trace and print messages. This feature is exposed
to the user by command line options of the test master.
6.3.3 Running Test Cases
To start a test case the test master sends a message to the test slave, which must be in the waiting-for-message
state. Upon reception the test slave will start the requested test slave procedure. This test slave procedure will run
the actual test. During this it may send messages for recording test results and emit log messages to the master.
If the test slave procedure returns, a special end message is sent. The test master procedure is responsible for
processing these messages. This processing is not done automatically. Instead the test master procedure must
call one of the message processing functions.
6.4 Interactive Tests
The test framework provides functions enabling an test case to ask questions, to be answered interactively by an
operator. These are
• tfw_interact_get_cb() (see section 8.4.1.5),
• tfw_interact_get_nl() (see section 8.4.1.2),
• tfw_interact_get_uint() (see section 8.4.1.3),
• tfw_interact_get_yn() (see section 8.4.1.4), and
• tfw_interact_put_msgf() (see section 8.4.1.1).
The function tfw_interact_put_msgf() is used to output questions. The other functions return the answers. Usually
the test case blocks until the question is answered. If tests are run without an operator this is undesired. In this
case this can be disabled using the RUN_OPTS parameter:
[ts_example] $ make run SET=<testsetname> RUN_OPTS=-u
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Interactive Tests 49
With the RUN_OPTS parameter set to -u the test master aborts immediately if a test case calls any of the
tfw_interact_*() functions.
Sometimes test cases contain no functional parts but are used only to ask questions and document the answers.
As the operator might need a lot of time to answer it is a waste to execute these tests on real target devices. In
this case it makes sense to put them into a separate test set and execute it on the "UNIX" target. For this the
sysconf.rc file must contain the following content:
1 export TFW_SLAVE_TYPE=UNIX
This configures the test framework to build the test slave for the UNIX target. It is then executed on the local
system without blocking any real target device.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
7 Parameter Sets Defined by Equivalence
Classes
In order to test the behavior of functions that are not subject to state changes with parameters which do not
influence each other, the test case developer may employ “equivalence classes”. Instead of exhaustively testing
all possible combinations of feasible and infeasible function parameters, equivalence classes offer a way to gather
several parameter values into classes and perform tests using their representatives.
Without the approaches explained in the later subsections equivalence classes are only usable for stateless
functions with independent parameters. It is important to understand this constraint as otherwise the created
test cases will not fully test the software under test and the reached code coverage will be quite low.
7.1 Defining Equivalence Classes
The set of possible input data or parameter values for a certain function is usually quite large and rather unman-
ageable for tests (e. g. testing all values of a 32 Bit quantity would require 232 runs of the same test). To remedy
this problem and to simplify analysis, equivalence classes are used. This means that all values of a particular
function parameter which are supposed to produce the same result are gathered into one set – an equivalence
class. This concept is best explained using the function from Figure 18 as example. The function is supposed to
succeed, if p1 ∈ P1 , p2 ∈ P2 , and p3 ∈ P3 , and to fail in all other cases. This function has the nice property that
the return value does not depend on the actual value of the parameters. It only depends on the class membership
of the parameters. These sets of input parameter values are called equivalence classes.
Using mathematical notation this property is defined as follows: Given a set S and a relation R, which is an
equivalence relation (i. e. reflexive, symmetric, and transitive), the element e ∈ S and all elements of S which are
(according to R) in relation to e form an equivalence class of S :
[e] = {x ∈ S|x ∼R e}
If there is such a relation, the set S can be split into disjoint subsets, which are the equivalence classes of S .
The example function validate() has arguments of the type uint32, so the possible input range for each parameter
is U = n ∈ N0 |n < 232 . Therefore, the equivalence classes of the parameter p1 are P1 and P1 = U \ P1 .

The equivalence classes for the parameters p2 and p3 are defined in the same manner.
To fully test the function validate(), it is sufficient to combine the equivalence classes of the parameters and then
choose for each of these combinations a few representatives of the equivalence classes. The rules for choosing
representatives depend on the tested function and additional needs and constraints.
In some cases, it might be wise to split the equivalence classes into distinct subsets. For example, functions might
behave differently depending on the size and the alignment of passed buffers. So although, if doing black box
testing, a parameter set may seem to form an equivalence class, it might not be one. Also, there might be a good
reason to work only with continuous sets. In both cases the solution is to split.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Combining Equivalence Classes 51
Assume, there is a function
1 int validate(uint32_t p1, uint32_t p2, uint32_t p3);
This function takes three 32 Bit unsigned integers as arguments. It is supposed to return 1, if all parameters are
simultaneously feasible and 0 if at least one of them is infeasible. Assume, the ranges of feasible input values for
the parameters are defined as follows:
• p1 may assume a value from P1 = {x ∈ N |10 ≤ x ≤ 20} ∪ {100, 200},
• p2 may assume a value from P2 = {x ∈ N |30 ≤ x ≤ 50} ∪ {1000}, and
• p3 may assume a value from P3 = {0, 5, 8}.
Figure 18: Example Function Used for Equivalence Classes.
7.2 Combining Equivalence Classes
In order to specify a particular test case for function validate(), every parameter of the function must be assigned
a concrete value and the expected result must be specified. As indicated above, only representatives of equiva-
lence classes should be used, instead of exhaustively enumerating all possible values for the parameters. First,
the equivalence classes must be combined. Second, it must be determined for each equivalence class, if the pa-
rameter set it represents is a valid input or an error case for the function under test. In the example, Pi is valid
and Pi is invalid.
Product Rule Combine the valid equivalence classes of the first parameter with the valid equivalence classes of
all other parameters. In the example, it is the combination of P1 × P2 × P3 . The product operator is used in
the notation to describe that multiple elements of each of the sets will be used and combined with multiple
elements of the other sets.
Sum Rule It is in general not meaningful to combine invalid values of multiple parameters for a function, as it is
usually not defined what will be returned, if two input parameters are invalid.
Therefore, equivalence classes with invalid parameter values are handled as follows: For each such class
of a parameter, combine it with all valid classes of the other parameters. In the example, there are the
following combinations: P1 . P2 − P3 , P1 / P2 . P3 , and P1 − P2 / P3 . The used operators describe
for example in the first combination that all elements of P1 and a single element of P2 and P3 are to be
combined. In the second combination a single element of P1 is combined with all elements of P2 and a
single element of P3 . Often, this is sufficient as functions usually handle error cases first. But to fully cover
a function, it might be necessary to add more values.
So in total, function validate() requires four equivalence class combinations (P1 × P2 × P3 , P1 . P2 − P3 ,
P1 / P2 . P3 , and P1 − P2 / P3 ).
7.3 Choosing Representatives
Different strategies for choosing representatives from equivalence class combinations are possible. Examples are:
• Combine representatives of equivalence classes so that each representative occurs in at least one test
case.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
52 Parameter Sets Defined by Equivalence Classes
• Combine all representatives into test cases but select only those cases for testing that are indeed relevant
for the user.
• Combine all representatives into test cases but prefer those test cases that contain combinations of bound-
ary values.
But the most important is picking the boundary values of the equivalence classes. This approach is described in
the following subsection.
7.3.1 Boundary Value Analysis
The analysis of boundary values is a promising strategy for choosing representatives for equivalence classes
which are a subset of the natural numbers. It is based on the experience that some of the most common errors in
computations are “off by one” errors. These errors are caused by the fact that due to an implementation error, a
quantity is computed with an absolute error of 1.
To improve the chances that such kinds of errors are revealed by appropriate test cases, five representatives are
used per equivalence class. Given an equivalence class {x ∈ N |a ≤ x ≤ b}, not one representative is picked,
but five:
• the lower bound and the adjacent value: {a, a + 1},
• the upper bound and the adjacent value: {b − 1, b}, and
j k
a+b
• an arbitrary value in the “middle” of the class, for example 2 .
This strategy yields multiple representatives for each equivalence class defined and each of them must be com-
bined with others in the same manner as before. Accordingly, the number of tests to be performed will increase.
To keep the number of tests to be executed in a tractable range, only the following combinations of representatives
are performed:
• For equivalence class combinations which yield valid parameter sets, combine the to be chosen represen-
tatives of the equivalence classes with each other using the product rule.
• For equivalence class combinations which yield invalid parameter sets, pick the invalid class and combine
its representatives with one representative from each of the valid classes.
The table below lists the representatives of the example to be used as input for validate(). The given sets are to
be combined using the product rule. The sum rule is reflected by sets with only a single member.
The presented example must be tested by 172 parameter sets. As the classes in the example are not continuous,
the picking rule is applied for every continuous subset of the classes.
There are two other ways on how to choose representatives in a meaningful way using a variant of the rule given
above:
Split all classes into continuous subsets, combine these using the rules given in section 7.2 and select represen-
tatives for these combinations. This approach would use the same representatives as given above, but generate
a much larger list of parameter sets to test. For example, it would also generate parameters for the case that p1
is invalid and p3 has a value of 8. This approach should be taken, if one must assume (given the interface and
design requirements) that different code paths are executed in these cases and thus the combinations give above
are not enough.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Extending Equivalence Classes by Logical Parameters 53
The number of invalid parameters to test could be reduced. The rule given asks to take the border value and its
neighbor. For example in the case of p1, the value 10 is a border value. Larger values are valid, smaller values
are invalid. To test this and detect all off-by-one errors it would be sufficient to test the values {10, 11} in the valid
case (first row of the table) and the value 9 in the invalid case (second row of the table). The combinations given
in the table consider both, 10 and 9 as border value. Thus in the invalid case, also the value 8 (second row) is
tested. If time is an issue, one could in these cases divert from the rule given above and remove tests for values
like the 8 in the example.
If parameter tables are generated automatically, it is sufficient to specify the boundaries of the sets to be tested.
The test framework will generate automatically the parameter combinations as described in this section. See the
example in section 7.6 for how to use this feature.
7.4 Extending Equivalence Classes by Logical Parameters
In many cases, it is necessary to test functions with parameters which are not independent. Consider for example
the C-function
1 void memcpy(void *dest, const void *src, size_t n);
copying n bytes from src to dest. The function fails, if the memory areas specified by (src, n) and (dest, n) overlap.
Even if src and dest point to valid memory areas, their combination may still be invalid, depending on the value of
n and on the fact whether the specified regions overlap. Consequently, function memcpy() cannot be tested using
equivalence classes as described above.
Nevertheless, equivalence classes can be applied to a certain extent to functions with non-independent param-
eters. To overcome the problems mentioned above, we introduce logical parameters. These parameters do not
comprise of a single value but represent a set of values. Using this approach function memcpy() into
1 test_memcpy(log_param *param)
2 {
3 memcpy(param->dest, param->src, param->n);
4 }
This function now only has one parameter param and this parameter is in fact a triple of values: it encodes the
source and destination address as well as the number of bytes to be copied. The equivalence classes for param
are now no longer subsets of N , but are now subsets of N 3 . Having converted the function memcpy() into a
function that takes only one parameter, we can now determine equivalence classes for this single parameter and
apply the equivalence class testing approach described above.
Note that it is not necessary that all parameters of a function are combined into a logical parameter. Only the
parameters which depend on each other are to be combined. The others stay as they are. In the end, we have a
function with a set of independent parameters, which can be tested using equivalence classes as described in the
preceding sections.
7.5 Extending Equivalence Classes by Virtual Parameters
In operating systems, (most) system calls depend on the current state of the system and can thus not be tested
by means of equivalence classes. Further, it is usually rather difficult to test parameter ranges in state-dependent
tests.
To overcome the before-mentioned problems, the use of equivalence classes can be extended to functions that
depend on a certain system state. The idea is to make the system state itself a parameter. Assume testing a
simple function
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
54 Parameter Sets Defined by Equivalence Classes
1 int func(int param);
that takes an integer as argument and returns an integer as well. This function is supposed to produce different
outputs for the same parameter value, depending on the current state of the system. Moreover, the function may
also be supposed to change the system’s state depending on its parameter. To test this function via equivalence
classes, a wrapper function is defined as follows:
1 int func_wrapper(int param, state_t in, state_t *out)
2 {
3 int ret
4
5 system_set_state(in);
6 ret = func(param);
7 *out = system_get_state();
8
9 return ret;
10 }
The datatype state_t identifies the system states. The function func_wrapper() first puts the system into the state
indicated by in, calls function func() and finally queries the state of the system and stores it in the location pointed
to by out i. e., apart from the return value function func_wrapper() has another output argument, namely out.
The function func_wrapper() is now independent of the current system state and may therefore be tested by the
equivalence class technique described above.
7.6 Using Equivalence Classes in Test Cases
The Test Framework supports Equivalence Classes in that it, given a description in the test cases XML file,
automatically generates a parameter table for it. It also puts a nicely formatted version of the equivalence class
description into the artifacts generated for the certification authority. Thus, this is the preferred approach to feed
parameters to test cases.
7.6.1 The Parameter List
The representatives listed in Table 2 are well-suited to test the example function. The XML code fragment shown
in Figure 19 can be used to instruct the test framework to use these values.
The name and type attribute of the parameter elements define the data type (ANSI-C) of the parameter and
the name to be used. The range attribute of the equiclass element describes the class. It will be used in the
generated artifacts for the certification authority. The attribute valid is used to describe if the representatives from
this equivalence class are valid input or an error case for the tested function. If an equivalence class is valid, one
of the representatives must be marked as the default one. This is the representative which will be used, if this
class is going to be combined with another invalid class.
If a fragment (<equitable/>) as described above exists in the test case description (XML file), the test case specific
header file will contain a type like this:
1 typedef struct {
2 [...]
3 } tc_NAME_NUM_params_t;
This type is used to store the values of the parameter set, the parameters given in the XML snippet are listed.
Thus, the structure of the example will be:
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Using Equivalence Classes in Test Cases 55
1 <equitable xmlns="http://www.sysgo.com/eq">
2 <parameterlist>
3 <parameter name="p1" type="uint32_t">
4 <equiclass range="valid input" valid="true">
5 <rep>10</rep> <rep>11</rep>
6 <rep default="true">15</rep>
7 <rep>19</rep> <rep>20</rep>
8 <rep>100</rep> <rep>200</rep>
9 </equiclass>
10 <equiclass range="invalid input" valid="false">
11 <rep>0</rep> <rep>1</rep> <rep>5</rep> <rep>8</rep>
12 <rep>9</rep> <rep>21</rep> <rep>21</rep> <rep>22</rep>
13 <rep>50</rep> <rep>98</rep> <rep>99</rep> <rep>101</rep>
14 <rep>102</rep> <rep>150</rep> <rep>198</rep> <rep>198</rep>
15 <rep>199</rep> <rep>201</rep> <rep>231</rep> <rep>232-2</rep>
16 <rep>232-1</rep>
17 </equiclass>
18 </parameter>
19 <parameter name="p2" type="uint32_t">
20 <equiclass range="valid input" valid="true">
21 <rep>30</rep> <rep>31</rep>
22 <rep default="true">40</rep>
23 <rep>49</rep> <rep>50</rep>
24 <rep>1000</rep>
25 </equiclass>
26 <equiclass range="invalid input" valid="false">
27 <rep>0</rep> <rep>1</rep> <rep>15</rep> <rep>28</rep>
28 <rep>29</rep> <rep>51</rep> <rep>52</rep> <rep>500</rep>
29 <rep>998</rep> <rep>999</rep> <rep>1001</rep><rep>1002</rep>
30 <rep>231</rep> <rep>232-2</rep> <rep>232-1</rep>
31 </equiclass>
32 </parameter>
33 <parameter name="p3" type="uint32_t">
34 <equiclass range="valid input" valid="true">
35 <rep>0</rep>
36 <rep default="true">5</rep>
37 <rep>8</rep>
38 </equiclass>
39 <equiclass range="invalid input" valid="false">
40 <rep>1</rep> <rep>2</rep> <rep>3</rep> <rep>4</rep>
41 <rep>6</rep> <rep>9</rep> <rep>10</rep> <rep>2^31</rep>
42 <rep>2^32-2</rep> <rep>2^32-1</rep>
43 </equiclass>
44 </parameter>
45 </parameterlist>
46 </equitable>
Figure 19: Equivalence Class Definition for validate()
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
56 Parameter Sets Defined by Equivalence Classes
Combination p1 p2 p3 Count Valid?
P1 × P2 × P3 {10, 11, 15, 19, 20, {30, 31, 40, 49, 50, {0, 5, 8} 126 Valid
100, 200} 1000}
P1 . P2 − P3 {0, 1, 5, 8, 9, 21, 22, {40} {5} 20 Invalid
50, 98, 99, 101, 102,
150, 198, 199, 201,
202, 231 , 232 − 2,
232 − 1 }
P1 / P2 . P 3 {15} {0, 1, 15, 28, 29, 51, {5} 15 Invalid
52, 500, 998, 999,
1001, 1002, 231 ,
232 − 2, 232 − 1}
P1 − P2 . Pe {15} {40} {1, 2, 3, 4, 6, 7, 9, 10, 11 Invalid
231 , 232 −2, 232 −1}
Table 2: Combinations, Chosen Representatives, and Total Number of Tests
1 typedef struct {
2 [...]
3 uint32_t p1;
4 uint32_t p2;
5 uint32_t p3;
6 [...]
7 } tc_NAME_NUM_params_t;
Note that it is only guaranteed that the parameters defined in the XML file will be there. But there is no guarantee
on the order. Also the structure might have additional members. Pointers to a structure of this type will can be
retrieved by the test procedures of the test slave and test master by calling the function tfw_param(). The content
of the representatives elements <rep/> is copied verbatim to the generated C source file and used to initialize the
elements of this structure.
Logical Parameters are also supported. For this, a normal <parameter/> element is used, but in the name and
type attribute, the range attribute of all included <equiclass/> elements and the content of all <rep/> elements a
semicolon is used to separate different values. For example:
1 <parameter name="first;second" type="int;char">
2 <equiclass range="valid value for first;valid value for second"
3 valid="true">
4 <rep default="true">10;’a’</rep>
5 </equiclass>
6 </parameter>
7.6.2 The Result and Objective List
Often not only the parameters but also the expected result and the objectives being tested by a given parameter
set must be passed to the test procedures. For this, the elements <resultlist/> and <objectivelist/> can be used.
For the function validate(), the <resultlist/> can be defined as follows:
1 <resultlist>
2 <result name="retval" type="int">
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Using Equivalence Classes in Test Cases 57
3 <value default="valid">1</value>
4 <value default="invalid">0</value>
5 </result>
6 </resultlist>
This snippet must follow after the <parameterlist/>. If it is present, it will add a member named retval of type int to
the parameter structure. Its value is 1, if the parameter set is valid and 0 otherwise.
It is possible to control in a more fine grained way which result is expected. For this, groups are used. The
<equiclass/> elements may have a group attribute. Its value must be a single word. Then in the <result/> element,
it is possible to use value elements like this:
1 <value if="group1 group2 group3">5</value>
This defines that the value shall be 5, if the parameter combination contains representatives from classes of
groups named group1, group2, and group3. The parameter table generator first evaluates all <value/> elements
with an if attribute. The first one that matches is used. It is in the responsibility of the programmer to assure that
only a single if attribute matches. If there is no <value/> element with matching if attribute, the value element with
matching default attribute is taken.
The <objectivelist/> element works like the <resultlist/> element. The only difference is that it gets a different
heading in the artifacts generated for the certification authority. The <objectivelist/> element must be the last
element in the <equitable/> element. An example is:
1 <objectivelist>
2 <objective name="obj" type="const char*">
3 <value default="valid">"OBJ_NORMAL_CASE"</value>
4 <value default="invalid">"OBJ_ERROR_CASE"</value>
5 </objective>
6 </objectivelist>
For an example of the C-code interface of these elements, see Appendix A.
7.6.3 The for Attribute
The elements <parameter>, <result>, and <objective> may have a for attribute. Its value controls if this parameter,
result, or objective shall be put in the slaves or the masters parameter table. The possible values are:
slave this element is to be put into the slaves table
master this element is to be put into the masters table
both this element is to be put into both tables
If there is no for attribute its value defaults to "slave".
7.6.4 Including Header Files
Sometimes, it is necessary to include additional header files in the c-file generated from the parameter table. This
is implemented using the <includes> element. If used, it must be the first element in the <equitable> element. A
full example is:
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
58 Parameter Sets Defined by Equivalence Classes
1 <includes>
2 <slave_type>
3 <file type="system">types.h</file>
4 </slave_type>
5 <slave_table>
6 <file type="local">local_data.h</file>
7 </slave_table>
8 <master_type>
9 <file type="system">host_types.h</file>
10 <master_type>
11 <master_table>
12 <file type="local">host_data.h</file>
13 </master_table>
14 </includes>
The <includes> element may contain elements of the type <slave_type>, <slave_table>, <master_type>, or <mas-
ter_table>. These elements contain <file> elements, which describe a header file to include each. The optional
type attribute specifies if angle brackets (if the value is "system") or double quotes (if the value is "local") shall be
used in the generated include statement. If no type attribute is given, its value defaults to "system".
The generated c-file consists of two major parts. One part for the test master and one part for the test slave.
These are divided into a part which is available at the time the data structure used for storing the parameter set
is defined and one part which is used to initialize the parameter table. The elements in <includes> corresponds
to these four parts. For example the include files from the <slave_table> element are available in the part of the
c-file initializing the slave part of the parameter table. Of course, files included in the type-section of the c-file are
also available in the table-section.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
8 Test Framework API Reference
This section presents all constants, data types, and functions available.
8.1 Common Master/Slave API
In this section the API available to test master and test slave is described.
8.1.1 Defines
TFW_PASS (OBJ, ...)
Record a PASS result for the given objectives.
Parameters:
OBJ The objectives for which to record a test result.
TFW_FAIL (OBJ, ...)
Record a FAIL result for the given objectives.
Parameters:
OBJ The objectives for which to record a test result.
TFW_UNSUPPORTED (OBJ, ...)
Record an UNSUPPORTED result for the given objectives.
Parameters:
OBJ The objectives for which to record a test result.
TFW_UNTESTED (OBJ, ...)
Record an UNTESTED result for the given objectives.
Parameters:
OBJ The objectives for which to record a test result.
TFW_EVAL (C, OBJ, ...)
If C evaluates to a true value record a PASS result for the given objectives. Otherwise record a FAIL
result.
Parameters:
C The condition to evaluate.
OBJ The objectives for which to record a test result.
TFW_EVALF (C, OBJ, FMT, ...)
If C evaluates to a true value record a PASS result for the given objective. Otherwise record a FAIL
result and output the message specified using the trace record type.
Parameters:
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
60 Test Framework API Reference
C The condition to evaluate.
OBJ The objective for which to record a test result.
FMT The format string to be used in case C evaluates to false.
... Variadic arguments needed by the format string.
TFW_INSPECT (OBJ)
Perform an Automatic Inspection.
Parameters:
OBJ The objective for which to perform the inspection.
Returns:
TFW_E_OK On success.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called
yet.
TFW_E_NOENT If no automated inspection result was recorded for objective OBJ.
TFW_REFRQ (OBJ)
Evaluate Result of Referenced Requirement Results.
Parameters:
OBJ The objective for which to get the referenced result.
Returns:
TFW_E_OK On success.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called
yet.
TFW_E_NOENT If no reference result was recorded for objective OBJ.
TFW_ASSERT (C)
Abort test run if argument does not evaluate to a true value.
Parameters:
C The condition to evaluate.
TFW_ASSERTF (C, FMT, ...)
Abort test run if first argument does not evaluate to a true value.
Parameters:
C The condition to evaluate.
FMT The format string to be used in case C evaluates to false.
... Variadic arguments needed by the format string.
TFW_DEBUGF (FMT, ...)
Allow printing a printf-like debug message via tfw_debugf(), if TFW_ENABLE_DEBUG is true.
Note:
If TFW_ENABLE_DEBUG is false, only the parameters are expanded to avoid unused variable warn-
ings.
Parameters:
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Common Master/Slave API 61
FMT The format string to be used.
... Variadic arguments needed by the format string.
TFW_COVERAGE_SEND
Used to instruct functions of type tfw_slave_coverage_t to send coverage data.
TFW_COVERAGE_RESET
Used to instruct functions of type tfw_slave_coverage_t to reset coverage data.
TFW_COVERAGE_SUBTCSTARTING
Used to signal to functions of type tfw_slave_coverage_t that a new sub test case is started.
TFW_COVERAGE_SUBTCFINISHED
Used to signal to functions of type tfw_slave_coverage_t that a sub test case is finished.
TFW_COVERAGE_TCSTARTING
Used to signal to functions of type tfw_slave_coverage_t that a new test case is started.
TFW_COVERAGE_TCFINISHED
Used to signal to functions of type tfw_slave_coverage_t that a test case is finished.
TFW_COVERAGE_CONNECTED
Used to signal to functions of type tfw_slave_coverage_t that the connection between test master and
test slave was successfully initiated.
TFW_COVERAGE_DISCONNECTING
Used to signal to functions of type tfw_slave_coverage_t that the connection between test master and
test slave will be disconnected.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
62 Test Framework API Reference
8.1.2 Enumerations
Enumeration type tfw_e_t
This enumeration contains the error codes returned by test framework functions. Except for TFW_E_OK (which
has the value zero) all error codes have a negative value.
Name Description
TFW_E_OK This value is returned upon success.
TFW_E_INTR This value is not used by the test framework. It can be used in user
supplied callback functions in the test master to make tfw_wait() (see
section 8.1.3.13) return.
TFW_E_RETURN This value is not used by the test framework. It can be used in user
supplied callback functions in the test master to make tfw_wait() (see
section 8.1.3.13) return.
TFW_E_TIMEOUT This value is returned if the connection timed out.
TFW_E_INVAL This value is returned if an invalid argument was supplied.
TFW_E_NOENT This value is returned if the specified argument does not exist.
TFW_E_NOMEM This value is returned in out of memory conditions.
TFW_E_IO This value is returned in case of I/O error.
TFW_E_EOF This value is returned by functions reading data if the end of file is en-
countered.
TFW_E_PROTO This value is returned if a protocol error is encountered.
TFW_E_UNSUPPORTED This value is returned if the the requested feature is not supported in the
current environment.
TFW_E_RECURSIVE This value is returned if a function is called recursively or by two threads
at the same time, which does not support this.
TFW_E_ASSERT This value is returned if an assertion failed.
TFW_E_INIT The function tfw_init() (see section 8.3.5.5) wasn’t called
TFW_E_STATE The function may not be called in the current connection state.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Common Master/Slave API 63
Enumeration type tfw_name_t
Arguments to the tfw_name() (see section 8.1.3.5) function.
Name Description
TFW_NAME_TESTCASE_FQ Used to request the fully qualified test case name.
TFW_NAME_TESTCASE Used to request the short test case name.
TFW_NAME_TESTSET Used to request the test set name.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
64 Test Framework API Reference
8.1.3 Functions
8.1.3.1 tfw_assertf
Record a failed assertion and abort test run.
Synopsis:
1 void tfw_assertf(const char *fmt,
2 ...)
Parameters:
fmt The format string describing the variadic parameters.
Description:
This function takes a format string as argument and writes it as assertion to the result log. Then the test run is
aborted. If no user specified assertion message is needed it is suggested to use the macro TFW_ASSERT() (see
section 8.1.1) instead.
Note:
The behavior of this function is unspecified if it is called in the test slave before calling tfw_init() (see section
8.3.5.5).
Returns:
This function does not return.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Common Master/Slave API 65
8.1.3.2 tfw_coverage
Request transmission of structural coverage data to the test master.
Synopsis:
1 tfw_e_t tfw_coverage(unsigned int opt)
Parameters:
opt Specifies the action to perform. Its value is the binary OR of TFW_COVERAGE_* values.
Description:
This function requests transmission of structural coverage data to the test master. Its argument is passed through
directly to the structural coverage callback registered in the test slave.
In the test slave this function simply sends out coverage data with the help of the callback. In the test master it
sends a request to the test slave to request sending structural coverage data. Upon reception the test slave sends
out coverage data. The test master switches into protocol processing mode to process the data sent by the slave.
If the test operator did not request transmission of structural coverage data the call back is not called. In this
case TFW_E_UNSUPPORTED is returned. If no structural coverage callback was registered by the test slave
TFW_E_UNSUPPORTED is returned, too.
In the test slave this function guards the structural coverage callback against recursive and concurrent usage. If it
detects this misbehavior it returns TFW_E_RECURSIVE.
In the test master this function may additionally return any value that can be returned by tfw_wait() (see section
8.1.3.13). This function does not support reentering in the way as tfw_wait() (see section 8.1.3.13) does.
Returns:
TFW_E_OK On success.
TFW_E_UNSUPPORTED The test operator did not enable structural coverage data transmission.
TFW_E_UNSUPPORTED The test slave did not register a structural coverage callback.
TFW_E_RECURSIVE This value is returned in the test slave if tfw_coverage() (see section 8.1.3.2) is either
called recursively or by two threads concurrently.
TFW_E_STATE This value is returned in the test master if the connection to the test slave is not established.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
TFW_E_* In the master this function may return any error code tfw_wait() (see section 8.1.3.13) might return.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
66 Test Framework API Reference
8.1.3.3 tfw_debugf
Write a Debug Record to the Result Log.
Synopsis:
1 tfw_e_t tfw_debugf(const char *fmt,
2 ...)
Parameters:
fmt The format string describing the variadic parameters.
Description:
This function writes a formatted message to the result log. It behaves mostly like printf(). In the test master the
printf() implementation of the libc is used in for formatting the message. So all of its features are available. In the
test slave a private implementation is used, which supports only a subset of the common features. The macro
TFW_DEBUGF is the preferred way of use.
Note:
Using the command line option -D it is possible to enable writing of messages with tfw_debugf() (see section
8.1.3.3).
Returns:
TFW_E_OK Upon success.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
TFW_E_* In the test slave error codes from the link layer are passed through.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Common Master/Slave API 67
8.1.3.4 tfw_dump
Output a hexadecimal dump of a binary object.
Synopsis:
1 tfw_e_t tfw_dump(const void *buffer,
2 size_t len)
Parameters:
buffer The address of the object to be output.
len The size of the object to be output.
Description:
This function takes as argument location and size of a binary object and writes it as a hexdump to the result log.
Returns:
TFW_E_OK On success.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
TFW_E_* In the test slave error codes from the link layer are passed through.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
68 Test Framework API Reference
8.1.3.5 tfw_name
Return name of a component.
Synopsis:
1 tfw_e_t tfw_name(tfw_name_t id,
2 const char **name)
Parameters:
id Test object identifier
name [OUT Upon success a pointer to the requested name is stored in name.
Description:
This function returns in its output parameter the name of the component identified by the input argument. It can
be used to retrieve e.g. the name of the current test case.
Returns:
TFW_E_OK On success.
TFW_E_INVAL If the argument provided is invalid.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
TFW_E_NOENT If the requested name is not available.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Common Master/Slave API 69
8.1.3.6 tfw_strerror
Return a string description of an error code.
Synopsis:
1 const char* tfw_strerror(int err)
Parameters:
err The error code to lookup.
Description:
This function takes as argument a TFW error code and returns a string describing int. In contrast to usual strerror()-
like implementation it is thread-safe.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
70 Test Framework API Reference
8.1.3.7 tfw_param
Return parameters of the current running sub test case.
Synopsis:
1 const void* tfw_param(void)
Description:
If parameter tables are supported and used by the test case this function is used to get a pointer to a data
structure with the actual parameter values. The return value must be casted to the correct data type, e.g. pointer
to tc_example_01_params_t, if the test case is named tc_example_01. If the test case does not use a parameter
table or the installation of the test framework does not support this feature, the function will return NULL.
Note:
The behavior of this function is unspecified if it is called outside of a test case (e.g. in the entry.c file of a test
slave).
Returns:
A pointer to the data structure containing the parameters if parameters are used. Otherwise it returns NULL.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Common Master/Slave API 71
8.1.3.8 tfw_printf
Write a formatted message to the result log.
Synopsis:
1 tfw_e_t tfw_printf(const char *fmt,
2 ...)
Parameters:
fmt The format string describing the variadic parameters.
Description:
This function writes a formatted message to the result log. In the test master the printf() from libc is used. In the
test slave a custom implementation supporting only a subset of the features is used.
Note:
Using a command line option it is possible to disable writing of messages with tfw_printf() (see section 8.1.3.8).
Returns:
TFW_E_OK Upon success.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
TFW_E_* In the test slave error codes from the link layer are passed through.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
72 Test Framework API Reference
8.1.3.9 tfw_sendf
Send Integer Data to the Peer.
Synopsis:
1 tfw_e_t tfw_sendf(const char *fmt,
2 ...)
Parameters:
fmt The format string describing the variadic parameters.
Description:
This function can be used to send data to the peer using a printf()-like interface. The peer must use the function
tfw_recvf() (see section 8.1.3.11) with a matching format string fmt to receive the data sent.
TFW_E_OK Upon success.
TFW_E_INVAL If an invalid format string is provided.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
TFW_E_NOMEM The internal buffer is too small for the request.
TFW_E_STATE This value is returned in the test master if the connection to the test slave is not established.
TFW_E_* Error codes from the link layer are passed through.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Common Master/Slave API 73
8.1.3.10 tfw_sync
Send synchronization message to Peer.
Synopsis:
1 tfw_e_t tfw_sync(void)
Description:
This function sends a synchronization message to its peer. The peer must wait for the message to arrive using
the function tfw_wait() (see section 8.1.3.13).
Returns:
TFW_E_OK On success.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
TFW_E_STATE This value is returned in the test master if the connection to the test slave is not established.
TFW_E_* Error codes from the link layer are passed through.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
74 Test Framework API Reference
8.1.3.11 tfw_recvf
Receive Integer Data from the Peer.
Synopsis:
1 tfw_e_t tfw_recvf(const char *fmt,
2 ...)
Parameters:
fmt The format string describing the variadic parameters.
Description:
This function can be used to receive data from the peer using a scanf()-like interface. The peer must use the
function tfw_sendf() (see section 8.1.3.9) with a matching format string fmt to send the data.
In the test master tfw_recvf() (see section 8.1.3.11) is implemented as normal message processing function. This
means that the callbacks registered with tfw_register() (see section 8.2.4.10) are called. Also the function supports
re-entry in the same manner as tfw_wait() (see section 8.1.3.13) and tfw_join() (see section 8.2.4.12).
TFW_E_OK Upon success.
TFW_E_INVAL If an invalid format string is provided.
TFW_E_INVAL If the received message does not match the format string.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
TFW_E_TIMEOUT This value is returned in the test master if no message from the peer was received for the
time specified in the applicable TFW tag.
TFW_E_PROTO If an unexpected protocol message was received.
TFW_E_STATE This value is returned in the test master if the connection to the test slave is not established.
TFW_E_* If termination was caused by a callback function returning an error code different from TFW_E_OK
it is passed through to the caller as error code of tfw_wait() (see section 8.1.3.13) (master only).
TFW_E_* In the test master error codes from the link layer are passed through.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Common Master/Slave API 75
8.1.3.12 tfw_tracef
Write a Trace Record to the Result Log.
Synopsis:
1 tfw_e_t tfw_tracef(const char *fmt,
2 ...)
Parameters:
fmt The format string describing the variadic parameters.
Description:
This function writes a formatted message to the result log. It behaves mostly like printf(). In the test master the
printf() implementation of the libc is used in for formatting the message. So all of its features are available. In the
test slave a private implementation is used, which supports only a subset of the common features.
Note:
Using a command line option it is possible to disable writing of messages with tfw_tracef() (see section 8.1.3.12).
Returns:
TFW_E_OK Upon success.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
TFW_E_* In the test slave error codes from the link layer are passed through.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
76 Test Framework API Reference
8.1.3.13 tfw_wait
Wait for Synchronization Message from Peer.
Synopsis:
1 tfw_e_t tfw_wait(void)
Description:
This function waits until a synchronization message is received from the peer or an error occurs. Additionally, it
terminates if a callback function registered by tfw_register() (see section 8.2.4.10) returns a value different from
TFW_E_OK.
After processing the event that caused tfw_wait() (see section 8.1.3.13) to terminate it can be called again to
continue processing.
In the test master it is possible to use this function as a generic function for non-proto message processing, even
before the connection to the test slave was initiated with tfw_connect() (see section 8.2.4.8) or after it was closed
with tfw_disconnect() (see section 8.2.4.9). For this callback functions processing the non-proto messages can be
registered with tfw_register() (see section 8.2.4.10). Then this function will process messages until either one of
the callbacks returns a value different from TFW_E_OK or a timeout occurs.
Note:
This function passes through the return values of the callback functions registered with tfw_register() (see section
8.2.4.10).
Returns:
TFW_E_OK If termination was caused by reception of a synchronization event message.
TFW_E_TIMEOUT If no message from the peer was received for the time specified in the applicable TFW tag
(master only).
TFW_E_PROTO If an unexpected protocol message was received.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
TFW_E_* If termination was caused by a callback function returning an error code different from TFW_E_OK
it is passed through to the caller as error code of tfw_wait() (see section 8.1.3.13) (master only).
TFW_E_* In the test master error codes from the link layer are passed through.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Common Master/Slave API 77
8.1.3.14 tfw_warnf
Output a warning message.
Synopsis:
1 tfw_e_t tfw_warnf(const char *fmt,
2 ...)
Parameters:
fmt The format string.
Description:
This function writes a warning record to the result file. It is printf-like with fmt being the format string.
Returns:
TFW_E_OK On success.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
TFW_E_* In the test slave error codes from the link layer are passed through.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
78 Test Framework API Reference
8.1.3.15 tfw_reset
Reset the Test Slave.
Synopsis:
1 void tfw_reset(void)
Description:
This function resets the test slave. For this different means are used. If the connection between test master and
test slave is established, protocol messages are used to call the tfw_slave_reset callback registered via tfw_init()
(see section 8.3.5.5). In this case coverage data is retrieved before actually performing the reset. If the connection
is not established a hard reset of the target is used.
If this function is called in the test slave, the reset is not performed immediately. Instead a message is sent to the
master requesting it to perform the reset procedure. The only message processing function able to process this
message is tfw_join() (see section 8.2.4.12). If another message processing function receives this message it will
return TFW_E_PROTO or it will abort the test run immediately with an error message.
If this function is called in the test master, the test slave is, when this function returned, in the same state as after
initial start-up of the test master. This means, if the test set is configured to call tfw_connect() (see section 8.2.4.8)
automatically the connection to the test slave is already established. If it is configured not to call tfw_connect()
(see section 8.2.4.8) automatically, the connection is not established and the test master procedure is responsible
to do so.
Note:
The behavior of this function is unspecified if it is called in the test slave before calling tfw_init() (see section
8.3.5.5).
Returns:
In the test slave this function does not return. In the test master it returns either successfully or not at all.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Master Only API 79
8.2 Master Only API
In this section the API available to the test master only is described.
8.2.1 Structure Definitions
8.2.1.1 struct tfw_np_cb_t
Data structure used to register callbacks.
The function tfw_register() (see section 8.2.4.10) allows to register callbacks for processing non-protocol data
received. For this it uses a variable of type tfw_np_cb_t. It is a typedef of a structure that contains at least the
members tfw_np_char and tfw_np_line. In the future more callback might be added. To allow to write future-proof
code the macro TFW_NP_CB_INITIALIZER is provided.
Note:
The structure may contain more members than described here.
Synopsis:
1 struct tfw_np_cb_t {
2 tfw_np_char_t * tfw_np_char;
3 tfw_np_line_t * tfw_np_line;
4 };
Structure Element Description:
tfw_np_char per character callback
tfw_np_line per line callback
8.2.2 Defines
TFW_CHANNEL_CONSOLE
Constant referring to the "console" channel.
TFW_CHANNEL_PROTOCOL
Constant referring to the "protocol" channel.
TFW_NP_CB_INITIALIZER
Initialize a variable of type tfw_np_cb_t.
TFW_TAG_SLAVE_RESET_BEFORE_TESTCASE
Value for the "slave_reset" tag. It is used to configure reset of the test slave before test case execution.
TFW_TAG_SLAVE_RESET_AFTER_TESTCASE
Value for the "slave_reset" tag. It is used to configure reset of the test slave after test case execution.
TFW_TAG_SLAVE_RESET_BEFORE_SUBTESTCASE
Value for the "slave_reset" tag. It is used to configure reset of the test slave before subtest case
execution.
TFW_TAG_SLAVE_RESET_AFTER_SUBTESTCASE
Value for the "slave_reset" tag. It is used to configure reset of the test slave after subtest case execution.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
80 Test Framework API Reference
8.2.3 Function Type Definitions
8.2.3.1 tfw_np_char_t
This is the prototype for functions processing non-protocol data one character at a time.
Synopsis:
1 typedef tfw_e_t tfw_np_char_t(int channel,
2 int c)
Description:
The function tfw_register() (see section 8.2.4.10) allows to register functions to be called by tfw_wait() (see section
8.1.3.13) and similar message processing functions if non-protocol data is received. This is the prototype for
functions to be called upon reception of a single non-protocol character.
Functions of this type have two arguments: channel and c. In channel the id of the channel on which the non-
protocol character was received is passed. The value of the character itself is passed in c as an unsigned char
casted to int.
The callback is also called if an error (e.g. read timeout) occurred. In this case the error code is passed in c. All
error codes in use have a negative value.
Returns:
TFW_E_OK On success.
TFW_E_* To interrupt message processing. The function tfw_wait() (see section 8.1.3.13) will return with this
value as return value.
8.2.3.2 tfw_np_line_t
This is the prototype for functions processing non-protocol data one line at a time.
Synopsis:
1 typedef tfw_e_t tfw_np_line_t(int channel,
2 const char *s)
Description:
The function tfw_register() (see section 8.2.4.10) allows to register functions to be called by tfw_wait() (see section
8.1.3.13) and similar message processing functions if non-protocol data is received. This is the prototype for
functions to be called upon reception of full line of non-protocol characters.
Function of this type have two arguments: channel and s. In channel the id of the channel on which the line of
non-protocol characters was received is passed. A pointer to the line itself is passed in s. This string is always
null-terminated. In the normal case it also contains the line break character ("\n"). If either an error occurred (e.g.
read timeout) or a 0x00 character was read from the channel, the string does not contain the line break character.
In contrast to the callback tfw_np_char_t the type of and reason for error is not passed to this callback.
Note:
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Master Only API 81
A corner case is if tfw_register() (see section 8.2.4.10) is called to remove this callback. In this case it is also
called with an incomplete line.
Returns:
TFW_E_OK On success.
TFW_E_* To interrupt message processing. The function tfw_wait() (see section 8.1.3.13) will return with this
value as return value.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
82 Test Framework API Reference
8.2.4 Functions
8.2.4.1 tfw_bootmsg
Get a buffer containing a copy of the boot messages.
Synopsis:
1 tfw_e_t tfw_bootmsg(char **msg,
2 size_t *len)
Parameters:
msg Upon success the address of a buffer containing the boot message is saved in msg.
len Upon success the length of the boot message is saved in len.
Description:
This functions returns in its out parameters a buffer containing the boot messages output by the target. Boot
messages are the non-protocol messages received whenever the connection to the slave is not up.
Note:
The buffer returned must be free()d by the caller.
Returns:
TFW_E_OK On success.
TFW_E_INVAL Provided arguments are invalid.
TFW_E_NOMEM Output buffer couldn’t be allocated.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Master Only API 83
8.2.4.2 tfw_channel_id
Get the id of a channel.
Synopsis:
1 tfw_e_t tfw_channel_id(const char *name,
2 int *channel)
Parameters:
name A channel name.
channel [OUT Upon success the channels Id is written to the output parameter channel.
Description:
Given the name of a channel this function returns its channel id in the output parameter channel.
TFW_E_OK Upon success.
TFW_E_INVAL If an invalid argument is provided.
TFW_E_NOENT If no channel with the supplied name exists.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
84 Test Framework API Reference
8.2.4.3 tfw_channel_name
Get the name of a channel.
Synopsis:
1 tfw_e_t tfw_channel_name(int channel,
2 const char **name)
Parameters:
channel The channel to act on.
name Upon success a pointer to channel name is stored in name.
Description:
Given a channel id this function returns in it’s out parameter a pointer to the channel name.
Returns:
TFW_E_OK Upon success.
TFW_E_NOENT If no channel with the supplied ID exists.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Master Only API 85
8.2.4.4 tfw_channel_pause
Pause thread reading from channel.
Synopsis:
1 tfw_e_t tfw_channel_pause(int channel,
2 int *fd)
Parameters:
channel The channel to act on.
fd Upon success the channels file descriptor is stored in the output parameter fd.
Description:
For each channel a thread is running reading characters from it and storing them into a buffer for retrieval by
tfw_channel_read() (see section 8.2.4.5). This function pauses this thread and returns the channels file descriptor
for use. The file descriptor must only be used while the thread is paused.
TFW_E_OK Upon success.
TFW_E_NOENT If no channel with the supplied ID exists.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
86 Test Framework API Reference
8.2.4.5 tfw_channel_read
Read from a channel.
Synopsis:
1 tfw_e_t tfw_channel_read(int channel,
2 time_t tout,
3 unsigned char *character)
Parameters:
channel The channel to act on.
tout Timeout to wait for a character to be received. A negative value means infinite timeout.
character Upon success the character read is stored in *character.
Description:
This functions reads a single character from the given channel. If no character is ready to be read it blocks for tout
seconds.
Returns:
TFW_E_OK If a character was received successfully.
TFW_E_EOF If the underlying functions report end-of-file. Usually this means that the peer closed the
connection.
TFW_E_IO If the underlying I/O functions report an error that is not mapped to any of the other return codes.
TFW_E_NOENT If no channel with the supplied ID exists.
TFW_E_TIMEOUT If no character was received within tout seconds.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Master Only API 87
8.2.4.6 tfw_channel_resume
Resume thread reading from channel.
Synopsis:
1 tfw_e_t tfw_channel_resume(int channel)
Parameters:
channel The channel to act on.
Description:
This function resumes the previously by tfw_channel_pause() (see section 8.2.4.4) paused thread. Before calling
this function the file descriptor returned by tfw_channel_pause() (see section 8.2.4.4) must be reconfigured to
exactly the state it had when it was returned by tfw_channel_pause() (see section 8.2.4.4).
TFW_E_OK Upon success.
TFW_E_NOENT If no channel with the supplied ID exists.
TFW_E_STATE If the channel with the provided ID is currently not paused.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
88 Test Framework API Reference
8.2.4.7 tfw_channel_write
Write to a channel.
Synopsis:
1 tfw_e_t tfw_channel_write(int channel,
2 const void *buffer,
3 size_t len)
Parameters:
channel The channel to act on.
buffer A pointer to the buffer to be written.
len The length of the buffer to be written.
Description:
This functions writes a buffer of data to the given channel.
Returns:
TFW_E_OK If the buffer was successfully written.
TFW_E_IO If a I/O error occurred.
TFW_E_NOENT If no channel with the supplied ID exists.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Master Only API 89
8.2.4.8 tfw_connect
Initialize the connection to the test slave.
Synopsis:
1 tfw_e_t tfw_connect(void)
Description:
This function initiates the connection between test master and test slave. When this function terminated success-
fully it is possible to start the test slave procedure and process it’s results.
With the function tfw_disconnect() (see section 8.2.4.9) an already established connection can be closed. To
reconnect a reset of the test slave with tfw_reset() (see section 8.1.3.15) is necessary.
Note:
This function passes through the return values of the callback functions registered with tfw_register() (see section
8.2.4.10).
Returns:
TFW_E_OK On success.
TFW_E_TIMEOUT If no message from the peer was received for the time specified in the applicable TFW
tag.
TFW_E_PROTO If an unexpected protocol message is received.
TFW_E_STATE This value is returned if the connection to the test slave is established or was established
before and tfw_reset() (see section 8.1.3.15) wasn’t called.
TFW_E_* If termination was caused by a callback function returning an error code different from TFW_E_OK
it is passed through to the caller.
TFW_E_* Error codes from the link layer are passed through.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
90 Test Framework API Reference
8.2.4.9 tfw_disconnect
Disconnect from test slave.
Synopsis:
1 tfw_e_t tfw_disconnect(void)
Description:
This function terminates the connection between test master and test slave. After calling this function it is no
longer possible to interact with the test slave using the normal test framework functions. It is only possible to
process non-protocol messages using tfw_register() (see section 8.2.4.10) and tfw_wait() (see section 8.1.3.13)
or perform a reset of the test slave using tfw_reset() (see section 8.1.3.15).
If a message processing function reports a connection error the caller is responsible to call tfw_disconnect() (see
section 8.2.4.9).
Returns:
TFW_E_OK On success.
TFW_E_STATE This value is returned if the connection to the test slave is not established.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Master Only API 91
8.2.4.10 tfw_register
Register callback functions processing non-protocol messages.
Synopsis:
1 tfw_e_t tfw_register(int channel,
2 tfw_np_cb_t *cb)
Parameters:
channel The channel to act on.
cb The callbacks to be registered.
Description:
This function registers callback functions for processing of non-protocol messages. Supported are two types of
callbacks: for single characters and for full lines. It is not possible to register both callbacks at the same time.
Pointers to the callbacks are passed in the cb argument. See the description of the data type tfw_np_cb_t for
details.
Callbacks must be unregistered. This is done by calling tfw_register() (see section 8.2.4.10) with the cb parameter
pointing to a value initialized using TFW_NP_CB_INITIALIZER.
Returns:
TFW_E_OK On success.
TFW_E_NOENT If no channel with the supplied ID exists.
TFW_E_INVAL If the cb argument is invalid.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
92 Test Framework API Reference
8.2.4.11 tfw_start
This function is used to start the test slave procedure.
Synopsis:
1 tfw_e_t tfw_start(void)
Description:
This function sends a message to the test slave requesting it to start the test slave procedure.
Returns:
TFW_E_OK On success.
TFW_E_STATE This value is returned if the connection to the test slave is not established.
TFW_E_* Error codes from the link layer are passed through.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Master Only API 93
8.2.4.12 tfw_join
Wait for the end of the test slave procedure.
Synopsis:
1 tfw_e_t tfw_join(void)
Description:
This function waits until the test slave procedure terminates, an error occurs, or a callback function registered with
tfw_register() (see section 8.2.4.10) returns a value different from TFW_E_OK. If the function returns due to a
callback returning a value different from TFW_E_OK, it can be called again to continue processing.
A test slave procedure can end in two possible ways: by reaching the end of the test slave procedure and by calling
the function tfw_reset() (see section 8.1.3.15). In the latter case tfw_join() (see section 8.2.4.12) transparently
restarts the test slave and (if automatic connecting is enabled) initiates the connection to the newly started test
slave. If automatic connecting is disabled the test master procedure is responsible to call tfw_connect() (see
section 8.2.4.8) when tfw_join() (see section 8.2.4.12) returned. In both cases tfw_join() (see section 8.2.4.12)
returns simply TFW_E_OK.
Note:
This function passes through the return values of the callback functions registered with tfw_register() (see section
8.2.4.10).
Returns:
TFW_E_OK If termination was caused by the end of the test slave procedure.
TFW_E_TIMEOUT If no message from the peer was received for the time specified in the TFW tag msg_time-
out.
TFW_E_STATE This value is returned if the connection to the test slave is not established.
TFW_E_PROTO If an unexpected protocol message was received.
TFW_E_* If termination was caused by a callback function returning an error code different from TFW_E_OK
it is passed through to the caller as error code of tfw_join() (see section 8.2.4.12).
TFW_E_* Error codes from the link layer are passed through.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
94 Test Framework API Reference
8.2.4.13 tfw_tag
This function is used to query the value of tfw tags.
Synopsis:
1 tfw_e_t tfw_tag(const char *name,
2 unsigned int *value)
Parameters:
name The name of the tag.
value If not NULL the value of the tag is stored here.
Description:
As argument this function gets the name of a tfw tag. If such a tag exists it will return the current value in its out
parameter value. Otherwise an error code is returned.
Returns:
TFW_E_OK On success.
TFW_E_NOENT If the tag specified does not exist.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Slave Only API 95
8.3 Slave Only API
In this section the API available to the test slave only is described.
8.3.1 Structure Definitions
8.3.1.1 struct tfw_slave_cb_t
Data structure used to register callbacks used by the test slave.
The function tfw_init() (see section 8.3.5.5) takes as argument a pointer to a data structure of this type. Its
content are pointers to functions to be called for certain low-level activities. The content of this data structure
might change in the future. To allow to write future-proof code the macros TFW_SLAVE_CB_INITIALIZER and
TFW_SECONDARY_SLAVE_CB_INITIALIZER are provided.
Note:
The structure may contain more members than described here.
Synopsis:
1 struct tfw_slave_cb_t {
2 tfw_slave_coverage_t * tfw_slave_coverage;
3 tfw_slave_halt_t * tfw_slave_halt;
4 tfw_slave_mutex_init_t * tfw_slave_mutex_init;
5 tfw_slave_mutex_destroy_t * tfw_slave_mutex_destroy;
6 tfw_slave_mutex_lock_t * tfw_slave_mutex_lock;
7 tfw_slave_mutex_unlock_t * tfw_slave_mutex_unlock;
8 tfw_slave_read_t * tfw_slave_read;
9 tfw_slave_reset_t * tfw_slave_reset;
10 tfw_slave_write_t * tfw_slave_write;
11 };
Structure Element Description:
tfw_slave_coverage Function to be called to transfer coverage data. For primary test slaves this callback is
optional, for secondary test slaves it is unsupported.
tfw_slave_halt Function to be called to stop execution, e.g. upon assertion. This callback is optional.
tfw_slave_mutex_init Function to be called to initialize a mutex. This callback is optional.
tfw_slave_mutex_destroy Function to be called to destroy a mutex. This callback is optional.
tfw_slave_mutex_lock Function to be called to lock a mutex. This callback is optional.
tfw_slave_mutex_unlock Function to be called to unlock a mutex. This callback is optional.
tfw_slave_read Function to be called to read data from the line connecting slave and master. For primary
test slaves this callback is mandatory, for secondary test slaves it is unsupported.
tfw_slave_reset Function to be called to reset the test slave. For primary test slaves this callback is optional.
For secondary test slaves it is unsupported.
tfw_slave_write Function to be called to write data to the line connecting slave and master. This callback is
mandatory.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
96 Test Framework API Reference
8.3.2 Defines
TFW_SLAVE_CB_INITIALIZER
Initialize a variable of type tfw_slave_cb_t to be used by the primary test slave.
8.3.3 Data Type Definitions
tfw_mutex_t This type used to implement mutexes.
8.3.4 Function Type Definitions
8.3.4.1 tfw_slave_halt_t
This is the prototype for a function called by the test framework to stop test execution.
Synopsis:
1 typedef void tfw_slave_halt_t(void)
Description:
The function tfw_init() (see section 8.3.5.5) allows to register low-level functions to be called in certain events. A
function of this type is called if an unrecoverable error occurred. It’s purpose is to halt test execution. A function
containing an infinite loop only is a suitable implementation.
Returns:
It is expected that this function does not return.
8.3.4.2 tfw_slave_mutex_init_t
This is the prototype for a function called by the test framework to initialize a mutex.
Synopsis:
1 typedef int tfw_slave_mutex_init_t(tfw_mutex_t *mutex)
Description:
The function tfw_init() (see section 8.3.5.5) allows to register low-level functions to be called in certain events. A
function of this type is called to initialize a mutex. A pointer to the mutex object is provided as argument. The
function must initialize it and return zero.
Returns:
0 If successful.
!=0 Otherwise.
8.3.4.3 tfw_slave_mutex_destroy_t
This is the prototype for a function called by the test framework to destroy a mutex.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Slave Only API 97
Synopsis:
1 typedef int tfw_slave_mutex_destroy_t(tfw_mutex_t *mutex)
Description:
The function tfw_init() (see section 8.3.5.5) allows to register low-level functions to be called in certain events. A
function of this type is called to destroy a mutex. A pointer to the mutex object is provided as argument. The
function must destroy it and return zero.
Returns:
0 If successful.
!=0 Otherwise.
8.3.4.4 tfw_slave_mutex_lock_t
This is the prototype for a function called by the test framework to lock a mutex.
Synopsis:
1 typedef int tfw_slave_mutex_lock_t(tfw_mutex_t *mutex)
Description:
The function tfw_init() (see section 8.3.5.5) allows to register low-level functions to be called in certain events. A
function of this type is called to lock a mutex. A pointer to the mutex object is provided as argument. The function
must lock it and return zero.
Returns:
0 If successful.
!=0 Otherwise.
8.3.4.5 tfw_slave_mutex_unlock_t
This is the prototype for a function called by the test framework to unlock a mutex.
Synopsis:
1 typedef int tfw_slave_mutex_unlock_t(tfw_mutex_t *mutex)
Description:
The function tfw_init() (see section 8.3.5.5) allows to register low-level functions to be called in certain events.
A function of this type is called to unlock a mutex. A pointer to the mutex object is provided as argument. The
function must unlock it and return zero.
Returns:
0 If successful.
!=0 Otherwise.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
98 Test Framework API Reference
8.3.4.6 tfw_slave_coverage_t
Manage structural coverage data.
Synopsis:
1 typedef void tfw_slave_coverage_t(unsigned int opt)
Description:
This function is called by the test framework to manage structural coverage data. Its purpose is to perform
whatever action is necessary to handle the request specified using the argument opt.
To transfer coverage data to the test master the functions tfw_coverage_beginf() (see section 8.3.5.1),
tfw_coverage_write() (see section 8.3.5.2) and tfw_coverage_end() must be used.
In the test case it is possible to call the function tfw_coverage() (see section 8.1.3.2), which will (after testing for
errors) call this callback.
8.3.4.7 tfw_slave_read_t
This is the prototype for a function called by the test framework to receive data from the test master.
Synopsis:
1 typedef int tfw_slave_read_t(unsigned int timeout)
Description:
The function tfw_init() (see section 8.3.5.5) allows to register low-level functions to be called in certain events. A
function of this type is called to receive data from the master. Upon each call of this function a single character is
read. If successful the value read is interpreted as a ’unsigned char’, casted to int and returned to the caller. If an
error occurred an appropriate error code is returned (the TFW_E_* failure codes are negative). As argument the
function gets the number of seconds it shall wait for a character to be received. The value 0 shall be interpreted
as infinite timeout. If nothing was received within the specified time the function shall return with the error code
TFW_E_TIMEOUT.
Returns:
value If the function is successful the function returns the value read as ’unsigned char’ casted to int.
TFW_E_TIMEOUT If timeout is different from zero and no character was received within timeout seconds.
TFW_E_* An appropriate error code otherwise.
8.3.4.8 tfw_slave_reset_t
This is the prototype for a function called by the test framework to reset the test slaves execution environment.
Synopsis:
1 typedef void tfw_slave_reset_t(void)
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Slave Only API 99
Description:
The function tfw_init() (see section 8.3.5.5) allows to register low-level functions to be called in certain events. A
function of this type is called if the test requests a reset of the test slaves execution environment.
Returns:
It is expected that this function does not return.
8.3.4.9 tfw_slave_write_t
This is the prototype for a function called by the test framework to send data to the test master.
Synopsis:
1 typedef tfw_e_t tfw_slave_write_t(const void *,
2 size_t len)
Description:
The function tfw_init() (see section 8.3.5.5) allows to register low-level functions to be called in certain events. A
function of this type is called to send data to the master. In the arguments the address and size of a buffer to be
send is specified. This function must send the entire buffer and return with TFW_E_OK. If it is not possible to send
the entire buffer it must return an error code.
Returns:
TFW_E_OK On success.
TFW_E_* An appropriate error code otherwise.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
100 Test Framework API Reference
8.3.5 Functions
8.3.5.1 tfw_coverage_beginf
Start transmission of a structural coverage data block.
Synopsis:
1 tfw_e_t tfw_coverage_beginf(const char *fmt,
2 ...)
Parameters:
fmt The format string describing the variadic parameters.
Description:
This function is used within the callback used by the test framework to manage structural coverage data to start a
new data transmission to the test master. Its printf()-like arguments can be used to write a string to the result log
to identify the structural coverage data block.
After successfully calling this function it is no longer possible to call other tfw_*() functions that might send
messages to the test master except for tfw_coverage_write() (see section 8.3.5.2) and tfw_assertf() (see section
8.1.3.1) until tfw_coverage_end() is called. This must be ensured by the user.
Returns:
TFW_E_OK On success.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
TFW_E_INVAL This value is returned if fmt is NULL.
TFW_E_* Error codes from the link layer are passed through.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Slave Only API 101
8.3.5.2 tfw_coverage_write
Send a block of structural coverage data to the master.
Synopsis:
1 tfw_e_t tfw_coverage_write(const void *buffer,
2 size_t len)
Parameters:
buffer The address of the buffer to transfer to the master.
len The size of the buffer to transfer to the master.
Description:
This function is used within the callback used by the test framework to manage structural coverage data to send a
single block of data to the master. As argument this function gets address and size of the data to transfer.
Before calling this function a transmission must be initiated by calling tfw_coverage_beginf() (see section 8.3.5.1).
When all data blocks were transferred by (possibly multiple) calls of tfw_coverage_write() (see section 8.3.5.2) the
transmission is finished by calling tfw_coverage_finish() (see section 8.3.5.3).
Returns:
TFW_E_OK On success.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
102 Test Framework API Reference
8.3.5.3 tfw_coverage_finish
Finish an ongoing transmission of structural coverage data to the master.
Synopsis:
1 tfw_e_t tfw_coverage_finish(void)
Description:
This function calls is used within the callback used by the test framework to manage structural coverage data to
finish an ongoing transmission of structural coverage data to the test master.
Returns:
TFW_E_OK On success.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Slave Only API 103
8.3.5.4 tfw_noop
Send a NOOP message to master.
Synopsis:
1 tfw_e_t tfw_noop(void)
Description:
This function sends a NOOP message to the the test master. It is used to keep the connection alive.
Returns:
TFW_E_OK On success.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
104 Test Framework API Reference
8.3.5.5 tfw_init
Initialize the test slave library.
Synopsis:
1 tfw_e_t tfw_init(tfw_slave_cb_t *cb)
Parameters:
cb Pointer to data structure with pointers to helper functions.
Description:
This function initializes the test slave library. As argument this function gets a pointer to a data structure containing
pointers to functions needed by the test framework for basic task like I/O or locking. Some of them are mandatory
while others are optional: mandatory are tfw_slave_read and tfw_slave_write. If tfw_slave_mutex_lock is defined,
tfw_slave_mutex_unlock must be defined, too. The same applies to tfw_slave_mutex_init and tfw_slave_mu-
tex_destroy.
Returns:
TFW_E_OK On success.
TFW_E_INVAL If the argument provided is invalid, e.g. due to a mismatch of the test framework API used for
compiling libtfw and the application using it.
TFW_E_INIT If the test slave library was already initialized.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Slave Only API 105
8.3.5.6 tfw_secondary
Start a secondary test slave connection.
Synopsis:
1 tfw_e_t tfw_secondary(void)
Description:
The function sends a message to the master to take over the connection. After this the primary test slave cannot
communicate with the test master anymore. Secondary test slaves are limited. They are not allowed to record test
results.
Returns:
TFW_E_OK On success.
TFW_E_INIT If tfw_init() (see section 8.3.5.5) was not called yet.
TFW_E_* Error codes from the link layer are passed through.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
106 Test Framework API Reference
8.3.5.7 tfw_slave
Execute test slave loop.
Synopsis:
1 void tfw_slave(void)
Description:
This function executes the test slave loop. It initiates a connection with the test master and waits for commands
from it.
Note:
The behavior of this function is unspecified if it is called in the test slave before calling tfw_init() (see section
8.3.5.5).
Returns:
This function does not return.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Slave Only API 107
8.3.5.8 tfw_read
Read one char from the master.
Synopsis:
1 tfw_e_t tfw_read(unsigned int read_timeout,
2 unsigned char *received_character)
Parameters:
read_timeout Timeout for the read operation in seconds. Infinity on 0.
received_character Upon success, the character read.
Description:
The tfw_read uses the tfw_slave_read function set by tfw_init. This function enables reads of one byte non_proto
data from the master, before tc_link_*() functions were called by the test case.
Note:
The type differs from tfw_slave_read_t.
TFW_E_OK Upon success.
TFW_E_INIT If tfw_init() (see section 8.3.5.5) was not called yet.
TFW_E_TIMEOUT If no character was received within read_timeout seconds.
TFW_E_* Error codes from the link layer are passed through.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
108 Test Framework API Reference
8.3.5.9 tfw_write
send an array of characters/bytes towards the master.
Synopsis:
1 tfw_e_t tfw_write(const void *,
2 size_t len)
Parameters:
len Size of the data to be sent.
Description:
This function is called to send data to the master. In the arguments the address and size of a buffer to be send
is specified. This function will send the entire buffer and return with TFW_E_OK. If it is not possible to send the
entire buffer, an error code is returned.
Returns:
TFW_E_OK On success.
TFW_E_INIT If tfw_init() (see section 8.3.5.5) was not called yet.
TFW_E_* Error codes from the link layer are passed through.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
User Interaction API 109
8.4 User Interaction API
This section describes the API provided to interact with a user. Except for tfw_interact_get_cb() (see section
8.4.1.5) all functions are available in the test slave and the test master. The function tfw_interact_get_cb() (see
section 8.4.1.5) is only available in the test master.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
110 Test Framework API Reference
8.4.1 Functions
8.4.1.1 tfw_interact_put_msgf
Output a message for user interaction.
Synopsis:
1 tfw_e_t tfw_interact_put_msgf(const char *fmt,
2 ...)
Parameters:
fmt The format string describing the variadic parameters.
Description:
This printf()-like function is used in user interactions to output messages.
TFW_E_OK Upon success.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
TFW_E_* In the test slave error codes from the link layer are passed through.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
User Interaction API 111
8.4.1.2 tfw_interact_get_nl
Wait for the user to press the ENTER key.
Synopsis:
1 tfw_e_t tfw_interact_get_nl(void)
Description:
This functions asks the user to press the ENTER key and waits for it. It is best used to pause the test run
after showing an informative message with tfw_interact_put_msgf() (see section 8.4.1.1). If the test is run non-
interactively this function aborts.
TFW_E_OK Upon success.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
TFW_E_PROTO In the test slave in case of a protocol error.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
112 Test Framework API Reference
8.4.1.3 tfw_interact_get_uint
Wait for the user to input an unsigned integer value.
Synopsis:
1 tfw_e_t tfw_interact_get_uint(unsigned int *value)
Parameters:
value The value provided by the user.
Description:
This functions asks the user to input an unsigned integer value and waits for him to do so. Values can be specified
in decimal, octal (with prefix 0), and hexadecimal (with prefix 0x). See tfw_interact_put_msgf() (see section 8.4.1.1)
for how to output a text explaining which number to input. If the test is run non-interactively this function aborts.
TFW_E_OK Upon success.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
TFW_E_PROTO In the test slave in case of a protocol error.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
User Interaction API 113
8.4.1.4 tfw_interact_get_yn
Wait for the user to enter "Yes" or "No".
Synopsis:
1 tfw_e_t tfw_interact_get_yn(int *value)
Parameters:
value Zero if the user entered "No", otherwise non-zero.
Description:
This functions asks the user to enter "Yes" or "No" and waits for him to do so. See tfw_interact_put_msgf() (see
section 8.4.1.1) for how to output a question. If the test is run non-interactively this function aborts.
TFW_E_OK Upon success.
TFW_E_INIT This value is returned in the test slave if tfw_init() (see section 8.3.5.5) wasn’t called yet.
TFW_E_PROTO In the test slave in case of a protocol error.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
114 Test Framework API Reference
8.4.1.5 tfw_interact_get_cb
Wait for the user while calling a provided interaction function.
Synopsis:
1 tfw_e_t tfw_interact_get_cb(void *value,
2 tfw_e_t(*cb)(void *))
Parameters:
value Pointer passed through to the callback cb.
cb Callback function called repeatedly by tfw_interact_get_cb() (see section 8.4.1.5).
Description:
This function gets as argument two pointers. The first points to some memory object and the second to a function.
This function is repeatedly called with the first argument as parameter until it returns TFW_E_OK. If the test is run
non-interactively this function aborts.
TFW_E_OK Upon success.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
9 Test Framework Tags
The test framework can be configured using so called tags.
Technically a tag is an integer variable with a value which is set at build time. A tag can be either global or local to
a test case. If for a tag a global and a local value is specified, the local value is used. It is also possible to override
the values using command line parameters of the test master.
Global tags are defined in the file tfwtags.xml in the test set directory, local tags are defined in the test case
xml file (see section 6.1.1.5 for a description of the format). At the command line the parameter TFW_TAGS can
be used:
[ts_example] $ make run SET=<testsetname> TFW_TAGS=msg_timeout=20
In tfwtags.xml and the test case XML file only the tags listed below are supported. On the command line
arbitrary tags can be set.
The function tfw_tag() (see section 8.2.4.13) can be used to retrieve tag values.
9.1 Global Only Tags
The following tags can be used in global tfwtags.xml file only:
wait_for_up If nonzero the test master waits for the handshake to succeed before starting the first test case. If
zero it does not wait and starts the first test case immediately. In this case the first test case needs a test
master procedure which is responsible to initiate the connection to the slave by calling tfw_connect() (see
section 8.2.4.8). The default value is 1.
9.2 Normal Tags
The following tags be used as global and as local tag.
msg_timeout Timeout in seconds used by the test master when reading from the protocol channel when the
connection between master and slave is up. The default value is 10.
msg_nonproto The number of nonprotocol messages (i. e. lines) the test master accepts between reception of
two protocol messages while the connection between master and slave is up. If it receives more the test
run is aborted with an error message. The default value is 100.
tfw_debug If this value is nonzero the test run will emit extra debug output (for example the protocol messages
transmitted). The default value is zero.
slave_reset If this value is nonzero the test slave is reset at certain points. For this the constants
• TFW_TAG_SLAVE_RESET_BEFORE_TESTCASE,
• TFW_TAG_SLAVE_RESET_AFTER_TESTCASE,
• TFW_TAG_SLAVE_RESET_BEFORE_SUBTESTCASE, and
• TFW_TAG_SLAVE_RESET_AFTER_SUBTESTCASE
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
116 Test Framework Tags
are defined (see section 8.2.2 for a description and Figure 16 on page 43 for usage). The actual reset is
done before executing a subtest. The variable slave_reset is used to specify when a reset is needed. Also
no reset is done if the test slave is still clean (i. e. no test was performed yet). The default is zero which
implies no reset.
up_timeout Timeout in seconds used by the test master when reading from the protocol channel before the
connection between master and slave is up. Usually this value is used before the first test case is started.
To take an effect in this case it must be specified globally. The test case local value is used if automatic
handshake is disabled or the test slave is reset. The default value is 60.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
10 Configuring Test Targets
10.1 Target Hooks
The test framework has built-in support for QEMU based targets. For all other targets a configuration file providing
shell hooks must be created. These files are either stored in the test suites config/ or sys/ directory. They
are named like the board with the prefix “target.”. The board name is what the tool pikeos-info outputs as
PIKEOS_BOARD if called like this:
[ts_example] $ pikeos-info -board -info=<*.bsp.dom>
To configure the test suite for the standard x86 board provided by PikeOS either the file config/target.x86-64
or the file sys/target.x86-64 is used. These files provide shell hooks called by the test framework. The
functions provided are
• the variables hook,
• the config hook, and
• the mandatory run hook.
All of them are optional. See Figure 20 for a skeleton.
10.1.1 The Variables Hook
This shell function is used to get the values for the options ARCH, PROC, BOARD, and BOOTSTRAT. Usually this
is not needed because the needed data is taken from pikeos-info. But sometimes the value of BOOTSTRAT
must be set, if the default boot strategy configured in the *.bsp.dom file and returned by pikeos-info is not
suitable.
10.1.2 The Config Hook
The purpose of the config hook is to set additional variables needed for MUXA setup. These are:
• TFW_MUXA_HOST_NAME,
• TFW_MUXA_HOST_IP and TFW_MUXA_HOST_PORT,
• TFW_MUXA_TARGET_IP and TFW_MUXA_TARGET_PORT,
• TFW_MUXA_CONFIG_PORT,
• TFW_MUXA_GW_IP, and
• TFW_MUXA_CHANNEL_0 to TFW_MUXA_CHANNEL_14.
If environment variables with these names are set and the integration project is configured to include the MUXA
these values are passed through directly to the corresponding options in the MUXA section of the integration
project.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
118 Configuring Test Targets
1 #! /bin/bash
2
3 # variables hook:
4 # arguments: none
5 variables_x86-64()
6 {
7 # set BOOTSTRAT
8 BOOTSTRAT=elf
9 }
10
11 # config hook:
12 # arguments: none
13 config_x86-64()
14 {
15 ...
16 }
17
18 # run hook:
19 # arguments: $1: name of master binary
20 # $2: name of slave image
21 # $3..$n: additional arguments
22 run_x86-64()
23 {
24 exec ...
25 }
Figure 20: Skeleton of the target.x86-64 file. It is a bash script implementing the hooks as functions with specific names.
While the variables and config hooks usually set variables and return the run hook calls exec.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
TCON Controlled Targets 119
1 config_x86-64()
2 {
3
4 TARGET_HOST=user@host
5 TARGET_NAME=x86
6
7 . ${TC_FRAMEWORK_DIR}/scripts/run/prebuild-tcon.sh
8 }
9
10 run_x86-64()
11 {
12 TARGET_HOST=user@host
13 TARGET_NAME=x86
14
15 . ${TC_FRAMEWORK_DIR}/scripts/run/run-tcon.sh
16 }
Figure 21: Example for how to configure a TCON controlled remote target. In this case the target hooks just include test
framework provided scripts after setting some variables.
10.1.3 The Run Hook
The run hook is used to execute tests. As argument it gets (in this order) the path to the test master plugin (or
the test master binary, if the plugin approach is not used), the path to the test slave plugin, and the options to be
passed to the test master. The task of the run hook is to execute the test.
10.2 TCON Controlled Targets
Often access to targets is managed by special software. One of them is TCON. It is supported directly by the test
framework. If it is used, the config and run target hooks simply include test framework provided scripts (see Figure
21 for an example). See the scripts for a description of all the variables available to configure their behavior.
10.3 Unmanaged Targets
All target hooks are optional. If access to the target is not controlled by special software, no MUXA is
used, and the default boot strategy is fine it is not necessary to create a target file. In this case the
options TARGET_CONSOLE_LINE and TFW_BOOTIMAGE must be set either at the command line or in the
config/testsuite.conf file. Then the test framework provided run script for unmanaged targets will be
called automatically. It supports the same options as the run-tcon.sh script.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
A Examples
A.1 A Test Case Using a Parameter Table
If a test needs to test the input range of a function it is recommended to use a parameter table generated from an
equivalence class description. During the build process C code is generated from this description. Definitions of
the data structures are made available in the test cases header file (e.g. tc_example_01.h). The actual values
can be retrieved by calling the function tfw_param(). A minimal example for a test case using this framework
feature is shown in Figure 22.
The return value of tfw_param() must be cast by the test procedure to the correct type. Then the parameter set
can be used. In the example it is used as argument of the function under test. The expected result and the id
of the objective which is tested are also taken from the parameter table. Using this approach it is possible to test
with a single, very short, test procedure and a well-defined set of equivalence classed the whole input range of a
function, including normal and robustness tests.
Usually the parameter table contains only values which are compile-time constants. But it is also possible to add
function calls to the parameter table. For this the parameter TFW_RUNTIME_PARAMETERS of the application
project must be set to "true".
A.2 A Test Master Processing Boot Messages
For some tests it is necessary to read the console output of the system under test and search in it for specific
strings. This is particularly difficult if the test is interested in the boot messages. Usually the first test case is
started after test master and test slave are up and running. But using the framework tag wait_for_up it is possible
to disable this behavior. In this case the first test case of the test set is started immediately.
The first thing the test slave will do after its start, is initiating the connection to the test master by sending
MSG_SYN messages. To conform to this behavior a test set using wait_for_up has to call the master part of the
handshaking function first. This is implemented by tfw_connect() (see section 8.2.4.8). This function implements
the whole master part of the handshake. It only returns if either the handshake was successful or an error (e. g. a
timeout) occurred. The caller may use tfw_register() (see section 8.2.4.10) to provide a callback function to
1 #include <tfw.h>
2 #include <tc_example_01.h>
3
4 void tc_example_01(void)
5 {
6 const tc_example_01_params_t *eq = tfw_param();
7 int rv;
8
9 rv = testedFunction(eq->param);
10 TFW_EVAL(rv == eq->retval, eq->obj);
11 }
Figure 22: Test Procedure Using a Parameter Table.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Testing a Communication Channel 121
tfw_connect() (see section 8.2.4.8) to evaluate the whole console output until the handshake succeeded. Figure
23 shows an example for a test case which searches in the console output during boot for a predefined pattern.
Important is to register the callback for the correct channel. If the TFW protocol is transmitted using the console
of the tested target the channel to use is TFW_CHANNEL_PROTOCOL, otherwise TFW_CHANNEL_CONSOLE
must be used. The result for the objective MESSAGE is S_PASS if and only if this string was found in the console
output and the handshake did not succeed. Otherwise it is S_FAIL.
Test cases like this are well suited to test that the system under test does not start under certain conditions, but
emits an error message and stops startup.
A.3 Testing a Communication Channel
Sometimes it is necessary to test message sending from the master to a slave over some communication channel.
In this case the functions available for master/slave synchronization must be used. If the communication channel
under test is the same as used by the test framework for communication tfw_channel_pause() and tfw_chan-
nel_resume() must be used. See Figure 24 for an example. It shows the necessary preparation and cleanup code
needed in the test master only. The actual test is performed by the function perform_test(), which is not shown.
A.4 Sending Data
The functions tfw_sendf() and tfw_recvf() allow to send integer data from the slave to the master and vice versa.
These functions do not process arbitrary protocol messages. I. e. for safe usage a synchronization message must
be send before calling tfw_sendf(). Likewise a function waiting for a synchronization message must be called
before calling tfw_recvf().
In Figure 25 a test master procedure receiving a signed integer and an unsigned char is shown. The format
string used by tfw_recvf() resembles the format of scanf(). Note, that tfw_wait() is called immediately before
calling tfw_recvf(). The implementation of the test slave is shown in Figure 26. It simply sends a synchronization
message followed by the data itself.
The functions tfw_sendf() and tfw_recvf() should use the same format string. Using a larger data type for reception
then for sending is supported (e. g. sending using %hd, reception using %d). The test case developer is responsi-
ble to ensure that the data type used for reception has a sufficient size. For example the data type int could have
size of 32 Bit in the test slave and 64 bit in the test master. In this case either the master must ensure that the
value to be transmitted does not exceed 232 − 1 or the slave must use a 64 Bit data type for reception (e. g. long
long int). Here it is important to understand that tfw_recvf() and tfw_sendf() do not transmit binary patterns, but
integer values, and tfw_recvf() does truncate the value silently if it does not fit into the provided argument.
A.5 Mutexes in the Test Slave
To make the test slave library thread safe mutex hooks must be implemented by the application project. These
are registered with tfw_init() (see section 8.3.5.5). To implement mutexes the elements tfw_slave_mutex_init,
tfw_slave_mutex_lock, tfw_slave_mutex_unlock, and tfw_slave_mutex_destroy of the callback structure must be
set. See Figure 27 for a possible implementation of these callbacks using PikeOS semaphores.
A.6 Ensure Master-Slave-Synchronisation
To ensure test cases with multiple synchronisation points between master and slave (using tfw_sync(), tfw_wait())
work as desired the following scheme may prove helpfull: The functions from Figure 28 implement numbered
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
122 Examples
1 #include <tfw.h>
2 #include <tc_example_01.h>
3 #include <string.h>
4
5 static enum { INIT, ERROR, FOUND } flag;
6
7 static tfw_e_t parse_boot_msg(int channel, const char *line)
8 {
9 if (strstr(line, "pattern to search for") != NULL)
10 flag = FOUND;
11
12 return TFW_E_OK;
13 }
14
15 void tc_example_01(void)
16 {
17 tfw_e_t ret;
18 tfw_np_cb_t np;
19
20 /* register the callback */
21 np = TFW_NP_CB_INITIALIZER;
22 np.tfw_np_line = parse_boot_msg;
23 ret = tfw_register(TFW_CHANNEL_CONSOLE, &np);
24 if (ret == TFW_E_NOENT)
25 ret = tfw_register(TFW_CHANNEL_PROTOCOL, &np);
26 TFW_ASSERT(ret == TFW_E_OK);
27
28 /* try to connect */
29 flag = INIT;
30 ret = tfw_connect();
31 TFW_EVAL(ret == TFW_E_TIMEOUT && flag == FOUND, OBJ_MESSAGE);
32
33 /* unregister the callback */
34 np = TFW_NP_CB_INITIALIZER;
35 ret = tfw_register(TFW_CHANNEL_CONSOLE, &np);
36 if (ret == TFW_E_NOENT)
37 ret = tfw_register(TFW_CHANNEL_PROTOCOL, &np);
38 TFW_ASSERT(ret == TFW_E_OK);
39 }
Figure 23: A Test Master Procedure processing Target Boot Messages
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Ensure Master-Slave-Synchronisation 123
1 #include <tfw.h>
2 #include <tc_example_01.h>
3
4 void tc_example_01()
5 {
6 int fd;
7 tfw_e_t ret;
8
9 ret = tfw_start();
10 TFW_ASSERT(ret == TFW_E_OK);
11
12 /* wait for a sync */
13 ret = tfw_wait();
14 TFW_ASSERT(ret == TFW_E_OK);
15
16 ret = tfw_channel_pause(TFW_CHANNEL_PROTOCOL, &fd);
17 TFW_ASSERT(ret == TFW_E_OK);
18 perform_test(fd);
19 ret = tfw_channel_resume(TFW_CHANNEL_PROTOCOL);
20 TFW_ASSERT(ret == TFW_E_OK);
21
22 /* wait for test case end */
23 ret = tfw_join();
24 TFW_ASSERT(ret == TFW_E_OK);
25 }
Figure 24: A Test Master Procedure Testing the Serial Line.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
124 Examples
1 #include <tfw.h>
2 #include <tc_example_01.h>
3
4 void tc_example_01(void)
5 {
6 tfw_e_t ret;
7 int value;
8 unsigned char c;
9
10 ret = tfw_start();
11 TFW_ASSERT(ret == TFW_E_OK);
12
13 /* wait for sync */
14 ret = tfw_wait();
15 TFW_ASSERT(ret == TFW_E_OK);
16
17 ret = tfw_recvf("%d%hhu", &value, &c);
18 TFW_ASSERT(ret == TFW_E_OK);
19
20 /* wait for end */
21 ret = tfw_join();
22 TFW_ASSERT(ret == TFW_E_OK);
23
24 tfw_printf("slave sent integer %d and char %c", value, c);
25 }
Figure 25: A Test Master Procedure Receiving Data from the Test Slave.
1 #include <tfw.h>
2 #include <tc_example_01.h>
3
4 void tc_example_01(void)
5 {
6 tfw_e_t ret;
7 int value = 0x12345678;
8 unsigned char c = ’A’;
9
10 tfw_printf("will sent integer %d and char %c", value, c);
11 ret = tfw_sync();
12 TFW_ASSERT(ret == TFW_E_OK);
13 ret = tfw_sendf("%d%hhu", value, c);
14 TFW_ASSERT(ret == TFW_E_OK);
15 }
Figure 26: A Test Slave Procedure Sending Data to the Test Master.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Ensure Master-Slave-Synchronisation 125
1 #include <p4.h>
2
3 static P4_sem_t semaphores[5];
4
5 static int cb_mutex_init(tfw_mutex_t * mutex)
6 {
7 static P4_atomic_t counter = 0;
8 unsigned int index;
9
10 index = p4_atomic_fetch_and_add(&counter, 1);
11 if (index >= sizeof(semaphores) / sizeof(semaphores[0]))
12 return TFW_E_NOMEM;
13
14 p4_sem_init(&semaphores[index], 1, 0);
15 *mutex = index;
16
17 return TFW_E_OK;
18 }
19
20 static int cb_mutex_lock(tfw_mutex_t * mutex)
21 {
22 return p4_sem_wait(&semaphores[*mutex], P4_TIMEOUT_INFINITE);
23 }
24
25 static int cb_mutex_unlock(tfw_mutex_t * mutex)
26 {
27 return p4_sem_post(&semaphores[*mutex]);
28 }
29
30 static int cb_mutex_destroy(tfw_mutex_t * mutex __attribute__ ((unused)))
31 {
32 return TFW_E_OK;
33 }
Figure 27: Code fragment implementing mutexes for the test slave on PikeOS. The functions can be directly assigned to
the members of tfw_slave_cb_t (see section 8.3.1). Supported are 5 mutexes, which is sufficient for the needs of the test
framework. Mutex destroy is not strictly needed, so it wasn’t implemented.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
126 Examples
1 #include <tfw.h>
2
3 void wait(int num)
4 {
5 tfw_e_t ret;
6 int value;
7
8 ret = tfw_recvf("%d", &value);
9 TFW_ASSERT(ret == TFW_E_OK && value == num);
10 }
11
12 void ping(int num)
13 {
14 tfw_e_t ret;
15
16 ret = tfw_sendf("%d", &value);
17 TFW_ASSERT(ret == TFW_E_OK);
18 }
Figure 28: Helper Functions to Enformce Synchronisation
synchronisation points. Using these funtions it is ensured that master and slave meet in the same synchronisation
points.
c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved.
Reference in a new issue View git blame Copy permalink