--- title: "Test Framework Reference Manual" source: "docs/development/test-framework-reference-manual.pdf" category: "development" pages: 126 extracted: "2026-07-06T23:05:35.500693" --- # Test Framework Reference Manual > Extracted from `docs/development/test-framework-reference-manual.pdf` (126 pages). > Figures, diagrams, and tables may not render accurately in plain text. PikeOS Test Framework Reference Manual Am Pfaffenstein 14, D-55270 Klein-Winternheim Notice: The contents of this document are proprietary to SYSGO AG and shall not be disclosed, disseminated, copied, or used except for purposes expressly authorized in writing by SYSGO AG. PikeOS Test Framework Reference Manual PikeOS D5.0, Document Version D5.0-3576 c 2005 – 2019 SYSGO GmbH SYSGO GmbH Email: office@sysgo.com Am Pfaffenstein 14 55270 Klein-Winternheim, Germany http://www.sysgo.com All rights reserved. PikeOS is a trademark of SYSGO GmbH. The designations used to identify other software or hardware products in this publication may be trademarks of their manufacturers or sellers. Contents 1 Basic Test Suite Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.1 Preparing a Test Suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.2 Build-In Help Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.3 Listing Test Cases and Test Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.4 Building and Running Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.4.1 Basic Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.4.2 Partial Test Set Builds and Runs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.4.3 Manual Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.4.4 Specifying Labels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.4.5 Temporarily Changing PikeOS Project Configuration . . . . . . . . . . . . . . . . . . . . 11 1.5 Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.6 Cleaning Up the Test Suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.6.1 Removing Intermediate Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.6.2 Resetting to Initial State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.6.3 Reverting Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2 Overlays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.1 Creating Overlays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.1.1 The Directory Structure of an Overlay . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.1.2 The Content of the Test Suite Specific Overlay . . . . . . . . . . . . . . . . . . . . . . . 14 2.1.3 The Content of the Test Suite Common Overlay . . . . . . . . . . . . . . . . . . . . . . . 14 2.1.4 The Content of the Generic Overlay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2 Using Overlays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3 Test Set Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.1 Creating Test Set Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.1.1 Naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.1.2 Directory Layout of a Test Set Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.1.3 The Test Case XML Description File V2.1 . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.1.3.1 The and Elements . . . . . . . . . . . . . . . . . . . . . . 17 3.1.3.2 The Element . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.1.3.3 The Element . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.1.3.4 The Element . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2 Using Test Set Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.3 Support for Multiple Test Set Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.4 Automatic Generation of Test Set Pool Variables from Trace-XML-files . . . . . . . . . . . . . . . 20 4 Verification Result Artifacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 4.1 Artifact Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 4.1.1 Building the Test Case Master Document . . . . . . . . . . . . . . . . . . . . . . . . . . 21 4.1.2 Building the Test Result Document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 4.1.3 Building the Structural Coverage Result Document . . . . . . . . . . . . . . . . . . . . . 22 4.2 Artifact Setup in XML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 4.2.1 XML Document Fragments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. 4 CONTENTS 4.2.2 Setup of the Test Case Master Document . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.2.2.1 The File doc/tc.xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.2.2.2 Configuring Label Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.2.2.3 Using description.xml Files . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.2.2.4 Additional Document Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.2.3 Setup of the Test Result Document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.2.3.1 The File doc/tr.xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.2.3.2 Additional Document Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 4.2.4 Setup of the Structural Coverage Result Document . . . . . . . . . . . . . . . . . . . . . 30 4.2.4.1 The File doc/sc.xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 4.2.4.2 Additional Document Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 4.3 Artifact Setup in TEX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 4.3.1 Managing Abbreviations and References with BibTeX . . . . . . . . . . . . . . . . . . . . 32 4.3.1.1 Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 4.3.1.2 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 4.4 Available Makefile Hooks for Document Creation . . . . . . . . . . . . . . . . . . . . . . . . . . 33 4.5 Changebars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 5 Test Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 5.1 Application and Integration Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 5.1.1 Requirements on the Application Project . . . . . . . . . . . . . . . . . . . . . . . . . . 36 5.1.2 Requirements on the Integration Project . . . . . . . . . . . . . . . . . . . . . . . . . . 37 5.2 Special PikeOS Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 6 Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 6.1 The Files in the Test Case Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 6.1.1 The Test Case XML Description File . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 6.1.1.1 The and Elements . . . . . . . . . . . . . . . . . . . . . . 40 6.1.1.2 The and Elements . . . . . . . . . . . . . . . . . . . . . . . . . 42 6.1.1.3 The and Elements . . . . . . . . . . . . . . . . . . . . . . 42 6.1.1.4 The and Elements . . . . . . . . . . . . . . . . . . . . 43 6.1.1.5 The Element . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 6.1.1.6 The Element . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 6.1.2 The Test Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 6.1.3 The symbol-global.*.txt Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 6.1.4 The t*_cflags.rc Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 6.1.5 The Makefile File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 6.2 General Rules for Writing Test Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 6.3 The Protocol Used by Master and Slave to Communicate . . . . . . . . . . . . . . . . . . . . . . 47 6.3.1 Three-Way Handshake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 6.3.2 Filtering Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 6.3.3 Running Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 6.4 Interactive Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 7 Parameter Sets Defined by Equivalence Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 7.1 Defining Equivalence Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 7.2 Combining Equivalence Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 7.3 Choosing Representatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 7.3.1 Boundary Value Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 7.4 Extending Equivalence Classes by Logical Parameters . . . . . . . . . . . . . . . . . . . . . . . 53 7.5 Extending Equivalence Classes by Virtual Parameters . . . . . . . . . . . . . . . . . . . . . . . 53 c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. CONTENTS 5 7.6 Using Equivalence Classes in Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 7.6.1 The Parameter List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 7.6.2 The Result and Objective List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 7.6.3 The for Attribute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 7.6.4 Including Header Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 8 Test Framework API Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 8.1 Common Master/Slave API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 8.1.1 Defines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 8.1.2 Enumerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 8.1.3 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 8.1.3.1 tfw_assertf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 8.1.3.2 tfw_coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 8.1.3.3 tfw_debugf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 8.1.3.4 tfw_dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 8.1.3.5 tfw_name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 8.1.3.6 tfw_strerror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 8.1.3.7 tfw_param . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 8.1.3.8 tfw_printf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 8.1.3.9 tfw_sendf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 8.1.3.10 tfw_sync . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 8.1.3.11 tfw_recvf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 8.1.3.12 tfw_tracef . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 8.1.3.13 tfw_wait . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 8.1.3.14 tfw_warnf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 8.1.3.15 tfw_reset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 8.2 Master Only API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 8.2.1 Structure Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 8.2.1.1 struct tfw_np_cb_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 8.2.2 Defines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 8.2.3 Function Type Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 8.2.3.1 tfw_np_char_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 8.2.3.2 tfw_np_line_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 8.2.4 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 8.2.4.1 tfw_bootmsg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 8.2.4.2 tfw_channel_id . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 8.2.4.3 tfw_channel_name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 8.2.4.4 tfw_channel_pause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 8.2.4.5 tfw_channel_read . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 8.2.4.6 tfw_channel_resume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 8.2.4.7 tfw_channel_write . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 8.2.4.8 tfw_connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 8.2.4.9 tfw_disconnect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 8.2.4.10 tfw_register . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 8.2.4.11 tfw_start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 8.2.4.12 tfw_join . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 8.2.4.13 tfw_tag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. 6 CONTENTS 8.3 Slave Only API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 8.3.1 Structure Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 8.3.1.1 struct tfw_slave_cb_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 8.3.2 Defines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 8.3.3 Data Type Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 8.3.4 Function Type Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 8.3.4.1 tfw_slave_halt_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 8.3.4.2 tfw_slave_mutex_init_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 8.3.4.3 tfw_slave_mutex_destroy_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 8.3.4.4 tfw_slave_mutex_lock_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 8.3.4.5 tfw_slave_mutex_unlock_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 8.3.4.6 tfw_slave_coverage_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 8.3.4.7 tfw_slave_read_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 8.3.4.8 tfw_slave_reset_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 8.3.4.9 tfw_slave_write_t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 8.3.5 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 8.3.5.1 tfw_coverage_beginf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 8.3.5.2 tfw_coverage_write . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 8.3.5.3 tfw_coverage_finish . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 8.3.5.4 tfw_noop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 8.3.5.5 tfw_init . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 8.3.5.6 tfw_secondary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 8.3.5.7 tfw_slave . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 8.3.5.8 tfw_read . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 8.3.5.9 tfw_write . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 8.4 User Interaction API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 8.4.1 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 8.4.1.1 tfw_interact_put_msgf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 8.4.1.2 tfw_interact_get_nl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 8.4.1.3 tfw_interact_get_uint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 8.4.1.4 tfw_interact_get_yn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 8.4.1.5 tfw_interact_get_cb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 9 Test Framework Tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 9.1 Global Only Tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 9.2 Normal Tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 10 Configuring Test Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 10.1 Target Hooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 10.1.1 The Variables Hook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 10.1.2 The Config Hook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 10.1.3 The Run Hook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 10.2 TCON Controlled Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 10.3 Unmanaged Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 A Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 A.1 A Test Case Using a Parameter Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 A.2 A Test Master Processing Boot Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 A.3 Testing a Communication Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 A.4 Sending Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 A.5 Mutexes in the Test Slave . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. CONTENTS 7 A.6 Ensure Master-Slave-Synchronisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. 1 Basic Test Suite Usage A test suite consists of test sets (containing test cases), and pre-defined configuration data used to compile and run the test sets. In the following sections, it is described step-by-step how to prepare an existing test suite, how to build and run test sets, and how to reset a test suite to the initial state it had before preparation. All functionality of the test framework is exposed to the user by a single Makefile provided by the test suite. This means that the user will usually call commands of the form: [ts_example] $ make OPTION1= OPTION2= ... Usually the test framework is non-verbose while performing commands. By setting the special option VERBOSE to yes the test framework can be instructed to be verbose about the commands being executed. Also various commands will output additional log messages. 1.1 Preparing a Test Suite Assume there is a test suite named ts_example and a shell is open in its base directory. At first a preparation step is necessary. For this the command prepare is used: [ts_example] $ make prepare This command will retrieve the test framework from a predefined location and make it available to the test suite. If the predefined location is not suitable it is possible to specify the location of the test framework using the option TFW_DIR: [ts_example] $ make prepare TFW_DIR= The directory must contain the modules framework, common, and contrib of the test framework. Further, the prepare command will create the directory sys/ in the test suite. Depending on the configuration other directories might be created. All of this is done automatically by the prepare command. The prepare command supports multiple additional options. See the detailed usage information of the prepare command for the options provided by the test framework. It is output by the following command: [ts_example] $ make help TFW_DIR= HELP=prepare An important feature provided by the test frameworks prepare command is support for test suite overlays, which can be used to extend or even replace parts of a test suite during test suite preparation. See section 2 for a description of this concept. The test suite itself might extend the prepare command with additional options, too. Refer to the test suite documentation for details on this possible existing additional options. 1.2 Build-In Help Messages The command help can be used to list all commands which are supported by the test framework installed: [ts_example] $ make help Further, the option HELP can be used to get detailed usage information for a specific command. For example, to get detailed information on the help command the following command can be used: c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. Listing Test Cases and Test Sets 9 [ts_example] $ make help HELP=help The test framework supports various commands which depend on configuration data in the test suite. If this configuration is not in place these commands are not shown in the output of the help command. This means the output contains only the commands that will actually work. 1.3 Listing Test Cases and Test Sets Two dedicated commands are available to list test cases (tclist) and test sets (setlist). See chapter 5 and 6 for a detailed description of test sets and test cases. For now it is only important to understand that a test case is the most basic building block of testing. A test set is a certain set of test cases which are compiled into the same binary. The test cases of a test suite can be listed using the following command: [ts_example] $ make tclist This will output all test cases of the test suite. The command supports various options that allow outputting only a subset of the test suite’s test cases. For example it is possible to output only the test cases of a test set: [ts_example] $ make tclist SET= In a similar manner it is possible to output the names of all test sets of the test suite: [ts_example] $ make setlist Again it is possible to filter the output. For example the following command will output the test set a given test case belongs to [ts_example] $ make setlist TC= For more information on the options available and their usage see the detailed usage information of tclist and setlist. 1.4 Building and Running Tests A test set is built by calling the build command and run by calling the run command. Both commands have the same options, so they are discussed together. 1.4.1 Basic Usage If no option is specified the build command builds all test sets for the default target board. But usually the user specifies which test set to build and which target board shall be used: [ts_example] $ make build SET= BOARD= Here it is important to understand that the only entity that can be built is the test set. The build command takes care of the entire build process. It will build the test master and the test slave. For PikeOS test slaves this includes cloning and building of e. g. application and integration projects. Output of the build command are binaries in sys/bin/, which are used by the run command. The same options are accepted by the run command: [ts_example] $ make run SET= BOARD= The run command depends on the build command. If the test framework installation supports validation of XML files the test cases XML files are validated before building. Both commands will output a log to the console and to a file. The output of the build process is stored by default in a file in the test suites log/ directory. c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. 10 Basic Test Suite Usage The output of the test run is stored in a file in the results/ directory. As file name the test set name with the postfix .log respectively .res is used. The binaries and all intermediate files are stored within the sys/ directory. See section 1.6 for a description how to clean up. 1.4.2 Partial Test Set Builds and Runs For development purposes the test framework supports building and running partial test sets. This means, it will build a test set but the binary will only contain a subset of the test cases the test set consists of. This is done by specifying e. g. the TC option: [ts_example] $ make build TC=" " This will build the test set to which the test cases testcasename1 and testcasenme2 belong. But the binary will only contain these two test cases. The other test cases of the test set are ignored. Of course, this makes only sense for development. It depends on the test suite if partial test set builds are supported. If the test suite does not support partial builds the build will usually fail with a linker error. If a partial test set build is run, it will automatically run all compiled-in test cases. This is also what happens if the TC option is specified when calling the run command: [ts_example] $ make run TC=" " It is also possible to build the full test set, but execute only a part of it. For this the parameter SKIP_TO can be used. Its value is the number of the test case to start at: [ts_example] $ make run SET= SKIP_TO=5 This will build the specified test fully (i. e. no partial build, all test cases are included), but execution will start at the fifth test case. The first four test case of the test set are skipped. Another possibility is to use the command line options of the test master binary. It has an option that allows to specify which test case of a test set shall be run. It can be set using the RUN_OPTS parameter: [ts_example] $ make run SET= RUN_OPTS="-t tc_example_01" Here the full test set is build, but by using the -t option the test master is instructed to run only the test case tc_example_01. This feature is helpful in debugging single test cases of long running test sets that do not support partial building. The -t also supports running parts of a test case: [ts_example] $ make run SET= RUN_OPTS="-t tc_example_01:3-8,10-15" This will build the test set fully, but execute only sub test cases 3 to 8 and 10 to 15 of test case tc_exmaple_01. In this case the result file generated by the test run will not contain objective results, but only sub results. 1.4.3 Manual Test Cases A test case might be marked as "‘manual"’ (see section 6.1.1.4 for details). To include test cases marked as manual the option INCLUDE_MANUAL_TC must be set to the value yes: [ts_example] $ make run SET= INCLUDE_MANUAL_TC=yes This will build and run the specified test set, including all test case marked as manual. Usually test cases are marked as manual if they require user interaction. For convenience test cases specified with the TC option are always included, no matter if marked as manual or not. c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. Identification 11 1.4.4 Specifying Labels The test framework supports the concept of labels. Labels are used to mark the association of test cases to a certain test set. If they are used on such an association it is only applicable if the label is configured (see section 6.1.1.4 for how to configure labels). By using labels it is possible to disable test cases completely or to put them into different test sets depending on configuration. Usually labels are configured statically in the testsuite.conf file. But it is possible to overwrite this configuration using the LABEL option: [ts_example] $ make build SET= LABEL=" " The value of the option LABEL is a white space separated list of the enabled labels. Note that the list of configured labels is replaced completely if this option is used. 1.4.5 Temporarily Changing PikeOS Project Configuration When building a test set it is possible to configure some aspects of the PikeOS projects used directly from the command line. This feature is usually used for debugging purposes. Supportedare the options APEX_OS_VARIANT, PIKEOS_COMPILATION_ASSERT, PIKEOS_COMPI- LATION_LIBGCC, PIKEOS_COMPILATION_VERBOSE, PIKEOS_COMPILATION_WARNING_STRICT, PIKEOS_COMPILATION_WERROR, PIKEOS_SMP, PIKEOS_TRACE, PSP_MEMSIZE, PSSW_COPY, UK_BOOT_MESSAGE, and UK_LOG_LEVEL. Their values are passed through directly to the corresponding option exposed in the PikeOS project configurator. This allows to select e. g. a multi-core kernel: [ts_example] $ make build SET= PIKEOS_SMP=true Please check the PikeOS documentation for a list of supported values. 1.5 Identification The test framework outputs a short identification of itself if the command tfw_version is called. Additionally a list of all installed test framework components is output. [ts_example] $ make tfw_version 1.6 Cleaning Up the Test Suite There are three different commands for clean-up: clean, distclean, and mrproper. These commands reliably clean up test framework generated files. If test suite provided tools create or modify additional files, additional steps might be necessary. 1.6.1 Removing Intermediate Files The command clean can be used to remove all intermediate files in the test suite: [ts_example] $ make clean Usually intermediate files are stored within the sys/ directory. This command does neither remove build logs, result logs, nor binaries. c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. 12 Basic Test Suite Usage 1.6.2 Resetting to Initial State The command distclean can be used to remove all files that did not exist in the test suite after preparation: [ts_example] $ make distclean Additionally to the actions performed by the clean command this command will remove build logs, result logs, and binaries, too. 1.6.3 Reverting Preparation The command mrproper can be used to remove all created files. Additionally it undoes the effect of the prepare command: [ts_example] $ make mrproper After this the test suite is in the same state as before the prepare command (see section 1.1) was executed. c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. 2 Overlays With overlays it is possible to add new files and directories to a test suite or to replace existing files and directories. Additionally it is possible to replace the content of the test frameworks contrib/ module and to make files available used in the PDF document generation process. These features directly correspond to the three types of overlays supported. These are: • The test suite specific overlay, which supports creation or replacement of files and directories in the test suites top level directory. The location of this overlay is configured using options named TE_OVER- LAY_*_PATH. • The test suites common overlay, which is used by the prepare command instead of the test frameworks contrib/ module to populate the sys/ directory. The location of this overlay is configured using options named TE_OVERLAY_*_COMMON_PATH. • The generic overlay providing files for PDF document generation only. The location of this overlay is configured using options named TE_*_COMMON_PATH. The three types of overlays can be used independently of each other. 2.1 Creating Overlays 2.1.1 The Directory Structure of an Overlay The general structure of an overlay is shown in Figure 1. In the example the overlay has the name “te_example”. The name can be arbitrary but is usually prefixed with “te_”. The overlay contains a common/ directory. This directory contains the generic overlay referenced to with the TE_*_COMMON_PATH. Additionally there are multiple directories named overlays.*/. Each of them contains one or more test suite specific overlays and the test suite common overlay for a certain architecture. In the example only the content of the directory overlays.$Y/ is shown. In its sub-directory named common the test suite common overlay is stored. Further it contains directories named like the test suites the overlay is for. These directories contain the test suite specific overlays. te_example/ + common/ + overlays.$X/ + overlays.$Y/ + common/ + ts_example/ + ts_example2/ + overlays.$Z/ Figure 1: The structure of an overlay. c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. 14 Overlays 2.1.2 The Content of the Test Suite Specific Overlay The test suite specific overlay has a very simplest structure. Its contains files and directories that shall be copied into the test suite. If the test suite contains already a file or directory with that name it is removed. Also this only works in the top level directory of the test suite. This means, it is possible to replace the entire config/ directory of the test suite, but not single files within it. The most common usage of this type of overlay is replacing the directories config/ and verify/ of the test suite. 2.1.3 The Content of the Test Suite Common Overlay The content of this overlay is used as starting content of the sys/ directory in the test suite. Most importantly it contains the seed of application and integration projects used by the test suite in its sub-directory seed/. It is also possible to store target.* files in the top level directory of a test suite common overlay. The test framework will use them unless a file with identical is also provided in the config/ directory of the test suite. Sometimes this type of overlay is used to provide files which are need by multiple test suites of a project. This is a supported use case. 2.1.4 The Content of the Generic Overlay The content of this type of overlay is used for PDF document generation only. Also it is not made available directly in the test suite but in a sub-directory of extern/. See section 4.3 for a description of which files are supported in it. 2.2 Using Overlays Overlays are applied to a test suite during preparation (see section 1.1). For this the location of the overlay to use is passed as option to the prepare command. The test framework supports drawing the overlay from a TAR archive, from CVS repository, and from a GIT repository. To use an overlay from a TAR archive the following options must be passed to the prepare command: TE_OVERLAY_TAR_FILE: This option specifies the path to the tar file containing the overlay. Usage of this option is mandatory if an overlay from a TAR file is used. TE_OVERLAY_TAR_PATH: This option is used to specify the location of the test suite specific overlay within the tar file. This option is optional. TE_OVERLAY_TAR_COMMON_PATH: This option is used to specify the location of the test suite common overlay within the tar file. This option is optional. TE_TAR_COMMON_PATH: This option specifies the location of the generic overlay within the tar file. This option is optional. For overlays from CVS the following options must be passed to the prepare command: TE_OVERLAY_CVS_REPO: This option specifies the name of the CVS repository containing the overlay. It will be passed directly to the cvs command and must contain everything needed by it (e. g. the user name) to identify the repository. Usage of this option is mandatory if an overlay from a CVS repository is used. c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. Using Overlays 15 TE_OVERLAY_CVS_BRANCH: This option specifies the branch in the CVS repository. It is optional. If it is not specified CVS HEAD is used. TE_OVERLAY_CVS_PATH: This option is used to specify the location of the test suite specific overlay within the CVS repository. This option is optional. TE_OVERLAY_CVS_COMMON_PATH: This option is used to specify the location of the test suite common overlay within the CVS repository. This option is optional. TE_CVS_COMMON_PATH: This option specifies the location of the generic overlay within the CVS repository. This option is optional. For overlays from GIT the following options must be passed to the prepare command: TE_OVERLAY_GIT_REPO: This option specifies the name of the GIT repository containing the overlay. Usage of this option is mandatory if an overlay from a GIT repository is used. TE_OVERLAY_GIT_BRANCH: This option specifies the branch in the GIT repository. It is optional. If it is not specified the "master" branch is used. TE_OVERLAY_GIT_PATH: This option is used to specify the location of the test suite specific overlay within the GIT repository. This option is optional. TE_OVERLAY_GIT_COMMON_PATH: This option is used to specify the location of the test suite common overlay within the GIT repository. This option is optional. TE_GIT_COMMON_PATH: This option specifies the location of the generic overlay within the GIT repository. This option is optional. c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. 3 Test Set Pools Test set pools enable the reuse of test sets in multiple test suites without error-prone manual copy-and-paste. A test set pool is separate from the test suite, and provides additional test set to the test suite. The following sections describe (1) the layout of the directory hierarchy to be used as a test set pool, (2) how to work with test set and test cases from the test set pool, and (3) how to interface with a test suite. 3.1 Creating Test Set Pools 3.1.1 Naming conventions Test set in a pool can be saved at arbitrary (at least two) directory levels. Valid directory names start with a letter and do not contain dashes, dots or spaces. 3.1.2 Directory Layout of a Test Set Pool A test set contains at least a sysconf.rc file. See Section 5 for a detailed explanation. A subdirectory tc/ contains the test cases comprising this test set. testsetpool/ + subdir1/ + subdir2/ + subsubdir2/ + description.xml + int.conf + snip.xml + sysconf.rc + tc/ + testcase_name_starting_with_a_letter_01/ + tc.xml + tools/ 3.1.3 The Test Case XML Description File V2.1 Test cases in the pool differ from traditional test cases (as described in Section 6) in the way that no requirements are referenced. Therefore, the V2.1-format of the XML description is missing those parts. The test case description comprises the following parts: • Objectives of the test case • A description part using a subset of HTML to explain: Setup and constraints c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. Using Test Set Pools 17 ∗ Test setup (reference to a setup described in the Test Case Master document ∗ Additional constraints on the test environment Test design Test procedure • Test input(s) (e.g. equivalence class definition) Figure 2 shows a template for a test case description to be filled by the test case developer. The XML document uses several elements. The root of the document is the element. Its attributes are described in section 6.1.1. 3.1.3.1 The and Elements The element contains a non-empty list of elements. Each of them describes exactly one test objective. The name of the test objective is given in the elements id attribute. The test objective itself is described in the text content of the element. 3.1.3.2 The Element After the element there may be an optional element. It can be used to configure different aspects of the test framework. In the example the maximum time the test master waits for messages from the slave is set to 30 seconds (default is 10) and the maximum number of non-protocol messages is set to 1000. See section 9 for a full list of tags supported. 3.1.3.3 The Element The format of the test case description is described in detail in section 6.1.1.6. 3.1.3.4 The Element Equivalence classes are an optional part of the test case XML file described in section 7. 3.2 Using Test Set Pools Test set pools are applied to a test suite during preparation (see section 1.1). There are four variants to specify the source of a test set pool: a git repository, a cvs repository, a tar file, or a local directory. To use a test set pool from a tar file, the following option is supported by the prepare command: TFW_POOL_TAR_FILE: This option specifies the path to the tar file containing the test set pool. To use a test set pool from a cvs repository, the following options are supported by the prepare command: TFW_POOL_CVS_REPO: This option specifies the name of the CVS repository containing the test set pool. It will be passed directly to the cvs command and must contain everything needed by it (e. g. the user name) to identify the repository. Usage of this option is mandatory if a test set pool from a CVS repository is used. c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. 18 Test Set Pools 1 2 11 12 13 description of objective EXAMPLE1 14 description of objective EXAMPLE2 15 16 17 18 19 20 21 22 23

Test Setup

...

24

Design

25

Preparation

...

26

Test

...

27

Cleanup

...

28 29

Equivalence Classes

...

30 31 32 33 [...] 34 35 36 Figure 2: Example for a test case description XML V2.1 c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. Support for Multiple Test Set Pools 19 TFW_POOL_CVS_MODULE: This option specify the CVS module to check out from the repository. This option is mandatory if a test set pool from a CVS repository is used. TFW_POOL_CVS_BRANCH: This option specifies the branch in the CVS repository. It is optional. If it is not specified CVS HEAD is used. To use a test set pool from a git repository, the following options are supported by the prepare command: TFW_POOL_GIT_REPO: This option specifies the name of the GIT repository containing the test set pool. Usage of this option is mandatory if an overlay from a GIT file is used. TFW_POOL_GIT_BRANCH: This option specifies the branch in the GIT repository. It is optional. If it is not specified the "master" branch is used. To use a test set pool from a local directory, the following option is supported by the prepare command: TFW_POOL_LOCAL_DIR: This option specifies the local directory containing the test set pool. Usage of this option is mandatory if a test set pool from a local directory is used. There are some common options for test set pools: TFW_POOL_TARGET: The test set from the pool are checked out to ./pool/. The default pool name is either the git repository name, the cvs module name, the basename of the tar file, or the lo- cal directory name. This option can be used to explicitly specify a target directory, where to place the test set relative to ./pool/. Setting this option to "arbitrary/path" results in a checkout to "pool/arbitrary/path". KEEP_VCS: By default the test set pool is checked out without version control files. Set this option, to be able to modify and commit files to the source repository. TFW_POOL_PATHS: By default the complete test set pool is checked out. This option can be specified, to limit the checkout to specific paths of the pool. Multiple paths are separated by white space. 3.3 Support for Multiple Test Set Pools A test suite can make use of multiple test set pools. This requires a Makefile, e.g. devel/Makefile.install to specify the target specific variable values for each test set pool. The variable TFW_POOLS specifies the target directories of the test set pools relative to the test suite directory. E.g. TFW_POOLS="pool/cvspool pool/gitpool" For each of the test set pool targets, the corresponding options are specified as target-specific makefile variables. E.g. 1 pool/cvspool: TFW_POOL_CVS_REPO := :ext:$(CVSUSER)@cvs.sysgo.com:/home/keeper/repos/00101-p4 2 pool/cvspool: TFW_POOL_TARGET := cvspool 3 pool/cvspool: TFW_POOL_CVS_MODULE := cert/tp 4 5 pool/gitpool: TFW_POOL_GIT_REPO := git@server:path/on/server/pool 6 pool/gitpool: TFW_POOL_TARGET := gitpool c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. 20 Test Set Pools It is important to specify TFW_POOL_TARGET to the same value as the target’s subdirectory in ./pool/, if the pool repository has a different name. 3.4 Automatic Generation of Test Set Pool Variables from Trace-XML-files Test suites specify the usage of test set from the pool via xml files in trace/ and verify/trace/ directories. Those xml files must comply to the http://www.sysgo.com/xsd/tfw/rqtrace-2.1.xsd. In those xml files, test cases coming from a test set pool are named by their dot-names, e.g. cvspool.subdir2.subsubdir2-testcase_name_starting_with_a_letter_01 Upon generation of the test suite the rqtrace-files are parsed for test cases from test set pools, i.e. counting dots. An appropriate Makefile is generated, which contains TFW_POOL_PATHS set for each identified test set. pool/cvspool: TFW_POOL_PATHS += subdir2/subsubdir2 Different pools are distinguished by the name of the pool, as used in TFW_POOL_TARGET, which is also the first directory part of the test set name. This implies that the appropriate variables have to be set, where to get the test set pool from, by means of either global options 3.2 or target specific makefile variables 3.3 as described in the corresponding sections above. When using trace-XML-files, the test framework automatically removes unused test cases from the checkout pool. To avoid removal of an otherwise unreferenced test case, append it to the variable TFW_POOL_TEST_CASE_LIST as follows: pool/cvspool: TFW_POOL_TEST_CASE_LIST += subdir2/subsubdir2/tc/tc_01 c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. 4 Verification Result Artifacts The following sections describe how to create verification result artifacts, including how to modify the inputs. 4.1 Artifact Creation This section describes how to create the artifacts if the test suite is already prepared for it. See Sections 4.2 and 4.3 for what must be set up before the commands described here are available. 4.1.1 Building the Test Case Master Document The test case master document is the artifact containing the description of test approaches, test sets, test cases, test tools, and everything else necessary to understand how the requirements under test are tested. For each test case it lists which requirements are tested. It does neither contain test results nor target specific data. This information is provided by the test result and the structural coverage result document. Usually, we refer to the test case master document as the TC document. To build the test case master document it is sufficient to call the tc command of the test framework: [ts_example] $ make tc This will create the TC document as HTML. It is also possible to create a PDF variant of the TC document. For this the option PDF must be set: [ts_example] $ make tc PDF=yes With this command the HTML and the PDF variant of the TC document will be created. The test framework differs between the “notrace”, the “specific”, and and the “generic” test case master document. The difference between them is which test cases and test sets are described and if trace data is contained. If the test suite uses either a test pool or manages trace data in the trace/ or verify/trace/ directory, only the “notrace” document is supported. It contains no trace data and only test cases and test sets that are currently enabled by labels (see section 6.1.1.4 for details on labels). If the test suite neither uses a test pool nor manages trace data in the trace/ or verify/trace/ directory the “specific” and “generic” documents are available, too. In this case the “specific” document is generated by default. But the “notrace” document is still available by setting the option TYPE accordingly. The “specific” document contains the same test cases and test sets as the “notrace” document. Additionally, it also contains trace data. This means it explains which requirements are tested by objectives of the test cases. By setting the option TYPE to generic the generic TC document can be created: [ts_example] $ make tc PDF=yes TYPE=generic It contains the description of all test cases and test sets that would be selected if all labels configured in the file doc/labels.xml were enabled. I. e. this call is equivalent to [ts_example] $ make tc PDF=yes LABEL="" c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. 22 Verification Result Artifacts This feature is usually used if multiple test environments are used within a single project: the generic test case master document lists all existing test cases and test sets and the test environment specific test result document specifies which of them are applicable. By this a single TC document can be used for all test environments. For development it is also possible to create a TC document containing only the description of specific test cases and test sets using the options TC and SET when calling the tc command. For example [ts_example] $ make tc PDF=yes TC=tc_example_01 will create a TC document containing only the description of the test case tc_example_01. 4.1.2 Building the Test Result Document The test result document is the artifact containing the test results for the requirements under test. Except for the introduction it is often completely auto-generated by the test framework. Usually, we refer to the test result document as the "‘TR"’ document. Before the test result document can be built test results must be available. Test results are taken from the result log created by the test framework when a test set is run (see section 1.4). So let’s assume result logs for all test sets of the test suite are available. Then the test result document can be built using the tr command: [ts_example] $ make tr BOARD= Similar to the tc command this will create the HTML variant of the test result document. But there is one important difference: When building the test result document the board must be specified (if it is different from the default board). The reason for this is that the test environment must be known to decide upon unsupported requirements. The option PDF can be used to create the PDF variant of the test result document, too: [ts_example] $ make tr BOARD= PDF=yes After creation of the document it must be checked that the test results listed in the document are as expected. Especially it must be checked that for all requirements with test result R_ERROR or R_FAIL a justification is given. If a justification is missing the reason for this must be investigated. Basically there are four possibilities: System under Test is broken The test suite detected a new, unknown error in the system under test. This must be reported (e. g. by creating a problem report). Then a justification referring to the report is added to the test result document. Test suite is broken The source of the error is in the test suite. In this case it must be fixed. After this the document creation process can be restarted. Temporary target error There are many sources for temporary errors, which are neither related to the system under test nor the test suite. This could be network errors like failing TFTP servers and unreachable license servers. In this case the process must be redone starting at the point where the temporary error happened. User error The user made an error in the process. A common error is a missing or wrong BOARD option. In this case the process must be redone, starting with the faulty command. 4.1.3 Building the Structural Coverage Result Document The structural coverage result document is the artifact containing detailed information upon the achieved structural coverage. We refer to this document as the “SC” document. Structural coverage is measured using test cases. The test results for these requirements are often added to an appendix of the structural coverage result document. c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. Artifact Setup in XML 23 They can be compared to the test results listed in the test result document. If the results are identical it can be assumed that the instrumentation process didn’t introduce errors. Before the structural coverage result document can be build result logs containing structural coverage data must be available and a report must have been generated. Then the structural coverage result document can be generated using the sc command: [ts_example] $ make sc BOARD= COV_TOOL= This will generated the HTML variant of the structural coverage result document. Again the option BOARD is necessary to decide upon unsupported requirements. The option COV_TOOL is used to specify the name of the tool used for measuring structural coverage. A PDF variant of the document can be created by setting the PDF option: sc command: [ts_example] $ make sc PDF=yes BOARD= COV_TOOL= After creation of the document it must be checked that the content is as expected: Full structural coverage for the system under test is needed. Any deviation must be justified. Test results for all requirements must be identical to the ones listed in the test result document. Any deviation must be qualified. If the content of the document is not as expected the reason for this must be investigated. Possible reasons include: Justifications are not attached correctly to uncovered code Justifications are attached to uncovered code using line numbers. Often even non-functional code changes affect the line numbers. In this case justi- fications will be missing in the structural coverage document. To fix this the line numbers must be updated and the report must be regenerated. After this a fixed structural coverage result document can be gener- ated. Requirement results are different from the ones listed in the TR document and no justification is given Apart from the possible reasons listed in section 4.1.2 it can happen that the test cases did detect the code changes introduced by instrumentation. A justification for this must be added to the document. Especially it must explain what causes the deviation and why this is not a problem for measuring structural coverage. Usually this is done in a separate section in the document. Temporary target error There are many sources of temporary errors that can happen during testing, which are neither related to the system under test nor the test suite. This could be network errors like failing TFTP servers and unreachable license servers. In this case the process must be redone starting at the point where the temporary error happened. User error The user made an error in the process. A common error is to forget to specify the BOARD option. In this case the process must be redone starting at the faulty command. 4.2 Artifact Setup in XML This section describes how to setup creation of artifacts using XML files. This is used for the first step of document creation. 4.2.1 XML Document Fragments This section describes the building blocks of the documents. The source of all documents is maintained in XML files. The test framework automatically creates various additional XML files based on e. g. current test results, c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. 24 Verification Result Artifacts test cases, and test environment. A mechanism to include these files into the document is provided. E. g. to build the test case master document a file doc/tc.xml must exist. In Figure 3 on page 25 a minimal variant of this file is shown. Using XSLT scripts it is converted to an HTML and a TeX document. The elements of the http://www.sysgo.com/tcm namespace used by this document have the following semantics: The element can be used to specify the names and the functions of the persons which are going to approve the document. It’s content is a nonempty set of elements. This element may be used as document root. The purpose of the optional element is to specify documents which shall always be added to the list of referenced documents. Even, if they are not used explicitly by the document (in which case they are added automatically). The individual documents are specified using child elements of type . This element may be used as document root. The element has a single attribute named ’key’. It’s value is the BibTeX key of a document that shall always be added to the list of referenced documents. This element is always the child of a element. A element contains in its context a single entry of the revision history as text. This element is always the child of a element. The element can be used to point to an XML file with any of the elements listed as “may be used as document root” here as root element. The file will be interpreted as if the ele- ment was replaced by the root element of the file pointed to. The file name must be specified relative to the file that contains the element. So if doc/tc.xml contains a element refer- ring ../sys/testsets.xml the file sys/testsets.xml is included. This element is always the child of a , a , or a element. Elements of type contain the actual text content. Allowed content are HTML code and elements only. This element may be used as document root. The elements of type may contain elements of type , , , , , , and . This element may be used as document root. The elements of type may contain elements of type , , and . The effect of the element is that the level of every HTML heading defined inside (either directly using or within a file included using is increased by one. This element is always the child of a , a , or a element. The element is used to specify the revision history of the document. Its content are elements of type , which contain the revision data for a single revision. This element may be used as document root. The element can be used inside a to provide LaTeX code which is copied verbatim to the TeX file. It is ignored completely for HTML output. In the example it is used to output the bibliography and to mark the beginning of the appendix. The element is the root element of the entire document. It’s content may be elements of the types , , , , , , and . All verification result documents are created using these document fragments. c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. Artifact Setup in XML 25 1 2 8 9 10 11 12 13 14 15 16 17 18 19 20 ... 21 22 23 24 25 Initial revision 26 27 28 29 30 31 32 33 34

Introduction

... 35 \makeBib 36 37 38 39

Overview

... 40 41 42 43 44 45 46 47 48 49 50 \appendix 51 52 53 54 Figure 3: Minimal Content of doc/tc.xml c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. 26 Verification Result Artifacts The element is used to specify the value of a single document property shown on the title page. It has the attributes ’key’ and ’value’. The following values for the ’key’ attribute are supported: Author To specify the author of the document. Date To specify the date of the document. The special value “today” is replaced with the current date. ID To specify the document ID. Revision To specify the revision of the document. State To specify the approval state of the document. Title To specify the title of the document. This element is always the child of a element. The element is used to specify the document properties which will show up on the title page of the PDF document generated. The individual properties are defined using child elements. These are the title, the ID, the revision and date, the state, and the author. If the value attribute of the date property has the value “today” it is replaced with the current date. The element specifies the revision history for a single revision. It has the attributes “revision” and “date”. The “revision” attribute contains the name of the revision. In the “date” attribute the date is given. Here the special value “today” is replaced with the current date. The actual revision history entries are provided in subelements of type . This element is always the child of a element. The element is used to specify name and function of a single signee. It has the attributes “function” and “name”. The “function” attribute is used to specify the role of the signee. Supported roles are Author, Functional Lead, Quality Assurance, Verifier, and Validator. The “name” attribute is used to specify the full name. This element is always the child of a element. 4.2.2 Setup of the Test Case Master Document The main entry point for the test case master document is the file doc/tc.xml. An example is shown in Figure 3. Usually the document explains after a short introduction the test approaches used. Then the description of the individual test cases and test sets is given. The following sub-sections describe the usual content of doc/tc.xml and how to modify it. 4.2.2.1 The File doc/tc.xml The file doc/tc.xml usually defines four sections only. All other content is encapsulated inside elements. By this it becomes a part of the previously defined section. Introduction This is the first section of the document. It describes the purpose of the document and contains the list of referenced documents. Overview This section is used to give a short overview of the test suite and to explain the test approaches used. In this section the file sys/labels_available.xml is included. It is automatically generated by the test framework. It contains a short description (see section 4.2.2.2) of all test case labels available in the test suite. c Copyright 2005 – 2019 SYSGO GmbH, all rights reserved. Artifact Setup in XML 27 1 2 3 4 8 9 13 14 Figure 4: Example for doc/labels.xml In the example the file doc/integration-projects.xml is included, too. It is used to explain which application and integration projects are used by the test suite. For PikeOS test suites this is a mandatory part of the test case master document, which is needed to understand the test approaches. none After this the file sys/tc_body.xml is included. It contains one or multiple sections describing all test cases and test sets of the test suite. This file is automatically generated by the test framework. Appendix The LaTeX command to switch to the appendix is emitted. After this the file sys/tc2set.xml is included. It contains a table associating test cases with test sets. 4.2.2.2 Configuring Label Descriptions For the test case master document a description of all labels in use is needed. The user provides this description in the file doc/labels.xml (see Figure 4 for an example). For each label in use a