portugalfuturista/universalisos
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
universalisos/docs/TESTING_GUIDELINES.md

10 KiB
Raw Blame History

UniversalisOS Testing Toolchain Manual

This document is the official manual for the UniversalisOS testing toolchain. It replaces generic testing guidelines with concrete instructions for the custom-built tools already present in the codebase.

1. Toolchain Overview

The UniversalisOS testing ecosystem consists of three integrated components:

Component Tool Purpose Location
Test Execution uos-target.py Compile tests into kernel payload, run via QEMU, parse UART results tools/uos-target/
Code Coverage uos-cover suite C instrumentation, trace parsing, MC/DC/branch/statement coverage tools/uos-cover/
Requirements Doorstop + Plane HLR management (Doorstop) and LLR management (Plane) tools/doorstop-integration/

2. Test Execution: uos-target.py

Purpose

uos-target.py manages the complete test execution pipeline:

  1. Compiles tests directly into the kernel payload/firmware
  2. Launches QEMU with the instrumented kernel
  3. Parses pass, fail, and skip results directly from UART output
  4. Generates JUnit XML reports for CI integration

Usage

# Run all tests for default target (qemu-arm-virt)
python tools/uos-target/uos-target.py --run

# Run tests for specific architecture
python tools/uos-target/uos-target.py --arch armv7 --platform qemu-arm-virt --run

# Run with specific test filter
python tools/uos-target/uos-target.py --run --filter "scheduler_*"

# Generate JUnit XML for CI
python tools/uos-target/uos-target.py --run --junit results.xml

# Run with coverage instrumentation enabled
python tools/uos-target/uos-target.py --run --coverage

Test Output Format

Tests embedded in the kernel must output results in this exact format:

[UOS-TEST] <test_name>: <pass|fail|skip> [<message>]

Example UART output:

[UOS-TEST] scheduler_round_robin: pass
[UOS-TEST] scheduler_preemption: pass
[UOS-TEST] memory_alloc_null: fail [null pointer not caught]
[UOS-TEST] ipc_message_size: skip [not implemented on this platform]

Configuration

uos-target.py reads target definitions from tools/uos-target/targets/:

# tools/uos-target/targets/qemu-arm-virt.yaml
name: qemu-arm-virt
arch: armv7
platform: qemu-arm-virt
qemu_binary: qemu-system-arm
qemu_args:
  - -M virt
  - -cpu cortex-a15
  - -m 512M
  - -nographic
uart_timeout: 10
banner: "UniversalisOS"

3. Code Coverage: uos-cover Suite

Purpose

The uos-cover suite provides precise, bare-metal ARINC/ISO 26262 compliant coverage analysis for firmware builds. It achieves MC/DC, branch, and statement coverage through source-level C instrumentation.

Tools

Tool Command Purpose
uos_cins.py python tools/uos-cover/uos_cins.py Instrument C/C++ source before compilation
uos_cover.py python tools/uos-cover/uos_cover.py Main coverage orchestrator
uos_covparse.py python tools/uos-cover/uos_covparse.py Parse trace data into coverage reports
uos_trace.py python tools/uos-cover/uos_trace.py Parse raw trace files from instrumented runs
uos_justify.py python tools/uos-cover/uos_justify.py Justify unreachable code paths
uos_verify.py python tools/uos-cover/uos_verify.py Verify coverage against requirements
uos_covexport.py python tools/uos-cover/uos_covexport.py Export coverage data (HTML, CSV, JSON)
uos_package.py python tools/uos-cover/uos_package.py Package coverage artifacts for submission
uos_xst.py python tools/uos-cover/uos_xst.py Cross-source traceability analysis

Coverage Workflow

# 1. Instrument source code
python tools/uos-cover/uos_cins.py \
    --input kernel/src/ \
    --output kernel/src_instrumented/

# 2. Build instrumented firmware
cd kernel
make ARCH=armv7 PLATFORM=qemu-arm-virt \
    SRC_DIR=src_instrumented/

# 3. Run tests with coverage (uos-target handles trace extraction)
python tools/uos-target/uos-target.py --run --coverage

# 4. Parse trace data
python tools/uos-cover/uos_covparse.py \
    --trace build/trace.txt \
    --output coverage.json

# 5. Verify against requirements
python tools/uos-cover/uos_verify.py \
    --coverage coverage.json \
    --requirements tools/doorstop-integration/

# 6. Generate HTML report
python tools/uos-cover/uos_covexport.py \
    --input coverage.json \
    --format html \
    --output coverage_report/

# 7. Package for submission
python tools/uos-cover/uos_package.py \
    --input coverage_report/ \
    --output coverage_package.zip

Coverage Levels (DO-178C / ISO 26262)

Level Metric Threshold Tool
Statement % of executable statements 100% uos_covparse.py
Branch % of decision branches 100% uos_covparse.py
MC/DC Modified Condition/Decision 100% uos_covparse.py --mcdc
Function % of functions called 100% uos_covparse.py

Justifying Unreachable Code

For code that cannot be exercised (e.g., hardware-failure paths, defensive checks):

python tools/uos-cover/uos_justify.py \
    --file kernel/src/core/scheduler.cpp \
    --line 245 \
    --reason "Unreachable: watchdog timeout triggers system reset before this path" \
    --category "hardware_failure"

Justifications are stored in .uos_justifications.json and included in verification reports.

4. Requirements Traceability

Higher-Level Requirements (HLR)

HLRs are managed by Doorstop at tools/doorstop-integration/:

# Create new HLR
cd tools/doorstop-integration
python doorsetup.py --create HLR --id REQ-006 \
    --text "The scheduler shall support round-robin partitioning"

# Validate all requirements
python doors_export.py --validate

# Generate traceability matrix
python doors_export.py --format csv --output trace_matrix.csv

HLRs are viewed and edited via:

  • Web: janela-do-desassossego-web (DOORS-like grid at /doors)
  • IDE: aurelio-theia sidebar widget (HLR Requirements view)
  • VS Code: aurelio-vscode panel (Aurelio: Open HLR Requirements command)

Lower-Level Requirements (LLR)

LLRs are managed in Plane (plane.portugalfuturista.org) and tied directly to the CI execution pipeline.

Bidirectional Sync

# Push HLRs to Plane (create/update LLRs)
python plane_bridge.py --sync

# Pull LLR status from Plane
python plane_bridge.py --pull --format yaml

5. CI Integration

GitHub Actions Pipeline

The CI pipeline runs all three toolchain components in sequence:

# .github/workflows/ci.yml (relevant jobs)
jobs:
  uos-target-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run firmware tests
        run: python tools/uos-target/uos-target.py --run --junit test-results.xml
      - name: Upload test results
        uses: actions/upload-artifact@v4
        with:
          name: test-results
          path: test-results.xml

  uos-cover-analysis:
    needs: uos-target-test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Instrument source
        run: python tools/uos-cover/uos_cins.py --input kernel/src/ --output kernel/src/
      - name: Build with coverage
        run: cd kernel && make ARCH=armv7 PLATFORM=qemu-arm-virt
      - name: Run tests with coverage
        run: python tools/uos-target/uos-target.py --run --coverage
      - name: Parse coverage
        run: python tools/uos-cover/uos_covparse.py --trace build/trace.txt --output coverage.json
      - name: Verify coverage
        run: python tools/uos-cover/uos_verify.py --coverage coverage.json --requirements tools/doorstop-integration/
      - name: Export HTML report
        run: python tools/uos-cover/uos_covexport.py --input coverage.json --format html --output coverage/
      - name: Upload coverage
        uses: actions/upload-artifact@v4
        with:
          name: coverage-report
          path: coverage/

  requirements-coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Validate Doorstop requirements
        run: python tools/doorstop-integration/doors_export.py --validate
      - name: Check traceability
        run: python tools/doorstop-integration/doors_export.py --check-links

6. Quick Reference

Running Tests Locally

# Full test suite with coverage
python tools/uos-target/uos-target.py --run --coverage

# Specific architecture only
python tools/uos-target/uos-target.py --arch riscv --run

# Quick smoke test (no coverage)
cd kernel && ./uos-check.sh test

Coverage Commands

# Instrument → Build → Test → Report (full pipeline)
python tools/uos-cover/uos_cover.py --full-pipeline --arch armv7

# Justify unreachable code
python tools/uos-cover/uos_justify.py --file <file> --line <line> --reason "<reason>"

# Export coverage for specific requirement
python tools/uos-cover/uos_covexport.py --input coverage.json --requirement REQ-001 --format html

Requirements Commands

# View requirements in browser (Doorstop backend must be running)
curl http://192.168.0.9:8100/api/v1/requirements

# Validate
python tools/doorstop-integration/doors_export.py --validate

# Sync with Plane
python tools/doorstop-integration/plane_bridge.py --sync

7. File Locations

Path Purpose
tools/uos-target/ Test execution engine (QEMU orchestration + UART parsing)
tools/uos-cover/ Coverage instrumentation and analysis suite
tools/doorstop-integration/ HLR requirements management (Doorstop backend)
tools/uos-pkg/ Package build tooling
kernel/uos-check.sh Legacy build + boot smoke tests
.github/workflows/ci.yml CI pipeline configuration

8. References