universalisos/docs/TESTING_GUIDELINES.md

UniversalisOS Testing & Quality Assurance Guidelines

As a safety-critical Type-1 hypervisor implementing PikeOS 5.0 patterns, UniversalisOS requires a rigorous, standards-compliant approach to testing. This document defines the strategies for requirements traceability, test creation, execution, and code coverage to align with high-assurance standards (e.g., DO-178C, ISO 26262).

---

1. Requirements Engineering & Traceability

Current State

• Tooling: Doorstop integration exists at tools/doorstop-integration/ with 5 HLRs (REQ-001.yml through REQ-005.yml).
• CI validation: The requirements-coverage GitHub Actions job validates Doorstop requirements and generates traceability matrices.
• Plane integration: plane_bridge.py provides bidirectional sync between Doorstop and Plane for LLRs.

Higher-Level Requirements (HLR)

HLRs describe what the system must do from a system integration perspective.

• Management: HLRs are managed via Doorstop (tools/doorstop-integration/).
• User Interface: Requirements engineers interact with HLRs via janela-do-desassossego-web (tabular UI) or IDE integrations in aurelio-theia/aurelio-vscode.
• Traceability: Every HLR must trace down to LLRs. Test engineers read HLRs, compare them with test framework keywords, and implement in-firmware tests directly targeting the requirements.

Lower-Level Requirements (LLR)

LLRs describe how the HLRs are implemented at the software and architecture level.

• Management: LLRs are managed in Plane (plane.portugalfuturista.org).
• Traceability: Every LLR must trace back to an HLR, and forward to specific in-firmware test cases executing within the CI pipeline.

Adding New Requirements

cd tools/doorstop-integration
python3 doorsetup.py --create HLR --id REQ-006 --text "Description of requirement"
python3 plane_bridge.py --sync  # Push to Plane for LLR management

---

2. Test Framework Architecture

Current State

UniversalisOS uses a dual-layer testing approach:

Layer	Tool	Status	Purpose
Host-side tooling	pytest suites	✅ Operational	Test the build/coverage/packaging tools
Target firmware	uos-check.sh	✅ Partial	Build verification + boot smoke tests
Target firmware	In-kernel test harness	❌ Not implemented	Structured firmware tests
Code coverage	uos-cover suite	✅ Operational	Instrumentation + coverage analysis
Test orchestration	uos-target	⚠️ Misnamed	Currently a target-definition validator, not a test runner

Host-Side Testing (Tooling)

The tools/ directory contains well-tested Python tooling:

Tool	Test Files	Coverage
tools/uos-cover/	8 pytest files + 1 C unit test	Instrumentation, trace parsing, justification, export
tools/uos-pkg/	3 pytest files	Package build, spec parsing, RPM format
tools/doorstop-integration/	CI validation	Requirements traceability

Run host-side tests:

# All tool tests
python -m pytest tools/uos-cover/test/ tools/uos-pkg/test/ -v

# Specific tool
python -m pytest tools/uos-cover/test/ -v

Target Firmware Testing (Current)

The kernel/uos-check.sh script provides build + boot verification:

cd kernel
./uos-check.sh test          # Build all architectures + boot smoke tests
./uos-check.sh test-armv7    # Build + boot ARMv7 only
./uos-check.sh test-aarch64  # Build + boot AArch64 only
./uos-check.sh test-riscv    # Build + boot RISC-V only
./uos-check.sh lint          # Static analysis (if available)
./uos-check.sh build-all     # Build all architectures (no boot)

What uos-check.sh test does:

1. Compiles the kernel for the target architecture
2. Launches QEMU with the compiled ELF
3. Waits up to 10 seconds for UART output
4. Greps for the "UniversalisOS" banner string
5. Reports PASS/FAIL based on banner detection

Limitations:

• Only checks for boot banner (no functional test validation)
• No structured pass/fail/skip keyword parsing
• No test result persistence or reporting
• Not integrated into CI (runs locally only)

Target Firmware Testing (Planned)

A structured in-kernel test harness is needed for DO-178C compliance. See Section 8: Roadmap.

---

3. Creating Test Cases

Host-Side Tool Tests

Host-side tests follow standard pytest conventions:

# tools/uos-cover/test/test_uos_trace.py example
import pytest
from uos_trace import parse_trace_file, generate_matrix

def test_parse_trace_file():
    """Verify trace file parsing handles valid input."""
    result = parse_trace_file("fixtures/valid_trace.txt")
    assert result is not None
    assert len(result.requirements) > 0

def test_generate_matrix():
    """Verify traceability matrix generation."""
    matrix = generate_matrix(requirements, test_cases)
    assert matrix覆盖率 >= 0.0  # Basic sanity check

Run:

python -m pytest tools/uos-cover/test/test_uos_trace.py -v

Firmware Test Cases (Current)

Firmware tests are currently inline smoke tests embedded in kernel source:

// kernel/src/core/kernel.cpp — inline demo test
void partition_lifecycle_test() {
    uart_puts("[TEST] partition_lifecycle: starting\n");
    // ... test logic ...
    uart_puts("[TEST] partition_lifecycle: pass\n");
}

Test output format:

[TEST] <test_name>: pass
[TEST] <test_name>: fail
[TEST] <test_name>: skip

Firmware Test Cases (Planned)

For DO-178C compliance, tests should be structured as:

// kernel/src/test/test_scheduler.cpp (planned)
#include "uos_test.h"

UOS_TEST(scheduler_round_robin) {
    // Arrange: create 3 partitions with equal priority
    // Act: run scheduler for 100 ticks
    // Assert: each partition ran approximately 33 ticks
    UOS_ASSERT(partition_a_ticks >= 30 && partition_a_ticks <= 36);
    UOS_ASSERT(partition_b_ticks >= 30 && partition_b_ticks <= 36);
    UOS_ASSERT(partition_c_ticks >= 30 && partition_c_ticks <= 36);
}

UOS_TEST(scheduler_preemption) {
    // Arrange: create high-priority and low-priority partitions
    // Act: high-priority partition runs
    // Assert: low-priority partition is preempted
    UOS_ASSERT(preemption_occurred == true);
}

Exhaustive Corner Cases

Tests must cover all corner cases, including:

• Integer overflow/underflow for all data types
• Memory boundary conditions (null pointers, buffer overflows)
• Race conditions in concurrent operations
• Error handling paths (invalid inputs, resource exhaustion)
• Hardware edge cases (device timeout, interrupt storms)

---

4. Running the Test Suite

Host-Side Tests (CI/CD)

Automated via GitHub Actions (.github/workflows/ci.yml):

# Already configured in CI:
- name: Run uos-cover tests
  run: python -m pytest tools/uos-cover/test/ -v

- name: Run uos-pkg tests
  run: python -m pytest tools/uos-pkg/test/ -v

- name: Validate requirements
  run: python tools/doorstop-integration/doors_export.py --validate

Target Firmware Tests (Manual)

cd kernel

# Quick smoke test (all architectures)
./uos-check.sh test

# Architecture-specific
make ARCH=armv7 PLATFORM=qemu-arm-virt
qemu-system-arm -M virt -cpu cortex-a15 -m 512M \
  -nographic -kernel build/armv7/qemu-arm-virt/universalisos.elf

# Watch for test output
# Press Ctrl+A then X to exit QEMU

Target Firmware Tests (CI - Planned)

See Section 8: Roadmap for CI integration of boot tests.

---

5. Code Coverage Analysis: uos-cover

Current State

The uos-cover suite is operational with 10 Python modules and comprehensive tests.

Coverage Tools

Tool	Purpose	Status
uos_cins.py	Instrument C/C++ source before compilation	✅ Working
uos_trace.py	Parse trace data from instrumented runs	✅ Working
uos_covparse.py	Generate coverage reports	✅ Working
uos_justify.py	Justify unreachable code paths	✅ Working
uos_verify.py	Verify coverage against requirements	✅ Working
uos_covexport.py	Export coverage data (HTML, CSV)	✅ Working
uos_package.py	Package coverage artifacts	✅ Working
uos_xst.py	Cross-source traceability	✅ Working
libuoscov	C runtime library for coverage ABI	✅ Working

Coverage Workflow

# 1. Instrument source code
python tools/uos-cover/uos_cins.py --input kernel/src/ --output kernel/src/

# 2. Build instrumented firmware
cd kernel
make ARCH=armv7 PLATFORM=qemu-arm-virt

# 3. Run instrumented firmware in QEMU
qemu-system-arm -M virt -cpu cortex-a15 -m 512M \
  -nographic -kernel build/armv7/qemu-arm-virt/universalisos.elf

# 4. Extract trace data (from UART output or memory dump)
# 5. Parse trace data
python tools/uos-cover/uos_covparse.py --trace trace.txt --output coverage.json

# 6. Generate report
python tools/uos-cover/uos_covexport.py --input coverage.json --format html --output coverage/

Coverage Requirements (DO-178C)

Level	Requirement	Current Status
Statement Coverage	100% of reachable statements	❌ Not measured for kernel
Branch Coverage	100% of all branches	❌ Not measured for kernel
MC/DC	100% for critical modules	❌ Not measured for kernel

Justifying Unreachable Code

For code that cannot be exercised (e.g., dead code after __builtin_unreachable()):

python tools/uos-cover/uos_justify.py \
  --file kernel/src/core/scheduler.cpp \
  --line 245 \
  --reason "Unreachable after watchdog timeout assertion"

---

6. Test Environment Requirements

Host-Side Development

• Python 3.8+
• pytest
• QEMU (for firmware testing)
• Cross-compilation toolchains (ARM, AArch64, RISC-V)

CI/CD (GitHub Actions)

• Ubuntu latest
• Python 3.10
• QEMU system packages
• Cross-compiler toolchains (via tools/uos-target/)

Target Hardware (Future)

• PikeOS-compatible evaluation boards
• JTAG/SWD debug probes
• Logic analyzers for timing verification

---

7. Test Prioritization

Risk-Based Testing

Tests are prioritized based on risk:

Priority	Module	Rationale
P0 - Critical	Scheduler, IPC, Memory Management	System stability; failures cause crashes
P1 - High	Device drivers, Health Monitoring	Hardware interaction; failures cause data loss
P2 - Medium	File system, Network stack	Feature failures degrade functionality
P3 - Low	Debug tools, Diagnostics	Failures affect observability only

Test Categories

Category	Description	Example
Smoke	Boot + basic functionality	Banner output, scheduler start
Functional	Feature-specific behavior	IPC message delivery, memory allocation
Regression	Prevent reintroduction of fixed bugs	Specific bug fix verification
Performance	Timing, throughput, latency	Scheduler tick accuracy, IPC latency
Stress	Under load, resource exhaustion	Memory pressure, task overflow
Security	Isolation, privilege boundaries	Partition isolation, capability checks

---

8. Roadmap: Building Out Testing Infrastructure

Phase 1: CI Boot Testing (Immediate)

Goal: Run firmware boot tests in CI for all architectures.

Tasks:

1. Add make test target to kernel/Makefile
2. Create tools/uos-boot-test/ — QEMU orchestrator with UART monitoring
3. Wire uos-check.sh test into .github/workflows/ci.yml
4. Add test result reporting (JUnit XML)

Estimated effort: 2-3 days

Phase 2: In-Kernel Test Harness (Short-term)

Goal: Structured test framework for firmware tests.

Tasks:

1. Create kernel/src/test/ directory
2. Implement uos_test.h — test macros (UOS_TEST, UOS_ASSERT, UOS_SKIP)
3. Implement test runner — collects test results, outputs to UART
4. Port inline smoke tests to structured test cases
5. Add test discovery and registration

Estimated effort: 1-2 weeks

Phase 3: Coverage in CI (Medium-term)

Goal: Automated code coverage measurement for kernel builds.

Tasks:

1. Integrate uos_cins.py into build process
2. Run instrumented firmware in QEMU
3. Extract and parse coverage data
4. Generate coverage reports in CI
5. Enforce coverage thresholds (start with 80%, increase to 100%)

Estimated effort: 2-3 weeks

Phase 4: MC/DC for Critical Modules (Long-term)

Goal: Achieve DO-178C Level A compliance for critical modules.

Tasks:

1. Identify critical modules (scheduler, IPC, memory)
2. Write MC/DC test cases for each decision point
3. Justify unreachable paths formally
4. Generate compliance reports

Estimated effort: 1-2 months

Phase 5: Performance & Stress Testing (Future)

Goal: Validate real-time guarantees and stability under load.

Tasks:

1. Define performance baselines (tick accuracy, IPC latency)
2. Create stress test scenarios (task overflow, memory exhaustion)
3. Automate performance regression detection

Estimated effort: 2-4 weeks

---

9. Appendices

A. File Locations

Path	Purpose
tools/uos-cover/	Code coverage instrumentation + analysis
tools/uos-pkg/	Package build tooling
tools/doorstop-integration/	Requirements traceability (HLR)
tools/uos-target/	Target definition validation (NOT test runner)
kernel/uos-check.sh	Build + boot smoke tests
kernel/src/test/	In-kernel test harness (planned)
.github/workflows/ci.yml	CI pipeline configuration

B. CI Pipeline Status

Job	Runs in CI	Tests
kernel-build	✅	Build only (no boot test)
uos-cover-test	✅	Tool-level pytest
uos-pkg-test	✅	Tool-level pytest
requirements-coverage	✅	Doorstop validation
mycelium-test	✅	Rust CLI tests
Boot smoke tests	❌	Not integrated
Kernel coverage	❌	Not implemented

C. Common Commands

# Host-side tests
python -m pytest tools/uos-cover/test/ -v
python -m pytest tools/uos-pkg/test/ -v

# Kernel build + smoke test
cd kernel && ./uos-check.sh test

# Kernel build only
cd kernel && make ARCH=armv7 PLATFORM=qemu-arm-virt

# QEMU boot
cd kernel && make run-qemu

# Coverage instrumentation
python tools/uos-cover/uos_cins.py --input kernel/src/ --output kernel/src/

# Requirements validation
python tools/doorstop-integration/doors_export.py --validate

D. References

• DO-178C — Software Considerations in Airborne Systems
• ISO 26262 — Road vehicles functional safety
• PikeOS 5.0 — Reference hypervisor implementation
• Doorstop — Requirements management tool
• uos-cover documentation — Coverage tooling details