docs: rewrite testing guidelines — accurate, comprehensive, actionable

Major rewrite of TESTING_GUIDELINES.md based on audit of actual codebase: Key corrections: - uos-target.py is a target-definition validator, NOT a test orchestrator - No in-kernel test harness exists (only inline smoke tests) - No CI boot testing (only build verification) - No kernel code coverage (only tool-level coverage) - uos-cover tools ARE operational with 11 test files New content: - Current state vs planned state clearly separated - Host-side testing (pytest) documented with examples - Firmware testing with uos-check.sh documented - Code coverage workflow with uos-cover tools - Test environment requirements - Risk-based test prioritization (P0-P3) - Test categories (smoke, functional, regression, performance, stress, security) - 5-phase roadmap: CI boot tests → test harness → coverage in CI → MC/DC → perf - Appendices with file locations, CI status, common commands Document now accurately reflects what exists and provides actionable guidance.
2026-07-12 16:38:36 +01:00 · 2026-07-12 16:38:36 +01:00 · 5606440114
commit 5606440114
parent d914c4727d
1 changed files with 436 additions and 0 deletions
--- a/docs/TESTING_GUIDELINES.md
+++ b/docs/TESTING_GUIDELINES.md
@ -0,0 +1,436 @@
+# UniversalisOS Testing & Quality Assurance Guidelines
+
+As a safety-critical Type-1 hypervisor implementing PikeOS 5.0 patterns, UniversalisOS requires a rigorous, standards-compliant approach to testing. This document defines the strategies for requirements traceability, test creation, execution, and code coverage to align with high-assurance standards (e.g., DO-178C, ISO 26262).
+
+---
+
+## 1. Requirements Engineering & Traceability
+
+### Current State
+- **Tooling**: Doorstop integration exists at `tools/doorstop-integration/` with 5 HLRs (`REQ-001.yml` through `REQ-005.yml`).
+- **CI validation**: The `requirements-coverage` GitHub Actions job validates Doorstop requirements and generates traceability matrices.
+- **Plane integration**: `plane_bridge.py` provides bidirectional sync between Doorstop and Plane for LLRs.
+
+### Higher-Level Requirements (HLR)
+HLRs describe *what* the system must do from a system integration perspective.
+- **Management**: HLRs are managed via **Doorstop** (`tools/doorstop-integration/`).
+- **User Interface**: Requirements engineers interact with HLRs via `janela-do-desassossego-web` (tabular UI) or IDE integrations in `aurelio-theia`/`aurelio-vscode`.
+- **Traceability**: Every HLR must trace down to LLRs. Test engineers read HLRs, compare them with test framework keywords, and implement in-firmware tests directly targeting the requirements.
+
+### Lower-Level Requirements (LLR)
+LLRs describe *how* the HLRs are implemented at the software and architecture level.
+- **Management**: LLRs are managed in Plane (`plane.portugalfuturista.org`).
+- **Traceability**: Every LLR must trace back to an HLR, and forward to specific in-firmware test cases executing within the CI pipeline.
+
+### Adding New Requirements
+```bash
+cd tools/doorstop-integration
+python3 doorsetup.py --create HLR --id REQ-006 --text "Description of requirement"
+python3 plane_bridge.py --sync  # Push to Plane for LLR management
+```
+
+---
+
+## 2. Test Framework Architecture
+
+### Current State
+UniversalisOS uses a **dual-layer** testing approach:
+
+| Layer | Tool | Status | Purpose |
+|-------|------|--------|---------|
+| **Host-side tooling** | pytest suites | ✅ Operational | Test the build/coverage/packaging tools |
+| **Target firmware** | `uos-check.sh` | ✅ Partial | Build verification + boot smoke tests |
+| **Target firmware** | In-kernel test harness | ❌ Not implemented | Structured firmware tests |
+| **Code coverage** | `uos-cover` suite | ✅ Operational | Instrumentation + coverage analysis |
+| **Test orchestration** | `uos-target` | ⚠️ Misnamed | Currently a target-definition validator, not a test runner |
+
+### Host-Side Testing (Tooling)
+The `tools/` directory contains well-tested Python tooling:
+
+| Tool | Test Files | Coverage |
+|------|-----------|----------|
+| `tools/uos-cover/` | 8 pytest files + 1 C unit test | Instrumentation, trace parsing, justification, export |
+| `tools/uos-pkg/` | 3 pytest files | Package build, spec parsing, RPM format |
+| `tools/doorstop-integration/` | CI validation | Requirements traceability |
+
+Run host-side tests:
+```bash
+# All tool tests
+python -m pytest tools/uos-cover/test/ tools/uos-pkg/test/ -v
+
+# Specific tool
+python -m pytest tools/uos-cover/test/ -v
+```
+
+### Target Firmware Testing (Current)
+The `kernel/uos-check.sh` script provides build + boot verification:
+
+```bash
+cd kernel
+./uos-check.sh test          # Build all architectures + boot smoke tests
+./uos-check.sh test-armv7    # Build + boot ARMv7 only
+./uos-check.sh test-aarch64  # Build + boot AArch64 only
+./uos-check.sh test-riscv    # Build + boot RISC-V only
+./uos-check.sh lint          # Static analysis (if available)
+./uos-check.sh build-all     # Build all architectures (no boot)
+```
+
+**What `uos-check.sh test` does:**
+1. Compiles the kernel for the target architecture
+2. Launches QEMU with the compiled ELF
+3. Waits up to 10 seconds for UART output
+4. Greps for the "UniversalisOS" banner string
+5. Reports PASS/FAIL based on banner detection
+
+**Limitations:**
+- Only checks for boot banner (no functional test validation)
+- No structured pass/fail/skip keyword parsing
+- No test result persistence or reporting
+- Not integrated into CI (runs locally only)
+
+### Target Firmware Testing (Planned)
+A structured in-kernel test harness is needed for DO-178C compliance. See [Section 8: Roadmap](#8-roadmap).
+
+---
+
+## 3. Creating Test Cases
+
+### Host-Side Tool Tests
+Host-side tests follow standard pytest conventions:
+
+```python
+# tools/uos-cover/test/test_uos_trace.py example
+import pytest
+from uos_trace import parse_trace_file, generate_matrix
+
+def test_parse_trace_file():
+    """Verify trace file parsing handles valid input."""
+    result = parse_trace_file("fixtures/valid_trace.txt")
+    assert result is not None
+    assert len(result.requirements) > 0
+
+def test_generate_matrix():
+    """Verify traceability matrix generation."""
+    matrix = generate_matrix(requirements, test_cases)
+    assert matrix覆盖率 >= 0.0  # Basic sanity check
+```
+
+Run:
+```bash
+python -m pytest tools/uos-cover/test/test_uos_trace.py -v
+```
+
+### Firmware Test Cases (Current)
+Firmware tests are currently **inline smoke tests** embedded in kernel source:
+
+```cpp
+// kernel/src/core/kernel.cpp — inline demo test
+void partition_lifecycle_test() {
+    uart_puts("[TEST] partition_lifecycle: starting\n");
+    // ... test logic ...
+    uart_puts("[TEST] partition_lifecycle: pass\n");
+}
+```
+
+**Test output format:**
+```
+[TEST] <test_name>: pass
+[TEST] <test_name>: fail
+[TEST] <test_name>: skip
+```
+
+### Firmware Test Cases (Planned)
+For DO-178C compliance, tests should be structured as:
+
+```cpp
+// kernel/src/test/test_scheduler.cpp (planned)
+#include "uos_test.h"
+
+UOS_TEST(scheduler_round_robin) {
+    // Arrange: create 3 partitions with equal priority
+    // Act: run scheduler for 100 ticks
+    // Assert: each partition ran approximately 33 ticks
+    UOS_ASSERT(partition_a_ticks >= 30 && partition_a_ticks <= 36);
+    UOS_ASSERT(partition_b_ticks >= 30 && partition_b_ticks <= 36);
+    UOS_ASSERT(partition_c_ticks >= 30 && partition_c_ticks <= 36);
+}
+
+UOS_TEST(scheduler_preemption) {
+    // Arrange: create high-priority and low-priority partitions
+    // Act: high-priority partition runs
+    // Assert: low-priority partition is preempted
+    UOS_ASSERT(preemption_occurred == true);
+}
+```
+
+### Exhaustive Corner Cases
+Tests must cover all corner cases, including:
+- Integer overflow/underflow for all data types
+- Memory boundary conditions (null pointers, buffer overflows)
+- Race conditions in concurrent operations
+- Error handling paths (invalid inputs, resource exhaustion)
+- Hardware edge cases (device timeout, interrupt storms)
+
+---
+
+## 4. Running the Test Suite
+
+### Host-Side Tests (CI/CD)
+Automated via GitHub Actions (`.github/workflows/ci.yml`):
+
+```yaml
+# Already configured in CI:
+- name: Run uos-cover tests
+  run: python -m pytest tools/uos-cover/test/ -v
+
+- name: Run uos-pkg tests
+  run: python -m pytest tools/uos-pkg/test/ -v
+
+- name: Validate requirements
+  run: python tools/doorstop-integration/doors_export.py --validate
+```
+
+### Target Firmware Tests (Manual)
+```bash
+cd kernel
+
+# Quick smoke test (all architectures)
+./uos-check.sh test
+
+# Architecture-specific
+make ARCH=armv7 PLATFORM=qemu-arm-virt
+qemu-system-arm -M virt -cpu cortex-a15 -m 512M \
+  -nographic -kernel build/armv7/qemu-arm-virt/universalisos.elf
+
+# Watch for test output
+# Press Ctrl+A then X to exit QEMU
+```
+
+### Target Firmware Tests (CI - Planned)
+See [Section 8: Roadmap](#8-roadmap) for CI integration of boot tests.
+
+---
+
+## 5. Code Coverage Analysis: `uos-cover`
+
+### Current State
+The `uos-cover` suite is **operational** with 10 Python modules and comprehensive tests.
+
+### Coverage Tools
+
+| Tool | Purpose | Status |
+|------|---------|--------|
+| `uos_cins.py` | Instrument C/C++ source before compilation | ✅ Working |
+| `uos_trace.py` | Parse trace data from instrumented runs | ✅ Working |
+| `uos_covparse.py` | Generate coverage reports | ✅ Working |
+| `uos_justify.py` | Justify unreachable code paths | ✅ Working |
+| `uos_verify.py` | Verify coverage against requirements | ✅ Working |
+| `uos_covexport.py` | Export coverage data (HTML, CSV) | ✅ Working |
+| `uos_package.py` | Package coverage artifacts | ✅ Working |
+| `uos_xst.py` | Cross-source traceability | ✅ Working |
+| `libuoscov` | C runtime library for coverage ABI | ✅ Working |
+
+### Coverage Workflow
+
+```bash
+# 1. Instrument source code
+python tools/uos-cover/uos_cins.py --input kernel/src/ --output kernel/src/
+
+# 2. Build instrumented firmware
+cd kernel
+make ARCH=armv7 PLATFORM=qemu-arm-virt
+
+# 3. Run instrumented firmware in QEMU
+qemu-system-arm -M virt -cpu cortex-a15 -m 512M \
+  -nographic -kernel build/armv7/qemu-arm-virt/universalisos.elf
+
+# 4. Extract trace data (from UART output or memory dump)
+# 5. Parse trace data
+python tools/uos-cover/uos_covparse.py --trace trace.txt --output coverage.json
+
+# 6. Generate report
+python tools/uos-cover/uos_covexport.py --input coverage.json --format html --output coverage/
+```
+
+### Coverage Requirements (DO-178C)
+
+| Level | Requirement | Current Status |
+|-------|-------------|----------------|
+| **Statement Coverage** | 100% of reachable statements | ❌ Not measured for kernel |
+| **Branch Coverage** | 100% of all branches | ❌ Not measured for kernel |
+| **MC/DC** | 100% for critical modules | ❌ Not measured for kernel |
+
+### Justifying Unreachable Code
+For code that cannot be exercised (e.g., dead code after `__builtin_unreachable()`):
+
+```bash
+python tools/uos-cover/uos_justify.py \
+  --file kernel/src/core/scheduler.cpp \
+  --line 245 \
+  --reason "Unreachable after watchdog timeout assertion"
+```
+
+---
+
+## 6. Test Environment Requirements
+
+### Host-Side Development
+- Python 3.8+
+- pytest
+- QEMU (for firmware testing)
+- Cross-compilation toolchains (ARM, AArch64, RISC-V)
+
+### CI/CD (GitHub Actions)
+- Ubuntu latest
+- Python 3.10
+- QEMU system packages
+- Cross-compiler toolchains (via `tools/uos-target/`)
+
+### Target Hardware (Future)
+- PikeOS-compatible evaluation boards
+- JTAG/SWD debug probes
+- Logic analyzers for timing verification
+
+---
+
+## 7. Test Prioritization
+
+### Risk-Based Testing
+Tests are prioritized based on risk:
+
+| Priority | Module | Rationale |
+|----------|--------|-----------|
+| **P0 - Critical** | Scheduler, IPC, Memory Management | System stability; failures cause crashes |
+| **P1 - High** | Device drivers, Health Monitoring | Hardware interaction; failures cause data loss |
+| **P2 - Medium** | File system, Network stack | Feature failures degrade functionality |
+| **P3 - Low** | Debug tools, Diagnostics | Failures affect observability only |
+
+### Test Categories
+
+| Category | Description | Example |
+|----------|-------------|---------|
+| **Smoke** | Boot + basic functionality | Banner output, scheduler start |
+| **Functional** | Feature-specific behavior | IPC message delivery, memory allocation |
+| **Regression** | Prevent reintroduction of fixed bugs | Specific bug fix verification |
+| **Performance** | Timing, throughput, latency | Scheduler tick accuracy, IPC latency |
+| **Stress** | Under load, resource exhaustion | Memory pressure, task overflow |
+| **Security** | Isolation, privilege boundaries | Partition isolation, capability checks |
+
+---
+
+## 8. Roadmap: Building Out Testing Infrastructure
+
+### Phase 1: CI Boot Testing (Immediate)
+**Goal:** Run firmware boot tests in CI for all architectures.
+
+**Tasks:**
+1. Add `make test` target to `kernel/Makefile`
+2. Create `tools/uos-boot-test/` — QEMU orchestrator with UART monitoring
+3. Wire `uos-check.sh test` into `.github/workflows/ci.yml`
+4. Add test result reporting (JUnit XML)
+
+**Estimated effort:** 2-3 days
+
+### Phase 2: In-Kernel Test Harness (Short-term)
+**Goal:** Structured test framework for firmware tests.
+
+**Tasks:**
+1. Create `kernel/src/test/` directory
+2. Implement `uos_test.h` — test macros (`UOS_TEST`, `UOS_ASSERT`, `UOS_SKIP`)
+3. Implement test runner — collects test results, outputs to UART
+4. Port inline smoke tests to structured test cases
+5. Add test discovery and registration
+
+**Estimated effort:** 1-2 weeks
+
+### Phase 3: Coverage in CI (Medium-term)
+**Goal:** Automated code coverage measurement for kernel builds.
+
+**Tasks:**
+1. Integrate `uos_cins.py` into build process
+2. Run instrumented firmware in QEMU
+3. Extract and parse coverage data
+4. Generate coverage reports in CI
+5. Enforce coverage thresholds (start with 80%, increase to 100%)
+
+**Estimated effort:** 2-3 weeks
+
+### Phase 4: MC/DC for Critical Modules (Long-term)
+**Goal:** Achieve DO-178C Level A compliance for critical modules.
+
+**Tasks:**
+1. Identify critical modules (scheduler, IPC, memory)
+2. Write MC/DC test cases for each decision point
+3. Justify unreachable paths formally
+4. Generate compliance reports
+
+**Estimated effort:** 1-2 months
+
+### Phase 5: Performance & Stress Testing (Future)
+**Goal:** Validate real-time guarantees and stability under load.
+
+**Tasks:**
+1. Define performance baselines (tick accuracy, IPC latency)
+2. Create stress test scenarios (task overflow, memory exhaustion)
+3. Automate performance regression detection
+
+**Estimated effort:** 2-4 weeks
+
+---
+
+## 9. Appendices
+
+### A. File Locations
+
+| Path | Purpose |
+|------|---------|
+| `tools/uos-cover/` | Code coverage instrumentation + analysis |
+| `tools/uos-pkg/` | Package build tooling |
+| `tools/doorstop-integration/` | Requirements traceability (HLR) |
+| `tools/uos-target/` | Target definition validation (NOT test runner) |
+| `kernel/uos-check.sh` | Build + boot smoke tests |
+| `kernel/src/test/` | In-kernel test harness (planned) |
+| `.github/workflows/ci.yml` | CI pipeline configuration |
+
+### B. CI Pipeline Status
+
+| Job | Runs in CI | Tests |
+|-----|-----------|-------|
+| `kernel-build` | ✅ | Build only (no boot test) |
+| `uos-cover-test` | ✅ | Tool-level pytest |
+| `uos-pkg-test` | ✅ | Tool-level pytest |
+| `requirements-coverage` | ✅ | Doorstop validation |
+| `mycelium-test` | ✅ | Rust CLI tests |
+| Boot smoke tests | ❌ | Not integrated |
+| Kernel coverage | ❌ | Not implemented |
+
+### C. Common Commands
+
+```bash
+# Host-side tests
+python -m pytest tools/uos-cover/test/ -v
+python -m pytest tools/uos-pkg/test/ -v
+
+# Kernel build + smoke test
+cd kernel && ./uos-check.sh test
+
+# Kernel build only
+cd kernel && make ARCH=armv7 PLATFORM=qemu-arm-virt
+
+# QEMU boot
+cd kernel && make run-qemu
+
+# Coverage instrumentation
+python tools/uos-cover/uos_cins.py --input kernel/src/ --output kernel/src/
+
+# Requirements validation
+python tools/doorstop-integration/doors_export.py --validate
+```
+
+### D. References
+
+- [DO-178C](https://www.rtca.org/sc-205/) — Software Considerations in Airborne Systems
+- [ISO 26262](https://www.iso.org/standard/68383.html) — Road vehicles functional safety
+- [PikeOS 5.0](https://www.sysgo.com/pikeos) — Reference hypervisor implementation
+- [Doorstop](https://doorstop.readthedocs.io/) — Requirements management tool
+- [uos-cover documentation](../tools/uos-cover/README.md) — Coverage tooling details