universalisos/docs/TESTING_GUIDELINES.md

# UniversalisOS Testing & Quality Assurance Guidelines

As a safety-critical Type-1 hypervisor implementing PikeOS 5.0 patterns, UniversalisOS requires a rigorous, standards-compliant approach to testing. This document defines the strategies for requirements traceability, test creation, execution, and code coverage to align with high-assurance standards (e.g., DO-178C, ISO 26262).

---

## 1. Requirements Engineering & Traceability

### Current State
- **Tooling**: Doorstop integration exists at `tools/doorstop-integration/` with 5 HLRs (`REQ-001.yml` through `REQ-005.yml`).
- **CI validation**: The `requirements-coverage` GitHub Actions job validates Doorstop requirements and generates traceability matrices.
- **Plane integration**: `plane_bridge.py` provides bidirectional sync between Doorstop and Plane for LLRs.

### Higher-Level Requirements (HLR)
HLRs describe *what* the system must do from a system integration perspective.
- **Management**: HLRs are managed via **Doorstop** (`tools/doorstop-integration/`).
- **User Interface**: Requirements engineers interact with HLRs via `janela-do-desassossego-web` (tabular UI) or IDE integrations in `aurelio-theia`/`aurelio-vscode`.
- **Traceability**: Every HLR must trace down to LLRs. Test engineers read HLRs, compare them with test framework keywords, and implement in-firmware tests directly targeting the requirements.

### Lower-Level Requirements (LLR)
LLRs describe *how* the HLRs are implemented at the software and architecture level.
- **Management**: LLRs are managed in Plane (`plane.portugalfuturista.org`).
- **Traceability**: Every LLR must trace back to an HLR, and forward to specific in-firmware test cases executing within the CI pipeline.

### Adding New Requirements
```bash
cd tools/doorstop-integration
python3 doorsetup.py --create HLR --id REQ-006 --text "Description of requirement"
python3 plane_bridge.py --sync  # Push to Plane for LLR management
```

---

## 2. Test Framework Architecture

### Current State
UniversalisOS uses a **dual-layer** testing approach:

| Layer | Tool | Status | Purpose |
|-------|------|--------|---------|
| **Host-side tooling** | pytest suites | ✅ Operational | Test the build/coverage/packaging tools |
| **Target firmware** | `uos-check.sh` | ✅ Partial | Build verification + boot smoke tests |
| **Target firmware** | In-kernel test harness | ❌ Not implemented | Structured firmware tests |
| **Code coverage** | `uos-cover` suite | ✅ Operational | Instrumentation + coverage analysis |
| **Test orchestration** | `uos-target` | ⚠️ Misnamed | Currently a target-definition validator, not a test runner |

### Host-Side Testing (Tooling)
The `tools/` directory contains well-tested Python tooling:

| Tool | Test Files | Coverage |
|------|-----------|----------|
| `tools/uos-cover/` | 8 pytest files + 1 C unit test | Instrumentation, trace parsing, justification, export |
| `tools/uos-pkg/` | 3 pytest files | Package build, spec parsing, RPM format |
| `tools/doorstop-integration/` | CI validation | Requirements traceability |

Run host-side tests:
```bash
# All tool tests
python -m pytest tools/uos-cover/test/ tools/uos-pkg/test/ -v

# Specific tool
python -m pytest tools/uos-cover/test/ -v
```

### Target Firmware Testing (Current)
The `kernel/uos-check.sh` script provides build + boot verification:

```bash
cd kernel
./uos-check.sh test          # Build all architectures + boot smoke tests
./uos-check.sh test-armv7    # Build + boot ARMv7 only
./uos-check.sh test-aarch64  # Build + boot AArch64 only
./uos-check.sh test-riscv    # Build + boot RISC-V only
./uos-check.sh lint          # Static analysis (if available)
./uos-check.sh build-all     # Build all architectures (no boot)
```

**What `uos-check.sh test` does:**
1. Compiles the kernel for the target architecture
2. Launches QEMU with the compiled ELF
3. Waits up to 10 seconds for UART output
4. Greps for the "UniversalisOS" banner string
5. Reports PASS/FAIL based on banner detection

**Limitations:**
- Only checks for boot banner (no functional test validation)
- No structured pass/fail/skip keyword parsing
- No test result persistence or reporting
- Not integrated into CI (runs locally only)

### Target Firmware Testing (Planned)
A structured in-kernel test harness is needed for DO-178C compliance. See [Section 8: Roadmap](#8-roadmap).

---

## 3. Creating Test Cases

### Host-Side Tool Tests
Host-side tests follow standard pytest conventions:

```python
# tools/uos-cover/test/test_uos_trace.py example
import pytest
from uos_trace import parse_trace_file, generate_matrix

def test_parse_trace_file():
    """Verify trace file parsing handles valid input."""
    result = parse_trace_file("fixtures/valid_trace.txt")
    assert result is not None
    assert len(result.requirements) > 0

def test_generate_matrix():
    """Verify traceability matrix generation."""
    matrix = generate_matrix(requirements, test_cases)
    assert matrix覆盖率 >= 0.0  # Basic sanity check
```

Run:
```bash
python -m pytest tools/uos-cover/test/test_uos_trace.py -v
```

### Firmware Test Cases (Current)
Firmware tests are currently **inline smoke tests** embedded in kernel source:

```cpp
// kernel/src/core/kernel.cpp — inline demo test
void partition_lifecycle_test() {
    uart_puts("[TEST] partition_lifecycle: starting\n");
    // ... test logic ...
    uart_puts("[TEST] partition_lifecycle: pass\n");
}
```

**Test output format:**
```
[TEST] <test_name>: pass
[TEST] <test_name>: fail
[TEST] <test_name>: skip
```

### Firmware Test Cases (Planned)
For DO-178C compliance, tests should be structured as:

```cpp
// kernel/src/test/test_scheduler.cpp (planned)
#include "uos_test.h"

UOS_TEST(scheduler_round_robin) {
    // Arrange: create 3 partitions with equal priority
    // Act: run scheduler for 100 ticks
    // Assert: each partition ran approximately 33 ticks
    UOS_ASSERT(partition_a_ticks >= 30 && partition_a_ticks <= 36);
    UOS_ASSERT(partition_b_ticks >= 30 && partition_b_ticks <= 36);
    UOS_ASSERT(partition_c_ticks >= 30 && partition_c_ticks <= 36);
}

UOS_TEST(scheduler_preemption) {
    // Arrange: create high-priority and low-priority partitions
    // Act: high-priority partition runs
    // Assert: low-priority partition is preempted
    UOS_ASSERT(preemption_occurred == true);
}
```

### Exhaustive Corner Cases
Tests must cover all corner cases, including:
- Integer overflow/underflow for all data types
- Memory boundary conditions (null pointers, buffer overflows)
- Race conditions in concurrent operations
- Error handling paths (invalid inputs, resource exhaustion)
- Hardware edge cases (device timeout, interrupt storms)

---

## 4. Running the Test Suite

### Host-Side Tests (CI/CD)
Automated via GitHub Actions (`.github/workflows/ci.yml`):

```yaml
# Already configured in CI:
- name: Run uos-cover tests
  run: python -m pytest tools/uos-cover/test/ -v

- name: Run uos-pkg tests
  run: python -m pytest tools/uos-pkg/test/ -v

- name: Validate requirements
  run: python tools/doorstop-integration/doors_export.py --validate
```

### Target Firmware Tests (Manual)
```bash
cd kernel

# Quick smoke test (all architectures)
./uos-check.sh test

# Architecture-specific
make ARCH=armv7 PLATFORM=qemu-arm-virt
qemu-system-arm -M virt -cpu cortex-a15 -m 512M \
  -nographic -kernel build/armv7/qemu-arm-virt/universalisos.elf

# Watch for test output
# Press Ctrl+A then X to exit QEMU
```

### Target Firmware Tests (CI - Planned)
See [Section 8: Roadmap](#8-roadmap) for CI integration of boot tests.

---

## 5. Code Coverage Analysis: `uos-cover`

### Current State
The `uos-cover` suite is **operational** with 10 Python modules and comprehensive tests.

### Coverage Tools

| Tool | Purpose | Status |
|------|---------|--------|
| `uos_cins.py` | Instrument C/C++ source before compilation | ✅ Working |
| `uos_trace.py` | Parse trace data from instrumented runs | ✅ Working |
| `uos_covparse.py` | Generate coverage reports | ✅ Working |
| `uos_justify.py` | Justify unreachable code paths | ✅ Working |
| `uos_verify.py` | Verify coverage against requirements | ✅ Working |
| `uos_covexport.py` | Export coverage data (HTML, CSV) | ✅ Working |
| `uos_package.py` | Package coverage artifacts | ✅ Working |
| `uos_xst.py` | Cross-source traceability | ✅ Working |
| `libuoscov` | C runtime library for coverage ABI | ✅ Working |

### Coverage Workflow

```bash
# 1. Instrument source code
python tools/uos-cover/uos_cins.py --input kernel/src/ --output kernel/src/

# 2. Build instrumented firmware
cd kernel
make ARCH=armv7 PLATFORM=qemu-arm-virt

# 3. Run instrumented firmware in QEMU
qemu-system-arm -M virt -cpu cortex-a15 -m 512M \
  -nographic -kernel build/armv7/qemu-arm-virt/universalisos.elf

# 4. Extract trace data (from UART output or memory dump)
# 5. Parse trace data
python tools/uos-cover/uos_covparse.py --trace trace.txt --output coverage.json

# 6. Generate report
python tools/uos-cover/uos_covexport.py --input coverage.json --format html --output coverage/
```

### Coverage Requirements (DO-178C)

| Level | Requirement | Current Status |
|-------|-------------|----------------|
| **Statement Coverage** | 100% of reachable statements | ❌ Not measured for kernel |
| **Branch Coverage** | 100% of all branches | ❌ Not measured for kernel |
| **MC/DC** | 100% for critical modules | ❌ Not measured for kernel |

### Justifying Unreachable Code
For code that cannot be exercised (e.g., dead code after `__builtin_unreachable()`):

```bash
python tools/uos-cover/uos_justify.py \
  --file kernel/src/core/scheduler.cpp \
  --line 245 \
  --reason "Unreachable after watchdog timeout assertion"
```

---

## 6. Test Environment Requirements

### Host-Side Development
- Python 3.8+
- pytest
- QEMU (for firmware testing)
- Cross-compilation toolchains (ARM, AArch64, RISC-V)

### CI/CD (GitHub Actions)
- Ubuntu latest
- Python 3.10
- QEMU system packages
- Cross-compiler toolchains (via `tools/uos-target/`)

### Target Hardware (Future)
- PikeOS-compatible evaluation boards
- JTAG/SWD debug probes
- Logic analyzers for timing verification

---

## 7. Test Prioritization

### Risk-Based Testing
Tests are prioritized based on risk:

| Priority | Module | Rationale |
|----------|--------|-----------|
| **P0 - Critical** | Scheduler, IPC, Memory Management | System stability; failures cause crashes |
| **P1 - High** | Device drivers, Health Monitoring | Hardware interaction; failures cause data loss |
| **P2 - Medium** | File system, Network stack | Feature failures degrade functionality |
| **P3 - Low** | Debug tools, Diagnostics | Failures affect observability only |

### Test Categories

| Category | Description | Example |
|----------|-------------|---------|
| **Smoke** | Boot + basic functionality | Banner output, scheduler start |
| **Functional** | Feature-specific behavior | IPC message delivery, memory allocation |
| **Regression** | Prevent reintroduction of fixed bugs | Specific bug fix verification |
| **Performance** | Timing, throughput, latency | Scheduler tick accuracy, IPC latency |
| **Stress** | Under load, resource exhaustion | Memory pressure, task overflow |
| **Security** | Isolation, privilege boundaries | Partition isolation, capability checks |

---

## 8. Roadmap: Building Out Testing Infrastructure

### Phase 1: CI Boot Testing (Immediate)
**Goal:** Run firmware boot tests in CI for all architectures.

**Tasks:**
1. Add `make test` target to `kernel/Makefile`
2. Create `tools/uos-boot-test/` — QEMU orchestrator with UART monitoring
3. Wire `uos-check.sh test` into `.github/workflows/ci.yml`
4. Add test result reporting (JUnit XML)

**Estimated effort:** 2-3 days

### Phase 2: In-Kernel Test Harness (Short-term)
**Goal:** Structured test framework for firmware tests.

**Tasks:**
1. Create `kernel/src/test/` directory
2. Implement `uos_test.h` — test macros (`UOS_TEST`, `UOS_ASSERT`, `UOS_SKIP`)
3. Implement test runner — collects test results, outputs to UART
4. Port inline smoke tests to structured test cases
5. Add test discovery and registration

**Estimated effort:** 1-2 weeks

### Phase 3: Coverage in CI (Medium-term)
**Goal:** Automated code coverage measurement for kernel builds.

**Tasks:**
1. Integrate `uos_cins.py` into build process
2. Run instrumented firmware in QEMU
3. Extract and parse coverage data
4. Generate coverage reports in CI
5. Enforce coverage thresholds (start with 80%, increase to 100%)

**Estimated effort:** 2-3 weeks

### Phase 4: MC/DC for Critical Modules (Long-term)
**Goal:** Achieve DO-178C Level A compliance for critical modules.

**Tasks:**
1. Identify critical modules (scheduler, IPC, memory)
2. Write MC/DC test cases for each decision point
3. Justify unreachable paths formally
4. Generate compliance reports

**Estimated effort:** 1-2 months

### Phase 5: Performance & Stress Testing (Future)
**Goal:** Validate real-time guarantees and stability under load.

**Tasks:**
1. Define performance baselines (tick accuracy, IPC latency)
2. Create stress test scenarios (task overflow, memory exhaustion)
3. Automate performance regression detection

**Estimated effort:** 2-4 weeks

---

## 9. Appendices

### A. File Locations

| Path | Purpose |
|------|---------|
| `tools/uos-cover/` | Code coverage instrumentation + analysis |
| `tools/uos-pkg/` | Package build tooling |
| `tools/doorstop-integration/` | Requirements traceability (HLR) |
| `tools/uos-target/` | Target definition validation (NOT test runner) |
| `kernel/uos-check.sh` | Build + boot smoke tests |
| `kernel/src/test/` | In-kernel test harness (planned) |
| `.github/workflows/ci.yml` | CI pipeline configuration |

### B. CI Pipeline Status

| Job | Runs in CI | Tests |
|-----|-----------|-------|
| `kernel-build` | ✅ | Build only (no boot test) |
| `uos-cover-test` | ✅ | Tool-level pytest |
| `uos-pkg-test` | ✅ | Tool-level pytest |
| `requirements-coverage` | ✅ | Doorstop validation |
| `mycelium-test` | ✅ | Rust CLI tests |
| Boot smoke tests | ❌ | Not integrated |
| Kernel coverage | ❌ | Not implemented |

### C. Common Commands

```bash
# Host-side tests
python -m pytest tools/uos-cover/test/ -v
python -m pytest tools/uos-pkg/test/ -v

# Kernel build + smoke test
cd kernel && ./uos-check.sh test

# Kernel build only
cd kernel && make ARCH=armv7 PLATFORM=qemu-arm-virt

# QEMU boot
cd kernel && make run-qemu

# Coverage instrumentation
python tools/uos-cover/uos_cins.py --input kernel/src/ --output kernel/src/

# Requirements validation
python tools/doorstop-integration/doors_export.py --validate
```

### D. References

- [DO-178C](https://www.rtca.org/sc-205/) — Software Considerations in Airborne Systems
- [ISO 26262](https://www.iso.org/standard/68383.html) — Road vehicles functional safety
- [PikeOS 5.0](https://www.sysgo.com/pikeos) — Reference hypervisor implementation
- [Doorstop](https://doorstop.readthedocs.io/) — Requirements management tool
- [uos-cover documentation](../tools/uos-cover/README.md) — Coverage tooling details