# UniversalisOS Testing Toolchain Manual

This document is the official manual for the UniversalisOS testing toolchain. It replaces generic testing guidelines with concrete instructions for the custom-built tools already present in the codebase.

---

## 1. Toolchain Overview

The UniversalisOS testing ecosystem consists of three integrated components:

| Component | Tool | Purpose | Location |
|-----------|------|---------|----------|
| **Test Execution** | `uos-target.py` | Compile tests into kernel payload, run via QEMU, parse UART results | `tools/uos-target/` |
| **Code Coverage** | `uos-cover` suite | C instrumentation, trace parsing, MC/DC/branch/statement coverage | `tools/uos-cover/` |
| **Requirements** | Doorstop + Plane | HLR management (Doorstop) and LLR management (Plane) | `tools/doorstop-integration/` |

---

## 2. Test Execution: `uos-target.py`

### Purpose
`uos-target.py` manages the complete test execution pipeline:
1. Compiles tests directly into the kernel payload/firmware
2. Launches QEMU with the instrumented kernel
3. Parses `pass`, `fail`, and `skip` results directly from UART output
4. Generates JUnit XML reports for CI integration

### Usage

```bash
# Run all tests for default target (qemu-arm-virt)
python tools/uos-target/uos-target.py --run

# Run tests for specific architecture
python tools/uos-target/uos-target.py --arch armv7 --platform qemu-arm-virt --run

# Run with specific test filter
python tools/uos-target/uos-target.py --run --filter "scheduler_*"

# Generate JUnit XML for CI
python tools/uos-target/uos-target.py --run --junit results.xml

# Run with coverage instrumentation enabled
python tools/uos-target/uos-target.py --run --coverage
```

### Test Output Format
Tests embedded in the kernel must output results in this exact format:

```
[UOS-TEST] <test_name>: <pass|fail|skip> [<message>]
```

Example UART output:
```
[UOS-TEST] scheduler_round_robin: pass
[UOS-TEST] scheduler_preemption: pass
[UOS-TEST] memory_alloc_null: fail [null pointer not caught]
[UOS-TEST] ipc_message_size: skip [not implemented on this platform]
```

### Configuration
`uos-target.py` reads target definitions from `tools/uos-target/targets/`:

```yaml
# tools/uos-target/targets/qemu-arm-virt.yaml
name: qemu-arm-virt
arch: armv7
platform: qemu-arm-virt
qemu_binary: qemu-system-arm
qemu_args:
  - -M virt
  - -cpu cortex-a15
  - -m 512M
  - -nographic
uart_timeout: 10
banner: "UniversalisOS"
```

---

## 3. Code Coverage: `uos-cover` Suite

### Purpose
The `uos-cover` suite provides precise, bare-metal ARINC/ISO 26262 compliant coverage analysis for firmware builds. It achieves MC/DC, branch, and statement coverage through source-level C instrumentation.

### Tools

| Tool | Command | Purpose |
|------|---------|---------|
| `uos_cins.py` | `python tools/uos-cover/uos_cins.py` | Instrument C/C++ source before compilation |
| `uos_cover.py` | `python tools/uos-cover/uos_cover.py` | Main coverage orchestrator |
| `uos_covparse.py` | `python tools/uos-cover/uos_covparse.py` | Parse trace data into coverage reports |
| `uos_trace.py` | `python tools/uos-cover/uos_trace.py` | Parse raw trace files from instrumented runs |
| `uos_justify.py` | `python tools/uos-cover/uos_justify.py` | Justify unreachable code paths |
| `uos_verify.py` | `python tools/uos-cover/uos_verify.py` | Verify coverage against requirements |
| `uos_covexport.py` | `python tools/uos-cover/uos_covexport.py` | Export coverage data (HTML, CSV, JSON) |
| `uos_package.py` | `python tools/uos-cover/uos_package.py` | Package coverage artifacts for submission |
| `uos_xst.py` | `python tools/uos-cover/uos_xst.py` | Cross-source traceability analysis |

### Coverage Workflow

```bash
# 1. Instrument source code
python tools/uos-cover/uos_cins.py \
    --input kernel/src/ \
    --output kernel/src_instrumented/

# 2. Build instrumented firmware
cd kernel
make ARCH=armv7 PLATFORM=qemu-arm-virt \
    SRC_DIR=src_instrumented/

# 3. Run tests with coverage (uos-target handles trace extraction)
python tools/uos-target/uos-target.py --run --coverage

# 4. Parse trace data
python tools/uos-cover/uos_covparse.py \
    --trace build/trace.txt \
    --output coverage.json

# 5. Verify against requirements
python tools/uos-cover/uos_verify.py \
    --coverage coverage.json \
    --requirements tools/doorstop-integration/

# 6. Generate HTML report
python tools/uos-cover/uos_covexport.py \
    --input coverage.json \
    --format html \
    --output coverage_report/

# 7. Package for submission
python tools/uos-cover/uos_package.py \
    --input coverage_report/ \
    --output coverage_package.zip
```

### Coverage Levels (DO-178C / ISO 26262)

| Level | Metric | Threshold | Tool |
|-------|--------|-----------|------|
| Statement | % of executable statements | 100% | `uos_covparse.py` |
| Branch | % of decision branches | 100% | `uos_covparse.py` |
| MC/DC | Modified Condition/Decision | 100% | `uos_covparse.py --mcdc` |
| Function | % of functions called | 100% | `uos_covparse.py` |

### Justifying Unreachable Code

For code that cannot be exercised (e.g., hardware-failure paths, defensive checks):

```bash
python tools/uos-cover/uos_justify.py \
    --file kernel/src/core/scheduler.cpp \
    --line 245 \
    --reason "Unreachable: watchdog timeout triggers system reset before this path" \
    --category "hardware_failure"
```

Justifications are stored in `.uos_justifications.json` and included in verification reports.

---

## 4. Requirements Traceability

### Higher-Level Requirements (HLR)
HLRs are managed by **Doorstop** at `tools/doorstop-integration/`:

```bash
# Create new HLR
cd tools/doorstop-integration
python doorsetup.py --create HLR --id REQ-006 \
    --text "The scheduler shall support round-robin partitioning"

# Validate all requirements
python doors_export.py --validate

# Generate traceability matrix
python doors_export.py --format csv --output trace_matrix.csv
```

HLRs are viewed and edited via:
- **Web**: `janela-do-desassossego-web` (DOORS-like grid at `/doors`)
- **IDE**: `aurelio-theia` sidebar widget (`HLR Requirements` view)
- **VS Code**: `aurelio-vscode` panel (`Aurelio: Open HLR Requirements` command)

### Lower-Level Requirements (LLR)
LLRs are managed in **Plane** (`plane.portugalfuturista.org`) and tied directly to the CI execution pipeline.

### Bidirectional Sync
```bash
# Push HLRs to Plane (create/update LLRs)
python plane_bridge.py --sync

# Pull LLR status from Plane
python plane_bridge.py --pull --format yaml
```

---

## 5. CI Integration

### GitHub Actions Pipeline
The CI pipeline runs all three toolchain components in sequence:

```yaml
# .github/workflows/ci.yml (relevant jobs)
jobs:
  uos-target-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run firmware tests
        run: python tools/uos-target/uos-target.py --run --junit test-results.xml
      - name: Upload test results
        uses: actions/upload-artifact@v4
        with:
          name: test-results
          path: test-results.xml

  uos-cover-analysis:
    needs: uos-target-test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Instrument source
        run: python tools/uos-cover/uos_cins.py --input kernel/src/ --output kernel/src/
      - name: Build with coverage
        run: cd kernel && make ARCH=armv7 PLATFORM=qemu-arm-virt
      - name: Run tests with coverage
        run: python tools/uos-target/uos-target.py --run --coverage
      - name: Parse coverage
        run: python tools/uos-cover/uos_covparse.py --trace build/trace.txt --output coverage.json
      - name: Verify coverage
        run: python tools/uos-cover/uos_verify.py --coverage coverage.json --requirements tools/doorstop-integration/
      - name: Export HTML report
        run: python tools/uos-cover/uos_covexport.py --input coverage.json --format html --output coverage/
      - name: Upload coverage
        uses: actions/upload-artifact@v4
        with:
          name: coverage-report
          path: coverage/

  requirements-coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Validate Doorstop requirements
        run: python tools/doorstop-integration/doors_export.py --validate
      - name: Check traceability
        run: python tools/doorstop-integration/doors_export.py --check-links
```

---

## 6. Quick Reference

### Running Tests Locally
```bash
# Full test suite with coverage
python tools/uos-target/uos-target.py --run --coverage

# Specific architecture only
python tools/uos-target/uos-target.py --arch riscv --run

# Quick smoke test (no coverage)
cd kernel && ./uos-check.sh test
```

### Coverage Commands
```bash
# Instrument → Build → Test → Report (full pipeline)
python tools/uos-cover/uos_cover.py --full-pipeline --arch armv7

# Justify unreachable code
python tools/uos-cover/uos_justify.py --file <file> --line <line> --reason "<reason>"

# Export coverage for specific requirement
python tools/uos-cover/uos_covexport.py --input coverage.json --requirement REQ-001 --format html
```

### Requirements Commands
```bash
# View requirements in browser (Doorstop backend must be running)
curl http://192.168.0.9:8100/api/v1/requirements

# Validate
python tools/doorstop-integration/doors_export.py --validate

# Sync with Plane
python tools/doorstop-integration/plane_bridge.py --sync
```

---

## 7. File Locations

| Path | Purpose |
|------|---------|
| `tools/uos-target/` | Test execution engine (QEMU orchestration + UART parsing) |
| `tools/uos-cover/` | Coverage instrumentation and analysis suite |
| `tools/doorstop-integration/` | HLR requirements management (Doorstop backend) |
| `tools/uos-pkg/` | Package build tooling |
| `kernel/uos-check.sh` | Legacy build + boot smoke tests |
| `.github/workflows/ci.yml` | CI pipeline configuration |

---

## 8. References

- [DO-178C](https://www.rtca.org/sc-205/) — Software Considerations in Airborne Systems
- [ISO 26262](https://www.iso.org/standard/68383.html) — Road vehicles functional safety
- [ARINC 653](https://www.aviation-ia.com/standards) — Avionics application standard interface
- [Doorstop](https://doorstop.readthedocs.io/) — Requirements management tool
- [uos-cover README](../tools/uos-cover/README.md) — Coverage tooling details
- [uos-target README](../tools/uos-target/README.md) — Test execution details