portugalfuturista/universalisos
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
universalisos/docs/TESTING_GUIDELINES.md

316 lines
10 KiB
Markdown
Raw Blame History

# UniversalisOS Testing Toolchain Manual
This document is the official manual for the UniversalisOS testing toolchain. It replaces generic testing guidelines with concrete instructions for the custom-built tools already present in the codebase.
---
## 1. Toolchain Overview
The UniversalisOS testing ecosystem consists of three integrated components:
| Component | Tool | Purpose | Location |
|-----------|------|---------|----------|
| **Test Execution** | `uos-target.py` | Compile tests into kernel payload, run via QEMU, parse UART results | `tools/uos-target/` |
| **Code Coverage** | `uos-cover` suite | C instrumentation, trace parsing, MC/DC/branch/statement coverage | `tools/uos-cover/` |
| **Requirements** | Doorstop + Plane | HLR management (Doorstop) and LLR management (Plane) | `tools/doorstop-integration/` |
---
## 2. Test Execution: `uos-target.py`
### Purpose
`uos-target.py` manages the complete test execution pipeline:
1. Compiles tests directly into the kernel payload/firmware
2. Launches QEMU with the instrumented kernel
3. Parses `pass`, `fail`, and `skip` results directly from UART output
4. Generates JUnit XML reports for CI integration
### Usage
```bash
# Run all tests for default target (qemu-arm-virt)
python tools/uos-target/uos-target.py --run
# Run tests for specific architecture
python tools/uos-target/uos-target.py --arch armv7 --platform qemu-arm-virt --run
# Run with specific test filter
python tools/uos-target/uos-target.py --run --filter "scheduler_*"
# Generate JUnit XML for CI
python tools/uos-target/uos-target.py --run --junit results.xml
# Run with coverage instrumentation enabled
python tools/uos-target/uos-target.py --run --coverage
```
### Test Output Format
Tests embedded in the kernel must output results in this exact format:
```
[UOS-TEST] <test_name>: <pass|fail|skip> [<message>]
```
Example UART output:
```
[UOS-TEST] scheduler_round_robin: pass
[UOS-TEST] scheduler_preemption: pass
[UOS-TEST] memory_alloc_null: fail [null pointer not caught]
[UOS-TEST] ipc_message_size: skip [not implemented on this platform]
```
### Configuration
`uos-target.py` reads target definitions from `tools/uos-target/targets/`:
```yaml
# tools/uos-target/targets/qemu-arm-virt.yaml
name: qemu-arm-virt
arch: armv7
platform: qemu-arm-virt
qemu_binary: qemu-system-arm
qemu_args:
- -M virt
- -cpu cortex-a15
- -m 512M
- -nographic
uart_timeout: 10
banner: "UniversalisOS"
```
---
## 3. Code Coverage: `uos-cover` Suite
### Purpose
The `uos-cover` suite provides precise, bare-metal ARINC/ISO 26262 compliant coverage analysis for firmware builds. It achieves MC/DC, branch, and statement coverage through source-level C instrumentation.
### Tools
| Tool | Command | Purpose |
|------|---------|---------|
| `uos_cins.py` | `python tools/uos-cover/uos_cins.py` | Instrument C/C++ source before compilation |
| `uos_cover.py` | `python tools/uos-cover/uos_cover.py` | Main coverage orchestrator |
| `uos_covparse.py` | `python tools/uos-cover/uos_covparse.py` | Parse trace data into coverage reports |
| `uos_trace.py` | `python tools/uos-cover/uos_trace.py` | Parse raw trace files from instrumented runs |
| `uos_justify.py` | `python tools/uos-cover/uos_justify.py` | Justify unreachable code paths |
| `uos_verify.py` | `python tools/uos-cover/uos_verify.py` | Verify coverage against requirements |
| `uos_covexport.py` | `python tools/uos-cover/uos_covexport.py` | Export coverage data (HTML, CSV, JSON) |
| `uos_package.py` | `python tools/uos-cover/uos_package.py` | Package coverage artifacts for submission |
| `uos_xst.py` | `python tools/uos-cover/uos_xst.py` | Cross-source traceability analysis |
### Coverage Workflow
```bash
# 1. Instrument source code
python tools/uos-cover/uos_cins.py \
--input kernel/src/ \
--output kernel/src_instrumented/
# 2. Build instrumented firmware
cd kernel
make ARCH=armv7 PLATFORM=qemu-arm-virt \
SRC_DIR=src_instrumented/
# 3. Run tests with coverage (uos-target handles trace extraction)
python tools/uos-target/uos-target.py --run --coverage
# 4. Parse trace data
python tools/uos-cover/uos_covparse.py \
--trace build/trace.txt \
--output coverage.json
# 5. Verify against requirements
python tools/uos-cover/uos_verify.py \
--coverage coverage.json \
--requirements tools/doorstop-integration/
# 6. Generate HTML report
python tools/uos-cover/uos_covexport.py \
--input coverage.json \
--format html \
--output coverage_report/
# 7. Package for submission
python tools/uos-cover/uos_package.py \
--input coverage_report/ \
--output coverage_package.zip
```
### Coverage Levels (DO-178C / ISO 26262)
| Level | Metric | Threshold | Tool |
|-------|--------|-----------|------|
| Statement | % of executable statements | 100% | `uos_covparse.py` |
| Branch | % of decision branches | 100% | `uos_covparse.py` |
| MC/DC | Modified Condition/Decision | 100% | `uos_covparse.py --mcdc` |
| Function | % of functions called | 100% | `uos_covparse.py` |
### Justifying Unreachable Code
For code that cannot be exercised (e.g., hardware-failure paths, defensive checks):
```bash
python tools/uos-cover/uos_justify.py \
--file kernel/src/core/scheduler.cpp \
--line 245 \
--reason "Unreachable: watchdog timeout triggers system reset before this path" \
--category "hardware_failure"
```
Justifications are stored in `.uos_justifications.json` and included in verification reports.
---
## 4. Requirements Traceability
### Higher-Level Requirements (HLR)
HLRs are managed by **Doorstop** at `tools/doorstop-integration/`:
```bash
# Create new HLR
cd tools/doorstop-integration
python doorsetup.py --create HLR --id REQ-006 \
--text "The scheduler shall support round-robin partitioning"
# Validate all requirements
python doors_export.py --validate
# Generate traceability matrix
python doors_export.py --format csv --output trace_matrix.csv
```
HLRs are viewed and edited via:
- **Web**: `janela-do-desassossego-web` (DOORS-like grid at `/doors`)
- **IDE**: `aurelio-theia` sidebar widget (`HLR Requirements` view)
- **VS Code**: `aurelio-vscode` panel (`Aurelio: Open HLR Requirements` command)
### Lower-Level Requirements (LLR)
LLRs are managed in **Plane** (`plane.portugalfuturista.org`) and tied directly to the CI execution pipeline.
### Bidirectional Sync
```bash
# Push HLRs to Plane (create/update LLRs)
python plane_bridge.py --sync
# Pull LLR status from Plane
python plane_bridge.py --pull --format yaml
```
---
## 5. CI Integration
### GitHub Actions Pipeline
The CI pipeline runs all three toolchain components in sequence:
```yaml
# .github/workflows/ci.yml (relevant jobs)
jobs:
uos-target-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run firmware tests
run: python tools/uos-target/uos-target.py --run --junit test-results.xml
- name: Upload test results
uses: actions/upload-artifact@v4
with:
name: test-results
path: test-results.xml
uos-cover-analysis:
needs: uos-target-test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Instrument source
run: python tools/uos-cover/uos_cins.py --input kernel/src/ --output kernel/src/
- name: Build with coverage
run: cd kernel && make ARCH=armv7 PLATFORM=qemu-arm-virt
- name: Run tests with coverage
run: python tools/uos-target/uos-target.py --run --coverage
- name: Parse coverage
run: python tools/uos-cover/uos_covparse.py --trace build/trace.txt --output coverage.json
- name: Verify coverage
run: python tools/uos-cover/uos_verify.py --coverage coverage.json --requirements tools/doorstop-integration/
- name: Export HTML report
run: python tools/uos-cover/uos_covexport.py --input coverage.json --format html --output coverage/
- name: Upload coverage
uses: actions/upload-artifact@v4
with:
name: coverage-report
path: coverage/
requirements-coverage:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Validate Doorstop requirements
run: python tools/doorstop-integration/doors_export.py --validate
- name: Check traceability
run: python tools/doorstop-integration/doors_export.py --check-links
```
---
## 6. Quick Reference
### Running Tests Locally
```bash
# Full test suite with coverage
python tools/uos-target/uos-target.py --run --coverage
# Specific architecture only
python tools/uos-target/uos-target.py --arch riscv --run
# Quick smoke test (no coverage)
cd kernel && ./uos-check.sh test
```
### Coverage Commands
```bash
# Instrument → Build → Test → Report (full pipeline)
python tools/uos-cover/uos_cover.py --full-pipeline --arch armv7
# Justify unreachable code
python tools/uos-cover/uos_justify.py --file <file> --line <line> --reason "<reason>"
# Export coverage for specific requirement
python tools/uos-cover/uos_covexport.py --input coverage.json --requirement REQ-001 --format html
```
### Requirements Commands
```bash
# View requirements in browser (Doorstop backend must be running)
curl http://192.168.0.9:8100/api/v1/requirements
# Validate
python tools/doorstop-integration/doors_export.py --validate
# Sync with Plane
python tools/doorstop-integration/plane_bridge.py --sync
```
---
## 7. File Locations
| Path | Purpose |
|------|---------|
| `tools/uos-target/` | Test execution engine (QEMU orchestration + UART parsing) |
| `tools/uos-cover/` | Coverage instrumentation and analysis suite |
| `tools/doorstop-integration/` | HLR requirements management (Doorstop backend) |
| `tools/uos-pkg/` | Package build tooling |
| `kernel/uos-check.sh` | Legacy build + boot smoke tests |
| `.github/workflows/ci.yml` | CI pipeline configuration |
---
## 8. References
- [DO-178C](https://www.rtca.org/sc-205/) — Software Considerations in Airborne Systems
- [ISO 26262](https://www.iso.org/standard/68383.html) — Road vehicles functional safety
- [ARINC 653](https://www.aviation-ia.com/standards) — Avionics application standard interface
- [Doorstop](https://doorstop.readthedocs.io/) — Requirements management tool
- [uos-cover README](../tools/uos-cover/README.md) — Coverage tooling details
- [uos-target README](../tools/uos-target/README.md) — Test execution details
Reference in a new issue View git blame Copy permalink