universalisos/RTOS_AUDIT.md

# RTOS Deep Audit — rtos_ref Collection

**Date**: 2026-07-14
**Auditor**: Hermes Agent (subagent)
**Scope**: f9-kernel, BORPH, ReconfROS, rteval + bonus RTOSes in `/home/fabiorafaelcoutada/portugalfuturista/rtos_ref/`

---

## Projects Found in rtos_ref/

| Project | Type | Relevance to UniversalisOS |
|---------|------|---------------------------|
| **f9-kernel** | L4 microkernel for ARM Cortex-M (MPU) | **PRIMARY REFERENCE** — closest to our no-MMU hypervisor |
| **seL4** | Formal-verified L4 microkernel | High — verification methodology, capability system |
| **threadx** | Azure RTOS (commercial RTOS) | High — PTS scheduling, industrial RTOS patterns |
| **ChibiOS** | Lightweight RTOS for Cortex-M | Medium — efficient RT primitives, HAL patterns |
| **rt-thread** | Chinese RTOS (IoT focus) | Medium — component-based architecture |
| **nuttx** | POSIX-compatible RTOS | Medium — POSIX compliance on no-MMU |
| **contiki** | IoT OS (proto-threads) | Low — cooperative multitasking model |
| **rodos** | Real-time OS for space | Low — deterministic scheduling reference |
| **rtems** | Real-time for embedded (POSIX) | Medium — full POSIX on embedded |
| **BORPH** | FPGA-aware Linux extension | Low — FPGA/hardware co-design |
| **ReconfROS** | ROS+FPGA trail follower | Low — not an RTOS, application project |
| **rteval** | Linux RT benchmark tool | Low — latency measurement, not an RTOS |

**FreeRTOS, Mbed OS, Zephyr**: Not found on disk (no repositories in `/home/fabiorafaelcoutada/portugalfuturista/`).
**Azure RTOS (ThreadX)**: Found at `rtos_ref/threadx/`.

---

## 1. f9-Kernel (DEEP AUDIT — PRIMARY REFERENCE)

### 1.1 Architecture Support
- **ARM Cortex-M4/M4F** (ARMv7-M) exclusively
- STM32F407VG, STM32F429ZI, STM32L475VG boards
- QEMU emulation (B-L475E-IOT01A machine with MPU+FPU emulation)
- No RISC-V, no x86, no AArch64 — pure Cortex-M

### 1.2 Memory Model
- **No virtual memory, no MMU** — physically addressed
- **ARM MPU (8 regions)** for hardware-enforced isolation
- Three-tier memory abstraction:
  - **Memory pools** (`mempool_t`): Static regions of physical address space with permission flags (KTEXT, UTEXT, KDATA, UDATA, UDEVICES, AVAILABLE)
  - **Flexible pages** (`fpage_t`): MPU-compatible regions (power-of-2 sized, aligned). Chains of fpages represent non-power-of-2 allocations (e.g., 96 bytes = 32+64 byte fpages)
  - **Address spaces** (`as_t`): Sorted linked list of fpages, refcounted, shared between threads
- Memory mapping operations: MAP, GRANT, UNMAP (L4 semantics)
- IPC-typed items carry MapItem/GrantItem for cross-AS memory transfer
- MPU region management: LRU eviction (`mpu_select_lru`) when 8 regions insufficient
- Stack overflow detection: canary value (0xDEADBEEF) at stack base, checked on context switch

**Key Design Pattern for UniversalisOS**: f9 proves that L4-style address spaces work on MPU-only hardware by mapping "flexible pages" to MPU regions instead of page tables. The 8-region constraint is managed via LRU eviction and priority-based allocation (PC region > always-mapped > others).

### 1.3 Kernel Primitives
- **Threads**: Full L4 thread model — TCB with parent/child/sibling tree, global IDs (14-bit shifted), UTCBs
- **Scheduling**: 32-level priority bitmap scheduler with O(1) CLZ selection
  - Preemption-Threshold Scheduling (PTS): ThreadX-style, controls which priorities can preempt
  - Priority Inheritance Protocol (PIP): automatic boost on IPC blocking
  - Round-robin within same priority level
  - Tickless operation for power efficiency
- **IPC**: Synchronous L4-style message passing
  - Short IPC: MR0-MR7 in hardware registers (R4-R11), zero-copy
  - Extended IPC: MR8-MR39 in TCB-embedded buffer (128 bytes), MR40-MR47 in UTCB
  - Fastpath for common case (~100 cycles), slowpath via softirq
  - Typed items for memory mapping (MapItem/GrantItem)
  - Combined send+receive in one syscall (request-response pattern)
- **Notifications**: Lightweight event-chaining system
  - Bit-mask notifications (32-bit `notify_bits`)
  - Async queue with softirq delivery (bounded batch size for RT safety)
  - Fast-path: `notification_post_softirq()` (~100 cycles)
  - Wait mask for selective blocking (`L4_NotifyWait`)
- **Timers**: Kernel timer subsystem with tickless scheduling
- **Interrupts**: NVIC integration, IRQ-to-thread IPC delivery

### 1.4 API Surface
**Native (L4-family) syscalls** — only 2 core syscalls:
- `L4_Ipc(to, from, timeout, *from)` — universal IPC (send/receive/both)
- `L4_ThreadControl(tid, spaceid, scheduler, pager, utcb)` — thread lifecycle
- Extensions: `L4_Schedule`, `L4_SpaceControl`, `L4_ExchangeRegisters`, `L4_SystemClock`
- Embedded extensions: `L4_TimerNotify`, `L4_NotifyWait`, `L4_NotifyPost`, `L4_NotifyClear`

**POSIX layer (user-space, PSE51/PSE52)**:
- Threads: `pthread_create/join/detach/cancel`
- Mutexes: normal, recursive, errorcheck; `pthread_mutex_timedlock`
- Condvars: wait/signal/broadcast with sequence-based atomicity
- Semaphores: `sem_init/wait/post/trywait/getvalue`
- RW locks and barriers (PSE52)
- Scheduling: `SCHED_FIFO`, `SCHED_RR`, `SCHED_OTHER`
- Signals: `sigwait`, `pthread_sigmask`, `sigaction`
- Clock/time: `clock_gettime`, `nanosleep`, `timer_create`
- Spinlocks: TTAS pattern with LDREX/STREX

### 1.5 Key Design Patterns for No-MMU Targets
1. **Flexible pages as MPU regions**: Power-of-2 splitting, chaining for arbitrary sizes
2. **LRU MPU region eviction**: Only 8 hardware regions → software manages priority-based allocation
3. **Physical address space as memory pools**: Static memmap table with permissions, no dynamic allocation
4. **UTCBs always-mapped**: Fast syscall argument access without MPU region change
5. **IPC fastpath in registers**: MR0-MR7 in R4-R11, zero memory access for small messages
6. **Stack canary protection**: Detects overflow without guard pages (no MMU to provide them)
7. **KIP (Kernel Interface Page)**: Always-mapped read-only region exposing kernel metadata to userspace

### 1.6 Isolation Mechanisms
- **MPU-based memory isolation**: Each address space gets its own MPU region configuration
- **Kernel/user separation**: KTEXT mapped kernel-only, UTEXT/UDATA mapped user-accessible
- **Device isolation**: Device memory regions mapped with specific permissions per AS
- **Reference-counted address spaces**: Threads sharing AS via refcount, cleanup on last put
- **IPC message validation**: Alignment checks on MapItem addresses (reject unaligned from userspace)
- **Privilege separation**: `thread_ispriviliged()` gates sensitive operations (UTCB writes, memory grants)
- **Stack overflow detection**: Canary-based (not guard pages — no MMU)

---

## 2. BORPH

### 2.1 Architecture Support
- **x86** (Linux kernel patch on x86/Kconfig)
- Targets FPGA-based reconfigurable computers (NetFPGA, RHINO, ROACH, BEE2-FPGA)

### 2.2 Memory Model
- Standard Linux virtual memory (BORPH is a Linux kernel extension, not standalone)
- FPGA hardware regions (HWRs) mapped as device memory

### 2.3 Kernel Primitives
- BOF (BORPH Object File) binary format — encapsulates ELF + FPGA bitstream
- Hardware Regions (HWRs): kernel-managed FPGA resources with UNIX process model
- `bkexecd`: kernel daemon for FPGA loading
- `/proc/borph/` filesystem interface for hardware status

### 2.4 API Surface
- Standard UNIX process model extended for FPGA
- FPGA hardware exposed via procfs
- BOF executable format (`binfmt_bof`)

### 2.5 Key Design Patterns for No-MMU Targets
- **Not applicable** — BORPH targets Linux with full MMU
- Interesting concept: treating FPGA regions as "processes" with UNIX semantics

### 2.6 Isolation Mechanisms
- Linux kernel isolation (standard process isolation)
- FPGA regions isolated by hardware (separate physical fabric)

**Relevance to UniversalisOS**: Low. BORPH is a Linux kernel extension for FPGA co-design. The HWR abstraction (treating hardware as schedulable resources) is conceptually interesting for hypervisor-level hardware partitioning, but the implementation is deeply tied to Linux/MMU.

---

## 3. ReconfROS

### 3.1 Architecture Support
- **ARM (Xilinx Zynq)** — PYNQ-Z2 FPGA board (xc7z020clg400-1)
- ROS Melodic on Linux

### 3.2 Memory Model
- Standard Linux (Zynq has MMU)
- FPGA HLS IPs communicate via memory-mapped AXI interfaces

### 3.3 Kernel Primitives
- **Not a kernel** — this is a ROS application for FPGA-accelerated computer vision
- Trail detection pipeline: camera → FPGA (HLS IP) → ROS → navigation
- Hardware IPs: `mm2vs`, `vs2mm`, `trail_detection` (Vivado HLS)

### 3.4 API Surface
- ROS topics/services for inter-node communication
- Vivado HLS C++ API for FPGA IP design
- Dynamic reconfiguration via `rqt_reconfigure`

### 3.5 Key Design Patterns for No-MMU Targets
- **Not applicable** — standard Linux/Zynq with MMU

### 3.6 Isolation Mechanisms
- Standard Linux process isolation
- FPGA fabric provides hardware-level isolation between IP blocks

**Relevance to UniversalisOS**: Low. An application-level ROS project, not an RTOS. The HLS IP design pattern (C++ → hardware) could inform our FPGA acceleration strategy if we ever add FPGA support.

---

## 4. rteval

### 4.1 Architecture Support
- **x86_64** (Linux userspace tool)

### 4.2 Memory Model
- Standard Linux userspace

### 4.3 Kernel Primitives
- **Not a kernel** — Python-based RT benchmarking tool
- Measures Linux PREEMPT_RT kernel latency under load
- Runs `cyclictest` + `hackbench` + parallel kernel compile
- Statistical analysis of timer latency histograms

### 4.4 API Surface
- Command-line tool: `rteval [options]`
- XML-RPC server for result aggregation
- DMI/system info collection

### 4.5 Key Design Patterns for No-MMU Targets
- **Not applicable** — Linux userspace benchmark tool

### 4.6 Isolation Mechanisms
- None (measurement tool, not isolation provider)

**Relevance to UniversalisOS**: Low as an implementation reference, but the **methodology** is valuable. rteval's approach of measuring worst-case latency under load (cyclictest + stress loads) is exactly what we need for validating UniversalisOS real-time guarantees.

---

## 5. Bonus RTOSes (Summary)

### 5.1 seL4 (Full L4 Microkernel)
- **Arch**: ARM (Cortex-A, Cortex-M experimental), RISC-V, x86, AArch64, x86_64
- **Memory**: Full MMU with formal verification of C implementation
- **Isolation**: Capability-based access control, formally verified
- **Relevance**: HIGH — verification methodology, capability model, L4 API surface

### 5.2 ThreadX (Azure RTOS)
- **Arch**: ARM (Cortex-M/A), RISC-V, x86, MIPS, PIC32, etc.
- **Memory**: MPU support on Cortex-M, MMU on Cortex-A
- **Isolation**: ThreadX-Modules (optional MPU-based module isolation)
- **Relevance**: HIGH — PTS scheduling (f9 already implements this), industrial RT patterns

### 5.3 ChibiOS
- **Arch**: ARM (Cortex-M), AVR, STM8, RISC-V
- **Memory**: No-MMU, flat memory with optional MPU support
- **Isolation**: MPU regions per thread (optional)
- **Relevance**: MEDIUM — efficient RT primitives, HAL abstraction

### 5.4 RT-Thread
- **Arch**: ARM (Cortex-M/A), RISC-V, MIPS, x86
- **Memory**: Component-based, supports both MMU and MPU targets
- **Isolation**: Process isolation on MMU targets (rt-smart)
- **Relevance**: MEDIUM — component-based architecture pattern

### 5.5 NuttX
- **Arch**: ARM, RISC-V, x86, MIPS, Xtensa, etc.
- **Memory**: Flat or protected (MPU on Cortex-M)
- **Isolation**: POSIX-compatible, optional MPU protection
- **Relevance**: MEDIUM — full POSIX compliance on no-MMU

### 5.6 RTEMS
- **Arch**: ARM, RISC-V, x86, PowerPC, SPARC, MIPS, etc.
- **Memory**: Supports both MMU and MPU targets
- **Isolation**: POSIX 1003.1b compliance, partition-based isolation
- **Relevance**: MEDIUM — POSIX on embedded, partition model

### 5.7 Contiki
- **Arch**: ARM, AVR, MSP430, x86
- **Memory**: Flat memory, proto-threads (cooperative, stackless)
- **Isolation**: None (single address space, cooperative)
- **Relevance**: LOW — cooperative model doesn't fit our preemption needs

### 5.8 RODOS
- **Arch**: ARM, x86, RISC-V
- **Memory**: Flat memory for embedded targets
- **Isolation**: Deterministic scheduling for space applications
- **Relevance**: LOW — niche (space), but deterministic scheduling patterns useful

---

## 6. Comparative Analysis: Isolation on Microcontrollers

| Feature | f9-kernel | seL4 | ThreadX | ChibiOS | NuttX |
|---------|-----------|------|---------|---------|-------|
| MPU regions | 8 (Cortex-M) | N/A (uses MMU) | 8 (optional) | 8 (optional) | 8 (optional) |
| Address spaces | Yes (L4-style) | Yes (capabilities) | Modules only | No | Optional |
| IPC isolation | Synchronous, typed | Capabilities | Message queues | None | Pipes/mqueues |
| Stack protection | Canary | Guard pages (MMU) | Canary | Canary | Canary |
| Memory mapping | Map/Grant via IPC | Cap-grant | None | None | None |
| Formal verification | No | Yes | No | No | No |

**Key insight for UniversalisOS**: f9-kernel is the **only** project that implements full L4-style address spaces on a Cortex-M MPU. seL4 has the verification methodology but targets MMU hardware. ThreadX has PTS scheduling but only optional module isolation. f9's approach of mapping fpages to MPU regions is the most directly applicable pattern for our hypervisor.

---

## 7. Recommendations for UniversalisOS

### From f9-kernel (most applicable):
1. **MPU-as-page-table pattern**: Use flexible pages mapped to MPU regions as the isolation primitive
2. **LRU eviction for limited MPU regions**: f9's `mpu_select_lru()` handles the 8-region constraint
3. **IPC fastpath in registers**: MR0-MR7 in R4-R11 for zero-copy small messages
4. **PTS scheduling**: ThreadX-style preemption-threshold for deterministic real-time
5. **Notification objects**: Lightweight event-chaining for interrupt→thread delivery
6. **Stack canary without guard pages**: Critical for no-MMU stack protection
7. **KIP pattern**: Always-mapped metadata region for fast kernel queries
8. **Softirq architecture**: Deferred processing for RT-safe bounded latency

### From seL4:
- Capability-based access control (formally verifiable)
- Verification methodology for kernel correctness proofs

### From ThreadX:
- Industrial RTOS scheduling patterns (f9 already implements PTS from ThreadX)
- Module isolation model for partitioned execution

### From rteval:
- RT benchmarking methodology: cyclictest + stress loads → statistical analysis
- Use this approach to validate UniversalisOS real-time guarantees

### Not found:
- FreeRTOS, Mbed OS, Zephyr repositories were NOT found on disk
- These should be cloned if we need them as references