universalisos/HARD_REALTIME_EVALUATION.md

# UniversalisOS — Hard Real-Time Re-Evaluation

## The Brutal Truth

We built a cute microkernel that boots, runs tasks, has semaphores and mutexes,
and compiles for 12 targets. But if you put it in a hard real-time system today,
it would **miss deadlines and kill people**.

This document re-evaluates every component against hard real-time requirements
and defines what must change to achieve **certifiable hard real-time**.

---

## 1. What "Hard Real-Time" Actually Means

Hard real-time is NOT "fast". It is **deterministic and bounded**:

| Requirement | Meaning | Failure Mode |
|---|---|---|
| **Bounded ISR latency** | Worst-case interrupt-to-handler time is provable | Missed deadline → catastrophic |
| **O(1) scheduling** | Context switch time independent of task count | Jitter → missed deadline |
| **Bounded critical sections** | Interrupt-disabled time has a provable upper bound | Extended latency → deadline miss |
| **Priority inheritance** | Low-priority task can't block high-priority indefinitely | Priority inversion → deadline miss |
| **No unbounded allocation** | Memory is static, no fragmentation possible | Allocation failure → crash |
| **WCET analysis** | Every code path has a measured worst-case time | Unanalyzable → uncertifiable |
| **Temporal isolation** | Tasks can't steal each other's CPU time | Interference → cascade failure |
| **Spatial isolation** | Tasks can't corrupt each other's memory | Corruption → crash |
| **Stack monitoring** | Overflow detected BEFORE corruption | Silent corruption → crash |

---

## 2. Current Deficiencies (Honest Audit)

### 🔴 CRITICAL — Will Miss Deadlines

#### 2.1 Tick Handler is O(N) — `uos_tick.c:21`
```c
for (uint32_t i = 0; i < UOS_MAX_TASKS; i++) {  // ← LINEAR SCAN
    uos_task_t* t = &g_task_pool[i];
    if (t->state == UOS_TASK_BLOCKED && t->delay_until != 0) { ... }
}
```
**Problem**: The SysTick ISR iterates ALL tasks every tick. With 16 tasks at 1kHz,
that's 16,000 iterations/second inside an interrupt. The ISR execution time
grows linearly with task count. This is **unacceptable for hard RT**.

**Fix**: Use a **delta-list (timer wheel)** — sorted by expiry time.
ISR only checks the head. O(1) for the common case.

#### 2.2 No Interrupt Latency Guarantee
**Problem**: Critical sections use `cpsid i` (global interrupt disable).
There is no bound on how long interrupts stay disabled. The mutex and semaphore
code disable interrupts during the entire blocking operation path.

**Fix**: Use **short, bounded critical sections**. Never hold `cpsid i`
across a context switch or blocking call. Use PRIMASK save/restore with
a maximum-disable-time assertion.

#### 2.3 Priority Inheritance is Incomplete — `uos_mutex.c`
**Problem**: The mutex boosts the owner's priority when a waiter blocks,
but `mutex_pi_restore()` is a **no-op** — it never restores the original
priority when the mutex is released. This means:
- Priority stays artificially high forever
- Medium-priority tasks starve
- The "inheritance" is actually a permanent promotion

**Fix**: Store `orig_prio` in the mutex (not just the task).
On unlock, restore from the mutex's own field.

#### 2.4 No Stack Overflow Detection
**Problem**: Stack overflow corrupts adjacent memory silently.
No canary, no guard page, no MPU region check.

**Fix**: Stack canary at stack base (pattern fill + check on context switch).
On MPU targets: MPU region with no-access guard page below each stack.

#### 2.5 No Timer Service
**Problem**: The API declares `uos_timer_t` but there is no implementation.
Software timers (one-shot and periodic) are essential for hard RT.

**Fix**: Delta-list timer service driven by the tick handler.

### 🟡 SERIOUS — Must Fix for Certification

#### 2.6 No Deadline Monitoring
**Problem**: No mechanism to detect if a task exceeds its WCET.
AUTOSAR has timing protection. APEX has HM. We have nothing.

**Fix**: Per-task execution budget. Reset on activation.
If budget exhausted → ProtectionHook → configurable action.

#### 2.7 No Preemption Threshold
**Problem**: Every ready task with higher priority preempts.
With 32 priority levels and many tasks, this causes unnecessary context switches.

**Fix**: Preemption threshold (ThreadX pattern) — task specifies
minimum priority that can preempt it.

#### 2.8 No ISR Latency Measurement
**Problem**: We can't prove worst-case interrupt latency.
DO-178C / ISO 26262 require evidence.

**Fix**: Instrument every ISR entry/exit with cycle counter.
Store min/max latency. Provide API to retrieve.

#### 2.9 Scheduler Not Tickless-Capable
**Problem**: SysTick fires at fixed 1kHz regardless of workload.
Tickless idle reduces power and jitter.

**Fix**: When idle, calculate next wake time, reprogram SysTick
to skip ticks. Standard FreeRTOS/RTX pattern.

#### 2.10 No Cache Analysis Support
**Problem**: On Cortex-M4/M7 with cache, cache misses add
unpredictable latency. Hard RT requires cache analysis or
locking.

**Fix**: Optional cache locking API. WCET analysis considers
cache state.

### 🟢 GOOD — Already Hard-RT Compatible

- **O(1) bitmap scheduler** — `sched_highest_prio()` uses CLZ, constant time ✓
- **Static allocation** — No malloc, no fragmentation ✓
- **Priority bitmap** — Ready-queue lookup is O(1) ✓
- **No virtual memory** — No page fault latency ✓
- **Freestanding** — No libc dependencies ✓
- **Preemptive** — PendSV context switch works ✓

---

## 3. The Hard Real-Time Roadmap

### Phase HRT-1: Fix the Critical Violations (1-2 weeks)

| Item | What | Impact |
|---|---|---|
| HRT-1.1 | Delta-list timer wheel | O(1) tick handler |
| HRT-1.2 | Bounded critical sections | Provable ISR latency |
| HRT-1.3 | Complete priority inheritance | No priority inversion |
| HRT-1.4 | Stack canary + overflow check | Corruption prevention |
| HRT-1.5 | Software timer service | Periodic task support |

### Phase HRT-2: Certification Hooks (2-3 weeks)

| Item | What | Impact |
|---|---|---|
| HRT-2.1 | Execution budget monitoring | WCET enforcement |
| HRT-2.2 | ISR latency instrumentation | Measurable worst-case |
| HRT-2.3 | Preemption threshold | Reduced context switches |
| HRT-2.4 | Stack high-water mark | Stack sizing evidence |
| HRT-2.5 | Deadline monitoring | Deadline miss detection |

### Phase HRT-3: Advanced Hard RT (3-4 weeks)

| Item | What | Impact |
|---|---|---|
| HRT-3.1 | Tickless idle | Power + jitter reduction |
| HRT-3.2 | Static schedule tables | Time-triggered scheduling |
| HRT-3.3 | Dual-core lockstep | Cortex-R safety |
| HRT-3.4 | Cache locking API | Deterministic memory access |
| HRT-3.5 | Formal verification hooks | seL4-style proofs |

### Phase HRT-4: Certifiability (ongoing)

| Item | What | Impact |
|---|---|---|
| HRT-4.1 | WCET trace generation | DO-178C / ISO 26262 evidence |
| HRT-4.2 | MC/DC test coverage | DO-178C Level A |
| HRT-4.3 | Requirements traceability | Certification artifact chain |
| HRT-4.4 | Code coverage analysis | 100% MC/DC on safety-critical paths |

---

## 4. What This Changes About the Vision

### What stays the same:
- Universal `uos_*` API across all architectures ✓
- Personality shells (FreeRTOS, APEX, AUTOSAR, Mbed OS) ✓
- 12+ hardware targets ✓
- Tier 0/1/2/3 architecture tiers ✓

### What changes:
- **Priority order shifts**: Hard RT correctness > feature count > target count
- **Every new feature must prove bounded execution time** before merging
- **No blocking operations inside ISRs** — ever
- **No unbounded loops in kernel code** — ever
- **All critical sections must have measured maximum duration**
- **Memory must be 100% static** — no dynamic allocation after init
- **Every API must have a documented worst-case execution time**

### The new rule:
> A feature is not "done" until its worst-case execution time is bounded
> and measured. If you can't bound it, it doesn't ship.

---

## 5. Re-Prioritized Implementation Order

```
NOW:    HRT-1: Fix critical violations (delta-list, PI fix, stack canary)
NEXT:   HRT-2: Certification hooks (budget monitoring, ISR latency)
THEN:   Personality shells get hard-RT audit (FreeRTOS/APEX/AUTOSAR)
AFTER:  HRT-3: Advanced (tickless, schedule tables, cache locking)
LATER:  More targets (but each must pass HRT audit)
```

The personality shells are already built. The hardware ports work.
The gap is **determinism**. That's what we fix now.