# ESP-QEMU CCOUNT/CCOMPARE Timer & ESP32 Memory Model Audit

**Source**: `/home/fabiorafaelcoutada/portugalfuturista/hypervisor_ref/esp-qemu/`
**Date**: 2026-07-15
**Context**: Debugging timer wheel hang — tasks wake once then system freezes. Timer fires repeatedly but tasks don't wake from second delay.

---

## 1. How QEMU Increments CCOUNT — Virtual Clock, NOT Instruction Count

**Answer: CCOUNT is derived from `QEMU_CLOCK_VIRTUAL` (nanosecond wall-clock of the VM), not from an instruction counter.**

### Mechanism

CCOUNT is lazily computed on every read/write. It is NOT incremented per-instruction. The value is derived from the elapsed virtual-clock nanoseconds converted to CPU clock ticks:

**`target/xtensa/op_helper.c:39-47`** — `HELPER(update_ccount)`:
```c
void HELPER(update_ccount)(CPUXtensaState *env)
{
    XtensaCPU *cpu = env_archcpu(env);
    uint64_t now = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);   // <-- virtual clock

    env->ccount_time = now;
    env->sregs[CCOUNT] = env->ccount_base +
        (uint32_t)clock_ns_to_ticks(cpu->clock, now - env->time_base);
}
```

- `ccount_base` is an offset adjusted when the guest writes CCOUNT (`wsr_ccount`, line 54).
- `time_base` is set once at IRQ init to the virtual clock value at that time (`pic_cpu.c:107`).
- `clock_freq_khz = 40000` for ESP32 → 240 MHz... wait, 40000 kHz = 40 MHz? No — `core-esp32.c:34`: `.clock_freq_khz = 40000` → **40 MHz base clock**. But the ESP32 actual CPU frequency can be changed at runtime via `esp32_clk_update` (`esp32.c:224-246`) which calls `clock_update_hz()`.

### When CCOUNT is Updated

CCOUNT is updated (re-synced from virtual clock) on these operations:
- **`rsr CCOUNT`** — `translate.c:2068-2076`: calls `gen_helper_update_ccount` before reading.
- **`wsr CCOUNT`** — `translate.c:2449-2455`: calls `gen_helper_wsr_ccount`.
- **`xsr CCOUNT`** — `translate.c:2640-2653`: calls `gen_helper_update_ccount` then `wsr_ccount`.
- **`wsr CCOMPAREn`** — indirectly via `update_ccompare` → `update_ccount` (`op_helper.c:67`).
- **`waiti`** — not directly, but `check_interrupts` reads INTSET/INTENABLE.

**Key Implication**: Between reads, CCOUNT in `env->sregs[CCOUNT]` is STALE. It only reflects the true virtual-clock-derived value when `update_ccount` runs. QEMU TCG does not increment it per-instruction.

### CCOUNT ↔ CCOMPARE Timer

The CCOMPARE timer is a QEMU `QEMUTimer` on the virtual clock:

**`hw/xtensa/pic_cpu.c:106-113`** — `xtensa_irq_init`:
```c
if (xtensa_option_enabled(env->config, XTENSA_OPTION_TIMER_INTERRUPT)) {
    env->time_base = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
    env->ccount_base = env->sregs[CCOUNT];
    for (i = 0; i < env->config->nccompare; ++i) {
        env->ccompare[i].env = env;
        env->ccompare[i].timer = timer_new_ns(QEMU_CLOCK_VIRTUAL,
                xtensa_ccompare_cb, env->ccompare + i);
    }
}
```

The timer is armed in `update_ccompare` (`op_helper.c:60-72`):
```c
void HELPER(update_ccompare)(CPUXtensaState *env, uint32_t i)
{
    XtensaCPU *cpu = env_archcpu(env);
    uint64_t dcc;

    qatomic_and(&env->sregs[INTSET], ~(1u << env->config->timerint[i]));  // clear timer IRQ
    HELPER(update_ccount)(env);
    dcc = (uint64_t)(env->sregs[CCOMPARE + i] - env->sregs[CCOUNT] - 1) + 1;
    timer_mod(env->ccompare[i].timer,
              env->ccount_time + clock_ticks_to_ns(cpu->clock, dcc));
    env->yield_needed = 1;
}
```

- `dcc` = delta in CCOUNT ticks between CCOMPARE and current CCOUNT.
- `timer_mod` schedules the QEMU timer at `ccount_time + dcc_in_ns` on the virtual clock.

---

## 2. CCOMPARE0 Firing: Does QEMU Auto-Clear INTSET?

**Answer: QEMU does NOT auto-clear the timer interrupt bit in INTSET when CCOMPARE fires. The guest MUST clear it.**

### What Happens When the Timer Fires

**`hw/xtensa/pic_cpu.c:85-92`** — `xtensa_ccompare_cb`:
```c
static void xtensa_ccompare_cb(void *opaque)
{
    XtensaCcompareTimer *ccompare = opaque;
    CPUXtensaState *env = ccompare->env;
    unsigned i = ccompare - env->ccompare;

    qemu_set_irq(env->irq_inputs[env->config->timerint[i]], 1);
}
```

This calls `qemu_set_irq(..., 1)` which routes to `xtensa_set_irq`:

**`hw/xtensa/pic_cpu.c:66-83`** — `xtensa_set_irq`:
```c
static void xtensa_set_irq(void *opaque, int irq, int active)
{
    CPUXtensaState *env = opaque;
    ...
    uint32_t irq_bit = 1 << irq;

    if (active) {
        qatomic_or(&env->sregs[INTSET], irq_bit);       // SET the bit
    } else if (env->config->interrupt[irq].inttype == INTTYPE_LEVEL) {
        qatomic_and(&env->sregs[INTSET], ~irq_bit);     // only LEVEL type auto-clears on deassert
    }
    check_interrupts(env);
}
```

### Critical: Timer Interrupts Are INTTYPE_TIMER, NOT INTTYPE_LEVEL

For ESP32, the timer interrupts map as follows:

**`core-esp32/core-isa.h:417, 426-427, 455-457`**:
```
Interrupt 6  (CCOMPARE0) → INTTYPE_TIMER  (XTHAL_INTTYPE_TIMER)
Interrupt 15 (CCOMPARE1) → INTTYPE_TIMER
Interrupt 16 (CCOMPARE2) → INTTYPE_TIMER
```

Since `inttype == INTTYPE_TIMER` (not `INTTYPE_LEVEL`), the `else if` branch in `xtensa_set_irq` **never fires** for timer interrupts. The bit stays set in INTSET once asserted.

### How the Guest Clears It

The guest must clear the timer interrupt via `wsr intclear` or `wsr ccompare` (which also clears):

**`target/xtensa/exc_helper.c:128-143`** — `HELPER(intset)` / `HELPER(intclear)`:
```c
void HELPER(intclear)(CPUXtensaState *env, uint32_t v)
{
    intclear(env, v & (env->config->inttype_mask[INTTYPE_SOFTWARE] |
                       env->config->inttype_mask[INTTYPE_EDGE]));
}
```

**⚠️ CRITICAL BUG-CANDIDATE**: `intclear` only allows clearing `INTTYPE_SOFTWARE` and `INTTYPE_EDGE` masked bits. `INTTYPE_TIMER` bits are NOT in either mask. The only way to clear a timer interrupt is via `wsr CCOMPAREn` (which calls `update_ccompare`, clearing the specific timer bit at `op_helper.c:65-66`).

**`core-esp32/core-isa.h:446-449`** — masks:
```
INTTYPE_MASK_SOFTWARE  = 0x20000080   (ints 7, 29)
INTTYPE_MASK_EXTERN_EDGE = 0x50400400 (ints 10, 22, 28, 30)
INTTYPE_MASK_TIMER     = 0x00018040   (ints 6, 15, 16) ← NOT clearable via intclear!
```

### Summary of Clearing Paths for Timer IRQ (int 6/15/16)

| Operation | Clears timer bit? | Reference |
|-----------|------------------|-----------|
| `wsr CCOMPAREn` (same index) | ✅ YES | `op_helper.c:65-66` |
| `wsr intclear` | ❌ NO (INTTYPE_TIMER not in mask) | `exc_helper.c:139-142` |
| `xtensa_set_irq(irq, 0)` | ❌ NO (not INTTYPE_LEVEL) | `pic_cpu.c:77-78` |
| QEMU auto-clear on delivery | ❌ NO | no such code path exists |

**⚠️ This is the most likely cause of the timer wheel hang.** If the guest ISR does not re-write CCOMPARE0 (or writes a new CCOMPARE value), the interrupt bit stays set in INTSET. With PS.EXCM set during ISR, and the interrupt still pending, the system enters a tight re-exception loop or deadlocks.

---

## 3. CCOMPARE Re-arming from ISR Context — Known Issues

### The Re-arm Sequence

For the timer to fire again, the guest MUST:
1. Read current CCOUNT (or know it implicitly).
2. Write the new compare value to `wsr CCOMPARE0`.
3. This triggers `update_ccompare` which: (a) clears the timer IRQ bit, (b) re-arms the QEMU timer.

### Potential Issues

#### Issue A: INTSET bit never cleared if CCOMPARE not re-written
As detailed in section 2, the timer interrupt bit (int 6) stays in INTSET until `wsr CCOMPARE0` runs. If the guest only does `wsr intclear` (common pattern in FreeRTOS/RTOS ISRs), the bit remains set → `check_interrupts` (`pic_cpu.c:35-64`) sees the interrupt is still pending → immediately re-enters ISR → **livelock or hang**.

#### Issue B: yield_needed flag
`update_ccompare` sets `env->yield_needed = 1` (`op_helper.c:71`). This triggers a `HELPER(exception)(env, EXCP_YIELD)` at the next TCG exit. If the yield mechanism is not properly handled in the hypervisor/OS context, this can cause the CPU to stall.

#### Issue C: Timer fires while in EXCM state
When the timer fires and the CPU is already in EXCM (exception mode, e.g., inside another ISR or exception handler), `handle_interrupt` (`exc_helper.c:161-203`) checks:
```c
if (level > xtensa_get_cintlevel(env) && ...)
```
For CCOMPARE0 at level 1 (`core-isa.h:379`: `XCHAL_INT6_LEVEL = 1`), if `cintlevel >= 1` (which it is when PS.EXCM is set, since `excm_level = 3`), the interrupt will NOT be delivered immediately. It remains pending. This is correct behavior but means:
- If the ISR runs at level ≥ 1 (which it always does with EXCM), a timer that fires DURING the ISR is deferred.
- When the ISR returns via `rfi`, EXCM is cleared, cintlevel drops, and the pending timer interrupt fires — but only if INTSET still has the bit set AND INTENABLE has it enabled.

#### Issue D: Timer interrupt at level 1 only
CCOMPARE0 → int 6 → **level 1** (`core-isa.h:379`). `XCHAL_EXCM_LEVEL = 3` (`core-isa.h:351`). This means PS.EXCM masks level 1-3. A level-1 timer interrupt CANNOT preempt any exception handler. The OS must use `rfi` to lower cintlevel to 0 for the timer ISR to run.

**For the timer wheel hang**: If the first delay works (timer fires, ISR runs, task wakes), but the second delay fails, the most likely scenario is:
1. First ISR clears the timer condition by writing CCOMPARE0 (re-arming).
2. Second fire happens, but either: (a) INTENABLE was cleared and not re-enabled, (b) the CCOMPARE write didn't happen because the ISR returned before re-arming, or (c) the `update_ccompare` `timer_mod` computed `dcc` from a stale CCOUNT (if `update_ccount` wasn't called recently, the virtual clock may have advanced significantly).

---

## 4. ESP32 DRAM Memory Map

### Primary Memory Regions

**`hw/xtensa/esp32.c:61-76`** — `esp32_memmap[]`:

| Region | Base | Size | Type | Notes |
|--------|------|------|------|-------|
| DROM | `0x3FF90000` | 0x10000 (64 KB) | ROM | Alias of IROM+0x60000, per-CPU |
| IROM | `0x40000000` | 0x70000 (448 KB) | ROM | Per-CPU, flash cache |
| **DRAM** | **`0x3FFAE000`** | **0x52000 (336 KB)** | **RAM** | **Main SRAM, shared** |
| IRAM | `0x40080000` | 0x40000 (256 KB) | RAM | Instruction RAM, shared |
| ICACHE0 | `0x40070000` | 0x8000 (32 KB) | RAM | Shared |
| ICACHE1 | `0x40078000` | 0x8000 (32 KB) | RAM | Shared |
| RTCSLOW | `0x50000000` | 0x2000 (8 KB) | RAM | RTC slow memory, shared |
| RTCFAST_I | `0x400C0000` | 0x2000 (8 KB) | RAM | PRO CPU only |
| RTCFAST_D | `0x3FF80000` | 0x2000 (8 KB) | RAM | Alias of RTCFAST_I, PRO CPU only |
| FRAMEBUF | `0x20000000` | variable | RAM | Virtual framebuffer |

### DRAM Details

- **Base**: `0x3FFAE000`
- **Size**: `0x52000` = 336 KB
- **End**: `0x3FFAE000 + 0x52000 = 0x40000000`
- **Type**: `memory_region_init_ram` — plain RAM, shared between both CPUs (`sys_mem`)
- **Allocation**: `esp32.c:300-302`:
  ```c
  memory_region_init_ram(dram, NULL, "esp32.dram", 0x52000, &error_fatal);
  memory_region_add_subregion(sys_mem, 0x3FFAE000, dram);
  ```

### Memory Aliases

1. **DROM is an alias of IROM**: `esp32.c:296`:
   ```c
   memory_region_init_alias(drom, NULL, name, irom, 0x60000, 0x10000);
   ```
   DROM at `0x3FF90000` maps to IROM offset `0x60000` → physical `0x40060000`.

2. **RTCFAST_D is an alias of RTCFAST_I**: `esp32.c:326`:
   ```c
   memory_region_init_alias(rtcfast_d, NULL, "esp32.rtcfast_d", rtcfast_i, 0, 0x2000);
   ```

3. **APB register mirror**: Peripheral devices registered at both DPORT base and APB base (`esp32.c:248-257`):
   ```c
   // DPORT base and APB base mirror
   memory_region_add_subregion_overlap(dest, dport_base_addr, mr, 0);
   memory_region_add_subregion_overlap(dest, dport_base_addr - DR_REG_DPORT_APB_BASE + APB_REG_BASE, mr_apb, 0);
   ```

4. **PSRAM (optional)**: If `s->dport.has_psram`, a cache region `dram1` is overlaid at `dram1->base` (`esp32.c:365-372`). This extends DRAM into the `0x3F800000` range via the flash cache controller.

### CPU-Specific vs Shared Memory

- **Per-CPU** (via `s->cpu_specific_mem[i]`): IROM, DROM, RTCFAST_I, RTCFAST_D, cache regions (drom0, iram0, dram1).
- **Shared** (via `sys_mem`): DRAM, IRAM, ICACHE0, ICACHE1, RTCSLOW, all peripherals.

---

## 5. Nested Exception / Double Exception Handling

### PS Register Fields

**`target/xtensa/cpu.h:658-665`** — `xtensa_get_cintlevel`:
```c
static inline int xtensa_get_cintlevel(const CPUXtensaState *env)
{
    int level = (env->sregs[PS] & PS_INTLEVEL) >> PS_INTLEVEL_SHIFT;
    if ((env->sregs[PS] & PS_EXCM) && env->config->excm_level > level) {
        level = env->config->excm_level;
    }
    return level;
}
```

For ESP32: `excm_level = 3` (`core-isa.h:351`). When PS.EXCM is set, cintlevel is at least 3, masking all level 1-3 interrupts.

### Exception Entry

**`target/xtensa/exc_helper.c:48-69`** — `HELPER(exception_cause)`:
```c
void HELPER(exception_cause)(CPUXtensaState *env, uint32_t pc, uint32_t cause)
{
    uint32_t vector;

    env->pc = pc;
    if (env->sregs[PS] & PS_EXCM) {
        // ALREADY in exception mode → DOUBLE EXCEPTION
        if (env->config->ndepc) {
            env->sregs[DEPC] = pc;      // save PC in DEPC
        } else {
            env->sregs[EPC1] = pc;
        }
        vector = EXC_DOUBLE;            // → double exception vector
    } else {
        env->sregs[EPC1] = pc;          // normal: save in EPC1
        vector = (env->sregs[PS] & PS_UM) ? EXC_USER : EXC_KERNEL;
    }

    env->sregs[EXCCAUSE] = cause;
    env->sregs[PS] |= PS_EXCM;          // set EXCM

    HELPER(exception)(env, vector);
}
```

### Interrupt-Driven Double Exception

**`target/xtensa/exc_helper.c:161-203`** — `handle_interrupt`:
```c
static void handle_interrupt(CPUXtensaState *env)
{
    int level = env->pending_irq_level;

    if ((level > xtensa_get_cintlevel(env) && ...) || level == env->config->nmi_level) {
        if (level > 1) {
            // High-priority interrupt
            env->sregs[EPC1 + level - 1] = env->pc;    // save PC
            env->sregs[EPS2 + level - 2] = env->sregs[PS]; // save PS
            env->sregs[PS] = (env->sregs[PS] & ~PS_INTLEVEL) | level | PS_EXCM;
            env->pc = relocated_vector(env, env->config->interrupt_vector[level]);
        } else {
            // Level-1 interrupt → converted to exception
            env->sregs[EXCCAUSE] = LEVEL1_INTERRUPT_CAUSE;

            if (env->sregs[PS] & PS_EXCM) {
                // Already in EXCM → DOUBLE EXCEPTION
                env->sregs[DEPC] = env->pc;   // (if ndepc)
                cs->exception_index = EXC_DOUBLE;
            } else {
                env->sregs[EPC1] = env->pc;
                cs->exception_index = (env->sregs[PS] & PS_UM) ? EXC_USER : EXC_KERNEL;
            }
            env->sregs[PS] |= PS_EXCM;
        }
    }
}
```

### Double Exception Behavior Summary

1. **Normal exception (PS.EXCM=0)**: PC→EPC1, vector=EXC_USER/EXC_KERNEL, PS.EXCM set.
2. **Exception while EXCM=1**: PC→DEPC (if `ndepc` configured), vector=EXC_DOUBLE. For ESP32, `ndepc` is set (DEPC exists).
3. **Level-1 interrupt while EXCM=1**: Treated as double exception (EXCCAUSE=LEVEL1_INTERRUPT_CAUSE, EXC_DOUBLE vector). **This is the path a timer interrupt takes if it fires while already handling another level-1 exception.**
4. **High-priority interrupt (level > 1)**: Uses separate EPCn/EPSn registers. Can nest. PS.EXCM is set, INTLEVEL raised to the interrupt level.

### Double Exception Vector

**`target/xtensa/exc_helper.c:224-250`** — `xtensa_cpu_do_interrupt`:
```c
case EXC_DOUBLE:
    if (env->config->exception_vector[cs->exception_index]) {
        vector = env->config->exception_vector[cs->exception_index];
        env->pc = relocated_vector(env, vector);
    }
```

The double exception vector is fetched from `exception_vector[EXC_DOUBLE]`, which comes from `EXCEPTION_VECTORS` in `overlay_tool.h`, derived from `XCHAL_DOUBLEEXC_VECTOR_*` in `core-isa.h`.

---

## Summary: Root Cause Hypotheses for Timer Wheel Hang

### Most Likely: Timer INTSET bit not cleared between fires

**Mechanism**: CCOMPARE0 fires → int 6 set in INTSET → ISR runs → ISR does NOT re-write CCOMPARE0 (or writes intclear which doesn't clear INTTYPE_TIMER) → int 6 stays in INTSET → `check_interrupts` sees pending interrupt → on `rfi` lowering cintlevel, immediately re-enters ISR → infinite loop / hang.

**Evidence**: `pic_cpu.c:77-78` — only `INTTYPE_LEVEL` interrupts auto-clear on deassert. Timer is `INTTYPE_TIMER`. `exc_helper.c:139-142` — `intclear` only clears `INTTYPE_SOFTWARE | INTTYPE_EDGE`.

**Fix**: The guest ISR MUST write `wsr CCOMPARE0` with the next compare value, which calls `update_ccompare` and clears the bit (`op_helper.c:65-66`). Writing `wsr INTCLR` is insufficient.

### Secondary: yield_needed causing unexpected TCG exit

`update_ccompare` sets `yield_needed = 1` (`op_helper.c:71`). If the hypervisor doesn't handle `EXCP_YIELD` properly, the CPU may stall after the first timer re-arm.

### Tertiary: Virtual clock drift vs instruction count

Since CCOUNT is tied to virtual clock (not instruction count), the timer fires based on wall-clock VM time, not execution progress. In a heavily loaded QEMU or under TCG single-step, the virtual clock and instruction count can diverge significantly, causing timers to fire at unexpected points relative to code execution.