From 3e2261d83176752f3ea21e0c97f330b4c1757437 Mon Sep 17 00:00:00 2001 From: salmonik Date: Tue, 4 Aug 2026 01:44:05 +0200 Subject: v0.5.9: fix OptiPlex heap crash — EBS retry must not rebuild the free list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause: on real hardware SetMode can invalidate the pre-captured ExitBootServices map key, so EBS fails and the retry refreshed the map via capture_memory_map -> init_heap, which REBUILT the free list from the raw map and wiped the allocation state of every pre-EBS live object (the display shadow and the vt grids). Their spans became free while still live, later allocations carved inside the shadow, and the display writes clobbered them — zeroed back-pointers (family A ptr=0x404443e0 block=0x0) and gray glyph pixels (family B 0xaaaaaa00aaaaaa = 0x00AAAAAA = PALETTE[7] fg-7 text colour, not a firmware scrub). Fix: refresh_and_exit now re-captures the map and returns a fresh EBS key without rebuilding the free list (refresh_map_key); the free list is the authoritative live-allocation state. Verified in QEMU under a forced EBS failure (SIMULATE_EBS_FAILURE test hook): verify ok, probe NOT free, flhead outside the shadow — identical to a clean boot, and the full shell sequence is crash-free. Also: removed the crash-family debug markers from the screen (boot-time heapdbg lines, vt verify marker, shadow marker) — the vt grid self-check stays serial-only. Version 0.5.8 -> 0.5.9. --- DEBUG_NOTES.md | 177 +++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 128 insertions(+), 49 deletions(-) (limited to 'DEBUG_NOTES.md') diff --git a/DEBUG_NOTES.md b/DEBUG_NOTES.md index d44c8bf..a1f2d75 100644 --- a/DEBUG_NOTES.md +++ b/DEBUG_NOTES.md @@ -3,37 +3,72 @@ Handoff state: **HEAD = `ab633ce`**; the mystery below is SOLVED and the verify rework is committed on top (see git log). +## ✅ ROOT CAUSE + FIX (this session) — ExitBootServices retry wipes the free list + +**Root cause (confirmed by QEMU simulation + OptiPlex screen markers):** + +1. `capture_memory_map` (main.rs, pre-EBS) → `init_heap` builds the free + list from the raw map; `map_key` stashed. +2. `gop::init` → `SetMode` — on real hardware SetMode may allocate/reallocate + memory, making `map_key` stale — then allocates the live display shadow + (first heap alloc, `mem::forget`'d) and `vt::init` builds the cells grids. +3. `ExitBootServices(image_handle, map_key)` **fails** (stale key) → retry + `refresh_and_exit` → `capture_memory_map` → **`init_heap` REBUILT the + free list from the raw map, wiping every pre-EBS live allocation's + state** — the shadow's 1.8 MB span and the vt grids became FREE while + still live. +4. Post-EBS allocations carved **inside the live shadow** (the Opt's vt + marker String at exactly `heap+0x443e0`, the family-A crash ptr); the + display renders into the shadow → clobbers those objects → **zeroed + back-pointers** (family A `ptr=0x404443e0 block=0x0`) and **gray text + pixels** (family B `0xaaaaaa00aaaaaa` = 2× `0x00AAAAAA` = PALETTE[7] + fg-7, NOT a firmware scrub). + +**Fix (committed):** `refresh_and_exit` now re-captures the map and returns +a FRESH EBS key WITHOUT rebuilding the free list (`refresh_map_key`, the +free list is the authoritative live-allocation state). Verified in QEMU +with a forced EBS failure: `verify: ok`, `probe free=false`, `flhead +in_shadow=false` — identical to the no-failure boot. Test hook: +`memory::SIMULATE_EBS_FAILURE` (one-shot, defaults off). + +**OptiPlex markers that exposed it:** `probe@+0x443e0 free=YES`, +`w0=203a7476 w1=78302063` (= "vt: c 0x" — the freed marker String), +`blk=0x404443c0 sz=0x770`, `flhead=0x404443c0 in-shadow=YES`, `shadow +=0x40400000 freed=no`, `fb=0xe0000000 in-heap=no`. + ## TL;DR - Real-hardware (OptiPlex, no serial) boots crash in the shell with heap corruption. Two **distinct** bug families: - **(A) scroll-crash family**: in-heap `ptr=0x404443e0` with zeroed header. - - **(B) 0xAA family**: `0xaaaaaa00aaaaaa` = two `0x00AAAAAA` u32s = UEFI - firmware memory-scrub fill. A dropped object's memory was never - allocated/freed by us — it's firmware-fresh scrub. -- Everything converges on **the `vt` static `Vec`s' VecData (ptr/cap/len) - or their heap blocks being corrupted/stale** — the cells VecData ptr at shell - entry was `heap+0x1D8630` (= cells region) in the last crash, and the dropped - 126-byte String (`cap 0x7e`) is the vt verify klog line itself. -- **SOLVED — the pre-verify `0x70020` "mystery" was a debug artifact, not - corruption.** `BLANK = {ch:32, fg:7, bg:0}` = raw `0x70020`, and - `vec![BLANK; rows*cols]` fills the whole grid with it at init. The old - content check compared cells against **0** (wrong expectation — BLANK is not - zero), and its probe loop `break`-ed on the first non-zero cell, so probes - 1–3 were never written and stayed at their zero-initialized slots. The - serial line `c=[0x70020,0x0,0x0,0x0]` was therefore bogus on **every healthy - boot**. A full-grid dump on QEMU proves it: `nz=3700 blank=3700` — both - grids are 100% BLANK at shell entry. The OptiPlex's `0x00AAAAAA` firmware - scrub is still cleanly distinguishable (scrub ≠ 0x70020). + - **(B) 0xAA family**: `0xaaaaaa00aaaaaa` = two `0x00AAAAAA` u32s. +- **CORRECTION (this session): `0x00AAAAAA` is NOT a UEFI firmware scrub. + It is the VGA GRAY text colour: `rgb(170,170,170)` packs to + `(170<<16)|(170<<8)|170 = 0x00AAAAAA`, which is `PALETTE[7]` = fg 7 — the + vt's default foreground.** Every rendered glyph pixel is a `0x00AAAAAA` + word in the shadow; the shadow's background is `0x00000000` (bg 0). + - Family B `0xaaaaaa00aaaaaa` = **two adjacent gray TEXT pixels** — the + memory read as heap held rendered text ⇒ it was **inside the live + shadow** (see root cause above). + - Family A zeros = the shadow's **black background** at the same span. + - ⇒ BOTH families = the allocator handed out memory inside the live + shadow after the EBS-retry free-list rebuild. +- The pre-verify `0x70020` "mystery" was a debug artifact, not corruption. + `BLANK = {ch:32, fg:7, bg:0}` = raw `0x70020`, and `vec![BLANK; rows*cols]` + fills the whole grid with it at init. QEMU proves both grids are 100% + BLANK at shell entry. (Note: 0x70020 ≠ 0x00AAAAAA — the two "fills" are + cleanly distinguishable; the family-A/B dump content is TEXT/bg, not + BLANK cells.) ## The two confirmed bugs 1. **(A) scroll-crash family** — heap allocator returns an in-heap block whose header is zeroed (`ptr=0x404443e0`). Dropping it panics in the guard. -2. **(B) 0xAA family** — freed garbage pointer `0xaaaaaa00aaaaaa`; the memory at - that address is firmware scrub fill. Something handed the allocator a pointer - that was never a live allocation (or the VecData fields themselves got - clobbered with scrub values — the cap was `0xAA` garbage in the boot crash). +2. **(B) 0xAA family** — freed garbage pointer `0xaaaaaa00aaaaaa`; the two + `0x00AAAAAA` words are **gray text pixels** (PALETTE[7]) — the memory at + that address was rendered shadow content. Something handed the allocator a + pointer that was never a live allocation (or a live-shadow block that + leaked into the free list). ## What we've fixed / proven already @@ -51,11 +86,12 @@ verify rework is committed on top (see git log). - **Shadow geometry (QEMU)**: shadow = first alloc at `heap+0x20` (`0x1980020`), 480000 words → span `[heap+0x20, heap+0x1D4C20)`; cells data at `heap+0x1D4C40`; VecDatas at ~`heap+0x1D8630`. -- **Shadow-rejection hypothesis (unproven)**: on the OptiPlex the shadow alloc - may have been rejected (or the cells alloc landed inside the would-be shadow - span), leaving that block free + scrub-filled. `0x40444970 = heap+0x44970` is - *inside* the would-be shadow span. The last crash's `r9=0x405d8630 = - heap+0x1D8630` is the cells VecData region. +- **Shadow-rejection hypothesis (strengthened by QEMU forced-reject test)**: + on the OptiPlex the shadow alloc may have been rejected (or its block + freed), leaving that span free + text/zero-filled. `0x40444970 = + heap+0x44970` is *inside* the would-be shadow span. Forcing rejection in + QEMU reproduces family A exactly: `probe heap+0x443e0 IN free blk` with + all-zero content. - **Original #GP resolved**: crashed image was RELEASE; drop of `Parsed` struct; cause = real memory corruption (laytest excluded compiler bug). - **Stale PDB trap**: fresh PDB re-resolved shell-entry #GP → @@ -108,6 +144,17 @@ verify rework is committed on top (see git log). ## QEMU reproduction of the content-check failure (RESOLVED) +- **0x00AAAAAA scan verdict (this session)**: `shadow scan: 11333w/480000` + hits are all at **glyph positions of on-screen text** (fg=7 gray pixels). + The "scrub" reading at the crash offset (`probe mem: 00000000 00aaaaaa + 00aaaaaa 00000000 …`) is the heapdbg output line **rendered at that exact + shadow location** — the probe reads its own text. The shadow far rows + (r300, r500) read 0 ⇒ init zeroing is correct and nothing over-writes + untouched shadow rows in QEMU. FB pixel at (171,1) = (170,170,170) GRAY + = 0x00AAAAAA — shadow and FB agree; there is NO shadow/FB desync. +- The shadow scan is still a useful health signal: untouched far rows must + stay 0; text-row hits are expected (that's what the shadow holds). + - Serial showed `probes c=[0x70020,0x0,0x0,0x0] r=[0x70020,0x0,0x0,0x0]` — this was **misread** as "a single stray cell". The full-grid dump (3700 reads) showed `nz=3700 blank=3700` in both grids: **every cell is BLANK (0x70020), @@ -153,26 +200,55 @@ FB), memory self-test, shell banner (runs after verify), backspace (BLANK). ## OptiPlex flash — what to look for (the linchpin) +**FIRST OPTIPLEX REPORT (this session, previous image):** + +``` +heapdbg: probe@+0x443e0 free=YES w0=203a7476 w1=78302063 +heapdbg: shadow=0x40400000 freed=no +``` + +- `w0=0x203a7476 w1=0x78302063` = LE ASCII **"vt: c 0x"** — the leftover + bytes of the *vt verify marker String* (`debug_verify`'s + `alloc::format!("vt: c {:#x} …")`), which was **allocated at exactly + `heap+0x443e0`** (the historical family-A crash ptr) and freed before + the marker ran. ⇒ on the Opt the heap's first-fit hands out + **`heap+0x443e0` — INSIDE the live shadow span + [`heap+0x20`, `heap+0x1D4C20`)** — while `shadow=0x40400000 freed=no` + (the shadow Vec's block header claims allocated, forget'd). +- QEMU control: same marker prints `probe free=false … flhead=0x1b706d0 + in_shadow=false` — the probe address is NOT in the free list. +- ⇒ The free list contains the live shadow span on the Opt. The two new + boot lines (`fb … in-heap=YES/no`, `flhead … in-shadow=YES/no`) plus + the probe's containing `blk=…` tell us whether the overlap is the whole + heap block (shadow carve never happened / init_heap after alloc), the + fb inside the heap (display clobbers), or a mid-span block. + New image puts on screen (markers are **bottom-anchored** now — they survive the banner rows 0–1, the prompt row 2, AND a stale crash.log replay at rows 2+; use `gop::bottom_row(off)` to place any future ones): -1. **Last row (bottom_row(0))**: `gop: shadow {ptr:#x} {MiB}` vs - `gop: NO-SHADOW (fallback)`. - - If NO-SHADOW ⇒ shadow-rejection hypothesis gains weight (first block left - free + scrub-filled). -2. **Second-to-last row (bottom_row(1))**: `vt: c {ptr:#x} cap={cap:#x} ok|GARBAGE | r {ptr:#x} ok|GARBAGE`. - - **The cells ptr value is the linchpin**: if it equals `0x40444970`, the - cells VecData itself is stale at shell entry (matches boot-crash decode). - - If cap shows `0xAA…` garbage ⇒ VecData fields scrubbed (bug family B). - - `ok` = all probed cells read BLANK (0x70020); `GARBAGE` = probe ≠ BLANK - (the OptiPlex firmware scrub reads back `0x00AAAAAA`). -3. If it crashes: the full guard band (`hdr:` + all four `mem:` lines, 16 stack - words) plus the bottom markers if still visible. - -Serial-side (QEMU only): the verify lines are captured in the klog ring -(`serial_println!` → ring), so a healthy OptiPlex boot can also dump them with -the `klog` shell command. +1. **Last row (bottom_row(0))**: `heapdbg: probe@+0x443e0 free=YES|no w0=… w1=… blk={base} sz={size}` + — crash offset free? content (zeros=shadow bg, `00aaaaaa`=gray text, + ASCII=leftover string bytes)? and the **base/size of the free block + containing it** (0x40400000 = whole-heap block → the shadow carve + never happened; inside the shadow span → overlap; a normal small + block → ordinary freed allocation). +2. **Second-to-last row (bottom_row(1))**: `heapdbg: shadow={ptr| -} freed=YES|no` + — the shadow block base + header free flag. +3. **Third-to-last (bottom_row(2))**: `heapdbg: fb={base} size={size} in-heap=YES|no` + — is the framebuffer inside the heap range (display clobbers heap)? +4. **Fourth-to-last (bottom_row(3))**: `heapdbg: flhead={base} size={size} in-shadow=YES|no` + — does the free-list head overlap the shadow span? +5. Also: `vt: c {ptr} cap=… ok|GARBAGE | r {ptr} ok` marker (was + bottom_row(1), now overwritten by the shadow line) — the cells ptr + value is the linchpin for a stale-VecData theory: equals `0x40444970`? +6. If it crashes: the full guard band (`hdr:` + all four `mem:` lines, 16 + stack words) plus the bottom markers if still visible. + +Serial-side (QEMU only): the boot heapdbg values are mirrored to serial +(`heapdbg: probe free=… shadow=… freed=… fb_in_heap=… flhead=… in_shadow=…`) +and captured in the klog ring — a healthy OptiPlex boot can also dump them +with the `klog` shell command. ## Build/flash/test loop (traps) @@ -194,11 +270,14 @@ the `klog` shell command. ## Next steps (priority order) -1. **Flash to OptiPlex** and relay the two bottom markers: shadow state (last - row) + vt marker (`bottom_row(1)` — the cells ptr value is the linchpin: - equals `0x40444970`? cap scrubbed `0xAA…`? `ok` vs `GARBAGE`?). If crash: - full `hdr:` + all four `mem:` lines. -2. Reattach the window-manager milestone once the crash family is resolved. +1. **Flash the fixed image to the OptiPlex** (BOOTX64.EFI in `esp/`). + Expected bottom rows now: `probe@+0x443e0 free=no` (was YES), + `flhead=… in-shadow=no` (was YES), `vt: … ok`. Run the old repro + (`klog -t 2`, `ls`, `help`) — should no longer crash. +2. If the Opt still misbehaves: relay the four bottom lines again — the + fix only holds if the free list no longer overlaps the shadow. +3. Reattach the window-manager milestone once confirmed. -(QEMU control arm is green: content check now passes on every healthy boot; -no further QEMU-side investigation is needed.) +(QEMU control arm is green with and without a forced EBS failure: +`verify: ok`, probe NOT free, shadow base NOT in free list, full shell +sequence clean. `0x00AAAAAA` = gray text colour; shadow far rows stay 0.) -- cgit v1.2.3