1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
|
# rsdos Debug Notes — heap/cells corruption family (v0.5.7)
Handoff state: **HEAD = `ab633ce`**; the mystery below is SOLVED and the
verify rework is committed on top (see git log).
## TL;DR
- Real-hardware (OptiPlex, no serial) boots crash in the shell with heap
corruption. Two **distinct** bug families:
- **(A) scroll-crash family**: in-heap `ptr=0x404443e0` with zeroed header.
- **(B) 0xAA family**: `0xaaaaaa00aaaaaa` = two `0x00AAAAAA` u32s = UEFI
firmware memory-scrub fill. A dropped object's memory was never
allocated/freed by us — it's firmware-fresh scrub.
- Everything converges on **the `vt` static `Vec<Cell>`s' VecData (ptr/cap/len)
or their heap blocks being corrupted/stale** — the cells VecData ptr at shell
entry was `heap+0x1D8630` (= cells region) in the last crash, and the dropped
126-byte String (`cap 0x7e`) is the vt verify klog line itself.
- **SOLVED — the pre-verify `0x70020` "mystery" was a debug artifact, not
corruption.** `BLANK = {ch:32, fg:7, bg:0}` = raw `0x70020`, and
`vec![BLANK; rows*cols]` fills the whole grid with it at init. The old
content check compared cells against **0** (wrong expectation — BLANK is not
zero), and its probe loop `break`-ed on the first non-zero cell, so probes
1–3 were never written and stayed at their zero-initialized slots. The
serial line `c=[0x70020,0x0,0x0,0x0]` was therefore bogus on **every healthy
boot**. A full-grid dump on QEMU proves it: `nz=3700 blank=3700` — both
grids are 100% BLANK at shell entry. The OptiPlex's `0x00AAAAAA` firmware
scrub is still cleanly distinguishable (scrub ≠ 0x70020).
## The two confirmed bugs
1. **(A) scroll-crash family** — heap allocator returns an in-heap block whose
header is zeroed (`ptr=0x404443e0`). Dropping it panics in the guard.
2. **(B) 0xAA family** — freed garbage pointer `0xaaaaaa00aaaaaa`; the memory at
that address is firmware scrub fill. Something handed the allocator a pointer
that was never a live allocation (or the VecData fields themselves got
clobbered with scrub values — the cap was `0xAA` garbage in the boot crash).
## What we've fixed / proven already
- **IST=1 stack collision → ROOT CAUSE of ACTIVE_VOL corruption**: vec 14 ran on
`PF_STACK`, a 4 KiB `.bss` static overlapping FAT `ACTIVE_VOL`. Fixed in
`gdt.rs`: heap-allocated 32 KiB IST stacks. (Done, earlier.)
- **AHCI/DMA EXONERATED**: user removed all disks; crashes persisted.
- **All QEMU repro attempts were clean** (mem/klog/tail-crash) until the content
probe upgrade — which tripped the *content check only* on a healthy boot for
the reason above (expectation bug + break bug). Fixed: the check now expects
`blank_raw` (0x70020), records all four probe values, and adds a full-grid
nz/blank summary line. QEMU now reports `verify: ok` on every boot.
- **Clean heap geometry (OptiPlex)**: heap = `0x40400000..0xc7e34000` (~2.2 GB),
boot stack ~1 MiB above heap end.
- **Shadow geometry (QEMU)**: shadow = first alloc at `heap+0x20` (`0x1980020`),
480000 words → span `[heap+0x20, heap+0x1D4C20)`; cells data at
`heap+0x1D4C40`; VecDatas at ~`heap+0x1D8630`.
- **Shadow-rejection hypothesis (unproven)**: on the OptiPlex the shadow alloc
may have been rejected (or the cells alloc landed inside the would-be shadow
span), leaving that block free + scrub-filled. `0x40444970 = heap+0x44970` is
*inside* the would-be shadow span. The last crash's `r9=0x405d8630 =
heap+0x1D8630` is the cells VecData region.
- **Original #GP resolved**: crashed image was RELEASE; drop of `Parsed` struct;
cause = real memory corruption (laytest excluded compiler bug).
- **Stale PDB trap**: fresh PDB re-resolved shell-entry #GP →
`rsnix::vt::put_byte+0x16a` (`mov %ax,(%rdx,%rcx,4)`) — u16 char at 4-byte
stride into cells buffer via garbage ptr from `r9=0x405d8630`. bt chain:
`shell::run → GopWriter::write_str → fmt::Formatter::pad_integral →
klog::LineBuf::write_str → core::fmt::write → Display(u32)`.
- **Crash decoded**: the dropped object in the last boot crash was a 126-byte
String (`cap 0x7e`) = the vt verify klog line (19-char prefix + 106-char text +
`\n`); `ptr=0x40444970` is STALE (inside the would-be shadow span, back-pointer
never written).
## Allocator layout (verified, `memory.rs`)
- `HEADER_SIZE=16`, `MINI_SIZE=8`, `MIN_BLOCK=32`, `MIN_ALIGN=16`
- payload = `align_up(block+24, align)`
- split: rewrites rem header + `free_list_insert` + back-pointer at `payload-8`
- `back_pointer_ok(ptr)` validates `[ptr-8]`; guard v0.5.7 (`0187b80`) reports +
halts on failure; `corrupt` QEMU hook verified.
## vt grid facts
- `put_cell(r,c,ch)` sets `ch` + fg/bg from statics `SGR_FG=7`/`SGR_BG=0`.
- Cell raw encoding: `ch | fg<<16 | bg<<24`; BLANK = 0; cursor-inverse blank =
`0x07000000` (bg and fg swapped). **`0x70020` is NOT a cursor-inverse cell** —
it's a literal space with fg=7 (ch=0x20, fg=7, bg=0).
- `flush()` diffs cells vs rendered; cursor pass writes inverse colors into both
FB and `rendered`.
- `term_println(s)` = `gop_print!` + blank line + explicit `serial::write_line`.
Display mirror to serial is NOT automatic for `gop_println!`.
- Boot order after vt init: shadow → boot markers → memory self-test → page
allocator + self-tests → paging adopt + self-test → sched init →
rtc/pci/ahci/fat/fs → **cursor spawn** → **`debug_verify()`** → `shell::run()`
(first banner `rs-dos v0.5.7 …`, then `=====`, then crash-log read).
## Current committed work (post-`ab633ce`)
- `gop::boot_marker_at(msg, row)` + `gop::text_rows()` + `gop::bottom_row(off)`
— shadow marker now on the LAST text row, vt verify marker one above it
(both survive the banner, prompt, and crash.log replay).
- `debug_verify()` fixed: compares probes against `blank_raw` (0x70020), not
zero; records all four probe values (no early break); probes BOTH cells and
rendered; adds a full-grid `grids c nz=… blank=… r nz=… blank=…` summary
(serial). Marker format:
`vt: c {ptr:#x} cap={cap:#x} ok|GARBAGE | r {ptr:#x} ok|GARBAGE` at
`bottom_row(1)`.
- QEMU verified: `verify: ok` every boot; screendump shows the banner (gray),
prompt, and both bottom markers. test-msg + test-fault PASS; debug and
release builds 0 warnings.
## QEMU reproduction of the content-check failure (RESOLVED)
- Serial showed `probes c=[0x70020,0x0,0x0,0x0] r=[0x70020,0x0,0x0,0x0]` — this
was **misread** as "a single stray cell". The full-grid dump (3700 reads)
showed `nz=3700 blank=3700` in both grids: **every cell is BLANK (0x70020),
the grids are healthy**. The probe array's 1–3 slots were zero-initialized
leftovers from the early-`break` bug.
- Memory dump proves grids are fully healthy on QEMU: cells block base
`0x1b54c20` `{size=0x39f0, free=0}`, back-ptr written; rendered block base
`0x1b58650`; both grids contain the shell banner post-draw. **No heap
corruption on QEMU.**
- QEMU verify geometry: cells ptr `0x1b54c40` cap `0xe74` len `0xe74`; rendered
ptr `0x1b58670`.
- ⇒ `CONTENT-GARBAGE` was never evidence of OptiPlex corruption — the check was
comparing against the wrong expectation. Now: probes compare against
`blank_raw` (0x70020); every probe value is logged; a `grids c nz=… blank=…`
summary is the definitive health signal.
## The pre-verify `0x70020` at grid(0,0) — MYSTERY SOLVED
**There was no writer.** `0x70020` is the grid's own init fill:
`BLANK = Cell{ch:32, fg:7, bg:0}` encodes as `0x70020`, and `vt::init` builds
both grids with `vec![BLANK; rows*cols]`. The old content check expected
all-zero cells (wrong), and the probe loop's early `break` made probes 1–3
look like zeros. Nothing writes a pre-banner space; the shell banner (gray
VGA-7 text, rows 0–1) and the `>` prompt (row 2) are the only vt writes
before the markers.
Also learned while verifying with a screendump:
- The shell banner occupies grid rows **0 and 1** (motd + `=====` line), and
the prompt sits at row 2 — so a marker at FB row 1 (the old shadow marker)
is **clobbered** by the banner, and row 2's first glyphs get eaten by the
prompt + cursor. Markers whose values matter now live at the **bottom**
(`gop::bottom_row(0)` = last row, `bottom_row(1)` = second-to-last), which
also survive a stale crash.log replay (~20 lines at grid rows 2+).
- Banner text renders VGA gray `0xAAAAAA` (fg 7), boot markers white `0xFFFFFF`
— a white-only pixel matcher misses the banner. QEMU screendump decode of
the bottom rows: `vt: c 0x1b54c40 cap=0xe74 ok | r 0x1b58670 ok` and
`gop: shadow 0x1980020 1.83 MiB`.
**Excluded candidates (all still true, now moot):** cursor task (FB/shadow
only), flush cursor-inverse pass (0x07000000 ≠ 0x70020), boot markers (direct
FB), memory self-test, shell banner (runs after verify), backspace (BLANK).
## OptiPlex flash — what to look for (the linchpin)
New image puts on screen (markers are **bottom-anchored** now — they survive
the banner rows 0–1, the prompt row 2, AND a stale crash.log replay at rows
2+; use `gop::bottom_row(off)` to place any future ones):
1. **Last row (bottom_row(0))**: `gop: shadow {ptr:#x} {MiB}` vs
`gop: NO-SHADOW (fallback)`.
- If NO-SHADOW ⇒ shadow-rejection hypothesis gains weight (first block left
free + scrub-filled).
2. **Second-to-last row (bottom_row(1))**: `vt: c {ptr:#x} cap={cap:#x} ok|GARBAGE | r {ptr:#x} ok|GARBAGE`.
- **The cells ptr value is the linchpin**: if it equals `0x40444970`, the
cells VecData itself is stale at shell entry (matches boot-crash decode).
- If cap shows `0xAA…` garbage ⇒ VecData fields scrubbed (bug family B).
- `ok` = all probed cells read BLANK (0x70020); `GARBAGE` = probe ≠ BLANK
(the OptiPlex firmware scrub reads back `0x00AAAAAA`).
3. If it crashes: the full guard band (`hdr:` + all four `mem:` lines, 16 stack
words) plus the bottom markers if still visible.
Serial-side (QEMU only): the verify lines are captured in the klog ring
(`serial_println!` → ring), so a healthy OptiPlex boot can also dump them with
the `klog` shell command.
## Build/flash/test loop (traps)
- Build 0 warnings, edition 2024, `no_std`, `x86_64-unknown-uefi`, workspace
v0.5.7. Zero new deps.
- Real-hardware boot chain: rsboot at `EFI/BOOT/BOOTX64.EFI` → kernel at
`bzImage`. QEMU dev loop boots kernel as BOOTX64.EFI.
- CRASHLOG stick must be FAT32; `mdir` grep gotcha: match `crash`
case-insensitively.
- **Stale qemu holds flock** → `pkill -9 -f 'qemu[-]system'` in a SEPARATE
command; `rm -f esp/NvVars` between boots.
- Use repo `scripts/` infra: test-msg/test-fault/test-mouse; monitor socket; HMP
`mouse_move`.
- Fault-context serial rule: never `serial_println!` in ISR/fault code; use
`serial::write_raw`; guard report must not allocate (stack `FmtBuf`).
- Marker on row 0 gets overwritten by the shell MOTD — use `boot_marker_at(msg, row>0)`.
- Stale PDBs misname symbols — re-resolve with a fresh PDB per build (PDB math
is decimal); `r9` = struct base with Vec layout `{ptr@0, cap@8, len@16}`.
## Next steps (priority order)
1. **Flash to OptiPlex** and relay the two bottom markers: shadow state (last
row) + vt marker (`bottom_row(1)` — the cells ptr value is the linchpin:
equals `0x40444970`? cap scrubbed `0xAA…`? `ok` vs `GARBAGE`?). If crash:
full `hdr:` + all four `mem:` lines.
2. Reattach the window-manager milestone once the crash family is resolved.
(QEMU control arm is green: content check now passes on every healthy boot;
no further QEMU-side investigation is needed.)
|
