path: root/CONTEXT.md

# CONTEXT.md — raisernics Kernel Glossary

## raisernics

Custom x86_64 kernel for the RNG ecosystem.  Toy project — self-contained,
no strict userspace separation, TempleOS-inspired design philosophy.
Built in Rust, compiled for the `x86_64-unknown-uefi` target (stable).

## Boot interface

The kernel is a **PE32+ EFI application** loaded via UEFI
`LoadImage`/`StartImage`.
The existing bootloader (rsbl) loads us from `\bzImage` and passes the UEFI
system table.  No bootloader changes needed.

## Output strategy

Two-phase console model:

- **Phase 1 (boot services alive):** UEFI `con_out->OutputString()` — works
  with whatever UEFI firmware routes (serial + framebuffer).
- **Phase 2 (native):** Raw serial UART 16550 driver on COM1 (I/O port
  `0x3F8`).  Activated after `ExitBootServices` when we own the hardware.

## Dependency philosophy

Aligns with RNG philosophy: minimal dependencies.  Freestanding Rust,
no libc, no external crates at the kernel/core level.  UEFI protocol
structs are hand-defined (not from `uefi-rs` crate) to stay lightweight.

## Paging strategy

Long mode mandates paging (`CR0.PG = 1`), so we accept that tax but keep it
simple.  Identity-mapped 4-level tables, no higher-half, no demand paging,
no guard pages except task stacks — each task's stack gets one unmapped
guard page below it, and a page-fault handler turns an overflow into a
task panic.  `paging.rs` provides identity map/unmap/query helpers.  At
boot the kernel runs on UEFI's tables; `adopt_tables()` rebuilds that map
into the kernel's own tables and switches CR3 — needed because UEFI maps its
page-table pages read-only, so the firmware tree can't be modified in place.
Firmware large pages (2 MB / 1 GB) are split into 4 KB tables on demand;
page-table pages come from the physical page allocator.

## Physical memory

Before `ExitBootServices`, the kernel reads the UEFI memory map to know
which physical regions are free.  The heap is a first-fit **free-list
allocator** with splitting and coalescing: blocks carry a 16-byte header
(size + free flag), free blocks are linked in address order, and an 8-byte
back-pointer before each payload lets `free` find its block even when
alignment left a gap.  When the free list is exhausted, the heap **grows**
by taking pages from the page allocator (proportional to the request),
keeping the two allocators cooperating.  A **bitmap page allocator** manages
physical pages: one bit per 4 KB page spanning the conventional regions, so
MMIO holes and firmware-reserved gaps are never handed out, and the heap
region itself is reserved to keep the allocators from double-allocating the
same RAM.  Contiguous runs with power-of-two alignment are supported
(`alloc_pages_aligned`).

## Input strategy

**PS/2 keyboard, interrupt-driven.**  IRQ1 pushes scancodes into a ring
buffer (`interrupts::KEYBUF`); the shell polls it via `read_scancode()`.
Port I/O to `0x60`/`0x64`.  Scancode set 1 (US QWERTY layout in the
initial build).

## Display strategy

Two-phase model matching output:

- **Phase 1:** UEFI `con_out->OutputString()` (framebuffer + serial handled
  by firmware).
- **Phase 2:** GOP (Graphics Output Protocol) framebuffer obtained before
  `ExitBootServices`, driven by a built-in 8×16 bitmap font.  Text rendered
  by drawing glyphs directly into the framebuffer.

The framebuffer address stays valid after `ExitBootServices` — the GPU's
linear frame buffer is system memory that we own once boot services are gone.

## Keyboard layout

Scancode set 1, US QWERTY layout.  Hardcoded scancode→ASCII table.
Other layouts can be added later by swapping the translation table.

## Task

A unit of concurrent execution in the kernel: its own kernel stack
(with a guard page below it), scheduler bookkeeping, and no
address-space separation — all tasks share the one identity-mapped
space and the kernel's facilities (TempleOS-style, not a process).
Tasks are preemptively round-robin-scheduled on the 100 Hz PIT tick;
the shell is task 0 and an idle `hlt` task is the always-ready
fallback.  A task may sleep, yield, and log to serial/klog; only the
shell touches the disk, filesystem, and terminal.  A panicking or
stack-overflowing task dies gracefully (logged, marked done, slot
reaped) without taking the kernel down.

## Development loop

Standalone project — no integration with the RNG initramfs build system.
Testing via QEMU with OVMF.  The kernel PE binary is placed on a FAT
image as `\bzImage` (matching rsbl's expected path).

## Project layout

Workspace of four crates, all built for the `x86_64-unknown-uefi` target:

- `rsnix` — the kernel library (module breakdown below)
- `shell` — command loop and built-ins
- `rsdos` — EFI entry point (`efi_main`) and boot orchestration
- `rsdosedit` — kernel-native terminal editor

`rsnix` module breakdown:

- `main.rs` — entry point, panic handler, top-level init
- `uefi.rs` — UEFI protocol type definitions
- `serial.rs` — UART 16550 driver
- `paging.rs` — identity page table helpers
- `memory.rs` — UEFI memory map, free-list heap allocator (grows from page
  allocator), bitmap page allocator
- `paging.rs` — identity page-table helpers (map/unmap/query, large-page split)
- `power.rs` — `ResetSystem`-based shutdown, hlt loop fallback
- `rtc.rs` — CMOS/RTC clock (ports 0x70/0x71), BCD/12h normalization
- `pci.rs` — legacy PCI config space (0xCF8/0xCFC), full
  enumeration; `config_write`
  - 64-bit BAR reader for the AHCI driver
- `ahci.rs` — SATA AHCI host driver (polling, DMA, no interrupts): scans PCI for
  class 1/subclass 6/prog-if 1, maps ABAR, starts port 0 (FRE/ST/DMAE), runs
  IDENTIFY + READ/WRITE DMA EXT through a 4 KiB DMA buffer
- `fat.rs` — FAT32: BPB parse, whole-FAT cache in RAM with
  write-back to both FAT
  copies, cluster chains, directory scan with checksum-validated LFN
  reassembly, LFN + 8.3 writing, FSInfo free-count maintenance,
  mkdir/rm/read/write
- `fs.rs` — VFS: dirs, ramfs files, char devices (`/dev/mem`, `rtc`, `pci`,
  `version`, `meminfo`, `null`, `zero`, `console`, `serial`; `/tmp` ramfs;
  `/etc/motd`); `/disk` mount node + 24-slot ephemeral node pool so shell
  commands work on disk paths unchanged
- `gop.rs` — framebuffer rasteriser (glyph blits, pixel shifts); the text
  surface itself lives in `vt`
- `vt.rs` — terminal emulator: cell grid, cursor, ANSI CSI parser (H/J/K,
  cursor moves, SGR 16-colour, `?25l/h`), dirty-cell blitting, UTF-8
  decoding (box-drawing `│`/`═`, arrows `←↑→↓`, em dash `—` render
  procedurally instead of `?`).  The inverse cursor is mirrored into the
  rendered grid so scrolling / `clear` can never strand stale inverse
  cells on the framebuffer.
- `tty.rs` — line discipline: canonical `read_line` (echo, line buffer,
  ^C/^D) and raw `read_byte` (terminal byte protocol: arrows → `ESC [ A–D`,
  ^X → `0x18`, …)
- `strbuf.rs` — stack fixed-capacity ASCII string (moved from shell)
- `font.rs` — 8×16 bitmap font data
- `ps2.rs` — PS/2 keyboard driver (polling); generic `Ctrl(char)` and
  `CtrlArrow` events
- `shell.rs` — command loop and built-in commands
- `rsdosedit/` — kernel-native port of the RNG `rsedit` terminal editor
  (no_std; talks to `tty` raw mode + `vt` ANSI; saves via the VFS)

## Milestone status (v0.1)

✅ **Booting** — kernel loads via rsbl, prints banner via UEFI
  console (verified on real hardware)
✅ **ExitBootServices** — memory map captured via static `.bss` buffer
  (see `docs/adr/0001-static-buffer-for-memory-map.md`)
✅ **Serial** — UART 16550 at COM1, 115200 8N1
✅ **Framebuffer** — GOP located, 8×16 bitmap font renders text
✅ **Keyboard** — PS/2 polling, scancode set 1, US QWERTY
✅ **Shell** — commands: `help`, `echo`, `clear`, `hello`, `info`,
  `mem`, `halt`, `poweroff`
✅ **Heap allocator** — first-fit free-list with splitting +
  coalescing, `alloc`/`free`
  (boot-time self-test verifies alignment, reuse, and full coalescing)
✅ **Physical pages** — bitmap allocator over conventional regions, contiguous
  runs, heap reservation, page-0 trap guard (boot-time self-test included)
✅ **Paging helpers** — identity map/unmap/query, 2 MB/1 GB large-page split,
  own tables adopted at boot (boot-time self-test included)
✅ **Heap growth** — heap expands into the page allocator when exhausted
✅ **Shell** — `mmap` dumps the UEFI map, `stress` exercises alloc/free + growth
✅ **Poweroff** — `poweroff`/`exit` shut down via UEFI `ResetSystem`
  (ACPI S5; `halt` keeps the plain hlt loop)
✅ **RTC** — CMOS clock read at boot and via `date` (BCD/12h handled)
✅ **PCI** — port-based config access, full bus enumeration via `pci`
✅ **VFS** — “everything is a file”: dirs, ramfs, char devices
  (`ls`/`cat`/`hexdump`/`mkdir`/`touch`/`rm`, `echo >`/`>>`; devices
  regenerate on read, `/dev/mem` is raw physical RAM)
✅ **^C / ^D** — Ctrl+C cancels the line or aborts a running stream
  (`cat`/`hexdump` check between chunks); Ctrl+D is EOF (no poweroff —
  removed this session; see session notes below)
✅ **tty + vt (terminal) — the 1:1 UNIX console split**: `tty` owns the
  line discipline (canonical echo/buffer/^C/^D for the shell; raw mode
  translates keys to the classic terminal byte protocol), `vt` owns the
  screen (cell grid + ANSI CSI parser + 16-colour SGR + dirty-cell blits
  into the GOP framebuffer).  `gop_println!` etc. now render through `vt`.
✅ **rsdosedit** — the RNG `rsedit` editor ported to the kernel (no_std,
  no libc): `edit <path>` opens a full-screen editor with undo, syntax
  highlighting, search (^W), save (^S), exit (^X, twice to discard), word
  nav (Ctrl+arrows), smart indent.  Files live in the ramfs.  Termios →
  `tty` raw mode, stdout → `vt`, `std::fs` → VFS, SIGWINCH → fixed
  geometry.  Verified in QEMU: edit/type/navigate/insert/save/exit, exact
  bytes via `cat`/`hexdump`, syntax colours via framebuffer dump.
✅ **AHCI + FAT32 (persistent disk)** — a SATA AHCI driver (polling DMA,
  no interrupts) plus a FAT32 filesystem, mounted under `/disk/<UID>/`;
  every shell command works on disk paths: `ls`/`cat`/`hexdump`/
  `echo >`/`>>`/`mkdir`/`touch`/`rm`/`edit`.  The driver scans **all**
  implemented ports (CAP.NP/PI) and starts every port with a device;
  `fat::init` mounts **every** valid FAT32 volume it finds, each under
  `/disk/<UID>/` where UID = `<LABEL>-<8-hex-serial>` (e.g.
  `RSDOS-5BD1E55F`) — sanitized uppercase label + volume serial, unique
  per format, stable across reboots, human-recognizable and
  machine-parseable (`^[A-Z0-9_]+-[0-9A-F]{8}$`); cloned volumes get a
  `-N` suffix.  So the kernel finds the disks no matter which AHCI port
  the firmware/QEMU put them on (real hardware included), and every
  disk is visible at once (verified: `RSDOS-…` + `DATA2-…` mounted
  side by side, writes to both, fsck-clean).  Whole-FAT cache per
  volume with write-back to both FAT copies, checksum-validated LFN
  read/write (multi-entry names >13 chars stored last-first on disk —
  both the writer and reader were reversed and cancelled out; fixed),
  multi-cluster files, space-padded `.`/`..` dot entries (`..` →
  cluster 0 under the root), FSInfo free-count maintenance
  (fsck-clean after a session, verified with `fsck.fat -n`).
  **Files survive reboot** — verified by poweroff → `mdir`/`mtype` on
  the image and by a fresh boot seeing the prior session's files.

✅ **Shell grammar (ADR 0002)** — one canonical grammar for every
  command: `cmd args [USE src] [TIL marker] [filters…] [output kw dst]`.
  Keyword redirection (`WRITING`/`WRITE` overwrite, `TO` append,
  `USE`/`USING` input source, `TIL`/`UNTIL` heredoc/terminator;
  uppercase-only, aliases valid, fully reserved), C-style quoting
  (`"…"` groups, `\x` escapes — quoted text is never an operator),
  post-filter stages applied left to right (`-h N` head kills the
  upstream, `-t N` tail holds back and flushes at EOF/^C, `-g PAT`
  smart-case literal substring, `-p` screenful pager; all repeatable),
  `>`/`<`/`<<`/`>>` removed (`>` is plain text), file sinks replace the
  screen, `cat`/`hexdump` consume input when path-less (stdin heredoc,
  `cat USE a TO b` copy, `USE f TIL m` truncation).  ^D is EOF
  everywhere and never powers off.  `help` rewritten; `test-disk.sh`
  and `sendkeys.py` updated to the keyword syntax.

  Verified in QEMU (26 grammar checks on a fresh *and* a reused disk,
  disk round trip, pager pacing): head kills at N lines, ^C flushes a
  `-t` window, smart-case grep, chained/repeated filters, `WRITING`+
  `TO`, `USE..TO` copy, `USE..TIL`, heredoc → file, quoting, `-p`.
  Two bugs found and fixed during verification: (1) `WRITING` over an
  existing file wrote after stale NUL padding — `fs::size()` returns
  the node-cached size, which is stale after `set_len(0)`; the sink
  now writes at literal offset 0 after truncation; (2) `USE f TIL m`
  leaked everything after the marker — the final partial-line emit
  ran even after the marker break; fixed with a `stopped` flag.

  QEMU setup: `-drive id=disk,if=none,format=raw,file=disk.img` +
  `-device ide-hd,drive=disk,bus=ide.0` puts the data disk on AHCI port
  0; the ESP gets `bus=ide.1`.  (The implicit `-drive file=fat:rw:esp`
  otherwise grabs port 0.)  Make targets (`run`, `run-gui`, `debug`)
  attach `$(DISK_IMG)` (default `/tmp/disk.img`) and auto-create it when
  missing.  OVMF writes its `NvVars` varstore file onto the first FAT
  volume — harmless.

  The whole disk workflow is scripted (no manual mkfs/mcopy):

- `make disk` / `./scripts/mkdisk.sh` — create the FAT32 image with
    `mkfs.fat -C` (the flag that actually creates the file; plain
    `mkfs.fat` needs an existing file), label `RSDOS`, seed from
    `seed/` via mtools; `--fresh` re-formats, `--size 128M` sizes it
- `make inspect` — `mdir -i` listing of the image (mtools)
- `make test-disk` / `./scripts/test-disk.sh` — full automated round
    trip: build → deploy → ensure image → boot QEMU headless → drive
    the shell via `sendkey` through the monitor (`ls /disk`, `mkdir`,
    `echo WRITING`, `cat`, `poweroff`) → fsck-clean check → `mtype` verifies
    the written file survived poweroff
- `./scripts/sendkeys.py` — types lines into the guest via the QEMU
    monitor (`--sock`, US layout, named keys for punctuation; bare
    `-`/`.`/`/` are not valid sendkey names)

✅ **`klog` kernel log ring** — a RAM ring of the last 128 serial log
  lines (128 chars each, ~16 KB), fed by `serial_println!` (one choke
  point — every boot/low-level message captured for free).  Sequence
  numbers per line + wrap/drop accounting; a dump header reports how
  many lines were lost to wrap.  Each line carries a `[ssss.mmm]`
  timestamp — monotonic ms since boot from `interrupts::uptime_ms()`:
  the 100 Hz PIT tick (10 ms) plus interpolation from the PIT's live
  1.19 MHz down-counter read twice around the tick load (a jump = a
  tick fired between reads) for true sub-ms resolution; `TICKS` widened
  to u64 so uptime never wraps.  **The CMOS RTC cannot provide
  milliseconds** — it is a 1 Hz BCD device (seconds only) — so the
  millis come from the PIT.  Early-boot lines are `[0000.000]`
  (PIT not yet configured; `PIT_READY` gates interpolation), and
  timestamps are clamped monotonic in the ring.

  Exposed as the `/dev/klog` char
  device; the shell `klog` command is a thin alias for `cat
  /dev/klog` (same filters/redirection), so `klog -t 5`, `klog -g
  fat`, and `klog WRITING /disk/…/boot.log` (persists the boot log)
  all work.  The ring never allocates (const-constructible static);
  only the dump allocates, at shell time.  `klog` rejects positional
  args (usage) and input keywords ("does not accept input").

  **Design consequence: shell output must not feed the ring.**  The
  plan's "hook `serial_println!`" only works if that macro is
  kernel-logging-only — but the shell printed prompts/errors/results
  through it, which would have flooded the boot log with echo/cat
  output (a `cat /dev/mem` would wrap the ring).  The shell now
  writes user-facing output (banner, prompt, `term_print`/
  `term_println`, pipeline sink, halt/poweroff) directly to the
  serial port via `serial::write_str`; tty echo already did.  Only
  kernel messages (rsnix modules + `rsdos/main.rs`) go through
  `serial_println!` now.  Verified in QEMU: 28 boot lines captured
  (seq 0..27, first `=== rsnix boot ===`, last `boot: entering
  shell`), `echo hello` output absent from the ring, `klog WRITING`
  persists a boot log (mtype-verified), wrap behavior verified with a
  temporary 140-line boot loop (128 kept / 40 dropped, seq 40..167,
  oldest live line correct — loop reverted after the test).

✅ **Runtime device detection (hotplug rescan)** — the boot discovery
  pipeline is now re-runnable and diffed against live hardware:

- `ahci`: stores ABAR/port bitmap; `poll_dirty()` is a cheap per-port
    `PxSSTS.DET == 3` probe (safe on every prompt), `rescan()` starts
    newly-present ports (full `start_port` bring-up) and tombstones
    slots of removed ones.  **Slots never shift** — indices stay stable
    so VFS disk handles and `ACTIVE` selection keep working; `select()`
    rejects tombstones.
- `fat`: `Volume` gains a `mounted` flag; `rescan()` unmounts volumes
    whose disk left and mounts every disk without a mounted volume
    (which also re-tries mounts that failed at boot, e.g. a non-FAT
    disk replaced by a FAT one).  UID dedup ignores tombstones, so a
    re-plugged disk keeps its `/disk/<UID>` name.
- `fs`: the init mount loop is factored into `rescan_disks()`, which
    lazily creates `/disk`, ensures a mount node per mounted volume and
    tombstones nodes of unmounted ones — `cat /disk/<gone>/…` fails
    cleanly with "no such file".
- shell: `rescan` command plus an automatic deep rescan before each
    prompt whenever `poll_dirty()` sees a change; a one-line summary
    reports added/removed counts.  All rescan events flow through
    `serial_println!` so they land in klog (`ahci: port N up`, `fat:
    mounted …`) — the boot log doubles as a device log.

  **QEMU testability reality (verified)**: PCIe ADD works end-to-end
  (`-device pcie-root-port,id=rp1` at boot, monitor `device_add
  e1000e,bus=rp1` → the kernel's `pci` re-scan sees it; q35 root bus
  itself rejects hotplug, root ports are required).  Disk hotplug is
  **not** QEMU-modelable: the ich9-ahci's IDE buses refuse
  `device_add ide-hd` ("Bus 'ide.N' does not support hotplugging") and
  `drive_add if=ide` ("Can't hot-add drive to type 1"); PCIe
  `device_del` is a guest-cooperative protocol (the device stays in
  config space until the guest powers off the slot), so removal is
  also untestable in QEMU.  The disk arrival/removal path (PxSSTS
  based) is verified on real hardware instead.

## Known issues / perf notes

- **Scroll cost** — scrolling still shifts the whole framebuffer (a
  ~3-8 MB pixel copy per row), but two mechanisms make it fast on real
  hardware:
  1. a RAM backbuffer (`gop::SHADOW`): every raster write goes to the
     shadow too; scrolling is a cached shadow memcpy + one write-only
     push to the display — the display is never *read* (uncached reads
     were the killer, since real-hardware GOP framebuffers are
     uncacheable/write-combining).
  2. deferred scroll batching (`vt::PENDING_SCROLLS`): the logical grid
     scrolls instantly in RAM while the framebuffer shift+push is
     deferred; N pending scrolls cost one push.  Bursts of streaming
     output (hexdump/cat) push every ~16 rows (or after ~40 ms of
     pause), and the shell syncs before each prompt.  Any CSI sequence
     (full-screen apps like the editor) or interactive input pushes
     immediately.
  Still on the someday list: a pan-based ring so scrolling never touches
  the whole framebuffer at all.
- **Editor redraw** re-blits changed cells per keystroke (fine at 1280×800);
  the "optimize drawing someday" item.
- **AHCI quirks learned (QEMU ground truth)**: the command header layout is
  FIS length in DW0 bits 4:0 and PRDTL in bits 31:16 (not the classic
  CFL-at-24 bit-fields); the H2D FIS sector count lives at bytes 12-13 with
  the device register at byte 7 (byte 10 is LBA[47:40] — no separate
  device byte); QEMU rejects H2D FISes with flags bits 6:4 set, so the
  write direction is signalled purely by the header W bit (1<<6).
  Real hardware follows the same FIS layout; the header layout matches
  Linux libahci.
- **ramfs only for `/`**: the root filesystem is still the ramfs (lost on
  reboot); persistent files live under `/disk` (AHCI/FAT32).

## Pending decisions (agreed, not yet implemented)

- **Disk recognition scope — keep simple for now**: no pre-EBS UEFI
  block-device census, no GPT/MBR partition parsing, no NVMe driver.
  (Revisit later: Linux-style recognition = PCI scan → ahci/nvme →
  partitions → superblock autodetect → by-label/by-uuid naming; our
  `/disk/<LABEL>-<SERIAL>` already mirrors the by-label/by-uuid idea.)

## Target audience

Learning exercise and toy.  Not a Linux replacement.  Not POSIX-compatible.

---

## Session notes — shell experience redesign (implemented)

Working from `shell_experience`.  Plan as stated:

- Commands accept **post-filters** as trailing flags: `-h N` (head),
  `-t N` (tail), `-g PAT` (grep).  Flag placement (at the end of the
  command) is respected.
- Redirection uses **keywords**, not `> << < >>`:
  `WRITING`/`WRITE` → overwrite, `UNTIL`/`TIL` → heredoc,
  `USE`/`USING` → input, `TO` → append.
- **No pipes.**  `head`/`tail`/`grep` needs are covered by the flags;
  a pager flag is a maybe.

Resolved this session:

- **Post-filter order = written order, left to right.**  Each flag is a
  filter stage applied to the command's output as a line stream:
  `cat file -h 50 -t 10 -g pat` = "lines containing `pat` among the last
  10 lines of the first 50 lines".  (The `shell_experience` prose for
  that example was loose; the left-to-right rule is canonical.)
- **`-t N` holds back.**  The tail stage buffers a rolling window of the
  last N lines and emits nothing until its upstream ends — EOF, or the
  user's ^C, which flushes the window to the screen.  `cat /dev/mem -g
  100 -t 10` therefore shows nothing while it runs and prints the last 10
  matches when ^C ends it.
- **`-h N` truncates.**  Once N lines have passed through, the head stage
  kills the upstream command — `cat /dev/mem -h 5` peeks 5 lines and
  stops instead of flooding.
- **`-g PAT` = smart-case literal substring.**  Match is case-insensitive
  unless PAT contains an uppercase letter, in which case it is
  case-sensitive (editor-style smart case).  No regex, no wildcards.
- **Post-filters repeat.**  `-h`, `-t` and `-g` may each appear more than
  once; every instance is a separate stage in the left-to-right chain
  (so `-h 10 -h 5` = first 5, `-t 10 -t 5` = last 5, `-g foo -g bar` =
  lines containing both).  After `-g`, the next token is the pattern
  verbatim (`-g -h` = pattern "-h").
- **Redirection keywords pair strictly.**  At most one input keyword and
  one output keyword per command line (both may appear together); the
  input keyword must precede the output keyword.  `USING f TO g` is
  accepted, `TO g USING f` is a syntax error, and so is any second
  keyword of the same kind (`TO a WRITE b`).
- **Trailing-region grammar (canonical order):**
  `cmd args [USE src] [TIL marker] [filters…] [output kw dst]` —
  input keywords first, then filter flags, then the output keyword.
  Rationale: "we feed the command input, parse it with filters, put
  that into output".  `USE` must precede `TIL`; filters come after the
  input keywords and before the output keyword.
- **`TIL` is a terminator, not a source.**  Unlike `USE` (which names an
  input source), `TIL <marker>` ends the input at a marker, so it may
  combine with `USE` — two "input" keywords are allowed, in the fixed
  order `USE` then `TIL`.
- **`TIL` truncates the source at the marker.**  The marker line is not
  part of the input and everything from the marker on is discarded —
  `USE f TIL m` reads only the text of `f` before `m`.  Marker match is
  an exact full-line match.
- **^D is EOF everywhere; ^D/^C no longer power off.**  Inside a `TIL`
  input block, ^D ends the input early exactly like a file's EOF.
  Shutting down via ^D or ^C is removed: at the prompt, ^D on an empty
  line is a no-op (fresh prompt); ^C keeps its line-cancel /
  stream-abort role; `poweroff`/`exit` remain the only shutdown verbs.
- **Input consumers: `cat` and `hexdump` only, and only when path-less.**
  With no path (or with an input keyword), they read the input stream
  (`cat USE a TO b` copies a→b; `cat TIL END` is the classic heredoc).
  All other commands reject input keywords with a syntax error — a
  command declares whether it takes input.  `USE`/`TIL` are general
  building blocks; more consumers will come later.
- **`cat <path> TIL <marker>` is sugar for `cat USE <path> TIL
  <marker>`** (refined this session): the paths are the input source,
  truncated at the first marker line across the combined stream —
  `cat a b TIL m` reads `a` then `b` and stops at the first `m` line.
  The single-source reading is unambiguous, so path args + `TIL` is
  valid; path args + `USE` stays a syntax error (two input sources).
- **Keywords: uppercase-only, aliases valid.**  `WRITE`==`WRITING`,
  `TIL`==`UNTIL`, `USE`==`USING`; lowercase forms are ordinary text, so
  `echo use the force` can never redirect.  Keywords are reserved
  operators — a keyword is always an operator, never an argument
  (collision escape: quoting, a future building block).
- **`-p` = pause flag (pager), more-style.**  A stage that pauses at
  every screenful of output and waits for a keypress; engages only when
  output exceeds one screenful, no args, no scrollback/search.  As a
  stage it chains with the others — `-h N -p -g pat -p` lets the user
  see intermediate steps of the filtering.
- **`> >> < <<` are gone.**  Keywords are the *only* redirection syntax;
  `>` is now ordinary text (`echo a > b` prints `a > b`).  Collateral:
  `scripts/test-disk.sh` and the `help` text must be rewritten to the
  keyword syntax; the milestone bullets above describing `echo >`/`>>`
  record the old behavior (superseded by these notes).
- **Quoting (C): `"..."` groups + literalizes; `\x` escapes one char.**
  Quoted/escaped text is always literal — never an operator, never
  split.  So `echo foo "WRITING" bar` prints `foo WRITING bar`; to
  print literal quote characters: `echo foo \"WRITING\" bar`.  Also
  enables spaces in filenames and multi-word `-g` patterns
  (`-g "a b"`).  Quote chars are delimiters, not content.

Assumed edge semantics (flag if wrong):

- Filtered output goes to both terminal (GOP) and serial, identically.
  A file sink (WRITING/TO) replaces terminal output entirely — screen
  is silent, as `echo >` behaved before.
- `-p` pauses terminal output only; with a file sink it is a no-op.
- `-h N` / `-t N` require N ≥ 1; otherwise syntax error.
- `cat p USE s` (path args *and* an explicit source keyword) is a
  syntax error — the input source is ambiguous.  `cat USE s`, path-less
  `cat TIL m`, and `cat <path> TIL m` (sugar for `USE <path> TIL m`)
  are the valid input forms.
- ADR 0002 (`docs/adr/0002-keyword-redirection-and-post-filters.md`)
  captures the keyword-redirection + post-filter decision; the ADR is
  the authoritative statement, these notes are the working detail.

Open questions (to resolve this session):

1. ~~Order of application of `-h`/`-t`/`-g`~~ — **resolved**: left to
   right.
2. ~~`-h`/`-t` streaming semantics~~ — **resolved**: `-t` hold-back +
   ^C flush; `-h` truncates the upstream.
3. ~~`-h`/`-t` semantics on infinite streams (e.g. `cat /dev/mem`)~~ —
   **resolved**: folded into 2.
4. ~~`-g` pattern semantics (literal substring vs regex; case
   sensitivity)~~ — **resolved**: smart-case literal substring;
   repeatable stages.
5. ~~Keyword details~~ — **resolved**: uppercase-only, both aliases
   valid, keywords fully reserved; quoting (also resolved this
   session) is the escape mechanism.
6. ~~Pager flag: in or out~~ — **resolved**: `-p` pause flag, in.
7. ~~Which commands consume input~~ — **resolved**: `cat` + `hexdump`,
   path-less only; other commands reject input keywords.
8. ~~Fate of the existing `>` / `>>` redirection~~ — **resolved**: full
   removal; `>` is plain text; scripts/help to be rewritten.

## Implementation (this session — done, verified in QEMU)

Everything above is now implemented in `shell/src/lib.rs`:

- **Tokenizer** — `"…"` groups text (delimiter, not content), `\x`
  escapes one char, a token containing any quoted/escaped char is
  flagged literal and can never be an operator or flag.
- **Parser** — canonical grammar for every command; keywords
  uppercase-only with aliases, fully reserved (quoted forms are
  literal data); strict ordering enforced (USE before TIL, input
  before filters, output keyword last) with syntax errors otherwise;
  `-h`/`-t` need N ≥ 1, `-g` takes the next token verbatim; unknown
  flags before the trailing region are ordinary arguments.
- **Filter pipeline** — stages over a line stream, left to right,
  each repeatable: `Head` kills the upstream (producers check the
  `line()` return), `Tail` holds a rolling window and flushes through
  the remaining stages at end-of-stream (EOF, ^C, or head kill),
  `Grep` is smart-case literal substring, `Pause` is a terminal-only
  screenful pager (^C aborts; dropped entirely with a file sink).
- **Sinks** — terminal (GOP + serial, identical) or file; a file sink
  replaces the screen, opens lazily on first output, `WRITING`
  truncates and writes from offset 0, `TO` appends at current size.
  Diagnostics (errors, usage) always go to the terminal, never into
  the pipeline or a file.
- **Input consumers** — `cat`/`hexdump`: no keyword = keyboard until
  ^D, `USE f` = file, `TIL m` = keyboard heredoc, `USE f TIL m` =
  file truncated at the exact marker line (marker and everything
  after discarded), and `cat <path> TIL m` / `hexdump <path> TIL m` =
  the paths as source, truncated at the first marker across the
  combined stream.  Other commands reject input keywords.
- **Ordering enforcement** — `USE`/`TIL` must precede the filters and
  the output keyword; filters must precede the output keyword;
  distinct messages for each violation ("must come before the
  filters" / "must come before the output keyword").
- **^D** — EOF everywhere, at the prompt it is a no-op (fresh
  prompt); ^C keeps line-cancel / stream-abort; `poweroff`/`exit`
  are the only shutdown verbs.
- **Collateral** — `help` text rewritten; `scripts/test-disk.sh` and
  the `sendkeys.py` docstring updated to keyword syntax.

Bugs found by QEMU verification and fixed:

1. `WRITING` over an existing file left stale NUL padding —
   `fs::size()` returns the node-cached size, stale after
   `set_len(0)`; the overwrite path now writes at literal offset 0
   (the pre-redesign echo wrote at 0 too).
2. `cat USE f TIL m` leaked everything after the marker — the
   final-partial-line emit ran even after the marker break; fixed
   with a `stopped` flag in `cat_file_until`/`hexdump_file_until`.
3. `cat file TO out TIL m` blamed the filters — the `trailing` flag
   was set by both filters and the output keyword, so the "must come
   before the filters" error fired for the output keyword too; the
   parser now distinguishes them (`TO dst TIL m` → "TIL must come
   before the output keyword").  Also: filters after the output
   keyword (`TO dst -h 2`) were silently accepted — now a syntax
   error.
4. `cat file TIL m` was rejected as ambiguous — path args + `TIL` is
   now sugar for `USE <path> TIL <marker>` (single source, truncated
   at the marker); path args + `USE` remains a syntax error.

Verified: 26 grammar checks pass on a fresh disk *and* on a reused
one (rewriting existing files), `make test-disk` round trip is
fsck-clean, the `-p` pager pauses at screenfuls and resumes on
keypress, `-h` stops `/dev/mem` at exactly N lines, and ^C flushes a
`-t` window on an infinite stream.

Follow-up verification (this session): 10 checks pass in QEMU for the
`cat <path> TIL <marker>` sugar — truncation at the marker, combined
multi-path stream truncated at the first marker, filters after the
sugar (`-h 1`), the heredoc round trip, all four ordering-error
messages, and the `USE`-with-paths ambiguity message; fsck clean.