shell: keyword redirection + post-filter flags (ADR 0002)

Implement the shell experience redesign: one canonical grammar for every
command — 'cmd args [USE src] [TIL marker] [filters...] [output kw dst]'.

- Keyword redirection (uppercase-only, aliases valid, fully reserved):
  WRITING/WRITE overwrite, TO append, USE/USING input source,
  TIL/UNTIL heredoc/terminator (exact full-line marker).  > >> < << are
  removed; '>' is plain text.
- Post-filter stages applied left to right over a line stream, each
  repeatable: -h N (head kills the upstream), -t N (tail holds a
  rolling window, flushes at EOF/^C), -g PAT (smart-case literal
  substring), -p (screenful pager, terminal only).  A file sink
  replaces the screen; -p is a no-op with one.
- C-style quoting: "..." groups and literalizes, \x escapes one char;
  quoted/escaped text is never an operator, never split.
- Input consumers: cat/hexdump, path-less only (stdin until ^D,
  USE f, TIL m heredoc, USE f TIL m truncation).  Other commands
  reject input keywords; paths + input keyword is a syntax error.
- ^D is EOF everywhere and never powers off; poweroff/exit are the
  only shutdown verbs.
- help rewritten; test-disk.sh and sendkeys.py use the keyword syntax.

Verified in QEMU: 26 grammar checks on fresh and reused disks, disk
round trip, pager pacing, -h stopping /dev/mem at N lines, ^C flushing
a -t window.  Two bugs found during verification and fixed:
1. WRITING over an existing file wrote after stale NUL padding —
   fs::size() is stale after set_len(0); the sink writes at offset 0
   after truncation.
2. USE f TIL m leaked the post-marker remainder — the final
   partial-line emit ran after the marker break; guarded with a
   stopped flag.

1 files changed, 204 insertions, 4 deletions

diff --git a/CONTEXT.md b/CONTEXT.md
index 1c73656..7514a6f 100644
--- a/CONTEXT.md
+++ b/CONTEXT.md
@@ -168,8 +168,8 @@ Workspace of four crates, all built for the `x86_64-unknown-uefi` target:
   (`ls`/`cat`/`hexdump`/`mkdir`/`touch`/`rm`, `echo >`/`>>`; devices
   regenerate on read, `/dev/mem` is raw physical RAM)
 ✅ **^C / ^D** — Ctrl+C cancels the line or aborts a running stream
-  (`cat`/`hexdump` check between chunks), Ctrl+D on an empty line
-  powers off (EOF)
+  (`cat`/`hexdump` check between chunks); Ctrl+D is EOF (no poweroff —
+  removed this session; see session notes below)
 ✅ **tty + vt (terminal) — the 1:1 UNIX console split**: `tty` owns the
   line discipline (canonical echo/buffer/^C/^D for the shell; raw mode
   translates keys to the classic terminal byte protocol), `vt` owns the
@@ -205,6 +205,32 @@ Workspace of four crates, all built for the `x86_64-unknown-uefi` target:
   **Files survive reboot** — verified by poweroff → `mdir`/`mtype` on
   the image and by a fresh boot seeing the prior session's files.
 
+✅ **Shell grammar (ADR 0002)** — one canonical grammar for every
+  command: `cmd args [USE src] [TIL marker] [filters…] [output kw dst]`.
+  Keyword redirection (`WRITING`/`WRITE` overwrite, `TO` append,
+  `USE`/`USING` input source, `TIL`/`UNTIL` heredoc/terminator;
+  uppercase-only, aliases valid, fully reserved), C-style quoting
+  (`"…"` groups, `\x` escapes — quoted text is never an operator),
+  post-filter stages applied left to right (`-h N` head kills the
+  upstream, `-t N` tail holds back and flushes at EOF/^C, `-g PAT`
+  smart-case literal substring, `-p` screenful pager; all repeatable),
+  `>`/`<`/`<<`/`>>` removed (`>` is plain text), file sinks replace the
+  screen, `cat`/`hexdump` consume input when path-less (stdin heredoc,
+  `cat USE a TO b` copy, `USE f TIL m` truncation).  ^D is EOF
+  everywhere and never powers off.  `help` rewritten; `test-disk.sh`
+  and `sendkeys.py` updated to the keyword syntax.
+
+  Verified in QEMU (26 grammar checks on a fresh *and* a reused disk,
+  disk round trip, pager pacing): head kills at N lines, ^C flushes a
+  `-t` window, smart-case grep, chained/repeated filters, `WRITING`+
+  `TO`, `USE..TO` copy, `USE..TIL`, heredoc → file, quoting, `-p`.
+  Two bugs found and fixed during verification: (1) `WRITING` over an
+  existing file wrote after stale NUL padding — `fs::size()` returns
+  the node-cached size, which is stale after `set_len(0)`; the sink
+  now writes at literal offset 0 after truncation; (2) `USE f TIL m`
+  leaked everything after the marker — the final partial-line emit
+  ran even after the marker break; fixed with a `stopped` flag.
+
   QEMU setup: `-drive id=disk,if=none,format=raw,file=disk.img` +
   `-device ide-hd,drive=disk,bus=ide.0` puts the data disk on AHCI port
   0; the ESP gets `bus=ide.1`.  (The implicit `-drive file=fat:rw:esp`
@@ -223,7 +249,7 @@ Workspace of four crates, all built for the `x86_64-unknown-uefi` target:
 - `make test-disk` / `./scripts/test-disk.sh` — full automated round
     trip: build → deploy → ensure image → boot QEMU headless → drive
     the shell via `sendkey` through the monitor (`ls /disk`, `mkdir`,
-    `echo >`, `cat`, `poweroff`) → fsck-clean check → `mtype` verifies
+    `echo WRITING`, `cat`, `poweroff`) → fsck-clean check → `mtype` verifies
     the written file survived poweroff
 - `./scripts/sendkeys.py` — types lines into the guest via the QEMU
     monitor (`--sock`, US layout, named keys for punctuation; bare
@@ -269,7 +295,7 @@ Workspace of four crates, all built for the `x86_64-unknown-uefi` target:
   128 chars ≈ 16 KB, sequence numbers, wrap/drop accounting.
 - **`klog` command** = thin alias for `cat /dev/klog` plus filters
   `-h` (head) and `-t` (tail); no log levels, no timestamps, nothing
-  fancier.  `/dev/klog` is a char device so `klog > /disk/…/boot.log`
+  fancier.  `/dev/klog` is a char device so `klog WRITING /disk/…/boot.log`
   persists a boot log via the existing redirection.
 - **Disk recognition scope — keep simple for now**: no pre-EBS UEFI
   block-device census, no GPT/MBR partition parsing, no NVMe driver.
@@ -280,3 +306,177 @@ Workspace of four crates, all built for the `x86_64-unknown-uefi` target:
 ## Target audience
 
 Learning exercise and toy.  Not a Linux replacement.  Not POSIX-compatible.
+
+---
+
+## Session notes — shell experience redesign (implemented)
+
+Working from `shell_experience`.  Plan as stated:
+
+- Commands accept **post-filters** as trailing flags: `-h N` (head),
+  `-t N` (tail), `-g PAT` (grep).  Flag placement (at the end of the
+  command) is respected.
+- Redirection uses **keywords**, not `> << < >>`:
+  `WRITING`/`WRITE` → overwrite, `UNTIL`/`TIL` → heredoc,
+  `USE`/`USING` → input, `TO` → append.
+- **No pipes.**  `head`/`tail`/`grep` needs are covered by the flags;
+  a pager flag is a maybe.
+
+Resolved this session:
+
+- **Post-filter order = written order, left to right.**  Each flag is a
+  filter stage applied to the command's output as a line stream:
+  `cat file -h 50 -t 10 -g pat` = "lines containing `pat` among the last
+  10 lines of the first 50 lines".  (The `shell_experience` prose for
+  that example was loose; the left-to-right rule is canonical.)
+- **`-t N` holds back.**  The tail stage buffers a rolling window of the
+  last N lines and emits nothing until its upstream ends — EOF, or the
+  user's ^C, which flushes the window to the screen.  `cat /dev/mem -g
+  100 -t 10` therefore shows nothing while it runs and prints the last 10
+  matches when ^C ends it.
+- **`-h N` truncates.**  Once N lines have passed through, the head stage
+  kills the upstream command — `cat /dev/mem -h 5` peeks 5 lines and
+  stops instead of flooding.
+- **`-g PAT` = smart-case literal substring.**  Match is case-insensitive
+  unless PAT contains an uppercase letter, in which case it is
+  case-sensitive (editor-style smart case).  No regex, no wildcards.
+- **Post-filters repeat.**  `-h`, `-t` and `-g` may each appear more than
+  once; every instance is a separate stage in the left-to-right chain
+  (so `-h 10 -h 5` = first 5, `-t 10 -t 5` = last 5, `-g foo -g bar` =
+  lines containing both).  After `-g`, the next token is the pattern
+  verbatim (`-g -h` = pattern "-h").
+- **Redirection keywords pair strictly.**  At most one input keyword and
+  one output keyword per command line (both may appear together); the
+  input keyword must precede the output keyword.  `USING f TO g` is
+  accepted, `TO g USING f` is a syntax error, and so is any second
+  keyword of the same kind (`TO a WRITE b`).
+- **Trailing-region grammar (canonical order):**
+  `cmd args [USE src] [TIL marker] [filters…] [output kw dst]` —
+  input keywords first, then filter flags, then the output keyword.
+  Rationale: "we feed the command input, parse it with filters, put
+  that into output".  `USE` must precede `TIL`; filters come after the
+  input keywords and before the output keyword.
+- **`TIL` is a terminator, not a source.**  Unlike `USE` (which names an
+  input source), `TIL <marker>` ends the input at a marker, so it may
+  combine with `USE` — two "input" keywords are allowed, in the fixed
+  order `USE` then `TIL`.
+- **`TIL` truncates the source at the marker.**  The marker line is not
+  part of the input and everything from the marker on is discarded —
+  `USE f TIL m` reads only the text of `f` before `m`.  Marker match is
+  an exact full-line match.
+- **^D is EOF everywhere; ^D/^C no longer power off.**  Inside a `TIL`
+  input block, ^D ends the input early exactly like a file's EOF.
+  Shutting down via ^D or ^C is removed: at the prompt, ^D on an empty
+  line is a no-op (fresh prompt); ^C keeps its line-cancel /
+  stream-abort role; `poweroff`/`exit` remain the only shutdown verbs.
+- **Input consumers: `cat` and `hexdump` only, and only when path-less.**
+  With no path (or with an input keyword), they read the input stream
+  (`cat USE a TO b` copies a→b; `cat TIL END` is the classic heredoc).
+  All other commands reject input keywords with a syntax error — a
+  command declares whether it takes input.  `USE`/`TIL` are general
+  building blocks; more consumers will come later.
+- **Keywords: uppercase-only, aliases valid.**  `WRITE`==`WRITING`,
+  `TIL`==`UNTIL`, `USE`==`USING`; lowercase forms are ordinary text, so
+  `echo use the force` can never redirect.  Keywords are reserved
+  operators — a keyword is always an operator, never an argument
+  (collision escape: quoting, a future building block).
+- **`-p` = pause flag (pager), more-style.**  A stage that pauses at
+  every screenful of output and waits for a keypress; engages only when
+  output exceeds one screenful, no args, no scrollback/search.  As a
+  stage it chains with the others — `-h N -p -g pat -p` lets the user
+  see intermediate steps of the filtering.
+- **`> >> < <<` are gone.**  Keywords are the *only* redirection syntax;
+  `>` is now ordinary text (`echo a > b` prints `a > b`).  Collateral:
+  `scripts/test-disk.sh` and the `help` text must be rewritten to the
+  keyword syntax; the milestone bullets above describing `echo >`/`>>`
+  record the old behavior (superseded by these notes).
+- **Quoting (C): `"..."` groups + literalizes; `\x` escapes one char.**
+  Quoted/escaped text is always literal — never an operator, never
+  split.  So `echo foo "WRITING" bar` prints `foo WRITING bar`; to
+  print literal quote characters: `echo foo \"WRITING\" bar`.  Also
+  enables spaces in filenames and multi-word `-g` patterns
+  (`-g "a b"`).  Quote chars are delimiters, not content.
+
+Assumed edge semantics (flag if wrong):
+
+- Filtered output goes to both terminal (GOP) and serial, identically.
+  A file sink (WRITING/TO) replaces terminal output entirely — screen
+  is silent, as `echo >` behaved before.
+- `-p` pauses terminal output only; with a file sink it is a no-op.
+- `-h N` / `-t N` require N ≥ 1; otherwise syntax error.
+- `cat p USE s` (path args *and* an input keyword) is a syntax error —
+  the input source is ambiguous.  `cat USE s` and path-less `cat TIL m`
+  are the valid input forms.
+- ADR 0002 (`docs/adr/0002-keyword-redirection-and-post-filters.md`)
+  captures the keyword-redirection + post-filter decision; the ADR is
+  the authoritative statement, these notes are the working detail.
+
+Open questions (to resolve this session):
+
+1. ~~Order of application of `-h`/`-t`/`-g`~~ — **resolved**: left to
+   right.
+2. ~~`-h`/`-t` streaming semantics~~ — **resolved**: `-t` hold-back +
+   ^C flush; `-h` truncates the upstream.
+3. ~~`-h`/`-t` semantics on infinite streams (e.g. `cat /dev/mem`)~~ —
+   **resolved**: folded into 2.
+4. ~~`-g` pattern semantics (literal substring vs regex; case
+   sensitivity)~~ — **resolved**: smart-case literal substring;
+   repeatable stages.
+5. ~~Keyword details~~ — **resolved**: uppercase-only, both aliases
+   valid, keywords fully reserved; quoting (also resolved this
+   session) is the escape mechanism.
+6. ~~Pager flag: in or out~~ — **resolved**: `-p` pause flag, in.
+7. ~~Which commands consume input~~ — **resolved**: `cat` + `hexdump`,
+   path-less only; other commands reject input keywords.
+8. ~~Fate of the existing `>` / `>>` redirection~~ — **resolved**: full
+   removal; `>` is plain text; scripts/help to be rewritten.
+
+## Implementation (this session — done, verified in QEMU)
+
+Everything above is now implemented in `shell/src/lib.rs`:
+
+- **Tokenizer** — `"…"` groups text (delimiter, not content), `\x`
+  escapes one char, a token containing any quoted/escaped char is
+  flagged literal and can never be an operator or flag.
+- **Parser** — canonical grammar for every command; keywords
+  uppercase-only with aliases, fully reserved (quoted forms are
+  literal data); strict ordering enforced (USE before TIL, input
+  before filters, output keyword last) with syntax errors otherwise;
+  `-h`/`-t` need N ≥ 1, `-g` takes the next token verbatim; unknown
+  flags before the trailing region are ordinary arguments.
+- **Filter pipeline** — stages over a line stream, left to right,
+  each repeatable: `Head` kills the upstream (producers check the
+  `line()` return), `Tail` holds a rolling window and flushes through
+  the remaining stages at end-of-stream (EOF, ^C, or head kill),
+  `Grep` is smart-case literal substring, `Pause` is a terminal-only
+  screenful pager (^C aborts; dropped entirely with a file sink).
+- **Sinks** — terminal (GOP + serial, identical) or file; a file sink
+  replaces the screen, opens lazily on first output, `WRITING`
+  truncates and writes from offset 0, `TO` appends at current size.
+  Diagnostics (errors, usage) always go to the terminal, never into
+  the pipeline or a file.
+- **Input consumers** — `cat`/`hexdump`, path-less only: no keyword =
+  keyboard until ^D, `USE f` = file, `TIL m` = keyboard heredoc,
+  `USE f TIL m` = file truncated at the exact marker line (marker and
+  everything after discarded).  Other commands reject input keywords.
+- **^D** — EOF everywhere, at the prompt it is a no-op (fresh
+  prompt); ^C keeps line-cancel / stream-abort; `poweroff`/`exit`
+  are the only shutdown verbs.
+- **Collateral** — `help` text rewritten; `scripts/test-disk.sh` and
+  the `sendkeys.py` docstring updated to keyword syntax.
+
+Bugs found by QEMU verification and fixed:
+
+1. `WRITING` over an existing file left stale NUL padding —
+   `fs::size()` returns the node-cached size, stale after
+   `set_len(0)`; the overwrite path now writes at literal offset 0
+   (the pre-redesign echo wrote at 0 too).
+2. `cat USE f TIL m` leaked everything after the marker — the
+   final-partial-line emit ran even after the marker break; fixed
+   with a `stopped` flag in `cat_file_until`/`hexdump_file_until`.
+
+Verified: 26 grammar checks pass on a fresh disk *and* on a reused
+one (rewriting existing files), `make test-disk` round trip is
+fsck-clean, the `-p` pager pauses at screenfuls and resumes on
+keypress, `-h` stops `/dev/mem` at exactly N lines, and ^C flushes a
+`-t` window on an infinite stream.