From 6eb14b64209b9d24554c587f610d7e300ef809ea Mon Sep 17 00:00:00 2001 From: salmonik Date: Sat, 1 Aug 2026 13:14:07 +0200 Subject: shell: keyword redirection + post-filter flags (ADR 0002) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implement the shell experience redesign: one canonical grammar for every command — 'cmd args [USE src] [TIL marker] [filters...] [output kw dst]'. - Keyword redirection (uppercase-only, aliases valid, fully reserved): WRITING/WRITE overwrite, TO append, USE/USING input source, TIL/UNTIL heredoc/terminator (exact full-line marker). > >> < << are removed; '>' is plain text. - Post-filter stages applied left to right over a line stream, each repeatable: -h N (head kills the upstream), -t N (tail holds a rolling window, flushes at EOF/^C), -g PAT (smart-case literal substring), -p (screenful pager, terminal only). A file sink replaces the screen; -p is a no-op with one. - C-style quoting: "..." groups and literalizes, \x escapes one char; quoted/escaped text is never an operator, never split. - Input consumers: cat/hexdump, path-less only (stdin until ^D, USE f, TIL m heredoc, USE f TIL m truncation). Other commands reject input keywords; paths + input keyword is a syntax error. - ^D is EOF everywhere and never powers off; poweroff/exit are the only shutdown verbs. - help rewritten; test-disk.sh and sendkeys.py use the keyword syntax. Verified in QEMU: 26 grammar checks on fresh and reused disks, disk round trip, pager pacing, -h stopping /dev/mem at N lines, ^C flushing a -t window. Two bugs found during verification and fixed: 1. WRITING over an existing file wrote after stale NUL padding — fs::size() is stale after set_len(0); the sink writes at offset 0 after truncation. 2. USE f TIL m leaked the post-marker remainder — the final partial-line emit ran after the marker break; guarded with a stopped flag. --- CONTEXT.md | 208 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 204 insertions(+), 4 deletions(-) (limited to 'CONTEXT.md') diff --git a/CONTEXT.md b/CONTEXT.md index 1c73656..7514a6f 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -168,8 +168,8 @@ Workspace of four crates, all built for the `x86_64-unknown-uefi` target: (`ls`/`cat`/`hexdump`/`mkdir`/`touch`/`rm`, `echo >`/`>>`; devices regenerate on read, `/dev/mem` is raw physical RAM) ✅ **^C / ^D** — Ctrl+C cancels the line or aborts a running stream - (`cat`/`hexdump` check between chunks), Ctrl+D on an empty line - powers off (EOF) + (`cat`/`hexdump` check between chunks); Ctrl+D is EOF (no poweroff — + removed this session; see session notes below) ✅ **tty + vt (terminal) — the 1:1 UNIX console split**: `tty` owns the line discipline (canonical echo/buffer/^C/^D for the shell; raw mode translates keys to the classic terminal byte protocol), `vt` owns the @@ -205,6 +205,32 @@ Workspace of four crates, all built for the `x86_64-unknown-uefi` target: **Files survive reboot** — verified by poweroff → `mdir`/`mtype` on the image and by a fresh boot seeing the prior session's files. +✅ **Shell grammar (ADR 0002)** — one canonical grammar for every + command: `cmd args [USE src] [TIL marker] [filters…] [output kw dst]`. + Keyword redirection (`WRITING`/`WRITE` overwrite, `TO` append, + `USE`/`USING` input source, `TIL`/`UNTIL` heredoc/terminator; + uppercase-only, aliases valid, fully reserved), C-style quoting + (`"…"` groups, `\x` escapes — quoted text is never an operator), + post-filter stages applied left to right (`-h N` head kills the + upstream, `-t N` tail holds back and flushes at EOF/^C, `-g PAT` + smart-case literal substring, `-p` screenful pager; all repeatable), + `>`/`<`/`<<`/`>>` removed (`>` is plain text), file sinks replace the + screen, `cat`/`hexdump` consume input when path-less (stdin heredoc, + `cat USE a TO b` copy, `USE f TIL m` truncation). ^D is EOF + everywhere and never powers off. `help` rewritten; `test-disk.sh` + and `sendkeys.py` updated to the keyword syntax. + + Verified in QEMU (26 grammar checks on a fresh *and* a reused disk, + disk round trip, pager pacing): head kills at N lines, ^C flushes a + `-t` window, smart-case grep, chained/repeated filters, `WRITING`+ + `TO`, `USE..TO` copy, `USE..TIL`, heredoc → file, quoting, `-p`. + Two bugs found and fixed during verification: (1) `WRITING` over an + existing file wrote after stale NUL padding — `fs::size()` returns + the node-cached size, which is stale after `set_len(0)`; the sink + now writes at literal offset 0 after truncation; (2) `USE f TIL m` + leaked everything after the marker — the final partial-line emit + ran even after the marker break; fixed with a `stopped` flag. + QEMU setup: `-drive id=disk,if=none,format=raw,file=disk.img` + `-device ide-hd,drive=disk,bus=ide.0` puts the data disk on AHCI port 0; the ESP gets `bus=ide.1`. (The implicit `-drive file=fat:rw:esp` @@ -223,7 +249,7 @@ Workspace of four crates, all built for the `x86_64-unknown-uefi` target: - `make test-disk` / `./scripts/test-disk.sh` — full automated round trip: build → deploy → ensure image → boot QEMU headless → drive the shell via `sendkey` through the monitor (`ls /disk`, `mkdir`, - `echo >`, `cat`, `poweroff`) → fsck-clean check → `mtype` verifies + `echo WRITING`, `cat`, `poweroff`) → fsck-clean check → `mtype` verifies the written file survived poweroff - `./scripts/sendkeys.py` — types lines into the guest via the QEMU monitor (`--sock`, US layout, named keys for punctuation; bare @@ -269,7 +295,7 @@ Workspace of four crates, all built for the `x86_64-unknown-uefi` target: 128 chars ≈ 16 KB, sequence numbers, wrap/drop accounting. - **`klog` command** = thin alias for `cat /dev/klog` plus filters `-h` (head) and `-t` (tail); no log levels, no timestamps, nothing - fancier. `/dev/klog` is a char device so `klog > /disk/…/boot.log` + fancier. `/dev/klog` is a char device so `klog WRITING /disk/…/boot.log` persists a boot log via the existing redirection. - **Disk recognition scope — keep simple for now**: no pre-EBS UEFI block-device census, no GPT/MBR partition parsing, no NVMe driver. @@ -280,3 +306,177 @@ Workspace of four crates, all built for the `x86_64-unknown-uefi` target: ## Target audience Learning exercise and toy. Not a Linux replacement. Not POSIX-compatible. + +--- + +## Session notes — shell experience redesign (implemented) + +Working from `shell_experience`. Plan as stated: + +- Commands accept **post-filters** as trailing flags: `-h N` (head), + `-t N` (tail), `-g PAT` (grep). Flag placement (at the end of the + command) is respected. +- Redirection uses **keywords**, not `> << < >>`: + `WRITING`/`WRITE` → overwrite, `UNTIL`/`TIL` → heredoc, + `USE`/`USING` → input, `TO` → append. +- **No pipes.** `head`/`tail`/`grep` needs are covered by the flags; + a pager flag is a maybe. + +Resolved this session: + +- **Post-filter order = written order, left to right.** Each flag is a + filter stage applied to the command's output as a line stream: + `cat file -h 50 -t 10 -g pat` = "lines containing `pat` among the last + 10 lines of the first 50 lines". (The `shell_experience` prose for + that example was loose; the left-to-right rule is canonical.) +- **`-t N` holds back.** The tail stage buffers a rolling window of the + last N lines and emits nothing until its upstream ends — EOF, or the + user's ^C, which flushes the window to the screen. `cat /dev/mem -g + 100 -t 10` therefore shows nothing while it runs and prints the last 10 + matches when ^C ends it. +- **`-h N` truncates.** Once N lines have passed through, the head stage + kills the upstream command — `cat /dev/mem -h 5` peeks 5 lines and + stops instead of flooding. +- **`-g PAT` = smart-case literal substring.** Match is case-insensitive + unless PAT contains an uppercase letter, in which case it is + case-sensitive (editor-style smart case). No regex, no wildcards. +- **Post-filters repeat.** `-h`, `-t` and `-g` may each appear more than + once; every instance is a separate stage in the left-to-right chain + (so `-h 10 -h 5` = first 5, `-t 10 -t 5` = last 5, `-g foo -g bar` = + lines containing both). After `-g`, the next token is the pattern + verbatim (`-g -h` = pattern "-h"). +- **Redirection keywords pair strictly.** At most one input keyword and + one output keyword per command line (both may appear together); the + input keyword must precede the output keyword. `USING f TO g` is + accepted, `TO g USING f` is a syntax error, and so is any second + keyword of the same kind (`TO a WRITE b`). +- **Trailing-region grammar (canonical order):** + `cmd args [USE src] [TIL marker] [filters…] [output kw dst]` — + input keywords first, then filter flags, then the output keyword. + Rationale: "we feed the command input, parse it with filters, put + that into output". `USE` must precede `TIL`; filters come after the + input keywords and before the output keyword. +- **`TIL` is a terminator, not a source.** Unlike `USE` (which names an + input source), `TIL ` ends the input at a marker, so it may + combine with `USE` — two "input" keywords are allowed, in the fixed + order `USE` then `TIL`. +- **`TIL` truncates the source at the marker.** The marker line is not + part of the input and everything from the marker on is discarded — + `USE f TIL m` reads only the text of `f` before `m`. Marker match is + an exact full-line match. +- **^D is EOF everywhere; ^D/^C no longer power off.** Inside a `TIL` + input block, ^D ends the input early exactly like a file's EOF. + Shutting down via ^D or ^C is removed: at the prompt, ^D on an empty + line is a no-op (fresh prompt); ^C keeps its line-cancel / + stream-abort role; `poweroff`/`exit` remain the only shutdown verbs. +- **Input consumers: `cat` and `hexdump` only, and only when path-less.** + With no path (or with an input keyword), they read the input stream + (`cat USE a TO b` copies a→b; `cat TIL END` is the classic heredoc). + All other commands reject input keywords with a syntax error — a + command declares whether it takes input. `USE`/`TIL` are general + building blocks; more consumers will come later. +- **Keywords: uppercase-only, aliases valid.** `WRITE`==`WRITING`, + `TIL`==`UNTIL`, `USE`==`USING`; lowercase forms are ordinary text, so + `echo use the force` can never redirect. Keywords are reserved + operators — a keyword is always an operator, never an argument + (collision escape: quoting, a future building block). +- **`-p` = pause flag (pager), more-style.** A stage that pauses at + every screenful of output and waits for a keypress; engages only when + output exceeds one screenful, no args, no scrollback/search. As a + stage it chains with the others — `-h N -p -g pat -p` lets the user + see intermediate steps of the filtering. +- **`> >> < <<` are gone.** Keywords are the *only* redirection syntax; + `>` is now ordinary text (`echo a > b` prints `a > b`). Collateral: + `scripts/test-disk.sh` and the `help` text must be rewritten to the + keyword syntax; the milestone bullets above describing `echo >`/`>>` + record the old behavior (superseded by these notes). +- **Quoting (C): `"..."` groups + literalizes; `\x` escapes one char.** + Quoted/escaped text is always literal — never an operator, never + split. So `echo foo "WRITING" bar` prints `foo WRITING bar`; to + print literal quote characters: `echo foo \"WRITING\" bar`. Also + enables spaces in filenames and multi-word `-g` patterns + (`-g "a b"`). Quote chars are delimiters, not content. + +Assumed edge semantics (flag if wrong): + +- Filtered output goes to both terminal (GOP) and serial, identically. + A file sink (WRITING/TO) replaces terminal output entirely — screen + is silent, as `echo >` behaved before. +- `-p` pauses terminal output only; with a file sink it is a no-op. +- `-h N` / `-t N` require N ≥ 1; otherwise syntax error. +- `cat p USE s` (path args *and* an input keyword) is a syntax error — + the input source is ambiguous. `cat USE s` and path-less `cat TIL m` + are the valid input forms. +- ADR 0002 (`docs/adr/0002-keyword-redirection-and-post-filters.md`) + captures the keyword-redirection + post-filter decision; the ADR is + the authoritative statement, these notes are the working detail. + +Open questions (to resolve this session): + +1. ~~Order of application of `-h`/`-t`/`-g`~~ — **resolved**: left to + right. +2. ~~`-h`/`-t` streaming semantics~~ — **resolved**: `-t` hold-back + + ^C flush; `-h` truncates the upstream. +3. ~~`-h`/`-t` semantics on infinite streams (e.g. `cat /dev/mem`)~~ — + **resolved**: folded into 2. +4. ~~`-g` pattern semantics (literal substring vs regex; case + sensitivity)~~ — **resolved**: smart-case literal substring; + repeatable stages. +5. ~~Keyword details~~ — **resolved**: uppercase-only, both aliases + valid, keywords fully reserved; quoting (also resolved this + session) is the escape mechanism. +6. ~~Pager flag: in or out~~ — **resolved**: `-p` pause flag, in. +7. ~~Which commands consume input~~ — **resolved**: `cat` + `hexdump`, + path-less only; other commands reject input keywords. +8. ~~Fate of the existing `>` / `>>` redirection~~ — **resolved**: full + removal; `>` is plain text; scripts/help to be rewritten. + +## Implementation (this session — done, verified in QEMU) + +Everything above is now implemented in `shell/src/lib.rs`: + +- **Tokenizer** — `"…"` groups text (delimiter, not content), `\x` + escapes one char, a token containing any quoted/escaped char is + flagged literal and can never be an operator or flag. +- **Parser** — canonical grammar for every command; keywords + uppercase-only with aliases, fully reserved (quoted forms are + literal data); strict ordering enforced (USE before TIL, input + before filters, output keyword last) with syntax errors otherwise; + `-h`/`-t` need N ≥ 1, `-g` takes the next token verbatim; unknown + flags before the trailing region are ordinary arguments. +- **Filter pipeline** — stages over a line stream, left to right, + each repeatable: `Head` kills the upstream (producers check the + `line()` return), `Tail` holds a rolling window and flushes through + the remaining stages at end-of-stream (EOF, ^C, or head kill), + `Grep` is smart-case literal substring, `Pause` is a terminal-only + screenful pager (^C aborts; dropped entirely with a file sink). +- **Sinks** — terminal (GOP + serial, identical) or file; a file sink + replaces the screen, opens lazily on first output, `WRITING` + truncates and writes from offset 0, `TO` appends at current size. + Diagnostics (errors, usage) always go to the terminal, never into + the pipeline or a file. +- **Input consumers** — `cat`/`hexdump`, path-less only: no keyword = + keyboard until ^D, `USE f` = file, `TIL m` = keyboard heredoc, + `USE f TIL m` = file truncated at the exact marker line (marker and + everything after discarded). Other commands reject input keywords. +- **^D** — EOF everywhere, at the prompt it is a no-op (fresh + prompt); ^C keeps line-cancel / stream-abort; `poweroff`/`exit` + are the only shutdown verbs. +- **Collateral** — `help` text rewritten; `scripts/test-disk.sh` and + the `sendkeys.py` docstring updated to keyword syntax. + +Bugs found by QEMU verification and fixed: + +1. `WRITING` over an existing file left stale NUL padding — + `fs::size()` returns the node-cached size, stale after + `set_len(0)`; the overwrite path now writes at literal offset 0 + (the pre-redesign echo wrote at 0 too). +2. `cat USE f TIL m` leaked everything after the marker — the + final-partial-line emit ran even after the marker break; fixed + with a `stopped` flag in `cat_file_until`/`hexdump_file_until`. + +Verified: 26 grammar checks pass on a fresh disk *and* on a reused +one (rewriting existing files), `make test-disk` round trip is +fsck-clean, the `-p` pager pauses at screenfuls and resumes on +keypress, `-h` stops `/dev/mem` at exactly N lines, and ^C flushes a +`-t` window on an infinite stream. -- cgit v1.2.3