1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
|
# CONTEXT.md — raisernics Kernel Glossary
## raisernics
Custom x86_64 kernel for the RNG ecosystem. Toy project — self-contained,
no strict userspace separation, TempleOS-inspired design philosophy.
Built in Rust, compiled for the `x86_64-unknown-uefi` target (stable).
## Boot interface
The kernel is a **PE32+ EFI application** loaded via UEFI
`LoadImage`/`StartImage`.
The existing bootloader (rsbl) loads us from `\bzImage` and passes the UEFI
system table. No bootloader changes needed.
## Output strategy
Two-phase console model:
- **Phase 1 (boot services alive):** UEFI `con_out->OutputString()` — works
with whatever UEFI firmware routes (serial + framebuffer).
- **Phase 2 (native):** Raw serial UART 16550 driver on COM1 (I/O port
`0x3F8`). Activated after `ExitBootServices` when we own the hardware.
## Dependency philosophy
Aligns with RNG philosophy: minimal dependencies. Freestanding Rust,
no libc, no external crates at the kernel/core level. UEFI protocol
structs are hand-defined (not from `uefi-rs` crate) to stay lightweight.
## Paging strategy
Long mode mandates paging (`CR0.PG = 1`), so we accept that tax but keep it
simple. Identity-mapped 4-level tables, no higher-half, no demand paging,
no guard pages except task stacks — each task's stack gets one unmapped
guard page below it, and a page-fault handler turns an overflow into a
task panic. `paging.rs` provides identity map/unmap/query helpers. At
boot the kernel runs on UEFI's tables; `adopt_tables()` rebuilds that map
into the kernel's own tables and switches CR3 — needed because UEFI maps its
page-table pages read-only, so the firmware tree can't be modified in place.
Firmware large pages (2 MB / 1 GB) are split into 4 KB tables on demand;
page-table pages come from the physical page allocator.
## Physical memory
Before `ExitBootServices`, the kernel reads the UEFI memory map to know
which physical regions are free. The heap is a first-fit **free-list
allocator** with splitting and coalescing: blocks carry a 16-byte header
(size + free flag), free blocks are linked in address order, and an 8-byte
back-pointer before each payload lets `free` find its block even when
alignment left a gap. When the free list is exhausted, the heap **grows**
by taking pages from the page allocator (proportional to the request),
keeping the two allocators cooperating. A **bitmap page allocator** manages
physical pages: one bit per 4 KB page spanning the conventional regions, so
MMIO holes and firmware-reserved gaps are never handed out, and the heap
region itself is reserved to keep the allocators from double-allocating the
same RAM. Contiguous runs with power-of-two alignment are supported
(`alloc_pages_aligned`).
## Input strategy
**PS/2 keyboard, interrupt-driven.** IRQ1 pushes scancodes into a ring
buffer (`interrupts::KEYBUF`); the shell polls it via `read_scancode()`.
Port I/O to `0x60`/`0x64`. Scancode set 1 (US QWERTY layout in the
initial build).
## Display strategy
Two-phase model matching output:
- **Phase 1:** UEFI `con_out->OutputString()` (framebuffer + serial handled
by firmware).
- **Phase 2:** GOP (Graphics Output Protocol) framebuffer obtained before
`ExitBootServices`, driven by a built-in 8×16 bitmap font. Text rendered
by drawing glyphs directly into the framebuffer.
The framebuffer address stays valid after `ExitBootServices` — the GPU's
linear frame buffer is system memory that we own once boot services are gone.
## Window
A rectangular display surface owned by exactly one task and composited
onto the 800×600 canvas by the window manager. The global console (vt)
becomes the first window, owned by the shell; further windows are spawned
per task (a terminal, a drawing canvas, …). Windows may overlap, so they
carry a z-order. Every window is a pixel surface: a terminal window is a
text grid rendered into its own pixels, a drawing canvas draws pixels
directly. The ruling constraint is "stupid simple".
## Chrome
Window-manager-owned UI drawn on the canvas but not a window: the
background/desktop fill, per-window title bars, and (later) a
launcher/menu. Chrome is not focusable and does not take input as a
window.
## Title bar
Per-window chrome strip at the top of a window showing the window's title
and its minimize and close buttons. Drawn by the window manager, not the
app.
## Minimized
A window state in which the window is not rendered (hidden); the app keeps
running. The inverse of the normal visible state.
## Close
The action that destroys a window and ends its app.
## App
A task that owns exactly one window and receives input events through its
message queue. A GUI program is an app; there is no separate "process"
concept — the app is its task plus its window. The home terminal is app 0
(window 0).
## Binary
A PE32+ file holding executable code that the kernel can load and run
without being recompiled into the kernel image. A binary runs against
the kernel's exported interface, never by linking kernel state directly.
Binaries come in two kinds, separated by **launch surface** rather than
implementation: **CLI binaries** live in /bin/cli and run inside a
terminal session (invoked by a shell, inheriting its input/output);
**GUI binaries** live in /bin/gui and are launchable with no terminal at
all — they own their windows — which makes enumerating launchable apps
as simple as listing /bin/gui. Every binary runs as its own task — a CLI binary's task is
waited on by the calling terminal, a GUI binary's task owns its window.
## System image
The single dd-able FAT filesystem image the build produces: bootloader,
kernel, /bin, /etc and seed content on one volume. Imaging it onto a disk
yields a complete installed system. Binaries live on the system image
under /bin/cli and /bin/gui; the shell finds them by scanning mounted
volumes — no volume is privileged. For user-facing paths, /bin is a
transparent VFS alias: `ls /bin/cli` resolves to the first mounted
volume whose root has \bin, so nobody needs /disk/<UID>/bin/... (ADR
0006).
## Loader
The kernel facility that takes a binary from disk, places it in memory,
and starts it. Loading changes no address-space setup — binaries run in
the one shared identity-mapped space.
## Exports table
The interface between a binary and the kernel: a versioned table of
functions the kernel hands to a binary at entry (output, filesystem,
memory, …). The caller constructs it per invocation, so the same binary's
output can be aimed at a terminal or into the shell's filter pipeline.
All kernel state stays behind the table — a binary never owns a copy of it.
## Terminal
A program that presents a command line in a window: a text terminal
emulator plus a shell command loop, rendered into the window's pixel
buffer. The home terminal is the persistent full-screen chrome-less
desktop; additional terminals are ordinary windows (F1).
## Focus
The single window (if any) that receives keyboard and mouse input. Focus
is set by clicking a window, which also raises it; input is delivered to
the focused app's message queue. With no focused window, a click only
sets focus.
## Keybinding
A global keyboard shortcut handled by the window manager before input is
routed to the focused window. It triggers a WM action (spawn a terminal,
focus/restore the next window, close, …) and is never delivered to an app.
## Keyboard layout
Scancode set 1, US QWERTY layout. Hardcoded scancode→ASCII table.
Other layouts can be added later by swapping the translation table.
## Task
A unit of concurrent execution in the kernel: its own kernel stack
(with a guard page below it), scheduler bookkeeping, and no
address-space separation — all tasks share the one identity-mapped
space and the kernel's facilities (TempleOS-style, not a process).
Tasks are preemptively round-robin-scheduled on the 100 Hz PIT tick;
the shell is task 0 and an idle `hlt` task is the always-ready
fallback. A task may sleep, yield, and log to serial/klog; disk access
is serialized by a shared disk controller (a re-entrant lock), and each
terminal is its own program instance. A panicking or
stack-overflowing task dies gracefully (logged, marked done, slot
reaped) without taking the kernel down.
## Development loop
Standalone project — no integration with the RNG initramfs build system.
Testing via QEMU with OVMF. The kernel PE binary is placed on a FAT
image as `\bzImage` (matching rsbl's expected path) and also as
`\EFI\BOOT\BOOTX64.EFI` (what the firmware boots directly).
Two commands, compile + boot:
```sh
./scripts/build.sh # cargo build --workspace --target x86_64-unknown-uefi --release
./scripts/run.sh # stage /tmp/system.img if stale, then boot QEMU
```
(`./scripts/run.sh` alone does both; `--headless`, `--debug`, `--monitor`,
`--fresh` as per its `--help`. `make` / `make run` are thin aliases.)
## Versioning
Workspace version is `a.b.c` (single version for all crates):
- **`a`** — very big overhaul (the kind of change that is 1.0.0 → 2.0.0
in other software). Reserved for fundamental rewrites; a future GUI
era could qualify.
- **`b`** — big feature (multitasking, the shell grammar, klog…).
- **`c`** — fix or small feature.
Bump the workspace version in the root `Cargo.toml` when a change of
that class lands — do not leave the version to drift. Current:
`0.5.4` — the GUI era landed with 0.5.0 (graphics layer + cursor
compositor); 0.5.1 fixed the canvas at 800×600; 0.5.2 added fault
diagnostics (screen + serial dumps for every CPU exception) and
real-hardware hardening (framebuffer-size clamping, fb-region exclusion
from the allocators, authoritative fb geometry logging); 0.5.3 changed
the scroll's full-frame push to per-word volatile stores like the
boot-time `clear()` instead of one 1.92 MB rep-movsb (a REP MOVS to an
uncached real-GPU framebuffer can stall forever — a stall, not an
exception — hence no crash dump); 0.5.4 allocates the shadow
backbuffer from the fb-aware page allocator (it can no longer land on
the framebuffer — the heap free list spans the fb hole, so a 1.9 MB
shadow could alias the display and turn every scroll into an MMIO→MMIO
memcpy) and converts the scroll's shadow memmove to per-word stores as
a belt-and-suspenders measure; 0.5.5 makes every boot stage visible on
real hardware (no serial console): `gop::boot_marker` prints the boot
progress straight to the framebuffer (the last marker before a hang
identifies the death site) and pre-shell panics now draw a white-on-red
`*** PANIC ***` band instead of a silent blank screen, and the shadow
fallback allocation is panic-proof (try_reserve; a heap OOM degrades
to no-shadow mode instead of halting); 0.5.6 makes every fault dump
carry the kernel image base + a stack backtrace (rbp-chain plus a raw
scan of the stack above the interrupted rsp, filtered to the in-memory
.text range parsed from the PE headers — so a real-hardware crash can
be resolved against the PDB offline); 0.5.7 (current) adds a full
register dump to every fault line (rax..rdi, rbx, r8..r15) and an
allocator corruption guard: `dealloc` validates the back-pointer at
`[ptr-8]` (in-heap block, sane size, in-use) and, when the header is
clobbered, prints `ptr`/`block`/heap range + the memory around the
block + the live stack chain, writes the crash band + crash.log, and
halts — instead of the opaque #GP inside `__rust_dealloc` that the
OptiPlex showed. A future major GUI overhaul
(graphics windows beyond the text surface) would be the `a` bump
(1.0.0).
## Project layout
Workspace of four crates, all built for the `x86_64-unknown-uefi` target:
- `rsnix` — the kernel library (module breakdown below)
- `shell` — command loop and built-ins
- `rsdos` — EFI entry point (`efi_main`) and boot orchestration
- `rsdosedit` — kernel-native terminal editor
`rsnix` module breakdown:
- `main.rs` — entry point, panic handler, top-level init
- `uefi.rs` — UEFI protocol type definitions
- `serial.rs` — UART 16550 driver
- `paging.rs` — identity page table helpers
- `memory.rs` — UEFI memory map, free-list heap allocator (grows from page
allocator), bitmap page allocator
- `paging.rs` — identity page-table helpers (map/unmap/query, large-page split)
- `power.rs` — `ResetSystem`-based shutdown, hlt loop fallback
- `rtc.rs` — CMOS/RTC clock (ports 0x70/0x71), BCD/12h normalization
- `pci.rs` — legacy PCI config space (0xCF8/0xCFC), full
enumeration; `config_write`
- 64-bit BAR reader for the AHCI driver
- `ahci.rs` — SATA AHCI host driver (polling, DMA, no interrupts): scans PCI for
class 1/subclass 6/prog-if 1, maps ABAR, starts port 0 (FRE/ST/DMAE), runs
IDENTIFY + READ/WRITE DMA EXT through a 4 KiB DMA buffer
- `fat.rs` — FAT32: BPB parse, whole-FAT cache in RAM with
write-back to both FAT
copies, cluster chains, directory scan with checksum-validated LFN
reassembly, LFN + 8.3 writing, FSInfo free-count maintenance,
mkdir/rm/read/write
- `fs.rs` — VFS: dirs, ramfs files, char devices (`/dev/mem`, `rtc`, `pci`,
`version`, `meminfo`, `null`, `zero`, `console`, `serial`; `/tmp` ramfs;
`/etc/motd`); `/disk` mount node + 24-slot ephemeral node pool so shell
commands work on disk paths unchanged
- `gop.rs` — framebuffer rasteriser (glyph blits, pixel shifts); the text
surface itself lives in `vt`
- `vt.rs` — terminal emulator: cell grid, cursor, ANSI CSI parser (H/J/K,
cursor moves, SGR 16-colour, `?25l/h`), dirty-cell blitting, UTF-8
decoding (box-drawing `│`/`═`, arrows `←↑→↓`, em dash `—` render
procedurally instead of `?`). The inverse cursor is mirrored into the
rendered grid so scrolling / `clear` can never strand stale inverse
cells on the framebuffer.
- `tty.rs` — line discipline: canonical `read_line` (echo, line buffer,
^C/^D) and raw `read_byte` (terminal byte protocol: arrows → `ESC [ A–D`,
^X → `0x18`, …)
- `strbuf.rs` — stack fixed-capacity ASCII string (moved from shell)
- `font.rs` — 8×16 bitmap font data
- `ps2.rs` — PS/2 keyboard driver (polling); generic `Ctrl(char)` and
`CtrlArrow` events
- `shell.rs` — command loop and built-in commands
- `rsdosedit/` — kernel-native port of the RNG `rsedit` terminal editor
(no_std; talks to `tty` raw mode + `vt` ANSI; saves via the VFS)
## Milestone status (v0.1)
✅ **Booting** — kernel loads via rsbl, prints banner via UEFI
console (verified on real hardware)
✅ **ExitBootServices** — memory map captured via static `.bss` buffer
(see `docs/adr/0001-static-buffer-for-memory-map.md`)
✅ **Serial** — UART 16550 at COM1, 115200 8N1
✅ **Framebuffer** — GOP located, 8×16 bitmap font renders text
✅ **Keyboard** — PS/2 polling, scancode set 1, US QWERTY
✅ **Shell** — commands: `help`, `echo`, `clear`, `hello`, `info`,
`mem`, `halt`, `poweroff`
✅ **Heap allocator** — first-fit free-list with splitting +
coalescing, `alloc`/`free`
(boot-time self-test verifies alignment, reuse, and full coalescing)
✅ **Physical pages** — bitmap allocator over conventional regions, contiguous
runs, heap reservation, page-0 trap guard (boot-time self-test included)
✅ **Paging helpers** — identity map/unmap/query, 2 MB/1 GB large-page split,
own tables adopted at boot (boot-time self-test included)
✅ **Heap growth** — heap expands into the page allocator when exhausted
✅ **Shell** — `mmap` dumps the UEFI map, `stress` exercises alloc/free + growth
✅ **Poweroff** — `poweroff`/`exit` shut down via UEFI `ResetSystem`
(ACPI S5; `halt` keeps the plain hlt loop)
✅ **RTC** — CMOS clock read at boot and via `date` (BCD/12h handled)
✅ **PCI** — port-based config access, full bus enumeration via `pci`
✅ **VFS** — “everything is a file”: dirs, ramfs, char devices
(`ls`/`cat`/`hexdump`/`mkdir`/`touch`/`rm`, `echo >`/`>>`; devices
regenerate on read, `/dev/mem` is raw physical RAM)
✅ **^C / ^D** — Ctrl+C cancels the line or aborts a running stream
(`cat`/`hexdump` check between chunks); Ctrl+D is EOF (no poweroff —
removed this session; see session notes below)
✅ **tty + vt (terminal) — the 1:1 UNIX console split**: `tty` owns the
line discipline (canonical echo/buffer/^C/^D for the shell; raw mode
translates keys to the classic terminal byte protocol), `vt` owns the
screen (cell grid + ANSI CSI parser + 16-colour SGR + dirty-cell blits
into the GOP framebuffer). `gop_println!` etc. now render through `vt`.
✅ **rsdosedit** — the RNG `rsedit` editor ported to the kernel (no_std,
no libc): `edit <path>` opens a full-screen editor with undo, syntax
highlighting, search (^W), save (^S), exit (^X, twice to discard), word
nav (Ctrl+arrows), smart indent. Files live in the ramfs. Termios →
`tty` raw mode, stdout → `vt`, `std::fs` → VFS, SIGWINCH → fixed
geometry. Verified in QEMU: edit/type/navigate/insert/save/exit, exact
bytes via `cat`/`hexdump`, syntax colours via framebuffer dump.
✅ **AHCI + FAT32 (persistent disk)** — a SATA AHCI driver (polling DMA,
no interrupts) plus a FAT32 filesystem, mounted under `/disk/<UID>/`;
every shell command works on disk paths: `ls`/`cat`/`hexdump`/
`echo >`/`>>`/`mkdir`/`touch`/`rm`/`edit`. The driver scans **all**
implemented ports (CAP.NP/PI) and starts every port with a device;
`fat::init` mounts **every** valid FAT32 volume it finds, each under
`/disk/<UID>/` where UID = `<LABEL>-<8-hex-serial>` (e.g.
`RSDOS-5BD1E55F`) — sanitized uppercase label + volume serial, unique
per format, stable across reboots, human-recognizable and
machine-parseable (`^[A-Z0-9_]+-[0-9A-F]{8}$`); cloned volumes get a
`-N` suffix. So the kernel finds the disks no matter which AHCI port
the firmware/QEMU put them on (real hardware included), and every
disk is visible at once (verified: `RSDOS-…` + `DATA2-…` mounted
side by side, writes to both, fsck-clean). Whole-FAT cache per
volume with write-back to both FAT copies, checksum-validated LFN
read/write (multi-entry names >13 chars stored last-first on disk —
both the writer and reader were reversed and cancelled out; fixed),
multi-cluster files, space-padded `.`/`..` dot entries (`..` →
cluster 0 under the root), FSInfo free-count maintenance
(fsck-clean after a session, verified with `fsck.fat -n`).
**Files survive reboot** — verified by poweroff → `mdir`/`mtype` on
the image and by a fresh boot seeing the prior session's files.
✅ **Shell grammar (ADR 0002)** — one canonical grammar for every
command: `cmd args [USE src] [TIL marker] [filters…] [output kw dst]`.
Keyword redirection (`WRITING`/`WRITE` overwrite, `TO` append,
`USE`/`USING` input source, `TIL`/`UNTIL` heredoc/terminator;
uppercase-only, aliases valid, fully reserved), C-style quoting
(`"…"` groups, `\x` escapes — quoted text is never an operator),
post-filter stages applied left to right (`-h N` head kills the
upstream, `-t N` tail holds back and flushes at EOF/^C, `-g PAT`
smart-case literal substring, `-p` screenful pager; all repeatable),
`>`/`<`/`<<`/`>>` removed (`>` is plain text), file sinks replace the
screen, `cat`/`hexdump` consume input when path-less (stdin heredoc,
`cat USE a TO b` copy, `USE f TIL m` truncation). ^D is EOF
everywhere and never powers off. `help` rewritten; `test-disk.sh`
and `sendkeys.py` updated to the keyword syntax.
Verified in QEMU (26 grammar checks on a fresh *and* a reused disk,
disk round trip, pager pacing): head kills at N lines, ^C flushes a
`-t` window, smart-case grep, chained/repeated filters, `WRITING`+
`TO`, `USE..TO` copy, `USE..TIL`, heredoc → file, quoting, `-p`.
Two bugs found and fixed during verification: (1) `WRITING` over an
existing file wrote after stale NUL padding — `fs::size()` returns
the node-cached size, which is stale after `set_len(0)`; the sink
now writes at literal offset 0 after truncation; (2) `USE f TIL m`
leaked everything after the marker — the final partial-line emit
ran even after the marker break; fixed with a `stopped` flag.
QEMU setup: `-drive id=disk,if=none,format=raw,file=disk.img` +
`-device ide-hd,drive=disk,bus=ide.0` puts the data disk on AHCI port
0; the ESP gets `bus=ide.1`. (The implicit `-drive file=fat:rw:esp`
otherwise grabs port 0.) Make targets (`run`, `run-gui`, `debug`)
attach `$(DISK_IMG)` (default `/tmp/disk.img`) and auto-create it when
missing. OVMF writes its `NvVars` varstore file onto the first FAT
volume — harmless.
The whole disk workflow is scripted (no manual mkfs/mcopy):
- `./scripts/build.sh` — the single cargo invocation: kernel +
every `/bin` binary (`bins/*.efi`), release profile by default
- `./scripts/mksystem.sh` / `make system` — assemble the dd-able
FAT32 image (`mkfs.fat -C`, label `RSDOS`, seed from `seed/` via
mtools, stage `/bin/cli` + `/bin/gui`); rebuilt only when a
kernel/binary/seed file is newer than it, `--force` wipes runtime
data, `--gpt` wraps it for USB sticks
- `./scripts/run.sh` / `make run` — build + stage + boot QEMU
(`--headless`, `--debug` GDB stub, `--monitor` HMP socket,
`--fresh`, `--mem/--smp/--bios/--image`); the two-command flow is
`./scripts/build.sh && ./scripts/run.sh`
- `make inspect` — `mdir -i` listing of the image (mtools)
- `make test` — all automated round trips; each boots QEMU headless,
drives the shell via `sendkeys.py` through the monitor and asserts
on the serial log
- `./scripts/tests/test-disk.sh` / `make test-disk` — full automated
round trip: build → stage → boot QEMU headless → drive
the shell via `sendkey` through the monitor (`ls /disk`, `mkdir`,
`echo WRITING`, `cat`, `poweroff`) → fsck-clean check → `mtype` verifies
the written file survived poweroff
- `./scripts/tests/sendkeys.py` — types lines into the guest via the QEMU
monitor (`--sock`, US layout, named keys for punctuation; bare
`-`/`.`/`/` are not valid sendkey names)
Shared config and helpers (image path, OVMF, QEMU flags, `build_all`,
`stage_image`, `test_boot`, `wait_log`, `fail`) live in
`scripts/lib.sh` — every script sources it; there is exactly one
`cargo build` and one QEMU invocation shape in the tree.
✅ **`klog` kernel log ring** — a RAM ring of the last 128 serial log
lines (128 chars each, ~16 KB), fed by `serial_println!` (one choke
point — every boot/low-level message captured for free). Sequence
numbers per line + wrap/drop accounting; a dump header reports how
many lines were lost to wrap. Each line carries a `[ssss.mmm]`
timestamp — monotonic ms since boot from `interrupts::uptime_ms()`:
the 100 Hz PIT tick (10 ms) plus interpolation from the PIT's live
1.19 MHz down-counter read twice around the tick load (a jump = a
tick fired between reads) for true sub-ms resolution; `TICKS` widened
to u64 so uptime never wraps. **The CMOS RTC cannot provide
milliseconds** — it is a 1 Hz BCD device (seconds only) — so the
millis come from the PIT. Early-boot lines are `[0000.000]`
(PIT not yet configured; `PIT_READY` gates interpolation), and
timestamps are clamped monotonic in the ring.
Exposed as the `/dev/klog` char
device; the shell `klog` command is a thin alias for `cat
/dev/klog` (same filters/redirection), so `klog -t 5`, `klog -g
fat`, and `klog WRITING /disk/…/boot.log` (persists the boot log)
all work. The ring never allocates (const-constructible static);
only the dump allocates, at shell time. `klog` rejects positional
args (usage) and input keywords ("does not accept input").
**Design consequence: shell output must not feed the ring.** The
plan's "hook `serial_println!`" only works if that macro is
kernel-logging-only — but the shell printed prompts/errors/results
through it, which would have flooded the boot log with echo/cat
output (a `cat /dev/mem` would wrap the ring). The shell now
writes user-facing output (banner, prompt, `term_print`/
`term_println`, pipeline sink, halt/poweroff) directly to the
serial port via `serial::write_str`; tty echo already did. Only
kernel messages (rsnix modules + `rsdos/main.rs`) go through
`serial_println!` now. Verified in QEMU: 28 boot lines captured
(seq 0..27, first `=== rsnix boot ===`, last `boot: entering
shell`), `echo hello` output absent from the ring, `klog WRITING`
persists a boot log (mtype-verified), wrap behavior verified with a
temporary 140-line boot loop (128 kept / 40 dropped, seq 40..167,
oldest live line correct — loop reverted after the test).
✅ **Runtime device detection (hotplug rescan)** — the boot discovery
pipeline is now re-runnable and diffed against live hardware:
- `ahci`: stores ABAR/port bitmap; `poll_dirty()` is a cheap per-port
`PxSSTS.DET == 3` probe (safe on every prompt), `rescan()` starts
newly-present ports (full `start_port` bring-up) and tombstones
slots of removed ones. **Slots never shift** — indices stay stable
so VFS disk handles and `ACTIVE` selection keep working; `select()`
rejects tombstones.
- `fat`: `Volume` gains a `mounted` flag; `rescan()` unmounts volumes
whose disk left and mounts every disk without a mounted volume
(which also re-tries mounts that failed at boot, e.g. a non-FAT
disk replaced by a FAT one). UID dedup ignores tombstones, so a
re-plugged disk keeps its `/disk/<UID>` name.
- `fs`: the init mount loop is factored into `rescan_disks()`, which
lazily creates `/disk`, ensures a mount node per mounted volume and
tombstones nodes of unmounted ones — `cat /disk/<gone>/…` fails
cleanly with "no such file".
- shell: `rescan` command plus an automatic deep rescan before each
prompt whenever `poll_dirty()` sees a change; a one-line summary
reports added/removed counts. All rescan events flow through
`serial_println!` so they land in klog (`ahci: port N up`, `fat:
mounted …`) — the boot log doubles as a device log.
**QEMU testability reality (verified)**: PCIe ADD works end-to-end
(`-device pcie-root-port,id=rp1` at boot, monitor `device_add
e1000e,bus=rp1` → the kernel's `pci` re-scan sees it; q35 root bus
itself rejects hotplug, root ports are required). Disk hotplug is
**not** QEMU-modelable: the ich9-ahci's IDE buses refuse
`device_add ide-hd` ("Bus 'ide.N' does not support hotplugging") and
`drive_add if=ide` ("Can't hot-add drive to type 1"); PCIe
`device_del` is a guest-cooperative protocol (the device stays in
config space until the guest powers off the slot), so removal is
also untestable in QEMU. The disk arrival/removal path (PxSSTS
based) is verified on real hardware instead.
✅ **Preemptive kernel tasks (ADR 0003)** — round-robin multitasking
on the 100 Hz PIT tick, one shared identity-mapped address space:
- `sched`: 16-slot static task table (no ISR-time allocation); task 0
= shell (boot stack), task 1 = idle `hlt` loop, tasks 2+ spawned
via `sched::spawn(entry, name)` with 3 pages each (1 unmapped
guard + 8 KiB stack); preemptive RR with `int $0x60` software
yield; `sleep_ms`/`yield_now`; lazy reaping of Done slots.
- `gdt`: kernel-owned GDT mirroring UEFI's selector layout (no
far-jump; CS/DS/SS keep working) plus a TSS with IST1 (page
fault) and IST2 (double fault) stacks — a stack overflow faults
inside the guard page, the CPU pushes the fault frame onto the
IST stack, and the task dies gracefully instead of triple-
faulting the machine (the ADR's guard-page promise, honored).
- `interrupts`: `isr_dispatch` returns the next frame pointer in
rax; asm does `mov rsp, rax` → pop → iretq. **QEMU quirk that
bit us**: QEMU's 64-bit interrupt entry always pushes
SS/RSP/RFLAGS/CS/RIP (40 bytes) and iretq always pops all five
(its same-privilege branch is never taken for 64-bit iretq), so
interrupt frames are 176 bytes and the synthetic task frames
must carry rsp=stack-top + ss — a 160-byte frame made the first
iretq pop rsp/ss from the unmapped page beyond the frame (task
started with rsp=0, or the iretq itself page-faulted). Real
hardware pushes only 24 bytes at same privilege, so the extra
fields are inert there; the synthetic frames set them
explicitly either way.
- `interrupts`/`sched` fault dumps (0.5.2): every CPU exception
vector 0-31 has an IDT entry (macro-generated stubs; 8/14 keep
their ISTs). Any fault in the shell (task ≤ 1) prints a
white-on-red band via `gop::dump_lines` (raw fb, bypasses vt,
clamped) *and* a parseable serial line — `!!! <kind> vec=N
err=… rip=… cr2=… rsp=… rflags=… task=N "name"` — then halts;
faults in background tasks abort the task instead. Real
hardware has no serial console, so the screen band is how a
crash reports itself; a non-canonical write shows vec=13 (GP,
cr2=0), an unmapped canonical address shows vec=14 with cr2=
the address. `fault` shell command + `make test-fault` verify
the path.
- Panic handling: a task panic (or guard-page overflow) is caught,
logged as `sched: task N "name" panicked: …`, the task is marked
Done, and the scheduler switches away — the shell keeps running.
Shell/idle/ISR/pre-scheduler panics still halt the kernel.
- Shell: `bg <builtin>` (spinner/ticker/counter demo + deliberate
`boom` panic + `overflow` recursion), `jobs` (state/ticks/
panic message), `sleep <ms>` (blocks only the shell), `kill
<id>`. Only the shell touches disk/fs/tty/vt; background tasks
are headless (sleep + serial/klog only).
- Boot order note: `sched::init` must run after the page allocator
and adopted page tables (the idle's stack + guard need both) —
it was originally before them, silently dropping the idle task.
Verified in QEMU (9-step script): demo interleaving at expected
cadences, sleep keeps background tasks logging, jobs states,
graceful `boom` panic with corpse in jobs, kill stops a task, and
`bg overflow` dies on the guard page with the machine intact; full
regression (rescan/til/klog/grammar/pager) re-passes.
✅ **PS/2 mouse** — auxiliary-channel driver on IRQ12, stream mode
(3-byte packets), the GUI's second input device:
- `interrupts`: the keyboard byte ring is generalised to `ByteRing`;
IRQ12 (vector 44) gets its own ring and stub. Both PS/2 handlers
route each data-port byte by the controller status register's aux
bit (bit 5), so keyboard and mouse never steal each other's bytes
even when both are pending at once. PIC unmask now covers IRQ2
(cascade) and IRQ12 (slave bit 4).
- `ps2`: `mouse_init()` enables the aux port, sets config bits
IRQ12+aux-clock (preserving the keyboard bit), then `set defaults`
(100 Hz / 4 counts/mm / 1:1) + `enable reporting` via the 0xD4
channel — each waits for the device ACK with a bounded timeout so
a machine without a mouse boots normally (log: `ps2: mouse not
present`). `try_poll_mouse()` reassembles packets in main
context from the raw byte ring; the packet-start marker (bit 3)
resyncs the stream after any lost byte. Events are **screen-
relative** (`dx` + = right, `dy` + = down — the PS/2 Y-up wire
convention is negated), buttons decoded; 9-bit sign extension via
the packet sign bits.
- `fs`: `/dev/mouse` is now a **position device** — `cat /dev/mouse`
prints the cursor's current state (`x=N y=N buttons=---`), since
the cursor task owns the PS/2 ring; `mouse: no PS/2 mouse`
otherwise.
- shell: `mouse` **watches the on-screen cursor** (x/y/buttons,
printed only on change, 10 ms poll) until ^C; `mouse <id>` also
routes every decoded event to that task's queue (`cursor still
tracks`, route cleared on exit). `cursor` prints a one-shot state
line (the parseable test surface).
Verified in QEMU (`make test-mouse`, headless, via the monitor's
`mouse_move`/`mouse_button` HMP commands): the cursor starts centred,
`mouse_move 20 7` moves it +20/+7 (asserted against `cursor` output),
`mouse_button 1` → `buttons=L--`, a held-button move (`-5,-3`)
tracks position *and* keeps the button, release clears, `/dev/mouse`
reports the same position snapshot, the live `mouse` watch prints the
changed line and ^C stops it, and `mouse 3` routes a real move into
the msgrecv task's queue. `make test-disk` regression re-passes.
✅ **Scheduler message queues (the GUI-era primitive from ADR 0003)** —
every task gets a static 16-slot ring of `TaskMsg` (4 KB in .bss,
no ISR-time allocation):
- `TaskMsg` = `Key(KeyEvent)` | `Mouse(MouseEvent)` | `Timer` |
`User(u64)` — input uses the same decoded types the PS/2 drivers
produce, so window input routing is literally "post what the
console would have gotten"; `User(u64)` is the generic escape
hatch. `KeyEvent`/`MouseEvent` derive Copy for ring storage.
- `sched::send(id, msg)` — posts from any context (task, shell, ISR)
under the scheduler lock; wakes a Blocked task straight to Ready;
errors cleanly on a gone task or a full queue (input is lossy
under extreme load by design). `recv()` pops without blocking;
`recv_blocking()` marks the task Blocked and yields. **Lost-wakeup
safety net**: the tick scan also wakes Blocked tasks whose queue is
non-empty, so a task that blocked *after* its message arrived (or
whose poster never woke it) is picked up within one 10 ms tick.
- shell: `msgrecv` spawns a task that blocks on recv and logs every
message to serial (`msgrecv[2]: Key(F3)` …); `send <id> <type>
[args]` posts `user <n>` / `timer` / `mouse <dx> <dy> <btn>`
(1=L 2=R 4=M) / `key <name>` (enter esc tab space bksp arrows
home end pgup pgdn del ctrl-c f1..f12, or a single char); `jobs`
now shows `q=` (pending message count).
- **Real-input routing, proven**: `mouse <id>` (angle brackets) streams
decoded IRQ12 events into that task's queue instead of the console
— the exact loop the window manager will run. `msgrecv` + `mouse 2`
- `mouse_move 20 7` showed `msgrecv[2]: Mouse(+20,+7 …)` in QEMU.
Verified in QEMU (`make test-msg`): all six message types delivered
in order (user, timer, key enter/ctrl-c/f3, synthesized mouse),
real routed mouse moves (including a held-button move), the 17th
send against a 16-cap queue reports `queue full` (tested against
the shell's never-consumed queue), and `jobs` shows q=0 + task back
to `blocked` after draining. `make test-disk` regression re-passes.
✅ **Graphics layer + cursor compositor** — the first visible GUI
surface: real rect primitives, a shadow→display push path, and an
on-screen mouse cursor that never races the text surface:
- `gop`: `fill_rect` + `push_rect(x,y,w,h)` — the **single display
write path**. Every raster op rasterises into the RAM shadow
first, then `push_rect` copies that rect to the display (write-
only; the display is still never *read*) and composites the
cursor on top. `blit_glyph` and scroll shifts now route through
it, so text redraws can never erase or corrupt the cursor. The
cursor lives *only* in the overlay — never in the shadow — which
makes save/restore impossible to get wrong: erasing = push the
old rect (the shadow simply doesn't contain the cursor there).
- cursor bitmap: 12×16 pointer built by a const fn from explicit
column lists (hand-mangled bit literals were the one real bug
this round — twice), drawn white with a black offset outline
(+1,+1). Position lives in one `AtomicU64` (x:high 32, y:low 32)
so a relaxed load never sees a torn pair — *and* the unpack must
mask the low word (a bare `as usize` cast is a no-op on 64-bit
and returned the whole packed value).
- `cursor` task (spawned at boot when the mouse is present, task 2):
polls the PS/2 ring every 10 ms, moves the cursor via
`cursor_move` (clamped to the screen; erase-old + draw-new are
both shadow pushes), tracks buttons, and forwards every event to
a task's queue when `mouse <id>` armed it. The cursor is drawn
once at spawn (the vt diff never re-blits idle cells) and stays
visible because every display write composites it; scrolls
re-overlay it after the bulk shift.
- shell: `gfx` draws a status-bar demo scene (navy strip + RGB
boxes); `gfx check` is an in-kernel pixel self-test — draws a red
rect, reads it back from the shadow *and* the display, asserts
the cursor tip is white on the display but absent from the shadow
(overlay purity), and prints PASS/FAIL per stage.
- **Canvas resolution fixed at 800×600** — `gop::init` scans the GOP
mode list and calls `SetMode` for an exact 800×600 (BGR/RGB 32-bit;
QEMU std-vga lists it as mode 3, real GPUs list it as a standard
mode), falling back to the smallest mode that fits, then to the
firmware's choice. Everything derives from it: the vt text grid,
the cursor clamp, the demo strip. `SetMode` may move the info
pointer and framebuffer base, so they are re-read after the switch.
Verified in QEMU (`make test-gfx`): `fill_rect/shadow: OK`,
`display push: OK`, `cursor overlay: OK`, `cursor not in shadow:
OK`, `gfx check: PASS`; the demo draws; the cursor tracks injected
mouse input (test-mouse) with the overlay intact. `make test-disk`
regression re-passes.
## Known issues / perf notes
- **Scroll cost** — scrolling still shifts the whole framebuffer (a
~3-8 MB pixel copy per row), but two mechanisms make it fast on real
hardware:
1. a RAM backbuffer (`gop::SHADOW`): every raster write goes to the
shadow too; scrolling is a cached shadow memcpy + one write-only
push to the display — the display is never *read* (uncached reads
were the killer, since real-hardware GOP framebuffers are
uncacheable/write-combining). **The push is per-word volatile
stores (0.5.3), not one bulk rep-movsb**: a 1.92 MB REP MOVS to an
uncached real-GPU framebuffer wedged the CPU on some hardware (a
stall, not an exception — no dump fired), while the boot-time
`clear()` — the same per-word pattern — always worked. QEMU's
RAM-backed framebuffer never showed it. **The shadow itself is
allocated from the fb-aware page allocator (0.5.4)** so it can
never land on the framebuffer: the heap's free list spans the fb
hole, so a 1.9 MB heap shadow could alias the display, making
every scroll an MMIO→MMIO memcpy that wedges real hardware. The
scroll's shadow memmove is also per-word stores (0.5.4) as a
belt-and-suspenders guard against any future fb-aliased shadow.
2. deferred scroll batching (`vt::PENDING_SCROLLS`): the logical grid
scrolls instantly in RAM while the framebuffer shift+push is
deferred; N pending scrolls cost one push. Bursts of streaming
output (hexdump/cat) push every ~16 rows (or after ~40 ms of
pause), and the shell syncs before each prompt. Any CSI sequence
(full-screen apps like the editor) or interactive input pushes
immediately.
Still on the someday list: a pan-based ring so scrolling never touches
the whole framebuffer at all.
- **Editor redraw** re-blits changed cells per keystroke (fine at 1280×800);
the "optimize drawing someday" item.
- **AHCI quirks learned (QEMU ground truth)**: the command header layout is
FIS length in DW0 bits 4:0 and PRDTL in bits 31:16 (not the classic
CFL-at-24 bit-fields); the H2D FIS sector count lives at bytes 12-13 with
the device register at byte 7 (byte 10 is LBA[47:40] — no separate
device byte); QEMU rejects H2D FISes with flags bits 6:4 set, so the
write direction is signalled purely by the header W bit (1<<6).
Real hardware follows the same FIS layout; the header layout matches
Linux libahci.
- **ramfs only for `/`**: the root filesystem is still the ramfs (lost on
reboot); persistent files live under `/disk` (AHCI/FAT32).
## Pending decisions (agreed, not yet implemented)
- **Disk recognition scope — keep simple for now**: no pre-EBS UEFI
block-device census, no GPT/MBR partition parsing, no NVMe driver.
(Revisit later: Linux-style recognition = PCI scan → ahci/nvme →
partitions → superblock autodetect → by-label/by-uuid naming; our
`/disk/<LABEL>-<SERIAL>` already mirrors the by-label/by-uuid idea.)
## Target audience
Learning exercise and toy. Not a Linux replacement. Not POSIX-compatible.
---
## Session notes — shell experience redesign (implemented)
Working from `shell_experience`. Plan as stated:
- Commands accept **post-filters** as trailing flags: `-h N` (head),
`-t N` (tail), `-g PAT` (grep). Flag placement (at the end of the
command) is respected.
- Redirection uses **keywords**, not `> << < >>`:
`WRITING`/`WRITE` → overwrite, `UNTIL`/`TIL` → heredoc,
`USE`/`USING` → input, `TO` → append.
- **No pipes.** `head`/`tail`/`grep` needs are covered by the flags;
a pager flag is a maybe.
Resolved this session:
- **Post-filter order = written order, left to right.** Each flag is a
filter stage applied to the command's output as a line stream:
`cat file -h 50 -t 10 -g pat` = "lines containing `pat` among the last
10 lines of the first 50 lines". (The `shell_experience` prose for
that example was loose; the left-to-right rule is canonical.)
- **`-t N` holds back.** The tail stage buffers a rolling window of the
last N lines and emits nothing until its upstream ends — EOF, or the
user's ^C, which flushes the window to the screen. `cat /dev/mem -g
100 -t 10` therefore shows nothing while it runs and prints the last 10
matches when ^C ends it.
- **`-h N` truncates.** Once N lines have passed through, the head stage
kills the upstream command — `cat /dev/mem -h 5` peeks 5 lines and
stops instead of flooding.
- **`-g PAT` = smart-case literal substring.** Match is case-insensitive
unless PAT contains an uppercase letter, in which case it is
case-sensitive (editor-style smart case). No regex, no wildcards.
- **Post-filters repeat.** `-h`, `-t` and `-g` may each appear more than
once; every instance is a separate stage in the left-to-right chain
(so `-h 10 -h 5` = first 5, `-t 10 -t 5` = last 5, `-g foo -g bar` =
lines containing both). After `-g`, the next token is the pattern
verbatim (`-g -h` = pattern "-h").
- **Redirection keywords pair strictly.** At most one input keyword and
one output keyword per command line (both may appear together); the
input keyword must precede the output keyword. `USING f TO g` is
accepted, `TO g USING f` is a syntax error, and so is any second
keyword of the same kind (`TO a WRITE b`).
- **Trailing-region grammar (canonical order):**
`cmd args [USE src] [TIL marker] [filters…] [output kw dst]` —
input keywords first, then filter flags, then the output keyword.
Rationale: "we feed the command input, parse it with filters, put
that into output". `USE` must precede `TIL`; filters come after the
input keywords and before the output keyword.
- **`TIL` is a terminator, not a source.** Unlike `USE` (which names an
input source), `TIL <marker>` ends the input at a marker, so it may
combine with `USE` — two "input" keywords are allowed, in the fixed
order `USE` then `TIL`.
- **`TIL` truncates the source at the marker.** The marker line is not
part of the input and everything from the marker on is discarded —
`USE f TIL m` reads only the text of `f` before `m`. Marker match is
an exact full-line match.
- **^D is EOF everywhere; ^D/^C no longer power off.** Inside a `TIL`
input block, ^D ends the input early exactly like a file's EOF.
Shutting down via ^D or ^C is removed: at the prompt, ^D on an empty
line is a no-op (fresh prompt); ^C keeps its line-cancel /
stream-abort role; `poweroff`/`exit` remain the only shutdown verbs.
- **Input consumers: `cat` and `hexdump` only, and only when path-less.**
With no path (or with an input keyword), they read the input stream
(`cat USE a TO b` copies a→b; `cat TIL END` is the classic heredoc).
All other commands reject input keywords with a syntax error — a
command declares whether it takes input. `USE`/`TIL` are general
building blocks; more consumers will come later.
- **`cat <path> TIL <marker>` is sugar for `cat USE <path> TIL
<marker>`** (refined this session): the paths are the input source,
truncated at the first marker line across the combined stream —
`cat a b TIL m` reads `a` then `b` and stops at the first `m` line.
The single-source reading is unambiguous, so path args + `TIL` is
valid; path args + `USE` stays a syntax error (two input sources).
- **Keywords: uppercase-only, aliases valid.** `WRITE`==`WRITING`,
`TIL`==`UNTIL`, `USE`==`USING`; lowercase forms are ordinary text, so
`echo use the force` can never redirect. Keywords are reserved
operators — a keyword is always an operator, never an argument
(collision escape: quoting, a future building block).
- **`-p` = pause flag (pager), more-style.** A stage that pauses at
every screenful of output and waits for a keypress; engages only when
output exceeds one screenful, no args, no scrollback/search. As a
stage it chains with the others — `-h N -p -g pat -p` lets the user
see intermediate steps of the filtering.
- **`> >> < <<` are gone.** Keywords are the *only* redirection syntax;
`>` is now ordinary text (`echo a > b` prints `a > b`). Collateral:
`scripts/tests/test-disk.sh` and the `help` text must be rewritten to the
keyword syntax; the milestone bullets above describing `echo >`/`>>`
record the old behavior (superseded by these notes).
- **Quoting (C): `"..."` groups + literalizes; `\x` escapes one char.**
Quoted/escaped text is always literal — never an operator, never
split. So `echo foo "WRITING" bar` prints `foo WRITING bar`; to
print literal quote characters: `echo foo \"WRITING\" bar`. Also
enables spaces in filenames and multi-word `-g` patterns
(`-g "a b"`). Quote chars are delimiters, not content.
Assumed edge semantics (flag if wrong):
- Filtered output goes to both terminal (GOP) and serial, identically.
A file sink (WRITING/TO) replaces terminal output entirely — screen
is silent, as `echo >` behaved before.
- `-p` pauses terminal output only; with a file sink it is a no-op.
- `-h N` / `-t N` require N ≥ 1; otherwise syntax error.
- `cat p USE s` (path args *and* an explicit source keyword) is a
syntax error — the input source is ambiguous. `cat USE s`, path-less
`cat TIL m`, and `cat <path> TIL m` (sugar for `USE <path> TIL m`)
are the valid input forms.
- ADR 0002 (`docs/adr/0002-keyword-redirection-and-post-filters.md`)
captures the keyword-redirection + post-filter decision; the ADR is
the authoritative statement, these notes are the working detail.
Open questions (to resolve this session):
1. ~~Order of application of `-h`/`-t`/`-g`~~ — **resolved**: left to
right.
2. ~~`-h`/`-t` streaming semantics~~ — **resolved**: `-t` hold-back +
^C flush; `-h` truncates the upstream.
3. ~~`-h`/`-t` semantics on infinite streams (e.g. `cat /dev/mem`)~~ —
**resolved**: folded into 2.
4. ~~`-g` pattern semantics (literal substring vs regex; case
sensitivity)~~ — **resolved**: smart-case literal substring;
repeatable stages.
5. ~~Keyword details~~ — **resolved**: uppercase-only, both aliases
valid, keywords fully reserved; quoting (also resolved this
session) is the escape mechanism.
6. ~~Pager flag: in or out~~ — **resolved**: `-p` pause flag, in.
7. ~~Which commands consume input~~ — **resolved**: `cat` + `hexdump`,
path-less only; other commands reject input keywords.
8. ~~Fate of the existing `>` / `>>` redirection~~ — **resolved**: full
removal; `>` is plain text; scripts/help to be rewritten.
## Implementation (this session — done, verified in QEMU)
Everything above is now implemented in `shell/src/lib.rs`:
- **Tokenizer** — `"…"` groups text (delimiter, not content), `\x`
escapes one char, a token containing any quoted/escaped char is
flagged literal and can never be an operator or flag.
- **Parser** — canonical grammar for every command; keywords
uppercase-only with aliases, fully reserved (quoted forms are
literal data); strict ordering enforced (USE before TIL, input
before filters, output keyword last) with syntax errors otherwise;
`-h`/`-t` need N ≥ 1, `-g` takes the next token verbatim; unknown
flags before the trailing region are ordinary arguments.
- **Filter pipeline** — stages over a line stream, left to right,
each repeatable: `Head` kills the upstream (producers check the
`line()` return), `Tail` holds a rolling window and flushes through
the remaining stages at end-of-stream (EOF, ^C, or head kill),
`Grep` is smart-case literal substring, `Pause` is a terminal-only
screenful pager (^C aborts; dropped entirely with a file sink).
- **Sinks** — terminal (GOP + serial, identical) or file; a file sink
replaces the screen, opens lazily on first output, `WRITING`
truncates and writes from offset 0, `TO` appends at current size.
Diagnostics (errors, usage) always go to the terminal, never into
the pipeline or a file.
- **Input consumers** — `cat`/`hexdump`: no keyword = keyboard until
^D, `USE f` = file, `TIL m` = keyboard heredoc, `USE f TIL m` =
file truncated at the exact marker line (marker and everything
after discarded), and `cat <path> TIL m` / `hexdump <path> TIL m` =
the paths as source, truncated at the first marker across the
combined stream. Other commands reject input keywords.
- **Ordering enforcement** — `USE`/`TIL` must precede the filters and
the output keyword; filters must precede the output keyword;
distinct messages for each violation ("must come before the
filters" / "must come before the output keyword").
- **^D** — EOF everywhere, at the prompt it is a no-op (fresh
prompt); ^C keeps line-cancel / stream-abort; `poweroff`/`exit`
are the only shutdown verbs.
- **Collateral** — `help` text rewritten; `scripts/tests/test-disk.sh` and
the `sendkeys.py` docstring updated to keyword syntax.
Bugs found by QEMU verification and fixed:
1. `WRITING` over an existing file left stale NUL padding —
`fs::size()` returns the node-cached size, stale after
`set_len(0)`; the overwrite path now writes at literal offset 0
(the pre-redesign echo wrote at 0 too).
2. `cat USE f TIL m` leaked everything after the marker — the
final-partial-line emit ran even after the marker break; fixed
with a `stopped` flag in `cat_file_until`/`hexdump_file_until`.
3. `cat file TO out TIL m` blamed the filters — the `trailing` flag
was set by both filters and the output keyword, so the "must come
before the filters" error fired for the output keyword too; the
parser now distinguishes them (`TO dst TIL m` → "TIL must come
before the output keyword"). Also: filters after the output
keyword (`TO dst -h 2`) were silently accepted — now a syntax
error.
4. `cat file TIL m` was rejected as ambiguous — path args + `TIL` is
now sugar for `USE <path> TIL <marker>` (single source, truncated
at the marker); path args + `USE` remains a syntax error.
Verified: 26 grammar checks pass on a fresh disk *and* on a reused
one (rewriting existing files), `make test-disk` round trip is
fsck-clean, the `-p` pager pauses at screenfuls and resumes on
keypress, `-h` stops `/dev/mem` at exactly N lines, and ^C flushes a
`-t` window on an infinite stream.
Follow-up verification (this session): 10 checks pass in QEMU for the
`cat <path> TIL <marker>` sugar — truncation at the marker, combined
multi-path stream truncated at the first marker, filters after the
sugar (`-h 1`), the heredoc round trip, all four ordering-error
messages, and the `USE`-with-paths ambiguity message; fsck clean.
|