path: root/CONTEXT.md

# CONTEXT.md — raisernics Kernel Glossary

## raisernics

Custom x86_64 kernel for the RNG ecosystem.  Toy project — self-contained,
no strict userspace separation, TempleOS-inspired design philosophy.
Built in Rust, compiled for the `x86_64-unknown-uefi` target (stable).

## Boot interface

The kernel is a **PE32+ EFI application** loaded via UEFI
`LoadImage`/`StartImage`.
The existing bootloader (rsbl) loads us from `\bzImage` and passes the UEFI
system table.  No bootloader changes needed.

## Output strategy

Two-phase console model:

- **Phase 1 (boot services alive):** UEFI `con_out->OutputString()` — works
  with whatever UEFI firmware routes (serial + framebuffer).
- **Phase 2 (native):** Raw serial UART 16550 driver on COM1 (I/O port
  `0x3F8`).  Activated after `ExitBootServices` when we own the hardware.

## Dependency philosophy

Aligns with RNG philosophy: minimal dependencies.  Freestanding Rust,
no libc, no external crates at the kernel/core level.  UEFI protocol
structs are hand-defined (not from `uefi-rs` crate) to stay lightweight.

## Paging strategy

Long mode mandates paging (`CR0.PG = 1`), so we accept that tax but keep it
simple.  Identity-mapped 4-level tables, no higher-half, no demand paging,
no guard pages.  `paging.rs` provides identity map/unmap/query helpers.  At
boot the kernel runs on UEFI's tables; `adopt_tables()` rebuilds that map
into the kernel's own tables and switches CR3 — needed because UEFI maps its
page-table pages read-only, so the firmware tree can't be modified in place.
Firmware large pages (2 MB / 1 GB) are split into 4 KB tables on demand;
page-table pages come from the physical page allocator.

## Physical memory

Before `ExitBootServices`, the kernel reads the UEFI memory map to know
which physical regions are free.  The heap is a first-fit **free-list
allocator** with splitting and coalescing: blocks carry a 16-byte header
(size + free flag), free blocks are linked in address order, and an 8-byte
back-pointer before each payload lets `free` find its block even when
alignment left a gap.  When the free list is exhausted, the heap **grows**
by taking pages from the page allocator (proportional to the request),
keeping the two allocators cooperating.  A **bitmap page allocator** manages
physical pages: one bit per 4 KB page spanning the conventional regions, so
MMIO holes and firmware-reserved gaps are never handed out, and the heap
region itself is reserved to keep the allocators from double-allocating the
same RAM.  Contiguous runs with power-of-two alignment are supported
(`alloc_pages_aligned`).

## Input strategy

**PS/2 keyboard, interrupt-driven.**  IRQ1 pushes scancodes into a ring
buffer (`interrupts::KEYBUF`); the shell polls it via `read_scancode()`.
Port I/O to `0x60`/`0x64`.  Scancode set 1 (US QWERTY layout in the
initial build).

## Display strategy

Two-phase model matching output:

- **Phase 1:** UEFI `con_out->OutputString()` (framebuffer + serial handled
  by firmware).
- **Phase 2:** GOP (Graphics Output Protocol) framebuffer obtained before
  `ExitBootServices`, driven by a built-in 8×16 bitmap font.  Text rendered
  by drawing glyphs directly into the framebuffer.

The framebuffer address stays valid after `ExitBootServices` — the GPU's
linear frame buffer is system memory that we own once boot services are gone.

## Keyboard layout

Scancode set 1, US QWERTY layout.  Hardcoded scancode→ASCII table.
Other layouts can be added later by swapping the translation table.

## Development loop

Standalone project — no integration with the RNG initramfs build system.
Testing via QEMU with OVMF.  The kernel PE binary is placed on a FAT
image as `\bzImage` (matching rsbl's expected path).

## Project layout

Workspace of four crates, all built for the `x86_64-unknown-uefi` target:

- `rsnix` — the kernel library (module breakdown below)
- `shell` — command loop and built-ins
- `rsdos` — EFI entry point (`efi_main`) and boot orchestration
- `rsdosedit` — kernel-native terminal editor

`rsnix` module breakdown:

- `main.rs` — entry point, panic handler, top-level init
- `uefi.rs` — UEFI protocol type definitions
- `serial.rs` — UART 16550 driver
- `paging.rs` — identity page table helpers
- `memory.rs` — UEFI memory map, free-list heap allocator (grows from page
  allocator), bitmap page allocator
- `paging.rs` — identity page-table helpers (map/unmap/query, large-page split)
- `power.rs` — `ResetSystem`-based shutdown, hlt loop fallback
- `rtc.rs` — CMOS/RTC clock (ports 0x70/0x71), BCD/12h normalization
- `pci.rs` — legacy PCI config space (0xCF8/0xCFC), full
  enumeration; `config_write`
  - 64-bit BAR reader for the AHCI driver
- `ahci.rs` — SATA AHCI host driver (polling, DMA, no interrupts): scans PCI for
  class 1/subclass 6/prog-if 1, maps ABAR, starts port 0 (FRE/ST/DMAE), runs
  IDENTIFY + READ/WRITE DMA EXT through a 4 KiB DMA buffer
- `fat.rs` — FAT32: BPB parse, whole-FAT cache in RAM with
  write-back to both FAT
  copies, cluster chains, directory scan with checksum-validated LFN
  reassembly, LFN + 8.3 writing, FSInfo free-count maintenance,
  mkdir/rm/read/write
- `fs.rs` — VFS: dirs, ramfs files, char devices (`/dev/mem`, `rtc`, `pci`,
  `version`, `meminfo`, `null`, `zero`, `console`, `serial`; `/tmp` ramfs;
  `/etc/motd`); `/disk` mount node + 24-slot ephemeral node pool so shell
  commands work on disk paths unchanged
- `gop.rs` — framebuffer rasteriser (glyph blits, pixel shifts); the text
  surface itself lives in `vt`
- `vt.rs` — terminal emulator: cell grid, cursor, ANSI CSI parser (H/J/K,
  cursor moves, SGR 16-colour, `?25l/h`), dirty-cell blitting, UTF-8
  decoding (box-drawing `│`/`═`, arrows `←↑→↓`, em dash `—` render
  procedurally instead of `?`).  The inverse cursor is mirrored into the
  rendered grid so scrolling / `clear` can never strand stale inverse
  cells on the framebuffer.
- `tty.rs` — line discipline: canonical `read_line` (echo, line buffer,
  ^C/^D) and raw `read_byte` (terminal byte protocol: arrows → `ESC [ A–D`,
  ^X → `0x18`, …)
- `strbuf.rs` — stack fixed-capacity ASCII string (moved from shell)
- `font.rs` — 8×16 bitmap font data
- `ps2.rs` — PS/2 keyboard driver (polling); generic `Ctrl(char)` and
  `CtrlArrow` events
- `shell.rs` — command loop and built-in commands
- `rsdosedit/` — kernel-native port of the RNG `rsedit` terminal editor
  (no_std; talks to `tty` raw mode + `vt` ANSI; saves via the VFS)

## Milestone status (v0.1)

✅ **Booting** — kernel loads via rsbl, prints banner via UEFI
  console (verified on real hardware)
✅ **ExitBootServices** — memory map captured via static `.bss` buffer
  (see `docs/adr/0001-static-buffer-for-memory-map.md`)
✅ **Serial** — UART 16550 at COM1, 115200 8N1
✅ **Framebuffer** — GOP located, 8×16 bitmap font renders text
✅ **Keyboard** — PS/2 polling, scancode set 1, US QWERTY
✅ **Shell** — commands: `help`, `echo`, `clear`, `hello`, `info`,
  `mem`, `halt`, `poweroff`
✅ **Heap allocator** — first-fit free-list with splitting +
  coalescing, `alloc`/`free`
  (boot-time self-test verifies alignment, reuse, and full coalescing)
✅ **Physical pages** — bitmap allocator over conventional regions, contiguous
  runs, heap reservation, page-0 trap guard (boot-time self-test included)
✅ **Paging helpers** — identity map/unmap/query, 2 MB/1 GB large-page split,
  own tables adopted at boot (boot-time self-test included)
✅ **Heap growth** — heap expands into the page allocator when exhausted
✅ **Shell** — `mmap` dumps the UEFI map, `stress` exercises alloc/free + growth
✅ **Poweroff** — `poweroff`/`exit` shut down via UEFI `ResetSystem`
  (ACPI S5; `halt` keeps the plain hlt loop)
✅ **RTC** — CMOS clock read at boot and via `date` (BCD/12h handled)
✅ **PCI** — port-based config access, full bus enumeration via `pci`
✅ **VFS** — “everything is a file”: dirs, ramfs, char devices
  (`ls`/`cat`/`hexdump`/`mkdir`/`touch`/`rm`, `echo >`/`>>`; devices
  regenerate on read, `/dev/mem` is raw physical RAM)
✅ **^C / ^D** — Ctrl+C cancels the line or aborts a running stream
  (`cat`/`hexdump` check between chunks), Ctrl+D on an empty line
  powers off (EOF)
✅ **tty + vt (terminal) — the 1:1 UNIX console split**: `tty` owns the
  line discipline (canonical echo/buffer/^C/^D for the shell; raw mode
  translates keys to the classic terminal byte protocol), `vt` owns the
  screen (cell grid + ANSI CSI parser + 16-colour SGR + dirty-cell blits
  into the GOP framebuffer).  `gop_println!` etc. now render through `vt`.
✅ **rsdosedit** — the RNG `rsedit` editor ported to the kernel (no_std,
  no libc): `edit <path>` opens a full-screen editor with undo, syntax
  highlighting, search (^W), save (^S), exit (^X, twice to discard), word
  nav (Ctrl+arrows), smart indent.  Files live in the ramfs.  Termios →
  `tty` raw mode, stdout → `vt`, `std::fs` → VFS, SIGWINCH → fixed
  geometry.  Verified in QEMU: edit/type/navigate/insert/save/exit, exact
  bytes via `cat`/`hexdump`, syntax colours via framebuffer dump.
✅ **AHCI + FAT32 (persistent disk)** — a SATA AHCI driver (polling DMA,
  no interrupts) plus a FAT32 filesystem, mounted under `/disk/<UID>/`;
  every shell command works on disk paths: `ls`/`cat`/`hexdump`/
  `echo >`/`>>`/`mkdir`/`touch`/`rm`/`edit`.  The driver scans **all**
  implemented ports (CAP.NP/PI) and starts every port with a device;
  `fat::init` mounts **every** valid FAT32 volume it finds, each under
  `/disk/<UID>/` where UID = `<LABEL>-<8-hex-serial>` (e.g.
  `RSDOS-5BD1E55F`) — sanitized uppercase label + volume serial, unique
  per format, stable across reboots, human-recognizable and
  machine-parseable (`^[A-Z0-9_]+-[0-9A-F]{8}$`); cloned volumes get a
  `-N` suffix.  So the kernel finds the disks no matter which AHCI port
  the firmware/QEMU put them on (real hardware included), and every
  disk is visible at once (verified: `RSDOS-…` + `DATA2-…` mounted
  side by side, writes to both, fsck-clean).  Whole-FAT cache per
  volume with write-back to both FAT copies, checksum-validated LFN
  read/write (multi-entry names >13 chars stored last-first on disk —
  both the writer and reader were reversed and cancelled out; fixed),
  multi-cluster files, space-padded `.`/`..` dot entries (`..` →
  cluster 0 under the root), FSInfo free-count maintenance
  (fsck-clean after a session, verified with `fsck.fat -n`).
  **Files survive reboot** — verified by poweroff → `mdir`/`mtype` on
  the image and by a fresh boot seeing the prior session's files.

  QEMU setup: `-drive id=disk,if=none,format=raw,file=disk.img` +
  `-device ide-hd,drive=disk,bus=ide.0` puts the data disk on AHCI port
  0; the ESP gets `bus=ide.1`.  (The implicit `-drive file=fat:rw:esp`
  otherwise grabs port 0.)  Make targets (`run`, `run-gui`, `debug`)
  attach `$(DISK_IMG)` (default `/tmp/disk.img`) and auto-create it when
  missing.  OVMF writes its `NvVars` varstore file onto the first FAT
  volume — harmless.

  The whole disk workflow is scripted (no manual mkfs/mcopy):

- `make disk` / `./scripts/mkdisk.sh` — create the FAT32 image with
    `mkfs.fat -C` (the flag that actually creates the file; plain
    `mkfs.fat` needs an existing file), label `RSDOS`, seed from
    `seed/` via mtools; `--fresh` re-formats, `--size 128M` sizes it
- `make inspect` — `mdir -i` listing of the image (mtools)
- `make test-disk` / `./scripts/test-disk.sh` — full automated round
    trip: build → deploy → ensure image → boot QEMU headless → drive
    the shell via `sendkey` through the monitor (`ls /disk`, `mkdir`,
    `echo >`, `cat`, `poweroff`) → fsck-clean check → `mtype` verifies
    the written file survived poweroff
- `./scripts/sendkeys.py` — types lines into the guest via the QEMU
    monitor (`--sock`, US layout, named keys for punctuation; bare
    `-`/`.`/`/` are not valid sendkey names)

## Known issues / perf notes

- **Scroll cost** — scrolling still shifts the whole framebuffer (a
  ~3-8 MB pixel copy per row), but two mechanisms make it fast on real
  hardware:
  1. a RAM backbuffer (`gop::SHADOW`): every raster write goes to the
     shadow too; scrolling is a cached shadow memcpy + one write-only
     push to the display — the display is never *read* (uncached reads
     were the killer, since real-hardware GOP framebuffers are
     uncacheable/write-combining).
  2. deferred scroll batching (`vt::PENDING_SCROLLS`): the logical grid
     scrolls instantly in RAM while the framebuffer shift+push is
     deferred; N pending scrolls cost one push.  Bursts of streaming
     output (hexdump/cat) push every ~16 rows (or after ~40 ms of
     pause), and the shell syncs before each prompt.  Any CSI sequence
     (full-screen apps like the editor) or interactive input pushes
     immediately.
  Still on the someday list: a pan-based ring so scrolling never touches
  the whole framebuffer at all.
- **Editor redraw** re-blits changed cells per keystroke (fine at 1280×800);
  the "optimize drawing someday" item.
- **AHCI quirks learned (QEMU ground truth)**: the command header layout is
  FIS length in DW0 bits 4:0 and PRDTL in bits 31:16 (not the classic
  CFL-at-24 bit-fields); the H2D FIS sector count lives at bytes 12-13 with
  the device register at byte 7 (byte 10 is LBA[47:40] — no separate
  device byte); QEMU rejects H2D FISes with flags bits 6:4 set, so the
  write direction is signalled purely by the header W bit (1<<6).
  Real hardware follows the same FIS layout; the header layout matches
  Linux libahci.
- **ramfs only for `/`**: the root filesystem is still the ramfs (lost on
  reboot); persistent files live under `/disk` (AHCI/FAT32).

## Pending decisions (agreed, not yet implemented)

- **`klog` kernel log ring**: capture all boot/low-level messages by
  appending to a RAM ring buffer inside `serial_println!` (one choke
  point, every existing message captured for free).  Ring: 128 lines ×
  128 chars ≈ 16 KB, sequence numbers, wrap/drop accounting.
- **`klog` command** = thin alias for `cat /dev/klog` plus filters
  `-h` (head) and `-t` (tail); no log levels, no timestamps, nothing
  fancier.  `/dev/klog` is a char device so `klog > /disk/…/boot.log`
  persists a boot log via the existing redirection.
- **Disk recognition scope — keep simple for now**: no pre-EBS UEFI
  block-device census, no GPT/MBR partition parsing, no NVMe driver.
  (Revisit later: Linux-style recognition = PCI scan → ahci/nvme →
  partitions → superblock autodetect → by-label/by-uuid naming; our
  `/disk/<LABEL>-<SERIAL>` already mirrors the by-label/by-uuid idea.)

## Target audience

Learning exercise and toy.  Not a Linux replacement.  Not POSIX-compatible.