docs: note robots.txt in the web layout and checks

1 files changed, 3 insertions, 1 deletions

diff --git a/web/README.md b/web/README.md
index 3e81468..fecb710 100644
--- a/web/README.md
+++ b/web/README.md
@@ -13,6 +13,7 @@ web/
   releases.php   shared scanner and hash cache
   install.sh     stable Linux installer
   install.ps1    stable Windows installer
+  robots.txt     disallows every crawler
   play/          browser app, served at /play/
   bin/gui/       apps, served at /bin/gui/...
   bin/client/    terminal clients
@@ -151,7 +152,8 @@ Then open <http://127.0.0.1:8899/>.
 ```
 
 `deploy.sh` rsyncs `web/` with `--delay-updates`, so a visitor never sees a
-half-written binary, then fetches the live page and every artifact it lists —
+half-written binary, then checks `robots.txt`, fetches the live page and every
+artifact it lists —
 GUI apps, installers, terminal clients and servers — and compares the
 downloaded bytes with the local SHA-256. It also fetches `/play/`, `app.js` and
 `style.css`. `--verify-only` skips the upload and only checks the live site.