fpga-archive-demo-box1-20261110 · EAGLE4 FPGA archive

Source reports/results/DEMO_FAST_MODEL_SWITCH_20261110.md · source md5 957a79473adef172f7854a1fc9f64326 · archived 2026-11-18

Download PDFOpen PDFRaw markdown

Demo model switch (Llama-3.2-3B ⇄ Qwen3.8-27B): where 2–2.5 min goes, and how to get it to seconds (2026-11-10)

Program: box1 demo, 4C1 drafter (ATT E2b 0xE4B713E2 on 01/42, MLPV 0xE4B71342 on 61/62) behind the H200 target, demo :8099 → FPGA endpoint :30020. Read-only on the live demo, engines and cards. No process stopped, no file removed, no engine restarted, no card opened. All new code is in NEW files, tested offline on the c4_mock world. Tags: [M] measured, with the file or command · [D] derived from [M] numbers · [P] predicted · [U] unknown, with the test that settles it.

0. Answer first

  1. Card side, 48 s (Qwen) / 36 s (Llama) [M]: about 60 % of it is host Python, not the cards.

    • 17.3 s re-hashes every pinned file: 14.3 GB at 0.83 GB/s, with the same 0.79–1.53 GB files hashed 2–4 times.
    • 5.7 s hashes the same payloads again inside the replay.
    • 5.3 s parses 250 MB of hex weight text.
    • About 17 s is real card I/O, done one card after another: 12.1 s of Python MMIO weight writes plus 5 s of H2C.
    • The design doc predicted 2–4 s for this step (QWEN27B_DRAFTER_20261020/DESIGN_DUAL_MODEL_4C1.md:149 [D]).
  2. Both models' weights cannot stay in URAM on today's images [M]. The ATT bank is 86.7 % full and the MLPV bank 62.0 %, so two models need 173 % and 124 %.

    • HBM can hold both models' tables easily: ≤ 4.8 GB of 32 GiB per card.
    • The RTL already has a boot DMA that loads HBM → URAM weight bank (weight_dma_loader, is_scale = 0).
    • So "a register flip plus a golden" needs an RTL rebuild. "An HBM → URAM reload plus a golden" (≈ 0.5 s of card I/O [P]) needs host changes only.
  3. Engine side, 50 s (Llama) / ~85 s (Qwen) [M], mostly process start-up (≈ 19 s) and the 27B weight load (52 s).

    • Both FPGA engines fit on the H200 at once: 117.9 GiB of 140.4 GiB with the two always-warm GPU engines, if the KV pools are capped (--max-total-tokens 8192) [D].
    • Today's FPGA engine holds 59.5 GiB, of which 44.8 GiB is a 379 K-token KV pool. Batch 1 at ctx ≤ 1,280 needs 0.14 GiB.
  4. The 19:49Z Qwen switch FAILED at engine start [M] (rc 137, "Raise --mem-fraction-static above 0.616"). It was restored to Llama at 19:53Z by demo_llama_d11_restore.sh.

    • The 0.55 fraction applies to the memory free at start. A GPU+EAGLE engine that warmed up at the same time took 18.1 GB of it during the 40 s weight load.
    • This is the failure that resident engines remove.
  5. Fastest safe plan (Option 3, §5): ≈ 10 s [P]. It has three parts:

    • resident engines with lazy card attach (c4_standby.py);
    • the same switch with the hashing cached, the stream parse cached and the 4 cards booted in parallel (c4_fastswitch.py);
    • an unchanged golden.

    With a C MMIO writer (Option 4) it is ≈ 5 s. With HBM-resident tables and M9 weight walks (Option 5) it is ≈ 3 s. Every gate runs as today.

1. Card side: the 48 s, by step

Sources:

Step of c4.py --model qwen27b switch Qwen s Llama s What it is Source
discover + rd_model_cfg (before) < 1 < 1 identity reads toggle.log, same second [M]
re-hash every pinned file (c4.py:546 _all_manifest_files_ok, 4 cards) 17.3 6.0 Qwen 14.31 GB pinned (per ATT rank 3.19 GB, per MLPV rank 3.96 GB: 1.53 GB host-only target embed + 2 × 0.79 GB draft embed + 0.79 GB lm-head + streams) at 0.83 GB/s host_timing B [M]
md5 inside the replay (c4_ops.py:82,129: every H2C / file BURST) 5.7 5.65 3.9 GB of the same payloads again host_timing D [M]
hex weight-stream parse (c4_bringup.py:632 _stream_table) 5.3 5.3 250 MB of lane addr word text → 13.5 M words host_timing C [M]
load_manifest × 2 per card 0.4 0.01 op-stream md5s, rope RAM emulation, e_norm check host_timing A [M]
MMIO weight / scale / gamma / GLM writes (boot.ops) 12.1 13.1 3.64 M words per ATT, 2.99 M per MLPV, one Python ctypes store each (~0.93 µs) [c4_ops] boot.ops secs 3.418+3.357+2.668+2.640 [M]
H2C embed / lm-head (payload.ops) 3.6 4.2 6 × 788 MB at ≈ 1.3 GB/s payload secs − D [D]
G8 lm-head duplicate + 8 MiB spot read-back (2 MLPV) ≈ 1.4 ≈ 1.4 2 × 788 MB (c4_bringup.py:599) [D] at the H2C rate
model ops (rope RAM + SubLN, qwen27b.ops) 0.02 — 5,645 writes [c4_ops] qwen27b.ops secs 0.012 ×2 [M]
MODEL_CFG writes + read-backs, fold-off, KVI apply, d2t < 0.1 < 0.1 ~20 MMIO + 0.03 s d2t [M]
residual (python start, imports, open 4 BARs, identity recheck) ≈ 2.1 ≈ 0.4 48 − sum [D]
arm + golden + read-back (after) < 1 < 1 record-identity golden 9/9 toggle.log 19:49:52Z, all three [M]

2. Can both models stay resident on the cards?

Sources:

Memory, per card Used by one model Capacity Two models Fits?
ATT weight bank (URAM, 512 URAM = 512 lanes × 8,192 × 32 b) 7,104 words/lane (3,637,248 words) 8,192 14,208 no (173 %)
ATT scale bank (BRAM) 888 /lane 2,048 1,776 yes
MLPV weight bank (URAM, 256 lanes × 16,384) 10,160 /lane 16,384 20,320 no (124 %)
MLPV scale bank 1,270 /lane 2,048 2,540 no
MLPV GLM lmdown (BRAM, 448 RAMB36) 393,216 words one model 2× no
rope RAM, gamma / SubLN slots one table each one 2× no (reload: 0.012 s)
HBM: embed 788 MB (ATT, MLPV), lm-head 788 MB + 1 GiB-offset duplicate (MLPV), weight + scale walk blobs ~15 MB ATT ≈ 0.8 GB, MLPV ≈ 2.4 GB 32 GiB ATT ≈ 1.6 GB, MLPV ≈ 4.8 GB yes

3. Engine side: the 50–85 s

Sources:

Phase Qwen3.8-27B s Llama-3.2-3B s
python + sglang import + args 7 8
scheduler spawn, imports, model config 10 11
target weights 52.1 (51.05 GB) 5.8 (6.06 GB)
draft weights 2.1 (2.80 GB) 2.9 (2.18 GB)
KV / mamba pools < 1 < 1
CUDA graphs 2.7 (verify) 7.1 (piecewise prefill 6.2 + verify 0.9)
warm-up (first FPGA tree: card process start, KVI, draft graph capture; + demo warm request) ≈ 9 (to 85) 4.3 ([stall44] took 4320.4 ms) + ≈ 4
total ≈ 85 50

4. Can both endpoints stay resident on the H200?

Now (nvidia-smi 20:01Z: 98,524 / 143,771 MiB used; re-read 20:19Z, copy DEMO_FAST_MODEL_SWITCH_20261110/nvidia_smi_1519.txt md5 5ebe413e: 98,568 MiB, the same three processes).

KV needed at batch 1, ctx ≤ 1,280 [D from the logs' own K sizes]:

Resident set GiB How
GPU-only Llama (other agent, always warm) 20.0 [M] unchanged
GPU+EAGLE Llama (other agent, always warm) 18.1 [M] unchanged
FPGA Llama endpoint, capped 14.5 [D] 2.27 context + 6.06 + 2.18 weights + 0.97 KV@8192 + 1.52 graphs + ~1.5 activations [U]
FPGA Qwen endpoint, capped 65.3 [D] 2.27 + 51.05 + 2.80 + 6.02 mamba + 0.59 KV@8192 + 0.60 graphs + ~2 activations [U]
total 117.9 of 140.4 22.5 GiB margin

Launch rules [D, SGLang 0.5.16]:

Lazy attach: can demo_server_4c1.py start without the cards? Partly.

5. Options and predicted switch time

All options keep the gates:

# What changes Card s Engine s Switch [P] Needs
0 today [M] 48 (Qwen) / 36 85 / 50 ~2–2.5 min —
1 c4_fastswitch F1+F2+F3 only ≈ 6–7 85 / 50 ~1–1.5 min host only (done, mock-proven)
2 resident engines only 48 / 36 ≈ 1–2 (attach + one tree) ~50 s host only (c4_standby done; resident server wrapper + active router to write)
3 resident engines + F1+F2+F3 ≈ 6–7 ≈ 1–2 + detach 0.5 ≈ 9–10 s host only
4 3 + C MMIO burst writer (like native/c4h.so) ≈ 3 ≈ 2 ≈ 5 s host C; per-write rate [U]
5 4 + both models' tables in HBM + M9 weight walks ≈ 1 ≈ 2 ≈ 3 s host only + an xsim gate + the NoC [U] (§2)
6 both models in URAM (register flip) ≈ 0.5 ≈ 2 ≈ 2.5 s RTL + 2 rebuilds; not recommended

Card-side details for Option 3 [D]:

Engine side: detach, then attach of a once-warmed engine.

6. Prototype (NEW files, offline, mock-tested)

File md5 What
scripts/4c1/c4_fastswitch.py a0a121bc F1 md5 memo by fingerprint (realpath, dev, inode, size, mtime_ns, ctime_ns) taken before and after the read. Only "settled" files are cached (mtime/ctime > 2 s old: the racy-timestamp rule). The persistent index is trusted only if owned by this euid and 0600/0700. F2 stream-table binary cache (blobs md5-checked on load). F3 one worker per card (fork on silicon, thread in the mock). prepare (no card, no lock), switch (= c4.py switch argv), selftest
scripts/4c1/c4_standby.py 477f50e1 lazy card attach / detach (§4)
scripts/4c1/sglang_site_standby/sitecustomize.py f4bd5c2b stock 4C1 sitecustomize + c4_standby.arm() when EAGLE4_4C1_STANDBY_CTL is set
scripts/4c1/demo/fastswitch_host_timing.py 84225066 offline timing of the host parts of a switch (§1)
scripts/4c1/tests/test_c4_fastswitch.py 0857a746 F3 fork/thread parallel (4 × 0.5 s → < 1.2 s, 4 pids), RED worker failure → rc 6 + text, F1 settle rule: 4/4 OK
scripts/4c1/tests/test_c4_standby.py 84761c30 detached → no build + lock free; attach → flock held + owner line; RED second engine refused; detach waits for the in-flight call, closes the driver once, frees the lock; hand-over; dead holder frees the lock; arm() wraps: 3/3 OK

Measurements:

Not done here (the live files are untouched):

7. The resident stack, built as COPIES (2026-11-10 ~20:45Z; coordinator go). Nothing is deployed.

Live files are unchanged: demo_server.py fe00155b, model_toggle.py 442b0d2b. No engine, port, card or lock was touched.

File (NEW) md5 Role
scripts/4c1/demo/demo_server_4c1_resident.py a4cd7b8c demo_server_4c1.py imported, not copied. Its engine and request path are unchanged. Differences: no FPGA warm-up at start (refuses EAGLE4_FA4_WARM=1); /health + resident/loaded/attach, warm true only while attached + warmed; POST /admin/attach {"run"} (lab lock via the agent, then the stall44 FA4 warm on the first attach + the demo warm prompt, through the unchanged c4_sglang.build gates); POST /admin/detach; /admin/shutdown detaches first; any generate while detached is refused before it reaches the engine
scripts/4c1/demo/resident_ctl.py acf32cdc the HTTP half: want.json / state.json nonce protocol with c4_standby. Detach = warm off → one router period → the generate lock HELD through the agent's detach
scripts/4c1/demo/router_active.py 304a17f3 :30020 front. Active = the ONE backend that is attached + warm (two → none). /health = the active engine's JSON verbatim + router. Everything else is forwarded (SSE relayed). /admin/shutdown → 403 always (the classic toggle's restart path fails closed); /admin/router-stop stops the router only
reports/arch_site/model_toggle_fast.py f9102ba2 COPY of model_toggle.py (same API and routes). Job: target loaded? → identity precheck → c4_fastswitch prepare (root, no lock) → detach the attached engine → the classic card steps with c4.py switch → c4_fastswitch.py switch → attach the target → router active. A refusal before any card write re-attaches the previous engine. free_gpu is never called
scripts/4c1/demo/demo_resident.sh d486c979 derive / start <m> / start-both (Qwen, then Llama, one at a time) / router / stop / status
scripts/4c1/demo/fast_toggle_cli.py d8e3af88 runs the SAME model_toggle_fast job from a shell (deploy, the silicon A/B)
scripts/4c1/demo/fastswitch_deploy.sh / fastswitch_rollback.sh b8b097b1 / fc5c5bc0 gated by FASTSWITCH_DEPLOY_GO=window-over / FASTSWITCH_ROLLBACK_GO=yes (both refuse otherwise, rc 2)
scripts/4c1/demo/fastswitch_demo_import.py 35b8b6ab the ONE live line the deploy changes: import model_toggle ⇄ import model_toggle_fast as model_toggle (exact line, create-only backup, parse check)
scripts/4c1/demo/mock_resident_endpoint.py 3d291981 test stand-in: the real resident_ctl + a real c4_standby agent (real flock on a temp lab) + a fake engine
scripts/4c1/c4_fastswitch.py f63ebc7a + mockswitch (mock world, fast do_switch, an rd_model_cfg-shaped readback)

demo_resident.sh start rules:

Offline gates (all green on the final files):

Silicon test, after the coordinator says the demo window is over:

  1. FASTSWITCH_DEPLOY_GO=window-over bash scripts/4c1/demo/fastswitch_deploy.sh (serves Llama at the end).
  2. fast_toggle_cli.py switch qwen27b (golden record identity + attach), then fast_toggle_cli.py switch llama3b. Record per-step seconds from the CLI output.
  3. If needed: FASTSWITCH_ROLLBACK_GO=yes bash scripts/4c1/demo/fastswitch_rollback.sh.

[U] measured only then: