fpga-archive-demo-box1-20261110 · EAGLE4 FPGA archive
Demo model switch (Llama-3.2-3B ⇄ Qwen3.8-27B): where 2–2.5 min goes, and how to get it to seconds (2026-11-10)
Program: box1 demo, 4C1 drafter (ATT E2b 0xE4B713E2 on 01/42, MLPV 0xE4B71342 on 61/62) behind the H200 target, demo :8099 → FPGA endpoint :30020.
Read-only on the live demo, engines and cards. No process stopped, no file removed, no engine restarted, no card opened. All new code is in NEW files, tested offline on the c4_mock world.
Tags: [M] measured, with the file or command · [D] derived from [M] numbers · [P] predicted · [U] unknown, with the test that settles it.
0. Answer first
-
Card side, 48 s (Qwen) / 36 s (Llama) [M]: about 60 % of it is host Python, not the cards.
- 17.3 s re-hashes every pinned file: 14.3 GB at 0.83 GB/s, with the same 0.79–1.53 GB files hashed 2–4 times.
- 5.7 s hashes the same payloads again inside the replay.
- 5.3 s parses 250 MB of hex weight text.
- About 17 s is real card I/O, done one card after another: 12.1 s of Python MMIO weight writes plus 5 s of H2C.
- The design doc predicted 2–4 s for this step (
QWEN27B_DRAFTER_20261020/DESIGN_DUAL_MODEL_4C1.md:149[D]).
-
Both models' weights cannot stay in URAM on today's images [M]. The ATT bank is 86.7 % full and the MLPV bank 62.0 %, so two models need 173 % and 124 %.
- HBM can hold both models' tables easily: ≤ 4.8 GB of 32 GiB per card.
- The RTL already has a boot DMA that loads HBM → URAM weight bank (
weight_dma_loader, is_scale = 0). - So "a register flip plus a golden" needs an RTL rebuild. "An HBM → URAM reload plus a golden" (≈ 0.5 s of card I/O [P]) needs host changes only.
-
Engine side, 50 s (Llama) / ~85 s (Qwen) [M], mostly process start-up (≈ 19 s) and the 27B weight load (52 s).
- Both FPGA engines fit on the H200 at once: 117.9 GiB of 140.4 GiB with the two always-warm GPU engines, if the KV pools are capped (
--max-total-tokens 8192) [D]. - Today's FPGA engine holds 59.5 GiB, of which 44.8 GiB is a 379 K-token KV pool. Batch 1 at ctx ≤ 1,280 needs 0.14 GiB.
- Both FPGA engines fit on the H200 at once: 117.9 GiB of 140.4 GiB with the two always-warm GPU engines, if the KV pools are capped (
-
The 19:49Z Qwen switch FAILED at engine start [M] (
rc 137, "Raise --mem-fraction-static above 0.616"). It was restored to Llama at 19:53Z bydemo_llama_d11_restore.sh.- The 0.55 fraction applies to the memory free at start. A GPU+EAGLE engine that warmed up at the same time took 18.1 GB of it during the 40 s weight load.
- This is the failure that resident engines remove.
-
Fastest safe plan (Option 3, §5): ≈ 10 s [P]. It has three parts:
- resident engines with lazy card attach (
c4_standby.py); - the same switch with the hashing cached, the stream parse cached and the 4 cards booted in parallel (
c4_fastswitch.py); - an unchanged golden.
With a C MMIO writer (Option 4) it is ≈ 5 s. With HBM-resident tables and M9 weight walks (Option 5) it is ≈ 3 s. Every gate runs as today.
- resident engines with lazy card attach (
1. Card side: the 48 s, by step
Sources:
- the switch logs
/mnt/data/eagle4_scratch/4c1_demo_toggle/qwen27b_20261110_144902/toggle.log(steps 19:49:04Z → 19:49:52Z), itsswitch.json, and the Llama restore…/4c1_runs/box1_e2b_demo_llama_back_20261110_145351/restore.log(switch 19:53:51Z → 19:54:27Z = 36 s); - the host-only parts re-run OFFLINE on the same rank dirs by
scripts/4c1/demo/fastswitch_host_timing.py→DEMO_FAST_MODEL_SWITCH_20261110/host_timing_qwen27b.json(md50a529982) andhost_timing_llama3b.json(49bd46b8).
Step of c4.py --model qwen27b switch |
Qwen s | Llama s | What it is | Source |
|---|---|---|---|---|
discover + rd_model_cfg (before) |
< 1 | < 1 | identity reads | toggle.log, same second [M] |
re-hash every pinned file (c4.py:546 _all_manifest_files_ok, 4 cards) |
17.3 | 6.0 | Qwen 14.31 GB pinned (per ATT rank 3.19 GB, per MLPV rank 3.96 GB: 1.53 GB host-only target embed + 2 × 0.79 GB draft embed + 0.79 GB lm-head + streams) at 0.83 GB/s | host_timing B [M] |
md5 inside the replay (c4_ops.py:82,129: every H2C / file BURST) |
5.7 | 5.65 | 3.9 GB of the same payloads again | host_timing D [M] |
hex weight-stream parse (c4_bringup.py:632 _stream_table) |
5.3 | 5.3 | 250 MB of lane addr word text → 13.5 M words |
host_timing C [M] |
load_manifest × 2 per card |
0.4 | 0.01 | op-stream md5s, rope RAM emulation, e_norm check | host_timing A [M] |
MMIO weight / scale / gamma / GLM writes (boot.ops) |
12.1 | 13.1 | 3.64 M words per ATT, 2.99 M per MLPV, one Python ctypes store each (~0.93 µs) | [c4_ops] boot.ops secs 3.418+3.357+2.668+2.640 [M] |
H2C embed / lm-head (payload.ops) |
3.6 | 4.2 | 6 × 788 MB at ≈ 1.3 GB/s | payload secs − D [D] |
| G8 lm-head duplicate + 8 MiB spot read-back (2 MLPV) | ≈ 1.4 | ≈ 1.4 | 2 × 788 MB (c4_bringup.py:599) |
[D] at the H2C rate |
model ops (rope RAM + SubLN, qwen27b.ops) |
0.02 | — | 5,645 writes | [c4_ops] qwen27b.ops secs 0.012 ×2 [M] |
| MODEL_CFG writes + read-backs, fold-off, KVI apply, d2t | < 0.1 | < 0.1 | ~20 MMIO + 0.03 s d2t | [M] |
| residual (python start, imports, open 4 BARs, identity recheck) | ≈ 2.1 | ≈ 0.4 | 48 − sum [D] | |
| arm + golden + read-back (after) | < 1 | < 1 | record-identity golden 9/9 | toggle.log 19:49:52Z, all three [M] |
- KV / state clear is not a switch cost. The switch does not clear KV: arm (< 1 s) sets the rings and PERSIST, and the next request's
KVRES_CLRsession boundary resets the KV. - The cards are booted sequentially (
c4_bringup.py:566-596, oneforloop). Per card, the I/O alone is ATT ≈ 4.0 s and MLPV ≈ 4.6 s [D].
2. Can both models stay resident on the cards?
Sources:
- capacity from each rank's
MANIFEST.jsonplan(~/eagle4_artifacts/4c1_boot_qwen27b/v3_ropestep256_enormk/0xE4B713E2/rank0,…/0xE4B71342/rank0); - URAM from the E2b placed util report
/mnt/data/eagle4_scratch/drafter_speed_20261107/evidence/E4B713E2_d9L512_E2b_hubleft_nobal/util/place_report_utilization_0.rpt(md5e2d9efa0):100(URAM 744 / 1,925 = 38.65 %) and:409(per SLR 232 / 171 / 341).
| Memory, per card | Used by one model | Capacity | Two models | Fits? |
|---|---|---|---|---|
| ATT weight bank (URAM, 512 URAM = 512 lanes × 8,192 × 32 b) | 7,104 words/lane (3,637,248 words) | 8,192 | 14,208 | no (173 %) |
| ATT scale bank (BRAM) | 888 /lane | 2,048 | 1,776 | yes |
| MLPV weight bank (URAM, 256 lanes × 16,384) | 10,160 /lane | 16,384 | 20,320 | no (124 %) |
| MLPV scale bank | 1,270 /lane | 2,048 | 2,540 | no |
| MLPV GLM lmdown (BRAM, 448 RAMB36) | 393,216 words | one model | 2× | no |
| rope RAM, gamma / SubLN slots | one table each | one | 2× | no (reload: 0.012 s) |
| HBM: embed 788 MB (ATT, MLPV), lm-head 788 MB + 1 GiB-offset duplicate (MLPV), weight + scale walk blobs ~15 MB | ATT ≈ 0.8 GB, MLPV ≈ 2.4 GB | 32 GiB | ATT ≈ 1.6 GB, MLPV ≈ 4.8 GB | yes |
-
Register flip (both models in URAM) = RTL + rebuild of both images. It needs four changes:
- the ATT bank to 16,384 deep (+512 URAM → 1,256 / 1,925; SLR2 already holds 341 of 624, so the per-die fit is [U] until placed);
- the MLPV bank to 24,576 deep (+256 URAM), or dropping G9's zero-filled HI half (an older doc says it is zero,
FOURCARD_DESIGN_20261016/4C1_HOST_WEIGHTS.md:23; [U] on these checkpoints); - a MODEL bank-select offset in the weight, scale and GLM read address generators;
- 2× scale, GLM and rope/gamma storage, or a reload of them.
It would save < 0.5 s over Option 5, so it is not recommended for a demo switch.
-
The host-only route already exists in the RTL. In the E2b fork (
verilog/forks/lutgemm_att_d9_e0_20261104,control_regs.svmd5b7284843:300-330),0x54 bit1 = 0selects the WEIGHT bank for the M9 boot DMA (boot_dma_is_scale // 0=weight (URAM),:334). Ineagle4_drafter.sv(md519782866), the path isu_dma_loader(:5499-5520) →dma_we_wt = dma_we & ~dma_tgt_emb & ~dma_tgt_hfc(:5615) → the same bank-write mux as the M1c MMIO trio (eff_boot_*,:5636-5645).- The kit loads scales this way today: 12
W 54 3+POLL 74per ATT rank inboot.ops. The weights still go through MMIO (4C1_HOST_WEIGHTS.md:33). - So both models' weights can be staged in HBM once per boot, and a switch becomes 6 M9 walks per ATT card + 2 per MLPV card. That is ~15 MB of on-card DMA, milliseconds [P]. The embed base is a host register (
W 40/44inpayload.ops). - [U] before Option 5:
- an xsim gate of M9 weight walks at L512 / LUTG, i.e.
tb_4c1_wloadwith M9 weight walks in place of the M1c lines (GREEN identical bank + RED swapped rank); - whether G8's 4 channels reach an lm-head base other than
0x4180000000(NoC aperture). If not, keep re-copying the lm-head: 1.2 s per MLPV, in parallel.
- an xsim gate of M9 weight walks at L512 / LUTG, i.e.
- The kit loads scales this way today: 12
3. Engine side: the 50–85 s
Sources:
- Qwen
reports/results/FOURCARD_DESIGN_20261016/demo_runs/demo_serve_fpga_qwen27b_w2d7_20261110_135119.log(start 18:51:20Z = firstclk.csvrow; WARM at log line 181, between 18:52:40Z and 18:53:01Z; the toggle's own step reading is 85 s); - Llama
…/demo_serve_fpga_llama3b_w2d11_20261110_135413.log(restore.log: "warm after 50 s").
| Phase | Qwen3.8-27B s | Llama-3.2-3B s |
|---|---|---|
| python + sglang import + args | 7 | 8 |
| scheduler spawn, imports, model config | 10 | 11 |
| target weights | 52.1 (51.05 GB) | 5.8 (6.06 GB) |
| draft weights | 2.1 (2.80 GB) | 2.9 (2.18 GB) |
| KV / mamba pools | < 1 | < 1 |
| CUDA graphs | 2.7 (verify) | 7.1 (piecewise prefill 6.2 + verify 0.9) |
| warm-up (first FPGA tree: card process start, KVI, draft graph capture; + demo warm request) | ≈ 9 (to 85) | 4.3 ([stall44] took 4320.4 ms) + ≈ 4 |
| total | ≈ 85 | 50 |
4. Can both endpoints stay resident on the H200?
Now (nvidia-smi 20:01Z: 98,524 / 143,771 MiB used; re-read 20:19Z, copy DEMO_FAST_MODEL_SWITCH_20261110/nvidia_smi_1519.txt md5 5ebe413e: 98,568 MiB, the same three processes).
- 20,492 MiB: GPU-only Llama
:30017(--mem-fraction-static 0.30). - 18,490 MiB: GPU+EAGLE
gpuspec_chat_server:30019. - 59,504 MiB: FPGA Llama
:30020(mem-frac 0.50 → KV pool 379,161 tokens = 40.5 GB + 4.3 GB draft KV; serve log 14:55:07).
KV needed at batch 1, ctx ≤ 1,280 [D from the logs' own K sizes]:
- Llama: 114,688 B/token (29.44 GiB / 551,276) → 0.14 GiB.
- Qwen: 65,534 B/token (2 × 8.68 GiB / 284,445) → 0.08 GiB, plus a fixed 6.02 GiB mamba / SSM cache.
- A cap of
--max-total-tokens 8192(6× headroom) costs 0.88 / 0.50 GiB.
| Resident set | GiB | How |
|---|---|---|
| GPU-only Llama (other agent, always warm) | 20.0 [M] | unchanged |
| GPU+EAGLE Llama (other agent, always warm) | 18.1 [M] | unchanged |
| FPGA Llama endpoint, capped | 14.5 [D] | 2.27 context + 6.06 + 2.18 weights + 0.97 KV@8192 + 1.52 graphs + ~1.5 activations [U] |
| FPGA Qwen endpoint, capped | 65.3 [D] | 2.27 + 51.05 + 2.80 + 6.02 mamba + 0.59 KV@8192 + 0.60 graphs + ~2 activations [U] |
| total | 117.9 of 140.4 | 22.5 GiB margin |
Launch rules [D, SGLang 0.5.16]:
--mem-fraction-staticis a fraction of the memory FREE at engine start (kv_cache_configurator.py:1527-1570).--max-total-tokenscaps the KV pool (:1592-1607). The rest of the "static" budget is not allocated.- The mamba start-up reserve is 512 + 1.5 × max(chunked_prefill, 2048) MB (
server_args.py:4135-4150): 12.6 GiB at the default chunk of 8192, 3.7 GiB at 2048. - So:
- Qwen:
--mem-fraction-static 0.85 --max-total-tokens 8192 --chunked-prefill-size 2048; - Llama FPGA:
--max-total-tokens 8192; - start them one after the other, never while another engine is warming (the 19:49Z failure, §0.4).
- Qwen:
- [U] the activation peaks. Settle them by starting each capped engine once in a window with the demo free.
Lazy attach: can demo_server_4c1.py start without the cards? Partly.
- The driver is already built lazily, on the first draft call (
cost_draft_tree_fpga_eaglefit.py:483-490→c4_sglang.build). - But today the endpoint runs inside
card_run.sh, which holds the lab lock for the whole process life. Its start-up warm-up (demo_server_4c1.py:361-362, stall44EAGLE4_FA4_WARM=1) runs an FPGA tree, which opens the cards. - Nothing in the build path checks the lock.
scripts/4c1/c4_standby.py(NEW) adds the missing gate in the scheduler process, armed byscripts/4c1/sglang_site_standby/sitecustomize.py(runs the stock 4C1 sitecustomize first):- detached (the start state): no lock and no card. The factory and the route body raise
Detached. - attach: takes
/sanctuary/lab/card.lockitself, non-blocking, writing the sameCARD_OWNERandcard_runs.loglines ascard_run.shwith ownerbox1-4c1-demo, somodel_toggle.lock_statereads it. It then setsEAGLE4_4C1_RUN. The unchangedc4_sglang.buildgates run on the next request. - detach: waits for the in-flight draft call, then
C4RemoteDrafter.close()(the card process's own_close, no signal). It dropscdt._fpga_tree_drvand releases the lock. - If the holder dies, the kernel releases the flock.
- detached (the start state): no lock and no card. The factory and the route body raise
- The resident engine starts with
EAGLE4_FA4_WARM=0, so it opens no card at start. It is warmed by one request right after its first attach (4.3 s once per engine life; later attaches reuse the draft CUDA graphs). - The front router on
:30020forwards to the one attached + warm backend, sodemo_server.pykeeps its URL. [U] the card-process start time inside an attach: measure with one attach on silicon.
5. Options and predicted switch time
All options keep the gates:
- discover identity + flash record;
rd_model_cfgdual check;do_switch's checks: MODEL_CFG canary, drained, manifests tagged, every pin compared with a digest of the file's present bytes, 0x1C8 = 0, word count, G8 spot read-back, MODEL_STATUS applied;- arm;
- record-identity golden;
- post read-back;
open_session's identity, arm and golden gates.
| # | What changes | Card s | Engine s | Switch [P] | Needs |
|---|---|---|---|---|---|
| 0 | today [M] | 48 (Qwen) / 36 | 85 / 50 | ~2–2.5 min | — |
| 1 | c4_fastswitch F1+F2+F3 only |
≈ 6–7 | 85 / 50 | ~1–1.5 min | host only (done, mock-proven) |
| 2 | resident engines only | 48 / 36 | ≈ 1–2 (attach + one tree) | ~50 s | host only (c4_standby done; resident server wrapper + active router to write) |
| 3 | resident engines + F1+F2+F3 | ≈ 6–7 | ≈ 1–2 + detach 0.5 | ≈ 9–10 s | host only |
| 4 | 3 + C MMIO burst writer (like native/c4h.so) |
≈ 3 | ≈ 2 | ≈ 5 s | host C; per-write rate [U] |
| 5 | 4 + both models' tables in HBM + M9 weight walks | ≈ 1 | ≈ 2 | ≈ 3 s | host only + an xsim gate + the NoC [U] (§2) |
| 6 | both models in URAM (register flip) | ≈ 0.5 | ≈ 2 | ≈ 2.5 s | RTL + 2 rebuilds; not recommended |
Card-side details for Option 3 [D]:
- host work is 0.43 s with a warm cache (measured below);
- the 4 cards boot in parallel: max(ATT 3.4 MMIO + 0.6 H2C, MLPV 2.7 + 1.2 + 0.7 dup) = 4.6 s;
- the ≈ 2 s residual stays;
- discover, arm and golden take < 1 s each.
Engine side: detach, then attach of a once-warmed engine.
6. Prototype (NEW files, offline, mock-tested)
| File | md5 | What |
|---|---|---|
scripts/4c1/c4_fastswitch.py |
a0a121bc |
F1 md5 memo by fingerprint (realpath, dev, inode, size, mtime_ns, ctime_ns) taken before and after the read. Only "settled" files are cached (mtime/ctime > 2 s old: the racy-timestamp rule). The persistent index is trusted only if owned by this euid and 0600/0700. F2 stream-table binary cache (blobs md5-checked on load). F3 one worker per card (fork on silicon, thread in the mock). prepare (no card, no lock), switch (= c4.py switch argv), selftest |
scripts/4c1/c4_standby.py |
477f50e1 |
lazy card attach / detach (§4) |
scripts/4c1/sglang_site_standby/sitecustomize.py |
f4bd5c2b |
stock 4C1 sitecustomize + c4_standby.arm() when EAGLE4_4C1_STANDBY_CTL is set |
scripts/4c1/demo/fastswitch_host_timing.py |
84225066 |
offline timing of the host parts of a switch (§1) |
scripts/4c1/tests/test_c4_fastswitch.py |
0857a746 |
F3 fork/thread parallel (4 × 0.5 s → < 1.2 s, 4 pids), RED worker failure → rc 6 + text, F1 settle rule: 4/4 OK |
scripts/4c1/tests/test_c4_standby.py |
84761c30 |
detached → no build + lock free; attach → flock held + owner line; RED second engine refused; detach waits for the in-flight call, closes the driver once, frees the lock; hand-over; dead holder frees the lock; arm() wraps: 3/3 OK |
Measurements:
- Mock selftest (
c4_fastswitch.py selftest: the real rank dirs on thec4_mockworld at E2b labels): PASS. The shippeddo_switchand the fast one (cold, then warm) give the same card state on all 4 cards: weight-write stream digest + count (7,286,528 words per ATT over the Llama bring-up + the switch), drop counters, register file, HBM blocks, the H2C file sequence. They also give the sameswitch.jsonminus timings.- RED arms pass: an in-place rewrite with the mtime put back is re-hashed; an unsettled file is not cached; a bad stream pin → Refused rc 4.
- Evidence:
DEMO_FAST_MODEL_SWITCH_20261110/selftest_qwen.json(ac035545) and the re-run on the final file (c4_fastswitch.py md5a0a121bc,selftest_qwen_final.md5):selftest_qwen_final.json(72db9794) PASS (shipped 45.9 s / fast cold 32.8 s, mock). - The mock times (shipped 45.7 s, fast 28–33 s; the threads share the GIL in the emulation) are mock, not silicon.
prepareon the real rank dirs, no card [M]: dedupe takes 14.3 GB + 3.9 GB → 4.93 GB unique.- Hashing: 1 thread 6.0 s, 8 threads 1.9 s (bound by the 1.53 GB file).
- Stream tables: parse 5.1 s, cached 0.19 s.
- Warm index: all host work of a Qwen switch = 0.43 s (was 28.7 s).
Not done here (the live files are untouched):
model_toggle.py/demo_server.pystill drivec4.py switchand the restart path.- Adoption is outside the 21:30Z setup, and the sequence is:
- run
c4_fastswitch.py prepareas root without the lock (no card) for both models; - in the toggle's
_card_sequence, callc4_fastswitch.py switchwith the same argv asc4.py switch; - for Options 3+: the resident server wrapper (no start-up warm-up,
/admin/attach|detachwritingwant.json) and an active-backend router on:30020, then a silicon A/B of one switch each way with the golden.
- run
7. The resident stack, built as COPIES (2026-11-10 ~20:45Z; coordinator go). Nothing is deployed.
Live files are unchanged: demo_server.py fe00155b, model_toggle.py 442b0d2b. No engine, port, card or lock was touched.
| File (NEW) | md5 | Role |
|---|---|---|
scripts/4c1/demo/demo_server_4c1_resident.py |
a4cd7b8c |
demo_server_4c1.py imported, not copied. Its engine and request path are unchanged. Differences: no FPGA warm-up at start (refuses EAGLE4_FA4_WARM=1); /health + resident/loaded/attach, warm true only while attached + warmed; POST /admin/attach {"run"} (lab lock via the agent, then the stall44 FA4 warm on the first attach + the demo warm prompt, through the unchanged c4_sglang.build gates); POST /admin/detach; /admin/shutdown detaches first; any generate while detached is refused before it reaches the engine |
scripts/4c1/demo/resident_ctl.py |
acf32cdc |
the HTTP half: want.json / state.json nonce protocol with c4_standby. Detach = warm off → one router period → the generate lock HELD through the agent's detach |
scripts/4c1/demo/router_active.py |
304a17f3 |
:30020 front. Active = the ONE backend that is attached + warm (two → none). /health = the active engine's JSON verbatim + router. Everything else is forwarded (SSE relayed). /admin/shutdown → 403 always (the classic toggle's restart path fails closed); /admin/router-stop stops the router only |
reports/arch_site/model_toggle_fast.py |
f9102ba2 |
COPY of model_toggle.py (same API and routes). Job: target loaded? → identity precheck → c4_fastswitch prepare (root, no lock) → detach the attached engine → the classic card steps with c4.py switch → c4_fastswitch.py switch → attach the target → router active. A refusal before any card write re-attaches the previous engine. free_gpu is never called |
scripts/4c1/demo/demo_resident.sh |
d486c979 |
derive / start <m> / start-both (Qwen, then Llama, one at a time) / router / stop / status |
scripts/4c1/demo/fast_toggle_cli.py |
d8e3af88 |
runs the SAME model_toggle_fast job from a shell (deploy, the silicon A/B) |
scripts/4c1/demo/fastswitch_deploy.sh / fastswitch_rollback.sh |
b8b097b1 / fc5c5bc0 |
gated by FASTSWITCH_DEPLOY_GO=window-over / FASTSWITCH_ROLLBACK_GO=yes (both refuse otherwise, rc 2) |
scripts/4c1/demo/fastswitch_demo_import.py |
35b8b6ab |
the ONE live line the deploy changes: import model_toggle ⇄ import model_toggle_fast as model_toggle (exact line, create-only backup, parse check) |
scripts/4c1/demo/mock_resident_endpoint.py |
3d291981 |
test stand-in: the real resident_ctl + a real c4_standby agent (real flock on a temp lab) + a fake engine |
scripts/4c1/c4_fastswitch.py |
f63ebc7a |
+ mockswitch (mock world, fast do_switch, an rd_model_cfg-shaped readback) |
demo_resident.sh start rules:
-
It derives argv and env from
demo_4c1.sh derive(the record launchers in DRYRUN), then changes only the documented items:- the resident server;
- ports
:30021/:30022and engine ports 31000 / 31010; --max-total-tokens 8192;- Qwen
--chunked-prefill-size 2048; - mem-frac = (base + 8 GiB) / free, clamped to 0.30–0.90;
sglang_site→sglang_site_standby;- standby ctl and owner
box1-4c1-demo; EAGLE4_FA4_WARM=0set last;- Qwen
MEASURE_ONLY=1 FASTGLUE=4, i.e.model_toggle._serve_env.
-
It is NOT under
card_run.sh.thermal_wrapruns with cooldown off. -
A start is refused unless:
- the H200's used memory moved ≤ 512 MiB over 4 samples 3 s apart;
- no engine of ours is up but not yet loaded;
- free ≥ need (Qwen 70 GiB, Llama 18 GiB).
It never stops anything to make room.
Offline gates (all green on the final files):
test_c4_fastswitch.py: 4/4.test_c4_standby.py: 3/3.test_fastswitch_deploy.py: 5/5.- The import edit on a COPY reverts byte-identical, and the live file is unchanged.
- REDs: no import line, or both lines.
- The resident derive transform for both models; no
card_run.shin the launch path. - GO refusals; no kill, rm, mv,
ln -sfor-p 1in the scripts.
test_resident_stack.py: 5/5, with logDEMO_FAST_MODEL_SWITCH_20261110/test_resident_stack_full.log.- CTL: attach, detach, a foreign lock holder, a warm failure leaves the lock free, detach waits for an in-flight chat.
- ROUTER: switching 503, the active engine's JSON passed through, the forward header,
/admin/shutdown403, a second attach refused (lock busy), the flip, a lying backend → none active. - TOGGLE: target not loaded → refused while Llama keeps serving; a refusal before any card write → re-attached.
- mock E2E Llama → Qwen through
c4_fastswitch mockswitch+ detach / attach: state ready, router active Qwen, owner line = the endpoint. 87 s mock time; detach took 0.12 s; mock card emulation dominates.
c4_fastswitch selftestre-run onf63ebc7a:selftest_qwen_v2.{json,log}.
Silicon test, after the coordinator says the demo window is over:
FASTSWITCH_DEPLOY_GO=window-over bash scripts/4c1/demo/fastswitch_deploy.sh(serves Llama at the end).fast_toggle_cli.py switch qwen27b(golden record identity + attach), thenfast_toggle_cli.py switch llama3b. Record per-step seconds from the CLI output.- If needed:
FASTSWITCH_ROLLBACK_GO=yes bash scripts/4c1/demo/fastswitch_rollback.sh.
[U] measured only then:
- the capped engines' real footprint (§4);
- the card-process start inside an attach;
- per-card parallel boot times on silicon (§5 Option 3 prediction ≈ 9–10 s).