fpga-archive-demo-box1-20261110 · EAGLE4 FPGA archive
Box1 4-card demo: software readiness for E2b (now) and F3t (after a flash) — 2026-11-10
Scope: code + mock only. No card, no lock, no GPU, no demo process was touched. Program: the 3-card-class Llama-3.2-3B DRAFTER (4C1) plus its dual-model Qwen-27B profile. Result: the two software blockers that stopped the box1 demo on its current images are fixed and tested offline; the Qwen side needed three kit ports (done) and has a short list of open items (section 5). Nothing here is proven on silicon by this work — section 6 lists what still needs a card.
Cards (from BOX1_DEMO_TEMPS_HUMANEVAL_20261110.md): 01/42 = ATT E2b 0xE4B713E2 (md5 2540c725, reg100 0xe4b711e0, tp_bid 0xe4b713e0 stale by design), 61/62 = MLPV 0xE4B71342 (md5 3e82ea1d, reg100 0xe4b71142). Flash record scripts/4c1/flash_records/flash_4c1_box1_e2b_20261107.json.
1. Blocker 1 — reports/arch_site/model_toggle.py was hard-wired to the 0x129C pair
Evidence of the blocker: /mnt/data/eagle4_scratch/box1_temp_20261109/start_demo_llama.out — state refused, c4 discover rc 3 (the default flash record has no 0x13E2 entry).
Fix: the accepted image sets are a table, DUAL_SETS:
| key | ATT label | ATT reg 0x100 | MLPV label | MLPV reg 0x100 | fold step | golden gate |
|---|---|---|---|---|---|---|
| 129C | 0xE4B7129C | 0xE4B7119C | 0xE4B7128B | not checked (as before) | yes | as before (measure-only) |
| E2b | 0xE4B713E2 | 0xE4B711E0 | 0xE4B71342 | 0xE4B71142 | skipped (G4_EMBFOLD 0) | record identity |
| F3t | 0xE4B713F9 | 0xE4B711F9 | 0xE4B71342 | 0xE4B71142 | skipped | record identity |
- Selection: the set is the one the flash record names (
EAGLE4_TOGGLE_FLASH_RECORD, default unchanged = the 0x129C Q3 record).c4 discoverchecks the cards against that record;check_dualthen checks the reg 0x100 read-back against the same row (ATT on every set, MLPV on E2b/F3t). - Every refusal path is kept. New refusals: a flash record that names an image outside the table ("unknown image"), a record that mixes sets, wrong MLPV on E2b/F3t. When the cards carry another known set, the refusal says which one and names the env to set.
- 0x129C behaviour is unchanged: same constants, same refusal strings (asserted byte for byte), same
/api/modelrequiresdict. - On E2b/F3t the switch requires the run's golden to be identical to the E2b record (section 2) before the endpoint starts; the Llama endpoint is then served under that gate, without the measure-only stamp. The Qwen endpoint keeps
MEASURE_ONLY=1(its launcher path), with the same identity requirement enforced by the toggle first. - The running demo server on :8099 imported the old module; it picks this up only when its owner restarts it.
2. Blocker 2 — the SGLang kit refused E2b because the bf16 golden never passes on a LUTG A8 image
Evidence: 4c1_box1_e2b_he80x256_temps_20261109_192011.log — golden.json verdict 'FAIL'. On E2b/F3t the ATT activations are 8-bit, so every tree matches the bf16 golden at L0 (8/8) and differs from L1 down. talon2 served both images on a different proof: the golden output must be identical, tree for tree and level for level, to the E2b record golden (tools_from_talon2/cmp_golden_levels.py, f3t_kit/win_f3t.sh, the STOP rule in chain_f3t.sh), then ran measure-only.
Fix: new module scripts/4c1/c4_recgate.py = that proof as a kit gate.
- Applies only when both ATT are
lutg+a8qrows ofc4_regmapin{0xE4B713E2, 0xE4B713F9}and both MLPV are0xE4B71342, 4-card topology. - Records are pinned by md5 (a record whose bytes changed is refused):
| record | model | trees | md5 | file under TRACK1_RECORD_F3T_20261109/e2b_same_day/ |
|---|---|---|---|---|
| e2b_llama_md15_7_2 | llama3b | 12 (56 levels) | 399b0e06415b601f4301a01803a9036f |
e2b_record_20261106/golden_llama_e2.json |
| e2b_llama_md7_2 | llama3b | 9 | 9a1206e9b9b6fca034cf97a7af7fa03a |
e2bwin/l/golden.json |
| e2b_qwen_md7_2_kc64 | qwen27b | 9 | 8906b857089b6a8f69b56046a88d2d35 |
e2bwin/q/golden.json |
- PASS = same tree list (golden, md, ctx, TREE_STOP rule), and per tree the same verdict, depth, compared depth and, per level, the same cand / surv / surv_blocks; same loop mode bits and fold; 0 discarded trees; L0 8/8 everywhere. An empty or shorter list is a FAIL.
c4.py goldenwrites the result intogolden.json["record_identity"](the bf16verdictstaysFAIL) and returns 0 when identity passes.c4_recgate.py stamp --run RUNdoes the same on an existing golden, file-only, and keeps the file's mtime so a stamp can never make an old golden look fresh.c4_sglang.open_session: when the golden is not a bf16 PASS, it needs the stamp and recomputes the comparison before any tree is timed. It still requires golden.json newer than the arm, the loop-mode and fold match, and a non-discarded TREE_STOP tree when TREE_STOP is on.- Not an override:
EAGLE4_4C1_ALLOW_UNGATEDis untouched and not needed;EAGLE4_4C1_RECORD_GATE=0restores the old behaviour exactly. - The golden step must run with the record's arguments and the default TREE_STOP theta (-1.25, i.e. do not export
EAGLE4_PAIR_TREE_STOP_THETAfor the golden step); the serve launcher sets -2.5 by itself.
3. Qwen side on box1 (checked offline)
Present and consistent:
| item | state |
|---|---|
| Qwen boot root | ~/eagle4_artifacts/4c1_boot_qwen27b/v3_ropestep256_enormk (the profile default) |
| MANIFEST md5, ATT 0xE4B713E2 rank0 / rank1 | 7def3a2a68106f269249dde62154c1b8 / f97bf560a8c539476b2c0188a1d78097 |
| MANIFEST md5, ATT 0xE4B713F9 rank0 / rank1 | dcd72eb3c3e52d1c828d904229aa352a / 128cf67c6e00ac03cc5fcccb3eb4a1e5 |
| MANIFEST md5, MLPV 0xE4B71342 rank0 / rank1 | c2b7e343555f0795f635750c1616ae91 / 6694caa3626f47980a87d4c88e514543 |
Llama boot root ~/eagle4_artifacts/4c1_boot, 13E2 r0/r1 |
ef71d0a2… / 8af9f2f9… (= talon2's) |
| Llama 13F9 r0/r1 | ddf7653b4105e455c33adfb2354ae375 / 89a855b80ea15f0566040f8f6370df35 |
| Llama 1342 r0/r1 | 1c8e12d7… / 8378ee10… |
| Qwen goldens | ~/eagle4_artifacts/4c1_boot_qwen27b/goldens_20261020/c4_golden_q27h200/goldens/md15_golden_q27_{p2_step001_ctx0089,p4_step000_ctx0159,p5_step002_ctx0187}.json present |
| Qwen switch on box1 E2b | already done once by the card owner: /mnt/data/eagle4_scratch/4c1_runs/box1_e2b_tempbench_20261110_q2/ (switch.json, golden 0/9 bf16). That golden is 9/9 identical to the Qwen record under the new gate. Its golden.log was not teed in the talon2 .lines form, so the 7aceae28 lines-md5 was not recomputed here; level identity is the stronger check. |
Missing from the box1 kit, now ported (all additive, only reached by E2b/F3t rows):
c4_bringup.VERIFY_LOGdefault list had no E2b/F3t/0x1342 verify logs (the 11-10 bring-up needed an env file). Added the box1 logs: Llama 13E2, Llama 1342, Qwen 13E2 in place; F3t Llama + Qwen and Qwen 1342 underBOX1_DEMO_READINESS_E2B_F3T_20261110/logs/.- The two F3t logs are new:
f3t_kit/verify_relabel_identity_13F9.pyrun on the box1 dirs,RELABEL_IDENTITY PASSfor both roots (37 and 48 files byte-identical to 13E2,boot.opsdiffers only in the label lines). verify4c1_qwen27b_mlpv_eaeb_1342_20261105.log(md5ceabce4f…) is a copy of the box1-twin verify of 11-05; it was only in scratch twins, not in the repo.
- The two F3t logs are new:
- LUTG zero-bank guard in
load_manifest(talon2patch_13de_host.py): a rank dir without the 6 zero-bank loads is refused. All eight box1 ATT dirs pass; a dir with one pin removed is refused. c4_profileqwen27benorm_rescale.images+= 0xE4B713E2, 0xE4B713F9, so the F2 e_norm check runs on them (it was skipped). All four box1 Qwen ATT dirs check exact.
With these, load_manifest accepts all 12 (model × card × rank) kits with the default verify list.
4. Operator runbook
All commands from the repo root as y344shi. PWF=~/.config/eagle4/ami_sudo_pw. Nothing below flashes or resets except step F1, which is the owner's -p 0 flash.
A. After every boot / cold AC (before anything is timed)
export PWF=~/.config/eagle4/ami_sudo_pw
bash scripts/restore_artifact_links.sh # every line LINKED:
lspci -d 10ee:50b4 | grep -c . # 4 (01 42 61 62)
sudo -p "" ami_tool overview < $PWF # 4 READY
# SMI / AER corrected-error mask (lost at every boot). Measure, mask all four V80s + their root ports, measure again.
sudo -p "" perf stat -a -e ls_smi_rx sleep 5 < $PWF 2>&1 | grep ls_smi_rx # storm ~136k / 5 s, clean 0-3k
MST=/sanctuary/lab/smi_aer_mask_state_$(date +%Y%m%d_%H%M%S); mkdir -p $MST
sudo -p "" python3 -I /sanctuary/pdi_inbox/SMI_AER_AB/v1/lib/smi_aer_ab.py apply-persist $MST --scope both --cards "01 42 61 62" < $PWF
sudo -p "" perf stat -a -e ls_smi_rx sleep 5 < $PWF 2>&1 | grep ls_smi_rx # must be clean; undo = smi_aer_ab.py restore $MST/saved_state.json
for B in 01 42 61 62; do OWNER=box1-4c1-qdma bash /sanctuary/lab/card_run.sh sudo -p "" EAGLE/v80_test_suite/03_setup_qdma.sh 4 $B:00.0 < $PWF | grep "Roundtrip test"; done
- The mask must go on before the demo endpoint takes the card lock: the tool skips cards named in
CARD_OWNER. - Card 01 alone is not enough since 10-18 (
post_coldac_4c1.shstep 4): mask all four. Use a fresh state dir each boot; the tool refuses to overwrite an undo point. scripts/4c1/post_coldac_4c1.shruns the same steps but then starts the default (non-E2b) bring-up; use the lines above for E2b/F3t.
B. Bring-up, Llama on E2b as flashed
export RUN=/mnt/data/eagle4_scratch/4c1_runs/box1_e2b_$(date +%Y%m%d_%H%M%S)
export FLASH_RECORD=$PWD/scripts/4c1/flash_records/flash_4c1_box1_e2b_20261107.json
RUN=$RUN PWF=$PWF FOLD=0 DEPTH=15 GOLDEN_MD=15,7,2 STEPS="discover linkfix link boot arm golden" bash scripts/4c1/run_4c1.sh
grep -E "^GOLDEN" $RUN/run.log # expect: GOLDEN FAIL (0/12 ...) + GOLDEN RECORD-IDENTITY PASS (trees identical 12/12, levels compared 56 ...) + GOLDEN GATE PASS by RECORD IDENTITY
- Arm depth 15 once per boot (PERSIST latches the deepest depth at the first run); it covers d7 and d11.
- Cards already armed this boot with an older golden:
python3 -I scripts/4c1/c4_recgate.py stamp --run $RUNstamps the existing golden.json (file-only, rc 0 = PASS). - If the identity line says FAIL, stop: that is a correctness failure, not a gate to step around.
C. Timed runs, Llama (single stream)
HE=~/haochen/eaglefit/inference/dataset/humaneval/question.jsonl
# humaneval 80 x 256, d7 (VTOK 20 = the record shape) and d11 -- CORRECTED 2026-11-10: run d11 with VTOK 20 (set it explicitly; the
# launcher's full-tree default is 43). VTOK 20 is the measured winner at d11 on talon2 (883.12, F3T_CONFIG_SWEEP) and on box1 (BOX1_E2B_D11_20261110.md)
RUN=$RUN PWF=$PWF BENCH=$HE MAXREQ=80 MAXNEW=256 DEPTH=7 WIDTH=2 VTOK=20 TAG=box1_e2b_he80x256_d7 bash scripts/4c1/run_4c1_sglang.sh
RUN=$RUN PWF=$PWF BENCH=$HE MAXREQ=80 MAXNEW=256 DEPTH=11 WIDTH=2 TAG=box1_e2b_he80x256_d11 bash scripts/4c1/run_4c1_sglang.sh
# the 15-prompt demo set (original humaneval rows, rank order)
python3 -I scripts/4c1/demo/demo_prompts_to_bench.py --model llama --out /mnt/data/eagle4_scratch/demo_prompts_bench_20261110/llama
RUN=$RUN PWF=$PWF BENCH=/mnt/data/eagle4_scratch/demo_prompts_bench_20261110/llama/question.jsonl MAXREQ=15 MAXNEW=256 DEPTH=7 WIDTH=2 VTOK=20 TAG=box1_e2b_demo15_d7 bash scripts/4c1/run_4c1_sglang.sh
- Launcher defaults already give FASTGLUE 4, K_C 64, TREE_STOP theta -2.5, KVI on, first draft on GPU, cuda graphs; the log header prints all of them.
- PQ+C is not available in this kit (section 5, item 1). Asking for it (
EXTRA_FPGA_ENV="EAGLE4_4C1_HP_PREPOISON=1 ...") is now refused with rc 62 instead of being silently ignored. The kit's own olderPREPOISON=1 PINPAR=1are on by default. - CORRECTED 2026-11-10: d11 is served with VTOK 20 (record config w2 d11 fa3 V20 K_C 64 theta -2.5), not the launcher's full-tree default 43. Measured on box1 E2b: d11 V20 820.20 (non-PQC) / 836.00 (PQ+C) @ 6.90.
- Each run takes the card lock itself and refuses while the demo endpoint holds it.
D. Demo endpoint and model switch
# endpoint by hand (holds the card lock while it serves; stop with POST /admin/shutdown)
RUN=$RUN PWF=$PWF MODEL=llama3b DEPTH=7 WIDTH=2 bash scripts/4c1/demo/demo_4c1.sh serve
For the web toggle, the demo server must be started by its owner with:
EAGLE4_TOGGLE_FLASH_RECORD=<repo>/scripts/4c1/flash_records/flash_4c1_box1_e2b_20261107.json EAGLE4_TOGGLE_LATCHED_DEPTH=15
(the cards are armed at 15; the toggle's arm step passes --accept-latched-depth). The toggle then runs discover → read-back → switch → arm → golden (md 7,2) → identity check → serve.
E. Qwen on the same cards
QG=~/eagle4_artifacts/4c1_boot_qwen27b/goldens_20261020/c4_golden_q27h200
export RUNQ=${RUN}_q; mkdir -p $RUNQ
c4() { OWNER=box1-4c1kit bash /sanctuary/lab/card_run.sh sudo -p "" -E env HOME=~ /usr/bin/python3 -I scripts/4c1/c4.py "$@" < $PWF; }
c4 --model qwen27b discover --out $RUNQ/discover.json --flash-record $FLASH_RECORD
c4 --model qwen27b switch --disc $RUNQ/discover.json --model qwen27b
c4 --model qwen27b arm --disc $RUNQ/discover.json --depth 15 --accept-latched-depth 15
c4 --model qwen27b golden --disc $RUNQ/discover.json --md 7,2 --kc 64 --gold-dir $QG \
--golden md15_golden_q27_p2_step001_ctx0089.json,md15_golden_q27_p4_step000_ctx0159.json,md15_golden_q27_p5_step002_ctx0187.json --out $RUNQ/golden.json
# expect GOLDEN RECORD-IDENTITY PASS (trees identical 9/9 ... record e2b_qwen_md7_2_kc64)
RUN=$RUNQ PWF=$PWF MOCK=0 BENCH=humaneval MAXREQ=80 MAXNEW=256 DEPTH=7 WIDTH=2 VTOK=20 FASTGLUE=4 TAG=box1_e2b_qwen_he80x256_d7 bash scripts/4c1/run_4c1_sglang_qwen.sh
- Back to Llama: the same four
c4lines with--model llama3b,--md 7,2, no--kc/--gold-dir/--golden, into a new run dir; identity is then againste2b_llama_md7_2(9/9). - The H200 must have room for the 27B target (the toggle's guard: about 62.7 GiB / 0.55 = 114 GiB free); the GPU demo engines hold memory.
- talon2 ran the Qwen cell with
MEASURE_ONLY=1; with the record gate it is no longer needed. The Qwen launcher takes a dataset name inBENCHand an explicit file inBENCH_PATH: for the 10-prompt Qwen demo set rundemo_prompts_to_bench.py --model qwen --out <dir>and passBENCH_PATH=<dir>/question.jsonl MAXREQ=10.
F. The same after an F3t flash of 01/42
- Flash (owner): the F3t PDI is not on box1 (
~/eagle4_artifacts/uni_oneflip/has no*13F9*). Copyoneflip_p2s_head_asgard_0xE4B713F9_66m_97d8445c.pdifrom talon2~/eagle4_artifacts/uni_oneflip/, check md597d8445cc6e64ff95af8894d6f267965and the FPT magic16 a5 f7 92, runscripts/preflash_gate.sh, flash 01 and 42 detached with-p 0, wait forFLASH_EXIT=0on both, then cold AC. - Write the flash record (it does not exist yet; a record is written from the flash logs, not ahead of them): copy
flash_4c1_box1_e2b_20261107.jsontoscripts/4c1/flash_records/flash_4c1_box1_f3t_<date>.json, set both ATT entries to label0xE4B713F9, pdi_md597d8445cc6e64ff95af8894d6f267965, the PDI path, and a_sourceline naming the flash log. - Section A, then B–E unchanged with
FLASH_RECORD(andEAGLE4_TOGGLE_FLASH_RECORD) pointing at the new file and run-dir names changed. F3t reads reg1000xe4b711f9and tp_bid0xe4b713f9(= label, not stale). The records are the same E2b records: F3t is level-identical to E2b on both models (f3t_run/goldens, 12/12, 9/9, 9/9 under the new gate). - One F3t-specific rule from the record: the 0x90 / 0xD4 / 0xD8 debug read-outs are cut on this image (TIMING_HARD) and are not evidence.
5. Still missing (not fixed here)
- Host pre-card levers PQ+C (
c4_hpre: HP_PREPOISON, HP_PINPAR, HP_CAPHOST, HP_IMGBYTES). The F3t 831.14 and E2b 816.47 records ran with them; the box1 repo kit has noc4_hpre.py. Copies exist at/mnt/data/eagle4_scratch/hostpre_20261107/kit/new/scripts/4c1/c4_hpre.pyand in the kit copy/mnt/data/eagle4_scratch/box1_temp_20261109/kit_e2b/4c1/. Porting touches the card-process path and needs its own gate; until then box1 numbers are the non-PQC arm (talon2 arm B on E2b was 784.06 vs 816.47 with PQC). - Mock endpoint:
reports/arch_site/mock_4c1_endpoint.pyis referenced by the toggle's mock mode and is not in the repo, so a full mock switch cannot reach "ready". The mock tests cover the card steps and the refusals. - F3t PDI and box1 F3t flash record (section F).
- Verify-log weakness (pre-existing): Qwen verify logs also carry un-prefixed
[verify4c1] 0x… rankN: PASSlines, and the Llama check matches un-prefixed lines, so a Qwen log can satisfy a Llama rank of the same label if it is newer than that MANIFEST. Not changed here; the Llama 1342 log is listed, so the right log is found first only by list order. demo_4c1.shcarries someone else's uncommitted--enable-metricsedit; left alone, not committed.- A stray empty-purpose file
BOX1_DEMO_READINESS_E2B_F3T_20261110/logs/.t1(timing output of the relabel run) was left in place, uncommitted (no removals).
6. What was tested, and what still needs a card
Offline tests (all pass):
| test | count | what |
|---|---|---|
scripts/4c1/tests/test_recgate.py |
33 | records pinned; GREEN on 6 real talon2 goldens (F3t + E2b, both models) and on 3 box1 E2b run dirs (12/12, 9/9, 9/9); 12 planted REDs; scope (0x129C refused as before); stamp / gate / stale / forged stamp; open_session with the card layer stubbed; c4.py golden exit codes; box1 kits under the default verify list; zero-bank RED |
scripts/4c1/tests/test_model_toggle_sets.py |
15 (278 s) | table vs c4_regmap; set per flash record; E2b and F3t identity; RED unknown image; 0x129C strings unchanged; the 0x129C mock switch card steps on c4_mock; E2b / F3t identity and two refused jobs on the mock |
test_profile_dual.py |
40/40 | one assertion updated for the two added e_norm labels |
test_sglang_seam.py (eaglefit python, no GPU), test_tp4_session7.py, test_golden_exact.py, test_demo_model.py |
pass | regression |
| launcher dry-run | — | d11 gives VTOK 43, K_C 64, FASTGLUE 4; HP lever request refused rc 62 |
Failing before and after, not caused by this work: test_regmap_cites.py (4, fork source md5 drift) and test_kit_units.py::test_att_image_alternates (duplicate reg100 ids in c4_regmap).
Needs a card (not done, not claimed):
- one real
c4.py goldenon box1 writing the stamp, then a served request throughopen_sessionon the cards (the gate decision ran only on archived goldens and stubs); - the toggle end to end on E2b (switch → identity → warm endpoint), both models;
- any tok/s or accept number: none was measured here;
- everything in section F.
Calculations and hypothesis
- "talon2 arm B 784.06 vs PQC 816.47" implies about +4% from PQ+C on E2b; whether box1 shows the same gap is not measured.
- The 114 GiB free-memory figure for Qwen is the toggle guard's own arithmetic ((56,525 + 6,144) MiB / 0.55), not a new measurement.