fpga-archive-demo-box1-20261110 · EAGLE4 FPGA archive

Source reports/results/BOX1_DEMO_READINESS_E2B_F3T_20261110.md · source md5 98e8c6be831de7c7b2e7a86bc1678bc5 · archived 2026-11-18

Download PDFOpen PDFRaw markdown

Box1 4-card demo: software readiness for E2b (now) and F3t (after a flash) — 2026-11-10

Scope: code + mock only. No card, no lock, no GPU, no demo process was touched. Program: the 3-card-class Llama-3.2-3B DRAFTER (4C1) plus its dual-model Qwen-27B profile. Result: the two software blockers that stopped the box1 demo on its current images are fixed and tested offline; the Qwen side needed three kit ports (done) and has a short list of open items (section 5). Nothing here is proven on silicon by this work — section 6 lists what still needs a card.

Cards (from BOX1_DEMO_TEMPS_HUMANEVAL_20261110.md): 01/42 = ATT E2b 0xE4B713E2 (md5 2540c725, reg100 0xe4b711e0, tp_bid 0xe4b713e0 stale by design), 61/62 = MLPV 0xE4B71342 (md5 3e82ea1d, reg100 0xe4b71142). Flash record scripts/4c1/flash_records/flash_4c1_box1_e2b_20261107.json.

1. Blocker 1 — reports/arch_site/model_toggle.py was hard-wired to the 0x129C pair

Evidence of the blocker: /mnt/data/eagle4_scratch/box1_temp_20261109/start_demo_llama.out — state refused, c4 discover rc 3 (the default flash record has no 0x13E2 entry).

Fix: the accepted image sets are a table, DUAL_SETS:

key ATT label ATT reg 0x100 MLPV label MLPV reg 0x100 fold step golden gate
129C 0xE4B7129C 0xE4B7119C 0xE4B7128B not checked (as before) yes as before (measure-only)
E2b 0xE4B713E2 0xE4B711E0 0xE4B71342 0xE4B71142 skipped (G4_EMBFOLD 0) record identity
F3t 0xE4B713F9 0xE4B711F9 0xE4B71342 0xE4B71142 skipped record identity

2. Blocker 2 — the SGLang kit refused E2b because the bf16 golden never passes on a LUTG A8 image

Evidence: 4c1_box1_e2b_he80x256_temps_20261109_192011.log — golden.json verdict 'FAIL'. On E2b/F3t the ATT activations are 8-bit, so every tree matches the bf16 golden at L0 (8/8) and differs from L1 down. talon2 served both images on a different proof: the golden output must be identical, tree for tree and level for level, to the E2b record golden (tools_from_talon2/cmp_golden_levels.py, f3t_kit/win_f3t.sh, the STOP rule in chain_f3t.sh), then ran measure-only.

Fix: new module scripts/4c1/c4_recgate.py = that proof as a kit gate.

record model trees md5 file under TRACK1_RECORD_F3T_20261109/e2b_same_day/
e2b_llama_md15_7_2 llama3b 12 (56 levels) 399b0e06415b601f4301a01803a9036f e2b_record_20261106/golden_llama_e2.json
e2b_llama_md7_2 llama3b 9 9a1206e9b9b6fca034cf97a7af7fa03a e2bwin/l/golden.json
e2b_qwen_md7_2_kc64 qwen27b 9 8906b857089b6a8f69b56046a88d2d35 e2bwin/q/golden.json

3. Qwen side on box1 (checked offline)

Present and consistent:

item state
Qwen boot root ~/eagle4_artifacts/4c1_boot_qwen27b/v3_ropestep256_enormk (the profile default)
MANIFEST md5, ATT 0xE4B713E2 rank0 / rank1 7def3a2a68106f269249dde62154c1b8 / f97bf560a8c539476b2c0188a1d78097
MANIFEST md5, ATT 0xE4B713F9 rank0 / rank1 dcd72eb3c3e52d1c828d904229aa352a / 128cf67c6e00ac03cc5fcccb3eb4a1e5
MANIFEST md5, MLPV 0xE4B71342 rank0 / rank1 c2b7e343555f0795f635750c1616ae91 / 6694caa3626f47980a87d4c88e514543
Llama boot root ~/eagle4_artifacts/4c1_boot, 13E2 r0/r1 ef71d0a2… / 8af9f2f9… (= talon2's)
Llama 13F9 r0/r1 ddf7653b4105e455c33adfb2354ae375 / 89a855b80ea15f0566040f8f6370df35
Llama 1342 r0/r1 1c8e12d7… / 8378ee10…
Qwen goldens ~/eagle4_artifacts/4c1_boot_qwen27b/goldens_20261020/c4_golden_q27h200/goldens/md15_golden_q27_{p2_step001_ctx0089,p4_step000_ctx0159,p5_step002_ctx0187}.json present
Qwen switch on box1 E2b already done once by the card owner: /mnt/data/eagle4_scratch/4c1_runs/box1_e2b_tempbench_20261110_q2/ (switch.json, golden 0/9 bf16). That golden is 9/9 identical to the Qwen record under the new gate. Its golden.log was not teed in the talon2 .lines form, so the 7aceae28 lines-md5 was not recomputed here; level identity is the stronger check.

Missing from the box1 kit, now ported (all additive, only reached by E2b/F3t rows):

  1. c4_bringup.VERIFY_LOG default list had no E2b/F3t/0x1342 verify logs (the 11-10 bring-up needed an env file). Added the box1 logs: Llama 13E2, Llama 1342, Qwen 13E2 in place; F3t Llama + Qwen and Qwen 1342 under BOX1_DEMO_READINESS_E2B_F3T_20261110/logs/.
    • The two F3t logs are new: f3t_kit/verify_relabel_identity_13F9.py run on the box1 dirs, RELABEL_IDENTITY PASS for both roots (37 and 48 files byte-identical to 13E2, boot.ops differs only in the label lines).
    • verify4c1_qwen27b_mlpv_eaeb_1342_20261105.log (md5 ceabce4f…) is a copy of the box1-twin verify of 11-05; it was only in scratch twins, not in the repo.
  2. LUTG zero-bank guard in load_manifest (talon2 patch_13de_host.py): a rank dir without the 6 zero-bank loads is refused. All eight box1 ATT dirs pass; a dir with one pin removed is refused.
  3. c4_profile qwen27b enorm_rescale.images += 0xE4B713E2, 0xE4B713F9, so the F2 e_norm check runs on them (it was skipped). All four box1 Qwen ATT dirs check exact.

With these, load_manifest accepts all 12 (model × card × rank) kits with the default verify list.

4. Operator runbook

All commands from the repo root as y344shi. PWF=~/.config/eagle4/ami_sudo_pw. Nothing below flashes or resets except step F1, which is the owner's -p 0 flash.

A. After every boot / cold AC (before anything is timed)

export PWF=~/.config/eagle4/ami_sudo_pw
bash scripts/restore_artifact_links.sh                       # every line LINKED:
lspci -d 10ee:50b4 | grep -c .                               # 4 (01 42 61 62)
sudo -p "" ami_tool overview < $PWF                       # 4 READY
# SMI / AER corrected-error mask (lost at every boot). Measure, mask all four V80s + their root ports, measure again.
sudo -p "" perf stat -a -e ls_smi_rx sleep 5 < $PWF 2>&1 | grep ls_smi_rx      # storm ~136k / 5 s, clean 0-3k
MST=/sanctuary/lab/smi_aer_mask_state_$(date +%Y%m%d_%H%M%S); mkdir -p $MST
sudo -p "" python3 -I /sanctuary/pdi_inbox/SMI_AER_AB/v1/lib/smi_aer_ab.py apply-persist $MST --scope both --cards "01 42 61 62" < $PWF
sudo -p "" perf stat -a -e ls_smi_rx sleep 5 < $PWF 2>&1 | grep ls_smi_rx      # must be clean; undo = smi_aer_ab.py restore $MST/saved_state.json
for B in 01 42 61 62; do OWNER=box1-4c1-qdma bash /sanctuary/lab/card_run.sh sudo -p "" EAGLE/v80_test_suite/03_setup_qdma.sh 4 $B:00.0 < $PWF | grep "Roundtrip test"; done

B. Bring-up, Llama on E2b as flashed

export RUN=/mnt/data/eagle4_scratch/4c1_runs/box1_e2b_$(date +%Y%m%d_%H%M%S)
export FLASH_RECORD=$PWD/scripts/4c1/flash_records/flash_4c1_box1_e2b_20261107.json
RUN=$RUN PWF=$PWF FOLD=0 DEPTH=15 GOLDEN_MD=15,7,2 STEPS="discover linkfix link boot arm golden" bash scripts/4c1/run_4c1.sh
grep -E "^GOLDEN" $RUN/run.log        # expect: GOLDEN FAIL (0/12 ...) + GOLDEN RECORD-IDENTITY PASS (trees identical 12/12, levels compared 56 ...) + GOLDEN GATE PASS by RECORD IDENTITY

C. Timed runs, Llama (single stream)

HE=~/haochen/eaglefit/inference/dataset/humaneval/question.jsonl
# humaneval 80 x 256, d7 (VTOK 20 = the record shape) and d11 -- CORRECTED 2026-11-10: run d11 with VTOK 20 (set it explicitly; the
# launcher's full-tree default is 43). VTOK 20 is the measured winner at d11 on talon2 (883.12, F3T_CONFIG_SWEEP) and on box1 (BOX1_E2B_D11_20261110.md)
RUN=$RUN PWF=$PWF BENCH=$HE MAXREQ=80 MAXNEW=256 DEPTH=7  WIDTH=2 VTOK=20 TAG=box1_e2b_he80x256_d7  bash scripts/4c1/run_4c1_sglang.sh
RUN=$RUN PWF=$PWF BENCH=$HE MAXREQ=80 MAXNEW=256 DEPTH=11 WIDTH=2         TAG=box1_e2b_he80x256_d11 bash scripts/4c1/run_4c1_sglang.sh
# the 15-prompt demo set (original humaneval rows, rank order)
python3 -I scripts/4c1/demo/demo_prompts_to_bench.py --model llama --out /mnt/data/eagle4_scratch/demo_prompts_bench_20261110/llama
RUN=$RUN PWF=$PWF BENCH=/mnt/data/eagle4_scratch/demo_prompts_bench_20261110/llama/question.jsonl MAXREQ=15 MAXNEW=256 DEPTH=7 WIDTH=2 VTOK=20 TAG=box1_e2b_demo15_d7 bash scripts/4c1/run_4c1_sglang.sh

D. Demo endpoint and model switch

# endpoint by hand (holds the card lock while it serves; stop with POST /admin/shutdown)
RUN=$RUN PWF=$PWF MODEL=llama3b DEPTH=7 WIDTH=2 bash scripts/4c1/demo/demo_4c1.sh serve

For the web toggle, the demo server must be started by its owner with: EAGLE4_TOGGLE_FLASH_RECORD=<repo>/scripts/4c1/flash_records/flash_4c1_box1_e2b_20261107.json EAGLE4_TOGGLE_LATCHED_DEPTH=15 (the cards are armed at 15; the toggle's arm step passes --accept-latched-depth). The toggle then runs discover → read-back → switch → arm → golden (md 7,2) → identity check → serve.

E. Qwen on the same cards

QG=~/eagle4_artifacts/4c1_boot_qwen27b/goldens_20261020/c4_golden_q27h200
export RUNQ=${RUN}_q; mkdir -p $RUNQ
c4() { OWNER=box1-4c1kit bash /sanctuary/lab/card_run.sh sudo -p "" -E env HOME=~ /usr/bin/python3 -I scripts/4c1/c4.py "$@" < $PWF; }
c4 --model qwen27b discover --out $RUNQ/discover.json --flash-record $FLASH_RECORD
c4 --model qwen27b switch --disc $RUNQ/discover.json --model qwen27b
c4 --model qwen27b arm --disc $RUNQ/discover.json --depth 15 --accept-latched-depth 15
c4 --model qwen27b golden --disc $RUNQ/discover.json --md 7,2 --kc 64 --gold-dir $QG \
   --golden md15_golden_q27_p2_step001_ctx0089.json,md15_golden_q27_p4_step000_ctx0159.json,md15_golden_q27_p5_step002_ctx0187.json --out $RUNQ/golden.json
# expect GOLDEN RECORD-IDENTITY PASS (trees identical 9/9 ... record e2b_qwen_md7_2_kc64)
RUN=$RUNQ PWF=$PWF MOCK=0 BENCH=humaneval MAXREQ=80 MAXNEW=256 DEPTH=7 WIDTH=2 VTOK=20 FASTGLUE=4 TAG=box1_e2b_qwen_he80x256_d7 bash scripts/4c1/run_4c1_sglang_qwen.sh

F. The same after an F3t flash of 01/42

  1. Flash (owner): the F3t PDI is not on box1 (~/eagle4_artifacts/uni_oneflip/ has no *13F9*). Copy oneflip_p2s_head_asgard_0xE4B713F9_66m_97d8445c.pdi from talon2 ~/eagle4_artifacts/uni_oneflip/, check md5 97d8445cc6e64ff95af8894d6f267965 and the FPT magic 16 a5 f7 92, run scripts/preflash_gate.sh, flash 01 and 42 detached with -p 0, wait for FLASH_EXIT=0 on both, then cold AC.
  2. Write the flash record (it does not exist yet; a record is written from the flash logs, not ahead of them): copy flash_4c1_box1_e2b_20261107.json to scripts/4c1/flash_records/flash_4c1_box1_f3t_<date>.json, set both ATT entries to label 0xE4B713F9, pdi_md5 97d8445cc6e64ff95af8894d6f267965, the PDI path, and a _source line naming the flash log.
  3. Section A, then B–E unchanged with FLASH_RECORD (and EAGLE4_TOGGLE_FLASH_RECORD) pointing at the new file and run-dir names changed. F3t reads reg100 0xe4b711f9 and tp_bid 0xe4b713f9 (= label, not stale). The records are the same E2b records: F3t is level-identical to E2b on both models (f3t_run/ goldens, 12/12, 9/9, 9/9 under the new gate).
  4. One F3t-specific rule from the record: the 0x90 / 0xD4 / 0xD8 debug read-outs are cut on this image (TIMING_HARD) and are not evidence.

5. Still missing (not fixed here)

  1. Host pre-card levers PQ+C (c4_hpre: HP_PREPOISON, HP_PINPAR, HP_CAPHOST, HP_IMGBYTES). The F3t 831.14 and E2b 816.47 records ran with them; the box1 repo kit has no c4_hpre.py. Copies exist at /mnt/data/eagle4_scratch/hostpre_20261107/kit/new/scripts/4c1/c4_hpre.py and in the kit copy /mnt/data/eagle4_scratch/box1_temp_20261109/kit_e2b/4c1/. Porting touches the card-process path and needs its own gate; until then box1 numbers are the non-PQC arm (talon2 arm B on E2b was 784.06 vs 816.47 with PQC).
  2. Mock endpoint: reports/arch_site/mock_4c1_endpoint.py is referenced by the toggle's mock mode and is not in the repo, so a full mock switch cannot reach "ready". The mock tests cover the card steps and the refusals.
  3. F3t PDI and box1 F3t flash record (section F).
  4. Verify-log weakness (pre-existing): Qwen verify logs also carry un-prefixed [verify4c1] 0x… rankN: PASS lines, and the Llama check matches un-prefixed lines, so a Qwen log can satisfy a Llama rank of the same label if it is newer than that MANIFEST. Not changed here; the Llama 1342 log is listed, so the right log is found first only by list order.
  5. demo_4c1.sh carries someone else's uncommitted --enable-metrics edit; left alone, not committed.
  6. A stray empty-purpose file BOX1_DEMO_READINESS_E2B_F3T_20261110/logs/.t1 (timing output of the relabel run) was left in place, uncommitted (no removals).

6. What was tested, and what still needs a card

Offline tests (all pass):

test count what
scripts/4c1/tests/test_recgate.py 33 records pinned; GREEN on 6 real talon2 goldens (F3t + E2b, both models) and on 3 box1 E2b run dirs (12/12, 9/9, 9/9); 12 planted REDs; scope (0x129C refused as before); stamp / gate / stale / forged stamp; open_session with the card layer stubbed; c4.py golden exit codes; box1 kits under the default verify list; zero-bank RED
scripts/4c1/tests/test_model_toggle_sets.py 15 (278 s) table vs c4_regmap; set per flash record; E2b and F3t identity; RED unknown image; 0x129C strings unchanged; the 0x129C mock switch card steps on c4_mock; E2b / F3t identity and two refused jobs on the mock
test_profile_dual.py 40/40 one assertion updated for the two added e_norm labels
test_sglang_seam.py (eaglefit python, no GPU), test_tp4_session7.py, test_golden_exact.py, test_demo_model.py pass regression
launcher dry-run — d11 gives VTOK 43, K_C 64, FASTGLUE 4; HP lever request refused rc 62

Failing before and after, not caused by this work: test_regmap_cites.py (4, fork source md5 drift) and test_kit_units.py::test_att_image_alternates (duplicate reg100 ids in c4_regmap).

Needs a card (not done, not claimed):

Calculations and hypothesis