fpga-archive-reports-process · EAGLE4 FPGA archive

Source tested_archive/README.md · source md5 3c9abf649ee2c685328ddf3e274db730 · archived 2026-11-18

Download PDFOpen PDFRaw markdown

EAGLE4 pair — tested archive (box1, snapshot 2026-09-27 ~02:10)

This folder is the reproducibility archive for every HEAD/TAIL V80 pair image tested on box1 in the 2026-09-21..27 silicon campaign. It holds:

Everything here is a copy; the sources stay where they were (see MANIFEST.md). To refresh after new runs, re-run /sanctuary/lab/tested_archive_build/copy_archive.sh. It is create-only (cp -n), so it adds new files and never overwrites or deletes. The A9b+F4 ladder was still running at snapshot time; its partial results are in results/lab/2026-09-27/.

Setup of every pair number below:

1. Results — best tok/s per image pair (mt_bench 50×32)

pair (HEAD+TAIL) best cell tok/s @ accept n (rounds) config source
A5+F4 w2 d2 268.01 @ 2.55 (3-round 264.29; fold ON 264.83) 1 (+3+3) KVI=1 + FASTGLUE=4 (HOST_GLUE v4.3), SGLv 0.5.16 results/pair_accuracy_20260921/width_ab_20260926_185929.txt, results/lab/2026-09-26/ADAPTIVE_TREE, A5F4_FOLD_V21
A5+F4 w1 d2 / w3 d2 259.45 @ 2.32 / 257.72 @ 2.61 1 same width_ab_20260926_193006.txt
A9b+F4 w2 d2 241.56 @ 2.55 (241.35 / 244.29 / 239.03) 3 same results/lab/2026-09-27/A9BF4_FLASH/REPORT.md, width_ab_20260927_020147.txt
A8+F4 w2 d2 234.62 @ 2.55 (234.56 / 234.71 / 234.60) 3 same results/lab/2026-09-26/HOST_GLUE_V4_FIXED
A8+F4 w2 d2, KVI off 203.94 (F3 fastglue) / 188.34 (plain) 3 FASTGLUE=3 / none, SGLv 0.5.14 HOST_GLUE_V3, A8F4_FLASH
A5+F4 w2 d2, KVI off 202.53 @ 1.83 3 protocol v1 results/lab/2026-09-26/A5F4_PROTOCOL
E7+F3 w2 d2 191.52 @ 1.83 3 KVI off, SGLv 0.5.14 results/lab/2026-09-26/E7F3_FLASH
C7+F4 w3 d2 181.13 @ 1.98 3 KVI off results/lab/2026-09-26/C7F4_FLASH
E3+CF w3 d2 180.74 @ 1.97 3 KVI off results/lab/2026-09-25/E3_FLASH
E1+CF w2 d2 178.16 @ 1.83 3 KVI off (the text reference for every later image) results/lab/2026-09-24/E1_W23_AB
C8+CF w2 d2 157.02 @ 1.82 1 FRSPEC=1 history
C9+B0 w8 d2 97.23 @ 2.22 4 KVONLY, older host history
0x1189+B0 w8 d2 (1 req) 57.65 @ 2.00 2 depth-N era history
GPU only (H200) w3 d5 449.68 @ 3.97 (fa3) / 446.58 @ 3.86 (fa4) 1 SGLang EAGLE4, same 50×32 results/gpu_width_depth_grid_20260924, HISTORY
GPU only w3 d2 392.23 @ 2.64 (fa4) / 401.39 @ 2.67 (fa3) 1 same cell as the pair's w3 d2 same

Headline: the best pair is A5+F4 at 268.01 tok/s, which is about 60% of GPU-only (~446–450). At matched w3 d2 the pair's accept (2.61) is 99% of the GPU's (2.64), so what's left is speed per step, not draft quality. Full dense tables:

Depth/width shape under KVI + FASTGLUE=4 (mt w3)

d A8+F4 A9b+F4 A5+F4 accept
2 220.13 230.88 257.72 2.61
3 203.88 213.51 247.16 3.20
5 173.75 180.68 221.21 (fold ON 225.80) 3.88
7 150.23 (running) 198.77 (fold ON 200.25) 4.35

2. Discoveries (each backed by the named result folder under results/lab/)

  1. KV import (KVI) is the largest single lever.
    • The GPU drafter's K/V is imported into the HEAD (Q5.10, one KV_LOAD item per draft), so the FPGA runs zero prime/catch-up launches.
    • On A8+F4 the draft tree is byte-identical to card priming, and priming is 9.4× faster (18 vs 170 ms for 639 rows).
    • Speed: 1.55–1.74× card-primed tok/s at w3 d2–7.
    • Accept is 97–103% of GPU: w3 d2 2.61 vs 2.64 on the GPU.
    • Folders: A8F4_KVI, KVI_PORT.
  2. The pair accept gap was the unprimed regime, not rounding. Without KVI the pair runs unprimed trees: accept 1.83–1.98 at d2 vs GPU 2.64. KVI closes the gap at no card cost.
  3. ATTN_PIPE (E3, and inside C7/A5) pays more as trees get deeper.
    • On E3, G5 attention drops 2.0× unprimed and 4.0–4.3× primed, with outputs identical.
    • Under KVI (every step primed), A5 beats A8 by +17% at d2, +21% at d3, +27% at d5 and +32% at d7. Humaneval shows +11% to +30%.
    • Folders: E3_FLASH, A5F4_FOLD.
  4. SEAM_WIDE 2 burst export (A5) gives +12–14% mt3 over C7 with KVI off: 202.53 / 202.22 vs 177.84 / 181.13.
  5. I12 ARRAY_FAST_TP (A9b) cuts HEAD G4 by exactly 9.74% in every census cell (HQ predicted −12.6%).
    • Every other phase, the step counts and accept are identical to A8.
    • Result: +3–5% over A8 under KVI. That is 3–4× smaller than the ATTN_PIPE gain, so the next HEAD should be A5 + I12, not A9b.
    • Folder: 2026-09-27/A9BF4_FLASH.
  6. G4 embedding fold (A5 + EMBFOLD v2.1) is correct and clean on silicon: host_miss 0, texts 50/50.
    • G4 items drop −8% to −35% by cell, but tok/s is neutral at d2: G4 is off the critical path there.
    • Deeper KVI trees gain +2.5% at d5 and +1.4% at d7.
    • Folders: A5F4_FOLD, A5F4_FOLD_V21.
  7. 256-lane GEMV (E7, A8) gains ~+7–9% over the 128-lane E1/E3 with KVI off (E7+F3 191.52 vs E1 178.16). Under KVI, A5's ATTN_PIPE outweighs it.
  8. Host glue is real time.
    • HOST_GLUE v3 (FASTGLUE=3, pipelined RPC) cuts −1.34 ms/step, giving +8.7% / +7.8% (A8 KVI off: 203.94 / 202.37).
    • v4 / v4.3 (FASTGLUE=4, composed with KVI) wins 3/3 rounds: +0.8% to +2.2% over KVI alone.
    • Folders: HOST_GLUE_V3, HOST_GLUE_V4_FIXED, HOST_GLUE_V43.
  9. TAIL F4 (G12X + TREE_P 32) cuts TAIL TREE cycles per step by exactly −33.2% vs F3 (C7F4_FLASH census).
  10. CHILD_ROPE_FIX (C7 and later) raises accept (w3 d2 1.97 → 1.98; primed d3 steps 12 → 11) without changing any text.
  11. ADAPTIVE_TREE (confidence stop).
    • The survivor-mass rule (dmin 1, θ −0.50) gives 246.8 tok/s (+11.5%). The held-out halves are +13.2% / +9.9%.
    • The launch rule (≥ +5% held-out) is MET, so HQ is building the F9 TREE_STOP TAIL.
    • Folder: ADAPTIVE_TREE.
  12. GPU drafter extend inside the pair step costs 0.79–0.97 ms per verify step (6–8% of the step). The earlier "≈12.6 ms/step" was a single 9.5 s warmup call dominating the average. Folder: GPU_EXTEND_IN_PAIR.
  13. Attention backend: keep the default (target flashinfer + draft fa4). A target-fa3 run is −5%; draft fa3 is a tie. Folder: A8F4_KVI_LEVERS.
  14. Correctness: every image from E1 to A9b is 50/50 text-identical to E1+CF at mt w2/w3 d2 (fixed chat date), and every KVI arm printed ### kvi: PROOF ok (0 FPGA prime/catch-up launches).

3. What failed or was retracted

4. Reproduce any number

Hard rules:

4.1 Flash (per card; HEAD = 61:00.0, TAIL = 41:00.0; both may be flashed in parallel)

Template: job_scripts/flash_a9b.sh. Pick the PDI and md5 from images/IMAGES.md.

PDI=images/oneflip_p2s_head_asgard_0xE4B711A5_66m_1f934249.pdi; MD5=1f934249815c340628f9c099bda288f5; DEV=61:00.0
[ "$(md5sum $PDI | cut -d' ' -f1)" = $MD5 ] && [ "$(od -An -tx4 -N4 $PDI | tr -d ' ')" = 92f7a516 ]
sudo -p "" env PCI_DEV=$DEV PREFLASH_ACK_QUARANTINE=1 bash scripts/R_scripts/preflash_gate.sh $PDI $MD5 < $PWF   # rc 0|3 = proceed
sudo -p "" ami_tool cfgmem_program -d $DEV -t primary -p 0 -y -q -i $PDI < $PWF                             # then COLD AC

Box1 runs every card job under /sanctuary/lab/card_run.sh (flock); a copy is in scripts/card_run.sh.

4.2 Bring-up + arm (loads all weights) after the cold AC

Template: job_scripts/bringup_a9bf4.sh; use bringup_a5f4.sh for the best pair. Set HEAD_ID/TAIL_ID to the flashed build ids.

bash scripts/R_scripts/restore_artifact_links.sh
lspci -d 10ee:                                                  # 4 functions
for D in 61:00.0 41:00.0; do sudo -p "" weights/loaders/v80_test_suite/03_setup_qdma.sh 4 $D < $PWF; done   # "Roundtrip test: PASS"
sudo -p "" -E scripts/W_scripts/pair_read_state.py < $PWF    # 0x100 must be the flashed HEAD id (else the old partition booted: 2nd cold AC)
HEAD_ID=0xe4b711a5 TAIL_ID=0xe4b711f4 PWF=$PWF bash scripts/W_scripts/postcoldac_arm_pair.sh      # FRONTIER GATE PASS + VERDICT PASS
HEAD_ID=0xe4b711a5 TAIL_ID=0xe4b711f4 sudo -p "" -E scripts/W_scripts/pair_state_check.py < $PWF   # STATE OK tail_width=8

postcoldac_arm_pair.sh runs reports/eagle4_drafter_m7/full_session_m7.py --osprey <boot_streams> --role head|tail --pci-dev … --stage-embed --do-scale for each card (copy: weights/loaders/eagle4_drafter_m7/). It loads:

Then it prints the [lanes] build-id … LANES= line and the 0x1c8 weight_drops=0 scale_drops=0 line. The arm log for every run is in results/pair_accuracy_20260921/arm_0x<head>_0x<tail>_*.log. Weights: weights/WEIGHTS.md (paths + md5). The FPGA boot streams are ~/eagle4_artifacts/osprey/boot_streams (small files copied to weights/boot_streams_small/). The SGLang target is models/Llama-3.2-3B-Instruct; the SGLang draft is ~/fetched_weights/bitnet_721/bf16_ckpt. Both serve from the weight_721/eaglefit vendored SGLang 0.5.14/0.5.16 in conda env eaglefit.

4.3 Tests (run from the worktree with scripts/W_scripts = $W/scripts; PWF = the 600 password file)

export PWF HEADBID=0xe4b711a5 TAILBID=0xe4b711f4 HEAD_ID=0xe4b711a5 TAIL_ID=0xe4b711f4 RACEFIX=1 PAIR=A5F4
R=<cleanup_EAGLE>; W=$R/.claude/worktrees/velvety-puzzling-blossom; FG=kits/HOST_GLUE/v4_3/host_glue_fastglue
BASEP="$W/scripts/tree_readback_racefix:$W/scripts/tree_width_cdt_graph_fix:$R/scripts/pair_sitecustomize_run"
# (a) standard protocol v1 (texts vs E1+CF, mt3, census, grid, primed, humaneval, csv) -- KVI off, fastglue off
bash scripts/W_scripts/pair_image_protocol.sh texts mt3 census grid primed hum csv
# (b) the 268 tok/s cell: KVI + FASTGLUE=4 v4.3, mt w2 d2 (and any W:D list)
KVI=1 ORDER="2:2 2:2 2:2" SGLV=0.5.16 ARM_ENV="PYTHONPATH=$FG:$W/scripts/kvi_host_shadow:$BASEP EAGLE4_PAIR_FASTGLUE=4" \
    bash scripts/W_scripts/pair_width_ab_kvi.sh          # every arm must print "### kvi: PROOF ok"
# (c) humaneval: add BENCH=~/haochen/eaglefit/inference/dataset/humaneval/question.jsonl ORDER="3:2 3:5 3:10 3:15"
# (d) census (1 request; primed adds EAGLE4_TREE_PRIME_PREFIX=1 EAGLE4_TREE_PRIME_KVONLY=1 EAGLE4_TREE_PRIME_MD=1)
ARM_ENV="PYTHONPATH=$BASEP" W=3 D=2 bash scripts/W_scripts/pair_census.sh
# (e) texts check against the E1+CF reference
python scripts/W_scripts/pair_cmp_perreq_texts.py results/pair_accuracy_20260921/wab20260924_223134_a1_w2d2_20260924_223134.perreq.jsonl <new perreq.jsonl>

Exact per-run commands are in:

EMBFOLD (A5 fold ON): job_scripts/a5_fold_v21.sh, plus the kit in kits/EMBFOLD/v2. The 9 GB table is not copied; its path and md5 are in weights/WEIGHTS.md.

5. Layout

dir contents
images/ 17 flashed PDIs, IMAGES.md (build id, md5, FPT magic, what each image is), inbox build notes
weights/ WEIGHTS.md, boot_streams_small/, model_configs/, loaders/ (arm/session-init/QDMA/relay/role-geometry + worktree overrides), EMBFOLD table md5
scripts/ R_scripts/ = the shared $R/scripts tree; W_scripts/ = the worktree $W/scripts tree (it wins on PYTHONPATH: KVI shadow, racefix, CDT fix, runners, protocol); card_run.sh
kits/ host kits as run: HOST_GLUE (v3/v4/v4_3), KVI_HOST, KVI_OVERLAP, ADAPTIVE_TREE, EMBFOLD (v1 host + v2/v2.1), HOST_PUSH v1, L256_HOST (v3/v4), E7_HOST
job_scripts/ every box1 driver script from this session (flash, bring-up, ladders, A/Bs) and their console logs
results/lab/ /sanctuary/lab/results/2026-09-24..27/*: every REPORT.md, raw .out, summaries, census, JSON
results/pair_accuracy_20260921/ every raw SGLang arm log, .perreq.jsonl, width_ab_*.txt summary and arm_*.log since 09-21
results/gpu_width_depth_grid_20260924/ GPU-only width/depth grid raw logs
results/HISTORY/ the dense CSVs + generator
MANIFEST.md every file: md5 + source path