# EAGLE4 pair — tested archive (box1, snapshot 2026-09-27 ~02:10)

This folder is the reproducibility archive for every HEAD/TAIL V80 pair image tested on box1 in the 2026-09-21..27 silicon campaign.
It holds:
- every flashed PDI (`images/`);
- the exact weights, with md5 (`weights/`), plus the loaders that put them on the cards;
- every test script, including the host kits;
- the raw logs and outputs.

Everything here is a **copy**; the sources stay where they were (see `MANIFEST.md`). To refresh after new runs, re-run
`/sanctuary/lab/tested_archive_build/copy_archive.sh`. It is create-only (`cp -n`), so it adds new files and never overwrites or deletes.
The A9b+F4 ladder was still running at snapshot time; its partial results are in `results/lab/2026-09-27/`.

**Setup of every pair number below:**
- System: Llama-3.2-3B-Instruct target on the H200 (SGLang). The drafter is EAGLE4 **bitnet_721 2-bit (OSPREY)**, split across two V80 cards: HEAD 61:00.0 (G4 QKV, G5 attention, G6, G7) and TAIL 41:00.0 (G9, GLM, tree/top-K).
- Benchmark: mt_bench 50 requests × 32 new tokens.
- Settings: fa4 draft attention, K_C 64 shortlist, RACEFIX=1, and the chat date pinned.
- tok/s is the SGLang `Throughput` line and accept is `Avg Accept Length`.

## 1. Results — best tok/s per image pair (mt_bench 50×32)

| pair (HEAD+TAIL) | best cell | tok/s @ accept | n (rounds) | config | source |
|---|---|---|---|---|---|
| **A5+F4** | **w2 d2** | **268.01 @ 2.55** (3-round 264.29; fold ON 264.83) | 1 (+3+3) | KVI=1 + FASTGLUE=4 (HOST_GLUE v4.3), SGLv 0.5.16 | `results/pair_accuracy_20260921/width_ab_20260926_185929.txt`, `results/lab/2026-09-26/ADAPTIVE_TREE`, `A5F4_FOLD_V21` |
| A5+F4 | w1 d2 / w3 d2 | 259.45 @ 2.32 / 257.72 @ 2.61 | 1 | same | `width_ab_20260926_193006.txt` |
| A9b+F4 | w2 d2 | 241.56 @ 2.55 (241.35 / 244.29 / 239.03) | 3 | same | `results/lab/2026-09-27/A9BF4_FLASH/REPORT.md`, `width_ab_20260927_020147.txt` |
| A8+F4 | w2 d2 | 234.62 @ 2.55 (234.56 / 234.71 / 234.60) | 3 | same | `results/lab/2026-09-26/HOST_GLUE_V4_FIXED` |
| A8+F4 | w2 d2, KVI off | 203.94 (F3 fastglue) / 188.34 (plain) | 3 | FASTGLUE=3 / none, SGLv 0.5.14 | `HOST_GLUE_V3`, `A8F4_FLASH` |
| A5+F4 | w2 d2, KVI off | 202.53 @ 1.83 | 3 | protocol v1 | `results/lab/2026-09-26/A5F4_PROTOCOL` |
| E7+F3 | w2 d2 | 191.52 @ 1.83 | 3 | KVI off, SGLv 0.5.14 | `results/lab/2026-09-26/E7F3_FLASH` |
| C7+F4 | w3 d2 | 181.13 @ 1.98 | 3 | KVI off | `results/lab/2026-09-26/C7F4_FLASH` |
| E3+CF | w3 d2 | 180.74 @ 1.97 | 3 | KVI off | `results/lab/2026-09-25/E3_FLASH` |
| E1+CF | w2 d2 | 178.16 @ 1.83 | 3 | KVI off (the text reference for every later image) | `results/lab/2026-09-24/E1_W23_AB` |
| C8+CF | w2 d2 | 157.02 @ 1.82 | 1 | FRSPEC=1 | history |
| C9+B0 | w8 d2 | 97.23 @ 2.22 | 4 | KVONLY, older host | history |
| 0x1189+B0 | w8 d2 (1 req) | 57.65 @ 2.00 | 2 | depth-N era | history |
| **GPU only (H200)** | w3 d5 | **449.68 @ 3.97 (fa3) / 446.58 @ 3.86 (fa4)** | 1 | SGLang EAGLE4, same 50×32 | `results/gpu_width_depth_grid_20260924`, HISTORY |
| GPU only | w3 d2 | 392.23 @ 2.64 (fa4) / 401.39 @ 2.67 (fa3) | 1 | same cell as the pair's w3 d2 | same |

**Headline:** the best pair is **A5+F4 at 268.01 tok/s**, which is about **60% of GPU-only** (~446–450). At matched w3 d2 the pair's accept (2.61) is **99%** of the GPU's (2.64), so what's left is speed per step, not draft quality.
Full dense tables:
- `results/HISTORY/ALL_RESULTS.csv` (560 runs) and `BEST_BY_CELL.csv` (311 cells), regenerated by `results/HISTORY/build_history.py`. It covers runs up to 2026-09-26 20:31; later runs are in the lab REPORTs.
- Per-image protocol CSVs: `results/lab/*/*_PROTOCOL/*_cells.csv`.

### Depth/width shape under KVI + FASTGLUE=4 (mt w3)
| d | A8+F4 | A9b+F4 | A5+F4 | accept |
|---|---|---|---|---|
| 2 | 220.13 | 230.88 | 257.72 | 2.61 |
| 3 | 203.88 | 213.51 | 247.16 | 3.20 |
| 5 | 173.75 | 180.68 | 221.21 (fold ON 225.80) | 3.88 |
| 7 | 150.23 | (running) | 198.77 (fold ON 200.25) | 4.35 |
- humaneval w3, A5+F4: d2/5/10/15 = 153.21 / 153.03 / 144.67 / 132.19 (accept 2.89 / 5.14 / 7.62 / 8.99). A8 at the same depths: 137.64 / 128.20 / 116.45 / 101.64.
- A8+F4 KVI grid (w1–7 × d2–7): `results/lab/2026-09-26/A8F4_KVI_GRID/REPORT.md`. The sweet spot is w2 d2; narrower and shallower is faster at every depth.

## 2. Discoveries (each backed by the named result folder under `results/lab/`)
1. **KV import (KVI) is the largest single lever.**
   - The GPU drafter's K/V is imported into the HEAD (Q5.10, one KV_LOAD item per draft), so the FPGA runs zero prime/catch-up launches.
   - On A8+F4 the draft tree is **byte-identical** to card priming, and priming is **9.4× faster** (18 vs 170 ms for 639 rows).
   - Speed: **1.55–1.74×** card-primed tok/s at w3 d2–7.
   - Accept is **97–103% of GPU**: w3 d2 2.61 vs 2.64 on the GPU.
   - Folders: `A8F4_KVI`, `KVI_PORT`.
2. **The pair accept gap was the unprimed regime, not rounding.** Without KVI the pair runs unprimed trees: accept 1.83–1.98 at d2 vs GPU 2.64. KVI closes the gap at no card cost.
3. **ATTN_PIPE (E3, and inside C7/A5) pays more as trees get deeper.**
   - On E3, G5 attention drops 2.0× unprimed and 4.0–4.3× primed, with outputs identical.
   - Under KVI (every step primed), A5 beats A8 by +17% at d2, +21% at d3, +27% at d5 and +32% at d7. Humaneval shows +11% to +30%.
   - Folders: `E3_FLASH`, `A5F4_FOLD`.
4. **SEAM_WIDE 2 burst export (A5)** gives +12–14% mt3 over C7 with KVI off: 202.53 / 202.22 vs 177.84 / 181.13.
5. **I12 ARRAY_FAST_TP (A9b)** cuts HEAD G4 by **exactly 9.74%** in every census cell (HQ predicted −12.6%).
   - Every other phase, the step counts and accept are identical to A8.
   - Result: +3–5% over A8 under KVI. That is 3–4× smaller than the ATTN_PIPE gain, so **the next HEAD should be A5 + I12**, not A9b.
   - Folder: `2026-09-27/A9BF4_FLASH`.
6. **G4 embedding fold (A5 + EMBFOLD v2.1)** is correct and clean on silicon: host_miss 0, texts 50/50.
   - G4 items drop −8% to −35% by cell, but tok/s is **neutral at d2**: G4 is off the critical path there.
   - Deeper KVI trees gain **+2.5% at d5 and +1.4% at d7**.
   - Folders: `A5F4_FOLD`, `A5F4_FOLD_V21`.
7. **256-lane GEMV (E7, A8)** gains ~+7–9% over the 128-lane E1/E3 with KVI off (E7+F3 191.52 vs E1 178.16). Under KVI, A5's ATTN_PIPE outweighs it.
8. **Host glue is real time.**
   - HOST_GLUE v3 (FASTGLUE=3, pipelined RPC) cuts **−1.34 ms/step**, giving +8.7% / +7.8% (A8 KVI off: 203.94 / 202.37).
   - v4 / v4.3 (FASTGLUE=4, composed with KVI) wins 3/3 rounds: +0.8% to +2.2% over KVI alone.
   - Folders: `HOST_GLUE_V3`, `HOST_GLUE_V4_FIXED`, `HOST_GLUE_V43`.
9. **TAIL F4 (G12X + TREE_P 32)** cuts TAIL TREE cycles per step by exactly −33.2% vs F3 (`C7F4_FLASH` census).
10. **CHILD_ROPE_FIX (C7 and later)** raises accept (w3 d2 1.97 → 1.98; primed d3 steps 12 → 11) without changing any text.
11. **ADAPTIVE_TREE (confidence stop).**
    - The survivor-mass rule (dmin 1, θ −0.50) gives 246.8 tok/s (+11.5%). The held-out halves are +13.2% / +9.9%.
    - The launch rule (≥ +5% held-out) is MET, so HQ is building the F9 TREE_STOP TAIL.
    - Folder: `ADAPTIVE_TREE`.
12. **GPU drafter extend inside the pair step** costs 0.79–0.97 ms per verify step (6–8% of the step). The earlier "≈12.6 ms/step" was a single 9.5 s warmup call dominating the average. Folder: `GPU_EXTEND_IN_PAIR`.
13. **Attention backend:** keep the default (target flashinfer + draft fa4). A target-fa3 run is −5%; draft fa3 is a tie. Folder: `A8F4_KVI_LEVERS`.
14. **Correctness:** every image from E1 to A9b is **50/50 text-identical to E1+CF** at mt w2/w3 d2 (fixed chat date), and every KVI arm printed `### kvi: PROOF ok` (0 FPGA prime/catch-up launches).

## 3. What failed or was retracted
- **"Pair ≈ 90% of H200"**: RETRACTED. The matched comparison shows ~37% (Aug) and today ~60% (268 vs ~446).
- **The Block 4 "0.139 ms / 4.73× GPU" headline is FALSE.** The HLS-era bench read a sticky `ap_done`.
- **HOST_GLUE v2:** F2 tied A (187.69 vs 188.34), and F1 lost 3/3; both were dropped. v4's first run was invalid because the runner forced OVERLAP_CATCHUP=0; this was fixed in `pair_width_ab_kvi.sh` (commit f536bad6).
- **EMBFOLD v1:** the host hook never attached (host_miss 13,172), because the serving path calls only `setup()`. v2.1 fixed it.
- **KVI canary crashes** (`_last_progress` AttributeError; the `blk==32` check vs the wide loader's blk=0) were fixed in `kvi_host_shadow/kvi_host.py`.
- **HEAD_PAR=8 (0x1190) accept 1.00** is CONFOUNDED with R2 PIPE_EXP_MULTI. HP8 attention itself is bit-exact (539/539).
- **The FA4 tree-draft on the H200 is broken:** accept 2.74 vs fa3 3.86 at deep trees. The GPU reference can be wrong, so use matched backends.
- **Census tok/s** (1 request) is NOT comparable to 50×32 throughput.

## 4. Reproduce any number
Hard rules:
- Flash `-p 0` only, a final FPT PDI only (magic `92f7a516`), detached.
- Never FLR/SBR/device_boot/PCIe remove.
- A cold AC (physical power cycle) is needed after every flash.
- sudo reads a mode-600 password file on stdin (`sudo -p "" … < $PWF`). No password is stored anywhere in this archive.

### 4.1 Flash (per card; HEAD = 61:00.0, TAIL = 41:00.0; both may be flashed in parallel)
Template: `job_scripts/flash_a9b.sh`. Pick the PDI and md5 from `images/IMAGES.md`.
```bash
PDI=images/oneflip_p2s_head_asgard_0xE4B711A5_66m_1f934249.pdi; MD5=1f934249815c340628f9c099bda288f5; DEV=61:00.0
[ "$(md5sum $PDI | cut -d' ' -f1)" = $MD5 ] && [ "$(od -An -tx4 -N4 $PDI | tr -d ' ')" = 92f7a516 ]
sudo -p "" env PCI_DEV=$DEV PREFLASH_ACK_QUARANTINE=1 bash scripts/R_scripts/preflash_gate.sh $PDI $MD5 < $PWF   # rc 0|3 = proceed
sudo -p "" ami_tool cfgmem_program -d $DEV -t primary -p 0 -y -q -i $PDI < $PWF                             # then COLD AC
```
Box1 runs every card job under `/sanctuary/lab/card_run.sh` (flock); a copy is in `scripts/card_run.sh`.

### 4.2 Bring-up + arm (loads all weights) after the cold AC
Template: `job_scripts/bringup_a9bf4.sh`; use `bringup_a5f4.sh` for the best pair. Set HEAD_ID/TAIL_ID to the flashed build ids.
```bash
bash scripts/R_scripts/restore_artifact_links.sh
lspci -d 10ee:                                                  # 4 functions
for D in 61:00.0 41:00.0; do sudo -p "" weights/loaders/v80_test_suite/03_setup_qdma.sh 4 $D < $PWF; done   # "Roundtrip test: PASS"
sudo -p "" -E scripts/W_scripts/pair_read_state.py < $PWF    # 0x100 must be the flashed HEAD id (else the old partition booted: 2nd cold AC)
HEAD_ID=0xe4b711a5 TAIL_ID=0xe4b711f4 PWF=$PWF bash scripts/W_scripts/postcoldac_arm_pair.sh      # FRONTIER GATE PASS + VERDICT PASS
HEAD_ID=0xe4b711a5 TAIL_ID=0xe4b711f4 sudo -p "" -E scripts/W_scripts/pair_state_check.py < $PWF   # STATE OK tail_width=8
```
`postcoldac_arm_pair.sh` runs `reports/eagle4_drafter_m7/full_session_m7.py --osprey <boot_streams> --role head|tail --pci-dev … --stage-embed --do-scale`
for each card (copy: `weights/loaders/eagle4_drafter_m7/`). It loads:
- the M1c qweights into URAM and the M9 scales into BRAM;
- the gammas;
- lm_head_full_fp16 and embed_tokens_full_bf16 into HBM (read back and md5-checked: `133c0866…`).

Then it prints the `[lanes] build-id … LANES=` line and the 0x1c8 `weight_drops=0 scale_drops=0` line. The arm log for every run is in `results/pair_accuracy_20260921/arm_0x<head>_0x<tail>_*.log`.
Weights: `weights/WEIGHTS.md` (paths + md5). The FPGA boot streams are `~/eagle4_artifacts/osprey/boot_streams` (small files copied to `weights/boot_streams_small/`).
The SGLang target is `models/Llama-3.2-3B-Instruct`; the SGLang draft is `~/fetched_weights/bitnet_721/bf16_ckpt`. Both serve from the `weight_721/eaglefit` vendored SGLang 0.5.14/0.5.16 in conda env `eaglefit`.

### 4.3 Tests (run from the worktree with `scripts/W_scripts` = `$W/scripts`; `PWF` = the 600 password file)
```bash
export PWF HEADBID=0xe4b711a5 TAILBID=0xe4b711f4 HEAD_ID=0xe4b711a5 TAIL_ID=0xe4b711f4 RACEFIX=1 PAIR=A5F4
R=<cleanup_EAGLE>; W=$R/.claude/worktrees/velvety-puzzling-blossom; FG=kits/HOST_GLUE/v4_3/host_glue_fastglue
BASEP="$W/scripts/tree_readback_racefix:$W/scripts/tree_width_cdt_graph_fix:$R/scripts/pair_sitecustomize_run"
# (a) standard protocol v1 (texts vs E1+CF, mt3, census, grid, primed, humaneval, csv) -- KVI off, fastglue off
bash scripts/W_scripts/pair_image_protocol.sh texts mt3 census grid primed hum csv
# (b) the 268 tok/s cell: KVI + FASTGLUE=4 v4.3, mt w2 d2 (and any W:D list)
KVI=1 ORDER="2:2 2:2 2:2" SGLV=0.5.16 ARM_ENV="PYTHONPATH=$FG:$W/scripts/kvi_host_shadow:$BASEP EAGLE4_PAIR_FASTGLUE=4" \
    bash scripts/W_scripts/pair_width_ab_kvi.sh          # every arm must print "### kvi: PROOF ok"
# (c) humaneval: add BENCH=~/haochen/eaglefit/inference/dataset/humaneval/question.jsonl ORDER="3:2 3:5 3:10 3:15"
# (d) census (1 request; primed adds EAGLE4_TREE_PRIME_PREFIX=1 EAGLE4_TREE_PRIME_KVONLY=1 EAGLE4_TREE_PRIME_MD=1)
ARM_ENV="PYTHONPATH=$BASEP" W=3 D=2 bash scripts/W_scripts/pair_census.sh
# (e) texts check against the E1+CF reference
python scripts/W_scripts/pair_cmp_perreq_texts.py results/pair_accuracy_20260921/wab20260924_223134_a1_w2d2_20260924_223134.perreq.jsonl <new perreq.jsonl>
```
Exact per-run commands are in:
- `job_scripts/`: `a5f4_ladder.sh`, `a5_fold_kvi.sh`, `a5_fold_v21.sh`, `a5_fold_deep.sh`, `a8f4_*.sh`, `v42_*.sh`, `a9bf4_ladder.sh` and the other job scripts;
- each `width_ab_*.txt` header, which records ORDER, SGLV, RACEFIX, KVI and ARM_ENV.

EMBFOLD (A5 fold ON): `job_scripts/a5_fold_v21.sh`, plus the kit in `kits/EMBFOLD/v2`. The 9 GB table is not copied; its path and md5 are in `weights/WEIGHTS.md`.

## 5. Layout
| dir | contents |
|---|---|
| `images/` | 17 flashed PDIs, `IMAGES.md` (build id, md5, FPT magic, what each image is), inbox build notes |
| `weights/` | `WEIGHTS.md`, `boot_streams_small/`, `model_configs/`, `loaders/` (arm/session-init/QDMA/relay/role-geometry + worktree overrides), EMBFOLD table md5 |
| `scripts/` | `R_scripts/` = the shared `$R/scripts` tree; `W_scripts/` = the worktree `$W/scripts` tree (it wins on PYTHONPATH: KVI shadow, racefix, CDT fix, runners, protocol); `card_run.sh` |
| `kits/` | host kits as run: HOST_GLUE (v3/v4/v4_3), KVI_HOST, KVI_OVERLAP, ADAPTIVE_TREE, EMBFOLD (v1 host + v2/v2.1), HOST_PUSH v1, L256_HOST (v3/v4), E7_HOST |
| `job_scripts/` | every box1 driver script from this session (flash, bring-up, ladders, A/Bs) and their console logs |
| `results/lab/` | `/sanctuary/lab/results/2026-09-24..27/*`: every REPORT.md, raw `.out`, summaries, census, JSON |
| `results/pair_accuracy_20260921/` | every raw SGLang arm log, `.perreq.jsonl`, `width_ab_*.txt` summary and `arm_*.log` since 09-21 |
| `results/gpu_width_depth_grid_20260924/` | GPU-only width/depth grid raw logs |
| `results/HISTORY/` | the dense CSVs + generator |
| `MANIFEST.md` | every file: md5 + source path |
