fpga-archive-demo-box1-20261110 · EAGLE4 FPGA archive

Source reports/results/DEMO_SPEED_AUDIT_20261110.md · source md5 3cb6ce81392f22dd8b7a26b9ddea602a · archived 2026-11-18

Download PDFOpen PDFRaw markdown

Demo speed audit: per-step budget of the best stack and what can land in 14 h (2026-11-10 07:3xZ)

Program 1: the Llama-3.2-3B DRAFTER on the 4C1 split (2 ATT + 2 MLPV), target Llama-3.2-3B on the talon2 H200. Read-only audit: no card, GPU or demo access, no new runs. Demo deadline ~21:30Z 11-10 (USER_DIALOGUE_LOG.md:1995). Tags: [M] measured (artifact cited) · [D] derived by arithmetic from [M] · [E] estimate / projection · [S] sim only. Short paths: T/ = reports/results/TRACK1_RECORD_F3T_20261109/, R/ = reports/results/DRAFTER_SPEED_RESEARCH_20261107/, B/ = reports/results/BATCH_BUBBLES_20261109/.

0. Answer first

1. Waterfall of one speculative step, F3t 0xE4B713F9 + MLPV 0x1342, Llama w2 V20 K_C 64

1.1 Reconciliation against tok/s ÷ accept

d7 (record n1) d11 fa3 (r1) source
tok/s, accept, steps, time 831.14, 5.89, 3,329, 23.60 s 882.38, 6.90, 2,843, 22.23 s T/f3t_run/serve/SUMMARY_llama3b.tsv:2; R/F3T_CONFIG_SWEEP_20261109.md:80 [M]
steps/s = tok/s ÷ accept 141.1 127.9 [D]
whole-run ms/step 7.088 7.818 time ÷ steps [D] (sweep doc :206 gives 7.116 / 7.812 on the 9-round means)

1.2 Step waterfall (ms, % of whole-run step)

# term d7 ms d7 % d11 ms d11 % tag source
1 GPU verify forward (target, 20 tokens, bs 1, cuda events p50) 3.452 48.7 3.452 44.2 [M] d7 on F3t same boot; d11 assumed equal (same V20 graph) B/INTERLEAVE_BATCHVERIFY_SILICON_20261109.md:76
2 card tree, doorbell → tree published, run mean ~2.04 28.8 ~2.63 33.6 [D] full-depth p50 minus tree-stop saving, see 1.4 R/F3T_CONFIG_SWEEP_20261109.md:154-156
3 host chain, UpdateResult end → ATT doorbell 0.95 13.4 0.95 12.2 [M] on E2b with PQ+C, same host kit lineage; not re-profiled on F3t R/HOST_PRECARD_OPT.md:52,179
4 residual: verify end → UpdateResult end (accept / sampling / update) + unmodelled 0.25 3.6 0.33 4.3 [D] closes d7 by construction; d11 = 0.25 + 0.08 unexplained –
5 per-request fixed cost spread over steps (prefill, first GPU draft, first tree, request boundary, detok) 0.384 5.4 0.45 5.8 [D] 15.96 ms/request, see 1.5 T/f3t_run/serve/*n1_154551*.perreq.jsonl
sum 7.088 100 7.818 100 d7 closes by row 4; d11 predicted 7.736 before the 0.08

Cross-check from the record run's own counters [M]: the host idles 1.330 ms/step (d7) and 1.937 ms/step (d11) waiting for the card (join_wait_ns 3,991,624,512 / 3,000 and 4,841,374,691 / 2,500), after 0.506 / 0.515 ms of host work that overlaps the card (host_ahead_ns). The card is already done at the join on only 5 % of steps (151 / 3,000).

1.3 Inside the card tree (µs per tree, chain census, 40 full-depth trees, card-only)

Source: T/config_sweep/census/chain_llama3b_f3t_w2d7_kc64_185201.analysis.json.gz (md5 25e30681) and …w2d11_kc64_185202… (md5 5b11b7f6), key occupancy2 [M]. The d7 values match R/F3T_SILICON_20261109.md:35-44.

card phase d7 µs d7 % of p50 d11 µs
ATT G1 83.5 3.7 136.9
ATT G4 QKV 390.2 17.5 630.0
ATT G5 attention 523.4 23.4 841.3
ATT G6 o-proj + TP exchange (the ATT↔ATT QSFP hop sits here) 125.2 5.6 195.1
ATT G7G 351.1 15.7 561.3
ATT G7U 186.4 8.3 302.4
ATT SILU + LVL_EMBG 88.4 4.0 144.8
ATT INGEST + EXPORT (ATT↔MLPV seam) 73.2 3.3 144.5
ATT busy total 1,821.5 81.5 2,956.5
ATT PIPE_WAIT (parked on the MLPV) 255.5 – 246.3
MLPV G9 167.8 – 268.9
MLPV G11 + GLM + G12 kick 138.5 – 229.6
MLPV TREE 437.7 – 701.7
MLPV G8 exact rescore 343.2 – 551.8
MLPV INGEST + XCH + LVL_SEL/OUT 152.9 – 243.3
MLPV busy total (mostly hidden under the ATT) 1,240.0 – 1,995.2
joint both busy / ATT only / MLPV only (exposed) / both parked 1,035.2 / 795.7 / 216.9 / 38.3 9.7 exposed 1,798.1 / 1,178.1 / 210.5 / 35.9
census window 2,086.9 3,221.5
outside the window (issue, SoA read-back / DMA out) 146.9 6.6 162.6
issue → done p50 2,233.8 100 3,384.1

1.4 Tree-stop correction to the card term [D]

The census runs full-depth trees. In serving, theta −2.5 stops trees early:

1.5 Per-request fixed cost [D]

Per request, service time = difference of consecutive e2e_latency values. A least-squares fit of service = a + b × verify steps over requests 2–80 gives:

run a ms/request b ms/step R²
F3t d7 n1 / n2 15.96 / 15.55 6.696 / 6.717 0.991 / 0.987
F3t d11 fa3 r1 / r2 35.71 / 35.35 6.807 / 6.798 0.969 / 0.970
GPU-only d7 fa3 / d9 / d11 7.80 / 8.30 / 8.11 7.032 / 7.686 / 8.318 0.997 / 0.997 / 0.998

1.6 Inside the 0.95 ms host chain (µs, p50, E2b + PQ profile; C then removed ~54)

Source R/HOST_PRECARD_OPT.md:38-52,124-132 [M].

class items µs
DMA in KV_LOAD in the card process 229 (4 pwrites ≈ 140) + root FASTIN to both ATT 141 370
GPU draft-side launch capture + EXTKV extend launch (113) + prepare_graph 229 → 210 with C
host glue (Python / IPC) KV gather 44, guards + plan 57, client KV pack/send 121, socket 13, image recv 46 → 34, stage/cfg/arm 19 + 17 + 40, poison 6, doorbell 8 ~360
root image graph replay (parallel path that ties with the KV path) 175 (not additive)

DMA out is not separately profiled; it sits in the 147 µs outside the census window (1.3).

2. Amdahl table

Bases [M]: humaneval-80 d7 828.06 (9-round mean) and d11 fa3 883.12; demo set (15 code prompts, 256 tokens) d7 953.0 and d11 1,059.3 (T/PER_PROMPT_BENCH_20261109.md:109-111). For card levers, Δstep = κ × Δtree. Two values of κ were measured: 1.0 (E2b vs 0x1340, R/MLPV_AND_STEP_BREAKDOWN_E2b.md:70) and 0.51 (E2b → F3t, R/SEVENCARD_PROJECTION_20261109.md:52). Both are shown where it matters.

lever evidence humaneval-80 demo set status time to land risk
depth 11 (with fa3) R/F3T_CONFIG_SWEEP_20261109.md:80,83 883.12 (+6.65 %) 1,059 (+11 %) MEASURED host-only, 0 h; PERSIST must be armed ≥ 11 before the first AP_START texts differ from d7 on 3/80; fa3 rounds identical
depth 8 / 9 / 10 :50,52,56 850 / 861 / 870 – MEASURED 0 h –
depth 12 / 13 / 15 :58-60 874.7 / 873.8 / 870.1 (−1.0 to −1.5 % vs d11) not measured MEASURED 0 h loss
width 3 (w4 skipped) :73-74 767 (−13 % at d11) – MEASURED – loss
verify tokens V16 / V24 / V27 :45-47,62-63 d7 828.6 / 799 / 800; d11 853 / 859 – MEASURED – loss or level
theta −2.0 / −3.0 :67-68 877 / 868 at d11 – MEASURED – level / loss
K_C 32 (128) :75-76; R/FAILED_DIRECTIONS_20261109.md:9 885.70 (+0.5 %, one round); E2b A/B +0.82 %; K_C 128 −4.9 % – MEASURED, inside noise host-only, 0 h changes 2–4/80 texts
FR-Spec 64K / 32K / 16K :141-143 −1.2 / −9.5 / −20.3 % – MEASURED – loss
FIRST_DRAFT_GPU already on: FIRSTGPU=1 at …154551.log:3 0 0 in the base – –
fa3 draft backend :170 +0.4 % at d11, neutral at d7 – MEASURED 0 h makes d11 deterministic
FASTGLUE variants / HP letters FASTGLUE 4 + P, Q, C on; R, K, E, W, A rejected (R/HOST_PRECARD_OPT.md:66-69,126) 0 0 MEASURED – –
PINPAR 1 :69-70 822 / 876 (no gain) – MEASURED – –
KVONC host fix +1.52 % on 0x130C + 0x128B at 749.57 (reports/results/FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:1414-1415) not applicable: the F3t regmap row has no kvoncard key (T/f3t_kit/patch_13f9_host.py:46); needs a KV_ONCARD image – MEASURED on another image a build not in 14 h
CUDA-graph / draft-graph warm target verify graph already captured (…154551.log:168-175); GRAPHWARM refuted, it changes accept (reports/results/FOURCARD_DESIGN_20261016/STEP_SPLIT_20261019.md:106) 0 0 MEASURED – –
CPU governor +0.65 %, below the bar (:171) ~+0.6 % – MEASURED 11-06 system-wide setting on a shared host low gain
GPU clocks / thermal talon2: throttled 0, max 69 °C in every row (T/f3t_run/serve/SUMMARY_llama3b.tsv:2-3); box1 H200 throttled at 87 °C on Qwen, SM 1,785 → 360–510 MHz (USER_DIALOGUE_LOG.md:1989) protects the number, adds nothing same MEASURED 0 h on box1, a hot GPU can cut the rate several-fold
prompt choice T/PER_PROMPT_BENCH_20261109.md:50-84 – d7 896–1,042; d11 943–1,193; mt_bench prose 404–497 MEASURED 0 h prose prompts halve the number
max_new_tokens 512 on code prompts decode-only rate d11 1,242–1,308 vs e2e 1,111–1,193 (:70-73) – ≈ +4–5 % [E] (halves the per-request share) estimate 0.5 h test early EOS; accept over a longer output not measured
per-request cost (FPGA 16 ms vs GPU 8 ms) 1.5 above ≤ +2.8 % [E] ≤ ~+4 % [E] estimate host dev, 3–6 h untested host change before a demo
2-stream PINGPONG / MPS 1.55× was the 2-card pair (memory feedback_batch1_latency_first_pingpong_is_batch2); on F3t, N2 without MPS = 221 (B/INTERLEAVE_BATCHVERIFY_SILICON_20261109.md:53) below single-stream – MEASURED loss without MPS; MPS arm not run hours –
batched verify BV4 + HBMREPLAY (aggregate) 730 / 717 vs 665 on the same launcher (:82); that launcher lacks the record levers 730 < 831 today; ~905 [E] if the record levers were ported – MEASURED (aggregate only) host dev, hours 6–8/80 texts change
G5_OVL in F3t (R/F3T_SILICON_20261109.md:14) 0 more 0 in the base – –
LANES 512 ATT in F3t (F3t = D9 L512, R/F3T_SILICON_20261109.md:4) 0 more 0 in the base – –
R3_EARLY −75 µs/tree [calc] (R/BOTTLENECK_TABLE.md:84); silicon F0b: no gain shown at equal host config, one unexplained accept-1.04 boot (R/FAILED_DIRECTIONS_20261109.md:11) +0.5 % (κ 0.51) to +1.1 % (κ 1) [E] same estimate a build on the F3t base not in 14 h
ATT TP4, 6 cards: B4 0x13B4 (GO, flashing now) + MLPV 0x134D R/SEVENCARD_PROJECTION_20261109.md:90; adjudication R/SEVENCARD_B1_B5_ADJUDICATION_20261110.md:11-13 d7 869 / d11 948 (κ 1); 848 / 915 (κ 0.51) not projected SIM only; no TP4 silicon census exists flash in progress + manual cold AC + first 6-card bring-up, 4–8 h [E] high: 0x134D PDI not written at 07:00Z (routed +0.001 / +0.010), new host kit, cables unverified
ATT TP4 B5 0x13B5 (SIBW W2) same doc :89 d7 885 / d11 979 (κ 1); 856 / 930 (κ 0.51) – SIM only as B4 as B4, plus hold exactly at the bar
G6_EARLY_ISSUE root item 11,152 → 9,670 cycles in TP4 xsim (reports/results/QWEN27B_DRAFTER_20261020/G6_EARLY_ISSUE_20261109.md:19) not on any Llama 4C1 image – SIM only, Q5120 lineage a build not in 14 h
MLPV L512 0x1351 died at place, 19,133 SLLs vs 18,870 (…4C1_DELIVERY_TRACKER.md:1502) – – dead – –
MLPV TREE_P 64 0x135C died at place (…4C1_DELIVERY_TRACKER.md:1531) – – dead so far – –
Q5120 drafter GPU-only 196.61 @ 3.976 (…4C1_DELIVERY_TRACKER.md:1516) vs BitNet Qwen drafter 249.62 @ 7.14 on F3t (R/F3T_SILICON_20261109.md:22); Q5120 MLPV boot refused at 07:02Z (…TRACKER.md:1535) other program (Qwen target); slower today – MEASURED GPU-only – does not raise the Llama number

3. What caps the number

what goes to zero d7 tok/s d11 tok/s how
card wait only (the measured host idle, 1.330 / 1.937 ms) 1,023 1,173 accept ÷ (step − join wait) [D]
whole card tree (2.04 / 2.63 ms) 1,167 1,330 upper bound; needs the 0.5 ms host prologue hidden as well [D]
host chain + residual (1.20 ms) 1,001 1,043 [D]
host chain + residual + per-request 1,071 1,120 [D]
GPU verify 1,620 1,580 [D]
card and host both (verify + per-request left) 1,535 – [D]

Reference: best GPU-only the same day is 835 (d9 fa3, R/F3T_CONFIG_SWEEP_20261109.md:120). Its marginal step is 7.03 ms at d7, so the GPU drafter + host costs 3.58 ms against the cards' 3.25 ms [D from 1.5].

4. Top 5 for the next 14 h, ranked by (gain × probability) ÷ hours

One-flash actions

  1. Put F3t 0xE4B713F9 (md5 97d8445c) back on 17/3d and 0x1342 (3e82ea1d) on 50/63, one cold AC, arm depth 15 before the first AP_START. This is the only route to the measured 883 / 1,059. Probability ~0.95 (same PDIs, kits and run dirs as 11-09: R/F3T_SILICON_20261109.md:62-86). ~2 h including the manual power cycle. Set a hard cutoff for starting it; ~17:30Z leaves 4 h.
  2. Time-box the 6-card TP4 attempt (B4 now flashing; B5 is the faster projection). Projected gain over F3t at d11: +3.6 % (κ 0.51) to +7.4 % (κ 1) for B4, sim only. It needs MLPV 0x134D on 50/63, which had no PDI at 07:00Z. Probability of a served, golden-checked number before the cutoff: low. If it is not serving by the cutoff, do action 1.

No-flash, host-only actions (on F3t once it is back)

  1. Serve at w2 d11, fa3, V20, K_C 64, theta −2.5. +6.65 % on humaneval-80 and +11 % on the demo prompts, measured, 0 h. Exact command: R/F3T_CONFIG_SWEEP_20261109.md:181-183.
  2. Use the high-accept code prompts (even_odd_palindrome, sort_array, is_sorted, pluck, words_in_sentence): 1,111–1,193 tok/s at d11, measured, 0 h. Avoid prose prompts (404–497).
  3. Test max_new_tokens 512 on those prompts to dilute the 18–26 ms per-request cost: ≈ +4–5 % [E], 0.5 h, unmeasured. Keep 256 if a prompt stops early or accept falls.

Not worth the hours: width 3, V ≠ 20, FR-Spec, K_C 128, PINPAR, GRAPHWARM, the governor, KVONC (wrong image), R3_EARLY / G6_EARLY_ISSUE (need builds), and the multi-stream modes (below single-stream today).

5. Not verified