fpga-archive-demo-box1-20261110 · EAGLE4 FPGA archive
Demo speed audit: per-step budget of the best stack and what can land in 14 h (2026-11-10 07:3xZ)
Program 1: the Llama-3.2-3B DRAFTER on the 4C1 split (2 ATT + 2 MLPV), target Llama-3.2-3B on the talon2 H200.
Read-only audit: no card, GPU or demo access, no new runs. Demo deadline ~21:30Z 11-10 (USER_DIALOGUE_LOG.md:1995).
Tags: [M] measured (artifact cited) · [D] derived by arithmetic from [M] · [E] estimate / projection · [S] sim only.
Short paths: T/ = reports/results/TRACK1_RECORD_F3T_20261109/, R/ = reports/results/DRAFTER_SPEED_RESEARCH_20261107/, B/ = reports/results/BATCH_BUBBLES_20261109/.
0. Answer first
- The binding term is GPU verify: 3.45 ms = 49 % of the 7.09 ms step. The card tree is ~2.0 ms (29 %), host glue ~1.2 ms (17 %), per-request cost 0.38 ms (5 %).
- The best measured numbers need no new image: F3t + 0x1342 at w2 d11, fa3, V20, K_C 64, theta −2.5 = 883 tok/s on humaneval-80 and 1,059 on the 15 demo prompts (top prompts 1,181–1,193).
- The record stack is not on any card right now. talon2 17/3d were re-flashed to Q5120 at 06:3xZ and are being flashed to TP4 B4 since 07:07Z; 50/63 hold the Q5120 MLPV 0x1349 (
USER_DIALOGUE_LOG.md:1990,1994). box1 holds E2b + 0x1342 (reports/results/BOX1_DEMO_TEMPS_HUMANEVAL_20261110.md:5-12). Getting 883 / 1,059 back needs F3t → 17/3d and 0x1342 → 50/63 plus one cold AC, and the talon2 PDU CLI is broken, so that cold AC is manual (USER_DIALOGUE_LOG.md:1992). I did not verify card state myself; this is from the logs. - Ceilings [D]: card time → 0 gives 1,023–1,167 tok/s at d7 (1,173–1,330 at d11). Host glue → 0 gives ~1,000 (d7) / ~1,043 (d11). Verify → 0 gives ~1,600. No card or host lever available in 14 h is worth more than the d11 + prompt-choice config already measured.
1. Waterfall of one speculative step, F3t 0xE4B713F9 + MLPV 0x1342, Llama w2 V20 K_C 64
1.1 Reconciliation against tok/s ÷ accept
| d7 (record n1) | d11 fa3 (r1) | source | |
|---|---|---|---|
| tok/s, accept, steps, time | 831.14, 5.89, 3,329, 23.60 s | 882.38, 6.90, 2,843, 22.23 s | T/f3t_run/serve/SUMMARY_llama3b.tsv:2; R/F3T_CONFIG_SWEEP_20261109.md:80 [M] |
| steps/s = tok/s ÷ accept | 141.1 | 127.9 | [D] |
| whole-run ms/step | 7.088 | 7.818 | time ÷ steps [D] (sweep doc :206 gives 7.116 / 7.812 on the 9-round means) |
1.2 Step waterfall (ms, % of whole-run step)
| # | term | d7 ms | d7 % | d11 ms | d11 % | tag | source |
|---|---|---|---|---|---|---|---|
| 1 | GPU verify forward (target, 20 tokens, bs 1, cuda events p50) | 3.452 | 48.7 | 3.452 | 44.2 | [M] d7 on F3t same boot; d11 assumed equal (same V20 graph) | B/INTERLEAVE_BATCHVERIFY_SILICON_20261109.md:76 |
| 2 | card tree, doorbell → tree published, run mean | ~2.04 | 28.8 | ~2.63 | 33.6 | [D] full-depth p50 minus tree-stop saving, see 1.4 | R/F3T_CONFIG_SWEEP_20261109.md:154-156 |
| 3 | host chain, UpdateResult end → ATT doorbell | 0.95 | 13.4 | 0.95 | 12.2 | [M] on E2b with PQ+C, same host kit lineage; not re-profiled on F3t | R/HOST_PRECARD_OPT.md:52,179 |
| 4 | residual: verify end → UpdateResult end (accept / sampling / update) + unmodelled | 0.25 | 3.6 | 0.33 | 4.3 | [D] closes d7 by construction; d11 = 0.25 + 0.08 unexplained | – |
| 5 | per-request fixed cost spread over steps (prefill, first GPU draft, first tree, request boundary, detok) | 0.384 | 5.4 | 0.45 | 5.8 | [D] 15.96 ms/request, see 1.5 | T/f3t_run/serve/*n1_154551*.perreq.jsonl |
| sum | 7.088 | 100 | 7.818 | 100 | d7 closes by row 4; d11 predicted 7.736 before the 0.08 |
Cross-check from the record run's own counters [M]: the host idles 1.330 ms/step (d7) and 1.937 ms/step (d11) waiting for the card (join_wait_ns 3,991,624,512 / 3,000 and 4,841,374,691 / 2,500), after 0.506 / 0.515 ms of host work that overlaps the card (host_ahead_ns). The card is already done at the join on only 5 % of steps (151 / 3,000).
- d7:
T/f3t_run/serve/hp_llama3b_PQC_w2d7_humaneval80x256_n1_154551_MEASUREONLY_goldendepth_unproven_20261109_154551.log:476. - d11:
T/config_sweep/serve3/hp_llama3b_PQC_w2d11_humaneval80x256_n1_182607_…_182607.log.gzline 454 (md5 9c9b32fe). - Counter meaning:
scripts/4c1/c4_verifypre.py:477-478. - d11 − d7 join wait = +0.606 ms, against the +0.59 ms of row 2 and the +0.73 ms whole-run step.
1.3 Inside the card tree (µs per tree, chain census, 40 full-depth trees, card-only)
Source: T/config_sweep/census/chain_llama3b_f3t_w2d7_kc64_185201.analysis.json.gz (md5 25e30681) and …w2d11_kc64_185202… (md5 5b11b7f6), key occupancy2 [M]. The d7 values match R/F3T_SILICON_20261109.md:35-44.
| card | phase | d7 µs | d7 % of p50 | d11 µs |
|---|---|---|---|---|
| ATT | G1 | 83.5 | 3.7 | 136.9 |
| ATT | G4 QKV | 390.2 | 17.5 | 630.0 |
| ATT | G5 attention | 523.4 | 23.4 | 841.3 |
| ATT | G6 o-proj + TP exchange (the ATT↔ATT QSFP hop sits here) | 125.2 | 5.6 | 195.1 |
| ATT | G7G | 351.1 | 15.7 | 561.3 |
| ATT | G7U | 186.4 | 8.3 | 302.4 |
| ATT | SILU + LVL_EMBG | 88.4 | 4.0 | 144.8 |
| ATT | INGEST + EXPORT (ATT↔MLPV seam) | 73.2 | 3.3 | 144.5 |
| ATT | busy total | 1,821.5 | 81.5 | 2,956.5 |
| ATT | PIPE_WAIT (parked on the MLPV) | 255.5 | – | 246.3 |
| MLPV | G9 | 167.8 | – | 268.9 |
| MLPV | G11 + GLM + G12 kick | 138.5 | – | 229.6 |
| MLPV | TREE | 437.7 | – | 701.7 |
| MLPV | G8 exact rescore | 343.2 | – | 551.8 |
| MLPV | INGEST + XCH + LVL_SEL/OUT | 152.9 | – | 243.3 |
| MLPV | busy total (mostly hidden under the ATT) | 1,240.0 | – | 1,995.2 |
| joint | both busy / ATT only / MLPV only (exposed) / both parked | 1,035.2 / 795.7 / 216.9 / 38.3 | 9.7 exposed | 1,798.1 / 1,178.1 / 210.5 / 35.9 |
| census window | 2,086.9 | 3,221.5 | ||
| outside the window (issue, SoA read-back / DMA out) | 146.9 | 6.6 | 162.6 | |
| issue → done p50 | 2,233.8 | 100 | 3,384.1 |
- The tree is ATT-bound: ATT busy is 87 % of the window and the MLPV is exposed for 217 µs.
- Link hops have no separate counter. The visible parts are G6 (TP exchange) and INGEST/EXPORT above;
R/MLPV_AND_STEP_BREAKDOWN_E2b.md:66says the same for E2b.
1.4 Tree-stop correction to the card term [D]
The census runs full-depth trees. In serving, theta −2.5 stops trees early:
- d7: 800 of 3,098 trees stopped, mean stop depth 4.68 (
…154551.log:482). Saving ≈ 26 % × 2.32 levels × 2 items × ~150 µs ≈ 0.18 ms → mean tree ≈ 2.04 ms. - d11: 1,400 of 2,526 stopped, mean stop depth 6.40 (
…182607.log.gzline 457). Saving ≈ 55 % × 4.6 × 2 × ~148 µs ≈ 0.75 ms → mean tree ≈ 2.63 ms. - Per-item cost 135–152 µs:
R/F3T_CONFIG_SWEEP_20261109.md:160.
1.5 Per-request fixed cost [D]
Per request, service time = difference of consecutive e2e_latency values. A least-squares fit of service = a + b × verify steps over requests 2–80 gives:
| run | a ms/request | b ms/step | R² |
|---|---|---|---|
| F3t d7 n1 / n2 | 15.96 / 15.55 | 6.696 / 6.717 | 0.991 / 0.987 |
| F3t d11 fa3 r1 / r2 | 35.71 / 35.35 | 6.807 / 6.798 | 0.969 / 0.970 |
| GPU-only d7 fa3 / d9 / d11 | 7.80 / 8.30 / 8.11 | 7.032 / 7.686 / 8.318 | 0.997 / 0.997 / 0.998 |
- The d11 fit is confounded: high-accept requests have fewer but longer (full-depth) steps, which moves time into the intercept. I use the d7 intercept for both columns.
- The FPGA path costs ~8 ms more per request than GPU-only (16.0 vs 7.8). On humaneval-80 that is 0.65 s = 2.8 % of the run.
- Demo prompts: the TTFT proxy (prefill + first tree + first verify) is 15–26 ms of a 215–286 ms request (
T/PER_PROMPT_BENCH_20261109.md:50-84).
1.6 Inside the 0.95 ms host chain (µs, p50, E2b + PQ profile; C then removed ~54)
Source R/HOST_PRECARD_OPT.md:38-52,124-132 [M].
| class | items | µs |
|---|---|---|
| DMA in | KV_LOAD in the card process 229 (4 pwrites ≈ 140) + root FASTIN to both ATT 141 | 370 |
| GPU draft-side launch | capture + EXTKV extend launch (113) + prepare_graph | 229 → 210 with C |
| host glue (Python / IPC) | KV gather 44, guards + plan 57, client KV pack/send 121, socket 13, image recv 46 → 34, stage/cfg/arm 19 + 17 + 40, poison 6, doorbell 8 | ~360 |
| root image graph replay (parallel path that ties with the KV path) | 175 | (not additive) |
DMA out is not separately profiled; it sits in the 147 µs outside the census window (1.3).
2. Amdahl table
Bases [M]: humaneval-80 d7 828.06 (9-round mean) and d11 fa3 883.12; demo set (15 code prompts, 256 tokens) d7 953.0 and d11 1,059.3 (T/PER_PROMPT_BENCH_20261109.md:109-111).
For card levers, Δstep = κ × Δtree. Two values of κ were measured: 1.0 (E2b vs 0x1340, R/MLPV_AND_STEP_BREAKDOWN_E2b.md:70) and 0.51 (E2b → F3t, R/SEVENCARD_PROJECTION_20261109.md:52). Both are shown where it matters.
| lever | evidence | humaneval-80 | demo set | status | time to land | risk |
|---|---|---|---|---|---|---|
| depth 11 (with fa3) | R/F3T_CONFIG_SWEEP_20261109.md:80,83 |
883.12 (+6.65 %) | 1,059 (+11 %) | MEASURED | host-only, 0 h; PERSIST must be armed ≥ 11 before the first AP_START | texts differ from d7 on 3/80; fa3 rounds identical |
| depth 8 / 9 / 10 | :50,52,56 |
850 / 861 / 870 | – | MEASURED | 0 h | – |
| depth 12 / 13 / 15 | :58-60 |
874.7 / 873.8 / 870.1 (−1.0 to −1.5 % vs d11) | not measured | MEASURED | 0 h | loss |
| width 3 (w4 skipped) | :73-74 |
767 (−13 % at d11) | – | MEASURED | – | loss |
| verify tokens V16 / V24 / V27 | :45-47,62-63 |
d7 828.6 / 799 / 800; d11 853 / 859 | – | MEASURED | – | loss or level |
| theta −2.0 / −3.0 | :67-68 |
877 / 868 at d11 | – | MEASURED | – | level / loss |
| K_C 32 (128) | :75-76; R/FAILED_DIRECTIONS_20261109.md:9 |
885.70 (+0.5 %, one round); E2b A/B +0.82 %; K_C 128 −4.9 % | – | MEASURED, inside noise | host-only, 0 h | changes 2–4/80 texts |
| FR-Spec 64K / 32K / 16K | :141-143 |
−1.2 / −9.5 / −20.3 % | – | MEASURED | – | loss |
| FIRST_DRAFT_GPU | already on: FIRSTGPU=1 at …154551.log:3 |
0 | 0 | in the base | – | – |
| fa3 draft backend | :170 |
+0.4 % at d11, neutral at d7 | – | MEASURED | 0 h | makes d11 deterministic |
| FASTGLUE variants / HP letters | FASTGLUE 4 + P, Q, C on; R, K, E, W, A rejected (R/HOST_PRECARD_OPT.md:66-69,126) |
0 | 0 | MEASURED | – | – |
| PINPAR 1 | :69-70 |
822 / 876 (no gain) | – | MEASURED | – | – |
| KVONC host fix | +1.52 % on 0x130C + 0x128B at 749.57 (reports/results/FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:1414-1415) |
not applicable: the F3t regmap row has no kvoncard key (T/f3t_kit/patch_13f9_host.py:46); needs a KV_ONCARD image |
– | MEASURED on another image | a build | not in 14 h |
| CUDA-graph / draft-graph warm | target verify graph already captured (…154551.log:168-175); GRAPHWARM refuted, it changes accept (reports/results/FOURCARD_DESIGN_20261016/STEP_SPLIT_20261019.md:106) |
0 | 0 | MEASURED | – | – |
| CPU governor | +0.65 %, below the bar (:171) |
~+0.6 % | – | MEASURED 11-06 | system-wide setting on a shared host | low gain |
| GPU clocks / thermal | talon2: throttled 0, max 69 °C in every row (T/f3t_run/serve/SUMMARY_llama3b.tsv:2-3); box1 H200 throttled at 87 °C on Qwen, SM 1,785 → 360–510 MHz (USER_DIALOGUE_LOG.md:1989) |
protects the number, adds nothing | same | MEASURED | 0 h | on box1, a hot GPU can cut the rate several-fold |
| prompt choice | T/PER_PROMPT_BENCH_20261109.md:50-84 |
– | d7 896–1,042; d11 943–1,193; mt_bench prose 404–497 | MEASURED | 0 h | prose prompts halve the number |
| max_new_tokens 512 on code prompts | decode-only rate d11 1,242–1,308 vs e2e 1,111–1,193 (:70-73) |
– | ≈ +4–5 % [E] (halves the per-request share) | estimate | 0.5 h test | early EOS; accept over a longer output not measured |
| per-request cost (FPGA 16 ms vs GPU 8 ms) | 1.5 above | ≤ +2.8 % [E] | ≤ ~+4 % [E] | estimate | host dev, 3–6 h | untested host change before a demo |
| 2-stream PINGPONG / MPS | 1.55× was the 2-card pair (memory feedback_batch1_latency_first_pingpong_is_batch2); on F3t, N2 without MPS = 221 (B/INTERLEAVE_BATCHVERIFY_SILICON_20261109.md:53) |
below single-stream | – | MEASURED loss without MPS; MPS arm not run | hours | – |
| batched verify BV4 + HBMREPLAY (aggregate) | 730 / 717 vs 665 on the same launcher (:82); that launcher lacks the record levers |
730 < 831 today; ~905 [E] if the record levers were ported | – | MEASURED (aggregate only) | host dev, hours | 6–8/80 texts change |
| G5_OVL | in F3t (R/F3T_SILICON_20261109.md:14) |
0 more | 0 | in the base | – | – |
| LANES 512 ATT | in F3t (F3t = D9 L512, R/F3T_SILICON_20261109.md:4) |
0 more | 0 | in the base | – | – |
| R3_EARLY | −75 µs/tree [calc] (R/BOTTLENECK_TABLE.md:84); silicon F0b: no gain shown at equal host config, one unexplained accept-1.04 boot (R/FAILED_DIRECTIONS_20261109.md:11) |
+0.5 % (κ 0.51) to +1.1 % (κ 1) [E] | same | estimate | a build on the F3t base | not in 14 h |
| ATT TP4, 6 cards: B4 0x13B4 (GO, flashing now) + MLPV 0x134D | R/SEVENCARD_PROJECTION_20261109.md:90; adjudication R/SEVENCARD_B1_B5_ADJUDICATION_20261110.md:11-13 |
d7 869 / d11 948 (κ 1); 848 / 915 (κ 0.51) | not projected | SIM only; no TP4 silicon census exists | flash in progress + manual cold AC + first 6-card bring-up, 4–8 h [E] | high: 0x134D PDI not written at 07:00Z (routed +0.001 / +0.010), new host kit, cables unverified |
| ATT TP4 B5 0x13B5 (SIBW W2) | same doc :89 |
d7 885 / d11 979 (κ 1); 856 / 930 (κ 0.51) | – | SIM only | as B4 | as B4, plus hold exactly at the bar |
| G6_EARLY_ISSUE | root item 11,152 → 9,670 cycles in TP4 xsim (reports/results/QWEN27B_DRAFTER_20261020/G6_EARLY_ISSUE_20261109.md:19) |
not on any Llama 4C1 image | – | SIM only, Q5120 lineage | a build | not in 14 h |
| MLPV L512 | 0x1351 died at place, 19,133 SLLs vs 18,870 (…4C1_DELIVERY_TRACKER.md:1502) |
– | – | dead | – | – |
| MLPV TREE_P 64 | 0x135C died at place (…4C1_DELIVERY_TRACKER.md:1531) |
– | – | dead so far | – | – |
| Q5120 drafter | GPU-only 196.61 @ 3.976 (…4C1_DELIVERY_TRACKER.md:1516) vs BitNet Qwen drafter 249.62 @ 7.14 on F3t (R/F3T_SILICON_20261109.md:22); Q5120 MLPV boot refused at 07:02Z (…TRACKER.md:1535) |
other program (Qwen target); slower today | – | MEASURED GPU-only | – | does not raise the Llama number |
3. What caps the number
- Binding term: GPU verify, 3.45 ms of 7.09 ms. It starts when the card publishes the tree (verify is pre-enqueued behind the card flag,
…154551.log:26), so verify and card are serial. - Second: the card tree, ~2.0 ms mean at d7. 87 % of it is ATT busy time, led by G5 (523 µs), G4 (390) and G7G (351).
- Third: host, 0.95 ms chain + 0.25 residual + 0.38 per-request. 0.37 ms of the chain is QDMA writes at ~28 µs fixed cost each (
R/HOST_PRECARD_OPT.md:171).
| what goes to zero | d7 tok/s | d11 tok/s | how |
|---|---|---|---|
| card wait only (the measured host idle, 1.330 / 1.937 ms) | 1,023 | 1,173 | accept ÷ (step − join wait) [D] |
| whole card tree (2.04 / 2.63 ms) | 1,167 | 1,330 | upper bound; needs the 0.5 ms host prologue hidden as well [D] |
| host chain + residual (1.20 ms) | 1,001 | 1,043 | [D] |
| host chain + residual + per-request | 1,071 | 1,120 | [D] |
| GPU verify | 1,620 | 1,580 | [D] |
| card and host both (verify + per-request left) | 1,535 | – | [D] |
Reference: best GPU-only the same day is 835 (d9 fa3, R/F3T_CONFIG_SWEEP_20261109.md:120). Its marginal step is 7.03 ms at d7, so the GPU drafter + host costs 3.58 ms against the cards' 3.25 ms [D from 1.5].
4. Top 5 for the next 14 h, ranked by (gain × probability) ÷ hours
One-flash actions
- Put F3t 0xE4B713F9 (md5 97d8445c) back on 17/3d and 0x1342 (3e82ea1d) on 50/63, one cold AC, arm depth 15 before the first AP_START. This is the only route to the measured 883 / 1,059. Probability ~0.95 (same PDIs, kits and run dirs as 11-09:
R/F3T_SILICON_20261109.md:62-86). ~2 h including the manual power cycle. Set a hard cutoff for starting it; ~17:30Z leaves 4 h. - Time-box the 6-card TP4 attempt (B4 now flashing; B5 is the faster projection). Projected gain over F3t at d11: +3.6 % (κ 0.51) to +7.4 % (κ 1) for B4, sim only. It needs MLPV 0x134D on 50/63, which had no PDI at 07:00Z. Probability of a served, golden-checked number before the cutoff: low. If it is not serving by the cutoff, do action 1.
No-flash, host-only actions (on F3t once it is back)
- Serve at w2 d11, fa3, V20, K_C 64, theta −2.5. +6.65 % on humaneval-80 and +11 % on the demo prompts, measured, 0 h. Exact command:
R/F3T_CONFIG_SWEEP_20261109.md:181-183. - Use the high-accept code prompts (
even_odd_palindrome,sort_array,is_sorted,pluck,words_in_sentence): 1,111–1,193 tok/s at d11, measured, 0 h. Avoid prose prompts (404–497). - Test max_new_tokens 512 on those prompts to dilute the 18–26 ms per-request cost: ≈ +4–5 % [E], 0.5 h, unmeasured. Keep 256 if a prompt stops early or accept falls.
Not worth the hours: width 3, V ≠ 20, FR-Spec, K_C 128, PINPAR, GRAPHWARM, the governor, KVONC (wrong image), R3_EARLY / G6_EARLY_ISSUE (need builds), and the multi-stream modes (below single-stream today).
5. Not verified
- Card state on talon2 and box1 is taken from the dialogue log and tracker, not read from registers.
- Row 3 of the waterfall (0.95 ms) is an E2b profile; no STEPPROF run exists on F3t. Row 4 is a residual.
- GPU verify 3.452 ms was timed in the batch-verify B1 arm (665 tok/s launcher), d7 only. A d11 verify timing does not exist.
- The mean tree time in serving (row 2) is an estimate from stop-depth counts; no serving-time card census exists.
- No 6-card TP4 silicon number exists; every TP4 figure is a projection from xsim cycles.