fpga-archive-reports-qwen · EAGLE4 FPGA archive

Source reports/results/QWEN_GPTQ2_20261116/D7_TREE_LEVERS_20261117.md · source md5 f6650861cbc3f2f6aeaa7b43e61510ae · archived 2026-11-18

Download PDFOpen PDFRaw markdown

D7 W2 tree levers for the next Qwen GPTQ-2 7-card image (qgptq2-d7-levers, 2026-11-17)

Program: the 7-card 4C1 drafter for the Qwen3.8-27B target (Q5120 GPTQ-2, H 5120). ATT x4 = 0xE4B7139C (TP4, L256), MLPV x2 = 0xE4B7139D (L256). Motto: a research track never ends by denying the idea; it engineers toward the fastest flashable PDI. Rules kept: read-only. Only window evidence FILES were read on talon2; no card access, no kills, no deletions. Tags: [M] measured (file + key or line + md5). [R] read from code (file:line + md5). Every number that is a projection, fit or hypothesis is in section 7, "Calculations and hypotheses".

Evidence copied to reports/results/QWEN_GPTQ2_20261116/d7_levers_evidence/, with md5s in MD5SUMS.txt:

The scratch copy is /mnt/hdd_data/eagle4_scratch/qwen_gptq2_20261116/d7_levers/.


0R. Re-rank v2: host path first (coordinator data folded in, 2026-11-17 ~05:20Z)

Ranked top 5, host first. Expected cuts are per serve round; the arithmetic is in sections 7.7-7.10.

rank lever kind expected cut per round, d2 / d7 what it removes
1 Pipeline / batch the KV-only rows inside the card server: issue rows back to back with the KVONLY arm held; drop the per-row quiescence and readback MMIO; PINPAR-interleave the 4 ranks' boot-DMA waits; one RPC carries the round's rows + tree; write the previous root's row while the GPU verify runs host -4.9 / -7.7 ms 1.64 ms/row x 3.96 / 6.2 rows per round. This is the bulk of the "12 ms host/bridge"
2 Do not run the GPU _grow when the card tree overwrites it (c4q5120_bridge.py:233-237) host -1.5..-3 / -3..-5 ms the GPU d-step drafting that every silicon round still pays
3 Throttle _dump(): write the stats every N trees + at exit host -1.35 / -1.0 ms (mean; grows to -2.5 ms by tree 588) the full STATS JSON, indent=1, rewritten after every card tree
4 Extend the golden suite to md 3 / 5 (W 2, K_C 64) so d3 / d5 can serve on the card host (golden run) turns the best shape on: modelled d5 W2 = 92 tok/s today vs 89 at d7, 78 at d2 the shape_not_in_golden refusal
5 EAGER + EARLY_A loop modes + the 1.47 ms host work per tree (PREPOISON, cfg diff, one SoA read) host (+ a mode-0x30 golden) -0.9..-2.3 ms (d7); -0.6..-1.5 (d2) tree-side host time and loop serialisation

After these, the card-tree RTL levers of section 7.4 follow: MLPV 132 MHz array, ATT L512, TREE floor.

0R.1 d2 W2 V6 serve (golden-covered) [M]

Sources: serve/qgserve_silicon_d2w2v6kc64_pwfifo_d2_20261117_045803.{client.log,jsonl,stats.json.26206} (jsonl md5 2ae2f597) and the GPU bar qgserve_none_d2w2v6kc64_gpubar_d2_20261117_050121.* (jsonl md5 310a4757).

quantity silicon d2 GPU-only d2 silicon d7 (section 1.5) GPU-only d7
tokens / rounds / accept 1,512 / 588 / 2.5714 1,512 / 588 / 2.5714 1,509 / 372 / 4.0565 1,509 / 379 / 3.9815
sum of walls / tok/s 20.0 s / 75.46 10.8 s / 139.51 17.0 s / 88.72 8.3 s / 182.50
ms per round 34.0 18.4 45.7 21.9
t_card_s (kvrows + tree RPCs) 6.308 s = 10.73 ms/round – 6.698 s = 18.0 ms/round –
t_fc_s + t_row_s 0.130 + 0.433 s = 0.96 ms/round – 0.107 + 0.418 s = 1.41 ms/round –
KV-only rows 2,327 (3.96 / round) – 2,316 (6.23 / round) –
card tree, sweep wall p50 (ctx 45 / 112) 3,751 / 3,715 us – 8,481 / (21,707 slow row) –

GPU-only bars at W2 (client logs): d2 V6 18.4 ms/round @ 2.5714; d3 V10 19.3 @ 3.0436; d5 V16 20.75 @ 3.7575 (181.44 tok/s); d7 V20 21.9 @ 3.9815.

0R.2 Per-request "silicon minus GPU" decode ms per round [M]

(wall - ttft) / rounds, from the jsonl. Rows/round = (C - 1 + completion) / rounds.

request d2 rows/rnd d2 extra ms d7 rows/rnd d7 extra ms d3 card-refused arm extra d5 card-refused arm extra
humaneval:0 (card-server bring-up inside) 6.39 21.69 16.23 47.09 6.56 20.48
humaneval:1 4.84 14.59 10.79 30.57 -0.03 -0.04
humaneval:2 5.82 16.81 14.82 38.40 0.02 -0.06
humaneval:3 6.11 17.43 15.94 39.16 -0.03 -0.07
mt_bench:81 2.52 12.36 2.91 18.31 0.00 -0.05
mt_bench:82 4.14 15.54 6.07 22.63 0.00 -0.02
mt_bench:83 3.16 14.01 4.41 19.88 0.00 -0.01
mt_bench:84 2.98 14.14 3.99 19.44 0.01 -0.01

0R.3 Code facts on the per-row and per-round path [R]

0R.4 Parity: RESOLVED (target near-tie flipped by tree shape) [M]

mt_bench:81, output tokens 27..30:

run tokens 27..30
GPU-only d2 bar 678, 10709, 6356, 1364
GPU-only d3 bar 488, 2923, 974, 279
GPU-only d5 bar 488, 2923, 974, 279
GPU-only d7 bar 678, 10709, 6356, 1364
card d2 678, 10709, 6356, 1364
card d7 488, 2923, 974, 279

0. Answer first (v1; for the end-to-end picture, 0R supersedes the ranking below)

  1. The "15.1 ms d7 W2" bar is an averaging artefact. It does not measure a d7 tree.
    • The sweep's p50_wall_us_by_shape is the arithmetic MEAN of the two prompts' p50s (qg_sweep.py:119,122).
    • md7_W2 = (21,707 + 8,481) / 2 = 15,094 us.
    • The 21,707 us row (req00, ctx 112) is the first shape timed in the sweep process. It is 2.6x the same shape at ctx 45 (8,481 us). It is also 2.2x the golden's md7 W2 trees at ctx 148 / 162 / 166 (10 ms each), which are longer contexts.
    • Every req00 row after md7 W8 matches req04 to within 0.6 ms.
    • The measured d7 W2 card tree is 8.5 ms wall / 7.0 ms issue-to-done (sweep, ctx 45). The golden gives 9 to 10 ms on all 7 non-first contexts (ctx 45 to 166). The serve gives about 8.4 to 10.8 ms per tree (fit, section 7.2).
  2. The d5 -> d7 step is linear, not superlinear, once that row is set aside.
    • At ctx 45 the W2 ladder is md2 3,751 / md3 4,514 / md5 6,231 / md7 8,481 us wall.
    • That is about +0.76 to +1.12 ms per level, at 2 items per level.
    • Host round trips per level: 0 (assists 0, TREE_STOP host verdicts 0).
    • Context growth adds tens of cycles per item (G5).
    • The slow block (req00 md7 W2 plus the tail of md7 W8, and a 1.9 ms/row prime just before them) has no confirmed mechanism. Section 2.3 lists what would decide it.
  3. For end-to-end tok/s, the two largest levers are on the host, not in the tree.
    • Per round, silicon pays the WHOLE GPU-only round (the GPU drafter _grow still runs before the card tree overwrites it, c4q5120_bridge.py:233-237).
    • On top of that it pays 12.8 ms + 1.68 ms per KV-only row (fit over 7 requests, residuals < 0.8 ms, section 7.2).
    • KV rows per round are about 3 (mt_bench) to 7 (humaneval) plus the amortised prime. So the per-row KV ingest costs as much as the tree, or more.
  4. Top 5 card-tree levers for the next image, by expected d7 W2 cut (numbers in section 7.4):
    • (1) MLPV array on the 132 MHz clock (the record M1361 mechanism; 0x139D has ARRAY_FAST_CLK 0);
    • (2) EAGER + EARLY_A loop modes (host plus a golden in that mode; 0x139D already carries EAGER_TPSAFE);
    • (3) cut the 1.47 ms constant host work per tree (PREPOISON, PINPAR, cfg diff, one SoA read);
    • (4) ATT L512 (needs 13A7's SLR2-URAM xdc fix);
    • (5) TREE tile floor (TREE4 form) on the MLPV.
    • The drafter clock 66 -> 100 MHz would cut more, but it is not a next-image lever.
  5. Text divergence (mt_bench:81 at output token 27) is reproduced by the GPU-only path when only the tree shape changes. It is not evidence of a drafter defect.
    • The GPU-only d3 bar diverges from the GPU-only d7 bar at the same token 27, with the same tokens as the card serve.
    • The card serve equals the GPU-only d3 bar through token 91.
    • Section 6 has the full evidence and the tests that would close it.

1. Measured d7 W2 decomposition

1.1 Card sweep (GPU out of the loop) [M]

Source: qg/sweep_q5120_023153.json (md5 21ddcb78), key rows[i]. Producer: ship/qg_sweep.py (md5 55c5eaba).

prompt (ctx) order md W items/rank nodes wall p50 us wall p10 / p90 issue_done p50 us
req00 (112), prime 111 rows in 0.21 s 1 7 2 13 26 21,707 19,266 / 22,827 (min 18,843) 18,106
2 7 8 49 392 19,616 19,002 / 31,621 (max 33,591) 17,451
3 5 2 9 18 6,618 6,213 / 7,378 5,172
4 5 8 33 264 14,503 14,122 / 15,212 12,712
5 3 2 5 10 4,653 4,169 / 4,938 3,153
6 2 2 3 6 3,715 3,384 / 3,879 2,176
7 2 8 9 72 5,852 5,313 / 6,720 4,368
req04 (45), prime 44 rows in 0.05 s 1 7 2 13 26 8,481 8,089 / 8,969 7,007
2 7 8 49 392 18,062 17,106 / 18,459 16,302
3 5 2 9 18 6,231 6,021 / 7,335 4,770
4 5 8 33 264 13,925 13,457 / 14,275 12,226
5 3 2 5 10 4,514 4,153 / 5,166 3,081
6 2 2 3 6 3,751 3,326 / 3,850 2,183
7 2 8 9 72 5,786 5,345 / 6,438 4,343

1.2 Golden: one tree per (prompt, shape), 8 contexts [M]

Source: qg/golden_q5120.json (md5 2f79f46b), results[i].tree_s / prime_s (ms resolution). Run order: req00 first. Every tree PASS.

ctx 112 (req00, first tree after arm) 166 162 148 45 69 80 62
md7 W2 tree_s ms 14 10 10 10 9 9 10 10
md7 W8 19 19 19 19 18 19 19 19
md2 W2 4 4 4 4 4 4 4 4
md2 W8 7 7 7 6 6 6 7 7
prime_s for (C-1) rows, first shape 0.18 (111) 0.17 (165) 0.16 (161) 0.15 (147) 0.04 (44) 0.07 (68) 0.08 (79) 0.06 (61)
prime_s, later shapes of the same prompt 0.12 / 0.11 / 0.11 0.17 0.16 0.15 0.04-0.05 0.07 0.08 0.06

1.3 Tree structure at W2 (what grows per level) [M]/[R]

1.4 Per-phase cycles per item (xsim, the image sources; not silicon) [M]

ATT (0x139C source, L256, TPX ideal 4-way sum TP_LAT 40) root L1 child MLPV (0x139D shape, L256 TREE_P 32) per item
G1 713 713 G9 S8 5,133
G4 (15 tiles) 4,825 4,825 XCH S26 46-61
G5 (ctx 50) 612 618 G11 S9 723
G6 + 4-way exchange 903 903 GLM S13 24
G7g (incl premlp) 3,937 3,937 G12 S10 + TREE S11 (G12 cheap-logit sweep inside S11) 13 + 6,803
G7u 2,585 2,585 G8 S14 (K_C 512 vectors) 1,896-2,109
SiLU / EXPORT 262 / 1,248 262 / 1,248 INGEST S23 865
EMBG / INGEST 0 / 0 329 / 337
busy 15,085 (229 us) 15,757 TOTAL 15,504-15,722 (235-238 us); mean 15,616 (TREE_P16 §6)

Sources: XSIM_ATT_20261116.md §3 (L512 control: busy 10,436 = G4 2,916, G6 701, G7g 2,668, G7u 1,316); XSIM_MLPV_20261116.md §5-§6.

1.5 End-to-end serve at d7 W2 V20 K_C 64 (card in the loop) [M]

Sources: serve/qgserve_silicon_d7w2v20kc64_pwfifo_20261117_043538.{client.log,jsonl,stats.json.20356} (md5 782b34ec / 1b906662 / c3239c9b) and the GPU bar qgserve_none_d7w2v20kc64_gpubar_20261117_043950.* (69ca21bb / 908dd89c).

quantity silicon GPU-only
tokens / rounds / accept 1,509 / 372 / 4.0565 1,509 / 379 / 3.9815
sum of request walls 17.008 s (88.72 tok/s) 8.269 s (182.50 tok/s)
card RPC time t_card_s (kvrows + tree RPCs) 6.698 s (18.0 ms/round) –
t_fc_s / t_row_s (eager fc + row build on the GPU host) 0.107 / 0.418 s –
KV-only rows kv_only_rows 2,316 (prime rows Σ(C-1) = 836; extend 1,480) –
card trees / GPU-tree rounds 372 / 0 –

Per request, decode ms/round = (wall - ttft) / verify rounds, from the jsonl:

request (C) humaneval:0 (112) :1 (166) :2 (162) :3 (148) mt_bench:81 (45) :82 (69) :83 (80) :84 (62)
silicon 68.9 50.7 58.5 59.2 38.4 42.7 39.9 39.5
GPU-only 21.8 20.1 20.1 20.0 20.0 20.0 20.0 20.1
accept (silicon) 7.692 6.564 7.500 7.778 2.485 3.800 3.368 3.127

Server-log progress lines (card trees N, kv-only rows, card s, 0.1 s resolution, *.server.excerpt.txt): 1 / 111 / 0.2; 10 / 183 / 0.3; 100 / 1,287 / 2.8; 200 / 1,615 / 4.2; 300 / 2,037 / 5.7; final 372 / 2,316 / 6.698.

[R] Silicon mode still runs the GPU drafter. DynamicDrafter._grow is wrapped: d = o_grow(...) (the full GPU tree) runs first, then _inject overwrites tree.tokens / scores / parents with the card tree (c4q5120_bridge.py:233-237, md5 b7383dbe). Other per-round costs in the same file:

1.6 d3 / d5 serves in the same window [M]


2. The d5 -> d7 jump: mechanism and evidence

2.1 What the number is [M]/[R]

The superlinear step (d5 -> d7 +8.7 ms against +0.9 and +1.8 ms) exists only in p50_wall_us_by_shape. That field is a mean across prompts (qg_sweep.py:119,122), and the req00 md7 W2 row inside it is anomalous. Per prompt:

2.2 Why the req00 md7 rows are not a depth or context effect [M]

  1. Context does not explain it. The golden ran md7 W2 at ctx 148 / 162 / 166 (longer than 112) at 10 ms each, and at ctx 45-80 at 9-10 ms (section 1.2). xsim G5 grows by only about 370 cycles per item from ctx 50 to 112 (section 1.4).
  2. Depth does not explain it. In the same prompt and the same process, md5 W2 (shape 3, run right after) is 6,618 us. The md7 W2 tree is a strict 2-level extension of it.
  3. The work does not vary, but the time does.
    • md7 W8 at req00 has p50 19,616 but p90 31,621 / max 33,591, while every one of its 20 SoAs is identical (nondeterministic 0).
    • md7 W8 at req04 has max 18,815.
    • So some trees paid about +12 to +14 ms for identical work.
  4. The slow block is the start of the process.
    • It is the first shape (md7 W2, all 20 trees) and part of the second shape (md7 W8) of the sweep process.
    • The prime just before them was also slow: 0.21 s for req00's 111 rows (1.9 ms/row), against 0.11 to 0.12 s for the same 111 rows on the same card in the golden.
    • The golden process shows the same "first" penalty, smaller: its first tree (req00 md7 W2) took 14 ms against 10 ms for every later context, and its first prime took 0.18 s against 0.11-0.12 s.
    • The sweep process started 50 s after the golden process ended (steps.lock: golden FREE 02:31:00Z, sweep OWNER 02:31:50Z).
  5. Not a token / emb-aperture effect. The req00 md7 W2 reference tree has no token >= 78,643 (the first-GiB emb boundary). The largest is 74,669, and other prompts have similar trees (suite qgptq2_golden_suite_nonfr_kc64.json, md5 6724495a, prompts[0].trees md 7 W 2).

2.3 What is not decided, and how to decide it (host-only, next window)

The slow-block mechanism (host warm-up, link idle exit, QDMA first use, or an on-card stall) is not identified. Nothing above separates host time from on-card time inside issue_done. These measurements would decide it:


3. Lever evidence: what applies to Q5120 and what each costs [M]/[R]

(Expected cuts are in section 7.4.)

# lever mechanism evidence it applies to Q5120 image cost correctness risk
L1 MLPV array on the 132 MHz clock (ARRAY_FAST_CLK 1) G9 and the G12 stream inside TREE run at 2x the 66 MHz cycle rate. MLPV appears twice per W2 level on the critical path The record MLPV M1361 ships ARRAY_FAST_CLK 1'D1 (verilog/clean7c/ref/mlpv_1361/CONFIG_MANIFEST.md:36). 0x139D has .ARRAY_FAST_CLK(1'D0) (BUILDS_20261116.md:43), and its FINAL rows show no clk_usr_1 (:157). The Fable GEMM ranks it first at A16 (DESIGN_FABLE_GPTQ2_GEMM_20261116.md:118-119) new MLPV PDI. The lineage refuses LUTG with ARRAY_FAST_CLK, so either the e3b DSP lane plus the exact 1-bit zero fold plus ACT_XPIPE, or the lifted interlock on the LUT lane (DESIGN_FABLE_GPTQ2_GEMM:86,137) medium. A new arithmetic lane needs bit-exact xsim against the bit-true reference and a new golden. The fold is "exact" on paper, not yet built
L2 EAGER + EARLY_A loop modes (mode_bits 0x30, the record default) The TAIL dispatches children early, and the MLPV exports a before G8 ends, so the next level's ATT work overlaps the current MLPV tail The window ran mode_bits 0 (qg_steps.sh:29, sweep mode_bits 0). Arm shows TPSAFE 0xe7010000 on both MLPVs (qg/run.log, arm7 lines), so EAGER_TPSAFE is built in. On the old Qwen v0 pair, EAGER+EARLY_A off cost 30.16 / 29.91 vs 29.28 / 28.56 ms/step (SESSION_CONTEXT_BACKUP_20261027.md:19) host only. A new golden must PASS in mode 0x30, because the card server refuses a serve whose mode_bits differ from the golden's low to medium. The EARLY_A count ERR 7 seen on the D7 PAIR_OVL MLPV (D7_PAIROVL_SILICON_RCA_20261113.md:68) is a different image. The golden decides
L3 Constant host work per tree (1.43-1.57 ms measured, section 1.3) PREPOISON=1 (poison in idle time, c4_loop.py:163-166); PINPAR=1 (program_inputs_interleaved, c4_loop.py:446); write cfg only when it changed (md / W / kc are constant per request); in serve, read the SoA from one MLPV (the rank cross-check is a golden tooth); the record's hx7 MMIO bundle (c4_hxo7.py) The levers exist in the kit. The window ran the defaults. The record serve runs PREPOISON / PINPAR 1 (C1_WINDOW_V2_RUNBOOK_20261112.md:204) host only low. Each one is a byte-identical register sequence, earlier or skipped. The one-SoA-read serve gives up the free rank cross-check
L4 ATT L512 ATT item 15,085 -> 10,436 cycles (xsim L512 control) 13A6 / 13A8 drafter OOCs exist: 1,290,8xx LUT, 1,169 URAM (BUILDS_20261116.md §5). 13A7 DIED at place (682 URAM in SLR2 > 624); the fix candidate is xdc 99a8d299 (BUILDS_20261116.md §8). All three are FR_A8 new ATT PDI with the fixed xdc. A non-FR L512 needs its own derive medium. FR_A8 is an accuracy-class change (golden class differs). L512 x FR_A8 L2 lanes 4-7 have not run in xsim (XSIM_ATT §4 limits)
L5 TREE tile floor (MLPV TREE4 form) Halves the G12 tile time inside TREE S11 (6,803 cycles, 44 % of the MLPV item) Llama MLPV TREE4: 2,020 -> 1,024 bit-exact. On Llama it was about 0 us alone, because G8 was then exposed (memory project_mlpv_tree4_exposes_g8_20261113). At Q5120 K_C 64, G8 is 1,896-2,109 cycles, against TREE 6,803 new MLPV PDI low to medium. TREE4 was bit-exact on Llama. The 248,320 vocab and 18-bit ids need re-gating
L6 K64 on the MLPV G9 64 codes per lane-cycle halves G9 (5,133 cycles) Fable GEMM §1.7: m -2.9K cycles; at A16 the DSP-half form does not apply (§1.8) new MLPV PDI; LUT form +163K LUT medium; area-bound
L7 Drafter clock 66 -> 100 MHz (integer-exact clk_wiz) Every on-card cycle x 0.66 0x139C FINAL_TRUE +0.124 / +0.012; 0x139D +0.000 / +0.009 margined, then rescued (BUILDS §9, §11) a closure campaign (fmax census) high. Not a next-image lever
L8 G5_OVL (Llama F3t) Overlaps G5 with the item G5 is 612-990 cycles of a 15,085-cycle ATT item at these contexts (section 1.4) new ATT PDI low
L9 Tree-seam pipeline register (DESIGN_ARCH_SHIFT §3.30) Timing hygiene: +1 cycle per tile INTO TREE 0x1399 crossing +0.084 ns, 23 levels new MLPV PDI low. It costs about 486 cycles per item, so it is an enabler for L1 / L7, not a speed lever
E1 Batch the KV-only rows (serve, per round) One row = one serialised PERSIST item plus about 12 MMIO and 1 H2C per rank (section 1.5). Rows of one round could be issued back to back, with the KVONLY latch held Per-row cost 1.68 ms in serve, about 1.0 ms in the golden prime (sections 1.2, 7.2) host first (hold the arm, drop per-row quiescence and readback, pipeline on IN_SEQ). RTL only if per-row CTX_LEN must ride in the header medium. KV order and position: the next golden must cover a burst
E2 Do not run the GPU _grow when the card tree will overwrite it Removes the GPU d7 drafting time from every silicon round c4q5120_bridge.py:233-237 host only (bridge) low for the tree (the card tree is what gets verified). The harness side effects of _grow need a read before cutting
E3 Throttle _dump() (every N trees + atexit + signal) 36 KB JSON rewritten per tree c4q5120_bridge.py:67-74, 514 host only none
E4 Extend the golden to md 3 / 5 (and W 4) Lets d3 / d5 serve on silicon at all d3 serve refused, shape_not_in_golden (section 1.6) host (golden run in a window) none

4. d3 / d5 effects (evidence)


5. HOST-only vs NEW-PDI

host only (no new image) needs a new PDI
L2 EAGER + EARLY_A (with a mode-0x30 golden) L1 MLPV 132 MHz array
L3 host work per tree (PREPOISON / PINPAR / cfg diff / one SoA read / hx7-class MMIO) L4 ATT L512 (xdc fix; FR_A8 or a non-FR derive)
Ring-H1 tooth off in timing runs (already off in serve) L5 TREE tile floor
E1 KV-row batching (first form) L6 K64 MLPV G9
E2 skip GPU _grow; E3 throttle _dump; E4 golden at md 3 / 5 L7 clock; L8 G5_OVL; L9 tree-seam register (enabler)
the §2.3 measurements (--census, --levels, raw per-tree lists)

6. The 1 / 8 text divergence (mt_bench:81, output token 27) -- RESOLVED (see 0R.4)

Facts [M] (output_ids compared across the five jsonl files; md5s in MD5SUMS.txt):

run first output token differing from the GPU-only d7 bar (req0, req4, req6; others identical) tokens 24..30 of req4
card d7 (pwfifo, 04:35) req4 27 488, 279, 4299, 488, 2923, 974, 279
GPU-only d7 bar (04:39) – 488, 279, 4299, 678, 10709, 6356, 1364
GPU-only d3 bar (04:47) req0 57, req4 27, req6 167 488, 279, 4299, 488, 2923, 974, 279
GPU-tree d3 (silicon arm, card refused: every round a GPU tree) req4 27, req6 167 (req0 equal) as the d3 bar
GPU-tree d7 (serve #1, card dead: every round a GPU tree) none (8 / 8 equal) as the d7 bar

Reading: with the drafter held fixed (the GPU drafter), changing only the tree shape (d3 vs d7) flips mt_bench:81 at token 27 to exactly the token sequence the card serve emits. Every emitted token is the target's own argmax under its verify pass. So the divergence comes from the target's bf16 / GDN verify numerics depending on the verify tree (shape and content): a near-tie at position 27. A drafter error cannot select an emitted token. This is the expected behaviour of lossless speculative decoding when the verify computation is not bit-identical across trees. It is not evidence of a drafter defect. The card trees themselves are golden-exact (32/32 + per-op, SESSION_CONTEXT_BACKUP_20261116.md:8).

What would close it (host-only, no card):


7. Calculations and hypotheses

(Everything in this section is arithmetic on the facts above, a fit, or a projection. 1 cycle = 15.15 ns at 66 MHz.)

7.1 d7 W2 per-level arithmetic (ctx 45 sweep)

7.2 Serve fits (card per round)

7.3 Where the 45.7 ms silicon round goes (d7, all 8 requests) [H, from 7.2]

GPU round 21.9 (incl. GPU drafting) + card tree ~9-10 + KV rows ~6.2 × 1.0-1.6 = 6.2-9.9 + fc/row 1.4 + residual (bring-up, _dump, RPC) ~3-5.

7.4 Ranked card-tree levers (expected cut on the d7 W2 tree; baseline 8.5 ms wall / 7.0 ms issue_done at ctx 45)

rank lever expected d7 W2 cut how computed [H]
0 read the bar correctly: 15.1 -> 8.5 ms (–6.6 ms of "bar", no silicon change) section 2
0b ring tooth off in timing runs -1.3 to -1.6 ms on the sweep metric; 0 in serve 52 reads × 25-31 us
1 L1 MLPV 132 MHz array -1.0 to -1.2 ms m 15,616 -> ~10,000 cycles (G9 -2,567, G12 stream about -3,000); MLPV twice per level: 6 × 2 × 85 us + root 85 us
2 L2 EAGER + EARLY_A -0.5 to -1.4 ms Qwen v0 pair measured -0.9 / -1.35 ms per step; EARLY_A alone hides G8 (~30 us) plus tail per level × 6
3 L3 host work per tree -0.4 to -0.9 ms (of the 1.47 ms measured) poison + cfg + one SoA read; PREPOISON moves the poison H2C off the path
4 L4 ATT L512 -0.4 to -0.5 ms a -4,649 cycles = -70 us; ATT once per level when a < m: 7 × 70 us
5 L5 TREE tile floor -0.3 to -0.4 ms alone; about half of that if stacked on L1 m -2K cycles × 2 per level × 6 + root
6 L6 K64 MLPV G9 -0.3 to -0.6 ms (overlaps L1) m -2.9K × 2 per level
7 L7 clock 66 -> 100 MHz -1.8 to -2.3 ms (not next image) on-card ~5.4-7.0 ms × 0.34
8 L8 G5_OVL -0.03 to -0.07 ms -200 to -300 cycles × 13 items
– L9 tree-seam register +0.1 ms (cost) +1 cycle × 486 tiles × 13 items

7.5 d3 / d5 effects of the same levers [H]

lever d3 W2 (4.51 ms; 3 levels) d5 W2 (6.23 ms; 5 levels) d7 W2 (8.48 ms; 7 levels)
L1 MLPV 132 MHz -0.4 to -0.5 -0.7 to -0.9 -1.0 to -1.2
L2 EAGER + EARLY_A -0.2 to -0.6 -0.4 to -1.0 -0.5 to -1.4
L3 host per tree (constant) -0.4 to -0.9 (largest share: 32 % of the d3 tree is host) -0.4 to -0.9 -0.4 to -0.9
L4 ATT L512 -0.2 -0.35 -0.5
L5 TREE floor -0.15 -0.25 -0.35
ring tooth (sweep metric only) -0.6 (20 reads) -1.1 (36 reads) -1.6 (52 reads)

7.6 Hypotheses for the slow block (section 2), not decided

7.7 Where the "12 ms host / bridge" per round goes (v2) [D from 0R.1-0R.3]

d2 W2 V6: extra 34.0 - 18.4 = 15.6 ms per round.

sub-step ms / round basis
card tree RPC 3.3-3.7 sweep wall 3.73 ms (ctx 45 / 112 mean), ring tooth ~0.4 ms not in serve
KV-only rows (card items, host-MMIO bound) 7.0-7.4 t_card 10.73 - tree; 3.96 rows / round, so 1.77-1.87 ms / row
eager fc + row build (t_fc + t_row) 0.96 measured
_dump() ~1.35 mean box1 replica, 0.2 -> 2.5 ms over 588 trees
tree copy-back, Python glue, req0 bring-up amortised (~0.25), unassigned ~2.5 remainder

d7 W2 V20: extra 45.7 - 21.9 = 23.8 ms per round. Tree ~6.9-8.5 + KV rows ~9.5 (6.23 rows x ~1.53) + fc / row 1.41 + _dump ~1.0 + remainder ~2-4.

7.8 Host-path levers, expected cut per round [H]

lever d2 d7 how computed
H1 KV rows pipelined / batched (target ~0.4 ms / row: one H2C + one issue per rank, arm held, waits overlapped; previous-root row sent during GPU verify) -4.9 -7.7 (1.64 - 0.4) x 3.96 / 6.23 rows
H2 skip GPU _grow -1.5..-3 -3..-5 GPU round slope across d2 -> d7 bars = +0.7 ms / level, including V growth, so drafting <= ~0.6-0.7 ms / level
H3 _dump every N trees -1.35 -1.0 replica mean
H4 one RPC per round (rows + tree) + server-side pipelining -0.2..-0.5 -0.2..-0.5 one IPC round trip + scheduler wake
H5 EAGER + EARLY_A + tree-side host work (L2 + L3) -0.6..-1.5 -0.9..-2.3 section 7.4 ranks 2-3, scaled by levels
H6 golden at md 3 / 5 enables shapes – section 7.9

7.9 Best (md, W) for tok/s given the overhead [H]

Model, validated on the two measured card serves: round(md) = G(md) + tree_wall(md) + 4.45 + 1.64 x rows/round.

shape (W2) accept (GPU bar) today + H1 H2 H3 + tree levers (section 7.4 stack, scaled) GPU-only
d2 V6 2.571 77.7 98.9 ~103 139.5
d3 V10 3.044 84.6 111.4 ~117 158.0
d5 V16 3.758 91.8 126.3 ~136 181.4
d7 V20 3.982 88.6 124.0 ~137 182.5

(The middle columns assume drafting ≈ 0.6 ms / level, 0.4 ms / row and a _dump saving of 1.0 ms.)

8. H6 landed on the host: golden at md 7,5,3,2 (qwen-gptq2-arch golden-md7532, 2026-11-17)

Lever H6 / 0R rank 4 is implemented and mock-proven. Silicon is unrun.

9. Host serve levers L1-L3 implemented (serve owner, 2026-11-17; code 3ecaa99c8, whose comments say "section 8" = this section)

Each lever has its own toggle. With every toggle unset, the serve path behaves as before: no lever keys are added to STATS and _queue_until_decision is not patched (test L2-0). All toggles are read by the bridge in the SGLang process. The sudo'd card server gets its options inside each RPC, so it needs no env.

toggle default effect
EAGLE4_Q5120_KVBATCH=1 (L1) off One round RPC per round carries the KV-only rows and the tree. On the server, ServeSession._kvrows_batch issues the same items in the same order: 0x15C KVONLY arm held across the batch (read back once, re-read after row 1, disarmed once); quiescence and the F24 RoPE tooth once per batch (max ctx); the 4 ranks' FASTIN programs interleaved (c4_card.program_inputs_interleaved); the per-item D956 audit and wait_out_seq kept; a host contiguity tooth on row ctx and root ctx
EAGLE4_Q5120_PREROOT=1 (L1 sub-toggle, needs KVBATCH) off After each tree reply, the server writes that tree's root row as a KV-only item while the GPU verifies. The next round's first row (byte-identical) is skipped; a failed pre-write raises at the next kvrows
EAGLE4_Q5120_SKIP_GPU_TREE=1 (L2) off The GPU draft steps 1..md-1 are deferred, and the card tree REPLACES them. Those steps are queued by DynamicDrafter._queue_until_decision, at prelaunch or at draft time; in mode static the literal _grow does no GPU work. Every round that does not inject a card tree reruns the deferred steps before rank/verify: card_dead, bring-up fail, shape_not_in_golden, request_dead, ctx>window, any card error or exception. STATS["l2"] gives deferred / replaced / gpu_reruns / rerun_ms
EAGLE4_Q5120_STATS_EVERY=N (L3) off (dump every tree) Stats are written every N card trees, plus at every request start (as before), plus an idle flush after EAGLE4_Q5120_STATS_IDLE_S (1.0 s; 0 = off), plus a SIGTERM flush (EAGLE4_Q5120_STATS_SIGTERM=0 = off; chains to the previous handler). Writes are atomic. stats_flush records why
EAGLE4_Q5120_RED=l1_droprow|l2_norerun|l3_noflush off Test-only defect injection, honoured only with CARD=mock

Also in 3ecaa99c8:

9.1 Gates and results [M]

9.2 Next-window serve command (step 9b, levers on)

9.3 v2: audit fixes of 3ecaa99c8 (commit 6fba0c315; audit 16b3337ca) [M]

Verdicts were L1 GO with fixes, PREROOT GO, L2 HOLD pending the F6 condition below, L3 GO with fixes, CTX forwarding GO.

fix what changed in 6fba0c315 gate
F1 the KV batch runs under try/finally; on any exception it best-effort disarms 0x15C bit3 on all 4 ranks (kv_batch_aborted counted) R14b, R21 (exception on row 2: no rank left armed)
F2 the disarm read checks bit3 is still set after the last row, else KVONLY cleared mid-batch (no extra MMIO) R20; E2E RED _v2_redkvc (mock RED l1_kvclear)
F3 interleaved FASTIN programs are now their own sub-toggle EAGLE4_Q5120_KVINTERLEAVE=1 (RPC opt, crosses sudo with no env). KVBATCH alone uses the default per-rank program_inputs. A separate silicon risk: never run on 20,480 B rows; bisect it first if L1 trips G6i; E2E arm _v2_l1i
F4 the inject builds the tokens / scores / parents tensors before the first copy_ (default path: same writes on success) E2E base --same
F7 L3 marks the file dirty on every round, so GPU-tree rounds get flushed too. Report: STATS STALE is a FAIL only with levers.stats_every, a NOTE otherwise; the default dump schedule is unchanged (chosen so the default path stays byte-identical) G9, R17
F8 the tmp name is unique per _dump call L3-1/2
F10 every L1 reply carries the card-server counters (srv_l1: kv_batches, preroot_written / used / errors, kv_batch_aborted) into STATS. The report fails with LEVER NOT ENGAGED when the client ### LEVERS line names a toggle the stats do not show running. --same requires equal CTX_WIN R22, R23, R24; E2E RED _v2_redf10 (KVBATCH=yes)
L2 site deferral and rerun are counted per site (prelaunch = after-verify prelaunch, draft = a new request's first round) E2E cw arms
low preroot_err is counted before begin() drops it; STATS_EVERY=abc becomes a recorded refusal (R25); a hostkit RuntimeError (EAGLE4_CTX_WIN above the built window) becomes a named ServeRefused R25

v2 tests. test_q5120_serve passes 56/56 for 139C+139D and 56/56 for 13A9+13A1. The regression set on the landed repo is all PASS: RECORD_PROFILE_IDENTITY, RECORD_GATE_IDENTITY 111/111, Q5120_HOSTKIT, QGPTQ2_HOSTGUARD 67/0, QGPTQ2_PROFILE 140/0/0. Logs: /mnt/hdd_data/eagle4_scratch/qwen_gptq2_20261116/levers_20261117/regress_v2/.

v1 chain.

v2 chain.

9.3.1 v2 chain results (CARD=mock World7, box1 H200 + real target, humaneval:4 + mt_bench:4, d7 W2 V20 K_C 64) [M]

arm env ACCEPT CARD_RESULT REPORT_RC SAME vs ref STATS rounds/verify l2 deferred (prelaunch/draft) / replaced / reruns (prelaunch/draft) / rerun ms srv_l1 kv_batches / preroot written/used fails
_v2_base (none) 1509/372 = 4.0565 CARD_RESULT PASS 0 - 372 vs sum spec_verify_ct 372 - -
_v2_l1 EAGLE4_Q5120_KVBATCH=1 EAGLE4_Q5120_PREROOT=1 1509/372 = 4.0565 CARD_RESULT PASS 0 output_ids + spec_verify_ct identical on 8/8 requests 372 vs sum spec_verify_ct 372 0 (0/0) / 0 / 0 (0/0) / 0.0 679 / 371/364
_v2_l1i EAGLE4_Q5120_KVBATCH=1 EAGLE4_Q5120_PREROOT=1 EAGLE4_Q5120_KVINTERLEAVE=1 1509/372 = 4.0565 CARD_RESULT PASS 0 output_ids + spec_verify_ct identical on 8/8 requests 372 vs sum spec_verify_ct 372 0 (0/0) / 0 / 0 (0/0) / 0.0 679 / 371/364
_v2_l2 EAGLE4_Q5120_SKIP_GPU_TREE=1 1509/372 = 4.0565 CARD_RESULT PASS 0 output_ids + spec_verify_ct identical on 8/8 requests 372 vs sum spec_verify_ct 372 379 (371/8) / 371 / 0 (0/0) / 0.0 -
_v2_l3 EAGLE4_Q5120_STATS_EVERY=50 1509/372 = 4.0565 CARD_RESULT PASS 0 output_ids + spec_verify_ct identical on 8/8 requests 372 vs sum spec_verify_ct 372 0 (0/0) / 0 / 0 (0/0) / 0.0 -
_v2_all EAGLE4_Q5120_KVBATCH=1 EAGLE4_Q5120_PREROOT=1 EAGLE4_Q5120_KVINTERLEAVE=1 EAGLE4_Q5120_SKIP_GPU_TREE=1 EAGLE4_Q5120_STATS_EVERY=50 1509/372 = 4.0565 CARD_RESULT PASS 0 output_ids + spec_verify_ct identical on 8/8 requests 372 vs sum spec_verify_ct 372 380 (372/8) / 372 / 0 (0/0) / 0.0 679 / 371/364
_v2_cwbase EAGLE4_CTX_WIN=200 1509/372 = 4.0565 CARD_RESULT PASS 0 - 330 vs sum spec_verify_ct 372 - -
_v2_cwl2 EAGLE4_CTX_WIN=200 EAGLE4_Q5120_SKIP_GPU_TREE=1 1509/372 = 4.0565 CARD_RESULT PASS 0 output_ids + spec_verify_ct identical on 8/8 requests 330 vs sum spec_verify_ct 372 124 (116/8) / 116 / 7 (4/3) / 14.0 -
_v2_cwall EAGLE4_CTX_WIN=200 EAGLE4_Q5120_KVBATCH=1 EAGLE4_Q5120_PREROOT=1 EAGLE4_Q5120_KVINTERLEAVE=1 EAGLE4_Q5120_SKIP_GPU_TREE=1 EAGLE4_Q5120_STATS_EVERY=50 1509/372 = 4.0565 CARD_RESULT PASS 0 output_ids + spec_verify_ct identical on 8/8 requests 372 vs sum spec_verify_ct 372 124 (116/8) / 116 / 8 (5/3) / 16.0 205 / 115/111
_v2_redl2 EAGLE4_CTX_WIN=200 EAGLE4_Q5120_SKIP_GPU_TREE=1 EAGLE4_Q5120_RED=l2_norerun 1509/377 = 4.0027 CARD_RESULT FAIL: SAME-C 1 output_ids + spec_verify_ct identical on 3/8 requests: ['req00 spec_verify_ct differ', 're 335 vs sum spec_verify_ct 377 124 (116/8) / 116 / 0 (0/0) / 0.0 - SAME-CONFIG MISMATCH (req00 spec_verify_ct differ; req01 spec_verify_ct differ; req02 spec_verify_ct differ; req03 spec_verify_ct differ)
_v2_redl1 TASKS=humaneval:1 MAXNEW=48 EAGLE4_Q5120_KVBATCH=1 EAGLE4_Q5120_PREROOT=1 EAGLE4_Q5120_RED=l1_droprow 48/6 = 8.0000 CARD_RESULT FAIL: LEVER 1 - 5 vs sum spec_verify_ct 6 0 (0/0) / 0 / 0 (0/0) / 0.0 1 / 0/0 LEVER NOT ENGAGED (PREROOT: levers.preroot True preroot_used 0 (extend rounds 1))
_v2_redkvc TASKS=humaneval:1 MAXNEW=48 EAGLE4_Q5120_KVBATCH=1 EAGLE4_Q5120_PREROOT=1 EAGLE4_Q5120_RED=l1_kvclear 48/6 = 8.0000 CARD_RESULT FAIL: 0 card 1 - 4 vs sum spec_verify_ct 6 0 (0/0) / 0 / 0 (0/0) / 0.0 - 0 card trees (rounds 4, skipped {'card_error_prime': 1, 'request_dead': 3}, errors {}): the accept above is a GPU-tree number
_v2_redl3 TASKS=humaneval:1 MAXNEW=48 EAGLE4_Q5120_STATS_EVERY=1000 EAGLE4_Q5120_STATS_IDLE_S=0 EAGLE4_Q5120_STATS_SIGTERM=0 EAGLE4_Q5120_RED=l3_noflush 48/6 = 8.0000 CARD_RESULT FAIL: 0 card 1 - 1 vs sum spec_verify_ct 6 0 (0/0) / 0 / 0 (0/0) / 0.0 - 0 card trees (rounds 1, skipped {}, errors {}): the accept above is a GPU-tree number
_v2_redf10 TASKS=humaneval:1 MAXNEW=48 EAGLE4_Q5120_KVBATCH=yes 48/6 = 8.0000 CARD_RESULT FAIL: LEVER 1 - 6 vs sum spec_verify_ct 6 - - LEVER NOT ENGAGED (['KVBATCH'] named but the stats carry no levers key)