fpga-archive-reports-arch · EAGLE4 FPGA archive

Source reports/results/DRAFTER_SPEED_RESEARCH_20261107/DESIGN_FABLE_DRAFT_AHEAD_20261117.md · source md5 cf8ae6a9df2fee96a6d80d6f630709a3 · archived 2026-11-18

Download PDFOpen PDFRaw markdown

Draft-ahead (card drafts tree t+1 while the H200 verifies tree t): measured hit rates, what is exact, and the price of the inexact variant (2026-11-17)

Program 1: ONE drafter on the talon2 V80s (4 ATT + 2 MLPV), single stream, H200 verifies. Llama-3.2-3B bitnet_721 and Qwen3.8-27B GPTQ-2 W2A16. Motto: a wall is a measured constraint to engineer around, toward the fastest flashable PDI. User directive: "make the drafter faster". Fable design agent for box1-two-v2. Read-only study + stdlib replay. No build, flash, card access, GPU run, kill, move or removal. talon2: one read-only tar of serve logs, nothing run. Rescoped mid-task by box1-two-v2: (1) verify that EXACT draft-ahead has hit rate 0; (2) the drafter-own-hidden variant is an ACCURACY TRADE (user ruling): quantify it, do not design toward shipping it; (3) use the window-2 / window-3 Qwen silicon logs; (4) list the exact overlap that remains. Tags: [M] measured (file cited), [D] arithmetic on [M], [P] projection or assumption. Short paths: R/ = this directory, E/ = R/draft_ahead_20261117/, Q/ = reports/results/QWEN_GPTQ2_20261116/, S/ = scripts/4c1/, B/ = scripts/qwen27b_open_release_20261102/c4q5120/.

0. Answer first

  1. Exact draft-ahead has hit rate 0 by construction, on both models. Confirmed in code [M]. The next tree's root row is embed[bonus token] | fc(target hidden of the last accepted node), and every KV row of an accepted token is built from the target's hidden of that token. All of these are outputs of the verify that draft-ahead wants to overlap. Citations in section 1. box1-qwen-junior's finding (Q/D7_HOSTGAP_20261117.md:278-296) is correct, and the same holds for Llama.
  2. The "hit rates" below are therefore hit rates of the INEXACT variant only (root = the drafter's own leaf state; output tokens stay exact because the target still verifies, but trees and accept change). That variant is an accuracy trade and a user ruling. Nothing here is designed toward shipping it; sections 3-4 are its price list.
  3. Measured hit rates [M].
    drafter, set, config full path accepted (upper bound of any single-path guess) h1: on the best leaf and next token = drafter top-1 h2: accepted path is a prefix of the best path h3: 2 leaves x top-1 / x top-2
    Qwen GPTQ-2 silicon, humaneval-80, d7 W2 (1,994 rounds) 85.96 % (1,714) about 82-84 % [D] (leaf top-1 not observable; x 0.975-0.987 level-1 rate) not observable (log holds level 1 only) not observable
    Qwen silicon, mt_bench:4, d7 W2 (275 rounds) 2.9 % (8) <= 2.9 % – –
    Qwen silicon, he4 + mt4 mix, d7 (363 rounds) 22.0 % (80) <= 22 % – –
    Qwen silicon, humaneval:4, d5 / d2 89.5 % / 98.6 % – – –
    Llama GPU fp16 drafter, humaneval 20 x 512, D = 7 (863 trees) 41.6 % 26.0 % (224) 57.2 % 32.6 % / 35.5 %
    same, D = 11 23.5 % 15.3 % (132) 50.3 % 18.8 % / 20.2 %
    • The ARCH_RETHINK fit for Qwen humaneval (0.86 full depth, hit 0.84) is now a measurement: 0.8596.
    • h2 buys nothing: when the accept stops inside the tree, the bonus token is by construction not a drafted child, so no resident node can be the next root.
    • A Qwen humaneval-80 silicon serve DOES exist (talon2 runs/qgwin_20261117w3c01/serve/*he80full*, copied to E/evidence/). The Llama 2-bit card drafter and Llama humaneval-80 still have no per-step log.
  4. Price of the inexact variant on Qwen humaneval-80 (replay of the 2,074 logged silicon rounds, section 3) [D on M, continuation accept is P].
    continuation accept assumption tokens per verify wall tok/s (measured base 140.15; replay base 140.7) gain
    upper bound: a continuation tree accepts what the logged fresh tree accepted 7.42 (unchanged) 200.6 +42.6 %
    hazard saturates at the level-7 value (what Llama's measured deep levels do) 7.07 (−4.7 %) 187.4 +33 %
    hazard keeps its measured linear growth with depth 6.31 (−15 %) 160-164 +14 to +16 %
    twice that slope 6.10 (−18 %) 155-157 +10 to +11 %
    • GPU-only on the same 80 prompts is 326.83 tok/s [M]. Even the upper bound does not reach GPU-only (200.6 / 326.8 = 0.61). The ARCH_RETHINK "36.5 → 25 ms" was on the he4 + mt4 round; on humaneval-80 the decode round is 49.5 ms [M] and falls to 33.5 ms at best.
    • Accept of a continuation tree is NOT-MEASURED for Qwen. Llama's measured analogue: a continuation collects 4.79 drafted tokens against 5.47 for a fresh tree at the same kind of position (−12 %), with a per-level hazard that is flat from level 3 to level 15 (section 2.2).
  5. Llama: unchanged verdict, marginal. Hit 15.3 % at d11. R/ARCH_RETHINK_7CARD_20261117.md:192-198: +3.3 % at zero miss cost, break-even at 0.3 ms miss cost. It is the same accuracy trade.
  6. Exact overlap that remains is small: about 1.5-2 ms of a 49.5 ms Qwen round (<= 4 %) and nothing new on Llama (section 5). The exact route to the same window is a faster verify-to-tree path: the in-RPC KV rows cost 1.27 ms each [D on M] and are 9.6 ms of a steady humaneval round plus 17.4 s of 102.7 s (16.9 %) as prompt rows; c5 KV_MROW addresses exactly that.
  7. One exact, config-only alternative has about the same modelled gain as the pessimistic draft-ahead rows: a deeper plain tree. With the measured hazards extended linearly, d11 gives about +11 % and d15 about +16 % decode tok/s on humaneval [P], no RTL, same exactness regime as every depth sweep. It needs goldens at md 11 / 15 and a window. It and draft-ahead rest on the same unknown, the Qwen hazard at levels 8-15.
  8. Cheapest measurement that decides both (no card): one CARD=mock (or GPU-only with a per-round trace) humaneval run at depth 15 on a free H200; E/qwen_hit_rates.py then prints the level 1-15 hazards. Recipe in section 6. Not run here: the box1 H200 belongs to the demo and talon2 is junior's.

1. Why the exact variant cannot hit [M, code]

input of tree t+1 Qwen GPTQ-2 path Llama 7-card path a verify-t output?
root token B/c4q5120_bridge.py:797-799 r = ext.last[0], tok = ext.tokens[r] (extend buffers written by the verify's accept); first round :808 b.spec_info.bonus_tokens[0] S/c4_cardproc.py:668 drv.root_image(ids[n - 1], …), ids = the verify-extend gather's input ids yes
root feature h_fc B/c4q5120_bridge.py:803-805 _c4q_orig_ptf(inner, ext.input_hidden[r:r+1]) = fc(target aux hidden, 15,360) S/c4_cardproc.py:654,668 entry = pm.get(S-1), entry[1] = the captured target 3H row; scripts/fpga_tree_marshaller.py:1414-1428 yes
KV rows of accepted tokens B/c4q5120_bridge.py:23-24 and :1047,1059: row j = `embed[tok_j] fc(aux row j)` sent as KV-only rows S/c4_cardproc.py:659-663 ses.add_rows(ps, k, v) + ses.sync: K/V of the GPU drafter's extend on target features, imported by KV_LOAD
ctx_len, md host arithmetic host arithmetic known once the accept length is known

2. Measured hit rates

2.1 Qwen GPTQ-2, silicon [M]

Source: STATS["trees_log"] of the serve bridge (B/c4q5120_bridge.py:1125-1126: request, ctx, root token, md, W, level-1 children) joined with the per-request output_ids. Advance per round = difference of consecutive root positions (accepted drafted tokens + bonus). Analyser E/qwen_hit_rates.py (md5 9ed11e89), output E/qwen_hit_rates.out. Asserted per round: root token == output_ids[ctx − prompt_tokens].

run (ATT + MLPV) set rounds (non-final) advance histogram full path mean advance level-1 accepted; by rank 0
…w3c1g_he80full_20261117_154118 (0x13A9 + 0x13E5), d7 W2 V20 K_C 64 humaneval 80 x 256 1,994 1:24 2:30 3:35 4:44 5:44 6:48 7:55 8:1714 0.8596 7.5075 98.80 %; 98.73 % of those
…w3c1b_full_20261117_152512 (same images) humaneval:4 88 8:72 0.818 7.35 98.9 %; 98.9 %
same mt_bench:4 275 1:64 2:69 3:52 4:33 5:21 6:22 7:6 8:8 0.029 3.03 76.7 %; 85.3 %
window 2 arm d Q/d7_hostgap_evidence/silicon_w2/…w2d_fullki… (0x139C + 0x139D) he4 / mt4 88 / 276 (8:72) / (8:7) 0.818 / 0.025 7.35 / 3.02 –
…w3c1d_full5… d5 he4 / mt4 114 / 274 – 0.895 / 0.131 5.71 / 3.00 –
…hgd2_best… d2 he4 / mt4 219 / 361 – 0.986 / 0.540 2.97 / 2.33 –

2.2 Llama [M, GPU fp16 drafter traces of 10-20; 2-bit card drafter: no per-step log]

Traces /mnt/data/eagle4_scratch/4c1_spec_20261020/e1_traces/d3trace_he_d15_20261020_214838.jsonl (863 trees) and d3trace_he_d7_20261020_215113.jsonl (1,099). Analyser E/llama_hit_rates.py (md5 c77f8504, reuses S/d3_trace/analyze_overlap.py e35292a5), output E/llama_hit_rates.out.

D = 7 D = 11
h1 (best leaf accepted and next token = drafter top-1) 26.0 % (224 / 863) 15.3 % (132 / 863)
h2 (accepted path, k >= 1, is a prefix of the best path) 57.2 % 50.3 %
h3, 2 leaves x top-1 (2 parallel continuations) 32.6 % 18.8 %
h3, 2 leaves x top-2 (4 continuations) 35.5 % 20.2 %

3. The inexact variant as a decision input (Qwen)

3.1 What it is, stated once (not a shipping design)

While verify t runs, the cards (a) body-pass the best leaf of tree t, (b) take its top-1 child as the root, (c) draft a d-level tree below it on the K/V the tree's own path left in the lane-chain slots. If verify t accepts the whole best path and its bonus token equals that root, tree t+1 is already there and goes straight to verify t+1; the exact KV rows of round t are ingested lazily in the next window. Otherwise the result is dropped and today's serial round runs. It is the same computation as a 2x deeper tree whose beam collapses to its best node at the split, verified in two halves.

3.2 Round costs used [M / D], all on the humaneval-80 silicon run

constant value derivation
decode per round, silicon 49.50 ms sum(wall − ttft) / 2,074 rounds, he80full.jsonl
decode per round, GPU-only 20.11 ms …w3c1h_he80gpu….jsonl (95491610), 2,081 rounds; wall 326.83 tok/s
TREE (card tree + IPC) 9.32 ms …hgt_tim….stats.json (f94e06ae): srv.tree_ms mean 8.73 over 363 rounds + (RPC − busy) median 0.60
KVROW (one in-RPC KV-only row) 1.275 ms (t_card 53.35 s − 2,074 x TREE) / 26,685 in-RPC rows; the timing arm fits 1.20 ms
ROWB (bridge row build) 0.158 ms / row t_row 4.87 s / rows
REST (verify + scheduler + inject) 21.43 ms (decode − t_card − t_row) / rounds
prompt rows 17.4 s of 102.7 s decode (16.9 %) 13,630 rows x KVROW; mean prompt 171 tokens
CONT_X (continuation tree / plain tree) 14 / 13 one extra body pass for the leaf [P]

3.3 Replay mock

E/draft_ahead_mock.py (stdlib; header documents policy and every assumption). It replays the logged rounds of each request on their real token stream and re-chunks them under the policy.

humaneval-80, 20 seeds, wall tok/s (base 140.7):

scenario C = 1 C = 2 C = 4 no cap hit per opportunity (no cap) tokens per verify (no cap)
opt 167.8 (+19.2 %) 179.2 (+27.4 %) 188.9 (+34.3 %) 200.6 (+42.6 %) 0.839 7.42
flat 164.3 (+16.7 %) 173.4 (+23.3 %) 180.7 (+28.5 %) 187.4 (+33.2 %) 0.740 7.07
lin 160.3 (+13.9 %) 163.7 (+16.3 %) 163.2 (+16.0 %) 162.5 (+15.5 %) 0.548 6.31
lin2 156.6 (+11.3 %) 157.0 (+11.6 %) 155.4 (+10.4 %) 155.2 (+10.3 %) 0.491 6.10

tok/s against hit rate (scenario opt, a logged full-path round hits with probability p): hit per round 0 → 140.7; 0.17 → 149.8; 0.35 → 160.5; 0.52 → 172.3; 0.69 → 186.2; 0.86 → 202.7. About +0.5 % tok/s per point of hit rate.

he4 + mt4 mix (w3c1b, base 107.7): opt +5.1 % (C = 1) to +8.6 % (no cap), hit per opportunity 0.12-0.18. The hazard scenarios are not meaningful on a two-task mix and are not quoted. mt_bench alone is at most 2.9 % hits: no gain.

3.4 Miss cost and the abort path [M code, U silicon]

3.5 What the variant would need (spec level only, in case the user rules for it)

  1. Loop sequencer: a CONTINUE arm rooted at a resident node (leaf body pass, then the tree below its top-1 child, K/V read from the lane-chain path slots). No array change.
  2. MLPV: export of the leaf's top-1 token before the continuation (the host needs it to test the hit).
  3. Host: queue of one pending tree; hit test = advance == md + 1 and bonus == exported token; lazy exact row ingest ordered before the next continuation; goldens for the continuation tree from a bit-true model (it is a deterministic function of the same inputs as today's tree t).
  4. A host abort only if depth is pushed past the verify window. First gate, before any RTL: the level 8-15 hazard measurement of section 6. Kill line from ARCH_RETHINK (R/ARCH_RETHINK_7CARD_20261117.md:214): humaneval hit rate below 0.4; measured 0.86, so the gate that remains is the continuation accept.

4. Llama

5. Exact overlap that remains (same trees, same accept)

item status ceiling per Qwen d7 round source
PREROOT: previous root's row written in the idle window on 1.26 ms (taken) Q/D7_HOSTGAP_20261117.md:101
EARLY (extend rows sent at prelaunch) built, default off 0.30-0.40 ms same :33
ROWVEC / TREEPIN / PREPOISON built 0.40-0.57 / <= 0.60 / about 0.1 ms same :31-34
cfg + loop-register pre-write in the idle window not built <= 0.2 ms same :287
pre-quantised embedding half of every row (static Q4.12 table instead of gather + q16 per round) not built <= about 1.0 ms: (t_row 2.35 − t_fc 0.32) x the emb share; overlaps ROWVEC [D] he80 stats
prefetch of top-k bonus candidates no ceiling 0 on a mid-tree stop the bonus is by construction not a drafted child; on a full path it is a child of the never-forwarded leaf
KV rows of predicted tokens during verify impossible exactly 0 row = f(target hidden)
batch-2 ping-pong preserved lever aggregate only, not single stream policy

6. One measurement decides the continuation accept and the plain-depth alternative

7. NOT-MEASURED

8. Files

E/qwen_hit_rates.py 9ed11e89, E/llama_hit_rates.py c77f8504, E/draft_ahead_mock.py (md5 in E/MD5SUMS.txt), outputs E/*.out, E/mock_constants.json, evidence copies E/evidence/ (md5 in E/MD5SUMS.txt).