Qwen drafters on the 7-card FPGA drafter: GPU first, then ours

Experiment QWEN_7CARD_COMPARE_20261117 · author box1-two-v2 · 2026-11-17 · status: running (host fixes and a multi-row KV image in flight) · target model Qwen3.8-27B (verify on one H200); 7 cards = 4 attention cards (ATT, tensor-parallel 4) + 2 MLP cards (MLPV) + verify GPU

On the GPU alone, the new 2-bit Qwen drafter (GPTQ-2) is the better model: +18.6% accept over the old BitNet drafter on the same 411 prompts, and +42% decode tok/s. On our 7-card FPGA drafter it is the opposite: 110.48 tok/s against 182.50 for the GPU alone on the same 8 prompts, where the old drafter ran at 250.27, near GPU parity. Accept is identical on the FPGA and the GPU. The whole loss is card speed: the new drafter puts 3.07× the weights through every tree node on an array that is the same width or narrower.

GPU only: new vs old Qwen drafter, accept, 411 prompts measured
3.990 vs 3.365
static tree w2 d7, 20 verify tokens
humaneval, GPU only: new vs old Qwen drafter measured
360.42 vs 307.36
same build, client, tree; 6 prompts
humaneval 80, 7-card: new vs old Qwen drafter measured
140.15 vs 249.62
GPU only, same 80 prompts: 326.83 vs 256.58
7-card, old Qwen drafter (humaneval 80) measured
250.27 tok/s
GPU only: 256.67
7-card, Llama-3.2-3B record (humaneval 80) measured
961.43 tok/s
GPU only best: 841.27
Card body weights per forward, new / old Qwen measured
435.16 M / 141.56 M
from the safetensors headers

Calculations and hypothesis

Not measured. Every ratio, conversion and hypothesis on this page lives in this bar. Everything below it is a fact or a measured result; a table cell marked derived is one of these ratios.

01GPU-only results (the bar we have to beat)

The H200 runs the target and the drafter itself, with no FPGA. Qwen rows come from one evaluation that used the same SGLang build, client, prompts and tree for both drafters: batch 1, greedy, thinking off, 256 new tokens.

Accept per task, Qwen3.8-27B, static tree width 2 × depth 7, 20 verify tokens

Task (prompts)Old drafter: BitNet step21491New drafter: GPTQ-2 2-bitChange
mt_bench (80)3.4994.028+15.1%
humaneval (30)7.2007.384+2.6%
gsm8k (30)5.8006.136+5.8%
math500 (30)5.1995.695+9.5%
alpaca chat (30)3.2503.729+14.7%
alpaca-zh Chinese (30)1.7762.408+35.6%
multilingual (30)2.1182.806+32.5%
All 411 prompts3.3653.990+18.6%

All values measured. Accept = Σ completion tokens ÷ Σ verify steps, bonus token included. The 2-bit drafter loses 2.0% accept against its own BF16 parent (4.125 vs 4.210, release serve config).

Decode tok/s, GPU only (cooled pass, every prompt starts at ≤ 65 °C, SM clock 1785 MHz)

PromptsTarget aloneOld drafter w2 d7New drafter w2 d7
humaneval (6)61.95307.36360.42
mt_bench (6)61.92105.08146.25
7 tasks × 6 = 42 prompts61.85132.15187.74

All measured. Decode tok/s = Σ(tokens − 1) ÷ Σ(wall − time to first token). The Llama-3.2-3B GPU-only best on the same H200 is 841.27 tok/s @ accept 6.63 (humaneval 80 × 256, depth 9, fa3) measured.

02Our 7-card FPGA drafter results

Same H200 verifies; the 7 cards draft the tree. Every row is paired with a GPU-only run on the same prompts.

DrafterPrompts × max tokensTreeAccept, 7-card / GPUtok/s, 7-cardtok/s, GPU only7-card ÷ GPU derived
Llama-3.2-3B record (bitnet_721)humaneval 80 × 256w2 d11, fa36.91 / 6.63961.43841.27 (d9, GPU best)1.143
Qwen, old: BitNet step21491humaneval 80 × 256w2 d77.14 / 7.13250.27256.670.975
Qwen, new: GPTQ-2, all host fixes so farhumaneval 0–3 + mt_bench 81–84, 8 × 256w2 d7, 20 verify4.0565 / 3.9815110.48182.500.605
· humaneval part, before host fixeshumaneval 0–3w2 d7, 20 verify7.185 / 7.185117.78296.520.397
· mt_bench part, before host fixesmt_bench 81–84w2 d7, 20 verify3.029 / 2.95574.41140.410.530

tok/s and accept measured. The 8-prompt split rows use completion tokens ÷ wall from the per-request records. Per request, the new drafter's completion tokens and verify steps on the cards equal the GPU's exactly on humaneval 0–3 (100/13, 256/39, 165/22, 140/18), so the cards lose no accept.

Read across rows only within the same prompt set. The new drafter's 7-card rows use 8 prompts, half of them mt_bench, and mt_bench decodes at about half humaneval's speed. Its 182.50 GPU bar is therefore not comparable with the old drafter's 256.67, which is humaneval only. On the same prompts, the new drafter is faster than the old one on the GPU: section 01, and the humaneval panel below.

Like for like: humaneval, all 80 prompts × 256 tokens, depth 7 width 2

DrafterGPU only tok/s @ accept7-card tok/s @ accept7-card ÷ GPU derived
Old Qwen: BitNet step21491 (2026-11-09)256.58 @ 7.13249.62 @ 7.140.973
New Qwen: GPTQ-2, FR-Spec images 0x13A9 + 0x13E5, all host fixes so far (2026-11-17)326.83 @ 7.40140.15 @ 7.420.429
humaneval 80: old vs new Qwen drafter, GPU only vs 7-card
bar length to scale, 0 – 400 tok/s
Old drafter · GPU
256.58
New drafter · GPU
326.83
Old drafter · 7-card
249.62
New drafter · 7-card
140.15
GPU only7-card FPGA drafter

All tok/s and accept measured. The new-drafter rows are from the Qwen track session's window 3, config 1 (golden 64/64 PASS, outputs identical 8/8; commit 99ceefe5a, reports/results/SESSION_CONTEXT_BACKUP_20261117.md). The old-drafter rows are from the 2026-11-09 run. The two GPU bars were measured on different days with different SGLang setups. On the 8-prompt mix the same new images give 107.33 (d7) and 108.50 (d5) against GPU-only 182.67 / 181.84.

Host fixes already measured on the new drafter (depth 7, images 0x139C + 0x139D, outputs identical 8/8)

Configurationtok/s
no fixes91.93
+ host lever L194.77
+ stats file written less often96.96
+ GPU no longer drafts its own tree100.77
+ KV rows interleaved across the 4 ATT cards110.48 (36.8 ms per round; GPU only 21.9 ms)

measured. Remaining per-round gap 14.7 ms: card RPCs 13.25 (KV-only row writes about 5.25 + tree 6.9–8.5), row building 0.96, tree copy 0.48.

03Model size, side by side (from the safetensors headers)

Each column is one drafter checkpoint. Shapes are as stored; GPTQ-2 shapes are [in/16, out] int32 (16 two-bit weights per word), shown as in × out. All values measured (header read), except the ratio column derived.

Tensor (card engine)Llama record: bitnet_721Old Qwen: BitNet step21491New Qwen: GPTQ-2 W2A16New ÷ old Qwen
hidden size · heads × head dim · MLP size3072 · 32 × 96 · 81923072 · 24 × 128 · 81925120 · 40 × 128 · 163841.67 · 1.67 · 2.00
q_proj (G4)[3072, 6144] = 18.87 M[3072, 6144] = 18.87 M10240 × 5120 = 52.43 M2.78×
k_proj (G4)18.87 M18.87 M52.43 M2.78×
v_proj (G4)18.87 M18.87 M52.43 M2.78×
o_proj (G6)[3072, 3072] = 9.44 M9.44 M5120 × 5120 = 26.21 M2.78×
gate_proj (G7g)[8192, 3072] = 25.17 M25.17 M5120 × 16384 = 83.89 M3.33×
up_proj (G7u)25.17 M25.17 M83.89 M3.33×
down_proj (G9)[3072, 8192] = 25.17 M25.17 M16384 × 5120 = 83.89 M3.33×
Card body total141.56 M141.56 M435.16 M3.07×
lm_head (G8 exact rescore)[128256, 3072] BF16 = 394.0 M[128256, 3072] BF16 = 394.0 M (128,256-token draft vocabulary, mapped)5120 × 248320 2-bit = 1,271.4 M (+ low-rank 128 head)3.23×
vocabulary the tree scans128,256128,256 (draft)248,3201.94×
fc (runs on the host GPU)[3072, 9216] = 28.31 M[3072, 15360] = 47.19 M15360 × 5120 = 78.64 M1.67×
embed_tokens (host)[128256, 3072][248320, 3072][248320, 5120]1.67×
quantization on the cardternary, packed 2 bitsternary, one scale per tensor2-bit, groups of 128, zero point + bf16 scale per group × column
extra normspost-MLP normattention + MLP sub-normsnone
file size1,915,799,320 B2,692,524,624 B3,030,701,984 B1.13×
Card body weights per forward (millions)
bar length to scale, 0 – 450 M
Llama record
141.56
Old Qwen (BitNet)
141.56
New Qwen (GPTQ-2)
435.16

04Our architecture: engine speed and dimensions

Dimensions of the card images

ItemLlama record setNew Qwen set
ImagesATT 0xE4B713D6 × 4 + MLPV M1361 × 2ATT 0xE4B7139C × 4 + MLPV 0xE4B7139D × 2
ATT tensor-parallel split4 ranks4 ranks, 10 heads each
ATT GEMV array lanes × K_PAR512 × 16256 × 16 (halved to fit 16-bit activations)
MLPV GEMV array lanes256256
Drafter clock66.001 MHz (15.151 ns)66 MHz
MLPV array clock132 MHz (2× fast clock on)66 MHz (fast clock off)
Tree top-K width (TREE_P)3232
Activationsint (record lane)16-bit (A16, user ruling)
Routed timing, true constraintsATT +0.044 / +0.010 ns; MLPV +0.041 / +0.016 nsMLPV 0x139D +0.030 / +0.022 ns

measured (build configs and timing reports). Vivado .rpt files give clock, slack and area only; engine time comes from on-card cycle counters (Llama) or xsim (Qwen).

Engine cycles per tree node

CardEngineLlama record, on cards measured→derivedNew Qwen, xsim measuredQwen ÷ Llama derived
ATTG1 norm4567131.6×
ATTG4 QKV1,4214,8253.4×
ATTG5 attention1,919 (context 566)612 (context 50)not comparable
ATTG6 o-proj + cross-card sum686903 (no link delay)1.3×
ATTG7g gate1,3973,9372.8×
ATTG7u up5572,5854.6×
ATTSiLU1332622.0×
ATTcard-to-card handoff8261,248–1,9141.5–2.3×
ATTtotal busy7,39415,085–15,7572.0–2.1×
MLPVG9 down8505,1336.0×
MLPVG11 norm4667231.6×
MLPVtree step (logit scan + top-K)2,2246,8163.1×
MLPVG8 exact rescore7471,896–2,1092.5–2.8×
MLPVtotal busy5,32715,6162.9×

Cycles at the 66 MHz drafter clock. Llama: median of 20 on-card census passes, depth-11 tree of 21 nodes, converted per node (formula in the top bar). Qwen: xsim of the image sources. No on-card census exists for Qwen yet.

Whole tree and whole round

ItemLlama recordOld QwenNew Qwen
Card time for one tree2.878 ms (depth 11)about 2.7 ms (depth 7)7.0 ms issue-to-done; 8.5 ms wall; 9–10 ms in the golden run (depth 7)
ms per round, 7-card7.187—36.8
ms per round, GPU only——21.9

measured. The old Qwen tree time comes from the Qwen track session's report.

05Conclusions

What these numbers do not say

06Evidence

WhatPath (repo-relative)md5 / commit
GPU accept and tok/s, old vs new Qwen drafterreports/results/QWEN27B_DRAFTER_20261020/OPEN_RELEASE_DRAFTER_EVAL_20261102.md §0.1, §0.2, :121old drafter model.safetensors 32f20d0d1c9d86fb5afb90088c8c5f02
New drafter checkpointreports/results/QWEN27B_DRAFTER_20261020/NEW_GPTQ2_LMHEAD2_CKPT_20261102.mdmodel.safetensors e41d16462c11f14eebc8b5573f0f110d; config 243b337fec280f1731cea9581772e905
Tensor listings, all three drafters (+ script)reports/results/QWEN_GPTQ2_20261116/weight_listing_20261117/commit 03a1e1ff6; configs 9fb170ca5518aee2103635eef66398b8 / 8d01c2517f998f715d15180cdb85ab8b / 243b337f
New Qwen 7-card and GPU-bar serves, per requestreports/results/QWEN_GPTQ2_20261116/d7_levers_evidence/serve/qgserve_{silicon_d7w2v20kc64_pwfifo_20261117_043538,none_d7w2v20kc64_gpubar_20261117_043950}.jsonlMD5SUMS.txt in that folder
New Qwen host-fix ladder, 110.48 tok/scommits 398a14af9, 2a73c0b63; reports/results/QWEN_GPTQ2_20261116/D7_TREE_LEVERS_20261117.md—
New Qwen per-node cycles (xsim)reports/results/QWEN_GPTQ2_20261116/D7_TREE_LEVERS_20261117.md §1.4 (:184-198)—
Llama per-engine census and step budgetreports/results/DRAFTER_SPEED_RESEARCH_20261107/SCALING_4C_VS_7C_PHASE_RANK_20261116.md :51-88census m1361_census_md11.json
Llama record 961.43, dataset and drafter pathreports/results/DRAFTER_SPEED_RESEARCH_20261107/ATT_COMBO_SILICON_20261116.md :16; record serve log t1 (humaneval 80 × 256, bitnet_721)—
Llama GPU-only 841.27reports/results/DRAFTER_SPEED_RESEARCH_20261107/BEST_VS_BEST_GPU_20261112.md :4-5—
Old Qwen 7-card 250.27 vs GPU 256.67reports/results/FPGA_QWEN_ALLSETS_TEMPS_20261110/, reports/results/GPU_BASELINE_QWEN_20261109/REPORT.md—
Array lanes and clocksverilog/clean7c/ref/att_13be/packaged_ip/src/eagle4_config.svh:332,444; verilog/clean7c/ref/mlpv_1361/packaged_ip/src/eagle4_config.svh:302,429,546; reports/results/QWEN_GPTQ2_20261116/DESIGN_ARCH_SHIFT.md:39,61—
Full write-upreports/results/QWEN_GPTQ2_20261116/WHY_SLOWER_LAYERS_PHASES_20261117.mdcommits 90b4b3fc2, 03a1e1ff6, 48fa4cf91