On the GPU alone, the new 2-bit Qwen drafter (GPTQ-2) is the better model: +18.6% accept over the old BitNet drafter on the same 411 prompts, and +42% decode tok/s. On our 7-card FPGA drafter it is the opposite: 110.48 tok/s against 182.50 for the GPU alone on the same 8 prompts, where the old drafter ran at 250.27, near GPU parity. Accept is identical on the FPGA and the GPU. The whole loss is card speed: the new drafter puts 3.07× the weights through every tree node on an array that is the same width or narrower.
Not measured. Every ratio, conversion and hypothesis on this page lives in this bar. Everything below it is a fact or a measured result; a table cell marked derived is one of these ratios.
The H200 runs the target and the drafter itself, with no FPGA. Qwen rows come from one evaluation that used the same SGLang build, client, prompts and tree for both drafters: batch 1, greedy, thinking off, 256 new tokens.
| Task (prompts) | Old drafter: BitNet step21491 | New drafter: GPTQ-2 2-bit | Change |
|---|---|---|---|
| mt_bench (80) | 3.499 | 4.028 | +15.1% |
| humaneval (30) | 7.200 | 7.384 | +2.6% |
| gsm8k (30) | 5.800 | 6.136 | +5.8% |
| math500 (30) | 5.199 | 5.695 | +9.5% |
| alpaca chat (30) | 3.250 | 3.729 | +14.7% |
| alpaca-zh Chinese (30) | 1.776 | 2.408 | +35.6% |
| multilingual (30) | 2.118 | 2.806 | +32.5% |
| All 411 prompts | 3.365 | 3.990 | +18.6% |
All values measured. Accept = Σ completion tokens ÷ Σ verify steps, bonus token included. The 2-bit drafter loses 2.0% accept against its own BF16 parent (4.125 vs 4.210, release serve config).
| Prompts | Target alone | Old drafter w2 d7 | New drafter w2 d7 |
|---|---|---|---|
| humaneval (6) | 61.95 | 307.36 | 360.42 |
| mt_bench (6) | 61.92 | 105.08 | 146.25 |
| 7 tasks × 6 = 42 prompts | 61.85 | 132.15 | 187.74 |
All measured. Decode tok/s = Σ(tokens − 1) ÷ Σ(wall − time to first token). The Llama-3.2-3B GPU-only best on the same H200 is 841.27 tok/s @ accept 6.63 (humaneval 80 × 256, depth 9, fa3) measured.
Same H200 verifies; the 7 cards draft the tree. Every row is paired with a GPU-only run on the same prompts.
| Drafter | Prompts × max tokens | Tree | Accept, 7-card / GPU | tok/s, 7-card | tok/s, GPU only | 7-card ÷ GPU derived |
|---|---|---|---|---|---|---|
| Llama-3.2-3B record (bitnet_721) | humaneval 80 × 256 | w2 d11, fa3 | 6.91 / 6.63 | 961.43 | 841.27 (d9, GPU best) | 1.143 |
| Qwen, old: BitNet step21491 | humaneval 80 × 256 | w2 d7 | 7.14 / 7.13 | 250.27 | 256.67 | 0.975 |
| Qwen, new: GPTQ-2, all host fixes so far | humaneval 0–3 + mt_bench 81–84, 8 × 256 | w2 d7, 20 verify | 4.0565 / 3.9815 | 110.48 | 182.50 | 0.605 |
| · humaneval part, before host fixes | humaneval 0–3 | w2 d7, 20 verify | 7.185 / 7.185 | 117.78 | 296.52 | 0.397 |
| · mt_bench part, before host fixes | mt_bench 81–84 | w2 d7, 20 verify | 3.029 / 2.955 | 74.41 | 140.41 | 0.530 |
tok/s and accept measured. The 8-prompt split rows use completion tokens ÷ wall from the per-request records. Per request, the new drafter's completion tokens and verify steps on the cards equal the GPU's exactly on humaneval 0–3 (100/13, 256/39, 165/22, 140/18), so the cards lose no accept.
Read across rows only within the same prompt set. The new drafter's 7-card rows use 8 prompts, half of them mt_bench, and mt_bench decodes at about half humaneval's speed. Its 182.50 GPU bar is therefore not comparable with the old drafter's 256.67, which is humaneval only. On the same prompts, the new drafter is faster than the old one on the GPU: section 01, and the humaneval panel below.
| Drafter | GPU only tok/s @ accept | 7-card tok/s @ accept | 7-card ÷ GPU derived |
|---|---|---|---|
| Old Qwen: BitNet step21491 (2026-11-09) | 256.58 @ 7.13 | 249.62 @ 7.14 | 0.973 |
| New Qwen: GPTQ-2, FR-Spec images 0x13A9 + 0x13E5, all host fixes so far (2026-11-17) | 326.83 @ 7.40 | 140.15 @ 7.42 | 0.429 |
All tok/s and accept measured. The new-drafter rows are from the Qwen track session's window 3, config 1 (golden 64/64 PASS, outputs identical 8/8; commit 99ceefe5a, reports/results/SESSION_CONTEXT_BACKUP_20261117.md). The old-drafter rows are from the 2026-11-09 run. The two GPU bars were measured on different days with different SGLang setups. On the 8-prompt mix the same new images give 107.33 (d7) and 108.50 (d5) against GPU-only 182.67 / 181.84.
| Configuration | tok/s |
|---|---|
| no fixes | 91.93 |
| + host lever L1 | 94.77 |
| + stats file written less often | 96.96 |
| + GPU no longer drafts its own tree | 100.77 |
| + KV rows interleaved across the 4 ATT cards | 110.48 (36.8 ms per round; GPU only 21.9 ms) |
measured. Remaining per-round gap 14.7 ms: card RPCs 13.25 (KV-only row writes about 5.25 + tree 6.9–8.5), row building 0.96, tree copy 0.48.
Each column is one drafter checkpoint. Shapes are as stored; GPTQ-2 shapes are [in/16, out] int32 (16 two-bit weights per word), shown as in × out. All values measured (header read), except the ratio column derived.
| Tensor (card engine) | Llama record: bitnet_721 | Old Qwen: BitNet step21491 | New Qwen: GPTQ-2 W2A16 | New ÷ old Qwen |
|---|---|---|---|---|
| hidden size · heads × head dim · MLP size | 3072 · 32 × 96 · 8192 | 3072 · 24 × 128 · 8192 | 5120 · 40 × 128 · 16384 | 1.67 · 1.67 · 2.00 |
| q_proj (G4) | [3072, 6144] = 18.87 M | [3072, 6144] = 18.87 M | 10240 × 5120 = 52.43 M | 2.78× |
| k_proj (G4) | 18.87 M | 18.87 M | 52.43 M | 2.78× |
| v_proj (G4) | 18.87 M | 18.87 M | 52.43 M | 2.78× |
| o_proj (G6) | [3072, 3072] = 9.44 M | 9.44 M | 5120 × 5120 = 26.21 M | 2.78× |
| gate_proj (G7g) | [8192, 3072] = 25.17 M | 25.17 M | 5120 × 16384 = 83.89 M | 3.33× |
| up_proj (G7u) | 25.17 M | 25.17 M | 83.89 M | 3.33× |
| down_proj (G9) | [3072, 8192] = 25.17 M | 25.17 M | 16384 × 5120 = 83.89 M | 3.33× |
| Card body total | 141.56 M | 141.56 M | 435.16 M | 3.07× |
| lm_head (G8 exact rescore) | [128256, 3072] BF16 = 394.0 M | [128256, 3072] BF16 = 394.0 M (128,256-token draft vocabulary, mapped) | 5120 × 248320 2-bit = 1,271.4 M (+ low-rank 128 head) | 3.23× |
| vocabulary the tree scans | 128,256 | 128,256 (draft) | 248,320 | 1.94× |
| fc (runs on the host GPU) | [3072, 9216] = 28.31 M | [3072, 15360] = 47.19 M | 15360 × 5120 = 78.64 M | 1.67× |
| embed_tokens (host) | [128256, 3072] | [248320, 3072] | [248320, 5120] | 1.67× |
| quantization on the card | ternary, packed 2 bits | ternary, one scale per tensor | 2-bit, groups of 128, zero point + bf16 scale per group × column | |
| extra norms | post-MLP norm | attention + MLP sub-norms | none | |
| file size | 1,915,799,320 B | 2,692,524,624 B | 3,030,701,984 B | 1.13× |
| Item | Llama record set | New Qwen set |
|---|---|---|
| Images | ATT 0xE4B713D6 × 4 + MLPV M1361 × 2 | ATT 0xE4B7139C × 4 + MLPV 0xE4B7139D × 2 |
| ATT tensor-parallel split | 4 ranks | 4 ranks, 10 heads each |
| ATT GEMV array lanes × K_PAR | 512 × 16 | 256 × 16 (halved to fit 16-bit activations) |
| MLPV GEMV array lanes | 256 | 256 |
| Drafter clock | 66.001 MHz (15.151 ns) | 66 MHz |
| MLPV array clock | 132 MHz (2× fast clock on) | 66 MHz (fast clock off) |
| Tree top-K width (TREE_P) | 32 | 32 |
| Activations | int (record lane) | 16-bit (A16, user ruling) |
| Routed timing, true constraints | ATT +0.044 / +0.010 ns; MLPV +0.041 / +0.016 ns | MLPV 0x139D +0.030 / +0.022 ns |
measured (build configs and timing reports). Vivado .rpt files give clock, slack and area only; engine time comes from on-card cycle counters (Llama) or xsim (Qwen).
| Card | Engine | Llama record, on cards measured→derived | New Qwen, xsim measured | Qwen ÷ Llama derived |
|---|---|---|---|---|
| ATT | G1 norm | 456 | 713 | 1.6× |
| ATT | G4 QKV | 1,421 | 4,825 | 3.4× |
| ATT | G5 attention | 1,919 (context 566) | 612 (context 50) | not comparable |
| ATT | G6 o-proj + cross-card sum | 686 | 903 (no link delay) | 1.3× |
| ATT | G7g gate | 1,397 | 3,937 | 2.8× |
| ATT | G7u up | 557 | 2,585 | 4.6× |
| ATT | SiLU | 133 | 262 | 2.0× |
| ATT | card-to-card handoff | 826 | 1,248–1,914 | 1.5–2.3× |
| ATT | total busy | 7,394 | 15,085–15,757 | 2.0–2.1× |
| MLPV | G9 down | 850 | 5,133 | 6.0× |
| MLPV | G11 norm | 466 | 723 | 1.6× |
| MLPV | tree step (logit scan + top-K) | 2,224 | 6,816 | 3.1× |
| MLPV | G8 exact rescore | 747 | 1,896–2,109 | 2.5–2.8× |
| MLPV | total busy | 5,327 | 15,616 | 2.9× |
Cycles at the 66 MHz drafter clock. Llama: median of 20 on-card census passes, depth-11 tree of 21 nodes, converted per node (formula in the top bar). Qwen: xsim of the image sources. No on-card census exists for Qwen yet.
| Item | Llama record | Old Qwen | New Qwen |
|---|---|---|---|
| Card time for one tree | 2.878 ms (depth 11) | about 2.7 ms (depth 7) | 7.0 ms issue-to-done; 8.5 ms wall; 9–10 ms in the golden run (depth 7) |
| ms per round, 7-card | 7.187 | — | 36.8 |
| ms per round, GPU only | — | — | 21.9 |
measured. The old Qwen tree time comes from the Qwen track session's report.
| What | Path (repo-relative) | md5 / commit |
|---|---|---|
| GPU accept and tok/s, old vs new Qwen drafter | reports/results/QWEN27B_DRAFTER_20261020/OPEN_RELEASE_DRAFTER_EVAL_20261102.md §0.1, §0.2, :121 | old drafter model.safetensors 32f20d0d1c9d86fb5afb90088c8c5f02 |
| New drafter checkpoint | reports/results/QWEN27B_DRAFTER_20261020/NEW_GPTQ2_LMHEAD2_CKPT_20261102.md | model.safetensors e41d16462c11f14eebc8b5573f0f110d; config 243b337fec280f1731cea9581772e905 |
| Tensor listings, all three drafters (+ script) | reports/results/QWEN_GPTQ2_20261116/weight_listing_20261117/ | commit 03a1e1ff6; configs 9fb170ca5518aee2103635eef66398b8 / 8d01c2517f998f715d15180cdb85ab8b / 243b337f |
| New Qwen 7-card and GPU-bar serves, per request | reports/results/QWEN_GPTQ2_20261116/d7_levers_evidence/serve/qgserve_{silicon_d7w2v20kc64_pwfifo_20261117_043538,none_d7w2v20kc64_gpubar_20261117_043950}.jsonl | MD5SUMS.txt in that folder |
| New Qwen host-fix ladder, 110.48 tok/s | commits 398a14af9, 2a73c0b63; reports/results/QWEN_GPTQ2_20261116/D7_TREE_LEVERS_20261117.md | — |
| New Qwen per-node cycles (xsim) | reports/results/QWEN_GPTQ2_20261116/D7_TREE_LEVERS_20261117.md §1.4 (:184-198) | — |
| Llama per-engine census and step budget | reports/results/DRAFTER_SPEED_RESEARCH_20261107/SCALING_4C_VS_7C_PHASE_RANK_20261116.md :51-88 | census m1361_census_md11.json |
| Llama record 961.43, dataset and drafter path | reports/results/DRAFTER_SPEED_RESEARCH_20261107/ATT_COMBO_SILICON_20261116.md :16; record serve log t1 (humaneval 80 × 256, bitnet_721) | — |
| Llama GPU-only 841.27 | reports/results/DRAFTER_SPEED_RESEARCH_20261107/BEST_VS_BEST_GPU_20261112.md :4-5 | — |
| Old Qwen 7-card 250.27 vs GPU 256.67 | reports/results/FPGA_QWEN_ALLSETS_TEMPS_20261110/, reports/results/GPU_BASELINE_QWEN_20261109/REPORT.md | — |
| Array lanes and clocks | verilog/clean7c/ref/att_13be/packaged_ip/src/eagle4_config.svh:332,444; verilog/clean7c/ref/mlpv_1361/packaged_ip/src/eagle4_config.svh:302,429,546; reports/results/QWEN_GPTQ2_20261116/DESIGN_ARCH_SHIFT.md:39,61 | — |
| Full write-up | reports/results/QWEN_GPTQ2_20261116/WHY_SLOWER_LAYERS_PHASES_20261117.md | commits 90b4b3fc2, 03a1e1ff6, 48fa4cf91 |