fpga-archive-reports-arch · EAGLE4 FPGA archive
Revising the drafter GEMM: the 7-card FPGA drafter is compute-bound on its 2-bit GEMV array (2026-11-17)
Author: box1-two-v2. Program: the 7-card FPGA drafter (4 ATT cards + 2 MLPV cards on V80, H200 verifies), Llama-3.2-3B record and the new Qwen3.8-27B GPTQ-2 drafter. Tags: [M] measured (artifact named), [D] arithmetic on measured numbers, [P] projection or hypothesis.
1. The problem in one paragraph
Every engine that dominates a tree node is a GEMV on the same weight-stationary 2-bit array: G4 QKV, G6 o-proj, G7 gate/up on the ATT cards; G9 down and the G12 cheap lm-head (inside TREE) on the MLPV cards. The array's time is fixed by one law, measured exact to the cycle on shipped images:
cycles(stage) = N * K / (LANES * K_PAR * RD_PORTS) K_PAR 16, RD_PORTS 2 (dual-read URAM)
The schedule is already tight: on an older image G4 + G9 measured 235,224 cycles against a floor of 235,008, i.e. 99.91 % of the bandwidth floor [M] (G4_G9_LEVER_INVENTORY_20260824.md). So de-serialising the schedule cannot help. Only the bandwidth term (LANES, K_PAR, RD_PORTS, bank entry width), the array clock, or re-using one weight sweep for several tree nodes can make it faster. When the drafter got 3x bigger (Qwen GPTQ-2), the cards fell to 0.43x of the GPU alone.
2. Measured shares [M] / [D]
Cycles per tree node at the 66 MHz drafter clock. Llama = on-card census (median of 20 passes); Qwen = xsim of the image sources (no on-card census yet).
| card | engine | Llama record | Qwen GPTQ-2 | Qwen / Llama |
|---|---|---|---|---|
| ATT | G4 QKV | 1,421 | 4,825 | 3.4x |
| ATT | G6 o-proj + cross-card sum | 686 | 903 (ideal link) | 1.3x |
| ATT | G7g gate | 1,397 | 3,937 | 2.8x |
| ATT | G7u up | 557 | 2,585 | 4.6x |
| ATT | GEMV total / busy | 4,061 / 7,394 = 55 % [D] | 12,250 / 15,085 = 81 % [D] | |
| MLPV | G9 down | 850 (array 132 MHz) | 5,133 (array 66 MHz) | 6.0x |
| MLPV | TREE (G12 cheap lm-head sweep + top-K) | 2,224 | 6,816 | 3.1x |
| MLPV | G9 + TREE / busy | 3,074 / 5,327 = 58 % [D] | 11,949 / 15,616 = 77 % [D] |
End to end, humaneval 80 x 256 [M]:
| drafter | GPU only | 7-card | 7-card / GPU |
|---|---|---|---|
| Llama-3.2-3B record | 841.27 | 961.43 | 1.14 |
| Qwen, old BitNet drafter (body 141.6 M weights) | 256.58 | 249.62 | 0.97 |
| Qwen, new GPTQ-2 drafter (body 435.2 M weights, 3.07x) | 326.83 | 140.15 | 0.43 |
3. Array dimensions today [M]
| ATT (record) | ATT (Qwen) | MLPV (record) | MLPV (Qwen) | |
|---|---|---|---|---|
| lanes x K_PAR x read ports | 512 x 16 x 2 | 256 x 16 x 2 (halved to fit A16 activations) | 256 x 16 x 2 | 256 x 16 x 2 |
| array clock | 66 MHz | 66 MHz | 132 MHz (2x fast clock) | 66 MHz |
| lane type | LUT-GEMM (W2 x A8) | LUT-GEMM (A16) | DSP58 packed dot (E3) | DSP58 packed dot |
Device walls on these images: SLICE ~90 % device-wide and ~99.7 % in the worst SLR (the binding resource), URAM 88.8 %, inter-die SLL 18,870 wires (kills wide sibling arrays).
4. What has already been tried [M]
- ARRAY_POC M2 (LANES 256 + E3 packed-DSP lane + locality floorplan + RET_SER + TILE_PIPE + FOLD): array alone routes at 166 MHz (+0.041 ns) and needs 14,316 vs 30,876 HEAD cycles (2.16x fewer). Never integrated into a full drafter image.
- Systolic vs LUT-GEMM study: both bit-exact; the shared bit-plane table lane (variant B, g=2) was recommended over the systolic chain.
- MLPV fast clock: the record MLPV runs its array at 132 MHz through a dual-clock bridge (shipped).
- Sibling batching (W): one weight sweep serving W tree siblings is bit-exact in RTL at W2/W4/W8 (+945 LUT per lane-column). On silicon only W4 rt2 gained (-6.1 %/level) because the MLPV serialised the batch release; W=8 is blocked by area, not SLL.
- Qwen GEMM levers (projected, not built): K64 bank entry (4 chunks per cycle), TREE tile floor, DSP58 half-lane, W=2 on G7/G9.
5. Options to revise the GEMM [P]
- More codes per cycle per lane (K64 bank entry / wider URAM word): halves G4/G7/G9 at the same lane count; costs LUT and bank assembly.
- Faster array clock everywhere (132-166 MHz array on both roles, drafter FSM stays at 66 MHz): the POC closed 166 MHz in isolation.
- Turn GEMV into a small GEMM (sibling batching): one weight read serves W tree nodes; needs the MLPV batch release fixed first.
- A different lane (bit-plane LUT tables, or DSP58 multi-pumping) to buy lanes inside the SLICE wall.
6. Questions for the team
- For 2-bit weights x int8/int16 activations with weights resident in URAM on V80/Versal, which lane design gives the most MAC/cycle per SLICE: LUT-GEMM tables, DSP58 packing, or a mix?
- Has anyone closed a URAM-fed array above 166 MHz on V80 across SLRs (or multi-pumped URAM)?
- Should we spend the remaining area on lanes/clock (option 1-2) or on sibling batching (option 3)?
- Would someone review the array RTL below (ATT LUT-GEMM lane, MLPV packed-DSP lane, array top)?
7. Key files (GitHub mirror: y344shi/eagle4-context)
- This report:
reports/results/GEMM_REVISION_20261117/GEMM_REVISION_REQUEST_20261117.md - Array architecture (clean7c):
reports/results/GEMM_REVISION_20261117/rtl_snapshot/ARRAY_clean7c.md(md5 d87e476c) - RTL snapshots:
rtl_snapshot/att_lutgemm_lane_w2.sv.md,att_gemv_stage_array_scaled_slr.sv.md,mlpv_gemv_lane_scaled_stream_dual_ext_opt.sv.md,mlpv_e3b_packed_dot.sv.md,mlpv_gemv_stage_array_scaled_slr.sv.md,mlpv_gemv_dualclk_bridge.sv.md - Why the Qwen drafter is slow (layer weights + per-engine cycles):
reports/results/QWEN_GPTQ2_20261116/WHY_SLOWER_LAYERS_PHASES_20261117.md - Qwen GEMM lever design:
reports/results/QWEN_GPTQ2_20261116/DESIGN_FABLE_GPTQ2_GEMM_20261116.md(md5 4991d5a4) - G4/G9 lever inventory (99.91 % of floor):
reports/results/G4_G9_LEVER_INVENTORY_20260824.md(md5 abd1e558) - Array speed PoC (166 MHz, 2.16x):
reports/results/ARRAY_POC_20260923.md(md5 1bc7ea2b) - Systolic vs LUT-GEMM:
reports/results/ARRAY_SYSTOLIC_LUTGEMM_20260923.md(md5 f8c8253b) - Llama per-engine census:
reports/results/DRAFTER_SPEED_RESEARCH_20261107/SCALING_4C_VS_7C_PHASE_RANK_20261116.md - Research page: https://hardware.research.yshicarrick.com/p/qwen-drafters-7card-vs-gpu-20261117/