# Revising the drafter GEMM: the 7-card FPGA drafter is compute-bound on its 2-bit GEMV array (2026-11-17)

Author: box1-two-v2. Program: the 7-card FPGA drafter (4 ATT cards + 2 MLPV cards on V80, H200 verifies), Llama-3.2-3B record and the new Qwen3.8-27B GPTQ-2 drafter.
Tags: [M] measured (artifact named), [D] arithmetic on measured numbers, [P] projection or hypothesis.

## 1. The problem in one paragraph
Every engine that dominates a tree node is a GEMV on the same weight-stationary 2-bit array: G4 QKV, G6 o-proj, G7 gate/up on the ATT cards; G9 down and the G12 cheap lm-head (inside TREE) on the MLPV cards. The array's time is fixed by one law, measured exact to the cycle on shipped images:

```
cycles(stage) = N * K / (LANES * K_PAR * RD_PORTS)     K_PAR 16, RD_PORTS 2 (dual-read URAM)
```

The schedule is already tight: on an older image G4 + G9 measured 235,224 cycles against a floor of 235,008, i.e. 99.91 % of the bandwidth floor [M] (`G4_G9_LEVER_INVENTORY_20260824.md`). So de-serialising the schedule cannot help. Only the bandwidth term (LANES, K_PAR, RD_PORTS, bank entry width), the array clock, or re-using one weight sweep for several tree nodes can make it faster. When the drafter got 3x bigger (Qwen GPTQ-2), the cards fell to 0.43x of the GPU alone.

## 2. Measured shares [M] / [D]
Cycles per tree node at the 66 MHz drafter clock. Llama = on-card census (median of 20 passes); Qwen = xsim of the image sources (no on-card census yet).

| card | engine | Llama record | Qwen GPTQ-2 | Qwen / Llama |
|---|---|---:|---:|---:|
| ATT | G4 QKV | 1,421 | 4,825 | 3.4x |
| ATT | G6 o-proj + cross-card sum | 686 | 903 (ideal link) | 1.3x |
| ATT | G7g gate | 1,397 | 3,937 | 2.8x |
| ATT | G7u up | 557 | 2,585 | 4.6x |
| ATT | **GEMV total / busy** | **4,061 / 7,394 = 55 %** [D] | **12,250 / 15,085 = 81 %** [D] | |
| MLPV | G9 down | 850 (array 132 MHz) | 5,133 (array 66 MHz) | 6.0x |
| MLPV | TREE (G12 cheap lm-head sweep + top-K) | 2,224 | 6,816 | 3.1x |
| MLPV | **G9 + TREE / busy** | **3,074 / 5,327 = 58 %** [D] | **11,949 / 15,616 = 77 %** [D] | |

End to end, humaneval 80 x 256 [M]:

| drafter | GPU only | 7-card | 7-card / GPU |
|---|---:|---:|---:|
| Llama-3.2-3B record | 841.27 | 961.43 | 1.14 |
| Qwen, old BitNet drafter (body 141.6 M weights) | 256.58 | 249.62 | 0.97 |
| Qwen, new GPTQ-2 drafter (body 435.2 M weights, 3.07x) | 326.83 | 140.15 | 0.43 |

## 3. Array dimensions today [M]
| | ATT (record) | ATT (Qwen) | MLPV (record) | MLPV (Qwen) |
|---|---|---|---|---|
| lanes x K_PAR x read ports | 512 x 16 x 2 | 256 x 16 x 2 (halved to fit A16 activations) | 256 x 16 x 2 | 256 x 16 x 2 |
| array clock | 66 MHz | 66 MHz | 132 MHz (2x fast clock) | 66 MHz |
| lane type | LUT-GEMM (W2 x A8) | LUT-GEMM (A16) | DSP58 packed dot (E3) | DSP58 packed dot |

Device walls on these images: SLICE ~90 % device-wide and ~99.7 % in the worst SLR (the binding resource), URAM 88.8 %, inter-die SLL 18,870 wires (kills wide sibling arrays).

## 4. What has already been tried [M]
- **ARRAY_POC M2** (LANES 256 + E3 packed-DSP lane + locality floorplan + RET_SER + TILE_PIPE + FOLD): array alone routes at 166 MHz (+0.041 ns) and needs 14,316 vs 30,876 HEAD cycles (2.16x fewer). Never integrated into a full drafter image.
- **Systolic vs LUT-GEMM study:** both bit-exact; the shared bit-plane table lane (variant B, g=2) was recommended over the systolic chain.
- **MLPV fast clock:** the record MLPV runs its array at 132 MHz through a dual-clock bridge (shipped).
- **Sibling batching (W):** one weight sweep serving W tree siblings is bit-exact in RTL at W2/W4/W8 (+945 LUT per lane-column). On silicon only W4 rt2 gained (-6.1 %/level) because the MLPV serialised the batch release; W=8 is blocked by area, not SLL.
- **Qwen GEMM levers (projected, not built):** K64 bank entry (4 chunks per cycle), TREE tile floor, DSP58 half-lane, W=2 on G7/G9.

## 5. Options to revise the GEMM [P]
1. **More codes per cycle per lane** (K64 bank entry / wider URAM word): halves G4/G7/G9 at the same lane count; costs LUT and bank assembly.
2. **Faster array clock everywhere** (132-166 MHz array on both roles, drafter FSM stays at 66 MHz): the POC closed 166 MHz in isolation.
3. **Turn GEMV into a small GEMM** (sibling batching): one weight read serves W tree nodes; needs the MLPV batch release fixed first.
4. **A different lane** (bit-plane LUT tables, or DSP58 multi-pumping) to buy lanes inside the SLICE wall.

## 6. Questions for the team
1. For 2-bit weights x int8/int16 activations with weights resident in URAM on V80/Versal, which lane design gives the most MAC/cycle per SLICE: LUT-GEMM tables, DSP58 packing, or a mix?
2. Has anyone closed a URAM-fed array above 166 MHz on V80 across SLRs (or multi-pumped URAM)?
3. Should we spend the remaining area on lanes/clock (option 1-2) or on sibling batching (option 3)?
4. Would someone review the array RTL below (ATT LUT-GEMM lane, MLPV packed-DSP lane, array top)?

## 7. Key files (GitHub mirror: y344shi/eagle4-context)
- This report: `reports/results/GEMM_REVISION_20261117/GEMM_REVISION_REQUEST_20261117.md`
- Array architecture (clean7c): `reports/results/GEMM_REVISION_20261117/rtl_snapshot/ARRAY_clean7c.md` (md5 d87e476c)
- RTL snapshots: `rtl_snapshot/att_lutgemm_lane_w2.sv.md`, `att_gemv_stage_array_scaled_slr.sv.md`, `mlpv_gemv_lane_scaled_stream_dual_ext_opt.sv.md`, `mlpv_e3b_packed_dot.sv.md`, `mlpv_gemv_stage_array_scaled_slr.sv.md`, `mlpv_gemv_dualclk_bridge.sv.md`
- Why the Qwen drafter is slow (layer weights + per-engine cycles): `reports/results/QWEN_GPTQ2_20261116/WHY_SLOWER_LAYERS_PHASES_20261117.md`
- Qwen GEMM lever design: `reports/results/QWEN_GPTQ2_20261116/DESIGN_FABLE_GPTQ2_GEMM_20261116.md` (md5 4991d5a4)
- G4/G9 lever inventory (99.91 % of floor): `reports/results/G4_G9_LEVER_INVENTORY_20260824.md` (md5 abd1e558)
- Array speed PoC (166 MHz, 2.16x): `reports/results/ARRAY_POC_20260923.md` (md5 1bc7ea2b)
- Systolic vs LUT-GEMM: `reports/results/ARRAY_SYSTOLIC_LUTGEMM_20260923.md` (md5 f8c8253b)
- Llama per-engine census: `reports/results/DRAFTER_SPEED_RESEARCH_20261107/SCALING_4C_VS_7C_PHASE_RANK_20261116.md`
- Research page: https://hardware.research.yshicarrick.com/p/qwen-drafters-7card-vs-gpu-20261117/
