fpga-archive-reports-arch · EAGLE4 FPGA archive

Source reports/results/GEMM_REVISION_20261117/GEMM_REVISION_REQUEST_20261117.md · source md5 55e86feb8f26b723994c8ddea6b697d5 · archived 2026-11-18

Download PDFOpen PDFRaw markdown

Revising the drafter GEMM: the 7-card FPGA drafter is compute-bound on its 2-bit GEMV array (2026-11-17)

Author: box1-two-v2. Program: the 7-card FPGA drafter (4 ATT cards + 2 MLPV cards on V80, H200 verifies), Llama-3.2-3B record and the new Qwen3.8-27B GPTQ-2 drafter. Tags: [M] measured (artifact named), [D] arithmetic on measured numbers, [P] projection or hypothesis.

1. The problem in one paragraph

Every engine that dominates a tree node is a GEMV on the same weight-stationary 2-bit array: G4 QKV, G6 o-proj, G7 gate/up on the ATT cards; G9 down and the G12 cheap lm-head (inside TREE) on the MLPV cards. The array's time is fixed by one law, measured exact to the cycle on shipped images:

cycles(stage) = N * K / (LANES * K_PAR * RD_PORTS)     K_PAR 16, RD_PORTS 2 (dual-read URAM)

The schedule is already tight: on an older image G4 + G9 measured 235,224 cycles against a floor of 235,008, i.e. 99.91 % of the bandwidth floor [M] (G4_G9_LEVER_INVENTORY_20260824.md). So de-serialising the schedule cannot help. Only the bandwidth term (LANES, K_PAR, RD_PORTS, bank entry width), the array clock, or re-using one weight sweep for several tree nodes can make it faster. When the drafter got 3x bigger (Qwen GPTQ-2), the cards fell to 0.43x of the GPU alone.

2. Measured shares [M] / [D]

Cycles per tree node at the 66 MHz drafter clock. Llama = on-card census (median of 20 passes); Qwen = xsim of the image sources (no on-card census yet).

card engine Llama record Qwen GPTQ-2 Qwen / Llama
ATT G4 QKV 1,421 4,825 3.4x
ATT G6 o-proj + cross-card sum 686 903 (ideal link) 1.3x
ATT G7g gate 1,397 3,937 2.8x
ATT G7u up 557 2,585 4.6x
ATT GEMV total / busy 4,061 / 7,394 = 55 % [D] 12,250 / 15,085 = 81 % [D]
MLPV G9 down 850 (array 132 MHz) 5,133 (array 66 MHz) 6.0x
MLPV TREE (G12 cheap lm-head sweep + top-K) 2,224 6,816 3.1x
MLPV G9 + TREE / busy 3,074 / 5,327 = 58 % [D] 11,949 / 15,616 = 77 % [D]

End to end, humaneval 80 x 256 [M]:

drafter GPU only 7-card 7-card / GPU
Llama-3.2-3B record 841.27 961.43 1.14
Qwen, old BitNet drafter (body 141.6 M weights) 256.58 249.62 0.97
Qwen, new GPTQ-2 drafter (body 435.2 M weights, 3.07x) 326.83 140.15 0.43

3. Array dimensions today [M]

ATT (record) ATT (Qwen) MLPV (record) MLPV (Qwen)
lanes x K_PAR x read ports 512 x 16 x 2 256 x 16 x 2 (halved to fit A16 activations) 256 x 16 x 2 256 x 16 x 2
array clock 66 MHz 66 MHz 132 MHz (2x fast clock) 66 MHz
lane type LUT-GEMM (W2 x A8) LUT-GEMM (A16) DSP58 packed dot (E3) DSP58 packed dot

Device walls on these images: SLICE ~90 % device-wide and ~99.7 % in the worst SLR (the binding resource), URAM 88.8 %, inter-die SLL 18,870 wires (kills wide sibling arrays).

4. What has already been tried [M]

5. Options to revise the GEMM [P]

  1. More codes per cycle per lane (K64 bank entry / wider URAM word): halves G4/G7/G9 at the same lane count; costs LUT and bank assembly.
  2. Faster array clock everywhere (132-166 MHz array on both roles, drafter FSM stays at 66 MHz): the POC closed 166 MHz in isolation.
  3. Turn GEMV into a small GEMM (sibling batching): one weight read serves W tree nodes; needs the MLPV batch release fixed first.
  4. A different lane (bit-plane LUT tables, or DSP58 multi-pumping) to buy lanes inside the SLICE wall.

6. Questions for the team

  1. For 2-bit weights x int8/int16 activations with weights resident in URAM on V80/Versal, which lane design gives the most MAC/cycle per SLICE: LUT-GEMM tables, DSP58 packing, or a mix?
  2. Has anyone closed a URAM-fed array above 166 MHz on V80 across SLRs (or multi-pumped URAM)?
  3. Should we spend the remaining area on lanes/clock (option 1-2) or on sibling batching (option 3)?
  4. Would someone review the array RTL below (ATT LUT-GEMM lane, MLPV packed-DSP lane, array top)?

7. Key files (GitHub mirror: y344shi/eagle4-context)