EAGLE4: A Speculative-Decoding Drafter Held in FPGA On-Chip Memory
Abstract
Speculative decoding speeds up a large language model by letting a small drafter propose a tree of tokens that the target model verifies in one pass. On a GPU the drafter and the target share one chip: in our GPU-only H200 profile, drafting takes 7.254 ms of a 12.0 ms step. EAGLE4 moves the EAGLE drafter onto four AMD Alveo V80 FPGAs. The drafter body is quantised to 2 bits and held in on-chip UltraRAM beside the multipliers, and the draft tree is decoded on chip, so the H200 only verifies. With Llama-3.2-3B as the target the system reaches 870.10 tok/s at accept 7.12, against 700.71 tok/s for the H200 running its own EAGLE drafter under the same configuration (humaneval 80 × 256, tree width 2, depth 15, fa4): 1.24× faster. At depth 7 it reaches 831.14 tok/s at accept 5.89. One full depth-7 tree takes 2.234 ms on the cards. The same ATT card image also drafts for Qwen3.8-27B (249.62 tok/s against 256.67 GPU-only).
1 Introduction
A language model generates one token per forward pass, so a single user's speed is set by how fast the hardware reads the model's weights. Speculative decoding [1][2] breaks this limit: a cheap drafter guesses several tokens, organised as a tree [7], and the target checks all of them at once and keeps the longest correct prefix. The text is exactly what the target would have produced. EAGLE [3][4][5] drafts from the target's own hidden features.
The drafter is small but sequential: each tree level depends on the level above. On a GPU a depth-15 tree costs 15 drafter forwards, and each forward streams 1.071 GB of 16-bit drafter weights from HBM (reports/results/WHY_SLOW_ROOFLINE_20260902.md:84-91, :108). EAGLE4 puts the drafter on dedicated FPGA cards, with its weights resident next to the arithmetic.
2 GPU compute style vs FPGA compute style
GPU. All weights live in HBM, off the compute die. Every drafter forward launches kernels that pull the drafter weights from HBM into the streaming multiprocessors, and the target uses the same SMs and the same HBM bandwidth. Draft levels and the verify pass run one after another on one timeline. Measured on box1's H200 as served (clean pass): draft step 2.528 / 2.572 ms at depth 7 and 3.933 / 3.903 ms at depth 11 (fa4 / fa3), plus a separate 0.40 ms extend per step (reports/results/GPU_DRAFT_ONLY_BOX1_20261110.md (Answer table; CUDA events, humaneval 20x256, w2 VTOK 20, bs 1)).
FPGA. The 2-bit drafter body and the cheap lm-head are loaded once at boot into UltraRAM (1,925 blocks per V80; 744 used by the ATT image), interleaved with the multiplier lanes: the weights stay put and only activations move (weight-stationary). Only the 16-bit exact lm-head lives in card HBM, and G8 gathers 64 shortlisted rows of it per pass. A tree state machine on the card runs level after level with no host round trip. The H200 only verifies. Measured: one full depth-7 tree 2.234 ms on the four cards (reports/results/DRAFTER_SPEED_RESEARCH_20261107/F3T_SILICON_20261109.md (chain census, 40 trees, md 7 w2 K_C 64); TRACK1_RECORD_F3T_20261109/EVALUATION.md).
3 Architecture
One drafter layer as a fixed pipeline. Hidden 3,072, MLP 8,192, vocabulary 128,256. Stages: input norm (G1), QKV projection (G4), attention over the KV cache (G5), output projection (G6), gate / up projections and SiLU (G7), down projection (G9), final norm and low-rank projection (G11 / GLM), a cheap 2-bit lm-head over the whole vocabulary (G12), on-chip top-K tree selection (TREE) and an exact 16-bit rescore of the shortlisted rows (G8).
Four cards, split by phase. An ATT pair runs G1 to G7 tensor-parallel across two cards and exchanges the G6 partial results over a QSFP link. An MLPV pair runs G9, G11 / GLM, G12, TREE and G8. The pairs are joined by QSFP links, and the children of each level go back to the ATT cards on those links. The host moves only the accepted tokens and the draft tree to and from the H200 over PCIe (Figure 2).
Inside one ATT card. The V80 has three dies (SLRs). Attention and its KV cache fill SLR0's block RAM, so the 2-bit weight array is split into lane groups on SLR1 and SLR2. Each GEMV pass broadcasts activations from SLR0 to the groups and returns per-lane results; the QSFP transceivers sit in SLR1 (Figure 3, Table 3). The binding resource is SLICE on the hub die: 87.60% in SLR0.
| resource | used | % of V80 |
|---|---|---|
| LUT | 1,062,125 | 41.26% |
| FF | 1,023,535 | 19.88% |
| DSP58 | 5,267 | 48.55% |
| BRAM36 | 2,194.5 | 58.66% |
| URAM | 744 | 38.65% |
| SLICE (device) | 236,676 | 73.55% |
| SLICE, worst die (SLR0) | 98,580 | 87.60% |
| SLL SLR1↔SLR0 / SLR2↔SLR1 | 8,888 / 1,990 of 18,870 | 47.10% / 10.55% |
| routed WNS / WHS | +0.040 / +0.011 ns | 66 MHz drafter clock |
4 Evaluation
Setup. Two servers: talon2 (H200 + seven V80, four or six used) and box1 (H200 + four V80, the public demo). Targets Llama-3.2-3B and Qwen3.8-27B. SGLang 0.5.16, batch 1, tree width 2. The baseline is the H200 running SGLang's own EAGLE drafter (GPU-only), measured on the same H200. Table 1 lists every end-to-end number.
| system | host, date | config | tok/s | accept |
|---|---|---|---|---|
| H200 + 4 V80 (F3t + 0x1342) | talon2, 11-09 | Llama, d15 fa4 | 870.10 | 7.12 |
| H200 GPU-only EAGLE, same configuration | talon2, 11-10 | Llama, d15 fa4 | 700.71 | 6.92 |
| H200 + 4 V80 (F3t + 0x1342) | talon2, 11-09 | Llama, d11 fa3 | 883.12 | 6.90 |
| H200 + 4 V80 (F3t + 0x1342) | talon2, 11-09 | Llama, d7 (record config) | 831.14 | 5.89 |
| H200 GPU-only EAGLE, best config | talon2, 11-09 | Llama, d9 fa3 | 835.12 | 6.63 |
| H200 + 4 V80 (E2b + 0x1342), demo launcher | box1, 11-10 | Llama, d11 | 820.20 | 6.90 |
| same, 15 demo prompts | box1, 11-10 | Llama, d11 | 951.19 | 8.72 |
| same, PQ+C host levers | box1, 11-10 | Llama, d11 · demo prompts | 836.00 · 990.88 | 6.90 · 8.72 |
| H200 + 6 V80 (TP4 ATT B4 + MLPV 0x134D) | talon2, 11-10 | Llama, d11 | 859.06 | 6.91 |
| H200 + 4 V80 (F3t + 0x1342) | talon2, 11-09 | Qwen3.8-27B, d7 | 249.62 | 7.14 |
| H200 GPU-only EAGLE | talon2, 11-10 | Qwen3.8-27B, d7 | 256.67 | 7.13 |
| Qwen3.8-27B target alone | talon2, 10-23 | no speculation | 61.08 | 1 |
| H200 + 4 V80, same tree (earlier images) | box1, 10-20 | Llama, humaneval 20×512, d15 | 846.20 | 7.29 |
| H200 GPU-only EAGLE, same tree | box1, 10-20 | Llama, humaneval 20×512, d15 | 741.27 | 7.97 |
| Llama-3.2-3B target alone | box1, 10-20 | humaneval 20×512, no speculation | 301.41 | 1 |
Throughput. Under the same configuration on both sides (width 2, depth 15, 20 verify tokens, fa4) the four cards reach 870.10 tok/s and the H200 drafting for itself 700.71 tok/s, 1.24× (Table 1). On the same H200 on the same day, the four cards at depth 11 reach 883.12 tok/s; the GPU-only drafter peaks at 835.12 tok/s at depth 9 and slows at greater depth (823.62 at d11, 778.82 at d13) (Figure 4). The depth ladder on the cards (Figure 5) shows why depth 11 wins: accept rises from 5.89 at d7 to 6.87–6.90 at d11 while the card tree grows from 2.234 to 3.384 ms.
accept: d5 4.97 · d7 5.89 · d9 6.47 · d11 6.87 · d13 7.03 · d15 7.12
Pure drafting speed. Table 2 gives the time for one draft tree. On the cards, one full depth-11 tree (21 forwards) takes 3.685 ms, 175 µs per forward; in serving the card time per tree averages 3.658 ms of an 8.330 ms step on box1. On the H200 the d11 draft step takes 3.933 ms (fa4) as served in the decode loop, plus a 0.400 ms extend, so in serving the cards build the tree in less time than the H200's draft + extend step. In an isolated back-to-back loop of draft() calls with no verify, the H200's draft call takes 2.886–2.894 ms, faster than today's card tree. Figure 6 breaks the depth-11 tree into phases: G5 attention is the largest phase (1,219.4 µs on ATT), and the ATT card bounds the tree.
| drafter | instrument (context) | depth | time per tree |
|---|---|---|---|
| 4 × V80, F3t + 0x1342 (talon2, 11-09) | card census, full-depth tree p50, no host glue | 7 | 2.234 ms |
| 4 × V80, F3t + 0x1342 (talon2, 11-09) | card census, full-depth tree p50, no host glue | 11 | 3.384 ms |
| 4 × V80, E2b + 0x1342 (box1, 11-10) | card bench, full-depth tree p50, no host glue | 7 | 2.449 ms |
| 4 × V80, E2b + 0x1342 (box1, 11-10) | card bench, full-depth tree p50, no host glue | 11 | 3.685 ms (21 forwards, 175 µs each) |
| 4 × V80, E2b + 0x1342 (box1, 11-10) | serving trees, card-process issue→done mean, includes its host work | 11 | 3.658 ms |
| H200 EAGLE drafter (box1, 11-10, clean pass) | CUDA events, draft step as served in SGLang, humaneval 20×256, p50, fa4 / fa3 | 7 | 2.528 / 2.572 ms + 0.399 / 0.406 ms extend |
| H200 EAGLE drafter (box1, 11-10, clean pass) | CUDA events, draft step as served in SGLang, humaneval 20×256, p50, fa4 / fa3 | 11 | 3.933 / 3.903 ms + 0.400 / 0.404 ms extend |
| H200 EAGLE drafter (box1, 11-10, first pass, flagged: overlapped demo traffic) | CUDA events, 200 back-to-back draft() calls on a fixed prefix, no verify, fa4 | 7 | 1.753–1.759 ms |
| H200 EAGLE drafter (box1, 11-10, first pass, flagged: overlapped demo traffic) | CUDA events, 200 back-to-back draft() calls on a fixed prefix, no verify, fa4 | 11 | 2.886–2.894 ms |
| 4 × V80 vs H200, per drafter forward (10-19 images) | on-card loop vs CUDA-event depth slope | 7 | 0.177 vs 0.37 ms |
What made F3t. F3t is the previous ATT image (E2b) plus the G5 overlap engine. It cut G5 attention per tree by 30.8% and the whole tree by 8.3% (Llama) and 8.5% (Qwen), with every other phase unchanged (Figure 7). Same-day serving at the identical host config: F3t 831.14 / 830.09 tok/s against E2b 812.22–816.61 over 7 rounds.
Qwen3.8-27B. The same ATT image, switched by registers, drafts for Qwen3.8-27B: 249.62 tok/s at accept 7.14 against 256.67 GPU-only and 61.08 for the target alone (Figure 8). On Qwen the cards are still below GPU-only.
Live demo. box1's public demo (https://demo.yshicarrick.com, login required) serves Llama at w2 d11 on ATT E2b + MLPV 0x1342: 820.20 tok/s on humaneval 80 × 256 and 951.19 tok/s on its 15 demo prompts; a real 256-token request decoded at 1,079 tok/s at accept 8.83 (reports/results/FOURCARD_DESIGN_20261016/BOX1_E2B_D11_20261110.md (Answer table; demo final state)). Its auto-run of the best prompts measured 1,000.1 tok/s @ 8.83 on 2 prompts and 977.7 tok/s @ 8.83 on 3 prompts on 2026-11-10 (reports/results/DEMO_AUTORUN_BOX1_20261110.md:114 (3 prompts, 18:55Z); USER_DIALOGUE_LOG.md:2094 (2 prompts, 19:0xZ live check)).
Correctness. The target verifies every token, so the output is the target's. Between card images the draft trees are compared level by level: F3t equals E2b on 12/12 Llama trees at depths 15, 7 and 2 (56 levels) and 9/9 Qwen trees; serving texts are 80/80 byte-identical to E2b on both models. The two d11 rounds are byte-identical to each other (80/80). Against a 16-bit reference drafter the 2-bit trees differ beyond the first level, as expected from 2-bit weights (reports/results/TRACK1_RECORD_F3T_20261109/EVALUATION.md (rounds table §1); reports/results/DRAFTER_SPEED_RESEARCH_20261107/F3T_SILICON_20261109.md).
5 Energy (measured on talon2, 2026-11-04)
Setup. talon2: 4 V80 (ATT pair + MLPV pair) + 1 H200 NVL; Llama-3.2-3B and Qwen3.8-27B, humaneval 80 × 256, w2 d7, batch 1. Power sampled at 1 Hz: each V80's AMC sensor, nvidia-smi for the H200, RAPL for the CPU package and IPMI DCMI for the whole chassis. These arms ran the card images of 2026-11-04 (Llama 723.09 tok/s on the cards vs 826.58 GPU-only that day); the speed has since risen to 831.14 / 883.12 tok/s, and power has not been re-measured. Energy = measured W × measured time; J/token = measured W ÷ measured tok/s.
Cards win · drafter forward
per drafter pass, 4 V80 vs H200 (box1, 2026-10-19, mt_bench 50 × 32 w2 d7; the H200 value is a depth-slope fit of CUDA events)
Cards win · energy per drafter forward
0.177 ms × 255.03 W (4 V80) vs 0.37 ms × 276.67 W (H200), each device alone, never summed
Cards win · H200 power while serving
H200 that only verifies vs H200 drafting for itself (Llama arms; Qwen 386.6 vs 433.3 W)
Cards win · H200 energy per token
H200 W ÷ tok/s, Llama (Qwen 1.637 vs 1.689 J)
Edge lost · V80 static power
the 4 V80s draw almost the same power idle and busy
Edge lost · host CPU
CPU package power, card arm vs GPU-only arm (RAPL)
| arm | tok/s | H200 W | CPU pkg W | chassis W (DCMI) | H200 J/token | chassis J/token |
|---|---|---|---|---|---|---|
| Llama-3.2-3B · H200 + 4 V80 | 723.09 | 227.0 | 289.6 | 1,110.1 | 0.314 | 1.535 |
| Llama-3.2-3B · GPU-only | 826.58 | 276.7 | 202.5 | 1,068.6 | 0.335 | 1.293 |
| Qwen3.8-27B · H200 + 4 V80 | 236.17 | 386.6 | 261.5 | 1,295.2 | 1.637 | 5.484 |
| Qwen3.8-27B · GPU-only | 256.58 | 433.3 | 204.0 | 1,236.9 (before a flash on other cards) | 1.689 | 4.821 |
| idle server | 0 | 80.4 | 132.9 | 751.9 | 0 | 0 |
| arm | 4 V80 W | 4 V80 J/token |
|---|---|---|
| idle server | 251.1 | 0 |
| Llama · cards drafting | 255.0 | 0.353 |
| Llama · GPU-only (cards idle) | 251.9 | 0 |
| Qwen · cards drafting | 252.4 | 1.069 |
| model | steady steps | card window ms | step ms | H200 verify ms | card busy | card idle |
|---|---|---|---|---|---|---|
| Llama-3.2-3B | 3,277 | 3.81 | 8.10 | 5.57 | ≤ 47.1% | ≥ 52.9% |
| Qwen3.8-27B | 2,091 | 4.00 | 26.94 | 20.20 | ≤ 14.9% | ≥ 85.1% |
6 Work in progress
- 6 cards (measured). The first 6-card drafter (four ATT ranks, TP4, + MLPV pair) passes every golden gate and serves 859.06 tok/s at accept 6.91 (d11) and 796.13 at 5.89 (d7); its d7 texts are byte-identical to the 4-card record. It is still below the 4-card record (reports/results/DRAFTER_SPEED_RESEARCH_20261107/C2_6CARD_TP4_20261110.md (Verdict)).
- 7 cards (simulation only). The item-parallel design C1 (two complete attention sets, one tree item each, + MLPV pair) passes its xsim gates; whole-tree cycles fall by 8.4% (with tree stop) to 12.1% (loop modes off) in the testbench. It has not run on silicon (reports/results/DRAFTER_SPEED_RESEARCH_20261107/C1_ITEMPAR_RTL_20261110.md:79-90 (tb_c1 whole-tree cycles)).
- Qwen. The FPGA drafter is 249.62 against 256.67 tok/s GPU-only on humaneval and below GPU-only on all eight prompt sets measured on 2026-11-10 (FPGA / GPU 0.961–0.975 in that report's table; reports/results/FPGA_QWEN_ALLSETS_TEMPS_20261110/REPORT.md:9-20; reports/results/DRAFTER_SPEED_RESEARCH_20261107/F3T_SILICON_20261109.md).
7 Related work
Speculative decoding [1][2] and tree verification [7] are the basis of this work; EAGLE [3][4][5] and Medusa [6] are the leading drafter designs, all run on the same GPU as the target. FPGA LLM accelerators such as DFX [9] and FlightLLM [10] run the whole model on FPGAs; we keep the target on the GPU and move only the latency-bound drafter, with its weights on chip [11]. Low-bit weights follow GPTQ-style quantisation [8].
8 Conclusion
Holding a 2-bit EAGLE drafter in FPGA on-chip memory, split by phase over four V80 cards, gives 870.10 tok/s on Llama-3.2-3B with an H200 that only verifies, against 700.71 tok/s for the H200 drafting for itself under the same configuration: 1.24× faster. The card drafts a full depth-7 tree in 2.234 ms, and the same image drafts for Qwen3.8-27B.
References
- Y. Leviathan, M. Kalman, Y. Matias. Fast Inference from Transformers via Speculative Decoding. ICML 2023.
- C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, J. Jumper. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv:2302.01318, 2023.
- Y. Li, F. Wei, C. Zhang, H. Zhang. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. ICML 2024.
- Y. Li, F. Wei, C. Zhang, H. Zhang. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. EMNLP 2024.
- Y. Li, F. Wei, C. Zhang, H. Zhang. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. arXiv:2503.01840, 2025.
- T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, T. Dao. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. ICML 2024.
- X. Miao et al. SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification. ASPLOS 2024.
- E. Frantar, S. Ashkboos, T. Hoefler, D. Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023.
- S. Hong et al. DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation. MICRO 2022.
- S. Zeng et al. FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs. FPGA 2024.
- AMD. Alveo V80 Compute Accelerator Data Sheet (DS1013).
Repository paths are relative to the project root. Builder: reports/results/WHITEPAPER_V7_20261110/build_v9.py; claims and sources: CLAIMS_V7_20261110.md. Earlier versions stay on the site at /v/1/ to /v/6/.