EAGLE4: A Speculative-Decoding Drafter Held in FPGA On-Chip Memory
[Authors — placeholder]
Calculations and hypothesis — every derived number and every projection on this page
Everything below the paper's title that is not a direct measurement is listed here with its formula. Projections are also collected in the clearly marked §6.2 Projections.
- Energy per drafter forward = forward time × device run-mean power: H200 0.37 ms × 276.67 W = 102.4 mJ; 4 V80 0.177 ms × 255.03 W = 45.1 mJ; ratio 2.27×. Per Llama step: H200 draft + extend (2.49 + 0.41) ms × 276.67 W = 0.80 J (box1 mt_bench split; the talon2 draft-time split is not yet measured); 4 V80 card time 3.81 ms × 255.03 W = 0.97 J (upper bound). Never summed.
- J/token = mean W ÷ (tokens ÷ time). H200 power change 1 − 227.0/276.7 = −17.9% (Qwen −10.8%). 2.1× = 0.37 / 0.177.
- 35.4 MB = sum of the 2-bit qweight tensors; 282 MB at 16 bit; ~69 MB UltraRAM = 1,925 × 36 KB.
- Duty: busy = card window ÷ step = 3.81/8.10 = 47.1%, 4.00/26.94 = 14.9% (measured ratio). Nodes per card set = 1 / busy = 2.1 → 2 (Llama), 6.7 → 6 (Qwen) — projection; time-sharing is not built.
- Cost (§6.2): capex = $H200 + n × $chip; 3-y energy = setup W × L / 3.6×10⁶ × $/kWh, L = 94,608,000 s; tokens per $ = tok/s × L / (capex + energy); tok/s to tie = TPD_GPU × (capex + energy) / L. Prices: H200 $34,500 (range $30,000–45,084), V80 $9,495, ASIC $5,000 (user projection), 14.04 ¢/kWh (EIA commercial 2026 YTD; range 9.09–14.47 ¢).
- Roadmap gains (§7): host path ≈0.75–1.0 ms/step ≈ +12–16% (modelled); KV-prefill card ≈0.17 ms/step (modelled); two streams 1.55× aggregate (measured on the 2-card pair, not on 4 cards).
Abstract
Speculative decoding makes a large language model faster by letting a small drafter propose a tree of tokens that the target model verifies in one pass. On a GPU the drafter and the target share the chip: in our GPU-only profile drafting takes 60.4% of each step. We move the EAGLE drafter onto four AMD Alveo V80 FPGAs, quantise its body to 2 bits and keep every body weight in on-chip UltraRAM, so the H200 only verifies. One drafter forward takes 0.177 ms on the four cards against 0.37 ms on the H200 (2.1×) and about 45 mJ against 102 mJ (2.3×). When it only verifies, the H200 draws 18% less power. With Llama-3.2-3B as the target, the system reaches 839.65 tok/s on humaneval 80 × 256 (accept 5.88). It also beats the H200 drafting for itself on the same depth-15 tree (846.20 vs 741.27 tok/s). The same four cards serve Qwen3.8-27B with a 262,144-token context. On a second server measured on 2026-11-04, the GPU-only baseline was faster end to end (826.58 vs 723.09 tok/s). We trace that gap to host overhead and the cards' static power, and measure the cards at least 52.9% (Llama) and 85.1% (Qwen) idle. That idle time is the headroom our roadmap and a drafter ASIC target.
1 Introduction
Large language models generate one token per forward pass, so a single user's speed is set by how fast the hardware can read the model's weights. Speculative decoding [1][2] breaks this limit: a cheap drafter guesses several tokens, organised as a tree [7], and the target model checks all of them at once and keeps the longest correct prefix. EAGLE [3][4][5] drafts from the target's own hidden features and is among the strongest drafters in open serving stacks.
The drafter is small but sequential. Each tree level depends on the level above, so on a GPU a depth-15 tree costs 15 drafter passes, and every pass re-reads the drafter's weights from HBM. In our GPU-only profile (Llama-3.2-3B, humaneval 80, depth 15, width 4) drafting takes 7.254 ms of a 12.0 ms step, 60.4% measured, and it runs on the same GPU the target needs.
This paper describes EAGLE4, a drafter that runs on dedicated FPGA silicon with its weights resident beside the multipliers. Our contributions are: (i) a hand-written RTL drafter with 2-bit weights in UltraRAM and on-chip tree decoding (§3); (ii) a four-card split of one drafter over QSFP links (§3–4); (iii) a measured evaluation of speed, latency, energy and duty cycle against the H200 drafting for itself (§5–6); and (iv) an honest account of where today's system loses end to end, and why (§7).
2 Background
Speculative decoding. A drafter proposes tokens and the target verifies them in parallel; accepted tokens are exactly those the target would have produced, so output quality is the target's [1][2]. Speed depends on the accept length (tokens kept per verification) and on the time per step: draft plus verify plus host work.
EAGLE. EAGLE [3] uses one transformer layer that takes the target's last hidden states and predicts the next features and tokens; EAGLE-2 [4] grows a dynamic tree; EAGLE-3 [5] scales training. Medusa [6] uses extra heads instead of a drafter layer.
Drafter cost. At batch 1 a drafter pass is bound by weight reads: our drafter's 16-bit weights are 1.071 GB per pass on the GPU, 74% of it the 128,256-word lm-head derived. A 2-bit body (GPTQ-style [8]) is 35.4 MB, small enough for the ~69 MB of UltraRAM on one V80 [11].
3 Architecture
On-chip 2-bit drafter. The drafter is one transformer layer (hidden 3,072, MLP 8,192, vocabulary 128,256) built as a fixed pipeline of stages: input norm (G1), QKV projection (G4), attention over the KV cache (G5), output projection (G6), gate/up projections and SiLU (G7), down projection (G9), final norm and low-rank projection (G11/GLM), a cheap 2-bit lm-head over the whole vocabulary (G12), on-chip top-K tree selection (TREE) and an exact 16-bit rescore of the shortlisted rows (G8).
Residency. 2-bit body weights and the cheap lm-head sit in UltraRAM; scales and norms in block RAM; the K cache in UltraRAM and the V cache in block RAM. Only the 16-bit exact lm-head (788 MB) lives in card HBM, and G8 gathers 64 of its rows per pass. All of it is loaded once at boot; no body weight is read from HBM during a pass.
Tree decode. Each tree level runs the full layer for the frontier, keeps the top candidates on chip and feeds the children to the next level, with no host round trip between levels.
Four-card split. An attention pair (ATT, tensor-parallel by two: G1, G4, G5, G6) and an MLP + lm-head pair (MLPV: G7, G9, G11/GLM, G12, TREE, G8) are joined by QSFP Aurora links. One width-2 tree level takes 342.4 µs on the four cards (context 566) measured.
4 Implementation
The drafter is hand-written SystemVerilog on AMD Alveo V80 cards (Versal HBM, three SLRs) [11], built with Vivado 2025.1 on the AVED platform and driven over QDMA. The drafter clock is 66 MHz, with a faster array clock for the GEMV pump. The current clock-fix image 0xE4B7133F closes timing at WNS +0.080 ns / WHS +0.011 ns measured. Table 1 gives the measured resources of the two card images. DSP58 binds on the attention card, and inter-die wires (SLLs) are the placement limit to watch.
| ATT 0x1288 | MLPV 0x1287 | |
|---|---|---|
| LUT | 518,096 (20.13%) | 795,271 (30.89%) |
| DSP58 | 7,377 (68.00%), binding | 6,288 (57.96%) |
| BRAM36 / URAM | 35.78% / 35.74% | 28.86% / 26.60% |
| SLICE device / worst SLR | 42.56% / 62.55% | 56.89% / 74.81% |
| SLL SLR0↔1 | 13,997 (74.18%) | 9,915 (52.54%) |
| routed WNS / WHS | +0.023 / +0.010 ns | +0.007 / +0.010 ns |
The host stack is SGLang with the cards as the drafter backend; a card process per pair exchanges the root state and the draft tree with the GPU process each step.
5 Evaluation
Setup. Two servers: box1 (one H200, four V80) and talon2 (one H200 NVL, seven V80, four used). Targets: Llama-3.2-3B and Qwen3.8-27B. Workloads: humaneval (80 prompts × 256 new tokens, or 20 × 512) and mt_bench (50 × 32), batch 1, tree width 2. The baseline is the H200 running SGLang's own EAGLE drafter (exec mode −1). Table 2 lists every end-to-end number used here.
| result | host, date | workload | tok/s | accept |
|---|---|---|---|---|
| H200 + 4 V80 | talon2, 11-04 | humaneval 80×256, d7 | 723.09 | 5.88 |
| H200 GPU-only EAGLE | talon2, 11-04 | humaneval 80×256, d7 | 826.58 | 5.97 |
| Qwen3.8-27B · H200 + 4 V80 | talon2, 11-04 | humaneval 80×256, d7 | 236.17 | 7.14 |
| Qwen3.8-27B · GPU-only EAGLE | talon2, 11-04 | humaneval 80×256, d7 | 256.58 | 7.13 |
| H200 + 4 V80, record | box1, 10-23 | humaneval 80×256, d7 | 839.65 | 5.88 |
| H200 + 4 V80, same tree | box1, 10-20 | humaneval 20×512, d15 | 846.20 | 7.29 |
| H200 drafting for itself, same tree | box1, 10-20 | humaneval 20×512, d15 | 741.27 | 7.97 |
| H200 + 4 V80 vs GPU self, deep tree | box1, 10-20 | mt_bench 50×32, d12 | 497.05 vs 413.4 | 4.06 vs 4.30 |
| live demo, one request | box1, 11-04 | chat | 1,047.12 | 7.111 |
| Qwen3.8-27B live demo, one request | box1, 11-04 | chat | 234.43 | 6.737 |
Throughput. Figure 2 compares the two systems on the same server. Figure 3 shows the best results on box1. On box1 the four-card drafter beats the H200 drafting for itself on the same depth-15 tree by 14%, and at depth 12 on mt_bench by 20%. On talon2 on 2026-11-04 the GPU-only baseline is faster, by 12.5% (Llama) and 7.9% (Qwen); §7 explains why. Accept lengths are within 0.1 of the GPU's on talon2.
Drafter latency and energy. One drafter forward takes 0.177 ms on the four cards against 0.37 ms on the H200, and about 45 mJ against 102 mJ (Figures 4 and 5). Each device is shown alone; the two are never added together.
Duty cycle. Per steady decode step the cards are busy at most 47.1% (Llama) and 14.9% (Qwen) of the time (Figure 6). The rest of the step is the H200 verifying.
H200 power when verify-only. With the drafting moved to the cards the H200 draws 227.0 W instead of 276.7 W (Llama), and 386.6 W instead of 433.3 W (Qwen) (Figure 7).
Correctness. The target verifies every token, so the output text is the target's. Draft trees on the 2-bit cards are compared level by level with the recorded image: 9/9 identical for Llama and for Qwen. The Qwen texts match the record on all 80 prompts measured (TALON2_POWER_ECON_20261104.md:42; SESSION_CONTEXT_BACKUP_20261104.md:91). Against a 16-bit reference drafter, the first tree level matches 8/8 and the second 6/8, which is the expected effect of 2-bit weights.
6 Cost and energy
6.1 Measured
Table 3 gives the measured power by device and Figure 8 the whole-server energy per token. The 4-V80 setup draws 255.0 W in its cards while serving and 251.1 W when idle: almost all of it is static. Per token the H200 itself uses less energy with the cards drafting: 0.314 vs 0.335 J (Llama) and 1.637 vs 1.689 J (Qwen). The whole server uses more: 1.535 vs 1.293 J (Llama).
| arm | 4 V80 | 3 other V80 | H200 | CPU package | chassis DCMI |
|---|---|---|---|---|---|
| idle | 251.1 | 119.9 | 80.4 | 132.9 | 751.9 |
| Llama · H200 + 4 V80 | 255.0 | 119.4 | 227.0 | 289.6 | 1,110.1 |
| Llama · GPU-only | 251.9 (idle) | 119.5 | 276.7 | 202.5 | 1,068.6 |
| Qwen · H200 + 4 V80 | 252.4 | 119.8 | 386.6 | 261.5 | 1,295.2 |
| Qwen · GPU-only | not read | not read | 433.3 | 204.0 | 1,236.9 (pre-flash) |
6.2 Projections projection
Everything in this subsection is calculated, not measured (formulas in the Calculations and hypothesis bar). Table 4 and Figure 9 price four configurations at market prices over 3 years. The H200 is $34,500 and the V80 $9,495. The ASIC is $5,000 per chip, the user's projection. The V80 and ASIC rows use today's measured FPGA-path speed and setup power; ASIC power is assumed equal to the V80 setup, because no ASIC silicon exists yet. At today's speed every FPGA or ASIC configuration buys fewer tokens per dollar than the GPU-only H200. One ASIC ties GPU-only at 962 tok/s on Llama, against the measured 723.09 (talon2) and 839.65 (box1 record). It ties at 298 tok/s on Qwen, against 236.17 measured. The duty cycle suggests one card set could draft for about 2 (Llama) or 6 (Qwen) H200 nodes if several streams time-shared it. That time-sharing is not implemented.
| configuration | capex $ | 3-y energy $ | $ per tok/s | M tokens per $ | tok/s to tie GPU-only |
|---|---|---|---|---|---|
| Llama · GPU-only (1 H200) | 34,500 | 2,572 | 41.74 | 2.109 | — |
| Llama · H200 + 4 V80 | 72,480 | 3,655 | 100.24 | 0.899 | 1,698 |
| Llama · H200 + 4 ASIC (projection) | 54,500 | 3,655 | 75.37 | 1.176 | 1,297 |
| Llama · H200 + 1 ASIC (projection) | 39,500 | 3,655 | 54.63 | 1.585 | 962 |
| Qwen · GPU-only (1 H200) | 34,500 | 3,195 | 134.46 | 0.644 | — |
| Qwen · H200 + 4 V80 | 72,480 | 4,337 | 306.90 | 0.291 | 523 |
| Qwen · H200 + 4 ASIC (projection) | 54,500 | 4,337 | 230.77 | 0.380 | 400 |
| Qwen · H200 + 1 ASIC (projection) | 39,500 | 4,337 | 167.25 | 0.510 | 298 |
7 Limitations
- GPU-only is faster end to end on talon2 today: 826.58 vs 723.09 tok/s (Llama) and 256.58 vs 236.17 (Qwen), same server, same prompts. The cards draft faster, but each step carries about 1.4 ms of serial host work out of a 6.25–6.46 ms step (SEVENCARD_BOARD_20261026/T2_HOST_TRAFFIC.md:17 (md5 8b9e3f84…)) measured. The host CPU draws +87.1 W more (289.6 vs 202.5 W). The four V80s draw ~251 W even when idle.
- Serial steps. One request at a time: draft and verify alternate, so the cards wait on the H200 for most of each step (§5).
- Measurement scope. Llama talon2 arms have 2 rounds, Qwen 1. The H200 draft-time split on talon2 is not yet measured; per-forward energies use run-mean power, not draft-window power. The Qwen GPU-only V80 power was not read.
- 2-bit drafter. Trees match the recorded image exactly but not a 16-bit drafter beyond the first level; the target verifies every token, so the text is the target's.
- Roadmap, not results. Each item lists its measured or modelled effect. Host path: one endpoint and fewer serial steps, modelled +12–16%. Export fix: unit gate measured 6,230 → 2,767 bridge cycles per item; full-drafter identity green. Clock-fix image: timing closed at WNS +0.080 ns, which enables the W4 speed image (child level −9.2% measured, Qwen). KV on card: modelled ≈0.17 ms/step. Batched verify / two streams: 1.55× aggregate measured on the 2-card pair, no gain yet on four cards. Sources: SEVENCARD_BOARD_20261026/T2_HOST_TRAFFIC.md:257, :261; TRACKLIST_20260923.md:613, :636 (EXPBURST unit gate; md15 identity GREEN); TRACKLIST_20260923.md:647 (0xE4B7133F FINAL_TRUE WNS +0.080 / WHS +0.011); TRACKLIST_20260923.md:585 (0xE4B7133B, child level −9.2% Qwen vs hold); QWEN27B_DRAFTER_20261020/NSTREAM_DRIVER_20261023.md:288; 4C1_DELIVERY_TRACKER.md:1156 (4-card N-stream: no gain).
8 Related work
Speculative decoding [1][2] and tree verification [7] are the basis of this work; EAGLE [3][4][5] and Medusa [6] are the leading drafter designs, all run on the same GPU as the target. FPGA LLM accelerators such as DFX [9] and FlightLLM [10] run the whole model on FPGAs; we instead keep the target on the GPU and move only the latency-bound drafter, with weights fully on chip. Low-bit weight quantisation follows GPTQ [8].
9 Conclusion
Holding a 2-bit drafter in FPGA on-chip memory makes each drafter forward 2.1× faster and 2.3× lower in energy than on the H200. It also lets the H200 verify at 18% lower power. On box1 it beat the GPU drafting for itself on the same tree. On talon2 today the host path and the cards' static power cost more than the drafting saves. The cards are idle at least half of every step, and that idle time is the headroom for the host fixes, for sharing one card set across GPU nodes and for a drafter ASIC.
References
- Y. Leviathan, M. Kalman, Y. Matias. Fast Inference from Transformers via Speculative Decoding. ICML 2023.
- C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, J. Jumper. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv:2302.01318, 2023.
- Y. Li, F. Wei, C. Zhang, H. Zhang. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. ICML 2024.
- Y. Li, F. Wei, C. Zhang, H. Zhang. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. EMNLP 2024.
- Y. Li, F. Wei, C. Zhang, H. Zhang. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. arXiv:2503.01840, 2025.
- T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, T. Dao. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. ICML 2024.
- X. Miao et al. SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification. ASPLOS 2024.
- E. Frantar, S. Ashkboos, T. Hoefler, D. Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023.
- S. Hong et al. DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation. MICRO 2022.
- S. Zeng et al. FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs. FPGA 2024.
- AMD. Alveo V80 Compute Accelerator Data Sheet (DS1013).
Repository paths are relative to reports/results/ unless they start with a top-level name. Evidence for this version: reports/results/FUNDRAISING_POWER_PRICING_20261104/ (SOURCES.md, econ.py, build_v6.py). Earlier versions of this page stay on the site at /v/1/ to /v/5/.