EAGLEEAGLE CORP
EAGLEEAGLE CORP
01 / 10 · EAGLE4

Keep the drafter's weights next to the math.

4 × V80FPGA cards hold the whole drafter on chip
1 × H200only verifies

A speculative-decoding drafter, quantised to 2 bits, resident in FPGA on-chip memory. measured system as built, 2026-11-04.

02 / 10 · Problem

On a GPU the drafter competes with the target for the same chip.

60.4%of a GPU speculative-decoding step is drafting
15×the drafter's weights re-read per depth-15 step

measured Llama-3.2-3B, humaneval 80, depth 15 width 4, GPU only, 2026-09-16 · GPU_DRAFTER_VERIFY_SPLIT_BITNET721_D15_20260916.md:12, :98-108; 15 drafter passes per depth-15 step · WHY_SLOW_ROOFLINE_20260902.md:91. Each tree level waits for the one above, so the GPU streams the drafter weights again per level.

03 / 10 · Insight

Move the drafter onto dedicated silicon where every weight already sits beside a multiplier.

35.4 MB2-bit drafter body (282 MB at 16 bit)
~69 MBUltraRAM on one V80
0 Bbody weights read from HBM per pass

derived sizes from the 2-bit checkpoint header and 1,925 × 36 KB UltraRAM; "0 B" is the design (body weights resident in UltraRAM; only 64 rows of the 16-bit lm-head are gathered from card HBM per pass) · tested_archive/weights/WEIGHTS.md:13 (788,004,864 B lm-head), :64 (2-bit checkpoint).

04 / 10 · What we built

A four-card V80 drafter beside an H200, serving two models live.

Llama-3.2-3Blive
Qwen3.8-27Blive, 262,144-token context
ATT ⇄ MLPVtwo card pairs over QSFP links

measured public demo at demo.yshicarrick.com (box1) · TRACKLIST_20260923.md:548; USER_DIALOGUE_LOG.md:1408 · split: FOURCARD_DESIGN_20261016/DOSSIER_4C1_SYSTEM_20261019.md:5-7 (system), :34 (66 MHz), :44-51 (resources) (md5 b4c31482…).

05 / 10 · Traction

Fast and live today.

1,047 tok/sLlama, accept 7.11 — one public demo request, box1, 2026-11-04
839.65 tok/sLlama record, accept 5.88, humaneval 80 × 256, box1, 2026-10-23
234 tok/sQwen3.8-27B live, accept 6.74 — one request, 2026-11-04

measured TRACKLIST_20260923.md:686 (17:31:22Z entry); USER_DIALOGUE_LOG.md:1652 · FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:1156 · TRACKLIST_20260923.md:664 (16:43:44Z entry); USER_DIALOGUE_LOG.md:1624. Identity: per-level trees 9/9 identical to the recorded image for both models; Qwen texts 80/80 identical to the record · TALON2_POWER_ECON_20261104.md:42; SESSION_CONTEXT_BACKUP_20261104.md:91.

06 / 10 · Efficiency edge

The cards draft faster and with less energy than the GPU drafting for itself.

2.1×faster drafter forward: 0.177 ms (4 V80) vs 0.37 ms (H200)
2.3×less energy per forward: 45 mJ (4 V80) vs 102 mJ (H200)
−18%H200 power when it only verifies: 227.0 W vs 276.7 W

measured forward times box1 2026-10-19, mt_bench 50 × 32 w2 d7 · FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:379 (GPU_VS_FPGA_DRAFTER_20261019.md); powers talon2 2026-11-04 run means · TALON2_POWER_ECON_20261104.md (commit 1b23c931, md5 0e9214f6…) · FUNDRAISING_POWER_PRICING_20261104/talon2_power_econ_analysis_20261104.jsonl. Energy = time × device power, each device alone, never summed derived.

07 / 10 · Headroom

The drafter cards sit idle most of every step.

≥ 85.1%idle per Qwen3.8-27B step (busy 4.00 of 26.94 ms)
≥ 52.9%idle per Llama-3.2-3B step (busy 3.81 of 8.10 ms)
~6 nodesH200 nodes one card set could draft for (Qwen) — time-sharing not built yet

measured idle fractions talon2 2026-11-04, batch 1 · TALON2_POWER_ECON_20261104.md:114-115. projection nodes = 1 / busy fraction, assuming multi-stream time-sharing, which is not implemented.

08 / 10 · Speed outlook

Moving the drafter from FPGA to an ASIC typically gives about 10× more speed.

0.177 msdrafter forward on 4 V80 today (measured)
~0.018 msthe same forward at ~10× (projection)
~223 µsone full d7 draft tree at ~10×; 2,234 µs on the cards today (projection)

projection ~10× is the industry rule of thumb for an FPGA-to-ASIC transition, applied to the measured numbers: 0.177 ms forward (box1 2026-10-19, FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:379) and 2,233.8 µs per full d7 tree (talon2 2026-11-09, DEMO_SPEED_AUDIT_20261110.md §1.3). The H200 verify time is not changed by it.

09 / 10 · Roadmap

Today on talon2 GPU-only EAGLE is faster end to end (827 vs 723 tok/s). The gap is on the host, and each item below attacks it.

+12–16%host path: one endpoint, fewer serial steps (modelled)
−55%export bridge cycles per item, 6,230 → 2,767 (unit-gate measured)
1.55×two streams per card set (measured on the 2-card pair; not yet on 4 cards)

measured/projection talon2 827 vs 723: TALON2_POWER_ECON_20261104.md (commit 1b23c931, md5 0e9214f6…) · FUNDRAISING_POWER_PRICING_20261104/talon2_power_econ_analysis_20261104.jsonl · host path projection SEVENCARD_BOARD_20261026/T2_HOST_TRAFFIC.md:257, :261 · export fix measured unit gate TRACKLIST_20260923.md:613, :636 (EXPBURST unit gate; md15 identity GREEN) · clock-fix image 0xE4B7133F closes timing (WNS +0.080 / WHS +0.011) measured TRACKLIST_20260923.md:647 (0xE4B7133F FINAL_TRUE WNS +0.080 / WHS +0.011); W4 speed image child level −9.2% Qwen TRACKLIST_20260923.md:585 (0xE4B7133B, child level −9.2% Qwen vs hold) · KV on card ≈0.17 ms/step projection SEVENCARD_BOARD_20261026/T2_HOST_TRAFFIC.md:257, :261 · two streams QWEN27B_DRAFTER_20261020/NSTREAM_DRIVER_20261023.md:288; 4C1_DELIVERY_TRACKER.md:1156 (4-card N-stream: no gain).

10 / 10 · Infrastructure

Two GPU + FPGA servers and a licensed build mesh run every experiment.

2 serversH200 + 4 V80 (box1) · H200 + 7 V80 (talon2)
4+ build hostslicensed Vivado image builds in parallel

measured infrastructure as operated, 2026-11-04.

EAGLEEAGLE CORP
White paper · 2026-11-04 · v6

EAGLE4: A Speculative-Decoding Drafter Held in FPGA On-Chip Memory

Calculations and hypothesis — every derived number and every projection on this page

Everything below the paper's title that is not a direct measurement is listed here with its formula. The speed outlook is in the clearly marked §6.2 Speed outlook.

  • Energy per drafter forward = forward time × device run-mean power: H200 0.37 ms × 276.67 W = 102.4 mJ; 4 V80 0.177 ms × 255.03 W = 45.1 mJ; ratio 2.27×. Per Llama step: H200 draft + extend (2.49 + 0.41) ms × 276.67 W = 0.80 J (box1 mt_bench split; the talon2 draft-time split is not yet measured); 4 V80 card time 3.81 ms × 255.03 W = 0.97 J (upper bound). Never summed.
  • J/token = mean W ÷ (tokens ÷ time). H200 power change 1 − 227.0/276.7 = −17.9% (Qwen −10.8%). 2.1× = 0.37 / 0.177.
  • 35.4 MB = sum of the 2-bit qweight tensors; 282 MB at 16 bit; ~69 MB UltraRAM = 1,925 × 36 KB.
  • Duty: busy = card window ÷ step = 3.81/8.10 = 47.1%, 4.00/26.94 = 14.9% (measured ratio). Nodes per card set = 1 / busy = 2.1 → 2 (Llama), 6.7 → 6 (Qwen) — projection; time-sharing is not built.
  • Speed outlook (§6.2, projection): An FPGA-to-ASIC move typically brings about 10× speed (industry rule of thumb). Applied to the measured drafter forward: 0.177 ms on four V80 cards → ~0.018 ms; one full d7 draft tree, 2,234 µs on the cards today (2026-11-09) → ~223 µs.
  • Roadmap gains (§7): host path ≈0.75–1.0 ms/step ≈ +12–16% (modelled); KV-prefill card ≈0.17 ms/step (modelled); two streams 1.55× aggregate (measured on the 2-card pair, not on 4 cards).

Abstract

Speculative decoding makes a large language model faster by letting a small drafter propose a tree of tokens that the target model verifies in one pass. On a GPU the drafter and the target share the chip: in our GPU-only profile drafting takes 60.4% of each step. We move the EAGLE drafter onto four AMD Alveo V80 FPGAs, quantise its body to 2 bits and keep every body weight in on-chip UltraRAM, so the H200 only verifies. One drafter forward takes 0.177 ms on the four cards against 0.37 ms on the H200 (2.1×) and about 45 mJ against 102 mJ (2.3×). When it only verifies, the H200 draws 18% less power. With Llama-3.2-3B as the target, the system reaches 839.65 tok/s on humaneval 80 × 256 (accept 5.88). It also beats the H200 drafting for itself on the same depth-15 tree (846.20 vs 741.27 tok/s). The same four cards serve Qwen3.8-27B with a 262,144-token context. On a second server measured on 2026-11-04, the GPU-only baseline was faster end to end (826.58 vs 723.09 tok/s). We trace that gap to host overhead and the cards' static power, and measure the cards at least 52.9% (Llama) and 85.1% (Qwen) idle. That idle time is the headroom our roadmap targets.

1 Introduction

Large language models generate one token per forward pass, so a single user's speed is set by how fast the hardware can read the model's weights. Speculative decoding [1][2] breaks this limit: a cheap drafter guesses several tokens, organised as a tree [7], and the target model checks all of them at once and keeps the longest correct prefix. EAGLE [3][4][5] drafts from the target's own hidden features and is among the strongest drafters in open serving stacks.

The drafter is small but sequential. Each tree level depends on the level above, so on a GPU a depth-15 tree costs 15 drafter passes, and every pass re-reads the drafter's weights from HBM. In our GPU-only profile (Llama-3.2-3B, humaneval 80, depth 15, width 4) drafting takes 7.254 ms of a 12.0 ms step, 60.4% measured, and it runs on the same GPU the target needs.

This paper describes EAGLE4, a drafter that runs on dedicated FPGA silicon with its weights resident beside the multipliers. Our contributions are: (i) a hand-written RTL drafter with 2-bit weights in UltraRAM and on-chip tree decoding (§3); (ii) a four-card split of one drafter over QSFP links (§3–4); (iii) a measured evaluation of speed, latency, energy and duty cycle against the H200 drafting for itself (§5–6); and (iv) an honest account of where today's system loses end to end, and why (§7).

2 Background

Speculative decoding. A drafter proposes tokens and the target verifies them in parallel; accepted tokens are exactly those the target would have produced, so output quality is the target's [1][2]. Speed depends on the accept length (tokens kept per verification) and on the time per step: draft plus verify plus host work.

EAGLE. EAGLE [3] uses one transformer layer that takes the target's last hidden states and predicts the next features and tokens; EAGLE-2 [4] grows a dynamic tree; EAGLE-3 [5] scales training. Medusa [6] uses extra heads instead of a drafter layer.

Drafter cost. At batch 1 a drafter pass is bound by weight reads: our drafter's 16-bit weights are 1.071 GB per pass on the GPU, 74% of it the 128,256-word lm-head derived. A 2-bit body (GPTQ-style [8]) is 35.4 MB, small enough for the ~69 MB of UltraRAM on one V80 [11].

3 Architecture

On-chip 2-bit drafter. The drafter is one transformer layer (hidden 3,072, MLP 8,192, vocabulary 128,256) built as a fixed pipeline of stages: input norm (G1), QKV projection (G4), attention over the KV cache (G5), output projection (G6), gate/up projections and SiLU (G7), down projection (G9), final norm and low-rank projection (G11/GLM), a cheap 2-bit lm-head over the whole vocabulary (G12), on-chip top-K tree selection (TREE) and an exact 16-bit rescore of the shortlisted rows (G8).

Residency. 2-bit body weights and the cheap lm-head sit in UltraRAM; scales and norms in block RAM; the K cache in UltraRAM and the V cache in block RAM. Only the 16-bit exact lm-head (788 MB) lives in card HBM, and G8 gathers 64 of its rows per pass. All of it is loaded once at boot; no body weight is read from HBM during a pass.

Tree decode. Each tree level runs the full layer for the frontier, keeps the top candidates on chip and feeds the children to the next level, with no host round trip between levels.

Four-card split. An attention pair (ATT, tensor-parallel by two: G1, G4, G5, G6) and an MLP + lm-head pair (MLPV: G7, G9, G11/GLM, G12, TREE, G8) are joined by QSFP Aurora links. One width-2 tree level takes 342.4 µs on the four cards (context 566) measured.

H200target verify ATT pair (2 × V80)G1 · G4 · G5 attention · G6 MLPV pair (2 × V80)G7 · G9 · G11/GLM · G12 · TREE · G8 QSFP PCIe on every V80 UltraRAM: 2-bit body + cheap lm-head, K cacheblock RAM: scales, norms, V cacheHBM: 16-bit exact lm-head (64 rows/pass)loaded once at boottree decode on chip, no host per level
Figure 1. System: the H200 verifies; four V80 cards draft. ATT and MLPV pairs exchange activations over QSFP; the host moves accepted tokens and the draft tree over PCIe.Source: FOURCARD_DESIGN_20261016/DOSSIER_4C1_SYSTEM_20261019.md:5-7 (system), :34 (66 MHz), :44-51 (resources) (md5 b4c31482…)

4 Implementation

The drafter is hand-written SystemVerilog on AMD Alveo V80 cards (Versal HBM, three SLRs) [11], built with Vivado 2025.1 on the AVED platform and driven over QDMA. The drafter clock is 66 MHz, with a faster array clock for the GEMV pump. The current clock-fix image 0xE4B7133F closes timing at WNS +0.080 ns / WHS +0.011 ns measured. Table 1 gives the measured resources of the two card images. DSP58 binds on the attention card, and inter-die wires (SLLs) are the placement limit to watch.

Table 1. Resources of the two 4-card images, post-place, as reported for the 2026-10-19 images (ATT 0xE4B71288, MLPV 0xE4B71287), and timing of the current clock-fix image.
ATT 0x1288MLPV 0x1287
LUT518,096 (20.13%)795,271 (30.89%)
DSP587,377 (68.00%), binding6,288 (57.96%)
BRAM36 / URAM35.78% / 35.74%28.86% / 26.60%
SLICE device / worst SLR42.56% / 62.55%56.89% / 74.81%
SLL SLR0↔113,997 (74.18%)9,915 (52.54%)
routed WNS / WHS+0.023 / +0.010 ns+0.007 / +0.010 ns
Source: FOURCARD_DESIGN_20261016/DOSSIER_4C1_SYSTEM_20261019.md:5-7 (system), :34 (66 MHz), :44-51 (resources) (md5 b4c31482…); current image TRACKLIST_20260923.md:647 (0xE4B7133F FINAL_TRUE WNS +0.080 / WHS +0.011)

The host stack is SGLang with the cards as the drafter backend; a card process per pair exchanges the root state and the draft tree with the GPU process each step.

5 Evaluation

Setup. Two servers: box1 (one H200, four V80) and talon2 (one H200 NVL, seven V80, four used). Targets: Llama-3.2-3B and Qwen3.8-27B. Workloads: humaneval (80 prompts × 256 new tokens, or 20 × 512) and mt_bench (50 × 32), batch 1, tree width 2. The baseline is the H200 running SGLang's own EAGLE drafter (exec mode −1). Table 2 lists every end-to-end number used here.

Table 2. End-to-end results used in this paper. All Llama-3.2-3B unless noted; batch 1; tree width 2.
resulthost, dateworkloadtok/saccept
H200 + 4 V80talon2, 11-04humaneval 80×256, d7723.095.88
H200 GPU-only EAGLEtalon2, 11-04humaneval 80×256, d7826.585.97
Qwen3.8-27B · H200 + 4 V80talon2, 11-04humaneval 80×256, d7236.177.14
Qwen3.8-27B · GPU-only EAGLEtalon2, 11-04humaneval 80×256, d7256.587.13
H200 + 4 V80, recordbox1, 10-23humaneval 80×256, d7839.655.88
H200 + 4 V80, same treebox1, 10-20humaneval 20×512, d15846.207.29
H200 drafting for itself, same treebox1, 10-20humaneval 20×512, d15741.277.97
H200 + 4 V80 vs GPU self, deep treebox1, 10-20mt_bench 50×32, d12497.05 vs 413.44.06 vs 4.30
live demo, one requestbox1, 11-04chat1,047.127.111
Qwen3.8-27B live demo, one requestbox1, 11-04chat234.436.737
Source: TALON2_POWER_ECON_20261104.md (commit 1b23c931, md5 0e9214f6…) · FUNDRAISING_POWER_PRICING_20261104/talon2_power_econ_analysis_20261104.jsonl; FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:1156; FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:868; EXP_K1_V59_HUMANEVAL_20261020.md:68-72; FOURCARD_DESIGN_20261016/HUMANEVAL_VS_MTBENCH_ACCEPT_20261020.md:89-93; WAVEB_DEEP_SWEEP_20261020.md:10-18; TRACKLIST_20260923.md:686 (17:31:22Z entry); USER_DIALOGUE_LOG.md:1652; TRACKLIST_20260923.md:664 (16:43:44Z entry); USER_DIALOGUE_LOG.md:1624

Throughput. Figure 2 compares the two systems on the same server. Figure 3 shows the best results on box1. On box1 the four-card drafter beats the H200 drafting for itself on the same depth-15 tree by 14%, and at depth 12 on mt_bench by 20%. On talon2 on 2026-11-04 the GPU-only baseline is faster, by 12.5% (Llama) and 7.9% (Qwen); §7 explains why. Accept lengths are within 0.1 of the GPU's on talon2.

Llama · H200 + 4 V80: 723.09 tok/sLlama · H200 + 4 V80723.09 tok/sLlama · H200 GPU-only EAGLE: 826.58 tok/sLlama · H200 GPU-only EAGLE826.58 tok/sQwen · H200 + 4 V80: 236.17 tok/sQwen · H200 + 4 V80236.17 tok/sQwen · H200 GPU-only EAGLE: 256.58 tok/sQwen · H200 GPU-only EAGLE256.58 tok/s
H200 + 4 V80 (this work)H200 GPU-only EAGLE
Figure 2. End-to-end throughput on the same server (talon2, 2026-11-04), humaneval 80 × 256, tree width 2 depth 7, batch 1. Accept length: Llama 5.88 vs 5.97; Qwen 7.14 vs 7.13. Llama 2 rounds per arm, Qwen 1 round. GPU-only is faster on this host today (§7).Source: TALON2_POWER_ECON_20261104.md (commit 1b23c931, md5 0e9214f6…) · FUNDRAISING_POWER_PRICING_20261104/talon2_power_econ_analysis_20261104.jsonl
live demo, one request (11-04): 1,047.12 tok/slive demo, one request (11-04)1,047.12 tok/shumaneval 20×512, d15 (10-20): 846.20 tok/shumaneval 20×512, d15 (10-20)846.20 tok/sGPU self-draft, same tree (10-20): 741.27 tok/sGPU self-draft, same tree (10-20)741.27 tok/shumaneval 80×256 record (10-23): 839.65 tok/shumaneval 80×256 record (10-23)839.65 tok/starget model alone, 20×512 (10-20): 301.41 tok/starget model alone, 20×512 (10-20)301.41 tok/s
H200 + 4 V80H200 drafting for itselftarget alone
Figure 3. Best measured results on box1 (Llama-3.2-3B, one H200). Same tree, humaneval 20 × 512, w2 d15, 59 verify tokens: 846.20 @ 7.29 (4 V80) vs 741.27 @ 7.97 (H200 drafting for itself). Record humaneval 80 × 256, w2 d7: 839.65 @ 5.88. Live demo: one request, 1,047.12 tok/s @ 7.111.Source: FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:868; EXP_K1_V59_HUMANEVAL_20261020.md:68-72; FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:1156; TRACKLIST_20260923.md:686 (17:31:22Z entry); USER_DIALOGUE_LOG.md:1652; FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:1171

Drafter latency and energy. One drafter forward takes 0.177 ms on the four cards against 0.37 ms on the H200, and about 45 mJ against 102 mJ (Figures 4 and 5). Each device is shown alone; the two are never added together.

4 V80: 0.177 ms4 V800.177 msH200: 0.370 msH2000.370 ms
Figure 4. Latency of one drafter forward, each device alone. box1, 2026-10-19, mt_bench 50 × 32, w2 d7. The H200 value is a depth-slope fit over its CUDA-event profile.Source: FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:379 (GPU_VS_FPGA_DRAFTER_20261019.md)
4 V80: 45 mJ4 V8045 mJH200: 102 mJH200102 mJ
Figure 5. Energy of one drafter forward, each device alone (never summed): forward time × the device's run-mean power on talon2 2026-11-04 (4 V80 255.03 W in the FPGA run; H200 276.67 W in the GPU-only run). Derived from measured values; the power is the run mean, not draft-window power.Source: FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:379 (GPU_VS_FPGA_DRAFTER_20261019.md); TALON2_POWER_ECON_20261104.md (commit 1b23c931, md5 0e9214f6…) · FUNDRAISING_POWER_PRICING_20261104/talon2_power_econ_analysis_20261104.jsonl

Duty cycle. Per steady decode step the cards are busy at most 47.1% (Llama) and 14.9% (Qwen) of the time (Figure 6). The rest of the step is the H200 verifying.

Llama-3.2-3BLlama-3.2-3B · busy ≤: 47.1%busy ≤ 47.1%Llama-3.2-3B · idle ≥: 52.9%idle ≥ 52.9%Qwen3.8-27BQwen3.8-27B · busy ≤: 14.9%busy ≤ 14.9%Qwen3.8-27B · idle ≥: 85.1%idle ≥ 85.1%
Figure 6. FPGA duty cycle per steady decode step, as a share of the step (talon2, 2026-11-04, batch 1): Llama 3.81 of 8.10 ms, Qwen 4.00 of 26.94 ms. Busy = card window (from the early tree send to the last card join, including host transfer), so it is an upper bound: 47.1% (Llama) and 14.9% (Qwen) busy. H200 verify takes 68.8% / 75.0% of the step.Source: TALON2_POWER_ECON_20261104.md:114-115

H200 power when verify-only. With the drafting moved to the cards the H200 draws 227.0 W instead of 276.7 W (Llama), and 386.6 W instead of 433.3 W (Qwen) (Figure 7).

Llama · H200 verify-only: 227.0 WLlama · H200 verify-only227.0 WLlama · H200 drafting itself: 276.7 WLlama · H200 drafting itself276.7 WQwen · H200 verify-only: 386.6 WQwen · H200 verify-only386.6 WQwen · H200 drafting itself: 433.3 WQwen · H200 drafting itself433.3 W
H200 in the 4-V80 setupH200 GPU-only
Figure 7. H200 power while serving (talon2, 2026-11-04, nvidia-smi run means). With the cards drafting the H200 only verifies and draws 18% (Llama) and 11% (Qwen) less.Source: TALON2_POWER_ECON_20261104.md (commit 1b23c931, md5 0e9214f6…) · FUNDRAISING_POWER_PRICING_20261104/talon2_power_econ_analysis_20261104.jsonl

Correctness. The target verifies every token, so the output text is the target's. Draft trees on the 2-bit cards are compared level by level with the recorded image: 9/9 identical for Llama and for Qwen. The Qwen texts match the record on all 80 prompts measured (TALON2_POWER_ECON_20261104.md:42; SESSION_CONTEXT_BACKUP_20261104.md:91). Against a 16-bit reference drafter, the first tree level matches 8/8 and the second 6/8, which is the expected effect of 2-bit weights.

6 Cost and energy

6.1 Measured

Table 3 gives the measured power by device and Figure 8 the whole-server energy per token. The 4-V80 setup draws 255.0 W in its cards while serving and 251.1 W when idle: almost all of it is static. Per token the H200 itself uses less energy with the cards drafting: 0.314 vs 0.335 J (Llama) and 1.637 vs 1.689 J (Qwen). The whole server uses more: 1.535 vs 1.293 J (Llama).

Table 3. Measured power by device (talon2, 2026-11-04, 1 Hz run means, W). The 3 other V80s in the chassis are not part of the setup.
arm4 V803 other V80H200CPU packagechassis DCMI
idle251.1119.980.4132.9751.9
Llama · H200 + 4 V80255.0119.4227.0289.61,110.1
Llama · GPU-only251.9 (idle)119.5276.7202.51,068.6
Qwen · H200 + 4 V80252.4119.8386.6261.51,295.2
Source: TALON2_POWER_ECON_20261104.md (commit 1b23c931, md5 0e9214f6…) · FUNDRAISING_POWER_PRICING_20261104/talon2_power_econ_analysis_20261104.jsonl; TALON2_POWER_ECON_20261104.md:81, :86-88
Llama · H200 + 4 V80: 1.535 JLlama · H200 + 4 V801.535 JLlama · GPU-only: 1.293 JLlama · GPU-only1.293 JQwen · H200 + 4 V80: 5.484 JQwen · H200 + 4 V805.484 JQwen · GPU-only: 4.821 JQwen · GPU-only4.821 J
Figure 8. Whole-server energy per token (IPMI DCMI input power ÷ tok/s, talon2 2026-11-04). The GPU-only arms include the 7 idle V80s in the same chassis (~371 W). Qwen GPU-only uses the pre-flash window (4.82 J).Source: TALON2_POWER_ECON_20261104.md (commit 1b23c931, md5 0e9214f6…) · FUNDRAISING_POWER_PRICING_20261104/talon2_power_econ_analysis_20261104.jsonl; TALON2_POWER_ECON_20261104.md:81, :86-88

6.2 Speed outlook projection

Everything in this subsection is calculated, not measured. An FPGA-to-ASIC move typically brings about 10× speed (industry rule of thumb). Applied to the measured drafter forward: 0.177 ms on four V80 cards → ~0.018 ms; one full d7 draft tree, 2,234 µs on the cards today (2026-11-09) → ~223 µs. The H200 verify step and the host work are outside this projection and keep their measured cost.

7 Limitations

  • GPU-only is faster end to end on talon2 today: 826.58 vs 723.09 tok/s (Llama) and 256.58 vs 236.17 (Qwen), same server, same prompts. The cards draft faster, but each step carries about 1.4 ms of serial host work out of a 6.25–6.46 ms step (SEVENCARD_BOARD_20261026/T2_HOST_TRAFFIC.md:17 (md5 8b9e3f84…)) measured. The host CPU draws +87.1 W more (289.6 vs 202.5 W). The four V80s draw ~251 W even when idle.
  • Serial steps. One request at a time: draft and verify alternate, so the cards wait on the H200 for most of each step (§5).
  • Measurement scope. Llama talon2 arms have 2 rounds, Qwen 1. The H200 draft-time split on talon2 is not yet measured; per-forward energies use run-mean power, not draft-window power.
  • 2-bit drafter. Trees match the recorded image exactly but not a 16-bit drafter beyond the first level; the target verifies every token, so the text is the target's.
  • Roadmap, not results. Each item lists its measured or modelled effect. Host path: one endpoint and fewer serial steps, modelled +12–16%. Export fix: unit gate measured 6,230 → 2,767 bridge cycles per item; full-drafter identity green. Clock-fix image: timing closed at WNS +0.080 ns, which enables the W4 speed image (child level −9.2% measured, Qwen). KV on card: modelled ≈0.17 ms/step. Batched verify / two streams: 1.55× aggregate measured on the 2-card pair, no gain yet on four cards. Sources: SEVENCARD_BOARD_20261026/T2_HOST_TRAFFIC.md:257, :261; TRACKLIST_20260923.md:613, :636 (EXPBURST unit gate; md15 identity GREEN); TRACKLIST_20260923.md:647 (0xE4B7133F FINAL_TRUE WNS +0.080 / WHS +0.011); TRACKLIST_20260923.md:585 (0xE4B7133B, child level −9.2% Qwen vs hold); QWEN27B_DRAFTER_20261020/NSTREAM_DRIVER_20261023.md:288; 4C1_DELIVERY_TRACKER.md:1156 (4-card N-stream: no gain).

8 Related work

Speculative decoding [1][2] and tree verification [7] are the basis of this work; EAGLE [3][4][5] and Medusa [6] are the leading drafter designs, all run on the same GPU as the target. FPGA LLM accelerators such as DFX [9] and FlightLLM [10] run the whole model on FPGAs; we instead keep the target on the GPU and move only the latency-bound drafter, with weights fully on chip. Low-bit weight quantisation follows GPTQ [8].

9 Conclusion

Holding a 2-bit drafter in FPGA on-chip memory makes each drafter forward 2.1× faster and 2.3× lower in energy than on the H200. It also lets the H200 verify at 18% lower power. On box1 it beat the GPU drafting for itself on the same tree. On talon2 today the host path and the cards' static power cost more than the drafting saves. The cards are idle at least half of every step, and that idle time is the headroom for the host fixes, and for sharing one card set across GPU nodes.

References

  1. Y. Leviathan, M. Kalman, Y. Matias. Fast Inference from Transformers via Speculative Decoding. ICML 2023.
  2. C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, J. Jumper. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv:2302.01318, 2023.
  3. Y. Li, F. Wei, C. Zhang, H. Zhang. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. ICML 2024.
  4. Y. Li, F. Wei, C. Zhang, H. Zhang. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. EMNLP 2024.
  5. Y. Li, F. Wei, C. Zhang, H. Zhang. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. arXiv:2503.01840, 2025.
  6. T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, T. Dao. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. ICML 2024.
  7. X. Miao et al. SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification. ASPLOS 2024.
  8. E. Frantar, S. Ashkboos, T. Hoefler, D. Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023.
  9. S. Hong et al. DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation. MICRO 2022.
  10. S. Zeng et al. FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs. FPGA 2024.
  11. AMD. Alveo V80 Compute Accelerator Data Sheet (DS1013).

Repository paths are relative to reports/results/ unless they start with a top-level name. Evidence for this version: reports/results/FUNDRAISING_POWER_PRICING_20261104/ (SOURCES.md, econ.py, build_v6.py). Earlier versions: /p/eagle4-white-paper/ (v1 to v6).