EAGLEEAGLE CORP
EAGLEEAGLE CORP
01 / 11 · EAGLE4

Keep the drafter's weights next to the math.

4 × V80FPGA cards hold the whole drafter on chip
1 × H200only verifies

A speculative-decoding drafter, quantised to 2 bits, resident in FPGA on-chip memory. measured system as built, 2026-11-04.

02 / 11 · Problem

On a GPU the drafter competes with the target for the same chip.

60.4%of a GPU speculative-decoding step is drafting
15×the drafter's weights re-read per depth-15 step

measured Llama-3.2-3B, humaneval 80, depth 15 width 4, GPU only, 2026-09-16 · GPU_DRAFTER_VERIFY_SPLIT_BITNET721_D15_20260916.md:12, :98-108; 15 drafter passes per depth-15 step · WHY_SLOW_ROOFLINE_20260902.md:91. Each tree level waits for the one above, so the GPU streams the drafter weights again per level.

03 / 11 · Insight

Move the drafter onto dedicated silicon where every weight already sits beside a multiplier.

35.4 MB2-bit drafter body (282 MB at 16 bit)
~69 MBUltraRAM on one V80
0 Bbody weights read from HBM per pass

derived sizes from the 2-bit checkpoint header and 1,925 × 36 KB UltraRAM; "0 B" is the design (body weights resident in UltraRAM; only 64 rows of the 16-bit lm-head are gathered from card HBM per pass) · tested_archive/weights/WEIGHTS.md:13 (788,004,864 B lm-head), :64 (2-bit checkpoint).

04 / 11 · What we built

A four-card V80 drafter beside an H200, serving two models live.

Llama-3.2-3Blive
Qwen3.8-27Blive, 262,144-token context
ATT ⇄ MLPVtwo card pairs over QSFP links

measured public demo at demo.yshicarrick.com (box1) · TRACKLIST_20260923.md:548; USER_DIALOGUE_LOG.md:1408 · split: FOURCARD_DESIGN_20261016/DOSSIER_4C1_SYSTEM_20261019.md:5-7 (system), :34 (66 MHz), :44-51 (resources) (md5 b4c31482…).

05 / 11 · Traction

Fast and live today.

1,047 tok/sLlama, accept 7.11 — one public demo request, box1, 2026-11-04
839.65 tok/sLlama record, accept 5.88, humaneval 80 × 256, box1, 2026-10-23
234 tok/sQwen3.8-27B live, accept 6.74 — one request, 2026-11-04

measured TRACKLIST_20260923.md:686 (17:31:22Z entry); USER_DIALOGUE_LOG.md:1652 · FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:1156 · TRACKLIST_20260923.md:664 (16:43:44Z entry); USER_DIALOGUE_LOG.md:1624. Identity: per-level trees 9/9 identical to the recorded image for both models; Qwen texts 80/80 identical to the record · TALON2_POWER_ECON_20261104.md:42; SESSION_CONTEXT_BACKUP_20261104.md:91.

06 / 11 · Efficiency edge

The cards draft faster and with less energy than the GPU drafting for itself.

2.1×faster drafter forward: 0.177 ms (4 V80) vs 0.37 ms (H200)
2.3×less energy per forward: 45 mJ (4 V80) vs 102 mJ (H200)
−18%H200 power when it only verifies: 227.0 W vs 276.7 W

measured forward times box1 2026-10-19, mt_bench 50 × 32 w2 d7 · FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:379 (GPU_VS_FPGA_DRAFTER_20261019.md); powers talon2 2026-11-04 run means · TALON2_POWER_ECON_20261104.md (commit 1b23c931, md5 0e9214f6…) · FUNDRAISING_POWER_PRICING_20261104/talon2_power_econ_analysis_20261104.jsonl. Energy = time × device power, each device alone, never summed derived.

07 / 11 · Headroom

The drafter cards sit idle most of every step.

≥ 85.1%idle per Qwen3.8-27B step (busy 4.00 of 26.94 ms)
≥ 52.9%idle per Llama-3.2-3B step (busy 3.81 of 8.10 ms)
~6 nodesH200 nodes one card set could draft for (Qwen) — time-sharing not built yet

measured idle fractions talon2 2026-11-04, batch 1 · TALON2_POWER_ECON_20261104.md:114-115. projection nodes = 1 / busy fraction, assuming multi-stream time-sharing, which is not implemented.

08 / 11 · Path to ASIC

One $5,000 drafter ASIC ties a GPU-only H200 on tokens per dollar at 962 tok/s.

2.109M tokens per $, GPU-only H200 (today, 3 years)
1.585M tokens per $, H200 + 1 ASIC at today's FPGA speed
962 tok/sLlama speed the 1-ASIC setup needs to tie; box1 record 839.65

projection ASIC price $5,000 (user projection); ASIC speed and power assumed = the measured 4-V80 path (723.09 tok/s, 990.6 W setup); H200 $34,500, 14.04 ¢/kWh, 3 years, 100% duty. Formulas in the white paper §6.2 · FUNDRAISING_POWER_PRICING_20261104/econ.py → econ_out.json.

09 / 11 · Roadmap

Today on talon2 GPU-only EAGLE is faster end to end (827 vs 723 tok/s). The gap is on the host, and each item below attacks it.

+12–16%host path: one endpoint, fewer serial steps (modelled)
−55%export bridge cycles per item, 6,230 → 2,767 (unit-gate measured)
1.55×two streams per card set (measured on the 2-card pair; not yet on 4 cards)

measured/projection talon2 827 vs 723: TALON2_POWER_ECON_20261104.md (commit 1b23c931, md5 0e9214f6…) · FUNDRAISING_POWER_PRICING_20261104/talon2_power_econ_analysis_20261104.jsonl · host path projection SEVENCARD_BOARD_20261026/T2_HOST_TRAFFIC.md:257, :261 · export fix measured unit gate TRACKLIST_20260923.md:613, :636 (EXPBURST unit gate; md15 identity GREEN) · clock-fix image 0xE4B7133F closes timing (WNS +0.080 / WHS +0.011) measured TRACKLIST_20260923.md:647 (0xE4B7133F FINAL_TRUE WNS +0.080 / WHS +0.011); W4 speed image child level −9.2% Qwen TRACKLIST_20260923.md:585 (0xE4B7133B, child level −9.2% Qwen vs hold) · KV on card ≈0.17 ms/step projection SEVENCARD_BOARD_20261026/T2_HOST_TRAFFIC.md:257, :261 · two streams QWEN27B_DRAFTER_20261020/NSTREAM_DRIVER_20261023.md:288; 4C1_DELIVERY_TRACKER.md:1156 (4-card N-stream: no gain).

10 / 11 · Team and infrastructure

[Team — to be filled in by the founders]

2 serversH200 + 4 V80 (box1) · H200 + 7 V80 (talon2)
4+ build hostslicensed Vivado image builds in parallel

measured infrastructure as operated, 2026-11-04. Team names, roles and advisors: placeholder for the founders.

11 / 11 · The ask

[Ask — amount, use of funds and milestones to be filled in by the founders]

$ —raise (placeholder)
— monthsrunway to ASIC tape-out decision (placeholder)

Placeholder slide: no numbers are claimed here.

EAGLEEAGLE CORP
White paper · 2026-11-04 · v6

EAGLE4: A Speculative-Decoding Drafter Held in FPGA On-Chip Memory

[Authors — placeholder]

Calculations and hypothesis — every derived number and every projection on this page

Everything below the paper's title that is not a direct measurement is listed here with its formula. Projections are also collected in the clearly marked §6.2 Projections.

  • Energy per drafter forward = forward time × device run-mean power: H200 0.37 ms × 276.67 W = 102.4 mJ; 4 V80 0.177 ms × 255.03 W = 45.1 mJ; ratio 2.27×. Per Llama step: H200 draft + extend (2.49 + 0.41) ms × 276.67 W = 0.80 J (box1 mt_bench split; the talon2 draft-time split is not yet measured); 4 V80 card time 3.81 ms × 255.03 W = 0.97 J (upper bound). Never summed.
  • J/token = mean W ÷ (tokens ÷ time). H200 power change 1 − 227.0/276.7 = −17.9% (Qwen −10.8%). 2.1× = 0.37 / 0.177.
  • 35.4 MB = sum of the 2-bit qweight tensors; 282 MB at 16 bit; ~69 MB UltraRAM = 1,925 × 36 KB.
  • Duty: busy = card window ÷ step = 3.81/8.10 = 47.1%, 4.00/26.94 = 14.9% (measured ratio). Nodes per card set = 1 / busy = 2.1 → 2 (Llama), 6.7 → 6 (Qwen) — projection; time-sharing is not built.
  • Cost (§6.2): capex = $H200 + n × $chip; 3-y energy = setup W × L / 3.6×10⁶ × $/kWh, L = 94,608,000 s; tokens per $ = tok/s × L / (capex + energy); tok/s to tie = TPD_GPU × (capex + energy) / L. Prices: H200 $34,500 (range $30,000–45,084), V80 $9,495, ASIC $5,000 (user projection), 14.04 ¢/kWh (EIA commercial 2026 YTD; range 9.09–14.47 ¢).
  • Roadmap gains (§7): host path ≈0.75–1.0 ms/step ≈ +12–16% (modelled); KV-prefill card ≈0.17 ms/step (modelled); two streams 1.55× aggregate (measured on the 2-card pair, not on 4 cards).

Abstract

Speculative decoding makes a large language model faster by letting a small drafter propose a tree of tokens that the target model verifies in one pass. On a GPU the drafter and the target share the chip: in our GPU-only profile drafting takes 60.4% of each step. We move the EAGLE drafter onto four AMD Alveo V80 FPGAs, quantise its body to 2 bits and keep every body weight in on-chip UltraRAM, so the H200 only verifies. One drafter forward takes 0.177 ms on the four cards against 0.37 ms on the H200 (2.1×) and about 45 mJ against 102 mJ (2.3×). When it only verifies, the H200 draws 18% less power. With Llama-3.2-3B as the target, the system reaches 839.65 tok/s on humaneval 80 × 256 (accept 5.88). It also beats the H200 drafting for itself on the same depth-15 tree (846.20 vs 741.27 tok/s). The same four cards serve Qwen3.8-27B with a 262,144-token context. On a second server measured on 2026-11-04, the GPU-only baseline was faster end to end (826.58 vs 723.09 tok/s). We trace that gap to host overhead and the cards' static power, and measure the cards at least 52.9% (Llama) and 85.1% (Qwen) idle. That idle time is the headroom our roadmap and a drafter ASIC target.

1 Introduction

Large language models generate one token per forward pass, so a single user's speed is set by how fast the hardware can read the model's weights. Speculative decoding [1][2] breaks this limit: a cheap drafter guesses several tokens, organised as a tree [7], and the target model checks all of them at once and keeps the longest correct prefix. EAGLE [3][4][5] drafts from the target's own hidden features and is among the strongest drafters in open serving stacks.

The drafter is small but sequential. Each tree level depends on the level above, so on a GPU a depth-15 tree costs 15 drafter passes, and every pass re-reads the drafter's weights from HBM. In our GPU-only profile (Llama-3.2-3B, humaneval 80, depth 15, width 4) drafting takes 7.254 ms of a 12.0 ms step, 60.4% measured, and it runs on the same GPU the target needs.

This paper describes EAGLE4, a drafter that runs on dedicated FPGA silicon with its weights resident beside the multipliers. Our contributions are: (i) a hand-written RTL drafter with 2-bit weights in UltraRAM and on-chip tree decoding (§3); (ii) a four-card split of one drafter over QSFP links (§3–4); (iii) a measured evaluation of speed, latency, energy and duty cycle against the H200 drafting for itself (§5–6); and (iv) an honest account of where today's system loses end to end, and why (§7).

2 Background

Speculative decoding. A drafter proposes tokens and the target verifies them in parallel; accepted tokens are exactly those the target would have produced, so output quality is the target's [1][2]. Speed depends on the accept length (tokens kept per verification) and on the time per step: draft plus verify plus host work.

EAGLE. EAGLE [3] uses one transformer layer that takes the target's last hidden states and predicts the next features and tokens; EAGLE-2 [4] grows a dynamic tree; EAGLE-3 [5] scales training. Medusa [6] uses extra heads instead of a drafter layer.

Drafter cost. At batch 1 a drafter pass is bound by weight reads: our drafter's 16-bit weights are 1.071 GB per pass on the GPU, 74% of it the 128,256-word lm-head derived. A 2-bit body (GPTQ-style [8]) is 35.4 MB, small enough for the ~69 MB of UltraRAM on one V80 [11].

3 Architecture

On-chip 2-bit drafter. The drafter is one transformer layer (hidden 3,072, MLP 8,192, vocabulary 128,256) built as a fixed pipeline of stages: input norm (G1), QKV projection (G4), attention over the KV cache (G5), output projection (G6), gate/up projections and SiLU (G7), down projection (G9), final norm and low-rank projection (G11/GLM), a cheap 2-bit lm-head over the whole vocabulary (G12), on-chip top-K tree selection (TREE) and an exact 16-bit rescore of the shortlisted rows (G8).

Residency. 2-bit body weights and the cheap lm-head sit in UltraRAM; scales and norms in block RAM; the K cache in UltraRAM and the V cache in block RAM. Only the 16-bit exact lm-head (788 MB) lives in card HBM, and G8 gathers 64 of its rows per pass. All of it is loaded once at boot; no body weight is read from HBM during a pass.

Tree decode. Each tree level runs the full layer for the frontier, keeps the top candidates on chip and feeds the children to the next level, with no host round trip between levels.

Four-card split. An attention pair (ATT, tensor-parallel by two: G1, G4, G5, G6) and an MLP + lm-head pair (MLPV: G7, G9, G11/GLM, G12, TREE, G8) are joined by QSFP Aurora links. One width-2 tree level takes 342.4 µs on the four cards (context 566) measured.

H200target verify ATT pair (2 × V80)G1 · G4 · G5 attention · G6 MLPV pair (2 × V80)G7 · G9 · G11/GLM · G12 · TREE · G8 QSFP PCIe on every V80 UltraRAM: 2-bit body + cheap lm-head, K cacheblock RAM: scales, norms, V cacheHBM: 16-bit exact lm-head (64 rows/pass)loaded once at boottree decode on chip, no host per level
Figure 1. System: the H200 verifies; four V80 cards draft. ATT and MLPV pairs exchange activations over QSFP; the host moves accepted tokens and the draft tree over PCIe.Source: FOURCARD_DESIGN_20261016/DOSSIER_4C1_SYSTEM_20261019.md:5-7 (system), :34 (66 MHz), :44-51 (resources) (md5 b4c31482…)

4 Implementation

The drafter is hand-written SystemVerilog on AMD Alveo V80 cards (Versal HBM, three SLRs) [11], built with Vivado 2025.1 on the AVED platform and driven over QDMA. The drafter clock is 66 MHz, with a faster array clock for the GEMV pump. The current clock-fix image 0xE4B7133F closes timing at WNS +0.080 ns / WHS +0.011 ns measured. Table 1 gives the measured resources of the two card images. DSP58 binds on the attention card, and inter-die wires (SLLs) are the placement limit to watch.

Table 1. Resources of the two 4-card images, post-place, as reported for the 2026-10-19 images (ATT 0xE4B71288, MLPV 0xE4B71287), and timing of the current clock-fix image.
ATT 0x1288MLPV 0x1287
LUT518,096 (20.13%)795,271 (30.89%)
DSP587,377 (68.00%), binding6,288 (57.96%)
BRAM36 / URAM35.78% / 35.74%28.86% / 26.60%
SLICE device / worst SLR42.56% / 62.55%56.89% / 74.81%
SLL SLR0↔113,997 (74.18%)9,915 (52.54%)
routed WNS / WHS+0.023 / +0.010 ns+0.007 / +0.010 ns
Source: FOURCARD_DESIGN_20261016/DOSSIER_4C1_SYSTEM_20261019.md:5-7 (system), :34 (66 MHz), :44-51 (resources) (md5 b4c31482…); current image TRACKLIST_20260923.md:647 (0xE4B7133F FINAL_TRUE WNS +0.080 / WHS +0.011)

The host stack is SGLang with the cards as the drafter backend; a card process per pair exchanges the root state and the draft tree with the GPU process each step.

5 Evaluation

Setup. Two servers: box1 (one H200, four V80) and talon2 (one H200 NVL, seven V80, four used). Targets: Llama-3.2-3B and Qwen3.8-27B. Workloads: humaneval (80 prompts × 256 new tokens, or 20 × 512) and mt_bench (50 × 32), batch 1, tree width 2. The baseline is the H200 running SGLang's own EAGLE drafter (exec mode −1). Table 2 lists every end-to-end number used here.

Table 2. End-to-end results used in this paper. All Llama-3.2-3B unless noted; batch 1; tree width 2.
resulthost, dateworkloadtok/saccept
H200 + 4 V80talon2, 11-04humaneval 80×256, d7723.095.88
H200 GPU-only EAGLEtalon2, 11-04humaneval 80×256, d7826.585.97
Qwen3.8-27B · H200 + 4 V80talon2, 11-04humaneval 80×256, d7236.177.14
Qwen3.8-27B · GPU-only EAGLEtalon2, 11-04humaneval 80×256, d7256.587.13
H200 + 4 V80, recordbox1, 10-23humaneval 80×256, d7839.655.88
H200 + 4 V80, same treebox1, 10-20humaneval 20×512, d15846.207.29
H200 drafting for itself, same treebox1, 10-20humaneval 20×512, d15741.277.97
H200 + 4 V80 vs GPU self, deep treebox1, 10-20mt_bench 50×32, d12497.05 vs 413.44.06 vs 4.30
live demo, one requestbox1, 11-04chat1,047.127.111
Qwen3.8-27B live demo, one requestbox1, 11-04chat234.436.737
Source: TALON2_POWER_ECON_20261104.md (commit 1b23c931, md5 0e9214f6…) · FUNDRAISING_POWER_PRICING_20261104/talon2_power_econ_analysis_20261104.jsonl; FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:1156; FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:868; EXP_K1_V59_HUMANEVAL_20261020.md:68-72; FOURCARD_DESIGN_20261016/HUMANEVAL_VS_MTBENCH_ACCEPT_20261020.md:89-93; WAVEB_DEEP_SWEEP_20261020.md:10-18; TRACKLIST_20260923.md:686 (17:31:22Z entry); USER_DIALOGUE_LOG.md:1652; TRACKLIST_20260923.md:664 (16:43:44Z entry); USER_DIALOGUE_LOG.md:1624

Throughput. Figure 2 compares the two systems on the same server. Figure 3 shows the best results on box1. On box1 the four-card drafter beats the H200 drafting for itself on the same depth-15 tree by 14%, and at depth 12 on mt_bench by 20%. On talon2 on 2026-11-04 the GPU-only baseline is faster, by 12.5% (Llama) and 7.9% (Qwen); §7 explains why. Accept lengths are within 0.1 of the GPU's on talon2.

Llama · H200 + 4 V80: 723.09 tok/sLlama · H200 + 4 V80723.09 tok/sLlama · H200 GPU-only EAGLE: 826.58 tok/sLlama · H200 GPU-only EAGLE826.58 tok/sQwen · H200 + 4 V80: 236.17 tok/sQwen · H200 + 4 V80236.17 tok/sQwen · H200 GPU-only EAGLE: 256.58 tok/sQwen · H200 GPU-only EAGLE256.58 tok/s
H200 + 4 V80 (this work)H200 GPU-only EAGLE
Figure 2. End-to-end throughput on the same server (talon2, 2026-11-04), humaneval 80 × 256, tree width 2 depth 7, batch 1. Accept length: Llama 5.88 vs 5.97; Qwen 7.14 vs 7.13. Llama 2 rounds per arm, Qwen 1 round. GPU-only is faster on this host today (§7).Source: TALON2_POWER_ECON_20261104.md (commit 1b23c931, md5 0e9214f6…) · FUNDRAISING_POWER_PRICING_20261104/talon2_power_econ_analysis_20261104.jsonl
live demo, one request (11-04): 1,047.12 tok/slive demo, one request (11-04)1,047.12 tok/shumaneval 20×512, d15 (10-20): 846.20 tok/shumaneval 20×512, d15 (10-20)846.20 tok/sGPU self-draft, same tree (10-20): 741.27 tok/sGPU self-draft, same tree (10-20)741.27 tok/shumaneval 80×256 record (10-23): 839.65 tok/shumaneval 80×256 record (10-23)839.65 tok/starget model alone, 20×512 (10-20): 301.41 tok/starget model alone, 20×512 (10-20)301.41 tok/s
H200 + 4 V80H200 drafting for itselftarget alone
Figure 3. Best measured results on box1 (Llama-3.2-3B, one H200). Same tree, humaneval 20 × 512, w2 d15, 59 verify tokens: 846.20 @ 7.29 (4 V80) vs 741.27 @ 7.97 (H200 drafting for itself). Record humaneval 80 × 256, w2 d7: 839.65 @ 5.88. Live demo: one request, 1,047.12 tok/s @ 7.111.Source: FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:868; EXP_K1_V59_HUMANEVAL_20261020.md:68-72; FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:1156; TRACKLIST_20260923.md:686 (17:31:22Z entry); USER_DIALOGUE_LOG.md:1652; FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:1171

Drafter latency and energy. One drafter forward takes 0.177 ms on the four cards against 0.37 ms on the H200, and about 45 mJ against 102 mJ (Figures 4 and 5). Each device is shown alone; the two are never added together.

4 V80: 0.177 ms4 V800.177 msH200: 0.370 msH2000.370 ms
Figure 4. Latency of one drafter forward, each device alone. box1, 2026-10-19, mt_bench 50 × 32, w2 d7. The H200 value is a depth-slope fit over its CUDA-event profile.Source: FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:379 (GPU_VS_FPGA_DRAFTER_20261019.md)
4 V80: 45 mJ4 V8045 mJH200: 102 mJH200102 mJ
Figure 5. Energy of one drafter forward, each device alone (never summed): forward time × the device's run-mean power on talon2 2026-11-04 (4 V80 255.03 W in the FPGA run; H200 276.67 W in the GPU-only run). Derived from measured values; the power is the run mean, not draft-window power.Source: FOURCARD_DESIGN_20261016/4C1_DELIVERY_TRACKER.md:379 (GPU_VS_FPGA_DRAFTER_20261019.md); TALON2_POWER_ECON_20261104.md (commit 1b23c931, md5 0e9214f6…) · FUNDRAISING_POWER_PRICING_20261104/talon2_power_econ_analysis_20261104.jsonl

Duty cycle. Per steady decode step the cards are busy at most 47.1% (Llama) and 14.9% (Qwen) of the time (Figure 6). The rest of the step is the H200 verifying.

Llama-3.2-3BLlama-3.2-3B · busy ≤: 47.1%busy ≤ 47.1%Llama-3.2-3B · idle ≥: 52.9%idle ≥ 52.9%Qwen3.8-27BQwen3.8-27B · busy ≤: 14.9%busy ≤ 14.9%Qwen3.8-27B · idle ≥: 85.1%idle ≥ 85.1%
Figure 6. FPGA duty cycle per steady decode step, as a share of the step (talon2, 2026-11-04, batch 1): Llama 3.81 of 8.10 ms, Qwen 4.00 of 26.94 ms. Busy = card window (from the early tree send to the last card join, including host transfer), so it is an upper bound: 47.1% (Llama) and 14.9% (Qwen) busy. H200 verify takes 68.8% / 75.0% of the step.Source: TALON2_POWER_ECON_20261104.md:114-115

H200 power when verify-only. With the drafting moved to the cards the H200 draws 227.0 W instead of 276.7 W (Llama), and 386.6 W instead of 433.3 W (Qwen) (Figure 7).

Llama · H200 verify-only: 227.0 WLlama · H200 verify-only227.0 WLlama · H200 drafting itself: 276.7 WLlama · H200 drafting itself276.7 WQwen · H200 verify-only: 386.6 WQwen · H200 verify-only386.6 WQwen · H200 drafting itself: 433.3 WQwen · H200 drafting itself433.3 W
H200 in the 4-V80 setupH200 GPU-only
Figure 7. H200 power while serving (talon2, 2026-11-04, nvidia-smi run means). With the cards drafting the H200 only verifies and draws 18% (Llama) and 11% (Qwen) less.Source: TALON2_POWER_ECON_20261104.md (commit 1b23c931, md5 0e9214f6…) · FUNDRAISING_POWER_PRICING_20261104/talon2_power_econ_analysis_20261104.jsonl

Correctness. The target verifies every token, so the output text is the target's. Draft trees on the 2-bit cards are compared level by level with the recorded image: 9/9 identical for Llama and for Qwen. The Qwen texts match the record on all 80 prompts measured (TALON2_POWER_ECON_20261104.md:42; SESSION_CONTEXT_BACKUP_20261104.md:91). Against a 16-bit reference drafter, the first tree level matches 8/8 and the second 6/8, which is the expected effect of 2-bit weights.

6 Cost and energy

6.1 Measured

Table 3 gives the measured power by device and Figure 8 the whole-server energy per token. The 4-V80 setup draws 255.0 W in its cards while serving and 251.1 W when idle: almost all of it is static. Per token the H200 itself uses less energy with the cards drafting: 0.314 vs 0.335 J (Llama) and 1.637 vs 1.689 J (Qwen). The whole server uses more: 1.535 vs 1.293 J (Llama).

Table 3. Measured power by device (talon2, 2026-11-04, 1 Hz run means, W). The 3 other V80s in the chassis are not part of the setup.
arm4 V803 other V80H200CPU packagechassis DCMI
idle251.1119.980.4132.9751.9
Llama · H200 + 4 V80255.0119.4227.0289.61,110.1
Llama · GPU-only251.9 (idle)119.5276.7202.51,068.6
Qwen · H200 + 4 V80252.4119.8386.6261.51,295.2
Qwen · GPU-onlynot readnot read433.3204.01,236.9 (pre-flash)
Source: TALON2_POWER_ECON_20261104.md (commit 1b23c931, md5 0e9214f6…) · FUNDRAISING_POWER_PRICING_20261104/talon2_power_econ_analysis_20261104.jsonl; TALON2_POWER_ECON_20261104.md:81, :86-88
Llama · H200 + 4 V80: 1.535 JLlama · H200 + 4 V801.535 JLlama · GPU-only: 1.293 JLlama · GPU-only1.293 JQwen · H200 + 4 V80: 5.484 JQwen · H200 + 4 V805.484 JQwen · GPU-only: 4.821 JQwen · GPU-only4.821 J
Figure 8. Whole-server energy per token (IPMI DCMI input power ÷ tok/s, talon2 2026-11-04). The GPU-only arms include the 7 idle V80s in the same chassis (~371 W). Qwen GPU-only uses the pre-flash window (4.82 J).Source: TALON2_POWER_ECON_20261104.md (commit 1b23c931, md5 0e9214f6…) · FUNDRAISING_POWER_PRICING_20261104/talon2_power_econ_analysis_20261104.jsonl; TALON2_POWER_ECON_20261104.md:81, :86-88

6.2 Projections projection

Everything in this subsection is calculated, not measured (formulas in the Calculations and hypothesis bar). Table 4 and Figure 9 price four configurations at market prices over 3 years. The H200 is $34,500 and the V80 $9,495. The ASIC is $5,000 per chip, the user's projection. The V80 and ASIC rows use today's measured FPGA-path speed and setup power; ASIC power is assumed equal to the V80 setup, because no ASIC silicon exists yet. At today's speed every FPGA or ASIC configuration buys fewer tokens per dollar than the GPU-only H200. One ASIC ties GPU-only at 962 tok/s on Llama, against the measured 723.09 (talon2) and 839.65 (box1 record). It ties at 298 tok/s on Qwen, against 236.17 measured. The duty cycle suggests one card set could draft for about 2 (Llama) or 6 (Qwen) H200 nodes if several streams time-shared it. That time-sharing is not implemented.

Table 4. PROJECTION. Cost of each configuration over 3 years at market prices (formulas in §6.2).
configurationcapex $3-y energy $$ per tok/sM tokens per $tok/s to tie GPU-only
Llama · GPU-only (1 H200)34,5002,57241.742.109—
Llama · H200 + 4 V8072,4803,655100.240.8991,698
Llama · H200 + 4 ASIC (projection)54,5003,65575.371.1761,297
Llama · H200 + 1 ASIC (projection)39,5003,65554.631.585962
Qwen · GPU-only (1 H200)34,5003,195134.460.644—
Qwen · H200 + 4 V8072,4804,337306.900.291523
Qwen · H200 + 4 ASIC (projection)54,5004,337230.770.380400
Qwen · H200 + 1 ASIC (projection)39,5004,337167.250.510298
Source: FUNDRAISING_POWER_PRICING_20261104/econ.py → econ_out.json; FUNDRAISING_POWER_PRICING_20261104/SOURCES.md (gpupoet.com, jarvislabs.ai, amcompute.com, AMD, EIA Table 5.3; accessed 2026-11-04)
Llama · GPU-only (1 H200): 2.109 MLlama · GPU-only (1 H200)2.109 MLlama · H200 + 4 V80: 0.899 M (projection)Llama · H200 + 4 V800.899 MLlama · H200 + 4 ASIC: 1.176 M (projection)Llama · H200 + 4 ASIC1.176 MLlama · H200 + 1 ASIC: 1.585 M (projection)Llama · H200 + 1 ASIC1.585 MQwen · GPU-only (1 H200): 0.644 MQwen · GPU-only (1 H200)0.644 MQwen · H200 + 4 V80: 0.291 M (projection)Qwen · H200 + 4 V800.291 MQwen · H200 + 4 ASIC: 0.380 M (projection)Qwen · H200 + 4 ASIC0.380 MQwen · H200 + 1 ASIC: 0.510 M (projection)Qwen · H200 + 1 ASIC0.510 M
GPU-only H200H200 + 4 V80H200 + ASIC (projection)
Figure 9. PROJECTION. Million tokens per dollar over 3 years (capex + energy at 14.04 ¢/kWh, 100% duty, batch 1). H200 $34,500, V80 $9,495, ASIC $5,000 (user projection). The V80 and ASIC rows use the measured FPGA-path tok/s and setup power; ASIC power is assumed equal to the V80 setup (no ASIC silicon yet). Hatched = calculated.Source: FUNDRAISING_POWER_PRICING_20261104/econ.py → econ_out.json; FUNDRAISING_POWER_PRICING_20261104/SOURCES.md (gpupoet.com, jarvislabs.ai, amcompute.com, AMD, EIA Table 5.3; accessed 2026-11-04)

7 Limitations

  • GPU-only is faster end to end on talon2 today: 826.58 vs 723.09 tok/s (Llama) and 256.58 vs 236.17 (Qwen), same server, same prompts. The cards draft faster, but each step carries about 1.4 ms of serial host work out of a 6.25–6.46 ms step (SEVENCARD_BOARD_20261026/T2_HOST_TRAFFIC.md:17 (md5 8b9e3f84…)) measured. The host CPU draws +87.1 W more (289.6 vs 202.5 W). The four V80s draw ~251 W even when idle.
  • Serial steps. One request at a time: draft and verify alternate, so the cards wait on the H200 for most of each step (§5).
  • Measurement scope. Llama talon2 arms have 2 rounds, Qwen 1. The H200 draft-time split on talon2 is not yet measured; per-forward energies use run-mean power, not draft-window power. The Qwen GPU-only V80 power was not read.
  • 2-bit drafter. Trees match the recorded image exactly but not a 16-bit drafter beyond the first level; the target verifies every token, so the text is the target's.
  • Roadmap, not results. Each item lists its measured or modelled effect. Host path: one endpoint and fewer serial steps, modelled +12–16%. Export fix: unit gate measured 6,230 → 2,767 bridge cycles per item; full-drafter identity green. Clock-fix image: timing closed at WNS +0.080 ns, which enables the W4 speed image (child level −9.2% measured, Qwen). KV on card: modelled ≈0.17 ms/step. Batched verify / two streams: 1.55× aggregate measured on the 2-card pair, no gain yet on four cards. Sources: SEVENCARD_BOARD_20261026/T2_HOST_TRAFFIC.md:257, :261; TRACKLIST_20260923.md:613, :636 (EXPBURST unit gate; md15 identity GREEN); TRACKLIST_20260923.md:647 (0xE4B7133F FINAL_TRUE WNS +0.080 / WHS +0.011); TRACKLIST_20260923.md:585 (0xE4B7133B, child level −9.2% Qwen vs hold); QWEN27B_DRAFTER_20261020/NSTREAM_DRIVER_20261023.md:288; 4C1_DELIVERY_TRACKER.md:1156 (4-card N-stream: no gain).

8 Related work

Speculative decoding [1][2] and tree verification [7] are the basis of this work; EAGLE [3][4][5] and Medusa [6] are the leading drafter designs, all run on the same GPU as the target. FPGA LLM accelerators such as DFX [9] and FlightLLM [10] run the whole model on FPGAs; we instead keep the target on the GPU and move only the latency-bound drafter, with weights fully on chip. Low-bit weight quantisation follows GPTQ [8].

9 Conclusion

Holding a 2-bit drafter in FPGA on-chip memory makes each drafter forward 2.1× faster and 2.3× lower in energy than on the H200. It also lets the H200 verify at 18% lower power. On box1 it beat the GPU drafting for itself on the same tree. On talon2 today the host path and the cards' static power cost more than the drafting saves. The cards are idle at least half of every step, and that idle time is the headroom for the host fixes, for sharing one card set across GPU nodes and for a drafter ASIC.

References

  1. Y. Leviathan, M. Kalman, Y. Matias. Fast Inference from Transformers via Speculative Decoding. ICML 2023.
  2. C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, J. Jumper. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv:2302.01318, 2023.
  3. Y. Li, F. Wei, C. Zhang, H. Zhang. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. ICML 2024.
  4. Y. Li, F. Wei, C. Zhang, H. Zhang. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. EMNLP 2024.
  5. Y. Li, F. Wei, C. Zhang, H. Zhang. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. arXiv:2503.01840, 2025.
  6. T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, T. Dao. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. ICML 2024.
  7. X. Miao et al. SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification. ASPLOS 2024.
  8. E. Frantar, S. Ashkboos, T. Hoefler, D. Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023.
  9. S. Hong et al. DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation. MICRO 2022.
  10. S. Zeng et al. FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs. FPGA 2024.
  11. AMD. Alveo V80 Compute Accelerator Data Sheet (DS1013).

Repository paths are relative to reports/results/ unless they start with a top-level name. Evidence for this version: reports/results/FUNDRAISING_POWER_PRICING_20261104/ (SOURCES.md, econ.py, build_v6.py). Earlier versions of this page stay on the site at /v/1/ to /v/5/.