Eagle Research
Research paper · 2026-11-13
Heterogeneous speculative decoding

A Six-FPGA Speculative-Decoding Drafter for Llama-3.2-3B

933.14 tokens per second single-stream on HumanEval at an accept length of 6.91, faster than GPU-only EAGLE on all five datasets measured on the same H200

EAGLE4 FPGA team · 2026-11-13
933.14tok/s
6 V80 + H200, HumanEval, depth 11, accept 6.91, measured
838.81tok/s
best GPU-only EAGLE (depth 9, accept 6.63), same H200, same day, measured
305.39tok/s
plain autoregressive decoding, same H200, measured
44.2mJ
energy per drafter forward, 6 V80, vs 97.6 mJ on the H200 (time × run-mean power, per device; derived [C3])

Abstract

Speculative decoding hides the latency of a target model behind a cheap drafter, but at batch 1 and high accept lengths the drafter's sequential tree levels become a large part of every step. We move the drafter of an EAGLE-style tree speculator for Llama-3.2-3B onto six AMD Alveo V80 FPGAs. Four attention cards run 4-way tensor parallel and two cards run the MLP and the 128K-token vocabulary, linked card-to-card by QSFP. A host kit hands each draft tree to an unmodified SGLang verifier on one NVIDIA H200. The draft model's 2-bit weights stay resident in on-chip memory, and the cards draw a width-2, depth-11 tree per verify step at 66 MHz. On HumanEval the system measures 933.1 tok/s at an accept length of 6.91, against 838.8 tok/s for the best GPU-only EAGLE configuration on the same H200 on the same day (1.11×) and 305.4 tok/s for plain autoregressive decoding (3.06×) [C1, C2]. It is faster than GPU-only EAGLE on all 5 datasets measured. A measurement-calibrated model of the loop between the two card roles guided four techniques: owner-gather of the exact vocabulary rescore across the two vocabulary cards (+1.5 %), two cuts in the host path around each tree (+2.9 % and +0.24 %), margin re-routes that close timing with +0.041 ns of slack, and a multiply-free 2-bit weight path. Across the five datasets the geometric-mean speed-up over GPU-only EAGLE is 1.138× (1.100–1.188×), and over plain decoding 2.08× [C1, C2]. Per drafter forward, the six V80s spend 44.2 mJ against 97.6 mJ on the H200, 2.21× less (time per forward × device run-mean power, per device, never summed; the H200 run-mean includes verification) [C3]. That advantage does not carry to the whole system: per drafting session the two are about level, and the FPGA system draws 566 W against 264 W, so it spends more energy per generated token [C3].

Calculations and assumptions

Every ratio, projection, model and energy or cost figure in this paper lives in this box, numbered C1 to C12. The tables and charts below carry only measured values; where the text needs a derived number it points here, like [C1].

  1. C1Speed-up over the best same-day GPU-only EAGLE (tok/s ratio, FPGA system / faster of GPU-only depth 9 and depth 11): HumanEval 933.1 / 838.8 = 1.113×; MT-Bench 557.8 / 487.6 = 1.144×; GSM8K 877.7 / 797.6 = 1.100×; Alpaca 530.5 / 462.5 = 1.147×; Natural Questions 430.4 / 362.3 = 1.188×. Geometric mean over the 5 datasets: 1.138×. Both arms use the same H200, SGLang build and prompts on the same boot.
  2. C2Speed-up over plain autoregressive decoding on the same H200: FPGA system HumanEval 3.06×; MT-Bench 1.82×; GSM8K 2.87×; Alpaca 1.73×; Natural Questions 1.40× (geometric mean 2.08×). GPU-only EAGLE over AR: HumanEval 2.75×; MT-Bench 1.59×; GSM8K 2.61×; Alpaca 1.51×; Natural Questions 1.18× (geometric mean 1.83×).
  3. C3(a) Energy per drafter forward, time per forward × device run-mean power, per device, never summed (white paper 11-04 method). FPGA: 0.137 ms per card item (census md11 w2 on the benchmark boot: 2.878 ms / 21 items) × 6-card run-mean W. GPU: 0.37 ms per forward (slope of GPU-only draft time over depth 2–7, H200, 2026-10-19) × H200 run-mean W of the GPU-only runs. HumanEval 0.137 × 323.5 = 44.3 mJ vs 0.37 × 262.9 = 97.3 mJ (d9), 97.6 mJ (d11): 2.20×; MT-Bench 0.137 × 322.6 = 44.2 mJ vs 0.37 × 264.3 = 97.8 mJ (d9), 96.7 mJ (d11): 2.21×; GSM8K 0.137 × 324.4 = 44.4 mJ vs 0.37 × 262.7 = 97.2 mJ (d9), 95.6 mJ (d11): 2.19×; Alpaca 0.137 × 321.5 = 44.0 mJ vs 0.37 × 263.4 = 97.4 mJ (d9), 96.6 mJ (d11): 2.21×; Natural Questions 0.137 × 321.2 = 44.0 mJ vs 0.37 × 265.6 = 98.3 mJ (d9), 97.8 mJ (d11): 2.23×. 5-dataset mean 44.2 vs 97.6 mJ = 2.21× less on the FPGA. The H200 run-mean includes verify, prefill and host gaps. With the 0.333 ms GPU slope measured on 2026-11-10 (humaneval) instead, GPU d9 is 87.5 mJ (1.97×).
    (b) Context, per drafting session (one step), humaneval at matched depth 11 (same method): FPGA card window 3.573 ms × 323.5 W = 1.156 J (tree only, doorbell to tree returned: 0.821 J) vs GPU draft + extend 4.307 ms × 263.8 W = 1.136 J (d9, interpolated step: 0.958 J). The FPGA session costs 1.02× the GPU d11 session; there is no FPGA energy advantage per session. Per accepted token: FPGA 167.3 mJ (card window) or 118.8 mJ (tree only) vs GPU d9 144.4 mJ and d11 161.1 mJ. With the cards charged for the whole wall step on every dataset, the FPGA spends 1.84–2.43× the GPU’s energy per accepted token. Above idle, the six cards add 17–38 mJ per step; the H200 adds 0.65–0.78 J per step above its 83.6 W idle. Source: DRAFT_SESSION_ENERGY_20261113 (offline recompute; the FPGA card window is assembled from p50s of separate segments of a 2026-11-12 profile; the GPU step times are from 2026-11-10, d9 interpolated).
    (c) System energy per token (J/token = mean power in the timed window / tok/s; power measured, ratio derived). FPGA system = H200 + 6 drafter cards; GPU-only and AR = H200 only. FPGA / GPU-only depth 9: HumanEval 0.600 / 0.314 = 1.91×; MT-Bench 1.018 / 0.542 = 1.88×; GSM8K 0.638 / 0.329 = 1.94×; Alpaca 1.070 / 0.570 = 1.88×; Natural Questions 1.335 / 0.733 = 1.82×. The FPGA system uses 1.82–1.94× the energy per token of GPU-only EAGLE. If the six cards are counted while idle-but-powered in the GPU-only runs too (a box where they stay on either way), the ratio becomes 0.83–0.87×. That accounting charges the GPU-only arm for cards it does not use, so it is a sunk-cost view of a box that already holds the cards, not an efficiency result; we do not claim one. Against plain AR: 0.54× (HumanEval) to 1.23× (Natural Questions). Power excludes the host CPU, memory, fans and the unused seventh card.
  4. C4Nameplate (TDP) bound, not a measurement. FPGA system: 6 × 190 W + 600 W = 1,740 W; GPU-only: 600 W. HumanEval tok/s per kW: FPGA 536, GPU-only 1,398, so on this basis GPU-only is 2.6× more power-efficient. The cards rarely draw their TDP, so C3 (measured) is the better number.
  5. C5Capital cost, list or estimated prices. 6 × $9,495 (V80 list) + $31–35k (H200 NVL estimate) = $88–92k; GPU-only $31–35k. HumanEval tok/s per $1k: FPGA 10.1–10.6; GPU-only 24.0–27.1. GPU-only is about 2.4× more cost-efficient. Server chassis, host CPU and the idle seventh card are excluded from both.
  6. C6Critical-loop model (Section 4.1). Clock 66 MHz = 15.15 ns. With a = 7,382 ATT cycles per item, m = 6,323 MLPV cycles per item and H = both hand-offs, each tree item costs max(a, (a + m + H) / 2). One free parameter, H ≈ 4,100 cycles, fits three measured depth-7 tree times (2,181 vs 2,160 µs; 2,116 vs 2,119; 1,974 vs 1,997). Since (a + m + H)/2 ≈ 8,900 > a, the loop sets the pace, and a 13-item tree moves by about 8.0 × Δa, 7.0 × Δm or 6.5 × ΔH cycles. For G8_OWN the model took G8 from 1,915 to about 835 cycles per item (Δm ≈ −1,080), predicted −115 µs per depth-7 tree and +1.2 to +2.1 % at depth 11, using a measured mapping of 0.8–1.4 µs of serving step per µs of modelled tree. Measured: +1.52 % (889.26 → 902.77). The hand-off H is fitted, not measured; a direct measurement on a four-card set gave 1,430 cycles.
  7. C7HX_O7 projection. From read counts (100 + 2 bulk reads → 39 + 1) and a measured unit cost of 1.75–2.15 µs per posted write + read on the stock register path, the end half was projected to shrink by 150–230 µs per step, i.e. +2.0 to +3.0 % at depth 11. Measured: −220 µs and +2.94 %.
  8. C8Value of the accept gap. 63 extra steps of 2,839 = 2.2 %; at an unchanged step time, closing it would add about 2.2 % tok/s. A fit of the 2026-11-12 runs gives T = 5.755 ms per step + 0.230 ms per width-2 level (residuals within 0.4 %), so any lever that buys the 63 steps with more than about 0.16 ms of step time is a net loss.
  9. C9Step remainder (Table 7). 7,447 − 1,130 (start half) − 2,452.6 (card wait) − 64.2 − 142.0 ≈ 3,660 µs per step for GPU verification, SGLang scheduling and the remaining glue. The card tree (2,878 µs in the bench census) is 2,878 / 7,447 ≈ 39 % of the step. The segments come from different runs, so this is an estimate.
  10. C10Accept-gap shares (Figure 7). 33 / 63 = 52 % TREE_STOP; 18 / 63 = 29 % cheap head at KC 64; 12 / 63 = 19 % from 3 text-divergent requests. Accept 2,839 vs 2,776 steps for 19,615 vs 19,560 tokens.
  11. C11Improvement over two days (Figure 6). 890.30 → 932.71 = +4.8 %; 859.06 → 932.71 = +8.6 %. Per change: G8_OWN +1.52 % (vs same-day parent 889.26), HX_O7 +2.94 %, HX_S7 v +0.24 % (each against its own alternating baseline).
  12. C12Consistency checks. tok/s = accept / step time: 6.91 / 7.447 ms = 928 tok/s, matching the 927.65 measured on that run. One ATT item = 7,382 × 15.15 ns = 111.8 µs; one MLPV item (before G8_OWN) = 95.8 µs.

1  Introduction

At batch 1, decoding a 3-billion-parameter model on a data-centre GPU is bound by latency, not arithmetic. Each new token needs one full pass over the weights, and the GPU spends much of that pass waiting on memory and kernel launches. On our NVIDIA H200, plain SGLang decoding of Llama-3.2-3B[32] measures 305.39 tok/s on HumanEval (Section 6).

Speculative decoding[1,2] raises single-stream speed without changing the output distribution. A small drafter proposes several tokens, and the target model checks them all in one forward pass. Tree-shaped drafts[3] and feature-level drafters such as EAGLE[4–6] now push the mean number of tokens accepted per verify step above six on code. At that point the drafter is no longer cheap. An EAGLE drafter runs one forward pass per tree level, so a depth-11 tree costs eleven dependent small forward passes per verify step. On a GPU-only system those passes share the device with the verifier, and each one pays the GPU's fixed per-kernel latency.

This paper moves the drafter onto a different device class. Six AMD Alveo V80 FPGAs[30] hold a 2-bit EAGLE-style draft model entirely in on-chip memory and draft the token tree, while one H200 runs the unmodified SGLang[13] verifier. Batch-1 drafting suits an FPGA. The matrices are small, the 2-bit weights fit on chip, the schedule is static, and a hardware pipeline has no per-kernel launch cost.

Our contributions:

  1. A six-card drafter: four attention cards in 4-way tensor parallelism (TP4) and two MLP/vocabulary cards, linked card-to-card by QSFP. It drafts width-2, depth-11 trees for a stock SGLang verifier on one H200, and every draft weight is resident on chip at 2 bits (Section 3).
  2. A critical-loop model of one tree level across the two card roles, calibrated on silicon counters. We used it to rank speed levers before spending a 10-hour build on any of them (Section 4.1).
  3. Four techniques that took the system from 889.97 to the low 930s of tok/s in two days: owner-gather of the exact vocabulary rescore across the two vocabulary cards, two cuts in the host path around each tree, margin re-routes that close timing with a safety margin, and the 2-bit weight path itself (Section 4).
  4. A same-day, same-GPU evaluation on five datasets against GPU-only EAGLE and plain autoregressive decoding, with measured power and a cost analysis that reports where the FPGA system loses (Sections 6 and 7).

2  Background and Related Work

2.1 Speculative decoding and token trees

Speculative decoding[1,2] drafts k tokens with a cheap model and accepts the longest prefix that the target model would also have produced; with rejection sampling the output distribution is exactly the target's. SpecInfer[3] verifies a whole token tree in one target pass using a tree attention mask, so several alternative continuations are checked at once. Medusa[7] adds extra decoding heads to the target; Hydra[10] makes those heads sequentially dependent; Sequoia[9] optimises the tree shape for the hardware; Lookahead decoding[8] drafts without a separate model.

2.2 EAGLE drafters

EAGLE[4] drafts at the feature level: a single transformer decoder layer consumes the target's hidden states, fused by a linear layer (fc), and predicts the next feature and token. EAGLE-2[5] grows the draft tree dynamically from the drafter's confidence, and EAGLE-3[6] trains on fused low-, mid- and high-level target features. EAGLE-3 reports a mean accept length of 6.74 on HumanEval for Llama-3.1-8B, and 373.25 tok/s at batch 1 in SGLang on one H100[6]. Recent block-diffusion drafters (DFlash[11]) report larger speedups again. A recent study of production serving[12] notes that speculative gains shrink as batch size grows, because verification comes to dominate. We therefore target, and claim only, the batch-1 regime.

Our drafter is EAGLE-style. Per draft forward it runs the phases in Table 2, with hidden size 3,072, 24 query heads and 24 key/value heads of dimension 128 (multi-head, not grouped), MLP width 8,192 and a 128,256-token vocabulary. It ranks the whole vocabulary with a cheap low-rank 2-bit head, keeps a shortlist of KC = 64 candidates per parent, and rescores that shortlist exactly against the fp16 LM head.

2.3 FPGA accelerators for LLM inference

Most FPGA LLM work runs the whole model on the FPGA without speculation. FlightLLM[15] maps LLaMA2-7B onto a Versal VHK158 at about 3.5-bit weights and reports 92.5 tok/s, with 6.0× the energy efficiency of a V100S on an Alveo U280. DFX[16] uses four U280s for GPT-2; the spatial-accelerator study of Chen et al.[17] and LoopLynx[18] target GPT-2-class models; TerEffic[19] runs ternary 370M and 2.7B models. The closest prior art to our split is DFVG[20] (draft on the FPGA, verify on the GPU), whose full text we could not retrieve, and HeteroReason[21], which reports 1.01–1.42× latency speedups for reasoning models with an FPGA drafter. SpecMamba[22] runs speculative decoding for state-space models on Versal parts. We found no published LLM throughput for the Alveo V80.

2.4 Where this work sits

Table 1 places the system among published single-stream numbers. The rows are not like-for-like: they use different model sizes (3B here; 8B to 400B elsewhere), precisions, hardware and dates, and most service numbers are vendor or third-party measurements we did not repeat. Batch-1 decode speed falls roughly with model size, so a 3B number cannot be ranked against an 8B or 70B number. The table is context, not a leaderboard.

SystemModelPer-user tok/sHardwareSource
This workLlama-3.2-3B, 2-bit drafter933.146× Alveo V80 + 1× H200 NVLSection 6
EAGLE-3 in SGLangLlama-3.1-8B373.251× H100[6]
NVIDIA TensorRT-LLM, 1B draftLlama-3.3-70B FP8181.741× H200[28]
NVIDIA DGX B200, EAGLE-3-based draftLlama-4 Maverick (400B MoE) FP8>1,0008× B200[29]
Cerebras InferenceLlama-3.1-8B, 16-bit1,800CS-3 wafer-scale system[25]
Groq LPU (Artificial Analysis, 2024-04)Llama-3-8B877GroqCloud[26]
SambaNova SN40L (Artificial Analysis)Llama-3-8B, 16-bit1,08416-chip SN40L[27]
FlightLLMLLaMA2-7B, ~3.5-bit92.5Versal VHK158 (whole model on FPGA)[15]

Table 1. Published single-stream (per-user) decode speeds. All rows except ours are quoted from the cited source and were not re-measured. Ratios against our number are deliberately not computed: model size dominates batch-1 speed.

3  System Architecture

3.1 Overview

Figure 1 shows the system. The H200 runs SGLang 0.5.16 with the Llama-3.2-3B target and verifies a 20-token tree per step (FlashAttention-3 backend[14], CUDA graphs). After each verify, the GPU also computes the drafter's fc fusion and the drafter's key/value rows for the newly accepted tokens. That is one small batched forward and the only drafter work left on the GPU. A host process ships those rows and the root of the next tree to the cards and rings their doorbells. The six cards then draft the whole tree without any host involvement. The host reads back the tree, and SGLang verifies it.

flowchart TB
  subgraph G["NVIDIA H200 NVL · SGLang 0.5.16"]
    direction LR
    T["Llama-3.2-3B target
verifies a 20-token tree"] -->|accepted tokens| E["drafter fc fusion +
K/V of accepted rows"] end H["Host card process
registers + DMA over PCIe"] subgraph A["4 ATT cards · TP4, 8 heads each"] direction LR A0["ATT r0"] --- A1["ATT r1"] --- A2["ATT r2"] --- A3["ATT r3"] end subgraph M["2 MLPV cards · vocabulary shards"] direction LR M0["MLPV r0"] --- M1["MLPV r1"] end E -->|K/V rows, tree root| H H -->|doorbell, root image| A A -->|per-item hand-off| M M -->|chosen parents, next input| A M -->|finished tree| H H -->|draft tree to verify| T

Figure 1. System overview. Thin lines between cards are QSFP card-to-card links: the four ATT ranks all-reduce the attention output projection over them, and the two MLPV ranks exchange partial sums and rescore results. The host touches the cards only at the start and end of each tree.

3.2 Splitting one draft forward across two card roles

Table 2 lists the phases of one draft forward and where each runs. The four ATT cards each own 8 of the 24 heads. Each computes the input RMSNorm, its quarter of the QKV projection, attention over its own key/value cache, its share of the output projection followed by a 4-rank all-reduce, and a quarter of the gate and up projections. The two MLPV cards finish the MLP and own the vocabulary side. That means the down projection, the final norm, the low-rank cheap LM head over all 128,256 tokens, tree selection, and the exact rescore of the KC = 64 shortlist against an fp16 LM head held in HBM.

PhaseOperationDimensionsCardWeights / data inCycles per item
fcfuse 3 target hidden states9,216 → 3,072GPUGPU memory–
G1RMSNorm (input)3,072ATTBRAM (gamma)456
G4QKV projection, rank quarter6,144 → 3 × 768ATT2-bit, URAM1,421
G5attention, 8 heads per rankhead dim 128ATTK/V cache in BRAM1,913
G6output projection + 4-rank all-reduce768 → 3,072ATT2-bit, URAM712
G7g / G7u / G7pgate, up, SiLU·up (quarter)3,072 → 8,192ATT2-bit, URAM1,397 / 557 / 133
–pipeline I/O + uncountedATT607 + ~186
G9down projection (+ next-level feature)8,192 → 6,144MLPV2-bit, URAM850
G11final RMSNorm3,072MLPVBRAM466
GLM + G12low-rank cheap LM head, full vocabulary3,072 → rank → 128,256MLPV2-bit, URAM226 + 22
TREEtop-KC survivor selectionKC = 64MLPVon chip2,019
G8exact rescore: gather 64 rows + dot64 × 3,072 fp16MLPVfp16 LM head in HBM1,915 (before owner-gather)
–ingest, level exchangeMLPV538 + ~287

Table 2. One draft forward ("item") and where it runs. Cycle counts are measured on silicon at 66 MHz by per-phase hardware counters on the ATT image 0xE4B713BC and the MLPV image 0xE4B71360 (mean of 20 census trees, depth 7, width 2, KC 64). They sum to 7,382 cycles per ATT item and 6,323 per MLPV item. The G8 row is the value before owner-gather (Section 4.2).

3.3 The tree schedule

A width-2 tree keeps two children per level, so a depth-11 tree is up to 21 items after the root. Two schedule features keep the cards busy. With EARLY_A, the MLPV sends the next level's input vector back early. With EAGER, the ATT starts a child before its level is formally committed. Even so, child c at level L+1 must wait for the MLPV result of child c at level L, so the two roles form one loop. A mass rule (TREE_STOP) ends a tree early when the drafter is unsure: it stops when the top-2 probability mass falls below 2−2.5. In the record runs, 1,400 of 2,534 trees stopped early, at a mean served depth of 8.45 of 11 levels.

3.4 Floorplan of an attention card

Each V80 is a three-die (three-SLR) Versal device. The die a block sits on is forced by memory, and that dictates where the inter-die wires (SLLs) go. The 2-bit GEMV array has one URAM per lane, so it is split into lane groups across dies. The attention key/value cache fills the BRAM of one die, so attention and the full-width "hub" share SLR0, and array groups 0 and 2 sit on SLR2 (Figure 2). Every GEMV pass broadcasts the activation from SLR0 to the other dies and returns per-lane results. The output projection also leaves through the QSFP transceivers in SLR1 to the other ranks.

SLR22-bit GEMV array, lane groups 0 and 2(one URAM per lane holds the 2-bit codes)SLICE 78.0 %DSP 30.1 %URAM 54.7 %SLR12-bit GEMV array, lane group 1QSFP transceivers: 4-rank all-reduce linkSLICE 86.3 %DSP 24.3 %URAM 27.4 %SLR0Attention: 8 heads, K/V cache in BRAMHub: result landing, gate/up latches, SiLU, all-reduce adapterSLICE 92.3 %DSP 85.4 %BRAM 54.8 %SLL SLR1-SLR2: 1,885 of 18,870 (10.0 %)SLL SLR0-SLR1: 9,646 of 18,870 (51.1 %)activation broadcastper-lane results returnQSFP toother ranks

Figure 2. Floorplan of the attention card. Placed utilisation per die is measured on the shipped ATT image 0xE4B713BC. The binding numbers are DSP in SLR0 (582 free) and SLICE in SLR0 (92.3 %). Arrows show the per-pass activation broadcast and the result return.

3.5 Implementation

Table 3 gives post-place utilisation and post-route timing for the two shipped images. Both run at a single 66 MHz drafter clock. Neither is limited by URAM. On the ATT the tightest numbers are DSP (85.4 %) and SLICE (92.3 %) in SLR0. On the MLPV they are SLICE in SLR1 and SLR2 (93.4 % and 94.4 %), DSP in SLR1 (84.5 %) and the SLR0–SLR1 inter-die wires (70.8 % of 18,870).

ImageLUTSLICE (worst die)DSP58BRAMURAMSLL 0–1 / 1–2Setup / hold slack (ns)
ATT 0xE4B713BC39.73 %85.65 % (92.31)48.55 %47.66 %38.65 %9,646 / 1,885+0.040 / +0.010
MLPV 0xE4B7136048.11 %92.34 % (94.04)69.17 %27.68 %26.60 %13,485 / 10,722+0.002 / +0.010
MLPV 0xE4B71361 (G8_OWN)45.55 %89.19 % (94.42)69.17 %27.68 %26.60 %13,360 / 12,331+0.000177 / +0.010
MLPV M1361 (margin re-route of 0xE4B71361)same netlist and placement as 0xE4B71361+0.041 / +0.016

Table 3. Post-place utilisation (percent of the V80) and true post-route slack at 66 MHz. The deployed set is four ATT 0xE4B713BC and two MLPV M1361. All slack values are positive. The device has 2.57M LUT, 10,848 DSP58, 3,741 BRAM36, 1,925 URAM and 18,870 SLLs per die crossing.

4  Key Techniques

4.1 A critical-loop model to rank levers

A full V80 image takes a median of 9.4 hours to build, so every lever must be ranked before it is built. Neither card is saturated. In a depth-7 census tree the ATT is busy 73 % of the in-tree time and the MLPV 63 %. What sets the pace is the loop formed by an ATT item, the hand-off, the MLPV item and the hand-off back. Figure 3 shows the two items side by side. Because child c of one level and child c' of the next interleave, each tree item costs about half the loop, not a whole one. A cycle cut anywhere in the loop therefore returns about half of itself per item [C6]. With one fitted parameter (the hand-off), the model reproduces three silicon tree times within ±23 µs. It ranked the exact-rescore gather on the MLPV first among the bit-exact levers, so we built that next.

Figure 3. Measured cycles per item for each phase, ATT image 0xE4B713BC and MLPV image 0xE4B71360, at 66 MHz (15.15 ns per cycle). The MLPV's TREE and G8 dominate its item. G5 attention and G4 QKV dominate the ATT item.

Show the data as a table

4.2 G8_OWN: owner-gather of the exact rescore

Before this change, both MLPV cards gathered the same 64 exact LM-head rows from HBM and dotted them redundantly. Both cards already hold an identical, globally ordered shortlist. Under G8_OWN, rank r gathers only the shortlist slots whose token id has parity r. After selection, the two cards swap the resulting dot scores over the card-to-card link they already share for the down-projection exchange. Slot positions are frozen and identical on both cards, so no token id travels, only {owned flag, score}. Each card lands its peer's scores on the non-owned slots and finishes as before. Both cards still compute every score with the same row, the same dot unit and the same normalisation, so the result is bit-exact. The change added 19,781 LUT and 2.1K FF, with no DSP, BRAM, URAM or new port. Its simulation gate checks every slot of every item on both ranks, with failure-injection arms that must fire.

On silicon, the image 0xE4B71361 ran three serves at 904.97, 902.40 and 900.93 tok/s (mean 902.77), against a same-day 889.26 for the parent set. Accept length (6.91), the 2,839 verify steps and all 80 output texts were identical. The slowest G8_OWN run beat the fastest parent run. The per-level card census fell from 280.0 to 269 µs. The model had predicted +1.2 to +2.1 % [C6].

4.3 HX_O7 and HX_S7: cutting the host path around each tree

More than half of a verify step is spent outside the card tree (Table 7). We split that time into an end half (cards done to tree handed to SGLang) and a start half (verify result to root doorbell) and instrumented both on silicon.

HX_O7 (end half). Counted on the record kit, the host made 100 register reads and 2 bulk tree reads between "cards done" and "tree returned". HX_O7 cuts this to 39 reads and 1 bulk read in three parts. Part l uses lean poll passes, part f exits on the done check without a further full pass, and part c runs the cross-checks after the tree is published to the GPU but before the reply. It issues no new card operation and changes no error path. Over three alternating A/B pairs it measured +2.94 % (901.14 → 927.65 tok/s), every B row above every A row, with identical texts. The step shrank by 220 µs, against a projected 150–230 µs [C7]. An instrumented pair measured the end half directly: wait entry to tree returned fell from 2,720.1 to 2,516.8 µs (p50).

HX_S7 (start half). The first on-silicon profile of the start half measured 1,130 µs p50 (Figure 4). Its largest parts are the scheduler-side work (620 µs, of which 201 µs is the GPU's drafter key/value extend), the tree request queued behind the key/value load (180 µs) and the staging-to-doorbell path (306 µs). Of two host levers, part v moves the 37 µs structural validation of the tree to after the publish, under the GPU verify. It measured +0.24 % over 4 of 4 positive pairs (930.45 → 932.71), with a best row of 933.28 tok/s. Part y, deferring a GPU synchronisation, measured −0.22 %. The trailer wait it would hide is only 5.9 µs, so we rejected it.

Figure 4. The start half of a verify step on the record path, from verify result to root doorbell: 1,130 µs p50, measured on silicon by stamping both host processes on one clock. Bars are the segments in order.

Show the data as a table
ArmChangetok/s (mean)RowsTexts vs recordVerdict
Arecord kit (G8_OWN image)901.143identicalbaseline
llean poll passes only901.671identicalno measurable effect
lf+ fast exit915.061identicalabout +1.2 %
lfc = HX_O7+ late checks after publish927.653identicaladopted
A′HX_O7 record (next window)930.455identicalbaseline
v = HX_S7validate after publish932.714identicaladopted
vy+ deferred GPU sync930.606identicalrejected

Table 4. Host-path A/B results on silicon, HumanEval 80 × 256, depth 11, width 2. Arms alternated with their neighbours on the same boot. Run-to-run noise within an arm is about ±0.5 %.

4.4 Timing closure with margin re-routes

Our rule is that an image ships only with positive setup and hold slack. A negative-slack image can hang the card or silently corrupt arithmetic. Over the week of 5–12 November we ran 213 distinct builds and closure arms (122 full builds). Of those, 91 closed. MLPV builds without a margin step consistently closed at +0.000 to +0.016 ns, which is positive but below our +0.030 ns release bar.

The margin re-route recipe starts from the routed checkpoint. It adds a clock uncertainty of 0.040 ns setup and 0.012 ns hold on all 49 clocks, runs physical optimisation, re-routes under that margin, and then removes the margin and re-times. This keeps the netlist and placement and buys slack that survives the true timing. On M1361 it took the G8_OWN image from +0.000 / +0.010 to +0.041 / +0.016 ns (Table 5). Across the week, 12 of 12 margin or closure arms on bases that had already routed with non-negative slack improved true slack, 6 of them by 0.030 ns or more. No arm rescued a base at −0.2 ns or worse (0 of 4). We re-flashed the drafter with M1361 and kept it: its serves were identical in text and within the record band.

Step (M1361)Margined WNS / WHSFailing endpointsTrue WNS / WHS
routed source, margin off–00.000 / 0.010
margin on (setup 0.040, hold 0.012)−0.040 / −0.0024,701 setup, 5 hold0.000 / 0.010
physical optimisation−0.024 / −0.0022,142 setup, 2 hold0.016 / 0.010
re-entrant route+0.001 / +0.00400.041 / 0.016
final, margin off–0+0.041 / +0.016

Table 5. The margin re-route of the G8_OWN MLPV image, step by step. Slack in ns at 66 MHz.

4.5 2-bit weights, resident on chip

The drafter checkpoint is trained with 2-bit body weights in the style of BitNet[24]. Every body tensor equals its 2-bit code times a scale, bit for bit, and the GPU-only baseline loads the same checkpoint. On the cards the 2-bit codes sit in URAM next to each GEMV lane, and the scales sit in BRAM. Activations stay on chip, so a forward pass issues no off-chip weight traffic. The only HBM read on the draft path is the exact rescore's gather of 64 fp16 LM-head rows.

The ATT array is multiply-free in the LUT-GEMM style[23]. Activations are quantised to 8 bits, each lane cluster shares a table of partial sums, and each lane selects from it with its own 2-bit codes. The zero-point term is common to the whole array and is computed once. This frees DSPs for attention. The ATT image uses 48.55 % of the device's DSP58, most of them (3,402) in SLR0 next to the attention engines.

Numerically, the cards track a GPU emulation of their own arithmetic closely. With TREE_STOP off, the cards and their GPU twin differ by 3 verify steps in total over the 77 requests whose texts agree. Most of the accept gap to GPU-only EAGLE (Section 6.3) comes from two settings we chose for speed, not from the 2-bit arithmetic [C10].

5  Methodology

5.1 Hardware and software

All runs used one server with seven AMD Alveo V80 cards and one NVIDIA H200 NVL. Six V80s drafted: four attention cards running ATT image 0xE4B713BC and two MLPV cards running M1361. The seventh card was not opened by any run. The target, Llama-3.2-3B-Instruct in fp16, ran in SGLang 0.5.16 with CUDA graphs, piecewise CUDA graphs and the FlashAttention-3 draft attention backend, at batch 1 (one running request). The FPGA system and GPU-only EAGLE load the same 2-bit drafter checkpoint.

5.2 Configurations

5.3 Workloads and metrics

Five prompt sets: HumanEval[33] (code), MT-Bench[34] (chat), GSM8K[35] (math), Alpaca[36] (instructions) and Natural Questions[37] (QA; 79 questions, filled to 80 by repetition for every configuration). Each run has 80 requests × up to 256 new tokens, a seed-42 shuffle and greedy decoding (temperature 0). Throughput is generated tokens divided by the wall time of the timed window, one request at a time. Accept length is tokens per verify step.

5.4 Protocol

Each configuration ran two timed rounds per dataset. Inside a dataset, round 1 was a random permutation of the four configurations and round 2 a permutation of the three speculative ones (fixed seed). AR round 2 ran afterwards. Before every run the H200 was idle and cooled to 60 °C or less. The H200 SM clock read 1,785 MHz in every 1 Hz sample of every timed window, with no throttle reason set, and peak GPU temperature was 57–73 °C. The HumanEval FPGA runs reproduced the record's 80 output texts byte for byte.

5.5 Power

H200 power is the driver's power.draw sampled at 1 Hz. Card power is the V80 management firmware's total-power sensor, read for all six drafting cards about every 2 s (4–19 samples per card per window). It was cross-checked against the cards' 1 Hz hardware-monitor reading, and the two sums agree within 1.7 W in every run. Power is averaged over each run's timed window. "System power" counts the H200 plus the six drafting cards for the FPGA system and the H200 alone for GPU-only and AR. The cards' idle draw during GPU-only and AR runs is reported but not counted. Host CPU, memory and fans are not measured.

6  Results

6.1 Five datasets, three systems, one GPU

Figure 5 and Table 6 give the measured throughput. The FPGA system is faster than the best GPU-only configuration on all five datasets, from 877.67 vs 797.59 tok/s on GSM8K to 430.44 vs 362.27 on Natural Questions [C1]. On HumanEval it reaches 933.14 tok/s at an accept length of 6.91. Plain decoding runs at 305–307 tok/s on every set, so the gain over it follows the accept length. It is largest on HumanEval and GSM8K (accept 6.91 and 6.46) and smallest on Natural Questions (accept 2.71) [C2]. The relative lead over GPU-only EAGLE is the other way round: largest on Natural Questions and smallest on GSM8K and HumanEval.

The FPGA system's accept length is below GPU-only depth 11 on all five sets (for example 6.91 vs 7.05 on HumanEval, 3.67 vs 3.85 on MT-Bench). It wins on speed because each draft level is cheaper, not because it accepts more (Section 6.3).

Output equality. Under greedy decoding, configurations that speculate differently (or not at all) do not produce byte-identical texts on every request, and the FPGA system is not special in this. Its texts match plain AR on 71–79 of 80 requests per set, equal to or higher than GPU-only depth 9 against AR (70–77 of 80) on every set. GPU-only depth 9 and depth 11 also differ from each other on 1–8 requests per set. We did not measure the mechanism of these divergences; fp16 verification over different tree shapes is one candidate.

Figure 5. Single-stream throughput per dataset, same H200, same day. "GPU-only EAGLE" is the faster of its depth-9 and depth-11 configurations on each dataset. Bars are means over the timed runs (Table 6).

Show the data as a table
Dataset6 V80 + H200acceptGPU-only d9acceptGPU-only d11acceptAutoregressive
HumanEval933.14
934.48 / 931.81
6.91838.81
837.62 / 840.00
6.63820.30
822.74 / 817.85
7.05305.39
306.18 / 304.61
MT-Bench557.76
559.63 / 555.90
3.67487.61
487.30 / 487.92
3.80454.07
453.59 / 454.56
3.85306.09
306.41 / 305.77
GSM8K877.67
878.55 / 876.80
6.46797.59
797.03 / 798.14
6.30768.33
771.35 / 765.30
6.61306.07
306.28 / 305.86
Alpaca530.46
531.66 / 529.26
3.46462.50
462.70 / 462.29
3.61427.20
428.54 / 425.85
3.62306.24
305.53 / 306.94
Natural Questions430.44
429.39 / 431.49
2.71362.27
363.27 / 361.28
2.81333.56
334.33 / 332.80
2.81306.63
305.47 / 307.79

Table 6. Measured tok/s (mean of n timed runs) and accept length (tokens per verify step) per dataset. Two timed rounds per configuration, alternated in a random order on one boot; the two rounds of every configuration produced identical texts. Bold marks the faster speculative configuration. Accept is not defined for autoregressive decoding (1 token per step).

6.2 Two days of improvement

Figure 6 traces the HumanEval record on the six-card set at a fixed configuration (depth 11, width 2, KC 64, FlashAttention-3, the same 80 prompts). Each step is one image or one host lever, and each was A/B-tested on silicon with texts identical to the record.

Figure 6. HumanEval tok/s on the six-card set after each change (mean of that change's runs; accept 6.91 throughout). B4 and BC are the ATT images 0xE4B713B4 and 0xE4B713BC; hx7 is an earlier host-register lever.

Show the data as a table

6.3 The accept gap

At depth 11 the FPGA system needs 2,839 verify steps for the 80 HumanEval prompts, and GPU-only EAGLE needs 2,776. Accept lengths are 6.91 and 7.05. Both load the same checkpoint. Silicon A/B runs split the 63 extra steps three ways (Figure 7). TREE_STOP accounts for 33 steps: with the rule off, the cards need 2,806. The card's cheap head at KC = 64 accounts for 18: a GPU twin with that head needs 2,794. The remaining 12 come from 3 requests whose texts diverge. Neither setting is free to undo. Turning TREE_STOP off lowered throughput to 876.12 tok/s, and no runtime setting beat the record on tok/s. A width-1 level costs the same card time as a width-2 level (263 vs 266 µs per level at depth 11), so thinning the tree buys no time either.

Figure 7. Extra verify steps of the FPGA system over GPU-only EAGLE at depth 11 (63 in total), by measured cause.

Show the data as a table

6.4 Anatomy of a verify step

Table 7 places the measured segments on the record step. The card tree itself, from doorbell to the last card done, is under half of the step. The rest is GPU verification, SGLang scheduling and host glue [C9]. This is why two host levers (Section 4.3) were worth more than the G8_OWN image.

Segmentµs (p50)Source run
Whole step, mean over the run (21.11 s / 2,839 steps)7,447HX_O7 B arm, 927.65 tok/s
Start half: verify result → root doorbell1,130HX_S7 profile
Card tree, issue → done (bench, depth 11)2,878M1361 census, 100 trees
Wait entry → poll-loop exit (serve, HX_O7)2,452.6HX_O7 instrumented arm
Loop exit → tree returned (HX_O7)64.2same
Tree returned → handler returned (build + publish)142.0same
Per tree level on the cards (census, depth 11, width 2)262M1361 census

Table 7. Measured segments of one verify step on the record configuration. Segments come from different instrumented runs on the same images and are not additive. The derived remainder is [C9].

7  Power and Cost Analysis

7.1 Drafter energy per forward

The drafter-energy metric we state is energy per drafter forward, computed as time per forward × device run-mean power, per device, never summed (white paper 11-04 method). A drafter forward is one pass of the draft model for one tree item. On the FPGA side it is one card item: 0.137 ms, measured by the on-card census at depth 11 and width 2 on the benchmark boot (2.878 ms per 21-item tree). On the GPU side it is 0.37 ms, the slope of GPU-only draft time against tree depth measured on an H200 on 2026-10-19. Each time is multiplied by its own device's run-mean power in the 2026-11-13 benchmark. That is the six drafting V80s' summed sensor power during the FPGA runs, and the H200's driver power during the GPU-only runs. The two devices are never added together.

By this method the six V80s spend 44.2 mJ per drafter forward against 97.6 mJ for the H200, 2.21× less (5-dataset mean; 2.15–2.23× on every dataset at either GPU depth, Table 8). The white paper of 2026-11-04 reported 45 vs 102 mJ with four V80s.

Dataset6 V80 WFPGA mJ / forwardH200 W, GPU-only d9GPU d9 mJ / forwardH200 W, GPU-only d11GPU d11 mJ / forwardGPU d9 / FPGA
HumanEval323.544.3262.997.3263.897.62.20×
MT-Bench322.644.2264.397.8261.496.72.21×
GSM8K324.444.4262.797.2258.495.62.19×
Alpaca321.544.0263.497.4261.196.62.21×
Natural Questions321.244.0265.698.3264.497.82.23×
5-dataset mean322.6544.2263.897.6261.896.92.21×
2026-11-04 (4 V80, depth 7)255.0345.1276.67102.4––2.27×

Table 8. Energy per drafter forward, time per forward × device run-mean power, per device, never summed (white paper 11-04 method). Power columns are measured run means. The mJ columns and ratios are derived: FPGA = 0.137 ms × 6-card W, GPU = 0.37 ms × H200 W [C3].

What this number includes and does not.

7.2 System power

Table 9 gives whole-accelerator power in every timed window. System power counts the H200 plus the six drafting cards for the FPGA configuration, and the H200 alone for GPU-only and plain decoding. Averaged over the five datasets, the FPGA system draws 565.9 W: 243.2 W on the H200 and 322.6 W on the six drafting cards. GPU-only EAGLE at depth 9 draws 263.8 W, and plain decoding 335.4 W. Three measured facts shape the comparison.

On system power, the FPGA system uses more energy per token than GPU-only EAGLE on every dataset, about 1.8–1.9 times as much [C3]. Against plain decoding it uses less energy per token on high-accept sets and more on low-accept sets [C3]. We make no efficiency claim.

DatasetConfigurationH200 W6 cards WSystem Wtok/sJ / token
HumanEval6 V80 + H200236.4323.5559.9933.140.600
GPU-only d9262.9318.1 (idle, not counted)262.9838.810.314
Autoregressive337.2319.3 (idle, not counted)337.2305.391.104
MT-Bench6 V80 + H200244.9322.6567.5557.761.018
GPU-only d9264.4319.2 (idle, not counted)264.4487.610.542
Autoregressive335.7318.9 (idle, not counted)335.7306.091.097
GSM8K6 V80 + H200235.6324.4559.9877.670.638
GPU-only d9262.7319.2 (idle, not counted)262.7797.590.329
Autoregressive335.5319.1 (idle, not counted)335.5306.071.096
Alpaca6 V80 + H200245.9321.5567.4530.461.070
GPU-only d9263.4319.0 (idle, not counted)263.4462.500.570
Autoregressive334.8318.8 (idle, not counted)334.8306.241.093
Natural Questions6 V80 + H200253.4321.2574.6430.441.335
GPU-only d9265.6318.4 (idle, not counted)265.6362.270.733
Autoregressive333.8318.5 (idle, not counted)333.8306.631.089

Table 9. System power: measured mean power over each run's timed window, averaged over the two rounds, and the resulting joules per token. System power counts the six cards only for the FPGA configuration. GPU-only depth 11 is omitted for space (it draws 258–264 W).

7.3 Nameplate power and capital cost

At nameplate ratings the FPGA system is 6 × 190 W of V80 plus 600 W of H200, or 1,740 W, against 600 W for the GPU alone[30,31]. At list or estimated prices, the six V80s ($9,495 each) roughly triple the accelerator cost of an H200 NVL (estimated at $31–35k). On both bases GPU-only EAGLE delivers more tokens per second per watt and per dollar [C4, C5]. Against rack-scale systems (Table 1), the FPGA system's envelope is far smaller: a Cerebras CS-3 is rated at 23 kW and a SambaNova SN40L rack at 10 kW typical[25,27]. But those systems serve larger models, so no per-token comparison is made.

In short, the six-FPGA drafter buys 10–19 % more single-stream speed on the same GPU. Per drafter forward it spends 2.21× less energy than the H200 by the white-paper method. Per drafting session it is about level, and per generated token the whole system uses about twice the energy and about three times the accelerator capital cost. It is worth deploying only where batch-1 latency is the product.

8  Discussion and Limitations

9  Conclusion

Six FPGAs drafting for one H200 deliver 933.1 tok/s single-stream on HumanEval for Llama-3.2-3B. That is faster than the best GPU-only EAGLE configuration on the same GPU on every dataset we measured, by 1.100–1.188×, and 1.40–3.06× plain autoregressive decoding [C1, C2]. The design keeps every draft weight on chip at 2 bits and splits one draft forward across attention and vocabulary cards. It improved by ranking levers with a silicon-calibrated loop model before building them: owner-gather of the exact rescore, cuts in the host path around each tree, and margin re-routes that give every shipped image positive setup and hold slack.

The result is a latency result. Per drafter forward the six V80s spend 2.21× less energy than the H200, but the whole FPGA system uses more energy per generated token and costs more [C3–C5], and more than half of each step is still GPU verification and host glue [C9]. The next steps follow from the measurements. They are a shorter start half (key/value load and staging on the critical path), recovering the 2 % accept gap without adding tree levels, and per-stream resident key/value state so the same cards can serve more than one request.

References

  1. [1]Y. Leviathan, M. Kalman, Y. Matias. Fast Inference from Transformers via Speculative Decoding. ICML 2023. https://arxiv.org/abs/2211.17192
  2. [2]C. Chen et al. Accelerating Large Language Model Decoding with Speculative Sampling. 2023. https://arxiv.org/abs/2302.01318
  3. [3]X. Miao et al. SpecInfer: Accelerating LLM Serving with Tree-based Speculative Inference and Verification. ASPLOS 2024. https://arxiv.org/abs/2305.09781
  4. [4]Y. Li, F. Wei, C. Zhang, H. Zhang. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. ICML 2024. https://arxiv.org/abs/2401.15077
  5. [5]Y. Li et al. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. EMNLP 2024. https://arxiv.org/abs/2406.16858
  6. [6]Y. Li et al. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. 2025 (Tables 1, 3, 4). https://arxiv.org/abs/2503.01840
  7. [7]T. Cai et al. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. ICML 2024. https://arxiv.org/abs/2401.10774
  8. [8]Y. Fu et al. Break the Sequential Dependency of LLM Inference Using Lookahead Decoding. ICML 2024. https://arxiv.org/abs/2402.02057
  9. [9]Z. Chen et al. Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding. NeurIPS 2024. https://arxiv.org/abs/2402.12374
  10. [10]Z. Ankner et al. Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding. 2024. https://arxiv.org/abs/2402.05109
  11. [11]DFlash: block-diffusion drafter for speculative decoding. 2026. https://arxiv.org/abs/2602.06036
  12. [12]Speculative Decoding: Performance or Illusion? MLSys 2026. https://arxiv.org/abs/2601.11580
  13. [13]L. Zheng et al. SGLang: Efficient Execution of Structured Language Model Programs. NeurIPS 2024. https://arxiv.org/abs/2312.07104
  14. [14]J. Shah et al. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. NeurIPS 2024. https://arxiv.org/abs/2407.08608
  15. [15]S. Zeng et al. FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs. FPGA 2024. https://arxiv.org/abs/2401.03868
  16. [16]S. Hong et al. DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation. MICRO 2022. https://arxiv.org/abs/2209.10797
  17. [17]H. Chen et al. Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference. ACM TRETS / FCCM 2024. https://arxiv.org/abs/2312.15159
  18. [18]LoopLynx: A Scalable Dataflow Architecture for Efficient LLM Inference. 2025. https://arxiv.org/abs/2504.09561
  19. [19]TerEffic: Highly Efficient Ternary LLM Inference on FPGA. 2025. https://arxiv.org/abs/2502.16473
  20. [20]DFVG: draft on FPGA, verify on GPU. ACM, doi 10.1145/3779212.3790153 (full text not retrieved). https://dl.acm.org/doi/10.1145/3779212.3790153
  21. [21]HeteroReason: FPGA drafting with GPU verification for reasoning models. 2026. https://arxiv.org/abs/2609.28717
  22. [22]SpecMamba: speculative decoding for state-space models on FPGA. 2025. https://arxiv.org/abs/2509.19873
  23. [23]G. Park et al. LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models. ICLR 2024. https://arxiv.org/abs/2206.09557
  24. [24]H. Wang et al. BitNet: Scaling 1-bit Transformers for Large Language Models. 2023. https://arxiv.org/abs/2310.11453
  25. [25]Cerebras. Introducing Cerebras Inference (2024-08-27); CS-3 power from the 2024-03-13 press release. https://www.cerebras.ai/blog/introducing-cerebras-inference-ai-at-instant-speed
  26. [26]Groq. Llama 3 8B and 70B on the LPU, Artificial Analysis numbers (2024-04). https://groq.com/12-hours-later-groq-is-running-llama-3-instruct-8-70b-by-meta-ai-on-its-lpu-inference-enginge/
  27. [27]SambaNova. Llama 3 8B speed test (2024-05-29). https://sambanova.ai/press/sambanova-scorches-nvidia-in-new-speed-test
  28. [28]NVIDIA. Boost Llama 3.3 70B inference throughput 3x with TensorRT-LLM speculative decoding (2024-12-17). https://developer.nvidia.com/blog/boost-llama-3-3-70b-inference-throughput-3x-with-nvidia-tensorrt-llm-speculative-decoding
  29. [29]NVIDIA. Blackwell breaks the 1,000 TPS/user barrier with Llama 4 Maverick (2025-05-22). https://developer.nvidia.com/blog/blackwell-breaks-the-1000-tps-user-barrier-with-metas-llama-4-maverick/
  30. [30]AMD Alveo V80: $9,495 MSRP (2024-05-17); 190 W maximum power (AMD product page). https://www.amd.com/en/products/accelerators/alveo/v80/a-v80-p64g-pq-g.html
  31. [31]NVIDIA H200 NVL price estimates, $31-35k (accessed 2026-11-13); 600 W maximum. https://www.trgdatacenters.com/resource/nvidia-h200-price-guide/
  32. [32]Meta. Llama-3.2-3B-Instruct model card. https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct
  33. [33]M. Chen et al. Evaluating Large Language Models Trained on Code (HumanEval). 2021. https://arxiv.org/abs/2107.03374
  34. [34]L. Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. https://arxiv.org/abs/2306.05685
  35. [35]K. Cobbe et al. Training Verifiers to Solve Math Word Problems (GSM8K). 2021. https://arxiv.org/abs/2110.14168
  36. [36]R. Taori et al. Stanford Alpaca: An Instruction-following LLaMA Model. 2023. https://crfm.stanford.edu/2023/03/13/alpaca.html
  37. [37]T. Kwiatkowski et al. Natural Questions: A Benchmark for Question Answering Research. TACL 2019. https://ai.google.com/research/NaturalQuestions