933.14 tokens per second single-stream on HumanEval at an accept length of 6.91, faster than GPU-only EAGLE on all five datasets measured on the same H200
Speculative decoding hides the latency of a target model behind a cheap drafter, but at batch 1 and high accept lengths the drafter's sequential tree levels become a large part of every step. We move the drafter of an EAGLE-style tree speculator for Llama-3.2-3B onto six AMD Alveo V80 FPGAs. Four attention cards run 4-way tensor parallel and two cards run the MLP and the 128K-token vocabulary, linked card-to-card by QSFP. A host kit hands each draft tree to an unmodified SGLang verifier on one NVIDIA H200. The draft model's 2-bit weights stay resident in on-chip memory, and the cards draw a width-2, depth-11 tree per verify step at 66 MHz. On HumanEval the system measures 933.1 tok/s at an accept length of 6.91, against 838.8 tok/s for the best GPU-only EAGLE configuration on the same H200 on the same day (1.11×) and 305.4 tok/s for plain autoregressive decoding (3.06×) [C1, C2]. It is faster than GPU-only EAGLE on all 5 datasets measured. A measurement-calibrated model of the loop between the two card roles guided four techniques: owner-gather of the exact vocabulary rescore across the two vocabulary cards (+1.5 %), two cuts in the host path around each tree (+2.9 % and +0.24 %), margin re-routes that close timing with +0.041 ns of slack, and a multiply-free 2-bit weight path. Across the five datasets the geometric-mean speed-up over GPU-only EAGLE is 1.138× (1.100–1.188×), and over plain decoding 2.08× [C1, C2]. Per drafter forward, the six V80s spend 44.2 mJ against 97.6 mJ on the H200, 2.21× less (time per forward × device run-mean power, per device, never summed; the H200 run-mean includes verification) [C3]. That advantage does not carry to the whole system: per drafting session the two are about level, and the FPGA system draws 566 W against 264 W, so it spends more energy per generated token [C3].
Every ratio, projection, model and energy or cost figure in this paper lives in this box, numbered C1 to C12. The tables and charts below carry only measured values; where the text needs a derived number it points here, like [C1].
At batch 1, decoding a 3-billion-parameter model on a data-centre GPU is bound by latency, not arithmetic. Each new token needs one full pass over the weights, and the GPU spends much of that pass waiting on memory and kernel launches. On our NVIDIA H200, plain SGLang decoding of Llama-3.2-3B[32] measures 305.39 tok/s on HumanEval (Section 6).
Speculative decoding[1,2] raises single-stream speed without changing the output distribution. A small drafter proposes several tokens, and the target model checks them all in one forward pass. Tree-shaped drafts[3] and feature-level drafters such as EAGLE[4–6] now push the mean number of tokens accepted per verify step above six on code. At that point the drafter is no longer cheap. An EAGLE drafter runs one forward pass per tree level, so a depth-11 tree costs eleven dependent small forward passes per verify step. On a GPU-only system those passes share the device with the verifier, and each one pays the GPU's fixed per-kernel latency.
This paper moves the drafter onto a different device class. Six AMD Alveo V80 FPGAs[30] hold a 2-bit EAGLE-style draft model entirely in on-chip memory and draft the token tree, while one H200 runs the unmodified SGLang[13] verifier. Batch-1 drafting suits an FPGA. The matrices are small, the 2-bit weights fit on chip, the schedule is static, and a hardware pipeline has no per-kernel launch cost.
Our contributions:
Speculative decoding[1,2] drafts k tokens with a cheap model and accepts the longest prefix that the target model would also have produced; with rejection sampling the output distribution is exactly the target's. SpecInfer[3] verifies a whole token tree in one target pass using a tree attention mask, so several alternative continuations are checked at once. Medusa[7] adds extra decoding heads to the target; Hydra[10] makes those heads sequentially dependent; Sequoia[9] optimises the tree shape for the hardware; Lookahead decoding[8] drafts without a separate model.
EAGLE[4] drafts at the feature level: a single transformer decoder layer consumes the target's hidden states, fused by a linear layer (fc), and predicts the next feature and token. EAGLE-2[5] grows the draft tree dynamically from the drafter's confidence, and EAGLE-3[6] trains on fused low-, mid- and high-level target features. EAGLE-3 reports a mean accept length of 6.74 on HumanEval for Llama-3.1-8B, and 373.25 tok/s at batch 1 in SGLang on one H100[6]. Recent block-diffusion drafters (DFlash[11]) report larger speedups again. A recent study of production serving[12] notes that speculative gains shrink as batch size grows, because verification comes to dominate. We therefore target, and claim only, the batch-1 regime.
Our drafter is EAGLE-style. Per draft forward it runs the phases in Table 2, with hidden size 3,072, 24 query heads and 24 key/value heads of dimension 128 (multi-head, not grouped), MLP width 8,192 and a 128,256-token vocabulary. It ranks the whole vocabulary with a cheap low-rank 2-bit head, keeps a shortlist of KC = 64 candidates per parent, and rescores that shortlist exactly against the fp16 LM head.
Most FPGA LLM work runs the whole model on the FPGA without speculation. FlightLLM[15] maps LLaMA2-7B onto a Versal VHK158 at about 3.5-bit weights and reports 92.5 tok/s, with 6.0× the energy efficiency of a V100S on an Alveo U280. DFX[16] uses four U280s for GPT-2; the spatial-accelerator study of Chen et al.[17] and LoopLynx[18] target GPT-2-class models; TerEffic[19] runs ternary 370M and 2.7B models. The closest prior art to our split is DFVG[20] (draft on the FPGA, verify on the GPU), whose full text we could not retrieve, and HeteroReason[21], which reports 1.01–1.42× latency speedups for reasoning models with an FPGA drafter. SpecMamba[22] runs speculative decoding for state-space models on Versal parts. We found no published LLM throughput for the Alveo V80.
Table 1 places the system among published single-stream numbers. The rows are not like-for-like: they use different model sizes (3B here; 8B to 400B elsewhere), precisions, hardware and dates, and most service numbers are vendor or third-party measurements we did not repeat. Batch-1 decode speed falls roughly with model size, so a 3B number cannot be ranked against an 8B or 70B number. The table is context, not a leaderboard.
| System | Model | Per-user tok/s | Hardware | Source |
|---|---|---|---|---|
| This work | Llama-3.2-3B, 2-bit drafter | 933.14 | 6× Alveo V80 + 1× H200 NVL | Section 6 |
| EAGLE-3 in SGLang | Llama-3.1-8B | 373.25 | 1× H100 | [6] |
| NVIDIA TensorRT-LLM, 1B draft | Llama-3.3-70B FP8 | 181.74 | 1× H200 | [28] |
| NVIDIA DGX B200, EAGLE-3-based draft | Llama-4 Maverick (400B MoE) FP8 | >1,000 | 8× B200 | [29] |
| Cerebras Inference | Llama-3.1-8B, 16-bit | 1,800 | CS-3 wafer-scale system | [25] |
| Groq LPU (Artificial Analysis, 2024-04) | Llama-3-8B | 877 | GroqCloud | [26] |
| SambaNova SN40L (Artificial Analysis) | Llama-3-8B, 16-bit | 1,084 | 16-chip SN40L | [27] |
| FlightLLM | LLaMA2-7B, ~3.5-bit | 92.5 | Versal VHK158 (whole model on FPGA) | [15] |
Table 1. Published single-stream (per-user) decode speeds. All rows except ours are quoted from the cited source and were not re-measured. Ratios against our number are deliberately not computed: model size dominates batch-1 speed.
Figure 1 shows the system. The H200 runs SGLang 0.5.16 with the Llama-3.2-3B target and verifies a 20-token tree per step (FlashAttention-3 backend[14], CUDA graphs). After each verify, the GPU also computes the drafter's fc fusion and the drafter's key/value rows for the newly accepted tokens. That is one small batched forward and the only drafter work left on the GPU. A host process ships those rows and the root of the next tree to the cards and rings their doorbells. The six cards then draft the whole tree without any host involvement. The host reads back the tree, and SGLang verifies it.
flowchart TB
subgraph G["NVIDIA H200 NVL · SGLang 0.5.16"]
direction LR
T["Llama-3.2-3B target
verifies a 20-token tree"] -->|accepted tokens| E["drafter fc fusion +
K/V of accepted rows"]
end
H["Host card process
registers + DMA over PCIe"]
subgraph A["4 ATT cards · TP4, 8 heads each"]
direction LR
A0["ATT r0"] --- A1["ATT r1"] --- A2["ATT r2"] --- A3["ATT r3"]
end
subgraph M["2 MLPV cards · vocabulary shards"]
direction LR
M0["MLPV r0"] --- M1["MLPV r1"]
end
E -->|K/V rows, tree root| H
H -->|doorbell, root image| A
A -->|per-item hand-off| M
M -->|chosen parents, next input| A
M -->|finished tree| H
H -->|draft tree to verify| T
Figure 1. System overview. Thin lines between cards are QSFP card-to-card links: the four ATT ranks all-reduce the attention output projection over them, and the two MLPV ranks exchange partial sums and rescore results. The host touches the cards only at the start and end of each tree.
Table 2 lists the phases of one draft forward and where each runs. The four ATT cards each own 8 of the 24 heads. Each computes the input RMSNorm, its quarter of the QKV projection, attention over its own key/value cache, its share of the output projection followed by a 4-rank all-reduce, and a quarter of the gate and up projections. The two MLPV cards finish the MLP and own the vocabulary side. That means the down projection, the final norm, the low-rank cheap LM head over all 128,256 tokens, tree selection, and the exact rescore of the KC = 64 shortlist against an fp16 LM head held in HBM.
| Phase | Operation | Dimensions | Card | Weights / data in | Cycles per item |
|---|---|---|---|---|---|
| fc | fuse 3 target hidden states | 9,216 → 3,072 | GPU | GPU memory | – |
| G1 | RMSNorm (input) | 3,072 | ATT | BRAM (gamma) | 456 |
| G4 | QKV projection, rank quarter | 6,144 → 3 × 768 | ATT | 2-bit, URAM | 1,421 |
| G5 | attention, 8 heads per rank | head dim 128 | ATT | K/V cache in BRAM | 1,913 |
| G6 | output projection + 4-rank all-reduce | 768 → 3,072 | ATT | 2-bit, URAM | 712 |
| G7g / G7u / G7p | gate, up, SiLU·up (quarter) | 3,072 → 8,192 | ATT | 2-bit, URAM | 1,397 / 557 / 133 |
| – | pipeline I/O + uncounted | ATT | 607 + ~186 | ||
| G9 | down projection (+ next-level feature) | 8,192 → 6,144 | MLPV | 2-bit, URAM | 850 |
| G11 | final RMSNorm | 3,072 | MLPV | BRAM | 466 |
| GLM + G12 | low-rank cheap LM head, full vocabulary | 3,072 → rank → 128,256 | MLPV | 2-bit, URAM | 226 + 22 |
| TREE | top-KC survivor selection | KC = 64 | MLPV | on chip | 2,019 |
| G8 | exact rescore: gather 64 rows + dot | 64 × 3,072 fp16 | MLPV | fp16 LM head in HBM | 1,915 (before owner-gather) |
| – | ingest, level exchange | MLPV | 538 + ~287 |
Table 2. One draft forward ("item") and where it runs. Cycle counts are measured on silicon at 66 MHz by per-phase hardware counters on the ATT image 0xE4B713BC and the MLPV image 0xE4B71360 (mean of 20 census trees, depth 7, width 2, KC 64). They sum to 7,382 cycles per ATT item and 6,323 per MLPV item. The G8 row is the value before owner-gather (Section 4.2).
A width-2 tree keeps two children per level, so a depth-11 tree is up to 21 items after the root. Two schedule features keep the cards busy. With EARLY_A, the MLPV sends the next level's input vector back early. With EAGER, the ATT starts a child before its level is formally committed. Even so, child c at level L+1 must wait for the MLPV result of child c at level L, so the two roles form one loop. A mass rule (TREE_STOP) ends a tree early when the drafter is unsure: it stops when the top-2 probability mass falls below 2−2.5. In the record runs, 1,400 of 2,534 trees stopped early, at a mean served depth of 8.45 of 11 levels.
Each V80 is a three-die (three-SLR) Versal device. The die a block sits on is forced by memory, and that dictates where the inter-die wires (SLLs) go. The 2-bit GEMV array has one URAM per lane, so it is split into lane groups across dies. The attention key/value cache fills the BRAM of one die, so attention and the full-width "hub" share SLR0, and array groups 0 and 2 sit on SLR2 (Figure 2). Every GEMV pass broadcasts the activation from SLR0 to the other dies and returns per-lane results. The output projection also leaves through the QSFP transceivers in SLR1 to the other ranks.
Figure 2. Floorplan of the attention card. Placed utilisation per die is measured on the shipped ATT image 0xE4B713BC. The binding numbers are DSP in SLR0 (582 free) and SLICE in SLR0 (92.3 %). Arrows show the per-pass activation broadcast and the result return.
Table 3 gives post-place utilisation and post-route timing for the two shipped images. Both run at a single 66 MHz drafter clock. Neither is limited by URAM. On the ATT the tightest numbers are DSP (85.4 %) and SLICE (92.3 %) in SLR0. On the MLPV they are SLICE in SLR1 and SLR2 (93.4 % and 94.4 %), DSP in SLR1 (84.5 %) and the SLR0–SLR1 inter-die wires (70.8 % of 18,870).
| Image | LUT | SLICE (worst die) | DSP58 | BRAM | URAM | SLL 0–1 / 1–2 | Setup / hold slack (ns) |
|---|---|---|---|---|---|---|---|
| ATT 0xE4B713BC | 39.73 % | 85.65 % (92.31) | 48.55 % | 47.66 % | 38.65 % | 9,646 / 1,885 | +0.040 / +0.010 |
| MLPV 0xE4B71360 | 48.11 % | 92.34 % (94.04) | 69.17 % | 27.68 % | 26.60 % | 13,485 / 10,722 | +0.002 / +0.010 |
| MLPV 0xE4B71361 (G8_OWN) | 45.55 % | 89.19 % (94.42) | 69.17 % | 27.68 % | 26.60 % | 13,360 / 12,331 | +0.000177 / +0.010 |
| MLPV M1361 (margin re-route of 0xE4B71361) | same netlist and placement as 0xE4B71361 | +0.041 / +0.016 | |||||
Table 3. Post-place utilisation (percent of the V80) and true post-route slack at 66 MHz. The deployed set is four ATT 0xE4B713BC and two MLPV M1361. All slack values are positive. The device has 2.57M LUT, 10,848 DSP58, 3,741 BRAM36, 1,925 URAM and 18,870 SLLs per die crossing.
A full V80 image takes a median of 9.4 hours to build, so every lever must be ranked before it is built. Neither card is saturated. In a depth-7 census tree the ATT is busy 73 % of the in-tree time and the MLPV 63 %. What sets the pace is the loop formed by an ATT item, the hand-off, the MLPV item and the hand-off back. Figure 3 shows the two items side by side. Because child c of one level and child c' of the next interleave, each tree item costs about half the loop, not a whole one. A cycle cut anywhere in the loop therefore returns about half of itself per item [C6]. With one fitted parameter (the hand-off), the model reproduces three silicon tree times within ±23 µs. It ranked the exact-rescore gather on the MLPV first among the bit-exact levers, so we built that next.
Figure 3. Measured cycles per item for each phase, ATT image 0xE4B713BC and MLPV image 0xE4B71360, at 66 MHz (15.15 ns per cycle). The MLPV's TREE and G8 dominate its item. G5 attention and G4 QKV dominate the ATT item.
Before this change, both MLPV cards gathered the same 64 exact LM-head rows from HBM and dotted them redundantly. Both cards already hold an identical, globally ordered shortlist. Under G8_OWN, rank r gathers only the shortlist slots whose token id has parity r. After selection, the two cards swap the resulting dot scores over the card-to-card link they already share for the down-projection exchange. Slot positions are frozen and identical on both cards, so no token id travels, only {owned flag, score}. Each card lands its peer's scores on the non-owned slots and finishes as before. Both cards still compute every score with the same row, the same dot unit and the same normalisation, so the result is bit-exact. The change added 19,781 LUT and 2.1K FF, with no DSP, BRAM, URAM or new port. Its simulation gate checks every slot of every item on both ranks, with failure-injection arms that must fire.
On silicon, the image 0xE4B71361 ran three serves at 904.97, 902.40 and 900.93 tok/s (mean 902.77), against a same-day 889.26 for the parent set. Accept length (6.91), the 2,839 verify steps and all 80 output texts were identical. The slowest G8_OWN run beat the fastest parent run. The per-level card census fell from 280.0 to 269 µs. The model had predicted +1.2 to +2.1 % [C6].
More than half of a verify step is spent outside the card tree (Table 7). We split that time into an end half (cards done to tree handed to SGLang) and a start half (verify result to root doorbell) and instrumented both on silicon.
HX_O7 (end half). Counted on the record kit, the host made 100 register reads and 2 bulk tree reads between "cards done" and "tree returned". HX_O7 cuts this to 39 reads and 1 bulk read in three parts. Part l uses lean poll passes, part f exits on the done check without a further full pass, and part c runs the cross-checks after the tree is published to the GPU but before the reply. It issues no new card operation and changes no error path. Over three alternating A/B pairs it measured +2.94 % (901.14 → 927.65 tok/s), every B row above every A row, with identical texts. The step shrank by 220 µs, against a projected 150–230 µs [C7]. An instrumented pair measured the end half directly: wait entry to tree returned fell from 2,720.1 to 2,516.8 µs (p50).
HX_S7 (start half). The first on-silicon profile of the start half measured 1,130 µs p50 (Figure 4). Its largest parts are the scheduler-side work (620 µs, of which 201 µs is the GPU's drafter key/value extend), the tree request queued behind the key/value load (180 µs) and the staging-to-doorbell path (306 µs). Of two host levers, part v moves the 37 µs structural validation of the tree to after the publish, under the GPU verify. It measured +0.24 % over 4 of 4 positive pairs (930.45 → 932.71), with a best row of 933.28 tok/s. Part y, deferring a GPU synchronisation, measured −0.22 %. The trailer wait it would hide is only 5.9 µs, so we rejected it.
Figure 4. The start half of a verify step on the record path, from verify result to root doorbell: 1,130 µs p50, measured on silicon by stamping both host processes on one clock. Bars are the segments in order.
| Arm | Change | tok/s (mean) | Rows | Texts vs record | Verdict |
|---|---|---|---|---|---|
| A | record kit (G8_OWN image) | 901.14 | 3 | identical | baseline |
| l | lean poll passes only | 901.67 | 1 | identical | no measurable effect |
| lf | + fast exit | 915.06 | 1 | identical | about +1.2 % |
| lfc = HX_O7 | + late checks after publish | 927.65 | 3 | identical | adopted |
| A′ | HX_O7 record (next window) | 930.45 | 5 | identical | baseline |
| v = HX_S7 | validate after publish | 932.71 | 4 | identical | adopted |
| vy | + deferred GPU sync | 930.60 | 6 | identical | rejected |
Table 4. Host-path A/B results on silicon, HumanEval 80 × 256, depth 11, width 2. Arms alternated with their neighbours on the same boot. Run-to-run noise within an arm is about ±0.5 %.
Our rule is that an image ships only with positive setup and hold slack. A negative-slack image can hang the card or silently corrupt arithmetic. Over the week of 5–12 November we ran 213 distinct builds and closure arms (122 full builds). Of those, 91 closed. MLPV builds without a margin step consistently closed at +0.000 to +0.016 ns, which is positive but below our +0.030 ns release bar.
The margin re-route recipe starts from the routed checkpoint. It adds a clock uncertainty of 0.040 ns setup and 0.012 ns hold on all 49 clocks, runs physical optimisation, re-routes under that margin, and then removes the margin and re-times. This keeps the netlist and placement and buys slack that survives the true timing. On M1361 it took the G8_OWN image from +0.000 / +0.010 to +0.041 / +0.016 ns (Table 5). Across the week, 12 of 12 margin or closure arms on bases that had already routed with non-negative slack improved true slack, 6 of them by 0.030 ns or more. No arm rescued a base at −0.2 ns or worse (0 of 4). We re-flashed the drafter with M1361 and kept it: its serves were identical in text and within the record band.
| Step (M1361) | Margined WNS / WHS | Failing endpoints | True WNS / WHS |
|---|---|---|---|
| routed source, margin off | – | 0 | 0.000 / 0.010 |
| margin on (setup 0.040, hold 0.012) | −0.040 / −0.002 | 4,701 setup, 5 hold | 0.000 / 0.010 |
| physical optimisation | −0.024 / −0.002 | 2,142 setup, 2 hold | 0.016 / 0.010 |
| re-entrant route | +0.001 / +0.004 | 0 | 0.041 / 0.016 |
| final, margin off | – | 0 | +0.041 / +0.016 |
Table 5. The margin re-route of the G8_OWN MLPV image, step by step. Slack in ns at 66 MHz.
The drafter checkpoint is trained with 2-bit body weights in the style of BitNet[24]. Every body tensor equals its 2-bit code times a scale, bit for bit, and the GPU-only baseline loads the same checkpoint. On the cards the 2-bit codes sit in URAM next to each GEMV lane, and the scales sit in BRAM. Activations stay on chip, so a forward pass issues no off-chip weight traffic. The only HBM read on the draft path is the exact rescore's gather of 64 fp16 LM-head rows.
The ATT array is multiply-free in the LUT-GEMM style[23]. Activations are quantised to 8 bits, each lane cluster shares a table of partial sums, and each lane selects from it with its own 2-bit codes. The zero-point term is common to the whole array and is computed once. This frees DSPs for attention. The ATT image uses 48.55 % of the device's DSP58, most of them (3,402) in SLR0 next to the attention engines.
Numerically, the cards track a GPU emulation of their own arithmetic closely. With TREE_STOP off, the cards and their GPU twin differ by 3 verify steps in total over the 77 requests whose texts agree. Most of the accept gap to GPU-only EAGLE (Section 6.3) comes from two settings we chose for speed, not from the 2-bit arithmetic [C10].
All runs used one server with seven AMD Alveo V80 cards and one NVIDIA H200 NVL. Six V80s drafted: four attention cards running ATT image 0xE4B713BC and two MLPV cards running M1361. The seventh card was not opened by any run. The target, Llama-3.2-3B-Instruct in fp16, ran in SGLang 0.5.16 with CUDA graphs, piecewise CUDA graphs and the FlashAttention-3 draft attention backend, at batch 1 (one running request). The FPGA system and GPU-only EAGLE load the same 2-bit drafter checkpoint.
v) host levers. This is the record configuration, unchanged.Five prompt sets: HumanEval[33] (code), MT-Bench[34] (chat), GSM8K[35] (math), Alpaca[36] (instructions) and Natural Questions[37] (QA; 79 questions, filled to 80 by repetition for every configuration). Each run has 80 requests × up to 256 new tokens, a seed-42 shuffle and greedy decoding (temperature 0). Throughput is generated tokens divided by the wall time of the timed window, one request at a time. Accept length is tokens per verify step.
Each configuration ran two timed rounds per dataset. Inside a dataset, round 1 was a random permutation of the four configurations and round 2 a permutation of the three speculative ones (fixed seed). AR round 2 ran afterwards. Before every run the H200 was idle and cooled to 60 °C or less. The H200 SM clock read 1,785 MHz in every 1 Hz sample of every timed window, with no throttle reason set, and peak GPU temperature was 57–73 °C. The HumanEval FPGA runs reproduced the record's 80 output texts byte for byte.
H200 power is the driver's power.draw sampled at 1 Hz. Card power is the V80 management firmware's total-power sensor, read for all six drafting cards about every 2 s (4–19 samples per card per window). It was cross-checked against the cards' 1 Hz hardware-monitor reading, and the two sums agree within 1.7 W in every run. Power is averaged over each run's timed window. "System power" counts the H200 plus the six drafting cards for the FPGA system and the H200 alone for GPU-only and AR. The cards' idle draw during GPU-only and AR runs is reported but not counted. Host CPU, memory and fans are not measured.
Figure 5 and Table 6 give the measured throughput. The FPGA system is faster than the best GPU-only configuration on all five datasets, from 877.67 vs 797.59 tok/s on GSM8K to 430.44 vs 362.27 on Natural Questions [C1]. On HumanEval it reaches 933.14 tok/s at an accept length of 6.91. Plain decoding runs at 305–307 tok/s on every set, so the gain over it follows the accept length. It is largest on HumanEval and GSM8K (accept 6.91 and 6.46) and smallest on Natural Questions (accept 2.71) [C2]. The relative lead over GPU-only EAGLE is the other way round: largest on Natural Questions and smallest on GSM8K and HumanEval.
The FPGA system's accept length is below GPU-only depth 11 on all five sets (for example 6.91 vs 7.05 on HumanEval, 3.67 vs 3.85 on MT-Bench). It wins on speed because each draft level is cheaper, not because it accepts more (Section 6.3).
Output equality. Under greedy decoding, configurations that speculate differently (or not at all) do not produce byte-identical texts on every request, and the FPGA system is not special in this. Its texts match plain AR on 71–79 of 80 requests per set, equal to or higher than GPU-only depth 9 against AR (70–77 of 80) on every set. GPU-only depth 9 and depth 11 also differ from each other on 1–8 requests per set. We did not measure the mechanism of these divergences; fp16 verification over different tree shapes is one candidate.
Figure 5. Single-stream throughput per dataset, same H200, same day. "GPU-only EAGLE" is the faster of its depth-9 and depth-11 configurations on each dataset. Bars are means over the timed runs (Table 6).
| Dataset | 6 V80 + H200 | accept | GPU-only d9 | accept | GPU-only d11 | accept | Autoregressive |
|---|---|---|---|---|---|---|---|
| HumanEval | 933.14 934.48 / 931.81 | 6.91 | 838.81 837.62 / 840.00 | 6.63 | 820.30 822.74 / 817.85 | 7.05 | 305.39 306.18 / 304.61 |
| MT-Bench | 557.76 559.63 / 555.90 | 3.67 | 487.61 487.30 / 487.92 | 3.80 | 454.07 453.59 / 454.56 | 3.85 | 306.09 306.41 / 305.77 |
| GSM8K | 877.67 878.55 / 876.80 | 6.46 | 797.59 797.03 / 798.14 | 6.30 | 768.33 771.35 / 765.30 | 6.61 | 306.07 306.28 / 305.86 |
| Alpaca | 530.46 531.66 / 529.26 | 3.46 | 462.50 462.70 / 462.29 | 3.61 | 427.20 428.54 / 425.85 | 3.62 | 306.24 305.53 / 306.94 |
| Natural Questions | 430.44 429.39 / 431.49 | 2.71 | 362.27 363.27 / 361.28 | 2.81 | 333.56 334.33 / 332.80 | 2.81 | 306.63 305.47 / 307.79 |
Table 6. Measured tok/s (mean of n timed runs) and accept length (tokens per verify step) per dataset. Two timed rounds per configuration, alternated in a random order on one boot; the two rounds of every configuration produced identical texts. Bold marks the faster speculative configuration. Accept is not defined for autoregressive decoding (1 token per step).
Figure 6 traces the HumanEval record on the six-card set at a fixed configuration (depth 11, width 2, KC 64, FlashAttention-3, the same 80 prompts). Each step is one image or one host lever, and each was A/B-tested on silicon with texts identical to the record.
Figure 6. HumanEval tok/s on the six-card set after each change (mean of that change's runs; accept 6.91 throughout). B4 and BC are the ATT images 0xE4B713B4 and 0xE4B713BC; hx7 is an earlier host-register lever.
At depth 11 the FPGA system needs 2,839 verify steps for the 80 HumanEval prompts, and GPU-only EAGLE needs 2,776. Accept lengths are 6.91 and 7.05. Both load the same checkpoint. Silicon A/B runs split the 63 extra steps three ways (Figure 7). TREE_STOP accounts for 33 steps: with the rule off, the cards need 2,806. The card's cheap head at KC = 64 accounts for 18: a GPU twin with that head needs 2,794. The remaining 12 come from 3 requests whose texts diverge. Neither setting is free to undo. Turning TREE_STOP off lowered throughput to 876.12 tok/s, and no runtime setting beat the record on tok/s. A width-1 level costs the same card time as a width-2 level (263 vs 266 µs per level at depth 11), so thinning the tree buys no time either.
Figure 7. Extra verify steps of the FPGA system over GPU-only EAGLE at depth 11 (63 in total), by measured cause.
Table 7 places the measured segments on the record step. The card tree itself, from doorbell to the last card done, is under half of the step. The rest is GPU verification, SGLang scheduling and host glue [C9]. This is why two host levers (Section 4.3) were worth more than the G8_OWN image.
| Segment | µs (p50) | Source run |
|---|---|---|
| Whole step, mean over the run (21.11 s / 2,839 steps) | 7,447 | HX_O7 B arm, 927.65 tok/s |
| Start half: verify result → root doorbell | 1,130 | HX_S7 profile |
| Card tree, issue → done (bench, depth 11) | 2,878 | M1361 census, 100 trees |
| Wait entry → poll-loop exit (serve, HX_O7) | 2,452.6 | HX_O7 instrumented arm |
| Loop exit → tree returned (HX_O7) | 64.2 | same |
| Tree returned → handler returned (build + publish) | 142.0 | same |
| Per tree level on the cards (census, depth 11, width 2) | 262 | M1361 census |
Table 7. Measured segments of one verify step on the record configuration. Segments come from different instrumented runs on the same images and are not additive. The derived remainder is [C9].
The drafter-energy metric we state is energy per drafter forward, computed as time per forward × device run-mean power, per device, never summed (white paper 11-04 method). A drafter forward is one pass of the draft model for one tree item. On the FPGA side it is one card item: 0.137 ms, measured by the on-card census at depth 11 and width 2 on the benchmark boot (2.878 ms per 21-item tree). On the GPU side it is 0.37 ms, the slope of GPU-only draft time against tree depth measured on an H200 on 2026-10-19. Each time is multiplied by its own device's run-mean power in the 2026-11-13 benchmark. That is the six drafting V80s' summed sensor power during the FPGA runs, and the H200's driver power during the GPU-only runs. The two devices are never added together.
By this method the six V80s spend 44.2 mJ per drafter forward against 97.6 mJ for the H200, 2.21× less (5-dataset mean; 2.15–2.23× on every dataset at either GPU depth, Table 8). The white paper of 2026-11-04 reported 45 vs 102 mJ with four V80s.
| Dataset | 6 V80 W | FPGA mJ / forward | H200 W, GPU-only d9 | GPU d9 mJ / forward | H200 W, GPU-only d11 | GPU d11 mJ / forward | GPU d9 / FPGA |
|---|---|---|---|---|---|---|---|
| HumanEval | 323.5 | 44.3 | 262.9 | 97.3 | 263.8 | 97.6 | 2.20× |
| MT-Bench | 322.6 | 44.2 | 264.3 | 97.8 | 261.4 | 96.7 | 2.21× |
| GSM8K | 324.4 | 44.4 | 262.7 | 97.2 | 258.4 | 95.6 | 2.19× |
| Alpaca | 321.5 | 44.0 | 263.4 | 97.4 | 261.1 | 96.6 | 2.21× |
| Natural Questions | 321.2 | 44.0 | 265.6 | 98.3 | 264.4 | 97.8 | 2.23× |
| 5-dataset mean | 322.65 | 44.2 | 263.8 | 97.6 | 261.8 | 96.9 | 2.21× |
| 2026-11-04 (4 V80, depth 7) | 255.03 | 45.1 | 276.67 | 102.4 | – | – | 2.27× |
Table 8. Energy per drafter forward, time per forward × device run-mean power, per device, never summed (white paper 11-04 method). Power columns are measured run means. The mJ columns and ratios are derived: FPGA = 0.137 ms × 6-card W, GPU = 0.37 ms × H200 W [C3].
What this number includes and does not.
Table 9 gives whole-accelerator power in every timed window. System power counts the H200 plus the six drafting cards for the FPGA configuration, and the H200 alone for GPU-only and plain decoding. Averaged over the five datasets, the FPGA system draws 565.9 W: 243.2 W on the H200 and 322.6 W on the six drafting cards. GPU-only EAGLE at depth 9 draws 263.8 W, and plain decoding 335.4 W. Three measured facts shape the comparison.
On system power, the FPGA system uses more energy per token than GPU-only EAGLE on every dataset, about 1.8–1.9 times as much [C3]. Against plain decoding it uses less energy per token on high-accept sets and more on low-accept sets [C3]. We make no efficiency claim.
| Dataset | Configuration | H200 W | 6 cards W | System W | tok/s | J / token |
|---|---|---|---|---|---|---|
| HumanEval | 6 V80 + H200 | 236.4 | 323.5 | 559.9 | 933.14 | 0.600 |
| GPU-only d9 | 262.9 | 318.1 (idle, not counted) | 262.9 | 838.81 | 0.314 | |
| Autoregressive | 337.2 | 319.3 (idle, not counted) | 337.2 | 305.39 | 1.104 | |
| MT-Bench | 6 V80 + H200 | 244.9 | 322.6 | 567.5 | 557.76 | 1.018 |
| GPU-only d9 | 264.4 | 319.2 (idle, not counted) | 264.4 | 487.61 | 0.542 | |
| Autoregressive | 335.7 | 318.9 (idle, not counted) | 335.7 | 306.09 | 1.097 | |
| GSM8K | 6 V80 + H200 | 235.6 | 324.4 | 559.9 | 877.67 | 0.638 |
| GPU-only d9 | 262.7 | 319.2 (idle, not counted) | 262.7 | 797.59 | 0.329 | |
| Autoregressive | 335.5 | 319.1 (idle, not counted) | 335.5 | 306.07 | 1.096 | |
| Alpaca | 6 V80 + H200 | 245.9 | 321.5 | 567.4 | 530.46 | 1.070 |
| GPU-only d9 | 263.4 | 319.0 (idle, not counted) | 263.4 | 462.50 | 0.570 | |
| Autoregressive | 334.8 | 318.8 (idle, not counted) | 334.8 | 306.24 | 1.093 | |
| Natural Questions | 6 V80 + H200 | 253.4 | 321.2 | 574.6 | 430.44 | 1.335 |
| GPU-only d9 | 265.6 | 318.4 (idle, not counted) | 265.6 | 362.27 | 0.733 | |
| Autoregressive | 333.8 | 318.5 (idle, not counted) | 333.8 | 306.63 | 1.089 |
Table 9. System power: measured mean power over each run's timed window, averaged over the two rounds, and the resulting joules per token. System power counts the six cards only for the FPGA configuration. GPU-only depth 11 is omitted for space (it draws 258–264 W).
At nameplate ratings the FPGA system is 6 × 190 W of V80 plus 600 W of H200, or 1,740 W, against 600 W for the GPU alone[30,31]. At list or estimated prices, the six V80s ($9,495 each) roughly triple the accelerator cost of an H200 NVL (estimated at $31–35k). On both bases GPU-only EAGLE delivers more tokens per second per watt and per dollar [C4, C5]. Against rack-scale systems (Table 1), the FPGA system's envelope is far smaller: a Cerebras CS-3 is rated at 23 kW and a SambaNova SN40L rack at 10 kW typical[25,27]. But those systems serve larger models, so no per-token comparison is made.
In short, the six-FPGA drafter buys 10–19 % more single-stream speed on the same GPU. Per drafter forward it spends 2.21× less energy than the H200 by the white-paper method. Per drafting session it is about level, and per generated token the whole system uses about twice the energy and about three times the accelerator capital cost. It is worth deploying only where batch-1 latency is the product.
Six FPGAs drafting for one H200 deliver 933.1 tok/s single-stream on HumanEval for Llama-3.2-3B. That is faster than the best GPU-only EAGLE configuration on the same GPU on every dataset we measured, by 1.100–1.188×, and 1.40–3.06× plain autoregressive decoding [C1, C2]. The design keeps every draft weight on chip at 2 bits and splits one draft forward across attention and vocabulary cards. It improved by ranking levers with a silicon-calibrated loop model before building them: owner-gather of the exact rescore, cuts in the host path around each tree, and margin re-routes that give every shipped image positive setup and hold slack.
The result is a latency result. Per drafter forward the six V80s spend 2.21× less energy than the H200, but the whole FPGA system uses more energy per generated token and costs more [C3–C5], and more than half of each step is still GPU verification and host glue [C9]. The next steps follow from the measurements. They are a shorter start half (key/value load and staging on the critical path), recovering the 2 % accept gap without adding tree levels, and per-stream resident key/value state so the same cards can serve more than one request.