Fast FPGA drafting for H200 serving.
The talon is a box of eight AMD V80 FPGA cards that does the drafting half of EAGLE speculative decoding for H200 GPU servers. The H200 keeps the target model and only verifies; the talon proposes the token trees. Same output as the GPU alone, sooner.
measured talon2 server, 2026-11-19: humaneval 80 prompts × 256 new tokens, greedy; accept length 6.91 vs 6.90 (Llama) and 10.42 vs 10.46 (Qwen), so the speed-up comes from faster drafting, not from accepting more tokens per step. All 80 replies are identical to the reference, three rounds each. Everything marked [P] below is projected.
A drafting box beside your GPUs, not a replacement for them.
EAGLE speculative decoding splits each step in two. A small drafter guesses a tree of next tokens; the large target checks the whole tree in one pass and keeps the longest correct branch. The result is exactly what the target would have written alone.
- Drafter on FPGA: the 2-bit drafter lives in on-chip memory on every card, so drafting is a fixed, short, predictable time.
- Target on H200: the GPU only verifies, the job it is best at.
- Shared: the cards are idle while a GPU verifies, so one talon takes turns serving several GPU servers.
On silicon today: one user gets ×1.44 on Llama and ×1.12 on Qwen.


Batch 1 is what one user feels: tokens per second on their own reply. The Llama result reproduced on three separate power-ups of the server (1,000.67, 996.00 and 998.68 tok/s).
measured talon2, 2026-11-19, humaneval 80 × 256, greedy, depth-11 width-2 tree (Llama) and depth-15 width-2 tree (Qwen) on the cards; H200 alone = SGLang EAGLE with the GPU drafting, same target, same drafter weights.
Drafting is where the talon is strongest: ×23.7 the drafting rate of two H200s at small batch.

Measured today
Today's six cards draft a full depth-15 tree in 3.13 ms: 319 trees/s, against 120.57 for an H200 drafting alone (×2.6). Today's cards draft one tree at a time, so the H200 overtakes them from about three trees in flight.
The talon design [P]
The talon drafts many trees at once (two 4-card sets taking turns) with the 4× card-time speedup built in. It stays ahead of two H200s at every batch from 1 to 128: ×23.7 at 1 tree, ×17.1 at 8, ×2.6 at 128.
Equal cost: one talon box vs two H200
Taking as a premise that one talon box costs the same as two H200s, these ratios are drafting throughput per unit cost. No prices are assumed.
| draft trees in flight | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| talon ÷ 2 H200, drafting throughput per unit cost [P] | ×23.7 | ×23.7 | ×22.4 | ×17.1 | ×10.6 | ×5.9 | ×3.6 | ×2.6 |
| same, against the H200's in-engine draft time | ×14.0 | ×14.0 | ×12.5 | ×9.8 | ×6.3 | ×3.9 | ×2.8 | ×2.5 |
talon (8x V80): 8 V80 with the 4× card-time speedup built in: 2× clock (66→132 MHz, roadmap) × 2× misc (placeholder) [P]. H200 line = standalone drafter bench (same tree, same context of 566 tokens, no target loaded). Inside SGLang the H200 drafts one tree faster (4.91 ms), so the second row is the conservative comparison. Drafted tokens per tree = 29.
While one GPU verifies, the talon drafts for the next: the cards are busy only 32% of each step.
Measured on one server: a Llama step takes 6.9 ms and the cards are busy for 2.2 ms of it (31.6%), so one card set has room for about 3 servers. On Qwen3.8-27B the GPU verifies longer and the cards are busy 18.7% of the step, room for about 5 servers per 4-card set.
The bigger the target, the more GPUs per talon
A longer verify on the GPU means a smaller busy share on the cards. A talon (8x V80, 4× card-time speedup built in) keeps 33 Qwen H200s drafted at batch 1 and 38 at batch 8 [P].

measured busy share = card time per step ÷ step time, from on-card counters and the serving rate (Llama record set, batch 1, 3 rounds; Qwen set, batch 1). Servers per set = floor(1 ÷ busy share); per talon (two sets) = floor(2 ÷ busy share per set) [P]. Interleaving three servers on one card set is a projection; one server per set is what has run.
Add one talon to an 8×H200 node: ×1.82 at batch 1, up to ×3.26 at batch 64 [P].

| batch per H200 | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| users on the node | 8 | 16 | 32 | 64 | 128 | 256 | 512 | 1,024 |
| Llama-3.2-3B: node + talon ÷ node [P] | ×1.82 | ×1.41 | ×2.14 | ×2.31 | ×2.44 | ×2.61 | ×3.26 | ×2.93 |
| Qwen3.8-27B: node + talon ÷ node [P] | ×1.44 | ×1.71 | ×1.72 | ×1.92 | ×1.44 | ×1.59 | ×1.90 | ×2.04 |
The H200s the talon feeds stop drafting and only verify; the others keep drafting for themselves. The Qwen node baseline above batch 1 comes from a verify-time model calibrated on measured H200 runs.
One talon + 6 H200 servers, Llama
| batch per H200 | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 6 H200 by themselves, tok/s | 4,179 | 7,245 | 8,889 | 14,802 | 21,487 | 28,031 | 29,854 | 35,116 |
| the same 6 H200 + one talon, tok/s [P] | 7,599 | 10,194 | 19,063 | 34,260 | 52,365 | 73,063 | 97,330 | 125,658 |
| gain | ×1.82 | ×1.41 | ×2.14 | ×2.31 | ×2.44 | ×2.61 | ×3.26 | ×3.58 |
talon (8x V80): 8 V80 with the 4× card-time speedup built in: 2× clock (66→132 MHz, roadmap) × 2× misc (placeholder) [P]. Node by itself = 8 × the measured single-H200 EAGLE rate at that batch.
Two factors of two on the card time; the charts above include both.
| [P] | talon design at today's 66 MHz clock card design in development | + 2× drafter clock (132 MHz) roadmap, timing not closed | + 2× misc = 4× card time placeholder factor; shown in every chart |
|---|---|---|---|
| Pure drafting vs 2 H200, 1 tree | ×5.9 | ×11.9 | ×23.7 |
| Pure drafting vs 2 H200, 8 trees | ×4.3 | ×8.5 | ×17.1 |
| Pure drafting vs 2 H200, 128 trees | ×0.65 | ×1.30 | ×2.61 |
| 8×H200 node gain, Llama, batch 1 | ×1.10 | ×1.36 | ×1.82 |
| 8×H200 node gain, Llama, batch 8 | ×1.25 | ×1.87 | ×2.31 |
| 8×H200 node gain, Llama, batch 64 | ×1.46 | ×2.06 | ×3.26 |
| 8×H200 node gain, Qwen, batch 1 | ×1.12 | ×1.23 | ×1.44 |
| 8×H200 node gain, Qwen, batch 8 | ×1.47 | ×1.63 | ×1.92 |
| H200s one talon feeds, Llama, batch 1 | 3 | 5 | 10 |
| H200s one talon feeds, Qwen, batch 1 | 10 | 19 | 33 |
2× drafter clock (66 → 132 MHz)
The cards' compute array already runs at 132 MHz; the rest of the drafter runs at 66 MHz. Closing the whole drafter at 132 MHz halves card time with the same cycle counts. Not closed yet.
2× misc
A placeholder for the remaining card-side and talon-side host savings: card time ÷ 2 again and the host's per-request cost ÷ 2. It is not tied to one specific change.
[P] projected every cell. Columns are cumulative: the last column (highlighted) is what the charts show. Pure drafting is Llama; ratios against two H200s drafting alone.
What the talon does not do yet, in numbers.
- Many users on today's cards. With 8 requests in flight, one card set + H200 serves 1,235.47 tok/s; the H200 alone at batch 8 serves 2,467.02. Today's card images are built for batch 1. At 8 in flight the host process driving the cards is 97% busy while the cards are about 37% busy: the host, not the cards, is the limit.
- Pure drafting today. An H200 drafting alone overtakes today's cards from about three trees in flight (measured).
- Without the clock roadmap. At today's 66 MHz the talon design would fall behind two H200s in pure drafting at 64 and 128 trees in flight (×0.90, ×0.65), and the 8×H200 Llama node gain at batch 2 would be ×0.98 [P].
- Against the H200's in-engine draft time the talon's drafting lead at one tree is ×14.0, not ×23.7 [P].
- Other drafters. With the Qwen GPTQ-2 drafter the H200 alone is faster today: 442.98 vs 255.69 tok/s at batch 1 (measured).
Measured means it ran on silicon; [P] means a model calibrated on silicon.
Measured vs projected
- measured read from a run on the talon2 server (7 V80 + 1 H200): every number in the hero, the two bar charts, today's diamonds and the H200 lines.
- [P] every talon line. It comes from a timing model of the cards and the H200 that predicts measured Llama serving rounds it was not fitted on within −2.9 % to +1.0 %.
- Ratios (×N) are the two plotted numbers divided, at the same batch.
Assumptions
- Today's drafter clock is 66 MHz; 132 MHz is a roadmap target, not closed.
- “2× misc” is a placeholder factor.
- The talon line models the high-batch card design (8 V80 as two 4-card sets taking turns), which is in development and not yet on silicon.
- Qwen memory wall: on one H200, batch × (1 + verified tokens) ≤ 440, so fewer tokens are verified per request at large batch, on both sides.
- Same drafter weights on both sides; measured accept lengths match (6.91 vs 6.90 Llama, 10.42 vs 10.46 Qwen), and the projections keep the same tree on both sides.
- Lossless: output equals the target's own greedy output.
- Equal cost (talon box = two H200) is a premise; no prices are used.