EAGLE talon: one talon drafting for five 8xH200 nodes
Carrick Shi · EAGLE · 2026-11-19
Downloads: PDF (presentation with breakdowns) · Markdown · All chart data (CSV)
Setup: one talon (8x V80) + five 8xH200 nodes = 40 H200. Step 1 is measured on silicon. Steps 2 to 6 are projections of the redesigned talon and carry the tag [P] projected.
One talon vs H200 nodes drafting themselves: Qwen3.8-27B, 1 / 3 / 4 / 5 nodes

Data: CSV (64 rows, both models)
Takeaway: blue = H200 nodes drafting themselves; green = the same nodes with one talon (8x V80, 4x speedup) drafting; labels = gain. [P] projected
One talon vs H200 nodes drafting themselves: Llama-3.2-3B, 1 / 3 / 4 / 5 nodes

How the 1 / 3 / 4 / 5-node charts are computed
Where the data comes from
- H200 verify times, measured on talon2 (GPU-only verify table, p50): Qwen BitNet target at V20, bs 1/2/4/8/16 = 20.31 / 21.16 / 23.88 / 31.93 / 51.66 ms; Llama FA4 bs 1/8/16/32/64/128 = 3.46 / 4.69 / 7.04 / 10.91 / 20.98 / 40.05 ms (scripts/hbcal/out/calibration.json, V_gpu). measured
- H200-alone serve at batch 1, measured: 289.49 tok/s (Qwen, accept 10.46) and 696.42 tok/s (Llama). measured
- Qwen H200 memory wall, measured: bs 32 / 64 / 128 at V20 fail at SGLang init (EXIT 137, 11-19); the hybrid target holds ~0.152 GB x (1+V) per request, so B x (1+V) <= 440 and the verified tokens per request V shrink with batch. measured
- Host cost: P 9.51 + h 0.73 x B ms per step, a least-squares fit on the measured SGLang H200 steps (step - verify - draft, bs 1-16); the talon side uses h / 2 (the '2x misc'). derived
- Accept: 10.42 measured (Qwen W2 V20); its change with V uses elasticity 0.13 from a measured Qwen mt_bench analogue; wider trees use a measured Llama analogue. derived
- Talon card time per tree: from bit-exact xsim per-unit cycle counts (attention item, MLPV fitting level F6, G8 v2) combined by the hbcal timing model; that model predicts today's Llama silicon within 2.9 % on runs it was not fitted on. projected
- The 4x: 2x drafter clock (66 -> 132 MHz, not yet closed) x 2x misc (a placeholder factor) is an assumption. assumption
How each line is calculated
- Blue = N_H200 x (one H200 running EAGLE alone at batch B), from the calibrated model: B x accept / (verify + draft + host step).
- Green = (H200s the talon feeds) x (per-H200 tok/s with the talon) + (remaining H200s) x (alone tok/s).
- Per-H200 tok/s with the talon = B x accept(V) / step period; period = verify(B, V) + talon-side host, under the overlap schedule (groups of requests alternate, so the H200 verify, the host and the card wave run side by side and the slowest sets the period).
- H200s the talon feeds = min(N_H200, floor(2 / card duty per set)); card duty per set = card time per period / period (two 4-V80 sets).
- V per B (Qwen) from the memory wall B x (1+V) <= 440: V = 20 / 14 / 10 / 10 / 10 / 10 / 5 / 2 at B = 1..128.
Worked example: Qwen, 3 nodes (24 H200), B = 32 (values from nodes1345_lines_std_20261119.csv and data_v16.json):
- Memory wall: 32 x (1 + V) <= 440, so V = 10.
- Blue, one H200 alone: 32 requests x accept 9.659 / step 96.05 ms = 3,217.9 tok/s; x 24 H200 = 77,230.
- With the talon: tree W3 V10, accept 9.907; schedule G2 (two groups of 16 alternate); period = 2 x verify(16 requests, V10) = 2 x 31.20 = 62.40 ms (the H200 verify binds; host 9.51 + 0.366 x 16 ms and the card wave overlap it); per H200 = 32 x 9.907 / 62.40 ms = 5,080.3 tok/s.
- Card duty per set = 5.35 ms / 62.40 ms = 0.0857, so the two sets feed floor(2 / 0.0857) = 23 H200 (of 24); the other 1 drafts on its own GPU.
- Green = 23 x 5,080.3 + 1 x 3,217.9 = 120,066; gain = 120,066 / 77,230 = x1.55.
The pitch: let the FPGA draft, let the H200 verify
- Speculative decoding = drafting, where a small model guesses ~20 tokens, plus verifying, where the big model on the H200 checks them all at once.
- Today the H200 does both.
- The talon is a box of 8 AMD V80 FPGA cards. It takes over the drafting, so the H200 only verifies.
Three strengths:
- Faster per user. Measured on silicon today.
- Mostly idle. The talon drafts so fast that it sits idle for much of each step: today's cards are ~30% busy.
- One talon, many H200s. The talon can take turns drafting for many H200s.
Definitions
- Node = one 8×H200 server (8 H200 GPUs).
- 5 nodes = 40 H200 GPUs. This deck is about one talon serving five nodes.
- talon = one box of 8 AMD V80 FPGA cards that drafts for the H200s. The H200s only verify.
- B = batch per H200, the number of requests each H200 serves at once. Five nodes at batch B serve 40 × B users.
- tok/s = accepted (output) tokens per second.
- [P] projected = projected, from the simulated and modelled redesign. Everything else is measured.
- Qwen = Qwen3.8-27B target with the BitNet drafter. Llama = Llama-3.2-3B target with the EAGLE4 2-bit drafter.
1. Today, measured: at batch 1 talon + H200 delivers 324.08 tok/s vs 289.49 for the H200 alone on Qwen (×1.12), and 1,000.67 vs 696.42 on Llama (×1.44).


Data: CSV (both models)
Takeaway: the output matches the H200 alone token for token, and accept length is the same (10.42 vs 10.46, 6.91 vs 6.90), so the whole gain comes from faster drafting.
How step 1 is computed
Formula: tok/s = accepted tokens ÷ wall time, over humaneval 80 prompts × 256 new tokens, greedy. Llama reports the mean of 3 rounds.
| Input | Value | Source | Tag |
|---|---|---|---|
| Llama, H200 + 6 V80, 3 rounds | 1,003.84 / 998.49 / 999.68 tok/s (19,615 tokens each) | WINDOW_RESULTS_20261119.md:96-100 | measured |
| Llama, H200 alone | 696.42 tok/s | HB_CALIBRATION.md:158 | measured |
| Qwen, H200 + 4 V80 | 324.08 tok/s, accept 10.42 (11-18 run, the number on the post) | WINDOW_RESULTS_20261119.md:185 | measured |
| Qwen, H200 + 4 V80, 11-19 retest | 324.74 tok/s = 15,510 tokens ÷ 47.76 s, 80/80 replies identical | WINDOW_RESULTS_20261119.md:190 | measured |
| Qwen, H200 alone | 289.49 tok/s, accept 10.46 | site data.json qwen.h200 | measured |
Worked example: nothing was measured at B=8, so this example uses batch 1. Llama round 1: 19,615 tokens ÷ 19.54 s = 1,003.84 tok/s. The mean of the 3 rounds is 1,000.67, and 1,000.67 ÷ 696.42 = ×1.44. Qwen: 324.08 ÷ 289.49 = ×1.12; the retest gives 15,510 ÷ 47.76 s = 324.74, also ×1.12.
2. The drafter is fast: in pure drafting the talon (8x V80) out-drafts two H200 drafters by ×24.66 at batch 1, still ×2.61 at batch 128 [P] projected.


Takeaway: these are drafting rates alone, with no verification. The H200 line is a measured drafter bench, and the H200 only catches up at very large batch (Llama: ×23.70 at 1, ×2.61 at 128).
How step 2 is computed
Formulas:
- 2 H200 = 2 × DBENCH trees/s at B/2. At B=1, one H200 drafts.
- talon = 2 sets × card-only set rate at B/2 × 4.
- Set rate = trees per set ÷ card time for those trees.
| Input | Value | Source | Tag |
|---|---|---|---|
| H200 drafter bench (DBENCH), depth 15 width 2, 29 nodes per tree, context 566, CUDA graph | Qwen, 4 trees: 409.55 trees/s | drafter_only/results_20261119/DBENCH.tsv (md5 def2c762) | measured |
| Card time per tree batch on one 4-V80 set, at 66 MHz | per-unit xsim cycle counts (ATT item, MLPV F6 947.6 cycles/node, G8 v2) | hbcal timing model (scripts/hbcal) | simulated + modelled |
| Set rate at 4 trees, 66 MHz | 1,822.28 trees/s (4 trees per 2.195 ms of card time) | f6_v9/data_v9.json red.Q | modelled |
| 4× card-time speedup | 2× drafter clock (66→132 MHz, not yet closed) × 2× misc (placeholder) | user ruling 2026-11-19 | assumption |
Worked example, Qwen, B=8 trees in flight:
- 2 H200: 2 × 409.55 = 819.10 trees/s.
- talon: 2 sets × 1,822.28 × 4 = 14,578.2 trees/s.
- Ratio: 14,578.2 ÷ 819.10 = ×17.80.
3. One talon keeps many H200s drafting: its card throughput covers all 40 H200s of five 8xH200 nodes up to batch 32 on Qwen and 2 on Llama; in the serving model of step 4 it feeds 33 / 40 / 40 / 40 / 40 / 33 / 18 / 13 of 40 on Qwen and 10 / 24 / 22 / 18 / 13 / 9 / 8 / 6 on Llama at B = 1..128 [P] projected.


Takeaway: there are two counts.
- Card throughput (this chart): Qwen 449 → 15 H200s, Llama 101 → 9. Above the 40-H200 line, the talon covers all five nodes.
- Served (step 4): fewer, because card duty per set caps the count at floor(2 ÷ duty). Unfed H200s draft on their own GPU.
How step 3 is computed
Formula: H200s fed = talon trees/s ÷ (B ÷ period per H200). Each request takes one tree per step.
- Realistic period = verify(B, V) + host, where host = P 9.51 ms + h·B with h = 0.73 ms halved to 0.366 ms.
- Worst-case period = verify only.
| Input | Value at B=8 | Source | Tag |
|---|---|---|---|
| talon supply = 2 sets × set rate at 4·B trees each × 4 | 2 × 2,244.86 × 4 = 17,958.9 trees/s | f6_v9/data_v14_dv.json | simulated + modelled + assumption (4×) |
| Qwen verify time at the memory-wall V (B=8 → V=10) | 22.78 ms | vbound V2D(B, V), fitted to measured V20 p50 at bs 1-16 | modelled (fit to measured) |
| Host fixed cost P, per-request cost h | 9.51 ms, 0.732 ms | vbound sgl_host_fit (least squares on measured SGLang runs) | modelled (fit to measured) |
| h halved by the 2× misc | 0.366 ms | user ruling 2026-11-19 | assumption |
Worked example, Qwen, B=8:
- Period = 22.78 + 9.51 + 0.366 × 8 = 35.22 ms, so one H200 needs 8 ÷ 0.03522 s = 227.1 trees/s.
- Realistic: 17,958.9 ÷ 227.1 = 79.06 H200s = 9.88 nodes.
- Verify only: 8 ÷ 0.02278 s = 351.2 trees/s, and 17,958.9 ÷ 351.2 = 51.14 H200s.
- Against five nodes: 79.06 ≥ 40, so the talon's card throughput covers all 40 H200s at B=8.
- Served count (step 4): with 40 H200s, card duty per set must be ≤ 2/40 = 0.05. The W2 V10 schedule has duty 0.0394, so the talon feeds all 40.
4. Five nodes serve more tokens: talon + 40 H200 gives ×1.34 to ×1.91 the tokens/s of the same 40 H200s alone on Qwen, ×1.20 to ×1.63 on Llama [P] projected.


Takeaway: the talon feeds as many of the 40 H200s as its duty allows: all 40 on Qwen at B=2 to 16, and 6 to 24 on Llama. The rest run EAGLE on their own.
How step 4 is computed (40 H200)
Formulas:
- 40 H200 alone = 40 × H200-alone tok/s(B). B=1 is measured; B≥2 comes from the vbound model fitted to measured verify times.
- talon + 40 H200 = n × per-fed tok/s + (40 − n) × H200-alone, where per-fed = B × accept ÷ period_talon.
- n = the number of H200s the talon feeds, chosen so card duty per set ≤ 2 ÷ n (best total).
- accept = 10.42 × width factor × (V/20)^0.13, with width factor 1.000 / 1.040 / 1.046 for W = 2 / 3 / 4.
- Per user = aggregate ÷ (40 × B).
| Input | Value at B=8 | Source | Tag |
|---|---|---|---|
| Base accept, W2 V20, batch 1 | 10.42 | measured Qwen serve | measured |
| Accept elasticity in V | 0.13 | measured on the Qwen mt_bench analogue (vbound.py:45-49); the humaneval value is not measured | modelled |
| Memory wall: B × (1+V) ≤ 440 on one H200 | V = 10 at B=8 | H200 memory | modelled |
| H200 alone: step, accept | 53.43 ms, 9.53 | vbound gpu_step | modelled (fit to measured) |
| Talon-fed config with duty ≤ 2/40 = 0.05 | W2 V10, period 27.87 ms, duty per set 0.0394 | vbound fpga_period, card time ÷ 4 (f6_v9/compute_v16.py) | simulated + modelled + assumption |
Worked example, Qwen, B=8, 40 H200:
- 40 H200 alone: 40 × 1,427.19 = 57,087.5 tok/s.
- Talon-fed H200: accept = 10.42 × 0.5^0.13 = 9.522, and 8 × 9.522 ÷ 0.02787 s = 2,732.9 tok/s. The talon feeds all n = 40: 40 × 2,732.87 = 109,314.9 tok/s.
- Gain: ×1.91. Per user: 109,314.9 ÷ 320 = 341.6 vs 178.4 tok/s.
5. One talon, three to five nodes: with 5 x 8xH200 nodes one talon reaches up to 183,660 tok/s on Qwen and 324,650 on Llama (×1.34 and ×1.39 at batch 128) [P] projected.


Takeaway: at five nodes the gain is ×1.34 to ×1.91 on Qwen and ×1.20 to ×1.63 on Llama. Fewer nodes per talon gives a larger gain per node (three nodes: ×1.44 to ×1.92 Qwen, ×1.34 to ×2.05 Llama). Host and link are treated as not binding.
How step 5 is computed
Formula: for N nodes (8N H200), the total is n × per-fed tok/s + (8N − n) × H200-alone, the same mixed model as step 4. Here n ≤ 8N and card duty per set ≤ 2 ÷ n.
| Input | Value at B=8 | Source | Tag |
|---|---|---|---|
| Per-fed tok/s, 3 or 4 nodes (duty cap 2/24, 2/32) | W3 V10: 2,736.88 | f6_v9/data_v16.json c2 | simulated + modelled + assumption |
| Per-fed tok/s, 5 nodes (duty cap 2/40) | W2 V10: 2,732.87 | f6_v9/data_v16.json c2 | simulated + modelled + assumption |
| H200 alone tok/s | 1,427.19 | vbound | modelled (fit to measured) |
| Duty interleave (idle card time drafts for other H200s) | user premise | assumption |
Worked example, Qwen, B=8:
- 3 nodes: 24 × 2,736.88 = 65,685.0 vs 24 × 1,427.19 = 34,252.5, ×1.92.
- 4 nodes: 32 × 2,736.88 = 87,580.0 vs 45,670.0, ×1.92.
- 5 nodes: 40 × 2,732.87 = 109,314.9 vs 57,087.5, ×1.91.
6. Same cost, more throughput: as the drafter for five 8xH200 nodes (40 verify H200) the talon gives ×1.01 to ×1.96 the tokens/s of 2xH200 drafters on Qwen, ×1.60 to ×2.35 on Llama [P] projected.


Takeaway: "2 H200" means two H200s used only as drafters. Equal cost (talon box = 2 H200) is a premise; no prices are used. On Qwen at small batch the verify side binds, so both drafters nearly tie; the talon pulls ahead from batch 16.
How step 6 is computed (40 verify H200)
Formula: tok/s = min(verify-limited tree rate, draft capacity) × accept.
- Verify-limited rate = 40 × B ÷ period.
- Draft capacity:
- talon: its supply at 40 × B trees;
- 2 H200: 2 × DBENCH at 40 × B ÷ 2, held at the measured 128-tree value beyond it.
- The host per-request cost h is halved on the talon side only.
| Input | Value at B=8 | Source | Tag |
|---|---|---|---|
| Verify time, host P, h | 22.78 ms, 9.51 ms, 0.732 ms | vbound (fit to measured) | modelled |
| Accept, W2 V10 | 10.42 × 0.5^0.13 = 9.522 | step 4 formula | modelled |
| Talon draft capacity, 320 trees | 19,865.1 trees/s | timing model × 4 | simulated + modelled + assumption |
| 2 H200 draft capacity, 320 trees | 2 × 5,063.02 = 10,126.04 trees/s | DBENCH.tsv, Qwen, 128 trees | measured |
| Equal cost, talon box = 2 H200 | premise | assumption |
Worked example, Qwen, B=8, 40 verify H200s:
- talon: period 22.78 + 9.51 + 0.366 × 8 = 35.22 ms, so the need is 40 × 8 ÷ 0.03522 s = 9,086.1 trees/s. That is below the 19,865.1 capacity, so verify binds: 9,086.1 × 9.522 = 86,519.2 tok/s.
- 2 H200: period 22.78 + 9.51 + 0.732 × 8 = 38.15 ms, so the need is 8,388.2 trees/s. That is below the 10,126.04 capacity, so verify binds: 8,388.2 × 9.522 = 79,874.2 tok/s.
- Ratio: ×1.08.
What is real and what is projected
- Measured: the batch-1 numbers (step 1); the H200 baselines and the H200 drafter bench; today's cards drafting at 335 trees/s (Qwen) and 319 trees/s (Llama).
- Simulated + modelled: the batched talon redesign, every talon point from step 2 on [P] projected. It rests on cycle-accurate simulated units and a calibrated system model, and is not yet on silicon.
- Assumption: the 4× card-time speedup = 2× drafter clock (66→132 MHz, not yet closed) × 2× misc (a placeholder factor, not a specific change).
- Premise: equal cost (one talon box = 2 H200), the duty interleave, and five 8xH200 nodes (40 H200) per talon. No prices are used.
- Qwen H200 memory wall: one H200 holds only B × (1+V) ≤ 440, so fewer tokens are verified per request at large batch, on both sides.
Thank you
EAGLE talon: let the FPGA draft, let the H200 verify.
- Live analysis: hardware.research.yshicarrick.com/p/eagle4-talon-vs-h200-eagle-20261119/
- Showcase: hardware.research.yshicarrick.com/p/eagle4-talon-showcase-20261119/
- This presentation: hardware.research.yshicarrick.com/p/eagle4-presentation-20261119/
Carrick Shi · EAGLE · 2026-11-19

