EAGLEEAGLE talon presentation
EAGLEEAGLE

EAGLE talon: one talon drafting for five 8xH200 nodes

Carrick Shi · EAGLE · 2026-11-19

Downloads: PDF (presentation with breakdowns) · Markdown · All chart data (CSV)

Setup: one talon (8x V80) + five 8xH200 nodes = 40 H200. Step 1 is measured on silicon. Steps 2 to 6 are projections of the redesigned talon and carry the tag [P] projected.

EAGLEEAGLE talon1 / 20

One talon vs H200 nodes drafting themselves: Qwen3.8-27B, 1 / 3 / 4 / 5 nodes

Qwen 1/3/4/5 nodes

Data: CSV (64 rows, both models)

Takeaway: blue = H200 nodes drafting themselves; green = the same nodes with one talon (8x V80, 4x speedup) drafting; labels = gain. [P] projected

EAGLEEAGLE talon2 / 20

One talon vs H200 nodes drafting themselves: Llama-3.2-3B, 1 / 3 / 4 / 5 nodes

Llama 1/3/4/5 nodes

Data: CSV (64 rows, both models)

EAGLEEAGLE talon3 / 20
Where the data comes from and how it is calculated

How the 1 / 3 / 4 / 5-node charts are computed

Where the data comes from

  • H200 verify times, measured on talon2 (GPU-only verify table, p50): Qwen BitNet target at V20, bs 1/2/4/8/16 = 20.31 / 21.16 / 23.88 / 31.93 / 51.66 ms; Llama FA4 bs 1/8/16/32/64/128 = 3.46 / 4.69 / 7.04 / 10.91 / 20.98 / 40.05 ms (scripts/hbcal/out/calibration.json, V_gpu). measured
  • H200-alone serve at batch 1, measured: 289.49 tok/s (Qwen, accept 10.46) and 696.42 tok/s (Llama). measured
  • Qwen H200 memory wall, measured: bs 32 / 64 / 128 at V20 fail at SGLang init (EXIT 137, 11-19); the hybrid target holds ~0.152 GB x (1+V) per request, so B x (1+V) <= 440 and the verified tokens per request V shrink with batch. measured
  • Host cost: P 9.51 + h 0.73 x B ms per step, a least-squares fit on the measured SGLang H200 steps (step - verify - draft, bs 1-16); the talon side uses h / 2 (the '2x misc'). derived
  • Accept: 10.42 measured (Qwen W2 V20); its change with V uses elasticity 0.13 from a measured Qwen mt_bench analogue; wider trees use a measured Llama analogue. derived
  • Talon card time per tree: from bit-exact xsim per-unit cycle counts (attention item, MLPV fitting level F6, G8 v2) combined by the hbcal timing model; that model predicts today's Llama silicon within 2.9 % on runs it was not fitted on. projected
  • The 4x: 2x drafter clock (66 -> 132 MHz, not yet closed) x 2x misc (a placeholder factor) is an assumption. assumption

How each line is calculated

  • Blue = N_H200 x (one H200 running EAGLE alone at batch B), from the calibrated model: B x accept / (verify + draft + host step).
  • Green = (H200s the talon feeds) x (per-H200 tok/s with the talon) + (remaining H200s) x (alone tok/s).
  • Per-H200 tok/s with the talon = B x accept(V) / step period; period = verify(B, V) + talon-side host, under the overlap schedule (groups of requests alternate, so the H200 verify, the host and the card wave run side by side and the slowest sets the period).
  • H200s the talon feeds = min(N_H200, floor(2 / card duty per set)); card duty per set = card time per period / period (two 4-V80 sets).
  • V per B (Qwen) from the memory wall B x (1+V) <= 440: V = 20 / 14 / 10 / 10 / 10 / 10 / 5 / 2 at B = 1..128.

Worked example: Qwen, 3 nodes (24 H200), B = 32 (values from nodes1345_lines_std_20261119.csv and data_v16.json):

  • Memory wall: 32 x (1 + V) <= 440, so V = 10.
  • Blue, one H200 alone: 32 requests x accept 9.659 / step 96.05 ms = 3,217.9 tok/s; x 24 H200 = 77,230.
  • With the talon: tree W3 V10, accept 9.907; schedule G2 (two groups of 16 alternate); period = 2 x verify(16 requests, V10) = 2 x 31.20 = 62.40 ms (the H200 verify binds; host 9.51 + 0.366 x 16 ms and the card wave overlap it); per H200 = 32 x 9.907 / 62.40 ms = 5,080.3 tok/s.
  • Card duty per set = 5.35 ms / 62.40 ms = 0.0857, so the two sets feed floor(2 / 0.0857) = 23 H200 (of 24); the other 1 drafts on its own GPU.
  • Green = 23 x 5,080.3 + 1 x 3,217.9 = 120,066; gain = 120,066 / 77,230 = x1.55.
EAGLEEAGLE talon4 / 20

The pitch: let the FPGA draft, let the H200 verify

  • Speculative decoding = drafting, where a small model guesses ~20 tokens, plus verifying, where the big model on the H200 checks them all at once.
  • Today the H200 does both.
  • The talon is a box of 8 AMD V80 FPGA cards. It takes over the drafting, so the H200 only verifies.

Three strengths:

  1. Faster per user. Measured on silicon today.
  2. Mostly idle. The talon drafts so fast that it sits idle for much of each step: today's cards are ~30% busy.
  3. One talon, many H200s. The talon can take turns drafting for many H200s.
EAGLEEAGLE talon5 / 20

Definitions

  • Node = one 8×H200 server (8 H200 GPUs).
  • 5 nodes = 40 H200 GPUs. This deck is about one talon serving five nodes.
  • talon = one box of 8 AMD V80 FPGA cards that drafts for the H200s. The H200s only verify.
  • B = batch per H200, the number of requests each H200 serves at once. Five nodes at batch B serve 40 × B users.
  • tok/s = accepted (output) tokens per second.
  • [P] projected = projected, from the simulated and modelled redesign. Everything else is measured.
  • Qwen = Qwen3.8-27B target with the BitNet drafter. Llama = Llama-3.2-3B target with the EAGLE4 2-bit drafter.
EAGLEEAGLE talon6 / 20

1. Today, measured: at batch 1 talon + H200 delivers 324.08 tok/s vs 289.49 for the H200 alone on Qwen (×1.12), and 1,000.67 vs 696.42 on Llama (×1.44).

Qwen3.8-27B batch 1 measured
Llama-3.2-3B batch 1 measured

Data: CSV (both models)

Takeaway: the output matches the H200 alone token for token, and accept length is the same (10.42 vs 10.46, 6.91 vs 6.90), so the whole gain comes from faster drafting.

EAGLEEAGLE talon7 / 20
How this number is computed

How step 1 is computed

Formula: tok/s = accepted tokens ÷ wall time, over humaneval 80 prompts × 256 new tokens, greedy. Llama reports the mean of 3 rounds.

Input Value Source Tag
Llama, H200 + 6 V80, 3 rounds 1,003.84 / 998.49 / 999.68 tok/s (19,615 tokens each) WINDOW_RESULTS_20261119.md:96-100 measured
Llama, H200 alone 696.42 tok/s HB_CALIBRATION.md:158 measured
Qwen, H200 + 4 V80 324.08 tok/s, accept 10.42 (11-18 run, the number on the post) WINDOW_RESULTS_20261119.md:185 measured
Qwen, H200 + 4 V80, 11-19 retest 324.74 tok/s = 15,510 tokens ÷ 47.76 s, 80/80 replies identical WINDOW_RESULTS_20261119.md:190 measured
Qwen, H200 alone 289.49 tok/s, accept 10.46 site data.json qwen.h200 measured

Worked example: nothing was measured at B=8, so this example uses batch 1. Llama round 1: 19,615 tokens ÷ 19.54 s = 1,003.84 tok/s. The mean of the 3 rounds is 1,000.67, and 1,000.67 ÷ 696.42 = ×1.44. Qwen: 324.08 ÷ 289.49 = ×1.12; the retest gives 15,510 ÷ 47.76 s = 324.74, also ×1.12.

EAGLEEAGLE talon8 / 20

2. The drafter is fast: in pure drafting the talon (8x V80) out-drafts two H200 drafters by ×24.66 at batch 1, still ×2.61 at batch 128 [P] projected.

Qwen pure drafting
Llama pure drafting

Data: Qwen CSV · Llama CSV

Takeaway: these are drafting rates alone, with no verification. The H200 line is a measured drafter bench, and the H200 only catches up at very large batch (Llama: ×23.70 at 1, ×2.61 at 128).

EAGLEEAGLE talon9 / 20
How this number is computed

How step 2 is computed

Formulas:

  • 2 H200 = 2 × DBENCH trees/s at B/2. At B=1, one H200 drafts.
  • talon = 2 sets × card-only set rate at B/2 × 4.
  • Set rate = trees per set ÷ card time for those trees.
Input Value Source Tag
H200 drafter bench (DBENCH), depth 15 width 2, 29 nodes per tree, context 566, CUDA graph Qwen, 4 trees: 409.55 trees/s drafter_only/results_20261119/DBENCH.tsv (md5 def2c762) measured
Card time per tree batch on one 4-V80 set, at 66 MHz per-unit xsim cycle counts (ATT item, MLPV F6 947.6 cycles/node, G8 v2) hbcal timing model (scripts/hbcal) simulated + modelled
Set rate at 4 trees, 66 MHz 1,822.28 trees/s (4 trees per 2.195 ms of card time) f6_v9/data_v9.json red.Q modelled
4× card-time speedup 2× drafter clock (66→132 MHz, not yet closed) × 2× misc (placeholder) user ruling 2026-11-19 assumption

Worked example, Qwen, B=8 trees in flight:

  • 2 H200: 2 × 409.55 = 819.10 trees/s.
  • talon: 2 sets × 1,822.28 × 4 = 14,578.2 trees/s.
  • Ratio: 14,578.2 ÷ 819.10 = ×17.80.
EAGLEEAGLE talon10 / 20

3. One talon keeps many H200s drafting: its card throughput covers all 40 H200s of five 8xH200 nodes up to batch 32 on Qwen and 2 on Llama; in the serving model of step 4 it feeds 33 / 40 / 40 / 40 / 40 / 33 / 18 / 13 of 40 on Qwen and 10 / 24 / 22 / 18 / 13 / 9 / 8 / 6 on Llama at B = 1..128 [P] projected.

Qwen H200s fed
Llama H200s fed

Data: Qwen CSV · Llama CSV

Takeaway: there are two counts.

  • Card throughput (this chart): Qwen 449 → 15 H200s, Llama 101 → 9. Above the 40-H200 line, the talon covers all five nodes.
  • Served (step 4): fewer, because card duty per set caps the count at floor(2 ÷ duty). Unfed H200s draft on their own GPU.
EAGLEEAGLE talon11 / 20
How this number is computed

How step 3 is computed

Formula: H200s fed = talon trees/s ÷ (B ÷ period per H200). Each request takes one tree per step.

  • Realistic period = verify(B, V) + host, where host = P 9.51 ms + h·B with h = 0.73 ms halved to 0.366 ms.
  • Worst-case period = verify only.
Input Value at B=8 Source Tag
talon supply = 2 sets × set rate at 4·B trees each × 4 2 × 2,244.86 × 4 = 17,958.9 trees/s f6_v9/data_v14_dv.json simulated + modelled + assumption (4×)
Qwen verify time at the memory-wall V (B=8 → V=10) 22.78 ms vbound V2D(B, V), fitted to measured V20 p50 at bs 1-16 modelled (fit to measured)
Host fixed cost P, per-request cost h 9.51 ms, 0.732 ms vbound sgl_host_fit (least squares on measured SGLang runs) modelled (fit to measured)
h halved by the 2× misc 0.366 ms user ruling 2026-11-19 assumption

Worked example, Qwen, B=8:

  • Period = 22.78 + 9.51 + 0.366 × 8 = 35.22 ms, so one H200 needs 8 ÷ 0.03522 s = 227.1 trees/s.
  • Realistic: 17,958.9 ÷ 227.1 = 79.06 H200s = 9.88 nodes.
  • Verify only: 8 ÷ 0.02278 s = 351.2 trees/s, and 17,958.9 ÷ 351.2 = 51.14 H200s.
  • Against five nodes: 79.06 ≥ 40, so the talon's card throughput covers all 40 H200s at B=8.
  • Served count (step 4): with 40 H200s, card duty per set must be ≤ 2/40 = 0.05. The W2 V10 schedule has duty 0.0394, so the talon feeds all 40.
EAGLEEAGLE talon12 / 20

4. Five nodes serve more tokens: talon + 40 H200 gives ×1.34 to ×1.91 the tokens/s of the same 40 H200s alone on Qwen, ×1.20 to ×1.63 on Llama [P] projected.

Qwen 40 H200 aggregate
Llama 40 H200 aggregate

Data: Qwen CSV · Llama CSV

Takeaway: the talon feeds as many of the 40 H200s as its duty allows: all 40 on Qwen at B=2 to 16, and 6 to 24 on Llama. The rest run EAGLE on their own.

EAGLEEAGLE talon13 / 20

4b. Per user: every user on the five nodes gets a faster reply [P] projected.

Qwen 40 H200 per user
Llama 40 H200 per user

Data: Qwen CSV · Llama CSV

Takeaway: per user = aggregate ÷ (40 × B users). The Qwen curve falls after B=32 on both lines because the H200 memory wall forces fewer verified tokens per request.

EAGLEEAGLE talon14 / 20
How this number is computed

How step 4 is computed (40 H200)

Formulas:

  • 40 H200 alone = 40 × H200-alone tok/s(B). B=1 is measured; B≥2 comes from the vbound model fitted to measured verify times.
  • talon + 40 H200 = n × per-fed tok/s + (40 − n) × H200-alone, where per-fed = B × accept ÷ period_talon.
  • n = the number of H200s the talon feeds, chosen so card duty per set ≤ 2 ÷ n (best total).
  • accept = 10.42 × width factor × (V/20)^0.13, with width factor 1.000 / 1.040 / 1.046 for W = 2 / 3 / 4.
  • Per user = aggregate ÷ (40 × B).
Input Value at B=8 Source Tag
Base accept, W2 V20, batch 1 10.42 measured Qwen serve measured
Accept elasticity in V 0.13 measured on the Qwen mt_bench analogue (vbound.py:45-49); the humaneval value is not measured modelled
Memory wall: B × (1+V) ≤ 440 on one H200 V = 10 at B=8 H200 memory modelled
H200 alone: step, accept 53.43 ms, 9.53 vbound gpu_step modelled (fit to measured)
Talon-fed config with duty ≤ 2/40 = 0.05 W2 V10, period 27.87 ms, duty per set 0.0394 vbound fpga_period, card time ÷ 4 (f6_v9/compute_v16.py) simulated + modelled + assumption

Worked example, Qwen, B=8, 40 H200:

  • 40 H200 alone: 40 × 1,427.19 = 57,087.5 tok/s.
  • Talon-fed H200: accept = 10.42 × 0.5^0.13 = 9.522, and 8 × 9.522 ÷ 0.02787 s = 2,732.9 tok/s. The talon feeds all n = 40: 40 × 2,732.87 = 109,314.9 tok/s.
  • Gain: ×1.91. Per user: 109,314.9 ÷ 320 = 341.6 vs 178.4 tok/s.
EAGLEEAGLE talon15 / 20

5. One talon, three to five nodes: with 5 x 8xH200 nodes one talon reaches up to 183,660 tok/s on Qwen and 324,650 on Llama (×1.34 and ×1.39 at batch 128) [P] projected.

Qwen 3/4/5 nodes
Llama 3/4/5 nodes

Data: Qwen CSV · Llama CSV

Takeaway: at five nodes the gain is ×1.34 to ×1.91 on Qwen and ×1.20 to ×1.63 on Llama. Fewer nodes per talon gives a larger gain per node (three nodes: ×1.44 to ×1.92 Qwen, ×1.34 to ×2.05 Llama). Host and link are treated as not binding.

EAGLEEAGLE talon16 / 20
How this number is computed

How step 5 is computed

Formula: for N nodes (8N H200), the total is n × per-fed tok/s + (8N − n) × H200-alone, the same mixed model as step 4. Here n ≤ 8N and card duty per set ≤ 2 ÷ n.

Input Value at B=8 Source Tag
Per-fed tok/s, 3 or 4 nodes (duty cap 2/24, 2/32) W3 V10: 2,736.88 f6_v9/data_v16.json c2 simulated + modelled + assumption
Per-fed tok/s, 5 nodes (duty cap 2/40) W2 V10: 2,732.87 f6_v9/data_v16.json c2 simulated + modelled + assumption
H200 alone tok/s 1,427.19 vbound modelled (fit to measured)
Duty interleave (idle card time drafts for other H200s) user premise assumption

Worked example, Qwen, B=8:

  • 3 nodes: 24 × 2,736.88 = 65,685.0 vs 24 × 1,427.19 = 34,252.5, ×1.92.
  • 4 nodes: 32 × 2,736.88 = 87,580.0 vs 45,670.0, ×1.92.
  • 5 nodes: 40 × 2,732.87 = 109,314.9 vs 57,087.5, ×1.91.
EAGLEEAGLE talon17 / 20

6. Same cost, more throughput: as the drafter for five 8xH200 nodes (40 verify H200) the talon gives ×1.01 to ×1.96 the tokens/s of 2xH200 drafters on Qwen, ×1.60 to ×2.35 on Llama [P] projected.

Qwen equal cost, 40 verify H200
Llama equal cost, 40 verify H200

Data: Qwen CSV · Llama CSV

Takeaway: "2 H200" means two H200s used only as drafters. Equal cost (talon box = 2 H200) is a premise; no prices are used. On Qwen at small batch the verify side binds, so both drafters nearly tie; the talon pulls ahead from batch 16.

EAGLEEAGLE talon18 / 20
How this number is computed

How step 6 is computed (40 verify H200)

Formula: tok/s = min(verify-limited tree rate, draft capacity) × accept.

  • Verify-limited rate = 40 × B ÷ period.
  • Draft capacity:
    • talon: its supply at 40 × B trees;
    • 2 H200: 2 × DBENCH at 40 × B ÷ 2, held at the measured 128-tree value beyond it.
  • The host per-request cost h is halved on the talon side only.
Input Value at B=8 Source Tag
Verify time, host P, h 22.78 ms, 9.51 ms, 0.732 ms vbound (fit to measured) modelled
Accept, W2 V10 10.42 × 0.5^0.13 = 9.522 step 4 formula modelled
Talon draft capacity, 320 trees 19,865.1 trees/s timing model × 4 simulated + modelled + assumption
2 H200 draft capacity, 320 trees 2 × 5,063.02 = 10,126.04 trees/s DBENCH.tsv, Qwen, 128 trees measured
Equal cost, talon box = 2 H200 premise assumption

Worked example, Qwen, B=8, 40 verify H200s:

  • talon: period 22.78 + 9.51 + 0.366 × 8 = 35.22 ms, so the need is 40 × 8 ÷ 0.03522 s = 9,086.1 trees/s. That is below the 19,865.1 capacity, so verify binds: 9,086.1 × 9.522 = 86,519.2 tok/s.
  • 2 H200: period 22.78 + 9.51 + 0.732 × 8 = 38.15 ms, so the need is 8,388.2 trees/s. That is below the 10,126.04 capacity, so verify binds: 8,388.2 × 9.522 = 79,874.2 tok/s.
  • Ratio: ×1.08.
EAGLEEAGLE talon19 / 20

What is real and what is projected

  • Measured: the batch-1 numbers (step 1); the H200 baselines and the H200 drafter bench; today's cards drafting at 335 trees/s (Qwen) and 319 trees/s (Llama).
  • Simulated + modelled: the batched talon redesign, every talon point from step 2 on [P] projected. It rests on cycle-accurate simulated units and a calibrated system model, and is not yet on silicon.
  • Assumption: the 4× card-time speedup = 2× drafter clock (66→132 MHz, not yet closed) × 2× misc (a placeholder factor, not a specific change).
  • Premise: equal cost (one talon box = 2 H200), the duty interleave, and five 8xH200 nodes (40 H200) per talon. No prices are used.
  • Qwen H200 memory wall: one H200 holds only B × (1+V) ≤ 440, so fewer tokens are verified per request at large batch, on both sides.
EAGLEEAGLE talon20 / 20

Thank you

EAGLE talon: let the FPGA draft, let the H200 verify.

Carrick Shi · EAGLE · 2026-11-19