# EAGLE talon: one talon drafting for five 8xH200 nodes

**Carrick Shi** · EAGLE · 2026-11-19

Downloads: [PDF (presentation with breakdowns)](https://hardware.research.yshicarrick.com/p/eagle4-presentation-20261119/#download) · [Markdown](https://hardware.research.yshicarrick.com/p/eagle4-presentation-20261119/talon_pitch_5nodes_20261119.md) · [All chart data (CSV)](https://hardware.research.yshicarrick.com/p/eagle4-presentation-20261119/talon_pitch_5nodes_all_chart_data_20261119.csv)

Setup: one talon (8x V80) + five 8xH200 nodes = 40 H200. Step 1 is measured on silicon. Steps 2 to 6 are projections of the redesigned talon and carry the tag [P].

---

## The pitch: let the FPGA draft, let the H200 verify

- **Speculative decoding** = **drafting**, where a small model guesses ~20 tokens, plus **verifying**, where the big model on the H200 checks them all at once.
- **Today the H200 does both.**
- **The talon** is a box of 8 AMD V80 FPGA cards. It takes over the drafting, so the H200 only verifies.

Three strengths:

1. **Faster per user.** Measured on silicon today.
2. **Mostly idle.** The talon drafts so fast that it sits idle for much of each step: today's cards are ~30% busy.
3. **One talon, many H200s.** The talon can take turns drafting for many H200s.

---

## Definitions

- **Node** = one 8×H200 server (8 H200 GPUs).
- **5 nodes** = 40 H200 GPUs. This deck is about one talon serving five nodes.
- **talon** = one box of 8 AMD V80 FPGA cards that drafts for the H200s. The H200s only verify.
- **B** = batch per H200, the number of requests each H200 serves at once. Five nodes at batch B serve 40 × B users.
- **tok/s** = accepted (output) tokens per second.
- **[P]** = projected, from the simulated and modelled redesign. Everything else is measured.
- **Qwen** = Qwen3.8-27B target with the BitNet drafter. **Llama** = Llama-3.2-3B target with the EAGLE4 2-bit drafter.

---

## 1. Today, measured: at batch 1 talon + H200 delivers 324.08 tok/s vs 289.49 for the H200 alone on Qwen (×1.12), and 1,000.67 vs 696.42 on Llama (×1.44).

![Qwen3.8-27B batch 1 measured](charts_5nodes/step1_measured_b1_qwen.png)
![Llama-3.2-3B batch 1 measured](charts_5nodes/step1_measured_b1_llama.png)

Data: [CSV (both models)](charts_5nodes/step1_measured_b1_qwen_llama.csv)

**Takeaway:** the output matches the H200 alone token for token, and accept length is the same (10.42 vs 10.46, 6.91 vs 6.90), so the whole gain comes from faster drafting.

---

## How step 1 is computed

**Formula:** tok/s = accepted tokens ÷ wall time, over humaneval 80 prompts × 256 new tokens, greedy. Llama reports the mean of 3 rounds.

| Input | Value | Source | Tag |
|---|---|---|---|
| Llama, H200 + 6 V80, 3 rounds | 1,003.84 / 998.49 / 999.68 tok/s (19,615 tokens each) | WINDOW_RESULTS_20261119.md:96-100 | [measured] |
| Llama, H200 alone | 696.42 tok/s | HB_CALIBRATION.md:158 | [measured] |
| Qwen, H200 + 4 V80 | 324.08 tok/s, accept 10.42 (11-18 run, the number on the post) | WINDOW_RESULTS_20261119.md:185 | [measured] |
| Qwen, H200 + 4 V80, 11-19 retest | 324.74 tok/s = 15,510 tokens ÷ 47.76 s, 80/80 replies identical | WINDOW_RESULTS_20261119.md:190 | [measured] |
| Qwen, H200 alone | 289.49 tok/s, accept 10.46 | site data.json qwen.h200 | [measured] |

**Worked example:** nothing was measured at B=8, so this example uses batch 1. Llama round 1: 19,615 tokens ÷ 19.54 s = 1,003.84 tok/s. The mean of the 3 rounds is 1,000.67, and 1,000.67 ÷ 696.42 = ×1.44. Qwen: 324.08 ÷ 289.49 = ×1.12; the retest gives 15,510 ÷ 47.76 s = 324.74, also ×1.12.

---

## 2. The drafter is fast: in pure drafting the talon (8x V80) out-drafts two H200 drafters by ×24.66 at batch 1, still ×2.61 at batch 128 [P].

![Qwen pure drafting](charts_5nodes/step2_pure_drafting_qwen.png)
![Llama pure drafting](charts_5nodes/step2_pure_drafting_llama.png)

Data: [Qwen CSV](charts_5nodes/step2_pure_drafting_qwen.csv) · [Llama CSV](charts_5nodes/step2_pure_drafting_llama.csv)

**Takeaway:** these are drafting rates alone, with no verification. The H200 line is a measured drafter bench, and the H200 only catches up at very large batch (Llama: ×23.70 at 1, ×2.61 at 128).

---

## How step 2 is computed

**Formulas:**

- 2 H200 = 2 × DBENCH trees/s at B/2. At B=1, one H200 drafts.
- talon = 2 sets × card-only set rate at B/2 × 4.
- Set rate = trees per set ÷ card time for those trees.

| Input | Value | Source | Tag |
|---|---|---|---|
| H200 drafter bench (DBENCH), depth 15 width 2, 29 nodes per tree, context 566, CUDA graph | Qwen, 4 trees: 409.55 trees/s | drafter_only/results_20261119/DBENCH.tsv (md5 def2c762) | [measured] |
| Card time per tree batch on one 4-V80 set, at 66 MHz | per-unit xsim cycle counts (ATT item, MLPV F6 947.6 cycles/node, G8 v2) | hbcal timing model (scripts/hbcal) | [simulated] + [modelled] |
| Set rate at 4 trees, 66 MHz | 1,822.28 trees/s (4 trees per 2.195 ms of card time) | f6_v9/data_v9.json red.Q | [modelled] |
| 4× card-time speedup | 2× drafter clock (66→132 MHz, not yet closed) × 2× misc (placeholder) | user ruling 2026-11-19 | [assumption] |

**Worked example, Qwen, B=8 trees in flight:**

- 2 H200: 2 × 409.55 = **819.10 trees/s**.
- talon: 2 sets × 1,822.28 × 4 = **14,578.2 trees/s**.
- Ratio: 14,578.2 ÷ 819.10 = **×17.80**.

---

## 3. One talon keeps many H200s drafting: its card throughput covers all 40 H200s of five 8xH200 nodes up to batch 32 on Qwen and 2 on Llama; in the serving model of step 4 it feeds 33 / 40 / 40 / 40 / 40 / 33 / 18 / 13 of 40 on Qwen and 10 / 24 / 22 / 18 / 13 / 9 / 8 / 6 on Llama at B = 1..128 [P].

![Qwen H200s fed](charts_5nodes/step3_h200s_fed40_qwen.png)
![Llama H200s fed](charts_5nodes/step3_h200s_fed40_llama.png)

Data: [Qwen CSV](charts_5nodes/step3_h200s_fed40_qwen.csv) · [Llama CSV](charts_5nodes/step3_h200s_fed40_llama.csv)

**Takeaway:** there are two counts.

- **Card throughput** (this chart): Qwen 449 → 15 H200s, Llama 101 → 9. Above the 40-H200 line, the talon covers all five nodes.
- **Served** (step 4): fewer, because card duty per set caps the count at floor(2 ÷ duty). Unfed H200s draft on their own GPU.

---

## How step 3 is computed

**Formula:** H200s fed = talon trees/s ÷ (B ÷ period per H200). Each request takes one tree per step.

- Realistic period = verify(B, V) + host, where host = P 9.51 ms + h·B with h = 0.73 ms halved to 0.366 ms.
- Worst-case period = verify only.

| Input | Value at B=8 | Source | Tag |
|---|---|---|---|
| talon supply = 2 sets × set rate at 4·B trees each × 4 | 2 × 2,244.86 × 4 = 17,958.9 trees/s | f6_v9/data_v14_dv.json | [simulated] + [modelled] + [assumption] (4×) |
| Qwen verify time at the memory-wall V (B=8 → V=10) | 22.78 ms | vbound V2D(B, V), fitted to measured V20 p50 at bs 1-16 | [modelled] (fit to measured) |
| Host fixed cost P, per-request cost h | 9.51 ms, 0.732 ms | vbound sgl_host_fit (least squares on measured SGLang runs) | [modelled] (fit to measured) |
| h halved by the 2× misc | 0.366 ms | user ruling 2026-11-19 | [assumption] |

**Worked example, Qwen, B=8:**

- Period = 22.78 + 9.51 + 0.366 × 8 = 35.22 ms, so one H200 needs 8 ÷ 0.03522 s = 227.1 trees/s.
- Realistic: 17,958.9 ÷ 227.1 = **79.06 H200s = 9.88 nodes**.
- Verify only: 8 ÷ 0.02278 s = 351.2 trees/s, and 17,958.9 ÷ 351.2 = **51.14 H200s**.
- Against five nodes: 79.06 ≥ 40, so the talon's card throughput covers all 40 H200s at B=8.
- Served count (step 4): with 40 H200s, card duty per set must be ≤ 2/40 = 0.05. The W2 V10 schedule has duty 0.0394, so the talon feeds **all 40**.

---

## 4. Five nodes serve more tokens: talon + 40 H200 gives ×1.34 to ×1.91 the tokens/s of the same 40 H200s alone on Qwen, ×1.20 to ×1.63 on Llama [P].

![Qwen 40 H200 aggregate](charts_5nodes/step4_nodes40_agg_qwen.png)
![Llama 40 H200 aggregate](charts_5nodes/step4_nodes40_agg_llama.png)

Data: [Qwen CSV](charts_5nodes/step4_nodes40_agg_qwen.csv) · [Llama CSV](charts_5nodes/step4_nodes40_agg_llama.csv)

**Takeaway:** the talon feeds as many of the 40 H200s as its duty allows: all 40 on Qwen at B=2 to 16, and 6 to 24 on Llama. The rest run EAGLE on their own.

---

## 4b. Per user: every user on the five nodes gets a faster reply [P].

![Qwen 40 H200 per user](charts_5nodes/step4_nodes40_user_qwen.png)
![Llama 40 H200 per user](charts_5nodes/step4_nodes40_user_llama.png)

Data: [Qwen CSV](charts_5nodes/step4_nodes40_user_qwen.csv) · [Llama CSV](charts_5nodes/step4_nodes40_user_llama.csv)

**Takeaway:** per user = aggregate ÷ (40 × B users). The Qwen curve falls after B=32 on both lines because the H200 memory wall forces fewer verified tokens per request.

---

## How step 4 is computed (40 H200)

**Formulas:**

- 40 H200 alone = 40 × H200-alone tok/s(B). B=1 is measured; B≥2 comes from the vbound model fitted to measured verify times.
- talon + 40 H200 = n × per-fed tok/s + (40 − n) × H200-alone, where per-fed = B × accept ÷ period_talon.
- n = the number of H200s the talon feeds, chosen so card duty per set ≤ 2 ÷ n (best total).
- accept = 10.42 × width factor × (V/20)^0.13, with width factor 1.000 / 1.040 / 1.046 for W = 2 / 3 / 4.
- Per user = aggregate ÷ (40 × B).

| Input | Value at B=8 | Source | Tag |
|---|---|---|---|
| Base accept, W2 V20, batch 1 | 10.42 | measured Qwen serve | [measured] |
| Accept elasticity in V | 0.13 | measured on the Qwen mt_bench analogue (vbound.py:45-49); the humaneval value is not measured | [modelled] |
| Memory wall: B × (1+V) ≤ 440 on one H200 | V = 10 at B=8 | H200 memory | [modelled] |
| H200 alone: step, accept | 53.43 ms, 9.53 | vbound gpu_step | [modelled] (fit to measured) |
| Talon-fed config with duty ≤ 2/40 = 0.05 | W2 V10, period 27.87 ms, duty per set 0.0394 | vbound fpga_period, card time ÷ 4 (f6_v9/compute_v16.py) | [simulated] + [modelled] + [assumption] |

**Worked example, Qwen, B=8, 40 H200:**

- 40 H200 alone: 40 × 1,427.19 = **57,087.5 tok/s**.
- Talon-fed H200: accept = 10.42 × 0.5^0.13 = 9.522, and 8 × 9.522 ÷ 0.02787 s = 2,732.9 tok/s. The talon feeds all n = 40: 40 × 2,732.87 = **109,314.9 tok/s**.
- Gain: **×1.91**. Per user: 109,314.9 ÷ 320 = **341.6** vs 178.4 tok/s.

---

## 5. One talon, three to five nodes: with 5 x 8xH200 nodes one talon reaches up to 183,660 tok/s on Qwen and 324,650 on Llama (×1.34 and ×1.39 at batch 128) [P].

![Qwen 3/4/5 nodes](charts_5nodes/step5_nodes345_qwen.png)
![Llama 3/4/5 nodes](charts_5nodes/step5_nodes345_llama.png)

Data: [Qwen CSV](charts_5nodes/step5_nodes345_qwen.csv) · [Llama CSV](charts_5nodes/step5_nodes345_llama.csv)

**Takeaway:** at five nodes the gain is ×1.34 to ×1.91 on Qwen and ×1.20 to ×1.63 on Llama. Fewer nodes per talon gives a larger gain per node (three nodes: ×1.44 to ×1.92 Qwen, ×1.34 to ×2.05 Llama). Host and link are treated as not binding.

---

## How step 5 is computed

**Formula:** for N nodes (8N H200), the total is n × per-fed tok/s + (8N − n) × H200-alone, the same mixed model as step 4. Here n ≤ 8N and card duty per set ≤ 2 ÷ n.

| Input | Value at B=8 | Source | Tag |
|---|---|---|---|
| Per-fed tok/s, 3 or 4 nodes (duty cap 2/24, 2/32) | W3 V10: 2,736.88 | f6_v9/data_v16.json c2 | [simulated] + [modelled] + [assumption] |
| Per-fed tok/s, 5 nodes (duty cap 2/40) | W2 V10: 2,732.87 | f6_v9/data_v16.json c2 | [simulated] + [modelled] + [assumption] |
| H200 alone tok/s | 1,427.19 | vbound | [modelled] (fit to measured) |
| Duty interleave (idle card time drafts for other H200s) | | user premise | [assumption] |

**Worked example, Qwen, B=8:**

- 3 nodes: 24 × 2,736.88 = **65,685.0** vs 24 × 1,427.19 = 34,252.5, **×1.92**.
- 4 nodes: 32 × 2,736.88 = **87,580.0** vs 45,670.0, **×1.92**.
- 5 nodes: 40 × 2,732.87 = **109,314.9** vs 57,087.5, **×1.91**.

---

## 6. Same cost, more throughput: as the drafter for five 8xH200 nodes (40 verify H200) the talon gives ×1.01 to ×1.96 the tokens/s of 2xH200 drafters on Qwen, ×1.60 to ×2.35 on Llama [P].

![Qwen equal cost, 40 verify H200](charts_5nodes/step6_equal_cost40_qwen.png)
![Llama equal cost, 40 verify H200](charts_5nodes/step6_equal_cost40_llama.png)

Data: [Qwen CSV](charts_5nodes/step6_equal_cost40_qwen.csv) · [Llama CSV](charts_5nodes/step6_equal_cost40_llama.csv)

**Takeaway:** "2 H200" means two H200s used only as drafters. Equal cost (talon box = 2 H200) is a premise; no prices are used. On Qwen at small batch the verify side binds, so both drafters nearly tie; the talon pulls ahead from batch 16.

---

## How step 6 is computed (40 verify H200)

**Formula:** tok/s = min(verify-limited tree rate, draft capacity) × accept.

- Verify-limited rate = 40 × B ÷ period.
- Draft capacity:
    - talon: its supply at 40 × B trees;
    - 2 H200: 2 × DBENCH at 40 × B ÷ 2, held at the measured 128-tree value beyond it.
- The host per-request cost h is halved on the talon side only.

| Input | Value at B=8 | Source | Tag |
|---|---|---|---|
| Verify time, host P, h | 22.78 ms, 9.51 ms, 0.732 ms | vbound (fit to measured) | [modelled] |
| Accept, W2 V10 | 10.42 × 0.5^0.13 = 9.522 | step 4 formula | [modelled] |
| Talon draft capacity, 320 trees | 19,865.1 trees/s | timing model × 4 | [simulated] + [modelled] + [assumption] |
| 2 H200 draft capacity, 320 trees | 2 × 5,063.02 = 10,126.04 trees/s | DBENCH.tsv, Qwen, 128 trees | [measured] |
| Equal cost, talon box = 2 H200 | | premise | [assumption] |

**Worked example, Qwen, B=8, 40 verify H200s:**

- talon: period 22.78 + 9.51 + 0.366 × 8 = 35.22 ms, so the need is 40 × 8 ÷ 0.03522 s = 9,086.1 trees/s. That is below the 19,865.1 capacity, so verify binds: 9,086.1 × 9.522 = **86,519.2 tok/s**.
- 2 H200: period 22.78 + 9.51 + 0.732 × 8 = 38.15 ms, so the need is 8,388.2 trees/s. That is below the 10,126.04 capacity, so verify binds: 8,388.2 × 9.522 = **79,874.2 tok/s**.
- Ratio: **×1.08**.

---

## What is real and what is projected

- **Measured:** the batch-1 numbers (step 1); the H200 baselines and the H200 drafter bench; today's cards drafting at 335 trees/s (Qwen) and 319 trees/s (Llama).
- **Simulated + modelled:** the batched talon redesign, every talon point from step 2 on [P]. It rests on cycle-accurate simulated units and a calibrated system model, and is **not yet on silicon**.
- **Assumption:** the 4× card-time speedup = 2× drafter clock (66→132 MHz, not yet closed) × 2× misc (a placeholder factor, not a specific change).
- **Premise:** equal cost (one talon box = 2 H200), the duty interleave, and five 8xH200 nodes (40 H200) per talon. No prices are used.
- **Qwen H200 memory wall:** one H200 holds only B × (1+V) ≤ 440, so fewer tokens are verified per request at large batch, on both sides.

---

## Thank you

**EAGLE talon: let the FPGA draft, let the H200 verify.**

- Live analysis: [hardware.research.yshicarrick.com/p/eagle4-talon-vs-h200-eagle-20261119/](https://hardware.research.yshicarrick.com/p/eagle4-talon-vs-h200-eagle-20261119/)
- Showcase: [hardware.research.yshicarrick.com/p/eagle4-talon-showcase-20261119/](https://hardware.research.yshicarrick.com/p/eagle4-talon-showcase-20261119/)
- This presentation: [hardware.research.yshicarrick.com/p/eagle4-presentation-20261119/](https://hardware.research.yshicarrick.com/p/eagle4-presentation-20261119/)

Carrick Shi · EAGLE · 2026-11-19
