Eagle Companion Chip · Evaluation stage

The draft model lives on the chip. The GPU just verifies.

A speculative-decoding draft accelerator built on three ideas: weights read straight from on-chip memory every clock, activations that never leave the chip between stages or tree levels, and RMSNorm done in dedicated hardware. Your GPU keeps doing what it does best: verifying tokens for the target model.

1.33×vanilla H200
HumanEval; 1.15× on MT-Bench, batch 1 derived
331tok/s
MT-Bench, batch 1, H200 target measured
383tok/s
HumanEval, batch 1 measured
~40W / card
accelerator power while decoding measured
Why it's fast

Three design choices, all in hardware

The drafter runs as a fixed hardware pipeline rather than as kernels on a GPU. Three choices make that pipeline fast.

01 · Weights

Weight reads straight from on-chip SRAM

All six 2-bit matrix stages (QKV, o-projection, gate, up, down and the full-vocabulary cheap head) are resident in on-chip SRAM, loaded once per session. The compute lanes read their weights directly every clock, with no DRAM or HBM fetch. The whole 41.58 MiB body fits on-chip: during a forward pass, no weight bit moves.1

02 · Activations

No round trip to re-dispatch activations

Activations never touch memory between stages: every seam is an on-chip register file. The tree control loop is in hardware too. The surviving branch's hidden state seeds the next tree level on-chip, so the architecture needs no host round trip between tree levels.2

03 · Normalization

Hardware-accelerated RMSNorm

The input, post-attention and final RMSNorm stages run in dedicated fixed-point hardware, inline in the pipeline, with Q4.12 gains held on-chip. No software or GPU pass is involved. Numerics are engineered stage by stage, and the output is verified identical to the reference.

More strengths

StrengthIn one lineBasis
Whole draft tree per launch8-node frontier × 8 children, global top-8 of 64 per level; log₂ cumulative scores that don't underflow at depth 15+architecture
LosslessOutput identical to the reference on every measured run (50/50 prompts). The drafter only proposes; the GPU verifies.measured
Accept parity96.8–101.5% of a GPU-hosted drafter's acceptance length, out to depth 15, under a matched protocolmeasured
2-bit where cheap, exact where it mattersGroup-128 2-bit matrices, plus an exact fp16 rescore of the candidate shortlist against the full vocabulary head in HBMarchitecture
Zero-copy tree contextEach branch gets its own KV rows through write-position addressing, so contexts are never duplicatedarchitecture
KV import from the GPUThe card takes the GPU's key/value rows instead of recomputing them: ≥ 99.4% of card-primed acceptance, 1.74× faster than primingmeasured
Adaptive depthEarly stop by commit mass: +10.3% at depth 7 (326.8 vs 296.4 tok/s)measured
Low power~37–40 W per card while decoding; about 0.68 J/token for the whole system at batch 1measured / derived
Frees the GPUDrafting is offloaded, so the GPU only verifies; two streams on one card pair reach 461 tok/s (batch 2 / aggregate)measured
Hardware-sequenced controlA state machine sequences the 15-phase forward and the tree loop, with a deterministic tie rule for selectionarchitecture
Scales out4 × QSFP56 (200G-capable) ports per card for direct links; 4 cards hold about 5× today's draft model (~0.9 B 2-bit weights)derived
Model can switchDrafter weights load at session start. See howarchitecture

In this table, the "architecture" label means a property of the design as built, verified in simulation and silicon bring-up; "derived" means arithmetic from measured or specified figures.

1 Scope: this covers the six matrix stages. The exact fp16 vocabulary head used to rescore the final shortlist is too large for on-chip residency in the prototype and is read from on-card HBM.
2 Scope: in today's two-card FPGA prototype, one card-to-card hop per item still passes through the host. A direct card-to-card QSFP link is in development, and the single-chip ASIC design removes the hop entirely.

What it is

A dedicated engine for the drafting half of speculative decoding

Speculative decoding pairs a small draft model with a large target model. The drafter proposes a tree of candidate tokens; the target model checks them in one pass. The Eagle Companion Chip takes the whole drafting side off the GPU.

01 · Weight-stationary

Draft model held on-chip

Every matrix of the drafter is stored at 2 bits in on-chip SRAM and loaded once per session. During a forward pass no weight bit moves.

02 · On-card tree

Draft tree built on the card

Candidate selection, cumulative scoring and the exact rescore of the shortlist all run on the accelerator. The host just supplies the root and collects the finished tree.

03 · GPU verifies

Lossless output

The GPU runs the target model and accepts a prefix of the draft, so the output is exactly what the target model alone would produce. Our tests check the text matches, prompt by prompt.

Model flexibility

The target model can switch

The accelerator is not tied to one model. Drafter weights are loaded into the card at session start, so switching target models means loading a different drafter. It does not mean new hardware.

  1. 1
    Train a drafter for the target. Each target model gets its own EAGLE-style drafter, trained against that model and quantized to 2 bits.
  2. 2
    Load it at session start. Weights and scales are runtime-loaded into on-chip memory. The low-rank vocabulary projection is runtime-loadable too; that path is verified in simulation.
  3. 3
    Serve through the usual engine. Any target model served through SGLang's EAGLE speculative-decoding path can be paired once its drafter is trained.

Tested today

Llama-3.2-3B-Instruct as the target, with our 2-bit drafter. Other target models need a drafter trained for them first. We have not tested other targets on the hardware yet.

Capacity

Today's drafter body is 41.58 MiB at 2 bits (one 3072-wide layer, Llama-3.2-3B class), and it fits on two cards with room to grow. Four cards would hold about 0.9 B 2-bit draft weights derived.

Measured on silicon

Results, with the conditions that produced them

Every number here was measured on our lab system and has the output text checked against the reference. They are absolute throughputs for the stated configuration, not a comparison claim.

Bars are measured tok/s measured; the ratios are vanilla vs Eagle runs from different sessions derived. The batch-2 bar was measured on an earlier card pair (A5+FA) with CUDA MPS.

Acceptance parity

Acceptance length stays within about 2–3% of a GPU-hosted drafter out to tree depth 15 measured. Moving the drafter onto the card does not cost accuracy.

Low accelerator power

Each card draws about 37–40 W while decoding measured, well under the card's 190 W rating. The GPU's power is separate.

Deeper trees now pay

An on-card early-stop rule lets depth-7 trees run as fast as depth-3 trees (+10% at depth 7 versus no stop) measured.

Test conditions

ItemSetting
Target modelLlama-3.2-3B-Instruct
Draft modelEagle 2-bit drafter (EAGLE-style), on 2 × AMD Alveo V80
Verifier GPU1 × NVIDIA H200
Serving engineSGLang 0.5.16, EAGLE speculative-decoding path
Batch1 (single stream). The batch-2 figure uses 2 streams with CUDA MPS, reported as aggregate.
BenchmarksMT-Bench, HumanEval; 50 prompts; greedy decoding
TreeMT-Bench record: width 2, depth 7 with on-card early stop and the host-side first-bell fixes; HumanEval: width 1, depth 4
BaselineVanilla H200: plain autoregressive decoding, same model and engine, no speculative decoding. MT-Bench 289.0, HumanEval 288.5 tok/s (3 rounds, 2026-09-30)
Correctnessoutput text identical to the reference, 50 / 50 prompts
DateMT-Bench record 2026-09-30; HumanEval 2026-09-29

Speed-ups are measured against plain autoregressive decoding on the same H200. GPU-hosted speculative decoding (EAGLE, same H200) measures 483 / 676 tok/s; Eagle offloads drafting from the GPU (see "Frees the GPU").

Only the H200 has been tested so far. Other GPUs, including RTX-class Blackwell cards, can be evaluated on request, since the GPU only needs to run target-model verification.

Hardware and software

What you install today

Form factor

ImplementationFPGA, on 2 × AMD Alveo V80
Cardsfull-height, dual-slot PCIe
Host linkPCIe Gen4 x16 / Gen5 x8x8
Networking4 × QSFP56 per card
Memory32 GB HBM2e per card
Power~37–40 W / card measured; 190 W rated
GPUseparate; runs the target model

Inference engines

We integrate through the engine's speculative-decoding hook plus a host library that drives the cards.

SGLang · supported vLLM · next target TensorRT-LLM · not supported

vLLM's speculative-decoding interface is close to SGLang's, which makes it the natural next integration.

Roadmap

From FPGA prototype to custom silicon

The FPGA design is the working prototype of the architecture. The custom ASIC is a roadmap item: it is not built today.

Now

FPGA evaluation

Two-card V80 system, SGLang integration, measured results above. Engineering prototype.

Roadmap

Custom silicon

An ASIC implementation of the same architecture, aimed at lower cost per token projected.

Live demo

See it run on the real hardware

A live demo is available on request, with controlled access. It runs on the same two-card system as the results above: you chat with the target model while the accelerator does the drafting.

What the demo shows

  • A chat interface served live by SGLang with the two-card drafter
  • Live telemetry for the GPU and both cards: temperatures and power
  • Card identity read live from the silicon, so you know which design is running
  • A serving on/off control for the session
  • Two-stream (batch-2) mode: planned

Demo panel illustrative

THROUGHPUT
331 tok/s
VS VANILLA
1.15×
CARD POWER
~40 W

Values shown are the MT-Bench batch-1 figures from the results section, not a live reading.

Why Eagle

What someone else would have to rebuild

Buying an FPGA or a GPU doesn't give you this. These strengths come from building the drafter, the numerics and the hardware together.

01 · Model–silicon co-design

2-bit drafting at GPU-level acceptance

A drafter quantized to 2 bits that keeps 96.8–101.5% of a GPU drafter's acceptance out to depth 15 measured. The quantization, integer softmax, log₂ tree scoring and exact rescore were designed together with the hardware. Matching it takes the drafter training, the numerics and the RTL together.

02 · A companion

Adds to the GPUs you already run

The accelerator sits next to the GPUs you already run. The GPU keeps verifying, so output stays lossless, and drafting moves to a small, low-power card. The target model is unchanged, and serving stays in SGLang with a host library.

03 · Drafting loop in hardware

A purpose-built drafting engine

The whole draft tree is built on-chip in one launch: the tree state machine, per-branch KV isolation by write-position addressing, and KV import from the GPU. It's a dedicated drafting engine, not a general matrix accelerator.

04 · Proven on silicon

Real hardware, working software path

Measured on real hardware: 331 tok/s at batch 1 (1.15–1.33× vanilla H200), 461 tok/s at batch 2 (aggregate), about 40 W per card measured. It is integrated with SGLang, and the output is verified identical to the reference.

05 · De-risked path to an ASIC

Validated before any tape-out spend

The architecture has been validated cycle-exact in simulation and on FPGA silicon across many design iterations, before any money is spent on custom silicon. The ASIC itself is on the roadmap.

Availability

Contact us for an evaluation or a demo session

We are at the engineering prototype and evaluation stage; there are no production units yet. Tell us which GPUs, target models and inference stack matter to you, and we will bring data for those.

Contact: via your Eagle representative.

Ask HQ about this page

For the project lead: ask for supporting data, a clarification or a re-test. The answerer can only read HQ records and queue a re-test for the lab; it cannot build, flash or change anything. Answers arrive within a few minutes.