Architecture update · 2026-10-07 · follows the 10-05 architecture report

EAGLE4 on four V80s: the variants we built, tested and chose

The EAGLE speculative-decoding drafter for Llama-3.2-3B, split across AMD Alveo V80 cards, with an H200 verifying. Since 10-03 the lab has four V80s. This post is a diff against the 10-05 architecture report: what happened to each 4-card variant, what we measured, and what we chose. The headline metric is batch-1 (single-stream) tok/s. Labels: M measured, D derived, P projected, each with its finding id.

339.12record pair, batch-1 tok/s MF0686
446.6–485.8GPU-only EAGLE, batch-1 tok/s MF0007, F0286
−30%SIBX-E 4-card vs 2-card, same host MF0845
366–368CL4 ring, chosen, untested PF0700, F0789
Where we stand, plainly. The FPGA pair has not beaten GPU-only EAGLE: 339.12 vs 446.6–485.8 tok/s at batch 1. The first measured 4-card design (SIBX-E) was slower than the pair. The 4-card chassis also still runs below the 10-01 record because of a firmware SMI storm MF0763, F0764, so 4-card results are compared only with a 2-card run on the same boot. All numbers here are batch 1; this post has no batch-2 / aggregate figures.

What changed since the 10-05 post

The 10-05 report covered the record pair (HX1 BD HEAD + TCLK75 TAIL) and five 4-card options: SIBX-E 2H2T over PCIe, the D2 card-side loop, the MP2 two-link HEAD, the CL4-L ring, and the SHAPE_RT runtime shape table. Two variants are new here: SG PAIR2 and HBM-streaming.

10-05 post → nowWhat changedWhyResultDecision
SIBX-E 2H2T over PCIeRe-measured on a healthy host (10-06), with a kit fix (v1.23stb3) that allows several 4-card runs per bootThe 10-05 number (50.00) came from a slowed chassis and could not judge the designX 2:7 114.35 vs P 2:7 163.47 tok/s, −30% MF0845Rejected
CL4-L ring (HEAD↔HEAD QSFP merge)Better-margin spare images (TAIL +0.018 / +0.009 ns); now the single 4-card candidateSIBX-E showed host merge + host KV cost more than the width split gainsFlash-ready, untested on silicon; PF0700, F0789 366–368Chosen (waits on 3 QSFP cables)
MP2 two-link HEADFirst clean hold-margin spare (HQ_MP2hm); re-scoped from a pair design to CL4's P2 transportP2 by itself adds 0 batch-1; its value is the HEAD↔HEAD port+0.001 / +0.010 ns MF0838Parked as a standalone; kept as CL4 transport
D2 card-side level loopBetter TAIL seed (TQ_D2s2); unchanged designMargin insurance; D2 is the sub-chain CL4 is built from+0.025 / +0.003 ns MF0835; 348.9 PF0644Queued for silicon (1 cable) as CL4's first step
SHAPE_RT 4-card (SG_CL4H/T)v4 HEAD failed; root cause = SLR-crossing broadcast + a dead pblock filter; three v5 fix arms buildingRuntime shapes are needed for 8B / Qwen-27B draftersv4 WNS −0.231 ns MF0865; v5 xsim cycle-exact MF0868Active
(new) SG PAIR2Shape-generic 2-card pair: SG_HEAD v3hm/v3hmx + SG_TAIL, timing PASS and stagedBring up the runtime shape table on 2 cards before 4+0.006 / +0.010 ns MF0864; 0 extra cycles on 3B MF0824, F0827Chosen (next silicon run)
(new) HBM-streamingStarted 10-07, design only: weights stream from HBM, URAM becomes a cacheURAM caps drafter size; 4 cards hold 128 GB of HBMNo numbers yet (design)Exploring
Record pair (baseline)Unchanged—339.12 tok/s MF0686Baseline
GPU-only EAGLE (reference)Unchanged—446.6–485.8 tok/s MF0007, F0286Reference; not yet beaten

Batch-1 numbers at a glance

0100200300400500GPU-only EAGLE (H200)GPU-only EAGLE (H200): 446.6–485.8 tok/s batch-1 [M] F0007, F0286446.6–485.8 [M]Record pair, 2-card chassis (10-01)Record pair, 2-card chassis (10-01): 339.12 tok/s batch-1 [M] F0686339.12 [M]Vanilla H200, no speculationVanilla H200, no speculation: 289 tok/s batch-1 [M] F0569289 [M]CL4 ring, 4 cardsCL4 ring, 4 cards: 366–368 tok/s batch-1 [P] F0700, F0789366–368 [P]D2 pairD2 pair: 348.9 tok/s batch-1 [P] F0644348.9 [P]2-card P 2:7, 4-card chassis (10-06)2-card P 2:7, 4-card chassis (10-06): 163.47 tok/s batch-1 [M] F0845163.47 [M]SIBX-E 4-card X 2:7 (10-06)SIBX-E 4-card X 2:7 (10-06): 114.35 tok/s batch-1 [M] F0845114.35 [M]
Batch-1 tok/s, mt_bench 50×32, Llama-3.2-3B target. Solid = measured on silicon, outlined = projected. The two 10-06 bars ran on the 4-card chassis, which still has a platform slowdown (see caveat), so compare them with each other, not with the 10-01 record.
HEAD cardTAIL cardhost / GPUPCIeQSFP28 card linkdata flow / merge

Baseline: the record pair

Measured · unchanged since 10-01
H200target verifyHost CPUrelay / KV / controlPCIeHEAD (HX1 BD)emb · QKV · attn · O · gate/upTAIL (TCLK75)down · LM · tree top-k · rescorePCIe x8PCIe x8seam 70 KB via hostOne level = HEAD item → host relay → TAIL item → host relay back. Host relay sits in every level.

One HEAD card (embedding, QKV, attention, O, gate/up GEMVs) and one TAIL card (SiLU, down projection, low-rank LM head, tree top-k, exact fp16 rescore), with the host relaying each draft level over PCIe. The H200 verifies.

  • Batch-1: 339.12 tok/s MF0686, TSP12QL tree, width 2, depth 7, accept length 3.62 MF0592.
  • Step 10.93 ms = FPGA chain ~5.4 ms (49%) + H200 verify ~3.6 ms (33%) + host glue ~1.9 ms (17%), all serial DF0602.
  • A cycle model of the same pair says the FPGA engines sit idle for ≥ 83% of the step PF0859. That is why every variant below is about removing host hops or overlap, not about more GEMV throughput.

Reference: GPU-only EAGLE

Measured · not yet beaten
H200target + EAGLE draft headHost CPUSGLang schedulerPCIeNo FPGA. Draft and verify share one GPU; no card hops, no host relay per level.

The honest comparison. The same EAGLE drafter on the H200 alone reaches 446.6–485.8 tok/s batch-1 MF0007, F0286. Vanilla decoding without speculation is 289.0 MF0569.

The FPGA pair is 1.17× vanilla but about 0.70–0.76× GPU-only EAGLE DF0686 / F0007, F0286. None of the 4-card projections below (≤ 368 PF0700) closes that gap either. The 4-card work is about learning how to scale out and about larger drafters; it is not a claim that FPGAs already win at 3B.

SIBX-E 2H2T: four cards over PCIe

Rejected · tested on silicon 10-06
H200target verifyHost CPUrelay / KV / controlPCIetop-W mergeon host, per levelHEAD Achain ATAIL ASIBX-E, even lanesHEAD Bchain BTAIL BSIBX-E, odd lanesLevel 0 runs on both chains; at level L ≥ 1 sibling j runs on chain j mod 2.Every seam, every KV row and the merge go through the host over PCIe; no card links.

Changed since 10-05: the 10-05 post showed the first valid 4-card run, 50.00 tok/s MF0819, on a chassis slowed 2–3.5× by a firmware SMI storm, plus a desktop service crash loop we later found and removed MF0833. After the fix, the same images and kit were re-run back to back:

ArmCell (width:depth)Batch-1 tok/sKV load
2-card P (same chassis)2:7163.47 (166.48 / 169.49 / 154.43) MF08451.80–1.89 ms
2-card P4:7136.29 MF0845—
4-card X2:7114.35, −30.05% vs P MF08453.12 ms
4-card X4:7~95 if it tracks the 2:7 ratio PF0853—
  • The design was projected at 348–357 PF0700. It measured 30% slower than the 2-card pair on the same host. The losses are the host-side merge per level and dual-chain KV loading through the host (3.12 vs 1.80 ms per load).
  • A TAIL status latch ("SIBX on-bit") stayed set after a 4-card run, so only one 4-card run per boot was possible. Kit v1.23stb3 clears it in software MF0849, F0851; confirmed on silicon MF0855. A 4:7 run was taken but is invalid: an unrelated workload loaded the shared host at the time MF0855.

Decision: rejected as a batch-1 design. The lesson carries over: at batch 1, a width split only pays if the merge stays off the host.

CL4-L ring: two D2 chains merged HEAD↔HEAD

Chosen · built, PASS, staged · untested (cables)
H200target verifyHost CPUrelay / KV / controlPCIeHEAD A (HQ_CL4)chain ATAIL A (TQ_CL4s2)even lanesHEAD B (HQ_CL4)chain BTAIL B (TQ_CL4s2)odd lanesP1P1P2 (HEAD↔HEAD)PART merge 16 B + 8 B/entryXITEM parent a, 6 KBPCIe: KV + control3 cables: A P1↔P1, B P1↔P1, HEAD A P2 ↔ HEAD B P2. Top-W merge happens on card, not on the host.

Changed since 10-05: design and images unchanged; it moved from "one option" to "the 4-card candidate" after SIBX-E. Insurance seeds landed with more margin:

  • HQ_CL4 HEAD: WNS +0.004 / WHS +0.010 ns MF0796, F0797; spare HQ_CL4s2 +0.001 / +0.010 MF0836.
  • TQ_CL4s2 TAIL: +0.018 / +0.009 ns (was 0.000 / +0.005), now the preferred TAIL MF0834.
  • Integrated 4-card xsim (4 loop sequencers + TAIL + link models): 0 frontier mismatches over 200 trees / 966 levels at w4 d7 MF0753; 8 of 8 fault injections caught MF0754.
  • About 46% of survivors need their parent a vector moved across chains (XITEM) DF0750.

Why it should beat SIBX-E: each chain runs its level loop card-to-card (D2), and the top-W merge is a 48-byte PART message at W=4 over the HEAD↔HEAD link, not a host round trip. Projection: 366–368 tok/s PF0700, F0789, about +8% over the record. Caveat: the same projection method put SIBX-E at 348–357, so we treat this as unproven until it runs.

Status: flash-ready with its host kit; it needs three QSFP28 DAC cables installed.

MP2: a two-port HEAD

Parked standalone · kept as CL4 transport
H200target verifyHost CPUrelay / KV / controlPCIeHEAD (HQ_MP2)link_dma + link_dma1TAIL (TQ_LV2ae)1 port (SLR1 full)2nd HEADin CL4 onlyP1P2Two Aurora links on one HEAD. P2 alone adds no batch-1 speed; it is the HEAD↔HEAD port CL4 needs.

Changed since 10-05: the MP2 HEAD has a recurring DSP58 cascade hold class. A plain reseed failed hold (WHS −0.023 ns MF0830), a hook-based fix was a no-op and the seed then failed setup (−0.301 ns MF0812). Adding a +0.020 ns hold margin to the router gave the first clean spare, HQ_MP2hm: +0.001 / +0.010 ns MF0838. A better TAIL seed also landed (TQ_LV2ae +0.019 / +0.008 MF0817).

Re-scoped: as a pair, MP2 runs the same drafter as the 10-01 link pair, so it projects to 305–320 tok/s PF0789, below the record. Its second link adds no batch-1 speed by itself. Its value is the HEAD↔HEAD port that CL4 uses. The hold-margin recipe has since been reused to close the shape-generic HEAD (below).

D2: card-side level loop

Queued for silicon as CL4's first step
H200target verifyHost CPUrelay / KV / controlPCIeHEAD (D2)loop_seq_d2 issues levelsTAIL (TQ_D2s2)survivors + parent a backKV / start / end onlyQSFP P1 seam →← survivorsThe level loop runs card to card. The host leaves the per-level path; it only loads KV and reads results.

Changed since 10-05: design unchanged; a better TAIL seed TQ_D2s2 landed at +0.025 / +0.003 ns with all link gates passing MF0835.

D2 removes the host from the per-level path of one pair: the TAIL sends survivors and the parent a back to the HEAD over QSFP, and the HEAD issues the next level itself. A 70 KB seam hop takes 39.9 µs on the link MF0608 against 81–88 µs per host relay hop (10-05 post). Projection 348.9 tok/s, +2.9% over the record PF0644, F0774.

Decision: run it first when cables arrive (one cable). It is the sub-chain of CL4, so a D2 pass de-risks half of CL4.

Shape-generic 4-card (SG_CL4H / SG_CL4T)

Active · v4 FAIL root-caused · v5 arms building
H200target verifyHost CPUrelay / KV / controlPCIeHEAD A (SG_CL4H)chain A + SHAPE_RTTAIL A (SG_CL4T)even lanesHEAD B (SG_CL4H)chain B + SHAPE_RTTAIL B (SG_CL4T)odd lanesP1P1P2 (HEAD↔HEAD)PART merge 16 B + 8 B/entryXITEM parent a, 6 KBPCIe: KV + control3 cables: A P1↔P1, B P1↔P1, HEAD A P2 ↔ HEAD B P2. Top-W merge happens on card, not on the host.

What it is: the CL4 ring with SHAPE_RT, a runtime table of GEMV geometry (weight base, columns, groups per stage). It is the path to larger drafters: an 8B drafter needs 2,554 URAM (33% of a 4-card pool of 7,700) and a Qwen-27B-class drafter 4,085 (53%) at 2-bit DF0821MF0820. Projected 4-card tensor-parallel batch-1: 8B ~303, Qwen-27B ~172 tok/s PF0825.

Correctness: on the 3B shape the table is cycle-exact against the fixed HEAD (0 of 1,292,916 cycles differ) MF0822; 8B and Qwen-27B slices match numpy 4,608/4,608 MF0823; both 4-card images pass full-card xsim MF0828.

Changed since 10-05:

  • TAIL SG_CL4T passes: v1 +0.003 / +0.001 ns MF0847, v3 +0.001 / +0.010 ns MF0862. Relayed.
  • HEAD v1 needed a constraint fix (an XDC if block was silently dropped MF0837) and closed only at zero margin; staged as a bring-up image.
  • HEAD v4 (shared multiply per group) failed setup: WNS −0.231 ns over 27,173 endpoints MF0865.
SLR0SLR1SLR2bcast replicasall pinned herelanes + DSP58next to URAMlanes + DSP58next to URAMlanes (grp 2)16,496 EP7,284 EPPer-group pblocks matched 0 cells (dead regex), so one SLR1 pblock pulled every broadcast register;each a_chunk net then crossed an SLR before fanning out at 132 MHz.

Root cause of v4 MF0867, F0870: about 22K of 27,159 failing endpoints are broadcast registers placed in SLR1 driving DSP58 inputs in SLR0 and SLR2 (16,496 and 7,284 crossings). The per-group placement constraints used one regex alternation and matched 0 cells; the same pattern was in 160 copies across HEAD lineages. Three fix arms are building:

  • A: per-lane reset replicas + KV-reader input register; xsim cycle-exact MF0868.
  • B: A + a DSP input register on the broadcast path; +4 of ~12,460 cycles per HEAD item (+0.03%), ~0 batch-1 effect because the HEAD is off the critical path MF0869.
  • C: B + repaired placement constraints (one filter per component, now non-empty) with a fail-closed lint before every build MF0870.

Shape-generic PAIR2 (new)

Chosen · PASS · staged for silicon
H200target verifyHost CPUrelay / KV / controlPCIeSG_HEAD v3hmHX1 BD + SHAPE_RT tableSG_TAILTCLK75 + SHAPE_RT tablePCIe x8PCIe x8same relay as record pairRecord-pair topology; GEMV shapes come from a runtime CSR table, not compile-time constants.

New since 10-05. The record-pair topology with SHAPE_RT on both cards, to prove the runtime table on silicon before the 4-card version.

  • HEAD v3 fixed the KV-reader clock domain (+0.075 ns on the 312 MHz domain, was −0.515) but failed hold by 0.016 ns on one DSP cascade pair MF0854. Route directive sweep: AggressiveExplore +0.006 > AlternateCLBRouting 0.000 > MoreGlobalIterations −0.261 MF0866.
  • Adding the MP2 hold margin closed it: SG_HEAD v3hm +0.006 / +0.010 ns MF0864; alternative v3hmx +0.010 / +0.010 ns MF0872. Both staged with SG_TAIL.
  • Speed: the 3B path is cycle-identical to the record pair on both HEAD and TAIL MF0824, F0827, so it should land at the record pair's speed on the same host DF0824, F0827, F0686.

HBM-streaming (new direction, design only)

Exploring · started 10-07 · projected only
H200target verifyHost CPUrelay / KV / controlPCIecard 0layer slice 1/4HBM 32 GBURAM cachecard 1layer slice 1/4HBM 32 GBURAM cachecard 2layer slice 1/4HBM 32 GBURAM cachecard 3layer slice 1/4HBM 32 GBURAM cacheConcept: weights live in HBM and stream through a fast-clocked GEMV; URAM only caches hot tiles.Goal: lift the on-chip URAM ceiling so larger drafters (and larger models) fit across 4 cards.

New since 10-05. Every design above keeps all weights in on-chip URAM: 1,925 URAM per card, 1,200 used by the HEAD and 896 by the TAIL today MF0820. That caps the drafter (a full-vocabulary Qwen-27B LM head does not fit even on 4 cards DF0821) and rules out serving larger models from the cards.

The new direction streams weights from HBM (32 GB per V80, 128 GB on four cards, vendor specification) through a fast-clocked GEMV, and uses URAM only as a cache for hot tiles. It started on 10-07 as a design study. There are no measured or modelled numbers yet; the first deliverable is a bandwidth and timing model, then an image. We will report it with labels when it exists.

Decision log

VariantDecisionWhy
SIBX-E 2H2T (PCIe merge)Rejected−30% vs the 2-card pair on the same healthy host MF0845; host merge and dual KV are on every level.
CL4-L ringChosenOnly 4-card design that keeps the merge on card; images PASS MF0834, F0836; needs 3 cables.
D2 pairChosen as step 1One cable; validates the card-side loop CL4 is made of; 348.9 PF0644.
MP2 pairParked305–320 PF0789 < record; kept as CL4's P2 transport.
SG PAIR2Chosen (next run)PASS MF0864, F0872; proves runtime shapes on silicon at 0 cycle cost.
SG 4-cardActivev4 FAIL root-caused MF0867; v5 arms A/B/C building.
Wider HEAD arrays (K_PAR 32, ternary dot)ParkedCross-SLR broadcast fails timing MF0815; ternary dot not data-exact MF0811.
HBM-streamingExploringUnblocks larger drafters and models; design only.

Next steps

  1. Re-measure the record pair on the 4-card chassis. The chassis still shows a firmware SMI storm MF0763; the record pair ran at 140.09 there before the host fixes MF0764. Until a fresh pair number exists, 4-card results are compared only with the 2-card arm on the same boot.
  2. SG PAIR2 on silicon: expect parity with the pair; any gap is a runtime-table bug.
  3. Cables in → D2, then CL4. D2 needs one QSFP28 DAC, CL4 three. CL4 is the first 4-card design that could clear the record (366–368 PF0700, F0789).
  4. SG 4-card v5: adjudicate arms A/B/C; relay the first PASS to replace the zero-margin bring-up HEAD.
  5. HBM-streaming: bandwidth/timing model first, then a GEMV streaming prototype image.
  6. Overlap draft with verify (draft-during-verify): the cycle model puts H200 verify at 38.7% and host relay at 19.5% of the step PF0859, the largest remaining lever for any card count.