Architecture update · 2026-10-07 · follows the 10-05 architecture report
EAGLE4 on four V80s: the variants we built, tested and chose
The EAGLE speculative-decoding drafter for Llama-3.2-3B, split across AMD Alveo V80 cards, with an H200 verifying. Since 10-03 the lab has four V80s. This post is a diff against the 10-05 architecture report: what happened to each 4-card variant, what we measured, and what we chose. The headline metric is batch-1 (single-stream) tok/s. Labels: M measured, D derived, P projected, each with its finding id.
Where we stand, plainly. The FPGA pair has not beaten GPU-only EAGLE: 339.12 vs 446.6–485.8 tok/s at batch 1. The first measured 4-card design (SIBX-E) was slower than the pair. The 4-card chassis also still runs below the 10-01 record because of a firmware SMI storm MF0763, F0764, so 4-card results are compared only with a 2-card run on the same boot. All numbers here are batch 1; this post has no batch-2 / aggregate figures.
What changed since the 10-05 post
The 10-05 report covered the record pair (HX1 BD HEAD + TCLK75 TAIL) and five 4-card options: SIBX-E 2H2T over PCIe, the D2 card-side loop, the MP2 two-link HEAD, the CL4-L ring, and the SHAPE_RT runtime shape table. Two variants are new here: SG PAIR2 and HBM-streaming.
10-05 post → now
What changed
Why
Result
Decision
SIBX-E 2H2T over PCIe
Re-measured on a healthy host (10-06), with a kit fix (v1.23stb3) that allows several 4-card runs per boot
The 10-05 number (50.00) came from a slowed chassis and could not judge the design
X 2:7 114.35 vs P 2:7 163.47 tok/s, −30% MF0845
Rejected
CL4-L ring (HEAD↔HEAD QSFP merge)
Better-margin spare images (TAIL +0.018 / +0.009 ns); now the single 4-card candidate
SIBX-E showed host merge + host KV cost more than the width split gains
Flash-ready, untested on silicon; PF0700, F0789 366–368
Chosen (waits on 3 QSFP cables)
MP2 two-link HEAD
First clean hold-margin spare (HQ_MP2hm); re-scoped from a pair design to CL4's P2 transport
P2 by itself adds 0 batch-1; its value is the HEAD↔HEAD port
+0.001 / +0.010 ns MF0838
Parked as a standalone; kept as CL4 transport
D2 card-side level loop
Better TAIL seed (TQ_D2s2); unchanged design
Margin insurance; D2 is the sub-chain CL4 is built from
+0.025 / +0.003 ns MF0835; 348.9 PF0644
Queued for silicon (1 cable) as CL4's first step
SHAPE_RT 4-card (SG_CL4H/T)
v4 HEAD failed; root cause = SLR-crossing broadcast + a dead pblock filter; three v5 fix arms building
Runtime shapes are needed for 8B / Qwen-27B drafters
Bring up the runtime shape table on 2 cards before 4
+0.006 / +0.010 ns MF0864; 0 extra cycles on 3B MF0824, F0827
Chosen (next silicon run)
(new) HBM-streaming
Started 10-07, design only: weights stream from HBM, URAM becomes a cache
URAM caps drafter size; 4 cards hold 128 GB of HBM
No numbers yet (design)
Exploring
Record pair (baseline)
Unchanged
—
339.12 tok/s MF0686
Baseline
GPU-only EAGLE (reference)
Unchanged
—
446.6–485.8 tok/s MF0007, F0286
Reference; not yet beaten
Batch-1 numbers at a glance
Batch-1 tok/s, mt_bench 50×32, Llama-3.2-3B target. Solid = measured on silicon, outlined = projected. The two 10-06 bars ran on the 4-card chassis, which still has a platform slowdown (see caveat), so compare them with each other, not with the 10-01 record.
HEAD cardTAIL cardhost / GPUPCIeQSFP28 card linkdata flow / merge
Baseline: the record pair
Measured · unchanged since 10-01
One HEAD card (embedding, QKV, attention, O, gate/up GEMVs) and one TAIL card (SiLU, down projection, low-rank LM head, tree top-k, exact fp16 rescore), with the host relaying each draft level over PCIe. The H200 verifies.
Step 10.93 ms = FPGA chain ~5.4 ms (49%) + H200 verify ~3.6 ms (33%) + host glue ~1.9 ms (17%), all serial DF0602.
A cycle model of the same pair says the FPGA engines sit idle for ≥ 83% of the step PF0859. That is why every variant below is about removing host hops or overlap, not about more GEMV throughput.
Reference: GPU-only EAGLE
Measured · not yet beaten
The honest comparison. The same EAGLE drafter on the H200 alone reaches 446.6–485.8 tok/s batch-1 MF0007, F0286. Vanilla decoding without speculation is 289.0 MF0569.
The FPGA pair is 1.17× vanilla but about 0.70–0.76× GPU-only EAGLE DF0686 / F0007, F0286. None of the 4-card projections below (≤ 368 PF0700) closes that gap either. The 4-card work is about learning how to scale out and about larger drafters; it is not a claim that FPGAs already win at 3B.
SIBX-E 2H2T: four cards over PCIe
Rejected · tested on silicon 10-06
Changed since 10-05: the 10-05 post showed the first valid 4-card run, 50.00 tok/s MF0819, on a chassis slowed 2–3.5× by a firmware SMI storm, plus a desktop service crash loop we later found and removed MF0833. After the fix, the same images and kit were re-run back to back:
Arm
Cell (width:depth)
Batch-1 tok/s
KV load
2-card P (same chassis)
2:7
163.47 (166.48 / 169.49 / 154.43) MF0845
1.80–1.89 ms
2-card P
4:7
136.29 MF0845
—
4-card X
2:7
114.35, −30.05% vs P MF0845
3.12 ms
4-card X
4:7
~95 if it tracks the 2:7 ratio PF0853
—
The design was projected at 348–357 PF0700. It measured 30% slower than the 2-card pair on the same host. The losses are the host-side merge per level and dual-chain KV loading through the host (3.12 vs 1.80 ms per load).
A TAIL status latch ("SIBX on-bit") stayed set after a 4-card run, so only one 4-card run per boot was possible. Kit v1.23stb3 clears it in software MF0849, F0851; confirmed on silicon MF0855. A 4:7 run was taken but is invalid: an unrelated workload loaded the shared host at the time MF0855.
Decision: rejected as a batch-1 design. The lesson carries over: at batch 1, a width split only pays if the merge stays off the host.
CL4-L ring: two D2 chains merged HEAD↔HEAD
Chosen · built, PASS, staged · untested (cables)
Changed since 10-05: design and images unchanged; it moved from "one option" to "the 4-card candidate" after SIBX-E. Insurance seeds landed with more margin:
TQ_CL4s2 TAIL: +0.018 / +0.009 ns (was 0.000 / +0.005), now the preferred TAIL MF0834.
Integrated 4-card xsim (4 loop sequencers + TAIL + link models): 0 frontier mismatches over 200 trees / 966 levels at w4 d7 MF0753; 8 of 8 fault injections caught MF0754.
About 46% of survivors need their parent a vector moved across chains (XITEM) DF0750.
Why it should beat SIBX-E: each chain runs its level loop card-to-card (D2), and the top-W merge is a 48-byte PART message at W=4 over the HEAD↔HEAD link, not a host round trip. Projection: 366–368 tok/sPF0700, F0789, about +8% over the record. Caveat: the same projection method put SIBX-E at 348–357, so we treat this as unproven until it runs.
Status: flash-ready with its host kit; it needs three QSFP28 DAC cables installed.
MP2: a two-port HEAD
Parked standalone · kept as CL4 transport
Changed since 10-05: the MP2 HEAD has a recurring DSP58 cascade hold class. A plain reseed failed hold (WHS −0.023 ns MF0830), a hook-based fix was a no-op and the seed then failed setup (−0.301 ns MF0812). Adding a +0.020 ns hold margin to the router gave the first clean spare, HQ_MP2hm: +0.001 / +0.010 ns MF0838. A better TAIL seed also landed (TQ_LV2ae +0.019 / +0.008 MF0817).
Re-scoped: as a pair, MP2 runs the same drafter as the 10-01 link pair, so it projects to 305–320 tok/s PF0789, below the record. Its second link adds no batch-1 speed by itself. Its value is the HEAD↔HEAD port that CL4 uses. The hold-margin recipe has since been reused to close the shape-generic HEAD (below).
D2: card-side level loop
Queued for silicon as CL4's first step
Changed since 10-05: design unchanged; a better TAIL seed TQ_D2s2 landed at +0.025 / +0.003 ns with all link gates passing MF0835.
D2 removes the host from the per-level path of one pair: the TAIL sends survivors and the parent a back to the HEAD over QSFP, and the HEAD issues the next level itself. A 70 KB seam hop takes 39.9 µs on the link MF0608 against 81–88 µs per host relay hop (10-05 post). Projection 348.9 tok/s, +2.9% over the record PF0644, F0774.
Decision: run it first when cables arrive (one cable). It is the sub-chain of CL4, so a D2 pass de-risks half of CL4.
Shape-generic 4-card (SG_CL4H / SG_CL4T)
Active · v4 FAIL root-caused · v5 arms building
What it is: the CL4 ring with SHAPE_RT, a runtime table of GEMV geometry (weight base, columns, groups per stage). It is the path to larger drafters: an 8B drafter needs 2,554 URAM (33% of a 4-card pool of 7,700) and a Qwen-27B-class drafter 4,085 (53%) at 2-bit DF0821MF0820. Projected 4-card tensor-parallel batch-1: 8B ~303, Qwen-27B ~172 tok/s PF0825.
Correctness: on the 3B shape the table is cycle-exact against the fixed HEAD (0 of 1,292,916 cycles differ) MF0822; 8B and Qwen-27B slices match numpy 4,608/4,608 MF0823; both 4-card images pass full-card xsim MF0828.
HEAD v1 needed a constraint fix (an XDC if block was silently dropped MF0837) and closed only at zero margin; staged as a bring-up image.
HEAD v4 (shared multiply per group) failed setup: WNS −0.231 ns over 27,173 endpoints MF0865.
Root cause of v4MF0867, F0870: about 22K of 27,159 failing endpoints are broadcast registers placed in SLR1 driving DSP58 inputs in SLR0 and SLR2 (16,496 and 7,284 crossings). The per-group placement constraints used one regex alternation and matched 0 cells; the same pattern was in 160 copies across HEAD lineages. Three fix arms are building:
B: A + a DSP input register on the broadcast path; +4 of ~12,460 cycles per HEAD item (+0.03%), ~0 batch-1 effect because the HEAD is off the critical path MF0869.
C: B + repaired placement constraints (one filter per component, now non-empty) with a fail-closed lint before every build MF0870.
Shape-generic PAIR2 (new)
Chosen · PASS · staged for silicon
New since 10-05. The record-pair topology with SHAPE_RT on both cards, to prove the runtime table on silicon before the 4-card version.
HEAD v3 fixed the KV-reader clock domain (+0.075 ns on the 312 MHz domain, was −0.515) but failed hold by 0.016 ns on one DSP cascade pair MF0854. Route directive sweep: AggressiveExplore +0.006 > AlternateCLBRouting 0.000 > MoreGlobalIterations −0.261 MF0866.
Adding the MP2 hold margin closed it: SG_HEAD v3hm +0.006 / +0.010 ns MF0864; alternative v3hmx +0.010 / +0.010 ns MF0872. Both staged with SG_TAIL.
Speed: the 3B path is cycle-identical to the record pair on both HEAD and TAIL MF0824, F0827, so it should land at the record pair's speed on the same host DF0824, F0827, F0686.
HBM-streaming (new direction, design only)
Exploring · started 10-07 · projected only
New since 10-05. Every design above keeps all weights in on-chip URAM: 1,925 URAM per card, 1,200 used by the HEAD and 896 by the TAIL today MF0820. That caps the drafter (a full-vocabulary Qwen-27B LM head does not fit even on 4 cards DF0821) and rules out serving larger models from the cards.
The new direction streams weights from HBM (32 GB per V80, 128 GB on four cards, vendor specification) through a fast-clocked GEMV, and uses URAM only as a cache for hot tiles. It started on 10-07 as a design study. There are no measured or modelled numbers yet; the first deliverable is a bandwidth and timing model, then an image. We will report it with labels when it exists.
Decision log
Variant
Decision
Why
SIBX-E 2H2T (PCIe merge)
Rejected
−30% vs the 2-card pair on the same healthy host MF0845; host merge and dual KV are on every level.
CL4-L ring
Chosen
Only 4-card design that keeps the merge on card; images PASS MF0834, F0836; needs 3 cables.
D2 pair
Chosen as step 1
One cable; validates the card-side loop CL4 is made of; 348.9 PF0644.
MP2 pair
Parked
305–320 PF0789 < record; kept as CL4's P2 transport.
SG PAIR2
Chosen (next run)
PASS MF0864, F0872; proves runtime shapes on silicon at 0 cycle cost.
Cross-SLR broadcast fails timing MF0815; ternary dot not data-exact MF0811.
HBM-streaming
Exploring
Unblocks larger drafters and models; design only.
Next steps
Re-measure the record pair on the 4-card chassis. The chassis still shows a firmware SMI storm MF0763; the record pair ran at 140.09 there before the host fixes MF0764. Until a fresh pair number exists, 4-card results are compared only with the 2-card arm on the same boot.
SG PAIR2 on silicon: expect parity with the pair; any gap is a runtime-table bug.
Cables in → D2, then CL4. D2 needs one QSFP28 DAC, CL4 three. CL4 is the first 4-card design that could clear the record (366–368 PF0700, F0789).
SG 4-card v5: adjudicate arms A/B/C; relay the first PASS to replace the zero-margin bring-up HEAD.
HBM-streaming: bandwidth/timing model first, then a GEMV streaming prototype image.
Overlap draft with verify (draft-during-verify): the cycle model puts H200 verify at 38.7% and host relay at 19.5% of the step PF0859, the largest remaining lever for any card count.