Weight-stationary inference silicon

LLM inference without moving the weights.

Every accelerator in production streams billions of parameters out of DRAM on every single token. We don't move them at all. The entire transformer body sits in on-chip SRAM permanently and stays there while it computes.

1,962 µs / drafter forward Measured on silicon AMD Alveo V80 · 1.62× over our previous image (3,167 µs) · fabricated hardware, not simulation
On-chip SRAM occupancy 88.83 %
1,710 occupied 215 free 3 super-logic regions

Every URAM block on the device, drawn to scale. The design is packed to 89.51 / 88.46 / 88.46 % across SLR0 / 1 / 2. URAM is the binding resource — deliberately.

The wall

Inference is a memory problem
wearing a compute costume.

Autoregressive decoding produces one token at a time. To produce that token, the machine must read every weight in the model — and then, for the next token, read every weight again. Nothing is reused between steps. When you are serving a single user's stream there is no batch dimension to amortise the traffic across.

So the ceiling is not arithmetic. It is memory bandwidth. A modern accelerator's multipliers spend most of a decode step idle, waiting on a memory controller to deliver parameters they will use exactly once and discard. Adding FLOPS does not move that ceiling. It is why serving cost per token is stubborn, why utilisation charts look the way they do, and why the industry's answer so far has been to buy more bandwidth rather than to need less of it.

The structural fix is to stop moving the weights. If the parameters never leave the die, there is nothing to be bandwidth-bound on. That is the entire premise of this architecture — and it is now running in fabricated silicon.

The approach

Weight-stationary, fully on-chip.

Five decisions taken together. Each one only works because of the other four.

Weights that never move

The whole transformer body — quantised to 2 bits, 41.58 MiB — is written into on-chip URAM once at boot and stays resident. During a forward pass there is zero HBM traffic for weights. Not reduced: none. Every matrix-vector product reads its parameters from SRAM sitting beside the multiplier that consumes them.

Activations in flip-flops

The intermediate tensors between stages live in registers — a counter-addressed flip-flop register file, not block RAM. That removes the last memory round-trip from the datapath, and it removes an entire class of silicon failure we learned about the expensive way: a registered block-RAM read whose timing has to cross a die boundary.

A state machine, not a host

All fifteen phases of a forward pass — norm, QKV projection, 24-head attention, output projection, gated MLP, final norm, low-rank projection, vocabulary logits, tree select, exact rescore — are sequenced by an on-chip FSM. There is no host round-trip per step. The CPU hands over a request and receives a result.

Speculation that stays on the die

Tree-structured speculative decoding runs entirely in hardware: a 512-candidate tree, an on-chip top-8 frontier, survivor selection and re-parenting all in the fabric. The branching structure that makes speculative decoding fast is exactly the structure that makes it chatty with a host — so we never talk to one.

One honest exception

A single tensor does not fit: the exact fp16 language-model head, 788 MB (788,004,864 bytes). It lives in HBM, and the final rescoring stage gathers 512 rows from it per forward. We name it rather than bury it — it is the one place the design still touches off-chip memory, it is the current throughput bottleneck, and on an ASIC it disappears entirely.

Measured, not estimated

Where a depth-2 draft step actually spends its time

Per-phase cycle budget — bit-true simulation of the shipping image's source, 2026-07-29 (66 MHz; 73-node depth-2 tree)
PhaseCyclesmsShare of draft
Level-0: root body pass + tree select + serialize≈44,0000.6711%
Level-1: 8 child body passes (sequential, 45,900 each)367,0005.5689%
Whole on-chip draft411,2816.23100%
System-level step budget on silicon — best config (K_C=64, strict, depth-2): 29.8 ms per verify step
ComponentmsShareNote
On-chip draft (all 15 phases × 9 passes + tree)≈6≈20%the fabric
Exact-rescore gather (off-chip, K_C=64)≈1≈3%was ≈9 ms at K_C=512 — shortlist shrink bought +30% throughput, accept-free
Host marshal + target verify + DMA≈23≈77%the machine is ~3× faster than the software around it — the current optimization frontier, and precisely the term that vanishes in a dedicated implementation

Instruments: launch/stage timestamps from bit-true simulation of the shipping source (memory-resident phases exact; off-chip gather idealized in sim, cross-checked against the silicon K_C sweep), and wall-clock step budgets from live strict-mode serving runs. The eight child passes run sequentially today — overlapping them through the existing register-file seams is the pipelining headroom the dedicated-silicon section describes.

Inside the machine

Two arithmetics,
one hundred and twenty-eight lanes,
three dies.

Filling 88.83 % of a device's SRAM is not an accident of synthesis. It is the constraint the entire microarchitecture was designed against.

Binding resource 1,710 / 1,925 URAM blocks · 88.83 %

The wall we chose to hit. Lane count, reduction width and head replication are all sized to fit underneath it.

Parallel structure 128 × 16 output lanes × reduction per SRAM read

128 output columns computed simultaneously, each consuming 16 weights per URAM access, spread across 3 super-logic regions.

Attention replication 6× HEAD_PAR = 6 · 24 heads

Six attention heads evaluated concurrently against a shared key/value stack — strapped and netlist-verified in the shipped image.

The unusual part

Engine A · the 2-bit body

A Wallace tree of LUT MACs

At 2 bits, a DSP is the wrong instrument — a hardened wide multiplier is far coarser than the operation needs, and we measured that rather than assumed it. So the body's matrix-vector products are multiplier-free: shift-add MACs built in pure logic, reduced through a Wallace 3:2 carry-save adder tree. Only the very end touches hard silicon — one DSP58 for the final 48-bit carry-propagate add, and one DSP58 per lane for the Q1.15 dequantisation scale.

Engine B · attention

A separate INT16 DSP58 datapath

Attention is a different precision problem, so it gets different hardware: a dedicated INT16 × INT16 DSP58 engine with its own carry-save reduction tree, serving 24 heads six at a time. Two arithmetic styles coexisting on one die, each matched to its own operand width, is precisely why the compute fits in the area the SRAM leaves behind.

Physical residency

Where every tensor actually lives

On-chip · the whole transformer body — zero weight streaming during compute Off-chip · HBM URAM · 1,710 blocks 2-bit transformer body — 41.58 MiB QKV 13.50 · o-proj 2.25 · gate 6.00 · up 6.00 · down 6.00 Cheap vocabulary head — 7.83 MiB 128,256 logits, 1,002 tiles, fully on-chip Key cache — K Block RAM · 3,017 tiles Value cache — V RMSNorm gains Q3.13 dequantisation scales Low-rank projection ROM — 1.5 MiB Q1.15, 16 lanes × 3,072 rows. Baked at synthesis — reload needs a rebuild. Flip-flops · the datapath All inter-stage activations. No block RAM on any seam. G1 norm · G4 QKV · G5 attention G6 o-proj · G7 gate/up · SiLU G9 down · G11 final norm GLM low-rank · G12 cheap logits TREE · 512-candidate select G8 · exact rescore Sequenced by an on-chip FSM — no host round-trip per step. resident · never streamed Every forward Exact fp16 lm-head 788 MB 788,004,864 bytes 512-row gather per forward Vanishes on an ASIC Tree depth ≥ 1 only Embedding table 788 MB bf16 verbatim, 128,256 × 3,072 One child-token row per branch; idle at depth 0 G8 rescore gather child-embedding gather
Model geometry, as implemented in the fabric
ParameterValueNote
Hidden dimension3,072Llama-3.2-3B family drafter
Attention heads24 Q / 24 KVTrue multi-head attention — not grouped-query
Head dimension128INT16 × INT16 DSP58 engine, 6-way replicated
MLP intermediate8,192Gate and up projections, SiLU-gated
Vocabulary128,256Cheap head on-chip; exact head in HBM
Candidate tree512Selected and rescored entirely in fabric
Frontier width8Top-8 survivors carried to the next tree level
Body weight precision2-bitGroup-quantised, Q3.13 dequantisation scales

Fabricated, flashed, measured

This runs. On real hardware.

Not an emulator, not a projection, not a synthesis estimate. A programmed AMD Alveo V80 with a host reading results back over PCIe.

Latency 1,962 µs per drafter forward · depth-0 root

Median of three consecutive on-card probes (1,960 / 1,962 / 1,960 µs). Derived rate: ≈510 drafter forwards / s.

Generation over generation 1.62× vs. 3,167 µs previous image

Both endpoints measured on the same card. The gain came from a staging fix, dual-port SRAM reads, 6-way head replication and a wider gather.

Timing closure +0.017 ns routed worst negative slack @ 66.002 MHz

Zero failing endpoints out of 3,562,588. Hold clean. A design this saturated with SRAM closes tight — but it closes.

Post-place utilisation — AMD Alveo V80, Versal HBM, 3 super-logic regions
ResourceUsedAvailableUtilisationRole in the design
URAM1,7101,92588.83 %Binding — resident weights + key cache
Block RAM tile3,0173,74180.65 %Value cache, norm gains, dequant scales
CLB LUT1,419,4992,574,20855.14 %Multiplier-free 2-bit MACs + carry-save trees
CLB register1,497,1815,148,41629.08 %All inter-stage activation staging
DSP582,49710,84823.02 %Attention engine, final CPA, dequant scale

What we are claiming, precisely. The figure above is the latency of one drafter forward pass on the card. It is not an end-to-end tokens-per-second number and we do not present it as one. End-to-end throughput is currently limited by an accept-length defect in the tree stage that is under active repair, and we will not publish an end-to-end comparison against a GPU until that is fixed. What the silicon proves today is the architecture: that a full transformer body can be held stationary in on-chip SRAM, sequenced entirely by hardware, and made to close timing across three dies at 88.83 % memory occupancy.

The next substrate

The FPGA was never
the destination.

An FPGA is a de-risking vehicle. It let us prove a weight-stationary transformer in fabricated silicon before anyone commits to a mask set. The same architecture on a custom ASIC changes by three multipliers — and loses its last off-chip tensor.

Projection — not measured

Memory density · projected ~10× SRAM macro vs. FPGA URAM

A hard SRAM macro is roughly an order of magnitude denser per unit area than a reconfigurable URAM block. At that density the entire model fits on-chip, including the 788 MB exact head. Zero off-chip memory. At all.

Compute density · projected ~10× standard-cell MAC vs. LUT-based

Our 2-bit MACs are currently assembled from general-purpose lookup tables. As standard cells they occupy a fraction of the area — so far more lanes fit into whatever space the on-chip memory leaves over.

Clock · projected ~15–30× GHz-class vs. our 66.002 MHz

We close timing at 66 MHz because that is what a fully-packed reconfigurable fabric with three die crossings permits. A custom implementation of the same datapath is not bound by that constraint.

Stacked, the same architecture plausibly reaches ~50–200 µs per forward.
This range is an engineering projection derived from the three density and frequency arguments above. It is not a measurement, there is no fabricated ASIC behind it, and it should be read as a target rather than a result. The only measured latency on this page is the 1,962 µs FPGA figure.

The port is not a rewrite. Everything expensive we learned on the FPGA transfers directly: the tree-decode state machine, the accept and re-parenting machinery, the register-staging discipline that keeps activations out of memory, the two-arithmetic split between the 2-bit body and the attention engine, the boot-time weight-residency protocol. Those are architecture, not device features. What changes on an ASIC is that they become faster, denser, and finally free of the one remaining off-chip table.

The strategic point is the sequencing. Most accelerator programmes ask for a mask set on the strength of a simulator. We have a fully packed, timing-closed, measured implementation of the architecture running on real hardware first — which means the risk remaining on an ASIC is a physical-design risk, not an architectural one.

Diligence

Every number, and where it came from.

This programme retracted a headline performance claim once. The standing rule since: a latency, utilisation or frequency figure appears only if it was pulled from an artifact — a silicon run, a synthesis report, or a place-and-route report.

Deliberately absent. No power figure, no energy per token, no tokens-per-watt, and no end-to-end throughput comparison against a GPU. None of those has an artifact behind it today. A number without a measurement is left off this page rather than estimated.