fpga-archive-reports-arch · EAGLE4 FPGA archive
Ideal drafter datapath "D2" — die-local residual stream, one fast clock, fused norms (box1-two-v2, 2026-11-18)
Author: box1-two-v2 (orchestrator-authored design; user 11-18: "feel free to do your own design for ideal datapath"). Program: the 7-card DRAFTER (Llama-3.2-3B record 961.43 tok/s @ 6.91 on COMBO3 0x13D6 x4 + M1361 x2; Qwen3.8-27B GPTQ-2 drafter, best humaneval 181.79 @ 12.14 d15). Motto: never end by denying the idea; engineer toward the fastest flashable PDI. Tags: [M] measured (artifact), [R] read from RTL/report, [P] projection.
0. Answer
The current ATT card is a single-card design grown by levers: a 66 MHz core with 132 MHz islands, a full-width HUB in SLR0 that every GEMV pass returns to, and attention KV filling the same die. SLR0 is 98.8 % SLICE (A8OV 0x13D3 placed) [M] while SLR1/SLR2 sit at 69 % / 78 % [M]. D2 changes three structural things, each provable on its own:
- Die-local residual stream (no hub). Each die owns a slice of the output columns of EVERY GEMV and the residual-stream slice for those columns. Attention heads live on the die whose lanes produce their Q/K/V columns, with their KV local. Only small vectors cross dies: the int8 activation all-gather (3,072 B per GEMV input for Llama), the 768-element attention output, and scalar partial sums for RMSNorm.
- Fused norms by scale invariance. RMSNorm followed by per-vector absmax A8 quantisation is invariant to the scalar 1/rms: quant(x∘g/rms) = quant(x∘g) with the dequant scale carrying 1/rms. So G1 (input norm), the pre-MLP norm (done: NFA8 NORMFUSE) and G11 never hold a GEMV: the GEMV starts on x∘g at once and rms (a scalar reduced in parallel) is applied at the output dequant.
- One fast clock for the whole datapath (target 133 MHz, = 2 x 66.0, integer clk_wiz ratio). Removes every 66<->132 bridge (AFC8 bridge = +24.9K FF) and halves every non-array phase too (G5, norms, hand-offs, FSM). The die-local layout is what makes this closable: the measured 100 MHz wall on the old single-card HEAD was WNS -4.850 ns [M, WHOLE_DRAFTER_FMAX_CENSUS_20260923.md running log 10:40], dominated by long full-width paths that D2 deletes.
Plus two scheduling changes that need no new datapath: sibling interleave (the attention engine works on sibling A while the array runs sibling B's GEMV) and one host message per step (batched KV rows, junior's c5 KV_MROW for Qwen).
1. Why (measured)
| fact | value | source |
|---|---|---|
| Llama ATT item on silicon | 7,394 cyc @ 66 MHz = 112 us (G1 456, G4 1,421, G5 1,919, G6 686, G7g 1,397, G7u 557, rest 958) | SCALING_4C_VS_7C_PHASE_RANK_20261116.md census |
| weight GEMV share of a Llama node | 24 % (2,832 of 11,860) | ARCH_RETHINK_7CARD_20261117.md |
| ATT item after NFA8 + AFC8 (xsim) | 4,126 cyc @ 66-equiv (G4 914, G6 632, G7g 486, G7u 220) | ATT_FASTCLK_20261117.md |
| SLR SLICE, A8OV placed | 111,216 / 70,765 / 83,010 = 98.83 / 69.02 / 77.78 % | /var/tmp/box1two_nfa8_20261117/watch_D3/STATUS 02:03Z |
| HUB contents (SLR0) | RET_SER landing/commit/per-lane shadow 512x48 b, g7g/g7u latches, R4/SiLU, u_tp (136,279 prims; 27,195 with TP_BEAT), q/k/v_full_buf | WHY_CROSS_DIE_RTL_READ_20261108.md, F_IMAGE_CONGESTION_DOSSIER_20261108.md |
| hub moved to SLR1 | 24,212 SLL > 18,870 available | same |
| cards' share of a Llama step | 2.0 ms of 7.19 ms | ARCH_RETHINK_7CARD_20261117.md |
| Qwen humaneval round | 49.5 ms; card RPC 25.7 ms of which KV rows 16.7 ms | DESIGN_FABLE_DRAFT_AHEAD_20261117.md |
| Qwen ATT item | 15,085 -> 8,102 with K64 + AFC (xsim) | DESIGN_FABLE_QWEN_ATT_K64_20261117.md |
| placement constraints | pblock_dc_grp0/1/2 NCELLS = 0 (vacuous) on junior's AFC images; 0x13CE pblockgate 22 dead filters | junior 11-18; GATE_VERDICTS s3LAtp4CE 00:46:55 |
2. D2 ATT card (TP4 rank), per die
broadcast x∘g (int8, all-gather) all-gather attn out (768) all-gather h (int8)
die d: [lanes N-slice d] --G4 Q/K/V cols of heads(d)--> [G5 heads(d), KV(d) local] --> [lanes N-slice d: G6] --> resid slice d
|-> partial Σx² (scalar) -> rms
[lanes N-slice d: G7g/G7u] -> SiLU·up slice d -> to MLPV (via link die)
cross-card: G6 partial sums all-reduced per column slice over the TP4 link (SLR1 GTM), each die sends only its slice.
- Llama per rank: 8 heads x 96 -> heads per die 3/3/2; Qwen per rank: 10 heads x 128 -> 4/3/3. Column slices follow the heads for G4; G6/G7 slices are even.
- KV BRAM spreads over three dies (today ~1,152 BRAM tiles in SLR0 [R]); SLR0 keeps only its share plus the PCIe/QDMA shell.
- No RET_SER return to a central die: results stay where they are produced; the next GEMV's input is an all-gather of int8 activations.
- MLPV card: same principles (die-local G9 column slices, G11 folded, TREE vocab slices per die, G8 candidates split per die's HBM channels).
3. Projection [P] (to be replaced by each stage's measurement)
| stage | ATT item (us) | basis |
|---|---|---|
| today (silicon) | 112 | [M] |
| NFA8 + AFC8 (building) | ~62 xsim-equiv (silicon will be higher by link waits) | [M] xsim 4,126 cyc |
| + G1 fold | ~56 | removes the 456-cycle G1 at 66 MHz |
| + one 133 MHz clock | ~36-44 | non-array phases halve; array already fast |
| + sibling interleave | ~30-36 | G5 hidden under the other sibling's GEMV |
| Llama: cards 2.0 -> ~0.8 ms per step => ~6.0 ms step => ~1,150 tok/s (+20 %) if the MLPV gets the same treatment. Qwen gains more (card-dominant round). |
4. Stages (each a unit gate first, then an image; one lever per image for attribution)
| # | stage | proof before image | kill / confirm |
|---|---|---|---|
| S0 | fix vacuous pblocks (dc_grp, dead filters) | NCELLS > 0 on a placed copy | SLR0 SLICE and WNS on re-place (SLR0_DECONGEST agent) |
| S1 | G1 fold (input norm by scale invariance) on the AFC8 fork | xsim bit-exact vs AFC8 on 4 ranks + RED; numerics proof of the scale identity incl. fp rounding of the scale | G1 cycles -> ~0 |
| S2 | 133 MHz fmax census on the routed NFA8 dcp (period override, every failing endpoint bucketed) | census tool exists (whole_drafter_fmax_census_20260923/census.tcl) | list of path classes to pipeline; if > ~2,000 distinct failing paths outside the hub/RET_SER/u_tp, D2's clock target drops to 100 MHz |
| S3 | die-local residual stream RTL (fork from AFC8) | xsim bit-exact vs AFC8 on 4 ranks; SLL estimate from an OOC place of the 3-die array top | SLR0 SLICE < 85 % and SLL < 15,000 on the OOC place |
| S4 | single 133 MHz clock on the S3 fork | OOC route of the S3 top at 7.5 ns | WNS >= 0 OOC, then full image |
| S5 | sibling interleave (attention vs array across W2 siblings) | xsim cycle drop, bit-exact | per-pair item time |
| Silicon order: NFA8/AFC8 (in flight) -> S1 image -> S3+S4 image (the D2 card) -> S5. |
5. Ownership
Orchestrator authors this spec and re-examines every critical RTL diff (S1, S3, S4); agents implement in their own forks with paired read-only auditors. Qwen images stay junior's; D2 applies to both models (dual-model ATT) and the recipe goes to junior once S3 is green.