fpga-archive-reports-arch · EAGLE4 FPGA archive

Source reports/results/DRAFTER_SPEED_RESEARCH_20261107/ARCH_RETHINK_7CARD_20261117.md · source md5 9bad620f099f24a5cf21a467b51c27bd · archived 2026-11-18

Download PDFOpen PDFRaw markdown

Rethinking the 7-card drafter from scratch: what the ideal architecture is, and what it buys (2026-11-17)

Program 1: ONE drafter on the talon2 V80s, single stream, H200 verifies. Two drafters are covered: Llama-3.2-3B bitnet_721 (record 961.43 tok/s @ 6.91, GPU-only 841.27) and Qwen3.8-27B GPTQ-2 W2A16 (140.15 vs GPU-only 326.83). Motto: a wall is a measured constraint to engineer around, toward the fastest flashable PDI. Fable architecture agent for box1-two-v2. Read-only study and a python model. No build, flash, card access, kill, move or removal. User question (verbatim): "If you rethink entirely the optimal architecture for the 7 cards, what's the ideal way to run these models compared to the current one?" Tags: [M] measured (file:line), [D] arithmetic on [M], [P] projection or assumption. Every [P] is collected in section 2. Short paths: R/ = reports/results/DRAFTER_SPEED_RESEARCH_20261107/, Q/ = reports/results/QWEN_GPTQ2_20261116/, F/ = reports/results/FOURCARD_DESIGN_20261016/, RR/ = reports/results/. Model: R/arch_rethink_20261117/arch_model.py (md5 3882bc44), output arch_model.out (md5 8294e164). Re-run: python3 arch_model.py.

0. Answer first

  1. The ideal is not "more tensor-parallel". It is three things in this order: (i) hide the card tree under the GPU verify, (ii) do one batched pass per tree level instead of one pass per tree node, (iii) only then spread every matrix over all cards on a faster array. Uniform tensor parallel by itself (candidate A) is worth about +0.2 % (6 cards) to +1 % (7 cards) on Llama [P].
  2. Why A alone is nearly worth nothing. Today's 4 + 2 role split is a two-stage pipeline that is already close to balanced: the attention cards work on node B while the MLP cards finish node A. A uniform image puts both stages on every card, so each node costs the full serial chain. On Llama that is 8,140 cycles per node at 6 cards against 8,240 measured today [P vs M]. A balanced pipeline and a uniform split use the same silicon equally well.
  3. Llama is not MAC-bound. The array is 2,832 of the 11,860 busy cycles per node (24 %) [D]. The rest is context sweeps (G5), serial norm and quantiser walks, the cross-card sum, the tile-bound vocabulary scan and the hand-offs. So more cards or DSP lanes move little; removing per-node fixed work moves a lot. Qwen is MAC-bound (81 % of the ATT node, 77 % of the MLPV node [M]), so B matters there.
  4. Ceiling. The cards own 2.010 ms of the 7.187 ms Llama step [M]. With a zero-time tree the step is 5.686 ms = 1,215 tok/s [D]. Any architecture lives between 961 and 1,215.
  5. What the full ideal buys on Llama [P]: one image on 7 cards, level-batched, array at 16,384 MAC per drafter cycle per card, prologue fusions and TREE4: d11 tree 2,622 → about 1,490 us, about 1,046 tok/s (+8.8 %); with a 2x array, 1,296 us and about 1,063. Evolving today's role split with the levers already in flight reaches about 993-1,008 [P]. So the rethink itself is worth roughly +4 to +5 % over the evolved role split.
  6. Qwen. The ideal card tree is about 1.7-2.5 ms per d7 tree against 5.82 ms measured today [P vs M]. That alone does not reach GPU-only, because the round is 36.5 ms against 21.9 and the tree is only 8.2 ms of it [M]. Qwen needs draft-ahead (D): the verify is 19.14 ms [M], long enough to hide the whole card path. At the fitted humaneval hit rate of about 0.84 [P] the round falls to about 25 ms; a hit rate of 1 gives 22.8 ms, i.e. GPU-only parity. Beating GPU-only then comes from depth: the cards can draft d11-d15 inside the verify window, which the GPU drafter cannot do for free.
  7. Draft-ahead on Llama is marginal: measured hit rate 15.3 % at d11 (26.0 % at d7) [M]. At zero miss cost that is +3.3 %; at the 0.3 ms miss cost of the earlier study it is break-even (break-even hit rate 0.167) [D].
  8. Binding term once the array shrinks: on Llama the G5 context sweeps (21 % of a batched level), then the array (27 % at 16,384, 16 % at 32,768), then the two all-reduces (16-19 %) [P]. The all-reduce is not the first wall, but it is the one that grows with N and with a bad topology: 150 → 1,200 cycles per all-reduce moves the result from 1,074 to 1,045 tok/s [P].
  9. Corrections to the brief. "DSP58 70 % idle" is not what the images show: ATT 0x13BE uses 48.6 % (51 % free), MLPV M1361 uses 69.2 % (31 % free) [M]. Card 99 is unused for cabling and tooling reasons, not a hardware fault (section 1.2).

1. Measured inputs

1.1 Inter-card link [M]

quantity value source
PHY per cage 4 x 25.78 Gb/s Aurora 64B/66B; raw 99.84 Gb/s per direction, PRBS31 600 s RR/SEVENCARD_BOARD_20261026/TALON2_QSFP_MAP_20261110.md:56-57; F/A2A_0x125D_AC18_REPORT.md:28
beat format 16 lanes x 48 bit = 768 b; a 3,072-lane vector = 192 beats verilog/forks/lutgemm_att_d9_tp4_20261108/link4/tp_ar4_hc.sv:24; RR/PER_PHASE_CYCLES_BY_IMAGE_20261110.md:175
link logic clock as built 200 MHz, 4 cycles per beat = 0.758 beat per drafter cycle; 192 beats ≈ 253 drafter cycles RR/PER_PHASE_CYCLES_BY_IMAGE_20261110.md:111; R/G6_TP4_EXCHANGE_RCA_20261110.md:101,105
all-reduce of 3,072 lanes, unit xsim of the real engines, 500 ns link K4 one step 291 cycles; hypercube two steps 360 (+69); +66 cycles per +500 ns of link F/TP4_DESIGN_20261018.md:10-16; R/ATT_TP4_20261108.md:67
G6 on silicon, 2 ranks (F3t) 636 cycles per node, array 288 RR/PER_PHASE_CYCLES_BY_IMAGE_20261110.md:57,175
G6 on silicon, 4 ranks before / after G6_REISSUE 1,213 (B4) → 686 (0x13BE), 693 (COMBO3); array 144, exchange ≈ 549 PER_PHASE...:57; R/SCALING_4C_VS_7C_PHASE_RANK_20261116.md:66,164; R/dw_sweep_combo3_20261116/census_combo3_P11w2.table.txt (G6 14,552 / 21)
one-way hop latency of the shipped stack on silicon NOT-MEASURED; the real-exchange testbench matches silicon at about 406 ns R/G6_TP4_EXCHANGE_RCA_20261110.md:115; R/G6_REISSUE_FIX_20261110.md:130
small-message round trip on silicon (prototype link, 402 MHz) 32 B 0.707 us (47 cycles); 6 KiB 1.20 us; 18,432 B 2.19 us reports/qsfp_payload_20260908/feasibility_20260909/test_1179_20260910_0056/README.md:11-23
ATT → MLPV hand-off fitted 40.6 us per node pair (≈ 2,680 cycles); seam 826 cycles per ATT node, 792 per MLPV node R/SCALING_4C_VS_7C_PHASE_RANK_20261116.md:168,179,182
retransmits while serving about 1 per verify step per ATT; cause = 16-deep cage RX FIFO; time cost NOT-MEASURED R/C2_6CARD_TP4_20261110.md:118-125; R/G6_TP4_EXCHANGE_RCA_20261110.md:131-161
engine slots per card 4 cages, 3 engine slots (NE = 3); a 4th needs a CSR re-layout link4/tp_link4.sv:27; R/ATT_TP4_20261108.md:33

1.2 Topology on talon2, and card 99 [M]

1.3 Per-card resources [M] (V80: 321,776 SLICE, 10,848 DSP58, 1,925 URAM, 3,741 BRAM36, 18,870 SLL per cut)

image SLICE device SLICE per die (0 / 1 / 2) DSP58 URAM BRAM SLL 1↔0 binding source
ATT 0x13BE (COMBO3's parent; no util exists for 0x13D6) 81.85 % 98.12 / 68.37 / 77.67 % 5,267 (48.6 %), 89 % on die 0 744 (38.7 %) 47.7 % 50.4 % SLICE on die 0 Q/SURVEY_PRECISION_AREA_20261116.md:163
MLPV M1361 89.6 % 81.33 / 93.44 / 94.55 % 7,504 (69.2 %) 512 (26.6 %) 27.7 % 78.2 % SLICE on dice 1-2, then DSP on die 1 SURVEY...:164; R/MLPV_G8_OWN_BUILD_20261111.md:114-120
Qwen ATT 0x139C (L256) – 87.13 / 73.06 / 5.12 % 2,757 913 1,651.5 69.7 % SLICE die 0 Q/BUILDS_20261116.md:34,168-169
Qwen MLPV 0x139D (L256) – 97.85 / 93.17 / 88.30 % 3,458 (31.9 %) 512 1,048 60.4 % SLICE die 0 (over the 95 % gate) Q/BUILDS_20261116.md:43,151-152,185

DSP owners [M]: ATT attention 3,416 (8 engines x 395) + array 1,635; MLPV array 5,120 (20 per lane) + GLM 2,048 + G8 84 (SURVEY...:183-188,229). Array today [M] (RR/GEMM_REVISION_20261117/GEMM_REVISION_REQUEST_20261117.md:38-42): Llama ATT 512 x 16 x 2 at 66 MHz (LUT lane); Llama MLPV 256 x 16 x 2 at 132 MHz (DSP lane); Qwen both 256 x 16 x 2 at 66 MHz. That is 16,384 MAC per drafter cycle per card on Llama and 8,192 on Qwen [D].

1.4 Weights per phase [M] (Q/weight_listing_20261117/*.tsv)

phase Llama bitnet_721 (tsv line) Qwen GPTQ-2 W2A16 (tsv line)
G4 q, k, v 3 x [3072, 6144] = 56.62 M (:14,16,17) 3 x 10240 x 5120 = 157.29 M (:30,38,42 qweight rows)
G6 o 9.44 M (:15) 26.21 M
G7 gate, up 2 x 25.17 M (:10,11) 2 x 83.89 M
G9 down 25.17 M (:9) 83.89 M
body 141.56 M = 35.4 MB at 2 bit 435.16 M = 108.8 MB + scales
exact lm_head (HBM) [128256, 3072] BF16, 788 MB (:6) 2-bit [320, 248320], 317.8 MB + scales
cheap lm_head built offline (not in the file) rank 128 x 248,320, 2-bit (efficient_lm_head.qweight [8, 248320])
Heads: Llama 32 x 96, Qwen 40 x 128 (Q/WHY_SLOWER_LAYERS_PHASES_20261117.md:90). The whole Llama body is 123 URAM-equivalents per sixth; weight memory is not a wall for a uniform split.

1.5 Per-node cycles today [M]

Llama, COMBO3 0xE4B713D6 x4 + M1361 x2, d11 W2, 21 nodes, census R/dw_sweep_combo3_20261116/census_combo3_P11w2.table.txt (JSON md5 a0603abb, R/DW_SWEEP_7CARD_COMBO3_20261116.md:45-49). Cycles per node = cycles per tree / 21.

ATT cyc of which array [D] MLPV cyc of which array [D]
G1 456 0 G9 850 768
G4 1,421 1,152 G11 466 0
G5 (ctx 566, P2W 8) 1,420 0 GLM + G12 kick 248 0
G6 + all-reduce 693 144 TREE (251 tiles x 8) 2,224 tile-bound
G7g (+R3, norm) 1,044 384 G8 tail 737 HBM-port-bound
G7u 557 384 PIPE_IO + uncounted 792 0
G7p 133 0
PIPE_IO + uncounted 809 0
busy 6,533 2,064 busy (0x13BE census) 5,327 768

Tree: 2.622 ms, 238.3 us per level, 124.8 us = 8,240 cycles per node [M]. Array closed forms: RR/PER_PHASE_CYCLES_BY_IMAGE_20261110.md:173-181. MLPV phases: R/SCALING_4C_VS_7C_PHASE_RANK_20261116.md:59-65. Qwen (xsim of the image sources, ideal link; no silicon census exists): ATT 15,085 root / 15,757 child, array 10,720; MLPV 15,616, G9 5,133, TREE 6,816 (Q/WHY_SLOWER_LAYERS_PHASES_20261117.md:37-52).

1.6 Step split [M]

Llama record (humaneval 80, d11 W2) Qwen GPTQ-2 (he4 + mt4, d7 W2)
step / round 7.187 ms 36.50 ms (GPU-only 21.9)
GPU verify 3.451 ms p50, measured on this set 19.14 ms
card path tree path 2.010 = host_ahead 0.510 + join_wait 1.501 card RPC 12.54 = KV rows 3.84 + tree 8.18 (issue→done 5.82) + IPC 0.52
rest host 1.72 ms [D] ≈ 4.8 ms [D]
source R/SCALING...:81-88; R/DW_SWEEP_7CARD_COMBO3_20261116.md:50-53 RR/SESSION_CONTEXT_BACKUP_20261117.md:97
Qwen humaneval 80: 140.15 @ 7.4166 (52.9 ms per round including prefill) vs 326.83 @ 7.3993 (22.6 ms) (SESSION_CONTEXT_BACKUP_20261117.md:86-87).
Host start half on 7 cards: verify result → root doorbell 1,130 us p50, of which KV_LOAD 357 and stage → doorbell 306 (R/DESIGN_FABLE_START_HALF_O_20261112.md:253-306).

The 0.516 factor. It is a ratio of deltas between two sets: (2.455 − 2.194) / (3.384 − 2.878) = 0.516 (R/SCALING...:82,71) [D]. It is not explained in any doc. Two other measured readings exist: served trees hold 15.9 of 21 nodes because of TREE_STOP = 0.757 (R/W4_SERVE_BOUND_20261115.md:124); 0x13BE → COMBO3 card-only gave 0.32 (tree path 2.194 → 2.112 with KV1H2C off, census 2.878 → 2.622; n = 1). I use 0.516 and quote the 0.32-0.757 range where it matters.

2. Calculations and hypotheses (everything that is not a measurement)

2.1 Model

One batch of W sibling nodes on N identical cards, T = MAC per drafter cycle per card: batch = once + W x per_sibling, with per_sibling = MAC / (N x T) + G7p + tree scan + G8 and once = G1 + quantiser prologues + R3/norm + G5 + 2 x all-reduce + G11 + GLM + merges + level select. Assumptions [P], each tied to a measured part:

2.2 Draft-ahead arithmetic

3. Results of the model [P]

3.1 Llama, per level (two nodes) and per d11 tree; today = 238.3 us per level, 2,622 us, 961.43 [M]

architecture N T level us tree us tok/s (0.516) range (0.32-0.757)
A uniform, one node at a time 6 16,384 246.7 2,590 963.6 –
A 7 16,384 236.1 2,479 971.4 968-976
A + B (2x array) 6 32,768 224.8 2,361 979.8 –
A + B 7 32,768 217.4 2,283 985.4 976-997
A + C (level batch) 6 16,384 173.3 1,857 1,017.3 –
A + C 7 16,384 162.8 1,746 1,026.0 1,001-1,059
A + C + fuse + TREE4 6 16,384 148.4 1,588 1,038.5 –
A + C + fuse + TREE4 7 16,384 139.3 1,492 1,046.3 1,012-1,091
A + B + C + fuse + TREE4 6 32,768 126.6 1,359 1,057.3 –
A + B + C + fuse + TREE4 7 32,768 120.6 1,296 1,062.6 1,022-1,118
same at 166 MHz 7 41,300 116.8 1,255 1,066.0 –
card time zero – – 0 0 1,215 –

Today's role split with the same levers (law P10):

step a m tree us tok/s
COMBO3 + M1361 6,533 5,327 2,622 [M] 961.4 [M]
+ NORMFUSE, G1FUSE, A8Q_OVL 5,562 5,327 2,474 971.8
+ ATT array at 132 MHz (−853, xsim, R/ATT_FASTCLK_20261117.md:45) 4,709 5,327 2,344 981.0
+ TREE4 4,709 4,227 2,177 993.2
+ hand-off halved 4,709 4,227 1,973 1,008.4

Reading:

3.2 Where a batched level goes (N 7, T 16,384, fuse + TREE4; 9,196 cycles)

G5 1,893 (21 %) · array 2 x 1,234 (27 %) · two all-reduces 1,482 (16 %) · vocabulary scan 2 x 288 · G8 2 x 211 · other once-per-level 2,208. At T 32,768 the order is G5 (24 %), all-reduces (19 %), array (16 %). G5 is the binding term; its lever is rows per cycle (P2W) or context-split engines, not cards.

3.3 All-reduce sensitivity (N 7, T 32,768, batched, fuse + TREE4)

cycles per all-reduce 150 300 549 800 1,200
share of the level 9.5 % 13.2 % 18.6 % 23.4 % 30.1 %
tok/s 1,073.9 1,069.7 1,062.6 1,055.6 1,044.6
Floor from measured parts: 192 beats at 0.758 beat per cycle = 253 cycles of serialisation at the 200 MHz link clock. Levers: link logic at 400 MHz, 24-bit partial sums (96 beats), K4-style one-step sum where cabling allows (−69).

3.4 Qwen, d7 W2 tree (13 nodes); today: issue→done 5.82 ms in serve [M]

architecture N T level us tree ms
A, today's lanes 6 / 7 8,192 548.8 / 497.1 3.57 / 3.23
A + C 7 8,192 418.1 2.76
A + B (132 MHz) 6 / 7 16,384 414.7 / 382.1 2.70 / 2.48
A + B + C + fuse + tile floor 7 16,384 260.0 1.72
A + B (512 lanes or DSP lanes too) + C + fuse + tile floor 7 32,768 202.5 1.35

4. (D) Drafting the next tree under the verify

4.1 Hit rates [M]

d7 d11 source
Llama, all drafts accepted (FPGA d11 with TREE_STOP: 22.96 %) 50.0 % 24.4 % R/accept_gap_20261112/evidence/level_model.out:1-11; R/DESIGN_FABLE_ACCEPT_GAP_7C_20261112.md:223-245
accept ends on the best-path leaf 30.4 % 17.4 % RR/VERIFY_OVERLAP_SPECULATIVE_TREE_20261110.md:49-61
hit: also bonus token = drafter top-1 there 26.0 % 15.3 % same
hit with top-2 continuations 28.0 % 16.5 % same
Qwen he4 + mt4, full depth 21.7 % – Q/D7_HOSTGAP_20261117.md:183
Qwen humaneval 80, full depth 0.86 [P fit] – section 2.2
The other 77 % of Llama steps end at a node whose bonus token is not in the tree (41.9 % at a forwarded node whose two children both miss, R/DESIGN_FABLE_ACCEPT_NOLEVEL_20261112.md:122-130); nothing drafted in advance can be reused there.

4.2 Gain as a function of hit rate [D]

Llama, tok/s:

miss cost h 0.153 h 0.204 h 0.26 h 0.4 h 0.6 break-even h
0 993 1,004 1,017 1,049 1,099 0
0.1 ms 981 993 1,006 1,040 1,092 0.062
0.3 ms 958 971 984 1,021 1,079 0.167
Qwen round (36.50 ms today, GPU-only 21.9): h 0.23 → 33.4 ms; h 0.5 → 29.7; h 0.84 → 25.0 (1.46x today's speed); h 1 → 22.8.

4.3 Verdict

5. The ideal, stated once

One image on every card, rank negotiated at power-on (the RING8 goal; 7-card autoneg converges in 18.25 us in xsim, F/PG1_EXPERIMENTS_20261022.md:13). Each card holds 1/N of every matrix, its own heads with their KV, 1/N of the vocabulary scan and a full copy of the exact head in HBM for its share of the candidates. One card faces the host; the others are fed over QSFP, so host time no longer grows with the card count (the bare 4 → 7 change cost −2.7 % for exactly that reason, R/SCALING...:95). The tree runs one batched pass per level. While the H200 verifies, the cards ingest the previous step's rows and keep extending the tree below its best leaf. Compared with today: same number of wide syncs per node, no hand-off, no idle role, one image instead of two, all 7 cards used, and batching is not blocked by a second role. The cost is area: attention plus vocabulary scan plus GLM on one card at 256 lanes, against a SLICE wall of 94-98 % on the fullest die today.

6. Migration in image-sized steps

# step image modelled gain [P] first measurement that confirms or kills it
0 No image: censuses that the model stands on – – (a) per-rank first-beat timestamp inside G6, to split hop latency, skew and retransmit (never measured); (b) Qwen silicon census (--census 20), none exists; (c) Qwen humaneval accept histogram and a GPU-only depth sweep d7 / d11 / d15 on the GPTQ-2 drafter. (c) decides whether section 4.3's depth claim is real, with no card
1 Qwen draft-ahead, host + "continue" item MLPV + ATT loop sequencer, no array change round 36.5 → about 25-30 ms replay of captured Qwen trees: hit rate and accept of continuation trees. Kill if the humaneval hit rate is below about 0.4
2 Fast array on both roles (B) Llama ATT AFC8 0xE4B713EF / 0x137F (building); Qwen MLPV at 132 MHz with the per-column zero Llama +1 %; Qwen tree −25 to −30 % AFC8 routed WNS, then its census: G4 1,421 → about 914 per node
3 MLPV vocabulary scan + G8 on more cards: card 99 as a third MLPV shard (two cables to 50 and 63, or relay through bd / e1) MLPV image with a 3-way shard map Llama: TREE 2,224 → about 1,560, G8 737 → 490 per node, ≈ +1 % (the MLPV is co-critical); Qwen more (TREE is 44 % of its MLPV node) joint census: does tree issue→done fall by more than a third of the MLPV saving? If not, the loop is ATT-bound and this step waits
4 Level batching on the role split, with the batch release fixed (dynamic groups) ATT W2 column path (B3 / B5 lineage) + MLPV two-column release level 238 → about 190 us [P from the −29 % 2-child xsim, R/SEVENCARD_PROJECTION_20261109.md:51] W2 census must be below serial 238 us per level; today's W2 is +26 %. Kill for the role split if the MLPV still serialises
5 Unified card, fit probe only out-of-context place of attention (6 engines) + 256-lane array + scan + GLM + G8 none (a probe) worst-die SLICE under 95 % and SLL under 18,870. Kill A if it does not place at 256 lanes
6 Unified image on 6 cards (17, 3d, bd, e1, 50, 63), existing cables + 4th engine slot one image, rank by negotiation Llama about 1,017-1,038; Qwen tree about 2.2-2.7 ms 6-rank all-reduce on silicon under 800 cycles, and a level below 190 us
7 Seventh card and single host-facing card same image, 2 cables for 99 (user's hands) Llama +0.8 % over step 6; Qwen −10 % tree host start half (1,130 us today) flat against the 6-card value
Steps 1-4 keep today's two images and are each useful alone. Step 5 costs no flash and decides whether 6 and 7 are worth building. If step 5 fails, the evolved role split of section 3.1 (about 1,000 on Llama) plus step 1 on Qwen is the plan.

7. NOT-MEASURED, stated plainly