Track 2 · talon 8×V80 ring · timing diagnosis · 2026-10-17

Why no ring image closes timing

The design that works on talon today has no slack left. Adding the QSFP ring pushes it over the edge in two separate places: the GT transmit clock and the 225 MHz engine clock.

Labels: [M] measured from Vivado reports · [D] derived by arithmetic · [P] projected. Every underlined number opens the report at that line, read-only, in the build-report browser.

Result, 2026-10-17 12:47 UTC: the recovery image closed timing. v8 (0xE4B71262: C_COLS 2, 225 MHz, GT island, crumbc fanout fix) finished post-route optimisation with WNS +0.003 / WHS +0.010, 0 failing endpoints [M]. The worst clock is hbms_clk +0.003, which is +0.093 true after adding back the 0.090 ns setup margin (0x1254 was released at +0.002 / +0.092 the same way). The GT clocks txoutclk_3 +0.047 and txoutclk_2 +0.073 were −1.813 and −0.487. Finding F1299.

Silicon, 2026-10-17 ~18:45 UTC [M]: flashed to partition 1 on all 8 talon cards (FLASH_EXIT=0 ×8, device_boot OK ×8). QM-38 PASS. HBMS ladder 52/52 PASS on all 8 cards. Ring: 16/16 links up; discovery finds one 8-card cycle; 30 s BIST PASS on all 16 directions with 0 word errors, about 4.19 GB/s each. Finding F1300.

Diagnosis. 0x1254 s2, the image flashed on all 8 talon cards, met 225 MHz with +0.002 ns of slack and contains no ring hardware. 0x1256 adds the talon ring core, 8 GT links and the HBMS all-reduce bridge. That logic lands mostly in SLR0, which goes from 89.8 % to 94.2 % SLICE, and the result fails by −1.813 ns [M].

The two failures need different fixes. The GT failure is a floorplan error and has a clean fix. The engine-clock failure is caused by how full the device is; fanout fixes help, and lowering the clock to 200 MHz is the sure fix.

Evidence 1

Known-good image against the failing ones

Final routed timing per clock. Setup slack must be at least +0.03 ns, and hold slack at least +0.010 ns, to release a PDI.

ClockDomain0x1254 s2 (on talon)0x1256 s1rR8_BEST s1c
hbms_clkengine, 225 MHz+0.002−0.492 · 61,366 ep−0.631 · 68,940 ep
hbms_mclkHBM NMU side, 400 MHznot presentnot present−1.210 · 97,123 ep
ch0_txoutclk_3QSFP GT transmitnot present−1.813 · 3,475 ep−1.766 · 2,689 ep
ch0_txoutclk_2QSFP GT transmitnot present−0.487 · 2,069 ep−0.536 · 871 ep
ch0_rxoutclk_2 / _3QSFP GT receivenot present+0.108 / +0.058+0.158 / +0.154
talon_clk_wizring core, 150 MHznot present+0.060+0.035
clk_pl_2shell+0.120+0.023+0.003
OverallWNS +0.002 · WHS +0.010WNS −1.813WNS −1.766

Units: ns of setup slack; "ep" = failing endpoints. [M] from route / post-route phys_opt timing summaries.

Evidence 2

The device is full

SLICE utilisation per SLR after placement. Above about 85 %, routing detours grow quickly. The known-good image was already at 96.9 % in SLR1, which is why it had only 0.002 ns to spare.

What the ring added (SLR0 unless noted)0x1254 s20x1256 s1rChange
SLICE used101,097105,987+4,890
CLB LUTs587,666615,056+27,390
LUT as memory (ring FIFOs)8,26812,288+4,020
Block RAM tiles120175+55
SLR1 ↔ SLR0 crossing wires used6,482 (34%)9,442 (50%)+2,960
Ring IPs in synthesis (talon core; 2 Aurora links)none30,323 + 2 × 10,355 LUT+51k LUT
Cause A

GT transmit clock: FIFO on the wrong die

What happens

The QSFP GT quads sit in SLR1. The hook put the whole talon core in SLR0, including each port's transmit FIFO and the first synchroniser stages, which run on the GT clock. A GT-clock path then crosses the SLR boundary twice inside a 2.48 ns period (402.8 MHz). That cannot close.

Worst path −1.813 ns on ch0_txoutclk_3; the same class in R8_BEST s1c is −1.766 [M].

slack −1.813 period 2.482 ns datapath 4.811 ns route 96.2% g_port[1].u_port/u_cdc/u_tx_fifo/.../rdp_inst/count_value_i_reg[2]_replica → (RAMD32, LUTRAM) → .../gen_sdpram.xpm_memory_base_inst/gen_rd_b... clock skew 0.586 ns

The FIFO's read side runs on the GT clock, yet its LUTRAM and read pointer sit in the far SLR. 4.6 ns of wire inside a 2.48 ns period.

Fix in flight: SerDes island (hook v4)

A new pblock in SLR1, next to the quads, holds every GT-clocked register, both async FIFOs, every synchroniser and the first capture flop. The FIFO becomes the only SLR crossing, on its 150 MHz side (6.67 ns).

checked on the netlist 12,428 of 12,428 island cells placed in SLR1; 630 TX and 1,258 RX FIFO RAM cells [M]. Not yet timed in a routed build.

Cause B

Engine clock: wires too long in a full fabric

The hbms_clk paths fail by about 0.49 ns. Almost all of the delay is routing, not logic. Many are single registers driving thousands of loads across a crowded region.

slack −0.492 levels 6 datapath 4.657 ns route 88.9% u_tile/g_port[6].u_port/g_dc.u_ceng/f1_reg → acc_reg[92][2][21] slack −0.491 levels 0 datapath 4.646 ns route 98.0% u_tile/g_port[3].u_port/g_dc.u_ceng/l2_reg → sh_reg[11][0][19]/CE slack −0.492 levels 1 datapath 5.136 ns route 97.3% g_tile[0].u_tile/bnd_p_reg[0][165] → g_port[10].u_port/g_dc.w2on_reg_rep__20 slack −0.491 levels 4 datapath 4.533 ns route 91.0% u_central/u_vpu/ex_v_reg[8] → u_vpu/tk2_am_reg[11]

Open each path in the report: path 1 · path 2 · path 3 · path 4 · all hbms_clk paths

Fix 1: cut fanout (v5)

In e8_crumbc_pd, the registers v1, f1 and l2 each drove all 128 rows × 4 columns of 27-bit accumulators (about 14k loads). Each is now replicated once per 8-row group, with the same latency and the same values.

xsim equivalence 200,000 random cycles, 0 mismatches; a deliberately broken copy was caught [M].

limit Only the u_ceng class is addressed. The tile broadcast and VPU paths are still open.

Fix 2: 200 MHz (v6, BUILD_ID 0xE4B71261)

A 5.000 ns period instead of 4.444 ns adds 0.556 ns, which turns the measured −0.492 into about +0.06 ns [D]. That clears the +0.03 gate on its own.

trade-off Engine and HBM-side throughput drop by about 11 % [D]. For a fallback ring image, a working PDI is worth more than the extra 25 MHz. Your call which one to flash.

Measured: every failing endpoint

Where the 66,910 failing paths are

Full netlist dump of 0x1256 s1r (one row per failing endpoint). Source: council H2, council H1.

Engine clock class (61,366 ep)EndpointsFix
crumbc f1/l2/v1 fanout23,416v5 replication ✓
crumbc product broadcast6,844v7 replication
ATTN-W (SLR1)14,146v7 registers
VPU (SLR1)5,190v7 registers + move to SLR0
other (attn, leng, t1x, cab, ring, axi)≈ 11,770density (C_COLS 2)
crossing an SLR boundary512not the problem
GT clock class (5,544 ep)ShareFix
link_rst_n as clock-enable into PRBS test registers45 %v9b: register + replicate the reset (BIST kept)
TX/RX CDC FIFOs29 %island + FIFOs in BRAM
Aurora IP internals14 %density in SLR1

Failing paths are 87–98 % routing delay on 0–1 logic levels and almost all stay inside one SLR: SLR0 28.5k, SLR1 24.2k, SLR2 8.1k. This is a density signature. The known-good 0x1254 needed a placement estimate of +0.144 ns to route at +0.002, so each new build must place at about +0.1 or better.

Builds now

What is building, and when it lands

BuildContentsHostExpected resultPDI ETA (UTC)
0x1256 v5 s1/s2/s3GT island + crumbc fanout fix; three directive setsolegvm3, olegvm2, midyardnear 0, unproven [P]17:30–19:00
0x1261 v6v5 at 200 MHzolegvm3 (queued)≈ +0.06 on hbms_clk [D]≈ 22:00
0x1256 v4 s1GT island onlyolegvm3≈ −0.49 [P]≈ 18:00
R8_BEST BD5A + B RTL, crumbc fix, floorplan D islandolegvm2 (queued)unknown≈ 22:00
R8_BEST kit_B / kit_BMolder fix setsolegvm3 / olegvm2−0.76 / −1.87 [M, mid-route]diagnostic only
Fix options

Ways to free the fabric, least impact first

✓ means the fix is in that build. Options 4–12 are being tried now by parallel agents (P = pipeline registers, R = no-cost trims, T = trade-offs). Agent results open in the council report browser.

Decision (user, 2026-10-17): C_COLS 2 at 225 MHz. Agent T found every talon serving result is W8, which does not use the W2 crumbc columns. C_COLS 4 → 2 frees about 150k LUT in each of SLR0 and SLR2 at 0 W8 tok/s cost [D]. 200 MHz would cost about 11 % on every model. The recovery image is now v8 (BUILD_ID 0xE4B71262): GT island, crumbc fanout fix, C_COLS 2, 225 MHz. v7 adds the pipeline registers and no-cost trims on top. v6 (200 MHz) stays as the safety net.

#OptionCostAgentv4v5v6BD5
1Pipeline registers on the remaining engine-clock paths (tile broadcast, VPU tk2, ATTN-W, kvnet)+1–2 cycles latencyP–––✓
2crumbc fanout replicationnone, bit-exactdone–✓✓✓
3SerDes island next to the GT quadsnonedone✓✓✓✓
4Ring FIFOs from LUTRAM to block RAMnoneR––––
5Compile out ring BIST and debugno on-card self-test unless strappedR––––
6Retiming and fanout options in synthesis and phys_optruntimeR––––
7Rebalance SLRs for the talon core and hub blocksnoneRpartpartpartpart
8400 MHz HBM-side registers (reset tree, read data, CE)nonedone–––✓
9Engine clock 200 MHz≈ −11 % engine throughputT––✓–
10crumbc C_COLS 4 → 3 or 2 (≈ 0.6M LUT at C=4)slower W2 / spec verifyT––––
11Narrower accumulatorsmust stay bit-exactT––––
12Fewer engine portsloses HBM bandwidthT––––

Will it fix setup at 225 MHz?

  • GT clock: likely yes with the island [P]. The receive side, already next to the quads, passes at +0.06 to +0.11 in the same build [M].
  • Engine clock: unlikely on v5 alone. The known-good image had only +0.002, so 0.52 ns must be recovered. v5 covers one class; with the pipeline registers it may reach about zero, with thin margin [P].

And at 200 MHz?

  • Likely yes, but thin [D]: −0.492 + 0.556 = +0.064, which clears the +0.03 gate by only 0.034 ns. Seed-to-seed noise can be 0.1–0.2 ns.
  • Real margin: v6 combined with the pipeline registers.
  • R8_BEST: not in the next build. It has two failing clocks and a fuller device.
What went wrong

Process failures on my side

  1. No baseline comparison. 0x1254 s2 had already closed at 225 MHz. Comparing it with 0x1256 in the first hour would have shown that the ring logic pushed a zero-margin design over the edge. I did that comparison only today.
  2. 14-hour builds used as the experiment loop. Each idea cost a full synthesis + place + route. Placement-only and implementation-from-checkpoint runs answer the same question in 2–3 h.
  3. Hooks launched without running them on a netlist. A g_port*/u_port pattern matched 0 cells, and a full build died at placement. A new floorplan was launched without its prebuild gate and was rejected.
  4. Builds not watched. One seed died and its host sat idle for 10 h. Another was frozen in synthesis for 15 h. A sub-agent's pattern kill destroyed two builds.
  5. Shallow verification. Goldens come from the Python ISA emulator on synthetic weights, not from GPU captures. The ring and QSFP sim was not re-run after changes, and there is no PCIe test case. Two speculative-decoding cases never finished.
  6. Build sprawl. Each new fix became a new build instead of replacing the old one, leaving 8 builds of which 4 can actually pass.
Changed from now

How the work runs from here

Fast loop first

  • Every idea is tried first as a placement-only run on an existing optimised checkpoint (2–3 h). It becomes a full build only if the estimate is positive.
  • Incremental implementation against the known-good 0x1254 routed checkpoint, so the ring logic is the only thing placed fresh.
  • Each failure is analysed per endpoint across the whole netlist: hierarchy, SLR, route share and congestion. Not just the top 8 paths.

Release discipline

  • At most 2 builds per image: the best candidate and one variant. Superseded builds are stopped at once.
  • Before any PDI reaches talon: the full xsim gate set on the final RTL, the ring 8-rank all-reduce test, and a GPU-captured golden case.
  • A council agent is now running these experiments in parallel (GT island check, failing-path histogram, 200 MHz margin, incremental from 0x1254).