# AMD Alveo V80: our platform (agent view)

## [Card] The V80 at a glance

One Versal HBM device (xcv80, VH1782) with three SLRs, 32 GB HBM2e, four QSFP cages and a PCIe Gen5 x8 host link; we run 12 of them across two machines.

The Alveo V80 is a PCIe card built around a single Versal HBM-series device, part xcv80-lsva4737-2MHP-e-S in our Vivado projects. Unlike a GPU there is no fixed compute engine: everything we run is a PDI (programmable device image) that we design, build and flash ourselves. The PDI always has two layers. AMD's AVED shell owns the fixed plumbing: the CIPS block (PCIe controller, platform management controller PMC, processors), the network-on-chip (NoC), the management controller firmware (AMC) and the SMBus/clocking IP. Our RTL sits in the programmable logic (PL) next to it and talks to the host through BAR registers on PCIe function 0 and through QDMA on function 1.
The device has no AI Engines (F0715, read from the Vivado part properties), so every multiply we do is either a DSP58 hard block or LUT logic. That single fact shapes both tracks: Track 1 keeps a 2-bit drafter resident in URAM next to packed DSP58 lanes; Track 2 streams 8-bit weights out of HBM through 32 lockstep ports into DSP58 columns.
We run V80s in two places. talon is an 8-card server (cards on PCIe buses 17, 3d, 50, 63, 99, bd, cf, e1) cabled as a QSFP ring; it carries Track 2 (HBM-streaming serving of Llama 1B/3B/8B/70B) and Track 3 (constant-folded compute). box1 holds 4 cards plus an H200 and carries Track 1 (FPGA EAGLE drafter + GPU target, the 350.16 tok/s batch-1 record, F1125).
How to read the rest of this explorer
Start with SLRs (why placement is three separate problems), then HBM and NoC (where bandwidth comes from), then PCIe / QDMA / AMI (how the host reaches the card), then the Operations group (flash, boot, recovery, ladder, cycle counters) and finally the Toolchain group (license, Vivado, synthesis, implementation, reading reports, xsim, mesh).Worked example: what "an image" means for us0xE4B71254 s2 (PDI md5 259a2800) is today's talon serving image. Its 32-bit BUILD_ID 0xE4B71254 is a strap in our RTL that the host reads at BAR0+0x10100 before doing anything (see *PCIe*). The same AVED shell, with different RTL, becomes the Track 1 drafter images (HEAD/TAIL) on box1. Changing anything in our RTL means a new full build (~13 h, see *Vivado*), a new BUILD_ID, a flash and a silicon ladder.

- LUT: 2,574,208 [M]
- registers: 5,148,416 [M]
- DSP58: 10,848 [M]
- BRAM36: 3,741 [M]
- URAM288: 1,925 [M]
- SLRs: 3 [M]
- HBM2e: 2 stacks, 32 pseudo-channels, 32 GB, 819.2 GB/s [S]
- AI Engines: none (F0715) [M]
- cards in use: talon 8, box1 4 [M]
- talon 3B serving, 1 card: 54.8 tok/s batch-1 (F1260) [M]
- aved: AVED Hardware Design: V80 (hierarchy, CIPS, NoC, memory pages) -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/Hardware+Design+-+V80.html
- aved: AVED V80 hierarchy overview -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BV80%2B-%2BHierarchy%2BOverview.html
- amd: DS960 Versal HBM Series data sheet -> https://docs.amd.com/r/en-US/ds960-versal-hbm
- path: V80 facts with sources -> /shared/rnd_hq/2026-10-15/tracks/v80_page/FROM_RESEARCH.md
- page: Architecture Book -> /p/eagle4-arch-book/
- page: Model Weights -> /p/eagle4-model-weights/

## [Fabric] SLRs: three dies on one package

SLR0 (bottom), SLR1 (middle, all four QSFP GTM quads), SLR2 (top, all 148 HBM sites), joined by 18,870 SLL wires per boundary.

The V80 device is built with stacked silicon interconnect (SSI): three super logic regions (SLRs) on one interposer. Each SLR is effectively its own FPGA with its own LUTs, DSPs and memories. Signals between SLRs travel on super long lines (SLLs): there are 18,870 of them per boundary [M], and every SLL crossing adds delay, so a design that ignores SLR boundaries both runs out of wires and loses timing.
Three physical facts decide our floorplans:
All 148 HBM sites and 89 of the 152 NoC NMUs are in SLR2 (F1113). Anything that streams from HBM through the direct BLI ports must live in SLR2 or pay SLLs to reach it.All four QSFP GTM quads are in SLR1 (F0193, F0882): e.g. cages P3/P4 are banks 112/111 = GTM_QUAD_X0Y10/X0Y9 in CLOCKREGION_X0Y7. Every QSFP link therefore puts logic into the middle SLR.SLR1 is where our hubs have historically lived, so SLR1 is where we run out of slices first.
Floorplans we have used
0x1254 (today's talon image): far port tile (16 ports + KV) in SLR2 next to HBM, central hub (activation buffer, sequencer, VPU, attention) in SLR1, near port tile (16 ports, 2,080 DSP) in SLR0 reaching HBM through register slices across two boundaries.Floorplan D (R8_BEST 0xE4B7125F, building): compute ports in SLR2 + SLR1 (next to HBM), hub in SLR0. Weights never cross into the hub SLR; SLR1-SLR0 SLL use is modelled to drop from ~69% to ~28% [D]. The ring's GT-side logic still has to sit in SLR1 next to the quads.How to read the slr reportSection 1 lists SLR2 &lt;-&gt; SLR1 | used | available | %. On 0x1254 s2 that is 7,923 of 18,870 (42%) and SLR1 &lt;-&gt; SLR0 6,482 (34%) [M]. Section 3 gives SLICE, LUT, FF, BRAM, URAM and DSP per SLR. Read SLICE %, not LUT %: a slice is the placement unit, and an SLR at 81% LUT can be 97% SLICE when its LUTs are spread thinly.

```
# per-SLR utilisation and SLL crossings of a placed / routed design
report_utilization -slr -file slr_util.rpt
report_utilization -pblocks [get_pblocks] -file pblock_util.rpt
# our report on the talon image: impl_1/hbms_np32s6_slr_util_placed.rpt
#   section 1 = SLR connectivity (SLLs used per boundary), section 3 = per-SLR CLB / DSP / RAM
```

- SLR0 LUT / DSP: ~900.2k / 3,984 [D]
- SLR1 LUT / DSP / URAM: ~820.2k / 3,456 / 624 [D]
- SLR2 LUT / DSP: ~853.8k / 3,408 [D]
- SLL per boundary: 18,870 [M]
- 0x1254 SLICE SLR0 / SLR1 / SLR2: 89.8% / 96.9% / 93.8% [M]
- 0x1254 LUT SLR0 / SLR1 / SLR2: 65.3% / 81.0% / 68.7% [M]
- 0x1254 URAM SLR0 / SLR1 / SLR2: 0 / 84 / 0 [M]
- 0x1254 SLL SLR2-SLR1 / SLR1-SLR0: 42% / 34% [M]
- PITFALL: Splitting HBM ports across SLR1/SLR2 (0x1218) needed 22,935 SLR1-SLR2 SLLs against 18,870 available: it does not fit (F1113).
- PITFALL: SLR1 SLICE is the wall on our drafter images (A8 99.56%, F4 98.17%); a TAIL at 99.44% failed Aurora TX timing with one link (F0193). Each extra GTM link costs ~5,376 slices of hard pblock per quad clock region (F0585).
- PITFALL: A central URAM bank feeding every port across SLRs needs ~32 kbit per cycle of SLLs: infeasible. Keep memories in the SLR of their consumer.
- amd: UG949 Super Long Line (SLL) routes -> https://docs.amd.com/r/2021.2-English/ug949-vivado-design-methodology/Super-Long-Line-SLL-Routes
- amd: UG949 SSI technology considerations -> https://docs.amd.com/r/2021.2-English/ug949-vivado-design-methodology/SSI-Technology-Considerations
- amd: UG906 Report SLR crossing -> https://docs.amd.com/r/en-US/ug906-vivado-design-analysis/Report-SLR-Crossing
- amd: UG835 report_utilization -> https://docs.amd.com/r/2025.2-English/ug835-vivado-tcl-commands/report_utilization
- path: FROM_RESEARCH.md sections 1 and 5 -> /shared/rnd_hq/2026-10-15/tracks/v80_page/FROM_RESEARCH.md
- path: floorplan D design -> /shared/rnd_hq/2026-10-15/tracks/slr_tp/DESIGN.md

## [Fabric] LUTs, slices and LUTRAM

2.57M six-input LUTs in slices; also usable as small distributed RAM (LUTRAM). Slice packing per SLR, not LUT count, is usually the limit.

The configurable logic block (CLB) is the general-purpose fabric. Each Versal slice holds LUT6s and flip-flops plus carry logic. A LUT6 is a 64-entry truth table: any function of 6 inputs, or two functions of 5 shared inputs. LUTs carry all our random logic: control FSMs, muxes, adder trees, and the 2-bit / 3-bit / ternary dot products that do not map well to DSPs (Track 2's W2/W3 "LUT engines", Track 1's G9 2-bit down projection).
Some slices (SLICEM) can turn their LUTs into LUTRAM: small distributed memories of 64 bits per LUT with asynchronous read. Track 3 (TUNIT constant fold) stores ternary weights as LUT constants rather than reading them from memory at all.
Why "81% LUT" can mean "full"
Placement happens in slices, and timing-driven placement spreads logic to keep wires short. On the talon image 0x1254 s2, SLR1 is at 81.0% LUT but 96.85% SLICE [M]: most slices are partly used. When SLICE reaches the high 90s, the placer has no room to legalise and routing congestion explodes, even though LUTs remain.How to read the congestion reportreport_design_analysis -congestion lists congested windows per direction (North/South/East/West) with a level (window size 2^level CLBs). Levels of 5 or more in an SLR mean the router will struggle there; cross-check which hierarchy owns the cells in that window ("Top 10 cells" column) before re-seeding.

```
report_utilization -hierarchical -hierarchical_depth 4 -file util_hier.rpt
report_design_analysis -congestion -file congestion.rpt     # after place/route
report_utilization -slr -file slr_util.rpt                   # SLICE per SLR
```

- device LUT: 2,574,208 [M]
- 0x1254 LUT SLR0 / SLR1 / SLR2: 587,666 / 664,520 / 586,207 [M]
- 0x1254 SLICE SLR1: 96.85% [M]
- 0x1254 LUT as memory: 29,563 (2.3%) [M]
- TUNIT fold: 3.01 M ternary weights in LUT constants [M]
- PITFALL: Vivado silently falls back to LUTRAM when `ram_style = "ultra"` is infeasible (`Synth 8-6849`): FOLDRING8 s3/s5 put ~50 Mbit of KV memory there and could never fit. Gate synthesis on 0 `Synth 8-6849`.
- PITFALL: The 0x1257 NP64 cut was forced by the crumbc C=4 W2 array (~35.7k LUT per port), not by the W2/W3 engines themselves (~7-9k LUT per port) [M OOC].
- amd: AM005 CLB resources -> https://docs.amd.com/r/en-US/am005-versal-clb/CLB-Resources
- amd: AM005 LUTRAM -> https://docs.amd.com/r/en-US/am005-versal-clb/LUTRAM
- amd: UG949 report_design_analysis congestion report -> https://docs.amd.com/r/2021.2-English/ug949-vivado-design-methodology/Report-Design-Analysis-Congestion-Report
- page: TUNIT constant fold -> /p/eagle4-tunit-constant-fold/

## [Fabric] DSP58 slices

10,848 hard multiply-accumulate blocks (27x24 multiplier, 58-bit accumulator); how Track 1 packs two products per DSP and why Track 2 leaves three of four columns idle at batch 1.

A DSP58 is a hard arithmetic block: a pre-adder, a 27 x 24 signed multiplier, and a 58-bit accumulator with cascade paths to its neighbours. It runs far faster and cheaper than the same multiply in LUTs. Versal also offers vector modes (INT8 dot products, complex and FP32 modes), but our designs use the plain multiplier with operand packing: put two small numbers into one wide operand so one multiply produces two products in separate bit fields.
Track 1: DOT_E3 packing
The Track 1 drafter computes a x (w_L + 2^23 * w_L+1), so one DSP58 yields two 2-bit-weight products in non-overlapping fields; 17 DSPs make 32 products, bit-exact against the golden model. With INT16 activations the packing caps at 2 products per DSP, because three fields of &gt;= 21 bits do not fit a 24/27-bit operand (F0715 [D]).
Track 2: the HBMS GEMV array
Each of the 32 HBM ports drives 32 rows x MD = 4 DSP58 columns (hbml_dspeng). One 256-bit HBM beat carries 32 int8 weights (one weight per row), and all 4 columns can multiply that same weight by up to 4 different activation rows. At batch-1 decode there is only one activation row, so 3 of 4 columns are idle [D]; they are there for speculative-decoding verify rows (M &lt;= 4). HBM-FOLD (R8_BEST 0xE4B7125F) reuses the idle columns to absorb up to 4 HBM beats per cycle (K-fold), which is how it raises the stream rate without adding DSPs.

```
report_utilization -cells [get_cells -hier -filter {REF_NAME =~ DSP*}]
report_utilization -slr -file slr_util.rpt     # &#x27;DSP Slices&#x27; per SLR
# check the DSP mapping of a critical multiply after synthesis:
report_cells -hier -filter {REF_NAME =~ DSP58*} [get_cells u_core/u_attn/*]
```

- device DSP58: 10,848 [M]
- 0x1254 DSP SLR0 / SLR1 / SLR2: 2,080 / 1,225 / 2,080 (49.6% total) [M]
- HBMS W8 MACs per port: 32 rows x 4 columns = 128 [D]
- columns busy at batch-1: 1 of 4 [D]
- Track 1 DOT_E3: 17 DSP per 32 products [M]
- packing cap with INT16 activations: 2 products / DSP (F0715) [D]
- PITFALL: Synthesis is free to absorb registers or split one logical multiply across DSP and fabric differently per lane. On 0x1250 lanes 4/5 of the attention normalise multiply mapped differently from the other six and failed on silicon; fixed in 0x1254 with explicit products + `keep_hierarchy` per lane.
- PITFALL: On FOLDRING8 the PV DSPs absorbed the URAM output register and made the URAM inference infeasible; a `keep` on the pipeline register fixed it.
- amd: AM004 DSP58 architecture -> https://docs.amd.com/r/en-US/am004-versal-dsp-engine/DSP58-Architecture
- amd: UG1344 DSP58 primitive -> https://docs.amd.com/r/en-US/ug1344-versal-architecture-libraries/DSP58
- page: HBMS engine architecture and cycles -> /p/talon-hbms-engine-arch-cycles-20261015/
- path: HBM-FOLD design (K-fold onto idle columns) -> /shared/rnd_hq/2026-10-15/tracks/arch_fanout/bandwidth/DESIGN.md

## [Fabric] URAM and BRAM

1,925 UltraRAM blocks (288 Kb each, ~67.7 MiB) and 3,741 BRAM36 (~16.4 MiB) of on-chip memory; how each track uses them and how to make Vivado actually infer them.

UltraRAM (URAM288) blocks are 4K x 72-bit synchronous RAMs (288 Kb), two ports, one-cycle registered read, cascadable into deeper memories without fabric. Block RAM (BRAM36) blocks are smaller (36 Kb), more flexible in width, and true dual-port. Together they give ~84 MiB of on-chip memory [D], read at aggregate bandwidths far above HBM, but only if the design keeps each memory next to its consumer.
How the tracks use them
Track 1 keeps the whole 2-bit drafter body (41.58 MiB, ~174M weights) resident in URAM beside each GEMV lane, so weights never move during a forward pass. That is why Track 1 is compute-bound, not bandwidth-bound.Track 2 (HBMS) uses one 2 MiB URAM block as the CAB (central activation buffer: activations and the short KV window), 84 URAM total, all in SLR1 [M]. Weights stream from HBM.URE (planned, 0xE4B712E0) makes Track 1's residency programmable: 384 lanes x (4 URAM + 4 BRAM36), ~56.6 MB per card [D].

```
// make Vivado infer URAM: one process, <= 2 ports, 64- or 72-bit wide banks, registered output
(* ram_style = &quot;ultra&quot; *) reg [63:0] bank [0:4095];
// inside one clocked always block on the rising edge of clk:
//   if (we) bank[wa] <= wd;
//   rd_q <= bank[ra];         // keep this output register; do not let a DSP absorb it
# gate every synthesis log:
grep -c &#x27;Synth 8-6849&#x27; synth.log    # infeasible ram_style -> must be 0
grep -c &#x27;Synth 8-11357&#x27; synth.log   # 3-D RAM / record RAM runtime issue -> must be 0
```

- URAM total: 1,925 x 36 KiB = 67.7 MiB [M]
- BRAM total: 3,741 x 4.5 KiB = 16.4 MiB [M]
- 0x1254 URAM used: 84 (4.4%), all SLR1 [M]
- 0x1254 BRAM used: 427.5 tiles (11.4%) [M]
- Track 1 drafter body: 41.58 MiB 2-bit [M]
- URE capacity: ~56.6 MB per card [D]
- PITFALL: Infeasible `ram_style = "ultra"` falls back to LUTRAM **without failing the build** (FOLDRING8 s3/s5: ~50 Mbit of KV in LUTRAM, could never fit). Gate on 0 `Synth 8-6849` and 0 `Synth 8-11357`.
- PITFALL: Wide 3-D arrays (e.g. 6,144-bit rows) are not inferable as URAM; write explicit 64-bit banks.
- PITFALL: A central URAM bank feeding ports in other SLRs is infeasible on SLLs; place banks in the consumer's SLR.
- amd: AM007 UltraRAM introduction -> https://docs.amd.com/r/en-US/am007-versal-memory/UltraRAM-Introduction
- amd: AM007 UltraRAM primitives -> https://docs.amd.com/r/en-US/am007-versal-memory/UltraRAM-Primitives
- amd: UG901 UltraRAM coding templates -> https://docs.amd.com/r/en-US/ug901-vivado-synthesis/UltraRAM-Coding-Templates
- amd: UG912 RAM_STYLE property -> https://docs.amd.com/r/2024.1-English/ug912-vivado-properties/RAM_STYLE
- aved: AVED V80 memory resources -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BV80%2B-%2BMemory%2BResources.html
- path: URAM residency handoff -> /shared/rnd_hq/2026-10-15/tracks/uram_resident/HANDOFF_TO_URE.md
- page: Weight residency -> /p/eagle4-weight-residency/

## [Memory] HBM2e and the 64 NMU ports

2 stacks x 16 pseudo-channels = 32 PCs of 1 GB; 64 PL ports (HBM_NMU / BLI) at up to 256 bit; 819.2 GB/s spec. Why today's image streams only ~230 GB/s and how HBM-FOLD lifts it.

The V80 package carries two HBM2e stacks. Each stack has 8 channels split into 2 pseudo-channels (PCs), so the card has 32 PCs of 1 GB, each 64 bits wide at 3.2 Gb/s per pin = 25.6 GB/s per PC, 819.2 GB/s total [S]. The PL reaches HBM through HBM_NMU ports (the "BLI" ports in our notes): there are 64, two per PC, each at most 256 bits wide [S]. At 400 MHz one NMU moves 12.8 GB/s, so 64 NMUs x 256 bit x 400 MHz = 819.2 GB/s exactly matches the PCs.
All HBM sites are in SLR2 (F1113). The alternative path through the general NoC from SLR1 delivered only 51.7 GB/s for 16 ports, against 163.9 GB/s on 16 direct BLI ports (F1152) [M]; our streaming engines therefore always use the BLI ports.
Why decode speed is an HBM-port question
At batch-1 decode every weight is used once per token, so a token costs "all weight bytes / stream rate". The silicon cycle profile shows 92.9% (1B) and 94.2% (3B) of a token is weight streaming [M], and every GEMV fits 333 + 1.0333 x beats per port cycles. Today's image 0x1254 uses 32 NMUs at 225 MHz, wired to HBM0..HBM7 PORT0..3: only 16 of 32 PCs, one stack, 16 GiB of reach, about 230 GB/s [D]. The HBM itself waits only 3.7-3.9% of the token (stall_tot) [M]; the limit is our port count and clock, not the memory.Worked example: 3b token timeLlama-3.2-3B at W8 streams ~3.2 GB per token. At 32 x 32 B x 225 MHz = 230.4 GB/s that is ~13.9 ms of pure beats; the measured card time is 15.65 ms (F1260), the rest being per-GEMV start-up (~333 cycles each), vector ops and attention. HBM-FOLD's 2 NMUs per port at 400 MHz with a 4-column K-fold reached 3.51 beats/cycle/port in xsim [M], i.e. ~3.4x the bytes per engine cycle.

```
# 0x1254 port window: 512 MiB per port (two ports share one PC)
addr = 0x40_0000_0000 + port_id * 0x2000_0000
# R8_BEST / HBM-FOLD: one PC per engine port (1 GiB window), 32 GiB reach
addr = 0x40_0000_0000 + port_id * 0x4000_0000
# NMU wiring rule in the block design: HBMnn_AXI -> HBM(nn/4)_PORT(nn%4), nn = 0..63
```

- spec bandwidth: 819.2 GB/s (32 PCs x 25.6) [S]
- NMUs: 64, 256-bit max, 12.8 GB/s at 400 MHz [S]
- 0x1254 ports: 32 NMUs at 225 MHz, 16 of 32 PCs [M]
- 0x1254 streamed: ~230 GB/s [D]
- HBM stall (stat 581): 3.7-3.9% of a token [M]
- NoC vs BLI, 16 ports: 51.7 vs 163.9 GB/s (F1152) [M]
- HBM-FOLD xsim: 3.51 beats/cycle/port [M]
- PITFALL: The port count, not HBM, is the bottleneck: per-port URAM prefetch FIFOs can only hide the 3.7-3.9% stall plus gaps (<= ~12% [D]).
- PITFALL: Lockstep coupling: any one of 32 port FIFOs dropping below 32 beats halts all 32; per port the dip is only 0.7-0.8%, the union 3.3% [M].
- PITFALL: An NP64 whole-core build at 400 MHz failed at WNS -3.507 ns (F1035): clock only the NMU side at 400 MHz, keep the engine at 225 MHz behind a dual-clock FIFO.
- amd: PG313 HBM_NMU -> https://docs.amd.com/r/en-US/pg313-network-on-chip/HBM_NMU
- amd: PG313 NMU selections (HBM system considerations) -> https://docs.amd.com/r/en-US/pg313-network-on-chip/NMU-Selections
- amd: PG313 routing use cases and HBM subsystems -> https://docs.amd.com/r/en-US/pg313-network-on-chip/Routing-Use-Cases-and-HBM-Subsystems
- aved: AVED V80 memory resources -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BV80%2B-%2BMemory%2BResources.html
- path: HBM-FOLD design -> /shared/rnd_hq/2026-10-15/tracks/arch_fanout/bandwidth/DESIGN.md
- path: silicon cycle profile -> /shared/rnd_hq/2026-10-15/tracks/cycle_profile/SILICON_PROFILE.txt
- page: HBMS cycle profile page -> /p/talon-hbms-engine-arch-cycles-20261015/

## [Memory] HBM address map and port windows

How a host address and an engine port reach a pseudo-channel: 32 PCs of 1 GB from 0x40_0000_0000, 64 NMUs (two per PC), 512 MiB port windows today vs 1 GiB windows in R8_BEST.

Every HBM byte has one physical home: a pseudo-channel (PC) on one of the two stacks. The V80 has 32 PCs of 1 GB [S]. Two address views matter to us:
The engine view. Each of our engine ports drives one AXI master into an HBM_NMU. The NMU wiring rule we use in the block design is HBMnn_AXI -&gt; HBM(nn/4)_PORT(nn%4): HBM controller nn/4, port nn%4. A controller serves two PCs, and each PC has two NMU ports (ports 0/1 -&gt; PC .0, ports 2/3 -&gt; PC .1).The host / NoC view. HBM appears in the system address map from 0x40_0000_0000; the host writes weights there through QDMA, and the 03_setup_qdma.sh roundtrip lands at 0x40C0000000.
Port windows
0x1254 (today): 32 ports, each with a 512 MiB window at 0x40_0000_0000 + p x 0x2000_0000. Two ports share one PC, so 32 ports cover only 16 PCs = one stack, 16 GiB of reach [M].R8_BEST 0xE4B7125F (HBM-FOLD): 32 engine ports, each owning one whole PC (1 GiB) at 0x40_0000_0000 + p x 0x4000_0000 and both of its NMUs (A streams the first half of each tile's K beats, B the second). All 64 NMUs and both stacks are used: 32 GiB of reach [D].
The compiler lays each weight tile out column-major (one beat = one K column of 32 rows), tile t on port t % 32, so per-port windows fill evenly.Worked example: does 70b fit?Llama-3.3-70B split 8 ways needs 10.86 GiB of weights per card: 347.5 MiB on each of 32 ports [M, TP8 image build]. That fits under today's 512 MiB windows, and leaves ~21 GB of the card's 32 GB free.

```
# 0x1254 base of port p
base = 0x40_0000_0000 + p * 0x2000_0000          # 512 MiB
# R8_BEST base of port p (one PC per port)
base = 0x40_0000_0000 + p * 0x4000_0000          # 1 GiB
# the host kits&#x27; REGION constant must match the image; F4 fold also needs K % 64 == 0 (compiler pads)
```

- PCs: 32 x 1 GB [S]
- NMUs: 64 (2 per PC) [S]
- 0x1254 window / reach: 512 MiB per port / 16 GiB [M]
- R8_BEST window / reach: 1 GiB per port / 32 GiB [D]
- 70B TP8 per port: 347.5 MiB [M]
- PITFALL: A REGION mismatch between image and host kit silently writes weights into the wrong PC; G0 identity plus the BUILD_ID-keyed allowlist in the host kit guard against it.
- PITFALL: Two ports per PC (0x1254) share that PC's 25.6 GB/s; one port per PC with two NMUs (R8_BEST) gives each engine port a whole PC.
- amd: PG313 HBM address map options tab -> https://docs.amd.com/r/en-US/pg313-network-on-chip/HBM-Address-Map-Options-Tab
- amd: PG313 HBM_NMU -> https://docs.amd.com/r/en-US/pg313-network-on-chip/HBM_NMU
- aved: AVED V80 memory map (HBM memory space) -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BV80%2B-%2BMemory%2BMap.html
- path: R8_BEST block design changes -> /shared/build/_inputs/workspace/eagle4_adaptation/hardware/cleanup_EAGLE/.claude/worktrees/r8best-0x125D/hw/r8best_0x125D/README.md

## [Memory] NoC and NMUs

The Versal network-on-chip connects CIPS/PCIe, the PL and memory; we use it for host access and the direct HBM_NMU (BLI) ports for bandwidth.

Versal has a hardened network-on-chip (NoC): a packet-switched fabric of NoC master units (NMUs, where traffic enters), NoC slave units (NSUs, where it leaves), switches and memory controllers. AVED's block design uses the NoC to connect CIPS (PCIe, processors, the PMC) to the PL control registers, to DDR and to HBM. The NoC compiler runs during implementation and must meet the QoS (bandwidth / latency) requested for every path; a path it cannot meet is reported SUBOPTIMAL.
For HBM the device offers two kinds of NMU: general NoC NMUs reachable from any SLR, and the localized HBM_NMUs in SLR2 that sit right next to the HBM controllers. Our measurements on the NP32 block design (F1152): 16 engine ports through SLR1 NoC NMUs got 51.7 GB/s total with a minimum of 2.6 GB/s per port (SUBOPTIMAL); 16 ports on HBM_NMUs got 163.9 GB/s [M]. So the host path (QDMA writes of weights into the HBM windows) uses the NoC, and the engines use HBM_NMUs.How to read the noc qos reportEach connection lists required and estimated read/write bandwidth and a status. Sum the estimated read bandwidth over the 64 HBM ports: anything below the requested total, or any port marked SUBOPTIMAL, means the engine will starve at runtime even if timing closes. The NP64 build's NoC compiler PASSED on all 64 BLI ports [M].

```
# AVED block design sources (per build tree):
#   hw/amd_v80_gen5x8_25.1_pdi23/src/create_design_hbms.tcl, src/bd/create_bd_design_hbms.tcl
validate_bd_design                      # BD-level checks
# after implementation: NoC QoS report
report_noc_qos -file noc_qos.rpt
# our parser: summarise per-port required vs estimated MB/s for HBM00..HBM63
python3 noc/qos_sum64.py noc_qos.rpt TAG     # tracks/arch_fanout/bandwidth/noc/
```

- NoC NMUs in SLR2: 89 of 152 (F1113) [M]
- NoC path, 16 ports: 51.7 GB/s, min 2.6 GB/s/port [M]
- HBM_NMU path, 16 ports: 163.9 GB/s [M]
- NP64 BD, 64 BLI ports: NoC compiler PASS [M]
- PITFALL: Do not route engine traffic through general NoC NMUs from SLR0/SLR1 to reach HBM: 3.2x slower than the BLI.
- PITFALL: QoS numbers in the BD are requests; read the post-implementation QoS report, not the BD.
- amd: PG313 HBM_NMU -> https://docs.amd.com/r/en-US/pg313-network-on-chip/HBM_NMU
- amd: PG313 HBM address map options tab -> https://docs.amd.com/r/en-US/pg313-network-on-chip/HBM-Address-Map-Options-Tab
- amd: UG1388 improving NoC performance (QoS vs fabric timing) -> https://docs.amd.com/r/en-US/ug1388-acap-system-integration-validation-methodology/Improving-NoC-Performance-by-Optimizing-NoC-QoS-Over-Fabric-Timing-QoR
- aved: AVED V80 NoC configuration -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BV80%2B-%2BNoC%2BConfiguration.html
- path: NoC QoS parser and probe -> /shared/rnd_hq/2026-10-15/tracks/arch_fanout/bandwidth/noc/

## [Host link] PCIe and the two functions

Gen5 x8 capable; trains Gen4 x8 on box1 and talon. PF0 = 10ee:50b4 (AMI + our BAR0 registers), PF1 = 10ee:50b5 (QDMA). MMIO costs 1.37 us per read, so the host never sits in the per-token loop.

Each V80 enumerates as two PCIe functions on its bus: xx:00.0 = vendor:device 10ee:50b4 (PF0) and xx:00.1 = 10ee:50b5 (PF1). PF0 carries the AMI management interface and our own register space: HBMS images expose BUILD_ID at BAR0+0x10100 and the magic word 'HBML' at BAR0+0x10104, and the descriptor tables, CSRs and stat counters behind them. PF1 is the QDMA engine for bulk transfers into the HBM windows (see *QDMA*).
The card supports Gen5 x8, but both our hosts' root ports run it at Gen4 x8 (16 GT/s) [M]. Bulk bandwidth is fine for loading weights; latency is the problem. A single 32-bit MMIO read costs 1.37 us, and a small QDMA transfer costs 130 us card-to-host and 250 us host-to-card [M]. That is why a tensor-parallel all-reduce through the host (56 per 3B token) cannot beat one card, and why every multi-card design moves partial sums over the QSFP ring instead.Worked example: why host all-reduce loses3B TP8 needs 56 all-reduces per token. Even at the best case of one small transfer each way, 56 x (130 + 250) us ~= 21 ms per token of host traffic [D], versus ~16 ms for the whole 3B model on one card [M/D]. Over the QSFP ring the same all-reduce measures 11.47 us in xsim [M].

```
lspci -nn -d 10ee: | sort                  # 2 lines per card: .0 = 50b4, .1 = 50b5
cat /sys/bus/pci/devices/0000:50:00.0/current_link_speed   # 16.0 GT/s PCIe = Gen4
cat /sys/bus/pci/devices/0000:50:00.0/current_link_width   # 8
cat /sys/bus/pci/devices/0000:50:00.0/aer_dev_correctable  # see the AER article
sudo ami_tool pcieinfo -d 50:00.0                          # AMI&#x27;s view of the link
# identity check our host kits do first (HBMS ladder G0):
#   BAR0+0x10100 == BUILD_ID (e.g. 0xE4B71254), BAR0+0x10104 == &#x27;HBML&#x27;
```

- link: Gen4 x8 trained (Gen5 x8 capable) [M]
- MMIO read: 1.37 us per 32-bit word [M]
- QDMA bulk h2c / c2h: 2,578 / 398 MB/s [M]
- small QDMA c2h / h2c: 130 / 250 us [M]
- talon BDFs (PF0): 17, 3d, 50, 63, 99, bd, cf, e1 [M]
- PITFALL: After `device_boot`, PF1 can report "Invalid config bar" (7 of 8 talon cards, 10-14): remove `XX:00.1` and rescan; no re-flash needed.
- PITFALL: box1 needs Above 4G decoding + Resizable BAR in BIOS to POST with 4 x V80; a CMOS reset loses them.
- PITFALL: A Python driver that serialises 8 cards in one thread halves a multi-card speed-up; drive cards in parallel.
- aved: AVED host to card communication -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2B-%2BHost%2Bto%2BCard%2BCommunication.html
- aved: ami_tool pcieinfo -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BManagement%2BInterface%2Buserguide%2B%28ami_tool%29.html#pcieinfo
- aved: AVED V80 CIPS configuration (PCIe controller) -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BV80%2B-%2BCIPS%2BConfiguration.html
- path: host link measurements -> /shared/rnd_hq/2026-10-15/tracks/v80_page/FROM_RESEARCH.md

## [Host link] PCIe AER errors

Advanced Error Reporting: correctable vs uncorrectable PCIe errors, where Linux shows them, and what we found on box1 (firmware-first RAS hiding a corrected-error storm on all 4 V80 links).

PCIe Advanced Error Reporting (AER) is a capability in each device's config space that counts and reports link and transaction errors. There are two classes:
Correctable (CE): the link recovered by itself (bad TLP/DLLP, replay timer, receiver error). Data is intact, but every event costs a replay and, on some platforms, an interrupt or an SMI.Uncorrectable (UE), non-fatal or fatal: lost or poisoned data, completion timeouts, surprise link down. A fatal error usually takes the device off the bus.
Where they are reported depends on the platform. With OS-first handling, Linux's AER driver logs to dmesg and exposes counters in sysfs. With firmware-first (APEI/GHES), the BIOS handles errors in system-management mode (SMM) and Linux may see nothing.
What we found on box1 (10-07)
Every 4-card boot logs corrected PCIe errors at the root port of card 01, then zero runtime GHES records (F0920).A live poll found self-clearing corrected errors on all 4 V80 links: lane error status counts of 1,719 / 840 / 586 / 28 over 3,000 polls in 41.5 s, with links at 16 GT/s x8 (F0925) [M].The OS had no AER path: root-port CE reporting was off and _OSC denied OS AER. Firmware cleared the errors in SMM, so the host's MMIO latency spikes lined up with an SMI storm of ~700-1,100 SMI/s per CPU.How to read itIn lspci -vvv, CESta: lists the correctable bits currently set (RxErr, BadTLP, BadDLLP, Rollover, Timeout, AdvNonFatalErr) and UESta: the uncorrectable ones. Non-zero CE bits that clear between reads mean something else (firmware) is servicing them; on a firmware-first platform a clean snapshot does not prove a clean link.

```
cat /sys/bus/pci/devices/0000:50:00.0/aer_dev_correctable      # CE counters (OS-first only)
cat /sys/bus/pci/devices/0000:50:00.0/aer_dev_fatal
sudo lspci -vvv -s 50:00.0 | grep -A8 &#x27;Advanced Error Reporting&#x27;  # UESta / CESta / CEMsk
sudo dmesg | grep -iE &#x27;aer|pcie bus error&#x27;
# box1 lab kit (no reboot, reversible): masks CE reporting per V80 and measures SMI/s per window -> SMI_AER_AB (F0928)
```

- box1 V80 links: 16 GT/s x8 [M]
- live CE counts, 41.5 s (F0925): 1,719 / 840 / 586 / 28 per card [M]
- SMI rate under load: ~700-1,100 per CPU per second [M]
- PITFALL: Firmware-first RAS hides errors from Linux; poll config space directly if MMIO latency looks odd.
- PITFALL: Masking CE reporting (CEMsk) is a diagnostic, not a fix; restore it after the A/B.
- amd: Linux kernel PCIe AER how-to (kernel.org, not AMD) -> https://docs.kernel.org/PCI/pcieaer-howto.html
- aved: ami_tool pcieinfo (link speed / width) -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BManagement%2BInterface%2Buserguide%2B%28ami_tool%29.html#pcieinfo
- aved: AVED debug techniques -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BDebug%2BTechniques.html
- path: host-gap RCA findings F0916-F0928 -> /shared/rnd_hq/hq_daily/findings.csv

## [Host link] QDMA driver and queues

Xilinx QDMA Linux driver bound to PF1 (10ee:50b5); memory-mapped queues move weights into the HBM windows. How we build it, set queues up and test a roundtrip.

QDMA is the multi-queue DMA engine in the Versal CPM PCIe block. Each queue is either memory-mapped (MM: host buffer &lt;-&gt; a card address) or streaming; we use MM queues, one set per card, to write weights and model images into the HBM windows and to read results back. The host side is the open-source dma_ip_drivers QDMA driver (qdma-pf module) plus its command-line tools (dma-ctl, dma-to-device, dma-from-device). The V80 needs one change: its PF1 device ID 0x50b5 added to the driver's PCI ID table.
On talon the driver is installed and bound on all 8 cards; each card gets device nodes /dev/qdma&lt;bus&gt;001-MM-0..3. Measured on the HBMS ladder (L2): host-to-card 2,578 MB/s and card-to-host 398 MB/s [M]. Loading a 1.48 GiB 1B model took 1.8 s; all 8 cards x 3B in parallel 17.5 s [M].How to read a roundtrip resultA PASS proves the driver, the queues and that the image maps the HBM window at 0x40C0000000. On an image without that window a FAIL says nothing about the driver. Always pass the PF0 BDF positionally: the script derives .1 itself and defaults to box1's 61:00.0.

```
grep -n 0x50b5 dma_ip_drivers/QDMA/linux-kernel/driver/src/pci_ids.h   # talon: line 469, V80 (AVED)
sudo apt-get install -y libaio-dev
make -j1                        # the tools&#x27; link races at -j16
sudo make install-mods &amp;&amp; sudo modprobe qdma-pf
lsmod | grep qdma_pf            # 864256 bytes on talon
# per card: 4 MM queues each way + a 64-byte roundtrip at 0x40C0000000 (inside the HBM window)
DRYRUN=1 bash 03_setup_qdma.sh 4 17:00.0      # touches nothing, prints the plan
sudo env PATH=$HOME/dma_ip_drivers/QDMA/linux-kernel/bin:$PATH bash 03_setup_qdma.sh 4 17:00.0
# expect: Roundtrip test: PASS  /  === QDMA ready ===
```

- talon driver: qdma_pf on all 8 cards (10-14) [M]
- box1 driver: qdma-pf 2024.1.0.0, dma_ip_drivers commit 77ceefe [M]
- bulk h2c / c2h: 2,578 / 398 MB/s [M]
- 3B load, 8 cards in parallel: 17.5 s [M]
- PITFALL: `make -j16` races at link: use `-j1` for the tools.
- PITFALL: `dma-ctl` is not on PATH on talon; pass PATH explicitly under sudo (sudo drops the environment).
- PITFALL: box1's `/etc/modprobe.d/qdma.conf` pins master_pf to card 01; the other cards load as non-master.
- PITFALL: PF1 "Invalid config bar" after device_boot: remove `XX:00.1` + rescan.
- amd: dma_ip_drivers QDMA Linux driver user guide -> https://xilinx.github.io/dma_ip_drivers/master/QDMA/linux-kernel/html/userguide.html
- amd: dma_ip_drivers QDMA build guide -> https://xilinx.github.io/dma_ip_drivers/master/QDMA/linux-kernel/html/build.html
- amd: PG347 CPM QDMA -> https://docs.amd.com/r/en-US/pg347-cpm-dma-bridge/QDMA
- aved: AVED QDMA -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BQDMA.html
- ours: 03_setup_qdma.sh -> /shared/rnd_hq/from_box1/tested_archive/weights/loaders/v80_test_suite/03_setup_qdma.sh

## [Host link] QSFP cages and Aurora links

4 QSFP28 cages per card, all four GTM quads in SLR1; Aurora 64B66B at 4 x 25.78 Gb/s per link. The talon ring passes BIST 16/16 at 99.84 Gb/s raw per direction.

Card-to-card traffic never goes through the host: it rides QSFP28 cables between cages. Each cage connects to one GTM transceiver quad (4 lanes); we run Aurora 64B/66B on 4 lanes x 25.78125 Gb/s (F0884), presented to our logic as a 256-bit AXI-Stream. Above Aurora, our own protocols add framing: the talon ring's hop_link frames (header with rank/len/seq, CRC, credit flow control) and the Track 1 link_dma engine.
On talon the 8 cards form a ring 17 &gt; 3d &gt; 50 &gt; 63 &gt; cf &gt; e1 &gt; 99 &gt; bd &gt; 17 using cages 0 and 3. The ring image 0xE4B712B7 passes BIST on 16/16 directions at frame lengths 64/256/960 with 0 errors (F1251) and a 600 s raw soak at 12.48 GB/s = 99.84 Gb/s per direction with BER &lt;= 5.01e-14 (F1253) [M]. Getting there needed one fix: each card's core clock differs by a few hundred ppm, so the receive CDC FIFO in hop_aurora_cdc slowly overflowed on the fast-to-slow direction. TX pacing (1 idle per 256 words), an overflow self-heal and DROP_RST (drop flagged frames) cured it.How to read link resultsA BIST line gives per-direction PASS/FAIL, error words, CRC errors, overflow flags and throughput. A raw soak gives error words per interval and a derived BER bound. Status registers only say a lane has a live peer, not which peer; only beacon discovery maps the cables.

```
# talon (host kit v4): discover cabling, then test
sudo python3 -I talon_ring_host.py status   --build 0xE4B712B7          # refuses writes on any other BUILD_ID
sudo python3 -I talon_ring_host.py discover --build 0xE4B712B7          # beacons -> RING_MAP (which cable goes where)
sudo python3 -I talon_ring_host.py bist     --build 0xE4B712B7 --len 256   # run again with --len 64 and --len 960
sudo python3 -I talon_ring_host.py raw      --build 0xE4B712B7 --secs 600 --every 60 --json raw.json
python3 talon_ring_host.py all --mock                                     # offline check, touches no card
# box1 read-only link status words (drafter images): 0x...039f = UP x4 lanes, 0x...0380 = DOWN (GT on, no peer)
```

- line rate: 4 lanes x 25.78125 Gb/s (F0884) [M]
- raw per direction: 99.84 Gb/s, BER <= 5.01e-14 (F1253) [M]
- framed BIST, len 256: 4.698 GB/s, 0 errors (F1251) [M]
- ring all-reduce P=8, 3,072 int32: 11.47 us [M xsim]
- talon ring order: 17>3d>50>63>cf>e1>99>bd [M]
- link_dma v2, 70 KB hop: 39.9 us [M xsim]
- PITFALL: All 4 GTM quads are in SLR1: every link costs SLR1 slices (~5,376-slice hard pblock per quad clock region, F0585); a TAIL at 99.44% SLR1 failed Aurora TX timing (F0193).
- PITFALL: A shared-clock testbench hides ppm-drift FIFO overflow; gate ring designs with per-card clocks at +-1000 ppm.
- PITFALL: CRC-flagged frames were delivered to the consumer until DROP_RST=1; set it in every ring image.
- PITFALL: P2 has never run on silicon; keep it free.
- amd: PG074 Aurora 64B/66B hardware debug -> https://docs.amd.com/r/12.0-English/pg074-aurora-64b66b/Hardware-Debug
- amd: AM017 Versal GTM transceivers -> https://docs.amd.com/r/en-US/am017-versal-gtm-transceivers
- path: talon ring report -> /shared/rnd_hq/2026-10-14/tracks/talon_ring/
- path: 4-card cable plan -> /shared/rnd_hq/2026-10-15/tracks/four_card_a2a/A2A_Q1_Q3.md
- page: RING8 serving design -> /p/ring8-hbm-llm-serving-20261011/

## [Host link] GTM transceivers and QSFP cages

The four QSFP28 cages map to four GTM quads, all in SLR1; which cage is which quad, the reference clock, and what each extra link costs in SLR1 area.

A GTM is Versal's high-speed serial transceiver (up to 58 Gb/s PAM4 or 32 Gb/s NRZ per lane, grouped in quads of 4 lanes with shared PLLs and reference clocks). Each QSFP28 cage on the V80 wires to one GTM quad; we run 4 lanes NRZ at 25.78125 Gb/s with Aurora 64B/66B on top (F0884), about 100 Gb/s per cage per direction.
The physical map matters for floorplanning: all four GTM quads sit in SLR1 (F0193). For example cages P3/P4 are banks 112/111 = GTM_QUAD_X0Y10 / GTM_QUAD_X0Y9 in CLOCKREGION_X0Y7 of SLR1, with reference clock pin AL17 = GTM_REFCLK_X0Y18 (bank 111) shared north to bank 112 (F0882). On talon the ring uses cages 0 and 3: GTM_QUAD_X1Y7 (CLOCKREGION_X9Y6, right edge) and GTM_QUAD_X0Y9 (CLOCKREGION_X0Y7, left edge, top row of SLR1).
So every link puts its GT-side logic (Aurora core, CDC, clock buffers) into the middle SLR, whatever the rest of the floorplan does. The ring's core-clock logic can live elsewhere (in R8_BEST, in the SLR0 hub), but the GT side cannot.

```
# find the quads and their clock regions in a placed design
get_sites -filter {SITE_TYPE =~ GTM*}
report_property [get_sites GTM_QUAD_X0Y9]       # CLOCK_REGION, SLR
# keep GT clock buffers inside the quad&#x27;s region: add gt_bufg / gt_rx_bufg to the link pblock (F0499)
```

- line rate: 4 lanes x 25.78125 Gb/s NRZ [M]
- per-cage throughput, raw: 99.84 Gb/s per direction (F1253) [M]
- P3/P4: banks 112/111, GTM_QUAD_X0Y10/X0Y9, CLOCKREGION_X0Y7, SLR1 [M]
- talon ring cages 0/3: GTM_QUAD_X1Y7 (X9Y6), GTM_QUAD_X0Y9 (X0Y7) [M]
- SLR1 cost per extra link: ~5,376-slice hard pblock per quad clock region (F0585) [M]
- PITFALL: A hard link pblock with EXCLUDE_PLACEMENT locked BUFG_GT out of the quad's region (Place 30-6836); include gt_bufg / gt_rx_bufg in the pblock (F0499).
- PITFALL: With SLR1 at 99.44% a TAIL failed Aurora TX timing with one link (F0193): budget SLR1 for every link before adding one.
- amd: AM017 Versal GTM transceivers architecture manual -> https://docs.amd.com/r/en-US/am017-versal-gtm-transceivers
- amd: AM017 GTME5_QUAD attributes -> https://docs.amd.com/r/en-US/am017-versal-gtm-transceivers/GTME5_QUAD-Attributes
- amd: PG074 Aurora 64B/66B hardware debug -> https://docs.amd.com/r/12.0-English/pg074-aurora-64b66b/Hardware-Debug
- path: ring8 merge floorplan (cage map) -> /shared/rnd_hq/2026-10-15/tracks/ring8_merge/FLOORPLAN.md

## [Operations] AMI and ami_tool

AMD's management interface: the ami kernel driver plus ami_tool talk to the on-card management controller (AMC) over PF0 to list cards, read sensors, program flash partitions and boot a partition.

AMI (AVED Management Interface) is the host half of the card's management path. A kernel driver (ami, 290,816 bytes loaded on talon [M]) binds PF0 and exchanges commands through a mailbox (the Generic Command Queue IP) with the AMC, the management firmware running on the card's real-time processor. ami_tool is the command-line front end. Through it we read the device table (BDF, UUID, AMC state), read sensors, inspect and program the flash partitions, and boot a chosen partition on a hot reset.
The UUID that ami_tool overview prints identifies the logic image (from its partition table), not the card. It is how we confirm what booted: e.g. 5bcfc08f4fcdd233fcf8cf05692ecee9 for the RING8 0xE4B71250 image, and a different factory UUID (5a73aca2...) when a cold boot fell back to partition 0 on 10-14.How to read the device tablePer card: the PF0 BDF, the image UUID and the AMC state. AMC READY plus the expected UUID plus two PCIe functions (.0 50b4 and .1 50b5) is the post-boot pass condition we use on talon; then the HBMS ladder's G0 checks BUILD_ID/'HBML' over BAR0.

```
sudo ami_tool overview                       # version block + device table: BDF, UUID, AMC state
sudo ami_tool sensors -d 50:00.0             # temperatures, voltages, power
sudo ami_tool cfgmem_info -d 50:00.0 -t primary    # partition table of the flash
sudo ami_tool cfgmem_program -d 50:00.0 -t primary -i image.pdi -p 1 -y -q   # see the flash article
sudo ami_tool device_boot -d 50:00.0 -p 1    # hot-reset boot of partition 1; no -y, answer y
ami_tool cfgmem_program -h                   # help works without sudo
```

- ami_tool: 2.4.0 [M]
- ami driver on talon: loaded, 290,816 bytes [M]
- RING8 0x1250 UUID: 5bcfc08f4fcdd233fcf8cf05692ecee9 [M]
- PITFALL: Without sudo, `ami_tool overview` prints an empty device table (10-14).
- PITFALL: `device_boot` into a bad partition dropped card 50 off PCIe on 10-14; recovery was JTAG (see JTAG recovery).
- PITFALL: On box1, flashing, device_boot and PCI remove belong to box1's own session; HQ never runs them there.
- aved: ami_tool user guide -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BManagement%2BInterface%2Buserguide%2B%28ami_tool%29.html
- aved: ami_tool overview -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BManagement%2BInterface%2Buserguide%2B%28ami_tool%29.html#overview
- aved: ami_tool sensors -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BManagement%2BInterface%2Buserguide%2B%28ami_tool%29.html#sensors
- aved: AVED Management Interface (AMI) -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BManagement%2BInterface%2B%28AMI%29.html
- aved: AMC architecture and design -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BManagement%2BController%2B%28AMC%29%2B-%2BArchitecture%2Band%2BDesign.html

## [Operations] Flashing and booting a card

Two flash partitions: a cold boot always loads partition 0; partition 1 is staging and is booted with device_boot on a hot reset. Flash p1 to test, p0 to persist.

The card's configuration flash holds a partition table (FPT) and two design partitions. At power-on the platform loads partition 0. ami_tool device_boot -p 1 reboots the device into partition 1 without a power cycle (a hot reset), so partition 1 is where we stage and test new images. A design only survives a cold AC once it is also written to partition 0.
What goes into a partition is the AMI flash-format PDI produced by the AVED build (it carries the FPT-compatible header). That file is different from the plain JTAG-loadable PDI (see PDI vs flash-format images).
The 10-14 lesson
All 8 talon cards were flashed to partition 1 only. A cold AC at 02:05 booted the factory partition 0 (UUID 5a73aca2..., one PCIe function) and the new image was gone. The rule since: flash p1, test with device_boot, then flash p0. The partition-0 copy survived the 05:02 reboot.Post-flash checklist (per card)lspci -nn -d 10ee: shows both .0 (50b4) and .1 (50b5).sudo ami_tool overview: expected UUID, AMC READY.HBMS ladder G0: BUILD_ID and 'HBML' over BAR0.For persistence: a cold reset after the p0 flash, a new boot ID, then 1-3 again.

```
md5sum image.pdi                       # compare with the build record before every flash
sudo ami_tool cfgmem_program -d 50:00.0 -t primary -i image.pdi -p 1 -y -q   # stage + test
sudo ami_tool cfgmem_program -d 50:00.0 -t primary -i image.pdi -p 0 -y -q   # persist
# 8 cards: one background job per card, each logging to its own file; success = log ends FLASH_EXIT=0
sudo ami_tool device_boot -d 50:00.0 -p 1         # no -y; answer y; hot reset of that card only
echo 1 | sudo tee /sys/bus/pci/devices/0000:50:00.1/remove; echo 1 | sudo tee /sys/bus/pci/rescan   # if PF1 shows Invalid config bar
```

- talon 8-card p1 flash: ~40 min (00:59-01:40 EDT, 10-14) [M]
- talon 8-card p0 flash: ~30 min in parallel [M]
- device_boot p1: 7 of 8 came up; card 50 dropped (10-14) [M]
- 0x1254 s2 after flash: 52/52 ladder on all 8 (F1257) [M]
- PITFALL: A partition-1-only flash does not survive a cold AC.
- PITFALL: Inside `sudo sh -c` use absolute paths: `~` is /root.
- PITFALL: `device_boot` into a bad partition can drop the card off PCIe; a rescan did not recover it; JTAG did.
- PITFALL: On box1 only box1's own session flashes, under its card lock, followed by the two-party cold-AC consent.
- aved: ami_tool cfgmem_program -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BManagement%2BInterface%2Buserguide%2B%28ami_tool%29.html#cfgmem-program
- aved: ami_tool device_boot -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BManagement%2BInterface%2Buserguide%2B%28ami_tool%29.html#device-boot
- aved: ami_tool cfgmem_info (partition table) -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BManagement%2BInterface%2Buserguide%2B%28ami_tool%29.html#cfgmem-info
- aved: Updating the design PDI in flash -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BUpdating%2BDesign%2BPDI%2Bin%2BFlash.html
- aved: Updating the FPT image in flash -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BUpdating%2BFPT%2BImage%2Bin%2BFlash.html
- aved: AVED device programming -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2B-%2BDevice%2BProgramming.html
- path: talon flash record -> /shared/rnd_hq/2026-10-11/tracks/talon_8fpga/TALON_FLASH_0x1250_20261014.md
- path: box1 delivery rules -> /shared/rnd_hq/2026-10-15/HQ_PDI_DELIVERY_RULES_20261015.md

## [Operations] PDI vs flash-format images

One build produces two files: the JTAG-loadable PDI (*_nofpt.pdi) and the AMI flash-format PDI that ami_tool writes into a flash partition. Which file goes where, and how we name and check them.

A Versal PDI (programmable device image) is the whole boot payload: the platform loader firmware (PLM), CIPS / NoC configuration, the PL bitstream and any processor software, packed by bootgen. The PMC (platform management controller) loads it at boot.
The AVED flow produces the image in two forms:
JTAG-loadable PDI (our files end in _nofpt.pdi): a plain PDI that the hardware server can push into the device over JTAG. Used for recovery when a card is off PCIe.AMI flash-format PDI (our main file, named with its md5, e.g. 0xE4B71250_b1_4e786830.pdi): the same design wrapped for the card's flash and its FPT (flash partition table). This is the file ami_tool cfgmem_program writes into partition 0 or 1, and what the card boots from on power-on or device_boot.
They are not interchangeable. Over JTAG the flash-format file is rejected with "No valid identification string"; cfgmem_program expects the flash format.How the pieces relate| artefact | made by | goes to | used for | |---|---|---|---| | bitstream + CIPS/NoC config | Vivado implementation | inside both PDIs | the design itself | | *_nofpt.pdi | write_device_image | JTAG (hw_server) | recovery | | flash-format *.pdi | AVED packaging step (needs the AMC firmware, amc.elf) | flash partition 0 / 1 | normal boot | | FPT image | AVED | flash header | partition table; rarely changed |
If the AMC firmware step fails (amc.elf missing, as on 0xE4B71214, F1035), you get a bitstream but no flash-format image, and the build cannot be flashed.

```
# naming: <BUILD_ID>_<seed>_<md5-prefix>.pdi (flash format), <BUILD_ID>_<seed>_nofpt.pdi (JTAG)
md5sum ~/eagle4_pdi/0xE4B71250_b1_4e786830.pdi       # 4e786830846f82f30de1f3727d4cc3a3
md5sum ~/eagle4_pdi/0xE4B71250_b1_nofpt.pdi          # d63d0b0d90477dd9d400c53c4dd841a2
sudo ami_tool cfgmem_program -d 50:00.0 -t primary -i ~/eagle4_pdi/0xE4B71250_b1_4e786830.pdi -p 1 -y -q
sudo ami_tool cfgmem_info -d 50:00.0 -t primary      # see what each partition holds
```

- 0x1250 flash-format md5: 4e786830846f82f30de1f3727d4cc3a3 [M]
- 0x1250 JTAG md5: d63d0b0d90477dd9d400c53c4dd841a2 [M]
- 0x1254 s2 flash-format: PDI md5 259a2800... [M]
- PITFALL: Over JTAG, the flash-format file fails with "No valid identification string": use `_nofpt.pdi`.
- PITFALL: Always md5 the file you flash against the build record; two seeds of one BUILD_ID are different images (0x1254 b1 failed 2 cards, s2 passed all 8).
- aved: Updating the design PDI in flash -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BUpdating%2BDesign%2BPDI%2Bin%2BFlash.html
- aved: Updating the FPT image in flash -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BUpdating%2BFPT%2BImage%2Bin%2BFlash.html
- aved: ami_tool cfgmem_program -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BManagement%2BInterface%2Buserguide%2B%28ami_tool%29.html#cfgmem-program
- aved: AVED device programming -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2B-%2BDevice%2BProgramming.html
- amd: UG1283 bootgen: PDI generation with a BIF example -> https://docs.amd.com/r/en-US/ug1283-bootgen-user-guide/PDI-Generation-with-BIF-Example

## [Operations] JTAG recovery

When a card drops off PCIe (a device_boot into a bad partition), push the JTAG-loadable PDI with hw_server, then rescan. The talon card-50 recovery of 10-14, step by step.

Every V80 has a JTAG chain reachable through the card's USB/JTAG path and the server's cabling. When a card no longer enumerates on PCIe, AMI cannot reach it, so the flash tools cannot help. JTAG still can: the Vivado hardware server (hw_server) loads a PDI straight into the device, which then comes up on PCIe again. The flash is untouched, so the next cold boot still loads partition 0.
The talon card-50 case (10-14)
After the 8-card flash, ami_tool device_boot -p 1 brought 7 of 8 cards up. Card 50 answered "could not select partition" and dropped off PCIe. A PCI rescan did not bring it back. Recovery:
Start the hardware server on talon directly from the Vivado install (not via settings64.sh, whose paths point at asgard).Pick the target by serial: XFL1C04XIDTGA was the only device with DONE=0 (BMC slot SL02 = bus 50).Program the JTAG-loadable file 0xE4B71250_b1_nofpt.pdi (the flash-format file is rejected).Rescan PCI. Result: DONE high, 50:00.0 and 50:00.1 present, UUID 5bcfc08f..., AMC READY.

```
~/xilinx_install/2025.1/Vivado/bin/hw_server -d        # on talon, no settings64.sh
# probe and program (on talon): ~/jtag_probe/probe2.tcl, ~/jtag_probe/prog50.tcl. prog50.tcl is:
#   open_hw_manager
#   connect_hw_server -allow_non_jtag
#   open_hw_target {localhost:3121/xilinx_tcf/Xilinx/XFL1C04XIDTGA}
#   set d [lindex [get_hw_devices xcv80*] 0]; current_hw_device $d
#   set_property PROGRAM.FILE {<path>/0xE4B71250_b1_nofpt.pdi} $d
#   program_hw_devices $d
#   refresh_hw_device -update_hw_probes false $d
#   puts &quot;AFTER DONE=[get_property REGISTER.JTAG_STATUS.SLR0.BIT\[34\]_DONE $d]&quot;
#   close_hw_target
echo 1 | sudo tee /sys/bus/pci/rescan
lspci -nn -d 10ee: | grep &#x27;^50:&#x27;                         # expect 50:00.0 (50b4) and 50:00.1 (50b5)
```

- talon cards up after device_boot (10-14): 7 of 8 [M]
- recovered card: bus 50, serial XFL1C04XIDTGA [M]
- JTAG serials mapped to a BDF: 1 of 8 so far [M]
- PITFALL: Record the JTAG-serial-to-BDF map each time a card is re-seated; only card 50 is mapped (7 serials unmapped).
- PITFALL: JTAG recovery is temporary: the device runs the pushed image until the next cold boot reloads flash partition 0.
- PITFALL: On talon every JTAG action needs the user's go; on box1 it belongs to box1's own session.
- aved: AVED JTAG boot recovery -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BJTAG%2BBoot%2BRecovery.html
- aved: AVED debug techniques -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BDebug%2BTechniques.html
- aved: ami_tool device_boot -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BManagement%2BInterface%2Buserguide%2B%28ami_tool%29.html#device-boot
- path: talon flash record -> /shared/rnd_hq/2026-10-11/tracks/talon_8fpga/TALON_FLASH_0x1250_20261014.md

## [Operations] Silicon ladder (HBMS gates)

hbms_host.py runs identity, caps/clocks and L1-L5 gates on one card: 52 gate results that are the go / no-go for an HBMS image. Never flashes or resets.

The ladder is our acceptance test for an HBMS image on silicon. It never flashes or resets anything: it checks that the BDF is a V80 PF0 running an allowlisted image, then runs gates of increasing scope and compares every output with a bit-exact golden.
| gate | checks | |---|---| | G0 | identity: 10ee:50b4 PF0, BUILD_ID, MAGIC 'HBML' (refuses any other image) | | G1 | CAPS, SPEC_CAPS, FEAT, LIMITS, clocks (control 50 MHz, stream clock via the FREQ register) | | L1 | 16 RW CSRs x 5 patterns, descriptor table, CAB port, NOP program, fail-closed errors and recovery | | L2 | QDMA write + read-back on every port | | L3 | GEMV W8 / W2 / W3 per engine, bit-exact | | L4 | full layer, KV and attention cases | | L5 | 21 speculative-decode cases, MQ off and on |
A full pass prints LADDER PASS: 52/52.How to read a ladder resultThe JSON (--out) lists every gate with PASS/FAIL; a FAIL line names the step and region (e.g. tKh1:87, KV words 72 bad 16). If the same FAIL set appears on every card, it is design behaviour (0x1250: ATTN_MODE 0 failed on all 8). If the failures differ per card and are deterministic per card, suspect timing margin (0x1254 b1: only cards 3d and 63 failed).

```
cd ~/HBMS_HOST_v1.4t_u
python3 -I hbms_host.py list                         # allowlist; no card access
python3 -I gate/test_offline.py --quick              # offline self-test: ALL GREEN
sudo python3 -I hbms_host.py run --image 0xE4B71254 --bdf 50:00.0 --keep-going --out run_50.json
```

- 0x1250: 40/52 on all 8 (attention lane mapping) [M]
- 0x1254 b1, WHS 0.000: 52/52 on 6 cards; 3d 41/52, 63 42/52 [M]
- 0x1254 s2, WHS +0.010: 52/52 on all 8 (F1257) [M]
- L2 QDMA: h2c 2,578 / c2h 398 MB/s [M]
- PITFALL: Hold slack 0.000 ns "passes timing" but broke silicon on 2 of 8 cards; release gate is true WHS >= +0.010 ns.
- PITFALL: The FREQ register tore on some cards (24-bit CDC); the v1.4t_u kit masks it. Do not trust a single read of a clock counter crossing domains.
- path: 0x1254 3d/63 RCA (timing margin) -> /shared/rnd_hq/2026-10-14/tracks/talon_1250_a0/A0_1254_3D63.md
- path: 0x1250 attention RCA -> /shared/rnd_hq/2026-10-14/tracks/talon_1250_a0/A0_ROOTCAUSE.md
- page: HBMS engine architecture -> /p/talon-hbms-engine-arch-cycles-20261015/

## [Operations] On-card cycle counters

The HBMS RTL counts cycles per op, per port and globally; read them after a token to see exactly where time goes. 92-94% of a decode token is weight streaming.

Measure before choosing a lever. The HBMS sequencer writes a stat block after every run: per-op cycles at stat word 512 + i (i &lt; 64) and 640 + i (i &gt;= 64), total cycles at 576/577, op count at 578, stall_tot (cycles the engine waited on HBM) at 581, and KV-prefetch counters at 587-594. Per-port stats (8 selectors x 32 ports) give each port's busy/ok counts.
On silicon (0x1254 s2, cards 17 and 50, bit-exact runs) a 3B token is 3,522,502 cycles = 15.656 ms at 225 MHz: 94.2% weight-streaming GEMVs, 3.9% vector ops, 1.5% attention, 0.3% sequencer [M]. Every GEMV fits 333 + 1.0333 x beats per port on both 1B and 3B.Worked example: deciding what to build| lever | what the counters say | verdict | |---|---|---| | per-port URAM prefetch FIFO | stall_tot 3.7-3.9% | &lt;= ~4% gain, not worth an image | | VPU_MODE 1 (4-bank VPU) | -3.05% (1B), -2.58% (3B), bit-exact | free: adopted as default | | more / faster HBM ports | 87-89% of the token is pure beats | the big lever: HBM-FOLD |

```
sudo python3 -I cycle_profile/profile_card.py img/llama32_3b --bdf 50:00.0 --load --steps 24 --vpu 0 1 --out cp_3b_50.json
python3 cycle_profile/analyze_profile.py cp_3b_50.json      # per-op / per-engine table
python3 cycle_profile/profile_card.py img/llama32_3b --mock --out mock.json   # offline, no card
```

- 3B token: 3,522,502 cycles, 15.656 ms [M]
- weight streaming: 92.9% (1B) / 94.2% (3B) [M]
- HBM stall: 3.9% (1B) / 3.7% (3B) [M]
- GEMV fit: 333 + 1.0333 x beats [M]
- ATTN per layer: 1B 965 + 32.3 L; 3B 1062 + 63.7 L cycles [M]
- VPU_MODE 1: -2.6 to -3.0%, bit-exact [M]
- PITFALL: Read stats after the run completes (done flag), and per segment when a program runs in several table segments.
- PITFALL: KV counters read 0 in KV_MODE 0; that is expected.
- path: silicon profile -> /shared/rnd_hq/2026-10-15/tracks/cycle_profile/SILICON_PROFILE.txt
- path: cycle model -> /shared/rnd_hq/2026-10-15/tracks/cycle_profile/cycle_model.py
- path: report -> /shared/rnd_hq/2026-10-15/tracks/cycle_profile/REPORT.md
- page: HBMS cycle profile page (interactive) -> /p/talon-hbms-engine-arch-cycles-20261015/

## [Toolchain] Vivado license

A floating 'smbus' license (for the SMBus IP in the AVED shell), 9999 seats, served by FlexLM from midyard with asgard's older server as fallback; how to test it properly.

Synthesising and implementing for the V80 part does not itself check out a paid license. The checkout that matters is the SMBus IP inside the AVED block design: when the BD is generated, Vivado requests the smbus feature from a FlexLM server. If no seat is reachable, the build fails at IP generation, hours in.
Our license (10-15) is floating, feature smbus, 9999 seats, permanent, locked to midyard's wired NIC. midyard runs lmgrd on port 2101 and the vendor daemon xilinxd on 2102. The build VMs reach both through reverse SSH tunnels on their own localhost, so their XILINXD_LICENSE_FILE lists port 2101 on localhost first. asgard's older server on port 2100 stays as the fallback, which matters right now: midyard is powered down for maintenance (GPU install), and both VMs passed an smbus checkout against the 2100 fallback [M].
Why the odd loader
Ubuntu 24.04 ships no /lib64/ld-lsb-x86-64.so.3, and the FlexLM binaries are LSB executables. We run lmgrd, xilinxd and lmutil through the normal glibc loader instead (no sudo needed).

```
# is a seat actually available? (the only meaningful test)
LIC=&quot;$XILINXD_LICENSE_FILE&quot;                 # on the build VMs: port 2101, then &#x27;at&#x27;, then localhost
lmutil lmdiag -c &quot;$LIC&quot; -n smbus               # expect: &quot;This license can be checked out&quot;
lmutil lmstat -c &quot;$LIC&quot; -a                     # server up, seats in use
# on Ubuntu 24.04 run FlexLM through the glibc loader
/lib64/ld-linux-x86-64.so.2 ./lmutil lmstat -a -c &quot;$LIC&quot;
# server start script and log on midyard: ~/flexlm_mid/start_lmgrd_mid.sh, ~/flexlm_mid/lmgrd.log
```

- seats: 9999 (smbus, floating, permanent) [M]
- servers: midyard 2101/2102 (primary), asgard 2100 (fallback) [M]
- fallback check, both VMs: smbus checkout PASS on 2100 (10-15) [M]
- PITFALL: A plain `synth_design` for xcv80 is **not** a license test: it passes even against a dead port.
- PITFALL: The SERVER hostname in the license file must resolve on the clients; we use `localhost` because clients reach the server through tunnels.
- PITFALL: Switching servers mid-build is safe only for builds that have already passed IP generation.
- aved: SMBus IP v1.1 product guide (the licensed IP) -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/SMBus%2BIP%2Bv1.1%2B-%2BProduct%2BGuide.html
- path: license migration record -> /shared/rnd_hq/2026-10-15/LICENSE_MIGRATION_MIDYARD_20261015.md

## [Toolchain] Vivado 2025.1 and the AVED shell

Every image is built with Vivado 2025.1 on the AVED amd_v80_gen5x8_25.1 shell (pdi23 flow): stage our RTL into an AVED tree, run its scripts, gate the result. ~13 h and ~75-78 GB per full build.

An image is the AVED block design (CIPS, NoC, clocking, AMC firmware, SMBus) plus our RTL. AVED ships a build tree (hw/amd_v80_gen5x8_25.1_pdi23/) with Tcl entry points: create_design.tcl / create_design_hbms.tcl build the block design and project, build_design.tcl runs synthesis, implementation, PDI generation and the flash-format packaging. Our pipelines (scripts/uni_pipeline_*.sh, per-image kits such as run_attnfix.sh, run_r8best.sh) do three things around that:
Stage: copy a pristine AVED tree (checked against its MD5 manifest), overlay our RTL and BD Tcl, write the BUILD_ID straps (STEP1-3, HBMS_PREBUILD checks).Build: run the AVED scripts on a mesh host with honest memory reservations.Gate: post-build checks (timing margin, netlist rules, lane uniformity, pblock non-empty) before anything is relayed to the cards.Worked example: one build's life0x1254 s2: Vivado session started 10-14 16:48, PDI written 10-15 05:52 on olegvm3: ~13 h wall [M], with place Explore / route AggressiveExplore. Its result: WNS +0.002, WHS +0.010 ns, and 52/52 on all 8 cards. The same RTL with default directives (b1) closed at WHS 0.000 and failed 2 cards.

```
# Vivado lives at /shared/tools/xilinxinstall/2025.1 (olegvm*), ~/workspace/xilinx_install/2025.1 (midyard)
# directive lint before every launch (fails on illegal place/route directives, incl. $::env ones)
bash /shared/rnd_hq/tools/lint_vivado_directives.sh <stage_dir> <vivado_bin>
# submit a full build through the mesh, detached from asgard
ssh -n -f olegvm2 &#x27;nohup setsid ~/mesh/mesh_run.sh --peak 80 --class pdi_build_<name> --host olegvm2     --cwd <dest> --log <dest>/build.log -- bash run_<kit>.sh <dest> < /dev/null > /dev/null 2>&amp;1 &amp;&#x27;
```

- full HBMS PDI wall: ~13 h (olegvm3) [M]
- RAM, measured eff: ~75-78 GB (r8merge peak 54 GB through early impl) [M]
- legal place_design directives (2025.1): Default, Quick, RuntimeOptimized, Explore, AggressiveExplore [M]
- PITFALL: `ExtraTimingOpt` / `SSI_SpreadLogic_high` are not legal place_design directives in 2025.1: one build died in create_design; lint v4 now resolves `$::env` directives and fails closed.
- PITFALL: Vivado runs under `setsid` in its own process group: killing the wrapper leaves orphans. Stop with `kill -TERM -- -PGID`, never by pattern.
- PITFALL: Never run Vivado on asgard (30 GB, OOM history) or talon.
- aved: AVED build scripts -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BBuild%2BScripts.html
- aved: How to rebuild an AVED design -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/How-to%2BRebuild%2Ban%2BAVED%2BDesign%2Bfor%2BYourself.html
- aved: AVED V80 source file overview -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BV80%2B-%2BSource%2BFile%2BOverview.html
- amd: UG904 directives used by opt_design and place_design -> https://docs.amd.com/r/2025.1-English/ug904-vivado-implementation/Directives-Used-by-opt_design-and-place_design-in-Implementation-Strategies
- amd: UG835 place_design -> https://docs.amd.com/r/2024.2-English/ug835-vivado-tcl-commands/place_design
- path: build scripts (box1-canonical repo snapshot) -> /shared/workspace/eagle4_adaptation/hardware/cleanup_EAGLE/scripts/

## [Toolchain] CIPS and the AVED block design

What AMD's AVED block design contains around our RTL: CIPS (PCIe, PMC, processors), the NoC, clocking, the base logic (UUID ROM, hardware discovery, command queue, SMBus), and where our engine cell plugs in.

Every V80 image is one Vivado block design (BD) created by AVED's Tcl (create_design.tcl / our create_design_hbms.tcl, with the BD itself in src/bd/create_bd_design_*.tcl). The BD has a fixed AVED part and a slot for user logic:
CIPS (Control, Interfaces and Processing System): the hard blocks. It contains the PMC (platform management controller, which boots the device from the PDI), the CPM PCIe controller with QDMA (our PF0/PF1), the real-time processor that runs the AMC firmware, and the DDR/NoC connection points. Its configuration is fixed by AVED.NoC instances (axi_noc_*): connect CIPS to the PL register space, DDR and HBM; the NoC compiler sizes them at implementation.Clocking and resets: AVED's clock wizard provides the user clocks. In the HBMS images clk_usr_0 is the control / CSR clock (50 MHz) and clk_usr_1 the stream / compute clock (225 MHz); R8_BEST adds a 400 MHz NMU-side clock. Processor-system resets (*_psr_*) sequence them.Base logic: UUID ROM (the image UUID that ami_tool overview prints), hardware-discovery IP (lets the host find BARs and endpoints), the generic command queue (the AMI mailbox to the AMC), PCIe management GPIOs, and the SMBus IP (the part that checks out our smbus license).Our cell: e.g. hbms_0 (the HBMS engine, through hbms_bd_wrap.v) and, on ring images, talon (the QSFP ring core). They connect to the NoC / HBM_NMU ports, the user clocks and AXI-Lite control from CIPS.How to change the bd safelyChange only through an overlay Tcl that (1) creates or connects exactly the cells and nets you need, (2) reads back every pin and clock frequency it set, and (3) fails closed with a clear tag. For R8_BEST the BD changes are: all 64 HBM_NMUs wired HBMnn_AXI -&gt; HBM(nn/4)_PORT(nn%4), a 400 MHz NMU clock on m00..m63_axi, 12,800 MB/s read QoS per NMU, 1 GiB windows, and the talon cell with its CDC constraint.

```
# AVED sub-IP runs you will see in a build tree (prj.runs/), e.g. on 0x1254 s2:
#   top_axi_noc_cips_0_synth_1   top_axi_noc_mc_ddr4_1_0_synth_1   top_usr_clk_wiz_0_synth_1
#   top_usr_1_psr_0_synth_1      top_pcie_psr_0_synth_1            top_pcie_mgmt_pdi_reset_gpio_0_synth_1
#   top_hw_discovery_0_synth_1   top_uuid_rom_0_synth_1            top_hbms_0_0_synth_1   (our engine)
# our overlays add cells and nets to the AVED BD, then read them back:
source src/create_design_hbms.tcl          # AVED project + BD + our hbms_0 cell
source ring/bd/talon_overlay_ar.tcl        # ring images: adds cell &#x27;talon&#x27;, six ring nets, prints TALON_BD_VALIDATED
validate_bd_design
```

- control clock: clk_usr_0 50 MHz [M]
- stream clock: clk_usr_1 225 MHz (measured 225.01 MHz on talon) [M]
- R8_BEST NMU clock: 400 MHz [P]
- BD dry run (create_design + overlay + validate): ~3-4 min, ~7 GB (F0954) [M]
- PITFALL: The BD wrapper (`hbms_bd_wrap.v`) must expose any new ports (e.g. ring_tx/rx); regenerate it with the anchored tool, never by hand.
- PITFALL: If the AMC firmware step fails, the build yields no flash-format PDI (0xE4B71214, F1035).
- aved: AVED V80 CIPS configuration -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BV80%2B-%2BCIPS%2BConfiguration.html
- aved: AVED V80 hierarchy overview -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BV80%2B-%2BHierarchy%2BOverview.html
- aved: AVED V80 base logic -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BV80%2B-%2BBase%2BLogic.html
- aved: AVED V80 clock reset module -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BV80%2B-%2BClock%2BReset%2BModule.html
- aved: AVED V80 NoC configuration -> https://xilinx.github.io/AVED/amd_v80_gen5x8_25.1_20250820/AVED%2BV80%2B-%2BNoC%2BConfiguration.html
- amd: UG1273 CIPS for PS, PMC and CPM -> https://docs.amd.com/r/2022.2-English/ug1273-versal-acap-design/CIPS-for-PS-PMC-and-CPM
- amd: PG352 CIPS presets -> https://docs.amd.com/r/3.1-English/pg352-cips/CIPS-Presets

## [Toolchain] Synthesis

RTL to netlist. What our synth gate checks before the long implementation: inferred memories, DSP mapping, unconnected ports, straps. Out-of-context (OOC) synthesis for fast experiments.

Synthesis turns RTL into a netlist of device primitives (LUTs, FFs, DSP58, URAM, BRAM, carry chains). Most expensive surprises are decided here, long before place and route: whether a memory became URAM or LUTRAM, whether a multiply became one DSP or a DSP plus fabric, whether a port was optimised away because it was never driven.
We run synthesis in two ways:
Full design, as the first stage of an AVED build. Our build_synth_gate.sh runs a quick synthesis first and fails fast on known hazards.Out-of-context (OOC) synthesis of one block (-mode out_of_context), often followed by OOC place/route at a target clock. It is how we test a timing idea (a 400 MHz NMU side, a URAM bank, a central x-generator in one SLR) in an hour instead of a 13 h build.How to read a synthesis logSearch, in order: CRITICAL WARNING (constraints and unconnected ports), Synth 8-6849 (an infeasible ram_style, so the memory moved to LUTRAM), Synth 8-11357 (3-D or record RAM that may blow up runtime), Synth 8-3332 / 8-3917 (sequential elements removed, unconnected), then the RAM and DSP inference tables near the end ("UltraRAM" / "Block RAM" / "DSP" mapping reports) to confirm every large memory landed where intended.

```
synth_design -top top -part xcv80-lsva4737-2MHP-e-S
synth_design -top hbml_lane -part xcv80-lsva4737-2MHP-e-S -mode out_of_context   # OOC block
report_utilization -file synth_util.rpt
# hazard gate on the log
grep -E &#x27;Synth 8-6849|Synth 8-11357|CRITICAL WARNING&#x27; synth.log
# FOLDRING8&#x27;s feasibility gate printed: FR8_SYNTHGATE_RESULT PASS (Infeasible 0, 3D-RAM 0, ERROR 0)
```

- FOLDRING8 s3/s5: Infeasible ultra -> ~50 Mbit LUTRAM, could never fit [M]
- FOLDRING8 fixed RTL: gate 0 Infeasible / 0 3D-RAM / 0 ERROR [M]
- LPT OOC (sequencer + tables): +317 LUT, +4 BRAM36, WNS +1.239 ns at 225 MHz [M]
- PITFALL: A DSP mapping difference in hbml_attn lanes 4/5 produced silicon lane errors on 0x1250; fixed with explicit products + `keep_hierarchy` per lane in 0x1254.
- PITFALL: A full synthesis can sit hours in "Final Netlist Cleanup" on huge flattened designs; consider `-flatten_hierarchy none` or per-block OOC + DCP.
- amd: UG901 UltraRAM coding templates -> https://docs.amd.com/r/en-US/ug901-vivado-synthesis/UltraRAM-Coding-Templates
- amd: UG912 RAM_STYLE -> https://docs.amd.com/r/2024.1-English/ug912-vivado-properties/RAM_STYLE
- amd: UG835 report_utilization -> https://docs.amd.com/r/2025.2-English/ug835-vivado-tcl-commands/report_utilization
- path: build_synth_gate.sh -> /shared/workspace/eagle4_adaptation/hardware/cleanup_EAGLE/scripts/build_synth_gate.sh

## [Toolchain] Implementation and timing closure

opt_design, place_design, phys_opt_design, route_design: what each stage does, our release gate (true WNS >= +0.03 ns, WHS >= +0.010 ns), and how to triage a failing build before re-seeding.

Implementation maps the synthesised netlist onto the physical device:
opt_design: logic optimisation and clean-up on the netlist.place_design: assigns every cell to a site; this is where SLR assignment, pblocks and congestion are decided. Its -directive (e.g. Explore, AggressiveExplore) changes the placer's effort and strategy.phys_opt_design: post-place fixes (replication, retiming, re-placement of critical cells).route_design: wires everything, then timing is final.In the AVED flow, write_device_image produces the PDI and the flash-format packaging follows.
Our release gate: true WNS &gt;= +0.03 ns and true WHS &gt;= +0.010 ns, 0 failing endpoints. "True" means the slack after removing the extra clock uncertainty (UU) we add as margin in the constraints; we compute it per clock. A WHS of 0.000 ns is not a pass: 0x1254 b1 closed at WNS 0.000 / WHS 0.000 and failed 2 of 8 cards on silicon, while s2 of the same RTL at WHS +0.010 passed 52/52 on all 8 (F1257).Triage before re-seedingWhen timing fails, group the failing endpoints by class (source clock -&gt; destination clock, hierarchy of the start and end points) before launching more seeds. A handful of paths in one block at -0.05 ns is a seed problem; -0.227 ns with 1,286 failing endpoints (box1's KVIFIX V1/V3) is structural and needs RTL or floorplan changes. Likewise -3.507 ns with 1.69M failing endpoints on a whole-core 400 MHz build (F1035) meant the clock target itself was wrong.

```
report_timing_summary -max_paths 20 -file timing_summary.rpt     # WNS / TNS / WHS / THS per clock
report_timing -delay_type min -max_paths 50 -file hold.rpt        # worst hold paths
report_clock_interaction -file clock_interaction.rpt              # unsafe / unconstrained clock pairs
report_utilization -slr -file slr_util.rpt
report_design_analysis -congestion -timing -file da.rpt
# place pre-hook: our pblocks and CDC constraints are applied in place_hbms.pre.tcl
```

- 0x1254 b1: WNS 0.000 / WHS 0.000 ns, 2 of 8 cards failed [M]
- 0x1254 s2 (Explore / AggressiveExplore): WNS +0.002 / WHS +0.010 ns, 52/52 x 8 [M]
- record TAIL TCLK75: WNS +0.016 / WHS +0.012 ns [M]
- ring bridge OOC with R8AR_CDC: WNS +1.678 / WHS +0.025 ns [M]
- NP64 at 400 MHz whole core (F1035): WNS -3.507 ns, 1.69M failing [M]
- PITFALL: Explore / AggressiveExplore seeds bought the hold margin that fixed 0x1254; default directives did not.
- PITFALL: In some flows the XPM CDC scoped constraints did not load (empty `report_exceptions`), leaving 60 xpm_cdc_sync_rst endpoints at -1.32 ns; an explicit `set_max_delay -datapath_only` hook (R8AR_CDC) fixed it.
- amd: UG904 directives used by opt_design and place_design -> https://docs.amd.com/r/2025.1-English/ug904-vivado-implementation/Directives-Used-by-opt_design-and-place_design-in-Implementation-Strategies
- amd: UG949 fixing large hold violations prior to routing -> https://docs.amd.com/r/2021.2-English/ug949-vivado-design-methodology/Fixing-Large-Hold-Violations-Prior-to-Routing
- amd: UG906 TIMING-24 overridden max delay datapath only -> https://docs.amd.com/r/2025.1-English/ug906-vivado-design-analysis/TIMING-24-Overridden-Max-Delay-Datapath-Only
- page: Reading timing and utilisation reports -> /p/eagle4-v80-platform/#reports

## [Toolchain] Reading timing and utilisation reports

How to read report_timing_summary (WNS/TNS/WHS/THS, true slack vs added uncertainty), report_utilization -slr, SLL crossings and clock interaction, with our real numbers as examples.

Two reports decide whether a build is releasable: the timing summary and the utilisation report. Both are written by every AVED build (impl_1/*.rpt) and both can be regenerated from a routed checkpoint.
Timing summary
report_timing_summary gives, per clock and overall:
WNS (worst negative slack, setup): the margin on the slowest path. Positive is good.TNS: sum of all negative setup slacks; with WNS it tells you whether a failure is one path or thousands.WHS / THS: the same for hold (paths that are too fast). Hold failures break silicon even when setup is fine.Failing endpoints: how many endpoints violate.
True slack. We add extra clock uncertainty (UU) in our constraints as a safety margin, so the reported slack already includes it. "True" WNS adds the UU back: 0x1254 b1 reported WNS 0.000 ns with a true WNS of +0.090 ns, yet its hold slack was exactly 0.000. Our release gate is on true slack: WNS &gt;= +0.03 ns, WHS &gt;= +0.010 ns, 0 failing [M lesson: WHS 0.000 broke 2 of 8 cards].
Utilisation by SLR
report_utilization -slr (or our *_slr_util_placed.rpt) gives SLICE, LUT, FF, BRAM, URAM, DSP per SLR plus SLL use per boundary. SLICE % is the number to watch; SLL % tells you how much of a boundary's 18,870 wires are used.Worked example: two builds of the same rtl| | 0x1254 b1 | 0x1254 s2 | |---|---|---| | directives | default | place Explore / route AggressiveExplore | | WNS / WHS (reported) | 0.000 / 0.000 ns | +0.002 / +0.010 ns | | silicon | cards 3d, 63 fail attention (41/52, 42/52) | 52/52 on all 8 (F1257) |
And the per-SLR picture of s2 [M]: SLICE 89.8 / 96.9 / 93.8%, LUT 65.3 / 81.0 / 68.7%, DSP 2,080 / 1,225 / 2,080, URAM 0 / 84 / 0, SLL 42% (SLR2-SLR1) and 34% (SLR1-SLR0). Reading: SLR1 is the congested region (SLICE nearly full while LUT is 81%), and DSP / URAM have lots of room.

```
open_checkpoint impl_1/top_wrapper_routed.dcp
report_timing_summary -max_paths 20 -report_unconstrained -file ts.rpt
report_timing -delay_type min -max_paths 50 -sort_by group -file hold.rpt
report_clock_interaction -delay_type min_max -file ci.rpt     # look for &#x27;Timed (unsafe)&#x27; / &#x27;No Common Clock&#x27;
report_utilization -slr -file slr.rpt
report_utilization -hierarchical -hierarchical_depth 4 -file hier.rpt
grep -A3 &#x27;Design Timing Summary&#x27; ts.rpt                         # WNS TNS ... WHS THS in one table
```

- release gate: true WNS >= +0.03 ns, WHS >= +0.010 ns, 0 failing [M]
- 0x1254 s2: WNS +0.002 / WHS +0.010 ns reported [M]
- KVIFIX V1/V3 (box1): WNS -0.227 ns, 1,286 failing (structural) [M]
- NP64 at 400 MHz (F1035): WNS -3.507 ns, 1.69M failing [M]
- PITFALL: "Timing met" with WHS 0.000 is a fail for us.
- PITFALL: Read hold paths per clock pair: CDC paths without proper constraints show up as huge unsafe hold violations; constrain them (`set_max_delay -datapath_only`), don't seed them away.
- PITFALL: Utilisation from synthesis is not placement: SLICE % per SLR only exists after place.
- amd: UG906 details of the timing summary report -> https://docs.amd.com/r/2025.1-English/ug906-vivado-design-analysis/Details-of-the-Timing-Summary-Report
- amd: UG906 report timing summary -> https://docs.amd.com/r/2025.1-English/ug906-vivado-design-analysis/Report-Timing-Summary
- amd: UG906 clock uncertainty -> https://docs.amd.com/r/2025.1-English/ug906-vivado-design-analysis/Clock-Uncertainty
- amd: UG906 hold/removal relationship -> https://docs.amd.com/r/en-US/ug906-vivado-design-analysis/Hold/Removal-Relationship
- amd: UG906 report clock interaction -> https://docs.amd.com/r/2020.2-English/ug906-vivado-design-analysis/Report-Clock-Interaction
- amd: UG906 report SLR crossing -> https://docs.amd.com/r/en-US/ug906-vivado-design-analysis/Report-SLR-Crossing
- amd: UG835 report_timing_summary -> https://docs.amd.com/r/en-US/ug835-vivado-tcl-commands/report_timing_summary
- amd: UG835 report_utilization -> https://docs.amd.com/r/2025.2-English/ug835-vivado-tcl-commands/report_utilization

## [Toolchain] xsim simulation

Vivado's simulator: unit gates, RED cases that must fail, full-engine runs bit-exact against Python emulators. All heavy runs on the mesh, always batch with a timeout.

Every image gets simulation gates before or during its build. A gate is a testbench run whose output is compared word-for-word with a Python emulator (the ISA model) or golden file. Three kinds:
GREEN cases: real workloads (GEMV formats, a full layer, attention, KV, spec-decode steps, ring all-reduce at 8 ranks) that must match bit-exactly.RED cases: deliberately planted bugs (a flipped weight bit, a swapped ring direction, a field cut from 4 to 3 bits) that the gate must detect. A gate that cannot fail proves nothing.Per-card clock cases: ring designs simulated with every card on its own clock at +-1000 ppm, because a shared-clock testbench hid the CDC FIFO overflow that broke the ring on silicon.
The BOOTSKIP flow (BR084) skips the long boot sequence for the drafter testbenches.

```
xvlog -sv rtl/*.sv tb/tb.sv &amp;&amp; xelab tb -s snap &amp;&amp; xsim snap -runall      # batch, never GUI
# through the mesh (never on asgard):
~/mesh/mesh_run.sh --class xsim_<name> --peak 8 --host olegvm3 --cwd <dir> --log <dir>/x.log -- bash gates.sh
# typical gate line we grep:
#   GREEN expect=PASS got=PASS case=c12_perf_f4 xsim_errors=0 ... mismatched words 0
```

- HBM-FOLD port gates: 12/12 bit-exact; 3.51 beats/cycle/port [M xsim]
- ring bridge G-AR, 8 ranks +-1000 ppm: PASS, emulator identical; REDs fire [M xsim]
- ring all-reduce P=8, 3,072 int32: 11.47 us [M xsim]
- FOLDRING8 G3-LINK: 6/6 tokens bit-exact [M xsim]
- PITFALL: Two link_dma testbenches ran 17 days hung in GUI sim mode on midyard: always `-runall` in batch with a wall-clock timeout.
- PITFALL: A self-imposed 3000 s timeout killed a gate that was still waiting for mesh admission; budget the timeout after admission, not from submission.
- amd: UG900 running the Vivado simulator in batch mode -> https://docs.amd.com/r/en-US/ug900-vivado-logic-simulation/Running-the-Vivado-Simulator-in-Batch-Mode
- amd: UG900 xsim executable options -> https://docs.amd.com/r/en-US/ug900-vivado-logic-simulation/xsim-Executable-Options
- path: ring bridge gates -> /shared/rnd_hq/2026-10-15/tracks/hbms_ar_ring_1256/FROZEN_FOR_125D.md

## [Toolchain] Build mesh

One queue for every heavy job across olegvm, olegvm2, olegvm3 and midyard, admitted by declared memory peak; pdi_build classes go first. How to submit, read status and unstick a queue.

mesh_core.py run --peak GB --class NAME -- cmd waits until a host has headroom for the declared peak, then runs the job and records its measured peak in the class history. mesh_run.sh wraps it. Admission rules that shape every night:
Classes starting with pdi_build are served first; the rest in waiting order.DRAIN: the oldest waiter that does not fit blocks everything behind it on that host (no back-filling), so one oversized job can stall a whole host.There is also a CPU cap: mesh threads &lt;= 1.25 x nproc.A reservation holds headroom until the job's process tree exits, even when its real memory (cur) is far lower.How to read mesh_statusA running job looks like [pdi_build_r8best 2889195 res=90.0 eff=75.0 cur=1.8 pk=1.8 0m]: res reserved, eff the effective reservation the admitter uses, cur real memory now, pk peak so far. A waiter looks like {WAIT pdi_build_kvifix_a 86.0G 43m DRAIN}. If headroom is below the first waiter's peak, everything behind it waits: that is the usual cause of "why is my job queued".Worked example (10-15)olegvm2 had 79 GB of headroom; the first waiter (a box1 PDI) needed 86 GB, so R8_BEST's 90 GB build and several small jobs sat in DRAIN while four long pathfinder builds held ~350 GB of reservations. Stopping the pathfinders (user order) admitted the box1 PDI and R8_BEST s1 within a minute.

```
~/mesh/mesh_status.sh             # per host: avail, headroom, thr, [running jobs], {WAIT waiters}
~/mesh/mesh_run.sh --peak 30 --class vivado_ooc_c --host olegvm2 --threads 8 --cwd <dir> --log <dir>/x.log -- vivado -mode batch -source x.tcl
bash /shared/rnd_hq/tools/withdraw_waiter.sh <host> <pid> --dry-run    # pre-admission waiters only
bash /shared/rnd_hq/tools/stale_kill.sh <host> <pid> --dry-run         # stale >= 2 h only
```

- hosts: olegvm 125 GB, olegvm2 / olegvm3 251 GB, midyard 94 GB [M]
- measured r8merge peak: ~54 GB through early impl [M]
- HBMS full build eff: ~75 GB [M]
- PITFALL: Honest but big reservations (89-92 GB) stall queues while real free RAM is high; declare measured peaks.
- PITFALL: A 90 GB job can never start on a 94 GB host that already runs another build.
- PITFALL: Withdrawing and resubmitting loses queue position unless `--waited-since` is passed.
- path: mesh-builds skill -> ~/.claude/skills/mesh-builds/SKILL.md
- page: Mesh queue design -> /p/mesh-queue-dispatch-20260924/
