mlpv_gemv_lane_scaled_stream_dual_ext_opt.sv.md
Source reports/results/GEMM_REVISION_20261117/rtl_snapshot/mlpv_gemv_lane_scaled_stream_dual_ext_opt.sv.md · source md5 a926cebe0eb6e16f0b811b04caf0dd76 · EAGLE4 FPGA archive, 2026-11-18
# Snapshot: verilog/clean7c/rtl/array/mlpv/gemv_lane_scaled_stream_dual_ext_opt.sv
Copied 2026-11-17T17:32:37Z from the clean7c tree (verilog repo commit 5998a7f1, md5 4a89d2e846002fd3f3fdbf4901dab089). Read-only copy for review; the live file is in the RTL repo.
```systemverilog
// =====================================================================================================================
// gemv_lane_scaled_stream_dual_ext_opt (clean7c rtl/array/mlpv, WP3)
// =====================================================================================================================
// Imported from the shipped packaged copy ref/mlpv_1361/packaged_ip/src/gemv_lane_scaled_stream_dual_ext_opt.sv md5 7ca4843b78210938893d328d0e718b92 (MLPV 0xE4B71361 (M1361, PDI e24deb3f)).
// Code = the shipped code; only this header, the items under 'clean7c edits' and the role include differ.
// Regenerate / verify: tools/wp3_import.py [--check]. Architecture + diagrams: docs/ARRAY.md.
//
// Purpose : one output column of the MLPV GEMV: dual-read-port scaled lane (2 K-chunks per cycle: even via URAM port A,
// odd via port B) whose two 16-term dot products are computed OUTSIDE the lane by the array's cross-lane
// packed-DSP58 unit (DOT_EXT). The lane runs the pair-walk FSM, exports its WLAT-aligned raw weight words
// (x_qw_e/x_qw_o), the activations (x_a_e, odd masked past nc) and the issue valid x_v, imports the two INT26
// sums EXT_LAT = 6 cycles later, folds them per GPTQ group (GW_N 26-bit, NARROW 1) and applies one scale multiply
// per group into the 48-bit result_q. TILE_PIPE 1: two accumulator banks (col_tag) so a new column can start on
// the terminal pair cycle; MASK_SUM 1: the odd partial-pair mask acts on the 26-b odd sum; ACC_DBG 0: acc_out = 0.
// (The file header says "_dual_ext": this module is the _opt copy; INVENTORY §5.)
// Users : u_array/g_grp[*].g_lane[*].g_dual_mo.g_e3.u_lane (256).
// Latency : WLAT 1 (g_wlat1): qw_word/qw_word_b/scale/issue index re-registered once to meet the +1-deeper seam
// activation; dot result EXT_LAT after x_v; capture when the column's own last pair result lands (lcol_now).
// Reset : synchronous active-low. KNOWN HAZARD (docs/ARRAY.md §7): g_wlat1 qw_d/qwb_d/sc_d/ec_d are pure datapath bits in
// the reset block of civ_d/tp_tag_d -- on M1364 place_design set IS_R_INVERTED 0 on 12 qw_d bits of lane 84
// (held in reset in operation: x_qw_e bits 0x5F077000 stuck 0). Also reset: the CSA_LAT tag/scale shift registers,
// the accumulator banks and result_q. Not changed here (lever L-RST-DP).
// Instances in the image: 256 (resolved parameters: rtl/array/SHIPPED_PARAMS.tsv, rows 'mlpv gemv_lane_scaled_stream_dual_ext_opt').
// Ports (generated from the port list; widths as declared, resolved values are in SHIPPED_PARAMS.tsv):
// input clk 1
// input rst_n 1
// input start 1
// input a_chunk signed [K_PAR*A_WIDTH-1:0]
// input a_chunk_b signed [K_PAR*A_WIDTH-1:0]
// input qw_word [31:0]
// input qw_word_b [31:0]
// input qz_raw [1:0]
// input scale_q signed [SCALE_WIDTH-1:0]
// input n_chunks_run [CW-1:0]
// output chunk_idx_out [CW-1:0]
// output chunk_idx_b_out [CW-1:0]
// output chunk_idx_valid 1
// output chunk_idx_next_out [CW-1:0]
// output chunk_idx_b_next_out [CW-1:0]
// output group_idx_out [GW-1:0]
// output group_idx_next_out [GW-1:0]
// output acc_out signed [31:0]
// output result_q signed [RES_WIDTH-1:0]
// output done 1
// output start_dropped 1
// input fold_i 1
// input fold_init_i signed [RES_WIDTH-1:0]
// output x_qw_e [31:0]
// output x_qw_o [31:0]
// output x_a_e [K_PAR*A_WIDTH-1:0]
// output x_a_o [K_PAR*A_WIDTH-1:0]
// output x_v 1
// input x_s_e signed [25:0]
// input x_s_o signed [25:0]
// input x_sv 1
// clean7c edits:
// - dead branch `g_wlat0` (WLAT 0) removed -> stub (shipped :341-349)
// - `ifdef RED_NOMASK arm removed (1 lines; no shipped build defines RED_NOMASK), `else text kept verbatim
// - `ifdef RED_GW arm removed (1 lines; no shipped build defines RED_GW), `else text kept verbatim
// - default K 64 -> 8192 (every gemv_lane_scaled_stream_dual_ext_opt instance of the image resolves 8192)
// - default SCALE_FRAC 15 -> 13 (every gemv_lane_scaled_stream_dual_ext_opt instance of the image resolves 13)
// - default WLAT 1'b0 -> 1'b1 (every gemv_lane_scaled_stream_dual_ext_opt instance of the image resolves 1'b1)
// - default TILE_PIPE 1'b0 -> 1'b1 (every gemv_lane_scaled_stream_dual_ext_opt instance of the image resolves 1'b1)
// - default EXT_LAT 7 -> 6 (every gemv_lane_scaled_stream_dual_ext_opt instance of the image resolves 6)
// - default ACC_DBG 1'b1 -> 1'b0 (every gemv_lane_scaled_stream_dual_ext_opt instance of the image resolves 1'b0)
// - default MASK_SUM 1'b0 -> 1'b1 (every gemv_lane_scaled_stream_dual_ext_opt instance of the image resolves 1'b1)
// - default NARROW 1'b0 -> 1'b1 (every gemv_lane_scaled_stream_dual_ext_opt instance of the image resolves 1'b1)
// =====================================================================================================================
`timescale 1ns/1ps
`default_nettype none
(* keep_hierarchy = "yes" *)
module gemv_lane_scaled_stream_dual_ext_opt #(
parameter int K = 8192,
parameter int K_PAR = 16,
parameter int A_WIDTH = 16,
parameter int GROUP = 128, // GPTQ group_size
parameter int SCALE_WIDTH = 16, // signed scale
parameter int SCALE_FRAC = 13, // fractional bits (informational)
parameter int RES_WIDTH = 48, // scaled fixed-point accumulator width
// WLAT (Stage-B' D1 activation-seam fix for the DUAL lane): mirror of the
// single lane's WLAT. On the SLR_SAFE seam the activation (a_chunk/a_chunk_b)
// arrives ONE register deeper than the weights (qw_word/qw_word_b), so at
// WLAT=0 the dual lane computes W[pair]·act[pair-1] on the body GEMV (the D1
// stale-by-1 class). WLAT=1 delays BOTH weight chunks + the MAC enable + the
// per-pair issue index (group tag + odd-mask) + scale_q by ONE cycle so the
// (already +1-deeper) activation and the delayed weights meet. BIT-EXACT: all
// operands and their tags shift by the SAME 1 cycle; the extra cycle is
// absorbed by S_DRAIN (CSA_LAT+2 margin). WLAT=0 = legacy (bit+cycle identical
// to the isolation-proven dual lane).
parameter bit WLAT = 1'b1,
// TILE_PIPE (L1, 2026-08-18) — see gemv_lane_scaled_stream.sv for the full
// rationale. Identical mechanism on the pair walk: `start` is also accepted
// on the TERMINAL PAIR cycle, a 1-bit col_tag rides the WLAT delay and the
// CSA_LAT shift register, and TWO accumulator banks keep the draining
// column separate from the one already running. At TILE_PIPE=0 col_tag is a
// constant 0 and every bank-0 register reproduces the legacy behaviour
// cycle-for-cycle.
parameter bit TILE_PIPE = 1'b1,
// G4_EMBFOLD (2026-09-23, fork g4_embfold_20260923): constant-fold the token-only
// embedding half of the K reduction. When EMBFOLD=1 and `fold_i` is high on the
// cycle a column is accepted, the column walk STARTS at chunk FOLD_BASE (not 0) and
// the 48-bit scaled accumulator starts at `fold_init_i` (not 0). BIT-EXACT by integer
// associativity: every term is an integer (per-group INT sum x integer scale, 48-bit
// wrap), so acc = P + sum_{groups >= FOLD_BASE/CPG} == sum_{all groups} exactly when
// P == sum_{groups < FOLD_BASE/CPG} (mod 2^48). FOLD_BASE must be a multiple of CPG
// (no group straddles the fold point) and even (pair walk). EMBFOLD=0: identical.
parameter bit EMBFOLD = 1'b0,
parameter int FOLD_BASE = 0,
// DOT_EXT lane (HX1 DSP-GEMV, 2026-09-27): the even/odd 16-term dot products
// are computed OUTSIDE the lane by the array's cross-lane packed-DSP58 unit
// (e3b_packed_dot: lanes L and L+1 share the activation broadcast, so one
// DSP58 multiply forms a*(w_L + 2^23*w_{L+1})). The lane EXPORTS its
// WLAT-aligned raw weight words, the (odd-masked) activations and the issue
// valid, and IMPORTS the two INT26 sums EXT_LAT cycles later. Everything else
// (FSM, WLAT, tags, group fold, scale, TILE_PIPE banks, EMBFOLD) is the
// unmodified gemv_lane_scaled_stream_dual body with CSA_LAT := EXT_LAT.
parameter int EXT_LAT = 6,
// ---- MODOPT array-lane options (defaults = bit/cycle-identical to HEAD) ----
parameter bit ACC_DBG = 1'b0, // 0: drop the dead unscaled debug accumulator (acc_out := 0)
parameter bit MASK_SUM = 1'b1, // 1: odd-chunk partial-pair mask applied to the 26-b odd SUM (delayed) instead of the 256-b activation
parameter bit NARROW = 1'b1, // 1: provably-sufficient group accumulator / scale-multiply width (+ narrow CSA when !EXT)
parameter int N_CHUNKS = K / K_PAR,
parameter int CW = (N_CHUNKS <= 1) ? 1 : $clog2(N_CHUNKS + 2),
parameter int CPG = GROUP / K_PAR, // chunks/group (8)
parameter int N_GROUPS = (K + GROUP - 1) / GROUP,
parameter int GW = (N_GROUPS <= 1) ? 1 : $clog2(N_GROUPS + 1),
// pair-walk count width (ceil(N_CHUNKS/2))
parameter int N_PAIRS = (N_CHUNKS + 1) / 2,
parameter int PW = (N_PAIRS <= 1) ? 1 : $clog2(N_PAIRS + 2)
) (
input wire clk,
input wire rst_n,
input wire start,
// Streaming activation: EVEN chunk (port A) + ODD chunk (port B) per cycle
input wire signed [K_PAR*A_WIDTH-1:0] a_chunk, // even chunk (2p)
input wire signed [K_PAR*A_WIDTH-1:0] a_chunk_b, // odd chunk (2p+1)
// Streaming qweight: even word (port A) + odd word (port B)
input wire [31:0] qw_word, // even chunk weights
input wire [31:0] qw_word_b, // odd chunk weights
// Per-group qzero (one 2-bit value); held for the current group
input wire [1:0] qz_raw,
// Per-group dequant scale, held for the CURRENT group
input wire signed [SCALE_WIDTH-1:0] scale_q,
// RUNTIME terminal chunk count for the FSM walk (G12_K_PATH FIX-1 idiom).
// The lane is ELABORATED for the worst-case K_MAX (N_CHUNKS) but only walks
// n_chunks_run chunks per start; 0 -> full N_CHUNKS (unconnected-safe).
input wire [CW-1:0] n_chunks_run,
// FSM exposes the EVEN chunk index it expects (port A); the ODD index
// (port B) is +1. Caller drives a_chunk/qw_word off chunk_idx_out and
// a_chunk_b/qw_word_b off chunk_idx_b_out.
output wire [CW-1:0] chunk_idx_out, // even (2p)
output wire [CW-1:0] chunk_idx_b_out, // odd (2p+1)
output wire chunk_idx_valid,
// Next-state EVEN/ODD chunk_idx for URAM 1-cycle lookahead (ports A/B).
output wire [CW-1:0] chunk_idx_next_out,
output wire [CW-1:0] chunk_idx_b_next_out,
// Group index the lane is currently consuming (drives scale-BRAM addr).
output wire [GW-1:0] group_idx_out,
output wire [GW-1:0] group_idx_next_out,
// Final UNSCALED INT32 accumulator (debug; == single-port lane's acc_out).
output reg signed [31:0] acc_out,
// Final SCALED fixed-point accumulator (Q with SCALE_FRAC frac bits).
output reg signed [RES_WIDTH-1:0] result_q,
output reg done,
// F1 (RD_FANOUT_AUDIT L1 follow-up): STICKY "a `start` was presented while
// this lane could not accept it". Under TILE_PIPE the acceptance window is
// exactly ONE cycle wide (the terminal chunk); if the 4-register issue chain
// lands +/-1 cycle off, the start is SILENTLY discarded -> no `done` ever
// -> the caller waits forever and the tile-tag FIFO desyncs. Nothing else in
// the design counts that. REPORT-ONLY: it drives an observability bit, it
// NEVER gates or aborts anything (feedback_gates_report_only_never_kill_builds).
output reg start_dropped,
// G4_EMBFOLD: sampled on col_accept (same cycle as the accepted `start`).
input wire fold_i,
input wire signed [RES_WIDTH-1:0] fold_init_i,
// ---- DOT_EXT export / import ----
output wire [31:0] x_qw_e, // WLAT-aligned even raw weight word
output wire [31:0] x_qw_o, // WLAT-aligned odd raw weight word
output wire [K_PAR*A_WIDTH-1:0] x_a_e, // even activation chunk
output wire [K_PAR*A_WIDTH-1:0] x_a_o, // odd activation chunk, zero-masked past nc
output wire x_v, // issue valid (== mac_in_valid)
input wire signed [25:0] x_s_e, // sum_i a_e[i]*w_e[i], EXT_LAT after x_v
input wire signed [25:0] x_s_o,
input wire x_sv // valid of x_s_e/x_s_o
);
`ifndef SYNTHESIS
initial if (CPG[0] != 1'b0)
$error("gemv_lane_scaled_stream_dual_ext: CPG (GROUP/K_PAR=%0d) must be EVEN for the dual-port pair/group alignment.", CPG);
`endif
localparam int CSA_LAT = EXT_LAT; // DOT_EXT: external dot latency
localparam int CPG_BITS = (CPG <= 1) ? 1 : $clog2(CPG);
// Forward declares for the FSM (driven by the two CSA trees below).
wire signed [25:0] csa_sum_e, csa_sum_o, csa_sum_o_raw;
wire csa_out_valid_e, csa_out_valid_o;
wire csa_out_valid; // == csa_out_valid_e (identical timing)
wire signed [26:0] csa_pair_sum; // csa_sum_e + csa_sum_o per cycle
// ----------------------------------------------------------------
// FSM — pair walk. Mirrors gemv_lane_scaled_stream but the run counter
// is the PAIR index (advances by 1 pair = 2 chunks per cycle).
// ----------------------------------------------------------------
typedef enum logic [1:0] { S_IDLE, S_RUN, S_DRAIN, S_DONE } state_t;
state_t state, state_n;
wire state_is_idle = (state == S_IDLE);
// ----------------------------------------------------------------
// Effective terminal chunk count, latched at start (same as single-port).
// ----------------------------------------------------------------
logic [CW-1:0] nc_run_q;
wire [CW-1:0] nc_eff = (nc_run_q == '0) ? CW'(N_CHUNKS) : nc_run_q;
// number of PAIRS to walk = ceil(nc_eff/2) (forward-used by col_accept)
wire [PW-1:0] np_eff = PW'((nc_eff + CW'(1)) >> 1);
logic [PW-1:0] pair_idx, pair_idx_n;
logic [CW-1:0] drain_count, drain_count_n;
logic [PW-1:0] outs_recvd, outs_recvd_n; // # pair-CSA results received
wire at_terminal_pair = (state == S_RUN) && (pair_idx == np_eff - 1'b1);
wire col_accept = (state_is_idle && start)
|| (TILE_PIPE && at_terminal_pair && start);
always_ff @(posedge clk) begin
if (!rst_n) nc_run_q <= '0;
else if (col_accept) nc_run_q <= n_chunks_run;
end
// G4_EMBFOLD: the accepted column is folded (start at FOLD_BASE, acc init = fold_init_i).
localparam int FOLD_PBASE = FOLD_BASE / 2;
wire fold_now = EMBFOLD && fold_i;
reg fold_q;
always_ff @(posedge clk) begin
if (!rst_n) fold_q <= 1'b0;
else if (col_accept) fold_q <= fold_now;
end
wire [PW-1:0] pair_start = fold_now ? PW'(FOLD_PBASE) : '0;
// pair-CSA results this column delivers (the drain-exit count)
wire [PW-1:0] np_run = (EMBFOLD && fold_q) ? (np_eff - PW'(FOLD_PBASE)) : np_eff;
`ifndef SYNTHESIS
initial if (EMBFOLD && ((FOLD_BASE % CPG) != 0 || (FOLD_BASE % 2) != 0 || FOLD_BASE <= 0))
$error("gemv_lane_scaled_stream_dual_ext: EMBFOLD needs FOLD_BASE (%0d) > 0, even and a multiple of CPG (%0d) -- a group must not straddle the fold point.", FOLD_BASE, CPG);
`endif
// F1: sticky dropped-start observability (see the port comment).
always_ff @(posedge clk) begin
if (!rst_n) start_dropped <= 1'b0;
else if (start && !col_accept && (state != S_IDLE)) start_dropped <= 1'b1;
end
`ifndef SYNTHESIS
// F4 (audit): ANNOTATED LIMIT — N_PAIRS is the ELABORATED worst case, not
// the runtime per-stage count; sizing sanity only, the runtime arm is below.
initial if (TILE_PIPE && (N_PAIRS <= CSA_LAT + 2))
$error("gemv_lane_scaled_stream_dual_ext: TILE_PIPE=1 needs N_PAIRS > CSA_LAT+2 (got %0d).", N_PAIRS);
// F5 (audit): police the INCOMING count (in PAIRS), and include WLAT. See
// the single-port lane for the full rationale. xsim exits 0 on $fatal.
always_ff @(posedge clk)
if (rst_n && TILE_PIPE && col_accept && (n_chunks_run != '0) &&
(((n_chunks_run + CW'(1)) >> 1) <= CW'(CSA_LAT + (WLAT?1:0))))
$fatal(1, "gemv_lane_scaled_stream_dual_ext: TILE_PIPE column too short (n_chunks_run=%0d pairs=%0d <= CSA_LAT+WLAT=%0d).",
n_chunks_run, (n_chunks_run+CW'(1))>>1, CSA_LAT + (WLAT?1:0));
`endif
always_ff @(posedge clk) begin
if (!rst_n) begin
state <= S_IDLE; pair_idx <= '0; drain_count <= '0; outs_recvd <= '0;
end else begin
state <= state_n; pair_idx <= pair_idx_n;
drain_count <= drain_count_n; outs_recvd <= outs_recvd_n;
end
end
always_comb begin
state_n = state; pair_idx_n = pair_idx;
drain_count_n = drain_count; outs_recvd_n = outs_recvd;
case (state)
S_IDLE: if (start) begin
state_n = S_RUN; pair_idx_n = pair_start;
drain_count_n = '0; outs_recvd_n = '0;
end
S_RUN: begin
if (pair_idx == np_eff - 1'b1) begin
// TILE_PIPE: accept the next column's start on the terminal
// pair cycle -> zero idle cycles between columns.
if (TILE_PIPE && start) begin
state_n = S_RUN; pair_idx_n = pair_start;
drain_count_n = '0; outs_recvd_n = '0;
end else begin
state_n = S_DRAIN; drain_count_n = '0;
end
end else pair_idx_n = pair_idx + 1'b1;
if (csa_out_valid) outs_recvd_n = outs_recvd + 1'b1;
end
S_DRAIN: begin
if (csa_out_valid) outs_recvd_n = outs_recvd + 1'b1;
if (outs_recvd == np_run) state_n = S_DONE;
else if (drain_count >= CSA_LAT + 2) state_n = S_DONE;
drain_count_n = drain_count + 1'b1;
end
S_DONE: state_n = S_IDLE;
default: state_n = S_IDLE;
endcase
end
// EVEN chunk = 2*pair_idx ; ODD chunk = 2*pair_idx + 1
wire [CW-1:0] even_cur = CW'(pair_idx) << 1;
wire [CW-1:0] even_next = CW'(pair_idx_n) << 1;
assign chunk_idx_out = even_cur;
assign chunk_idx_b_out = even_cur + CW'(1);
assign chunk_idx_next_out = even_next;
assign chunk_idx_b_next_out = even_next + CW'(1);
assign chunk_idx_valid = (state == S_RUN);
// Group index = even_chunk / CPG (== odd_chunk/CPG for even CPG). Current +
// next for the scale-BRAM 1-cycle lookahead.
wire [CW-1:0] grp_cur_full = (CPG <= 1) ? even_cur : (even_cur >> CPG_BITS);
wire [CW-1:0] grp_next_full = (CPG <= 1) ? even_next : (even_next >> CPG_BITS);
assign group_idx_out = grp_cur_full[GW-1:0];
assign group_idx_next_out = grp_next_full[GW-1:0];
// ----------------------------------------------------------------
// WLAT (D1 dual-lane seam fix): delay BOTH weight chunks + MAC enable +
// the per-pair issue index (group tag + odd-mask) + scale_q by ONE cycle
// to meet the +1-deeper SLR-seam activation. Pass-through at WLAT=0
// (bit+cycle identical to the isolation-proven dual lane). chunk_idx_out /
// group_idx_out stay UNDELAYED (the caller's fetch index; the seam adds the
// register externally) — same idiom as the single lane.
// ----------------------------------------------------------------
wire [31:0] qw_word_eff, qw_word_b_eff;
wire mac_in_valid;
wire [CW-1:0] even_issue;
wire signed [SCALE_WIDTH-1:0] scale_q_eff;
// TILE_PIPE column tag (see the single-port lane). Inert at TILE_PIPE=0.
reg col_tag;
always_ff @(posedge clk) begin
if (!rst_n) col_tag <= 1'b0;
else if (TILE_PIPE && col_accept) col_tag <= ~col_tag;
end
wire tag_pre = col_tag;
wire lastcol_pre = (pair_idx == np_eff - 1'b1);
wire tag_issue;
wire lastcol_issue;
generate
if (!WLAT) begin : g_wlat0
// clean7c WP3: branch `g_wlat0` (WLAT 0) is elaborated by NO instance of MLPV 0xE4B71361 (M1361, PDI e24deb3f) and was removed
// (shipped text: ref/mlpv_1361/packaged_ip/src/gemv_lane_scaled_stream_dual_ext_opt.sv:341-349). Selecting it is an elaboration
// error, never silently different hardware; restoring it is a lever (PLAN.md §6), not a cleanup.
C7_REMOVED_BRANCH__gemv_lane_scaled_stream_dual_ext_opt__g_wlat0 u_c7_removed ();
end else begin : g_wlat1
reg [31:0] qw_d, qwb_d;
reg civ_d;
reg [CW-1:0] ec_d;
reg signed [SCALE_WIDTH-1:0] sc_d;
reg tp_tag_d, tp_lcol_d;
always_ff @(posedge clk) begin
if (!rst_n) begin
qw_d <= '0; qwb_d <= '0; civ_d <= 1'b0; ec_d <= '0; sc_d <= '0;
tp_tag_d <= 1'b0; tp_lcol_d <= 1'b0;
end else begin
qw_d <= qw_word; qwb_d <= qw_word_b; civ_d <= chunk_idx_valid;
ec_d <= even_cur; sc_d <= scale_q;
tp_tag_d <= tag_pre; tp_lcol_d <= lastcol_pre;
end
end
assign qw_word_eff = qw_d;
assign qw_word_b_eff = qwb_d;
assign mac_in_valid = civ_d;
assign even_issue = ec_d;
assign scale_q_eff = sc_d;
assign tag_issue = tp_tag_d;
assign lastcol_issue = tp_lcol_d;
end
endgenerate
// ----------------------------------------------------------------
// ODD-chunk validity mask: when 2p+1 >= nc_eff (odd nc final pair), the
// odd MAC must contribute 0. Force a_odd = 0 so csa_o = 0 exactly.
// (Uses even_issue = the WLAT-aligned index.)
// ----------------------------------------------------------------
wire odd_valid_issue = ((even_issue + CW'(1)) < nc_eff);
wire signed [K_PAR*A_WIDTH-1:0] a_odd_eff =
(odd_valid_issue || MASK_SUM) ? a_chunk_b : '0;
// ----------------------------------------------------------------
// DOT_EXT: dequant + 2 x 16 MAC + 2 Wallace trees are REPLACED by the
// array-level packed-DSP unit. Export the exact operands the in-lane
// datapath would have consumed (same WLAT-aligned words, same odd mask).
// qz_raw is broadcast per group, so the unit takes it from the array.
// ----------------------------------------------------------------
assign x_qw_e = qw_word_eff;
assign x_qw_o = qw_word_b_eff;
assign x_a_e = a_chunk;
assign x_a_o = a_odd_eff;
assign x_v = mac_in_valid;
assign csa_sum_e = x_s_e;
assign csa_sum_o_raw = x_s_o;
assign csa_out_valid_e = x_sv;
assign csa_out_valid_o = x_sv;
assign csa_out_valid = csa_out_valid_e;
// ----------------------------------------------------------------
// UNSCALED INT32 accumulator (debug; matches single-port acc_out exactly).
// Per cycle accumulate (csa_e + csa_o). INT26 + INT26 = up to INT27.
// ----------------------------------------------------------------
// MASK_SUM: odd_valid_issue delayed CSA_LAT deep (aligned with the dot result) gates the 26-b odd sum.
logic ov_sr [0:CSA_LAT-1];
integer ovs;
always_ff @(posedge clk) begin
if (!rst_n) begin
for (ovs = 0; ovs < CSA_LAT; ovs = ovs + 1) ov_sr[ovs] <= 1'b1;
end else begin
ov_sr[0] <= odd_valid_issue;
for (ovs = 1; ovs < CSA_LAT; ovs = ovs + 1) ov_sr[ovs] <= ov_sr[ovs-1];
end
end
assign csa_sum_o = (MASK_SUM && !ov_sr[CSA_LAT-1]) ? 26'sd0 : csa_sum_o_raw;
assign csa_pair_sum = $signed(csa_sum_e) + $signed(csa_sum_o);
// TILE_PIPE column-tag / last-pair-of-column shift register, CSA_LAT deep.
logic tag_sr [0:CSA_LAT-1];
logic lcol_sr [0:CSA_LAT-1];
integer ts;
always_ff @(posedge clk) begin
if (!rst_n) begin
for (ts = 0; ts < CSA_LAT; ts = ts + 1) begin
tag_sr[ts] <= 1'b0; lcol_sr[ts] <= 1'b0;
end
end else begin
tag_sr[0] <= tag_issue;
lcol_sr[0] <= lastcol_issue && mac_in_valid;
for (ts = 1; ts < CSA_LAT; ts = ts + 1) begin
tag_sr[ts] <= tag_sr[ts-1]; lcol_sr[ts] <= lcol_sr[ts-1];
end
end
end
wire tag_now = tag_sr[CSA_LAT-1];
wire lcol_now = lcol_sr[CSA_LAT-1];
wire new_tag = TILE_PIPE ? ~col_tag : 1'b0;
logic signed [31:0] acc_b [0:1];
wire signed [31:0] acc_next = acc_b[tag_now] + $signed(csa_pair_sum);
genvar ab;
for (ab = 0; ab < 2; ab = ab + 1) begin : g_accbank
always_ff @(posedge clk) begin
if (!rst_n) acc_b[ab] <= '0;
else if (col_accept && (new_tag == ab[0])) acc_b[ab] <= '0;
else if (csa_out_valid && (tag_now == ab[0])) acc_b[ab] <= acc_next;
end
end
// ================================================================
// SCALED PATH — per-group fold then one DSP scale.
// ================================================================
// Tag each PAIR at issue with its group, last-in-group, and the scale the
// caller drove for that group; shift CSA_LAT deep to line up with the
// pair-CSA result. Identical idiom to gemv_lane_scaled_stream.
wire [CW-1:0] grp_issue_full = (CPG <= 1) ? even_issue : (even_issue >> CPG_BITS);
wire [GW-1:0] grp_issue = grp_issue_full[GW-1:0];
// group's last VALID chunk = min((grp+1)*CPG-1, nc_eff-1).
wire [CW-1:0] grp_end_full = ((grp_issue_full + CW'(1)) << CPG_BITS) - CW'(1);
wire [CW-1:0] grp_last = (grp_end_full < (nc_eff - CW'(1)))
? grp_end_full : (nc_eff - CW'(1));
// does THIS pair contain grp_last? odd member if odd_valid, else even member.
wire last_in_grp_issue = odd_valid_issue ? ((even_issue + CW'(1)) == grp_last)
: (even_issue == grp_last);
wire issue_valid = mac_in_valid;
logic [GW-1:0] grp_sr [0:CSA_LAT-1];
logic last_sr [0:CSA_LAT-1];
logic vld_sr [0:CSA_LAT-1];
logic signed [SCALE_WIDTH-1:0] sc_sr [0:CSA_LAT-1];
integer s;
always_ff @(posedge clk) begin
if (!rst_n) begin
for (s = 0; s < CSA_LAT; s = s + 1) begin
grp_sr[s] <= '0; last_sr[s] <= 1'b0; vld_sr[s] <= 1'b0; sc_sr[s] <= '0;
end
end else begin
grp_sr[0] <= grp_issue;
last_sr[0] <= last_in_grp_issue;
vld_sr[0] <= issue_valid;
sc_sr[0] <= scale_q_eff;
for (s = 1; s < CSA_LAT; s = s + 1) begin
grp_sr[s] <= grp_sr[s-1]; last_sr[s] <= last_sr[s-1];
vld_sr[s] <= vld_sr[s-1]; sc_sr[s] <= sc_sr[s-1];
end
end
end
wire last_now = last_sr[CSA_LAT-1];
wire signed [SCALE_WIDTH-1:0] scale_now = sc_sr[CSA_LAT-1];
// Per-group INT accumulator (fold pair sums), flush*scale at group end.
// NARROW: |group sum| <= GROUP * 2^(A_WIDTH+1) (|a|<=2^(A_WIDTH-1), |w|<=4) = 2^(A_WIDTH+2+log2 GROUP-?);
// signed width A_WIDTH+3+clog2(GROUP) (=26 at A=16,GROUP=128). Arithmetic is modulo-2^GW_N and the true value
// fits, so the sign-extension to 32 is exact.
localparam int GW_N = NARROW ? (A_WIDTH + 3 + $clog2(GROUP)) : 32;
logic signed [GW_N-1:0] group_acc_b [0:1];
logic signed [RES_WIDTH-1:0] scaled_acc_b [0:1];
wire signed [GW_N-1:0] g_total_w = group_acc_b[tag_now] + GW_N'($signed(csa_pair_sum));
wire signed [RES_WIDTH-1:0] scaled_next_w =
last_now ? (scaled_acc_b[tag_now] +
($signed(g_total_w) * $signed(scale_now)))
: scaled_acc_b[tag_now];
genvar sb;
for (sb = 0; sb < 2; sb = sb + 1) begin : g_scbank
always_ff @(posedge clk) begin
if (!rst_n) begin
group_acc_b[sb] <= '0; scaled_acc_b[sb] <= '0;
end else if (col_accept && (new_tag == sb[0])) begin
group_acc_b[sb] <= '0;
scaled_acc_b[sb] <= fold_now ? fold_init_i : '0; // G4_EMBFOLD init (0 when EMBFOLD=0)
end else if (csa_out_valid && (tag_now == sb[0])) begin
group_acc_b[sb] <= last_now ? '0 : g_total_w;
scaled_acc_b[sb] <= scaled_next_w;
end
end
end
// --- Outputs ---
// TILE_PIPE=0 : capture on S_DONE (legacy). TILE_PIPE=1 : capture when THIS
// column's own last csa output lands, identified by its tag.
wire cap_ev = TILE_PIPE ? (csa_out_valid && lcol_now) : (state == S_DONE);
always_ff @(posedge clk) begin
if (!rst_n) begin
acc_out <= '0; result_q <= '0; done <= 1'b0;
end else begin
done <= cap_ev;
if (cap_ev) begin
acc_out <= !ACC_DBG ? 32'sd0 : (TILE_PIPE ? acc_next : acc_b[tag_now]);
result_q <= TILE_PIPE ? scaled_next_w : scaled_acc_b[tag_now];
end
end
end
endmodule
`default_nettype wire
```