mlpv_gemv_lane_scaled_stream_dual_ext_opt.sv.md

Source reports/results/GEMM_REVISION_20261117/rtl_snapshot/mlpv_gemv_lane_scaled_stream_dual_ext_opt.sv.md · source md5 a926cebe0eb6e16f0b811b04caf0dd76 · EAGLE4 FPGA archive, 2026-11-18

# Snapshot: verilog/clean7c/rtl/array/mlpv/gemv_lane_scaled_stream_dual_ext_opt.sv

Copied 2026-11-17T17:32:37Z from the clean7c tree (verilog repo commit 5998a7f1, md5 4a89d2e846002fd3f3fdbf4901dab089). Read-only copy for review; the live file is in the RTL repo.

```systemverilog
// =====================================================================================================================
// gemv_lane_scaled_stream_dual_ext_opt   (clean7c rtl/array/mlpv, WP3)
// =====================================================================================================================
// Imported from the shipped packaged copy ref/mlpv_1361/packaged_ip/src/gemv_lane_scaled_stream_dual_ext_opt.sv md5 7ca4843b78210938893d328d0e718b92 (MLPV 0xE4B71361 (M1361, PDI e24deb3f)).
// Code = the shipped code; only this header, the items under 'clean7c edits' and the role include differ.
// Regenerate / verify: tools/wp3_import.py [--check]. Architecture + diagrams: docs/ARRAY.md.
//
// Purpose  : one output column of the MLPV GEMV: dual-read-port scaled lane (2 K-chunks per cycle: even via URAM port A,
//            odd via port B) whose two 16-term dot products are computed OUTSIDE the lane by the array's cross-lane
//            packed-DSP58 unit (DOT_EXT). The lane runs the pair-walk FSM, exports its WLAT-aligned raw weight words
//            (x_qw_e/x_qw_o), the activations (x_a_e, odd masked past nc) and the issue valid x_v, imports the two INT26
//            sums EXT_LAT = 6 cycles later, folds them per GPTQ group (GW_N 26-bit, NARROW 1) and applies one scale multiply
//            per group into the 48-bit result_q. TILE_PIPE 1: two accumulator banks (col_tag) so a new column can start on
//            the terminal pair cycle; MASK_SUM 1: the odd partial-pair mask acts on the 26-b odd sum; ACC_DBG 0: acc_out = 0.
//            (The file header says "_dual_ext": this module is the _opt copy; INVENTORY §5.)
// Users    : u_array/g_grp[*].g_lane[*].g_dual_mo.g_e3.u_lane (256).
// Latency  : WLAT 1 (g_wlat1): qw_word/qw_word_b/scale/issue index re-registered once to meet the +1-deeper seam
//            activation; dot result EXT_LAT after x_v; capture when the column's own last pair result lands (lcol_now).
// Reset    : synchronous active-low. KNOWN HAZARD (docs/ARRAY.md §7): g_wlat1 qw_d/qwb_d/sc_d/ec_d are pure datapath bits in
//            the reset block of civ_d/tp_tag_d -- on M1364 place_design set IS_R_INVERTED 0 on 12 qw_d bits of lane 84
//            (held in reset in operation: x_qw_e bits 0x5F077000 stuck 0). Also reset: the CSA_LAT tag/scale shift registers,
//            the accumulator banks and result_q. Not changed here (lever L-RST-DP).
// Instances in the image: 256 (resolved parameters: rtl/array/SHIPPED_PARAMS.tsv, rows 'mlpv gemv_lane_scaled_stream_dual_ext_opt').
// Ports (generated from the port list; widths as declared, resolved values are in SHIPPED_PARAMS.tsv):
//   input  clk                   1
//   input  rst_n                 1
//   input  start                 1
//   input  a_chunk               signed [K_PAR*A_WIDTH-1:0]
//   input  a_chunk_b             signed [K_PAR*A_WIDTH-1:0]
//   input  qw_word               [31:0]
//   input  qw_word_b             [31:0]
//   input  qz_raw                [1:0]
//   input  scale_q               signed [SCALE_WIDTH-1:0]
//   input  n_chunks_run          [CW-1:0]
//   output chunk_idx_out         [CW-1:0]
//   output chunk_idx_b_out       [CW-1:0]
//   output chunk_idx_valid       1
//   output chunk_idx_next_out    [CW-1:0]
//   output chunk_idx_b_next_out  [CW-1:0]
//   output group_idx_out         [GW-1:0]
//   output group_idx_next_out    [GW-1:0]
//   output acc_out               signed [31:0]
//   output result_q              signed [RES_WIDTH-1:0]
//   output done                  1
//   output start_dropped         1
//   input  fold_i                1
//   input  fold_init_i           signed [RES_WIDTH-1:0]
//   output x_qw_e                [31:0]
//   output x_qw_o                [31:0]
//   output x_a_e                 [K_PAR*A_WIDTH-1:0]
//   output x_a_o                 [K_PAR*A_WIDTH-1:0]
//   output x_v                   1
//   input  x_s_e                 signed [25:0]
//   input  x_s_o                 signed [25:0]
//   input  x_sv                  1
// clean7c edits:
//   - dead branch `g_wlat0` (WLAT 0) removed -> stub (shipped :341-349)
//   - `ifdef RED_NOMASK arm removed (1 lines; no shipped build defines RED_NOMASK), `else text kept verbatim
//   - `ifdef RED_GW arm removed (1 lines; no shipped build defines RED_GW), `else text kept verbatim
//   - default K 64 -> 8192 (every gemv_lane_scaled_stream_dual_ext_opt instance of the image resolves 8192)
//   - default SCALE_FRAC 15 -> 13 (every gemv_lane_scaled_stream_dual_ext_opt instance of the image resolves 13)
//   - default WLAT 1'b0 -> 1'b1 (every gemv_lane_scaled_stream_dual_ext_opt instance of the image resolves 1'b1)
//   - default TILE_PIPE 1'b0 -> 1'b1 (every gemv_lane_scaled_stream_dual_ext_opt instance of the image resolves 1'b1)
//   - default EXT_LAT 7 -> 6 (every gemv_lane_scaled_stream_dual_ext_opt instance of the image resolves 6)
//   - default ACC_DBG 1'b1 -> 1'b0 (every gemv_lane_scaled_stream_dual_ext_opt instance of the image resolves 1'b0)
//   - default MASK_SUM 1'b0 -> 1'b1 (every gemv_lane_scaled_stream_dual_ext_opt instance of the image resolves 1'b1)
//   - default NARROW 1'b0 -> 1'b1 (every gemv_lane_scaled_stream_dual_ext_opt instance of the image resolves 1'b1)
// =====================================================================================================================
`timescale 1ns/1ps
`default_nettype none

(* keep_hierarchy = "yes" *)
module gemv_lane_scaled_stream_dual_ext_opt #(
    parameter int K           = 8192,
    parameter int K_PAR       = 16,
    parameter int A_WIDTH     = 16,
    parameter int GROUP       = 128,      // GPTQ group_size
    parameter int SCALE_WIDTH = 16,       // signed scale
    parameter int SCALE_FRAC  = 13,       // fractional bits (informational)
    parameter int RES_WIDTH   = 48,       // scaled fixed-point accumulator width
    // WLAT (Stage-B' D1 activation-seam fix for the DUAL lane): mirror of the
    // single lane's WLAT. On the SLR_SAFE seam the activation (a_chunk/a_chunk_b)
    // arrives ONE register deeper than the weights (qw_word/qw_word_b), so at
    // WLAT=0 the dual lane computes W[pair]·act[pair-1] on the body GEMV (the D1
    // stale-by-1 class). WLAT=1 delays BOTH weight chunks + the MAC enable + the
    // per-pair issue index (group tag + odd-mask) + scale_q by ONE cycle so the
    // (already +1-deeper) activation and the delayed weights meet. BIT-EXACT: all
    // operands and their tags shift by the SAME 1 cycle; the extra cycle is
    // absorbed by S_DRAIN (CSA_LAT+2 margin). WLAT=0 = legacy (bit+cycle identical
    // to the isolation-proven dual lane).
    parameter bit WLAT        = 1'b1,
    // TILE_PIPE (L1, 2026-08-18) — see gemv_lane_scaled_stream.sv for the full
    // rationale. Identical mechanism on the pair walk: `start` is also accepted
    // on the TERMINAL PAIR cycle, a 1-bit col_tag rides the WLAT delay and the
    // CSA_LAT shift register, and TWO accumulator banks keep the draining
    // column separate from the one already running. At TILE_PIPE=0 col_tag is a
    // constant 0 and every bank-0 register reproduces the legacy behaviour
    // cycle-for-cycle.
    parameter bit TILE_PIPE    = 1'b1,
    // G4_EMBFOLD (2026-09-23, fork g4_embfold_20260923): constant-fold the token-only
    // embedding half of the K reduction. When EMBFOLD=1 and `fold_i` is high on the
    // cycle a column is accepted, the column walk STARTS at chunk FOLD_BASE (not 0) and
    // the 48-bit scaled accumulator starts at `fold_init_i` (not 0). BIT-EXACT by integer
    // associativity: every term is an integer (per-group INT sum x integer scale, 48-bit
    // wrap), so acc = P + sum_{groups >= FOLD_BASE/CPG} == sum_{all groups} exactly when
    // P == sum_{groups < FOLD_BASE/CPG} (mod 2^48). FOLD_BASE must be a multiple of CPG
    // (no group straddles the fold point) and even (pair walk). EMBFOLD=0: identical.
    parameter bit EMBFOLD      = 1'b0,
    parameter int FOLD_BASE    = 0,
    // DOT_EXT lane (HX1 DSP-GEMV, 2026-09-27): the even/odd 16-term dot products
    // are computed OUTSIDE the lane by the array's cross-lane packed-DSP58 unit
    // (e3b_packed_dot: lanes L and L+1 share the activation broadcast, so one
    // DSP58 multiply forms a*(w_L + 2^23*w_{L+1})). The lane EXPORTS its
    // WLAT-aligned raw weight words, the (odd-masked) activations and the issue
    // valid, and IMPORTS the two INT26 sums EXT_LAT cycles later. Everything else
    // (FSM, WLAT, tags, group fold, scale, TILE_PIPE banks, EMBFOLD) is the
    // unmodified gemv_lane_scaled_stream_dual body with CSA_LAT := EXT_LAT.
    parameter int EXT_LAT      = 6,
    // ---- MODOPT array-lane options (defaults = bit/cycle-identical to HEAD) ----
    parameter bit ACC_DBG   = 1'b0,   // 0: drop the dead unscaled debug accumulator (acc_out := 0)
    parameter bit MASK_SUM  = 1'b1,   // 1: odd-chunk partial-pair mask applied to the 26-b odd SUM (delayed) instead of the 256-b activation
    parameter bit NARROW    = 1'b1,   // 1: provably-sufficient group accumulator / scale-multiply width (+ narrow CSA when !EXT)
    parameter int N_CHUNKS    = K / K_PAR,
    parameter int CW          = (N_CHUNKS <= 1) ? 1 : $clog2(N_CHUNKS + 2),
    parameter int CPG         = GROUP / K_PAR,                    // chunks/group (8)
    parameter int N_GROUPS    = (K + GROUP - 1) / GROUP,
    parameter int GW          = (N_GROUPS <= 1) ? 1 : $clog2(N_GROUPS + 1),
    // pair-walk count width (ceil(N_CHUNKS/2))
    parameter int N_PAIRS     = (N_CHUNKS + 1) / 2,
    parameter int PW          = (N_PAIRS <= 1) ? 1 : $clog2(N_PAIRS + 2)
) (
    input  wire                            clk,
    input  wire                            rst_n,
    input  wire                            start,

    // Streaming activation: EVEN chunk (port A) + ODD chunk (port B) per cycle
    input  wire signed [K_PAR*A_WIDTH-1:0] a_chunk,      // even chunk (2p)
    input  wire signed [K_PAR*A_WIDTH-1:0] a_chunk_b,    // odd  chunk (2p+1)
    // Streaming qweight: even word (port A) + odd word (port B)
    input  wire        [31:0]              qw_word,       // even chunk weights
    input  wire        [31:0]              qw_word_b,      // odd  chunk weights
    // Per-group qzero (one 2-bit value); held for the current group
    input  wire        [1:0]               qz_raw,
    // Per-group dequant scale, held for the CURRENT group
    input  wire signed [SCALE_WIDTH-1:0]   scale_q,

    // RUNTIME terminal chunk count for the FSM walk (G12_K_PATH FIX-1 idiom).
    // The lane is ELABORATED for the worst-case K_MAX (N_CHUNKS) but only walks
    // n_chunks_run chunks per start; 0 -> full N_CHUNKS (unconnected-safe).
    input  wire        [CW-1:0]            n_chunks_run,

    // FSM exposes the EVEN chunk index it expects (port A); the ODD index
    // (port B) is +1. Caller drives a_chunk/qw_word off chunk_idx_out and
    // a_chunk_b/qw_word_b off chunk_idx_b_out.
    output wire        [CW-1:0]            chunk_idx_out,     // even (2p)
    output wire        [CW-1:0]            chunk_idx_b_out,   // odd  (2p+1)
    output wire                            chunk_idx_valid,
    // Next-state EVEN/ODD chunk_idx for URAM 1-cycle lookahead (ports A/B).
    output wire        [CW-1:0]            chunk_idx_next_out,
    output wire        [CW-1:0]            chunk_idx_b_next_out,
    // Group index the lane is currently consuming (drives scale-BRAM addr).
    output wire        [GW-1:0]            group_idx_out,
    output wire        [GW-1:0]            group_idx_next_out,

    // Final UNSCALED INT32 accumulator (debug; == single-port lane's acc_out).
    output reg  signed [31:0]              acc_out,
    // Final SCALED fixed-point accumulator (Q with SCALE_FRAC frac bits).
    output reg  signed [RES_WIDTH-1:0]     result_q,
    output reg                             done,
    // F1 (RD_FANOUT_AUDIT L1 follow-up): STICKY "a `start` was presented while
    // this lane could not accept it". Under TILE_PIPE the acceptance window is
    // exactly ONE cycle wide (the terminal chunk); if the 4-register issue chain
    // lands +/-1 cycle off, the start is SILENTLY discarded -> no `done` ever
    // -> the caller waits forever and the tile-tag FIFO desyncs. Nothing else in
    // the design counts that. REPORT-ONLY: it drives an observability bit, it
    // NEVER gates or aborts anything (feedback_gates_report_only_never_kill_builds).
    output reg                             start_dropped,
    // G4_EMBFOLD: sampled on col_accept (same cycle as the accepted `start`).
    input  wire                            fold_i,
    input  wire signed [RES_WIDTH-1:0]     fold_init_i,
    // ---- DOT_EXT export / import ----
    output wire [31:0]                     x_qw_e,   // WLAT-aligned even raw weight word
    output wire [31:0]                     x_qw_o,   // WLAT-aligned odd  raw weight word
    output wire [K_PAR*A_WIDTH-1:0]        x_a_e,    // even activation chunk
    output wire [K_PAR*A_WIDTH-1:0]        x_a_o,    // odd activation chunk, zero-masked past nc
    output wire                            x_v,      // issue valid (== mac_in_valid)
    input  wire signed [25:0]              x_s_e,    // sum_i a_e[i]*w_e[i], EXT_LAT after x_v
    input  wire signed [25:0]              x_s_o,
    input  wire                            x_sv      // valid of x_s_e/x_s_o
);

`ifndef SYNTHESIS
    initial if (CPG[0] != 1'b0)
        $error("gemv_lane_scaled_stream_dual_ext: CPG (GROUP/K_PAR=%0d) must be EVEN for the dual-port pair/group alignment.", CPG);
`endif

    localparam int CSA_LAT  = EXT_LAT;   // DOT_EXT: external dot latency
    localparam int CPG_BITS = (CPG <= 1) ? 1 : $clog2(CPG);

    // Forward declares for the FSM (driven by the two CSA trees below).
    wire signed [25:0] csa_sum_e, csa_sum_o, csa_sum_o_raw;
    wire               csa_out_valid_e, csa_out_valid_o;
    wire               csa_out_valid;     // == csa_out_valid_e (identical timing)
    wire signed [26:0] csa_pair_sum;      // csa_sum_e + csa_sum_o per cycle

    // ----------------------------------------------------------------
    // FSM — pair walk. Mirrors gemv_lane_scaled_stream but the run counter
    // is the PAIR index (advances by 1 pair = 2 chunks per cycle).
    // ----------------------------------------------------------------
    typedef enum logic [1:0] { S_IDLE, S_RUN, S_DRAIN, S_DONE } state_t;
    state_t state, state_n;
    wire state_is_idle = (state == S_IDLE);

    // ----------------------------------------------------------------
    // Effective terminal chunk count, latched at start (same as single-port).
    // ----------------------------------------------------------------
    logic [CW-1:0] nc_run_q;
    wire  [CW-1:0] nc_eff = (nc_run_q == '0) ? CW'(N_CHUNKS) : nc_run_q;
    // number of PAIRS to walk = ceil(nc_eff/2)  (forward-used by col_accept)
    wire [PW-1:0] np_eff = PW'((nc_eff + CW'(1)) >> 1);

    logic [PW-1:0] pair_idx,    pair_idx_n;
    logic [CW-1:0] drain_count, drain_count_n;
    logic [PW-1:0] outs_recvd,  outs_recvd_n;   // # pair-CSA results received

    wire at_terminal_pair = (state == S_RUN) && (pair_idx == np_eff - 1'b1);
    wire col_accept = (state_is_idle && start)
                   || (TILE_PIPE && at_terminal_pair && start);
    always_ff @(posedge clk) begin
        if (!rst_n)          nc_run_q <= '0;
        else if (col_accept) nc_run_q <= n_chunks_run;
    end
    // G4_EMBFOLD: the accepted column is folded (start at FOLD_BASE, acc init = fold_init_i).
    localparam int FOLD_PBASE = FOLD_BASE / 2;
    wire fold_now = EMBFOLD && fold_i;
    reg  fold_q;
    always_ff @(posedge clk) begin
        if (!rst_n)          fold_q <= 1'b0;
        else if (col_accept) fold_q <= fold_now;
    end
    wire [PW-1:0] pair_start = fold_now ? PW'(FOLD_PBASE) : '0;
    // pair-CSA results this column delivers (the drain-exit count)
    wire [PW-1:0] np_run     = (EMBFOLD && fold_q) ? (np_eff - PW'(FOLD_PBASE)) : np_eff;
`ifndef SYNTHESIS
    initial if (EMBFOLD && ((FOLD_BASE % CPG) != 0 || (FOLD_BASE % 2) != 0 || FOLD_BASE <= 0))
        $error("gemv_lane_scaled_stream_dual_ext: EMBFOLD needs FOLD_BASE (%0d) > 0, even and a multiple of CPG (%0d) -- a group must not straddle the fold point.", FOLD_BASE, CPG);
`endif
    // F1: sticky dropped-start observability (see the port comment).
    always_ff @(posedge clk) begin
        if (!rst_n)                                         start_dropped <= 1'b0;
        else if (start && !col_accept && (state != S_IDLE)) start_dropped <= 1'b1;
    end
`ifndef SYNTHESIS
    // F4 (audit): ANNOTATED LIMIT — N_PAIRS is the ELABORATED worst case, not
    // the runtime per-stage count; sizing sanity only, the runtime arm is below.
    initial if (TILE_PIPE && (N_PAIRS <= CSA_LAT + 2))
        $error("gemv_lane_scaled_stream_dual_ext: TILE_PIPE=1 needs N_PAIRS > CSA_LAT+2 (got %0d).", N_PAIRS);
    // F5 (audit): police the INCOMING count (in PAIRS), and include WLAT. See
    // the single-port lane for the full rationale. xsim exits 0 on $fatal.
    always_ff @(posedge clk)
        if (rst_n && TILE_PIPE && col_accept && (n_chunks_run != '0) &&
            (((n_chunks_run + CW'(1)) >> 1) <= CW'(CSA_LAT + (WLAT?1:0))))
            $fatal(1, "gemv_lane_scaled_stream_dual_ext: TILE_PIPE column too short (n_chunks_run=%0d pairs=%0d <= CSA_LAT+WLAT=%0d).",
                   n_chunks_run, (n_chunks_run+CW'(1))>>1, CSA_LAT + (WLAT?1:0));
`endif

    always_ff @(posedge clk) begin
        if (!rst_n) begin
            state <= S_IDLE; pair_idx <= '0; drain_count <= '0; outs_recvd <= '0;
        end else begin
            state <= state_n; pair_idx <= pair_idx_n;
            drain_count <= drain_count_n; outs_recvd <= outs_recvd_n;
        end
    end

    always_comb begin
        state_n = state; pair_idx_n = pair_idx;
        drain_count_n = drain_count; outs_recvd_n = outs_recvd;
        case (state)
            S_IDLE: if (start) begin
                state_n = S_RUN; pair_idx_n = pair_start;
                drain_count_n = '0; outs_recvd_n = '0;
            end
            S_RUN: begin
                if (pair_idx == np_eff - 1'b1) begin
                    // TILE_PIPE: accept the next column's start on the terminal
                    // pair cycle -> zero idle cycles between columns.
                    if (TILE_PIPE && start) begin
                        state_n = S_RUN; pair_idx_n = pair_start;
                        drain_count_n = '0; outs_recvd_n = '0;
                    end else begin
                        state_n = S_DRAIN; drain_count_n = '0;
                    end
                end else pair_idx_n = pair_idx + 1'b1;
                if (csa_out_valid) outs_recvd_n = outs_recvd + 1'b1;
            end
            S_DRAIN: begin
                if (csa_out_valid) outs_recvd_n = outs_recvd + 1'b1;
                if (outs_recvd == np_run) state_n = S_DONE;
                else if (drain_count >= CSA_LAT + 2) state_n = S_DONE;
                drain_count_n = drain_count + 1'b1;
            end
            S_DONE: state_n = S_IDLE;
            default: state_n = S_IDLE;
        endcase
    end

    // EVEN chunk = 2*pair_idx ; ODD chunk = 2*pair_idx + 1
    wire [CW-1:0] even_cur  = CW'(pair_idx)      << 1;
    wire [CW-1:0] even_next = CW'(pair_idx_n)    << 1;
    assign chunk_idx_out        = even_cur;
    assign chunk_idx_b_out      = even_cur  + CW'(1);
    assign chunk_idx_next_out   = even_next;
    assign chunk_idx_b_next_out = even_next + CW'(1);
    assign chunk_idx_valid      = (state == S_RUN);

    // Group index = even_chunk / CPG (== odd_chunk/CPG for even CPG). Current +
    // next for the scale-BRAM 1-cycle lookahead.
    wire [CW-1:0] grp_cur_full  = (CPG <= 1) ? even_cur  : (even_cur  >> CPG_BITS);
    wire [CW-1:0] grp_next_full = (CPG <= 1) ? even_next : (even_next >> CPG_BITS);
    assign group_idx_out      = grp_cur_full[GW-1:0];
    assign group_idx_next_out = grp_next_full[GW-1:0];

    // ----------------------------------------------------------------
    // WLAT (D1 dual-lane seam fix): delay BOTH weight chunks + MAC enable +
    // the per-pair issue index (group tag + odd-mask) + scale_q by ONE cycle
    // to meet the +1-deeper SLR-seam activation. Pass-through at WLAT=0
    // (bit+cycle identical to the isolation-proven dual lane). chunk_idx_out /
    // group_idx_out stay UNDELAYED (the caller's fetch index; the seam adds the
    // register externally) — same idiom as the single lane.
    // ----------------------------------------------------------------
    wire [31:0]                   qw_word_eff, qw_word_b_eff;
    wire                          mac_in_valid;
    wire [CW-1:0]                 even_issue;
    wire signed [SCALE_WIDTH-1:0] scale_q_eff;
    // TILE_PIPE column tag (see the single-port lane). Inert at TILE_PIPE=0.
    reg  col_tag;
    always_ff @(posedge clk) begin
        if (!rst_n)                       col_tag <= 1'b0;
        else if (TILE_PIPE && col_accept) col_tag <= ~col_tag;
    end
    wire tag_pre     = col_tag;
    wire lastcol_pre = (pair_idx == np_eff - 1'b1);
    wire tag_issue;
    wire lastcol_issue;
    generate
    if (!WLAT) begin : g_wlat0
        // clean7c WP3: branch `g_wlat0` (WLAT 0) is elaborated by NO instance of MLPV 0xE4B71361 (M1361, PDI e24deb3f) and was removed
        // (shipped text: ref/mlpv_1361/packaged_ip/src/gemv_lane_scaled_stream_dual_ext_opt.sv:341-349). Selecting it is an elaboration
        // error, never silently different hardware; restoring it is a lever (PLAN.md §6), not a cleanup.
        C7_REMOVED_BRANCH__gemv_lane_scaled_stream_dual_ext_opt__g_wlat0 u_c7_removed ();
    end else begin : g_wlat1
        reg [31:0]                   qw_d, qwb_d;
        reg                          civ_d;
        reg [CW-1:0]                 ec_d;
        reg signed [SCALE_WIDTH-1:0] sc_d;
        reg                          tp_tag_d, tp_lcol_d;
        always_ff @(posedge clk) begin
            if (!rst_n) begin
                qw_d <= '0; qwb_d <= '0; civ_d <= 1'b0; ec_d <= '0; sc_d <= '0;
                tp_tag_d <= 1'b0; tp_lcol_d <= 1'b0;
            end else begin
                qw_d  <= qw_word;  qwb_d <= qw_word_b; civ_d <= chunk_idx_valid;
                ec_d  <= even_cur; sc_d  <= scale_q;
                tp_tag_d <= tag_pre; tp_lcol_d <= lastcol_pre;
            end
        end
        assign qw_word_eff   = qw_d;
        assign qw_word_b_eff = qwb_d;
        assign mac_in_valid  = civ_d;
        assign even_issue    = ec_d;
        assign scale_q_eff   = sc_d;
        assign tag_issue     = tp_tag_d;
        assign lastcol_issue = tp_lcol_d;
    end
    endgenerate

    // ----------------------------------------------------------------
    // ODD-chunk validity mask: when 2p+1 >= nc_eff (odd nc final pair), the
    // odd MAC must contribute 0. Force a_odd = 0 so csa_o = 0 exactly.
    // (Uses even_issue = the WLAT-aligned index.)
    // ----------------------------------------------------------------
    wire odd_valid_issue = ((even_issue + CW'(1)) < nc_eff);
    wire signed [K_PAR*A_WIDTH-1:0] a_odd_eff =
        (odd_valid_issue || MASK_SUM) ? a_chunk_b : '0;

    // ----------------------------------------------------------------
    // DOT_EXT: dequant + 2 x 16 MAC + 2 Wallace trees are REPLACED by the
    // array-level packed-DSP unit. Export the exact operands the in-lane
    // datapath would have consumed (same WLAT-aligned words, same odd mask).
    // qz_raw is broadcast per group, so the unit takes it from the array.
    // ----------------------------------------------------------------
    assign x_qw_e = qw_word_eff;
    assign x_qw_o = qw_word_b_eff;
    assign x_a_e  = a_chunk;
    assign x_a_o  = a_odd_eff;
    assign x_v    = mac_in_valid;
    assign csa_sum_e       = x_s_e;
    assign csa_sum_o_raw   = x_s_o;
    assign csa_out_valid_e = x_sv;
    assign csa_out_valid_o = x_sv;
    assign csa_out_valid   = csa_out_valid_e;

    // ----------------------------------------------------------------
    // UNSCALED INT32 accumulator (debug; matches single-port acc_out exactly).
    // Per cycle accumulate (csa_e + csa_o). INT26 + INT26 = up to INT27.
    // ----------------------------------------------------------------
    // MASK_SUM: odd_valid_issue delayed CSA_LAT deep (aligned with the dot result) gates the 26-b odd sum.
    logic ov_sr [0:CSA_LAT-1];
    integer ovs;
    always_ff @(posedge clk) begin
        if (!rst_n) begin
            for (ovs = 0; ovs < CSA_LAT; ovs = ovs + 1) ov_sr[ovs] <= 1'b1;
        end else begin
            ov_sr[0] <= odd_valid_issue;
            for (ovs = 1; ovs < CSA_LAT; ovs = ovs + 1) ov_sr[ovs] <= ov_sr[ovs-1];
        end
    end
    assign csa_sum_o = (MASK_SUM && !ov_sr[CSA_LAT-1]) ? 26'sd0 : csa_sum_o_raw;
    assign csa_pair_sum = $signed(csa_sum_e) + $signed(csa_sum_o);
    // TILE_PIPE column-tag / last-pair-of-column shift register, CSA_LAT deep.
    logic tag_sr  [0:CSA_LAT-1];
    logic lcol_sr [0:CSA_LAT-1];
    integer ts;
    always_ff @(posedge clk) begin
        if (!rst_n) begin
            for (ts = 0; ts < CSA_LAT; ts = ts + 1) begin
                tag_sr[ts] <= 1'b0; lcol_sr[ts] <= 1'b0;
            end
        end else begin
            tag_sr[0]  <= tag_issue;
            lcol_sr[0] <= lastcol_issue && mac_in_valid;
            for (ts = 1; ts < CSA_LAT; ts = ts + 1) begin
                tag_sr[ts] <= tag_sr[ts-1]; lcol_sr[ts] <= lcol_sr[ts-1];
            end
        end
    end
    wire tag_now  = tag_sr[CSA_LAT-1];
    wire lcol_now = lcol_sr[CSA_LAT-1];
    wire new_tag  = TILE_PIPE ? ~col_tag : 1'b0;

    logic signed [31:0] acc_b [0:1];
    wire signed [31:0] acc_next = acc_b[tag_now] + $signed(csa_pair_sum);
    genvar ab;
    for (ab = 0; ab < 2; ab = ab + 1) begin : g_accbank
        always_ff @(posedge clk) begin
            if (!rst_n)                                   acc_b[ab] <= '0;
            else if (col_accept    && (new_tag == ab[0])) acc_b[ab] <= '0;
            else if (csa_out_valid && (tag_now == ab[0])) acc_b[ab] <= acc_next;
        end
    end

    // ================================================================
    // SCALED PATH — per-group fold then one DSP scale.
    // ================================================================
    // Tag each PAIR at issue with its group, last-in-group, and the scale the
    // caller drove for that group; shift CSA_LAT deep to line up with the
    // pair-CSA result. Identical idiom to gemv_lane_scaled_stream.
    wire [CW-1:0] grp_issue_full = (CPG <= 1) ? even_issue : (even_issue >> CPG_BITS);
    wire [GW-1:0] grp_issue      = grp_issue_full[GW-1:0];

    // group's last VALID chunk = min((grp+1)*CPG-1, nc_eff-1).
    wire [CW-1:0] grp_end_full = ((grp_issue_full + CW'(1)) << CPG_BITS) - CW'(1);
    wire [CW-1:0] grp_last     = (grp_end_full < (nc_eff - CW'(1)))
                                 ? grp_end_full : (nc_eff - CW'(1));
    // does THIS pair contain grp_last? odd member if odd_valid, else even member.
    wire last_in_grp_issue = odd_valid_issue ? ((even_issue + CW'(1)) == grp_last)
                                             : (even_issue == grp_last);
    wire issue_valid = mac_in_valid;

    logic [GW-1:0]                 grp_sr  [0:CSA_LAT-1];
    logic                         last_sr [0:CSA_LAT-1];
    logic                         vld_sr  [0:CSA_LAT-1];
    logic signed [SCALE_WIDTH-1:0] sc_sr  [0:CSA_LAT-1];
    integer s;
    always_ff @(posedge clk) begin
        if (!rst_n) begin
            for (s = 0; s < CSA_LAT; s = s + 1) begin
                grp_sr[s] <= '0; last_sr[s] <= 1'b0; vld_sr[s] <= 1'b0; sc_sr[s] <= '0;
            end
        end else begin
            grp_sr[0]  <= grp_issue;
            last_sr[0] <= last_in_grp_issue;
            vld_sr[0]  <= issue_valid;
            sc_sr[0]   <= scale_q_eff;
            for (s = 1; s < CSA_LAT; s = s + 1) begin
                grp_sr[s] <= grp_sr[s-1]; last_sr[s] <= last_sr[s-1];
                vld_sr[s] <= vld_sr[s-1]; sc_sr[s]   <= sc_sr[s-1];
            end
        end
    end
    wire                          last_now  = last_sr[CSA_LAT-1];
    wire signed [SCALE_WIDTH-1:0] scale_now = sc_sr[CSA_LAT-1];

    // Per-group INT accumulator (fold pair sums), flush*scale at group end.
    // NARROW: |group sum| <= GROUP * 2^(A_WIDTH+1) (|a|<=2^(A_WIDTH-1), |w|<=4) = 2^(A_WIDTH+2+log2 GROUP-?);
    // signed width A_WIDTH+3+clog2(GROUP) (=26 at A=16,GROUP=128). Arithmetic is modulo-2^GW_N and the true value
    // fits, so the sign-extension to 32 is exact.
    localparam int GW_N = NARROW ? (A_WIDTH + 3 + $clog2(GROUP)) : 32;
    logic signed [GW_N-1:0]      group_acc_b  [0:1];
    logic signed [RES_WIDTH-1:0] scaled_acc_b [0:1];
    wire signed [GW_N-1:0] g_total_w = group_acc_b[tag_now] + GW_N'($signed(csa_pair_sum));
    wire signed [RES_WIDTH-1:0] scaled_next_w =
         last_now ? (scaled_acc_b[tag_now] +
                     ($signed(g_total_w) * $signed(scale_now)))
                  : scaled_acc_b[tag_now];
    genvar sb;
    for (sb = 0; sb < 2; sb = sb + 1) begin : g_scbank
        always_ff @(posedge clk) begin
            if (!rst_n) begin
                group_acc_b[sb]  <= '0; scaled_acc_b[sb] <= '0;
            end else if (col_accept && (new_tag == sb[0])) begin
                group_acc_b[sb]  <= '0;
                scaled_acc_b[sb] <= fold_now ? fold_init_i : '0;   // G4_EMBFOLD init (0 when EMBFOLD=0)
            end else if (csa_out_valid && (tag_now == sb[0])) begin
                group_acc_b[sb]  <= last_now ? '0 : g_total_w;
                scaled_acc_b[sb] <= scaled_next_w;
            end
        end
    end

    // --- Outputs ---
    // TILE_PIPE=0 : capture on S_DONE (legacy). TILE_PIPE=1 : capture when THIS
    // column's own last csa output lands, identified by its tag.
    wire cap_ev = TILE_PIPE ? (csa_out_valid && lcol_now) : (state == S_DONE);
    always_ff @(posedge clk) begin
        if (!rst_n) begin
            acc_out  <= '0; result_q <= '0; done <= 1'b0;
        end else begin
            done <= cap_ev;
            if (cap_ev) begin
                acc_out  <= !ACC_DBG ? 32'sd0 : (TILE_PIPE ? acc_next : acc_b[tag_now]);
                result_q <= TILE_PIPE ? scaled_next_w : scaled_acc_b[tag_now];
            end
        end
    end

endmodule
`default_nettype wire
```