fpga-archive-reports-qwen · EAGLE4 FPGA archive

Source reports/results/QWEN_GPTQ2_20261116/D7_HOSTGAP_20261117.md · source md5 1db2d31ef9a660382b508f40a3f194e4 · archived 2026-11-18

Download PDFOpen PDFRaw markdown

D7 W2 host gap: where the 14.8 ms per round goes, and the next host levers (qgptq2-d7-hostgap, 2026-11-17)

Program: the 7-card 4C1 drafter for the Qwen3.8-27B target (Q5120 GPTQ-2 EAGLE-4 drafter, A16). ATT 0xE4B7139C TP4 on 17/3d/bd/e1, MLPV 0xE4B7139D TP2 on 50/63, served by SGLang EAGLE_HARNESS through the c4q5120 shim and bridge. Motto: a research track never ends by denying the idea; it engineers toward the fastest correct result and claims nothing it has not measured.

Rules kept:

Tags:

Evidence:


0. Answer first

  1. The d7 W2 gap is the card RPC.
    • Arm d (full + KI, window 2) decodes at 34.94 ms/round. CARD=none decodes at 20.13 ms/round. Gap: 14.81 ms/round.
    • Of that, 13.25 ms is the card "round" RPC: in-RPC KV-only rows + the tree + IPC.
    • 0.96 ms is the bridge's root/fc/row build.
    • ~0.48 ms is the tree copy-in.
    • The GPU and scheduler parts are equal in both modes. They are measured with the new TIMING instrument on box1, on the same harness.
  2. What is left on the host side is small. Measured or bounded cuts per round:
    • ROWVEC: -0.40 to -0.57 [M box1];
    • TREEPIN: up to -0.60 [H, silicon-derived bound];
    • EARLY: -0.30 to -0.40 [M box1 scheduler gap];
    • PREPOISON: about -0.1 [H];
    • H2CPAR: not bounded until window 3 measures the silicon H2C share.
    • Stacked [H]: about -1.4 to -1.7 ms, plus H2CPAR. That is 34.9 -> about 33.3 ms/round, about 116 tok/s, against 110.48.
  3. The ceiling is structural.
    • The card is idle for a median 20.2 ms per round while the GPU verifies, and the GPU is idle for the 13.25 ms card RPC.
    • No exact host scheme moves card work into the verify window: the root row and the KV rows are verify outputs (section 6).
    • The candidates above 3 ms change the trees, so they are user accuracy rulings.
  4. Everything is implemented behind default-off EAGLE4_Q5120_* toggles and mock-proven.
    • The toggles: TIMING, ROWVEC, TREEPIN, PREPOISON, EARLY, H2CPAR.
    • Results: --same vs base 8/8, counters engaged, a RED per lever, and unit tests 73/73 on both 139C+139D and 13A9+13A1 (section 5).
    • The default path is unchanged: the _hg_def arm is --same against the v2 base.
  5. New finding: box1 mocks ran at a different loop mode from silicon.
    • Every box1 mock (levers-v2 and this chain) ran with loop modes EAGER + EARLY_A (0x30). c4_levers.loop_mode_bits() defaults to the record values when the env is unset.
    • The silicon windows force 0.
    • _hg_max2 reruns the full lever set at mode 0.

1. Measured decomposition, d7 W2 V20 K_C 64

1.1 Silicon, window 2 [M]

Sources: d7_hostgap_evidence/silicon_w2/, from talon2 runs/qgwin_20261117b/serve/.

arm (levers) decode ms/round t_card ms/round t_row (incl fc) t_fc delta decode vs previous
a (off) 42.34 16.94 0.96 0.23 -
b (KVBATCH + PREROOT) 40.98 15.66 1.00 0.24 -1.36
c (+ STATS_EVERY 50) 40.07 16.19 0.95 0.22 -0.91
c2 (+ SKIP_GPU_TREE) 38.47 16.42 0.94 0.23 -1.60
d (+ KVINTERLEAVE) 34.94 13.25 0.96 0.23 -3.53 (t_card -3.17)
CARD=none 20.13 - - - -

1.2 GPU / scheduler / bridge parts: TIMING on box1 [M]

part median ms in GPU-only too?
draft() exit -> prelaunch (verify launch, host) 1.65 yes
prelaunch (extend queued) 0.14 yes
prelaunch -> verify + draft-extend fence 17.12 yes
(verify GPU, CUDA events) (18.38) yes
(draft-extend GPU, side stream) (0.42) yes; on the critical path, but in silicon mode only its static buffers are used
fence done -> next draft() (scheduler) 0.30 yes
draft() entry -> _grow 0.04 yes
_grow -> rows built (root + eager fc + FASTIN rows) 1.14 (mean 1.33; silicon t_row 0.96) no
card RPC: IPC + pickle (RPC - server busy) 0.35 no (inside t_card)
reply -> tree injected (SoA -> tensors, copy_) 0.48 no
injected -> draft() exit (rank, verify input) 0.25 (mean 0.44: the L3 dump every 50 trees costs 6-13 ms) yes
card server idle before each request 20.24 (the verify window)
card server after-work (PREROOT row) 1.26 inside the idle window

Silicon round model. Box1 GPU/host parts plus talon2 t_card / t_row:

1.65 + 0.14 + 17.12 + 0.30 + 0.04 + 0.96 + 13.25 + 0.48 + 0.25 = 34.19 ms, against 34.94 measured. The residual 0.75 is the mean-versus-median difference and the dumps.

Gap. silicon - GPU-only = 0.96 + 13.25 + 0.48 + ~0.1 = 14.8 ms.

2. Ranked levers (per round, d7 W2)

rank lever toggle cut ms/round basis
1 Inside the card RPC: KV rows 4.75-6.35, tree 6.9-8.5 (window-3 TIMING measures the split; then the levers below) - bounded, section 1.1
2 ROWVEC: one embed gather + one q16 per half for all of the round's rows (default: 2 q16 GPU->host copies per row) EAGLE4_Q5120_ROWVEC=1 -0.40 (median) / -0.57 (mean) [M box1] _hg_tim 1.141 / 1.332 -> _hg_max 0.738 / 0.761
3 TREEPIN: the tree's root FASTIN row interleaved over the 4 ATT ranks (the same program_inputs_interleaved KI runs per KV row) EAGLE4_Q5120_TREEPIN=1 (needs KVBATCH) <= 0.60 [H] = the measured KI saving per program_inputs call on silicon (0.60 ms/row, section 1.1); one call per tree
4 EARLY: the next round's "round" RPC is sent from DynamicDrafter.prelaunch once the verify + extend fence is done; the card overlaps the scheduler's post-verify work EAGLE4_Q5120_EARLY=1 (needs KVBATCH) -0.30 to -0.40 [M box1] scheduler 0.30 + pre-draft 0.04 median; the default sends at fence + 0.30 + 0.04 + rows, EARLY at fence + rows
5 H2CPAR (mock: +0.37 ms of thread overhead per round; run on silicon only if the H2C share is above about 0.3 ms): each KV-only row's 4 FASTIN H2Cs (20,480 B each, one QDMA char device per card) in parallel threads, then the identical boot-DMA program EAGLE4_Q5120_H2CPAR=1 (needs KVBATCH + KVINTERLEAVE) <= 3/4 x (4 x H2C) per in-RPC row x 5.25 [H]: the mock H2C (22 us/row) is not silicon; window-3 TIMING reports h2c_ms
6 PREPOISON: the MLPV SoA poison (4 small H2Cs) moved to the idle window after the reply EAGLE4_Q5120_PREPOISON=1 (needs KVBATCH) ~-0.1 [H]; mock 0.016 ms
7 tree copy-in: one pinned tensor + non_blocking copies (not implemented) <= 0.48 [M box1]
8 IPC / pickle of the rows (not implemented) <= 0.35 [M box1]
9 skip the draft-extend forward in silicon mode (keep its buffer staging) (not implemented: graph-captured; changes the GPU fallback tree's draft KV) <= 0.42 [M box1] CUDA events
10 card loop modes EAGER + EARLY_A env only + a golden in that mode -0.5 to -1.4 [H] D7_TREE_LEVERS §7.4; section 6.1
11 cfg diff + loop-register pre-write in the idle window (not implemented) <= 0.2 [H] section 6.2 c3

The stack of 2 + 3 + 4 + 6 [H] is about -1.4 to -1.7 ms. That is 34.9 -> about 33.3 ms/round, i.e. about 4.06 / 33.3 ms = 122 tok/s on the decode-only basis. With TTFT included, about 116 tok/s.

3. What was built (code, default off)

Files:

c4_loop.py / c4_card.py are not touched, so the record path is unaffected: RECORD_PROFILE_IDENTITY passes (section 5).

Toggle rules:

toggle bridge card server engagement counter (report: LEVER NOT ENGAGED if named and 0)
EAGLE4_Q5120_TIMING=1 per-round stamps d0 / g0 / fr / r0 / r1 / end / d1 / p0 / p1 / ready + CUDA events (verify GPU, draft extend) in STATS["timing"]["rounds"] (_tm*, _patch_prelaunch) round opt timing -> reply srv_t: kv_ms, kv_prog_ms, kv_wait_ms, h2c_ms, tree_ms, tree_wait_ms, busy_ms, idle_ms, after_prev_ms (preroot / prepoison) timing rounds > 0; report prints the per-round table (qgserve_report.py --timing-only)
EAGLE4_Q5120_ROWVEC=1 _rows_vec: index_select + one q16 per half for all of the round's rows (prefill C rows, extend last+1 rows) - hostgap.rowvec_rows
EAGLE4_Q5120_TREEPIN=1 round opt tree_pin ServeSession._tree_levers: c4_loop's own EAGLE4_4C1_PINPAR branch for THIS run_tree only (env restored; never on the DW route) srv_l1.tree_pinpar
EAGLE4_Q5120_PREPOISON=1 round opt prepoison ServeSession.prepoison_after() after the reply and the preroot row (c4_loop.Session.prepoison; run_tree skips its own poison on a key match) srv_l1.prepoison_used (c4_loop's counter)
EAGLE4_Q5120_EARLY=1 _early_launch in the prelaunch hook:<br>- waits on the prelaunch fence, then builds the SAME rows / root the extend branch would build;<br>- CardClient.send_async;<br>- _inject_qg consumes the reply with take() after checking rid / last / positions / tokens / md / W / prev-root against the live extend buffers (mismatch -> drain + card_error_early, request dead);<br>- rpc() drains an unconsumed early reply first (request end), counting early_discard_err - (FIFO, unchanged) hostgap.early_used; FAIL on early_discard_err / early_mismatch (EARLY ERROR)
EAGLE4_Q5120_H2CPAR=1 round opt h2cpar ServeSession._program_h2cpar: a 4-thread pool does the 4 H2Cs (joined, errors raised), then program_inputs_interleaved on proxies whose h2c is a no-op = the same boot-DMA writes / polls / order. In-RPC rows and the preroot row srv_l1.h2cpar_rows

Correctness arguments.

4. Window-2 observations kept for the record [M]

5. Mock E2E (box1 H200 + real Qwen3.8-27B target, CARD=mock World7, humaneval:4 + mt_bench:4, d7 W2 V20 K_C 64) [M]

Code under test. The final code is commit 913a2690c (bridge md5 f6e243a3, cardsrv 2bc28789, q5120_serve 1f6c687f, report 52e00a45, runner 2300c804).

arm env (beyond the runner's) ref result lever counters md5 (jsonl)
_hg_smoke 1 req x 48 tok, ALL + HG - CARD_RESULT PASS; ROWS MATCH tree_pinpar 6, prepoison_used 5, early_used 5, rowvec_rows 112 6d2e193d
_hg_tim ALL + TIMING v2_base SAME 8/8, PASS, accept 4.0565 364 timing records d9bc61ed
_hg_max ALL + TIMING ROWVEC TREEPIN PREPOISON EARLY v2_base SAME 8/8, PASS, ROWS MATCH 8/8 tree_pinpar 379, prepoison_used 378 / written 378, early_sent 372 / used 364 / discarded 7 / discard_err 0 / mismatch 0, rowvec_rows 844 c7284ef2
_hg_cwmax CTX_WIN 200 + ALL + HG (requests die at the prime and mid-request) v2_cwbase SAME 8/8, PASS early_used 111, early_not_sent {cap 5, request 256}; l2 reruns 8 (prelaunch 5 / draft 3); tree_pinpar 116, prepoison_used 115 2c417178
_hg_def none (default path) v2_base SAME 8/8, PASS. STATS keys == v2_base's (no levers / hostgap / timing) - 67cb1878
_hg_s1ref 1 req x 48 tok, ALL - PASS (the REDs' reference) - 7f9cdc1c
_hg_redtp RED s1 + ALL + TREEPIN, RED treepin_drop s1ref FAIL as required: mock exact fail 47 + SAME-CONFIG MISMATCH - b9dde94b
_hg_redearly RED s1 + ALL + EARLY + TIMING, RED early_prev s1ref FAIL as required: EARLY ERROR (mismatch 1, discard_err 1) + CARD ERROR card_error_early (server contiguity tooth tree root ctx 16 != 113) early_sent 1 9e5ac626
_hg_redrv RED (v1) s1 + ALL + ROWVEC, RED rowvec_flip (one extend row's h half zeroed) s1ref PASSED: NOT caught red_fired 5 ce43945e
_hg_redrv2 RED (v2) as v1, RED now also hits prefill row 1 + the extend root row s1ref FAIL as required: ROWS mismatch rows_req00_C112.bin. --same stayed 1/1: the root-row corruption did not move the 6-round verify count, so the ROWS tooth is the one that catches it red_fired 6, rowvec_rows 152 c710f4f6
_hg_redh2c RED s1 + ALL + H2CPAR, RED h2cpar_drop s1ref FAIL as required: mock exact fail 47 + SAME-CONFIG MISMATCH h2cpar_rows 157 10481cd4
_hg_max2 ALL + HG + H2CPAR at loop mode 0 (= silicon: EAGLE4_PAIR_EAGER/EARLY_A/SPEC0=0, confirmed in the card server's /proc environ, mock_chain/max2_cardsrv_env.txt) v2_base SAME 8/8, PASS, ROWS MATCH 8/8, accept 4.0565 h2cpar_rows 2,363, tree_pinpar 379, prepoison_used 378, early_used 364 / discarded 7 / discard_err 0, rowvec_rows 844 31e20c9b

H2CPAR costs time on the mock. The thread pool adds work there: rows program median 3.29 ms (_hg_max) -> 3.79 ms (_hg_max2), and H2C 0.02 -> 0.39 ms per round, because a mock H2C is an in-process memcpy.

The v1 rowvec RED is a gate-shape lesson.

Unit tests (scripts/4c1/tests/test_q5120_serve.py, CPU, the live bit-true mock): 73/73 for 139C+139D (155 s) and 73/73 for 13A9+13A1. Logs are in d7_hostgap_evidence/regress/. New arms:

Regression, all PASS:

6. Structural lever: card work inside the verify window (coordinator request, 2026-11-17)

Measured ceiling. The card server sits idle while the GPU verifies: the median idle gap before each round request is 20.2 ms ([M] box1 TIMING, server idle before this request, 364 rounds; the verify GPU time is 18.4 ms). Meanwhile the GPU sits idle for the 13.25 ms card RPC. Overlapping the two would take the round from 34.9 ms toward about 21 ms. The question is whether any EXACT scheme (same trees, --same 8/8) can move card work into that window.

6.1 The record overlap toggles (scripts/4c1/c4_levers.py:40-42,66-70,99-132; RTL loop_seq_d2.sv:1-31)

toggle what it does which window it overlaps reuse on Q5120
EAGLE4_PAIR_EAGER (LOOP_CTL[4]) The TAIL's loop_seq issues a level's child ITEMs as soon as each survivor rank is proven (_eager_poll on 0x1D0), instead of waiting for the whole level to COMMIT inside ONE tree: the levels of the card loop pipeline ATT -> MLPV Env-only. No bridge code is needed. ServeSession already takes LV.loop_mode_bits() (q5120_serve.py:405-406), and open_silicon refuses a golden whose mode_bits differ (q5120_serve.py:370-372). So a window runs the golden AND the serve with EAGLE4_PAIR_EAGER=1 EAGLE4_PAIR_EARLY_A=1. 0x139D carries EAGER_TPSAFE (c4_tp4.py:565).
EAGLE4_PAIR_EARLY_A (LOOP_CTL[5], needs EAGER) A child may start when its parent's a export is counted early (0x1D0[7:4]), not only after the parent's full item (0x184) inside one tree (per level) As above
EAGLE4_PAIR_SPEC0 (LOOP_CTL[6], needs EAGER) Speculative child 0 is sent before COMMIT; CANCEL retires it on a miss inside one tree Not on 4C1. It is rank-asymmetric (c4_levers.py:66-70, FOURCARD_DEFAULT 0: TP-SUM wedge RCA §10)
EAGLE4_PAIR_TREE_STOP A mass-rule early stop after level L (host-assisted inside the guard band) shortens a tree Changes the tree (fewer levels): this is an accept trade, not --same. A USER ruling

None of the four moves card work into the GPU verify window. They shorten the card tree itself.

Finding [M]. c4_levers.loop_mode_bits() defaults to the record values. In a clean env it returns (48, ['eager', 'early_a']), i.e. 0x30, so every box1 mock arm ran EAGER + EARLY_A:

The silicon windows force all modes to 0 (serveW2_remote.sh: EAGLE4_PAIR_EAGER=0 EAGLE4_PAIR_EARLY_A=0 EAGLE4_PAIR_SPEC0=0 EAGLE4_PAIR_TREE_STOP=0).

Consequences:

6.2 Candidates

candidate exact? bound per round why
(a) build round r+1's tree during verify r from a predicted accepted path no: hit rate 0 by construction 0 ms exact The root FASTIN row of round r+1 is `embed[bonus] Q4.12
(a') same, with the drafter's own hidden / top child as a stand-in for the target's no (different trees) up to 13.25 + 0.96 ms hidden, if the accept held up The output ids stay exact (the target verifies, greedy), but the trees, and therefore the accept, change. That is an accuracy trade: a USER ruling. Upper bound on any single-path prediction [M, silicon arm d trees_log ctx deltas, 364 rounds]: accept length 8 (full top-depth path) in 21.7 % of rounds; histogram 1:64 2:71 3:58 4:35 5:22 6:27 7:8 8:79. Not implemented.
(b) KV-only rows for the predicted accepted tokens during verify no 0 ms exact Row j = `embed[tok_j]
(c1) defer the accepted rows: draft round r+1's tree on the K/V the card already holds for those nodes (the tree's own lane-chain slots, drafter-hidden K/V), and recompute the exact rows inside the verify window no (the tree's prefix K/V differs from SGLang's draft extend) 4.75-6.35 ms (the bounded in-RPC row share, §1) > 3 ms, but an accuracy trade plus RTL/host KV-slot work: USER ruling. Not implemented.
(c2) batch 2 ping-pong (request B's card round during request A's verify) yes per stream aggregate only The bridge refuses bs != 1 (c4q5120_bridge.py:907 bs!=1). This is the preserved "free batch 2" (single-stream policy). Not a bs-1 lever.
(c3) pre-issue tree work in the idle window: cfg diff (K_F / MAX_DEPTH / KC / N_TILES / OUT bases / TREE_WIDTH are constant per serve; only CTX_LEN changes) + loop-register pre-write (all but ROOT / CTL) yes <= 0.2 ms [H] About 30 + 70 MMIO writes and 4 reads per tree (c4_loop.py:338-363, 498-531). Below the 3 ms bar; listed for a later lever pack.
(c4) PREPOISON (done) / EARLY (done) yes 0.3-0.4 ms (EARLY, measured scheduler gap) §4
(c5) RTL: one multi-row KV-only item (N accepted positions through ONE weight sweep, as W siblings share it in a tree level) yes (same K/V) up to ~(N-1)/N of the row CARD time A tree level at W8 costs 299 us per item against 482 us at W2 (D7_TREE_LEVERS §7.1): items amortise the sweep. A new image + golden; not host work.

Verdict. No EXACT host scheme moves the 13.25 ms off the critical path. The data dependency is the root row and the KV rows, which are both verify outputs.

7. Next-window commands (silicon, d7 W2 V20 K_C 64; the coordinator runs the window)

The kit must carry the levers. The window-3 kit staged from 20298af07 does NOT have them. Stage a new versioned kit from the lever commit or later (box1, create-only):

bash scripts/qwen27b_open_release_20261102/c4q5120/stage_kits_talon2.sh 913a2690c ship      # or the doc commit on top of it

Then the window's usual flash / cold AC / bring-up / golden (GMDS=7,5,3,2) runs, unchanged.

Launcher. Serve arms use d7_hostgap_evidence/serveW3_remote.sh. It is window 2's serveW2_remote.sh with V and RB as parameters, the same FIFO credential handling, and no password in the file. Invoke it exactly as serveW2_remote.sh, prefixed with:

Shorthand: ALL='EAGLE4_Q5120_KVBATCH=1 EAGLE4_Q5120_PREROOT=1 EAGLE4_Q5120_KVINTERLEAVE=1 EAGLE4_Q5120_SKIP_GPU_TREE=1 EAGLE4_Q5120_STATS_EVERY=50'

order TG LEV REFJ what it decides
1 w3a_all $ALL - the reference (= window-2 arm d, 110.48 tok/s)
2 w3b_tim $ALL EAGLE4_Q5120_TIMING=1 w3a the silicon decomposition inside the 13.25 ms RPC (rows prog / wait / h2c, tree, tree wait, IPC) + TIMING is non-perturbing (--same, tok/s about w3a)
3 w3c_rowvec + EAGLE4_Q5120_ROWVEC=1 w3a ROWVEC on silicon
4 w3d_pin + EAGLE4_Q5120_TREEPIN=1 EAGLE4_Q5120_PREPOISON=1 w3a TREEPIN + PREPOISON
5 w3e_h2c + EAGLE4_Q5120_H2CPAR=1 w3a H2CPAR (the h2c_ms row of arm 2 says what it can win)
6 w3f_early + EAGLE4_Q5120_EARLY=1 (= every lever) w3a EARLY; the best stack
7 w3g_bar CM=none - GPU-only bar

8. Commits and evidence