EAGLE4 hardware research · author box1-qwen-junior · 2026-11-09 · source reports/results/DRAFTER_INTERFACE_GUIDE_4C1_20261109.md at repo commit 5ea8c7c7e · file references are repo-relative paths (path:line).
4C1 drafter interface guide: access, host driver and the per-step protocol (talon2, 2026-11-09)
For the user and Igor. Program 1 (the 4-card "4C1" EAGLE4 drafter for Llama-3.2-3B / Qwen3.8-27B, verified today by an H200) seen from the side of Program 2 (the target model running entirely on FPGAs), which wants to drive this drafter instead of the GPU. Written from the code: every claim carries a repo-relative path:line; files that live only on talon2 are marked talon2:~/... and were read read-only. Short version: [DRAFTER_INTERFACE_ONEPAGER_4C1_20261109.md](DRAFTER_INTERFACE_ONEPAGER_4C1_20261109.md). Nothing was flashed, run on a card, locked or removed to write this.
0. Answer first
- The drafter is four V80 cards on talon2 driven by one Python host process over PCIe (BAR0 registers + QDMA). Two ATT cards (PCI 17, 3d) hold half the attention heads each; two MLPV cards (63, 50) hold the MLP, vocab head and tree search. The cards talk to each other over QSFP; the host never relays card-to-card data during a tree.
- Per step the drafter needs, per accepted token: the token id and the target model's 3 auxiliary hidden states (3 x 3,072 for Llama-3.2-3B, 3 x 5,120 for Qwen3.8-27B). From those the host builds one 12,288-byte row per token (embedding +
fc projection, fixed-point int16). It returns a tree of W + W^2(D-1) nodes (42 at width 2, depth 11): token id, first-child / next-sibling links, and a log2 score.
- Today the GPU is also used for three things beyond verify: the
fc projection, the drafter prompt prefill whose K/V is imported into the ATT cards ("KVI"), and the first draft of each request ("lever Y"). All three have card-only replacements in the kit; the card-only K/V path is measured on silicon with unchanged accept and a −13.7 % / −24.6 % tok/s cost (section 5.3).
- Option (a), the target FPGAs talk to the drafter host process, is software-only: the card side already runs as a separate, torch-free card-server process with a request/reply API (
scripts/4c1/c4_cardproc.py:191-212).
- Option (b), a direct QSFP link from the target cards, is not built. The link layer has free cages and one idle engine slot per ATT card, but no "verifier -> drafter" frame exists in RTL, and the per-item cross-card protocol currently needs host register traffic.
1. System picture
1.1 Cards and roles (talon2, 2026-11-09)
| slot | PCI (PF0) | image | reg 0x100 | tp_link4 build id | role | source |
|---|
| ATT0 | 0000:17:00.0 | F3t 0xE4B713F9 (md5 97d8445c…) | 0xe4b711f9 | 0xe4b713f9 | attention, heads 0-15, rank 0 | serve log header CARDS: in reports/results/TRACK1_RECORD_F3T_20261109/config_sweep/serve3/*183008*.log.gz line 12 |
| ATT1 | 0000:3d:00.0 | F3t 0xE4B713F9 | 0xe4b711f9 | 0xe4b713f9 | attention, heads 16-31, rank 1 | same |
| MLPV0 | 0000:63:00.0 | 0xE4B71342 (md5 3e82ea1d…) | 0xe4b71142 | 0xe4b71342 | MLP + vocab + tree, rank 0 | same; TRACK1_RECORD_F3T_20261109/README.md:3 |
| MLPV1 | 0000:50:00.0 | 0xE4B71342 | 0xe4b71142 | 0xe4b71342 | MLP + vocab + tree, rank 1 | same |
| (other) | 99, bd, e1 | Program 2 / other owners | – | – | not part of the drafter | .claude/skills/eagle4-talon2-connect/SKILL.md:37 |
- talon2 also carries one H200 NVL (the verifier today) and 31 GB of RAM (
SKILL.md:37).
- The ATT image row in the host register map:
scripts/4c1/c4_regmap.py:192 (fork verilog/forks/lutgemm_att_d9_e0_20261104, :565, lanes 512, KV_COMMIT present, dual model). The MLPV fork is verilog/forks/fourcard_4c1_mlpv_g11f_lsi_eaearly_20261104 (c4_regmap.py:644).
- Note: the box1 copy of
scripts/4c1/c4_bringup.py:26-34 still names the old box1 cards 01/42/61/62. The talon2 kit copy has the talon2 map (talon2:~/eagle4_talon2_4c1/sibw/repo/scripts/4c1/c4_bringup.py:27-38, from the patch scripts/4c1/talon2/patch_talon2_k4_link_20261023.py:3-25).
1.2 QSFP links (the "K4" map in use)
| link | cable (card + cage) | link engine | carries | source |
|---|
| ATT TP pair | 17c0 – 3dc3 | e1 (tp_xchg) | the G6 partial-sum exchange between the two attention halves | talon2:…/c4_bringup.py:32 |
| MLPV TP pair | 63c3 – 50c0 | e1 (tp_xchg) | G9 partial sums + vocab-shard merge | :33 |
| stage rank 0 | 17c1 – 63c2 | e0 (hop_xport) | ATT0 -> MLPV0 per-item "BELL" (22,528 B) and MLPV0 -> ATT0 next item (6,144 B) | :34 |
| stage rank 1 | 3dc0 – 50c3 | e0 (hop_xport) | ATT1 <-> MLPV1, same | :35 |
| idle diagonal | 17c2 – 50c2, 3dc2 – 63c1 | – | cabled, unused | :36-37 |
| empty cages | 17c3, 3dc1, 50c1; 63c0 has a stray cable (link up, no beacon) | – | free | scripts/4c1/talon2/patch_talon2_k4_link_20261023.py:6 |
PORTMAP words written at bring-up: 17 = 0x45, 3d = 0x74, 63 = 0x76, 50 = 0x47 (talon2:…/c4_bringup.py:38). The link layer is tp_link4 (4 cages, Aurora 64B66B x4 per cage, beacon discovery, run-time engine-to-cage binding; verilog/forks/fourcard_4c1_att_dual_sibw_b_20261027/link4/x1_snapshot_2fcaa3d_rtofix/tp_link4.sv:1-24).
flowchart LR
subgraph HOST["talon2 host"]
V["verifier today: SGLang on the H200"]
S["scheduler-side client: root row (embed + fc), KV rows, tree -> token/parent/score"]
P["card-server process c4_cardproc.py (no torch): registers + QDMA"]
V --> S --> P
end
P -- "PCIe: BAR0 regs + QDMA H2C/C2H" --> A0["ATT0 card 17 (heads 0-15)"]
P -- PCIe --> A1["ATT1 card 3d (heads 16-31)"]
P -- PCIe --> M0["MLPV0 card 63"]
P -- PCIe --> M1["MLPV1 card 50"]
A0 <-- "QSFP 17c0-3dc3 TP (G6 sums)" --> A1
M0 <-- "QSFP 63c3-50c0 TP (G9 sums, vocab merge)" --> M1
A0 <-- "QSFP 17c1-63c2 stage (BELL / next item)" --> M0
A1 <-- "QSFP 3dc0-50c3 stage" --> M1
M0 -. "tree SoA read by host (C2H)" .-> P
1.3 What runs where
| on the host (talon2 CPU / GPU) | on the cards |
|---|
| target model + verify (SGLang, H200) | the whole drafter layer for every tree item: G1 norm, QKV, attention over the drafter KV cache (ATT); o-proj, MLP, final norm, lm-head, top-K and tree search (MLPV) |
root row: embed_tokens(token) and fc(target 3 x hidden), quantised to int16 (scripts/fpga_tree_marshaller.py:1414-1467) | item-to-item hand-off of the tree, level by level, over QSFP with no host involvement (loop_seq_d2, scripts/4c1/c4_loop.py:5-17) |
drafter K/V import for the prompt and accepted tokens (KVI) unless the card-only path is enabled (scripts/4c1/c4_kvi.py:1-3) | KV-only items: the cards compute K/V rows from host rows (c4_loop.py:225-253) |
the first draft of each request on the GPU when lever Y is on (EAGLE4_4C1_FIRST_GPU=1 in the serving env) | – |
SoA -> func_info (tokens, parents, scores) for SGLang (fpga_tree_marshaller.py:586-660) | the tree result written by MLPV's OUT-DMA into card HBM (verilog/forks/fourcard_4c1_mlpv_g11f_lsi_eaearly_20261104/src_full/top/tree_dma_writer.sv:7-35) |
2. Access
2.1 Reaching talon2
- From box1: ssh through the Cloudflare access proxy (
cloudflared access ssh) as user carrick; plain ssh fails because the public name is IPv6-only and box1 has no IPv6 route. Exact command and the scratch known-hosts file: skill .claude/skills/eagle4-talon2-connect/SKILL.md:15-30. A second hop via asgard is listed there too.
- Before any card work: read talon2's flash logs (other teams flash cards there) and do the two-factor identity check (
SKILL.md:37-38,63).
- Power: a talon2 cold AC is a PDU outlet cycle with a graceful shutdown first; procedure in skill
.claude/skills/eagle4-talon-pdu-coldac/SKILL.md. PDU/BMC credentials are held by the user only.
- Accounts: carrick is not a sudoer on talon2. Card bring-up and serving drivers run as root through a wrapper (
talon2:~/eagle4_talon2_4c1/rootrun.sh:2); the sudo password lives in a mode-600 credential file on the host that the scripts read by path. SGLang itself must run as carrick (trap 7.5).
2.2 The card lock
- Wrapper:
talon2:~/eagle4_talon2_4c1/lock/card_run.sh (repo copy reports/results/FOURCARD_DESIGN_20261016/talon2_stream_a_20261022/card_run.sh). Usage OWNER=<who> bash ~/eagle4_talon2_4c1/lock/card_run.sh <cmd> (card_run.sh:3,11).
- Two layers: a
flock on lock/card.lock (:14-15), then a wait while the last line of the host-wide text lock ~/TALON2_CARD_LOCK starts with OWNER (polled every 30 s, :16-18). It appends OWNER <owner> <ts> <cmd> and later FREE <ts> (<owner>: rc=N), also on INT/TERM (:20,27,31-32), and writes lock/CARD_OWNER and lock/card_runs.log (:21-22,28-29).
~/TALON2_CARD_LOCK is shared with Program 2's jobs on cards bd/e1 (owner t2-ure-runall seen 2026-11-09 20:43Z). A long hold blocks them too.
- Scripts that are already inside a hold set
CARD_LOCK_HELD=1 and must not nest another card_run (it would deadlock: TRACK1_RECORD_F3T_20261109/config_sweep/scripts/serve_cfg_drv.sh:5-6,18).
2.3 QDMA and device files
- Driver: QDMA PF kernel module loaded at boot by
talon2:~/eagle4_talon2_4c1/eagle4-qdma-pf.service:9-14 (fallback in scripts/talon2/post_boot_talon2.sh:33).
- Queues per card after every boot:
EAGLE/v80_test_suite/03_setup_qdma.sh 4 <bus>:00.0 (sets qmax, adds/starts 4 MM queues each way, then a 64 B round trip that must print Roundtrip test: PASS; 03_setup_qdma.sh:21-23,40-60). The F3t window ran it for 17/3d/50/63 (TRACK1_RECORD_F3T_20261109/f3t_kit/win_f3t.sh:22).
- Device file per card:
/dev/qdma<bus><slot>1-MM-0 on PF1, e.g. /dev/qdma17001-MM-0 (scripts/4c1/c4_bar.py:175-178). Root-owned, mode 0600.
- Registers: PF0 BAR0 is mmapped; it maps AXI 0x0201_0000_0000 + 256 MiB (
c4_bar.py:10-16). Windows the kit may write: drafter CSRs 0x0201_0001_0000 (64 KB), loop sequencer 0x0201_0104_F000 (4 KB), link enable 0x0201_0104_B008, tp_link4 CSRs 0x0201_0105_A000 (4 KB); any other write is refused (c4_bar.py:24-35,43). Every 32-bit store is one ctypes store, because a slice store splits into two AXI-Lite beats (c4_bar.py:4-6).
- **Only plain
pread/pwrite on the QDMA device** (c4_bar.py:166-205). A vectored read (preadv) hit a kernel BUG in the QDMA cdev and froze a host on 2026-10-19 (memory feedback_qdma_cdev_no_vectored_io_kernel_bug). Do not use readv/writev/preadv/pwritev, aio or io_uring on these files.
2.4 Identity checks (before every run)
python3 -I scripts/4c1/c4.py --model llama3b discover --out RUN/discover.json --flash-record F reads reg 0x100 and the tp_link4 BUILD_ID (CSR 0x0201_0105_A004) of every card and binds them to the flash record md5 (scripts/4c1/c4.py:4).
- reg 0x100 reads
{0xE4B7, 0x11, PROBE} and collides between images; the tp_link4 BUILD_ID is the unique label (c4_regmap.py:24-31,348; runbook reports/results/FOURCARD_DESIGN_20261016/4C1_SILICON_RUNBOOK.md:15, blocker B7). Expected today: ATT 0xe4b711f9 / 0xe4b713f9, MLPV 0xe4b71142 / 0xe4b71342.
scripts/4c1/c4_sglang.py:225-240 re-checks identity and the arm state from RUN/discover.json + RUN/arm_state.json when the serving process opens the cards.
2.5 PERSIST: arm the deepest depth before the first AP_START
- The cards run a persistent kernel: AP_START is issued once per boot, only while
pipe_out_seq (0x184) is 0 (scripts/4c1/c4_card.py:157-170). The OUT-DMA owner select is latched on that start as outdma_sel_tree_q <= (max_depth > 1) (ATT verilog/forks/lutgemm_att_d9_e0_20261104/src_full/top/eagle4_drafter.sv:2272; MLPV …mlpv_g11f_lsi_eaearly_20261104/src_full/top/eagle4_drafter.sv:2323; start_pulse at ATT control_regs.sv:632).
- So
c4.py arm --depth D must use the deepest depth that will ever run until the next cold AC: 0x158 = 0x2F (reads 0x4E53002F), 0x20 = D, 0x1C8 == 0 (4C1_SILICON_RUNBOOK.md:52-55). The talon2 cards were armed at 15 (win_f3t.sh:31).
- The host refuses an md above the armed depth, and a first tree at an md other than the armed one (
scripts/4c1/c4_loop.py:280-286,292-298).
3. Host driver stack, layer by layer
Kit root scripts/4c1/ (stdlib only, runs under python3 -I; the talon2 kit is a copy at talon2:~/eagle4_talon2_4c1/f3twin_20261109/kit/repo plus the image patch TRACK1_RECORD_F3T_20261109/f3t_kit/patch_13f9_host.py).
| layer | file | purpose | key functions (line) |
|---|
| L0 access | c4_bar.py | BAR0 mmap with write windows + QDMA pread/pwrite | SiliconBar, QdmaDev :165, pwrite :186, pread :197 |
| L0 map | c4_regmap.py | the ONE register table, each row citing the RTL decode line; image table; ring bases | _r(...) rows :338-450, image rows :81-267 (F3t :192), ring bases :841-856, BELL :858-859, loop CSRs LC :809, tp_link4 CSRs TP :796 |
| L1 card | c4_card.py | one card = drafter / loop / tp_link4 accessors + item helpers | program_inputs :109, issue_item :157, wait_out_seq :172, wait_tree_done :183, poison :194, read_soa :205, census :271 |
| L1 bring-up | c4_bringup.py, c4.py | discover / link / boot weights / arm / golden / census CLI | c4.py subcommands and rc table :4-22 |
| L2 tree | c4_tree.py | header word, pool geometry, SoA decode | p2_hdr :27, n_nodes_of :39, unpack_soa :51, link_problems :61 |
| L2 session | c4_loop.py | one request / KV-only row / whole tree on four cards | begin_request :204, kvonly_row :225, enable_links :255, program_cfg :291, run_tree :346, _run_tree_wait :434 |
| L2 chain/census | c4_chain.py | back-to-back trees with per-phase dwell (--set fsm) | used for the census in section 4.7 |
| L3 KV | c4_kvi.py, c4_kvi_hooks.py | import the GPU drafter's K/V into both ATT ranks (16 heads each) | c4_kvi.py:1-45 contract; hooks c4_kvi_hooks.py:1-12 |
| L3 KV (card-only) | c4_kvpo.py, c4_kvx.py | accepted-token (and prompt) K/V computed on the cards from host rows | c4_kvpo.py:1-20, c4_kvx.py (send, run_batch) |
| L3 KV commit | c4_kvcommit.py | keep drafted K/V of accepted nodes on the card (COMMIT_PATH), images with KV_COMMIT | header :1-60; off in today's serving env |
| L4 SGLang backend | c4_sglang.py | FourC1Fpga behind the stock FpgaTreeDrafter: captures register writes, runs run_tree | FourC1Fpga :48, launch :126, open_session :225, build :415, arm :472 |
| L4 card process | c4_cardproc.py | FASTGLUE=4: all card ops in a separate torch-free process, pickled RPC over a socketpair | op table :191-212, client marshal_inputs :459, remote_stage_cfg_launch :478 |
| L4 tree math | scripts/fpga_tree_marshaller.py | root row build, SoA -> func_info, structural checks | q412 :305, qhfc :424, marshal_inputs :1374, root_image :1414, soa_to_func_info :586, validate_func_info :783 |
| L5 SGLang hook | scripts/4c1/sglang_site/sitecustomize.py | put on PYTHONPATH by the launcher (run_4c1_sglang.sh:96), so it runs in every interpreter incl. SGLang's scheduler: arms the eaglefit FPGA hook, points its driver factory at c4_sglang.build, arms the KVI hooks; edits nothing inside SGLang | :1-40 |
| L5 SGLang patches | scripts/cost_draft_tree_fpga_eaglefit.py, scripts/cost_draft_tree_fpga.py (in this repo, no other copies) | import-time patches of SGLang 0.5.16's CostDraftTree: draft_model_decode -> one whole-tree FPGA launch _draft_fpga_tree_v14 (:219-221, :638), prefill capture of the target 3H hidden (:223-225), verify-extend capture _capture_extend_feed (cost_draft_tree_fpga.py:394-440), marshal_inputs + launch_and_read (:852-853) | |
| L5 launchers | run_4c1.sh, run_4c1_sglang.sh, run_4c1_sglang_qwen.sh, run_4c1_nstream.sh, demo/demo_4c1.sh | bring-up chain; serving env + engine command | run_4c1_sglang.sh:60-101 |
| extras | c4_tp4.py, c4_nstream.py, c4_pingpong.py, c4_hbmreplay.py, c4_batchverify.py, native/ | 7-card TP4 map; N-stream / ping-pong serving; HBM replay; batched verify; C fast path (native/c4h.c) and a dma-buf probe (native/qdma_dmabuf) | not on the 883 tok/s path |
Serving levers of the 883 tok/s config, "PQ+C": P = EAGLE4_4C1_HP_PREPOISON, Q = EAGLE4_4C1_HP_PINPAR, C = EAGLE4_4C1_HP_CAPHOST + EAGLE4_4C1_HP_IMGBYTES (mapped in TRACK1_RECORD_F3T_20261109/f3t_kit/serve_f3twin.sh:11,41-44; implemented in c4_hpre.py, which exists only in the talon2 kit, repo copy reports/results/SEVENCARD_BOARD_20261026/sibw_talon2/hostpre_20261107/kit/c4_hpre.py:14-21). run_4c1_sglang.sh refuses W > 2 unless EAGLE4_4C1_ALLOW_WIDE=1 (:48-51), takes the card lock itself unless CARD_LOCK_HELD=1 (:108-115), and runs a link check before SGLang starts (:128-138). Discover/link/boot/arm/golden are done before it by run_4c1.sh (:4,38).
Call order in a serving run (in-process view; with FASTGLUE=4 the card half runs in c4_cardproc):
sitecustomize -> cost_draft_tree_fpga_eaglefit.arm() -> c4_sglang.arm() replaces _build_drv_v14 with c4_sglang.build (sitecustomize.py:13-35, c4_sglang.py:472-475).
build(cdt) -> c4_cardproc.build_remote (FASTGLUE=4) or build_local -> open_session (identity, arm state, golden record) -> FourC1Fpga + the drafter class + c4_kvi.attach (c4_sglang.py:415-469).
- Per request:
begin_request pulses 0x154 on both ATT ranks (c4_loop.py:204-213); KVI imports the prompt K/V.
- Per step (
scripts/cost_draft_tree_fpga_eaglefit.py:638-859): after the previous verify the extend capture stores {pos: (token, target 3H row)}, the GPU drafter extend runs and its K/V rows are gathered (c4_kvi_hooks.py:65-109) -> the next draft joins pending card work, pulses 0x154 on a new request (eaglefit.py:736-762) -> picks the root input (the captured 3H row at seq_len-1, :764-781) -> KV catch-up c4_kvi.catchup loads the K/V rows into both ATT ranks (:784) -> marshal_inputs (root row + config capture, :852) -> launch_and_read (:853) -> run_tree on four cards -> SoA -> rank cross-check -> soa_to_func_info -> func_info_to_torch -> SGLang verify.
4. The per-step protocol
4.1 Registers the host writes (drafter CSR block, offsets from 0x0201_0001_0000)
A/ = verilog/forks/lutgemm_att_d9_e0_20261104/src_full/top/, M/ = verilog/forks/fourcard_4c1_mlpv_g11f_lsi_eaearly_20261104/src_full/top/. (The _r rows in c4_regmap.py cite the older v0/v1 forks; the lines below are for the two images on the cards.)
| off | name | written when | meaning | ATT decode | MLPV decode |
|---|
| 0x000 | AP_CTRL | once per boot (first item) | W[0] start | A/control_regs.sv:862-865 | M/control_regs.sv:811 |
| 0x010 | ctx_len | every tree / KV row | sequence position (md-aware clamp by host, c4_loop.py:59) | :872 | :821 |
| 0x018 | K_F | every tree | frontier 8 (c4_loop.py:304) | – | – |
| 0x020 | max_depth | arm + every tree | tree depth D (<= armed depth) | :900 | :834 |
| 0x028-0x03C | out0/1/2 base | every tree (MLPV) | OUT-DMA bases 0x41_4000_0000 / +0x1000 / +0x2000 | :901-906 | :836-841 |
| 0x054-0x074 | boot-DMA | per input row | HBM -> input buffers walk (emb 0x54[2], hfc 0x54[3]), status 0x74 | :918-925 | – |
| 0x100 | BUILD_ID | read only | identity | :1185 | :1141 |
| 0x14C | n_tiles | every tree (MLPV) | vocab shortlist tiles, 0 = full vocab | :991 | :920 |
| 0x150 | K_C | every tree (MLPV) | exact-rescore shortlist (64 in serving) | :994 | :923 |
| 0x154 | KV reset | every new request (ATT) | W[0] pulse, R {0x4B56, cnt} must read cnt 0 | :998 | :927 |
| 0x158 | NORM_SEL | arm | 0x2F (reads 0x4E53002F) | :1003 | :932 |
| 0x15C | ACCFIX | per KV-only item (ATT) | bit3 = the next item is KV-only; R {0xACF1, cfg} | :1005 | :934 |
| 0x180 | IN_SEQ | per item | doorbell k+1 | (c4_regmap.py:362) | |
| 0x184 | OUT_SEQ | read | items finished | A/control_regs.sv:1078 | M/control_regs.sv:1034 |
| 0x198 | HDR | per item | {new_tree[31], level[27:24], branch[23:21], parent[20:18], token[17:0]} (c4_tree.py:4,27-31) | | |
| 0x1C0 | CONSUMED | read | items started; must equal k after the header write (c4_card.py:160-163) | | |
| 0x1CC | rank / width | arm (ATT rank), every tree (MLPV width) | ATT: rank + sticky item error; MLPV: tree width W | :885 / R :1097 | :835 / R :1052 |
| 0x004, 0x00C | COMMIT_LANES / CFG | KV commit only | ATT KV_COMMIT (not used in today's serving) | :883-884 | absent |
Loop sequencer CSRs (0x0201_0104_F000 + off, c4_regmap.py:809-813): per tree the host writes TREE id, KH0/KT0 (item counters), DW = (W << 4) | md, WANT, ROOT header, THETA (TREE_STOP threshold, Q12.20), DMIN, the four ring bases, and CTL = EN | mode bits | ARM, on the MLPV first and then its ATT (c4_loop.py:380-392).
4.2 Card HBM buffers (addresses as the host sees them through QDMA)
| address | what | written by | source |
|---|
| 0x42_0040_0000 | FASTIN: the 12,288 B root/KV row, staged per item on both ATT cards | host H2C | c4_regmap.py:847, c4_card.py:109-128 |
| 0x42_0041_0000 + t x 12,288 | batched KV-only rows (KVX) | host H2C | KVONLY_EXTEND_PHASE0_20261108.md:15-17 |
| 0x41_4010_0000 | ATT OUT ring (MLPV reads it over the link) | ATT | c4_regmap.py:841 |
| 0x42_B000_0000 | MLPV / ATT IN ring (slot stride 0x14000, item offset 0x11800) | link | c4_regmap.py:842-846 |
| 0x41_4000_0000 | tree result SoA (20,480 B), MLPV only | MLPV OUT-DMA | c4_regmap.py:851, c4_tree.py:18-21 |
| 0x41_8000_0000, 0x41_C000_0000 | exact fp16 lm-head + duplicate | boot | c4_regmap.py:849-850 |
| KVI base (0x1F0/0x1F4) | imported K, V at +128 MiB | host KV_LOAD | c4_kvi.py:12-14 |
4.3 One tree, step by step (scripts/4c1/c4_loop.py:346-405)
- Poison the SoA on both MLPV (0xEEEEEEEE at region0[0] and the last score;
c4_card.py:194-196) so a stale read cannot pass.
- Program config on all four cards (ctx, K_F, md) and on MLPV (K_C, n_tiles, out bases, width with read-back) (
c4_loop.py:291-315).
- Enable the link DMA once per card and prove it in
st_misc (c4_loop.py:255-276).
- Arm the loop sequencer of each pair, TAIL (MLPV) first then HEAD (ATT), with identical tree/depth/width/root (
:380-392).
- Stage the root row on both ATT cards (FASTIN H2C + two boot-DMA walks;
c4_card.py:109-128).
- Issue only the root item on both ATT ranks: header 0x198, audit 0x1C0, doorbell 0x180 (
c4_card.py:157-170, c4_loop.py:403-404). From here the cards run the whole tree themselves: ATT -> MLPV BELL, MLPV -> ATT next item, level by level.
- Poll the four loop STATUS words; TREE_STOP verdicts (early stop when the level's probability mass is below theta) are taken by the host only when both MLPV ask at the same level with identical scores (
c4_loop.py:317-344).
- Done = four DONE, links drained, loop DIGEST of HEAD == TAIL per pair and pair 0 == pair 1 (
:484-497).
- Read the SoA from BOTH MLPV, require byte-identical regions, identity retrieve-index, per-item status clean (
:510-536). Any failure discards the tree (never returned).
4.4 BELL / frame contract ATT -> MLPV (card to card, no host)
| message | bytes per item per rank | layout | source |
|---|
| BELL, ATT -> MLPV | 22,528 | r5 product half 8 KB @ +0, unwritten gap @ +0x2000, h1 6,144 B @ +0x4000 | c4_regmap.py:858-859; A/pipe_seam_bridge.sv:5,20-24 |
| ITEM, MLPV -> ATT | 6,144 | next item's hidden half, TAIL OUT slot(parent)+0x11800 -> HEAD IN slot k+0x11800 | M/eagle4_drafter.sv:2297; loop_seq_d2.sv:58,333 |
| control frame | 256-bit | [51:48] type (BELL 1, ITEM 2, SPEC 3, COMMIT 4, CANCEL 5, STOP 6), [63:52] tree, [95:64] header, [127:96] k | verilog/forks/fourcard_4c1_mlpv_g11f_lsi_eaearly_20261104/common_overlay_strap1/rtl/loop_seq_d2.sv:7-13,101 |
4.5 How the tree comes back (OUT-DMA SoA, written by MLPV only)
Writer M/tree_dma_writer.sv:7-35,66-69; host decode scripts/4c1/c4_tree.py:51-54.
| region | byte offset | int32 per node |
|---|
| 0 | +0x0000 | draft token id (18-bit) |
| 1 | +0x1000 | retrieve index = identity (integrity check) |
| 2 | +0x2000 | first child (pool index, −1 none) |
| 3 | +0x3000 | next sibling (−1 none) |
| 4 | +0x4000 | cumulative log2 probability, Q12.20 |
- Node count
W + W^2(D-1); level 0 holds W nodes, level L >= 1 holds W^2 at pool index W + (L-1)W^2 + parent_slot*W + child (c4_tree.py:39-48).
- There is no parent array: the host rebuilds parents from the links, prepends the root, and converts scores to linear
2^score * 0.999^depth (float32) (fpga_tree_marshaller.py:620-660). Output to SGLang: tokens int64, scores float32, prev/next/side, count = nodes + 1.
- Qwen: tokens are draft-vocab ids; the host maps them to target ids with
target = draft + d2t[draft] (c4_sglang.py:340-347, c4_profile.py:18).
4.6 KV after verify (what the verifier must send back)
- Today's serving (KVI=1, KVCOMMIT off): after verify, the GPU drafter's own extend forward over the accepted tokens produces K/V; the hook gathers those rows from SGLang's pool (
c4_kvi_hooks.py:1-12), the host quantises them (round_half_even(x * s * 2^10), per-tensor scales) and loads 16 heads into each ATT rank via KV_LOAD (c4_kvi.py:4-26).
- Card-only path (KVPO / KVX): per accepted token the host builds the same 12,288 B row from
(token, target 3 x hidden) and runs one KV-only item (0x15C bit3) on both ATT ranks; the card computes and appends that row's K/V (c4_kvpo.py:1-20, c4_loop.py:225-253). KVX batches all rows of a step into one card request (KVONLY_EXTEND_PHASE0_20261108.md:12-21); KVX_PROMPT does the same for the prompt (:96-104).
- KV commit (images with KV_COMMIT, F3t has it): the card can keep the K/V it already drafted for accepted nodes; the host sends the lane of each accepted node and the card copies rows (
c4_kvcommit.py:4-35). Only the last (leaf) row still needs a KV-only item. Not enabled in the 883 tok/s config.
- Either way the verifier's output per step is: the accepted token ids (in order) and the target hidden states (3 aux layers) of each accepted position; the next tree's root is the last accepted token, with ctx = old ctx + accepted + 1 (
c4_kvcommit.py:30-33).
4.7 Measured timing
| what | value | source |
|---|
| card tree issue -> done, p50, w2 K_C 64 | 2,234 µs at d7; 2,774 µs at d9; 3,384 µs at d11 (about 135-152 µs per extra item) | reports/results/DRAFTER_SPEED_RESEARCH_20261107/F3T_CONFIG_SWEEP_20261109.md:154-157 |
| ATT G5 attention per item | ~40 µs | same table |
| GPU verify-only time, batch 1 | 3.45 ms | reports/results/BATCH_BUBBLES_20261109/INTERLEAVE_BATCHVERIFY_SILICON_20261109.md:76 |
| end-to-end, d11 w2 fa3 | 883.87 tok/s @ accept 6.90 (19,615 tokens, 2,843 verify steps, 22.19 s) | serve log config_sweep/serve3/*183008*.log.gz (md5 f11c5ff1) |
| card-only KV path (E2b image, w2 d7) | KVX 696.24 tok/s (−13.7 %), KVX+PROMPT 607.83 (−24.6 %), accept 5.88-5.89 unchanged; step p50 6.94 -> 8.24 ms | KVONLY_EXTEND_PHASE0_20261108.md:152-166 |
| QSFP link | 12.48 GB/s per direction; echo round trip 1.20 µs at 6 KiB | reports/results/MULTIFPGA_INTERCONNECT_CENSUS_20260915.md:13,71 |
| QDMA host relay (card -> host -> card) | 1.85-1.90 GB/s at 1 MiB; 28.7-29.3 µs one-way at 6 KiB | MULTIFPGA_INTERCONNECT_CENSUS_20260915.md:14,78 |
5. Verifier interface (for Program 2)
5.1 What a non-GPU verifier must provide and consume, per step and per stream
| dir | item | Llama-3.2-3B | Qwen3.8-27B | dtype / notes |
|---|
| in | accepted token ids (this step), in order | 1 .. D+1 ids | same | target vocab ids (128,256 / 248,320) |
| in | target aux hidden of every accepted position | 3 x 3,072 per token (18,432 B fp16) | 3 x 5,120 per token, layers 3/31/59 (30,720 B bf16) | c4_profile.py:47,69-70 |
| in | prompt (once per request): ids + aux hidden of every prompt position | as above per token | as above | for the prompt K/V (KVX_PROMPT) or for a GPU/CPU drafter prefill (KVI) |
| in | per-step config | ctx_len, depth D (<= armed), width W, K_C | same | the host writes the CSRs |
| host-internal | root / KV row | 12,288 B = emb Q4.12 int16 x 3,072 + fc-out Q5.11 int16 x 3,072 | same 12,288 B (fc 15,360 -> 3,072, with fc_norm) | fpga_tree_marshaller.py:305-314,424-428; EAGLE4_TREE_HFC_FRAC=11 in the serving env |
| out | tree | W + W^2(D-1) nodes + root (43 at W2 D11) | same; tokens mapped draft -> target | tokens int64, scores float32, prev/next/side int |
| out | verify candidates | the verifier prunes to VTOK nodes (20 in the 883 config) | same | SGLang's own prune today |
Needed on the drafter side regardless of who verifies: the drafter's embed_tokens table and fc weights (Llama fc 9,216 -> 3,072; Qwen fc_norm x3 + fc 15,360 -> 3,072) to build rows, and for Qwen the d2t table (c4_profile.py:13-18,74-75).
5.2 Latency budget (measured parts)
The cards need 3.38 ms per d11 tree (2.23 ms at d7). Today's whole step at d11 is 7.81 ms (derived in the bar below) with a 3.45 ms GPU verify. A target-FPGA verifier is inside today's budget if its verify + the two hand-offs stay at or below what the H200 path uses now.
5.3 Option (a): target FPGAs talk to the drafter host process; the host keeps driving the cards
| exists today | must be built |
|---|
card-server process with a request/reply API, torch-free: ops stage_cfg, stage_cfg_launch(emb, hfc, ctx, md, vid) -> func_info, launch_and_read, kvonly_rows(imgs, ctxs, toks), kv_residency_clear, kvi_load, stats (c4_cardproc.py:132-212) | a network endpoint in front of it (today: pickle over a local socketpair, c4_cardproc.py:21) |
row builder root_image / prep_rows (embed + fc + quantise) | a CPU (or target-FPGA) implementation of embed + fc, today done on the GPU in the client (c4_cardproc.py:449-457) |
| card-only K/V: KV-only items, KVX batching, KVX_PROMPT for the prompt; accept unchanged on silicon | run with EAGLE4_4C1_FIRST_GPU=0 (first draft on the cards) and no GPU drafter prefill; not measured together end to end |
| per-tree checks (rank cross-check, digests, poison) inside the card process | the verifier-side tree prune and accept logic (SGLang does it today) |
5.4 Option (b): direct QSFP from target cards to drafter cards
| exists today | must be built |
|---|
tp_link4: 4 QSFP cages per card, Aurora 64B66B, beacon discovery, run-time PORTMAP binding (tp_link4.sv:1-24) | a "verifier -> drafter" frame (accepted ids + hidden rows, or ready-made 12,288 B rows) and an RTL consumer that writes FASTIN and rings the doorbell without the host |
free cages: 17c3, 3dc1, 50c1 empty; 63c0 cabled (stray); per the 7-card plan 17c3 may already run to card bd and 63c0 to e1 (ATT_TP4_20261108.md:44,49), unverified by beacon | cables / beacon proof to the target cards |
one idle engine slot per ATT card: tp_link4 has 3 engine slots and at TP2 the ATT's e2 is tied idle (ATT_TP4_20261108.md:32; tp_link4_bd.v:99-108) | a third engine (e.g. hop_xport2) bound to that slot on the ATT image |
precedents: KV-PREFILL design fed the ATT through a free cage/engine (DESIGN_KV_PREFILL_CARD_20261022.md:15,17,77); MLPV-SOLO and TP4 "ring B" (second ingest ring at IN + 0x140000 with hop_xport2) (MLPV_SOLO_20261022.md:27-28, SEVENCARD_MLPV_0x134D_20261109.md:27-47) | the fc (and Qwen fc_norm) on the target or drafter card, today host-side |
| tree result already in MLPV HBM at 0x41_4000_0000 | a "drafter -> verifier" return path over QSFP; and moving today's host-side per-tree work (config writes, loop arming, TREE_STOP verdicts, SoA cross-check) onto the cards |
6. How to run it today (talon2, from the committed docs)
All card steps as root inside one card lock; SGLang as carrick.
- Bring-up after a cold AC (
TRACK1_RECORD_F3T_20261109/f3t_kit/win_f3t.sh:18-31): QDMA queues on 17/3d/50/63 (03_setup_qdma.sh 4 <bdf>), then c4.py --model llama3b discover, link, boot, arm --depth 15.
- Golden / identity check (
win_f3t.sh:32-33):
c4.py --model llama3b golden --disc RUN/discover.json --md 15,7,2 --out RUN/golden.json, then python3 -I cmp_golden_levels.py <reference golden.json> RUN/golden.json (rc 0 = level-identical). The F3t window passed 12/12 trees vs the E2b record (TRACK1_RECORD_F3T_20261109/EVALUATION.md:81).
- Serving, the 883 tok/s config (
F3T_CONFIG_SWEEP_20261109.md:178-191; as root inside a lock, the driver starts SGLang and chowns flashinfer back to carrick after each round):
OWNER=<you> bash ~/eagle4_talon2_4c1/lock/card_run.sh env MODEL=llama3b ORDER=PQC \
RUNDIR=/home/carrick/eagle4_talon2_4c1/runs/cfgsweep_20261109/l2 OUTD=<out> \
DEPTH=11 WIDTH=2 VTOK=20 KC=64 BACKEND=fa3 EAGLE4_PAIR_TREE_STOP_THETA=-2.5 ARM_PINPAR=0 EAGLE4_4C1_ALLOW_WIDE=1 \
bash /home/carrick/eagle4_talon2_4c1/cfgsweep_20261109/serve_cfg_drv.sh
The engine command it produces: engine_test.py … --draft-tree-depth 11 --draft-tree-width 2 --verify-tokens-num 20 --max-new-tokens 256 --max-requests-num 80 --speculative-exec-mode 1 --speculative-draft-attention-backend fa3 --sglang-version 0.5.16 --enable-cuda-graph --enable-piecewise-cuda-graph with the FPGA env in the log header (serve3/*183008*.log.gz lines 5, 8).
- Census (card µs per tree, per phase):
c4_chain.py --set fsm, 40 trees (F3T_CONFIG_SWEEP_20261109.md:151; scripts config_sweep/scripts/census.sh, chain_b*.sh).
- Offline (no card):
python3 -I scripts/4c1/c4_selftest.py | tail -1 runs the whole flow against the mock four-card world (4C1_SILICON_RUNBOOK.md:113,134).
7. Traps
- PERSIST latch. The first AP_START after boot fixes the OUT-DMA ABI; arm the deepest depth first (section 2.5). Recovery needs a cold AC.
- Build-id guards.
TREE_EXPECT_BUILDID (scripts/fpga_tree_marshaller.py:193-197) is read only by the single-card driver builder (scripts/cost_draft_tree_fpga_eaglefit.py:428-433), which the 4C1 path replaces, so setting it does nothing for 4C1. The 4C1 kit keys on its regmap image rows (c4_regmap.py:81-267) plus the flash-record md5 (:36-41), re-checked when the serving process opens the cards (c4_sglang.py:238). A new image needs a row (talon2 uses patch_13f9_host.py); adding ids to the old pair allowlists would misdrive the cards (4C1_SILICON_RUNBOOK.md:11 blocker B3, :85-96).
- Root-owned lock files.
card.lock, CARD_OWNER and card_runs.log are root-owned in a carrick directory, so the lock only works as root; give outputs back to carrick after root jobs (scripts/talon2/post_boot_talon2.sh:24,51).
- Chat-date drift. The Llama-3.2 chat template inserts today's date; text-identity gates across midnight change with no defect. Pin
EAGLE4_CHAT_DATE ('26 Jul 2024', scripts/4c1/run_4c1_sglang.sh:87; memory feedback_chat_template_date_drifts_texts_across_days).
- SGLang as carrick, not root. A root run leaves root-owned flashinfer JIT locks in
~carrick/.cache/flashinfer that crash later carrick runs; the serve driver chowns them back after every round (serve_cfg_drv.sh:57; memory feedback_talon2_sglang_as_carrick_not_root).
- fa4 tree-draft defect. FA4 paged-KV non-causal attention is wrong/nondeterministic on the H200; use
--speculative-draft-attention-backend fa3 (makes d11 deterministic, F3T_CONFIG_SWEEP_20261109.md:21-25; memory project_gpu_fa4_tree_draft_defect).
- 1-token requests crash the harness.
engine_test.py:286 reads meta_info["spec_verify_ct"], absent when no verify ran (TRACK1_RECORD_F3T_20261109/PER_PROMPT_BENCH_20261109.md:39). Use max_tokens >= 2.
- Hardcoded REPO paths.
run_4c1_nstream.sh hardcoded the talon2 kit copy sibw/repo and refused F3t; now REPO=${EAGLE4_HP_REPO:-…} (scripts/4c1/run_4c1_nstream.sh:19, commit 1753f9c37). The box1 copies of run_4c1_sglang.sh:28, run_4c1_sglang_qwen.sh:20 and demo/demo_4c1.sh:32 still hardcode box1's repo path; the talon2 kit copies of the two serving launchers read EAGLE4_HP_REPO. Set EAGLE4_HP_REPO to the kit you mean to run.
- MEASURE-ONLY stamp. The serving launcher runs with
MEASURE_ONLY_GOLDEN_DEPTH_UNPROVEN=1 and stamps every line (run_4c1_sglang.sh:79-85): the bf16 GPU golden at depth >= 1 prints FAIL (2-bit vs bf16); correctness is carried by level identity vs the previous image and text agreement with GPU-only (75/80 at d11 fa3) (EVALUATION.md:21-26, F3T_CONFIG_SWEEP_20261109.md:21-25).
- **Wide trees need
EAGLE4_4C1_ALLOW_WIDE=1 in the launcher's own env**, or it refuses with rc 61 (F3T_CONFIG_SWEEP_20261109.md:87-91; run_4c1_sglang.sh:48-51).
Calculations and hypothesis
- [derived] Step time at d11 fa3 = 19,615 tokens / 883.12 tok/s / 2,843 steps = 7.81 ms; at d7 7.12 ms (
F3T_CONFIG_SWEEP_20261109.md:206-207).
- [derived] Host-to-card data per step today ~ one 12,288 B row per ATT card + K/V import; at the measured 1.85 GB/s QDMA rate a 12,288 B row is ~6.6 µs, small next to the 3.4 ms tree.
- [derived] Verifier -> drafter payload per accepted token: Llama 18,432 B (3 x 3,072 fp16), Qwen 30,720 B (3 x 5,120 bf16); at accept 6.90 about 127 KB (Llama) per step. At 12.48 GB/s one QSFP lane carries that in ~10 µs.
- [hypothesis] Option (a) with the card-only K/V path should keep accept (measured unchanged on E2b) and cost about the measured KVX/KVX_PROMPT step increase (+1.3 ms at d7) unless the row build and K/V items are overlapped with the target's verify.
- [hypothesis] Option (b) removes the PCIe/host hops but needs on-card replacements for host-side work (fc, loop arming, TREE_STOP verdicts, SoA cross-check); its gain is bounded by today's host share of the step (step 7.81 ms minus card 3.38 ms minus GPU verify 3.45 ms ≈ 1.0 ms at d11).