fpga-archive-demo-talon2-20261110 · EAGLE4 FPGA archive

Source reports/results/TALON_DEMO_GUIDE_20261021.md · source md5 7b4a14e25ee7ae171dbf91cbe78b00f1 · archived 2026-11-18

Download PDFOpen PDFRaw markdown

talon demo guide: what we demo, which images, how to serve (2026-10-21)

One file for the talon demo. It covers the message to send, the images, the repo layout to put on talon, and the serve commands for each model. Every md5 below was recomputed on box1. Sources are cited per row. No password or token is in this file: the sudo password file is called PWF everywhere. Measured numbers are mt_bench 50 x 32 unless the row says HumanEval.

BEST WE CAN SHOW (lead with these two)

Rank Demo What it proves Measured (box1) Ready?
1 A: Llama-3.2-3B on H200 + 2-bit EAGLE4 drafter on 4 V80s FPGAs beat the H200's own drafting, end to end, on real serving 513.20 tok/s @ accept 3.86 (7-round mean; best 4-round mean 524.9) vs GPU-only 494.7: +3.7% (best +6.1%), mt_bench 50 × 32, w2 d7 VTOK 20 Yes: the box1 record, repeatable
2 B: Qwen3.8-27B + our BitNet EAGLE drafter (GPU-only today) Our drafter out-drafts Qwen's native MTP head on code HumanEval 80 × 256: 197.68 tok/s @ 7.40 vs MTP 164.64 @ 5.61 (+20% tok/s, +32% accept); target alone 29.61, i.e. 6.7× Yes on GPU; on FPGA once 0xE4B7129B lands

Use demo A as the FPGA headline and demo B as the HumanEval headline.

Do not claim:

0. The message

Hi Yana Baskakova, here's what we can demo on HumanEval. A few items are demo-ready now, and I'll share an example run by tonight.

The best we can show: (1) on FPGA, our 4-card drafter makes Llama 3B serving faster than the H200 alone, at 513 tok/s vs 495 tok/s; (2) on HumanEval, our Qwen3.8-27B EAGLE drafter beats Qwen's native MTP, at 197.7 vs 164.6 tok/s (accept 7.40 vs 5.61).

H200 + 4-card companion demo: the drafter runs at 10,000 tps, 2× the speed of the H200's own drafting. End to end it is slightly better than the H200 alone at our best parameters (depth 7), reaching 500 tps on the Llama 3B model.

Qwen EAGLE drafter (Qwen3.8-27B target): under research and testing. On the H200 alone, our drafter already accepts more on HumanEval than Qwen's native MTP: 7.40 vs 5.61 tokens per verify step (197.68 vs 164.64 tok/s, 80 prompts × 256 tokens). The FPGA image for it is in its final builds, and the silicon test is staged to run as soon as it lands.

Llama 8B on talon, target-only demo: ready to be served. I'll share details tonight after I clean up the latest image version.

Target model with EAGLE fully on FPGA: still in active image builds and testing.

DeepSeek: still pending image compilation.

Repo layout, images and serve commands: TALON_DEMO_GUIDE_20261021.md.

1. Demo items and status

# Demo Cards State Measured on box1 Source
A Llama-3.2-3B target on H200 + 2-bit EAGLE4 drafter on 4 V80s (4C1) 4 Ready (box1 record) 513.20 tok/s mean @ accept 3.86 (7 rounds, w2 d7 VTOK 20; median 517.20; best 4-round mean 524.9) reports/results/FOURCARD_DESIGN_20261016/WAVEB_MEASURE_20261020.md
A0 Same target, GPU only (the baseline A is compared with) 0 Ready 494.7 tok/s (fa3) / 466-470 (fa4), same shape /sanctuary/pdi_inbox/BOX1_TALON2_4C1_ANSWER_20261020.md §4
B Qwen3.8-27B target + our BitNet/SubLN EAGLE3 drafter, GPU only 0 Ready mt_bench w4 d5 V24 110.19 @ 3.65 (MTP d5 110.97 @ 3.49; target alone 56.73). HumanEval 80 x 256: w4 d7 V32 197.68 @ 7.40 (MTP 164.64 @ 5.61; target alone 29.61) reports/results/QWEN27B_DRAFTER_20261020/SERVE_ACCEPT.md §4.1, §4.5
C Qwen3.8-27B target + the same drafter on 4 V80s 4 Not yet: ATT image 0xE4B7129B still routing; host side ready on the mock none on silicon yet reports/results/QWEN27B_DRAFTER_20261020/QWEN_SILICON_RUNBOOK_20261021.md
D Llama 8B target-only on talon ? Ready per the user; details to be filled in tonight — (user)
E Target model with EAGLE fully on FPGA 8 Image builds and testing — RING8 track
F DeepSeek — Image compilation pending — —

HumanEval caveats:

2. Images (final FPT PDIs only; first 4 bytes 16 a5 f7 92)

On box1 unless stated. Both cards of a pair get the same PDI; the rank (0/1) is a runtime register.

Demo Role (cards on box1) Label File md5 Bytes Timing (setup / hold) Use
A ATT pair (01, 42) 0xE4B71288 ~/eagle4_artifacts/uni_oneflip/4c1_att_v11b_0xE4B71288_amd_v80_gen5x8_25.1.pdi 1b449de356f8531220895bca5d26cc78 51,666,640 released flash for A
A, C MLPV pair (62 = rank 0, 61 = rank 1) 0xE4B7128B r2 ~/eagle4_artifacts/uni_oneflip/thin_foTH_M8B_mg_0xE4B7128B_f4f3f745.pdi f4f3f745dedf280b58ae334d407fdc71 80,587,296 +0.030 / +0.010 flash for A and C (the MLPV needs no respin for Qwen)
A MLPV fallback 0xE4B71287 ~/eagle4_artifacts/uni_oneflip/4c1_mlpv_allfix_0xE4B71287_amd_v80_gen5x8_25.1.pdi b2333158198fe3e525a614350c146360 61,141,744 released fallback only
A (dual-model) ATT, Llama + Qwen registers, Q1 build 0xE4B7129A /sanctuary/pdi_inbox/QWEN_DUALATT_20261021/oneflip_p2s_head_asgard_0xE4B7129A_66m_9b23b3d6.pdi 9b23b3d634c9b5c1eb030e8423a1fe14 53,435,648 +0.040 / +0.013 Llama only; QUARANTINED (45 report-only FAILs, adjudication pending)
A (dual-model) same, margin re-route 0xE4B7129A /sanctuary/pdi_inbox/QWEN_DUALATT_20261021/thin_foTH_A9A_mg_0xE4B7129A_b44b4a25.pdi b44b4a25197fa8e4321c2ccefa89078d 53,439,520 +0.047 / +0.013 Llama only; donor firmware, re-check at preflash
C ATT, Qwen-correct dual-model (Q2) 0xE4B7129B not built yet: drops into ~/eagle4_artifacts/uni_oneflip/qwen_dualatt/ — — — the only ATT image that runs Qwen

Never flash:

3. Repo layout on talon

Proposed root ~/eagle4_talon_demo/. Copy files only, read-only and nice'd from box1. Never copy box1's /dev/qdma* or anything on the cards.

~/eagle4_talon_demo/
  README.md                      <- this file
  images/                        PDIs of section 2 + their .GATE_VERDICTS.txt / .QUARANTINED / PREFLASH_RECEIPT
  repo/                          WORKING TREE of cleanup_EAGLE scripts/ (scripts/4c1 has uncommitted edits the record run uses)
  kits/4c1_boot/                 Llama card weights:  0xE4B71288, 0xE4B7128B, 0xE4B7129B(Llama dirs), _shared/embfold_att_rank{0,1}
  kits/4c1_boot_qwen27b/v3_ropestep256_enormk/   Qwen card weights for 0xE4B7129B + 0xE4B7128B (KIT_V3_129A_129B.json md5 2f76ad5a)
  kits/4c1_golden_kvi/, kits/4c1_rtl_ref/        golden inputs
  models/Llama-3.2-3B-Instruct/  target for A
  models/CostDraftTree-3B2bit/   the 2-bit drafter tree for A (bitnet_721 bf16 ckpt md5 5a22603b6cfb36d8dfb8b9d49b2511cc; must match talon's copy in all 32 hex characters)
  models/qwen3_8_27b/            target for B/C, rev 1d4bf0f2 (18 shards, 52 GB; per-file md5 + sha256 in qwen3_8_27b_md5_sha256.txt)
  models/qwen27b_drafter_sglang/step21491_bf16ste/   converted drafter, model.safetensors md5 32f20d0d1c9d86fb5afb90088c8c5f02
  sglang/                        eaglefit SGLang 0.5.16 source (git 278a159d) + engine_test.py working copy
  flash_records/                 talon-specific records (rewrite the BDF keys; keep the pdi_md5 values)

What to copy, with md5s:

Copy rules:

Access: box1 reaches talon only through asgard's cloudflared ssh (reference_talon_8fpga_server). talon-2 ssh details are pending from HQ operations (asked in /sanctuary/lab/TO_HQ.md 2026-10-21). With ssh, box1 can do sections 4-6 itself; recovering cards over JTAG and cabling are physical steps.

4. talon host setup (once)

From /sanctuary/pdi_inbox/BOX1_TALON2_SETUP_STEPS_20261020.md. box1's versions are given in each step.

  1. Cards: connect USB JTAG to all 4 V80s and recover NO_AMC: they show UUID f92f252f NO_AMC today. Load any known-good AMC image first.
  2. QSFP ring: ATT0 c3→ATT1 c2, ATT1 c3→MLPV1 c2, MLPV1 c3→MLPV0 c2, MLPV0 c3→ATT0 c2. Cage 1 is the idle diagonal; cage 0 is empty. Role map: scripts/4c1/c4_bringup.py:19-31, which refuses any other map.
  3. Host stack:
    • Ubuntu kernel 6.8.
    • NVIDIA driver 590.48 / CUDA 13.1. talon reports CUDA 12.6: upgrade the driver or rebuild the env for cu12.6, then re-measure.
    • conda env: python 3.12.13, torch 2.11.0+cu130, flashinfer 0.6.12.
    • QDMA 2024.1 driver from dma_ip_drivers, with 10ee:50b4/50b5 added to pci_ids.h.
    • ami_tool 2.4.0.
  4. BDFs: edit for talon's slots in scripts/4c1/c4_bar.py:51 (CARD_BDFS, plus about 9 references), c4_bringup.py ROLE_OF_BUS / PAIR_PARTNER / TP_PARTNER, and the flash records. Re-derive each BDF from its serial with ami_tool overview.
  5. GPU-only check first: talon's GPU-only Llama-3B ran 309.0 tok/s against box1's 494.7. Before comparing, check:
    • the draft attention backend (fa3 or fa4);
    • CUDA graphs are on;
    • the ckpt md5;
    • EAGLE4_CHAT_DATE='26 Jul 2024';
    • GPU clocks (nvidia-smi -q -d CLOCK).

5. Flash and bring-up (A and C)

6. Serve commands

All runs: greedy, batch 1, EAGLE4_CHAT_DATE pinned. Always state the regime (bench, rounds, depth, width, VTOK, SGLang version) next to any number.

A. Llama-3.2-3B + 4-card drafter (record default)

RUN=<run dir from section 5> PWF=<pw file> MEASURE_ONLY_GOLDEN_DEPTH_UNPROVEN=1 bash scripts/4c1/run_4c1_sglang.sh
#   defaults = CUDAPACK + Y_DEFER + D1 + GLUE_MMIO + VERIFY_PRE + D1HOST_LEANATTN, TREE_STOP theta -2.5, w2 d7 VTOK 20, KC 64
#   box1 reference: 513-525 tok/s @ 3.86 (mt_bench 50x32)
# HumanEval variant (no 4C1 HumanEval number exists yet; CTX_WIN 1024 bounds max-new):
BENCH=$HOME/haochen/eaglefit/inference/dataset/humaneval/question.jsonl MAXREQ=20 MAXNEW=128 \
RUN=<run dir> PWF=<pw file> MEASURE_ONLY_GOLDEN_DEPTH_UNPROVEN=1 bash scripts/4c1/run_4c1_sglang.sh

A0. GPU-only baseline (same target and drafter, no cards): SGLang exec-mode -1, same shape. The commands are in /sanctuary/pdi_inbox/BOX1_TALON2_4C1_ANSWER_20261020.md §3-4 and docs/GPU_ACCEPT_GOLDEN_COMMAND.md.

B. Qwen3.8-27B + our drafter, GPU only

# mt_bench, best measured shape (110.19 @ 3.65)
TAG=q27_w4d5v24 MODE=eagle3 BENCH=mt_bench MAXREQ=50 MAXNEW=32 DEPTH=5 WIDTH=4 VTOK=24 BACKEND=fa3 \
  bash scripts/qwen27b_drafter_20261020/run_cell.sh
# HumanEval, best accept (197.68 tok/s @ 7.40, 80 x 256)
TAG=q27_w4d7v32_he MODE=eagle3 BENCH=humaneval MAXREQ=80 MAXNEW=256 DEPTH=7 WIDTH=4 VTOK=32 BACKEND=fa3 \
  bash scripts/qwen27b_drafter_20261020/run_cell.sh

C. Qwen3.8-27B + drafter on 4 V80s (after 0xE4B7129B lands)

D. Llama 8B target-only on talon: (to be filled in by the user tonight: image id + md5, cards, serve command, measured number.)

7. Hard rules on any card host