fpga-archive-demo-talon2-20261110 · EAGLE4 FPGA archive
talon demo guide: what we demo, which images, how to serve (2026-10-21)
One file for the talon demo. It covers the message to send, the images, the repo layout to put on talon, and the serve commands for each model.
Every md5 below was recomputed on box1. Sources are cited per row. No password or token is in this file: the sudo password file is called PWF everywhere.
Measured numbers are mt_bench 50 x 32 unless the row says HumanEval.
BEST WE CAN SHOW (lead with these two)
| Rank | Demo | What it proves | Measured (box1) | Ready? |
|---|---|---|---|---|
| 1 | A: Llama-3.2-3B on H200 + 2-bit EAGLE4 drafter on 4 V80s | FPGAs beat the H200's own drafting, end to end, on real serving | 513.20 tok/s @ accept 3.86 (7-round mean; best 4-round mean 524.9) vs GPU-only 494.7: +3.7% (best +6.1%), mt_bench 50 × 32, w2 d7 VTOK 20 | Yes: the box1 record, repeatable |
| 2 | B: Qwen3.8-27B + our BitNet EAGLE drafter (GPU-only today) | Our drafter out-drafts Qwen's native MTP head on code | HumanEval 80 × 256: 197.68 tok/s @ 7.40 vs MTP 164.64 @ 5.61 (+20% tok/s, +32% accept); target alone 29.61, i.e. 6.7× | Yes on GPU; on FPGA once 0xE4B7129B lands |
Use demo A as the FPGA headline and demo B as the HumanEval headline.
Do not claim:
- An FPGA HumanEval number: none measured yet.
- Qwen-on-FPGA numbers: not on silicon yet.
- A large FPGA speedup over GPU: it is +3.7–6.1%, because the H200 verify is the floor.
0. The message
Hi Yana Baskakova, here's what we can demo on HumanEval. A few items are demo-ready now, and I'll share an example run by tonight.
The best we can show: (1) on FPGA, our 4-card drafter makes Llama 3B serving faster than the H200 alone, at 513 tok/s vs 495 tok/s; (2) on HumanEval, our Qwen3.8-27B EAGLE drafter beats Qwen's native MTP, at 197.7 vs 164.6 tok/s (accept 7.40 vs 5.61).
H200 + 4-card companion demo: the drafter runs at 10,000 tps, 2× the speed of the H200's own drafting. End to end it is slightly better than the H200 alone at our best parameters (depth 7), reaching 500 tps on the Llama 3B model.
Qwen EAGLE drafter (Qwen3.8-27B target): under research and testing. On the H200 alone, our drafter already accepts more on HumanEval than Qwen's native MTP: 7.40 vs 5.61 tokens per verify step (197.68 vs 164.64 tok/s, 80 prompts × 256 tokens). The FPGA image for it is in its final builds, and the silicon test is staged to run as soon as it lands.
Llama 8B on talon, target-only demo: ready to be served. I'll share details tonight after I clean up the latest image version.
Target model with EAGLE fully on FPGA: still in active image builds and testing.
DeepSeek: still pending image compilation.
Repo layout, images and serve commands: TALON_DEMO_GUIDE_20261021.md.
1. Demo items and status
| # | Demo | Cards | State | Measured on box1 | Source |
|---|---|---|---|---|---|
| A | Llama-3.2-3B target on H200 + 2-bit EAGLE4 drafter on 4 V80s (4C1) | 4 | Ready (box1 record) | 513.20 tok/s mean @ accept 3.86 (7 rounds, w2 d7 VTOK 20; median 517.20; best 4-round mean 524.9) | reports/results/FOURCARD_DESIGN_20261016/WAVEB_MEASURE_20261020.md |
| A0 | Same target, GPU only (the baseline A is compared with) | 0 | Ready | 494.7 tok/s (fa3) / 466-470 (fa4), same shape | /sanctuary/pdi_inbox/BOX1_TALON2_4C1_ANSWER_20261020.md §4 |
| B | Qwen3.8-27B target + our BitNet/SubLN EAGLE3 drafter, GPU only | 0 | Ready | mt_bench w4 d5 V24 110.19 @ 3.65 (MTP d5 110.97 @ 3.49; target alone 56.73). HumanEval 80 x 256: w4 d7 V32 197.68 @ 7.40 (MTP 164.64 @ 5.61; target alone 29.61) | reports/results/QWEN27B_DRAFTER_20261020/SERVE_ACCEPT.md §4.1, §4.5 |
| C | Qwen3.8-27B target + the same drafter on 4 V80s | 4 | Not yet: ATT image 0xE4B7129B still routing; host side ready on the mock | none on silicon yet | reports/results/QWEN27B_DRAFTER_20261020/QWEN_SILICON_RUNBOOK_20261021.md |
| D | Llama 8B target-only on talon | ? | Ready per the user; details to be filled in tonight | — | (user) |
| E | Target model with EAGLE fully on FPGA | 8 | Image builds and testing | — | RING8 track |
| F | DeepSeek | — | Image compilation pending | — | — |
HumanEval caveats:
- Demo A on HumanEval: no 4C1 HumanEval capture exists yet. The card context window (CTX_WIN 1024) bounds max-new, so run HumanEval with
MAXREQ=20 MAXNEW=128(FOURCARD_DESIGN_20261016/ADAPTIVE_TREE_20261020.md:203). The 500 tok/s and 10,000 tps figures are mt_bench. - Demo B at 256 tokens is thermally throttled on box1's H200 (345-1700 MHz at 87-95 C;
SERVE_ACCEPT.md§6). The tok/s is a floor; the accept is unaffected.
2. Images (final FPT PDIs only; first 4 bytes 16 a5 f7 92)
On box1 unless stated. Both cards of a pair get the same PDI; the rank (0/1) is a runtime register.
| Demo | Role (cards on box1) | Label | File | md5 | Bytes | Timing (setup / hold) | Use |
|---|---|---|---|---|---|---|---|
| A | ATT pair (01, 42) | 0xE4B71288 | ~/eagle4_artifacts/uni_oneflip/4c1_att_v11b_0xE4B71288_amd_v80_gen5x8_25.1.pdi |
1b449de356f8531220895bca5d26cc78 |
51,666,640 | released | flash for A |
| A, C | MLPV pair (62 = rank 0, 61 = rank 1) | 0xE4B7128B r2 | ~/eagle4_artifacts/uni_oneflip/thin_foTH_M8B_mg_0xE4B7128B_f4f3f745.pdi |
f4f3f745dedf280b58ae334d407fdc71 |
80,587,296 | +0.030 / +0.010 | flash for A and C (the MLPV needs no respin for Qwen) |
| A | MLPV fallback | 0xE4B71287 | ~/eagle4_artifacts/uni_oneflip/4c1_mlpv_allfix_0xE4B71287_amd_v80_gen5x8_25.1.pdi |
b2333158198fe3e525a614350c146360 |
61,141,744 | released | fallback only |
| A (dual-model) | ATT, Llama + Qwen registers, Q1 build | 0xE4B7129A | /sanctuary/pdi_inbox/QWEN_DUALATT_20261021/oneflip_p2s_head_asgard_0xE4B7129A_66m_9b23b3d6.pdi |
9b23b3d634c9b5c1eb030e8423a1fe14 |
53,435,648 | +0.040 / +0.013 | Llama only; QUARANTINED (45 report-only FAILs, adjudication pending) |
| A (dual-model) | same, margin re-route | 0xE4B7129A | /sanctuary/pdi_inbox/QWEN_DUALATT_20261021/thin_foTH_A9A_mg_0xE4B7129A_b44b4a25.pdi |
b44b4a25197fa8e4321c2ccefa89078d |
53,439,520 | +0.047 / +0.013 | Llama only; donor firmware, re-check at preflash |
| C | ATT, Qwen-correct dual-model (Q2) | 0xE4B7129B | not built yet: drops into ~/eagle4_artifacts/uni_oneflip/qwen_dualatt/ |
— | — | — | the only ATT image that runs Qwen |
Never flash:
impl_1/top_wrapper.pdior any nofpt PDI: no AMC firmware, it bricks management.- 0xE4B7128B r1 (
b7012211…): superseded at WNS +0.001. - Any image whose setup or hold is negative.
- 0x129A in Qwen mode: it has no SubLN and has the e_norm sign-wrap, so it is not the trained drafter.
3. Repo layout on talon
Proposed root ~/eagle4_talon_demo/. Copy files only, read-only and nice'd from box1. Never copy box1's /dev/qdma* or anything on the cards.
~/eagle4_talon_demo/
README.md <- this file
images/ PDIs of section 2 + their .GATE_VERDICTS.txt / .QUARANTINED / PREFLASH_RECEIPT
repo/ WORKING TREE of cleanup_EAGLE scripts/ (scripts/4c1 has uncommitted edits the record run uses)
kits/4c1_boot/ Llama card weights: 0xE4B71288, 0xE4B7128B, 0xE4B7129B(Llama dirs), _shared/embfold_att_rank{0,1}
kits/4c1_boot_qwen27b/v3_ropestep256_enormk/ Qwen card weights for 0xE4B7129B + 0xE4B7128B (KIT_V3_129A_129B.json md5 2f76ad5a)
kits/4c1_golden_kvi/, kits/4c1_rtl_ref/ golden inputs
models/Llama-3.2-3B-Instruct/ target for A
models/CostDraftTree-3B2bit/ the 2-bit drafter tree for A (bitnet_721 bf16 ckpt md5 5a22603b6cfb36d8dfb8b9d49b2511cc; must match talon's copy in all 32 hex characters)
models/qwen3_8_27b/ target for B/C, rev 1d4bf0f2 (18 shards, 52 GB; per-file md5 + sha256 in qwen3_8_27b_md5_sha256.txt)
models/qwen27b_drafter_sglang/step21491_bf16ste/ converted drafter, model.safetensors md5 32f20d0d1c9d86fb5afb90088c8c5f02
sglang/ eaglefit SGLang 0.5.16 source (git 278a159d) + engine_test.py working copy
flash_records/ talon-specific records (rewrite the BDF keys; keep the pdi_md5 values)
What to copy, with md5s:
- Demo A:
/sanctuary/pdi_inbox/BOX1_TALON2_MANIFEST_20261020.mdsections 1-6 (about 21.7 GB), md5s inBOX1_TALON2_MD5SUMS_20261020.txt(1,499 lines). - Demos B/C, in addition:
~/eagle4_artifacts/4c1_boot_qwen27b/;/mnt/data/models/qwen3_8_27b/and/mnt/data/models/qwen27b_drafter_sglang/;scripts/qwen27b_drafter_20261020/, includingsubln_shim/andcdt_gdn_site/;- the new Llama dirs
~/eagle4_artifacts/4c1_boot/0xE4B7129B/.
Copy rules:
- Use
rsync -L: the boot streams symlink into~/eagle4_artifacts/osprey/boot_streams/. - Re-run
md5sum -con talon before any flash.
Access: box1 reaches talon only through asgard's cloudflared ssh (reference_talon_8fpga_server). talon-2 ssh details are pending from HQ operations (asked in /sanctuary/lab/TO_HQ.md 2026-10-21). With ssh, box1 can do sections 4-6 itself; recovering cards over JTAG and cabling are physical steps.
4. talon host setup (once)
From /sanctuary/pdi_inbox/BOX1_TALON2_SETUP_STEPS_20261020.md. box1's versions are given in each step.
- Cards: connect USB JTAG to all 4 V80s and recover NO_AMC: they show UUID f92f252f NO_AMC today. Load any known-good AMC image first.
- QSFP ring: ATT0 c3→ATT1 c2, ATT1 c3→MLPV1 c2, MLPV1 c3→MLPV0 c2, MLPV0 c3→ATT0 c2. Cage 1 is the idle diagonal; cage 0 is empty. Role map:
scripts/4c1/c4_bringup.py:19-31, which refuses any other map. - Host stack:
- Ubuntu kernel 6.8.
- NVIDIA driver 590.48 / CUDA 13.1. talon reports CUDA 12.6: upgrade the driver or rebuild the env for cu12.6, then re-measure.
- conda env: python 3.12.13, torch 2.11.0+cu130, flashinfer 0.6.12.
- QDMA 2024.1 driver from
dma_ip_drivers, with 10ee:50b4/50b5 added topci_ids.h. - ami_tool 2.4.0.
- BDFs: edit for talon's slots in
scripts/4c1/c4_bar.py:51(CARD_BDFS, plus about 9 references),c4_bringup.pyROLE_OF_BUS / PAIR_PARTNER / TP_PARTNER, and the flash records. Re-derive each BDF from its serial withami_tool overview. - GPU-only check first: talon's GPU-only Llama-3B ran 309.0 tok/s against box1's 494.7. Before comparing, check:
- the draft attention backend (fa3 or fa4);
- CUDA graphs are on;
- the ckpt md5;
EAGLE4_CHAT_DATE='26 Jul 2024';- GPU clocks (
nvidia-smi -q -d CLOCK).
5. Flash and bring-up (A and C)
- Preflash:
echo … | sudo -p "" env PCI_DEV=<bdf> PREFLASH_ACK_QUARANTINE=1 bash scripts/preflash_gate.sh <pdi> <md5>. Pass is rc 0, or rc 3 with UNVER only and 0 FAIL. - Flash:
-p 0ONLY, detached. Use the pattern ofscripts/4c1/flash_4c1_waveB_20261019.sh(A) orflash_4c1_waveC_20261020.sh <pdi> <md5> ATT_0x129B(C).- It is done when the log shows "Image programming complete" and
FLASH_EXIT=0. - Then a cold AC, which needs the user's words.
- It is done when the log shows "Image programming complete" and
- Bring-up:
FLASH_RECORD=<talon record> RUN=<run dir> PWF=<pw file> DEPTH=7 bash scripts/4c1/post_coldac_4c1.sh. Do not wrap it in card_run.sh; it locks per step.- It runs the SMI/AER mask, QDMA × 4, discover, link, boot, fold, arm and golden.
- Pass: every link LINKED and 0 MISSING.
- If a cable shows no peer on both ends after a cold boot, another cold AC retrained it on box1; link-local GT resets did not.
6. Serve commands
All runs: greedy, batch 1, EAGLE4_CHAT_DATE pinned. Always state the regime (bench, rounds, depth, width, VTOK, SGLang version) next to any number.
A. Llama-3.2-3B + 4-card drafter (record default)
RUN=<run dir from section 5> PWF=<pw file> MEASURE_ONLY_GOLDEN_DEPTH_UNPROVEN=1 bash scripts/4c1/run_4c1_sglang.sh
# defaults = CUDAPACK + Y_DEFER + D1 + GLUE_MMIO + VERIFY_PRE + D1HOST_LEANATTN, TREE_STOP theta -2.5, w2 d7 VTOK 20, KC 64
# box1 reference: 513-525 tok/s @ 3.86 (mt_bench 50x32)
# HumanEval variant (no 4C1 HumanEval number exists yet; CTX_WIN 1024 bounds max-new):
BENCH=$HOME/haochen/eaglefit/inference/dataset/humaneval/question.jsonl MAXREQ=20 MAXNEW=128 \
RUN=<run dir> PWF=<pw file> MEASURE_ONLY_GOLDEN_DEPTH_UNPROVEN=1 bash scripts/4c1/run_4c1_sglang.sh
A0. GPU-only baseline (same target and drafter, no cards): SGLang exec-mode -1, same shape. The commands are in /sanctuary/pdi_inbox/BOX1_TALON2_4C1_ANSWER_20261020.md §3-4 and docs/GPU_ACCEPT_GOLDEN_COMMAND.md.
B. Qwen3.8-27B + our drafter, GPU only
# mt_bench, best measured shape (110.19 @ 3.65)
TAG=q27_w4d5v24 MODE=eagle3 BENCH=mt_bench MAXREQ=50 MAXNEW=32 DEPTH=5 WIDTH=4 VTOK=24 BACKEND=fa3 \
bash scripts/qwen27b_drafter_20261020/run_cell.sh
# HumanEval, best accept (197.68 tok/s @ 7.40, 80 x 256)
TAG=q27_w4d7v32_he MODE=eagle3 BENCH=humaneval MAXREQ=80 MAXNEW=256 DEPTH=7 WIDTH=4 VTOK=32 BACKEND=fa3 \
bash scripts/qwen27b_drafter_20261020/run_cell.sh
run_cell.shdrives~/haochen/eaglefit/inference/scripts/engine_test.py: SGLang 0.5.16 eaglefit,--speculative-exec-mode -1,--enable-cuda-graph, conda env eaglefit, mem-fraction capped at 0.88.- The SubLN shim (
scripts/qwen27b_drafter_20261020/subln_shim/sitecustomize.py) must be onPYTHONPATH. Stockllama_eagle3has no SubLN. - Native-MTP comparison:
--speculative-model NEXTNwith no draft path. - GPU memory: target 51.05 GB + draft 2.65 GB + KV pools; the peak was 138.9 GB of 143 GB at mem-fraction 0.88.
C. Qwen3.8-27B + drafter on 4 V80s (after 0xE4B7129B lands)
- Follow
reports/results/QWEN27B_DRAFTER_20261020/QWEN_SILICON_RUNBOOK_20261021.mdsteps 1-12:- fill the flash record;
- preflash, then flash 01/42;
- cold AC, then
post_coldac_4c1.sh; - Llama back-compat (texts md5 fefeff71, accept 3.86);
- read the model-config registers (
scripts/4c1/rd_model_cfg.py); c4.py --model qwen27b switch, thenarm --depth 7;- the Qwen golden;
- Qwen serving through
scripts/4c1/run_4c1_sglang_qwen.sh: w2 d7 V20 first, w4 d5 V24 last.
- Required host fixes, all committed:
cdt_gdn_site/(9748e2a3): without it SGLang's FPGA verify path gives wrong text on the Qwen hybrid target, even GPU-only;- the fc_norm chain (d4e47ce6);
- model-tagged goldens (3eae843d);
- the MODEL_CFG switch on dual-model images only (28325ed6).
- Qwen mode runs with EMBFOLD off (
FOLD=0). - Width 4 wedged the Llama 4C1 run twice; a wedge needs a cold AC.
D. Llama 8B target-only on talon: (to be filled in by the user tonight: image id + md5, cards, serve command, measured number.)
7. Hard rules on any card host
- Flash
-p 0only. Never-p 1, FLR, SBR, device_boot or PCIe remove: these corrupt QDMA until a cold AC. - QDMA cdev: plain
pread/pwriteonly.preadv/readv/aio froze box1 with a kernel BUG. - PERSIST images latch the tree-depth ABI at the first AP_START after boot: arm the deepest depth you will use first.
- Never put a password or token in a file. Never kill a process you did not start. Never remove files on a shared host.