4-card drafter: how to drive it from another verifier (one page, 2026-11-09)
What it is. Four AMD V80 cards on the talon2 server draft a whole speculative-decoding token tree per step for a target LLM (Llama-3.2-3B or Qwen3.8-27B). Two "ATT" cards split the attention heads (16 each); two "MLPV" cards run the MLP, the vocabulary head and the tree search. A Python host process on talon2 drives them over PCIe; today an H200 GPU verifies. Measured on talon2 2026-11-09: 883 tok/s at accept 6.90 (Llama, depth 11, width 2); the cards take 3.4 ms per depth-11 tree.
verifier --(PCIe/host)--> host process --PCIe--> ATT 17 <--QSFP--> ATT 3d
^ |QSFP |QSFP
+------- tree (tokens, parents, scores) <-- MLPV 63 <--QSFP--> MLPV 50
Each step, per request (stream). All integers little-endian.
| direction | field | size Llama-3.2-3B | size Qwen3.8-27B | dtype |
|---|---|---|---|---|
| verifier -> drafter | last accepted token id (new tree root) | 1 id | 1 id | int (vocab 128,256 / 248,320) |
| verifier -> drafter | target hidden states of that token: 3 layers | 3 x 3,072 | 3 x 5,120 (layers 3, 31, 59) | fp16 / bf16 |
| verifier -> drafter | same pair (token + 3 hidden) for every other accepted token | per token as above | per token as above | as above |
| verifier -> drafter | context length, tree depth, width | 3 ints | 3 ints | int |
| host -> cards (internal) | embedding + fused hidden of the root | 12,288 B | 12,288 B | 2 x 3,072 int16 fixed point |
| drafter -> verifier | tree nodes: token, parent, score | W + W^2(D-1) nodes + root (43 at width 2, depth 11) | same | int64 token, int parent index, float32 probability |
The host turns the 3 hidden layers into the card row (embedding lookup + one matrix multiply, fc). Today the GPU also computes the small drafter's KV cache; a card-only mode that builds it from the same rows is measured on the cards with the same accept and 14-25 % lower speed.
Two ways to connect.
- (a) Through the talon2 host process (software only; the card side exists today). The cards already sit behind a card-server process with a request/reply API (
stage_cfg_launch,kvonly_rows, ...), used today by the GPU-side client over a local socket. To build: a network front end, and thefcstep + Qwen token-id map moved off the GPU. - (b) Direct QSFP from the target cards to the drafter cards (not built). The cards speak a 4-cage QSFP link layer with spare cages and one idle link engine on each ATT card, but no frame format for "verifier -> drafter" exists in RTL yet.
Driver files to read.
- [
scripts/4c1/c4_cardproc.py](../../scripts/4c1/c4_cardproc.py): the card-server process and its request API (the seam for option a). - [
scripts/4c1/c4_loop.py](../../scripts/4c1/c4_loop.py): one tree on four cards, register by register. - [
scripts/4c1/c4_regmap.py](../../scripts/4c1/c4_regmap.py): every register, with the RTL line that decodes it. - [
scripts/4c1/c4_tree.py](../../scripts/4c1/c4_tree.py): the tree result layout. - [
scripts/fpga_tree_marshaller.py](../../scripts/fpga_tree_marshaller.py): how the root row is built and how the tree becomes token/parent/score arrays.
Full guide (access, every register, timings, run commands, traps): [DRAFTER_INTERFACE_GUIDE_4C1_20261109.md](DRAFTER_INTERFACE_GUIDE_4C1_20261109.md).