Qwen3.8-27B drafters: old vs new weights, per phase
Both lists printed on 2026-11-18 straight from the safetensors headers of the two checkpoint files. Old: Finn's BitNet drafter step21491 (bf16, ternary-trained), file md5 32f20d0d…, 2,692,524,624 B. New: GPTQ 2-bit W2A16 drafter, file md5 e41d1646…, 3,030,701,984 B.
Model size in bytes
Old = Finn's Qwen3.8-27B BitNet drafter, step 21491 (model.safetensors md5 32f20d0d…). New = GPTQ 2-bit W2A16 drafter (model.safetensors md5 e41d1646…). Names are the raw tensor names in each file.
| old tensor (Finn BitNet step21491) | old file bytes | new tensor(s) (GPTQ-2 W2A16) | new file bytes | new ÷ old | old bytes at 2 bit/weight | new bytes at 2 bit/weight | new ÷ old |
|---|---|---|---|---|---|---|---|
| embed_tokens.weight | 1,525,678,080 | embed_tokens.weight | 2,542,796,800 | 1.67× | 190,709,760 | 317,849,600 | 1.67× |
| fc.weight | 94,371,840 | fc.g_idx fc.qweight fc.qzeros fc.scales | 21,104,640 | 0.22× | 11,796,480 | 19,660,800 | 1.67× |
| fc_norm.0.weight | 10,240 | fc_norm.0.weight | 10,240 | 1.00× | 1,280 | 1,280 | 1.00× |
| fc_norm.1.weight | 10,240 | fc_norm.1.weight | 10,240 | 1.00× | 1,280 | 1,280 | 1.00× |
| fc_norm.2.weight | 10,240 | fc_norm.2.weight | 10,240 | 1.00× | 1,280 | 1,280 | 1.00× |
| midlayer.hidden_norm.weight | 6,144 | layers.0.hidden_norm.weight | 10,240 | 1.67× | 768 | 1,280 | 1.67× |
| midlayer.input_layernorm.weight | 6,144 | layers.0.input_layernorm.weight | 10,240 | 1.67× | 768 | 1,280 | 1.67× |
| midlayer.self_attn.q_proj.weight | 37,748,736 | layers.0.self_attn.q_proj.g_idx layers.0.self_attn.q_proj.qweight layers.0.self_attn.q_proj.qzeros layers.0.self_attn.q_proj.scales | 14,069,760 | 0.37× | 4,718,592 | 13,107,200 | 2.78× |
| midlayer.self_attn.k_proj.weight | 37,748,736 | layers.0.self_attn.k_proj.g_idx layers.0.self_attn.k_proj.qweight layers.0.self_attn.k_proj.qzeros layers.0.self_attn.k_proj.scales | 14,069,760 | 0.37× | 4,718,592 | 13,107,200 | 2.78× |
| midlayer.self_attn.v_proj.weight | 37,748,736 | layers.0.self_attn.v_proj.g_idx layers.0.self_attn.v_proj.qweight layers.0.self_attn.v_proj.qzeros layers.0.self_attn.v_proj.scales | 14,069,760 | 0.37× | 4,718,592 | 13,107,200 | 2.78× |
| midlayer.self_attn.attn_sub_norm.weight | 6,144 | (none) | 0 | — | 768 | 0 | — |
| midlayer.self_attn.o_proj.weight | 18,874,368 | layers.0.self_attn.o_proj.g_idx layers.0.self_attn.o_proj.qweight layers.0.self_attn.o_proj.qzeros layers.0.self_attn.o_proj.scales | 7,034,880 | 0.37× | 2,359,296 | 6,553,600 | 2.78× |
| midlayer.post_attention_layernorm.weight | 6,144 | layers.0.post_attention_layernorm.weight | 10,240 | 1.67× | 768 | 1,280 | 1.67× |
| midlayer.mlp.gate_proj.weight | 50,331,648 | layers.0.mlp.gate_proj.g_idx layers.0.mlp.gate_proj.qweight layers.0.mlp.gate_proj.qzeros layers.0.mlp.gate_proj.scales | 22,466,560 | 0.45× | 6,291,456 | 20,971,520 | 3.33× |
| midlayer.mlp.up_proj.weight | 50,331,648 | layers.0.mlp.up_proj.g_idx layers.0.mlp.up_proj.qweight layers.0.mlp.up_proj.qzeros layers.0.mlp.up_proj.scales | 22,466,560 | 0.45× | 6,291,456 | 20,971,520 | 3.33× |
| midlayer.mlp.ffn_sub_norm.weight | 16,384 | (none) | 0 | — | 2,048 | 0 | — |
| midlayer.mlp.down_proj.weight | 50,331,648 | layers.0.mlp.down_proj.g_idx layers.0.mlp.down_proj.qweight layers.0.mlp.down_proj.qzeros layers.0.mlp.down_proj.scales | 22,511,616 | 0.45× | 6,291,456 | 20,971,520 | 3.33× |
| norm.weight | 6,144 | norm.weight | 10,240 | 1.67× | 768 | 1,280 | 1.67× |
| lm_head.weight | 788,004,864 | lm_head.g_idx lm_head.qweight lm_head.qzeros lm_head.scales | 340,218,880 | 0.43× | 98,500,608 | 317,849,600 | 3.23× |
| (none) | 0 | efficient_lm_head.g_idx efficient_lm_head.qweight efficient_lm_head.qzeros efficient_lm_head.scales efficient_lm_head_down_proj.weight | 9,816,192 | — | 0 | 8,110,080 | — |
| d2t | 1,026,048 | (none) | 0 | — | 0 | 0 | — |
| t2d | 248,320 | (none) | 0 | — | 0 | 0 | — |
| TOTAL | 2,692,522,496 | 3,030,697,088 | 1.13× | 336,406,016 | 772,268,800 | 2.30× |
File bytes = what each safetensors file stores (old: bf16; new: packed 2-bit + scales/qzeros/g_idx). Bytes at 2 bit/weight = weights × 2 ÷ 8 (a GPTQ qweight int32 holds 16 weights), the format the FPGA holds the drafter body in.
Per-phase summary
| phase | old weights | new weights | N× | on-card bytes old (2 bit/weight) | on-card bytes new (2 bit/weight) | N× | file bytes old | file bytes new | N× file |
|---|---|---|---|---|---|---|---|---|---|
| Token embedding (host) | 762,839,040 | 1,271,398,400 | 1.67× | 190,709,760 | 317,849,600 | 1.67× | 1,525,678,080 | 2,542,796,800 | 1.67× |
| fc input norms (3 target hiddens) | 15,360 | 15,360 | 1.00× | 3,840 | 3,840 | 1.00× | 30,720 | 30,720 | 1.00× |
| fc: fuse 3 target hiddens to H | 47,185,920 | 78,643,200 | 1.67× | 11,796,480 | 19,660,800 | 1.67× | 94,371,840 | 21,104,640 | 0.22× |
| G1 RMSNorm (input + hidden) | 6,144 | 10,240 | 1.67× | 1,536 | 2,560 | 1.67× | 12,288 | 20,480 | 1.67× |
| G4 Q/K/V projection (ATT) | 56,623,104 | 157,286,400 | 2.78× | 14,155,776 | 39,321,600 | 2.78× | 113,246,208 | 42,209,280 | 0.37× |
| G5 attention sub-norm (BitNet only) | 3,072 | 0 | removed | 768 | 0 | removed | 6,144 | 0 | removed |
| G6 o-projection (ATT) | 9,437,184 | 26,214,400 | 2.78× | 2,359,296 | 6,553,600 | 2.78× | 18,874,368 | 7,034,880 | 0.37× |
| R3 post-attention norm | 3,072 | 5,120 | 1.67× | 768 | 1,280 | 1.67× | 6,144 | 10,240 | 1.67× |
| G7 gate + up (ATT) | 50,331,648 | 167,772,160 | 3.33× | 12,582,912 | 41,943,040 | 3.33× | 100,663,296 | 44,933,120 | 0.45× |
| G7 FFN sub-norm (BitNet only) | 8,192 | 0 | removed | 2,048 | 0 | removed | 16,384 | 0 | removed |
| G9 down projection (MLPV) | 25,165,824 | 83,886,080 | 3.33× | 6,291,456 | 20,971,520 | 3.33× | 50,331,648 | 22,511,616 | 0.45× |
| G11 final norm (MLPV) | 3,072 | 5,120 | 1.67× | 768 | 1,280 | 1.67× | 6,144 | 10,240 | 1.67× |
| G12 cheap low-rank lm-head (MLPV) | 0 | 32,440,320 | new only | 0 | 8,110,080 | new only | 0 | 9,816,192 | new only |
| lm-head (vocabulary projection) | 394,002,432 | 1,271,398,400 | 3.23× | 98,500,608 | 317,849,600 | 3.23× | 788,004,864 | 340,218,880 | 0.43× |
| draft-to-target vocab maps | 0 | 0 | — | 0 | 0 | — | 1,274,368 | 0 | removed |
| Drafter body on the cards (G4 + G6 + G7 + G9) | 141,557,760 | 435,159,040 | 3.07× | 35,389,440 | 108,789,760 | 3.07× | |||
Every tensor, side by side
Old: Finn's BitNet step21491 (bf16)
| tensor (raw name) | dtype | shape | bytes |
|---|---|---|---|
| Token embedding (host) | |||
| embed_tokens.weight | BF16 | [248320, 3072] | 1,525,678,080 |
| fc input norms (3 target hiddens) | |||
| fc_norm.0.weight | BF16 | [5120] | 10,240 |
| fc_norm.1.weight | BF16 | [5120] | 10,240 |
| fc_norm.2.weight | BF16 | [5120] | 10,240 |
| fc: fuse 3 target hiddens to H | |||
| fc.weight | BF16 | [3072, 15360] | 94,371,840 |
| G1 RMSNorm (input + hidden) | |||
| midlayer.hidden_norm.weight | BF16 | [3072] | 6,144 |
| midlayer.input_layernorm.weight | BF16 | [3072] | 6,144 |
| G4 Q/K/V projection (ATT) | |||
| midlayer.self_attn.k_proj.weight | BF16 | [3072, 6144] | 37,748,736 |
| midlayer.self_attn.q_proj.weight | BF16 | [3072, 6144] | 37,748,736 |
| midlayer.self_attn.v_proj.weight | BF16 | [3072, 6144] | 37,748,736 |
| G5 attention sub-norm (BitNet only) | |||
| midlayer.self_attn.attn_sub_norm.weight | BF16 | [3072] | 6,144 |
| G6 o-projection (ATT) | |||
| midlayer.self_attn.o_proj.weight | BF16 | [3072, 3072] | 18,874,368 |
| R3 post-attention norm | |||
| midlayer.post_attention_layernorm.weight | BF16 | [3072] | 6,144 |
| G7 gate + up (ATT) | |||
| midlayer.mlp.gate_proj.weight | BF16 | [8192, 3072] | 50,331,648 |
| midlayer.mlp.up_proj.weight | BF16 | [8192, 3072] | 50,331,648 |
| G7 FFN sub-norm (BitNet only) | |||
| midlayer.mlp.ffn_sub_norm.weight | BF16 | [8192] | 16,384 |
| G9 down projection (MLPV) | |||
| midlayer.mlp.down_proj.weight | BF16 | [3072, 8192] | 50,331,648 |
| G11 final norm (MLPV) | |||
| norm.weight | BF16 | [3072] | 6,144 |
| G12 cheap low-rank lm-head (MLPV) | |||
| (no tensor) | |||
| lm-head (vocabulary projection) | |||
| lm_head.weight | BF16 | [128256, 3072] | 788,004,864 |
| draft-to-target vocab maps | |||
| d2t | I64 | [128256] | 1,026,048 |
| t2d | BOOL | [248320] | 248,320 |
| file total (tensor bytes) | 2,692,522,496 | ||
New: GPTQ-2 W2A16
| tensor (raw name) | dtype | shape | bytes |
|---|---|---|---|
| Token embedding (host) | |||
| embed_tokens.weight | BF16 | [248320, 5120] | 2,542,796,800 |
| fc input norms (3 target hiddens) | |||
| fc_norm.0.weight | BF16 | [5120] | 10,240 |
| fc_norm.1.weight | BF16 | [5120] | 10,240 |
| fc_norm.2.weight | BF16 | [5120] | 10,240 |
| fc: fuse 3 target hiddens to H | |||
| fc.g_idx | I32 | [15360] | 61,440 |
| fc.qweight | I32 | [960, 5120] | 19,660,800 |
| fc.qzeros | I32 | [120, 320] | 153,600 |
| fc.scales | BF16 | [120, 5120] | 1,228,800 |
| G1 RMSNorm (input + hidden) | |||
| layers.0.hidden_norm.weight | BF16 | [5120] | 10,240 |
| layers.0.input_layernorm.weight | BF16 | [5120] | 10,240 |
| G4 Q/K/V projection (ATT) | |||
| layers.0.self_attn.k_proj.g_idx | I32 | [10240] | 40,960 |
| layers.0.self_attn.k_proj.qweight | I32 | [640, 5120] | 13,107,200 |
| layers.0.self_attn.k_proj.qzeros | I32 | [80, 320] | 102,400 |
| layers.0.self_attn.k_proj.scales | BF16 | [80, 5120] | 819,200 |
| layers.0.self_attn.q_proj.g_idx | I32 | [10240] | 40,960 |
| layers.0.self_attn.q_proj.qweight | I32 | [640, 5120] | 13,107,200 |
| layers.0.self_attn.q_proj.qzeros | I32 | [80, 320] | 102,400 |
| layers.0.self_attn.q_proj.scales | BF16 | [80, 5120] | 819,200 |
| layers.0.self_attn.v_proj.g_idx | I32 | [10240] | 40,960 |
| layers.0.self_attn.v_proj.qweight | I32 | [640, 5120] | 13,107,200 |
| layers.0.self_attn.v_proj.qzeros | I32 | [80, 320] | 102,400 |
| layers.0.self_attn.v_proj.scales | BF16 | [80, 5120] | 819,200 |
| G5 attention sub-norm (BitNet only) | |||
| (no tensor) | |||
| G6 o-projection (ATT) | |||
| layers.0.self_attn.o_proj.g_idx | I32 | [5120] | 20,480 |
| layers.0.self_attn.o_proj.qweight | I32 | [320, 5120] | 6,553,600 |
| layers.0.self_attn.o_proj.qzeros | I32 | [40, 320] | 51,200 |
| layers.0.self_attn.o_proj.scales | BF16 | [40, 5120] | 409,600 |
| R3 post-attention norm | |||
| layers.0.post_attention_layernorm.weight | BF16 | [5120] | 10,240 |
| G7 gate + up (ATT) | |||
| layers.0.mlp.gate_proj.g_idx | I32 | [5120] | 20,480 |
| layers.0.mlp.gate_proj.qweight | I32 | [320, 16384] | 20,971,520 |
| layers.0.mlp.gate_proj.qzeros | I32 | [40, 1024] | 163,840 |
| layers.0.mlp.gate_proj.scales | BF16 | [40, 16384] | 1,310,720 |
| layers.0.mlp.up_proj.g_idx | I32 | [5120] | 20,480 |
| layers.0.mlp.up_proj.qweight | I32 | [320, 16384] | 20,971,520 |
| layers.0.mlp.up_proj.qzeros | I32 | [40, 1024] | 163,840 |
| layers.0.mlp.up_proj.scales | BF16 | [40, 16384] | 1,310,720 |
| G7 FFN sub-norm (BitNet only) | |||
| (no tensor) | |||
| G9 down projection (MLPV) | |||
| layers.0.mlp.down_proj.g_idx | I32 | [16384] | 65,536 |
| layers.0.mlp.down_proj.qweight | I32 | [1024, 5120] | 20,971,520 |
| layers.0.mlp.down_proj.qzeros | I32 | [128, 320] | 163,840 |
| layers.0.mlp.down_proj.scales | BF16 | [128, 5120] | 1,310,720 |
| G11 final norm (MLPV) | |||
| norm.weight | BF16 | [5120] | 10,240 |
| G12 cheap low-rank lm-head (MLPV) | |||
| efficient_lm_head.g_idx | I32 | [128] | 512 |
| efficient_lm_head.qweight | I32 | [8, 248320] | 7,946,240 |
| efficient_lm_head.qzeros | I32 | [1, 15520] | 62,080 |
| efficient_lm_head.scales | BF16 | [1, 248320] | 496,640 |
| efficient_lm_head_down_proj.weight | BF16 | [128, 5120] | 1,310,720 |
| lm-head (vocabulary projection) | |||
| lm_head.g_idx | I32 | [5120] | 20,480 |
| lm_head.qweight | I32 | [320, 248320] | 317,849,600 |
| lm_head.qzeros | I32 | [40, 15520] | 2,483,200 |
| lm_head.scales | BF16 | [40, 248320] | 19,865,600 |
| draft-to-target vocab maps | |||
| (no tensor) | |||
| file total (tensor bytes) | 3,030,697,088 | ||
Source files: old_step21491_bf16ste.tsv and new_gptq2_W2A16.tsv printed by hdr.py (safetensors header reader). Downloads: old_step21491_bf16ste.tsv · new_gptq2_W2A16.tsv · hdr.py.