Qwen3.5 Native MTP Speculative Decoding on MLX

Reproduction of mlx-lm PR #990 (AirRunner's feat/mtp-native branch): quantize Qwen3.5-9B and -27B to 4-bit while preserving the MTP head, then compare decode speed with and without --mtp.

📝 Blog post: https://blog.web-of-anion.top/archives/qwen-3-5-mtp-mlx

Headline results

Model	Condition	Decode tok/s	Speedup	Acceptance
Qwen3.5-9B 4bit	Baseline	54.94	1.00×	—
Qwen3.5-9B 4bit	+MTP	65.87	1.199×	45.5%
Qwen3.5-27B 4bit	Baseline	17.70	1.00×	—
Qwen3.5-27B 4bit	+MTP	23.25	1.314×	44.3%

Run on Apple M5 Pro / macOS 26.4, temp=0 greedy, 400 tokens. Full data, cross-hardware comparison, and analysis in research_log.md and the blog post.

Repo layout

.
├── README.md               # this file
├── research_log.md         # full research log (decisions, data, corrections)
├── blog/
│   └── qwen35-mtp-story.md # blog post source
├── scripts/
│   ├── README.md           # reproduction guide
│   ├── quant_bench.py      # quantize + instrumented timing / memory
│   ├── run_benchmark.py    # baseline vs MTP tok/s + acceptance
│   └── verify_weights.py   # MTP weight structure check
└── results/
    ├── bench_result_{9B,27B}.json    # benchmark output
    ├── quant_result_{9B,27B}.json    # quantization timing / memory peaks
    └── quant_timeline_{9B,27B}.csv   # RSS / MLX mem time series during quantization

Model weights are intentionally not checked in (see .gitignore).

Quick reproduction

# 1. Environment (isolated venv, does not touch system mlx-lm)
git clone -b feat/mtp-native https://github.com/AirRunner/mlx-lm.git ~/mlx-lm-mtp
cd ~/mlx-lm-mtp
uv venv .venv --python 3.12 && source .venv/bin/activate
uv pip install -e .
uv pip install "httpx[socks]"   # required behind a SOCKS proxy

# 2. Clone this repo
git clone https://github.com/Anionex/qwen35-mtp-mlx.git ~/qwen35-mtp-mlx
cd ~/qwen35-mtp-mlx

# 3. Quantize (9B ≈ 4 s, 27B ≈ 9 s; both with ~11 GB RSS peak)
python scripts/quant_bench.py Qwen/Qwen3.5-9B \
    ~/.lmstudio/models/mlx-community/Qwen3.5-9B-MLX-4bit-MTP 9B

# 4. Verify MTP weights
python scripts/verify_weights.py \
    ~/.lmstudio/models/mlx-community/Qwen3.5-9B-MLX-4bit-MTP

# 5. Benchmark (baseline vs MTP)
python scripts/run_benchmark.py \
    ~/.lmstudio/models/mlx-community/Qwen3.5-9B-MLX-4bit-MTP 9B

Detailed steps in scripts/README.md.

Measurement protocol (used for every result in this repo)

• Prompt: "Write a Python function that implements a LRU cache with O(1) get and put operations. Include tests."
• max_tokens = 400, greedy (temp = 0)
• mx.reset_peak_memory() before each measurement, 20-token warmup
• Quantization: --q-bits 4 --q-group-size 64, mtp.fc kept at bf16

Hardware / software

• Apple M5 Pro, macOS 26.4
• mlx-lm AirRunner fork, branch feat/mtp-native
• Python 3.12, uv-managed venv

References

• mlx-lm PR #990
• AirRunner/mlx-lm feat/mtp-native
• sammcj M5 Max benchmark gist
• Leviathan et al. 2023 — Fast Inference from Transformers via Speculative Decoding

License

MIT