Local MoE models on an M5 Max: server and speculative-decoding benchmarks

TL;DR

• Fastest setup: Qwen3.6-35B-A3B on mlx-serve with its MTP head. It runs at 206–273 tok/s for one request and 412 tok/s in total across four, against 104–109 on mlx_lm.server .
• Most of the gain comes from the engine. mlx-serve reaches 188–217 tok/s with speculation off, about twice the other servers. The MTP head adds 25–45% on code and long prompts, and nothing on prose.
• Rapid-MLX comes second. It ran at 165–240 tok/s with MTP for one request and 348 tok/s across four.
• Gemma 4 26B-A4B tops out at 166 tok/s, on Rapid-MLX with a DFlash drafter. On mlx-serve, its drafters made it slower, and four clients barely beat one (142 vs 136 tok/s).
• MoE beats dense on speed. On the same server, Qwen3.6-35B-A3B ran at about 105 tok/s and the dense Qwen3.8-27B at about 31. I compared speed only, not answer quality.
• Caveats:
  • Temperature 0 only, one run per setup, and repeated runs varied by up to 15%.
  • All runs were on AC power; on battery the 27B was far slower.
  • The mlx-serve MTP result uses the mlx-serve author's own 4-bit build.
  • It's an October 2026 snapshot, and these servers ship updates monthly.

---

On a MacBook Pro M5 Max with 128 GB, Qwen3.6-35B-A3B generates 107 tokens per second on a coding prompt under  mlx_lm.server  and 271 tokens per second under mlx-serve with its MTP head. Both runs use 4-bit MLX weights. The model stays the same; the server and the speculative-decoding setup change.

Measured on 19 October 2026. Servers in this space ship releases every few weeks, so these numbers describe one snapshot.

Setup

• Hardware. MacBook Pro, M5 Max, 128 GB unified memory, on AC power.
• Models. Qwen3.6-35B-A3B (35B parameters, about 3B active per token) and Gemma 4 26B-A4B (26B parameters, about 4B active), both 4-bit MLX. Qwen3.8-27B, a dense model, serves as a reference point.
• Servers. mlx_lm.server 0.32.2 (Apple's reference), vllm-mlx 0.5.0, Rapid-MLX 0.16.1, dflash-mlx (August 2026 main branch) and mlx-serve 26.10.2.
• Workloads. A 300-word explanation (prose), a Python class with tests (code), and a three-bullet summary of a 2,400-token prompt. Each generated up to 512 tokens, or 256 for the summary.
• Method. Temperature 0, streamed responses, one warm-up request, one measured run per setup. The figure reported is generation speed after the first token.

Qwen3.6-35B-A3B, one request at a time

Without speculation, three of four servers land at 87 to 109 tok/s. mlx_lm.server , vllm-mlx and Rapid-MLX produce the same speed within noise.

mlx-serve runs at 188 to 217 tok/s without any speculation. A wall-clock check with prompt-lookup drafting also disabled gave 216 tok/s against 118 for mlx_lm.server . Its log reports fused Metal kernels for MoE routing and the MLP layers. mlx-serve is written in Zig on MLX's C API, with no Python in the decode loop.

MTP adds 25% to 150%, depending on server and workload. Qwen models ship a multi-token-prediction head, a small extra layer that drafts the next few tokens for the main model to verify in one pass. Rapid-MLX attaches it automatically and goes from 96 to 240 tok/s on code. mlx-serve with its own MTP build goes from 217 to 271 on code and from 188 to 273 on the long prompt.

Code benefits most. Drafted tokens are accepted more often when output is predictable. dflash-mlx accepted 72% of drafted tokens on the code task and 65% on prose, and its speed followed: 210 against 142 tok/s.

Gemma 4 26B-A4B, one request at a time

Rapid-MLX with a DFlash drafter is fastest on two of three workloads , at 166 tok/s on code and 144 on the long prompt. DFlash is a block-diffusion drafter trained for one specific target model.

mlx-serve is fastest on prose, at 135 tok/s, and slowest on the long prompt. Its drafters gave no gain on this model. mlx-serve disables drafters for MoE targets by default; forced on, DFlash had low acceptance and switched itself off, and the Gemma assistant drafter dropped code speed from 136 to 82 tok/s. The long-prompt runs on mlx-serve stopped after 72 tokens, so those three figures carry more noise than the rest.

One request versus four

Four clients keep the same ranking and narrow the gaps.  For Qwen3.6-35B-A3B, four simultaneous 400-token coding requests produced 412 tok/s in total on mlx-serve with MTP, 320 on Rapid-MLX with MTP, 280 on Rapid-MLX without speculation and 249 on vllm-mlx with continuous batching. Each request used a distinct prompt, and I recounted output tokens from the returned text.

Batching gains shrink as single-request speed rises.  In the same test, vllm-mlx went from 86 tok/s for one request to 249 for four (2.9×); mlx-serve went from 282 to 412 (1.5×). Single-request figures here include prompt processing. A likely reason: when one request already keeps the GPU busy, batching has less idle time to fill.

Gemma 4 26B-A4B barely gains from batching on either server.  Four requests produced 150 tok/s in total on Rapid-MLX with DFlash (146 for one) and 132 on mlx-serve (127 for one).

Dense versus mixture-of-experts

On the same server with no speculation, Qwen3.8-27B (dense) generates about 31 tok/s and Qwen3.6-35B-A3B (MoE) about 105. Token generation on Apple silicon is limited by memory bandwidth: a dense model reads all 27B weights per token, and the MoE model reads its 3B active parameters plus routing. The dense model is the newer release and may give better answers; this test measured speed only.

Caveats

• One run per setup. Where I repeated a test, runs differed by up to 15%.
• Temperature 0 only. MTP and DFlash acceptance rates usually fall at higher sampling temperatures.
• Builds differ in one row. mlx-serve's MTP result uses the mlx-serve author's 4-bit build, which bundles the MTP head; all other Qwen rows use the mlx-community 4-bit build.
• Battery power cuts speed. Qwen3.8-27B ran at about 12 tok/s on battery and about 31 on AC. The likely cause is macOS limiting GPU power on battery in its default power mode.
• Quality was not compared. The outputs I spot-checked were coherent. Speculative decoding verifies drafts against the main model, so at temperature 0 the output should match the model without speculation.
• Other quantizations. NVFP4 matched 4-bit speed under mlx_lm.server (about 110 tok/s). OptiQ-4bit ran at about 83, because it keeps some layers at higher precision.

---

Try it yourself

Tested on macOS 26.6 with an M5 Max and 128 GB. Each model's 4-bit weights take about 21 GB of disk and the same in memory. Keep the Mac on AC power while testing; generation on battery was markedly slower in these runs.

Option 1: mlx-serve with Qwen3.6-35B-A3B (fastest in these tests)

mlx-serve needs macOS 26.2 or later on Apple silicon.

brew tap ddalcu/mlx-serve <https://github.com/ddalcu/mlx-serve>
brew trust ddalcu/mlx-serve
brew install mlx-serve

brew trust lets this third-party tap install and update without prompting. brew untap ddalcu/mlx-serve removes it.

Download the 4-bit build that bundles the MTP draft head (about 21 GB), then serve it:

hf download ddalcu/Qwen3.6-35B-A3B-MLX-Serve-4bit
mlx-serve --host 127.0.0.1 --model ~/.cache/huggingface/hub/models--ddalcu--Qwen3.6-35B-A3B-MLX-Serve-4bit/snapshots/*/

hf comes with pip install -U huggingface_hub . Without --host 127.0.0.1 , mlx-serve listens on every network interface.

The server exposes an OpenAI-compatible API on port 11234. List the model id, then send a request:

curl -s <http://127.0.0.1:11234/v1/models>
curl -s <http://127.0.0.1:11234/v1/chat/completions> -H 'Content-Type: application/json' \
  -d '{"model": "MODEL_ID_FROM_ABOVE", "messages": [{"role": "user", "content": "Write a Python LRU cache with tests."}], "max_tokens": 512}'

Option 2: Rapid-MLX (also runs Gemma 4 with a DFlash drafter)

Rapid-MLX installs commands named vllm-mlx , so it goes in its own virtual environment to avoid clashing with an existing vllm-mlx install:

uv venv -p 3.12 ~/.venvs/rapid-mlx
VIRTUAL_ENV=~/.venvs/rapid-mlx uv pip install 'rapid-mlx==0.16.1'

Qwen3.6-35B-A3B with its MTP head (the head downloads automatically):

~/.venvs/rapid-mlx/bin/vllm-mlx serve mlx-community/Qwen3.6-35B-A3B-4bit \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' --host 127.0.0.1 --port 8000

Gemma 4 26B-A4B with a DFlash drafter:

~/.venvs/rapid-mlx/bin/vllm-mlx serve lmstudio-community/gemma-4-26B-A4B-it-QAT-MLX-4bit \
  --speculative-config '{"method":"dflash","model":"z-lab/gemma-4-26B-A4B-it-DFlash"}' --host 127.0.0.1 --port 8000

Both serve an OpenAI-compatible API at http://127.0.0.1:8000/v1 . Any OpenAI client works by setting its base URL to that address.

Notes

• Versions tested: mlx-serve 26.10.2, Rapid-MLX 0.16.1, October 2026. Flags change between releases; check each project's README before relying on these.
• Speculative decoding gave the largest gains at temperature 0 and on code. Expect smaller gains on free-form text at higher temperatures.