TL;DR: With tree-based speculative decoding, Qwen3.8-27B [1] runs at 23 to 52 instead of about 8 tokens per second on a base M5 MacBook Pro.
Run it yourself
uv tool install git+https://github.com/JonasLoos/qwen-spec && qwen-spec
By default, language models write their answers token by token. Each forward pass reads the text so far and gives a probability for every possible next token. One of them is selected and the next pass starts.
This means that for each token, the full 15.1 GB of 4-bit quantized weights of Qwen3.8 27B have to pass through the GPU. At 153 GB/s that's 100 ms in theory and 125 ms in practice, i.e. about 8 tokens per second (tps).
One way to speed this up is speculative decoding [2, 3]: a much smaller drafter model (DFlash2 [4]) suggests multiple tokens in advance, which the main Qwen model then checks in parallel. The drafter proposes a block of up to 16 tokens in one pass of about 13 ms. The check also gives the main model's own next token after the last accepted guess (green in the figure), so every pass yields at least one token. Because the bottleneck is memory bandwidth, processing multiple tokens at once is nearly free.
On a base M5, checking 16 tokens in one pass costs about the same as checking one (with our optimized kernel); only beyond that does compute become the bottleneck and more parallel tokens get expensive. However, plainly checking the drafter's block of 16 tokens doesn't help that much: often only a few of them are correct, and everything after the first wrong one has to be thrown away. The mean number of accepted tokens per pass turns out to be around 2.5 to 7.5 depending on the content (open chat is hard to predict, math is easy), yielding 15 to 50 tps, 2 to 6 times faster than plain decoding.
Instead of predicting one linear sequence of next tokens, we can build a tree of possible completions [5, 6, 7]. Then, if the main guess of the drafter is wrong, a backup guess is already computed, and we can accept additional tokens per pass.
The tree comes for free: the drafter also scores alternatives for every position, and the 16 nodes it is most confident about are kept. On a base M5 this pushes the number of accepted tokens per pass to 3.5 to 8 and the speed to 23 to 52 tps. The gain over the single sequence is largest where the text is hard to predict: on open chat the tree accepts 50 % more tokens per pass, on math only 5 %.
How big should the tree be? More nodes give more accepted tokens per pass, but above about 16 nodes the extra tokens stop paying off: the pass gets slower, and the drafter's deeper guesses are rarely right. The budget comes from the cost curve of the machine, which is measured once at the first start. With the stock MLX kernel, the best budget would be only about 4 nodes.
Tokens per pass, time per pass and tokens per second against the number of nodes checked in one pass, for our kernel and the stock kernel.
How much faster it gets depends on how predictable the text is. Math and code are easy to guess and reach 35 to 52 tps, open-ended chat is the hardest at about 23 tps. In the model's default thinking mode with sampling, chat and code run at 27 tps and math at 53; these use the slightly relaxed acceptance rule explained in the last section. One pass takes about 150 ms whatever the tree looks like, about the time of one plain token, and the context length barely matters.
The GitHub repo is public and you can run this yourself:
uv tool install git+https://github.com/JonasLoos/qwen-spec
qwen-spec # chat
qwen-spec "your question" # one answer
qwen-spec-server # OpenAI-compatible API
Notes:
Checking 16 tokens in one pass is only free with the right kernels. The stock 4-bit matmul of MLX [8] re-reads the weights beyond a few rows, so a pass over 16 tokens costs 2.7 times a pass over one. Three custom Metal kernels for the M5 tensor units fix this: a 4-bit matmul that reads every weight tile once and multiplies all rows at once, a tree-attention kernel that reads the KV cache in place, and a fused sampling kernel that draws all tree nodes at once in about 1 ms.
Left: with our matmul kernel a pass over 16 tokens costs 1.02 times a pass over one token, with the stock MLX kernel 2.7 times; the two floors are reading the weights once and the tensor-unit compute. Right: the tree-attention kernel halves the time of an attention layer at every context length.
The tree itself also needs care, because Qwen3.8 is a hybrid model [9]: only 16 of its 64 layers use attention, the other 48 are Gated DeltaNet layers with a single recurrent state that cannot be rewound. The attention layers see the tree through a mask, so that every node only sees the context and its own ancestors. For the recurrent layers, every node's output is computed from the shared state after the context plus the updates along its own path, so the state is read once per layer and no per-node state exists. After the check, the accepted path is written into the state and the cache is compacted.
One pass over a draft tree: the attention layers see it through a mask; the recurrent layers read one shared state and add, per node, only the updates along its own path.
With greedy decoding, the tree gives exactly the same output as plain decoding. Sampling (the model's default: temperature 1.0, top-p 0.95, top-k 20) can be kept exact too: the check draws one token per tree node from the model's own distribution and follows the tree while the draw matches a drafted token. But this is slower, because a drafted token is only accepted when the random draw happens to land on it.
The default therefore relaxes this a bit. When the draw misses every drafted token, the walk still takes the drafted token with the longest verified continuation, if the probability mass moved, divided by the number of tokens gained, is at most 0.3.
One recorded pass in thinking mode. Twice the model's draw (green, dashed) was not among the drafted tokens, and the relaxed rule took the drafted branch anyway (orange) because it had a long verified continuation.
Over the recorded thinking-mode passes below, this forces about 4 % of the tokens and gives about 14 % more tokens per pass. Accuracy on math and code benchmarks is unchanged in paired tests, and the samples of a prompt become somewhat more alike, about as much as lowering the temperature from 1.0 to 0.85. --accept lossless restores exact sampling, and greedy decoding is exact under every rule.
Over 800 recorded passes in thinking mode (math and chat): extra tokens per pass against tokens forced, for thresholds from 0 to 1. The default 0.3 is marked.