TL;DR: With tree-based speculative decoding, Qwen3.8-27B [1] runs at 28 to 61 instead of about 8 tokens per second on a base M5 MacBook Pro.
Run it yourself
uv tool install git+https://github.com/JonasLoos/qwen-spec && qwen-spec
By default, language models write their answers token by token. Each forward pass reads the text so far and gives a probability for every possible next token. One of them is selected and the next pass starts.
This means that for each token, the full 15.1 GB of 4-bit quantized weights of Qwen3.8 27B have to pass through the GPU. At 153 GB/s that's 100 ms in theory and 125 ms in practice, i.e. about 8 tokens per second (tps).
One way to speed this up is speculative decoding [2, 3]: a much smaller drafter model (DFlash2 [4]) suggests multiple tokens in advance, which the main Qwen model then checks in parallel. The drafter proposes a block of up to 32 tokens in one pass of about 16 ms. The check also gives the main model's own next token after the last accepted guess (green in the figure), so every pass yields at least one token. Because the bottleneck is memory bandwidth, processing multiple tokens at once is nearly free.
On a base M5, checking 16 tokens in one pass costs about the same as checking one (with our optimized kernel); only beyond that does compute become the bottleneck and more parallel tokens get expensive. However, plainly checking the drafter's block doesn't help that much: often only a few of them are correct, and everything after the first wrong one has to be thrown away. The mean number of accepted tokens per pass turns out to be around 2.7 to 7.9 depending on the content (open chat is hard to predict, math is easy), yielding about 16 to 45 tps, 2 to 6 times faster than plain decoding.
Instead of predicting one linear sequence of next tokens, we can build a tree of possible completions [5, 6, 7]. Then, if the main guess of the drafter is wrong, a backup guess is already computed, and we can accept additional tokens per pass.
The tree comes for free: the drafter also scores alternatives for every position, and the nodes it is most confident about are kept, up to 32. On a base M5 this pushes the number of accepted tokens per pass to 4-13 and the speed to 25-77 tps. The gain over the single sequence is largest where the text is hard to predict: on open chat the tree accepts almost 40 % more tokens per pass, on code and math about 10 %.
How big should the tree be? More nodes give more accepted tokens per pass, but the pass gets slower: on a base M5 checking up to 16 nodes costs almost the same as one (5 % more), 17 nodes cost 12 % more, and 32 about 17 % more. Whether the extra nodes pay off depends on how good the drafter's deeper guesses are. The engine measures the cost curve of the machine once at the first start, calibrates how often a guess at each depth is right, and picks the number of nodes per pass from the two: between 16 and 32, depending on the text. With the stock MLX kernel, the best budget would be only about 4 nodes.
Tokens per pass, time per pass and tokens per second against the number of nodes checked in one pass, for our kernel and the stock kernel.
How much faster it gets depends on how predictable the text is. In the model's thinking block, decoding runs at 26 to 39 tps, while the answers of the code and math prompts reach 56 to 77 tps under exact sampling and 70 to 94 with the relaxed acceptance rule explained in the last section. Chat answers stay at about 25 tps. Together this means whole responses come out at 28 (chat), 44 (code) and 61 (math) tps on average. Each inference pass takes about 165 ms, about a quarter more than one plain token, and the context length barely matters.
The GitHub repo is public and you can run this yourself:
uv tool install git+https://github.com/JonasLoos/qwen-spec
qwen-spec # chat
qwen-spec "your question" # one answer
qwen-spec-server # OpenAI-compatible API
Notes:
The published DFlash2 drafter was trained for blocks of 8 tokens, and its guesses beyond the 15th position are rarely right, so with exact acceptance, trees deeper than 16 nodes bought nothing. We continued its training at a block size of 32 on 110 M tokens of Qwen3.8's own thinking-mode answers. The first 8 positions stay as good as before, positions 9 to 31 get much better, and with a 32-node tree the fine-tuned head accepts about 10 % more tokens per pass than the original at the same tree, most on code and least on chat. The head [8] and the permissively licensed half of the training data [9] are on Hugging Face, and the tool uses the head by default.
Checking 16 tokens in one pass is only free with the right kernels. The stock 4-bit matmul of MLX [10] re-reads the weights beyond a few rows, so a pass over 16 tokens costs 2.7 times a pass over one. Three custom Metal kernels for the M5 tensor units fix this: a 4-bit matmul that reads every weight tile once and multiplies all rows at once, a tree-attention kernel that reads the KV cache in place, and a fused sampling kernel that draws all tree nodes at once in about 1 ms.
Left: with our matmul kernel a pass over 16 tokens costs 1.05 times a pass over one token, with the stock MLX kernel 2.7 times; the two floors are reading the weights once and the tensor-unit compute. Right: the tree-attention kernel halves the time of an attention layer at every context length.
The tree itself also needs care, because Qwen3.8 is a hybrid model [11]: only 16 of its 64 layers use attention, the other 48 are Gated DeltaNet layers with a single recurrent state that cannot be rewound. The attention layers see the tree through a mask, so that every node only sees the context and its own ancestors. For the recurrent layers, every node's output is computed from the shared state after the context plus the updates along its own path, so the state is read once per layer and no per-node state exists. After the check, the accepted path is written into the state and the cache is compacted.
One pass over a draft tree: the attention layers see it through a mask; the recurrent layers read one shared state and add, per node, only the updates along its own path.
With greedy decoding, the tree gives exactly the same output as plain decoding. Sampling (the model's default: temperature 1.0, top-p 0.95, top-k 20) can be kept exact too: the check draws one token per tree node from the model's own distribution and follows the tree while the draw matches a drafted token. But this is slower, because a drafted token is only accepted when the random draw happens to land on it.
The default therefore relaxes this a bit. When the draw misses every drafted token, the walk still takes the drafted token with the longest verified continuation, if the probability mass moved, divided by the number of tokens gained, is at most 0.3.
Example of a drafted tree (simplified). Twice the relaxed rule forces a drafted token instead of the model's draw (" simple" instead of " user", " reason" instead of " provide"), so ten tokens get sampled instead of one.
Over the recorded thinking-mode passes below, this forces about 4 % of the tokens and gives about 20 % more tokens per pass. Accuracy on math and code benchmarks is unchanged in paired tests, and the samples of a prompt become somewhat more alike, about as much as lowering the temperature from 1.0 to 0.85. --accept lossless restores exact sampling, and greedy decoding is exact under every rule.
Over about 700 recorded passes in thinking mode (math and chat): extra tokens per pass against tokens forced, for thresholds from 0 to 1. The default 0.3 is marked.