MoE vs Dense: Why the Larger Model Was Four Times Faster
A local document benchmark on a Mac mini with M4 Pro shows why active parameters matter more for throughput than total model size.
After switching models, my automated document pipeline suddenly took five to twelve minutes per document. Three parallel runs were sometimes enough to throw the Ollama runner off. Switching back to the Mixture-of-Experts model I had used before made the problem disappear.
My first suspicion was the architecture. The dense 27B model has to process all 27 billion parameters for every token. The larger MoE has roughly 36 billion parameters in total, but only activates around three billion of them at a time.
That sounded plausible. I had never measured it.
So I turned the complete pipeline from my automated Paperless workflow into a reproducible benchmark. Not MMLU, multiple-choice questions, or a single synthetic prompt, but the work the model actually does in production: check OCR, read handwritten pages, classify documents, and generate metadata.
The full benchmark, including the corpus, prompts, and raw results, is available on GitHub.
The larger model computes with fewer parameters
The most useful comparison is between two models from the same Qwen generation:
| Model | Architecture | Total parameters | Active per token |
|---|---|---|---|
qwen3.6:35b-a3b | Mixture of Experts | 36.0B | around 3B |
qwen3.6:27b | Dense | 27.8B | 27.8B |
With an MoE, a router decides which experts are needed for each token. Qwen3.6-35B-A3B has 256 experts. For every token, eight routed experts plus one shared expert are activated.1 The remaining weights are not used for that computation step.
That reduces compute, not memory requirements. In a normal Ollama or llama.cpp setup, all experts still have to stay in memory. In simplified terms, the model behaves like a large model when it comes to RAM usage and like a much smaller one during decoding.
On Apple Silicon, that is a useful combination. I ran the benchmark on a Mac mini with an M4 Pro and 48 GB of unified memory. All four models fit fully into memory using Q4_K_M quantization.
I benchmarked the real pipeline
The production pipeline has four stages. First, the model decides whether the existing OCR text is usable. If it is not, a vision model reads the page image. Classification and metadata extraction follow afterwards.
| Stage | Task | Evaluation |
|---|---|---|
| OCR check | Is the existing text usable? | correct yes/no decision |
| Vision OCR | Read the page image | character and word error rate |
| Classification | Correspondent, document type, and tags | exact match, F1 for tags |
| Metadata | Title and document date | token F1, exact match |
The prompts are unchanged from the production Windmill script. I measured directly against Ollama’s /api/generate endpoint because the API exposes generation, prompt processing, and loading separately through values such as eval_count and eval_duration. In production, LlamaIndex sits in between. Measuring through the wrapper would have mixed model runtime with framework overhead.
The corpus contains eight fictional documents: four printed pages, two genuinely handwritten and scanned pages, and two documents rendered with a handwriting font. The content is synthetic because a precisely known reference text is required to calculate character error rate, and because private documents do not belong in a public repository.
I restarted Ollama before every model. Each model ran the complete benchmark three times, with only one document processed at a time. The test plan contained 96 model-document combinations. The 8B model skipped twelve handwriting cases because it has no vision support, leaving 84 combinations that were actually processed. None failed. Memory use, swap, and thermal throttling were recorded alongside the runs.
In addition to the two Qwen3.6 models, I included qwen3.8:27b as a newer dense model and qwen3:8b as a small text-only variant. The 8B model has no vision capability, so it only processed the printed documents.
The MoE model runs 4.6 times faster
The difference is hard to miss.
| Model | Generation rate across stages | Seconds per document |
|---|---|---|
qwen3.6:35b-a3b | 67.1 to 68.0 tokens/s | 85 s |
qwen3.6:27b | 14.5 to 14.8 tokens/s | 367 s |
qwen3.8:27b | 15.0 to 20.6 tokens/s | 96 s |
qwen3:8b | 45.8 to 46.4 tokens/s | 37 s* |
* Printed documents only, without Vision OCR. The wall-clock time is therefore not directly comparable to the other models.
For the architecture comparison, qwen3.6:35b-a3b and qwen3.6:27b matter most. They come from the same model generation, use the same quantization, and received the same inputs. The MoE reached roughly 68 tokens per second. The dense model stayed just below 15. That is a 4.6x difference in decoding throughput.
Wall-clock time tells a similar story. The MoE needed an average of 85 seconds per document, while the dense counterpart took 367 seconds. One document therefore finished in less than a quarter of the time.
The gap remained across all four stages:
| Stage | MoE 35B-A3B | Dense 27B |
|---|---|---|
| OCR check | 68.0 tokens/s | 14.8 tokens/s |
| Vision OCR | 67.1 tokens/s | 14.5 tokens/s |
| Classification | 67.7 tokens/s | 14.7 tokens/s |
| Metadata | 68.0 tokens/s | 14.7 tokens/s |
The repeated runs were stable as well. The dense model took 359, 367, and 375 seconds per document. The MoE came in at 99, 76, and 80 seconds. There is some spread, but nowhere near enough to explain the architecture gap.
The comparison with the small 8B model was even more striking. The MoE generated roughly 68 tokens per second, around 1.5 times faster than the dense 8B model at about 46 tokens per second. A model with 36 billion total parameters was faster at decoding than one with 8.2 billion. Total parameter count says surprisingly little about decoding speed here. What matters more is how many parameters are actually active for each token.
Fast decoding does not make reasoning free
Tokens per second are only half the story. A model can decode very quickly and still take a long time if it generates many more tokens.
That is exactly what happened with the MoE. Across the pipeline, it generated an average of 5,549 tokens per document. qwen3.8:27b generated 1,685. Even so, the MoE still finished slightly faster at 85 seconds versus 96 seconds for the newer dense model.
The effect is particularly clear on handwritten document d05. In the Vision OCR stage alone, the MoE generated 5,262 tokens and took 84 seconds. qwen3.8:27b generated 408 tokens and finished after 30 seconds. A large part of the extra output was reasoning.
The high decoding rate offsets part of those additional reasoning tokens.
That objection disappears when comparing the two Qwen3.6 models. Their total output per document was almost the same: 5,549 tokens for the MoE and 5,222 for the dense model. The wall-clock result was still 85 versus 367 seconds. Shorter answers or less reasoning cannot explain that gap.
For a production workflow, both measurements therefore matter. Token rate tells me how fast the architecture computes. Wall-clock time tells me how long the job actually takes.
There is no overall winner on accuracy
The models are easy to separate on speed. Accuracy is much less clear-cut.
| Metric | MoE 35B-A3B | Dense 27B, Qwen3.6 | Dense 27B, Qwen3.8 |
|---|---|---|---|
| Character error rate, real handwriting | 23.5% | 34.0% | 22.4% |
| Word error rate, real handwriting | 41.1% | 56.5% | 41.1% |
| Correspondent correct | 88% | 88% | 88% |
| Document type correct | 79% | 75% | 88% |
| Tags F1 | 0.88 | 0.83 | 0.87 |
| Date correct | 100% | 100% | 100% |
| Title F1 | 0.53 | 0.38 | 0.43 |
On average, the MoE read the two real handwritten pages substantially better than its dense Qwen3.6 counterpart. Almost all of that difference, however, comes from one difficult document. On the easier handwritten page, all three vision models were around eleven percent character error rate. On the difficult one, the MoE reached 35.9 percent, qwen3.8:27b 34.7 percent, and the dense Qwen3.6 model 57.1 percent.
So the MoE was better than its dense counterpart on this corpus. That is not enough evidence for a general claim about MoE models and handwriting. There are only two genuinely handwritten pages, and only one of them clearly separates the models.
On printed documents, model size mattered surprisingly little. All four models identified the correspondent correctly in 100 percent of cases, the date in 100 percent, and the document type in 75 percent. Even the 8B model handled these structured text tasks reliably. Larger models mainly improved tags and titles, not the basic assignment.
One repeated document-type error was also worth looking at. An insurance policy change was labelled Vertrag (contract) in the reference data. Every model returned Bestätigung (confirmation) on every run. The letter really does confirm a change. If four models reproduce the same supposed error twelve times, the ground truth deserves scrutiny as well.
A handwriting font does not test handwriting
The two documents rendered with a handwriting font initially looked like a reasonable substitute for real scans. In practice, they were useless for this benchmark.
All three vision models read both pages in every run with zero percent character and word error rate. On the real handwritten pages, character error rate ranged from 22.4 to 34.0 percent.
A handwriting font is still a clean vector font with unusual letter shapes. It has no inconsistent stroke direction, overwritten characters, uneven pressure, or slanted lines. Testing OCR on rendered text mainly measures whether the model can read a particular typeface.
Fluent wrong text is more dangerous than gibberish
The hardest page was a handwritten reminder from a library. Dates, amounts, and the general document structure remained surprisingly stable across the models. Terms carrying the actual meaning, however, often turned into plausible alternatives.
| On the page | MoE 35B-A3B | Dense Qwen3.8 | Dense Qwen3.6 |
|---|---|---|---|
Ausweisnummer 4471 | Kassennummer 4971 | Kundennummer 4471 | Kunstenzwaams 49 t.t |
Mahngebühr 6.00 CHF | Mahngebühr 6 CHF | Mahnschreiben 6 CHF | Makroplastiekken 6 CHF |
Bitte innert zehn Tagen zurückbringen | Bitte sofort an Tiger Journal Premier | Bitte umsofort 10 Tage vor dem 2. Mahnung | Bilte simost 1e Tegen landd kremer |
The dense Qwen3.6 model had the highest error rate, but its output was visibly broken and sometimes looked vaguely Dutch. At least that was easy to recognize as wrong.
Plausible OCR output is more dangerous. Kundennummer 4471 looks perfectly normal on a form, even though the page actually says Ausweisnummer 4471. Errors like that pass quietly into the next stage. On this document, none of the models found the correct correspondent. In several runs, qwen3.8:27b even selected a plausible but wrong entry from the provided list.
Character error rate alone does not capture that risk. A document pipeline also needs plausibility checks for known senders, amounts, dates, and document types. A model that produces obvious nonsense can be less dangerous operationally than one that expresses the same mistake fluently.
The first benchmark run partly measured the machine
The benchmark itself had a few false starts. In an early run, qwen3.8:27b occasionally dropped from roughly 24 to 26 tokens per second down to one or two. Prompt size and generated token count were comparable, but runtime increased by roughly a factor of 20.
It would have been tempting to use that as evidence against the dense model. The later run on the Mac mini did not reproduce it. Ollama was restarted before every model, token rates remained stable, swap stayed below one gigabyte, and there was no thermal throttling. Even the slow dense Qwen3.6 model ran for two and a half hours without collapsing.
The earlier slowdown was real, but its cause remains open. Server state, the operating system, Ollama version, concurrency, or some other runtime condition are all plausible explanations in addition to the model itself. The benchmark also processes only one document at a time, while the original production problem appeared with three concurrent runs.
A second correction was much simpler. An early evaluation contained a combination of token count and runtime that was mathematically impossible at the measured token rate. I had copied the number manually from the JSON, and it happened to support the conclusion I wanted particularly well.
Since then, bench/report.py has generated all tables directly from the result JSON. I should have done that from the start; the manually copied number was simply wrong.
Other measurements show the same order of magnitude
The numbers fit surprisingly well with two other Apple Silicon benchmarks I found.
A reproducible llama.cpp comparison on a MacBook Pro with an M1 Pro and 32 GB measured 5.3 tokens per second for the dense 27B model and 25.4 for the MoE. That is a 4.8x difference. The MoE also used more memory and scored 66 instead of 73 in that benchmark’s aggregate quality score.2
mlx-coding-bench measured 16 tokens per second for the dense model and 86 for the MoE on an M4 Pro with 64 GB using 4-bit MLX quantization, a factor of 5.4. In its mix of coding, reasoning, tool use, mathematics, and writing tasks, the dense model scored 90.1 versus 86.2 percent.3
The absolute numbers are not directly comparable to my Ollama run. Hardware, runtime, quantization, context, prompts, and token budgets differ. The ratios are nevertheless close: in all three tests, the MoE decodes several times faster, while the dense models perform somewhat better in the external quality benchmarks and use less memory.
A recent paper shows the same effect with a different MoE architecture. DECO activates only 20 percent of its experts, reaches dense-comparable performance at the same parameter budget according to the paper, and achieves a threefold speedup on real hardware with a specialized kernel.4 It is not directly comparable to Qwen, but the underlying mechanism is the same.
Active parameter count is therefore a useful first indicator of decoding throughput, but not a complete explanation. Expert routing, memory bandwidth, kernels, quantization, context length, and model generation all influence whether the theoretical advantage turns into a factor of two or a factor of five.
There is not one best model for the pipeline
For my document pipeline, I would still choose qwen3.6:35b-a3b in this setup if one model has to handle all four stages. Compared with its dense counterpart, it is more than four times faster without showing a general quality disadvantage on this task.
Another result is useful as well: the text-only part often does not need a large model. On printed documents, the 8B model handled correspondent, document type, and date just as reliably as the 27B and 36B variants.
That does not automatically mean that two models would make the production pipeline better. In a local setup, switching models can add loading time, while keeping both loaded increases memory pressure. For my pipeline, a single vision-capable MoE is probably the more practical choice. The 8B result is still useful because it shows that much of the structured text processing requires far less model capacity than the difficult OCR cases.
The limits of the test are straightforward: one machine, eight documents, two genuinely handwritten pages, three repetitions, one quantization, only Qwen models, and no concurrency. The benchmark demonstrates a substantial speed difference on this hardware. It is not a general ranking of dense and sparse models.
On this machine, the test confirms my original suspicion. Active parameter count explains decoding throughput much better than total model size. Total size still matters because all model weights have to fit into memory.
The practical takeaway is simple: a 36B MoE can be substantially faster than a 27B dense model on the same hardware, as long as the larger model fits into memory.