How to Speed Up Local LLMs with vLLM

Local language models are becoming a practical alternative, and how they are served makes a real difference in performance.

In his latest article, Hıdır Bozkurt explores how to speed up local LLMs with vLLM, covering quantization, speculative decoding and their impact on accuracy.

1. Why local LLMs are slow: memory bandwidth

The DGX Spark is a small desktop machine built around NVIDIA’s GB10 chip. Its selling point is 128 GB of unified memory (about 121 GB usable), enough to hold models that would never fit on a gaming GPU. Its weak point is that this memory is LPDDR5X with about 273 GB/s of bandwidth, several times less than a data-center GPU.

That matters because an LLM does two very different kinds of work:

Figure 1. Reading the prompt is limited by compute. Writing the answer is limited by memory bandwidth.

When the model reads your prompt (prefill), it processes thousands of tokens in parallel, and the GPU cores are busy. When it writes the answer (decode), it produces one token at a time, and for every single token it has to stream the active model weights from memory. The GPU spends most of that time waiting for data.

A useful rule of thumb: decode speed ≈ memory bandwidth ÷ bytes read per token. A 9B-parameter model in BF16 is about 18 GB, so the ceiling is 273 ÷ 18 ≈ 15 tokens/s. We measured 12.4. Every trick in this post therefore does one of two things: read fewer bytes per token, or get more than one token out of each read.

The same holds on any single GPU or workstation, so the methods below carry over to other hardware. Only the absolute numbers and the kernel choice (section 4.4) are specific to DGX Spark.

2. Test setup

All speeds are generated tokens per second as seen by the client. For the quality runs we also record what matters to a user: the wall-clock time to finish the whole test set.

3. Results per model: speed and accuracy

We tuned three models separately. Each one is compared with its own untuned starting point: the bars below are not one model getting faster step by step.

Figure 2. Single-request decode speed for each model, from its untuned baseline to its best configuration. Same machine, same prompts.

To check that the speed did not cost accuracy, every model answered the same 1,550 benchmark questions in its slowest setting (BF16 weights, BF16 KV cache, no speculative decoding) and its fastest one, with thinking off and temperature 0. Qwen3.6 also ran in Qwen’s official FP8 release as a reference for the cost of 4-bit weights.

Figure 3. Benchmark scores at each model’s slowest (light) and fastest (dark) setting. The fastest settings are the last bars in Figure 2.

Single-request speeds come from the speed test (section 2), where the untuned baselines already used FP8 KV cache. The 16-request speeds and scores come from the benchmark runs. The fastest settings also use FP8 KV cache. A needle-in-a-haystack test at 32K and 64K tokens found every needle in all seven settings.

Within each model, the same two levers do most of the work: lower-precision weights (fewer bytes per token) and speculative decoding (more tokens per read). The Qwen3.6 checkpoint is already 4-bit when you download it, so its untuned baseline is fast and the remaining gain comes from speculative decoding alone.

• No measurable loss from any speedup. Slowest and fastest settings differ by 1–2 points in both directions (MMLU-Pro −0.4, +0.9 and −0.8). Even Qwen3.6’s two NVFP4 settings, which differ only in FP8 KV cache and the lossless DSpark drafter, moved 2 points on MBPP+, so differences of that size are noise.
• 4-bit costs about one point. Against Qwen’s FP8 release, NVFP4 is 0.7 points lower on MMLU-Pro, equal on HumanEval+ and 2.7 lower on MBPP+, while generating 1.5x faster.
• FP8 KV cache keeps long context intact: no needle was missed at 32K or 64K tokens.
• The fastest model is also the best one. Qwen3.6 beats Qwen3.5-35B-A3B, which beats the 9B, on every test. Its fastest setting finished all 1,550 questions in 36 minutes.

Benchmarks: MMLU-Pro via lm-evaluation-harness · HumanEval+ and MBPP+ via EvalPlus · needle test modelled on RULER niah_single_1.

Comparing the models with each other

Across models the picture is different, and it is a separate question: which model to run. Fully tuned, the 35B MoE is 1.9x faster than the dense 9B (68.1 vs 36.5 tokens/s), even though it is four times larger, and it also scores higher on every benchmark. Qwen3.6 in NVIDIA’s NVFP4 release is a newer version of the same MoE architecture. At 4 bits it reaches 129.2 tokens/s. On this kind of hardware, the number that matters is not total parameters but bytes that have to be read for each token.

4. What worked, one by one

4.1 Pick a Mixture-of-Experts model

A dense 9B model touches all 9B parameters for every token. Qwen3.5/3.6-35B-A3B has 35B parameters split into 256 small “experts”, but a router picks only 8 of them per token, so roughly 3B parameters are active. All 35B must fit in memory (fine with 121 GB), but only a tenth is read per token. On a bandwidth-starved machine this is the single most effective choice.

Figure 4. A dense model reads all its weights for every token. An MoE reads only the experts the router picked.

4.2 Lower-precision weights: BF16 → FP8 → NVFP4

Weights are stored as numbers. Using fewer bits per number means fewer bytes to read.

• BF16 (16-bit): the original release. Highest fidelity, slowest.
• FP8 (8-bit): vLLM can convert on the fly with –quantization fp8. 9B went from 12.4 to 21.9 tokens/s without MTP.
• NVFP4 (4-bit): NVIDIA’s 4-bit format with small per-block scales, published as a ready checkpoint (nvidia/Qwen3.6-35B-A3B-NVFP4). With MTP 2 it reached 118 tokens/s, against 68 for the same-architecture Qwen3.5-35B-A3B in FP8.
• FP8 KV cache (–kv-cache-dtype fp8): the attention cache is read for every token too. With long inputs it made MoE decode 45% faster and raised cache capacity from 2.38M to 3.97M tokens.

Section 3 shows that, in our runs, the 4-bit model lost about one benchmark point against FP8.

4.3 Speculative decoding: more than one token per read

Since decode is waiting on memory anyway, checking several tokens costs almost the same as producing one. Speculative decoding exploits that: a cheap drafter guesses the next few tokens, and the main model verifies all guesses in a single pass. Correct guesses are kept, and at the first wrong one the main model’s own token is used instead. The output follows exactly the same distribution as the main model alone. It is only faster.

Figure 5. The drafter proposes four tokens. The main model accepts three and corrects the fourth, giving four tokens for the price of one read.

Figure 6. Asking MTP for more tokens helps up to 2–3, then wasted guesses cost more than they save.

The first guess gives the biggest jump. On Qwen3.6, one MTP token took the speed from 75.9 to 103.5 tokens/s, and two tokens to 117.8. A third added only 2 tokens/s more. The 9B model peaks at 2–3 tokens (36.6 tokens/s) and slows down after that: with 5 tokens it drops to 33.9.

The reason is that each extra guess is less likely to be right. The MTP layer was trained to look one token ahead, so its second and third guesses build on its own earlier guesses and errors add up. On 9B, each step produced 1.9 tokens with 1 guess, 3.0 with 3 guesses and only 3.7 with 5. Every rejected guess still costs compute in the verification pass, so past 2–3 tokens the extra work outweighs the extra accepted tokens. DSpark drafts the whole block at once instead of one token at a time, which is why it stays useful at 4 tokens.

4.4 The right kernel for GB10

GB10 is new (compute capability 12.1), and many optimized kernels simply refuse to run on it. For FP8 MoE only the triton and marlin backends worked. For NVFP4 only marlin did. Setting –moe-backend marlin explicitly matters: a pre-quantized FP8 checkpoint otherwise selected DeepGEMM automatically and was slower.

4.5 Smaller but welcome

• Prefix caching (–enable-prefix-caching): agents resend the same growing conversation every turn. Time-to-first-token on later turns fell from 1.64 to 0.32 s on 9B, and from 0.90 to 0.64 s on the MoE.
• fastsafetensors (–load-format fastsafetensors): weight loading went from 151–188 s to 22 s. Total time until the server was ready from 228–302 s to 96 s. Use it with a ready-made FP8 or NVFP4 checkpoint. Combined with on-the-fly FP8 conversion (–quantization fp8) of a large BF16 model, it holds the BF16 weights in memory while the FP8 copy is built and can run out of the shared 121 GB.
• Async scheduling and –max-num-batched-tokens 8192: small but consistent gains. 8192 was the best chunk size we tried.

5. Many users at once

A single request leaves most of the GPU idle, so serving several at once is nearly free at first: the same weight read serves every request in the batch.

Figure 7. Top: total throughput of the box. Bottom: average time until one 256-token answer is complete, as seen by each user.

Total throughput and the wait each user feels move in opposite directions. Going from 1 to 16 requests, the box produces 4x more tokens in total, but each answer takes 4x longer to finish (2.1 s → 8.2 s for Qwen3.6), because every request now shares the same weight reads with fifteen others. For a chat user 33 tokens/s is still faster than reading. For an agent that waits on every answer before its next step, the longer finish time adds up across hundreds of steps.

Two more observations. The 9B dense model barely slows down per answer (6.9 s → 9.2 s): a dense model reads the same weights no matter how many requests share them. And the dense 9B catches up with the Qwen3.5 MoE and overtakes it from 8 requests on. With one request the MoE reads only 8 experts per layer. With 16 different requests the union of chosen experts grows toward all 256, so its per-token advantage shrinks.

“Concurrent” means answers being generated at the same moment. A chat service with 20 active users typically has only a few requests in flight. An agent, by contrast, keeps roughly one request running all the time.

6. What did not work and DGX Spark pitfalls

• Many FP4 MoE kernels did not run on GB10 in our vLLM nightly build. flashinfer_trtllm, cutedsl and b12x reported an unsupported device, and flashinfer_b12x crashed. Only marlin worked for us. Kernel support changes quickly between vLLM releases, so check your own version.
• Auto-tuning the MoE kernel (vLLM’s benchmark_moe) for the FP8 MoE at batch sizes 1–16 gave no measurable gain.
• Temperature had no effect on speed, except DFlash, which accepts more guesses at temperature 0.

7. The recipe

The fastest configuration we found for one DGX Spark:

vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --moe-backend marlin \
  --speculative-config '{"method":"dspark",
     "model":"RedHatAI/Qwen3.6-35B-A3B-speculator.dspark",
     "num_speculative_tokens":4}' \
  --kv-cache-dtype fp8 --load-format fastsafetensors \
  --enable-prefix-caching --enable-chunked-prefill --async-scheduling \
  --max-num-batched-tokens 8192 \
  --reasoning-parser qwen3 --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder --trust-remote-code

If you do not want an extra drafter download, replace the speculative config with the built-in MTP ({“method”:”mtp”,”num_speculative_tokens”:2}) for about 118 tokens/s.

8. Takeaways and limits

• On a bandwidth-limited box, choose the model by active parameters and bytes per token, not by total size.
• Quantization and speculative decoding stack: together they made each model 1.7–2.9x faster with no measurable quality loss: across three models, the fastest settings scored within 1–2 points of the slowest on MMLU-Pro, HumanEval+ and MBPP+.
• Measure time-to-finish as well as tokens/s. Total throughput grows with more users, but each answer takes longer.
• GB10 is still new: check which kernels actually run before trusting a recipe from another GPU.

Limits: one machine, one model family, coding prompts for the speed tests, thinking off in the quality tests, a vLLM nightly build, and single quality runs with real noise. Treat small differences as indicative, not final.

Hıdır Bozkurt

Data Scientist
Hıdır Bozkurt conducts R&D across various areas of data science, primarily focusing on NLP and computer vision.