Field Guide: Running Qwen3.8 Locally
Local LLMs have quietly crossed an important threshold.
Qwen3.8-27B is the model local users most often name when they say they're getting real coding work done on their own hardware β on a single consumer GPU or a well-specced Mac. And if your machine is smaller than that, the Qwen family still has something that runs.
This guide gets you from zero to a running Qwen model in a few minutes, then helps you pick the right variant for your hardware.
About the numbers in this guide: speeds marked (user report) come from individual people's setups in public discussions, not controlled benchmarks. Treat them as ballpark figures β your hardware, driver and build will differ. The 4B figures were measured directly for this guide.
Step 0 β Install llama.cpp
llama.cpp now uses CMake, so older make commands will fail.
Mac (fastest path)
brew install llama.cpp
Mac / Linux (recommended, latest version)
New Qwen releases often need a recent build, so building from source is the safer bet.
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j 8
NVIDIA GPU (CUDA)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j 8
Windows via WSL (CUDA)
CUDA works inside WSL through the normal Windows NVIDIA driver β don't install a Linux GPU driver in the distro. You do still need the CUDA toolkit inside WSL, otherwise CMake can't find nvcc. This is the most common stumbling block.
# inside your WSL distro (Ubuntu)
nvidia-smi # should list your GPU; if not, update the Windows driver
sudo apt install -y build-essential cmake git nvidia-cuda-toolkit
nvcc --version # confirm the toolkit is on PATH
Then build with -DGGML_CUDA=ON as above. If CMake picks the wrong GPU architecture, set it explicitly β for example -DCMAKE_CUDA_ARCHITECTURES=75 for a Turing card (T1000, RTX 20xx).
AMD GPU (try Vulkan before ROCm)
On RDNA3/RDNA4 cards (7900 XTX, 9070 XT), several users report that Vulkan is as fast or faster than ROCm and uses less VRAM β for example ~63 tok/s on Vulkan vs ~52 tok/s on ROCm for Qwen3.8-27B on a 7900 XTX (user report). Benchmark both on your card if you care about the last few percent.
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release -j 8
Verify install
./build/bin/llama-cli --version
Step 1 β Run Qwen3.8-27B (One Line)
Start with this:
./build/bin/llama-cli -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 -c 32768 --jinja -ngl 99
That's it.
-ngl 99 puts every layer on the GPU. Recent builds often do this automatically, but setting it explicitly means you'll notice if offloading fails β check the startup log for a line saying how many layers were offloaded. If it reports 0, your build doesn't have GPU support.
Hardware note: at Q4 this wants roughly 24GB of VRAM (RTX 3090/4090/5090, 7900 XTX) or a Mac with 48GB+ β that's an estimate from the file size plus context, not a measured figure. On a 32GB Mac it loads but crawls (2β4 tok/s (user report)); use the MoE model below instead.
Single-shot mode: to send one prompt and exit (handy for scripts), add -st and -p "your prompt". Older tips use -no-cnv for this; current builds reject it.
Running Other Model Variants
Small GPU / Laptop (Qwen3.5-4B)
Most machines don't have 24GB of VRAM. If you have a 4β8GB GPU or a 16GB-RAM laptop, start here. At Q4 the 4B is about 2.7GB and fits entirely on a small card.
./build/bin/llama-cli -hf unsloth/Qwen3.5-4B-GGUF:Q4_K_M --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 -c 32768 --jinja -ngl 99
Measured for this guide on a Quadro T1000 (4GB VRAM, 15GB RAM) under WSL with CUDA: 62 tok/s prompt processing, 35 tok/s generation, all layers on the GPU. Asked to write a prime-checking function, it produced a reasoning block and then a correct-looking is_prime.
Set expectations accordingly: a 4B is good for quick questions, small functions, tagging and extraction, not for multi-file agent work. If you have 8GB of VRAM, Qwen3.5-9B is the next step up. Its Q4_K_M file is about 5.7GB, so it fits on an 8GB card with room for a moderate context:
./build/bin/llama-cli -hf unsloth/Qwen3.5-9B-GGUF:Q4_K_M --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 -c 16384 --jinja -ngl 99
Fast MoE (Qwen3.6-35B-A3B)
Only ~3B parameters are active per token, so it runs much faster than the 27B on the same GPU (~129 tok/s on a 7900 XTX (user report)) and copes better with partial CPU offload. This is the pick for 32GB Macs, Strix Halo boxes, and 16GB GPUs.
./build/bin/llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 -c 32768 --jinja -ngl 99
Big MoE (Qwen3.8 Flash Next)
For 128GB machines β Framework Desktop / Ryzen AI Max 395, 128GB Macs. Users describe it as stronger on broad knowledge than the 27B, at the cost of most of your RAM even at Q4.
./build/bin/llama-cli -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 -c 32768 --jinja -ngl 99
Coding Agent Mode (server)
To plug Qwen into OpenCode, pi, or any OpenAI-compatible client, run the server instead of the CLI:
./build/bin/llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 -c 131072 --jinja -ngl 99 --cache-type-k q8_0 --cache-type-v q8_0 --reasoning-budget 4096 --host 127.0.0.1 --port 8080
Point your harness at http://localhost:8080/v1. Swap in the model tag for your tier (for example unsloth/Qwen3.5-4B-GGUF:Q4_K_M with a smaller -c on a 4GB card).
Security:
llama-serverhas no authentication by default. Keep--host 127.0.0.1unless you need other machines to reach it. If you do bind to0.0.0.0, anyone on your network can use it β put it behind a firewall or set--api-key.
Model Comparison
This is the only section where you need to think about model choice.
Quick Summary
| Model | Speed | Quality | Hardware | Use Case |
|---|---|---|---|---|
| Qwen3.5-4B | β‘β‘β‘ | LowβMed | 4β8GB GPU / 16GB laptop | quick Q&A, small functions |
| Qwen3.6-35B-A3B | β‘β‘β‘ | Medium | 16GB GPU / 32GB Mac | fast chat, tools, pipelines |
| Qwen3.8-27B | β‘β‘ | High | 24GB GPU / 48GB+ Mac | default, coding agents |
| Qwen3.8 Flash Next | β‘β‘ | Very High | 128GB unified memory | max quality, broad knowledge |
Tuning That Actually Matters
Most flags don't move the needle. These do. Every flag in this section is present in llama.cpp build 12623; flag names move fast, so on another build check llama-server --help if one is rejected.
Cap the thinking. Qwen at high reasoning effort is wordy: even the 4B writes a full reasoning block for a one-line prompt. --reasoning-budget 4096 keeps it in check, and --reasoning-budget-message "... I am thinking for too long β let me act." helps it wrap up instead of being cut off mid-thought.
Quantize the KV cache, not just the weights. --cache-type-k q8_0 --cache-type-v q8_0 roughly halves context memory with little quality loss. One user reports fitting ~190k tokens of context next to a Q4 27B on a 24GB card (user report).
Try speculative decoding. Current builds support MTP drafting (--spec-type draft-mtp), which needs a model that ships MTP heads, such as the Qwen MTP variants on Hugging Face. One user reports draft acceptance above 80% on code (user report). The speedup wasn't measured for this guide.
For MoE models that don't fit in VRAM, --cpu-moe (-cmoe) keeps the expert weights in system RAM while the rest stays on the GPU, and --moe-cache-mib N uses N MiB of spare VRAM as a cache for the experts that get used most. Raising the micro-batch with -ub 2048 (or 4096; the default is 512) speeds up prompt processing a lot in that setup. Users report up to twice the prefill speed with these settings (user report). The speedup wasn't measured for this guide.
On Macs, macOS caps GPU memory at about β of RAM by default. Raise it (here for a 64GB machine):
sudo sysctl iogpu.wired_limit_mb=54000
This resets on reboot.
What Changed With Qwen3.8
For a long time, the honest answer to "can I code with a local model?" was "sort of." The 7β8B models that fit on ordinary hardware could autocomplete, but struggled once a task needed planning across files. Every few months someone announced that small models were finally good enough, and real-world use often said otherwise.
Qwen3.8-27B is where a lot of local users say that changed. It isn't a frontier model, and nobody claims it is. But people report using it as a coding agent β reading files, running commands, fixing their own mistakes β when they give it a clear plan and the right context. The common view in those discussions is that around 27B is where local models stop needing constant hand-holding. That's practitioner opinion rather than a benchmark result, so test it on your own tasks before you rely on it.
The MoE variants matter just as much, because they change who can run this class of model. A dense 27B needs all its weights in fast memory on every token, which is why it crawls on a 32GB laptop. The 35B-A3B only touches a small slice of its weights per token, so it runs fast on hardware that would choke on the dense model, and degrades more gracefully when part of it spills into system RAM. Flash Next stretches the same idea upward for 128GB machines. And at the bottom end, a 4B runs at a usable speed on a 4GB laptop GPU β limited, but real.
The rough edges are real too. Qwen thinks too much if you let it, its knowledge has gaps that search tools need to fill, and new releases usually need a fresh llama.cpp build. Those are mostly setup problems, not capability problems. The question used to be "is the model good enough?" For many workloads, it's now "how do I set it up well?"