Top stories from r/LocalLLaMA and r/localllm — curated & summarized daily
Community guide for running Qwen 3.6 27B with Multi-Token Prediction in llama.cpp, achieving 2.5x faster generation with 262K context on 48GB of memory. MTP is now out of experimental mode — enable with --spec-type mtp for noticeably faster long-form output (code, writing) with zero quality loss, though short Q&A sees less benefit.
llama.cpp has merged support for MiniMax M3, including its MiniMax Sparse Attention (MSA) architecture. Multi-GPU setups with two RTX PRO 6000s can run the full model, while consumer rigs need aggressive quantization. The EAGLE3 speculative decoding fork adds draft-model acceleration. Watch out for broken UD-Q4_K_XL quantizations from early GGUF conversions.
Google pushed a stealth update to Gemma 4 that fixes broken tool-calling JSON — payloads now close cleanly, escape properly, and no longer double-emit closing braces on retries. A companion truncation fix shipped together. This makes Gemma 4 viable for local agentic workflows in Ollama, vLLM, and llama.cpp without manual JSON repair.
Community benchmarks show ik_llama.cpp (a performance-focused llama.cpp fork) raising Qwen 3.5 27B prompt evaluation from ~43 tok/s to 1,122 tok/s on an RTX PRO 4000, with generation climbing from 7.5 to 26 tok/s. The fork offers SOTA quantization types and better CPU performance. Best for long-context workloads where prefill time dominates.
A practical checklist born from a failed local tool-calling setup: OpenWebUI configuration, native tool-call support, quantization level, and runtime choice (Ollama vs llama.cpp vs vLLM) all affect reliability. Key takeaway — tool calling breaks more often from harness misconfiguration than model capability. Start with Qwen 3.6 or Gemma 4 post-fix, use Q4_K_M or higher, and verify your chat template.