Once DeepSeek V4.1 Flash could run, the harder problem was keeping 261K and eventually one million cold input tokens moving while preserving official sparse semantics and complete output. This final article follows the real long-prefill bottlenecks through cross-layer candidate selection, lossless integer indexing, one-time expansion, dynamic LDS, dense tiling, chunk sizing, and arena reuse. It reports the retained 1,553.91-token/s long-input result, a complete one-million-token run, and current single- and eight-request decode throughput.
Read the full articleFrom official cross-layer candidate selection and lossless index expansion to dynamic LDS, dense tiling, chunk sizing, and arenas: the final DeepSeek V4.1 Flash optimization path and performance data on eight ROCm GPUs
From image_url through the vision tower, aligner, and dual-bias MoE routing: tracing tensor differences while integrating DeepSeek V4.1 Flash vision
ROCm integration in practice: interpreting FP8 weights and carrying shared KV, sparse selections, and mHC state across GPU boundaries
ROCm engineering in practice: from memory lookup to AVX-512, NUMA placement, and CPU/GPU overlap
We put MiniCPM5-2B inside zllm-app and ran it fully offline on a Qualcomm NPU. With one phone, a time traveler can grow food, make fertilizer, brew, translate, train teams, and bootstrap civilization.
Using 1 TiB of RAM per node to target 15+ times the active context capacity, with almost no decode loss measured at 50K. The decisions, failed approaches, and evidence behind the result.
Dual Intel Xeon E5-2696 v4 CPUs and a 12 GiB GeForce RTX 3060 running Qwen3.8-Flash-Next (UD-Q4_K_XL).
We added K2-Horizon MoVA to zLLM, ran its IQ3_XS GGUF with Q8 KV cache and Metal replay on a 24 GiB Apple M5, then tested its reasoning and Chinese writing.
This may be the lowest-cost setup for a genuinely usable GLM-5.3 deployment.
A token-to-token tour of Transformer inference, mapped to zLLM's CPU oracle, attention, KV cache, FFN/MoE, and prefill/decode modules.
Why quantization makes large models fit and can make them faster; why 4-bit is often the sweet spot; and the engineering boundaries around calibration, quality, output fences, and reasoning drift.
How zLLM separates model specifications, full-model runtime, domain semantics, backend capabilities, and device implementation, with platform backends and parameterized kernels as peer tracks.
A short prompt to Claude Code or Codex may carry tens or hundreds of thousands of tokens of hidden working context. This article explains the three major HTTP APIs, prefix-cache hits, VRAM-to-SSD backup, and token-level tool-call fences.
A native executable measured in megabytes plus Gemma 4 E4B is enough for local chat and image understanding on a Mac—and the same engine embeds directly into Rust applications.
Why zLLM chose Rust and a clean-sheet design to own model execution, resource lifecycles, and hardware capabilities in a native inference engine.