A Vintage Machine Can Run a Top-Tier LLM

Two old Xeons, one RTX 3060, and about 125 GiB of system memory: that is the machine used to run Qwen3.8-Flash-Next.

zLLM now runs the complete 48-layer text model on this system and has connected it to a resident service. With CUDA-driver-owned pinned host memory, the latest service regression delivers about 6.0–6.4 tokens/s, passing 50 requests and four long generations without failure.

The current highest result is a separate, single-process MTP benchmark: 13.158 tokens/s. It reproduces the original 42-token template prompt and 128-token output with a 5.25 GiB expert cache, confidence 0.6, prefetch 0, chunk size 64, a 0.25 GiB MTP cache, four draft tokens, Q8G64 KV, frequency cache and expert_transfer_group=1. Acceptance was 97.09%. The previous 12.708-token/s run used the same workload and parameters; pinned target and draft sources reduced verify time from 8.960 s to 8.662 s and draft time from 0.758 s to 0.713 s, with identical upload bytes and token sequence. This is a benchmark record, not the resident service baseline, and it excludes model preloading.

The hardware and model

Component	Specification
CPU	Dual Intel Xeon E5-2696 v4
GPU	NVIDIA GeForce RTX 3060, 12 GiB VRAM
Memory	125 GiB RAM
OS	Fedora 44
GPU stack	NVIDIA driver 610.57.04, CUDA 13.3
Link	PCIe 3.0 x16
Model	Qwen3.8-Flash-Next, GGUF architecture qwen4exp
Weights	unsloth/Qwen3.8-Flash-Next-GGUF, four UD-Q4_K_XL shards

The model's quantized expert weights alone occupy 77,017,907,200 bytes, about 71.73 GiB. A 12 GiB GPU cannot hold them all. The model is a mixture-of-experts network, however: each layer routes a token to 10 experts out of 512, plus shared experts. System memory holds the complete expert set while the GPU receives only the experts needed for the current layer.

How the model fits in 12 GiB

Non-expert weights remain quantized in VRAM. The 71.73 GiB of experts stay resident in host memory and are uploaded on demand. Large PLE tables and embeddings are accessed by selecting only the compressed rows needed for the current token history.

The runtime keeps experts in their GGUF formats instead of expanding everything to F16. CUDA kernels consume Q4_K, Q5_K, Q5_1 and Q8_0 data directly. A fixed-size VRAM arena tracks ownership and reuse, so long generations cannot exhaust the card through allocator fragmentation.

“On demand” here means host-to-device transfer: the experts are already in RAM. PLE and embedding access can still cause limited disk reads.

Why the architecture makes new models easier to add

Qwen3.8-Flash-Next combines Hyper-Connection, GDN, PLE, sparse QSA attention and 512-way expert routing. zLLM can still add it without cloning an entire inference stack because model semantics and device execution are separate.

1. Describe the model. src/model_spec/qwen4exp.rs declares the 48-layer layout, tensor shapes, routing constants, QSA/PLE/HC parameters and GGUF metadata.
2. Implement the model. src/runtime/qwen4exp/mod.rs loads weights and assembles the forward pass. cpu.rs is the reference implementation; cuda.rs and cuda_mtp.rs compose the required CUDA and MTP paths.
3. Register the service. Configuration and Node registration add Qwen4ExpNodeModelConfig and Qwen4ExpCudaEngine, while reusing chat templates, streaming, cancellation, stop/EOS handling and the scheduler protocol.

Those five model-specific files total about 2,228 lines of Rust. They reuse the existing GGUF loader, KV cache, CUDA context and streams, fixed VRAM arena, expert LRU, quantized kernels, service protocol and lifecycle management. Expert transfer lives in backend/cuda/expert.rs; packed Q4/Q5/Q8, attention, routing and tensor kernels are shared modules. New models define how weights and computation are interpreted, rather than rebuilding the runtime.

Correctness and optimization are verified separately. CPU reference comparisons, tensor-format checks, operator tests and token-by-token output comparisons answer “is it correct?” Arena reuse, grouped DMA, cache policy and MTP draft length are then measured with complete requests to answer “is it fast?” A faster microbenchmark is not enough to ship a change; it must also pass reference comparisons, long-output stability and end-to-end throughput checks.

MTP results and their limits

Shared MTP uses a draft layer to propose tokens and the full 48-layer target to verify them.

Workload	Prompt / output	Ordinary	MTP
Coding task	42 / 128 tokens	10.779	12.708
Coding task	42 / 256 tokens	10.682	11.239
Chinese chat	43 / 256 tokens	11.371	11.238

A later run used a different 59-token Chinese coding prompt and reached 8.32 tokens/s with 74.6% acceptance. It is not comparable with the 13.158 result. MTP works in the standalone benchmark loop, but the resident Node MTP path is still converging, so the current service runs without MTP.

The bottleneck is data movement. In one 256-token test, each output token corresponded to about 632.5 MB of expert uploads. At a measured 11.90 GB/s transfer rate, the lower bound is already about 53.2 ms per token, before computation and synchronization. That is why simply increasing draft depth does not guarantee 20 tokens/s.

Service lessons

The first persistent-service failures came from two independent bugs: an incorrect PLE row width and an unsafe asynchronous DMA path reading registered ordinary heap memory. The final implementation loads experts into CUDA-driver-allocated pinned host memory and uploads from there. It restores the 6.0–6.4 tokens/s range and passed the 50-request regression.

Pinned memory needs operational care. A normal exit can return roughly 95 GiB to the driver's host pool without immediately restoring the operating system's MemAvailable; the next allocation can reuse it, but a memory preflight may reject the run. A forced SIGKILL can leave NVIDIA driver accounting behind and require a reboot. Production shutdown should therefore be graceful.

Long-context prefill is a separate cost. A recorded 10.6K-token input took 351.2 seconds after sparse QSA integration, so first-token latency can be measured in minutes. The tested context configuration reaches 65,536 tokens, but that is not a claim of a full 65,536-token benchmark. Vision support and complete logits alignment remain future work.

This is what the vintage machine provides: enough RAM to hold a modern MoE model, a consumer GPU to execute its active experts, and a runtime that keeps compressed weights, transfers and lifetimes under control. The result is practical text inference on hardware that appears far too small on a VRAM specification sheet.

---

Based on the zLLM Qwen3.8-Flash-Next integration record and its 2026-09-07 follow-up. Historical benchmark records and current service validation are reported separately; benchmarks were not rerun for this article.