The Essential App for Time Travelers: Modern Civilization in One Phone

2026-09-09T00:00:00+00:00

Suppose you wake up tomorrow in the ancient world.

There is no power grid, no internet, no search engine, and certainly no cloud LLM. You cannot recite an encyclopedia of preindustrial technology from memory. You vaguely remember crop rotation from a video, but when a harvest depends on it, all that remains is one terrifying question: how exactly was that done?

Other time travelers bring a lighter. You bring a phone running zllm-app.

Inside it is the complete MiniCPM5-2B model. The weights are the knowledge. No network, no server, and no prewritten question list are required. Growing grain, making fertilizer, brewing alcohol, translating languages, building waterwheels, organizing workshops, training a disciplined force—if you can ask it, you can start working on it.

Add a foldable solar panel and a power bank: harvest sunlight by day, summon modern civilization by night.

This is not time travel. This is administrator access. </video>

First we type a time-traveler opening question. MiniCPM5-2B immediately lays out a path through farming, fertilizer, brewing, and disciplined organization. Then we type one Chinese sentence and get English, Japanese, and Korean on the spot. Airplane mode cannot stop it, because every answer is generated inside the phone.

Put the artifact in your pocket</h2> Download zllm-app 0.1 SM8635 Preview APK (8.7 MB)</a>
SHA-256: d3cbc48b928da4932cea37c4538dc0ed687336b1c4d5d93d18692b3f92d2cb5a</code>
This is a directly installable technical preview. The APK does not cram a 2B model into the installer. On first launch, it races Hugging Face against ModelScope and downloads the faster copy of the roughly 2.6 GiB core model, with resumable transfers and SHA-256 verification. Once the model is in place, conversations work without a network.
Current device restrictions</h3> Item</th> Current support boundary</th></tr></thead> Fully validated phone</td> Motorola XT2451-4</td></tr> SoC / NPU</td> Qualcomm SM8635 (Snapdragon 8s Gen 3) / HTP V73</td></tr> Memory</td> 12 GB validated; lower-memory devices are not currently supported</td></tr> OS and ABI</td> Android 16 and arm64-v8a validated</td></tr> Storage</td> Core model is about 2.6 GiB; reserve at least 4 GiB</td></tr> </tbody></table> The current model package and QNN context graphs are built specifically for SM8635 / HTP V73. Other Snapdragon chips and HTP versions, as well as MediaTek, Samsung, and Kirin platforms, are outside the current support boundary. The artifact has arrived—but this release answers to one NPU only. Civilization is not in the cloud. It is in the weights.</h2> Most AI apps become an empty input box when the network disappears. zllm-app is different: the MiniCPM5-2B weights, tokenizer, chat template, and inference engine all live on the phone. You ask; the phone computes. The model answers; the text is generated locally. Nothing needs to be sent to a server, and no reply has to travel back from an internet connection a thousand years in the future. Text ───────────────────────┐ ↓ Voice → local SenseVoice → MiniCPM5-2B ↓ Qualcomm HTP/NPU ↓ Streaming output and saved chats </code></pre> As long as the phone can light up, the knowledge is still there. We actually ran it on a Qualcomm NPU</h2> “An LLM on a phone” is easy to put on a slide. Making it real is a different kind of battle. Our test device is a Motorola XT2451-4 with an SM8635, 12 GB of RAM, and HTP V73, running QAIRT 2.38. We did not dump the model onto the CPU and wait. We integrated Qualcomm QNN directly: dynamically loading libQnnHtp.so</code>, creating the backend, device, context, and execution graphs, then sending MiniCPM5-2B's core computation to the phone's NPU. The embedding, all 42 Transformer layers, final normalization, and vocabulary projection now run on HTP. The SenseVoice-Small encoder and CTC head run there as well: hold the microphone, speech becomes text locally, MiniCPM5-2B receives the question, and the answer streams onto the screen. The integration produced enough traps to fill a book: Activation outliers in the fixed prefix distorted the quantization range, driving K/V errors across all 42 layers to unacceptable levels.</li> On this hardware and SDK combination, QNN could not use the S32/F16/F32 output paths we originally expected, so we redesigned the path around S8 input, S8 output, and scale restoration.</li> Full INT4 was fast but drifted during long generations; full INT8 preserved quality but reached only 8.97 tok/s.</li> DSpark speculative decoding ran end to end, but weak draft acceptance reduced it to 5.723 tok/s—slower than ordinary decoding.</li> Reloading 43 graphs for every request cost several seconds, so decode and ASR received resident runners that load the graphs once and reuse them.</li> </ul> The final 2B INT8 configuration sustains roughly 8.7–9.8 tok/s on long answers. Reusing resident graphs reduced the second in-app question from 14.1 seconds to 9.65 seconds. For long inputs, batched AR8 prefill takes over: a fresh 831-token conversation fell from 95–143 seconds to 36.9 seconds. This is not “NPU support” in a presentation. It is a real phone, real QNN graphs, and a complete path from the text box to streaming output. Question one: how do you survive the first week?</h2> While everyone else is composing poetry, tracing family trees, and looking for a patron, you open zllm-app and solve the problems that actually matter: water, food, shelter, safety, local climate, and the local power structure. Seconds later, the phone gives you a plan for day one, the first three days, and the first week. They are still trying to identify the dynasty. You are already building a base. Question two: how do you double the harvest?</h2> In the ancient world, the real hard currency is not gold. It is grain. You do not need to memorize an entire agronomy library. Tell zllm-app the climate, the crop, the sowing date, the color of the leaves, and last year's yield. It breaks the problem apart: seed selection, seedlings, rotation, compost, drainage, spacing, weeds, pests, harvest, drying, and storage. In year one, farmers keep control plots and record seed, fertilizer, and yield for every field. In year two, “the elders say so” becomes a set of comparable records. Which seed produces more? Which compost works? Which field floods? Now you have evidence. Everyone else depends on the weather. You begin agricultural experiments. Once the harvest grows reliably, you are no longer feeding only yourself. You can support craftsmen, caravans, guards, and an expanding city. Question three: how do you make fertilizer without a chemical plant?</h2> A time traveler cannot build a chemical factory on day one. With a model, however, you know where the technology tree begins. Start with mature compost, green manure, crop rotation, nitrogen-fixing legumes, wood ash, and drainage. Then distinguish nitrogen, phosphorus, and potassium deficiencies from plant symptoms. Change one variable at a time, keep small control plots, and record inputs against yield. The frightening advantage is not memorizing one recipe. It is knowing how to experiment, debug, and turn a lucky accident into a repeatable method. One village learns it; ten villages copy it. Ten villages prove it; the yield of an entire region begins to rise. That is what knowledge can do. Question four: what comes after surplus grain?</h2> Once agriculture creates a surplus, fermentation is the next branch of the technology tree. Temperature, sanitation, containers, timing, batch records—zllm-app turns mystical “ancestral intuition” into a process. Record the ingredients, the day each jar was filled, when bubbling began, and when an off smell appeared. Reliable alcohol becomes trade goods. Vinegar seasons and preserves food. The same fermentation discipline extends to sauces, pickles, and long-term storage. Other households brew by luck. Your workshop begins quality control. Question five: no common language? Your phone is the foreign office.</h2> You may not land in the imperial heartland. You might appear on a trade route, at a frontier town, in a port, or among people whose language you cannot understand at all. Translation is not a luxury there. It is survival. MiniCPM5-2B's on-device quality regression includes translation samples. The SenseVoice-Small front end supports Chinese, English, Japanese, Korean, and Cantonese; Chinese, English, and Cantonese all passed the phone's HTP transcription gates. A visitor can speak to the phone, while zllm-app extracts the meaning, translates it, and rewrites your reply to sound more polite, formal, or suitable for trade. Others meet a foreign caravan and resort to gestures. You negotiate prices, ask for routes, read contracts, and compose replies. One phone becomes translator, scribe, and diplomatic adviser. Trade routes stop being lines on a map. They become your intelligence network, commercial network, and talent pipeline. Question six: how do you build a formidable ancient force?</h2> Weapons alone have never determined the strength of an army. zllm-app gives you an organizational system: teams of ten with clear responsibility; standard commands, formations, and assembly times; regular conditioning, load carrying, marching, watch duty, first aid, and evacuation drills; inventories for provisions and equipment; records for illness and injury; consistent rewards and discipline; a mandatory review after every operation. Other commanders train according to their mood. You use a standard schedule. Other columns scatter on the march, lose track of supplies, and garble commands. Your people know who is responsible for whom, where the supplies are, and where to report a problem. In peace they repair levees, fight fires, guard grain, and move cargo. In crisis they act together instead of dissolving into chaos at the first alarm. What you brought is not a legendary weapon. It is modern organizational science. That is more powerful than a legendary weapon. One phone is an entire technology tree</h2> The classic time-traveler mistake is trying to build a steam engine immediately. The real sequence is different: secure water and food, then establish records and measurement. Once the food supply is stable, organize craftsmen. With craftsmen, improve tools, kilns, waterwheels, and workshops. Only after surplus capacity exists do you advance metallurgy, machinery, transport, and education. What you have</th> What comes next</th></tr></thead> A field and some seed</td> Rotation, seedlings, compost, control plots, grain storage</td></tr> Surplus grain</td> Brewing, vinegar, food processing, warehouses</td></tr> Contact with other languages</td> Translation, trade, diplomacy, technical intelligence</td></tr> Carpenters and blacksmiths</td> Better tools, waterwheels, standardized dimensions</td></tr> Reliable tax grain</td> Support craftsmen, open schools, build roads, train teams</td></tr> Mature workshops</td> Metallurgy, machinery, printing, scaled production</td></tr> </tbody></table> The model does not conjure machines from thin air. It ensures that you always know the objective, the principle, the missing prerequisites, and the first step. The time traveler's real cheat code</h2> One phone. MiniCPM5-2B. zllm-app. A Qualcomm NPU. One foldable solar panel. No network, yet questions still receive answers. No cloud, yet text still appears. Speak once and SenseVoice recognizes it locally. Leave mid-conversation and the chat and KV state remain saved. Start another topic and it receives a completely independent conversation. Other time travelers rely on whatever fragments they can recall. You carry an agronomist, process engineer, translator, historian, drillmaster, and chief technology officer who never sleeps. From the first cup of clean water to the first high-yield field; from the first reliable jar of alcohol to the first disciplined organization; from the first cross-language trade to a technological system that keeps compounding—it all begins with one sentence typed onto a phone. This is zllm-app. Completely offline. Modern civilization in your pocket. If we must find one missing piece, this time-travel artifact still lacks TTS. Once it can speak an answer aloud, the modern strategist inside the phone will not merely write your plans. It will have a voice. Perhaps that is for the best. An artifact this powerful should not be revealed lightly. We will save that voice for the next release. From 1M Total to 15+ × 1M Contexts: GLM-5.3 KV Cache Offloading 2026-09-08T00:00:00+00:00 The biggest benefit of this optimization is capacity: moving from roughly 1M tokens of total context toward 15+ concurrent decoding sessions, each carrying a 1M-token history. The same GPUs can support a projected 15+ times as much active history. Previously, one million-token conversation could consume almost the entire system's context budget. Now, each machine's 1 TiB of host RAM holds MLA history, while the GPUs keep a 32K hot cache for current computation. The aim is to let multiple users keep generating against their own million-token histories, beyond simply archiving completed sessions. 15+ × 1M is a capacity estimate, not a completed concurrency benchmark. We show the memory accounting below. The nearly lossless speed result comes from completed 50K comparisons. These answer two separate questions: how much history the system can hold, and how much speed offloading costs. Our measured result has a specific scope: with 50,023 input tokens, 1,024 output tokens, and a 32K GPU hot cache, storing MLA history in host RAM reduced throughput by only 0.15%–1.30% across three MTP=3 pairs. Without MTP, three pairs were about 2% faster. Every complete output matched its corresponding baseline. Getting there took several reversals. We first tried moving indexing to the CPU, then backed away. The hot-cache path once lost roughly 30% of throughput. Several convincing local optimizations failed to improve the full model. The remaining instability eventually led us from attention kernels into Linux memory management and the GPU driver. This article follows those decisions: why we tried each path, which evidence justified keeping it, and what forced a change of direction. Separate historical capacity from the current working set</h2> We used the two-node system from our previous GLM-5.3 optimization article</a>: eight AMD Radeon PRO W7900D GPUs per machine, 48 GiB per card, an EPYC 9334 platform, and 1 TiB of RAM per node. Intra-node communication uses HIP P2P; TCP carries activations across node boundaries. There is no XGMI. This work changes KV placement. Weights and layer computation stay on their assigned devices, and historical data stays in the host memory of the node responsible for those layers. The opportunity comes from GLM-5.3's sparse attention path. The DSA indexer selects the top 2,048 historical positions; MLA attention consumes their KV. Some subsequent layers reuse the selection, while retaining their own KV contents. The amount of history that must be preserved can therefore be managed separately from the amount consumed by one attention operation. In this Q8G64 MLA representation, each row contains 512 B of latent data, 16 B of scales, and 128 B of RoPE data: 656 B in total. The engineering records give the following payload sizes. These cover MLA only, excluding DSA, MTP layers, metadata, alignment, and scratch space. Object</th> MLA payload</th></tr></thead> 50,000 tokens, one layer, one replica</td> 31.281 MiB</td></tr> 50,000 tokens, 78 main-model layers, both replicas</td> 4.765 GiB</td></tr> 2,048 selected rows, one layer, one replica</td> 1.28125 MiB</td></tr> 32,768 hot-cache rows, one layer, one replica</td> 20.5 MiB</td></tr> </tbody></table> One 50K history is manageable, but its cost accumulates with conversation length and session count. We wanted the main GPU MLA history allocation to be bounded by the hot-cache size, with growth absorbed by RAM. This hot cache is not a sliding window that discards everything older than 32K tokens. Old positions remain in RAM, DSA can still select them, and a cache miss reads them back. Physical placement changes without deliberately truncating accessible history. Decision one: should the CPU store history and run the indexer?</h2> The first proposal was natural: the host has abundant memory and CPU cores, so let it score the full history and select Top-K, then send the selected KV to the GPU. There was also a scheduling opportunity. Once selection was known, layers reusing it could prefetch historical KV while independent GPU work continued. We already had CPU Q8 scoring, blocked layouts, and vectorized implementations. We added a fixed worker team, local histograms, reusable Top-K workspaces, incremental mirrors, rollback, and prefetching. This deserved a test rather than dismissal based solely on CPU versus GPU arithmetic speed. Hardware testing produced two independent reasons to reject it for this iteration. First, performance: the 50K GPU baseline delivered about 14.03 tokens/s, while CPU DSA candidates reached only 10.92–11.32 tokens/s. Query readback, thread coordination, selection upload, and GPU waiting joined CPU arithmetic on the critical path. Older CPU measurements at 131K could not simply be multiplied by 50/131</code>, because the number of active workers changed with context length too. Second, correctness: final selection using CPU Q8 queries differed around the cutoff from the GPU BF16-query exact path. Fixed greedy generation diverged. Internal CPU consistency did not establish equivalence to the original GPU path. Decision: keep exact DSA on the GPU, disable CPU selection, and continue moving MLA history into RAM. This narrowed the problem. Storage expansion did not need to wait for CPU indexing to succeed. Prefetching remained a useful overlap opportunity, but could not be credited as zero overhead in advance. Decision two: let the CPU manage the GPU hot cache first</h2> With selection still computed on the GPU, the early hot-cache path sent it back to the host. The CPU looked up mappings, checked hits, packed missing rows, and submitted uploads and attention. It established the capacity direction, but performance was poor. During the September 7 experiments, the real 32K hot-cache configuration, after packed replication and batched-mirror fixes, still had a three-run median of about 10.41 tokens/s. That phase reported roughly a 29% loss against its reference. Those early results included different diagnostic conditions and must not be combined with the final tests as one strict A/B comparison. Two concrete problems appeared first. The offload branch had lost the existing compressed replication path. Previously, the owner quantized once and copied 656 B of packed KV per row to its peer. In hot-cache mode, fallback appended and requantized on both sides, also losing existing channel overlap. We restored packed replication to the hot-cache path. The number of small transfers mattered. Updating host mirrors per layer and token could create 156 stream operations across 78 layers and their owner/peer replicas. Splitting each row into three transfers and increasing outstanding work raised that to 468 operations. Throughput fell to roughly 8.8 tokens/s. That experiment rejected the assumption that more asynchronous concurrency would solve transfer overhead. The payload was already small; additional submissions could cost more than additional bytes. Copying back in batches of 64 rows sharply reduced operation count and improved the full-cache phase. A substantial gap remained once the cache actually entered hot mode. We then moved peer-mirror maintenance into the worker's locked section and made the append kernel also write a contiguous log. The three-run medians were about 10.39 and 10.06 tokens/s, respectively. Neither delivered the expected improvement. Decision: retain batched logs and necessary lifecycle fixes, but withdraw the claim that moving locks and adding log writes would close the main gap. Completing an implementation and validating its performance hypothesis are separate milestones. A mistaken diagnosis: the largest share does not identify the regression</h2> We added synchronized tracing around hot-cache attention. Kernels accounted for 77.4% of measured time, and host locking plus remapping for about 22%. We initially concluded that the remaining problem was primarily the window-attention kernel. Reviewing the evidence invalidated that method of attribution. The profile described the candidate's absolute time breakdown. It did not establish where the difference between GPU residency and offloading arose. The resident path also runs attention. Moreover, synchronization and tracing reduced full-run throughput to about 4.21 tokens/s, substantially changing submission and overlap behavior. Source inspection also found that selected-gather functionality, proposed as a next step, already existed. We needed to verify the actual path and its dependencies rather than treat an existing mechanism as missing work. We changed the measurement procedure: diagnostics locate problems; formal throughput runs disable profiling, tracing, and extra synchronization. Each candidate gets a same-version GPU baseline, and complete outputs must match before comparing speed. The quantity to explain is the difference between the two paths. Decision three: let the GPU manage the hot cache directly</h2> New evidence brought us back to the submission chain. The GPU had already computed selection, yet the CPU waited for readback and performed mapping and packing before attention could proceed. Host-observed selection readback waits reached about 0.7–1.2 ms. They included earlier queued GPU work, so they were not pure copy times, but they exposed the control round trip. We moved cache management. Selection stays on the GPU, which directly checks hot slots. Hits read VRAM; misses read the registered RAM mirror. Gather produces a compact selection for attention. GPU exact DSA ──→ selected historical positions │ ▼ GPU hot gather ├─ hit: read VRAM hot slot Local RAM history ─────└─ miss: read registered host memory │ ▼ compact KV → GPU attention New KV → GPU recent ring → batched copyback → RAM history </code></pre> Host memory supplies historical capacity while indexing, cache lookup, and attention continue on the GPU. The CPU still manages resources and lifetimes, but no longer coordinates lookup and transfer for every selection. PCIe reads still have a cost. Some early sampling windows had less than 1% misses, showing reuse in this workload; that does not promise the same hit rate for every input. The first GPU-managed version reached about 11.72 tokens/s, with an initial registration stall. Moving registration earlier and completing multirow handling even produced a run at 5.13 tokens/s. We kept investigating the direction without calling near-baseline steady-state event spacing a passing full-run result. GPU cache management has hazards too</h2> MTP exposed problems that were less visible on the single-row path. Verification processes multiple rows, and several queries can select the same missing position. An early implementation allowed one block within a gather invocation to read a slot another block had just filled. A real MTP test produced abnormal scales, followed by a failure to find a finite selectable token. We tightened visibility: slots filled in the current round become hit sources only in the next round; duplicate misses within the same round read RAM directly. Slots used by the current round are pinned, and a recent ring protects new tokens that have not yet reached RAM. The complete 1,024-token MTP output then matched the same-version resident baseline, but throughput was 16.175 versus 27.530 tokens/s, a loss of about 41.2%. Both accepted 665 draft tokens, ruling out acceptance-rate differences as the explanation. Restoring owner-to-peer packed KV replication for small verification batches raised the candidate to 20.035 tokens/s. This was a full execution-path repair worth keeping. Replacement policy brought another surprise. With clock replacement and reference bits, a synthetic case with a full hot cache, all reference bits set, and only 1% misses took about 1.03–1.07 ms per gather. A few miss blocks had to scan long stretches. More misses could actually run faster because more blocks participated in scanning. Replacing it with circular eviction that skips currently pinned slots reduced the same probe to about 0.032 ms for 2,048 rows. Yet full MTP throughput was only 19.767 tokens/s. The probe's large gain did not materialize in the complete model. Decision: keep the simpler replacement policy and remove the demonstrated pathological case, but reject the inference that this local hotspot explained the full performance gap. Decision four: inspect the submission threads beyond attention</h2> CPU sampling restricted to active decode threads found about 21.1% of CPU cycles in host memcpy. The call chain ended in clone</code> on read-only MoE weights: submitting work to a worker could deep-copy the dense router. This was unrelated to KV semantics, but affected the same execution chain. We shared the read-only weight group through Arc</code>, preserving operators and numerical values. The resident baseline received the same fix. The v15 MTP comparison reached 25.425 versus 26.832 tokens/s, narrowing the loss to 5.24%. That was substantial progress, but still just outside the 5% gate. The 21.1% CPU sampling share could not be translated into an equal end-to-end speed gain. Further work reused the recent ring for peer logging, removed duplicate copies, transferred existing prefill buffers into hot-cache ownership, and used compute copies and packing for small logs to reduce DMA queue operations. These changes did not solve everything at once. One no-MTP pair reached a 4.80% loss; a later MTP pair lost 9.87%. Logs kept showing 0.6–1.2 second stalls, even when most event intervals were near the resident baseline. We did not discard the first 128 tokens or declare victory using p50 alone. Those stalls were part of the user's wait and remained in the original complete-decode timing window. Decision five: follow the stalls into OS page migration</h2> Device timelines showed that the long stalls did not necessarily occur in hot gather. One capture showed maximum gather times of only about 0.054–0.056 ms, alongside a roughly 792 ms synchronization wait and almost no device work across the pipeline. Blaming whichever kernel appeared stretched in the profiler would repeat our earlier error. We began comparing system settings and driver behavior across the two machines. Disabling automatic NUMA balancing helped, but was insufficient</h3> Amd-1 had kernel.numa_balancing=0</code>; Amd-2 had it set to 1</code>. Long stalls concentrated on the latter. Disabling it consistently produced a first MTP pair with only 0.51% loss. But the third interleaved pair lost 5.49%, stopping the gate. We retained consistent settings and continued investigating, instead of averaging away the failed pair. Pre-touching registered tail pages did not solve the problem</h3> Another hypothesis was that unused tail pages in the RAM mirror's reserved capacity had not yet been written, and first-touch writes during append disrupted GPU mappings. We zeroed spare capacity before registration without changing valid KV bytes or logical length. The candidate still showed 7.35% loss and second-scale gaps. Pre-touching did not resolve the instability and could not be presented as the final cause or solution. Proactive compaction finally yielded call-chain evidence</h3> BPF tracing tied the actual inference process to invalidation addresses and restore workers, capturing this path: kcompactd → proactive_compact_node → migrate_pages → try_to_migrate_one → amdgpu_hmm_invalidate_hsa → related queue restoration work </code></pre> This established a path through proactive page compaction, HMM invalidation, and queue restoration. Many sampled invalidation addresses did not intersect the explicitly registered KV mirrors, so they could not be explained as first writes to KV tail pages. This capture did not reproduce the earlier full one-second stall; we did not claim to have attributed every long pause individually. We changed vm.compaction_proactiveness</code> from 20 to 0 on both machines, with automatic NUMA balancing still disabled. Linux documents that zero disables proactive compaction and that page migration can cause application latency spikes. The system-specific conclusion still required complete paired runs under identical settings. Linux kernel documentation</a> These were runtime changes on the experimental machines, with original values and restoration commands recorded. Boot configuration was unchanged. These settings are part of the test conditions, not a universal tuning prescription for every GPU system. Six interleaved pairs establish the nearly lossless result</h2> The final gate used the same v23 binary, 50,023 input tokens, 1,024 output tokens, greedy decoding, and fixed seed01. The offloaded configuration used a 32,768-row hot cache; the baseline kept KV on the GPU. Both disabled automatic NUMA balancing, proactive compaction, profiling, tracing, attach support, and CPU DSA. We restarted the experimental runtime for each run and interleaved resident and offloaded ordering. Throughput retained the original first-text-event-to-SSE-DONE window, with no stalls removed. Mode</th> Pair</th> RAM history + GPU hot cache (tokens/s)</th> GPU resident (tokens/s)</th> Offload throughput loss</th></tr></thead> MTP=3</td> 1</td> 27.839</td> 27.883</td> 0.157%</td></tr> MTP=3</td> 2</td> 27.923</td> 27.966</td> 0.150%</td></tr> MTP=3</td> 3</td> 27.738</td> 28.102</td> 1.297%</td></tr> No MTP</td> 1</td> 14.042</td> 13.788</td> −1.846%</td></tr> No MTP</td> 2</td> 14.029</td> 13.753</td> −2.007%</td></tr> No MTP</td> 3</td> 14.029</td> 13.747</td> −2.056%</td></tr> </tbody></table> All 12 requests completed, and full output-file hashes matched the reference for their respective modes. Negative loss means offloading was slightly faster. We interpret that as essentially matching the baseline on this workload, not a guarantee that offloading accelerates inference. This validates single-session decoding with long history. Initial prefill, appended inputs, multi-session capacity, and concurrent throughput each need their own tests. Capacity optimization must survive the session lifecycle</h2> One uninterrupted generation is not enough for real conversations. Users append inputs, MTP rejects drafts and rolls back, and sessions are saved and restored. The successful path has an explicit consistency boundary. New KV may still be in the GPU recent ring or an in-flight copyback. Before exporting or restoring full history, submit outstanding logs, wait for copyback, and verify that CPU row count has caught up with logical row count. An enqueued asynchronous copy is not yet a completed RAM copy. Large append prefill also differs from decode. To process many new rows, the current implementation restores the required full-history GPU MLA KV from RAM, performs the append, and returns to hot-cache mode for decode. Small prefill tail chunks must not be mistaken for verification, and owner and peer must remain consistent. The first real large-append test failed on MTP terminal hidden state. Request completion had saved a raw BF16 residual where the later path expected F32 hidden state after final normalization. The fix aligned normalization boundaries and addressed old-cache restoration. The subsequent v25 same-version comparison reused 50,085 historical rows, appended 9,854 tokens, reached a final prompt of 59,939 tokens, and generated 1,024 tokens. Complete outputs matched. Large append comparison</th> Offloaded</th> GPU resident</th></tr></thead> Time to first token</td> 17.905 s</td> 18.350 s</td></tr> Decode</td> 26.850 tokens/s</td> 27.004 tokens/s</td></tr> </tbody></table> Decode loss was 0.57% in this pair. It established that an offloaded long conversation could continue with a large append, but remained a single append test rather than a replacement for the six single-session pairs. Ten sessions exposed a physical-allocation problem</h2> While preparing ten independent 50K histories, the sixth prefill ran out of memory on the tail node. The highest 1 Hz VRAM sample was about 47.98 GiB. Ten-session simultaneous decoding had not yet begun. The problem was the general GPU memory pool. Small long-lived KV, hot-cache, and session buffers could receive large allocations returned by prefill through best-fit reuse, then hold them indefinitely. A logical allocation of tens of KiB or MiB could retain much more physical space. Accounting reported logical bytes, further understating usage. We enforced exact capacity limits for long-lived caches and changed primary KV/DSA accounting to actual allocation size, while retaining the existing pool policy for temporary activations. Tests first populated the pool with oversized blocks, then checked that small hot-cache buffers did not retain them. After the fix, v28 completed all ten histories and subsequent concurrent appends, delivering 98.56 tokens/s over the full request group. The highest observed VRAM sample was about 45.12 GiB. Concurrency tuning followed. Reducing MTP depth to 1 raised full-request throughput to 125.26 tokens/s. Reducing each stage's decode batch limit from 4 to 1 reached 145.67 tokens/s, with matching inputs, complete outputs, and usage between those latter two configurations. Multiple requests could keep the pipeline moving; deeper speculation and larger batches that help one session need not help ten. Separately increasing the A0 ready-work batch limit produced 145.28 tokens/s, essentially unchanged. With no demonstrated benefit, we removed that change. These concurrent results use actual completion-token totals divided by wall time for the entire append request group, including request boundaries. SSE text-event counts were not treated as token counts. There was no corresponding fully GPU-resident concurrency comparison, so these runs do not establish lossless concurrent offloading. The latest concurrency candidate still needed single-session regression gates. The capacity gain: from roughly 1M total toward 15+ sessions of 1M each</h2> Consider the user-facing change. With an original total budget of about 1M tokens, one user carrying a million-token history could effectively occupy it. After offloading, the target is 15+ such users decoding concurrently, each retaining their own million-token history. Dimension</th> Original capacity basis</th> Estimated offloaded capacity target</th></tr></thead> Total history across active sessions</td> About 1M tokens</td> 15M+ tokens</td></tr> Sessions carrying 1M history each</td> About 1</td> 15+ decoding concurrently</td></tr> Main storage for full MLA history</td> GPU VRAM</td> 1 TiB host RAM per node</td></tr> MLA history payload on GPUs</td> Grows with history length</td> 32K hot cache per session; old positions read on demand</td></tr> </tbody></table> This is the practical meaning of opening up capacity: targeting 15+ times as much active long history on the same GPUs. It expands multi-session historical capacity. Per-session speed at that concurrency still depends on compute, indexing, transfers, and scheduling. Does RAM fit fifteen million-token histories?</h3> Using the same main-model MLA encoding, 1M = 1,048,576 tokens</code>, 78 layers, and both owner/peer replicas: One 1M history = 1,048,576 × 656 B × 78 × 2 ≈ 99.94 GiB 15 sessions ≈ 1,499.06 GiB (about 1.464 TiB) of main-model MLA history Total host RAM across both machines = 2 TiB </code></pre> The budget is 2 TiB across two machines. All history does not reside on one 1 TiB host. MLA history is distributed according to layer ownership. The owner/peer replicas are already included above; they do not mean each host stores the entire model's history. For 15 sessions, the main-model MLA history totals about 1.464 TiB; roughly equal distribution would put about 0.732 TiB on each node. Actual placement follows each node's layer count and requires checking headroom separately. The main-model MLA payload for 15 sessions does not exhaust the two-node RAM budget, leaving room to explore 15+ sessions. This is an encoding-based capacity calculation, not a measured whole-system memory peak or maximum concurrency. Meanwhile, fifteen 32K hot caches require about 46.85 GiB of main-model payload across all GPUs, including both replicas. That component does not grow proportionally when each history expands from 50K to 1M. This is the central capacity advantage. The complete service budget must also cover GPU DSA history, token mappings, MTP layers, per-session workspaces, allocation and registration overhead, and transient VRAM during full-history restoration for prefill or large append. Fitting the MLA bytes in RAM provides a basis for the 15+ session target, but does not constitute whole-system validation at that scale. Completed measurements cover nearly lossless 50K single-session decoding, a large append reaching about 60K, and complete concurrent requests over ten 50K histories. The next meaningful capacity test is to extend each history to 1M and active concurrency to 15+, recording real memory usage, time to first token, and sustained decode throughput. The current 1M admission configuration must also be aligned with that larger aggregate target. The decisions that survived</h2> Approach or claim</th> Outcome</th> Decisive evidence</th></tr></thead> Final DSA selection on CPU</td> Rejected for this iteration</td> Lower speed and divergent fixed greedy output</td></tr> MLA history in RAM with GPU hot working sets</td> Retained</td> Six passing single-session pairs with matching complete outputs</td></tr> CPU coordinates hot slots for every selection</td> Main path moved to GPU management</td> Readback waiting interrupted submission</td></tr> More per-row asynchronous copies</td> Rejected</td> More operations reduced throughput</td></tr> Batched logs and packed owner/peer replication</td> Retained</td> Reduced redundant work and repaired single-row/MTP paths</td></tr> Moving locks or dual-writing logs closes the gap</td> Rejected</td> No substantial complete-run improvement</td></tr> High kernel share proves the regression is in kernels</td> Withdrawn</td> Absolute shares did not explain the resident/offload difference</td></tr> Simpler GPU replacement</td> Retained with limited claims</td> Large probe improvement without matching full-model gain</td></tr> Share read-only weights instead of deep-copying</td> Retained</td> Same-version full-run gap narrowed</td></tr> Disabling automatic NUMA balancing is sufficient</td> Sufficiency rejected</td> Repeated gate still failed</td></tr> Pre-touching registered tail pages fixes the stalls</td> Rejected</td> Second-scale gaps and a 7.35% failed pair remained</td></tr> Consistent NUMA settings and disabled proactive compaction</td> Retained for these machines</td> Kernel-path evidence and six interleaved pairs</td></tr> Small logical buffers imply small VRAM occupancy</td> Rejected</td> Preparation OOM and oversized physical allocations</td></tr> Increase A0 batching under concurrency</td> Removed in this iteration</td> Full-request throughput was essentially unchanged</td></tr> </tbody></table> The reusable lesson is to manage historical capacity, current working sets, and control dependencies separately. RAM provides capacity; GPU hot caches preserve local access; the GPU consumes selection directly; batched logs maintain new history. Host and driver memory behavior also belong in end-to-end measurement. Every change of direction used the same standard: did complete requests improve, did outputs match, and could failures be reproduced? That standard turned spare host RAM into usable long-history capacity, with a path from roughly 1M total toward 15+ simultaneous million-token sessions. Sources and measurement scope: based on zLLM engineering records in docs/glm53-cpu-kv-decode.md</code> (September 5–8, 2026), the two-node baseline in docs/glm53-rocm-amd12-baseline-20260906.md</code>, and inspection of the local GPU hot-cache implementation. Formal single-session results are the six cp0-v23</code> pairs; append results are from append-v25</code>; concurrency results are from concurrency-v28</code> and concurrency-tuning</code>. Historical exploration, diagnostic probes, formal pairs, and later concurrency candidates are identified separately and must not be merged into a single strict cross-version performance curve. No remote benchmarks were rerun for this article. A Vintage Machine Can Run a Top-Tier LLM 2026-09-07T00:00:00+00:00 Two old Xeons, one RTX 3060, and about 125 GiB of system memory: that is the machine used to run Qwen3.8-Flash-Next. zLLM now runs the complete 48-layer text model on this system and has connected it to a resident service. With CUDA-driver-owned pinned host memory, the latest service regression delivers about 6.0–6.4 tokens/s, passing 50 requests and four long generations without failure. The current highest result is a separate, single-process MTP benchmark: 13.158 tokens/s. It reproduces the original 42-token template prompt and 128-token output with a 5.25 GiB expert cache, confidence 0.6, prefetch 0, chunk size 64, a 0.25 GiB MTP cache, four draft tokens, Q8G64 KV, frequency cache and expert_transfer_group=1</code>. Acceptance was 97.09%. The previous 12.708-token/s run used the same workload and parameters; pinned target and draft sources reduced verify time from 8.960 s to 8.662 s and draft time from 0.758 s to 0.713 s, with identical upload bytes and token sequence. This is a benchmark record, not the resident service baseline, and it excludes model preloading. The hardware and model</h2> Component</th> Specification</th></tr></thead> CPU</td> Dual Intel Xeon E5-2696 v4</td></tr> GPU</td> NVIDIA GeForce RTX 3060, 12 GiB VRAM</td></tr> Memory</td> 125 GiB RAM</td></tr> OS</td> Fedora 44</td></tr> GPU stack</td> NVIDIA driver 610.57.04, CUDA 13.3</td></tr> Link</td> PCIe 3.0 x16</td></tr> Model</td> Qwen3.8-Flash-Next, GGUF architecture qwen4exp</code></td></tr> Weights</td> unsloth/Qwen3.8-Flash-Next-GGUF</code>, four UD-Q4_K_XL shards</td></tr> </tbody></table> The model's quantized expert weights alone occupy 77,017,907,200 bytes, about 71.73 GiB. A 12 GiB GPU cannot hold them all. The model is a mixture-of-experts network, however: each layer routes a token to 10 experts out of 512, plus shared experts. System memory holds the complete expert set while the GPU receives only the experts needed for the current layer. How the model fits in 12 GiB</h2> Non-expert weights remain quantized in VRAM. The 71.73 GiB of experts stay resident in host memory and are uploaded on demand. Large PLE tables and embeddings are accessed by selecting only the compressed rows needed for the current token history. The runtime keeps experts in their GGUF formats instead of expanding everything to F16. CUDA kernels consume Q4_K, Q5_K, Q5_1 and Q8_0 data directly. A fixed-size VRAM arena tracks ownership and reuse, so long generations cannot exhaust the card through allocator fragmentation. “On demand” here means host-to-device transfer: the experts are already in RAM. PLE and embedding access can still cause limited disk reads. Why the architecture makes new models easier to add</h2> Qwen3.8-Flash-Next combines Hyper-Connection, GDN, PLE, sparse QSA attention and 512-way expert routing. zLLM can still add it without cloning an entire inference stack because model semantics and device execution are separate. Describe the model. src/model_spec/qwen4exp.rs</code> declares the 48-layer layout, tensor shapes, routing constants, QSA/PLE/HC parameters and GGUF metadata.</li> Implement the model. src/runtime/qwen4exp/mod.rs</code> loads weights and assembles the forward pass. cpu.rs</code> is the reference implementation; cuda.rs</code> and cuda_mtp.rs</code> compose the required CUDA and MTP paths.</li> Register the service. Configuration and Node registration add Qwen4ExpNodeModelConfig</code> and Qwen4ExpCudaEngine</code>, while reusing chat templates, streaming, cancellation, stop/EOS handling and the scheduler protocol.</li> </ol> Those five model-specific files total about 2,228 lines of Rust. They reuse the existing GGUF loader, KV cache, CUDA context and streams, fixed VRAM arena, expert LRU, quantized kernels, service protocol and lifecycle management. Expert transfer lives in backend/cuda/expert.rs</code>; packed Q4/Q5/Q8, attention, routing and tensor kernels are shared modules. New models define how weights and computation are interpreted, rather than rebuilding the runtime. Correctness and optimization are verified separately. CPU reference comparisons, tensor-format checks, operator tests and token-by-token output comparisons answer “is it correct?” Arena reuse, grouped DMA, cache policy and MTP draft length are then measured with complete requests to answer “is it fast?” A faster microbenchmark is not enough to ship a change; it must also pass reference comparisons, long-output stability and end-to-end throughput checks. MTP results and their limits</h2> Shared MTP uses a draft layer to propose tokens and the full 48-layer target to verify them. Workload</th> Prompt / output</th> Ordinary</th> MTP</th></tr></thead> Coding task</td> 42 / 128 tokens</td> 10.779</td> 12.708</td></tr> Coding task</td> 42 / 256 tokens</td> 10.682</td> 11.239</td></tr> Chinese chat</td> 43 / 256 tokens</td> 11.371</td> 11.238</td></tr> </tbody></table> A later run used a different 59-token Chinese coding prompt and reached 8.32 tokens/s with 74.6% acceptance. It is not comparable with the 13.158 result. MTP works in the standalone benchmark loop, but the resident Node MTP path is still converging, so the current service runs without MTP. The bottleneck is data movement. In one 256-token test, each output token corresponded to about 632.5 MB of expert uploads. At a measured 11.90 GB/s transfer rate, the lower bound is already about 53.2 ms per token, before computation and synchronization. That is why simply increasing draft depth does not guarantee 20 tokens/s. Service lessons</h2> The first persistent-service failures came from two independent bugs: an incorrect PLE row width and an unsafe asynchronous DMA path reading registered ordinary heap memory. The final implementation loads experts into CUDA-driver-allocated pinned host memory and uploads from there. It restores the 6.0–6.4 tokens/s range and passed the 50-request regression. Pinned memory needs operational care. A normal exit can return roughly 95 GiB to the driver's host pool without immediately restoring the operating system's MemAvailable</code>; the next allocation can reuse it, but a memory preflight may reject the run. A forced SIGKILL</code> can leave NVIDIA driver accounting behind and require a reboot. Production shutdown should therefore be graceful. Long-context prefill is a separate cost. A recorded 10.6K-token input took 351.2 seconds after sparse QSA integration, so first-token latency can be measured in minutes. The tested context configuration reaches 65,536 tokens, but that is not a claim of a full 65,536-token benchmark. Vision support and complete logits alignment remain future work. This is what the vintage machine provides: enough RAM to hold a modern MoE model, a consumer GPU to execute its active experts, and a runtime that keeps compressed weights, transfers and lifetimes under control. The result is practical text inference on hardware that appears far too small on a VRAM specification sheet. Based on the zLLM Qwen3.8-Flash-Next integration record and its 2026-09-07 follow-up. Historical benchmark records and current service validation are reported separately; benchmarks were not rerun for this article. How Good Is K2-Horizon from Abu Dhabi? On an M5 It Solves the Problems, Then Stumbles over Chinese Prose 2026-09-06T00:00:00+00:00 How good is K2-Horizon, the new model family from Abu Dhabi? Give it a reasoning problem and then ask it to write an essay, and the two results tell very different stories. After adding K2-Horizon-MoVA-36B-A4B to zLLM, I ran the IQ3_XS quantization on an Apple M5 Mac with 24 GiB of unified memory. It correctly solved a snail-climbing problem and the classic cats-and-mice rate problem. When asked for a Chinese essay about a market after the rain, however, it began mixing English, Japanese, and Arabic into the prose and eventually hit repetition detection. Lowering the temperature to zero improved the output, but did not eliminate the problem. Before evaluating the model, though, there is a more basic engineering question to answer: what did zLLM actually have to implement to run it end to end on Metal? K2-Horizon is not a conventional Transformer that can be supported by swapping in a new set of matrix dimensions. It places sparse experts in the value path of attention, creating two independent routing systems inside a layer. The first half of this article explains how zLLM connects the GGUF, MoVA, MoE, KV cache, and complete decode path. The second half examines the distance between solving a problem and writing consistently good Chinese. Start with the source: IFM is headquartered in Abu Dhabi</h2> K2-Horizon was developed by the Institute of Foundation Models, or IFM. According to IFM, the institute is part of Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), headquartered in Abu Dhabi, UAE, with research centers in Paris and Silicon Valley. The K2 Horizon family was released on September 3, 2026. See the IFM overview</a> and official announcement</a>. Calling it simply a European model would therefore be inaccurate. A better description is a model released by a global research team headquartered in Abu Dhabi, with research centers in Paris and Silicon Valley. 36B parameters of capacity, about 4B active per token</h2> The complete model name is K2-Horizon-MoVA-36B-A4B. IFM describes it as a 36B-parameter model that activates about 4B parameters per token, with a native context length of 524,288 tokens. Those are model specifications; this local test did not attempt a 512K context. See the official model card</a>. Conventional MoE models apply sparse routing mainly to the feed-forward network. K2-Horizon's MoVA, or Mixture-of-Value Attention, extends expert routing into the value path of attention. IFM describes the design in its K2 Horizon architecture overview</a>. The local GGUF metadata describes 48 layers, a hidden size of 2,560, 32 query heads, and 8 KV heads. The value path selects 4 of 64 experts. After three leading dense layers, the feed-forward path selects 8 of 100 routed experts and also executes a shared expert. zLLM now supports the corresponding grouped RMSNorm, attention output gate, MoVA, MoE, complete Metal prefill and decode, and Q8g64 KV cache. The file used here was: K2-Horizon-MoVA-36B-A4B-IQ3_XS.gguf </code></pre> It is 15,695,083,968 bytes, or about 14.62 GiB. This is a third-party low-bit quantization rather than an official IFM BF16 checkpoint. Its tensors mix IQ3_S, IQ3_XXS, Q3_K, Q6_K, and F32; the file is not made entirely from one “three-bit” tensor format. Activating only a subset of experts reduces computation. It does not mean that only 4B parameters need to be loaded. The quality results below belong to this exact GGUF and runtime configuration and should not be treated as measurements of the official BF16 model. Integration step one: K2 cannot be treated as ordinary GQA</h2> zLLM does not bake K2 dimensions into Metal kernels. Model dimensions, layer counts, normalization, attention geometry, expert counts, and Top-K values live in a dedicated K2HorizonConfig</code>. When a GGUF is opened, zLLM reads and validates the metadata field by field. The validation goes beyond hidden size. The current implementation confirms sigmoid expert routing, normalization of selected expert weights, one shared expert per sparse layer, matching key and value head dimensions, and MoE in every layer after the dense prefix. A structurally incompatible GGUF fails during loading instead of carrying an invalid shape into GPU execution. The model schedule lives in runtime/k2_horizon/</code>. It describes the actual flow of one K2 layer: hidden → grouped RMSNorm → query + attention gate → key → dense value (first 3 layers) or 4 of 64 value experts (last 45 layers) → RoPE + GQA + KV append → attention gate + output projection + residual → grouped RMSNorm → dense FFN (first 3 layers) or 8 of 100 routed experts + shared expert → residual </code></pre> The value path is the easiest part to wire incorrectly. In ordinary GQA, value is a single linear projection. From layer four onward, K2 first applies a sigmoid router and bias, chooses 4 of 64 value experts, computes their outputs, and combines them with normalized routing weights. The result has 8 KV heads of 128 dimensions each, for a 1,024-dimensional value vector, and only then enters attention. The FFN has a separate router. It selects 8 of 100 routed experts, executes a shared expert, normalizes the selected weights, and applies a scaling factor of 2.5. The two routers own different weights and serve different dataflows. Treating both as one generic “expert” branch would hide a real dependency. The third-party GGUF quantizes the small but sensitive MoVA router as IQ3_S. zLLM decodes only that router to resident F32 during loading to keep routing scores stable. The much larger value-expert weights remain quantized and are consumed directly by Metal. This keeps the exception local instead of weakening the general weight-loading contract or expanding all expert weights. Integration step two: IQ3_XS is not one kernel</h2> Opening the tensor directory reveals five storage types: 322 IQ3_S tensors, 240 IQ3_XXS tensors, 3 Q3_K tensors, 1 Q6_K tensor, and 232 F32 tensors. A single generic “IQ3” entry point is not enough. To execute the whole file, the Metal path needs ordinary quantized GEMV, fused gate/up projection, expert-indexed gate/up/down projection, the 4-of-64 MoVA value operation, and a Q6_K LM head over a 250,624-token vocabulary. These kernels read packed GGUF weights directly. Fully dequantizing the weights before every generated token would throw away much of the capacity and bandwidth advantage of the 14.62 GiB file. Prefill and decode also need different execution shapes. Prefill processes multiple token rows, routes the batch, groups work by expert, and then executes each group. Decode has one row and uses device-resident Top-K results with indexed expert kernels. The last 45 layers contain 100 FFN experts each: 4,500 expert groups totaling about 10.00 GiB. The zero-configuration run preloads them into Metal resources visible through unified memory so that generation does not stop for per-layer disk I/O. Integration step three: a 24 GiB machine still needs room for KV</h2> A 14.62 GiB weight file does not leave every other byte available for context. Execution also needs resident expert resources, activations, scratch space, Metal objects, and memory for the operating system. At startup, zLLM reads the model geometry and machine budget, then computes KV cost per token. The M5 run reported: Weights 14.62 GiB KV budget 0.96 GiB KV cost per token ~101,376 bytes Selected context length 9,216 tokens </code></pre> For K2 on a 24 GiB machine, zLLM permits a working-set ceiling of three quarters of physical memory while preserving at least 2 GiB for the system. KV cache defaults to Q8g64, with one quantization group per 64 values. Short contexts use direct attention. Beyond 256 tokens, split-KV divides history into parallel segments and merges the partial results. This is why the model may advertise 524,288 tokens while the zero-configuration entry point chooses 9,216 on this machine. Flattening complete decode into replay</h2> One decode round crosses all 48 layers. An early profile showed roughly 1,100 Metal commands per token. Rebuilding and submitting that graph from the CPU for every round would repeat substantial scheduling work. zLLM therefore implements complete single-token decode replay for K2. The first pass records the sequence from an uploaded token embedding through 48 layers of attention, both expert routers, MoE, final norm, LM head, and argmax. Later rounds update the token embedding, position, and KV write slots, then replay the same plan. Q8 attention replay selects its direct or split-KV pipeline from the current position, so crossing the 256-token boundary does not silently preserve the short-context path. There is an important limitation behind the performance figures. Greedy generation can replay the complete round. Temperature/top-p sampling must retrieve logits and maintain sampling state, so the current K2 path does not enter the same replay fast path. The roughly 30 tok/s greedy result therefore does not describe the sampled writing tests later in this article. Prove the numerical path before judging the text</h2> Integration did not stop when the model began emitting tokens. The validation record includes: K2 Q8 direct and split KV append comparisons against the reference;</li> a BF16 split-KV vectorized-kernel argument-binding regression, fixed and rechecked against the CPU oracle;</li> Q6_K GEMV compared with a dot product over decoded weights;</li> Metal top-p sampling over a 250,624-token vocabulary checked point by point against the CPU reference;</li> full real-weight loading, prefill, decode, Q8 KV cache, and replay on the M5.</li> </ul> Under low-reasoning greedy settings, zLLM and llama.cpp produced the same four wrong answers for the same GGUF. With high reasoning and enough token budget, the archived llama.cpp test answered all four questions correctly across three seeds, for 12/12. This comparison does not prove that every numerical path is identical, but it rules out the simple explanation that those four mistakes occurred only in zLLM. The console exposed another problem that was subtler than a kernel bug. The shared client initially sent enable_thinking=false</code> explicitly. K2's chat template interprets that as an empty thinking fence, forcing a direct answer and making mixed-language degradation more likely. The corrected client leaves that field unset for K2, preserving the template's default high-thinking mode. MiniCPM5 and Qwen retain their separately validated settings. Model support includes template semantics as well as matrix math. Current speed and the remaining bottlenecks</h2> In an archived run with the same 55-token prompt, greedy decoding, and a 256-token completion, llama.cpp used F16 KV and measured 0.322 seconds for prefill plus 7.800 seconds for decode, or 32.691 tok/s. zLLM used Q8 KV and its best recorded run measured 5.999 seconds for prefill plus 8.623 seconds for decode, or 29.690 tok/s, 9.18% slower. This is not a strict same-KV-format comparison, but it makes the current position clear: decode is close, while short-prompt prefill remains much slower. In a single-token profile, the two sigmoid routers—the 8-of-100 FFN router and the 4-of-64 value router—ranked among the largest GPU costs. Split-KV beyond 256 tokens and replay submission also leave room for improvement. The machine eventually thermal-throttled during the last stress run, increasing direct decode from about 30 ms/token to 80–100 ms/token. The article therefore uses the reproducible pre-throttling measurements instead of presenting an overheated transient result as normal performance. Give it enough reasoning budget, then inspect the answer</h2> The quality test used high reasoning, temperature 1.0, and top-p 0.95, matching IFM's recommended settings</a>. Each request had a 9,216-token context limit and up to 4,096 generated tokens. The questions were independent and did not share chat history. Reasoning mode is not merely a display option. The local template changes its thinking marker based on the selected effort; disabling thinking inserts an empty thinking fence. If a run spends a tiny completion budget reasoning and ends with finish=length</code>, it should be recorded as incomplete rather than graded as if its intermediate text were the final answer. Both reasoning problems below reached a normal stop: Problem</th> Correct answer</th> Observed answer</th> Generated tokens, including reasoning</th></tr></thead> A snail climbs 3 m by day and slips 2 m at night. On which day does it leave a 10 m well?</td> Day 8</td> Day 8</td> 1,344</td></tr> Three cats catch three mice in three minutes. How long do nine cats need for nine mice?</td> 3 minutes</td> 3 minutes</td> 700</td></tr> </tbody></table> The final climb has no following slip, which makes day eight correct. The cats work in parallel, so scaling both cats and mice by three leaves the time unchanged. The model got both conclusions right. Its Chinese explanation of the snail problem nevertheless included this sentence: 每天 daytime 爬升 3 米， nighttime 回滑 2 米 </blockquote> It means “climbs three meters in the daytime and slips two at night,” but inserts the English words daytime</code> and nighttime</code> into an otherwise Chinese sentence. It also produced the malformed phrase 白天下爬到</code>. A correct numeric answer did not make the explanation naturally written. The market essay makes the problem obvious</h2> The first writing prompt, in Chinese, asked for a restrained 400–600-character slice-of-life essay titled The Market after the Rain, organized around concrete people, actions, sounds, and smells. It explicitly requested only the title and body, in Chinese. The generated body began: 雨刚停，菜市场还不全是人。薄荷黑布遮在丑毛さらに码头上，粪水从竹筐的缝隙里滴着Becoming一条条银白色的短河。 </blockquote> The sentence starts in Chinese, inserts Japanese さらに</code> and English Becoming</code>, and is semantically broken even if those words are removed. Later text includes Granny</code>, Japanese へえ</code>, Arabic السجائر</code>, and ends with ポリ袋ákááááááááá</code>. The runtime stopped it with finish=repetition</code>. Counting only Han characters after the title, it produced 88—far short of the requested essay. The second prompt requested a 400–600-character explanatory essay titled If a Small Model Can Solve Problems, Can It Write Well? It asked the model to distinguish answer correctness, reliable reasoning, and natural language, with one concrete example. This time the output formed complete paragraphs, but still contained lines such as: Such表述语言自然。 </blockquote> 最后把推理过程写成一段连贯的文字 describing the solving journey, 语言自然。 </blockquote> And the conclusion inserted the word fingernail</code> where it had no meaningful role. The argument also treated natural language as exclusive to creative writing and tried to infer writing ability from one correct equation solution. Its finish=stop</code> means generation stopped normally; it does not mean the essay passed review. How much does temperature zero help?</h2> I kept high reasoning and the same two prompts, changed temperature to 0, and ran each writing task once more. Task</th> temperature=1.0</th> temperature=0</th></tr></thead> Market essay</td> Mixed languages, repetition stop; 88 Han characters</td> Normal stop, 489 Han characters; still contains English</td></tr> Evaluation essay</td> Normal stop, 481 Han characters; meaningless English insertions</td> Normal stop, 652 Han characters; much better Chinese, but over the requested length</td></tr> </tbody></table> The counts include only Han characters in the body after the title, excluding punctuation, Latin letters, and Markdown markers. The zero-temperature market essay has a complete scene, but still writes freshly 捕来的鲫鱼</code>, Somehow 却不显得刺鼻</code>, and 女孩 grabs 住鱼</code>. It also renders the sound of bargaining as a camera-like “click.” The evaluation essay improves much more and has no comparable meaningless foreign-language insertion; AI</code> is an ordinary abbreviation. It still misses the requested length. The result should not be compressed into “the model cannot write Chinese.” A more accurate conclusion is that under these prompts and this runtime configuration, completion, language consistency, and instruction following in Chinese writing are unstable, and both genre and generation settings affect the result. One engineering question remains unresolved: where does the degradation originate? The writing prompts were not repeated on the BF16 checkpoint or another runtime, so this test cannot separate quantization loss, runtime numerics, template behavior, and the model itself. Temperature zero also changes sampling state and eligibility for the current zLLM replay path, so it is not a clean one-variable ablation. The evidence is enough to describe this local configuration. It is not enough to judge the entire model family. Running K2-Horizon with zLLM</h2> From the zLLM repository, pass the GGUF path to the compiled release client: ./target/release/zllm-metal \ /Volumes/ORICO/models/K2-Horizon-MoVA-36B-A4B-IQ3_XS.gguf </code></pre> --release</code> is a Cargo build option and is not passed to zllm-metal</code>. Once loading completes, enter a message directly; /reset</code> clears the conversation and /exit</code> quits. On this machine the client selected a 9,216-token context automatically. The advertised 512K model limit is not the usable capacity of this 24 GiB configuration. This integration gives zLLM a complete executable path for a MoVA model. The writing tests add the other half of the story: an engine can run every layer correctly while a particular quantization and generation setup still produces poor prose. After looking at reasoning scores, it is worth asking a model for one concrete slice-of-life essay. Tasks without a single correct answer often reveal whether the language holds together, whether details make sense, and whether the result can actually be handed to a reader. Test record</h2> The tests ran on September 6, 2026, on an Apple M5 with 24 GiB of unified memory, using the IQ3_XS GGUF, Metal, and Q8g64 KV cache. There were six independent requests: four at temperature 1.0 without a fixed seed and two zero-temperature writing comparisons. Every request allowed up to 4,096 generated tokens, including reasoning and answer text. The sampled runs were not replay performance measurements. The production entry point is zllm-metal</code>, which identifies and loads K2 from the supplied GGUF path. Quality tests reused the same production inference path and an existing release build. The recorded source HEAD describes the workspace at test time and is not a verified build commit for that binary. The runtime returned a combined text stream rather than separate reasoning and content fields. Essay excerpts begin after an explicit title; problem grading uses the final answer at the end. English inside hidden reasoning was not counted as an error in the Chinese essay body. The JSON record</a> contains every prompt, all six raw outputs, finish reasons, and a runtime fingerprint. The sampled log</a>, zero-temperature log</a>, sampled configuration</a>, and zero-temperature configuration</a> are also available. The raw model output remains in Chinese and has not been rewritten. Paths in the configuration files belong to the test machine and must be changed for reproduction elsewhere. Operator validation and earlier performance measurements are preserved in the integration report</a>. Its measurements and the writing tests above are separate records. Running GLM-5.3 on Two 8-GPU AMD W7900D Machines 2026-09-05T00:00:00+00:00 Sixteen GPUs, 768 GiB of total VRAM—and GLM-5.3 initially generated just 4.08 tokens/s, taking roughly 245 milliseconds per token. Having enough memory to fit the model does not make compute scale automatically. Each of our two machines has eight 48 GiB W7900D cards. Communication uses PCIe 4.0 within a node and TCP between nodes, with no XGMI high-speed interconnect available to this execution path. The model has to span multiple GPUs, but every additional collaboration point can add another wait to each token. This constraint shaped our ROCm optimization work in zLLM: reduce communication inside layers, improve the work performed on each GPU, then use MTP speculative decoding to reduce the number of full-model execution rounds. Checking the original result files on Amd-1 gives 11.321 tokens/s for a complete run with one GPU per stage and MTP disabled. The latest two-GPU operator-pair version achieved 13.842–14.043 tokens/s across three runs, with a median of 13.968—about 14 tokens/s. The single-GPU-stage path is about 2.77 times faster than the initial 4.08 tokens/s baseline. A complete run of an older paired version with MTP also recorded 25.275 tokens/s. This establishes that the combination has been tested; it does not establish that MTP has been validated on the latest version delivering approximately 14 tokens/s without it. Measurement notes: the main results come from JSON files and runtime logs on Amd-1. The corpus is named 50k-allgpu.txt</code>, but the actual requests contained 46,152–46,158 input tokens and generated 1,024 tokens each. Decode throughput is the remaining 1,023 generated tokens divided by generation time after the first text event. “50K” below retains the corpus name; 50K in operator profiles describes their workload size. Results with different commits, nonces, or MTP settings are not strict same-version A/B comparisons. </blockquote> Decide where data lives before deciding how to parallelize</h2> Single-request autoregressive generation has a hard dependency: the next token must wait for the current token to finish sampling. For the 78-layer model tested here, a token passes through every layer, the final output projection (LM Head), and sampling before the next round can begin. Distributing those layers across 16 GPUs divides the execution chain into stages. It does not let one token pass through all layers simultaneously. We started by assigning each GPU a contiguous range of complete layers. Weights, KV cache, sparse-attention index state, expert weights, and scratch buffers remain local. Only the hidden activation at a stage boundary needs to be handed off: </a> Click to enlarge. Traffic and latency comparisons have separate scales. Layer profiles contain nested scopes and must not be added up into a whole-token time breakdown. “Single-GPU stage” means one GPU executes each stage; the complete model still uses all 16 GPUs. An earlier design used finer-grained collaboration between two GPUs, repeatedly exchanging queries, selections, and expert partial results—four to six handoffs per layer. The tensors were not necessarily large, but event waits after each exchange put the difference in GPU progress onto the critical path. Across 78 layers, coordination costs absorbed the savings from parallel computation. The approximate latency budget for single-request decode is therefore: Per-token latency ≈ sum of serial execution time across all stages + stage handoffs + LM Head, sampling, and the autoregressive return path </code></pre> This distinction also matters for load balancing. When multiple requests or prefill chunks overlap, the slowest stage limits pipeline throughput. For single-request, token-by-token decode, total chain time comes first. Moving layer boundaries helps only if it reduces waits, improves execution efficiency, or relieves resource constraints. From 4.08 to 11.32: where the time went</h2> The archived tests used GLM-5.3 UD-IQ4_XS GGUF, 78 layers, Q8G64 KV, the “50K” corpus—actually about 46.15K input tokens—and 1,024 generated tokens. MTP, DSpark, and the old cooperative-pair path were disabled. Stage</th> Decode (tokens/s)</th> Latency per token</th> Main changes</th></tr></thead> Initial single pipeline</td> 4.084</td> 244.8 ms</td> Topology corrected; single-row operators still unoptimized</td></tr> One GPU per stage, no intra-layer pairing</td> 11.321</td> About 88.3 ms</td> Dedicated IQ kernels, MLA tuning, Q8 shared expert, direct W8 reads, scheduling improvements</td></tr> </tbody></table> At 11.321 tokens/s, throughput is about 2.77 times the initial baseline and per-token latency is about 63.9% lower. Several changes account for the improvement. A single decode row is not a large matrix multiplication</h3> Prefill processes multiple input rows, allowing matrix multiplication to amortize weight reads. Ordinary decode processes one row at a time. It is closer to matrix-vector multiplication, or GEMV: a large set of weights is read to produce one token's output. Weight traffic, dequantization instructions, thread mapping, and kernel launch overhead become prominent. We wrote HIP/ROCm operators specifically for gfx1100, with dedicated paths for IQ3_S, IQ4_XS, and Q8/W8A16. For example, IQ3_S gate/up uses an 8-lane subgroup, followed by wider u16</code> loads to reduce loading overhead. With layout and output bit patterns unchanged, probe time fell from approximately 207–211 microseconds to 188–190 microseconds. Similar code does not guarantee similar gains. IQ4_XS down already achieved about 465 GB/s of effective bandwidth. Porting a different codebook-reading implementation increased its time from 102 to 144 microseconds. For this shape, further work on codebook caching did not address the main bottleneck. Keep quantized weights compressed in memory</h3> Multi-head Latent Attention, or MLA, is a major aggregate hotspot executed in every layer. The resident representation of kv_b</code> was wasting substantial bandwidth. The old path decoded W8 weights into F16 and then kept a dense F32 representation resident for absorb/PV operations. Although the stored weights were quantized, execution expanded them again: those stages read approximately 58.6 MB per layer. The new path consumes W8 weights and scales directly, reducing traffic to about 14.7 MB, roughly one quarter of the original. Complete runs improved from about 9.76 to 10.50/10.28 tokens/s, an average gain of roughly 6.5% across the two runs. This was more direct than another tile adjustment: if the hot path ultimately reads F32, quantization has saved file size without fully delivering its runtime bandwidth advantage. Other MLA changes also helped. Vectorizing contiguous PV loads reduced the device scope from approximately 0.484 to 0.439 ms/layer, yielding about 2.5%–3.5% for the full model. Selected scan distributed QK work previously handled mainly by one wave across eight waves, cutting partial-plus-merge time from approximately 90.6 to 57–59 microseconds. These local savings do not automatically translate into equal reductions in token latency. A faster kernel can still leave the full chain slower</h2> The DSA sparse-attention indexer selects historical positions for subsequent attention computation. At a 50K workload, score computation took about 91 microseconds, already reaching roughly 68% of the profile's estimated compute ceiling. The serial tail of radix selection was a more promising target. The original last two levels rescanned serially in one thread block. Splitting the work into multiple kernels reduced the relevant kernel time from about 105 to 96 microseconds. But two additional launches, each costing about 12 microseconds, left complete selection at approximately 139 microseconds. The expected end-to-end gain did not materialize. These experiments expanded our unit of optimization: first a kernel, then an entire operator, and ultimately a stage's contribution to the token critical path. Main constraint</th> Examples</th> Useful direction</th></tr></thead> Weight-read bandwidth</td> q_b</code>, kv_b</code>, MoE weight streams</td> Compressed residency, wide contiguous loads, shared weight reads</td></tr> Compute and dequantization instructions</td> DSA score, IQ decoding</td> Wave mapping, shorter instruction dependencies</td></tr> Synchronization and scheduling</td> Multi-level radix selection, launches, pair joins</td> Fewer handoffs, fused dependency chains, no bridge copies</td></tr> </tbody></table> The updated source notes provide the following bandwidth breakdown: </a> Effective bandwidth here is weight bytes divided by kernel time. It is an analytical metric. Usable underlying memory-traffic and occupancy counters were not available in this investigation, so these values are not hardware-counter measurements. The Graph experiment: faster launches, but 4%–5% lower throughput</h2> HIP Graph records a repeated kernel dependency chain and replays it, reducing the cost of submitting kernels individually. This looks attractive for single-row decode with many short kernels. Our probes confirmed launch savings, but integrating the graph into the production path did not deliver an end-to-end gain. The probe improved; the complete request did not</h3> A chain of 64 small kernels took approximately 0.76–0.80 ms on the eager device timeline, compared with 0.23–0.27 ms for warmed static Graph replay. That is roughly 12 versus 4 microseconds per node. Graph reduced dispatch gaps; it did not accelerate the computation or weight reads inside each kernel. After correctness fixes, complete-request tests of the five-node MoE Graph produced: Same binary, “50K” corpus, 1,024-token decode</th> Run 1</th> Run 2</th></tr></thead> Graph disabled</td> 9.509 tokens/s</td> 9.607 tokens/s</td></tr> Graph enabled</td> 9.119 tokens/s</td> 9.124 tokens/s</td></tr> Throughput change</td> −4.1%</td> −5.0%</td></tr> </tbody></table> This was a historical Graph experiment, fixed in b4433700</code> and recorded in 51a4f30d</code>. These figures describe a same-version Graph toggle comparison. The verified single-GPU-stage result used elsewhere in this article is 11.321 tokens/s. The historical regression percentage must not be applied directly to the latest version. Why saving launches still lost time</h3> The crucial cost was bridging into fixed addresses. This implementation's graph nodes recorded device pointers, while upstream outputs had to be moved into the graph's fixed slots. That added three device-to-device copies per layer, plus their host submission overhead. The graph covered only five nodes, limiting the dispatch gaps it could eliminate. The bridge operations, meanwhile, recurred every layer. Applying the per-node probe difference gives only about 40 microseconds of potential savings for five nodes. This is a scale estimate, not the production graph's net benefit. The real budget must include bridging, replay, and subsequent dependency waits; complete-request measurements showed a loss. The small MoE graph also left the long serial attention chain outside its scope. It did not reduce expert weight reads, dequantization, or inter-GPU waits. It optimized a small part of the critical path while adding fixed costs to enter that part. Correctness was fixed; the performance problem remained</h3> Earlier versions also suffered hangs, NaNs, and illegal memory accesses. Intermediate buffers used while building the graph were returned to the memory pool when construction finished, but graph nodes retained their device pointers. Once the pool reused those addresses, replay could access another object's memory. The isolated probe did not reproduce this because it retained its buffers throughout the test. Production execution had a different resource lifetime. The fix made the graph object retain the required buffers, keeping their addresses valid during replay. Additional synchronization temporarily hid symptoms, but long-context tests still failed: synchronization cannot repair a dangling pointer. The negative results in the table were measured after the fix. Stable replay and lower end-to-end latency require separate validation. When another attempt is worthwhile</h3> The production path remained at decode_graph=false</code>. A worthwhile next experiment is a whole-layer graph covering a longer dependency chain: upstream operators write directly into fixed workspaces, eliminating extra bridge copies, and the graph explicitly retains its referenced resources. Workspace growth and memory-pool reuse must not invalidate captured pointers. Even with broader coverage, the final test is a complete request using the same binary, input, and output-correctness requirements, with the profiler disabled. The acceptance metric is token latency—not node count or replay-probe speed. Why two-GPU parallelism did not come close to doubling speed</h2> After optimizing the single-GPU path, we revisited paired execution. The new parallel_operator_pairs</code> path partitions MLA by query heads and MoE by intermediate dimension, merging results at larger attention and FFN boundaries. We also changed the exchange format. The owner quantizes KV once, transmitting only newly appended Q8 latent values, scales, and RoPE information. Per-token, per-layer KV exchange fell from 2,304 B to 656 B, a 71.5% reduction. Each side recomputes the router to avoid another routing-table handoff. The non-MTP results are: Execution</th> Decode (tokens/s)</th> Latency per token</th></tr></thead> One GPU per stage, no intra-layer pairing</td> 11.321</td> About 88.3 ms</td></tr> Two-GPU operator pairs, median of three runs</td> 13.968, about 14</td> About 71.6 ms</td></tr> </tbody></table> The paired median is approximately 23.4% faster than 11.321 tokens/s, reducing latency by about 16.7 milliseconds per token. This compares different versions, rather than a strict same-version parallelism toggle. Profiles explain why the gain is far below ideal halving. First, splitting query heads did not split the historical scan. Both GPUs still gathered Top-2048 positions and scanned the complete logical KV. The paired MLA scope took approximately 0.35–0.45 ms/layer versus 0.35–0.36 ms/layer on one GPU, providing essentially no speedup. Second, splitting large weight operations did not halve the entire dependency chain. Norms, routing, DSA selection, some shared-expert work, and merges remained. The complete FFN/MoE section became only about 30%–40% faster. Finally, attention and MoE each exchanged partial results in both directions every layer. Peer-copy kernels alone totaled about 3.8 ms/token across 78 layers, with additional costs from events, joins, and unequal progress. These bottlenecks come from profiles of an earlier paired version. They explain the optimization direction, not the exact breakdown of the latest da1e40be</code> version. In those profiles, raw inter-node TCP transmission took only tens of microseconds per token. Most losses arose from layer-by-layer collaboration within a node. Reducing message size alone was insufficient; duplicate scans and synchronization waits mattered more. Amdahl's law limits overall acceleration when serial work remains, and added communication further reduces the gain. Two busy GPUs do not imply that the token critical path has been cut in half. Further work aims to make the owner hold the canonical hidden state, reduce partials toward it in one direction, and remove reverse exchanges and duplicate residual work. Partitioning DSA/MLA across history would then allow the GPUs to scan different portions of that history. The next performance step requires changing the division of data and work. MTP saves full-model execution rounds</h2> Once ordinary decode reaches about 88.3 milliseconds per token, shaving off a few more microseconds has diminishing returns. MTP takes another approach: propose several candidate tokens, then verify them with the target model so that one expensive verification round can commit several valid tokens. Autoregressive semantics remain intact. Candidate prefixes turn some sequential work into multi-row verification. The benefit depends on accepted tokens per round and the combined cost of drafting, verification, rollback, and state catch-up. Amd-1's paired-mtp5-g2-50k-d1024-r1.json</code> records an actual paired-plus-MTP run: 46,152 input tokens, 1,024 output tokens, 40.475 seconds of decode, and 25.275 tokens/s. The configuration uses parallel_operator_pairs=true</code>, draft depth 5, and verify group rows 2. Runtime logs confirm that MTP L78 also uses an operator pair. The corresponding complete-request log records 315 target rounds, 1,567 verified candidates, and 709 accepted candidates, for a 45.25% acceptance rate. That is approximately 3.25 committed tokens per target round under the log's accounting, or 1,024 / 315. Accepted counts by draft depth were 241, 179, 130, 92, and 67, showing declining useful output at deeper candidate positions. Other complete runs on the same general path reached 23.793 tokens/s with depth 3 / group 2, and 20.647 tokens/s with depth 5 / group 1. Verification organization affects end-to-end behavior, but because settings and request nonces differ, we do not attribute the entire difference to one factor. 25.275 is one complete measurement of an older paired configuration. It is neither a result for MTP on the latest da1e40be</code> paired version nor a stable multi-run baseline. The 1,024-token request completed before a later rocprof-attached diagnostic run appeared in the log. We do not mix those profiler results into this throughput figure. The latest paired version still requires separate validation of MTP speed and correctness. The next priority is not simply increasing draft depth. Multi-row verification should share weight reads across projections, MLA, and MoE. The cheaper each extra verification row becomes, the more valuable additional accepted candidates are. Correctness must include state. After a mismatch, target KV, DSA selection, and MTP caches must return to the correct position. After accepting a prefix, the corresponding caches must catch up. Throughput is meaningful only when these states remain consistent. Why 4-bit today: room for 5/6-bit, but not FP8</h2> We currently choose 4-bit quantization to preserve VRAM headroom for long contexts, MTP, and multi-GPU execution. Four bits is the starting point for this deployment; five or six bits are possible upgrades. Our assessment of this execution path is that they can improve model quality noticeably with essentially no performance loss. For quality-sensitive use, that is more attractive than pushing weight precision lower. Increasing bit width does not increase end-to-end time proportionally. Single-request decode also includes DSA, attention, kernel submission, inter-GPU synchronization, and sampling. Low-bit formats themselves incur codebook, scale, sign-handling, and dequantization costs. More weight traffic at 5/6-bit therefore does not directly imply the same percentage increase in token latency. This assessment applies to this model, hardware, and execution path; this article does not present separate 5/6-bit speed or quality measurements. FP8 crosses a different boundary: the current hardware cannot meet the memory budget of the complete deployment described here. The high-quality 4-bit checkpoint is approximately 362 GiB. Roughly doubling weight storage gives about 724 GiB at 8-bit, against 768 GiB of physical VRAM across all 16 GPUs. The remaining approximately 44 GiB must hold long-context KV, DSA state, MTP, peer replicas, workspaces, HIP runtime allocations, and fragmentation. That is insufficient for this deployment. This is a capacity-and-headroom conclusion, not a claim that an API cannot express FP8. Weight format</th> Choice and assessment for this project</th></tr></thead> 4-bit</td> Current path; preserves context and execution headroom</td></tr> 5/6-bit</td> Possible upgrade with little expected performance loss and noticeable quality gains on this path; requires recalculating memory headroom</td></tr> FP8</td> Current hardware cannot meet the full deployment's VRAM budget</td></tr> </tbody></table> Execution does not require every tensor to use the same bit width. Routed experts currently use IQ3_S/IQ4_XS, suitable projections retain W8A16, KV uses Q8G64, and LM Head uses Q8G128. A precision upgrade should still choose formats per operator and validate quality on the intended tasks. Whether using four, five, or six bits, the hot path should consume compressed weights directly instead of keeping an expanded F32 representation resident. Additional memory should improve model quality, rather than pay for unnecessary intermediate representations. Keep measuring complete requests</h2> A historical cooperative prefill path has an archived 1331.62 tokens/s result under the 50K test setting. It is a different execution configuration and must not be combined with current decode figures as if they came from one run. The next goal is to complete multi-row operators and improve chunk and stage scheduling toward 1500 tokens/s, validated through complete-prompt time to first token. For decode, the next step is to revalidate MTP on the latest paired version: check output and cache state first, then measure acceptance, multi-row verification cost, and complete-request throughput across repeated runs. The older configuration's 25.275 tokens/s provides experimental evidence, but 30+ remains a target. Gains from different versions cannot simply be multiplied. The latest notes also report effective activity above 70% during a GPU's local stage execution window. That does not mean all 16 GPUs sustain 70% utilization throughout single-request decode, and it is not interchangeable with hardware occupancy. Local work is more compact; end-to-end timing still determines whether the user waits less. We retained direct W8 consumption because it reduces weight reads, and rejected experiments whose better microbenchmarks did not improve the complete chain. The standard remains the same: use a reference to check values and state, use profiling to locate waits, then disable the profiler and measure complete requests with matching inputs and settings. The path every token must finish is the thing we ultimately have to optimize. Sources and accounting: algorithms and historical experiments follow the project notes updated on September 5, 2026. Current speeds were cross-checked against original JSON files, configurations, and logs on Amd-1. The complete requests cited below were inspected, not rerun for this article. Path / result file</th> Input tokens</th> Output tokens</th> Decode (tokens/s)</th></tr></thead> Single-GPU stage: c343909e-current-routepair-no-mtp-strict1024-r1.json</code></td> 46,158</td> 1,024</td> 11.321</td></tr> Paired: paired-directjoin-da1e40be-formal-no-mtp.json</code></td> 46,153</td> 1,024</td> 14.043</td></tr> Paired recheck: paired-directjoin-da1e40be-formal-recheck-no-mtp.json</code></td> 46,153</td> 1,024</td> 13.842</td></tr> Paired recheck: paired-directjoin-da1e40be-formal-recheck-no-mtp-r2.json</code></td> 46,153</td> 1,024</td> 13.968</td></tr> Older paired + MTP: paired-mtp5-g2-50k-d1024-r1.json</code></td> 46,152</td> 1,024</td> 25.275</td></tr> </tbody></table> The files are in /workspace/zllm-kv-eval-run/runs/</code> and results/</code> on Amd-1. Their base-corpus SHA-256 matches, while nonces are not all identical. The three latest paired runs have identical output SHA-256 values. This establishes output consistency across those runs, not an independent model-quality evaluation. Some runner topology</code> fields retained a “16 serial stages” template; runtime logs confirm four owner-peer pairs per node, or eight logical stages across both nodes, for paired execution. The Algorithm Layer: From Attention and KV Cache to FFN and MoE 2026-09-04T00:00:00+00:00 The algorithm layer answers not “which GPU runs this?” but “what result should this operation produce mathematically?” In zLLM it begins with device-independent specifications, dataflow, and CPU f32</code> references. The CPU oracle provides a readable and testable correctness anchor; Metal, CUDA, ROCm, Vulkan, and NPU kernels are accelerated implementations of the same semantics. This distinction matters. A fast kernel may change layouts, fuse operations, lower precision, or submit work in parallel, but it must not silently change the causal mask, RoPE, Top-K routing, KV write position, or numerical definition. New algorithms first establish shapes, boundaries, and results in CPU/reference code, then device implementations are compared against that oracle. Passing an operator oracle establishes local semantic agreement; complete prefill/decode execution still has to validate output and performance. 1. From a token to the next token</h2> A Transformer layer can be compared to one structured step of thought, but the analogy is not a biological claim. “Angle, shape, vision, semantics, history, and relationships” are not six predefined slots. After training, such features are distributed across directions in a high-dimensional representation, and an individual coordinate rarely has one stable human interpretation. The complete path is: Tokenization. The tokenizer maps text or multimodal placeholders to token IDs. A token may be a word, subword, byte fragment, or special symbol; it need not equal one written word.</li> Vectorization. An embedding table maps each discrete ID to a hidden_size</code>-dimensional vector. Positional encoding then lets attention distinguish order and relative position.</li> Layer-by-layer transformation. Each layer uses attention to collect relevant information from current and historical tokens, then an FFN or MoE transforms each token representation. Normalization and residual connections run throughout the block.</li> Output. Final normalization and the LM head project hidden state to vocabulary logits. Sampling or greedy selection chooses a token, and decode feeds it into the next round until EOS or a length limit is reached.</li> </ol> What expansion, matrix multiplication, merging, and reduction mean</h3> For one hidden row x ∈ R^d</code>, a linear layer is fundamentally y = xW</code>. Matrix multiplication forms weighted combinations of the old coordinates in a new coordinate system. It may project d</code> dimensions into a wider d_ff</code> space, form Q/K/V, or map concatenated head outputs back to d</code> dimensions. Expansion does not create facts from nothing. It supplies a wider intermediate workspace in which gates and nonlinearities can express more feature combinations.</li> Multiple heads project the same hidden state into several subspaces. Heads may learn locality, semantics, position, reference, or visual relations, but no rule says “head 3 is shape.”</li> Merging usually means a weighted sum of values followed by concatenation across heads. It aggregates information rather than copying the source sentence.</li> Reduction uses an output projection or FFN down projection to return to hidden_size</code>, so the result can join the residual stream and enter the next layer. It is learned projection, not truncating the last few coordinates.</li> </ul> A layer's “thinking” is therefore better described as alternating cross-token retrieval in attention with per-token feature transformation in the FFN. Depth provides successive revisions of the representation; width provides the workspace available to each revision. 2. Attention: choosing what information to read now</h2> Classic scaled dot-product attention is: Attention(Q,K,V) = softmax(QKᵀ / √dₖ + mask)V</code> Q asks “what am I looking for now?”, K is the index under which each piece of history can be found, and V is the content retrieved after a match. A causal mask prevents token N from seeing the future. “Attention Is All You Need”</a> replaced the recurrent backbone common in earlier sequence models with global self-attention, but modern models now use several compute/storage trade-offs. These techniques are not all forms of “gated grouping.” They differ in where compression occurs: Design</th> Core idea</th> Primarily reduces</th> zLLM mapping</th></tr></thead> MHA, multi-head attention</td> Every Q head has its own K/V head</td> Baseline; expressive but KV-heavy</td> General multi-head geometry and block/reference semantics</td></tr> MQA</td> All Q heads share one K/V pair</td> KV cache and K/V bandwidth</td> The num_kv_heads = 1</code> limit of GQA</td></tr> GQA, grouped-query attention</td> A group of Q heads shares one K/V head</td> KV cache, K/V projection, bandwidth</td> attention/gqa.rs</code>, including full/sliding windows and hybrid layers</td></tr> MLA / Gated MLA</td> Compress KV into a low-rank latent and reconstruct it when needed; optionally gate output</td> KV representation and projection cost</td> attention/mla.rs</code>, composed by GLM-5.2, DeepSeek-V3, Kimi-K3, and other runtimes</td></tr> Sliding / block attention</td> Read only a recent window or explicitly visible blocks</td> Long-context attention compute</td> gqa::CausalWindow</code>, attention/block.rs</code></td></tr> DSA / MSA</td> A learned indexer or block index selects Top-K tokens/blocks per query</td> Long-context QK and AV work</td> attention/dsa.rs</code>, attention/msa.rs</code></td></tr> Compressed sparse attention</td> Keep a recent window and pool older history before selection or full reading</td> Distant-history storage and compute</td> attention/compressed_sparse.rs</code></td></tr> Gated DeltaNet / KDA</td> Recursively summarize history in short-convolution and fixed-size recurrent state</td> Avoid a full KV cache that grows with context</td> attention/gated_delta_net.rs</code>, attention/kda.rs</code></td></tr> </tbody></table> Gating, grouping, low rank, and sparse selection are four different mechanisms. A gate controls how much information passes; GQA shares KV among query heads; MLA compresses representation; DSA/MSA let the current query focus on only part of history. They resemble selective human attention: a question need not examine all memories at equal strength. Engineering still has to ask whether selection itself scans all history—sparse output does not automatically imply sparse computation. 3. KV cache: a reusable representation of history</h2> Autoregressive generation adds only one token per round. Recomputing K and V for every earlier token each time would repeat a large amount of matrix multiplication. A KV cache stores the historical K/V representation—or equivalent state—already produced by each layer. Decode only computes the new token, queries the saved history, and appends a new record. It is useful to compare this with an index and contents in working memory, but it is not a lossless copy of the original conversation: GQA stores fewer KV heads; MHA/GQA capacity still generally grows linearly with token count.</li> MLA stores a normalized low-rank latent plus a RoPE component and reconstructs the required representation when read.</li> zLLM's MLA cache supports F16</code> and also defines a layout with per-group INT8 latent values while retaining RoPE in F16. Quantization saves capacity at the cost of measurable error.</li> Sliding windows retain only the effective window; DSA/MSA also maintain indexing information.</li> Gated DeltaNet and KDA retain fixed-size recurrent and short-convolution state. Their update semantics differ from full-attention KV and must not be forced into one storage abstraction.</li> </ul> kv_cache/mod.rs</code> owns device-independent semantics such as logical-layer-to-slot mapping, GQA/MLA shapes, formats, strides, capacity, and valid length. The backend owns physical buffer allocation, placement, quantization kernels, and synchronization. KV stays with the device that computes its layer; when multi-node execution partitions consecutive complete layers, per-layer KV does not travel across the network. 4. FFN: feature processing inside each layer</h2> Attention answers “what should I retrieve from other tokens?” The FFN answers “how should this token transform the information it now has?” A mainstream gated MLP can be summarized as: y = W_down(act(xW_gate) ⊙ (xW_up))</code> Gate and up projections expand hidden state into a wider intermediate space; activation and elementwise multiplication make a nonlinear selection; down projection returns to hidden size. zLLM's moe/dense_mlp.rs</code> preserves this device-independent dataflow. Its current activation specifications cover SiLU, clamped SiLU, SiTU, OpenAI-style SwiGLU, and GELU-Tanh, while the backend capability implements the matrix operations. Dense FFN and MoE</h3> In a dense FFN, the complete gate/up/down weight set participates for every token. It is casually described as “all experts working in a layer,” but in source terms a dense MLP is one complete network block, not a collection of pre-existing experts that were all selected. MoE instead provides many expert FFNs. A router scores them and activates only the Top-K routed experts for each token; shared experts may always run. This resembles the idea that different brain regions become active for different tasks: total parameter capacity can be large while active computation per token remains smaller. The analogy explains sparse activation only; it does not mean Transformer experts map to fixed brain regions or possess named personalities. zLLM divides FFN/MoE responsibility as follows: Module</th> Responsibility</th></tr></thead> moe/dense_mlp.rs</code></td> Dense gated-MLP specification, activations, and gate/up → activation → down</code> dataflow</td></tr> moe/routing.rs</code></td> Routing scores, Top-K, grouped assignments, and active-expert accounting</td></tr> moe/topk_moe.rs</code></td> Routed/shared expert composition, post-routing scaling, and output accumulation</td></tr> moe/latent_moe.rs</code></td> Route from original hidden state while routed experts consume a low-dimensional latent; shared MLP still reads original hidden</td></tr> moe/prefill.rs</code></td> Grouped expert execution and merging for multi-token prefill</td></tr> moe/expert_predictor.rs</code></td> Predict later experts from real route history for asynchronous prefetch; owns neither weights nor I/O</td></tr> </tbody></table> MoE primarily reduces computation per token; it does not automatically reduce total model weights. High-throughput, low-latency deployments generally keep many or all expert weights resident in memory/VRAM, so memory usage remains close to that of the complete MoE model. The router saves FLOPs but does not remove unselected experts from the checkpoint. When capacity is tight and performance requirements are lower, cold experts can reside in CPU memory or on SSD and be streamed after routing. zLLM accordingly places expert source, prefetch, and prediction feedback at backend/runtime boundaries. This introduces I/O latency, prefetch hit rate, concurrent working-set, and jitter problems. Streaming trades residency for acceptable latency only when loading genuinely overlaps current-layer compute and prediction hits often enough. 5. A map of zLLM's algorithm modules</h2> Directory/module</th> What it defines</th> What it does not own</th></tr></thead> attention/mod.rs</code></td> Device-independent entry point for attention-family specifications</td> Device buffers or command submission</td></tr> attention/rope.rs</code></td> Positional rotation, layouts, and reference behavior</td> Model tokenization</td></tr> attention/gqa.rs</code>, mla.rs</code></td> GQA/MLA geometry, windows, projection relationships, and f32 references</td> Platform-specific fused kernels</td></tr> attention/dsa.rs</code>, msa.rs</code></td> Token/block scoring, causal Top-K, and sparse-selection semantics</td> A claim that every selection path is inherently O(K)</td></tr> attention/compressed_sparse.rs</code></td> Recent window, compressed history, visibility, and selection plans</td> Physical placement of compression buffers</td></tr> attention/gated_delta_net.rs</code>, kda.rs</code></td> Recurrent/conv state shapes, update semantics, and references</td> Pretending this state is ordinary KV cache</td></tr> attention/hybrid.rs</code></td> Combined full-attention and linear/recurrent layer state</td> Device-placement policy</td></tr> attention/hyper_connection.rs</code>, attn_res.rs</code></td> Multi-residual-stream and attention-residual semantics</td> HTTP sessions or cross-node transport</td></tr> moe/*</code></td> Dense FFN, Top-K/latent MoE, routing, shared experts, and prefetch feedback semantics</td> SSD/VRAM allocation and DMA</td></tr> kv_cache/*</code></td> Logical layouts, formats, capacity, valid length, and persistence boundaries</td> Platform allocation, synchronization, and kernels</td></tr> runtime/prefill.rs</code></td> Generic chunk, batch, stage, and complete-layer loops</td> A specific model's layer order</td></tr> runtime/generation.rs</code></td> Token generation, EOS, and position-advance lifecycle</td> Attention/FFN mathematics</td></tr> runtime/<model>/</code></td> Compose embedding, per-layer attention + FFN, normalization, LM head from model specifications</td> Duplicating backend algorithms</td></tr> </tbody></table> Production execution has only three complete task forms: NewPrefill</code> creates a session and its cache, AppendPrefill</code> appends a token span after existing history, and DecodeRound</code> consumes one token, runs every layer, appends state, and produces the next token. The algorithm layer defines the correct order, runtime composes a complete model, backend manages resources and submission, and kernels accelerate local computation. 6. How the CPU oracle protects correctness</h2> The CPU oracle is valuable not because “the CPU can just about run the model,” but because it turns complex device optimization into testable problems: Use small but real shapes to verify post-tokenization positions, masks, RoPE, matrix dimensions, and cache append behavior.</li> Compare deterministic CPU f32</code> references with device kernels. Check shape, finite values, absolute/relative error, and discrete results such as sparse indices and MoE expert IDs.</li> Compare hidden state, routing, and logits layer by layer to find where error first grows.</li> Run prefill/decode regressions on fixed token sequences, including cache boundaries, appended prefixes, and long context.</li> Finally run the complete model task. Successful compilation, a passing operator oracle, fixed-token agreement, and acceptable on-device performance are four distinct states.</li> </ol> Tolerance must follow dtype, quantization method, and error propagation; one generous global threshold is not correctness. Top-K routing, causal selection, cache length, and commit position often require exact equality. Evidence-based atol/rtol</code> belongs on F16/BF16/quantized matrix outputs. Even a kernel that passes its local oracle may hurt end-to-end performance through extra conversion, synchronization, or unsuitable shapes, so complete prefill/decode throughput, peak memory, and stability still need validation. That is the role of zLLM's algorithm layer: not another device implementation, but the mathematical contract shared by every implementation. CPU references make the contract executable and regression-testable; attention/KV define how history is read, FFN/MoE define how the current representation is processed, and runtime composes them into one complete, verifiable generation. Quantization Is Not a Zip File: Model Layers and the Reality of 4-bit Inference 2026-09-03T00:00:00+00:00 LLM quantization is often reduced to one sentence: turn 16-bit weights into 4-bit weights and the model becomes four times smaller. That is true, but it omits nearly every difficult part. Quantization first solves a capacity problem: the weights do not fit in VRAM or unified memory. It also offers a possible second benefit—higher speed. Decode is frequently memory-bandwidth-bound, and the weights must be read again for every token. Yet a smaller file does not automatically run faster. If the runtime expands 4-bit weights to F16/F32, or the device lacks a packed kernel, much of the capacity and bandwidth advantage disappears. Quantization is therefore not a file-conversion option. It is an execution path spanning model specification, weight assembly, device residency, kernels, and quality validation. 1. Start with layers: model architecture and weight encoding are different concerns</h2> zLLM does not put everything behind one opaque Model</code> object: model_spec/<model> architecture constants: layers, hidden size, attention, MoE runtime/<model> LayerSpec expansion and complete prefill/decode orchestration weight/model checkpoint names, shape validation, model assembly weight/format byte layouts such as FP8, W4A16, and GGUF K-quants weight/codec IO-free reference decoding and layout conversion backend + kernel residency, format dispatch, and direct device computation </code></pre> The important consequence is that model architecture is not weight encoding. The same linear layer may come from BF16, FP8, NVFP4, W4A16, or Q4_K. The format layer interprets bytes, the model adapter knows names and shapes, runtime describes dataflow, and the backend selects a kernel according to device capability. 2. What quantization actually solves</h2> Capacity comes first</h3> Ignoring scales, metadata, and tensors retained at higher precision, an N-parameter model needs approximately: BF16 / FP16: 2N bytes INT8: 1N bytes INT4: 0.5N bytes </code></pre> A 70B model is therefore about 140 GB in BF16 and ideally 35 GB at 4-bit. Real Q4_K_M files are larger because block scales, minima, alignment, and protected tensors also consume space. Quantization only shrinks the data it covers: KV cache, activations, scratch buffers, and runtime memory still need separate budgets. On discrete GPUs this determines whether weights fit in VRAM. On Apple UMA it determines whether the model, KV cache, OS, and application can coexist without severe memory pressure. Eliminating a traditional host-to-VRAM staging copy does not make capacity infinite. Moving fewer bytes can make decode faster</h3> Single-token decode is dominated by GEMV-like operations with little weight reuse. If weights stay packed and a kernel decodes each local block directly into accumulation, 4-bit storage offers close to four times the effective weight density of BF16. That speedup requires a native device kernel, low enough decode/scale/activation overhead, and end-to-end measurement. In one real zLLM Qwen3.8 CPU path, packed Q4_K AVX2/FMA dot products reduced a 16-token decode from 107.483 to 7.578 seconds, and paired gate/up traversal reached 7.183 seconds. This roughly 15× change combined removal of repeated F32 expansion with specialized vectorization; it is not a universal 4-bit-versus-BF16 ratio. Conversely, a Gemma 4 Q4_K dual-output kernel passed its CPU oracle but reduced end-to-end throughput from 35.9 to 35.5 tok/s and was reverted. 3. The core idea: local structure, not global rounding</h2> Naive quantization shares one range across an entire matrix: q = round(w / scale) w_hat = scale * q </code></pre> Outliers stretch that range and leave too few useful grid points for ordinary values. Modern weight quantization therefore makes three local decisions. Block locality. Groups of 32, 64, 128, or 256 weights receive their own scale. Smaller groups fit local distributions better but add metadata and kernel cost. Tensor and channel sensitivity. Attention value/output projections, FFN down projections, embeddings, and the LM head need not tolerate the same error. AWQ similarly uses activation statistics to identify and protect salient weights. Layer locality. Error propagates through depth: a hidden-state shift changes later routing, attention, and token ranking. Layerwise calibration controls accumulated error better than one global conversion. MoE calibration must also exercise a representative expert-routing distribution. How to read common GGUF quantization names</h3> GGUF is a container, not a quantization algorithm. It stores metadata, tokenizer data, a tensor directory, and tensors that may independently use F32, F16, Q4_K, Q6_K, IQ4_XS, and other types. A Q4_K_M</code> filename does not mean every tensor uses an identical 4-bit block. Family</th> Common names</th> Structure and use</th> Practical guidance</th></tr></thead> Classic blocks</td> Q4_0, Q5_0, Q8_0</td> Integer codes plus a block scale; simple and widely implemented</td> Q4_0 favors compatibility; Q8_0 is a useful quality baseline or activation format</td></tr> K-quants</td> Q3_K, Q4_K, Q5_K, Q6_K</td> Sub-block scales/minima inside 256-weight super-blocks</td> Start with Q4_K_M; use Q5_K_M/Q6_K when memory permits</td></tr> I-quants</td> IQ2_, IQ3_, IQ4_NL, IQ4_XS</td> Importance-aware or nonlinear codebooks</td> Strong quality per bit, but verify native backend kernels</td></tr> Mixed recipes</td> Q3_K_S/M/L, Q4_K_S/M, Q5_K_S/M, UD-*</td> Different tensor types according to sensitivity</td> Read the tensor directory; never infer support from the filename alone</td></tr> </tbody></table> S/M/L</code> usually means Small, Medium, or Large mixed-precision policy, not three integer widths. IQ4_XS</code> and Q4_K_M</code> also are not interchangeable: the former uses nonlinear/codebook ideas, while the latter uses hierarchical K-quant scales. A custom name such as UD-Q4_K_XL</code> may mix Q4_K, Q5_K, Q6_K, and IQ4_XS across layers. 4. Why 4-bit is often the sweet spot</h2> The sweet spot is where three curves meet: capacity falls sharply, devices can still decode efficiently, and quality has not entered the steep decline often seen at 2–3 bits. For Mistral 7B on WikiText-2 at context 512, one same-method llama.cpp comparison reported: Format</th> Perplexity</th> Increase over FP16</th></tr></thead> Q3_K_S</td> 6.0021</td> 5.44%</td></tr> Q3_K_M</td> 5.8489</td> 2.75%</td></tr> Q4_K_S</td> 5.7349</td> 0.75%</td></tr> Q4_K_M</td> 5.7259</td> 0.59%</td></tr> Q5_K_S</td> 5.7100</td> 0.31%</td></tr> </tbody></table> Moving from Q3_K_M to Q4_K_M cuts the relative increase from 2.75% to 0.59%; moving again to Q5_K_S only reaches 0.31% while consuming more memory bandwidth. Llama 33B results show a similar curve: FP16, Q6_K, Q5_K_M, Q4_K_M, and Q3_K_M scored 4.1557, 4.1598, 4.1675, 4.2081, and 4.3594 respectively. These numbers establish a trend only for that model, dataset, context, and quantizer. Perplexity is not synonymous with chat quality, code correctness, or long-reasoning stability. Small local errors can flip tokens near a decision boundary or change MoE Top-K routing. 5. Why models can be quantized at all</h2> Neural networks are not bit-exact symbolic programs. Many parameters form redundant statistical representations, while residual connections, normalization, and overparameterization tolerate small perturbations. Quantization exploits that redundancy. Generation is probabilistic as well. Logits pass through softmax, temperature, top-k/top-p, and sampling. At nonzero temperature, identical weights can produce different answers; even greedy decoding can flip when two logits are extremely close. This does not mean errors are harmless. Open-ended generation often distinguishes good from better rather than one exact string, so quality must be evaluated statistically and by task. Mathematics, code, tool arguments, and JSON schemas still have hard correctness boundaries. A quantized model may occasionally score above the floating-point baseline. Quantization noise may break an existing bad preference, but finite samples, sampling variance, and evaluator noise can do the same. Only repeated, multi-seed, multi-dataset results with uncertainty justify a claim of genuine improvement. 6. What training and calibration compensate for</h2> Post-training quantization (PTQ) uses representative samples to estimate scales, clipping, channel importance, or Hessian approximations. GPTQ compensates layerwise for induced weight error; AWQ uses activations to protect salient weights. The target is not merely “each weight remains close,” but “the layer output remains close on real inputs.” Quantization-aware training (QAT) simulates rounding and clipping during training so parameters adapt to the low-precision grid. It costs more but becomes increasingly important for 3/2-bit weights, activation quantization, and sensitive tasks. Calibration data must match deployment: short prose is a poor calibration set for long reasoning, code, or broad MoE expert routing. Weight, activation, and KV-cache quantization are separate choices. W4A16 keeps activations at 16-bit; W4A8 adds dynamic-range and fused-kernel constraints; KV quantization changes long-context capacity and continuously perturbs attention. Calling all three a “4-bit model” hides essential information. 7. Fences constrain structure; they do not manufacture correctness</h2> Three engineering fences are needed: Operator fence: reference codecs, CPU oracles, layerwise hidden/logit comparisons, and real shape validation.</li> Generation fence: token-level JSON Schema, tool-name, closing-tag, stop-token constraints, plus an independent repetition guard.</li> Quality fence: greedy regression prompts, perplexity, task suites, long contexts, multi-seed evaluation, and human review for critical tasks.</li> </ol> A generation fence can guarantee allowed structure, not correct semantics. A JSON repair pass must not hide tool-selection drift. Chain-of-thought should not be judged only by verbatim equality either: visible reasoning is sampling-sensitive and is not the model's internal computation. Evaluate final answers and tool outcomes, key intermediate constraints, finish reasons, loop rate, length distribution, and success-rate changes under the same decoding policy. Short greedy prompts can produce a dangerous false pass. Small logit shifts accumulate over thousands of reasoning tokens. Long tests should track correctness, semantic restarts, repetition, premature EOS, and pathological long tails. Fences prevent protocol disasters; evaluations discover semantic regression. 8. Engineering pitfalls</h2> “Support” must be split into four states: file can be parsed ≠ CPU reference can decode it ≠ device has a packed direct kernel ≠ model uses it in the default end-to-end path </code></pre> Packed data must remain packed until the kernel. Expanding it to F32 at load time erases the memory and bandwidth advantage. The desired lifecycle is container lookup → format interpretation → name/shape validation → backend residency → block-local decode and accumulation. Do not forget non-weight memory. KV cache, prefill activations, MoE working sets, vision encoders, logits, and command buffers can become the new peak. Measure TTFT, prefill and decode throughput, peak memory, SSD wait, GPU utilization, and thermal steady state. Finally, “the output looks fluent” is not correctness: validate codecs, operator oracles, layerwise differences, fixed-token regressions, and full task suites. Conclusion</h2> Quantization reallocates a budget among precision, capacity, bandwidth, and compute. Four bits is often attractive because it crosses the capacity threshold, retains near-baseline statistical quality, and maps well to packed device kernels. It is not a magic number independent of model, data, hardware, and task. A trustworthy deployment answers four questions: what is quantized at what granularity; whether the device consumes the packed representation directly; whether speed improves end to end; and whether quality fences cover long reasoning, structured output, and real tasks. Only then does “the model runs” become “the model is deployable.” References</h2> GPTQ</a></li> AWQ</a></li> llama.cpp Mistral 7B quantization comparison</a></li> llama.cpp K-quant perplexity records</a></li> llama.cpp quantization formats</a></li> </ul> Decoupling: Let Models, Algorithms, Backends, and Kernels Evolve Independently 2026-09-01T00:00:00+00:00 First drafted on 2026-08-28; rewritten on 2026-09-01. This article follows the current model_spec/</code>, runtime/</code>, backend/</code>, kernel/</code>, weight/</code>, and server/</code> source trees. It describes boundaries that exist in the implementation rather than presenting a generic planner or remote Expert RPC as completed architecture. Decoupling in zLLM is not about making the directory tree look tidy. It is about making each kind of change travel along the right axis: model architecture changes belong mainly in the spec and runtime, new mathematical semantics enter the domain layer first, device differences stop at the backend, and local performance work stays in kernels. A complete inference request still uses all of these regions, but no region needs to know every implementation below it. 1. The Architecture at a Glance</h2> The main dependency direction runs from top to bottom. Weight formats form a side data path: model_spec</code> supplies shapes, weight</code> parses containers and encodings, and the backend selects the final resident representation. This side path does not control model execution. The five-layer spine and the device implementation region are responsibility boundaries, not six runtime processes. Inside device implementation, backend/<platform></code> and kernel/<platform></code> are peers: the platform backend owns tensors, weights, caches, placement, submission, and transfer;</li> parameterized kernels own local computation;</li> together they implement the capabilities required by the upper layers.</li> </ul> Production requests still execute a complete New Prefill, Append Prefill, or Decode Round. Decoupling changes code dependencies and resource ownership; it does not turn full-model inference into disconnected partial demonstrations. The responsibility of each region is concise: bin/runtime</code> and bin/tools</code> compose platform entry points driven by --config <yaml></code>. Model-specific validation and profiling binaries live outside the main runtime.</li> embedded</code> exposes an in-process Engine</code> with the same structured JSON semantics as service mode, without starting HTTP, the scheduler, or iroh.</li> server/</code> owns HTTP/H3 protocols, Chat/Anthropic/Responses compatibility, nodes, the scheduler, and iroh stage transport.</li> model_spec/<model></code> contains execution- and platform-independent architecture data. It is the single specification shared by runtime orchestration and weight assembly.</li> runtime/<model></code> expands layer specs and owns platform-independent full-model orchestration. Platform composition files remain beside the model runtime.</li> attention/</code>, moe/</code>, kv_cache/</code>, and norm.rs</code> contain domain specs, reusable algorithms, and reference semantics.</li> backend/</code> contains capability contracts and platform resource implementations, including cache, completion, streams or command buffers, residency, and scheduling.</li> kernel/</code> contains parameterized local computation. Model dimensions may not be hard coded. ROCm HIP bodies are stored in separate source.hip</code> files; Rust launchers load, validate, and submit them.</li> weight/</code> parses standard containers and encodings. Model-specific code is limited to naming and shape adaptation.</li> </ul> 2. Dependency Rules</h2> Shared scheduling such as runtime::prefill</code> does not depend on a concrete model or backend.</li> model_spec</code> contains architecture data only and does not depend on other crate modules. Runtime and weight assembly depend on it without depending on each other.</li> Platform-independent code in runtime/<model>/mod.rs</code> depends on capability traits. Only adjacent platform composition files may depend on a concrete backend.</li> backend/</code> and kernel/</code> do not name or branch on concrete models. Cooperative two-GPU MoE and Metal replay express device capabilities; they may not encode GLM layer numbers.</li> Domain modules contain specs, algorithms, and references. Concrete storage, kernels, and synchronization belong to device implementation.</li> weight/</code> parses and loads formats. Algorithms do not enter the weight tree, and platform resources do not enter format parsing.</li> </ol> A violation usually indicates dependency inversion. Any exception must be a concrete platform composition file or a measurable production execution path. It must not be an empty flow, factory, or protocol introduced for hypothetical reuse. 3. The Three Decoupling Boundaries</h2> Boundary</th> Upstream expression</th> Downstream responsibility</th> Current constraint</th></tr></thead> A · Model × Domain</td> Layer dataflow and complete prefill/decode ordering</td> Mathematical semantics for attention, MoE, KV, and their references</td> Define and test new semantics before accelerating them</td></tr> B · Domain × Capability</td> Required capabilities declared by generic bounds</td> Tensor/cache/weight residency, submission, and transfer contracts</td> Runtime does not touch HIP events or Metal command buffers</td></tr> C · Capability × Implementation</td> Capability and resource-lifecycle contracts</td> Peer platform-backend and parameterized-kernel tracks</td> Both tracks implement capabilities; kernels contain no model names or constants</td></tr> </tbody></table> Boundary A: Model × Domain</h3> Model orchestration calls domain operations and passes a generic backend through the dataflow. The domain describes what a computation means; device implementation decides how it runs. A new operation such as kpool first receives a reference definition and tests. GPU kernels are accelerated substitutes for that definition rather than independent sources of semantics. CPU remains the primary correctness oracle, but it is not limited to that role. Current GLM-5.2 paths also use CPU work in production for DSA selection and the DSpark drafter. The boundary is semantic ownership, not a rule that all CPU code must be diagnostic. Boundary B: Domain × Capability</h3> The base Backend</code> trait contains operations shared across models. Domain capabilities such as KdaKernel</code>, HyperConnectionKernel</code>, DsaPrefillBackend</code>, and ExpertPrefillBackend</code> remain separate. A model's generic bounds state exactly what it needs. New capabilities may use explicit unsupported defaults so existing backends can adopt them incrementally without pretending to support the new semantics. Stage execution uses the same principle. Runtime sees completion, available resources, and submission classes; it does not see HIP events or queue handles. The ROCm backend may choose latency or background streams and retire completions out of order without leaking those mechanics into model code. Boundary C: Capability × Device Implementation</h3> backend/<platform></code> and kernel/<platform></code> sit on the same side of this boundary as peer tracks. The backend owns resources, resident representations, shape/format dispatch, and submission semantics. Kernels own parameterized local computation. Weights enter device implementation through LinearWeight</code> variants such as F32, F16, Bf16Bytes, FP8, MXFP8, MXFP4, NVFP4, W4A16, W8A16, and GGUF. A platform backend selects a kernel using weight format and shape. Quantized paths decode inside the kernel instead of expanding weights to F32. This is a collaboration relationship, not an architectural hierarchy from platform backend down to kernel. 4. Key Decisions</h2> Decision</th> Rationale</th></tr></thead> Keep model_spec</code> data-only instead of introducing a behavioral thin model layer</td> Runtime orchestration and weight assembly share one specification without depending on each other</td></tr> Use associated Tensor</code>, Weight</code>, and Cache</code> types instead of one universal object</td> CPU values, Metal buffers, ROCm dual host/device state, and wgpu resources are fundamentally different</td></tr> Split capability traits by domain</td> Model dependency surfaces stay readable and new capabilities can enter incrementally</td></tr> Define reference semantics before optimized kernels</td> Cross-backend behavior remains testable while optimization proceeds independently</td></tr> Keep the single model flow in runtime and platform composition next to it</td> Avoid N-model × M-platform copies of the same execution algorithm</td></tr> Drive entry points through YAML configuration</td> Keep service and tool entry points converged; validation and profiling remain separate concerns</td></tr> </tbody></table> 5. Benefits</h2> Combination cost grows linearly. A new model built from existing operations requires no backend changes. A new operation on a new backend requires a kernel and a capability implementation, not another full runtime.</li> Correctness has an anchor. Reference behavior and tests allow a backend to start with a slower correct path and optimize later.</li> Platforms evolve independently. Metal replay, ROCm stage pipelines and CPU/GPU DSA cooperation, CUDA paths, Vulkan, and experimental NPU work can progress without changing model semantics.</li> Reviews have a clear target. A change to mathematical meaning, device resources, or a local kernel has a mostly unique home.</li> </ol> 6. Costs and Trade-offs</h2> Hot semantics still need multiple implementations. Rust references, MSL, HIP, CUDA, WGSL, and AscendC do not optimize themselves.</li> Generic instantiation increases compile time and binary size. Capability bounds also become longer as models add domains.</li> Capability growth requires discipline. Every new architecture concept creates pressure to add shared trait methods even when the capability is not yet general.</li> Platform-specific behavior must be documented. Fallbacks, shape constraints, replay, CPU/GPU cooperation, and multi-GPU paths differ by backend.</li> Peer backend and kernel tracks require careful ownership. Dispatch and resource lifetime belong to the backend; local math belongs to the kernel. Moving either concern to the other track makes the boundary harder to audit.</li> </ol> 7. Series Guide</h2> 02 · Model Layer: formats, config, weight assembly, and quantization</li> 03 · Algorithm Layer: attention, MoE, KV cache, prefill, and decode orchestration</a></li> 04 · Backend: device abstraction, resources, submission, and platform behavior</li> 05 · Kernels: primitive and fused kernels, precision selection, and hardware mapping</li> 06 · Service Layer: fences, protocols, input/output, and the embedded library</li> 07 · Multi-node: scheduling, admission, cross-node state, and stage pipelines</li> 08 · Speculative Decoding: transactional verification, MTP, DSpark, and fences</li> </ul> Why Did One Sentence Consume Hundreds of Thousands of Tokens? 2026-08-30T00:00:00+00:00 Behind one short sentence is a huge token stream containing system instructions, tool definitions, project context, and history. Hot state stays in VRAM; evicted state is backed up to SSD and restored when needed. You typed one sentence. Why did the usage report show tens of thousands—or even hundreds of thousands—of input tokens? Because agents such as Claude Code and Codex do not send only the text visible in the input box. Before calling the model, the client assembles a complete working context: system and developer instructions, tool descriptions, JSON Schemas, workspace rules, conversation history, previous tool calls and results, plus file excerpts or summaries needed to continue the task. Your sentence may be only 20 tokens, but it is the final sliver of a very large prompt. In other words, every request must tell the model three things: who it is, what it can do, and what has already happened. This is not training and it does not permanently write knowledge into the model. It is context supplied for this inference request, and much of it normally has to be present again in the next request. That is exactly why prefix caching matters. A client may resend tens of thousands of identical tokens, but a server that retained the model state for that prefix does not need to recompute it from the beginning. At first glance, an LLM service looks like an ordinary JSON-over-HTTP application: send messages in, stream tokens out. Once it must support several client protocols, long conversations, persistent cache state, and reliable tool calls, the problem becomes much deeper than adding a few routes. In zLLM we implemented three common interfaces—OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages—plus a two-level prefix cache with VRAM as the active tier and SSD as its backup, and token-level generation fences for DeepSeek DSML tool calls. They appear to be separate features, but all enforce the same invariant: protocol state, the tokens seen by the model, and the execution state stored on the device must agree exactly. The tens of thousands of tokens behind one sentence</h2> A typical Claude Code or Codex request contains roughly these parts: Context component</th> What it contains</th> Why it may reappear every turn</th></tr></thead> System and developer instructions</td> Identity, safety boundaries, output rules, coding conventions, workflow</td> A model does not remember these rules across independent HTTP requests</td></tr> Tool definitions</td> Names, descriptions, and parameter Schemas for shell, files, search, browser, and other tools</td> The model must know which actions exist and how to invoke them legally</td></tr> Workspace knowledge</td> Current directory, repository rules, AGENTS.md, environment and permissions</td> The agent must act in the correct project under the correct constraints</td></tr> Conversation history</td> User requests, prior reasoning, completed work, intermediate conclusions</td> Keeps a multi-turn task coherent</td></tr> Tool calls and results</td> Commands, file contents, errors, logs, and returned data</td> Later decisions depend on these observations</td></tr> Current input</td> The sentence you just typed</td> Usually only the tail of the full prompt</td></tr> </tbody></table> This is what people mean when they say an agent first “injects tens of K of knowledge” into the model. Here, K means a thousand tokens. Tokens are not characters: a tokenizer splits prose, punctuation, paths, source code, and JSON differently. Tool Schemas are particularly expensive because every field name, description, enum, nested object, and protocol example counts as input. With dozens of enabled tools, their declarations alone can occupy a substantial part of the context window. The UI usually shows only the user's messages and the final answer, making this cost easy to mistake for secret model output. In practice, usage has at least three relevant categories: input_tokens</code>: the complete context supplied this turn, not just the newest user message;</li> cached_tokens</code>: the prefix within that input whose execution state the server reused;</li> output_tokens</code>: newly generated reasoning, text, or tool calls.</li> </ul> Billing rules differ by provider, but computationally, cached tokens are still part of the input. They really are present in the prompt; the server simply avoids repeating the same prefill work. This produces a result that looks contradictory but is not: a request can contain 100,000 input tokens while only the final few hundred tokens require new computation. Without a cache hit, all 100,000 tokens must pass through every layer of the model again, increasing time to first token. With a hit, the server restores the previous state and runs append prefill only for the new user message and assistant prefix. Three major APIs: the differences go beyond field names</h2> zLLM exposes three primary endpoints: API</th> Typical clients</th> Main input</th> Streaming output</th> Continuation model</th></tr></thead> POST /v1/chat/completions</code></td> OpenAI SDKs and general chat clients</td> messages</code></td> Chat chunk SSE</td> Client resends history and may supply a cache identifier</td></tr> POST/GET /v1/responses</code></td> Codex, ZCode, newer agent clients</td> input</code> items</td> Responses SSE or WebSocket events</td> previous_response_id</code></td></tr> POST /v1/messages</code></td> Claude Code and Anthropic SDKs</td> top-level system</code> plus messages</code></td> Anthropic event SSE</td> Client resends history</td></tr> </tbody></table> All three eventually become the same internal tasks. Structured messages pass through the model's chat template and tokenizer, followed by NewPrefill</code>, AppendPrefill</code>, and DecodeRound</code>. The adapter layer, however, cannot be a collection of field renames. Chat Completions: the most direct common representation</h3> Chat Completions centers on an ordered messages</code> array. Tools use the OpenAI function-tool shape: { "model": "deepseek-v4", "messages": [{"role": "user", "content": "Check the weather in Shanghai"}], "tools": [{ "type": "function", "function": { "name": "get_weather", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"] } } }], "stream": true } </code></pre> A non-streaming response collects text and tool_calls</code> into one assistant message. A streaming response must keep each tool call's index</code>, ID, function name, and argument deltas stable. Tool-call IDs cannot be derived from the function name alone: the same function can appear in multiple requests or be called more than once in a response. zLLM hashes a request-level scope, call index, and function name together. Chat Completions is a useful internal canonical form because roles, tools, and sampling controls are explicit. Responses and Anthropic requests are carefully converted into this representation before they reach the scheduler and runtime. Responses: an event protocol, not Chat with different JSON</h3> The Responses API accepts a stream of input</code> items rather than just a message array. Items may include messages, function_call</code>, function_call_output</code>, images, and the additional_tools</code> prefix used by Codex Responses Lite. Those tools may be wrapped in namespaces, so conversion must expand executable functions while skipping hosted tool types that the local runtime cannot execute. Output is not a growing delta.content</code> string either. Text, reasoning, and function calls are separate output items with their own created, delta, and completed events. Reasoning must be exposed as a distinct output[type=reasoning]</code> item; folding it into answer text prevents clients such as Codex from rendering and restoring state correctly. zLLM supports both HTTP POST with SSE and a WebSocket upgrade on GET /v1/responses</code>. The latter serially reuses one connection for multiple response.create</code> requests, which suits long-running agent sessions. previous_response_id</code> needs special attention. zLLM persists the protocol messages and cache key of the previous response, so it can reconstruct a conversation chain after restart. But that record does not own model KV state. Restoring protocol history only means the complete input can be rebuilt; whether VRAM or SSD state is available remains a separate runtime decision. Anthropic Messages: content blocks and event order are semantic</h3> Anthropic puts system</code> at the top level and represents content as blocks. Tool definitions use name</code>, description</code>, and input_schema</code>; invocations and results are tool_use</code> and tool_result</code> blocks. The adapter must convert these structures in both directions instead of flattening them into strings. Streaming event order is also strict: message_start</code>, then content_block_start</code>, the corresponding deltas, a block stop, and finally message_delta</code> and message_stop</code>. Text and multiple tool calls may occupy different block indexes. Renaming Chat SSE fields produces incorrectly ordered or unclosed blocks. Claude CLI also calls /v1/messages/count_tokens</code> to decide when to compact the context. Even an estimated count must include system content, messages, and tools. Counting only user prose delays compaction far beyond the intended threshold. What a cache hit actually means</h2> In an inference service, a cache hit does not mean “we found a similar question,” and it does not mean “return the old answer.” Its precise meaning is: The beginning of the new request's token sequence is exactly equal, token for token, to a saved session. The runtime can restore every layer's KV and auxiliary DSA/MTP state at the end of that prefix and continue from there. </blockquote> Suppose the saved terminal state represents: [system][tools][user-1][assistant-1] </code></pre> The next request is: [system][tools][user-1][assistant-1][user-2][assistant-prefix] |<----------- cached_tokens ----------->|<-- append prefill --> </code></pre> The first cached_tokens</code> do not run through the full Transformer again. Only the appended suffix receives prefill before decoding begins. A hit reduces repeated prefill and time to first token; it does not eliminate decoding for the new answer. Several consequences follow: Tokens must match; visually similar text is insufficient. Whitespace, JSON serialization, tool-schema order, thinking controls, templates, and special tokens can all fork the prefix.</li> A cache ID is a lookup hint, not proof of correctness. The runtime still verifies the token prefix and namespace, falling back to a longest-prefix search or cold prefill if they disagree.</li> The cached length must be shorter than the new prompt. A complete terminal state without appended tokens is not a valid append boundary.</li> A hit does not imply residency. The matching state may be in VRAM or backed up on SSD. Both avoid recomputing the prefix, but restore latency differs greatly.</li> </ol> Terminal-cache identity includes not only user-visible messages, but also model and template-related request state, assistant output, and structured tool calls. Otherwise, two requests that look identical on screen could share incompatible execution state. Why VRAM needs an SSD backup tier</h2> Long-context KV is expensive. Agent conversations with tens or hundreds of thousands of tokens benefit enormously from retaining historical branches, but VRAM cannot grow without bound. A pure LRU that simply discards old sessions makes a recently evicted long conversation restart from zero. The actual hierarchy is: Request arrives ├─ exact cache ID resident in VRAM ─────────┐ ├─ longest matching prefix in VRAM ─────────┤ ├─ exact cache ID backed up on SSD ─────────┤→ restore to VRAM → append prefill → decode ├─ longest matching prefix on SSD ──────────┤ └─ no match → new prefill in VRAM ──────────┘ VRAM: active primary cache, smallest capacity, lowest latency │ ├── eviction backup ──> SSD └── on-demand restore <── SSD SSD: larger backup store; it never serves as a parallel inference destination </code></pre> Level one: resident terminal state in VRAM</h3> After a successful request, the engine retains the terminal session: token sequence, namespace, parent and round relationship, and model-private execution state. The common cache layer owns entries, prefix graphs, and eviction; each model runtime defines its own KV, compressed rings, DSpark target cache, and other state. Admission must use a real residency budget, not a limit such as “keep N sessions.” Sessions with similar token counts can differ by several times in allocated bytes due to fixed and stepped costs: recent/compressed rings, batch scratch, speculative blocks, and target caches. zLLM accounts using actual allocated_bytes</code> and reserves room for KV growth during decode. The cache is not a flat LRU list. Agent conversations branch from shared history, so the memory tier maintains a shared prefix block graph. Parents may be reused by several descendants, while only dead branches are released. A prefix-round limit prevents every intermediate terminal state from remaining resident forever. Level two: SSD backup and restore</h3> When VRAM evicts a terminal state worth retaining, the runtime downloads and encodes the model state into a versioned SSD snapshot. The index records cache ID, complete token sequence, namespace, round, head status, resident byte count, and blob generation; large KV data is stored in chunks. Writes need a commit boundary. The manifest switches to a new generation only after its blob is complete, so a crash cannot leave an index pointing to a half-written cache. Reads treat disk snapshots as untrusted input: version, lengths, counts, and trailing bytes are validated before allocation. On an SSD hit, the engine first obtains a reusable device session, uploads each stage cache, restores DSpark and other auxiliary state, and only then continues computation in VRAM. SSD is therefore a backup capacity tier, not another compute path. SSD also supports longest-prefix lookup. Matching remains strict: tokens.starts_with(snapshot.tokens)</code>, isolated by namespace. To diagnose almost-hits, zLLM logs snapshots sharing at least 1,024 tokens and more than 90% of the old prefix. Those reports often reveal a template, tool echo, or serialization fork immediately. The hardest cache problems we encountered</h2> Eviction order and admission deadlock</h3> An early implementation tried to reserve memory for a new request before evicting old cache entries. A request could be rejected even though reclaimable state existed. The correct order is to identify disposable resident entries, finish any required backup and release, and only then perform admission—while still reserving capacity for decode growth. Asynchronous restore still needs explicit dependencies</h3> SSD reads, CPU decoding, and multi-GPU host-to-device uploads can run concurrently, but their dependencies cannot disappear. We encountered restore jobs writing cache buffers after a compute stream had begun reading them, as well as cache-open commands whose resources were reused before draining. The fix was not a blanket global synchronization. Each stage publishes a restore-completion event; independent I/O and uploads overlap, while the first consumer waits on the exact dependency it needs. Reusable pools can still leak capacity</h3> If every SSD restore creates a new session, old device buffers may accumulate indefinitely in a “reusable” pool without actually being reused. Restore must take and overwrite a pooled session first. Failure and cancellation paths must reset and return sessions just as carefully as the success path. Tool echoes are a frequent cache-chain breakpoint</h3> A tool call travels through four forms: native model text, structured API call, client execution, and re-encoded next-turn history. If forward parsing and history rendering are not inverse operations, the next turn forks at the tool position. DSML continuation failures we saw came from differences in tags, string-versus-JSON parameter encoding, and case normalization. Each ToolDialect</code> therefore owns four related operations: instruction injection, tool-call history rendering, tool-result rendering, and model-output parsing. They must come from the same protocol implementation rather than being scattered between HTTP adapters and backends. Tool-call fences: why parsing after generation is not enough</h2> Prompting a model to “return valid JSON” is not a guarantee. It may invent a tool, repeat a parameter, omit a required field, emit a closing tag inside JSON, or fall into a loop after long reasoning. Parsing the final output can only report failure after the decode work has already been wasted. zLLM therefore implements tool support in three layers: Protocol prompting and history rendering encode a common function schema into GLM XML, ChatML JSON, or DeepSeek DSML.</li> Incremental parsing separates visible text from structured tool regions without leaking half a tag to the client.</li> Token-level generation fences force or exclude candidates before sampling so structured regions can follow only valid paths.</li> </ol> The third layer is the actual fence. For a named DeepSeek DSML tool, the state machine is: fixed tool_calls/invoke prefix ↓ choose one unseen parameter ↓ generate its value under JSON Schema ↓ close parameter; are all required fields present? ├─ no: another parameter is required └─ yes: invoke/tool_calls may close </code></pre> The fence forces fixed tags, the tool name, and parameter names. JsonSchemaFence</code> determines legal value tokens. A parameter can appear only once, required fields must be complete before closing, markup-opening tokens are excluded inside strings, and terminal tokens remain illegal while the structure is unfinished. tool_choice=auto</code> is harder. The model must remain free while writing ordinary text; the fence activates only after detecting the DSML trigger. Several tool candidates may remain possible, so their state machines advance in parallel and incompatible branches are discarded. The allowed next-token set is the union of surviving branches. Forcing a tool from the beginning would incorrectly turn auto</code> into required</code>. The fence sits between the output head and sampling. Ordinary decode, every batch row, MTP or DSpark draft generation, and speculative verification must all use the same request-level fence state. Missing one path creates the classic failure mode: tools are valid without speculative decoding and intermittently malformed when it is enabled. Tool-fence bugs that taught us the most</h2> Tokenizer boundaries are not string boundaries</h3> A closing tag may span tokens or share a special </</code> token with other text. A fence cannot assume one character or one tag per token. At initialization, all literals are encoded with the active model tokenizer and closing boundaries are verified. An incompatible tokenizer causes fence construction to fail early instead of entering an unreachable generation state. Auto branches require a union, not the wrong intersection</h3> Different candidate tools may permit different next tokens. A token can be excluded only when every surviving branch rejects it. Naively merging excluded sets kills valid branches too early. After each generated token, branches incompatible with that token must also be removed. Draft tokens need a cloned state advanced one token at a time</h3> Speculative decoding proposes several tokens at once. The current fence cannot validate the whole batch as one unit, and drafts must not mutate committed state. zLLM clones the guard, computes a fence for each position, advances the probe token by token, and advances the real state only for the accepted prefix. Positions already forced by the grammar can bypass unnecessary logit work. Parser patches are not a substitute for generation constraints</h3> We added compatibility for case differences, escaped DSML tags, and stray closing tags. Parser tolerance helps with existing model behavior, but predictable structural errors should still be prevented before sampling. Otherwise a non-streaming result may appear repaired after the fact while a streaming client has already received the malformed fragment. Syntax fences and loop protection must compose</h3> Tool grammar prevents invalid structure; it does not stop a model from repeating its reasoning indefinitely. zLLM's GenerationGuard</code> detects repeated tokens, repeated patterns, and evidence-backed semantic restarts. It excludes the known looping successor before the next sample; if a speculative batch already contains a loop, it truncates accepted rows and may force the reasoning-end token. Tool grammar and loop protection combine into one TokenFence</code>, but remain separate state machines so model protocol logic does not contaminate the general generation layer. The complete path of one request</h2> Putting the APIs, cache hierarchy, and tool fence together, a real request follows this path: Chat / Responses / Anthropic request ↓ validation and normalization common messages + function tools ↓ model dialect, template, tokenizer complete prompt tokens + cache identity ↓ exact/longest prefix in VRAM → exact/longest backup on SSD → cold start ↓ │ └──────── restore to VRAM ───────┘ ↓ NewPrefill or AppendPrefill ↓ output head → request token fence → sampling/speculative verification ↓ incremental native tool-stream parser ↓ Chat chunks / Responses events / Anthropic content blocks ↓ successful terminal state remains resident in VRAM; evicted state is backed up to SSD </code></pre> The HTTP layer does not own KV. The cache layer does not understand model syntax. The backend does not know tool names. The model runtime does not emit a particular external SSE format. Each layer owns only the semantics it needs, but passes verifiable state across its boundary rather than a vague “session ID.” The final test</h2> A prefix cache is not correct because a log line says hit=true</code>. Restored tokens, positions, per-layer KV, and auxiliary state must match a full prefill. Performance is measured by real time to first token, not only by the number of skipped tokens. A tool implementation is not correct merely because a final string passes a JSON parser once. Streaming event order, history replay, next-turn cache continuation, batch decode, and speculative paths must all obey the same constraints. The lesson is simple: API compatibility does not end when the JSON looks similar. A cache hit does not end when the ID matches. Structured generation does not end when malformed output can be repaired afterward. Reliable inference ultimately comes down to the exact tokens seen by the model and the exact state held by the device. MacBook Air M5: Run a Multimodal LLM with Zero Runtime Dependencies 2026-08-29T00:00:00+00:00 zLLM requires no Python, PyTorch, Conda, Docker, or resident model server. End users receive one native executable and model weights; Rust developers can put the same engine directly inside their own process. </blockquote> On a 10-core MacBook Air M5 with 24 GB of unified memory, the current release build of zllm-metal</code> is 8.1 MB. Add the Q4_K_M Gemma 4 E4B model and its vision projector, and this small program can run multi-turn chat and image understanding directly through Metal. “Zero dependencies” here means zero external framework dependencies at deployment and runtime. There is no Python environment, PyTorch dynamic library, or separately managed C++ inference runtime. Model weights remain required data, and source builds naturally compile zLLM's Rust crates and macOS system bindings through Cargo. Two interfaces, one inference path</h2> zLLM offers both a library and a command-line interface. They share the same model Runtime, Metal Backend, Kernels, weight loading, KV cache, and generation state machine. Interface</th> Best for</th> Starts a service?</th></tr></thead> zllm::embedded::Engine</code></td> Rust desktop apps, tools, and services</td> No; it runs in-process</td></tr> zllm-metal</code></td> Immediate local chat and image understanding</td> No; it is one native CLI</td></tr> </tbody></table> Library mode is not an HTTP client wrapper. Engine</code> owns the model and KV cache directly. Requests use familiar OpenAI Chat JSON semantics, while generated tokens arrive through a Rust callback. Embed zLLM in a Rust program</h2> The zLLM source is now public on GitHub</a>. Use the Git dependency directly, or clone it and switch to a local path</code> during development. </blockquote> Add zLLM to Cargo.toml</code> with the source path available on your machine: [dependencies] zllm = { git = "https://github.com/zllm-lab/zllm" } serde_json = "1" </code></pre> Load the engine once and call generate</code> repeatedly: use std::io::{self, Write}; use serde_json::json; use zllm::embedded::Engine; fn main() -> Result<(), Box<dyn std::error::Error>> { let mut engine = Engine::from_config("gemma4-metal.yaml")?; let cancellation = engine.cancellation(); let result = engine.generate( &json!({ "model": "gemma4", "messages": [{ "role": "user", "content": "Explain in one sentence why an inference engine belongs in-process" }], "max_completion_tokens": 128 }), &cancellation, |_token, text| { print!("{text}"); io::stdout().flush().is_ok() }, )?; println!("\nfinish={} tokens={}", result.finish_reason, result.completion_tokens); Ok(()) } </code></pre> Return false</code> from the callback or call cancellation.cancel()</code> from another thread to stop generation promptly. GenerationResult</code> also contains token counts, the finish reason, and a cache_id</code> for subsequent turns. Application code never manages Metal command buffers or platform KV objects. Image input uses the same interface with ordered multimodal content parts: let cancellation = engine.cancellation(); let result = engine.generate( &json!({ "model": "gemma4", "messages": [{ "role": "user", "content": [ {"type": "text", "text": "Describe this image"}, {"type": "image_url", "image_url": { "url": "/absolute/path/to/photo.png" }} ] }], "max_completion_tokens": 256 }), &cancellation, |_token, text| { print!("{text}"); true }, )?; </code></pre> Local paths, HTTP(S) URLs, and data:</code> URLs share the same image materialization and vision-encoding path. Gemma 4 E4B's mmproj</code> loads lazily on the first image request, so text-only startup does not pay the vision-tower cost. Get Gemma 4 E4B</h2> This guide uses the instruction-tuned Gemma 4 E4B release: Hugging Face: unsloth/gemma-4-E4B-it-GGUF</a></li> ModelScope: unsloth/gemma-4-E4B-it-GGUF</a></li> </ul> Download these two files into the same directory: models/gemma-4-E4B-it-GGUF/ ├── gemma-4-E4B-it-Q4_K_M.gguf # main model, about 4.98 GB └── mmproj-F16.gguf # vision encoder/projector, about 990 MB </code></pre> With the Hugging Face CLI: hf download unsloth/gemma-4-E4B-it-GGUF \ gemma-4-E4B-it-Q4_K_M.gguf mmproj-F16.gguf \ --local-dir ./models/gemma-4-E4B-it-GGUF </code></pre> The ModelScope page above exposes the same filenames for networks where it is faster. Q4_K_M is a balanced choice for this 24 GB MacBook Air; retaining the roughly 1 GB F16 vision projector favors image-understanding quality. The minimal gemma4-metal.yaml</code> configuration is: version: 1 kind: standalone http: listen: 127.0.0.1:8000 public_base_url: http://127.0.0.1:8000 artifacts: directory: ./artifacts node: cache_directory: ./cache/gemma4 persist_kv_cache: false max_concurrency: 1 model: architecture: gemma4 weights_directory: ./models/gemma-4-E4B-it-GGUF/gemma-4-E4B-it-Q4_K_M.gguf lm_head_quantization: native max_sequence_length: 49152 execution: prefill_chunk_size: 2048 backend: kind: metal device: default </code></pre> An 8 MB Metal command-line program</h2> For immediate use, no configuration file or server is needed. Clone the public repository and build the native CLI: git clone https://github.com/zllm-lab/zllm.git cd zllm cargo build --release --bin zllm-metal </code></pre> Then pass the model directly to the release executable: ./target/release/zllm-metal \ ./models/gemma-4-E4B-it-GGUF/gemma-4-E4B-it-Q4_K_M.gguf </code></pre> zllm-metal</code> automatically: detects Gemma 4 from GGUF metadata;</li> discovers a sibling mmproj-*.gguf</code>;</li> enables MTP speculative decoding when a matching sibling mtp-*.gguf</code> exists;</li> derives a KV budget from unified memory, weight size, and a safety margin;</li> loads the Metal Runtime, enters multi-turn chat, and reuses the terminal KV cache.</li> </ol> Type normally to chat: > Introduce yourself in one sentence and say that you are running locally on this Mac. </code></pre> For a local image, put the question and path in the same message. Bare, quoted, backtick-wrapped, and space-containing paths are supported: > Describe this image in two sentences `/Users/me/Pictures/demo.png` </code></pre> Or copy an image and use: > /paste Describe the image on the clipboard </code></pre> Useful commands are: /stats Show context and KV memory /reset Clear the conversation /compact Summarize older history /paste Read a PNG image from the macOS clipboard /exit Exit </code></pre> Demo on real hardware</h2> The interaction video below uses commands, startup figures, token counts, and response text from a real run on the same MacBook Air M5. The 8.1 MB release executable selected a 49,152-token context; repeated warm-file-cache starts took 0.9–1.6 seconds. Three cold-start text runs averaged 40.47 token/s over tg50</code>, while the measured 323-token image request took 6.469 seconds for prefill and 7.312 seconds to first token. Results vary with model revision, context length, sampling, file cache state, and thermals. Your browser does not support embedded video; download the demo file instead. </video> The 23-second video validates the complete loading, text-chat, and vision-input path. The terminal output is line-wrapped and paced for the frame; it is not a model-quality evaluation. </blockquote> The important boundary</h2> zLLM is not replacing a Python environment with another heavyweight runtime. It makes model execution part of the application: one Rust object, one model, and one native backend. The application owns its threads, cancellation, cache, and lifetime instead of delegating its core capability to another process. That turns “add local AI to an existing Rust program” from a deployment project into an ordinary library call—and gives an executable measured in megabytes complete multimodal inference on a Mac. Why We Are Rewriting an Inference Engine in Rust 2026-08-29T00:00:00+00:00 zLLM is not a Rust wrapper around an existing framework. It is a native inference engine designed around model execution, data lifecycles, and hardware resources, with Rust at its core and only the third-party dependencies that are truly necessary. </blockquote> This article answers three questions: why zLLM needs to own its engine boundaries, why Rust is a natural language for expressing those boundaries, and how that choice appears in real code and complete inference paths. Building another LLM inference engine today invites two obvious questions: why not use a mature engine, and why choose Rust? If the only goal is to get a model running quickly, an existing framework is usually the economical choice. zLLM is solving a broader problem. The same engine must work across Apple UMA, CPU, CUDA, ROCm, Vulkan, NPUs, local SSDs, massive MoE weights, long-context KV caches, and eventually layer-partitioned multi-node execution. These resources differ in location, synchronization, bandwidth, lifetime, and failure boundaries. The hard problem is not wrapping another operator API; it is making the resource relationships of the complete inference process explicit enough to verify, optimize, and evolve. That is why zLLM starts from a clean design. Not because existing projects are inadequate, and not for the sake of being self-built, but because the boundaries we need differ from the historical boundaries of many existing systems. What “rewriting from scratch” means</h2> Starting over does not mean rejecting standards or rebuilding every piece of infrastructure. zLLM continues to read Safetensors, GGUF/GGML, compressed-tensors, and ModelOpt weight formats. It uses Metal, CUDA, ROCm, Vulkan, and other platform capabilities, along with proven general-purpose Rust libraries. We do not reinvent file formats, network protocols, or operating-system interfaces. What we redesign is the inference engine itself: how model specifications are represented;</li> how model algorithms are written once;</li> which capabilities a backend must provide;</li> how kernels stay parameterized instead of binding themselves to a model;</li> who owns weights, KV cache, activations, experts, and scratch memory;</li> how complete prefill, append-prefill, and decode paths are scheduled;</li> how single-node, embedded, and layer-partitioned multi-node execution share one set of semantics;</li> how CPU reference and cross-backend consistency anchor correctness.</li> </ul> In short, zLLM reuses open standards and platform capabilities while owning model execution and resource semantics. Reused</th> Owned by zLLM</th></tr></thead> Safetensors, GGUF/GGML, compressed-tensors, ModelOpt</td> Model specifications, algorithm orchestration, and weight assembly</td></tr> Metal, CUDA, ROCm, Vulkan, and NPU interfaces</td> Backend capabilities, placement, and synchronization</td></tr> Mature serialization, networking, and async Rust libraries</td> KV, expert, activation, and scratch lifecycles</td></tr> Platform-native GPU kernel languages</td> Complete prefill, append-prefill, and decode rounds</td></tr> </tbody></table> This is also the difference between redesigning and merely rewriting. A rewrite can reproduce an old structure in a new language. A redesign returns to data dependencies, resource locations, and real workloads, then asks whether every module deserves to exist. A minimal engine, not a minimal feature set</h2> For zLLM, “minimal” does not mean a demo with limited capability. It means minimizing the number of concepts in the system. Our long-term rule is Simple / Short / Straight: Simple: solve a responsibility in one clear module rather than splitting it into layers of abstraction;</li> Short: keep code, scopes, and object lifetimes as short as possible;</li> Straight: define things near their use, so control flow and dependencies can be followed directly.</li> </ul> Directories, files, traits, wrappers, managers, registries, and contexts are not free. Every additional entity adds a name to remember, another forwarding layer to understand, and another lifetime to trace. An entity is justified only when it owns distinct data, behavior, lifetime, or dependency boundaries. zLLM therefore avoids generic flows without real implementations, speculative abstract factories, and facade shells that only forward calls. There is one model execution flow. Platform differences belong to capability traits and concrete backends. Kernels express operators and do not know model names. The goal is not to minimize capability, but to minimize the mental load required to understand the system. The fewest necessary third-party dependencies</h2> Minimal dependencies do not mean zero dependencies. There is no reason to reimplement mature password hashing, serialization, async runtimes, HTTP, or standard-format parsing for ideological purity. Our principle is simple: a dependency may provide general-purpose capability, but it must not own the core semantics of our inference engine. Model execution, resource residency, KV lifecycles, weight assembly, scheduling policy, and cross-backend boundaries must remain visible in zLLM itself. A third-party library should not invisibly decide where a tensor lives, when it moves, when it synchronizes, when it is reclaimed, or how a model executes. This has three direct benefits. First, the dependency graph is easier to audit. Updating a network library should not change model numerics. Adding a weight format should not invade the runtime. Replacing a platform binding should not require rewriting model algorithms. Second, performance costs stay visible. Inference systems are especially vulnerable to a convenient abstraction hiding an implicit copy, a device synchronization, or temporary dequantization. Owning the critical path lets us trace the origin, destination, and lifetime of every large allocation. Third, long-term evolution is not constrained by an upstream framework. zLLM can reuse standards and general libraries without handing its architecture to a large tensor runtime or Python extension ecosystem. Why Rust fits the job</h2> Rust matters for more than “no GC” or “faster than Python.” For an inference engine, its strongest advantage is the ability to encode resource relationships in program structure and reject many illegal states at compile time. 1. Ownership and lifetimes match inference resources</h3> An inference engine manages expensive resources: memory-mapped weights, device buffers, KV pages, command buffers, asynchronous I/O, distributed sessions, and temporary scratch space. Every one of them raises the same questions: who owns it, who may borrow it, when may it be released, and will it remain valid until an asynchronous task finishes? Rust ownership, borrowing, and lifetimes are not incidental restrictions; they are direct language for these problems. A prefetch task cannot safely reference a buffer that has been released. A cache cannot be reclaimed while an execution queue still uses it. Shared state must declare its synchronization boundary. In C or C++, these rules often rely on convention and review. In garbage-collected languages, delayed reclamation and external resource lifetimes can bypass the collector. Rust turns a meaningful portion of these failures into compilation errors. 2. Predictable execution without a GC</h3> Decode is a continuous, fine-grained loop sensitive to tail latency. Unpredictable pauses, implicit allocation, and delayed destruction make performance analysis harder. Rust has no garbage collector, and value lifetimes normally follow lexical scope. Combined with preallocation and scratch reuse, this makes allocation, release, and synchronization on hot paths more predictable. It does not guarantee performance automatically, but it provides the foundation for building a predictable system. 3. Zero-cost abstractions express backend capabilities</h3> Backend differences are real. A CPU tensor may be ordinary memory, a Metal tensor a device-resource handle, and ROCm may have a completely different placement and submission model. Forcing all of them into a single dynamic object tends to produce a lowest-common-denominator API, indirect calls, and runtime checks across hot paths. zLLM uses traits and associated types to express capabilities. A model runtime declares what it needs; a concrete backend supplies those capabilities at compile time. Generics cost compilation time and binary size, but in return provide static composition, a readable capability set, and less dynamic dispatch in performance-critical paths. 4. Enums and pattern matching make format states explicit</h3> Weights may be F32, F16, or BF16, or they may remain in FP8, FP4, W4A16, or GGUF block encoding until a kernel consumes them. Explicit enums require every backend to process or reject each legal state deliberately instead of guessing across strings, pointers, and implicit conventions. This is particularly important for quantization. Quantization is not a loading-time detail that always expands to floating point. It is a data path orthogonal to model specifications and platform kernels. The more explicit the state, the harder it is for accidental expansion or incorrect dispatch to enter the system. 5. Rust spans the systems layer and the service layer</h3> Many inference stacks are naturally split: C/C++ or GPU languages at the bottom, Python, Go, or another service framework on top. Each side can be reasonable alone, but together they create duplicated data models, cross-language FFI, separate error systems, and lifetimes that are hard to unify. Rust can handle weight parsing, CPU reference implementations, backend bindings, runtimes, schedulers, HTTP/SSE services, and embedded library APIs. GPU kernels still use platform-native languages—Metal Shading Language, HIP/CUDA, WGSL, or AscendC—but the engine around them shares one type system, error model, and resource semantics. That is what zLLM means by “pure Rust.” It does not deny the existence of native platform kernels. It means there is no Python control plane and no delegation of core execution to another large tensor runtime: Rust directly owns the inference engine. 6. One toolchain keeps cross-platform engineering consistent</h3> Cargo build, features, tests, and dependency management let CPU, Metal, CUDA, ROCm, and Vulkan paths maintain clear boundaries in one project. Platform capabilities are enabled by feature and target. CPU unit tests require no real weights, while GPU paths use the CPU oracle as their numerical anchor. Rust cannot replace real-device testing or catch an ABI mismatch inside runtime-compiled MSL or HIP at compile time. It can, however, confine platform-specific unsafe boundaries to a relatively small region while the rest of the system retains static checking. How zLLM’s architecture reflects these choices</h2> zLLM divides the system into regions with distinct responsibilities and one-way dependencies: The directory names are less important than the direction of dependency: model specifications do not depend on platforms;</li> model algorithms are written once rather than copied into every backend;</li> backends and kernels do not know concrete model names;</li> domains such as attention, MoE, and KV cache keep device-independent semantics and reference implementations;</li> the weight layer owns standard formats, naming, shape validation, and lifetimes—not model algorithms;</li> CPU is the correctness oracle, while GPU kernels are accelerated implementations verified against it.</li> </ul> This avoids the most common form of combinatorial explosion. Adding a model does not mean copying its execution flow for every platform. Adding a backend does not mean relearning every model. A new model mainly adds specifications, orchestration, and weight adaptation. A new backend mainly implements existing capabilities. Domain interfaces expand only when a genuinely new operator semantic appears. Start from data flow, not semantic labels</h2> A clean-sheet design lets zLLM organize the system around real data dependencies instead of paper sections or model terminology. If A produces the input consumed by B, they form a serial pipeline and should stay close while sharing intermediate state. If A and B independently read the same hidden state, they should remain separate so a backend can schedule them concurrently. Weight structures, kernel fusion, and resource lifetimes follow this same rule. Production inference recognizes only three complete tasks: prefill for a new session, append-prefill for an existing session, and a complete decode round. Stopping at one layer, running one kernel, or comparing a partial output can be useful diagnostics, but cannot replace an end-to-end workload. Optimization must ultimately return to TTFT, decode latency, throughput, peak memory, SSD stalls, and device utilization. This keeps the system from being distracted by attractive but irrelevant local numbers. Rust does not automatically deliver correctness or performance</h2> Choosing Rust does not mean the language replaces engineering discipline. Rust cannot prove a GPU kernel numerically correct, prevent a bad algorithm, or guarantee that an elegant abstraction has no performance cost. Multiple backends mean one kernel semantic may need optimized Rust reference, MSL, HIP, CUDA, WGSL, or AscendC implementations. Generics increase compilation cost. Capability traits can still grow without discipline. zLLM therefore combines the language with explicit rules: every new operator starts with reference semantics and unit tests;</li> CPU and device backends are checked for consistency;</li> kernel dimensions are parameterized rather than embedding model constants;</li> isolated kernel benchmarks never substitute for complete-path metrics;</li> complexity is not added for performance without measurement;</li> source presence, successful compilation, oracle validation, and real-device end-to-end validation remain distinct states;</li> every change is the smallest one required for the current task.</li> </ul> Rust provides solid ground. Engine quality still depends on clear boundaries, reproducible validation, and sustained restraint toward complexity. Why starting over is worth it</h2> Rewriting an engine is expensive. Weight formats must be understood directly, kernels optimized platform by platform, services and schedulers refined in-house, and no upstream framework absorbs the mistakes. In the short term, this path is unquestionably slower than integrating a mature runtime. What it buys is long-term ownership of every critical decision. When a model changes, we know whether to modify its specification, orchestration, or domain capabilities. When a platform changes, we know whether the backend or kernel should change. When performance regresses, we can follow weights, activations, KV, experts, scratch, and synchronization points one by one. When the system expands across machines, it sends layer-boundary activations rather than mixing model-internal state with remote expert RPC on the hot path. That is the purpose of zLLM: not to become the largest general AI framework with the most abstractions, but to remain a small, direct, native inference engine that truly owns hardware resources and model execution. We chose Rust because it matches that goal: safe without surrendering low-level control; abstract without mandatory runtime cost; cross-platform without erasing platform differences; capable of writing the code nearest to a kernel and continuing all the way to services and distributed scheduling. We chose a clean-sheet design because only then can Simple, Short, and Straight shape the entire engine from its first line instead of becoming local patches on an inherited system.

From 1M Total to 15+ × 1M Contexts: GLM-5.3 KV Cache Offloading

A Vintage Machine Can Run a Top-Tier LLM

How Good Is K2-Horizon from Abu Dhabi? On an M5 It Solves the Problems, Then Stumbles over Chinese Prose

Running GLM-5.3 on Two 8-GPU AMD W7900D Machines

The Algorithm Layer: From Attention and KV Cache to FFN and MoE

Quantization Is Not a Zip File: Model Layers and the Reality of 4-bit Inference

Decoupling: Let Models, Algorithms, Backends, and Kernels Evolve Independently

Why Did One Sentence Consume Hundreds of Thousands of Tokens?

MacBook Air M5: Run a Multimodal LLM with Zero Runtime Dependencies

Why We Are Rewriting an Inference Engine in Rust

Why Rust fits the job</h2>
Rust matters for more than “no GC” or “faster than Python.” For an inference engine, its strongest advantage is the ability to encode resource relationships in program structure and reject many illegal states at compile time.</p>

zLLM · Building the Best Inference Engine

DeepSeek V4.1 Flash Engineering, Part 4: Long Prefill and the Final Optimization Mile

DeepSeek V4.1 Flash Engineering, Part 3: Integrating Vision—Finding Differences with Intermediate Tensors

DeepSeek V4.1 Flash Engineering, Part 2: From Weight Loading to an Eight-GPU Text Pipeline

Step 1: Assemble file tensors into usable model weights</h2>
Weight loading starts with three questions: what is the tensor called, what is its logical shape, and how should its bytes be interpreted?</p>

Engram in DeepSeek V4.1 Flash: Knowledge Without Recomputing It Every Time

The Essential App for Time Travelers: Modern Civilization in One Phone

zLLM · Building the Best Inference Engine

DeepSeek V4.1 Flash Engineering, Part 4: Long Prefill and the Final Optimization Mile

DeepSeek V4.1 Flash Engineering, Part 3: Integrating Vision—Finding Differences with Intermediate Tensors

DeepSeek V4.1 Flash Engineering, Part 2: From Weight Loading to an Eight-GPU Text Pipeline

Step 1: Assemble file tensors into usable model weights</h2> Weight loading starts with three questions: what is the tensor called, what is its logical shape, and how should its bytes be interpreted?</p>

Engram in DeepSeek V4.1 Flash: Knowledge Without Recomputing It Every Time

The Essential App for Time Travelers: Modern Civilization in One Phone

From 1M Total to 15+ × 1M Contexts: GLM-5.3 KV Cache Offloading

A Vintage Machine Can Run a Top-Tier LLM

How Good Is K2-Horizon from Abu Dhabi? On an M5 It Solves the Problems, Then Stumbles over Chinese Prose

Running GLM-5.3 on Two 8-GPU AMD W7900D Machines

The Algorithm Layer: From Attention and KV Cache to FFN and MoE

Quantization Is Not a Zip File: Model Layers and the Reality of 4-bit Inference

Step 1: Assemble file tensors into usable model weights</h2>
Weight loading starts with three questions: what is the tensor called, what is its logical shape, and how should its bytes be interpreted?</p>