From 1M Total to 15+ × 1M Contexts: GLM-5.3 KV Cache Offloading
Using 1 TiB of RAM per node to target 15+ times the active context capacity, with almost no decode loss measured at 50K. The decisions, failed approaches, and evidence behind the result.
The biggest benefit of this optimization is capacity: moving from roughly 1M tokens of total context toward 15+ concurrent decoding sessions, each carrying a 1M-token history. The same GPUs can support a projected 15+ times as much active history.
Previously, one million-token conversation could consume almost the entire system's context budget. Now, each machine's 1 TiB of host RAM holds MLA history, while the GPUs keep a 32K hot cache for current computation. The aim is to let multiple users keep generating against their own million-token histories, beyond simply archiving completed sessions.
15+ × 1M is a capacity estimate, not a completed concurrency benchmark. We show the memory accounting below. The nearly lossless speed result comes from completed 50K comparisons. These answer two separate questions: how much history the system can hold, and how much speed offloading costs.
Our measured result has a specific scope: with 50,023 input tokens, 1,024 output tokens, and a 32K GPU hot cache, storing MLA history in host RAM reduced throughput by only 0.15%–1.30% across three MTP=3 pairs. Without MTP, three pairs were about 2% faster. Every complete output matched its corresponding baseline.
Getting there took several reversals. We first tried moving indexing to the CPU, then backed away. The hot-cache path once lost roughly 30% of throughput. Several convincing local optimizations failed to improve the full model. The remaining instability eventually led us from attention kernels into Linux memory management and the GPU driver.
This article follows those decisions: why we tried each path, which evidence justified keeping it, and what forced a change of direction.
Separate historical capacity from the current working set
We used the two-node system from our previous GLM-5.3 optimization article: eight AMD Radeon PRO W7900D GPUs per machine, 48 GiB per card, an EPYC 9334 platform, and 1 TiB of RAM per node. Intra-node communication uses HIP P2P; TCP carries activations across node boundaries. There is no XGMI.
This work changes KV placement. Weights and layer computation stay on their assigned devices, and historical data stays in the host memory of the node responsible for those layers.
The opportunity comes from GLM-5.3's sparse attention path. The DSA indexer selects the top 2,048 historical positions; MLA attention consumes their KV. Some subsequent layers reuse the selection, while retaining their own KV contents.
The amount of history that must be preserved can therefore be managed separately from the amount consumed by one attention operation.
In this Q8G64 MLA representation, each row contains 512 B of latent data, 16 B of scales, and 128 B of RoPE data: 656 B in total. The engineering records give the following payload sizes. These cover MLA only, excluding DSA, MTP layers, metadata, alignment, and scratch space.
| Object | MLA payload |
|---|---|
| 50,000 tokens, one layer, one replica | 31.281 MiB |
| 50,000 tokens, 78 main-model layers, both replicas | 4.765 GiB |
| 2,048 selected rows, one layer, one replica | 1.28125 MiB |
| 32,768 hot-cache rows, one layer, one replica | 20.5 MiB |
One 50K history is manageable, but its cost accumulates with conversation length and session count. We wanted the main GPU MLA history allocation to be bounded by the hot-cache size, with growth absorbed by RAM.
This hot cache is not a sliding window that discards everything older than 32K tokens. Old positions remain in RAM, DSA can still select them, and a cache miss reads them back. Physical placement changes without deliberately truncating accessible history.
Decision one: should the CPU store history and run the indexer?
The first proposal was natural: the host has abundant memory and CPU cores, so let it score the full history and select Top-K, then send the selected KV to the GPU.
There was also a scheduling opportunity. Once selection was known, layers reusing it could prefetch historical KV while independent GPU work continued.
We already had CPU Q8 scoring, blocked layouts, and vectorized implementations. We added a fixed worker team, local histograms, reusable Top-K workspaces, incremental mirrors, rollback, and prefetching. This deserved a test rather than dismissal based solely on CPU versus GPU arithmetic speed.
Hardware testing produced two independent reasons to reject it for this iteration.
First, performance: the 50K GPU baseline delivered about 14.03 tokens/s, while CPU DSA candidates reached only 10.92–11.32 tokens/s. Query readback, thread coordination, selection upload, and GPU waiting joined CPU arithmetic on the critical path. Older CPU measurements at 131K could not simply be multiplied by 50/131, because the number of active workers changed with context length too.
Second, correctness: final selection using CPU Q8 queries differed around the cutoff from the GPU BF16-query exact path. Fixed greedy generation diverged. Internal CPU consistency did not establish equivalence to the original GPU path.
Decision: keep exact DSA on the GPU, disable CPU selection, and continue moving MLA history into RAM.
This narrowed the problem. Storage expansion did not need to wait for CPU indexing to succeed. Prefetching remained a useful overlap opportunity, but could not be credited as zero overhead in advance.
Decision two: let the CPU manage the GPU hot cache first
With selection still computed on the GPU, the early hot-cache path sent it back to the host. The CPU looked up mappings, checked hits, packed missing rows, and submitted uploads and attention.
It established the capacity direction, but performance was poor. During the September 7 experiments, the real 32K hot-cache configuration, after packed replication and batched-mirror fixes, still had a three-run median of about 10.41 tokens/s. That phase reported roughly a 29% loss against its reference. Those early results included different diagnostic conditions and must not be combined with the final tests as one strict A/B comparison.
Two concrete problems appeared first.
The offload branch had lost the existing compressed replication path. Previously, the owner quantized once and copied 656 B of packed KV per row to its peer. In hot-cache mode, fallback appended and requantized on both sides, also losing existing channel overlap. We restored packed replication to the hot-cache path.
The number of small transfers mattered. Updating host mirrors per layer and token could create 156 stream operations across 78 layers and their owner/peer replicas. Splitting each row into three transfers and increasing outstanding work raised that to 468 operations. Throughput fell to roughly 8.8 tokens/s.
That experiment rejected the assumption that more asynchronous concurrency would solve transfer overhead. The payload was already small; additional submissions could cost more than additional bytes.
Copying back in batches of 64 rows sharply reduced operation count and improved the full-cache phase. A substantial gap remained once the cache actually entered hot mode.
We then moved peer-mirror maintenance into the worker's locked section and made the append kernel also write a contiguous log. The three-run medians were about 10.39 and 10.06 tokens/s, respectively. Neither delivered the expected improvement.
Decision: retain batched logs and necessary lifecycle fixes, but withdraw the claim that moving locks and adding log writes would close the main gap. Completing an implementation and validating its performance hypothesis are separate milestones.
A mistaken diagnosis: the largest share does not identify the regression
We added synchronized tracing around hot-cache attention. Kernels accounted for 77.4% of measured time, and host locking plus remapping for about 22%. We initially concluded that the remaining problem was primarily the window-attention kernel.
Reviewing the evidence invalidated that method of attribution.
The profile described the candidate's absolute time breakdown. It did not establish where the difference between GPU residency and offloading arose. The resident path also runs attention. Moreover, synchronization and tracing reduced full-run throughput to about 4.21 tokens/s, substantially changing submission and overlap behavior.
Source inspection also found that selected-gather functionality, proposed as a next step, already existed. We needed to verify the actual path and its dependencies rather than treat an existing mechanism as missing work.
We changed the measurement procedure: diagnostics locate problems; formal throughput runs disable profiling, tracing, and extra synchronization. Each candidate gets a same-version GPU baseline, and complete outputs must match before comparing speed. The quantity to explain is the difference between the two paths.
Decision three: let the GPU manage the hot cache directly
New evidence brought us back to the submission chain. The GPU had already computed selection, yet the CPU waited for readback and performed mapping and packing before attention could proceed. Host-observed selection readback waits reached about 0.7–1.2 ms. They included earlier queued GPU work, so they were not pure copy times, but they exposed the control round trip.
We moved cache management. Selection stays on the GPU, which directly checks hot slots. Hits read VRAM; misses read the registered RAM mirror. Gather produces a compact selection for attention.
GPU exact DSA ──→ selected historical positions
│
▼
GPU hot gather
├─ hit: read VRAM hot slot
Local RAM history ─────└─ miss: read registered host memory
│
▼
compact KV → GPU attention
New KV → GPU recent ring → batched copyback → RAM history
Host memory supplies historical capacity while indexing, cache lookup, and attention continue on the GPU. The CPU still manages resources and lifetimes, but no longer coordinates lookup and transfer for every selection.
PCIe reads still have a cost. Some early sampling windows had less than 1% misses, showing reuse in this workload; that does not promise the same hit rate for every input.
The first GPU-managed version reached about 11.72 tokens/s, with an initial registration stall. Moving registration earlier and completing multirow handling even produced a run at 5.13 tokens/s. We kept investigating the direction without calling near-baseline steady-state event spacing a passing full-run result.
GPU cache management has hazards too
MTP exposed problems that were less visible on the single-row path. Verification processes multiple rows, and several queries can select the same missing position.
An early implementation allowed one block within a gather invocation to read a slot another block had just filled. A real MTP test produced abnormal scales, followed by a failure to find a finite selectable token.
We tightened visibility: slots filled in the current round become hit sources only in the next round; duplicate misses within the same round read RAM directly. Slots used by the current round are pinned, and a recent ring protects new tokens that have not yet reached RAM.
The complete 1,024-token MTP output then matched the same-version resident baseline, but throughput was 16.175 versus 27.530 tokens/s, a loss of about 41.2%. Both accepted 665 draft tokens, ruling out acceptance-rate differences as the explanation.
Restoring owner-to-peer packed KV replication for small verification batches raised the candidate to 20.035 tokens/s. This was a full execution-path repair worth keeping.
Replacement policy brought another surprise. With clock replacement and reference bits, a synthetic case with a full hot cache, all reference bits set, and only 1% misses took about 1.03–1.07 ms per gather. A few miss blocks had to scan long stretches. More misses could actually run faster because more blocks participated in scanning.
Replacing it with circular eviction that skips currently pinned slots reduced the same probe to about 0.032 ms for 2,048 rows. Yet full MTP throughput was only 19.767 tokens/s. The probe's large gain did not materialize in the complete model.
Decision: keep the simpler replacement policy and remove the demonstrated pathological case, but reject the inference that this local hotspot explained the full performance gap.
Decision four: inspect the submission threads beyond attention
CPU sampling restricted to active decode threads found about 21.1% of CPU cycles in host memcpy. The call chain ended in clone on read-only MoE weights: submitting work to a worker could deep-copy the dense router.
This was unrelated to KV semantics, but affected the same execution chain. We shared the read-only weight group through Arc, preserving operators and numerical values.
The resident baseline received the same fix. The v15 MTP comparison reached 25.425 versus 26.832 tokens/s, narrowing the loss to 5.24%. That was substantial progress, but still just outside the 5% gate. The 21.1% CPU sampling share could not be translated into an equal end-to-end speed gain.
Further work reused the recent ring for peer logging, removed duplicate copies, transferred existing prefill buffers into hot-cache ownership, and used compute copies and packing for small logs to reduce DMA queue operations.
These changes did not solve everything at once. One no-MTP pair reached a 4.80% loss; a later MTP pair lost 9.87%. Logs kept showing 0.6–1.2 second stalls, even when most event intervals were near the resident baseline.
We did not discard the first 128 tokens or declare victory using p50 alone. Those stalls were part of the user's wait and remained in the original complete-decode timing window.
Decision five: follow the stalls into OS page migration
Device timelines showed that the long stalls did not necessarily occur in hot gather. One capture showed maximum gather times of only about 0.054–0.056 ms, alongside a roughly 792 ms synchronization wait and almost no device work across the pipeline.
Blaming whichever kernel appeared stretched in the profiler would repeat our earlier error. We began comparing system settings and driver behavior across the two machines.
Disabling automatic NUMA balancing helped, but was insufficient
Amd-1 had kernel.numa_balancing=0; Amd-2 had it set to 1. Long stalls concentrated on the latter. Disabling it consistently produced a first MTP pair with only 0.51% loss.
But the third interleaved pair lost 5.49%, stopping the gate. We retained consistent settings and continued investigating, instead of averaging away the failed pair.
Pre-touching registered tail pages did not solve the problem
Another hypothesis was that unused tail pages in the RAM mirror's reserved capacity had not yet been written, and first-touch writes during append disrupted GPU mappings. We zeroed spare capacity before registration without changing valid KV bytes or logical length.
The candidate still showed 7.35% loss and second-scale gaps. Pre-touching did not resolve the instability and could not be presented as the final cause or solution.
Proactive compaction finally yielded call-chain evidence
BPF tracing tied the actual inference process to invalidation addresses and restore workers, capturing this path:
kcompactd
→ proactive_compact_node
→ migrate_pages
→ try_to_migrate_one
→ amdgpu_hmm_invalidate_hsa
→ related queue restoration work
This established a path through proactive page compaction, HMM invalidation, and queue restoration. Many sampled invalidation addresses did not intersect the explicitly registered KV mirrors, so they could not be explained as first writes to KV tail pages. This capture did not reproduce the earlier full one-second stall; we did not claim to have attributed every long pause individually.
We changed vm.compaction_proactiveness from 20 to 0 on both machines, with automatic NUMA balancing still disabled. Linux documents that zero disables proactive compaction and that page migration can cause application latency spikes. The system-specific conclusion still required complete paired runs under identical settings. Linux kernel documentation
These were runtime changes on the experimental machines, with original values and restoration commands recorded. Boot configuration was unchanged. These settings are part of the test conditions, not a universal tuning prescription for every GPU system.
Six interleaved pairs establish the nearly lossless result
The final gate used the same v23 binary, 50,023 input tokens, 1,024 output tokens, greedy decoding, and fixed seed01. The offloaded configuration used a 32,768-row hot cache; the baseline kept KV on the GPU. Both disabled automatic NUMA balancing, proactive compaction, profiling, tracing, attach support, and CPU DSA.
We restarted the experimental runtime for each run and interleaved resident and offloaded ordering. Throughput retained the original first-text-event-to-SSE-DONE window, with no stalls removed.
| Mode | Pair | RAM history + GPU hot cache (tokens/s) | GPU resident (tokens/s) | Offload throughput loss |
|---|---|---|---|---|
| MTP=3 | 1 | 27.839 | 27.883 | 0.157% |
| MTP=3 | 2 | 27.923 | 27.966 | 0.150% |
| MTP=3 | 3 | 27.738 | 28.102 | 1.297% |
| No MTP | 1 | 14.042 | 13.788 | −1.846% |
| No MTP | 2 | 14.029 | 13.753 | −2.007% |
| No MTP | 3 | 14.029 | 13.747 | −2.056% |
All 12 requests completed, and full output-file hashes matched the reference for their respective modes. Negative loss means offloading was slightly faster. We interpret that as essentially matching the baseline on this workload, not a guarantee that offloading accelerates inference.
This validates single-session decoding with long history. Initial prefill, appended inputs, multi-session capacity, and concurrent throughput each need their own tests.
Capacity optimization must survive the session lifecycle
One uninterrupted generation is not enough for real conversations. Users append inputs, MTP rejects drafts and rolls back, and sessions are saved and restored.
The successful path has an explicit consistency boundary. New KV may still be in the GPU recent ring or an in-flight copyback. Before exporting or restoring full history, submit outstanding logs, wait for copyback, and verify that CPU row count has caught up with logical row count. An enqueued asynchronous copy is not yet a completed RAM copy.
Large append prefill also differs from decode. To process many new rows, the current implementation restores the required full-history GPU MLA KV from RAM, performs the append, and returns to hot-cache mode for decode. Small prefill tail chunks must not be mistaken for verification, and owner and peer must remain consistent.
The first real large-append test failed on MTP terminal hidden state. Request completion had saved a raw BF16 residual where the later path expected F32 hidden state after final normalization. The fix aligned normalization boundaries and addressed old-cache restoration.
The subsequent v25 same-version comparison reused 50,085 historical rows, appended 9,854 tokens, reached a final prompt of 59,939 tokens, and generated 1,024 tokens. Complete outputs matched.
| Large append comparison | Offloaded | GPU resident |
|---|---|---|
| Time to first token | 17.905 s | 18.350 s |
| Decode | 26.850 tokens/s | 27.004 tokens/s |
Decode loss was 0.57% in this pair. It established that an offloaded long conversation could continue with a large append, but remained a single append test rather than a replacement for the six single-session pairs.
Ten sessions exposed a physical-allocation problem
While preparing ten independent 50K histories, the sixth prefill ran out of memory on the tail node. The highest 1 Hz VRAM sample was about 47.98 GiB. Ten-session simultaneous decoding had not yet begun.
The problem was the general GPU memory pool. Small long-lived KV, hot-cache, and session buffers could receive large allocations returned by prefill through best-fit reuse, then hold them indefinitely. A logical allocation of tens of KiB or MiB could retain much more physical space. Accounting reported logical bytes, further understating usage.
We enforced exact capacity limits for long-lived caches and changed primary KV/DSA accounting to actual allocation size, while retaining the existing pool policy for temporary activations. Tests first populated the pool with oversized blocks, then checked that small hot-cache buffers did not retain them.
After the fix, v28 completed all ten histories and subsequent concurrent appends, delivering 98.56 tokens/s over the full request group. The highest observed VRAM sample was about 45.12 GiB.
Concurrency tuning followed. Reducing MTP depth to 1 raised full-request throughput to 125.26 tokens/s. Reducing each stage's decode batch limit from 4 to 1 reached 145.67 tokens/s, with matching inputs, complete outputs, and usage between those latter two configurations. Multiple requests could keep the pipeline moving; deeper speculation and larger batches that help one session need not help ten.
Separately increasing the A0 ready-work batch limit produced 145.28 tokens/s, essentially unchanged. With no demonstrated benefit, we removed that change.
These concurrent results use actual completion-token totals divided by wall time for the entire append request group, including request boundaries. SSE text-event counts were not treated as token counts. There was no corresponding fully GPU-resident concurrency comparison, so these runs do not establish lossless concurrent offloading. The latest concurrency candidate still needed single-session regression gates.
The capacity gain: from roughly 1M total toward 15+ sessions of 1M each
Consider the user-facing change. With an original total budget of about 1M tokens, one user carrying a million-token history could effectively occupy it. After offloading, the target is 15+ such users decoding concurrently, each retaining their own million-token history.
| Dimension | Original capacity basis | Estimated offloaded capacity target |
|---|---|---|
| Total history across active sessions | About 1M tokens | 15M+ tokens |
| Sessions carrying 1M history each | About 1 | 15+ decoding concurrently |
| Main storage for full MLA history | GPU VRAM | 1 TiB host RAM per node |
| MLA history payload on GPUs | Grows with history length | 32K hot cache per session; old positions read on demand |
This is the practical meaning of opening up capacity: targeting 15+ times as much active long history on the same GPUs. It expands multi-session historical capacity. Per-session speed at that concurrency still depends on compute, indexing, transfers, and scheduling.
Does RAM fit fifteen million-token histories?
Using the same main-model MLA encoding, 1M = 1,048,576 tokens, 78 layers, and both owner/peer replicas:
One 1M history = 1,048,576 × 656 B × 78 × 2
≈ 99.94 GiB
15 sessions ≈ 1,499.06 GiB (about 1.464 TiB) of main-model MLA history
Total host RAM across both machines = 2 TiB
The budget is 2 TiB across two machines. All history does not reside on one 1 TiB host. MLA history is distributed according to layer ownership. The owner/peer replicas are already included above; they do not mean each host stores the entire model's history.
For 15 sessions, the main-model MLA history totals about 1.464 TiB; roughly equal distribution would put about 0.732 TiB on each node. Actual placement follows each node's layer count and requires checking headroom separately. The main-model MLA payload for 15 sessions does not exhaust the two-node RAM budget, leaving room to explore 15+ sessions. This is an encoding-based capacity calculation, not a measured whole-system memory peak or maximum concurrency.
Meanwhile, fifteen 32K hot caches require about 46.85 GiB of main-model payload across all GPUs, including both replicas. That component does not grow proportionally when each history expands from 50K to 1M. This is the central capacity advantage.
The complete service budget must also cover GPU DSA history, token mappings, MTP layers, per-session workspaces, allocation and registration overhead, and transient VRAM during full-history restoration for prefill or large append. Fitting the MLA bytes in RAM provides a basis for the 15+ session target, but does not constitute whole-system validation at that scale.
Completed measurements cover nearly lossless 50K single-session decoding, a large append reaching about 60K, and complete concurrent requests over ten 50K histories. The next meaningful capacity test is to extend each history to 1M and active concurrency to 15+, recording real memory usage, time to first token, and sustained decode throughput. The current 1M admission configuration must also be aligned with that larger aggregate target.
The decisions that survived
| Approach or claim | Outcome | Decisive evidence |
|---|---|---|
| Final DSA selection on CPU | Rejected for this iteration | Lower speed and divergent fixed greedy output |
| MLA history in RAM with GPU hot working sets | Retained | Six passing single-session pairs with matching complete outputs |
| CPU coordinates hot slots for every selection | Main path moved to GPU management | Readback waiting interrupted submission |
| More per-row asynchronous copies | Rejected | More operations reduced throughput |
| Batched logs and packed owner/peer replication | Retained | Reduced redundant work and repaired single-row/MTP paths |
| Moving locks or dual-writing logs closes the gap | Rejected | No substantial complete-run improvement |
| High kernel share proves the regression is in kernels | Withdrawn | Absolute shares did not explain the resident/offload difference |
| Simpler GPU replacement | Retained with limited claims | Large probe improvement without matching full-model gain |
| Share read-only weights instead of deep-copying | Retained | Same-version full-run gap narrowed |
| Disabling automatic NUMA balancing is sufficient | Sufficiency rejected | Repeated gate still failed |
| Pre-touching registered tail pages fixes the stalls | Rejected | Second-scale gaps and a 7.35% failed pair remained |
| Consistent NUMA settings and disabled proactive compaction | Retained for these machines | Kernel-path evidence and six interleaved pairs |
| Small logical buffers imply small VRAM occupancy | Rejected | Preparation OOM and oversized physical allocations |
| Increase A0 batching under concurrency | Removed in this iteration | Full-request throughput was essentially unchanged |
The reusable lesson is to manage historical capacity, current working sets, and control dependencies separately. RAM provides capacity; GPU hot caches preserve local access; the GPU consumes selection directly; batched logs maintain new history. Host and driver memory behavior also belong in end-to-end measurement.
Every change of direction used the same standard: did complete requests improve, did outputs match, and could failures be reproduced? That standard turned spare host RAM into usable long-history capacity, with a path from roughly 1M total toward 15+ simultaneous million-token sessions.
Sources and measurement scope: based on zLLM engineering records in docs/glm53-cpu-kv-decode.md (September 5–8, 2026), the two-node baseline in docs/glm53-rocm-amd12-baseline-20260906.md, and inspection of the local GPU hot-cache implementation. Formal single-session results are the six cp0-v23 pairs; append results are from append-v25; concurrency results are from concurrency-v28 and concurrency-tuning. Historical exploration, diagnostic probes, formal pairs, and later concurrency candidates are identified separately and must not be merged into a single strict cross-version performance curve. No remote benchmarks were rerun for this article.