DeepSeek V4.1 Flash Engineering, Part 4: Long Prefill and the Final Optimization Mile

Part 3 aligned image preprocessing, the vision tower, the aligner, and dual-bias routing through intermediate tensors. At that point, DeepSeek V4.1 Flash could run its text path, Engram, eight-GPU language backbone, and multimodal path.

The final article asks one remaining question: when an input grows from a few thousand tokens to 261K and then one million, how can the complete request keep making progress while cold-prefill time actually falls?

This was not a matter of replacing one matrix kernel. Long context amplifies sparse indexing, temporary tensors, VRAM fragmentation, stage handoffs, and chunk scheduling at the same time. A microbenchmark can improve by several times without moving the complete model. A configuration can touch 1,600 token/s yet leave only a few dozen MiB of VRAM, making it unsuitable as the retained deployment path.

We therefore accepted three kinds of evidence: operator comparison against the official reference, complete-output gates on fixed inputs, and end-to-end timing with profiling disabled.

Restore the official cross-layer sparse semantics first

The first long-prefill step was a correctness fix.

DeepSeek V4.1 Flash DSA does more than select Top-K positions from all history independently at every layer. Layer 20 first aggregates index scores into blocks and chooses a candidate region. Layers 24, 28, 32, and 36 then apply their own scores within those candidates. Sparsity therefore extends across depth: one source layer publishes a bounded search region and later consumers reuse it.

An early local configuration disabled this chain and let later layers select directly from full history. When the context was short, candidate capacity covered all visible history and the paths looked equivalent. Beyond roughly 16K tokens, candidates began to exclude positions and the difference became real.

zLLM restored three pieces of state:

1. Candidate blocks produced by the source layer remain alive across stages and travel with the hidden state.
2. Consumer layers score only candidate slots and map local selections back to global positions.
3. The newest visible block, causal prefixes, and partial tail blocks follow the official boundaries.

Real-shape tests extended to 1,048,576 historical positions. Candidate blocks and both Top-512 stages matched the official PyTorch result. The fix also reshaped the performance problem: the four downstream consumers are capped at 16,384 scored positions, while the earlier source index layers still traverse history that grows with context.

The latest prefill path: stop decoding the same index data repeatedly

After candidate semantics were fixed, profiling pointed to front-end FP4 index scoring. The straightforward implementation repeatedly decoded the same query and key values inside every score tile. The cost became increasingly visible with longer history.

The final path has four parts.

1. A provably lossless integer representation

When the official FP4 mantissas and E8M0 scales satisfy a checked exponent range, they can be aligned into small integers. INT8 inputs and an INT32 dot product then produce the exact integer sum. The implementation restores the power-of-two scale and preserves the original head order for ReLU, weighting, and reduction.

This is not approximate quantization. Each tile is checked at runtime. If any proof condition fails, the tile falls back to the original FP32 accumulation path. Tests cover different head counts, dimensions, tails, causal prefixes, and million-position history; GPU scores and Top-K results remain bit-identical.

2. Expand each query and key once

An integer dot product alone is insufficient if every tile still parses FP4 codes and scales repeatedly. The new path expands the query batch and historical keys once, after which scoring reads prepared data directly.

In isolated same-shape measurements, 128 queries over 131,072 historical positions fell from about 47.6 ms to 6.55 ms. Thirty-two queries over 524,288 positions fell from about 47.6 ms to 6.62 ms. These are index-score microbenchmarks, not complete-model speedups, but they clearly identify repeated decode as removable work.

3. Dynamic LDS follows the actual head count

An early kernel reserved shared memory for as many as 64 heads even though the front index used 32. Dynamic LDS computes shared-memory use from the actual head count, allowing more workgroups to remain active on a CU.

On the same 261,933-token cold input, this reduced TTFT from 357.10 to 306.66 seconds and raised input processing from 733.51 to 854.15 token/s. The complete 1,024-token output remained identical to the control.

4. Recompute only the heads that fail the integer conditions

If one head in a query falls outside the exponent proof, falling back an entire tile to scalar FP32 discards most of the gain. The final implementation builds a warp mask for exceptional heads and recomputes only those dot products in the original order. It uses full-tile fallback only when the exceptional share is too large.

After one-time expansion, candidate-score reuse, and local fallback, the same 261K request reached 1,188.42 token/s at chunk 256 while preserving the complete output hash.

Dense and MoE: keep only improvements visible in the complete model

Once indexing improved, the bottleneck moved into routed MoE, attention projections, and dense temporaries.

The dense path gained direct fragment-load branches for 16, 64, and 128 rows while preserving the original accumulation order for each 16-element K segment. It raised the 261K cold-prefill result from 1,023.58 to 1,075.85 token/s, about 5.1%.

For MoE, we tried larger K tiles, input conversion, merged LDS, route-block changes, and early weight reads. Most changes did not survive complete-model testing or helped only very small route groups. The retained path preserves matrix accumulation order and passes CPU/GPU oracles. We do not present a local 3%–5% microbenchmark gain as an end-to-end result.

This became a recurring rule of the final optimization cycle: hotspot order changes whenever the previous bottleneck is removed. The top item in an old profile may no longer be the best target in the current build.

A larger chunk is useful only while its working set fits

After operators and allocation became stable, we retested prefill chunks on the same 261,933-token input:

Chunk	TTFT	Cold-prefill rate	Minimum sampled VRAM headroom
256	220.40 s	1,188.42 token/s	about 5.05 GiB
512	177.29 s	1,477.44 token/s	about 5.00 GiB
1,024	173.45 s	1,510.12 token/s	about 2.63 GiB
2,048	170.67 s	1,534.76 token/s	about 0.26 GiB

A larger chunk lets one weight load serve more tokens and reduces pipeline fill and drain cycles. It also enlarges activations, routed output, RoPE, and index workspaces. Reducing the work window from 32 to 16 changed throughput by only about 0.31% and did not restore VRAM headroom, so it was not retained as an optimization.

Different chunk sizes enter different BF16 row-count dispatches, and long generated text diverged across batch shapes. Runs with the same chunk, build, and fixed input reproduce; operator oracles also remain aligned. We still do not claim that the cross-batch-shape full-text difference is resolved. These numbers describe the current engineering path and do not replace a full model-quality evaluation.

The final arena step: split returned large blocks again

Long prefill continuously changes temporary-tensor sizes. The old L1 reuse layer could use a large completed buffer for a small request while lending out the entire block. The unused remainder could not immediately rejoin the arena, so a later large request fell through to hipMalloc, potentially introducing an implicit device synchronization.

The fix applies when a completed block came from the arena, is at least twice the requested size, and leaves a useful remainder. Under the same arena lock, the reuse layer returns the complete block and allocates the requested size again. The remainder becomes immediately available for coalescing and reuse.

With a 3 GiB arena and chunk 2,048, the complete request reached 1,553.91 token/s with a 168.56-second TTFT. One 5 GiB arena experiment reached 1,600.04 token/s, but the tightest GPU had only about 38 MiB of sampled headroom. We did not retain that capacity-edge configuration. The final reporting point uses 3 GiB instead of selecting the one run that barely crossed 1,600.

Final performance data

The test system used two AMD EPYC 9334 processors, 64 physical cores in total, roughly 1 TiB of host memory, and eight Radeon Pro W7900D GPUs with about 48 GiB each. The model used the official safetensors checkpoint on the ROCm backend. Every row below is a complete HTTP request; model loading is excluded.

Scenario	Input / output	Result	Scope
261K cold-prefill recovery baseline	261,933 / 1,024	332.73 token/s, TTFT 787.22 s	Same input after restoring a runnable production configuration, before optimization
Retained 261K cold prefill	261,933 / 1,024	1,553.91 token/s, TTFT 168.56 s	3 GiB arena; complete SSE termination
Complete single-request 1M context	1,000,411 / 1,024	732.59 token/s, TTFT 1,365.59 s	Complete generation; output matched the previous 1M run
Single-request DSpark decode	about 2K generated tokens	25.82 / 26.14 token/s	Two no-profiler runs
Eight-request aggregate DSpark decode	eight long requests	107.61 / 109.47 token/s	Current reliable two-run baseline
Historical eight-request best range	eight-request steady state	111.15–114.43 token/s	Shows the observed ceiling; not the default result

From the recovery baseline to the final retained 261K result, TTFT fell by about 78.6% and cold-input throughput rose by about 4.67×. No single kernel produced this result. It came from the official sparse semantics, index representation, shared-memory sizing, dense tiling, chunk size, and memory lifetime working together.

TTFT runs from the HTTP request to the first non-empty text event. Cold-prefill rate is input tokens divided by TTFT. It includes request handling, eight-GPU prefill, and first-output cost; it is not pure GPU-kernel throughput. VRAM figures are whole-card samples taken every five seconds and may miss instantaneous peaks. Prefill and decode rows come from different requests and cannot be added into one run's total throughput.

The article includes the raw summary: final performance JSON.

What remains after the optimization series

The four articles began with Engram, continued through weight assembly, eight-GPU state transfer, and vision-tensor alignment, and end with long-context performance. Together they show that model capability no longer lives only in dense Transformer computation.

Knowledge can be retrieved through Engram. Cross-layer candidates can restrict attention history. Visual semantics can enter the language backbone as typed rows. Speculative decoding can reduce the number of full target-model rounds. An inference engine now manages a combination of computation, retrieval, state, and resources.

The effective optimization order followed the same structure: restore the model's semantics, locate repeated work, pass operator oracles, verify complete output, and only then decide from end-to-end TTFT, throughput, and memory whether a change should remain.

Reaching a peak shows that an experiment worked. Repeatedly completing requests within correctness, capacity, and stability boundaries shows that the engineering path works.