Why Did One Sentence Consume Hundreds of Thousands of Tokens?

Behind one short sentence is a huge token stream containing system instructions, tool definitions, project context, and history. Hot state stays in VRAM; evicted state is backed up to SSD and restored when needed.

You typed one sentence. Why did the usage report show tens of thousands—or even hundreds of thousands—of input tokens?

Because agents such as Claude Code and Codex do not send only the text visible in the input box. Before calling the model, the client assembles a complete working context: system and developer instructions, tool descriptions, JSON Schemas, workspace rules, conversation history, previous tool calls and results, plus file excerpts or summaries needed to continue the task. Your sentence may be only 20 tokens, but it is the final sliver of a very large prompt.

In other words, every request must tell the model three things: who it is, what it can do, and what has already happened. This is not training and it does not permanently write knowledge into the model. It is context supplied for this inference request, and much of it normally has to be present again in the next request.

That is exactly why prefix caching matters. A client may resend tens of thousands of identical tokens, but a server that retained the model state for that prefix does not need to recompute it from the beginning.

At first glance, an LLM service looks like an ordinary JSON-over-HTTP application: send messages in, stream tokens out. Once it must support several client protocols, long conversations, persistent cache state, and reliable tool calls, the problem becomes much deeper than adding a few routes.

In zLLM we implemented three common interfaces—OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages—plus a two-level prefix cache with VRAM as the active tier and SSD as its backup, and token-level generation fences for DeepSeek DSML tool calls. They appear to be separate features, but all enforce the same invariant: protocol state, the tokens seen by the model, and the execution state stored on the device must agree exactly.

The tens of thousands of tokens behind one sentence

A typical Claude Code or Codex request contains roughly these parts:

Context component	What it contains	Why it may reappear every turn
System and developer instructions	Identity, safety boundaries, output rules, coding conventions, workflow	A model does not remember these rules across independent HTTP requests
Tool definitions	Names, descriptions, and parameter Schemas for shell, files, search, browser, and other tools	The model must know which actions exist and how to invoke them legally
Workspace knowledge	Current directory, repository rules, AGENTS.md, environment and permissions	The agent must act in the correct project under the correct constraints
Conversation history	User requests, prior reasoning, completed work, intermediate conclusions	Keeps a multi-turn task coherent
Tool calls and results	Commands, file contents, errors, logs, and returned data	Later decisions depend on these observations
Current input	The sentence you just typed	Usually only the tail of the full prompt

This is what people mean when they say an agent first “injects tens of K of knowledge” into the model. Here, K means a thousand tokens. Tokens are not characters: a tokenizer splits prose, punctuation, paths, source code, and JSON differently. Tool Schemas are particularly expensive because every field name, description, enum, nested object, and protocol example counts as input. With dozens of enabled tools, their declarations alone can occupy a substantial part of the context window.

The UI usually shows only the user's messages and the final answer, making this cost easy to mistake for secret model output. In practice, usage has at least three relevant categories:

• input_tokens: the complete context supplied this turn, not just the newest user message;
• cached_tokens: the prefix within that input whose execution state the server reused;
• output_tokens: newly generated reasoning, text, or tool calls.

Billing rules differ by provider, but computationally, cached tokens are still part of the input. They really are present in the prompt; the server simply avoids repeating the same prefill work. This produces a result that looks contradictory but is not: a request can contain 100,000 input tokens while only the final few hundred tokens require new computation.

Without a cache hit, all 100,000 tokens must pass through every layer of the model again, increasing time to first token. With a hit, the server restores the previous state and runs append prefill only for the new user message and assistant prefix.

Three major APIs: the differences go beyond field names

zLLM exposes three primary endpoints:

API	Typical clients	Main input	Streaming output	Continuation model
POST /v1/chat/completions	OpenAI SDKs and general chat clients	messages	Chat chunk SSE	Client resends history and may supply a cache identifier
POST/GET /v1/responses	Codex, ZCode, newer agent clients	input items	Responses SSE or WebSocket events	previous_response_id
POST /v1/messages	Claude Code and Anthropic SDKs	top-level system plus messages	Anthropic event SSE	Client resends history

All three eventually become the same internal tasks. Structured messages pass through the model's chat template and tokenizer, followed by NewPrefill, AppendPrefill, and DecodeRound. The adapter layer, however, cannot be a collection of field renames.

Chat Completions: the most direct common representation

Chat Completions centers on an ordered messages array. Tools use the OpenAI function-tool shape:

{
  "model": "deepseek-v4",
  "messages": [{"role": "user", "content": "Check the weather in Shanghai"}],
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "parameters": {
        "type": "object",
        "properties": {"city": {"type": "string"}},
        "required": ["city"]
      }
    }
  }],
  "stream": true
}

A non-streaming response collects text and tool_calls into one assistant message. A streaming response must keep each tool call's index, ID, function name, and argument deltas stable. Tool-call IDs cannot be derived from the function name alone: the same function can appear in multiple requests or be called more than once in a response. zLLM hashes a request-level scope, call index, and function name together.

Chat Completions is a useful internal canonical form because roles, tools, and sampling controls are explicit. Responses and Anthropic requests are carefully converted into this representation before they reach the scheduler and runtime.

Responses: an event protocol, not Chat with different JSON

The Responses API accepts a stream of input items rather than just a message array. Items may include messages, function_call, function_call_output, images, and the additional_tools prefix used by Codex Responses Lite. Those tools may be wrapped in namespaces, so conversion must expand executable functions while skipping hosted tool types that the local runtime cannot execute.

Output is not a growing delta.content string either. Text, reasoning, and function calls are separate output items with their own created, delta, and completed events. Reasoning must be exposed as a distinct output[type=reasoning] item; folding it into answer text prevents clients such as Codex from rendering and restoring state correctly.

zLLM supports both HTTP POST with SSE and a WebSocket upgrade on GET /v1/responses. The latter serially reuses one connection for multiple response.create requests, which suits long-running agent sessions.

previous_response_id needs special attention. zLLM persists the protocol messages and cache key of the previous response, so it can reconstruct a conversation chain after restart. But that record does not own model KV state. Restoring protocol history only means the complete input can be rebuilt; whether VRAM or SSD state is available remains a separate runtime decision.

Anthropic Messages: content blocks and event order are semantic

Anthropic puts system at the top level and represents content as blocks. Tool definitions use name, description, and input_schema; invocations and results are tool_use and tool_result blocks. The adapter must convert these structures in both directions instead of flattening them into strings.

Streaming event order is also strict: message_start, then content_block_start, the corresponding deltas, a block stop, and finally message_delta and message_stop. Text and multiple tool calls may occupy different block indexes. Renaming Chat SSE fields produces incorrectly ordered or unclosed blocks.

Claude CLI also calls /v1/messages/count_tokens to decide when to compact the context. Even an estimated count must include system content, messages, and tools. Counting only user prose delays compaction far beyond the intended threshold.

What a cache hit actually means

In an inference service, a cache hit does not mean “we found a similar question,” and it does not mean “return the old answer.” Its precise meaning is:

The beginning of the new request's token sequence is exactly equal, token for token, to a saved session. The runtime can restore every layer's KV and auxiliary DSA/MTP state at the end of that prefix and continue from there.

Suppose the saved terminal state represents:

[system][tools][user-1][assistant-1]

The next request is:

[system][tools][user-1][assistant-1][user-2][assistant-prefix]
|<----------- cached_tokens ----------->|<-- append prefill -->

The first cached_tokens do not run through the full Transformer again. Only the appended suffix receives prefill before decoding begins. A hit reduces repeated prefill and time to first token; it does not eliminate decoding for the new answer.

Several consequences follow:

1. Tokens must match; visually similar text is insufficient. Whitespace, JSON serialization, tool-schema order, thinking controls, templates, and special tokens can all fork the prefix.
2. A cache ID is a lookup hint, not proof of correctness. The runtime still verifies the token prefix and namespace, falling back to a longest-prefix search or cold prefill if they disagree.
3. The cached length must be shorter than the new prompt. A complete terminal state without appended tokens is not a valid append boundary.
4. A hit does not imply residency. The matching state may be in VRAM or backed up on SSD. Both avoid recomputing the prefix, but restore latency differs greatly.

Terminal-cache identity includes not only user-visible messages, but also model and template-related request state, assistant output, and structured tool calls. Otherwise, two requests that look identical on screen could share incompatible execution state.

Why VRAM needs an SSD backup tier

Long-context KV is expensive. Agent conversations with tens or hundreds of thousands of tokens benefit enormously from retaining historical branches, but VRAM cannot grow without bound. A pure LRU that simply discards old sessions makes a recently evicted long conversation restart from zero.

The actual hierarchy is:

Request arrives
  ├─ exact cache ID resident in VRAM ─────────┐
  ├─ longest matching prefix in VRAM ─────────┤
  ├─ exact cache ID backed up on SSD ─────────┤→ restore to VRAM → append prefill → decode
  ├─ longest matching prefix on SSD ──────────┤
  └─ no match → new prefill in VRAM ──────────┘

VRAM: active primary cache, smallest capacity, lowest latency
  │
  ├── eviction backup ──> SSD
  └── on-demand restore <── SSD

SSD: larger backup store; it never serves as a parallel inference destination

Level one: resident terminal state in VRAM

After a successful request, the engine retains the terminal session: token sequence, namespace, parent and round relationship, and model-private execution state. The common cache layer owns entries, prefix graphs, and eviction; each model runtime defines its own KV, compressed rings, DSpark target cache, and other state.

Admission must use a real residency budget, not a limit such as “keep N sessions.” Sessions with similar token counts can differ by several times in allocated bytes due to fixed and stepped costs: recent/compressed rings, batch scratch, speculative blocks, and target caches. zLLM accounts using actual allocated_bytes and reserves room for KV growth during decode.

The cache is not a flat LRU list. Agent conversations branch from shared history, so the memory tier maintains a shared prefix block graph. Parents may be reused by several descendants, while only dead branches are released. A prefix-round limit prevents every intermediate terminal state from remaining resident forever.

Level two: SSD backup and restore

When VRAM evicts a terminal state worth retaining, the runtime downloads and encodes the model state into a versioned SSD snapshot. The index records cache ID, complete token sequence, namespace, round, head status, resident byte count, and blob generation; large KV data is stored in chunks.

Writes need a commit boundary. The manifest switches to a new generation only after its blob is complete, so a crash cannot leave an index pointing to a half-written cache. Reads treat disk snapshots as untrusted input: version, lengths, counts, and trailing bytes are validated before allocation.

On an SSD hit, the engine first obtains a reusable device session, uploads each stage cache, restores DSpark and other auxiliary state, and only then continues computation in VRAM. SSD is therefore a backup capacity tier, not another compute path.

SSD also supports longest-prefix lookup. Matching remains strict: tokens.starts_with(snapshot.tokens), isolated by namespace. To diagnose almost-hits, zLLM logs snapshots sharing at least 1,024 tokens and more than 90% of the old prefix. Those reports often reveal a template, tool echo, or serialization fork immediately.

The hardest cache problems we encountered

Eviction order and admission deadlock

An early implementation tried to reserve memory for a new request before evicting old cache entries. A request could be rejected even though reclaimable state existed. The correct order is to identify disposable resident entries, finish any required backup and release, and only then perform admission—while still reserving capacity for decode growth.

Asynchronous restore still needs explicit dependencies

SSD reads, CPU decoding, and multi-GPU host-to-device uploads can run concurrently, but their dependencies cannot disappear. We encountered restore jobs writing cache buffers after a compute stream had begun reading them, as well as cache-open commands whose resources were reused before draining. The fix was not a blanket global synchronization. Each stage publishes a restore-completion event; independent I/O and uploads overlap, while the first consumer waits on the exact dependency it needs.

Reusable pools can still leak capacity

If every SSD restore creates a new session, old device buffers may accumulate indefinitely in a “reusable” pool without actually being reused. Restore must take and overwrite a pooled session first. Failure and cancellation paths must reset and return sessions just as carefully as the success path.

Tool echoes are a frequent cache-chain breakpoint

A tool call travels through four forms: native model text, structured API call, client execution, and re-encoded next-turn history. If forward parsing and history rendering are not inverse operations, the next turn forks at the tool position. DSML continuation failures we saw came from differences in tags, string-versus-JSON parameter encoding, and case normalization.

Each ToolDialect therefore owns four related operations: instruction injection, tool-call history rendering, tool-result rendering, and model-output parsing. They must come from the same protocol implementation rather than being scattered between HTTP adapters and backends.

Tool-call fences: why parsing after generation is not enough

Prompting a model to “return valid JSON” is not a guarantee. It may invent a tool, repeat a parameter, omit a required field, emit a closing tag inside JSON, or fall into a loop after long reasoning. Parsing the final output can only report failure after the decode work has already been wasted.

zLLM therefore implements tool support in three layers:

1. Protocol prompting and history rendering encode a common function schema into GLM XML, ChatML JSON, or DeepSeek DSML.
2. Incremental parsing separates visible text from structured tool regions without leaking half a tag to the client.
3. Token-level generation fences force or exclude candidates before sampling so structured regions can follow only valid paths.

The third layer is the actual fence. For a named DeepSeek DSML tool, the state machine is:

fixed tool_calls/invoke prefix
        ↓
choose one unseen parameter
        ↓
generate its value under JSON Schema
        ↓
close parameter; are all required fields present?
        ├─ no: another parameter is required
        └─ yes: invoke/tool_calls may close

The fence forces fixed tags, the tool name, and parameter names. JsonSchemaFence determines legal value tokens. A parameter can appear only once, required fields must be complete before closing, markup-opening tokens are excluded inside strings, and terminal tokens remain illegal while the structure is unfinished.

tool_choice=auto is harder. The model must remain free while writing ordinary text; the fence activates only after detecting the DSML trigger. Several tool candidates may remain possible, so their state machines advance in parallel and incompatible branches are discarded. The allowed next-token set is the union of surviving branches. Forcing a tool from the beginning would incorrectly turn auto into required.

The fence sits between the output head and sampling. Ordinary decode, every batch row, MTP or DSpark draft generation, and speculative verification must all use the same request-level fence state. Missing one path creates the classic failure mode: tools are valid without speculative decoding and intermittently malformed when it is enabled.

Tool-fence bugs that taught us the most

Tokenizer boundaries are not string boundaries

A closing tag may span tokens or share a special </ token with other text. A fence cannot assume one character or one tag per token. At initialization, all literals are encoded with the active model tokenizer and closing boundaries are verified. An incompatible tokenizer causes fence construction to fail early instead of entering an unreachable generation state.

Auto branches require a union, not the wrong intersection

Different candidate tools may permit different next tokens. A token can be excluded only when every surviving branch rejects it. Naively merging excluded sets kills valid branches too early. After each generated token, branches incompatible with that token must also be removed.

Draft tokens need a cloned state advanced one token at a time

Speculative decoding proposes several tokens at once. The current fence cannot validate the whole batch as one unit, and drafts must not mutate committed state. zLLM clones the guard, computes a fence for each position, advances the probe token by token, and advances the real state only for the accepted prefix. Positions already forced by the grammar can bypass unnecessary logit work.

Parser patches are not a substitute for generation constraints

We added compatibility for case differences, escaped DSML tags, and stray closing tags. Parser tolerance helps with existing model behavior, but predictable structural errors should still be prevented before sampling. Otherwise a non-streaming result may appear repaired after the fact while a streaming client has already received the malformed fragment.

Syntax fences and loop protection must compose

Tool grammar prevents invalid structure; it does not stop a model from repeating its reasoning indefinitely. zLLM's GenerationGuard detects repeated tokens, repeated patterns, and evidence-backed semantic restarts. It excludes the known looping successor before the next sample; if a speculative batch already contains a loop, it truncates accepted rows and may force the reasoning-end token. Tool grammar and loop protection combine into one TokenFence, but remain separate state machines so model protocol logic does not contaminate the general generation layer.

The complete path of one request

Putting the APIs, cache hierarchy, and tool fence together, a real request follows this path:

Chat / Responses / Anthropic request
        ↓  validation and normalization
common messages + function tools
        ↓  model dialect, template, tokenizer
complete prompt tokens + cache identity
        ↓
exact/longest prefix in VRAM → exact/longest backup on SSD → cold start
        ↓                                │
        └──────── restore to VRAM ───────┘
        ↓
NewPrefill or AppendPrefill
        ↓
output head → request token fence → sampling/speculative verification
        ↓
incremental native tool-stream parser
        ↓
Chat chunks / Responses events / Anthropic content blocks
        ↓
successful terminal state remains resident in VRAM;
evicted state is backed up to SSD

The HTTP layer does not own KV. The cache layer does not understand model syntax. The backend does not know tool names. The model runtime does not emit a particular external SSE format. Each layer owns only the semantics it needs, but passes verifiable state across its boundary rather than a vague “session ID.”

The final test

A prefix cache is not correct because a log line says hit=true. Restored tokens, positions, per-layer KV, and auxiliary state must match a full prefill. Performance is measured by real time to first token, not only by the number of skipped tokens.

A tool implementation is not correct merely because a final string passes a JSON parser once. Streaming event order, history replay, next-turn cache continuation, batch decode, and speculative paths must all obey the same constraints.

The lesson is simple: API compatibility does not end when the JSON looks similar. A cache hit does not end when the ID matches. Structured generation does not end when malformed output can be repaired afterward. Reliable inference ultimately comes down to the exact tokens seen by the model and the exact state held by the device.