Decoupling: Let Models, Algorithms, Backends, and Kernels Evolve Independently

First drafted on 2026-08-28; rewritten on 2026-09-01. This article follows the current model_spec/, runtime/, backend/, kernel/, weight/, and server/ source trees. It describes boundaries that exist in the implementation rather than presenting a generic planner or remote Expert RPC as completed architecture.

Decoupling in zLLM is not about making the directory tree look tidy. It is about making each kind of change travel along the right axis: model architecture changes belong mainly in the spec and runtime, new mathematical semantics enter the domain layer first, device differences stop at the backend, and local performance work stays in kernels. A complete inference request still uses all of these regions, but no region needs to know every implementation below it.

1. The Architecture at a Glance

The main dependency direction runs from top to bottom. Weight formats form a side data path: model_spec supplies shapes, weight parses containers and encodings, and the backend selects the final resident representation. This side path does not control model execution.

The five-layer spine and the device implementation region are responsibility boundaries, not six runtime processes. Inside device implementation, backend/<platform> and kernel/<platform> are peers:

• the platform backend owns tensors, weights, caches, placement, submission, and transfer;
• parameterized kernels own local computation;
• together they implement the capabilities required by the upper layers.

Production requests still execute a complete New Prefill, Append Prefill, or Decode Round. Decoupling changes code dependencies and resource ownership; it does not turn full-model inference into disconnected partial demonstrations.

The responsibility of each region is concise:

• bin/runtime and bin/tools compose platform entry points driven by --config <yaml>. Model-specific validation and profiling binaries live outside the main runtime.
• embedded exposes an in-process Engine with the same structured JSON semantics as service mode, without starting HTTP, the scheduler, or iroh.
• server/ owns HTTP/H3 protocols, Chat/Anthropic/Responses compatibility, nodes, the scheduler, and iroh stage transport.
• model_spec/<model> contains execution- and platform-independent architecture data. It is the single specification shared by runtime orchestration and weight assembly.
• runtime/<model> expands layer specs and owns platform-independent full-model orchestration. Platform composition files remain beside the model runtime.
• attention/, moe/, kv_cache/, and norm.rs contain domain specs, reusable algorithms, and reference semantics.
• backend/ contains capability contracts and platform resource implementations, including cache, completion, streams or command buffers, residency, and scheduling.
• kernel/ contains parameterized local computation. Model dimensions may not be hard coded. ROCm HIP bodies are stored in separate source.hip files; Rust launchers load, validate, and submit them.
• weight/ parses standard containers and encodings. Model-specific code is limited to naming and shape adaptation.

2. Dependency Rules

1. Shared scheduling such as runtime::prefill does not depend on a concrete model or backend.
2. model_spec contains architecture data only and does not depend on other crate modules. Runtime and weight assembly depend on it without depending on each other.
3. Platform-independent code in runtime/<model>/mod.rs depends on capability traits. Only adjacent platform composition files may depend on a concrete backend.
4. backend/ and kernel/ do not name or branch on concrete models. Cooperative two-GPU MoE and Metal replay express device capabilities; they may not encode GLM layer numbers.
5. Domain modules contain specs, algorithms, and references. Concrete storage, kernels, and synchronization belong to device implementation.
6. weight/ parses and loads formats. Algorithms do not enter the weight tree, and platform resources do not enter format parsing.

A violation usually indicates dependency inversion. Any exception must be a concrete platform composition file or a measurable production execution path. It must not be an empty flow, factory, or protocol introduced for hypothetical reuse.

3. The Three Decoupling Boundaries

Boundary	Upstream expression	Downstream responsibility	Current constraint
A · Model × Domain	Layer dataflow and complete prefill/decode ordering	Mathematical semantics for attention, MoE, KV, and their references	Define and test new semantics before accelerating them
B · Domain × Capability	Required capabilities declared by generic bounds	Tensor/cache/weight residency, submission, and transfer contracts	Runtime does not touch HIP events or Metal command buffers
C · Capability × Implementation	Capability and resource-lifecycle contracts	Peer platform-backend and parameterized-kernel tracks	Both tracks implement capabilities; kernels contain no model names or constants

Boundary A: Model × Domain

Model orchestration calls domain operations and passes a generic backend through the dataflow. The domain describes what a computation means; device implementation decides how it runs. A new operation such as kpool first receives a reference definition and tests. GPU kernels are accelerated substitutes for that definition rather than independent sources of semantics.

CPU remains the primary correctness oracle, but it is not limited to that role. Current GLM-5.2 paths also use CPU work in production for DSA selection and the DSpark drafter. The boundary is semantic ownership, not a rule that all CPU code must be diagnostic.

Boundary B: Domain × Capability

The base Backend trait contains operations shared across models. Domain capabilities such as KdaKernel, HyperConnectionKernel, DsaPrefillBackend, and ExpertPrefillBackend remain separate. A model's generic bounds state exactly what it needs. New capabilities may use explicit unsupported defaults so existing backends can adopt them incrementally without pretending to support the new semantics.

Stage execution uses the same principle. Runtime sees completion, available resources, and submission classes; it does not see HIP events or queue handles. The ROCm backend may choose latency or background streams and retire completions out of order without leaking those mechanics into model code.

Boundary C: Capability × Device Implementation

backend/<platform> and kernel/<platform> sit on the same side of this boundary as peer tracks. The backend owns resources, resident representations, shape/format dispatch, and submission semantics. Kernels own parameterized local computation.

Weights enter device implementation through LinearWeight variants such as F32, F16, Bf16Bytes, FP8, MXFP8, MXFP4, NVFP4, W4A16, W8A16, and GGUF. A platform backend selects a kernel using weight format and shape. Quantized paths decode inside the kernel instead of expanding weights to F32. This is a collaboration relationship, not an architectural hierarchy from platform backend down to kernel.

4. Key Decisions

Decision	Rationale
Keep model_spec data-only instead of introducing a behavioral thin model layer	Runtime orchestration and weight assembly share one specification without depending on each other
Use associated Tensor, Weight, and Cache types instead of one universal object	CPU values, Metal buffers, ROCm dual host/device state, and wgpu resources are fundamentally different
Split capability traits by domain	Model dependency surfaces stay readable and new capabilities can enter incrementally
Define reference semantics before optimized kernels	Cross-backend behavior remains testable while optimization proceeds independently
Keep the single model flow in runtime and platform composition next to it	Avoid N-model × M-platform copies of the same execution algorithm
Drive entry points through YAML configuration	Keep service and tool entry points converged; validation and profiling remain separate concerns

5. Benefits

1. Combination cost grows linearly. A new model built from existing operations requires no backend changes. A new operation on a new backend requires a kernel and a capability implementation, not another full runtime.
2. Correctness has an anchor. Reference behavior and tests allow a backend to start with a slower correct path and optimize later.
3. Platforms evolve independently. Metal replay, ROCm stage pipelines and CPU/GPU DSA cooperation, CUDA paths, Vulkan, and experimental NPU work can progress without changing model semantics.
4. Reviews have a clear target. A change to mathematical meaning, device resources, or a local kernel has a mostly unique home.

6. Costs and Trade-offs

1. Hot semantics still need multiple implementations. Rust references, MSL, HIP, CUDA, WGSL, and AscendC do not optimize themselves.
2. Generic instantiation increases compile time and binary size. Capability bounds also become longer as models add domains.
3. Capability growth requires discipline. Every new architecture concept creates pressure to add shared trait methods even when the capability is not yet general.
4. Platform-specific behavior must be documented. Fallbacks, shape constraints, replay, CPU/GPU cooperation, and multi-GPU paths differ by backend.
5. Peer backend and kernel tracks require careful ownership. Dispatch and resource lifetime belong to the backend; local math belongs to the kernel. Moving either concern to the other track makes the boundary harder to audit.

7. Series Guide

• 02 · Model Layer: formats, config, weight assembly, and quantization
• 03 · Algorithm Layer: attention, MoE, KV cache, prefill, and decode orchestration
• 04 · Backend: device abstraction, resources, submission, and platform behavior
• 05 · Kernels: primitive and fused kernels, precision selection, and hardware mapping
• 06 · Service Layer: fences, protocols, input/output, and the embedded library
• 07 · Multi-node: scheduling, admission, cross-node state, and stage pipelines
• 08 · Speculative Decoding: transactional verification, MTP, DSpark, and fences