Why We Are Rewriting an Inference Engine in Rust

zLLM is not a Rust wrapper around an existing framework. It is a native inference engine designed around model execution, data lifecycles, and hardware resources, with Rust at its core and only the third-party dependencies that are truly necessary.

This article answers three questions: why zLLM needs to own its engine boundaries, why Rust is a natural language for expressing those boundaries, and how that choice appears in real code and complete inference paths.

Building another LLM inference engine today invites two obvious questions: why not use a mature engine, and why choose Rust?

If the only goal is to get a model running quickly, an existing framework is usually the economical choice. zLLM is solving a broader problem. The same engine must work across Apple UMA, CPU, CUDA, ROCm, Vulkan, NPUs, local SSDs, massive MoE weights, long-context KV caches, and eventually layer-partitioned multi-node execution. These resources differ in location, synchronization, bandwidth, lifetime, and failure boundaries. The hard problem is not wrapping another operator API; it is making the resource relationships of the complete inference process explicit enough to verify, optimize, and evolve.

That is why zLLM starts from a clean design. Not because existing projects are inadequate, and not for the sake of being self-built, but because the boundaries we need differ from the historical boundaries of many existing systems.

What “rewriting from scratch” means

Starting over does not mean rejecting standards or rebuilding every piece of infrastructure.

zLLM continues to read Safetensors, GGUF/GGML, compressed-tensors, and ModelOpt weight formats. It uses Metal, CUDA, ROCm, Vulkan, and other platform capabilities, along with proven general-purpose Rust libraries. We do not reinvent file formats, network protocols, or operating-system interfaces.

What we redesign is the inference engine itself:

• how model specifications are represented;
• how model algorithms are written once;
• which capabilities a backend must provide;
• how kernels stay parameterized instead of binding themselves to a model;
• who owns weights, KV cache, activations, experts, and scratch memory;
• how complete prefill, append-prefill, and decode paths are scheduled;
• how single-node, embedded, and layer-partitioned multi-node execution share one set of semantics;
• how CPU reference and cross-backend consistency anchor correctness.

In short, zLLM reuses open standards and platform capabilities while owning model execution and resource semantics.

Reused	Owned by zLLM
Safetensors, GGUF/GGML, compressed-tensors, ModelOpt	Model specifications, algorithm orchestration, and weight assembly
Metal, CUDA, ROCm, Vulkan, and NPU interfaces	Backend capabilities, placement, and synchronization
Mature serialization, networking, and async Rust libraries	KV, expert, activation, and scratch lifecycles
Platform-native GPU kernel languages	Complete prefill, append-prefill, and decode rounds

This is also the difference between redesigning and merely rewriting. A rewrite can reproduce an old structure in a new language. A redesign returns to data dependencies, resource locations, and real workloads, then asks whether every module deserves to exist.

A minimal engine, not a minimal feature set

For zLLM, “minimal” does not mean a demo with limited capability. It means minimizing the number of concepts in the system.

Our long-term rule is Simple / Short / Straight:

• Simple: solve a responsibility in one clear module rather than splitting it into layers of abstraction;
• Short: keep code, scopes, and object lifetimes as short as possible;
• Straight: define things near their use, so control flow and dependencies can be followed directly.

Directories, files, traits, wrappers, managers, registries, and contexts are not free. Every additional entity adds a name to remember, another forwarding layer to understand, and another lifetime to trace. An entity is justified only when it owns distinct data, behavior, lifetime, or dependency boundaries.

zLLM therefore avoids generic flows without real implementations, speculative abstract factories, and facade shells that only forward calls. There is one model execution flow. Platform differences belong to capability traits and concrete backends. Kernels express operators and do not know model names.

The goal is not to minimize capability, but to minimize the mental load required to understand the system.

The fewest necessary third-party dependencies

Minimal dependencies do not mean zero dependencies. There is no reason to reimplement mature password hashing, serialization, async runtimes, HTTP, or standard-format parsing for ideological purity.

Our principle is simple: a dependency may provide general-purpose capability, but it must not own the core semantics of our inference engine.

Model execution, resource residency, KV lifecycles, weight assembly, scheduling policy, and cross-backend boundaries must remain visible in zLLM itself. A third-party library should not invisibly decide where a tensor lives, when it moves, when it synchronizes, when it is reclaimed, or how a model executes.

This has three direct benefits.

First, the dependency graph is easier to audit. Updating a network library should not change model numerics. Adding a weight format should not invade the runtime. Replacing a platform binding should not require rewriting model algorithms.

Second, performance costs stay visible. Inference systems are especially vulnerable to a convenient abstraction hiding an implicit copy, a device synchronization, or temporary dequantization. Owning the critical path lets us trace the origin, destination, and lifetime of every large allocation.

Third, long-term evolution is not constrained by an upstream framework. zLLM can reuse standards and general libraries without handing its architecture to a large tensor runtime or Python extension ecosystem.

Why Rust fits the job

Rust matters for more than “no GC” or “faster than Python.” For an inference engine, its strongest advantage is the ability to encode resource relationships in program structure and reject many illegal states at compile time.

1. Ownership and lifetimes match inference resources

An inference engine manages expensive resources: memory-mapped weights, device buffers, KV pages, command buffers, asynchronous I/O, distributed sessions, and temporary scratch space. Every one of them raises the same questions: who owns it, who may borrow it, when may it be released, and will it remain valid until an asynchronous task finishes?

Rust ownership, borrowing, and lifetimes are not incidental restrictions; they are direct language for these problems. A prefetch task cannot safely reference a buffer that has been released. A cache cannot be reclaimed while an execution queue still uses it. Shared state must declare its synchronization boundary. In C or C++, these rules often rely on convention and review. In garbage-collected languages, delayed reclamation and external resource lifetimes can bypass the collector. Rust turns a meaningful portion of these failures into compilation errors.

2. Predictable execution without a GC

Decode is a continuous, fine-grained loop sensitive to tail latency. Unpredictable pauses, implicit allocation, and delayed destruction make performance analysis harder.

Rust has no garbage collector, and value lifetimes normally follow lexical scope. Combined with preallocation and scratch reuse, this makes allocation, release, and synchronization on hot paths more predictable. It does not guarantee performance automatically, but it provides the foundation for building a predictable system.

3. Zero-cost abstractions express backend capabilities

Backend differences are real. A CPU tensor may be ordinary memory, a Metal tensor a device-resource handle, and ROCm may have a completely different placement and submission model. Forcing all of them into a single dynamic object tends to produce a lowest-common-denominator API, indirect calls, and runtime checks across hot paths.

zLLM uses traits and associated types to express capabilities. A model runtime declares what it needs; a concrete backend supplies those capabilities at compile time. Generics cost compilation time and binary size, but in return provide static composition, a readable capability set, and less dynamic dispatch in performance-critical paths.

4. Enums and pattern matching make format states explicit

Weights may be F32, F16, or BF16, or they may remain in FP8, FP4, W4A16, or GGUF block encoding until a kernel consumes them. Explicit enums require every backend to process or reject each legal state deliberately instead of guessing across strings, pointers, and implicit conventions.

This is particularly important for quantization. Quantization is not a loading-time detail that always expands to floating point. It is a data path orthogonal to model specifications and platform kernels. The more explicit the state, the harder it is for accidental expansion or incorrect dispatch to enter the system.

5. Rust spans the systems layer and the service layer

Many inference stacks are naturally split: C/C++ or GPU languages at the bottom, Python, Go, or another service framework on top. Each side can be reasonable alone, but together they create duplicated data models, cross-language FFI, separate error systems, and lifetimes that are hard to unify.

Rust can handle weight parsing, CPU reference implementations, backend bindings, runtimes, schedulers, HTTP/SSE services, and embedded library APIs. GPU kernels still use platform-native languages—Metal Shading Language, HIP/CUDA, WGSL, or AscendC—but the engine around them shares one type system, error model, and resource semantics.

That is what zLLM means by “pure Rust.” It does not deny the existence of native platform kernels. It means there is no Python control plane and no delegation of core execution to another large tensor runtime: Rust directly owns the inference engine.

6. One toolchain keeps cross-platform engineering consistent

Cargo build, features, tests, and dependency management let CPU, Metal, CUDA, ROCm, and Vulkan paths maintain clear boundaries in one project. Platform capabilities are enabled by feature and target. CPU unit tests require no real weights, while GPU paths use the CPU oracle as their numerical anchor.

Rust cannot replace real-device testing or catch an ABI mismatch inside runtime-compiled MSL or HIP at compile time. It can, however, confine platform-specific unsafe boundaries to a relatively small region while the rest of the system retains static checking.

How zLLM’s architecture reflects these choices

zLLM divides the system into regions with distinct responsibilities and one-way dependencies:

The directory names are less important than the direction of dependency:

• model specifications do not depend on platforms;
• model algorithms are written once rather than copied into every backend;
• backends and kernels do not know concrete model names;
• domains such as attention, MoE, and KV cache keep device-independent semantics and reference implementations;
• the weight layer owns standard formats, naming, shape validation, and lifetimes—not model algorithms;
• CPU is the correctness oracle, while GPU kernels are accelerated implementations verified against it.

This avoids the most common form of combinatorial explosion. Adding a model does not mean copying its execution flow for every platform. Adding a backend does not mean relearning every model. A new model mainly adds specifications, orchestration, and weight adaptation. A new backend mainly implements existing capabilities. Domain interfaces expand only when a genuinely new operator semantic appears.

Start from data flow, not semantic labels

A clean-sheet design lets zLLM organize the system around real data dependencies instead of paper sections or model terminology.

If A produces the input consumed by B, they form a serial pipeline and should stay close while sharing intermediate state. If A and B independently read the same hidden state, they should remain separate so a backend can schedule them concurrently. Weight structures, kernel fusion, and resource lifetimes follow this same rule.

Production inference recognizes only three complete tasks: prefill for a new session, append-prefill for an existing session, and a complete decode round. Stopping at one layer, running one kernel, or comparing a partial output can be useful diagnostics, but cannot replace an end-to-end workload. Optimization must ultimately return to TTFT, decode latency, throughput, peak memory, SSD stalls, and device utilization.

This keeps the system from being distracted by attractive but irrelevant local numbers.

Rust does not automatically deliver correctness or performance

Choosing Rust does not mean the language replaces engineering discipline.

Rust cannot prove a GPU kernel numerically correct, prevent a bad algorithm, or guarantee that an elegant abstraction has no performance cost. Multiple backends mean one kernel semantic may need optimized Rust reference, MSL, HIP, CUDA, WGSL, or AscendC implementations. Generics increase compilation cost. Capability traits can still grow without discipline.

zLLM therefore combines the language with explicit rules:

• every new operator starts with reference semantics and unit tests;
• CPU and device backends are checked for consistency;
• kernel dimensions are parameterized rather than embedding model constants;
• isolated kernel benchmarks never substitute for complete-path metrics;
• complexity is not added for performance without measurement;
• source presence, successful compilation, oracle validation, and real-device end-to-end validation remain distinct states;
• every change is the smallest one required for the current task.

Rust provides solid ground. Engine quality still depends on clear boundaries, reproducible validation, and sustained restraint toward complexity.

Why starting over is worth it

Rewriting an engine is expensive. Weight formats must be understood directly, kernels optimized platform by platform, services and schedulers refined in-house, and no upstream framework absorbs the mistakes. In the short term, this path is unquestionably slower than integrating a mature runtime.

What it buys is long-term ownership of every critical decision.

When a model changes, we know whether to modify its specification, orchestration, or domain capabilities. When a platform changes, we know whether the backend or kernel should change. When performance regresses, we can follow weights, activations, KV, experts, scratch, and synchronization points one by one. When the system expands across machines, it sends layer-boundary activations rather than mixing model-internal state with remote expert RPC on the hot path.

That is the purpose of zLLM: not to become the largest general AI framework with the most abstractions, but to remain a small, direct, native inference engine that truly owns hardware resources and model execution.

We chose Rust because it matches that goal: safe without surrendering low-level control; abstract without mandatory runtime cost; cross-platform without erasing platform differences; capable of writing the code nearest to a kernel and continuing all the way to services and distributed scheduling.

We chose a clean-sheet design because only then can Simple, Short, and Straight shape the entire engine from its first line instead of becoming local patches on an inherited system.