MacBook Air M5: Run a Multimodal LLM with Zero Runtime Dependencies

zLLM requires no Python, PyTorch, Conda, Docker, or resident model server. End users receive one native executable and model weights; Rust developers can put the same engine directly inside their own process.

On a 10-core MacBook Air M5 with 24 GB of unified memory, the current release build of zllm-metal is 8.1 MB. Add the Q4_K_M Gemma 4 E4B model and its vision projector, and this small program can run multi-turn chat and image understanding directly through Metal.

“Zero dependencies” here means zero external framework dependencies at deployment and runtime. There is no Python environment, PyTorch dynamic library, or separately managed C++ inference runtime. Model weights remain required data, and source builds naturally compile zLLM's Rust crates and macOS system bindings through Cargo.

Two interfaces, one inference path

zLLM offers both a library and a command-line interface. They share the same model Runtime, Metal Backend, Kernels, weight loading, KV cache, and generation state machine.

Interface	Best for	Starts a service?
zllm::embedded::Engine	Rust desktop apps, tools, and services	No; it runs in-process
zllm-metal	Immediate local chat and image understanding	No; it is one native CLI

Library mode is not an HTTP client wrapper. Engine owns the model and KV cache directly. Requests use familiar OpenAI Chat JSON semantics, while generated tokens arrive through a Rust callback.

Embed zLLM in a Rust program

The zLLM source is now public on GitHub. Use the Git dependency directly, or clone it and switch to a local path during development.

Add zLLM to Cargo.toml with the source path available on your machine:

[dependencies]
zllm = { git = "https://github.com/zllm-lab/zllm" }
serde_json = "1"

Load the engine once and call generate repeatedly:

use std::io::{self, Write};
use serde_json::json;
use zllm::embedded::Engine;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut engine = Engine::from_config("gemma4-metal.yaml")?;
    let cancellation = engine.cancellation();

    let result = engine.generate(
        &json!({
            "model": "gemma4",
            "messages": [{
                "role": "user",
                "content": "Explain in one sentence why an inference engine belongs in-process"
            }],
            "max_completion_tokens": 128
        }),
        &cancellation,
        |_token, text| {
            print!("{text}");
            io::stdout().flush().is_ok()
        },
    )?;

    println!("\nfinish={} tokens={}",
        result.finish_reason, result.completion_tokens);
    Ok(())
}

Return false from the callback or call cancellation.cancel() from another thread to stop generation promptly. GenerationResult also contains token counts, the finish reason, and a cache_id for subsequent turns. Application code never manages Metal command buffers or platform KV objects.

Image input uses the same interface with ordered multimodal content parts:

let cancellation = engine.cancellation();
let result = engine.generate(
    &json!({
        "model": "gemma4",
        "messages": [{
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image"},
                {"type": "image_url", "image_url": {
                    "url": "/absolute/path/to/photo.png"
                }}
            ]
        }],
        "max_completion_tokens": 256
    }),
    &cancellation,
    |_token, text| {
        print!("{text}");
        true
    },
)?;

Local paths, HTTP(S) URLs, and data: URLs share the same image materialization and vision-encoding path. Gemma 4 E4B's mmproj loads lazily on the first image request, so text-only startup does not pay the vision-tower cost.

Get Gemma 4 E4B

This guide uses the instruction-tuned Gemma 4 E4B release:

• Hugging Face: unsloth/gemma-4-E4B-it-GGUF
• ModelScope: unsloth/gemma-4-E4B-it-GGUF

Download these two files into the same directory:

models/gemma-4-E4B-it-GGUF/
├── gemma-4-E4B-it-Q4_K_M.gguf   # main model, about 4.98 GB
└── mmproj-F16.gguf               # vision encoder/projector, about 990 MB

With the Hugging Face CLI:

hf download unsloth/gemma-4-E4B-it-GGUF \
  gemma-4-E4B-it-Q4_K_M.gguf mmproj-F16.gguf \
  --local-dir ./models/gemma-4-E4B-it-GGUF

The ModelScope page above exposes the same filenames for networks where it is faster. Q4_K_M is a balanced choice for this 24 GB MacBook Air; retaining the roughly 1 GB F16 vision projector favors image-understanding quality.

The minimal gemma4-metal.yaml configuration is:

version: 1
kind: standalone
http:
  listen: 127.0.0.1:8000
  public_base_url: http://127.0.0.1:8000
artifacts:
  directory: ./artifacts
node:
  cache_directory: ./cache/gemma4
  persist_kv_cache: false
  max_concurrency: 1
model:
  architecture: gemma4
  weights_directory: ./models/gemma-4-E4B-it-GGUF/gemma-4-E4B-it-Q4_K_M.gguf
  lm_head_quantization: native
  max_sequence_length: 49152
  execution:
    prefill_chunk_size: 2048
backend:
  kind: metal
  device: default

An 8 MB Metal command-line program

For immediate use, no configuration file or server is needed. Clone the public repository and build the native CLI:

git clone https://github.com/zllm-lab/zllm.git
cd zllm
cargo build --release --bin zllm-metal

Then pass the model directly to the release executable:

./target/release/zllm-metal \
  ./models/gemma-4-E4B-it-GGUF/gemma-4-E4B-it-Q4_K_M.gguf

zllm-metal automatically:

1. detects Gemma 4 from GGUF metadata;
2. discovers a sibling mmproj-*.gguf;
3. enables MTP speculative decoding when a matching sibling mtp-*.gguf exists;
4. derives a KV budget from unified memory, weight size, and a safety margin;
5. loads the Metal Runtime, enters multi-turn chat, and reuses the terminal KV cache.

Type normally to chat:

> Introduce yourself in one sentence and say that you are running locally on this Mac.

For a local image, put the question and path in the same message. Bare, quoted, backtick-wrapped, and space-containing paths are supported:

> Describe this image in two sentences `/Users/me/Pictures/demo.png`

Or copy an image and use:

> /paste Describe the image on the clipboard

Useful commands are:

/stats    Show context and KV memory
/reset    Clear the conversation
/compact  Summarize older history
/paste    Read a PNG image from the macOS clipboard
/exit     Exit

Demo on real hardware

The interaction video below uses commands, startup figures, token counts, and response text from a real run on the same MacBook Air M5. The 8.1 MB release executable selected a 49,152-token context; repeated warm-file-cache starts took 0.9–1.6 seconds. Three cold-start text runs averaged 40.47 token/s over tg50, while the measured 323-token image request took 6.469 seconds for prefill and 7.312 seconds to first token. Results vary with model revision, context length, sampling, file cache state, and thermals.

The 23-second video validates the complete loading, text-chat, and vision-input path. The terminal output is line-wrapped and paced for the frame; it is not a model-quality evaluation.

The important boundary

zLLM is not replacing a Python environment with another heavyweight runtime. It makes model execution part of the application: one Rust object, one model, and one native backend. The application owns its threads, cancellation, cache, and lifetime instead of delegating its core capability to another process.

That turns “add local AI to an existing Rust program” from a deployment project into an ordinary library call—and gives an executable measured in megabytes complete multimodal inference on a Mac.