Agens Engine is open source: Volundr's context window in a 16 GiB GPU budget

Why we wrote our own engine

Agens Volundr is shaped differently from most models: only 18 of its 72 layers keep a KV cache, while the other 54 hold a fixed-size state. General-purpose servers treat that as a special case. We wanted an engine built around it, so long agent sessions could stay on a single GPU people actually own.

Where the memory goes

On the 16gb profile, the compute weights, output layer included (mixed 2/3/4-bit, chosen per layer), stay on the GPU at about 11 GB. The embedding table and the Engram lookup tables, which are read only a few rows per token, live in system RAM. At long context the KV cache is tiered to pinned host memory, up to about 24 GB at the full window, instead of growing GPU memory. You choose the budget; the engine never overruns it.

What it does

Speculative decoding uses a companion drafter plus the model's own MTP head, for greedy and sampled requests (temperature, top-k, top-p, min-p). Outputs are unchanged; drafting only changes speed. Images go in through the same API. Two or four identical GPUs in one machine work with --tp 2 or --tp 4. With a session ID, a follow-up turn reuses the conversation instead of re-reading it.

Since 1.0 we have added a prefix cache, reproducible sampling (the same tokens alone or batched, with or without speculation), server mode with batching up to 64 requests, and in 1.3.5 a KV pool that sizes itself and fairer scheduling for long agent sessions.

What it measures

All of the following were measured on one 48 GB GPU (compute capability 8.9) held to a 16 GiB budget by the engine's hard cap, single conversation, not on a physical 16 GB card. With the drafter, greedy decoding runs at 98–150 tok/s at short context and 67–93 tok/s after a 219K-token document. With the drafter and MTP: 108–155 tok/s greedy and 93–136 tok/s sampled (T 0.6) at short context. One conversation reached 239K tokens with a 15.5 GiB peak.

The first numbers on a physical card came from the community. Contributor @mmssix ran the 16 GB build on the 24gb profile on a 24 GB card (compute capability 8.6), one stream, thinking off, greedy: 65.0 / 52.9 / 79.0 tok/s for chat, code and reasoning on Engine 1.1.0 with the drafter, and 70.7 / 59.4 / 93.5 tok/s on 1.3.0 with drafter and MTP. A follow-up turn on a 54K-token conversation took 5.7 s with a session ID, against 77 s without.

At the other end, on two 141 GB GPUs (compute capability 9.0) with --tp 2, Volundr INT4, server mode (--gpu-gib 128 --max-batch 48 --max-convs 48 --vision 1) and MTP: with the updated MTP head v1.1 (Blockway/Agens-Volundr-32B-v1-MTP), one greedy request decodes at about 375 tok/s on a short prompt and about 327 tok/s at a 22K-token prompt; aggregate throughput is 1,329 / 1,620 / 1,857 tok/s aggregate at 16 / 32 / 48 concurrent requests. 1.3.5 is about long agent sessions under load: with 48 concurrent ~30K-token sessions, failed requests went from 14 on 1.3.4 to 0, aggregate decode from 66 to 144 tok/s, and the median time to first token from 25.5 s to 1.1 s. In three hours of mixed traffic (chat, images, documents and agent sessions, about 7,000 requests) nothing failed and GPU memory stayed flat.

Limits, honestly

Agens Engine runs on Linux with Docker and the NVIDIA container toolkit; Windows and macOS are not supported; for Mac users there is a community MLX port, volundr-mlx (MIT), linked from our README. Kernels are prebuilt for compute capability 8.6, 8.9, 9.0 and 12.0: 8.9 and 9.0 are validated by us, 8.6 by a community contributor, and 12.0 is built but not validated by us. The full context needs 32 GB or more of system RAM.

With thinking on over long documents, the 16 GB build can loop; for long documents turn thinking off or use the 4-bit build. Volundr's retrieval is validated to 107K tokens; the full 262K window has not been re-validated on this checkpoint.

What's next

Windows support, starting with WSL2, and 12 GB GPU support are in development. Upstream, the Volundr model PR is open in SGLang, and we have committed to upstreaming the architecture to llama.cpp as small PRs. If you run the engine on your own card, please tell us how it goes: every report makes the next release better.