OPEN SOURCE (MIT) · v1.3.5

Agens Engine

Name: Agens Engine
Author: Blockway

The inference engine we built for Agens Volundr. Volundr's full 262K-token context within a 16 GiB GPU memory budget.

OpenAI-compatible API
262K context
16 / 24 / 48 GB profiles
Speculative decoding
Image input
Multi-GPU

Volundr has a 262K-token context window. Retrieval is validated to 107K tokens (needle-in-a-haystack 100/100); the full 262K window has not been re-validated on this checkpoint.

View on GitHub About Agens Volundr

WHERE THE MEMORY GOES

Built around the model, not bolted on

Only 18 of Volundr's 72 layers keep a KV cache; the other 54 hold a fixed-size state. The engine is built around that shape: compute stays on the GPU, lookup tables live in system RAM, and long-context KV moves to host memory instead of growing GPU memory. You set the budget; the engine never overruns it.

GPU

16 GiB budget

Compute weights incl. output layer, mixed 2/3/4-bit~11 GB
KDA states, 54 layersfixed size
Recent KV of the 18 attention layers, drafterin budget

System RAM

32 GB+ for 262K

Embedding table~2.5 GB
Engram n-gram tables~4.3 GB
Long-context KV, tieredup to ~24 GB

16 GB build on the 16gb profile. Sizes are approximate.

WHAT IT DOES

One model, served properly

OpenAI-compatible API

/v1/chat/completions and /v1/completions. Point existing tools and agent frameworks at it.

Mixed 2/3/4-bit weights

Bit width chosen per layer, repacked at load time so the compute weights fit in about 11 GB of GPU memory.

KV cache tiered to host memory

At long context the KV cache moves to pinned system RAM. A follow-up turn reuses the conversation instead of re-reading it.

Speculative decoding

A companion drafter plus the model's own MTP head, for greedy and sampled requests. Outputs are unchanged; drafting only changes speed.

Image input

Send images with the same API. On by default in the 24 GB and 48 GB profiles.

Multi-GPU

Tensor parallel across 2 or 4 identical GPUs in one machine with --tp 2 or --tp 4.

MEASURED

What it does on a 16 GiB budget

Measured on one 48 GB GPU (compute capability 8.9) held to a 16 GiB budget by the engine's hard cap, single conversation. Not measured on a physical 16 GB card.

98–150

tok/s, greedy, short context, with drafter

67–93

tok/s, greedy, after a 219K-token document

108–155

tok/s, greedy, short context, drafter + MTP

93–136

tok/s, sampled (T 0.6), short context, drafter + MTP

15.5 GiB

peak GPU memory with the conversation at 239K tokens

239K

tokens in one conversation, inside the 16 GiB cap

From the community

Community contributor @mmssix ran the 16 GB build on the 24gb profile on a 24 GB card (compute capability 8.6), one stream, thinking off, greedy:

Engine 1.1.0, drafter

chat 65.0 · code 52.9 · reasoning 79.0 tok/s

Engine 1.3.0, drafter + MTP

chat 70.7 · code 59.4 · reasoning 93.5 tok/s

Follow-up turn on a 54K-token conversation

5.7 s with a session ID, 77 s without

And at the other end

Agens Engine 1.3.5 on two 141 GB GPUs (compute capability 9.0), --tp 2, Volundr INT4, server mode, --gpu-gib 128 --max-batch 48 --max-convs 48 --vision 1, with --mtp and MTP head v1.1 (Blockway/Agens-Volundr-32B-v1-MTP).

~375

tok/s, greedy, one request, short prompt (~320 at a 6K-token prompt, ~327 at 22K)

1,857

tok/s aggregate, 48 concurrent requests

failed requests across 48 concurrent ~30K-token agent sessions (1.1 s median time to first token)

1,329 / 1,620

tok/s aggregate at 16 / 32 concurrent requests, default sampling

~7,000

requests in 3 hours of mixed traffic (chat, images, documents, agent sessions), none failed

0.49 s

median time to first token for short chats during that run

QUICKSTART

Running in two commands

Download the model and drafter from Hugging Face, then start the server with a profile that matches your GPU.

# 16 GB GPU: 262K context, one conversation, speculative decoding
scripts/agens-run.sh --profile 16gb --model /path/to/model --drafter /path/to/drafter

# the same with the MTP head as a second draft source (context ~237K)
scripts/agens-run.sh --profile 16gb --model /path/to/model --drafter /path/to/drafter --mtp /path/to/mtp

# then, from another shell
curl http://localhost:30000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model": "agens", "messages": [{"role": "user", "content": "Hello!"}]}'

Or with Docker directly

docker run --gpus device=0 --ipc=host -p 30000:30000 \
  -v /path/to/model:/model:ro -v /path/to/drafter:/drafter:ro \
  ghcr.io/blockwayz/agens-engine:1.3.5 --profile 16gb --model /model --drafter /drafter

Profiles

Profile	GPU budget	Context	Mode
16gb	16 GiB	262K (225K with images)	one conversation, images off by default
24gb	23 GiB	262K	server, up to 4 concurrent requests
48gb	46 GiB	262K	server, up to 16 concurrent requests

Add --mtp to any profile (16gb context: ~237K, 237,572 measured). Use --tp 2 or --tp 4 for 2 or 4 identical GPUs.

REQUIREMENTS

What you need

Operating system

Linux with Docker, the NVIDIA container toolkit and a CUDA 12.9-capable driver

GPU memory

16 GB, 24 GB or 48 GB

Prebuilt kernels

compute capability 8.6, 8.9, 9.0 and 12.0. Validated: 8.9 and 9.0 by us, 8.6 by a community contributor. 12.0 is built, not validated by us