OPEN MODEL · PREVIEW · COMING SOON
Long context, without the memory bill. Only 18 of its 72 layers keep a KV cache. A dense 32B open model built for tool use, coding and long working sessions, on machines a company can actually own.
THE PROBLEM
In a conventional model, every attention layer keeps a KV cache: a copy of what it has seen, which grows with every token. Stack seventy-two of them and, at long context, the cache, not the weights, decides what fits on your machines.
Volundr changes the shape of that curve. 54 layers hold a fixed-size state. Only 18 keep a cache.
FIVE IDEAS, ONE MODEL
KDA · 54 LAYERS
Kimi Delta Attention is linear attention: instead of keeping every past token, each layer folds the conversation into a recurrent state of fixed size.
Why it matters: Three quarters of the model carry the whole session forward in the same amount of memory on turn 5 as on turn 500. That's what makes long agent sessions practical on hardware you can buy.
TOKENS IN
BCSA · 17 LAYERS
Blockway Compressed-Sparse Attention reads the latest 4,096 tokens exactly. Everything older is pooled 4:1 into blocks, and a learned indexer picks the 512 most relevant blocks to read.
Why it matters: Every token reads a fixed budget: the recent past in full detail, plus the parts of the distant past that matter. The model can look back across the whole window without paying to look at all of it.
FAR FIELD · POOLED 4:1 INTO BLOCKS
indexer picks the top 512 blocks
EXACT WINDOW
last 4,096 tokens, every one
DENSE · 1 LAYER
A single layer of classic full attention closes the stack: here, every token can see every other token.
Why it matters: Fifty-four layers stay lean and seventeen stay selective, and one keeps the complete picture.
ENGRAM · 2 LAYERS
Engram is a hashed n-gram memory, a large lookup of patterns, and it lives in host RAM rather than on the accelerator.
Why it matters: GPU memory is the scarcest resource in a self-hosted setup. Engram's capacity is paid for in ordinary system RAM, which is far cheaper and far larger.
GPU MEMORY
model weights
and caches
HOST RAM
Engram
n-gram memory
mHC · 4 STREAMS
A conventional transformer passes information between layers along one residual stream. Volundr uses four in parallel.
Why it matters: More room to carry information through 72 layers of depth, so the model doesn't have to squeeze everything into a single path.
DENSE, NOT MIXTURE-OF-EXPERTS
About 32B parameters, all of them doing work on every token. No hidden expert pool to keep resident, so the footprint is exactly what it says on the label, and capacity planning for your own machines stays simple.
AT LAUNCH
BF16
full precision
Runs on
two 48 GB GPUs, or one H200
Blockway/Agens-Volundr-32B-Preview
INT4
31.7 GiB
Runs on
one 48 GB GPU
Blockway/Agens-Volundr-32B-Preview-INT4
262K-token context
retrieval verified to 127K tokens
Apache-2.0 open weights
on Hugging Face, for commercial use
Serving
Blockway's sglang build at launch; stock sglang and vLLM can't load it yet
GGUF / llama.cpp
planned after launch
Model card and evaluation details are published with the weights.
Agens Volundr 32B Preview is coming soon. Follow along and be first to get the weights.
