Kindi is a custom inference engine for Prism ML's Ternary-Bonsai-2 language models (2B, 4B, 9B and 27B parameters, ternary weights). It started on the NVIDIA Quadro K420 (Kepler, 1 GB), where it runs the 2B at 20-23 tokens per second. Version 0.2.0 adds the Quadro K1200, two cards in tensor parallel, the GeForce GTX 1080, the GeForce RTX 3080, a CPU engine and a one-command installer. It has an OpenAI-compatible server for chat clients, scripts and agents that write code. Its default output is token-for-token identical to the output of llama.cpp for the same model file. On the K420, with the same file and the same context, it generates about 13x faster than llama.cpp. The comparisons for each card are in docs/BENCHMARKS.md. The one-page summary is WHY.md.

_{Recorded live: the pi coding agent (with a Kindi colour theme) connected to kindi serve on one Quadro K420, default settings. Token streaming is real time. We shortened pauses longer than 2.5 s. The full video (123 s, no pauses shortened) and its asciicast are in the 0.1.0 release.}

By Amir Husain. Watch the launch video on YouTube.

On Linux x86-64 with an NVIDIA card (Kepler to Ampere, one or two cards) or with no GPU:

curl -fsSL https://amirhusain.com/kindi/install.sh | bash

The script finds your cards and selects the model and the settings for them. It downloads the model and checks its sha256, converts it, checks that the output is token-for-token identical to llama.cpp, and starts the server on 127.0.0.1:8093. It explains each step, and it asks before it installs Docker or the NVIDIA Container Toolkit. To see each step before anything changes, add -s -- --dry-run after bash. Run it again to upgrade or repair; --uninstall removes it. To check the script's sha256 before you run it, and for the model that each card gets, see docs/INSTALL.md.

• Quick start: one command
• Requirements
• Install - with Docker, or natively
• Get the model and convert it
• Run the server
• Use it - curl, Python, streaming, tools, agents
• Exact and fast output heads
• The prompt cache
• Commands
• Configuration
• Performance
• Agent benchmark
• Limits
• Troubleshooting
• License
• Acknowledgements

docs/CARDS.md gives the model and the settings for each card. The card must be free: Kindi fits the context to the free card memory. On the K420 1 GB, Kindi uses about 870 MB of its 980 MB with the default 24,576-token context. A desktop session on the same card takes a few MB and is fine.

CUDA 10.2 and CuPy 9.6 are the last releases that support Kepler, and they need an old userland; the image packages exactly that.

git clone https://github.com/ahtx/kindi.git && cd kindi
docker build -t kindi:0.2.0 -f docker/Dockerfile .   # downloads the CUDA 10.2 toolkit (~2.5 GB), ~15 minutes

You can also use the release archive. Download kindi-0.2.0.tar.gz from the v0.2.0 release, then run tar xzf kindi-0.2.0.tar.gz && cd kindi-0.2.0 and build the image with the same docker build command.

Check that Docker sees the GPU (create ~/.kindi first, so it belongs to you and not to root):

mkdir -p ~/.kindi
docker run --rm --gpus all -u $(id -u):$(id -g) -v ~/.kindi:/data kindi:0.2.0 info

More in docs/DOCKER.md.

You need CUDA 10.2 (toolkit with NVRTC and cuBLAS), Python 3.6 or later that CuPy 9.6 supports, and:

pip install "numpy==1.19.5" "fastrlock==0.8.3" "cupy-cuda102==9.6.0" "fastapi==0.79.1" "uvicorn==0.16.0" "pydantic==1.9.2"

Then run Kindi from the repository root with python3 -m kindi ... (put the repository on PYTHONPATH to run it from elsewhere).

These steps use the 2B. For the 4B, 9B and 27B, use their PTQ1_0 files from the same publisher. The installer selects and downloads the right one; docs/INSTALL.md lists the files and their sha256. Download Prism ML's PTQ1_0 file (430 MB, Apache 2.0) from Hugging Face:

mkdir -p ~/.kindi && cd ~/.kindi
wget https://huggingface.co/prism-ml/Ternary-Bonsai-2-2B-gguf/resolve/main/Ternary-Bonsai-2-2B-PTQ1_0.gguf

Convert it once into Kindi's weight pack (about a minute; exact - the same weights, re-laid out):

# Docker
docker run --rm -u $(id -u):$(id -g) -v ~/.kindi:/data kindi:0.2.0 convert /data/Ternary-Bonsai-2-2B-PTQ1_0.gguf
# native
python3 -m kindi convert ~/.kindi/Ternary-Bonsai-2-2B-PTQ1_0.gguf          # writes ~/.kindi/pack

Check the engine against llama.cpp's reference output:

docker run --rm --gpus all -u $(id -u):$(id -g) -v ~/.kindi:/data kindi:0.2.0 verify

prompt 1 'The sky is blue because'          PASS  40/40 tokens match
prompt 2 'The capital of France is'         PASS  40/40 tokens match
prompt 3 'Water boils at a lower temper...' PASS  40/40 tokens match
prompt 4 'def add_two_numbers('             PASS  40/40 tokens match
ALL PASS: token-exact with llama.cpp - 22.1 tokens/s over the run

The speed in the last line includes the prompt tokens, so it is not the generation speed. It also depends on the fast-head file of the next step. Without this file, verify uses a plain layout of the exact head and shows about 20 tokens/s. With the file, verify shows about 22 tokens/s. In both cases, the output is token-exact. kindi bench measures the generation speed (see Performance).

Optional - build the cluster file for the fast output head (about 2 minutes):

docker run --rm --gpus all -u $(id -u):$(id -g) -v ~/.kindi:/data kindi:0.2.0 fast-head

docker run -d --name kindi --restart unless-stopped --gpus all -u $(id -u):$(id -g) \
    -p 127.0.0.1:8093:8093 -v ~/.kindi:/data kindi:0.2.0
# native:  python3 -m kindi serve

The first start compiles Kindi's CUDA kernels for your card (about 1.5 minutes on the K420). Kindi caches them under state_dir, and later starts take about 10 seconds. Then:

curl -s localhost:8093/health

The server listens on 127.0.0.1:8093 (natively) or on the port you publish (Docker). To serve your network, publish -p 8093:8093 (Docker) or set host = 0.0.0.0, and set an api_key.

The endpoint is http://localhost:8093/v1, the model id bonsai2-2b (configurable).

curl -s localhost:8093/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "bonsai2-2b",
  "messages": [{"role": "user", "content": "Write a bash one-liner that counts the .txt files here."}]
}'

Python (pip install openai):

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8093/v1", api_key="none")   # your api_key if you set one
r = client.chat.completions.create(model="bonsai2-2b",
                                   messages=[{"role": "user", "content": "What is the capital of Italy?"}])
print(r.choices[0].message.content)

Kindi supports these request fields: messages (system / user / assistant / tool, text content), tools and tool_choice ("none" disables tools), and stream and stream_options.include_usage. It also supports temperature (0 = greedy), top_p, top_k, max_tokens / max_completion_tokens, stop and seed. It also supports chat_template_kwargs.enable_thinking (off by default). When thinking is on, the thinking comes back as reasoning_content (also streamed as delta.reasoning_content). thinking_budget_tokens caps the thinking. At the budget, the server closes the thinking and the model then gives its answer. Responses carry OpenAI usage (with prompt_tokens_details.cached_tokens) and a timing block.

Tool calls come back as OpenAI tool_calls with finish_reason: "tool_calls"; send the results back as role: "tool" messages.

Agents and chat UIs. Point any OpenAI-compatible client at http://localhost:8093/v1 with model bonsai2-2b. Two settings matter on the smaller cards - see docs/AGENTS.md:

• Give the client a long request timeout. An agent's first prompt of 5,000+ tokens takes a few minutes to read the first time. The cache makes later turns fast. Do not use automatic retries, because they queue behind the current request.
• Tell the client the context window that /health shows (on the K420 1 GB: 24,576 tokens).

kindi chat is a terminal chat that needs no server.

The output layer scores all 248,320 vocabulary entries for every generated token - about 9 ms of a 46 ms step.

• exact (default): the full head. Greedy output is token-for-token identical to llama.cpp.
• fast: a shortlist head. kindi fast-head clusters the vocabulary once. For each token, Kindi scores the 1,024 cluster centres. Then it computes exact scores only for the 256 clusters with the best scores (~13 % of the vocabulary). Generation is about 15 % faster. On 310 measured steps, it chose the same token as the full head 99.4 % of the time.

To change the head while the server runs, with no restart, use kindi head fast / kindi head exact (Docker: docker exec kindi python3 -m kindi head fast). /health shows the current mode.

Bonsai mixes attention with recurrent Gated-DeltaNet layers, whose state cannot be rewound. Kindi caches anyway, exactly:

• checkpoints of the recurrent state every 256 tokens. A request that shares a start with the previous request resumes at the last checkpoint before the first difference.
• parked conversations: Kindi keeps up to 3 other conversations in host RAM. It resumes each conversation when its next turn comes. (Agents alternate between their main prompt and side prompts.)
• saved openings on disk. When two requests share an opening of 512+ tokens and then differ, Kindi saves the state of that opening under state_dir. Typically, this opening is a system prompt with tool definitions. Later sessions load it in a fraction of a second, also after restarts.

Resumed output is identical to the output when Kindi reads the prompt from scratch.

Global options: --config FILE, --pack DIR, --state-dir DIR, --ctx N.

Defaults, then a config file (--config, $KINDI_CONFIG, ./kindi.conf or ~/.kindi/kindi.conf), then environment variables KINDI_<KEY>, then command-line flags. Copy kindi.conf.example and edit it. The file docs/CONFIGURATION.md describes the keys: paths, server address, API key, context length, head mode, cache sizes and generation defaults.

Quadro K420, Ternary-Bonsai-2-2B, kindi bench (dev build de83d2f, blocking sync off, the installer's setting):

Prompt reading on the same card (Kindi 0.2.0, exact head): 33.3 tokens/s at 512 tokens, 24.2 at 24,000.

The other cards, at a short context, exact head:

llama.cpp on the same card, with the same model file and the same context: on the K420, it generates 1.65 tokens/s and reads prompts at 1.2 tokens/s (512 tokens). The tables for each card, with their method, are in docs/BENCHMARKS.md. How Kindi works: docs/HOW-IT-WORKS.md. The settings for each tested card (K420, K1200, GTX 1080, RTX 3080, CPU): docs/CARDS.md.

kbench has 65 agent tasks with hidden tests and a runner. It also has the results of our runs with Kindi on the K420. kbench works with any OpenAI-compatible server.

• One request at a time. Concurrent requests queue (the card holds one engine).
• On the K420, Kindi reads prompts at about 25-33 tokens/s, so a long new prompt takes time. A new prompt of 5,000 tokens takes about 3 minutes there. The cache makes repeated prompts fast. Kindi reads text that is not in the cache (for example, a large file or long command output) at this speed.
• The Ternary-Bonsai-2 models only (the PTQ1_0 GGUFs: 2B, 4B, 9B, 27B). The 27B needs 8 GB of card memory (GTX 1080, RTX 3080) or two 4 GB cards (2x K1200).
• At most two GPUs per model. Two cards run one model in tensor parallel. More cards run more servers.
• Small models make mistakes. The 2B is good at short code, shell and factual questions, tool calls and summaries. Check what it writes.

See docs/TROUBLESHOOTING.md. The usual causes are:

• Another program uses the card (out of memory at start).
• On Kepler, the driver is not the 470 branch.
• A client timed out while it waited for a long first prompt.

Amir Husain. The Deep Tech with Amir series on VectorFall covers this work. The Kindi launch video shows Kindi on the K420. More broadly, it also shows how to get the most from inexpensive GPUs.

AGPL-3.0; commercial licenses available.

Kindi is free software under the GNU Affero General Public License v3.0. If you distribute Kindi or a changed version, the AGPL says that you must give the source code to the recipients. This is also true when users use your version over a network. For use in a closed product or a hosted service, a paid commercial license is available. See COMMERCIAL-LICENSE.md.

The model is Prism ML's Ternary-Bonsai-2-2B under Apache 2.0. You download it from its publisher. The third-party notices are in NOTICE.

Kindi exists because of the model it runs. Thank you to Prism ML for Ternary-Bonsai-2-2B and for its release under Apache 2.0. Thank you to the Qwen team at Alibaba Cloud for Qwen3.5-2B, the base model of Ternary-Bonsai-2-2B. Thank you also to the llama.cpp authors. Their implementation is the reference that we check Kindi's output against.