Graph-Based Captioning (GBC) as one KIR program on the Togeari dataflow machine, with the engines that run it: a VLM over any OpenAI-compatible or Anthropic API, vLLM in-process or a specialised Qwen3.5 engine; OWLv2 or SAM 3 for detection; EmbeddingGemma for names; on one GPU, many GPUs or other nodes.

Producing one GBC graph is a branching process: ask the VLM, detect what it named, describe each region, detect its parts, merge regions that are the same object, ask relational questions. Its control flow differs per image and cannot be batched across images by hand. KohakuGBC writes that process once, as a sequential KIR program; Togeari runs many images at once, overlaps every call the sequential run cannot tell apart, and turns their model calls into batched engine calls.

Contents: Highlights · Install · Quick start · How it works · Configuration · Output · Faithful to GBC · Measured · Repository layout · Documentation · Checks

• 2.06 images/s on one RTX 5090, 8.17 on four. Qwen3.5-4B, SAM 3 and EmbeddingGemma on each card, about 15 vertices and 27 captions per graph; each image is 41 s in the machine at the median, however long the source runs. Four cards give 3.96x one card (measured).
• Engines in their own processes. Every card's engines run in an engine host process; the machine's process holds connectors and the dispatcher only, over shared memory. That is +9% over the same engines in the machine's process, and the same connector reaches engines on other nodes.
• vLLM 3.1x faster on this workload. KohakuvLLM (a drop-in vLLM plugin) takes vllm serve from 11.4 to 35.0 requests/s on one RTX 5090: FP8 GDN projections, a bf16 or INT8 GDN state, a Rust JSON grammar, and a phased scheduler. Every reply is schema-valid.
• GBC at scale, with the models you choose. Thousands to millions of images through a hosted API, your own vLLM or SGLang servers, or engines on your GPUs; engines on other nodes join the same lanes.
• Exactly the sequential procedure. Every job can be recorded and replayed through KIR's reference interpreter. On recorded runs of GBC's proof of concept, every graph equals the PoC's to the digest.
• The procedure as data. The root (an image query, existing captions, tags), the node budget, the fan-out and the relational questions are per-image controls; the detector is one setting.
• One engine contract. Every engine, from an HTTP server to the qwen35 engine written for this workload, sits behind Togeari's contract and is checked against it.

pip install kohakugbc-data            # graphs and datasets only: no machine, no GPU stack
pip install kohakugbc                 # program, machine, API and replay engines
pip install "kohakugbc[torch]"        # detector (OWLv2, SAM 3) and embedder on torch
pip install "kohakugbc[vllm]"         # the VLM in-process through vLLM (Linux)
pip install "kohakuvllm @ git+https://github.com/KohakuBlueleaf/KohakuvLLM"   # our vLLM parts
pip install "kohakugbc[qwen35]"       # the qwen35 VLM engine (Windows and Linux)

Two packages from this repo:

• kohakugbc-data (packages/kohakugbc-data, on PyPI): the GBC graph (store, resolution, prompt rendering) and the GBC dataset format, with its Rust extension; depends on numpy, pillow, pyarrow and kohakuvault only. Training code and dataset tools use this alone.
• kohakugbc (pure Python): the program, machine and engines, on top of kohakugbc-data.

From a clone, with a Rust toolchain:

git clone https://github.com/KohakuBlueleaf/KohakuGBC && cd KohakuGBC
uv venv -p 3.13
uv pip install -e packages/kohakugbc-data -e ".[torch,qwen35,dev]"   # torch: the index matching your driver
gbc doctor

gbc doctor prints versions, devices with their free memory, every entry per slot and, given configuration layers, their placement plan and whether it fits.

On one GPU, the VLM, the detector and the embedder in this process (configs/machine/ qwen35_local.py: Qwen3.5-4B on the qwen35 engine, OWLv2, EmbeddingGemma on the CPU; 24 GiB):

GBC_IMAGES=path/to/images gbc run configs/machine/qwen35_local.py
# SAM 3 as the detector (illustrations):
GBC_IMAGES=path/to/images gbc run configs/machine/qwen35_local.py configs/machine/qwen35_sam3_local.py

One graph per image lands in runs/machine/qwen35_local/graphs.jsonl; the teaser above is the graph of its image (29 vertices, the figure shows 20; 51 edges, 48 captions), drawn by scripts/docs/teaser.py:

{"vertices":[{"vertex_id":"","bbox":{"left":0.0,"top":0.0,"right":1.0,"bottom":1.0,"confidence"
:null},"label":"image","descs":[{"text":"An anime-style illustration depicts a young girl with
blonde hair and fox-like ears sitting on a weathered wooden window sil ...

Over an API, the VLM on a server (vLLM, SGLang, llama.cpp, OpenAI, OpenRouter, Anthropic), the vision models here:

# my_run.py
engines = {
    "vlm": {
        "entry": "api",
        "protocol": "openai",
        "flavour": "vllm",
        "model": "Qwen/Qwen3.5-4B",
        "endpoints": [{"base_url": "http://10.0.0.5:8000/v1", "max_inflight": 64}],
        "metrics": "vllm",
    },
    "detect": {"entry": "torch", "model": "facebook/sam3", "dtype": "bfloat16"},
    "embed": {"entry": "torch", "devices": ["cpu"]},
}
source = {"kind": "folder", "path": "images"}
sink = {"kind": "graphs", "path": "runs/my_run/graphs.jsonl"}
machine = {"report": "runs/my_run/report"}

gbc check my_run.py      # build the machine without starting a unit; every problem at once
gbc run my_run.py

Without models, the recorded PoC runs replayed: the program and the machine on pseudo engines, every graph equal to the PoC's.

gbc run configs/machine/replay_text.py

From Python:

from kohakugbc.machine import build

machine, source, sink = build(
    "configs/machine/qwen35_local.py", {"source": {"kind": "folder", "path": "images"}}
)
with machine:
    machine.run(source, sink)

A GBC graph describes one image as a DAG. The image vertex carries a short and a detailed caption; the objects those captions name are entity vertices with their own captions and boxes; their parts hang below them; composition vertices arrange several instances of one object; relation vertices describe how objects relate. Every edge is grounded three ways: it is a graph edge, its label is a phrase of the parent's caption, and the child's box lies inside the parent's.

The root describes the image and finds the objects it names; a round runs every waiting region as its own job (seen before? describe it, find its parts), and the parts it finds wait for the next round. A decision then runs another round one level deeper, asks the VLM what the image still holds and adds those objects, or ends with the relation questions. Every step marked ×? spawns as many jobs as its reply names, and the decision how many rounds run, so each image grows its own workflow while it runs: from 3 VLM requests on one image to more than 30 on another. That work cannot be batched stage by stage across images without waiting for the slowest one; the machine instead runs every job as soon as its inputs exist and batches whatever calls are ready, across images (scripts/docs/workflow.py).

gbc.kir is a plain sequential program; this is its top level:

@def (image, captions = None, tags = None, controls = None)gbc(graph, stats):
    (image, captions, tags, controls)root(records, found)      # the image vertex, what it names
    ...
    decide:
        (records, frontier, rounds, breadth, stop, controls)next_step(step)
        (step)switch("depth"=>`depth`, "breadth"=>`widen`, _=>`done`)
        widen:                                                 # one more root sentence
            (image, records, controls)coverage(records, found, grew)
        depth:                                                 # one round of regions
            (image, frontier, records, seen, controls)depth_round(frontier, records, seen)
        done:
    (image, records, controls)relate(records)                  # relations, compositions
    (image, records, controls)to_gbc(graph, stats)             # records to the GBC graph

Model calls are transport-only nodes (chat, detector, embedder); requests are built and replies parsed by plain nodes, so the same requests and parsing run on every engine.

One job input holds every choice about what the graph contains; a configuration's controls are the defaults under each job's own.

Control presets are configuration layers in configs/controls/: small, mid, large (graph size), and sam3 (SAM 3's selection: box_score 0.3, nms_iou 0.7; the PoC's 0.2 removes overlapping objects, two people embracing at IoU 0.5), stacked after a machine configuration (configs/machine/<m>.py configs/controls/mid.py configs/controls/sam3.py). On the 30 PoC images with SAM 3 (Qwen3.5-4B FP8 on vLLM, one RTX 4090, 64 VLM requests in flight):

flowchart LR
    subgraph proc["machine process"]
        direction LR
        src[("source")] --> adm["admission"] --> jobs["image jobs<br/>gbc.kir, out of order"]
        jobs <-->|"calls"| lanes["lanes<br/>vlm · detect · embed · cpu"]
        lanes <--> conn["connectors<br/>remote units · API units"]
        jobs --> sink[("graphs.jsonl")]
    end
    conn <-->|"shm / inband"| h0["gbc host, card 0<br/>vLLM · SAM 3 · EmbeddingGemma"]
    conn <-->|"shm / inband"| h1["gbc host, card N"]
    conn <-->|"TCP"| far["gbc host on another node"]
    conn <-->|"HTTP"| api["OpenAI-compatible or Anthropic API"]

Togeari runs each image job like a thread on an out-of-order CPU core: statements issue as soon as their inputs exist, a branch still being decided parks while the code after it runs ahead, a parallel or each fans out, and a window retires everything in program order, so a job's result is exactly the sequential run's. Across jobs, every model call of one kind joins one lane: the detect lane batches crops from many images into one forward, the VLM lane keeps its units' queues full.

Each lane's units are connectors. With GBC_HOSTING=process (the default of configs/machine/qwen35_card.py), the machine starts one gbc host per card running that card's engines, and the lane holds a remote unit for each. The dispatcher then has its process's GIL to itself: it does not share it with vLLM's frontend or SAM 3's preparation. Images and tensors travel in shared-memory blocks, everything else as small messages, and capacity and load are reported back at least every 0.2 s (engine hosts). The same engines in the machine's process (GBC_HOSTING=inprocess) remain one setting away.

One run of the machine from its event log (scripts/docs/machine_view.py). It is 240 COCO images on one RTX 5090 with the defaults of configs/machine/qwen35_card.py (engines in a host process, 96 resident images), taking 126 s for 19,849 engine calls (1.91 img/s, loading excluded). Left, each image is a row and each engine call a bar in its lane's colour. The first 96 images are admitted at once, and each later image enters as one finishes, so images overlap on the shared engines. Right, the VLM lane fills to its 384 lanes as the first images fan out, then holds 100–300 requests in flight as images come and go. A single image keeps up to 10 of its own VLM requests in flight at once.

Admission keeps the VLM lane at its target load (vlm_waiting): an image is admitted while the lane's queue and its units' waiting counts are below it, every other lane in waiting is below its limit, and fewer than max_jobs images are resident. max_jobs bounds an image's time in the machine (about max_jobs / throughput) however long the source runs. Each slot's entry is selected once; the placement plan adds up what every engine holds on every device and refuses a configuration that does not fit before any weight loads.

Detection on illustrations. On an anime-style portrait, OWLv2 (the PoC's detector) gives its best "girl" box (0.415) to her legs alone, and the GBC rule for a single object keeps it; her parts are then asked of that crop, where her tail is the only "hair" in sight. SAM 3 boxes the whole girl (0.89), her hair (0.94) and her tail (0.91) apart. The detector is one setting:

engines = {"detect": {"entry": "torch", "model": "facebook/sam3", "dtype": "bfloat16"}}

Engines on other nodes. gbc host serves a slot's units; the machine names them as a remote entry, and its lane batches, retries and fails over as for local units (engine host):

KOHAKUGBC_HOST_KEY=... gbc host my_vlm.py --slot vlm --listen 0.0.0.0:7501

engines = {"vlm": {"entry": "remote", "hosts": ["10.0.0.5:7501/qwen35-0"], "runs": {...}}}

Configuration files are Python files (or dicts); their lower-case module-level names form one layer, and later layers merge over earlier ones key by key, so a run is usually a base file plus a small file of overrides.

Every engine entry, with all its settings: docs/guides/engines.md.

graphs.jsonl holds one GBC graph per line, the JSON ml-gbc writes (vertices with vertex_id, bbox, label image / entity / composition / relation, descs with their kind, in_edges and out_edges labelled by their grounding phrase). Beside it, graphs.failures.jsonl (one line per failed job: key, kind, message) and graphs.stats.jsonl (per job: vertices by kind, edges, captions, depth, leaves). A run report (machine.report) holds the configuration, the environment, per-second metrics of every lane and unit, events, recorded jobs and a summary.

Captioned images ship as a GBC dataset (kohakugbc_data.dataset, in kohakugbc-data): sharded folders of KohakuClip image shards, graphs and per-image metadata in KohakuVault, joined by key (dataset format).

The node library ports GBC's proof of concept (its round engine, JSON reply layer and seeded root) and ml-gbc's parsers, detection post-processing, same-object merge and relational queries. Six PoC arms (text, image and text, tags and image-query roots; entity batch 4; leaf captions from the parent; 30 images each) were recorded as traces of every request and reply.

Where the program differs from the PoC (prompt files read as UTF-8, seeded draws where the PoC's were unseeded, relational questions asked once at the end by default), each difference and why: docs/internals/program.md.

Full tables and methods: docs/performance.

A Linux box with four RTX 5090s; configs/machine/qwen35_card.py with the mid and sam3 control presets on COCO val2017. Qwen3.5-4B (FP8, FP8 GDN projections) runs through vLLM 0.32 with KohakuvLLM's PhasedScheduler, grammar and INT8 GDN state, at 384 lanes and a 12 GiB KV cache. GBC's SAM 3 and EmbeddingGemma share the card. 96 images are resident per card, and job seconds are each image's time from admission to its graph (flagship):

Both one-card runs make the same engine calls: 16.5k VLM, 11.1k detector and 12.1k embedder. The card is power-bound at 98% busy. With the engines in a host, the dispatcher keeps its GIL and the detector's queue drains (61 to 14 requests waiting). One machine over four cards gives 3.96x one card with its job times unchanged. Throughput and job time hold however long the source runs, because admission bounds the resident images.

The VLM behind vllm serve, one RTX 5090, the same workload's 1724 recorded requests (about 1000 prompt tokens, 100 reply tokens, one image and one JSON schema each), 512 in flight, mean of four cards (KohakuvLLM):

In the machine, the grammar is also the fastest way to get structured replies: 1.589 img/s, against 1.539 for prompt + one retry, 1.426 for guidance and 1.226 for xgrammar. On RTX 3090s (SmoothQuant INT8, 6 GiB KV cache, the same configuration), the machine gives 0.630 img/s per card.

An RTX 4090 (24 GiB) on Windows 11, Python 3.13, torch 2.14.

The machine on pseudo engines (scripts/bench/pseudo_machine.py: 120 recorded jobs per configuration, replayed latencies divided by 4). The VLM lane is the limit; the machine keeps it 85–94% of the way to its floor (every VLM latency over the lane's capacity):

The machine on real engines (scripts/bench/real_machine.py configs/machine/qwen35_local.py: the recorded jobs of two PoC arms, 30 images each; the qwen35 VLM, OWLv2 and EmbeddingGemma on the one GPU; wall time includes loading the models):

One portrait image on the same GPU: 34 s with OWLv2, 79 s with transformers' SAM 3 (the device fills with SAM 3 beside the VLM).

The VLM alone (scripts/bench/vlm_units.py: the 425 VLM requests of the text arm, 409 measured, 64 in flight, Qwen3.5-4B, stock vLLM):

GBC's own SAM 3 and EmbeddingGemma against their reference libraries: 45 recorded detector requests through the detect engine and 925 names (six GPUs and the CPU):

flowchart LR
    cli["cli"] --> machine["machine"] --> program["program"] --> nodes["nodes"] --> gbcgraph["graph (Python, Rust)"]
    machine --> togeari["Togeari"]
    machine --> engines["engines<br/>vlm · torch · host · replay"] --> models["kohakugbc_models<br/>detect · embed · vlm · kernels"]
    engines --> kvllm["KohakuvLLM"]
    program --> kir["KohakuNodeIR"]

src/kohakugbc/
  program/        gbc.kir, controls, the reference run
  nodes/          the node library gbc: root, regions, relations, specs (chat, detect, parse)
  graph/          records, resolution to GBC graphs, digests
  machine/        build, configuration layers, the machine module
  engines/        registry, placement, adapters, vlm/ (api, vLLM in-process, qwen35), torch/,
                  host/, replay/
  io/             sources (folder, jsonl, trace), sinks (graphs, memory)
  cli/            gbc
src/kohakugbc_models/     the models GBC runs itself, standalone of the program and machine
  detect/         GBC's own SAM 3 (vision, fusion, masks, memory plan), mask boxes
  embed/          GBC's own EmbeddingGemma over packed token streams
  vlm/qwen35/     the Qwen3.5 engine: forward, ops (Triton, FLA), pools, grammar, graphs, loop
  kernels/        op registry, per-card tuning, CUDA graphs, fp8; backends eager / compile /
                  triton / tilelang (attention, varlen, box bias, mask box, fp8 producers)
  numerics/       error profiles against an fp64 ruler
packages/kohakugbc-data/  the data package (PyPI kohakugbc-data)
  rust/           kohakugbc_data._rs: graph resolution, the graph store, prompts
  src/kohakugbc_data/     prompt/ (renderers), dataset/ (the GBC dataset format)
configs/              machine and engine configurations
scripts/              trace recording, benchmarks, docs figures; ci/ (versions, release notes)
docs/                 concepts, guides, internals, performance
tests/                data and format checks: program, graph, engines; fixtures: recorded PoC traces

python -m pytest tests -q                             # data and format checks, CPU only
python scripts/bench/pseudo_machine.py                # the machine on pseudo engines
scripts/bench/machine/card.sh CARD TAG                # one machine per card, every metric on disk
python scripts/bench/real_machine.py configs/machine/qwen35_local.py --name local   # real engines
python scripts/bench/vlm_units.py --trace runs/traces/poc30/text --name qwen35 --engine '{"entry": "qwen35", "model": "..."}'

GBC is a compute project: the tests check data and formats only, and everything else is checked by runs on the cards. tests/program replays recorded jobs of every fixture arm through the reference interpreter and compares graphs by digest; tests/graph holds both resolutions to ml-gbc's; tests/engines checks the API entry's request shapes, the batch formers and SAM 3's mask boxes. Engines, kernels and the machine are measured by the benches above, every metric on disk (performance). CI checks the code's form (lint, import boundary, rustfmt, clippy).

Open an issue first for a feature. Conventions: black and ruff at line length 100, no imports inside functions, nested folders, comments that say what the code does, docs that describe what exists, and every check able to fail on a broken implementation. Run the checks above before a pull request.

apple/ml-gbc.

Apache-2.0.