kling01/RubricLoop

Live demo · Quick start · Usage · Roadmap · Human Value Allocation · 简体中文

Rubric-driven evaluation for AI agents. RubricLoop turns traces into inspectable results: define executable rubrics, combine code rules with model judges, and review each decision alongside its trace in a standalone HTML report.

RubricLoop is an open-source evaluation project from 人在回路. The starting point is a practical data-quality workflow: which behavior needs attention, which criterion caught it, and what should a person review?

Try the evaluation workflow with three hand-authored traces: run code rules locally, inspect each result, and open the generated report.

Actual output from rubricloop demo: 3 traces, 9 checks, 6 pass, 2 fail, and 1 not applicable. Open the live report, or run the command below to open a local report.

Requires Python 3.10 or newer.

git clone https://github.com/kling01/RubricLoop.git
cd RubricLoop
python -m venv .venv
source .venv/bin/activate
pip install -e .

rubricloop demo --out runs/demo

Open runs/demo/report.html in your browser. The same run writes runs/demo/results.json.

Previously released as AgentEval: v0.1.0a2 introduces the RubricLoop name. The agenteval command and AGENTEVAL_* environment variables remain compatible; Python imports continue to use agenteval. See migration notes.

rubricloop run examples/traces.json \
  --rules examples/rules.yaml \
  --out runs/rules

Start with a JSON array of conversations:

[
  {
    "id": "example-001",
    "messages": [
      {"role": "user", "content": "What is 17 + 25?"},
      {"role": "assistant", "content": "42"}
    ]
  }
]

Use code rules for checks you can state precisely. Add a model judge for criteria that require semantic judgment:

export RUBRICLOOP_BASE_URL="https://your-provider.example/v1"
export RUBRICLOOP_MODEL="your-model"
export RUBRICLOOP_API_KEY="your-api-key"

rubricloop run examples/traces.json \
  --rules examples/rules.yaml \
  --evaluators examples/judge.yaml \
  --concurrency 4 \
  --n-runs 1 \
  --out runs/judged

The configured provider receives the trace content selected by your evaluator prompts. This command uses that provider's API and may incur charges. The offline demo needs no provider or API key.

See Usage for input formats, rule semantics, model configuration, and result interpretation.

Trace / message history
          │
          ▼
Normalize and extract features
          │
          ├── Code rules ───────────────┐
          │                            │
          └── Model / hybrid judges ───┤
                                       ▼
                              HTML + JSON results
                                       │
                                       ▼
                              Evidence and review

Each criterion retains its own outcome:

A code rule's check describes a violation: True produces a fail. Rules are trusted Python expressions, executed locally; they are not sandboxed. Review rule files before running them.

The report counts passes, failures, non-applicable results, and errors separately.

This release packages the evaluation engine, a CLI, authored examples, and standalone reports. The engine includes structured, versioned rubric clauses; the public CLI focuses on trace-level runs.

The next steps are evaluator feedback and revision, regression checks, and integrations into data-production workflows. Policy-to-rubric generation, automatic repair loops, model training, and intervention-value prediction are tracked in the roadmap.

Human Value Allocation is a separate project for inspecting visible human contributions in human–agent conversations.

Read the related paper, From Tasks to Turns: Measuring Human Labor Contributions in Human–Agent Joint Production, for the research on human participation. RubricLoop provides the reusable agent-evaluation component.

Concrete evaluation cases are especially useful: a minimal trace, an intended criterion, the observed result, and the expected result. Remove private data and credentials before sharing a reproduction in Issues.

See Usage for the runnable interface and Roadmap for the current boundary and development direction.

MIT.