https://github.com/google/ml-workload-debugging-wiki

Diagnose failures in large-scale ML training and inference workloads — Cloud TPU, GPU, Pathways, JAX, XLA and GKE — using a knowledge base that both people and coding agents can execute.

Copyright 2026 Google LLC. Licensed under the Apache License, Version 2.0.

This is not an officially supported Google product. We share it as a reference implementation of agent-executable operational knowledge.

Debugging a stalled 2,048-chip training run is mostly an exercise in knowing where to look: which log holds the first real error, which of 56 restart attempts matters, whether a DATA_LOSS message names the culprit host or just a device that noticed. That knowledge usually lives in the heads of a few on-call engineers and in scattered incident docs.

This project writes it down in a form you can run. Prose pages explain a failure mode; JSON workflows execute the diagnostic steps against your cluster.

• What you get
• Quick start
• Example: diagnosing a workload
• How actions run
• Using it with a coding agent
• Repository layout
• Documentation
• Getting help
• Contributing
• License

Around 250 pages, organised by page type, each with a schema defined in _wiki/SCHEMA.md:

Pages cross-reference each other with [[wikilink]] syntax, and _wiki/INDEX.md indexes all of them.

The distinction that matters most is playbook vs. workflow. Reach for a playbook when you are reasoning heuristically and the picture is unclear. Reach for a workflow when you want the same parameterised steps to run the same way every time.

You need Python 3.9 or newer (tested on 3.13). Actions that query live infrastructure also need the Google Cloud SDK — gcloud and kubectl — authenticated against the project you are debugging.

git clone https://github.com/google/ml-workload-debugging-wiki
cd ml-workload-debugging-wiki

python3 -m venv .venv
source .venv/bin/activate
pip install -r _wiki/requirements.txt

Confirm the wiki is intact:

python3 _wiki/wiki_linter.py

The linter takes no arguments. It validates frontmatter against the schema, link integrity, orphaned pages, tag vocabulary and duplicate actions, printing a JSON report. A healthy tree reports "status": "PASS" with zero issues. Run it before you send a change.

Start from the symptom. If a training job hangs and the logs show workers stuck on a synchronisation flag, open _wiki/symptoms/wait-timeout-on-sflag.md. Its ## Connections section names the concepts and workflows that apply.

Run one step of that investigation directly:

python3 _wiki/tools/scripts/python_glue_runner.py \
  resolve-workload-nodes '{"workload_name": "my-job", "project_id": "my-project"}'

The runner reads _wiki/actions/resolve-workload-nodes.json, interpolates your parameters into its command, runs it, and prints the action's declared summary_fields — here the node set and whether it is complete.

Read the action file before running anything unfamiliar. Actions that change cluster state carry "requires_approval": true, and several of them cordon nodes or remove hosts from service.

An action is not necessarily a shell command. Dispatching one the wrong way is the most common integration mistake:

The ai- filename prefix and "execution_mode": "ai" always agree. The prefix is the at-a-glance signal; the field decides.

The wiki ships agent definitions for two harnesses: .claude/agents/ for Claude Code, and _agents/agents/ for the Antigravity harness. A test fails if the two drift apart.

Under Claude Code, /troubleshoot <workload_id> runs the whole sequence.

Every profile carries the same grounding rules: cite the page behind each claim, label whatever the wiki does not support, never invent an action or host name, and separate what you read from what you inferred. Those rules exist because a confident but unsupported diagnosis is worse than none — the reader cannot tell the difference.

_wiki/              Knowledge base, schema, tooling
  SCHEMA.md           Page types, required frontmatter, tag taxonomy
  INDEX.md            Index of every page
  TUTORIAL.md         End-to-end walkthrough
  LOG.md              Append-only record of changes and lint runs
  wiki_linter.py      Structural and schema validation
  query_interface.py  Programmatic search over the wiki
  tools/scripts/      Diagnostic runners, including the glue runner
_agents/            Agent definitions for the Antigravity harness
.claude/            Agent definitions and slash commands for Claude Code
docs/               Contributing guide and code of conduct

• _wiki/SCHEMA.md — page types, frontmatter, tags. Read this before adding a page.
• _wiki/INDEX.md — the index; the best way to browse.
• _wiki/TUTORIAL.md — a walkthrough of diagnosing a failing workload end to end.
• _wiki/LOG.md — what changed, when, and why.

_wiki/query_interface.py offers a programmatic way to search the wiki when you would rather not grep.

Open an issue at https://github.com/google/ml-workload-debugging-wiki/issues. For a page that contradicts reality, say which page and what you observed instead — that is the most useful bug report this project receives.

See docs/contributing.md. Contributions need a Google CLA and follow the Community Guidelines.

The most valuable changes: a symptom page for a failure mode nobody has written up, a workflow that codifies steps people still do by hand, or a correction where the documented behaviour has drifted from the real system. New pages must satisfy _wiki/SCHEMA.md and pass the linter.

Apache License 2.0. See LICENSE.