blendi-remade/prompt2rig

Type a motion. A video model acts it out, and any rigged character performs it, fingers included.
A prompt goes to a fast video model on fal, your GPU recovers clean 3D body and hand motion from the clip, and it plays live on a Mixamo character in the browser, ready to export to Blender or Unity.

A universal-modder tool. Share what you make, get help, and follow along on discord and x.

flowchart LR
  P[prompt] --> V["FLUX 3 Max / H3 Max Turbo<br/>on fal"]
  I["performer image<br/>or the last clip's final frame"] --> V
  V --> L["MediaPipe · browser<br/>instant preview"]
  V --> H["GVHMR · your GPU<br/>3D body motion"]
  V --> F["WiLoR · your GPU<br/>wrists + fingers"]
  L --> R[Mixamo rig]
  H -- cross-fade --> R
  F --> R
  R --> G[".glb · Blender render"]

The live pass covers the wait; the HQ pass is the result. GVHMR sees the whole clip at once, outputs joint rotations on a fixed-proportion body and recovers a gravity-aware root path, so there's no jitter, limb stretching or twisted forearms. WiLoR adds the palm and all 15 finger joints per hand.

Needs: Node 18+ and a fal API key. For the HQ pass: Windows, an NVIDIA GPU, uv and ffmpeg on PATH.

git clone https://github.com/blendi-remade/prompt2rig && cd prompt2rig
cp .env.example .env          # add your FAL_KEY
node server.mjs               # → http://localhost:5177 (the live preview works now)

HQ pass (recommended):

hq\setup.ps1                  # venv, CUDA PyTorch, GVHMR + ~5 GB of checkpoints

Download SMPL-X v1.1 (NPZ) from smpl-x.is.tue.mpg.de (free registration) and put SMPLX_NEUTRAL.npz in hq/body_models/smplx/.

Finger tracking (optional): WiLoR needs the MANO hand model. Register at mano.is.tue.mpg.de, download Models & Code (mano_v1_2.zip), then:

hq\.venv\Scripts\python hq\setup_hands.py $HOME\Downloads\mano_v1_2.zip   # WiLoR weights + MANO

Without MANO, fingers fall back to MediaPipe Hands (rougher depth). Then start the worker next to the web server:

hq\.venv\Scripts\python hq\worker.py    # → 127.0.0.1:5178, loads once and stays warm

• Generate from a prompt or a preset. Every clip starts from the same flux-made performer (one person, full body, plain backdrop, static camera), which is the input pose estimation likes best.
• Video model. FLUX 3 Max for the most realistic motion; H3 Max Turbo for the fastest loop and stream mode.
• Character. X Bot, Michelle or Soldier (textured Mixamo rigs from the three.js examples), or load mixamo file… for any Mixamo .fbx / .glb from mixamo.com (T-pose).
• Track fingers. WiLoR runs on the full-resolution clip; hands are matched to the body's wrists, rotated into the motion's world frame and smoothed. Untracked hands rest in a natural curl.
• Stream mode. One continuous performance: each clip picks up from the last one's final frame, plays once its HQ motion is in while the next one generates, and short idle clips fill the gaps between prompts.
• Swap character every loop, cinematic camera, portrait or landscape clips.
• ↓ glb on any take exports the skinned character with its animation, and keeps a copy plus the source clip's URL in takes/.

Prompts that retarget well face the camera with the limbs clear of the body: dances, kicks, kata, sport mimes, celebrations, clear hand shapes. Flips, spins, floor work and crossed or hidden arms confuse any single-camera pose estimator; the presets and the prompt template steer away from them.

Blender. Turn a saved take into a studio clip (cyclorama, three-point light, slow camera arc), headless:

& "C:\Program Files\Blender Foundation\Blender 5.2\blender.exe" -b -P tools/render_blender.py -- takes/<take>.glb --side-by-side

It writes renders/<take>.mp4 (1080×1920, or --res 1920x1080) and, with --side-by-side, the source clip next to the render.

App recordings. tools/capture.mjs drives an off-screen Chrome and records takes frame by frame at 2560×1440, so nothing on your screen is touched:

node tools/capture.mjs --prompts tools/prompts_dynamic.txt --character xbot --out captures
node tools/capture.mjs --clip <video url> --prompt "..." --passes 2 --zoom 5.2-10.0 --zoom-bone RightHand

• GVHMR (ZJU3DV) via the gvhmr package · WiLoR via WiLoR-mini · Motion2MixamoRig by @MaxLiu09, which started this
• MediaPipe · three.js (X Bot, Michelle and Soldier from its examples) · video and image models on fal

Made by the universal-modder crew. Discord · X @universalmodhq · universalmodder.com

prompt2rig's own code is Apache-2.0 (see NOTICE). It runs alongside third-party components that keep their own licenses; none are included in this repo:

In short: the web app, the live preview and video generation are commercially usable; the HQ path (GVHMR, SMPL-X, WiLoR, MANO) is research / non-commercial, so a commercial product would need to replace or license those pieces.

Tested on Windows 11 with an RTX 4090.