A humanoid that walks — and dribbles — inside a Gaussian splat

A Unitree G1, 29 degrees of freedom, driven by a ProtoMotions motion-tracking policy exported to ONNX. Every 20 ms it reads the robot's own state — torso orientation, joint positions and velocities, root angular velocity — and emits joint targets through the robot's PD loop. Physics runs underneath at 1 kHz, twenty substeps per control step, in Isaac Sim 6.0 / Isaac Lab 3.0.

The ball is a real rigid body — 0.43 kg, 109 mm, a proper collider — not a prop stuck to the foot.

Over 3.7 seconds the robot takes eight steps, covers 3.3 m, and never falls. It measures out as real walking: a 0.99 s stride at 121 steps per minute, and the pelvis rises and falls at exactly twice the sway frequency — the geometric signature of a gait, not a scripted loop.

The garden behind it is a real scene, reconstructed as a Gaussian splat:

Study 01 was Scene→Sim: turning a real capture into a reconstruction clean enough to train in. This is the next stage — Sim→Skill: a policy actually doing something inside that reconstructed world.

The physics is real and runs in the engine live, so contact, balance and force are the policy's problem, not an animator's. That is the whole point of a skill layer: skills are learned and validated in simulation on a faithful twin of the real scene, then transferred onto the machine — where the same forces apply.

The policy is tracking a reference walk. It has no observation of the ball at all — nothing in what it senses says the ball exists. The contacts are emergent: the foot happens to meet the ball because the walk carries it there.

The reconstruction also came out 1.8° off level, so the flat collision plane the robot walks on did not quite match the lawn it appears to walk on. We caught it by planting reference rods of known world-vertical in frame, and corrected it before this render.

It shows in how the run ends, past the clip above: the ball decelerates like a ball on grass and rolls to a stop, and the robot — still walking — catches up, taps it, and then walks straight past it. Nothing steers it back. That is the tell that this is physics and tracking, not a choreographed shot.