Diffusion Policy

How it works

You record a person teleoperating the robot. The model learns to turn camera images and the robot's current state into the next second or so of movement. Instead of guessing one action, it starts from random noise and cleans it up step by step into a plausible action sequence, the same trick image generators use. That lets it represent several valid ways to do a task without averaging them into a bad middle. At run time it predicts a chunk of actions, executes part of it, then re-plans.

What it needs

Typically 50–300 teleoperated demonstrations per task; one or two RGB cameras; a GPU for training (hours) and for inference at ~10 Hz. No reward design and no simulator needed.

Strengths

• Handles tasks with several valid solutions
• Strong real-robot results from a few hundred demos
• Open source and widely re-implemented

Limits

• Needs demonstrations for every new task and object
• Does not improve after training without extra machinery
• Struggles when the scene differs from the demos