Sim-to-real
ResMimic (residual RL on motion tracking)
Teaches a humanoid to carry boxes and sit in chairs by adding a small learned correction on top of a general motion-tracking policy, trained in simulation and transferred to a Unitree G1.
How it works
First, train one policy in simulation that can imitate ~42 hours of human motion capture on the humanoid's body. That policy knows how to move but not how to handle objects. Second, for each task (carry a box, sit on a chair), train a second small policy that outputs corrections to the first one, with rewards for object tracking and contact. Train with PPO in Isaac Gym with domain randomization, check in MuJoCo, then run on the real G1 without further training.
What it needs
A good motion-capture dataset (AMASS, OMOMO) retargeted to your robot, GPU simulation (Isaac Gym/Lab), and a real humanoid. Real-world runs used proprioception only, or motion capture for object state.
Strengths
- Real G1 carried 4.5 kg boxes and chairs beyond the arm payload by using the whole body
- Two-stage design reuses one base policy across tasks
- Code released
Limits
- Real-world results are qualitative; no success rate reported
- Object-aware behaviour needs motion-capture object tracking
- Tasks are coarse (carry, sit), not precise manipulation
Where it has been shown to work (1)
| Task | Robot | Maturity | Success | n | Grade | Year |
|---|---|---|---|---|---|---|
| Carrying and whole-body manipulation | Unitree G1 | Lab | — | 11 | B | 2025 |