Sounds like science fiction, doesn’t it? However, with the advent of the revolutionary SLoMo framework, this future may be closer than we think. SLoMo, short for ‘Skilled Locomotion from Monocular Videos,’ is a groundbreaking method that enables legged robots to imitate animal and human motions by transferring these skills from casual, real-world videos.
The SLoMo Framework: A Three-Stage Approach
The SLoMo framework works in three stages:
Stage 1: Synthesizing Physically Plausible Reconstructed Key-Point Trajectory
In the first stage of the SLoMo framework, a physically plausible reconstructed key-point trajectory is synthesized from monocular videos. This involves using a combination of computer vision and machine learning algorithms to extract the most relevant features from the video footage.
Stage 2: Optimizing Dynamically Feasible Reference Trajectory Offline
In the second stage of the SLoMo framework, a dynamically feasible reference trajectory for the robot is optimized offline that includes body and foot motion, as well as contact sequences that closely track the key points. This involves using a variety of optimization algorithms to find the most efficient and effective motion sequence.
Stage 3: Tracking Reference Trajectory Online Using Model-Predictive Controller
In the third stage of the SLoMo framework, the reference trajectory is tracked online using a general-purpose model-predictive controller on robot hardware. This involves using real-time feedback from sensors to adjust the motion of the robot and ensure that it stays on track.
Successful Demonstrations and Comparisons
SLoMo has been demonstrated across a range of hardware experiments on a Unitree Go1 quadruped robot and simulation experiments on the Atlas humanoid robot. This approach has proven more general and robust than previous motion imitation methods, handling unmodeled terrain height mismatch on hardware and generating offline references directly from videos without annotation.
Limitations and Future Work
Despite its promise, SLoMo does have limitations, such as key model simplifications and assumptions, as well as manual scaling of reconstructed characters. To further refine and improve the framework, future research should focus on:
Extending the Work to Use Full-Body Dynamics
One potential area for improvement is extending the work to use full-body dynamics in both offline and online optimization steps. This would allow for more realistic and dynamic motion sequences.
Automating the Scaling Process
Another potential area for improvement is automating the scaling process and addressing morphological differences between video characters and corresponding robots. This would make it easier to adapt SLoMo to different robot platforms and environments.
Investigating Improvements and Trade-Offs
Investigating improvements and trade-offs by using combinations of other methods in each stage of the framework is also an area for future research. For example, leveraging RGB-D video data could provide more accurate and detailed motion sequences.
Deploying SLoMo on Humanoid Hardware
Finally, deploying the SLoMo pipeline on humanoid hardware, imitating more challenging behaviors, and executing behaviors on more challenging terrains is a key area for future work.
Conclusion
As SLoMo continues to evolve, the possibilities for robot locomotion and motion imitation are virtually limitless. This innovative framework may well be the key to unlocking a future where robots can seamlessly blend in with the natural world, walking, running, and even playing alongside their animal and human counterparts.
Author Information
SLoMo: A General System for Legged Robot Motion Imitation from Casual Videos
- John Z. Zhang
- Shuo Yang
- Gengshan Yang
- Arun L. Bishop
- Deva Ramanan
- Zachary Manchester
- Robotics Institute, Carnegie Mellon University
References
https://arxiv.org/pdf/2304.14389.pdf
