Pi0 and Lerobot with Rerun

A repo to explore training robots with Pi0 and Lerobot

Installation

Currently only linux is supported

Using Pixi

Make sure you have the Pixi package manager installed

git clone https://github.com/rerun-io/pi0-lerobot.git
cd pi0-lerobot
pixi install

Usage

Human Pose and Kinematics

For assembly101 dataset go to this link to get/download dataset

pixi install
pixi shell
python tools/visualize_assembly.py --root-directory $ROOT-DIR-FOR-ASSEMBLY-DATA --example-name $STR-NAME-OF-EXAMPLE-FOLDER

Jupyter Notebook Tutorials

pixi run notebook_tutorial

TODO

Part 1. Human Pose and Kinematics

• Basic Triangulation from 2D Detection (body pose)
• Basic Triangulation from 2D detection (hand pose)
• Detection by Tracking (extrapolate 3d views-> Generate bounding box based on extrapolated -> check based on kpts confidence)
• Mano + SMPL fitting for skeleton kinematics
• Add EgoCentric (first person) headset views and poses

Part 2. Body and Hand Pose Retargeting

• Implement pose retargeting to isaac sim

Part 3. Immitation Learning and Teleoperation

• Notebook Tutorials
• Finetune and evaluation on Aloha simulator with Pi0
• Finetune base Pi0 on SO-ARM100 from lerobot