LeRobot pi0.5#

This page documents the FGManip workflow based on LeRobot pi0.5. The paper is available at pi0.5: a Vision-Language-Action Model with Open-World Generalization.

Integration status: pi0.5 currently runs through the LeRobot toolchain. It shares dataset conventions with other policies where possible, but runtime and conversion steps are still policy-specific.

Overview#

Current implementation reuses LeRobot pi0.5 for policy inference and evaluation in FGManip. Because the pipeline is built on LeRobot, this page can also serve as a template for other VLA policies supported by LeRobot.

  • Policy entrypoint: FGManip/core/policies/pi05/evaluate.py

  • Data converter: utils/convert_to_lerobot.py

  • Checkpoint format: LeRobot pretrained_model directory with config.json and *.safetensors

1. Evaluation Script Usage#

Evaluation script: FGManip/core/policies/pi05/evaluate.py. It supports loading LeRobot fine-tuned checkpoints and running closed-loop tests in FGManip simulation.

1.1 Generic Task Evaluation#

For tasks that do not require specific object or part identifiers (e.g. plug_charger).

python core/policies/pi05/evaluate.py \
  --policy-path /Your/path/to/pretrained_model \
  --env-id plug_charger \
  --obs-mode rgb \
  --control-mode pd_joint_delta_pos \
  --n-episodes 50 \
  --device cuda \
  --task "Description of this task" \
  --record-dir /Your/path/to/save \
  --save-video

Arguments:

  • --policy-path: path to LeRobot training output pretrained_model.

  • --env-id: FGManip environment ID.

  • --task: natural language description aligned with training semantics.

1.2 Specific-Object Task Evaluation#

For tasks requiring explicit object and part selection (e.g. toggle_switch).

python FGManip/core/policies/pi05/evaluate.py \
  --policy-path /Your/path/to/pretrained_model \
  --env-id toggle_switch \
  --object-name 100920 \
  --part-name button \
  --obs-mode rgb \
  --control-mode pd_joint_delta_pos \
  --n-episodes 50 \
  --device cuda \
  --task "Description of this task" \
  --record-dir /Your/path/to/save \
  --save-video

Differences from generic evaluation:

  • Add --object-name for asset/object ID.

  • Add --part-name for target part.

2. Data Conversion Notes#

Before LeRobot training/inference, convert ManiSkill-format trajectories to LeRobot dataset format. 

python utils/convert_to_lerobot.py \
  --traj-path demos/trajectory.h5 \
  --output-dir demos/lerobot_grasp_part \
  --task-name "Grasp Part" \
  --fps 30 \
  --robot-type panda

2.1 Quantile Normalization Stats#

The converter should generate stats.json with q01 and q99 for action/state normalization. 

'q01': np.percentile(actions, 1, axis=0).tolist(),
'q99': np.percentile(actions, 99, axis=0).tolist()

Reason: pi0.5 uses quantile normalization by default. Missing quantile stats causes: 

ValueError: QUANTILES normalization mode requires q01 and q99 stats

3. Environment Setup#

Since this workflow depends on LeRobot pi0.5 and ManiSkill simulation, install both stacks.

ManiSkill

pip install --upgrade mani_skill
pip install torch

LeRobot

pip install -e ".[pi]"

For LeRobot 0.4.0, use:

pip install "lerobot[pi]@git+https://github.com/huggingface/lerobot.git"

Compatibility Tips

pip install numpy==1.26.4

For RTX 50-series GPUs:

pip install torch==2.10.0 torchvision torchaudio

4. Maintenance Checklist#

Use this structure when contributors update LeRobot-based policy pages.

  • Model Source: paper link, upstream repo, exact version tag.

  • Environment Mapping: supported FGManip env IDs and task prompts.

  • Data Contract: input trajectory format, conversion script, required stats fields.

  • Runbook: train/eval command matrix for generic and object-specific tasks.

  • Validation: required metrics, number of episodes, video logging policy.

  • Troubleshooting: common install/runtime errors and fixes.