LeRobot documentation
Simulation
Simulation
A simulator in LeRobot is a Gymnasium environment that follows LeRobot’s conventions (dictionary observations with camera images and robot state, continuous actions, success in info["is_success"]; see The environment), written once and used everywhere LeRobot runs a robot:
- to evaluate a policy, as a batched env (
--env.*): many environments of one task, stepped together.lerobot-evaland evaluation during training use it. - as a sim robot (
--robot.type=sim --robot.env.*): one environment behind theRobotinterface, swappable with real hardware.lerobot-rollout,lerobot-record,lerobot-teleoperateandlerobot-replayuse it.
Either runs locally, in your Python environment, or served by lerobot-env-server, usually in its own Docker image, so that the simulator’s dependencies never meet the policy’s. Setting an endpoint selects served execution; both paths give the policy the same observations (HWC uint8 images, flat float32 state), the same episode ends, and the same control conventions.
| Locally | Served | |
|---|---|---|
Batched env: lerobot-eval | --env.type=<sim> | --env.type=<sim> --env.server.endpoint=tcp/... |
Sim robot: lerobot-rollout, -record, -teleoperate, -replay | --robot.type=sim --robot.env.type=<sim> | --robot.type=sim --robot.env.type=<sim> --robot.env.server.endpoint=tcp/... |
| Install | the simulator, next to LeRobot (its extra) | lerobot[remote]; the simulator lives in the server’s image |
| To | Read |
|---|---|
| Evaluate, record or teleoperate in simulation | this page, and the simulator’s own page (LIBERO, Meta-World, …) |
| Use or publish a simulator from the Hugging Face Hub | EnvHub |
| Add a benchmark that LeRobot maintains in its CI | Adding a benchmark |
Evaluate a checkpoint
Locally, with the simulator installed next to LeRobot (see LIBERO):
lerobot-eval \
--policy.path=lerobot/smolvla_libero \
--policy.pretrained_revision=a24f1f380461b665158d52acd140152c79e8b36d \
--policy.device=cuda \
--env.type=libero --env.task=libero_spatial --env.task_ids='[0]' \
--rename_map='{"observation.images.image": "observation.images.camera1", "observation.images.image2": "observation.images.camera2"}' \
--eval.n_episodes=1 --eval.batch_size=1 \
--output_dir=outputs/eval_liberoServed: start the server with the same environment (see Serve an environment), and add its endpoint:
lerobot-env-server --config_path=docker/sim/configs/libero.yaml --docker=true
lerobot-eval ... --env.server.endpoint=tcp/127.0.0.1:7448An EnvHub environment works the same way, with --env.type=envhub --env.hub_path=<user/repo>, served with lerobot-env-server --envhub=<user/repo> --trust_remote_code=true --docker=true once you have read its code.
Every option for the server is under --env.server.*, and none applies to a local run. The client asks for the deployment named after --env.type, as the server configurations name it; pass --env.server.deployment for another. A wrong endpoint or deployment fails within --env.server.handshake_timeout_s (10 s) and names what the endpoint serves. Against a server secured with TLS, pass its Zenoh JSON5 file with --env.server.zenoh_config_path, as for remote inference. The server must run the environment the command describes: the same type, task, control conventions and simulated body (robot_type), or the evaluation refuses to start and lists the differences. Other options that differ (resolution, episode length) are logged as warnings.
Observation names come from the environment (observation.images.<camera>, observation.state). When a checkpoint was trained with other camera names, map them with --rename_map, as for any robot, and preserve each camera’s identity even when several have the same shape. Placeholder cameras (empty_cameras) are a policy setting saved with the checkpoint; override it with --policy.empty_cameras only when needed.
Before the first step, every checkpoint input must exist in the environment after renaming, with matching state and action dimensions and RGB channel count. Image height and width may differ: policy preprocessing owns resizing. Controller conventions (delta vs absolute actions, rotation format) are not checked; use a checkpoint trained on the environment’s convention.
Batches and served throughput
Each step() advances every unfinished environment in the batch once. An environment that ends keeps its terminal observation until the next reset; stepping the others never restarts it. With --eval.use_async_envs=true (the default), each environment runs in its own worker process, locally or on the server, and the batch steps in parallel. A server admits up to max_envs environments per batch (32 by default).
--env.server.encoding=jpeg (or --robot.env.server.encoding=jpeg, with --env.server.jpeg_quality, 90 by default) sends served camera frames as JPEG: smaller replies for a remote server and larger batches, at the cost of encoding time and lossy frames. On localhost, raw frames are faster.
A reply carries the frames of every environment in the batch and must fit max_payload_bytes (32 MiB by default) at both ends, or the server refuses the batch when it opens. For large batches or high resolution cameras, raise it on the server (max_payload_bytes in the YAML’s zenoh section, next to listen_endpoints) and on the client (--env.server.max_payload_bytes, or --robot.env.server.max_payload_bytes).
Run a sim robot
The same environment as a batched env and as a sim robot:
lerobot-eval --env.type=libero --env.task=libero_spatial --env.task_ids='[0]' ...
lerobot-rollout --robot.type=sim --robot.env.type=libero --robot.env.task=libero_spatial --robot.task_id=0 ...For example:
lerobot-rollout \
--robot.type=sim --robot.env.type=libero --robot.env.task=libero_spatial --robot.task_id=0 \
--task="pick up the black bowl between the plate and the ramekin and place it on the plate" \
--policy.path=lerobot/smolvla_libero \
--policy.pretrained_revision=a24f1f380461b665158d52acd140152c79e8b36d \
--rename_map='{"observation.images.image": "observation.images.camera1", "observation.images.image2": "observation.images.camera2"}' \
--device=cuda --fps=20 --inference.type=sync \
--display_data=true --display_mode=rerun \
--return_to_initial_position=false --play_sounds=false--robot.env.*takes the same options as--env.*in evaluation.--robot.task_idpicks the one task the robot runs, where evaluation runs every task of the suite or those in--env.task_ids. EnvHub environments are--robot.env.type=envhub --robot.env.hub_path=<user/repo>, with--robot.env.trust_remote_code=trueto run them locally.--robot.env.server.endpoint=tcp/...drives the environment on an env server instead, as--env.server.endpointdoes for a batched env. The robot checks that the server runs that environment.- Each
send_action()advances the environment exactly once. Observation reads, policy inference, and waiting do not advance simulation; the existing control loop owns wall-clock pacing. --fpsmust equal the environment’s control rate, divided by--interpolation_multiplierwhen interpolating, since every command advances the environment once. When recording,--dataset.fpsmust equal--fps. Rollout refuses to start otherwise.
A sim robot is a robot: swap --robot.type=sim --robot.env.* for the real follower’s --robot.*, and everything else stays. A teleoperator drives it as it drives the real follower (--teleop.type=so101_leader for an SO-101 twin such as lerobot/so101-tabletop-mujoco), when the environment declares the follower’s action names and units; a mismatch is reported at the first action.
Robot camera features use the same observation.images.<camera> names as real-hardware recording. A single unnamed camera is observation.images.image. Batched evaluation keeps the environment’s own keys, such as observation.image for a single camera; when deploying a checkpoint with that name through a sim robot, use --rename_map='{"observation.images.image": "observation.image"}'.
When the policy is late, holding a sim robot does not advance physics. Simulator failures propagate to rollout shutdown.
Record in simulation
lerobot-record stores the body the environment declares as the dataset’s robot_type (panda for LIBERO, so_follower for an SO-101 twin), or sim_robot when it declares none.
Recorded episodes follow the environment’s. lerobot-record and the episodic rollout strategy restart the simulated environment at the start of each episode, and end it when the environment’s episode ends (success, termination or its step limit) or when its duration is up, whichever comes first. Each recorded episode is therefore one simulated episode, from its initial state. Nothing needs resetting by hand, so skip the reset phase between episodes with --dataset.reset_time_s=0, and with the episodic rollout strategy also pass --strategy.reset_to_initial_position=false.
When an episode ends, the robot stops: it ignores actions and keeps its final state, and latest keeps the final reward, termination and success. Teleoperation and the base rollout strategy then start a new episode (the strategy also resets the policy), and replay stops. To record with a policy, use the episodic rollout strategy. Real robots are unaffected: their episodes are timed, and their scene is reset by hand. For success rates, evaluate with lerobot-eval and a batched env.
Remote inference
Keep the same --robot.* options and replace the local policy and inference options with:
--inference.type=remote \ --inference.endpoint=tcp/127.0.0.1:7447 \ --inference.deployment=libero \ --inference.semantics=libero-delta-v1
Start the policy server separately using remote inference setup. Its schema must match the robot’s observations after --rename_map. Omit --policy.path from the rollout command because the policy server loads the checkpoint. For local RTC, use --inference.type=rtc with the policy’s RTC settings.
Serve an environment
From a LeRobot source checkout (the image is built from it), run from the repository root:
# An in-tree simulator
lerobot-env-server --config_path=docker/sim/configs/libero.yaml --docker=true
# An EnvHub repository, once you have read its code
lerobot-env-server --envhub=<user/repo>@<commit> --trust_remote_code=true --docker=true--docker=true uses docker/sim/compose.server.yaml, selects the image from sim.type (or builds one from the EnvHub repository), mounts the effective YAML, and runs the container on the host network, so the server listens on localhost:7448 unless --zenoh.listen_endpoints says otherwise. Swap the YAML to select another simulator. Each simulator’s control rate, action convention, component names and robot type come from its integration’s conventions() (an EnvHub repository’s LEROBOT_SIM), so a YAML only names the task and the simulator’s options.
As for lerobot-policy-server, every YAML field accepts a nested command-line override, and the file stays unchanged. A dictionary value such as sim.kwargs is merged into the YAML’s:
lerobot-env-server --config_path=docker/sim/configs/libero.yaml --docker=true \
--sim.task=libero_goal \
--sim.kwargs='{gym_kwargs: {observation_height: 256, observation_width: 256}}'The effective configuration is kept under $HF_LEROBOT_HOME/sim, so a restarted container finds it. Switching configuration recreates the server and ends active sessions. The image is built when missing; add --build=true to rebuild it after changing the simulator or LeRobot source, or --stop=true to shut down the server. The container requires Docker GPU access for EGL rendering.
If a simulator that renders with Vulkan (Isaac Sim, or ManiSkill/SAPIEN as in RoboMME, whose image ships its own Vulkan file) fails in the container with
vkCreateInstance failedorERROR_INCOMPATIBLE_DRIVERwhile it works outside Docker, the container has the NVIDIA driver’s libraries but not the loader files that point at them (/usr/share/vulkan/icd.d/nvidia_icd.json,/usr/share/glvnd/egl_vendor.d/10_nvidia.json). The NVIDIA Container Toolkit’s default mode, used bygpus: all, may not mount them; its CDI mode does. Request the GPU through CDI (--device nvidia.com/gpu=allfordocker run,devices: [nvidia.com/gpu=all]in place ofgpus: allindocker/sim/compose.server.yaml), or add the two files to the image.
Without --docker=true, the same command runs the server in the current environment: install the simulator’s native dependencies in an isolated environment with Python 3.12 or newer, install LeRobot with --no-deps, and install docker/sim/requirements.txt. The server is part of LeRobot, so it needs the same Python as LeRobot; a simulator that only supports an older Python can’t be served.
Where the server runs
The server only accepts connections from its own machine by default, since the connection is neither encrypted nor authenticated: anyone who reaches it can drive the environment. To run it on another machine, such as a workstation or a cloud instance with a GPU, while the policy runs on yours, pass the address of the network interface yours reaches with --zenoh.listen_endpoints='[tcp/<address>:7448]', and connect with --env.server.endpoint=tcp/<address>:7448 (--robot.env.server.endpoint for a sim robot). Keep this to a trusted network, or add TLS and access control with a zenoh.config_file in the server YAML and --env.server.zenoh_config_path on the client, as for remote inference. Over a slower link, JPEG frames need less bandwidth.
Server requests are never retried. After an ambiguous timeout, the client fails; close it and open a new session.
Add a simulator
To share an environment of your own, publish it on the Hub as an EnvHub environment: an env.py in a Hub repository is the whole integration, and it gets everything on this page. Benchmarks that LeRobot maintains and tests in its CI, with a pinned checkpoint and a Docker stage, are added to LeRobot itself: see Adding a benchmark.