Enable multi-policy experiments on OSMO

[alexmillane]

Summary

Enable multi-policy experiments on OSMO

Detailed description

• Why: Previously an OSMO Arena-experiment submission served one policy type, selected with --policy_server,`
• What:
  • remove --policy_server and automatically determine which policy server task should be launched from the experiment config.
  • this requires a reigstry maintaining a mapping between policy clients and their server tasks.

[greptile-apps]

Greptile Summary

The PR generalizes Arena Experiment OSMO submissions to derive and co-schedule a policy server for each Run.

• Adds pi0, GR00T, and Cosmos server bindings with per-server deployment configuration.
• Adds a watchdog that restarts silent experiment runners after clearing their output.
• Adds Cosmos serving support, camera recording defaults, dry-run and override-listing CLI options, and OSMO workflow documentation.

Confidence Score: 4/5

The PR should not merge until active evaluations cannot be mistaken for stalls and effective OSMO resources are checked against the derived server requirements.

Newline-only watchdog activity tracking can repeatedly kill healthy runs, while post-derivation pool and platform overrides can silently schedule policy servers outside their declared resources.

Files Needing Attention: isaaclab_arena/evaluation/experiment_runner_watchdog.py, osmo/workflows/arena_experiment_workflow.py, osmo/submit_arena_experiment.py

Important Files Changed

Filename	Overview
isaaclab_arena/evaluation/experiment_runner_watchdog.py	Adds process-group stall detection and restart cleanup, but newline-based liveness tracking can terminate active evaluations.
osmo/tasks/experiment_runner_task.py	Enables watchdog execution and camera recording for every OSMO experiment runner.
osmo/workflows/server_bindings.py	Introduces typed bindings that map pi0, GR00T, and Cosmos clients to server tasks and client wiring.
osmo/workflows/arena_experiment_workflow.py	Generalizes per-Run server fan-out, but does not validate effective workflow resources against server bindings.
osmo/submit_arena_experiment.py	Derives server configuration from the Experiment and adds dry-run and override-listing support.
osmo/tasks/cosmos_server_task.py	Adds the Cosmos inference-server task and safely quotes its generated command.
isaaclab_arena_gr00t/policy/gr00t_remote_closedloop_policy.py	Broadens the scheduler annotation for Hydra composition while retaining constructor validation.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  CLI[submit_arena_experiment] --> LOAD[Load typed Experiment]
  LOAD --> DERIVE[Derive required ServerBindings]
  DERIVE --> HYDRA[Apply submission overrides]
  HYDRA --> WF[ArenaExperimentWorkflow]
  WF --> RUNS{For each Run}
  RUNS -->|Local policy| RUNNER[ExperimentRunnerTask]
  RUNS -->|Remote policy| PAIR[Runner and dedicated policy server]
  RUNNER --> WATCHDOG[Watchdog launches runner]
  PAIR --> WATCHDOG
  WATCHDOG --> OUTPUT[Per-Run output]
  OUTPUT --> COLLECT[Collect Experiment outputs]

Loading

_{Reviews (1): Last reviewed commit: "Derive per-Run policy servers in the OSM..." | Re-trigger Greptile}

[arena-review-bot]

🤖 Isaac Lab-Arena Review Bot

Summary

This PR generalizes the OSMO Arena-experiment workflow from a single pi0-only path to a policy-agnostic one: a server_bindings.py registry maps each remote client-policy type to a ServerBinding (server task, resource, client wiring, compatibility check), and ArenaExperimentWorkflow derives each Run's server from its policy so pi0/GR00T/Cosmos Runs can be mixed in one Experiment. It also adds a Cosmos server task, a stall-restart watchdog, --dry_run/--list_overrides, and video recording. The refactor is clean and well-tested; my comments are a couple of default-behavior questions and two small simplifications — no blockers.

Findings

🟡 experiment_runner_task.py:44 — The watchdog wraps every Run by default and treats 600 s of silent stdout as a hang (kill + wipe output + relaunch). Isaac Sim's cold-start first-output latency can be long and silent, so a still-starting Run risks being killed in a loop — consider a longer grace before the first output.
🟡 experiment_runner_task.py:38 — Camera-video recording is now on by default for every OSMO Experiment (was off), adding encode/upload cost for all submissions. Confirm on-by-default is intended vs. opt-in.
🔵 submit_arena_experiment.py:78 — The "servers share one pool" set-and-assert is duplicated in ArenaExperimentWorkflow._assert_servers_share_one_pool; consider a single helper in server_bindings.py.
🔵 gr00t_remote_closedloop_policy.py:59 — scheduler dropped from Literal to plain str; an Enum composes under Hydra (OmegaConf supports enums) while keeping config-layer validation.

Test Coverage

Good — the workflow changes are covered (mixed pi0+GR00T fan-out, all-local no-server path, different-pool rejection, GR00T dry-run through the real CLI, --dry_run/--list_overrides). These are non-sim config/rendering tests, so the inner/outer run_simulation_app_function pattern doesn't apply. The watchdog (experiment_runner_watchdog.py) ships without unit tests and was only smoke-tested by hand; a small test of the stall/restart/give-up state machine would be worthwhile given it now gates every Run.

Verdict

Minor fixes needed

[alexmillane]

Self review 1

[alexmillane]

Self review

[alexmillane]

moar self review.

[alexmillane]

self review.

[alexmillane]

Self review.

[alexmillane]

Self review.

[cvolkcvolk]

Great change!