acceptodds
Under review as a conference paper at ICLR 2027

RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineers

Abstract

Coding agents are beginning to move beyond purely digital tasks to tackle physical-world challenges, particularly in robotics. Existing robotics benchmarks, however, primarily focus on the performance of individual artifacts, such as policies or controllers, offering limited coverage of coding agents’ broader engineering capabilities. Real-world robotics extends beyond control: agents must build, integrate, diagnose, and improve heterogeneous artifacts under resource constraints and reason from multimodal feedback. To evaluate these broader capabilities, we introduce RLE-Bench, a benchmark of robot-learning tasks spanning four representative robotics development workflows: interactive control, policy learning, perception and estimation, and mechanical design. We use diverse task-specific metrics to evaluate the artifacts submitted by the coding agents, from the success rate the agents achieved to the policy agents trained, the harness agent built, and the mechanical structures the agent designed. We aggregate these metrics into an overall RLE Index and report workflow-specific capability profiles, enabling systematic comparison of coding agents' capabilities across multiple capability dimensions. Beyond performance ranks, we also conduct in-depth case studies examining agent behavior on representative tasks, highlighting both current capabilities and limitations, and pointing to the opportunities robotics tasks have to offer for future agent training.

Primary area:
datasets and benchmarks
Keywords:
coding agent benchmark, robot learning, agentic robotics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_E3h7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces RLE-Bench, a benchmark for evaluating coding agents across a broader robotics-engineering workflow than controller generation alone. It contains 51 tasks grouped into nine task families and four workflows: interactive control, policy learning, perception and estimation, and mechanical design. Agents develop heterogeneous artifacts under explicit resource budgets, and benchmark-controlled verifiers evaluate them under held-out conditions. The paper compares 11 model–harness combinations using a hierarchical RLE Index and workflow-specific profiles, and analyzes development trajectories, perception failures, robustness gaps, and mechanical-design limitations.

Critical strengths
  1. A broader and practically relevant evaluation scope: Including reusable harness construction, VLA training recipes, pose estimation, and hardware–controller co-design makes the benchmark meaningfully different from a collection of code-as-policy tasks. The submitted artifacts reflect several kinds of work that robotics engineers actually perform (Sections 3.1–3.2).

  2. Careful separation of development and authoritative evaluation: The artifact contracts, metered interaction budgets, and verifier-controlled hidden environments provide a useful foundation for reproducible agent evaluation. The task cards specify what information is available, what must be submitted, and how it is tested, including hidden physical variations and retraining of submitted recipes (Section 3.3 and Appendix A).

  3. Informative capability diagnostics beyond a scalar leaderboard: The paper supplements the RLE Index with per-family results, native physical metrics, and concrete behavior analyses. The differences between development and robustness scores, between static and occluded pose estimation, and between structural validity and physical stability are useful observations that aggregate rankings alone would miss (Section 4.3 and Appendix B).

Critical weaknesses
  1. The overall index is insufficiently calibrated for the breadth of its interpretation: Equal workflow weighting is defensible, but its implications are substantial: the two mechanical-design tasks together receive 25% of the index, the same weight as all 35 interactive-control tasks. Each mechanical-design task therefore contributes 12.5% despite representing a narrow part of that workflow. The normalized rewards also combine different nonlinear penalties and gates, and the Family04 task card states that four motion-specific threshold sets remain provisional. The mobile-base reference solution is useful, but does not calibrate the other workflows. Reference-based anchoring, as illustrated by MLE-bench [1], and sensitivity analysis would make the index more interpretable without implying that a particular score establishes engineering qualification.

  2. Independent-run uncertainty and the run-retention policy are unclear: Appendix B describes selected or retained runs, but the number of independent development attempts and the rule for choosing the reported run are not clearly documented. Multiple hidden evaluation seeds test the robustness of a submitted artifact; they do not measure variability in the agent's artifact-development process. Without repeated-development uncertainty or rank-stability analysis, close leaderboard differences, such as 26.8 versus 26.5, are difficult to interpret. This does not negate the large gaps, but limits the precision of the ranking claims.

  3. A central artifact/evaluation contract is inconsistent between the main text and appendix: Table 2 lists a model checkpoint as the Family05 submission and validation loss as its metric. In contrast, the Family05 task card specifies submission of solution.py, retraining in the verifier, no transfer of development checkpoints, and scoring by closed-loop success rate. These are materially different evaluation contracts, not interchangeable descriptions. The appendix appears consistent with the reported rollout analyses, but the manuscript should unambiguously identify which contract produced the leaderboard and align the main table with it.

References

[1] Jun Shern Chan, Neil Chowdhury, et al. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. arXiv:2410.07095, 2024.

Questions for authors
  1. How stable are the rankings under equal-family weighting, leave-one-family-out analysis, and reasonable variations of the provisional scoring thresholds? Reporting the unchanged supplied PPO baseline alongside agent-developed policies would also help interpret how much engineering improvement Family04 measures.

  2. How many independent development attempts were made per task–system pair, and how were selected/retained runs chosen? Can a small stratified subset be repeated to estimate agent-development variability and distinguish meaningful ranking differences from noise?

  3. Can the authors reconcile the Family05 submission and scoring contract in Table 2 with Appendix A? For Family02, please also specify whether the fresh evaluation agents use a fixed model and configuration across all submitted harnesses, so that harness quality can be interpreted separately from recipient-agent capability.

3 comments

Thank you for the detailed review and we provide further explanations in below.

1.Task score weighting and its stability (W1, Q1):

Equal-family weighting. We found that weighting all nine families equally (11.1% each) leaves the ranking of all 11 models unchanged (Kendall τ = 1.0), and no model's index changes by more than 2.0 points. The ranking therefore is not sensitive to the workflow-level weighting. We keep the workflow-balanced index so that each workflow contributes equally, however many families it contains.

Leave-one-family-out. We recompute the index nine times, each time leaving out one family, and compare each result with the full nine-family ranking. The ranking is generally stable. The mean Kendall τ is 0.943, and 74 of the 99 model–family combinations keep their rank. The top five models never move across each leave-one-family-out setup, except that GPT-5.6 Sol and Claude Opus 5 swap third and fourth when T06 is left out. Index values are similarly robust: every model except one stays within 5 points of its full-data index, and the largest change is 6.9 points (GPT-5.6 Sol).

Among all the families, leaving out T01 or T03 changes nothing. In seven of the nine cases, no model moves by more than one position. The only clear outlier is T08. Without it, T09 carries the whole Mechanical Design weight, and Claude Opus 4.8 rises from ninth to sixth (τ = 0.855) because it scores 52.0 on T09 but only 15.0 on T08.

2.Motion-specific threshold used in family 04 (W1, Q1)

We would like to clarify our use of the word provisional. In Family04, the reward is computed as

score = clip_[0,1]( 0.7 · tracking + 0.3 · survival − 0.15 · min(jerk / jerk_cap, 1) − 0.05 · fell )

where tracking measures how closely the humanoid follows the reference motion, survival measures the fraction of frames the humanoid stays upright, the jerk term penalizes non-smooth actions, and fell penalizes falling to the ground. Only one quantity, jerk_cap, is calibrated, and it is calibrated only on the dance motion; the same value is then used for all other motions. We did set this cap from the reference PPO trajectory on dance. However, the choice transfers well to the other motions. To check this, we rescored all 55 Family04 runs (11 systems) with the cap on the four non-dance motions scaled from ×0.25 to ×4, and with the jerk penalty removed entirely. In no setting does any system change rank, either within Family04 or in the RLE Index. Family04 scores change by at most 7.7 points, and the RLE Index by at most 0.96 points. All systems shift together because their median jerk on these motions (0.13–0.18) is close to that on dance (0.16) and stays below the cap.

J_max scaleRank changes (Family04 / RLE-Index)Max abs. Δs, Family04Max abs. Δs, Index
×0.250 / 07.70.96
×0.50 / 05.30.67
×20 / 03.70.46
×40 / 05.60.70
No jerk penalty0 / 07.40.93

The reference PPO scores are in the following table.

MotionReference Score
dance0.38
fight0.00
fall-and-get-up0.00
run0.03
sprint0.31

3.Reference-based anchoring (W1)

We thank the reviewer and agree that an RLE-Bench Index should not be read as certifying engineering qualification. We will state clearly that the RLE Index is a relative measure under a fixed protocol. We also agree that if every task had a reference solution, anchoring as in MLE-Bench would be a reasonable way to scale per-task scores. However, many RLE-Bench tasks have no reference solution, notably the interactive-control and harness-engineering tasks. Moreover, stronger systems tend to do better on both easier and harder tasks, so the index remains informative when used to compare systems.

4.Independent run uncertainty (W2, Q2)

We thank the reviewer for this suggestion. Within the rebuttal period, we re-ran T07, T08 and T09 four more times for nine of the 11 paper systems. We chose these families because each has a single subtask, so its score rests on one development run. We recomputed the RLE Index with each attempt's T07–T09 scores, keeping the published T01–T06 scores. Index values vary across attempts (std 0.6–5.0 points), but the ranking is stable. The top five never change, no system moves by more than one position, and Kendall τ against the paper ranking is at least 0.89. We will extend these repeated runs to all task families in the next revision.

ModelA1 (paper)A2A3A4A5Index range
GPT-6 Astra1111172.0–73.5
Claude Fable 5.12222260.6–64.0
Claude Opus 53333353.4–59.2
GPT-5.6 Sol4444435.8–48.7
Grok 4.65555529.0–34.3
GLM-5.3 Flash6767719.3–27.6
Claude Opus 4.87676624.3–29.8
GPT-5.6 Luna8988813.0–20.3
GPT-5.6 Terra9899913.0–16.8

5.Run retention policy (W2, Q2)

We apologize for the ambiguity in Appendix B. In the draft, we only run 1 seed per task for each system and report its reward, without cherry-picking or choosing the best from multiple runs. We have revised Appendix B to avoid this ambiguity.

6.Family05 contract mismatch (W3, Q3)

Thank you for spotting this inconsistency. The protocol reported in the appendix is correct: agents submit solution.py, the verifier retrains the model from scratch, and evaluates the model on test suites. We have updated Table 2 accordingly.

7.Family02 evaluation agents (Q3)

In Family02, the fresh evaluation agent is not a single fixed model. Each harness is evaluated by a fresh instance of the same model that developed it, and all other settings are held fixed across harnesses. This is deliberate: Family02 measures an agent’s ability to build tools for its own use, not harness quality in isolation. Different agents behave differently, and a tool's usefulness depends on who uses it. Self-use evaluation therefore better predicts how an agent will perform when it relies on tools it has built itself.

The task02 itself is also designed so that the score depends on the library rather than on raw recipient capability. Each Task02 test episode allows only 5k steps, compared with 50k steps for Task01 development. Under this budget, the task is very hard to solve zero-shot, so the agent must rely on what its library provides.

Thank you for the thorough responses and the substantial additional experiments. I will maintain my original score.

Reviewer_46k1OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

RLE-Bench evaluates coding agents on robot-learning engineering tasks spanning interactive control, policy development, perception and estimation, and mechanical design. It comprises 51 tasks across nine families, evaluates executable artifacts under resource constraints, and aggregates performance into an RLE Index.

Critical strengths

A meaningful expansion of robotics evaluation. The benchmark addresses an important gap by evaluating agents’ ability to build controllers, training pipelines, reusable tools, perception systems, and mechanical designs. This broader scope provides useful coverage of engineering capabilities beyond policy execution.

Thoughtful artifact-based evaluation. Benchmark-owned verification, hidden evaluation conditions, and explicit artifact contracts strengthen the evaluation. Particularly valuable designs include transferring a generated harness to fresh agents and replaying submitted training code in an isolated environment. These test whether agents produce reusable engineering artif

Critical weaknesses

The scoring rules are insufficiently calibrated against engineering quality. In Family08, all evaluated systems receive zero worst-arm credit for static stability, yet submissions can still achieve substantial overall scores, including 61.3/100. Although this does not imply failure under every operating condition, it raises concerns about allowing other attributes to compensate for a fundamental design limitation. Family04 also explicitly describes four of its five motion-specific threshold sets as provisional. Without stronger calibration against reference solutions or expert judgments, the engineering meaning of these scores remains uncertain.

The reliability of the aggregate ranking is not sufficiently established. The 51 tasks include related configurations, while mechanical design is represented by only two joint tasks that together determine 25% of the RLE Index. Equal weighting across workflows is defensible, but the resulting ranking may depend strongly on a small number of task choices and scoring conventions. The paper does not systematically report sensitivity to these choices or variability across repeated, complete agent-development sessions. Multiple hidden evaluation seeds do not capture the latter source of uncertainty.

Questions for authors

How were scoring thresholds and weights calibrated? In particular, what justifies the provisional motion-tracking thresholds and the compensatory scoring of static stability in Family08?

2 comments

1. How the scoring rules are calibrated (W1, Q1)?

As in other multi-criteria benchmarks such as PaperBench and HealthBench, the scoring weights in RLE-Bench are set by experts according to the importance of each criterion. We acknowledge that this inevitably introduces some subjectivity, which no fixed weighting can fully avoid. In the below, we address the two concerns in turn.

For the Family08, the static stability test holds the arm at full horizontal reach with the maximum payload and sweeps it through its entire slewing range while the base is at rest. A zero score means that the design fails to maintain the required static margin, which reflects the core difficulty of designing a mobile base. The overall score is a weighted sum of five criteria, and static stability does not come at the expense of the others. Our reference design meets the static requirement while performing well on the other criteria, scoring 90.4 compared with 61.3 for the best system. Stronger designs should try to achieve stability, not to trade it away.

For the motion-tracking thresholds used in Family04, Each episode is scored as

score = clip_[0,1]( 0.7 · tracking + 0.3 · survival − 0.15 · min(jerk / jerk_cap, 1) − 0.05 · fell )

where

  • tracking measures how closely the humanoid follows the reference motion;
  • survival is the fraction of the clip completed before the humanoid deviates too far from the reference;
  • the jerk term penalizes non-smooth actions;
  • fell penalizes such early termination.

Only one quantity, jerk_cap, requires calibration. It is set to twice the action jerk of a one-hour reference PPO policy on dance, and the same value is reused for the other four motions. We had two observations suggesting that this choice transfers well. First, the penalty visibly reduces jerky motion on all five motions. Second, scaling jerk_cap on the four non-dance motions from ×0.25 to ×4, or removing the jerk penalty entirely, changes no system's rank, either within Family04 or in the RLE Index (table below). The rankings are therefore robust to this design choice.

jerk_cap scaleRank changes (Family04 / Index)Max Δ, Family04Max Δ, Index
×0.250 / 07.70.96
×0.50 / 05.30.67
×20 / 03.70.46
×40 / 05.60.70
No jerk penalty0 / 07.40.93

2. Reliability of the aggregate ranking (W2):

We interpret this question as two things: whether any single family has too much influence on the RLE Index, and whether run-to-run variance within a family can change the index substantially.

Influence of individual families. We ran a leave-one-family-out analysis. We recomputed the index nine times, each time omitting one family, and compared each result with the full nine-family ranking. The ranking is generally stable. The mean Kendall τ is 0.943, and 74 of the 99 model–family combinations keep their rank. The top five models never move, except that GPT-5.6 Sol and Claude Opus 5 swap third and fourth when T06 is omitted. Omitting T01 or T03 changes nothing, and in seven of the nine cases no model moves by more than one position. The only clear outlier is T08. Without it, T09 carries the full Mechanical Design weight, and Claude Opus 4.8 rises from ninth to sixth (τ = 0.855), because it scores 52.0 on T09 but only 15.0 on T08. Index values are similarly robust. Every model except one stays within 5 points of its full-data index, and the largest change is 6.9 points (GPT-5.6 Sol).

Run-to-run variance. To measure that variability, we ran four additional independent attempts of T07, T08 and T09 for nine of the 11 paper systems during the rebuttal period. We chose these families because each has a single subtask, so each family score comes from one development run. We recomputed the RLE Index with each attempt's T07–T09 scores, keeping the published T01–T06 scores. Index values vary across attempts (std 0.6–5.0 points), but the ranking is stable. The top five never change, no system moves by more than one position, and Kendall τ against the paper ranking is at least 0.89.

ModelA1 (paper)A2A3A4A5Index range
GPT-6 Astra1111172.0–73.5
Claude Fable 5.12222260.6–64.0
Claude Opus 53333353.4–59.2
GPT-5.6 Sol4444435.8–48.7
Grok 4.65555529.0–34.3
GLM-5.3 Flash6767719.3–27.6
Claude Opus 4.87676624.3–29.8
GPT-5.6 Luna8988813.0–20.3
GPT-5.6 Terra9899913.0–16.8

Both analyses show that the ranking is informative and robust when comparing model performances. No single family dominates the workflow-balanced index, and run-to-run variance changes the order of models only through swaps between close neighbours. We agree that the most complete remedy is to add more tasks to each workflow to amortize the stochasticity, and we leave this to future work.

Reviewer_KpHxOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the exciting new topic of Code-as-Policy where AI agents write code for robots. The objective of this paper is to provide a more extensive suite of benchmarks noting that AI agents can code interactve control, but they can also serve as "robot learning engineers" that create protocols for policy learning, algorithms (skills) for perception, and design of mechanical artifacts. The benchmark includes 51 simulated tasks organized into 4 categories and 9 families, with associated performance metrics. The benchmark also specifies a score for each task in the range [0,1] so that scores can be averaged to produce an overall Robot-Learning-Engineer score: RLE-Index. The paper then uses this benchmark to evaluate 11 model and harness combinations (eg, Claude Fable 5.1 with Claude Code) and provide a leaderboard in Figure 1.

Critical strengths

The idea of benchmarking capabilities beyond robot control is very valuable. The methodology is reasonable and extensive results are reported.

The 7 "Findings" listed are very helpful summaries of the results!

Critical weaknesses

One weakness is that all evaluations are solely in simulation. It would be very helpful to also include some physical (real-world) evaluations.

Also, the combinations of Models and Harnesses in Table 3 should include more combinations across companies / sources, eg, Claude Fable 5.1 as Model withj GPT-6 Astra as Harness and vice versa.

The overall RLE-Index should be weighted by difficulty. Ie, some tasks are very easy and they should not be weighted less than more difficult tasks.

Questions for authors

Could you include some physical (real-world) evaluations?

Could you include more combinations across companies / sources, eg, Claude Fable 5.1 as Model withj GPT-6 Astra as Harness and vice versa?

Could you weight the overall RLE-Index by difficulty? Ie, some tasks are very easy and they should not be weighted less than more difficult tasks.

1 comment

Thank you for the careful review and for the kind suggestions. We answer each point below.

1. Simulation only, no real-world tests (W1, Q1).

We agree that real-world tests would be valuable. We chose simulation on purpose: every agent must be scored under the same hidden conditions, many times, and at a manageable cost. With real hardware, it is hard to keep conditions fair across 11 systems, since hardware conditions may vary over time. We have stated this limitation in the conclusion section of the paper.

2. More combinations across companies (W2, Q2).

Thank you for this suggestion. We would like to note that even when frontier models are trained across multiple harnesses, they may be further optimized for their providers’ own harnesses, such as Claude Code for Claude models and Codex for GPT models. To test how harness choice affects performance, we evaluated Claude Opus 5.5 in both Claude Code and OpenAI’s Codex CLI across all 15 task01 cells.

HarnessL1L2L3task01 meanMean successCost
Claude Code0.7130.5580.7470.6730.76$151
Codex CLI0.6080.5860.4260.5400.63$279

With Codex CLI, the same model scored 0.133 lower on average (0.540 vs. 0.673), and its mean success rate fell by 13 percentage points (from 76% to 63%). Given this observation, we chose the benchmark these models paired with their providers' own harness in our draft.

For open-weight models, we selected the harness empirically. In a preliminary comparison using DeepSeek-v4.1-flash, we evaluated OpenCode, Claude Code, and Codex CLI. Claude Code achieved the highest score in that comparison, so we selected it for our open-weight model evaluations. We also would like to note that Claude Code is a popular choice for evaluating open-weight models, for example, Terminal Bench v4.1 (https://www.tbench.ai/).

3. Weighting the RLE Index by difficulty (W3, Q3).

We thank the reviewer for this suggestion. We chose the uniform weighting for the RLE-Index on purpose. Task difficulty has no objective, stable definition: expert labels are subjective, while labels derived from model performances depend on which models are evaluated. Performance-based weights would also have to be recomputed as models improve. The score of an unchanged model could then shift just because new models were added, and rankings would stop being comparable over time, which contradicts the goal of a lasting reference benchmark. Uniform weighting, instead, fixes the scoring rule in advance and keeps it independent of any model population. It also already lets harder tasks drive the separation between models, since easy tasks generally contribute nearly the same score to every model at the frontier level.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.