- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces RLE-Bench, a benchmark for evaluating coding agents across a broader robotics-engineering workflow than controller generation alone. It contains 51 tasks grouped into nine task families and four workflows: interactive control, policy learning, perception and estimation, and mechanical design. Agents develop heterogeneous artifacts under explicit resource budgets, and benchmark-controlled verifiers evaluate them under held-out conditions. The paper compares 11 model–harness combinations using a hierarchical RLE Index and workflow-specific profiles, and analyzes development trajectories, perception failures, robustness gaps, and mechanical-design limitations.
Critical strengths
-
A broader and practically relevant evaluation scope: Including reusable harness construction, VLA training recipes, pose estimation, and hardware–controller co-design makes the benchmark meaningfully different from a collection of code-as-policy tasks. The submitted artifacts reflect several kinds of work that robotics engineers actually perform (Sections 3.1–3.2).
-
Careful separation of development and authoritative evaluation: The artifact contracts, metered interaction budgets, and verifier-controlled hidden environments provide a useful foundation for reproducible agent evaluation. The task cards specify what information is available, what must be submitted, and how it is tested, including hidden physical variations and retraining of submitted recipes (Section 3.3 and Appendix A).
-
Informative capability diagnostics beyond a scalar leaderboard: The paper supplements the RLE Index with per-family results, native physical metrics, and concrete behavior analyses. The differences between development and robustness scores, between static and occluded pose estimation, and between structural validity and physical stability are useful observations that aggregate rankings alone would miss (Section 4.3 and Appendix B).
Critical weaknesses
-
The overall index is insufficiently calibrated for the breadth of its interpretation: Equal workflow weighting is defensible, but its implications are substantial: the two mechanical-design tasks together receive 25% of the index, the same weight as all 35 interactive-control tasks. Each mechanical-design task therefore contributes 12.5% despite representing a narrow part of that workflow. The normalized rewards also combine different nonlinear penalties and gates, and the Family04 task card states that four motion-specific threshold sets remain provisional. The mobile-base reference solution is useful, but does not calibrate the other workflows. Reference-based anchoring, as illustrated by MLE-bench [1], and sensitivity analysis would make the index more interpretable without implying that a particular score establishes engineering qualification.
-
Independent-run uncertainty and the run-retention policy are unclear: Appendix B describes selected or retained runs, but the number of independent development attempts and the rule for choosing the reported run are not clearly documented. Multiple hidden evaluation seeds test the robustness of a submitted artifact; they do not measure variability in the agent's artifact-development process. Without repeated-development uncertainty or rank-stability analysis, close leaderboard differences, such as 26.8 versus 26.5, are difficult to interpret. This does not negate the large gaps, but limits the precision of the ranking claims.
-
A central artifact/evaluation contract is inconsistent between the main text and appendix: Table 2 lists a model checkpoint as the Family05 submission and validation loss as its metric. In contrast, the Family05 task card specifies submission of
solution.py, retraining in the verifier, no transfer of development checkpoints, and scoring by closed-loop success rate. These are materially different evaluation contracts, not interchangeable descriptions. The appendix appears consistent with the reported rollout analyses, but the manuscript should unambiguously identify which contract produced the leaderboard and align the main table with it.
References
[1] Jun Shern Chan, Neil Chowdhury, et al. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. arXiv:2410.07095, 2024.
Questions for authors
-
How stable are the rankings under equal-family weighting, leave-one-family-out analysis, and reasonable variations of the provisional scoring thresholds? Reporting the unchanged supplied PPO baseline alongside agent-developed policies would also help interpret how much engineering improvement Family04 measures.
-
How many independent development attempts were made per task–system pair, and how were selected/retained runs chosen? Can a small stratified subset be repeated to estimate agent-development variability and distinguish meaningful ranking differences from noise?
-
Can the authors reconcile the Family05 submission and scoring contract in Table 2 with Appendix A? For Family02, please also specify whether the fresh evaluation agents use a fixed model and configuration across all submitted harnesses, so that harness quality can be interpreted separately from recipient-agent capability.
3 comments
4.Independent run uncertainty (W2, Q2)
We thank the reviewer for this suggestion. Within the rebuttal period, we re-ran T07, T08 and T09 four more times for nine of the 11 paper systems. We chose these families because each has a single subtask, so its score rests on one development run. We recomputed the RLE Index with each attempt's T07–T09 scores, keeping the published T01–T06 scores. Index values vary across attempts (std 0.6–5.0 points), but the ranking is stable. The top five never change, no system moves by more than one position, and Kendall τ against the paper ranking is at least 0.89. We will extend these repeated runs to all task families in the next revision.
| Model | A1 (paper) | A2 | A3 | A4 | A5 | Index range |
|---|---|---|---|---|---|---|
| GPT-6 Astra | 1 | 1 | 1 | 1 | 1 | 72.0–73.5 |
| Claude Fable 5.1 | 2 | 2 | 2 | 2 | 2 | 60.6–64.0 |
| Claude Opus 5 | 3 | 3 | 3 | 3 | 3 | 53.4–59.2 |
| GPT-5.6 Sol | 4 | 4 | 4 | 4 | 4 | 35.8–48.7 |
| Grok 4.6 | 5 | 5 | 5 | 5 | 5 | 29.0–34.3 |
| GLM-5.3 Flash | 6 | 7 | 6 | 7 | 7 | 19.3–27.6 |
| Claude Opus 4.8 | 7 | 6 | 7 | 6 | 6 | 24.3–29.8 |
| GPT-5.6 Luna | 8 | 9 | 8 | 8 | 8 | 13.0–20.3 |
| GPT-5.6 Terra | 9 | 8 | 9 | 9 | 9 | 13.0–16.8 |
5.Run retention policy (W2, Q2)
We apologize for the ambiguity in Appendix B. In the draft, we only run 1 seed per task for each system and report its reward, without cherry-picking or choosing the best from multiple runs. We have revised Appendix B to avoid this ambiguity.
6.Family05 contract mismatch (W3, Q3)
Thank you for spotting this inconsistency. The protocol reported in the appendix is correct: agents submit solution.py, the verifier retrains the model from scratch, and evaluates the model on test suites. We have updated Table 2 accordingly.
7.Family02 evaluation agents (Q3)
In Family02, the fresh evaluation agent is not a single fixed model. Each harness is evaluated by a fresh instance of the same model that developed it, and all other settings are held fixed across harnesses. This is deliberate: Family02 measures an agent’s ability to build tools for its own use, not harness quality in isolation. Different agents behave differently, and a tool's usefulness depends on who uses it. Self-use evaluation therefore better predicts how an agent will perform when it relies on tools it has built itself.
The task02 itself is also designed so that the score depends on the library rather than on raw recipient capability. Each Task02 test episode allows only 5k steps, compared with 50k steps for Task01 development. Under this budget, the task is very hard to solve zero-shot, so the agent must rely on what its library provides.
Thank you for the thorough responses and the substantial additional experiments. I will maintain my original score.
Thank you for the detailed review and we provide further explanations in below.
1.Task score weighting and its stability (W1, Q1):
Equal-family weighting. We found that weighting all nine families equally (11.1% each) leaves the ranking of all 11 models unchanged (Kendall τ = 1.0), and no model's index changes by more than 2.0 points. The ranking therefore is not sensitive to the workflow-level weighting. We keep the workflow-balanced index so that each workflow contributes equally, however many families it contains.
Leave-one-family-out. We recompute the index nine times, each time leaving out one family, and compare each result with the full nine-family ranking. The ranking is generally stable. The mean Kendall τ is 0.943, and 74 of the 99 model–family combinations keep their rank. The top five models never move across each leave-one-family-out setup, except that GPT-5.6 Sol and Claude Opus 5 swap third and fourth when T06 is left out. Index values are similarly robust: every model except one stays within 5 points of its full-data index, and the largest change is 6.9 points (GPT-5.6 Sol).
Among all the families, leaving out T01 or T03 changes nothing. In seven of the nine cases, no model moves by more than one position. The only clear outlier is T08. Without it, T09 carries the whole Mechanical Design weight, and Claude Opus 4.8 rises from ninth to sixth (τ = 0.855) because it scores 52.0 on T09 but only 15.0 on T08.
2.Motion-specific threshold used in family 04 (W1, Q1)
We would like to clarify our use of the word provisional. In Family04, the reward is computed as
score = clip_[0,1]( 0.7 · tracking + 0.3 · survival − 0.15 · min(jerk / jerk_cap, 1) − 0.05 · fell )
where tracking measures how closely the humanoid follows the reference motion, survival measures the fraction of frames the humanoid stays upright, the jerk term penalizes non-smooth actions, and fell penalizes falling to the ground. Only one quantity, jerk_cap, is calibrated, and it is calibrated only on the dance motion; the same value is then used for all other motions. We did set this cap from the reference PPO trajectory on dance. However, the choice transfers well to the other motions. To check this, we rescored all 55 Family04 runs (11 systems) with the cap on the four non-dance motions scaled from ×0.25 to ×4, and with the jerk penalty removed entirely. In no setting does any system change rank, either within Family04 or in the RLE Index. Family04 scores change by at most 7.7 points, and the RLE Index by at most 0.96 points. All systems shift together because their median jerk on these motions (0.13–0.18) is close to that on dance (0.16) and stays below the cap.
The reference PPO scores are in the following table.
3.Reference-based anchoring (W1)
We thank the reviewer and agree that an RLE-Bench Index should not be read as certifying engineering qualification. We will state clearly that the RLE Index is a relative measure under a fixed protocol. We also agree that if every task had a reference solution, anchoring as in MLE-Bench would be a reasonable way to scale per-task scores. However, many RLE-Bench tasks have no reference solution, notably the interactive-control and harness-engineering tasks. Moreover, stronger systems tend to do better on both easier and harder tasks, so the index remains informative when used to compare systems.