acceptodds
Under review as a conference paper at ICLR 2027

Dr.RL-Bench: Diagnosing Reinforcement Learning Runs with AI Agents

Abstract

Autonomous reinforcement-learning (RL) post-training requires agents to understand a training run before deciding how to improve it. Existing benchmarks primarily measure end-to-end model improvement, conflating training diagnosis with intervention design and experiment execution. We introduce Dr.RL-Bench, a benchmark that isolates diagnosis as an evidence-grounded reasoning task before intervention. Its 196 tasks are constructed from real GPU training runs and cover anomalies in data, training parameters, infrastructure, rewards, and environments. Agents freely investigate the provided code, configurations, logs, and generated responses, then report checkable Facts and evidence-supported Inferences. Expert-authored rubrics evaluate whether these findings identify relevant issues and explain their underlying causes. Experiments with eight LLM agents reveal substantial diagnostic gaps: the strongest model achieves 47.52/100, and all agents score lower on Inferences than on Facts. These results and case studies show how agents can recognize symptoms yet fail to connect dispersed evidence into a supported explanation. Dr.RL-Bench complements outcome-based evaluation by assessing what agents understand about a training run before they act.

Primary area:
datasets and benchmarks
Keywords:
Agent Evaluation, Diagnostic Analysis, AI for AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6QbPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces DR.RL-Bench, a benchmark to systematically analyze how LLM agents identify and diagnose issues in RL post-training.

Critical strengths
  • The paper tackles an important, well-motivated problem in the current literature.
Critical weaknesses
  • "Facts" and "Inferences" are used from the beginning in the paper but are only defined in Section 3.
  • How do the authors ensure the task packages were not part of the LLMs' training data? Without verification, this reintroduces the evaluation contamination that Dr.RL-Bench aims to resolve.
  • In Section 3.4, defining "the task score is 30 times Fact coverage plus 70 times Inference coverage" feels arbitrary. The following explanation is shallow and ungrounded. The authors should provide empirical justification or sensitivity ablations for this 30/70 split.
  • Should the diagnostic score be weighted by efficiency/runtime? Similar to what authors show in Fig 4 but with a proper definition of a metric and comparing methods. Factoring in efficiency might make GLM-5.3 competitive with or superior to Opus 5.
  • Crucial details are buried in the supplementary material. Important sections from App A and C should be moved to the main paper.
  • The subsections in Section 4.2 feel disconnected from each other and the main text, making them difficult to parse and synthesize.
  • Minor typo (line 256): "we'll" -> "we will".
Questions for authors

See weaknesses

Reviewer_v45TOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Dr.RL-Bench isolates diagnosis from intervention in autonomous RL post-training. Its 196 tasks provide training artifacts and hidden Fact/Inference rubrics. Eight agents investigate the scenes with CPU tools. The strongest scores 47.52/100, and all models cover fewer Inferences than Facts. Detailed cases illustrate failures to connect observations to causes.

Critical strengths
  • Separating diagnosis from repair addresses an important evaluation gap.
  • GPU-backed validation and expert rubric construction provide concrete task grounding.
  • The three case studies explain technically meaningful failure mechanisms.
Critical weaknesses
  • The score rewards Fact and Inference coverage without explicitly penalizing additional incorrect claims. This could favor speculative reports over precise diagnoses. The reported hit ratio measures rubric matches, so it cannot distinguish false claims from valid findings outside the rubric.
  • GPT-5.6 Luna determines whether findings match expert rubric items, but the paper reports no human–judge agreement statistics or judge error rates. Expert review of the rubrics does not establish the reliability of this automated matching step, particularly for subtle causal inferences.
  • Although agents share a common harness, the paper does not fully specify investigation budgets or report uncertainty across repeated runs. Provider-specific reasoning settings further complicate comparison. Thus, the robustness of model rankings to resource allocation and sampling remains unclear.
Questions for authors
  • How are incorrect claims and valid findings outside the rubric handled? Can you report expert-verified precision and test whether adding unsupported claims affects scores or rankings?
  • Can you provide a blinded human audit of accepted and rejected matches, separately for Facts and Inferences? Do model rankings remain stable with another judge?
  • What token, time, and tool-call limits were used? Can you report repeated-run confidence intervals and assess whether the rankings remain stable under matched resource budgets?
Reviewer_GL2wOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Dr.RL-Bench, a benchmark evaluating LLM agents' ability to diagnose issues in RL post-training. It contains 196 tasks across five domains, constructed from real GPU training runs with experimentally validated anomalies. Agents investigate training artifacts and report Facts and Inferences, which are evaluated against expert-annotated rubrics. Experiments with eight frontier agents reveal substantial diagnostic gaps, particularly in deriving underlying causes from observed evidence.

Critical strengths
  1. Meaningful problem formulation. Isolating diagnosis from end-to-end training optimization provides a useful perspective on autonomous AI research capabilities.

  2. Substantial benchmark construction. The benchmark covers diverse RL training issues, with real GPU experiments and controlled ablations supporting the validity of the constructed anomalies.

  3. Informative analysis. The Fact/Inference distinction and detailed case studies highlight interesting limitations in agents' ability to connect observations with underlying causes.

Critical weaknesses
  1. The anomalies are primarily designed by human experts rather than collected from actual agent-driven training failures. It remains unclear whether they represent the issues frontier agents typically encounter or introduce during autonomous RL optimization.

  2. Limited evidence of downstream relevance. Poor diagnostic performance on curated tasks does not necessarily imply that agents frequently make similar mistakes in autonomous training. The paper does not establish a relationship between benchmark scores and end-to-end optimization performance.

  3. The coverage-based scoring does not explicitly penalize incorrect or speculative diagnoses, potentially rewarding excessive reporting rather than reliable diagnosis.

  4. Insufficient evaluation validation. The paper lacks human validation of the LLM judge, human baselines, and sensitivity analysis of the manually chosen Fact/Inference weighting.

Questions for authors
  1. Are the benchmark representative of failures commonly encountered or introduced by frontier agents during actual autonomous RL post-training? The authors could provide supporting evidence from real agent-driven training trajectories.

  2. Does performance on Dr.RL-Bench correlate with agents' success in end-to-end autonomous post-training benchmarks?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

We thank all reviewers for the constructive feedback. For your convenience, we summarize our responses and revisions below, with references to the corresponding point-by-point response so that each reviewer can quickly find where their concerns are addressed (in these references, Q and W refer to the numbered questions and weaknesses listed in your review):

  • Evaluation reliability (reviewer-v45T: Q2, Q3(ii); reviewer-GL2w: W4; reviewer-6QbP: W3): we added a blinded human audit, repeated answerer runs with confidence intervals, repeated judge evaluations, and a sensitivity analysis of the Fact/Inference weights.

  • Excessive reporting and unsupported claims (reviewer-v45T: Q1; reviewer-GL2w: W3): we added original and expert-verified precision results, precision-adjusted scoring, and an experiment testing the effect of adding unsupported claims. The adjusted scores preserve the original ranking of all eight agents.

  • Resource constraints and efficiency (reviewer-v45T: Q3(i), Q3(iii); reviewer-6QbP: W4): we clarified the original resource limits and added comparisons under a tighter tool-call budget, together with a score-per-minute efficiency analysis.

  • Downstream relevance (reviewer-GL2w: W2): we added downstream evaluations and a comparison with PostTrainBench. Downstream performance improved after repair. The four models shared with PostTrainBench have exactly the same relative ranking on both benchmarks (Opus 5 > GLM-5.3 > Kimi K3 > GLM-5.2).

  • Task construction and representativeness (reviewer-GL2w: W1): we clarified why we use expert-constructed anomalies and explained what training problems the benchmark is intended to represent.

  • Training-data contamination (reviewer-6QbP: W2): we clarified that the task packages had not been publicly released or made available to external parties.

  • Presentation (reviewer-6QbP: W1, W5–W7): we clarified the definitions of Facts and Inferences at their first mention in the paper, moved important parts of appendix into the main text, strengthened the connections among the case studies, and corrected wording issues.

Beyond these specific revisions, we were encouraged that the reviews collectively recognized the importance of evaluating diagnosis as a distinct stage of RL post-training, the benchmark’s grounding in real GPU runs, and the insights into agents’ diagnostic limitations.

We have highlighted all the manuscript changes in blue in the revised PDF. We hope these additions and clarifications help address your concerns.

Thank you again for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.