- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces DR.RL-Bench, a benchmark to systematically analyze how LLM agents identify and diagnose issues in RL post-training.
Critical strengths
- The paper tackles an important, well-motivated problem in the current literature.
Critical weaknesses
- "Facts" and "Inferences" are used from the beginning in the paper but are only defined in Section 3.
- How do the authors ensure the task packages were not part of the LLMs' training data? Without verification, this reintroduces the evaluation contamination that Dr.RL-Bench aims to resolve.
- In Section 3.4, defining "the task score is 30 times Fact coverage plus 70 times Inference coverage" feels arbitrary. The following explanation is shallow and ungrounded. The authors should provide empirical justification or sensitivity ablations for this 30/70 split.
- Should the diagnostic score be weighted by efficiency/runtime? Similar to what authors show in Fig 4 but with a proper definition of a metric and comparing methods. Factoring in efficiency might make GLM-5.3 competitive with or superior to Opus 5.
- Crucial details are buried in the supplementary material. Important sections from App A and C should be moved to the main paper.
- The subsections in Section 4.2 feel disconnected from each other and the main text, making them difficult to parse and synthesize.
- Minor typo (line 256): "we'll" -> "we will".
Questions for authors
See weaknesses
Summary of Responses and Revisions
We thank all reviewers for the constructive feedback. For your convenience, we summarize our responses and revisions below, with references to the corresponding point-by-point response so that each reviewer can quickly find where their concerns are addressed (in these references, Q and W refer to the numbered questions and weaknesses listed in your review):
Evaluation reliability (reviewer-v45T: Q2, Q3(ii); reviewer-GL2w: W4; reviewer-6QbP: W3): we added a blinded human audit, repeated answerer runs with confidence intervals, repeated judge evaluations, and a sensitivity analysis of the Fact/Inference weights.
Excessive reporting and unsupported claims (reviewer-v45T: Q1; reviewer-GL2w: W3): we added original and expert-verified precision results, precision-adjusted scoring, and an experiment testing the effect of adding unsupported claims. The adjusted scores preserve the original ranking of all eight agents.
Resource constraints and efficiency (reviewer-v45T: Q3(i), Q3(iii); reviewer-6QbP: W4): we clarified the original resource limits and added comparisons under a tighter tool-call budget, together with a score-per-minute efficiency analysis.
Downstream relevance (reviewer-GL2w: W2): we added downstream evaluations and a comparison with PostTrainBench. Downstream performance improved after repair. The four models shared with PostTrainBench have exactly the same relative ranking on both benchmarks (Opus 5 > GLM-5.3 > Kimi K3 > GLM-5.2).
Task construction and representativeness (reviewer-GL2w: W1): we clarified why we use expert-constructed anomalies and explained what training problems the benchmark is intended to represent.
Training-data contamination (reviewer-6QbP: W2): we clarified that the task packages had not been publicly released or made available to external parties.
Presentation (reviewer-6QbP: W1, W5–W7): we clarified the definitions of Facts and Inferences at their first mention in the paper, moved important parts of appendix into the main text, strengthened the connections among the case studies, and corrected wording issues.
Beyond these specific revisions, we were encouraged that the reviews collectively recognized the importance of evaluating diagnosis as a distinct stage of RL post-training, the benchmark’s grounding in real GPU runs, and the insights into agents’ diagnostic limitations.
We have highlighted all the manuscript changes in blue in the revised PDF. We hope these additions and clarifications help address your concerns.
Thank you again for your time and effort.