- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Agentic Auto-Rubrics. It masks responses in multi-turn user utterances, compiles them into requirements, compresses those requirements into a parent rubric, and uses a reviewer with a read-only workspace to give a success probability. Experiments cover Arena, SWE-Together, R2E-Gym, SWE-Gym, and SWE-bench Verified, and the main table compares direct scoring, checklists, and two existing rubric methods under the same GPT-5.5 reviewer.
Critical strengths
The main table places these methods under the same reviewer. Tables 4 and 5 report AUROC and same-task paired accuracy, and Section 4.2 states which gaps are not distinguished.
Critical weaknesses
The stable advantage in the abstract is not in Table 4. Under the same GPT-5.5 reviewer, AgenticRubrics is .777 and AAR is .763 on SWE-Together. The main text states that on multi-turn benchmarks only the gap versus AgenticRubrics on Arena is distinguished, and that on SWE-Together none of these gaps are distinguished. On single-turn SWE-Gym, AdaRubric is .901 and AAR is .881. In Table 5, paired selection on SWE-bench is .778 for direct scoring and .775 for AAR.
Parent-level compression and "does not drop as the session grows longer" also have no corresponding advantage. AAR-ATOMIC and AAR differ by no more than .013 on every benchmark, and on Arena the scores are .832 and .835. In Figure 2, AAR is .721 on the shortest Arena and the checklist is .750. Section 5 states that under the same reviewer these rubrics differ from each other by no more than .07 on SWE-Together and on single-turn settings.
Questions for authors
- What are the differences and intervals between AAR and AgenticRubrics, and between AAR and AdaRubric, on Arena and SWE-Together? By what rule is "resolved" decided?
- What is the AUROC of each reviewer in each bin of Figure 2? Is the abstract statement that baselines drop as sessions grow longer consistent with these numbers?