acceptodds
Under review as a conference paper at ICLR 2027

Agentic Auto-Rubrics: Verifiable Rubrics for Judging Multi-Turn Coding Agents

Abstract

Judging whether a coding agent fulfills a user's request is challenging in multi-turn sessions. In multi-turn sessions, user requirements constantly change and replace earlier requests, while user reactions can unintentionally reveal task outcome and bias evaluation. We introduce , a framework that compiles a multi-turn interaction into a verifiable specification of what the agent is responsible for. Given reaction-masked user turns, AAR constructs the active requirements and compiles them into fine-grained atomic rubrics, which specify explicit evidence types and codebase locations. To increase the judge's efficiency, AAR compresses similar atomic rubrics into a concise set of high-level parent rubrics. An agentic verifier then inspects the agent's workspace through read-only tools, evaluating these rubrics from concrete file-based evidence and references. Across multi- and single-turn coding benchmarks, AAR consistently outperforms direct scoring judges and existing rubric-based verifiers. While baseline verifiers degrade as sessions lengthen, AAR maintains stable accuracy across session lengths and five judge models, showing that compiling evolving interactions into verifiable specifications enables robust evaluation of coding agents.

Primary area:
interpretability and explainable AI
Keywords:
AutoRubrics, Coding Agents, Multi-Turn Evaluation, LLM-as-a-Judge

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8XDWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Agentic Auto-Rubrics. It masks responses in multi-turn user utterances, compiles them into requirements, compresses those requirements into a parent rubric, and uses a reviewer with a read-only workspace to give a success probability. Experiments cover Arena, SWE-Together, R2E-Gym, SWE-Gym, and SWE-bench Verified, and the main table compares direct scoring, checklists, and two existing rubric methods under the same GPT-5.5 reviewer.

Critical strengths

The main table places these methods under the same reviewer. Tables 4 and 5 report AUROC and same-task paired accuracy, and Section 4.2 states which gaps are not distinguished.

Critical weaknesses

The stable advantage in the abstract is not in Table 4. Under the same GPT-5.5 reviewer, AgenticRubrics is .777 and AAR is .763 on SWE-Together. The main text states that on multi-turn benchmarks only the gap versus AgenticRubrics on Arena is distinguished, and that on SWE-Together none of these gaps are distinguished. On single-turn SWE-Gym, AdaRubric is .901 and AAR is .881. In Table 5, paired selection on SWE-bench is .778 for direct scoring and .775 for AAR.

Parent-level compression and "does not drop as the session grows longer" also have no corresponding advantage. AAR-ATOMIC and AAR differ by no more than .013 on every benchmark, and on Arena the scores are .832 and .835. In Figure 2, AAR is .721 on the shortest Arena and the checklist is .750. Section 5 states that under the same reviewer these rubrics differ from each other by no more than .07 on SWE-Together and on single-turn settings.

Questions for authors
  1. What are the differences and intervals between AAR and AgenticRubrics, and between AAR and AdaRubric, on Arena and SWE-Together? By what rule is "resolved" decided?
  2. What is the AUROC of each reviewer in each bin of Figure 2? Is the abstract statement that baselines drop as sessions grow longer consistent with these numbers?
Reviewer_nzk1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents AAR: Agentic Auto Rubrics, a method for producing a probability of success on a given SWE problem using self-generated and evaluated rubrics. This method has applications in automated evaluation of coding agents, test time roll-out selection, and providing training rewards. Authors find AAR improves over strong baselines as a classifier (AUROC, Table 4) and selector (paired-acc, Table 5) of SWE agent trajectories.

Critical strengths
  1. The method is novel and well motivated. While relatively complex, consisting of multiple steps, I found the motivation for each component to be clear.
  2. The baselines are well chosen and defined: those from prior work are highly relevant, and the provided baselines carefully ablate the key components of the proposed method. The datasets are similarly well motivated.
  3. Despite some shortcomings addressed in the next section, the results are promising.
  4. The writing and presentation is quite clear and reflect careful work from the authors.
Critical weaknesses
  1. I think the main weakness of this work is in the results:
    • No analysis of cost (tokens used, or $ cost for estimation) supports using AAR over other rubrics or evaluation methods. The method is novel but relatively complex and likely costly to run at inference time. Adding analysis which compares the cost-performance trade-off of this evaluator over others would be helpful.
    • AdaRubrics and AgenticRubrics are omitted from Figure 2 and included in Appendix E, but the separation isn’t well justified: Figure 2 makes it appear that AAR is the best performing across turns on SWE-Together but the other rubric methods actually perform best. I think Figure 2 should just have all lines included.
    • AAR doesn’t show consistent improvement over existing rubric methods (AgenticRubrics & AdaRubrics), seeing sizable improvement on Arena (un-cited?) but shortcomings on SWE-Together. It isn’t conclusive which method will best generalize to new evaluation settings.
    • Figure 3 only compares to the holistic judge, which is the weakest baseline. While I think including all other methods might be costly, maybe one additional method would help convincingly show that AAR is the best rubric method across judges.
    • I think Table 1 would be improved with significance tests or confidence intervals, as the evaluation sets vary substantially in size.
  2. I think the paper is about a page short and can use the additional space to add further analyses of the method and its components.
Questions for authors
  1. “Arena is the one benchmark whose requirements were written by real users over real sessions.” (L416). Is this actually true? My understanding is SWE-Together constructs tasks from the real-user conversations in SWE-Chat. While single-turn, SWE-Bench problems also come from real-users.
  2. I think the read-only evaluator scope is justifiable but maybe not by cost alone: is there a cost-reason to not allow the evaluator or judge to use rubrics that run self-generated execution tests?
Reviewer_spp2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Agentic Auto Rubrics (AAR), a framework to create eval rubrics for multi-turn, interactive agent sessions for coding agents. ARR extracts user requirements from the conversation and updates them as the session proceeds to capture evolving requirements by the user. User reactions are masked to avoid shortcuts. The final rubric is scored against the workspace artefacts.

Critical strengths
  • S1: the central idea of evolving the requirements as users add, refine or drop them during an interaction is interesting and well motivated.

  • S2: the paper is overall well written and clearly presented. The difference between different methods is contrasted in table 2. However, there are some aspects where I am not sure I understand the paper correctly - see questions

Critical weaknesses
  • W1: experiments in table 4 include 2 experiments in multi-turn becnhmarks (where the focus of the AAR is) but AAR only improves over the baslines on one. While ARR is better on average, the method is not clearly better on all benchmarks (not a criticla weakness

  • W2: some of the differences in the results are quite small , eg. ARR 0.713 vs ReqCHecklist 0.716 in table 5. Statistical significance tests or error bars would help to understand if these differences are significant or not

Questions for authors

Q1: What is the motivation for masking reactions? If AAR is mainly intended for "offline" evals where the user-agent interaction is already known before the rubrics are created and the interaction is scored, then I don't see a strong need for masking. l. 130 "Give an interaction session .." suggests that the full session is already available. Of course the user reactions can give away whether the agent did the correct action (eg. "great", "thank you") but in an offline setting I don't see this as a problem, frankly. Of course a rubric that relies on these user reactions will likely not generalise well to new interactions, eg in online monitoring of agents.

Q2: Are the rubrics created for each test case separately or are they supposed to generalise to an entire test set? The example is figure 1 "Check cli flag with -A" suggests that rubrics are tailor-mode for each test case. However, l. 61 says "allowing the same rubrics to evaluate diverse candidate models fairly and consistently." which suggests that they the same rubric is re-used for the different models under test but is the same rubric supposed to capture general rules/policies across test cases?

Q3: Section 3.4. How can you ensure that you can compress the rubrics into at most 10 parent rubrics? Is it possible that some rubrics need to be dropped? Figure 1 suggests that similar rubrics are clustered together but this might not always be possible.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.