acceptodds
Under review as a conference paper at ICLR 2027

RAF: RUBRICS AUTO-GENERATION FRAMEWORK FOR AUDITABLE EVALUATION OF LONG-HORIZON LLM AGENTS

Abstract

Evaluating LLM agents with rubrics is a popular approach, which specifies a set of textual criteria with per-criterion weights, together with a scoring rule that ag- gregates criterion-level judgments into a single score. However, it is challenging to design effective rubrics due to rubric bias and score instability, which can lead to unreliable evaluation results that deviate substantially from human judgments. In this paper, we address these challenges by proposing a novel rubric generation method. First, we introduce an AND–OR tree as the underlying rubric structure, which organizes criteria into two types of nodes: leaf nodes, each representing an atomic and verifiable criterion, and condition nodes, which hierarchically ag- gregate the judgments of leaf nodes through AND/OR conditions. Secondly, we introduce Rubric Auto-generation Framework (RAF) that automatically and itera- tively optimizes rubrics within AND-OR tree structure to improve their alignment with human judgments. Furthermore, we construct a benchmark comprising 121 long-horizon tasks from personal-assistant scenarios, annotated by human experts. On this benchmark, RAF outperforms baselines by a clear margin: it improves the Pearson correlation with human by 0.136 over the strongest baseline, reaching 0.8102, and reduces the average scoring variance from 10.18 to 3.36.

Primary area:
datasets and benchmarks
Keywords:
rubric-based evaluation, LLM agents, agent benchmarking, automatic rubric generation, LLM-as-a-judge, open-ended evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ihudOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies rubric-based evaluation of long-horizon LLM agents and targets two problems: "rubric bias", where criteria are too coarse or too fine and drift from human judgment, and "score instability" across repeated judging. It proposes an AND–OR criteria tree with binary leaves, where AND nodes take the minimum of their children and OR nodes route to the branch whose applicability condition holds (§3.1). It also proposes RAF, a human-in-the-loop loop in which an LLM generates a rubric, an expert reviews it, a judge agent scores trajectories, and LLM-proposed revisions are audited by the expert until a composite of 0.8·Pearson + 0.2·QWK against a human gold standard exceeds 0.8 (§3.2, Fig. 2). Finally, it introduces DailyClawBench, 121 personal-assistant tasks built from real queries of 210 OpenClaw users and run in sandbox snapshots (§4.1).

Six frontier models are evaluated on DailyClawBench, with Claude Opus 4.7 best (Table 1). On GLM-5 trajectories scored by three expert annotators, RAF's final rubrics reach a composite of 0.8102, against 0.6746 for a WildBench generic prompt and 0.6397 for AutoRubric, and the mean per-case variance over three judge runs falls from 51.25 (WildBench) and 10.18 (AutoRubric) to 3.36 (Table 2). The paper also traces the refinement rounds and an ablation without expert gating (Table 3), shows that the refined rubrics help other judges (Table 4), and reports success rates close to the official ones on τ²-bench airline and AppWorld (Table 5).

Critical strengths
  • DailyClawBench is built from real user queries (§4.1), each task is a reproducible container snapshot (App. B), and the rubric-generation prompt, judge prompt and a worked rubric are given in full (App. E–G). If the promised release (p. 10) happens, this is a useful artifact for the community, and it is the main reason I do not rate the paper lower.
  • The paper gives clear evidence that binary, structured criteria stabilise LLM judging: the mean per-case variance drops from 51.25 to 10.18 to 3.36 under the same judge (Table 2). The ablation rows of Table 3 also show that ungated LLM revision lowers alignment below the unrefined start (0.6050 vs. 0.7146 after one round) while slightly raising leaf-level kappa, which is a useful caution for anyone automating rubric refinement. These findings are the least affected by my main concern below.
Critical weaknesses
  • The headline alignment result is in-sample by construction. RAF scores trajectories against the human gold standard, passes the gold-derived inconsistencies to the LLM and expert for revision (Steps 3–5, p. 4–5), and stops once the composite exceeds 0.8. The reported final composite of 0.8102 is computed on the same GLM-5 trajectories and gold labels (§4.3), and as far as I can tell there is no held-out set. RAF's advantage over WildBench and AutoRubric in Table 2 therefore compares rubrics fitted to the evaluation labels with baselines that never saw them, and calling this "empirically validating" the threshold (p. 8) is circular. Table 4 does not resolve this, since rubrics fitted to the labels on fixed trajectories should help any competent judge on those trajectories, and Table 5 runs RAF "without any per-benchmark tuning", which presumably means only the Step-1 generator is tested. Since verified alignment with human preference is the paper's main methodological claim, I cannot tell whether the gain reflects rubric quality or fitting to the labels.
  • The AND–OR tree, which the paper presents as its first contribution and calls "a novel rubric structure" (p. 2), is neither new nor ablated. Hierarchical rubric trees for agent evaluation already exist, e.g., the requirement trees in PaperBench and the hierarchical requirements in Agent-as-a-Judge/DevAI, and neither is cited or discussed. The paper also gives no evidence that its specific choices matter: there is no comparison of AND against weighted-mean aggregation, of OR routing against none, or of binary against graded leaves. The preference for AND rests on one sentence (p. 4), and the rubrics average only 0.59 OR groups per task (Table 10), so the OR part of the structure is rarely used. Since the tree is a significant part of the claimed contribution, the paper needs to position it against prior hierarchical rubrics and show what the AND–OR design adds.

Other, less critical points:

  • No table reports confidence intervals or significance tests. With 121 tasks, the SE on a pass rate near 80% is about 3.6 points, so most gaps in Table 1 are within noise, and n=30–50 in Table 5 gives SEs of roughly 6–9 points. Table 5 also compares only aggregate success rates, which can match even when per-task verdicts disagree.
  • The claim that pass rate and quality are "non-redundant" and that "any scalar aggregate would misrepresent most of the field" (§4.2) is not supported by Table 1. The example given, GPT-5.4 ranking second in quality but third in pass rate, is a difference of one rank, and no model moves by more than two ranks. Across the six models, the two metrics have a Pearson correlation of 0.85 and a Spearman correlation of 0.77, and most of the rank differences are within the noise above.
  • Claude Opus 4.7 is both the judge and the top-ranked model (Table 1). Apart from the GPT-5.4 re-scoring in App. C (Table 9), the claim that binary leaves "structurally limit" self-preference (p. 9) is not supported.
  • Several pass rates in Table 1 are not multiples of 1/121 (e.g., 77.75, 82.47), which suggests averaging over runs that are not described.
  • nit: the abstract calls 0.8102 a "Pearson correlation", but it is the composite.
Questions for authors
  1. (Clarification) What is the unit over which Pearson, Spearman and QWK are computed in Tables 2–4: one GLM-5 trajectory per task pooled across tasks, or several trajectories per task? The 0.8 threshold is applied per rubric (§3.2), and a per-task Pearson over few trajectories would be close to meaningless. How are continuous quality scores discretised for QWK, and how are leaf-level completion metrics computed against annotators who were blind to the rubrics (§4.3.2)?
  2. (Clarification) Did any evaluation use trajectories or annotations that RAF never saw during refinement? If not, please report RAF's frozen rubrics on held-out trajectories (e.g., from another model in Table 1) against fresh human annotations, next to WildBench and AutoRubric on the same data. A clear held-out gain would substantially change my assessment.
  3. (Suggestion) How does the AND–OR tree differ from the hierarchical rubrics in PaperBench and Agent-as-a-Judge/DevAI? Please ablate the structural choices on a fixed rubric set (AND vs. weighted mean, with and without OR routing, binary vs. graded leaves), and report bootstrap CIs for Tables 1–5 and per-task agreement (accuracy or kappa) with the gold outcomes in Table 5.
  4. (Clarification) Which LLM generates and revises the rubrics in Steps 1 and 5, what is the inter-annotator agreement among the three gold annotators before consensus, and how many runs underlie the pass rates in Table 1? The inter-annotator agreement sets the ceiling against which 0.81 should be read.
  5. (Clarification) Do you have evidence beyond Table 9 that binary leaves limit self-preference? For example, how well does Opus-as-judge agree with the human gold on Opus trajectories compared with other models' trajectories?

My score could increase if the authors (i) clarify how alignment is computed (Q1), (ii) show a held-out alignment gain over the baselines on freshly annotated trajectories (Q2), and (iii) position the AND–OR tree against prior hierarchical rubrics and show through ablations what it adds (Q3).

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors(Part 1/3)

Response to Reviewer ihud

We thank the reviewer for the precise critique and for stating explicitly what would raise the score — (i) clarifying how alignment is computed, (ii) a held-out gain on freshly annotated trajectories, and (iii) positioning and ablating the AND–OR tree (full experiment details in our global response, §1–§4).

W1: "The headline alignment result is in-sample by construction"

We agree with the mechanism the reviewer describes. The circular sentence ("empirically validating both the threshold choice") is deleted — the threshold is now a stated design choice with a sensitivity check. The reviewer is also correct that Table 4 controls judge identity, not trajectory overfitting (we now say so), and that the external experiment tests only the Step-1 generator, frozen after refinement on our own benchmark.

Held-out evaluation (new). Refinement and termination used only GLM-5 trajectories. We evaluate RAF's frozen refinement outcome on held-out trajectories from Kimi-K2.6 and GPT-5.4 — two Table-1 models entirely excluded from rubric optimization — against fresh human annotations (same disjoint annotator pool, same rubric-blind protocol):

Held-out modelInitial promptAfter RAF (refined on GLM-5 only)
Kimi-K2.60.71920.7834
GPT-5.40.68130.7632

(composite; full metric breakdown in §1 of the global response.) The drop from 0.8102 is real and reported openly — but the refinement gain transfers fully to unseen models (+0.064 / +0.082 over the unrefined control on the same held-out data), and both values remain far above the unverified-generation range (WildBench 0.6746, AutoRubric 0.6397 in-sample). The revision reports in-sample and held-out numbers side by side and softens "certified" to "verified against a specified criterion sample".

W2: "The AND–OR tree… is neither new nor ablated"

W2.1 (positioning). We should have cited this literature; the revision adds both. PaperBench (Starace et al., 2025) uses requirement trees with binary leaves — but they are hand-built over weeks with the original papers' authors, and internal nodes aggregate by weighted average, diluting single failures and ruling out conditionally divergent success modes. Agent-as-a-Judge (Zhuge et al., 2025) uses DevAI's hand-specified dependency DAG of requirements — again expert-authored and conjunctive in spirit. We differ on two axes: (i) the rubric is auto-generated, then verified/refined against measured human alignment — the loop, not the data structure alone, is the contribution; (ii) AND–OR semantics make blocking failures non-dilutable (AND) and circumstance-dependent success expressible (OR).

W2.2 (ablations on a fixed rubric set). Identical binary leaves, prompts, judge, and gold: AND → weighted mean −0.131 composite; OR removed −0.076; fully flattened checklist −0.239. The preference for AND no longer rests on one sentence. OR groups are rare (0.59/task, Table 10) but load-bearing where they occur. More details in §2 of the global response.

W2.3 (binary vs. graded leaves). We did not run this variant, and we want to be transparent about why: graded leaves reintroduce exactly the failure mode RAF removes — multi-point LLM judgments are unstable (Zheng et al., 2023), and our stability results quantify the difference (per-case variance 51.25 vs. 3.36 for the binary-leaf tree, a 15× reduction; Table 2). An indirect signal also comes from the baselines: WildBench and AutoRubric both rely on continuous/graded scoring and both trail RAF on composite under the same judge and gold — leaf content differs, so we treat this as corroborating rather than controlled evidence. A controlled graded-leaf comparison is future work.

AuthorsOpenReview ↗

Rebuttal by Authors(Part 2/3)

Other, less critical points

LC1 (CIs/significance). We agree most Table 1 gaps are within noise at these sample sizes. We now report Wilson 95% CIs and per-task agreement for the external benchmarks (Q3), and will temper ranking claims accordingly.

LC2 (non-redundancy, §4.2). We concede this: with a cross-model Pearson of 0.85 and rank moves of at most two, "any scalar aggregate would misrepresent most of the field" overclaims. The revision removes that sentence and reframes the split as per-case diagnosis, not model re-ranking.

LC3 (judge-as-contestant / "structurally limit self-preference"). Direct test using Tables 1 and 10 (the appendix's GPT-5.4 re-judging): computing each judge's own-model advantage over the field mean, Opus's self-preference delta is negative on both metrics (pass −1.26, quality −1.34) — Opus-as-judge does not inflate Opus trajectories. We also disclose GPT-5.4's small positive delta (+0.69 pass, +1.47 quality), and soften "structurally limit" to state the mechanism precisely. Details in §4 of the global response.

LC4 (pass rates not multiples of 1/121). Correct — each case's score is the mean of its 3 evaluation runs (values in {0, 1/3, 2/3, 1}), so scene denominators are 3 × N_scene. This will be stated in §4.1 and Table 1's caption.

LC5 (abstract nit). Fixed: 0.8102 is the composite; the abstract now names it and reports Pearson = 0.8147 (+0.132 over the strongest baseline) separately.


Q1: "What is the unit over which Pearson, Spearman and QWK are computed?"

Each case has 3 runs, each independently expert-scored, then averaged per case — yielding 121 case-level (rubric score, gold score) pairs. Pearson/Spearman/QWK in Tables 2–4 are computed over these pairs at the dataset level — never per-task over 3 points, which would indeed be near-meaningless. The 0.8 threshold is likewise dataset-level: all rubrics freeze together once the pooled composite exceeds 0.8, after 4 refinement rounds; "Final/End" is uniform across cases. For QWK, both sides' scores (on [0,1]) are discretized with identical edges into five equal-width bins — [0,0.2), [0.2,0.4), [0.4,0.6), [0.6,0.8), [0.8,1.0] — N = 5 categories. Completion metrics: the gold completion label is a trajectory-level binary pass/fail from annotators blind to the rubrics, compared against the rubric's conjunctive completion verdict (Acc/κ/F1). The revision states all of this explicitly.

Q2: "Did any evaluation use trajectories or annotations that RAF never saw during refinement?"

Yes — Kimi-K2.6 and GPT-5.4 trajectories (models excluded from refinement) with fresh annotations; the RAF-refined prompt improves over the unrefined control on the same held-out data (+0.064 / +0.082 composite; 0.7834 / 0.7632 absolute). Details in §1 of the global response.

Q3: "How does the AND–OR tree differ from… PaperBench and Agent-as-a-Judge/DevAI? …ablations… bootstrap CIs… per-task agreement in Table 5"

Positioning and ablations: see W2. Per-task agreement with the gold outcomes, replacing the aggregate-only table:

BenchmarkNAgreement (95% Wilson CI)Cohen's κMcNemar pConfusion (TN/FP/FN/TP)
τ²-bench (airline)5090.0% [78.6%, 95.7%]0.7661.00013 / 2 / 3 / 32
AppWorld3083.3% [66.4%, 92.7%]0.6671.00012 / 2 / 3 / 13
Pooled8087.5%0.7340.75425 / 4 / 6 / 45

Disagreements are directionally symmetric (3 FN / 2 FP each) — no systematic strictness or leniency; the two benchmarks' agreement CIs overlap almost completely; κ is lower on AppWorld because its near-50/50 split raises chance agreement p_e. We present these as directional evidence with explicit uncertainty, not parity claims (full confusion matrices in §3 of the global response).

AuthorsOpenReview ↗

Rebuttal by Authors(Part 3/3)

Q3: "How does the AND–OR tree differ from… PaperBench and Agent-as-a-Judge/DevAI? …ablations… bootstrap CIs… per-task agreement in Table 5"

Positioning and ablations: see W2. Per-task agreement with the gold outcomes, replacing the aggregate-only table:

BenchmarkNAgreement (95% Wilson CI)Cohen's κMcNemar pConfusion (TN/FP/FN/TP)
τ²-bench (airline)5090.0% [78.6%, 95.7%]0.7661.00013 / 2 / 3 / 32
AppWorld3083.3% [66.4%, 92.7%]0.6671.00012 / 2 / 3 / 13
Pooled8087.5%0.7340.75425 / 4 / 6 / 45

Disagreements are directionally symmetric (3 FN / 2 FP each) — no systematic strictness or leniency; the two benchmarks' agreement CIs overlap almost completely; κ is lower on AppWorld because its near-50/50 split raises chance agreement p_e. We present these as directional evidence with explicit uncertainty, not parity claims (full confusion matrices in §3 of the global response).

Q4: "Which LLM generates and revises the rubrics… inter-annotator agreement… how many runs underlie Table 1?"

Steps 1 and 5 use Claude Opus 4.7 (fixed version, §4.1) — the same model as the default judge; we disclose this explicitly: rubric content is constrained by the gold-anchored refinement loop and expert gatekeeping, and rankings are preserved under GPT-5.4 judging (App. C; Opus's self-preference delta is negative — §4 of the global response). Pre-consensus annotator disagreement (121 cases): negligible (< 0.2) 93; adjacent-level (0.2–0.4) 19; cross-level (> 0.4) 9 — resolved by discussion plus an independent adjudicator. Pre-discussion Krippendorff's α = 0.861 — above the 0.800 reliability threshold; the final composite reaches 94% of this ceiling (0.8102/0.861), and the revision reports α alongside the alignment numbers. Table 1 pass rates average 3 runs per case.

Q5: "Evidence beyond Table 9 that binary leaves limit self-preference?"

Yes — the judge-gap analysis above: Opus's self-preference delta is negative on both metrics (pass −1.26, quality −1.34), computed from Table 1 and the appendix's Table 10 under two different judges. Details in §4 of the global response.


Ethics Concern

We apologize for the unfilled template; the actual mechanisms (condensed; full documentation available upon request — see also the global response):

Consent. Data collection operated under a layered consent mechanism: registration-level consent via the platform's ToS (explicit research-use clause covering anonymized usage data), a collection-level in-product notice when connecting to our self-deployed API (March–July 2026), and an opt-out under which a withdrawing user's data is excluded at the case filtering and selection stage — so no data from opted-out users enters the benchmark or any released artifact. More details about the consent mechanism can be found in the Ethics section of the global response.

Release safeguards / residual-PII checking. De-identification is three-layered: (1) automated scrubbing of names, emails, phone numbers, account identifiers, and credentials at collection; (2) all files referenced or manipulated by a query are replaced with mock equivalents — no real user file content enters the benchmark; (3) a full manual audit of the final 121 cases and all associated files, including residual-PII checking of everything to be released. Only de-identified artifacts will be released. The collection and release plan was reviewed by our internal privacy board. We will replace the template with the above in the revision.

We believe the held-out results (W1/Q2), the prior-work positioning and fixed-set ablations (W2/Q3), and the clarified computation units (Q1) address the reviewer's three stated conditions for a higher score.

Reviewer_L12QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RAF, a framework for constructing task-specific evaluation rubrics for long-horizon LLM agents. Rubrics use binary criteria organized into AND–OR trees and are iteratively refined through LLM suggestions, expert auditing, and alignment with human judgments.

The authors introduce DailyClawBench, comprising 121 personal-assistant tasks, and report improvements in human agreement and scoring stability, alongside cross-judge and external-benchmark evaluations.

Critical strengths

Relevant problem and interpretable design. Reliable evaluation of open-ended agent tasks is important. The explicit criteria, conditional branches, and separation of completion from quality make the evaluation process easier to inspect.

Useful benchmark construction. Tasks derived from real user interactions, preserved environment snapshots, and expert annotations provide a potentially valuable resource for evaluating personal-assistant agents.

Encouraging empirical results. The reported improvements in agreement and stability are promising. The expert-auditing ablation and cross-judge experiments provide useful evidence about the refinement process.

Critical weaknesses

The separation between rubric optimization and evaluation is unclear. RAF repeatedly uses human reference scores to refine rubrics and determine termination, but the paper does not clearly describe a held-out evaluation protocol. Using different people for annotation and rubric auditing does not prevent overfitting to the reference labels. Consequently, the reported gains may reflect calibration to the optimization data rather than generalization.

The contribution of the tree structure is insufficiently isolated. The experiments do not directly compare AND–OR trees against flat binary checklists or alternative aggregation rules under matched prompts, judges, and expert effort. Since RAF receives iterative human auditing, comparisons with single-pass baselines cannot establish how much improvement comes from the proposed structure versus additional supervision.

External generalization is evaluated through insufficient metrics. Table 5 reports aggregate success rates rather than agreement on individual trajectories. Similar overall rates can conceal substantial disagreement about which tasks succeed. The small external samples and absence of uncertainty estimates further limit the strength of the generalization claim.

Questions for authors

Which trajectories and human labels are used for refinement, stopping, and final evaluation? Could you report results on held-out trajectories, including outputs from agent models excluded from rubric optimization?

Could you compare the proposed trees with flat checklists and alternative aggregation rules under matched expert effort, and report the human time and computational cost required for refinement?

Could you provide per-trajectory accuracy, confusion matrices, and confidence intervals for the external benchmarks, rather than only aggregate success rates?

If all my concerns addressed, I will consider improve my score

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors(Part 1/2)

Response to Reviewer L12Q

We thank the reviewer for the clear statement of concerns, which we read as three requests: a held-out evaluation protocol, an isolation of the tree structure's contribution, and per-trajectory agreement metrics on the external benchmarks. New experiments address all three (full results in our global response: §1, §2, §3).

W1: "The separation between rubric optimization and evaluation is unclear"

We agree that using different people for annotation and rubric auditing does not by itself prevent overfitting to the reference labels — it establishes independence of the target, not generalization of the artifact. We have therefore added the held-out protocol the reviewer asks for.

Protocol. RAF refinement — including stopping — uses only GLM-5 trajectories with their expert labels. The held-out evaluation uses trajectories from Kimi-K2.6 and GPT-5.4, two agent models entirely excluded from rubric optimization, annotated with fresh human labels by the same disjoint annotator pool under the same rubric-blindness protocol. Nothing from these models was visible to any RAF step.

Results. Held-out composites: 0.7834 (Kimi-K2.6) and 0.7632 (GPT-5.4) — below the in-sample 0.8102, reported openly. The key finding is that the gain from refinement transfers to models and trajectories the loop never saw: +0.064 / +0.082 composite over the initial (unrefined) prompt on the same held-out data, remaining well above the unverified-generation range (AutoRubric 0.6397 in-sample). The reported gains therefore do not merely reflect calibration to the optimization data. The revision will (i) describe this protocol explicitly in §4.3, (ii) report in-sample and held-out numbers side by side, and (iii) soften "certified" to "verified against a specified criterion sample" throughout. More details about the held-out evaluation protocol and full results can be found in our response to §1 in the global response.

W2: "The contribution of the tree structure is insufficiently isolated"

We have added the requested comparison — AND–OR trees against flat binary checklists and alternative aggregation rules — under matched prompts, judge, gold, and identical binary leaf sets; only the structure/aggregation changes:

ConditionCompositeΔ vs. full
Full AND–OR tree (RAF, End)0.8102—
AND → (weighted) mean0.6793−0.131
OR removed0.7338−0.076
Fully flattened checklist0.5713−0.239

Because the leaf sets are identical, these deltas isolate the contribution of structure with zero difference in supervision. We will add this ablation table to §4 in the revision. More details can be found in our response to §2 in the global response.

On the "structure versus additional supervision" decomposition specifically: two further controls in the paper already bear on it. (i) AutoRubric (Table 2) is single-pass case-level generation: it scores 0.6397, below even the generic prompt — per-case granularity without verified refinement is not sufficient. (ii) Table 4 runs the loop without expert gatekeeping (accepting all LLM suggestions, same rounds, same judge): this degrades alignment below the unrefined starting point (0.6050 < 0.7146 after one round) — iteration without expert selection is actively harmful, and the loop's value lies in expert selection, not iteration volume. A flat-checklist condition refined under matched expert effort is a natural further control; we leave it to future work.

Human time and computational cost (also Q2 below): gold annotation is a one-time 10–20 expert-minutes per case; one refinement round is 2–3 hours wall-clock, dominated by LLM API latency, of which the expert's own time is 5–30 minutes; with a uniform 4 rounds per case (dataset-level termination), the total is ≈8–12 wall-clock hours (unattended) plus ≈20–120 expert-minutes.

AuthorsOpenReview ↗

Rebuttal by Authors(Part 2/2)

W3: "External generalization is evaluated through insufficient metrics"

Agreed — aggregate success rates can conceal substantial disagreement about which tasks succeed. We now report per-trajectory agreement with the official gold outcomes, with confusion matrices, Cohen's κ, and confidence intervals:

τ²-bench (airline), N = 50: per-trajectory agreement 45/50 = 90.0%, 95% Wilson CI [78.6%, 95.7%]; κ = 0.766; McNemar p = 1.000 (TN 13 / FP 2 / FN 3 / TP 32).

AppWorld, N = 30: agreement 25/30 = 83.3%, 95% Wilson CI [66.4%, 92.7%]; κ = 0.667; McNemar p = 1.000 (TN 12 / FP 2 / FN 3 / TP 13).

Pooled (N = 80): agreement 87.5%, κ = 0.734. Disagreements are directionally symmetric on both benchmarks (3 FN / 2 FP each) — no systematic strictness or leniency. We fully acknowledge the small external samples: the official-vs-ours gaps (2.0 / 3.3 points) sit far inside the Wilson intervals, so we now frame this experiment as directional evidence with explicit uncertainty estimates, not parity — the revision reports these per-task tables and CIs in place of the aggregate-only table. More details can be found in our response to §3 in the global response.


Q1: "Which trajectories and human labels are used for refinement, stopping, and final evaluation?"

Refinement and stopping: GLM-5 trajectories only — 3 runs per case, each independently expert-scored and then averaged per case into 121 case-level (rubric score, gold score) pairs; all alignment metrics are computed over these pairs at the dataset level, and the 0.8 stopping threshold is likewise dataset-level, so all rubrics freeze together after a uniform 4 refinement rounds. Final evaluation in the revision: both the in-sample numbers and the new held-out evaluation on trajectories from agent models excluded from rubric optimization (Kimi-K2.6, GPT-5.4) with fresh human labels. More details can be found in our response to §1 in the global response.

Q2: "Compare the proposed trees with flat checklists and alternative aggregation rules under matched expert effort, and report the human time and computational cost"

See W2: the ablation holds prompts, judge, gold, and leaf sets constant, changing only structure/aggregation — composite drops 0.131 (AND→mean), 0.076 (no OR), 0.239 (flat). Costs: 10–20 expert-minutes per case for gold annotation; 2–3 wall-clock hours per refinement round (5–30 minutes of expert time within it), uniform 4 rounds per case — ≈8–12 wall-clock hours plus ≈20–120 expert-minutes of refinement per case in total.

Q3: "Per-trajectory accuracy, confusion matrices, and confidence intervals for the external benchmarks"

All three are now reported: per-trajectory agreement 90.0% (τ²-bench) / 83.3% (AppWorld), full confusion matrices, κ = 0.766 / 0.667, Wilson 95% CIs, and McNemar tests, replacing the aggregate-only table in the revision. More details can be found in our response to §3 in the global response.


We hope these additions — held-out trajectories from excluded models, matched-leaf structure ablations, and per-trajectory external-benchmark analyses — address the reviewer's concerns, and we would be grateful for a reconsideration of the score.

Thank you for the detailed response and additional experiments. The held-out evaluations, structural ablations, and per-trajectory analyses address several of my main concerns and strengthen the empirical evidence.

However, I still have reservations about the generalizability and scalability of the framework. While the held-out evaluations demonstrate transfer to unseen model trajectories, the external benchmark evaluations remain limited in scale. Moreover, although the structural ablations are informative, the benefits of the framework relative to independently refined alternatives under comparable expert effort remain insufficiently established. I therefore maintain my original score.

AuthorsOpenReview ↗

Thank you for your response

Thank you for the thoughtful follow-up. We clarify the scope of our findings below.

Generalizability and refinement effectiveness. Although external benchmark evaluations are limited in scale and involve only Step 1 (initial rubric generation), our held-out model evaluations provide complementary evidence: rubrics refined on GLM-5 transfer to unseen Kimi-K2.6 and GPT-5.4 trajectories with fresh human annotations, improving alignment by +0.0642 and +0.0819, respectively. Regarding refinement alternatives, we acknowledge the absence of a matched-expert-effort comparison. Our structural ablations instead isolate the contribution of RAF's design: replacing AND aggregation, removing OR routing, and flattening the tree reduce alignment by 0.131, 0.076, and 0.239, respectively. Removing expert auditing also reduces first-round alignment from 0.7146 to 0.6050. These results support the importance of structured aggregation and expert guidance, but do not establish superiority over independently refined alternatives.

Scalability and expert effort. Importantly, RAF performs dataset-level optimization: all 121 cases are jointly evaluated, and their rubrics are refined through four uniform rounds. Across the entire dataset and all four rounds, active expert refinement takes only approximately 20–120 minutes in total, while the overall process takes 8–12 hours of wall-clock time, primarily due to LLM API latency. This is distinct from the one-time gold annotation cost of 10–20 expert-minutes per case. Thus, expert refinement effort is not incurred independently for every case. We believe this demonstrates bounded expert involvement, while acknowledging that larger-scale validation and matched-budget comparisons remain future work.

We will clarify these distinctions and narrow our claims accordingly. Our central contribution is a structured, expert-controlled refinement framework that improves rubric alignment while keeping active expert involvement limited.

Reviewer_ArTGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes a way to build case-specific rubrics for evaluating long-horizon LLM agents. There are three contributions:

  • AND–OR rubric tree. Leaves are binary, verifiable checks. AND nodes take the minimum of their children, and OR nodes pick the branch whose applicability condition matches the trajectory. Each rubric has two parts: a strictly conjunctive completion tree that gives pass/fail, and a quality tree whose top-level entries carry weights from 1 to 6.
  • RAF, an iterative refinement loop. An LLM drafts a rubric and a human expert screens it. A judge agent (Claude Opus 4.7) then scores trajectories, and the result is compared with a human gold standard using 0.8⋅Pearson+0.2⋅QWK0.8\cdot\text{Pearson} + 0.2\cdot\text{QWK}. Below 0.8, the LLM proposes revisions, the expert accepts or rejects each one, and the loop repeats.
  • DailyClawBench. 121 OpenClaw personal-assistant tasks taken from 210 real users, each with a sandbox snapshot. Six frontier models are evaluated on it.

Validation: Baselines (GLM-5 trajectories, Opus judge). RAF rubrics reach a composite score of 0.8102, against 0.6746 for a WildBench-style generic prompt and 0.6397 for AutoRubric. Per-case score variance over three runs drops from 51.25 to 3.36. Ablation. Removing the expert gate makes alignment worse. Judge transfer. The refined rubrics also improve GPT-5.4 and DeepSeekV4-Flash as judges. External benchmarks. Aggregate success rates come within 2–3.3 points of the official outcomes on τ²-bench (50 trajectories) and AppWorld (30).

Critical strengths
  • Weighted-mean checklists let critical failures get diluted by many trivially satisfied checks. Splitting completion from quality, and using conjunctive gating with explicit conditional branches, addresses that directly. The rubric-generation and judge prompts in Appendices E and G are detailed and reusable
  • The roughly 15× variance reduction holds up even though the alignment numbers are fit to the gold labels (see Weakness 1), because stability doesn’t depend on those labels. DailyClawBench, built from real user queries with reproducible snapshots, is a useful resource. It also yields an informative finding: completion criteria are the binding constraint, and reflection/repair is the weakest capability.
Critical weaknesses
  1. The headline alignment results are in-sample fits to the gold standard. RAF refines each rubric until its scores agree with the human gold labels on the same GLM-5 trajectories, and it stops once the score passes 0.8. Table 2 then reports alignment on those same trajectories and labels. So the final 0.8102 is close to guaranteed by the stopping rule. It is not evidence that the rubric generalizes. Neither baseline gets access to the gold labels, so comparing RAF with WildBench or AutoRubric mostly measures access to the labels, not the quality of the rubric. Keeping annotators separate from the auditors doesn’t solve this. The auditors act on the “quantitative diagnosis” computed from the gold labels, so label information still reaches the rubric. The judge-transfer result is measured on the same trajectories and labels, so it can’t rule out overfitting to the trajectories either. The real use case is scoring new trajectories: the five other models in Table 1, which were never checked against human judgment. The paper has no held-out evaluation of that. Without held-out trajectories (other models, or new runs) with fresh human labels, the central claim of a human-aligned rubric is unsupported.
  2. The paper says AND “aligns better” than a weighted mean “empirically”, but shows no AND-vs-mean or with/without-OR comparison. OR nodes are rare (0.59 per task), so it’s unclear what the AND–OR structure contributes beyond case-level binary checklists.
  3. Running RAF “without any per-benchmark tuning” seems to contradict the method, which requires gold labels and expert review. Was only Step 1 run?
Questions for authors
  • Can you report alignment on trajectories not used during refinement, such as other models’ trajectories with fresh human labels? A drop from 0.81 back toward the baseline range would change the paper’s conclusion.
  • How is alignment computed per case? If each case has only one GLM-5 trajectory, per-case Pearson or QWK is undefined. Is the 0.8 threshold applied to each case or to the whole dataset? How many rounds does “Final” represent, and does it differ by case?
  • What is the inter-annotator agreement among the three experts before discussion? How are continuous scores binned for QWK?
  • The pass rates in Table 1 aren’t multiples of 1/121 or of the per-scene task counts. For example, overall 80.90 and Personal 76.44. Several cells are also identical across models (82.64/79.92 for GPT-5.4 and DeepSeekV4-Pro, and 76.44 for GLM-5 and Kimi). How are pass rates computed: averaged over runs, or something else?
  • For the external benchmarks: per-task agreement with the gold outcomes, and which RAF steps were actually run.
  • How much expert effort does RAF take per case (rounds, hours)? This bears directly on the scalability claim.
  • The abstract calls 0.8102 a “Pearson correlation”. It is the composite score; Pearson is 0.8147, and the Pearson gain over the strongest baseline is 0.132.
  • §4.3.2 says Pearson captures magnitude while Spearman captures ranking, which contradicts §3.2.
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors(Part 1/2)

Response to Reviewer ArTG

We thank the reviewer for the detailed and constructive critique. The central concern — that "the headline alignment results are in-sample fits to the gold standard" — is well taken, and we have run new experiments to address it (full results in our global response: §1 held-out evaluation, §2 structure ablation, §3 external-benchmark analysis).

W1: "The headline alignment results are in-sample fits to the gold standard"

W1.1 (in-sample / no held-out evaluation). We agree that the 0.8102 composite is, by construction, influenced by the stopping rule, and that in-sample alignment alone cannot establish generalization. We have therefore added the held-out evaluation the reviewer asks for: RAF refinement uses only GLM-5 trajectories, and we now evaluate on held-out trajectories from Kimi-K2.6 and GPT-5.4 — models never seen during refinement — with freshly annotated human labels. Held-out composites are 0.7834 (Kimi-K2.6) and 0.7632 (GPT-5.4), versus 0.7192 / 0.6813 for the initial (unrefined) prompt on the same held-out data. The drop from 0.8102 is real, and we report it openly — but the refinement gain transfers (+0.064 / +0.082 over the initial prompt), and both values remain well above the in-sample baselines (WildBench 0.6746, AutoRubric 0.6397). Details: §1 of the global response.

W1.2 ("Neither baseline gets access to the gold labels"). Correct — and this is precisely the granularity dilemma we highlight: the comparison isolates what verified refinement buys over equally-structured but unverified generation (AutoRubric is case-level yet scores 0.6397, below the generic prompt). The held-out experiment is the cleaner test of transfer, and we now lead with it.

W1.3 ("Label information still reaches the rubric through the auditors"). Correct, and by design: RAF is a human-in-the-loop fitting procedure. What the disjoint-annotator protocol establishes is that the gold labels are not produced by the people who edited the rubrics; what the new held-out experiment establishes is that the fitted artifact generalizes beyond the fitting sample.

W1.4 ("The judge-transfer result can't rule out overfitting to the trajectories"). Agreed — it controls for judge identity, not trajectory overfitting, and we now say so explicitly; the held-out experiment is the control for trajectory overfitting.

W1.5 ("No held-out evaluation of scoring new trajectories"). Now provided: the five other models in Table 1 include exactly the two we evaluate held-out, with fresh human judgment (§1 of the global response).

W2: "No AND-vs-mean or with/without-OR comparison"

Now added: on GLM-5, same judge, same gold, identical leaf sets, changing only aggregation/structure. AND → mean costs 0.131 composite; removing OR costs 0.076; a fully flattened checklist trails by 0.239. OR nodes are indeed rare (0.59 per task), yet load-bearing where they occur. The "empirically" sentence in §3.1 now refers to this ablation. Details: §2 of the global response.

W3: "Running RAF 'without any per-benchmark tuning' seems to contradict the method"

Thank you — the phrasing was ambiguous. Only Step 1 (rubric generation) was run, using the generation prompt as refined by RAF on our own benchmark; no gold labels, expert review, or refinement rounds were performed on τ²-bench or AppWorld. The purpose is narrower than the full pipeline: whether the rubric-generation prompt optimized on our own benchmark generalizes to similar office/general-purpose agent tasks. "Without any per-benchmark tuning" means exactly this; the sentence is rewritten accordingly.


Q1: "Alignment on trajectories not used during refinement"

See W1: 0.7834 (Kimi-K2.6) and 0.7632 (GPT-5.4) held-out versus 0.8102 in-sample — a real but modest drop, with the refinement gain fully preserved relative to the initial prompt on the same held-out data. A drop back toward the baseline range would indeed have changed the conclusion; that is not what we observe.

Q2: "How is alignment computed per case?"

Each case has 3 runs, independently expert-scored, then averaged per case — yielding 121 case-level (rubric score, gold score) pairs, over which all alignment metrics are computed at the dataset level, never per case. The 0.8 threshold is likewise dataset-level: all rubrics freeze together once the pooled composite exceeds 0.8, after 4 refinement rounds — "Final" denotes this same point for every case. Completion metrics are computed over binary leaf verdicts. The revision makes both the computation unit and the termination unit explicit.

AuthorsOpenReview ↗

Rebuttal by Authors(Part 2/2)

Q3: "Inter-annotator agreement… how are continuous scores binned for QWK?"

Pre-discussion disagreement (121 cases): negligible (gap < 0.2) 93; adjacent-level (0.2–0.4) 19 (resolved by discussion then averaging); cross-level (> 0.4) 9 (additionally adjudicated by an independent fourth expert). Pre-discussion Krippendorff's α = 0.861 (interval level) — above the conventional 0.800 threshold; the final composite reaches 94% of this annotation ceiling (0.8102/0.861). For QWK, both sides' scores (on [0,1]) are discretized with identical edges into five equal-width bins — [0,0.2), [0.2,0.4), [0.4,0.6), [0.6,0.8), [0.8,1.0] — N = 5 categories; the revision states this next to Eq. (4) and reports α alongside the alignment numbers.

Q4: "Pass rates aren't multiples of 1/121… several cells are identical across models"

As the reviewer suspects, each case's score is the mean of its 3 evaluation runs (a value in {0, 1/3, 2/3, 1}), and scene/overall pass rates aggregate these case-level means — effective denominators are 3 × N_scene, hence not multiples of 1/121. Identical cells (82.64 for GPT-5.4 and DeepSeekV4-Pro; 76.44 for GLM-5 and Kimi-K2.6) occur when the two models pass the same number of (task, run) pairs in that scene. One further detail: for a small number of cases (5–10, varying by model with API conditions), the initial API timeout prevented any of the 3 runs from returning a valid result; for these cases we relaxed the timeout and ran 5 additional evaluation runs, with the case score taken as the mean over those reruns. This will be stated in §4.1 and Table 1's caption.

Q5: "For the external benchmarks: per-task agreement, and which RAF steps were actually run"

Only Step 1 was run (see W3). Per-trajectory agreement with the official gold outcomes: τ²-bench (N = 50) agreement 45/50 = 90.0% [78.6%, 95.7%], κ = 0.766, McNemar p = 1.000 (TN 13 / FP 2 / FN 3 / TP 32); AppWorld (N = 30) agreement 25/30 = 83.3% [66.4%, 92.7%], κ = 0.667, McNemar p = 1.000 (TN 12 / FP 2 / FN 3 / TP 13). Disagreements are directionally symmetric on both benchmarks (3 FN / 2 FP) — no systematic strictness or leniency. We present these as directional evidence with explicit uncertainty, given the sample sizes. Details (confusion matrices and CIs): §3 of the global response.

Q6: "How much expert effort does RAF take per case?"

Two separate costs. Gold annotation (one-time per case): approximately 10–20 expert-minutes. RAF refinement: one round takes 2–3 hours wall-clock, dominated by LLM API latency; of this, the expert's own time is 5–30 minutes reviewing the LLM's revision suggestions. With the uniform 4 rounds (see Q2), full refinement of a case totals ≈8–12 wall-clock hours plus ≈20–120 expert-minutes. We will add this breakdown to §4.3.2, as it bears directly on the scalability claim.

Q7: "The abstract calls 0.8102 a 'Pearson correlation'"

Correct — 0.8102 is the composite (0.8·Pearson + 0.2·QWK); Pearson is 0.8147, and the Pearson gain over the strongest baseline is +0.132. We will fix the abstract accordingly. Thank you for catching this.

Q8: "§4.3.2 says Pearson captures magnitude while Spearman captures ranking, which contradicts §3.2"

Agreed, and §4.3.2's wording is the incorrect one: Pearson is invariant to affine transformations, so it cannot capture magnitude; absolute-agreement is exactly QWK's role in the composite. §4.3.2 will be aligned with §3.2's definition ("Pearson correlation captures whether the rubric preserves the ordering of human preferences, while QWK captures whether its absolute ratings agree with human judgments", Eq. 3; Appendix A).


Ethics Concern

We apologize for the unresolved template placeholder — that was an oversight, and we have rewritten the statement with the actual mechanisms (condensed; full documentation available upon request, see global response).

Consent. Data collection operated under a layered consent mechanism: registration-level consent via the platform's ToS (explicit research-use clause covering anonymized usage data), a collection-level in-product notice when connecting to our self-deployed API (March–July 2026), and an opt-out under which a withdrawing user's data is excluded at the case filtering and selection stage — so no data from opted-out users enters the benchmark or any released artifact. More details about the consent mechanism can be found in the Ethics section of the global response.

De-identification (three layers). (1) Automated scrubbing of common PII (names, emails, phone numbers, account identifiers, credentials) at collection; (2) all files referenced or manipulated by a query are replaced with mock equivalents — no real user file content enters the benchmark; (3) a full manual audit of the final 121 cases and all associated files. Only de-identified artifacts (task descriptions, rubrics, mock-file snapshots) will be released.

Reviewer_ArTGOpenReview ↗

Response to rebuttal

Thank you for the detailed response and the new experiments. The held-out evaluation on Kimi-K2.6 and GPT-5.4 trajectories with fresh labels is the right test, and it shows that refinement gains carry over to unseen trajectories (+0.064 / +0.082 over the initial rubrics). The clarifications on QWK binning, the Pearson/Spearman wording, the abstract, and the external-benchmark protocol are appreciated, and I have raised my soundness score. Several points still need attention before I can update my overall rating.

  1. Held-out baselines. The held-out RAF composites (0.7834 / 0.7632) are compared with WildBench and AutoRubric scores measured on GLM-5 trajectories. Please report both baselines on the same held-out Kimi-K2.6 and GPT-5.4 sets, with bootstrap CIs on the differences (n = 121). This matters because the unrefined initial rubrics already exceed WildBench in-sample (0.7146 vs. 0.6746), so the share of RAF’s advantage due to refinement rather than Step 1 is unclear. A held-out set of Claude Opus 4.7 trajectories would also be informative, since Opus is both judge and contestant.

  2. Structure ablation. The AND-vs-mean and OR ablations are run on GLM-5, on rubrics refined under AND–OR aggregation against the same labels, so they favor the structure the rubrics were fitted under. Could you repeat them on the held-out sets? Please also define “removing OR”: if both branches become mandatory, the inapplicable branch fails by construction. Finally, the flattened checklist (≈0.571) falls below both WildBench and AutoRubric, which suggests the refined leaves are tied to the aggregation they were tuned under.

  3. Table 1 pass rates. The explanation does not reproduce the reported values. If each case score is a mean over 3 runs (or 5 reruns), each scene’s pass rate × N should be a multiple of 1/15. GPT-5.4 Personal gives 82.64% × 46 = 38.01, while the nearest attainable values give 82.61% or 82.75%, and most non-Claude cells fail the same check. Claude’s cells, by contrast, are exact single-run fractions (41/46, 37/44, 26/31, 104/121). Could you release the per-case, per-run outcomes and the aggregation script? Separately, granting extra reruns with a relaxed timeout only to cases that timed out, with the set differing by model, makes the protocol non-uniform across models. Please report Table 1 under a uniform protocol, with timeouts counted as failures, as a robustness check.

  4. Termination unit. Since the 0.8 threshold is dataset-level and all rubrics freeze after 4 rounds, individual rubrics are never verified against the threshold. Section 3.2’s per-case description (“above a threshold the rubric is accepted”) should be revised accordingly.

  5. Annotation ceiling. The ratio 0.8102 / 0.861 compares a Pearson/QWK composite with Krippendorff’s α and should be removed. Because the gold labels are a consensus of three experts plus adjudication, α is also not the relevant ceiling.

  6. External benchmarks. The paper says the evaluator is “consistently more conservative”, while the rebuttal reports “no systematic strictness or leniency”. With 5 discordant pairs, neither conclusion is supported, and both should be dropped. A generic-judge baseline (e.g., WildBench) on the same trajectories is needed to show that the AND–OR structure contributes. Please also clarify what “the generation prompt as refined by RAF” means, since RAF refines rubrics, not the generation prompt, and which model produced the AppWorld trajectories.

  7. Cost. At roughly 30–140 expert-minutes per case, plus three annotators for gold labels, the 121-case benchmark implies on the order of 100+ expert-hours. Please report the total and scope the scalability claim accordingly.

  8. Ethics. Thank you for the revised statement. It does not mention IRB or equivalent ethics review (or a formal exemption). Please state this in the paper and include the consent documentation rather than providing it on request.

If points 1 and 3 are resolved, I would be inclined to raise my rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response(Part1/2)

We thank all three reviewers for their detailed and constructive reviews. Reading across the reviews, the concerns converge on four core questions: is the reported alignment a genuine property of the rubrics or a fit to the gold labels (§1); what does the AND–OR structure itself contribute (§2); does the evaluator transfer to external benchmarks at the level of individual tasks (§3); and can the judge be trusted given that it is also a contestant (§4). We have run new experiments and analyses for each, summarized here once and referenced from the individual responses. Clarification questions specific to a single reviewer are answered in the corresponding individual response. All results here will be incorporated into the revision.


1. Generalization of the verified alignment

Core concern: the headline alignment is computed on the same trajectories and labels used for refinement, so it may reflect fitting rather than rubric quality (raised by Reviewer ArTG and Reviewer ihud; Reviewer L12Q further asks for a clearly separated optimization/evaluation protocol).

  • Held-out trajectories with fresh human labels. RAF refinement (including the termination criterion) uses only GLM-5 trajectories. We evaluate on held-out trajectories from Kimi-K2.6 and GPT-5.4 — two models entirely excluded from rubric optimization — with fresh human annotations (same guideline, same disjoint annotator pool, same rubric-blindness protocol). For each held-out case we compare rubrics from the initial generation prompt against the RAF-refined prompt (refined exclusively on GLM-5 data):

    Model (Hold-out)Initial Prompt PearsonInitial Prompt SpearmanInitial Prompt QWKInitial Prompt CompositeAfter RAF (GLM-5) PearsonAfter RAF (GLM-5) SpearmanAfter RAF (GLM-5) QWKAfter RAF (GLM-5) Composite
    KIMI-K2.60.71450.50670.73780.71920.78260.54540.78640.7834
    GPT5.40.69150.53570.64040.68130.76850.47330.74190.7632
  • Results. Held-out composites (0.7834 / 0.7632) are below the in-sample 0.8102 — a real drop, reported openly — but the refinement gain transfers to models and trajectories the loop never saw (+0.064 / +0.082 composite over the initial prompt on the same held-out data), and both remain well above the unverified-generation range (WildBench 0.6746, AutoRubric 0.6397 in-sample). The reported gains therefore do not merely reflect calibration to the optimization data.

2. Contribution of the AND–OR structure

Core concern: the design preference for conjunctive aggregation and OR routing rests on assertion rather than comparison (Reviewer ArTG, Reviewer ihud); and single-pass baselines cannot separate the structure's contribution from the additional supervision (Reviewer L12Q).

  • Ablation under matched conditions. Identical binary leaf sets, identical prompts, judge, and gold; only structure/aggregation changes (GLM-5):

    ConditionPearsonSpearmanQWKCompositeΔ vs. full
    Full AND–OR tree (RAF, End)0.81470.62970.79200.8102—
    AND → (weighted) mean0.67160.50020.71020.6793−0.131
    OR routing removed0.73370.53190.73420.7338−0.076
    Fully flattened checklist0.60310.54340.44420.5713−0.239
  • Results. Replacing AND with a mean over the same leaves costs 0.131 composite; removing the (rare, 0.59/task) OR nodes costs 0.076; a flat checklist trails by 0.239. Because the leaf sets are identical, these deltas isolate the contribution of structure with zero difference in supervision. Two further controls in the paper bound the supervision axis from the other side: AutoRubric (single-pass case-level generation, 0.6397 — below the generic prompt) and the no-gatekeeping loop (Table 4, degrading below the unrefined starting point).

3. Transfer to external benchmarks

Core concern: aggregate success rates can conceal substantial disagreement about which tasks succeed, and it was unclear which RAF steps were actually run (Reviewer L12Q, Reviewer ArTG; Reviewer ihud asks for per-task agreement with the gold outcomes).

  • Protocol. Only Step 1 (rubric generation, frozen prompt) was run on the external benchmarks — no gold labels, expert review, or refinement on τ²-bench or AppWorld.

    • τ²-bench (airline), N = 50 — official Claude Sonnet 4.5 trajectories:

      MetricPoint estimate95% Wilson CI
      Official accuracy35/50 = 70.0%[56.3%, 80.9%]
      Our rubrics accuracy34/50 = 68.0%[54.2%, 79.2%]
      Per-trajectory agreement45/50 = 90.0%[78.6%, 95.7%]
      Cohen's κ0.766—
      McNemar exact p1.000—

    The confusion matrix in Part 2

AuthorsOpenReview ↗

Global Response(Part2/2)

Connecting to the above, the τ²-bench Confusion matrix:

Confusion matrix (rows = official verdict, columns = our rubrics):

Predicted: FailPredicted: PassTotal
Actual: Fail13 (TN)2 (FP)15
Actual: Pass3 (FN)32 (TP)35
Total163450

Only 5 trajectories disagree, split 3 FN / 2 FP — directionally symmetric (McNemar p = 1.0), so the rubrics are neither systematically stricter nor more lenient than the official checker. Sensitivity 91.4% (32/35); specificity 86.7% (13/15).

  • AppWorld, N = 30:

    MetricPoint estimate95% Wilson CI
    Official accuracy16/30 = 53.3%[36.1%, 69.8%]
    Our rubrics accuracy15/30 = 50.0%[33.2%, 66.9%]
    Per-trajectory agreement25/30 = 83.3%[66.4%, 92.7%]
    Cohen's κ0.667—
    McNemar exact p1.000—

    Confusion matrix (rows = official verdict, columns = our rubrics):

    Predicted: FailPredicted: PassTotal
    Actual: Fail12 (TN)2 (FP)14
    Actual: Pass3 (FN)13 (TP)16
    Total151530

    Sensitivity 81.3% (13/16); specificity 85.7% (12/14). κ is lower than on τ²-bench despite similar agreement because AppWorld's pass/fail split is near 50/50, which raises chance agreement p_e and structurally deflates κ.

  • Cross-benchmark summary:

    BenchmarkNOfficial accOur accAgreementCohen's κMcNemar p
    τ²-bench (airline)5070.0%68.0%90.0%0.7661.000
    AppWorld3053.3%50.0%83.3%0.6671.000
    Pooled8063.8%61.3%87.5%0.7340.754

Results. Disagreements are directionally symmetric on both benchmarks (3 FN / 2 FP each) — no systematic strictness or leniency. We note openly that at N = 50/30 the Wilson intervals are wide, and the official-vs-ours accuracy gaps (2.0 / 3.3 points) are statistically indistinguishable; the agreement-rate difference between the two benchmarks (90.0% vs. 83.3%) is itself within noise — the CIs [78.6%, 95.7%] and [66.4%, 92.7%] overlap almost completely, and each disagreement moves AppWorld's rate by 3.3 points. We present these as directional agreement evidence with explicit uncertainty, not parity claims.

4. Judge reliability

Core concern: Claude Opus 4.7 is both the judge and the top-ranked model, so the headline ranking could reflect self-preference rather than merit (Reviewer ihud).

  • Judge-gap analysis (Tables 1 and 10). Using Table 1 (Opus 4.7 as judge) and Table 10 in the appendix (the same main results re-judged by GPT-5.4), we compute each model's judge gap and each judge's own-model advantage over the field mean:

    ModelPass gap (Opus−GPT)Quality gap (Opus−GPT)
    GPT-5.4−4.20+1.57
    Claude Opus 4.7−4.77+1.70
    GLM-5−4.64+2.68
    Kimi-K2.6−3.81+3.78
    DSV4-Flash+0.43+4.44
    DSV4-Pro−4.05+4.09
    Self-preference deltaPassQuality
    Claude Opus 4.7−1.26−1.34
    GPT-5.4+0.69+1.47
  • Results. Opus's advantage over the field is smaller under its own judging than under GPT-5.4's — its self-preference delta is negative on both metrics, so nothing suggests Opus-as-judge inflates Opus trajectories. We also disclose GPT-5.4's small positive delta. (Raw gaps confound judge leniency with self-preference; the signal is the cross-model variation reported here.)


Ethics and data governance (condensed)

Core concern: the consent basis for collecting real user queries and the safeguards for releasing derived data (raised by Reviewer ArTG and flagged by Reviewer ihud).

Consent: layered mechanism — (1) registration-level ToS with an explicit research-use clause covering anonymized usage data; (2) collection-level in-product notice when connecting to the self-deployed API (window March–July 2026); (3) opt-out — withdrawal excludes all of a user's data at the case filtering and selection stage (service data retention remains governed by the platform's standard policy). No data beyond what the service already processes; no minors' data to our knowledge. De-identification is three-layered: automated PII scrubbing at collection; replacement of all referenced/manipulated files with mock equivalents; full manual audit of the final 121 cases and associated files. Only de-identified artifacts will be released. The collection and release plan was reviewed by our internal privacy board. Complete documentation (ToS clauses, notice text, opt-out records, de-identification specification) is available to reviewers and the ethics chair upon request.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.