- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies rubric-based evaluation of long-horizon LLM agents and targets two problems: "rubric bias", where criteria are too coarse or too fine and drift from human judgment, and "score instability" across repeated judging. It proposes an AND–OR criteria tree with binary leaves, where AND nodes take the minimum of their children and OR nodes route to the branch whose applicability condition holds (§3.1). It also proposes RAF, a human-in-the-loop loop in which an LLM generates a rubric, an expert reviews it, a judge agent scores trajectories, and LLM-proposed revisions are audited by the expert until a composite of 0.8·Pearson + 0.2·QWK against a human gold standard exceeds 0.8 (§3.2, Fig. 2). Finally, it introduces DailyClawBench, 121 personal-assistant tasks built from real queries of 210 OpenClaw users and run in sandbox snapshots (§4.1).
Six frontier models are evaluated on DailyClawBench, with Claude Opus 4.7 best (Table 1). On GLM-5 trajectories scored by three expert annotators, RAF's final rubrics reach a composite of 0.8102, against 0.6746 for a WildBench generic prompt and 0.6397 for AutoRubric, and the mean per-case variance over three judge runs falls from 51.25 (WildBench) and 10.18 (AutoRubric) to 3.36 (Table 2). The paper also traces the refinement rounds and an ablation without expert gating (Table 3), shows that the refined rubrics help other judges (Table 4), and reports success rates close to the official ones on τ²-bench airline and AppWorld (Table 5).
Critical strengths
- DailyClawBench is built from real user queries (§4.1), each task is a reproducible container snapshot (App. B), and the rubric-generation prompt, judge prompt and a worked rubric are given in full (App. E–G). If the promised release (p. 10) happens, this is a useful artifact for the community, and it is the main reason I do not rate the paper lower.
- The paper gives clear evidence that binary, structured criteria stabilise LLM judging: the mean per-case variance drops from 51.25 to 10.18 to 3.36 under the same judge (Table 2). The ablation rows of Table 3 also show that ungated LLM revision lowers alignment below the unrefined start (0.6050 vs. 0.7146 after one round) while slightly raising leaf-level kappa, which is a useful caution for anyone automating rubric refinement. These findings are the least affected by my main concern below.
Critical weaknesses
- The headline alignment result is in-sample by construction. RAF scores trajectories against the human gold standard, passes the gold-derived inconsistencies to the LLM and expert for revision (Steps 3–5, p. 4–5), and stops once the composite exceeds 0.8. The reported final composite of 0.8102 is computed on the same GLM-5 trajectories and gold labels (§4.3), and as far as I can tell there is no held-out set. RAF's advantage over WildBench and AutoRubric in Table 2 therefore compares rubrics fitted to the evaluation labels with baselines that never saw them, and calling this "empirically validating" the threshold (p. 8) is circular. Table 4 does not resolve this, since rubrics fitted to the labels on fixed trajectories should help any competent judge on those trajectories, and Table 5 runs RAF "without any per-benchmark tuning", which presumably means only the Step-1 generator is tested. Since verified alignment with human preference is the paper's main methodological claim, I cannot tell whether the gain reflects rubric quality or fitting to the labels.
- The AND–OR tree, which the paper presents as its first contribution and calls "a novel rubric structure" (p. 2), is neither new nor ablated. Hierarchical rubric trees for agent evaluation already exist, e.g., the requirement trees in PaperBench and the hierarchical requirements in Agent-as-a-Judge/DevAI, and neither is cited or discussed. The paper also gives no evidence that its specific choices matter: there is no comparison of AND against weighted-mean aggregation, of OR routing against none, or of binary against graded leaves. The preference for AND rests on one sentence (p. 4), and the rubrics average only 0.59 OR groups per task (Table 10), so the OR part of the structure is rarely used. Since the tree is a significant part of the claimed contribution, the paper needs to position it against prior hierarchical rubrics and show what the AND–OR design adds.
Other, less critical points:
- No table reports confidence intervals or significance tests. With 121 tasks, the SE on a pass rate near 80% is about 3.6 points, so most gaps in Table 1 are within noise, and n=30–50 in Table 5 gives SEs of roughly 6–9 points. Table 5 also compares only aggregate success rates, which can match even when per-task verdicts disagree.
- The claim that pass rate and quality are "non-redundant" and that "any scalar aggregate would misrepresent most of the field" (§4.2) is not supported by Table 1. The example given, GPT-5.4 ranking second in quality but third in pass rate, is a difference of one rank, and no model moves by more than two ranks. Across the six models, the two metrics have a Pearson correlation of 0.85 and a Spearman correlation of 0.77, and most of the rank differences are within the noise above.
- Claude Opus 4.7 is both the judge and the top-ranked model (Table 1). Apart from the GPT-5.4 re-scoring in App. C (Table 9), the claim that binary leaves "structurally limit" self-preference (p. 9) is not supported.
- Several pass rates in Table 1 are not multiples of 1/121 (e.g., 77.75, 82.47), which suggests averaging over runs that are not described.
- nit: the abstract calls 0.8102 a "Pearson correlation", but it is the composite.
Questions for authors
- (Clarification) What is the unit over which Pearson, Spearman and QWK are computed in Tables 2–4: one GLM-5 trajectory per task pooled across tasks, or several trajectories per task? The 0.8 threshold is applied per rubric (§3.2), and a per-task Pearson over few trajectories would be close to meaningless. How are continuous quality scores discretised for QWK, and how are leaf-level completion metrics computed against annotators who were blind to the rubrics (§4.3.2)?
- (Clarification) Did any evaluation use trajectories or annotations that RAF never saw during refinement? If not, please report RAF's frozen rubrics on held-out trajectories (e.g., from another model in Table 1) against fresh human annotations, next to WildBench and AutoRubric on the same data. A clear held-out gain would substantially change my assessment.
- (Suggestion) How does the AND–OR tree differ from the hierarchical rubrics in PaperBench and Agent-as-a-Judge/DevAI? Please ablate the structural choices on a fixed rubric set (AND vs. weighted mean, with and without OR routing, binary vs. graded leaves), and report bootstrap CIs for Tables 1–5 and per-task agreement (accuracy or kappa) with the gold outcomes in Table 5.
- (Clarification) Which LLM generates and revises the rubrics in Steps 1 and 5, what is the inter-annotator agreement among the three gold annotators before consensus, and how many runs underlie the pass rates in Table 1? The inter-annotator agreement sets the ceiling against which 0.81 should be read.
- (Clarification) Do you have evidence beyond Table 9 that binary leaves limit self-preference? For example, how well does Opus-as-judge agree with the human gold on Opus trajectories compared with other models' trajectories?
My score could increase if the authors (i) clarify how alignment is computed (Q1), (ii) show a held-out alignment gain over the baselines on freshly annotated trajectories (Q2), and (iii) position the AND–OR tree against prior hierarchical rubrics and show through ablations what it adds (Q3).
3 comments
Rebuttal by Authors(Part 2/3)
Other, less critical points
LC1 (CIs/significance). We agree most Table 1 gaps are within noise at these sample sizes. We now report Wilson 95% CIs and per-task agreement for the external benchmarks (Q3), and will temper ranking claims accordingly.
LC2 (non-redundancy, §4.2). We concede this: with a cross-model Pearson of 0.85 and rank moves of at most two, "any scalar aggregate would misrepresent most of the field" overclaims. The revision removes that sentence and reframes the split as per-case diagnosis, not model re-ranking.
LC3 (judge-as-contestant / "structurally limit self-preference"). Direct test using Tables 1 and 10 (the appendix's GPT-5.4 re-judging): computing each judge's own-model advantage over the field mean, Opus's self-preference delta is negative on both metrics (pass −1.26, quality −1.34) — Opus-as-judge does not inflate Opus trajectories. We also disclose GPT-5.4's small positive delta (+0.69 pass, +1.47 quality), and soften "structurally limit" to state the mechanism precisely. Details in §4 of the global response.
LC4 (pass rates not multiples of 1/121). Correct — each case's score is the mean of its 3 evaluation runs (values in {0, 1/3, 2/3, 1}), so scene denominators are 3 × N_scene. This will be stated in §4.1 and Table 1's caption.
LC5 (abstract nit). Fixed: 0.8102 is the composite; the abstract now names it and reports Pearson = 0.8147 (+0.132 over the strongest baseline) separately.
Q1: "What is the unit over which Pearson, Spearman and QWK are computed?"
Each case has 3 runs, each independently expert-scored, then averaged per case — yielding 121 case-level (rubric score, gold score) pairs. Pearson/Spearman/QWK in Tables 2–4 are computed over these pairs at the dataset level — never per-task over 3 points, which would indeed be near-meaningless. The 0.8 threshold is likewise dataset-level: all rubrics freeze together once the pooled composite exceeds 0.8, after 4 refinement rounds; "Final/End" is uniform across cases. For QWK, both sides' scores (on [0,1]) are discretized with identical edges into five equal-width bins — [0,0.2), [0.2,0.4), [0.4,0.6), [0.6,0.8), [0.8,1.0] — N = 5 categories. Completion metrics: the gold completion label is a trajectory-level binary pass/fail from annotators blind to the rubrics, compared against the rubric's conjunctive completion verdict (Acc/κ/F1). The revision states all of this explicitly.
Q2: "Did any evaluation use trajectories or annotations that RAF never saw during refinement?"
Yes — Kimi-K2.6 and GPT-5.4 trajectories (models excluded from refinement) with fresh annotations; the RAF-refined prompt improves over the unrefined control on the same held-out data (+0.064 / +0.082 composite; 0.7834 / 0.7632 absolute). Details in §1 of the global response.
Q3: "How does the AND–OR tree differ from… PaperBench and Agent-as-a-Judge/DevAI? …ablations… bootstrap CIs… per-task agreement in Table 5"
Positioning and ablations: see W2. Per-task agreement with the gold outcomes, replacing the aggregate-only table:
| Benchmark | N | Agreement (95% Wilson CI) | Cohen's κ | McNemar p | Confusion (TN/FP/FN/TP) |
|---|---|---|---|---|---|
| τ²-bench (airline) | 50 | 90.0% [78.6%, 95.7%] | 0.766 | 1.000 | 13 / 2 / 3 / 32 |
| AppWorld | 30 | 83.3% [66.4%, 92.7%] | 0.667 | 1.000 | 12 / 2 / 3 / 13 |
| Pooled | 80 | 87.5% | 0.734 | 0.754 | 25 / 4 / 6 / 45 |
Disagreements are directionally symmetric (3 FN / 2 FP each) — no systematic strictness or leniency; the two benchmarks' agreement CIs overlap almost completely; κ is lower on AppWorld because its near-50/50 split raises chance agreement p_e. We present these as directional evidence with explicit uncertainty, not parity claims (full confusion matrices in §3 of the global response).
Rebuttal by Authors(Part 3/3)
Q3: "How does the AND–OR tree differ from… PaperBench and Agent-as-a-Judge/DevAI? …ablations… bootstrap CIs… per-task agreement in Table 5"
Positioning and ablations: see W2. Per-task agreement with the gold outcomes, replacing the aggregate-only table:
| Benchmark | N | Agreement (95% Wilson CI) | Cohen's κ | McNemar p | Confusion (TN/FP/FN/TP) |
|---|---|---|---|---|---|
| τ²-bench (airline) | 50 | 90.0% [78.6%, 95.7%] | 0.766 | 1.000 | 13 / 2 / 3 / 32 |
| AppWorld | 30 | 83.3% [66.4%, 92.7%] | 0.667 | 1.000 | 12 / 2 / 3 / 13 |
| Pooled | 80 | 87.5% | 0.734 | 0.754 | 25 / 4 / 6 / 45 |
Disagreements are directionally symmetric (3 FN / 2 FP each) — no systematic strictness or leniency; the two benchmarks' agreement CIs overlap almost completely; κ is lower on AppWorld because its near-50/50 split raises chance agreement p_e. We present these as directional evidence with explicit uncertainty, not parity claims (full confusion matrices in §3 of the global response).
Q4: "Which LLM generates and revises the rubrics… inter-annotator agreement… how many runs underlie Table 1?"
Steps 1 and 5 use Claude Opus 4.7 (fixed version, §4.1) — the same model as the default judge; we disclose this explicitly: rubric content is constrained by the gold-anchored refinement loop and expert gatekeeping, and rankings are preserved under GPT-5.4 judging (App. C; Opus's self-preference delta is negative — §4 of the global response). Pre-consensus annotator disagreement (121 cases): negligible (< 0.2) 93; adjacent-level (0.2–0.4) 19; cross-level (> 0.4) 9 — resolved by discussion plus an independent adjudicator. Pre-discussion Krippendorff's α = 0.861 — above the 0.800 reliability threshold; the final composite reaches 94% of this ceiling (0.8102/0.861), and the revision reports α alongside the alignment numbers. Table 1 pass rates average 3 runs per case.
Q5: "Evidence beyond Table 9 that binary leaves limit self-preference?"
Yes — the judge-gap analysis above: Opus's self-preference delta is negative on both metrics (pass −1.26, quality −1.34), computed from Table 1 and the appendix's Table 10 under two different judges. Details in §4 of the global response.
Ethics Concern
We apologize for the unfilled template; the actual mechanisms (condensed; full documentation available upon request — see also the global response):
Consent. Data collection operated under a layered consent mechanism: registration-level consent via the platform's ToS (explicit research-use clause covering anonymized usage data), a collection-level in-product notice when connecting to our self-deployed API (March–July 2026), and an opt-out under which a withdrawing user's data is excluded at the case filtering and selection stage — so no data from opted-out users enters the benchmark or any released artifact. More details about the consent mechanism can be found in the Ethics section of the global response.
Release safeguards / residual-PII checking. De-identification is three-layered: (1) automated scrubbing of names, emails, phone numbers, account identifiers, and credentials at collection; (2) all files referenced or manipulated by a query are replaced with mock equivalents — no real user file content enters the benchmark; (3) a full manual audit of the final 121 cases and all associated files, including residual-PII checking of everything to be released. Only de-identified artifacts will be released. The collection and release plan was reviewed by our internal privacy board. We will replace the template with the above in the revision.
We believe the held-out results (W1/Q2), the prior-work positioning and fixed-set ablations (W2/Q3), and the clarified computation units (Q1) address the reviewer's three stated conditions for a higher score.
Rebuttal by Authors(Part 1/3)
Response to Reviewer ihud
We thank the reviewer for the precise critique and for stating explicitly what would raise the score — (i) clarifying how alignment is computed, (ii) a held-out gain on freshly annotated trajectories, and (iii) positioning and ablating the AND–OR tree (full experiment details in our global response, §1–§4).
W1: "The headline alignment result is in-sample by construction"
We agree with the mechanism the reviewer describes. The circular sentence ("empirically validating both the threshold choice") is deleted — the threshold is now a stated design choice with a sensitivity check. The reviewer is also correct that Table 4 controls judge identity, not trajectory overfitting (we now say so), and that the external experiment tests only the Step-1 generator, frozen after refinement on our own benchmark.
Held-out evaluation (new). Refinement and termination used only GLM-5 trajectories. We evaluate RAF's frozen refinement outcome on held-out trajectories from Kimi-K2.6 and GPT-5.4 — two Table-1 models entirely excluded from rubric optimization — against fresh human annotations (same disjoint annotator pool, same rubric-blind protocol):
(composite; full metric breakdown in §1 of the global response.) The drop from 0.8102 is real and reported openly — but the refinement gain transfers fully to unseen models (+0.064 / +0.082 over the unrefined control on the same held-out data), and both values remain far above the unverified-generation range (WildBench 0.6746, AutoRubric 0.6397 in-sample). The revision reports in-sample and held-out numbers side by side and softens "certified" to "verified against a specified criterion sample".
W2: "The AND–OR tree… is neither new nor ablated"
W2.1 (positioning). We should have cited this literature; the revision adds both. PaperBench (Starace et al., 2025) uses requirement trees with binary leaves — but they are hand-built over weeks with the original papers' authors, and internal nodes aggregate by weighted average, diluting single failures and ruling out conditionally divergent success modes. Agent-as-a-Judge (Zhuge et al., 2025) uses DevAI's hand-specified dependency DAG of requirements — again expert-authored and conjunctive in spirit. We differ on two axes: (i) the rubric is auto-generated, then verified/refined against measured human alignment — the loop, not the data structure alone, is the contribution; (ii) AND–OR semantics make blocking failures non-dilutable (AND) and circumstance-dependent success expressible (OR).
W2.2 (ablations on a fixed rubric set). Identical binary leaves, prompts, judge, and gold: AND → weighted mean −0.131 composite; OR removed −0.076; fully flattened checklist −0.239. The preference for AND no longer rests on one sentence. OR groups are rare (0.59/task, Table 10) but load-bearing where they occur. More details in §2 of the global response.
W2.3 (binary vs. graded leaves). We did not run this variant, and we want to be transparent about why: graded leaves reintroduce exactly the failure mode RAF removes — multi-point LLM judgments are unstable (Zheng et al., 2023), and our stability results quantify the difference (per-case variance 51.25 vs. 3.36 for the binary-leaf tree, a 15× reduction; Table 2). An indirect signal also comes from the baselines: WildBench and AutoRubric both rely on continuous/graded scoring and both trail RAF on composite under the same judge and gold — leaf content differs, so we treat this as corroborating rather than controlled evidence. A controlled graded-leaf comparison is future work.