- Rating
- 2
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper argues that the deterministic rule metrics commonly used to score coding-agent trajectories should be treated as fallible, versioned sensors rather than as labels. Each rule is wrapped in a "sensor contract" consisting of an applicability precondition, a set of admissible detector versions, a reliability set, and an evidence requirement. The reliability sets are built from two LLM proxies (GPT-5.5 and Claude Opus 4.6) that independently judge 1,800 trace-rule pairs, and the authors are explicit that proxy agreement is not taken as correctness. Criterion scores then become set-valued over versions, applicability states, and reliability weights, and a pairwise order between two trajectories is certified only when their intervals do not overlap. On top of this, a Qwen3-8B judge is trained with supervised initialization on teacher outputs and then with GRPO under four matched rule-reward variants (Naive, Gate, Proxy-LCB, and the proposed worst-case RAVEL-Set objective). The empirical part has two halves. A production audit of 402K trajectories shows that recovery and alignment sit at their default value on 99.74% of traces, that one parser correction changes 36.9% of letter grades, that only 58.0% of sampled pairwise orders are certified when both detector versions are treated as admissible, and that a logistic router predicts grade changes on a temporal holdout with AUROC 0.743. A diagnostic meta-evaluation on 670 automatically extracted execution anchors, each with metamorphic variants, shows that supervised initialization raises strict schema coverage from 0% to 88.1%, while the four reward variants have overlapping intervals and swap rank between 128 and 256 training steps. The authors say plainly that RAVEL-Set shows no stable advantage and position the contribution as the measurement framework itself.
Critical strengths
-
The reporting is more careful and candid than most papers in this area. Thresholds are frozen on a session-disjoint split, invalid outputs count as errors, AUROC is recomputed on the common valid subset, intervals use cluster bootstrap, and both training budgets are reported side by side instead of picking the favourable one. The authors also say directly that their proposed objective does not win, and they note that an always-success predictor would beat every row on Execution-Risk@90. I appreciated this, and the budget-sensitivity point (a reward ablation can flip ranking between 128 and 256 steps) is a useful caution for anyone running small-budget GRPO ablations on judges.
-
Some of the production audit findings are useful to practitioners building agent dashboards. The observation that recovery and alignment look near-perfect only because 99.74% of traces never contain a correction episode, that explicit "not applicable" statements account for 26.8% of zero scores in the existing LLM criterion judgments, and that grade changes across detector versions are predictable from provenance features (AUROC 0.743) while distance to a grade boundary is not (0.460) are concrete and well-quantified. Separating applicability from score as a set A_ik, and pushing that into an interval, is a sensible way to formalize the problem.
Critical weaknesses
-
Once the negative result is taken at face value, very little of the proposed method is left with empirical support. RAVEL-Set is the only learning contribution, and it is indistinguishable from Naive averaging: 0.666 vs 0.635 AUROC at 128 steps, 0.619 vs 0.663 at 256, with all intervals overlapping. Only two of the eight GRPO checkpoints beat the supervised initialization's AUROC of 0.656. The other core objects of the framework, namely the fused judge-sensor interval of Eq. (7) and the conflict-based selector g_eta built on Eq. (8), are never evaluated. The Discussion and the Limitations both say so, and the reported risk metric relies on self-reported confidence rather than the proposed selector. What remains is an audit of one production system plus a formalism whose central components are untested. Honesty about a negative result is welcome, but it does not by itself make a contribution, and the paper does not yet show that treating rules as set-valued sensors leads to better evaluation decisions than simply fixing and gating the rules.
-
The headline evidence for "implementation fragility" looks more like a bug fix than an instance of two defensible measurements disagreeing. The abstract and conclusion lean on the parser correction that changes 36.9% of grades and leaves 42.0% of pairwise orders uncertified. Table 10, however, shows that Task evidence (TES) had a mean of exactly 0.0000 before the fix across all 402K trajectories, and Appendix C.5 describes an earlier matcher that attached only the first tool result after a multi-call message, with build-pass rate moving from 10.7% to 76.8% after correction. A detector that returns zero for every trace is not an admissible measurement, and treating the broken and fixed versions as equally valid endpoints of an interval inflates the uncertainty the framework is meant to capture. The more convincing case, two reasonable loop detectors that agree on binary detection but differ in counts by up to 19.4x, is based on only 100 traces. With a single correction and two versions, which the authors acknowledge, the version-set analysis shows that bugs change scores but does not show that a version-interval representation is the right response to them.
-
The meta-evaluation cannot resolve the effects the paper cares about, and it measures something different from the stated motivation. The test split has 29 failures and the development split has 7. As a result, strict balanced accuracy falls below 0.5 for most rows, and Execution-Risk@90 is uninformative given the 517/29 class imbalance. The outcome-flip views include synthetic markers that are visible to the model, and the directional contracts can be satisfied by an unchanged prediction. The prompted Claude judge scores below chance on AUROC (0.476) yet has the best directional consistency (0.841), which suggests the directional aggregate is not tracking judgment quality. More fundamentally, the paper's motivation concerns process-level criteria such as recovery, efficiency, and intent alignment, where rules are conditional and unreliable. The objective suite only tests whether the judge recovers the exit code of a pattern-matched verification command, which is the setting where a deterministic rule is already reliable. Even if RAVEL-Set had won, this instrument would not show that it handles uncertain process rules better. Add one training seed, 512 prompts, and four candidates per prompt, and the ablation is underpowered by design.
-
The reliability sets have no external anchor, and the closest existing approach is not compared against. Q_k is the hull of Beta posterior intervals over agreement between two frontier LLMs on 180 trajectories, with kappa between 0.49 and 0.80 depending on the field. Point supports range from 0.0168 to 0.6611, so some rules are effectively switched off by the proxies alone. Without even a small human-audited sample, it is impossible to tell whether wide sets coincide with rules that are actually unreliable or only with rules the two models find ambiguous, and shared biases between the proxies, which the authors raise themselves, would go unnoticed. On positioning, the paper cites data programming but does not compare against a label-model style fusion of the ten detectors with abstention, which is the standard way to treat heuristics as noisy, conditionally applicable sources. Much of the remaining framing (record detector versions, gate rules by preconditions, do not average undefined metrics) is good measurement practice, but it is not yet a research contribution that ICLR readers can build on.
Questions for authors
-
Why should the pre-fix detector, with TES identically zero across all 402K trajectories, be treated as an admissible version in V_k? Can you repeat the version-set and order-certification analysis at production scale on a pair of implementations that are both defensible, for instance the strict and broad loop detectors, and report how many orders remain uncertified?
-
Can you evaluate the fused interval of Eq. (7) and the conflict selector g_eta end to end, and compare them with the confidence-ranked selector currently reported? Without this, the set-valued inference half of the framework has no empirical result.
-
Is there any human-labeled subset, even a few hundred trace-rule pairs, against which you can check whether the width of Q_k tracks actual rule error? A rank correlation between set width and human-judged rule failure would go a long way toward justifying the dual-proxy construction.
-
Can you build an objective suite with substantially more failures, for example by stratified sampling of failing executions from the 402K pool, and with at least one process-level criterion where rules are known to be unreliable, and rerun the four-way reward ablation with several seeds? As the paper stands, it is unclear whether the null result reflects the objective or the instrument.
2 comments
Thank authors for the detailed response and revisions on the paper for the claims and the scope. However, this boiled down the paper to some good practice for the measurement and raised further questions for the novelty and effectiveness of the paper in presence of null results. Overall, while the revision addresses some concerns, I do not think it sufficiently resolves my main reservations about novelty, scope and practical applicability. I therefore keep my original rating.
Response to xCvV: Negative Result, TES Degeneracy, Meta-Evaluation, and Reliability
We thank Reviewer xCvV for the most detailed critique. The TES observation and the point that honesty is not itself a contribution changed how we frame the paper.
W1 (Negative result). We agree. The thesis is now that the measurement layer, not the judge or reward layer, is the bottleneck. We rewrote the abstract and contributions and added a non-claims box. We confirm the count: two of eight checkpoints exceed the SFT AUROC of 0.656, and the common-subset recomputation moves AUROC by at most 0.006. A failure-enriched suite is not possible under the data agreement; the power analysis (MDD 0.22; spread 0.03-0.05) and the descriptive label are in the paper.
W2 & Q1 (TES = 0). We agree. Section 3 now has V_obs/V_adm and the predicate; only the post-fix version passes, and the 42.0% is a bound under V_obs. For the loop pair we report the 100-trace aggregates (25 vs 33; 92.0%; κ = 0.807; ρ = 0.884; 19.4×) and note that without per-trace counts no certified-order rate is computed.
W3 & Q4 (Meta-evaluation). We agree that the suite tests execution-outcome recovery, not process-level judgment; Sections 5 and 7 say so. Strict response rates are now reported, and the Claude pattern (0.476 AUROC, 0.841 directional) is explained as a direction-versus-ranking dissociation. The power analysis and descriptive label are in place. Marker-blinded and process-level audits are not possible under the data agreement; Section 7 states both.
W4 & Q3 (Q_k anchor and label model). We agree that Q_k has no external anchor and that a label-model baseline is the natural comparison; neither is available under the data agreement. Section 7 says that Q_k captures proxy ambiguity, not rule error; the [0.0168, 0.6611] point-support range is reported; and the remaining framing is measurement practice, not a new learning method.
Q2 (End-to-end Eq. 7-8). Eqs. (7)-(8) are not evaluated end to end; Eq. (8) is a formal extension, and the evaluated rule is version-interval certification.
The manuscript has been revised accordingly.