acceptodds
Under review as a conference paper at ICLR 2027

Preserving Rule Uncertainty for Auditable Coding-Agent Evaluation

Abstract

Evaluating coding agents requires interpreting both execution outcomes and trajectory-level behavior, yet deterministic rules can be misleading when their preconditions fail, their implementations change, or supporting evidence is absent. We present RAVEL, a framework that represents each rule as a versioned sensor contract specifying applicability, provenance, evidence requirements, and an admissible reliability set. Two model proxies inform these sets, but their agreement is not treated as semantic correctness and their disagreement is retained as uncertainty. A compact generative judge is supervised to emit criterion scores and event citations, then trained with matched reward variants, including a worst-case rule-agreement objective over admissible reliability weights. An audit of 402K production trajectories shows that one parser correction changes 36.9% of grades and that 42.0% of sampled pairwise orders remain uncertified across two detector versions. For diagnostic evaluation, we construct 670 instances from automatically extracted execution signals and 2,017 metamorphic relations. Across 2,680 evaluation views, supervised initialization raises strict structured-output coverage from 0% to 88.1%. The four reward variants, however, have overlapping uncertainty intervals and change rank between 128 and 256 training steps; the proposed set objective does not show a stable advantage. These results support the measurement framework—auditing applicability, provenance, and version sensitivity—while delimiting what the present execution-anchored study can establish about process-level judgment.

Primary area:
datasets and benchmarks
Keywords:
coding agents, LLM-as-a-judge, agent evaluation, rule uncertainty, weak supervision, selective prediction, metamorphic testing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xCvVOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that the deterministic rule metrics commonly used to score coding-agent trajectories should be treated as fallible, versioned sensors rather than as labels. Each rule is wrapped in a "sensor contract" consisting of an applicability precondition, a set of admissible detector versions, a reliability set, and an evidence requirement. The reliability sets are built from two LLM proxies (GPT-5.5 and Claude Opus 4.6) that independently judge 1,800 trace-rule pairs, and the authors are explicit that proxy agreement is not taken as correctness. Criterion scores then become set-valued over versions, applicability states, and reliability weights, and a pairwise order between two trajectories is certified only when their intervals do not overlap. On top of this, a Qwen3-8B judge is trained with supervised initialization on teacher outputs and then with GRPO under four matched rule-reward variants (Naive, Gate, Proxy-LCB, and the proposed worst-case RAVEL-Set objective). The empirical part has two halves. A production audit of 402K trajectories shows that recovery and alignment sit at their default value on 99.74% of traces, that one parser correction changes 36.9% of letter grades, that only 58.0% of sampled pairwise orders are certified when both detector versions are treated as admissible, and that a logistic router predicts grade changes on a temporal holdout with AUROC 0.743. A diagnostic meta-evaluation on 670 automatically extracted execution anchors, each with metamorphic variants, shows that supervised initialization raises strict schema coverage from 0% to 88.1%, while the four reward variants have overlapping intervals and swap rank between 128 and 256 training steps. The authors say plainly that RAVEL-Set shows no stable advantage and position the contribution as the measurement framework itself.

Critical strengths
  • The reporting is more careful and candid than most papers in this area. Thresholds are frozen on a session-disjoint split, invalid outputs count as errors, AUROC is recomputed on the common valid subset, intervals use cluster bootstrap, and both training budgets are reported side by side instead of picking the favourable one. The authors also say directly that their proposed objective does not win, and they note that an always-success predictor would beat every row on Execution-Risk@90. I appreciated this, and the budget-sensitivity point (a reward ablation can flip ranking between 128 and 256 steps) is a useful caution for anyone running small-budget GRPO ablations on judges.

  • Some of the production audit findings are useful to practitioners building agent dashboards. The observation that recovery and alignment look near-perfect only because 99.74% of traces never contain a correction episode, that explicit "not applicable" statements account for 26.8% of zero scores in the existing LLM criterion judgments, and that grade changes across detector versions are predictable from provenance features (AUROC 0.743) while distance to a grade boundary is not (0.460) are concrete and well-quantified. Separating applicability from score as a set A_ik, and pushing that into an interval, is a sensible way to formalize the problem.

Critical weaknesses
  • Once the negative result is taken at face value, very little of the proposed method is left with empirical support. RAVEL-Set is the only learning contribution, and it is indistinguishable from Naive averaging: 0.666 vs 0.635 AUROC at 128 steps, 0.619 vs 0.663 at 256, with all intervals overlapping. Only two of the eight GRPO checkpoints beat the supervised initialization's AUROC of 0.656. The other core objects of the framework, namely the fused judge-sensor interval of Eq. (7) and the conflict-based selector g_eta built on Eq. (8), are never evaluated. The Discussion and the Limitations both say so, and the reported risk metric relies on self-reported confidence rather than the proposed selector. What remains is an audit of one production system plus a formalism whose central components are untested. Honesty about a negative result is welcome, but it does not by itself make a contribution, and the paper does not yet show that treating rules as set-valued sensors leads to better evaluation decisions than simply fixing and gating the rules.

  • The headline evidence for "implementation fragility" looks more like a bug fix than an instance of two defensible measurements disagreeing. The abstract and conclusion lean on the parser correction that changes 36.9% of grades and leaves 42.0% of pairwise orders uncertified. Table 10, however, shows that Task evidence (TES) had a mean of exactly 0.0000 before the fix across all 402K trajectories, and Appendix C.5 describes an earlier matcher that attached only the first tool result after a multi-call message, with build-pass rate moving from 10.7% to 76.8% after correction. A detector that returns zero for every trace is not an admissible measurement, and treating the broken and fixed versions as equally valid endpoints of an interval inflates the uncertainty the framework is meant to capture. The more convincing case, two reasonable loop detectors that agree on binary detection but differ in counts by up to 19.4x, is based on only 100 traces. With a single correction and two versions, which the authors acknowledge, the version-set analysis shows that bugs change scores but does not show that a version-interval representation is the right response to them.

  • The meta-evaluation cannot resolve the effects the paper cares about, and it measures something different from the stated motivation. The test split has 29 failures and the development split has 7. As a result, strict balanced accuracy falls below 0.5 for most rows, and Execution-Risk@90 is uninformative given the 517/29 class imbalance. The outcome-flip views include synthetic markers that are visible to the model, and the directional contracts can be satisfied by an unchanged prediction. The prompted Claude judge scores below chance on AUROC (0.476) yet has the best directional consistency (0.841), which suggests the directional aggregate is not tracking judgment quality. More fundamentally, the paper's motivation concerns process-level criteria such as recovery, efficiency, and intent alignment, where rules are conditional and unreliable. The objective suite only tests whether the judge recovers the exit code of a pattern-matched verification command, which is the setting where a deterministic rule is already reliable. Even if RAVEL-Set had won, this instrument would not show that it handles uncertain process rules better. Add one training seed, 512 prompts, and four candidates per prompt, and the ablation is underpowered by design.

  • The reliability sets have no external anchor, and the closest existing approach is not compared against. Q_k is the hull of Beta posterior intervals over agreement between two frontier LLMs on 180 trajectories, with kappa between 0.49 and 0.80 depending on the field. Point supports range from 0.0168 to 0.6611, so some rules are effectively switched off by the proxies alone. Without even a small human-audited sample, it is impossible to tell whether wide sets coincide with rules that are actually unreliable or only with rules the two models find ambiguous, and shared biases between the proxies, which the authors raise themselves, would go unnoticed. On positioning, the paper cites data programming but does not compare against a label-model style fusion of the ten detectors with abstention, which is the standard way to treat heuristics as noisy, conditionally applicable sources. Much of the remaining framing (record detector versions, gate rules by preconditions, do not average undefined metrics) is good measurement practice, but it is not yet a research contribution that ICLR readers can build on.

Questions for authors
  • Why should the pre-fix detector, with TES identically zero across all 402K trajectories, be treated as an admissible version in V_k? Can you repeat the version-set and order-certification analysis at production scale on a pair of implementations that are both defensible, for instance the strict and broad loop detectors, and report how many orders remain uncertified?

  • Can you evaluate the fused interval of Eq. (7) and the conflict selector g_eta end to end, and compare them with the confidence-ranked selector currently reported? Without this, the set-valued inference half of the framework has no empirical result.

  • Is there any human-labeled subset, even a few hundred trace-rule pairs, against which you can check whether the width of Q_k tracks actual rule error? A rank correlation between set width and human-judged rule failure would go a long way toward justifying the dual-proxy construction.

  • Can you build an objective suite with substantially more failures, for example by stratified sampling of failing executions from the 402K pool, and with at least one process-level criterion where rules are known to be unreliable, and rerun the four-way reward ablation with several seeds? As the paper stands, it is unclear whether the null result reflects the objective or the instrument.

2 comments
AuthorsOpenReview ↗

Response to xCvV: Negative Result, TES Degeneracy, Meta-Evaluation, and Reliability

We thank Reviewer xCvV for the most detailed critique. The TES observation and the point that honesty is not itself a contribution changed how we frame the paper.

W1 (Negative result). We agree. The thesis is now that the measurement layer, not the judge or reward layer, is the bottleneck. We rewrote the abstract and contributions and added a non-claims box. We confirm the count: two of eight checkpoints exceed the SFT AUROC of 0.656, and the common-subset recomputation moves AUROC by at most 0.006. A failure-enriched suite is not possible under the data agreement; the power analysis (MDD 0.22; spread 0.03-0.05) and the descriptive label are in the paper.

W2 & Q1 (TES = 0). We agree. Section 3 now has V_obs/V_adm and the predicate; only the post-fix version passes, and the 42.0% is a bound under V_obs. For the loop pair we report the 100-trace aggregates (25 vs 33; 92.0%; κ = 0.807; ρ = 0.884; 19.4×) and note that without per-trace counts no certified-order rate is computed.

W3 & Q4 (Meta-evaluation). We agree that the suite tests execution-outcome recovery, not process-level judgment; Sections 5 and 7 say so. Strict response rates are now reported, and the Claude pattern (0.476 AUROC, 0.841 directional) is explained as a direction-versus-ranking dissociation. The power analysis and descriptive label are in place. Marker-blinded and process-level audits are not possible under the data agreement; Section 7 states both.

W4 & Q3 (Q_k anchor and label model). We agree that Q_k has no external anchor and that a label-model baseline is the natural comparison; neither is available under the data agreement. Section 7 says that Q_k captures proxy ambiguity, not rule error; the [0.0168, 0.6611] point-support range is reported; and the remaining framing is measurement practice, not a new learning method.

Q2 (End-to-end Eq. 7-8). Eqs. (7)-(8) are not evaluated end to end; Eq. (8) is a formal extension, and the evaluated rule is version-interval certification.

The manuscript has been revised accordingly.

Thank authors for the detailed response and revisions on the paper for the claims and the scope. However, this boiled down the paper to some good practice for the measurement and raised further questions for the novelty and effectiveness of the paper in presence of null results. Overall, while the revision addresses some concerns, I do not think it sufficiently resolves my main reservations about novelty, scope and practical applicability. I therefore keep my original rating.

Reviewer_41ETOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses the problem of evaluating coding-agent trajectories using deterministic heuristics throughout couple with an LLM-as-a-judge. The authors collect and audit a 402,000 dataset of production trajectories to show that deterministic rules may be sensitive and misleading. To combat these deficiencies, they propose the RAVEL framework which yield interval-valued scores rather than scalar estimates. They fine-tune a judge using SFT and four GRPO variants, showing that SFT raises valid output from 0 to 88.1%, yet the RAVEL RL objective shows no consistent advantage over baselines.

Critical strengths
  • The empirical audit of production failure models, especially at large-scale. This work in identifying the scale at which practical pitfalls emerge in automated agent evaluation is valuable towards improving agent evals.
Critical weaknesses
  • Overall, the paper was incredibly hard for me to parse. This may be due to my own inexperience in this literature, but there appears to be a severe lack of clarity and numerous undefined terms. Core concepts are noted without concrete definitions such as "conditionality," "implementation uncertainty," "recovery" and "alignment" (naturally, the last word has meaning in the literature, but needs careful definition when scoped to the present problem). Notably, the paper states these "equal their default value for 99.74% of traces" without defining these metrics or how the underlying rules are specified.
  • It is very difficult to understand what problem the paper seeks to resolve beyond the data audit. The initial introduction of RAVEL is hard to parse, and there is a massive disconnect between the 402K production audit, theoretical set-valued fusion of Section 3-4, and the GRPO experiments.
  • Additionally, in working to parse the empirical results, the proposed RAVEL-set method does not improve over baselines? The test split seems to contain only 29 failure instances of 546 giving an accuracy worse than random guessing.
Questions for authors
  • Can the authors succinctly and intuitively define the core takeaway of their work, given that RAVEL does not outperform naive rewards Eq. 7 is not evaluated end-to-end? Is this purely meant to show the observational data audit?
  • Can you clearly/formally define the used terms noted above from section 1?
2 comments
AuthorsOpenReview ↗

Response to 41ET: Readability, Thesis, and the 29/546 Accuracy

We thank Reviewer 41ET for the candid comments on readability. They led to a rewrite of the introduction and a new definitions table.

W1 & Q2 (Terminology). We added a definitions table (Appendix A) for conditionality, implementation uncertainty, recovery, alignment, default value, and applicability. Section 1 now explains the 99.74% default: recovery and alignment are at their maximum on 99.74% of traces because almost no trace contains a correction episode, and 99.38% contain at most one user turn. The worked example is at the start of Section 4; the ten detectors and their preconditions are listed in the appendix rule library; and the difference between strict accuracy and BAcc/AUROC is now explicit.

W2 & Q1 (Thesis and takeaway). The introduction now opens with the takeaway: the measurement layer, not the judge, is the bottleneck. The paper follows one line: audit, formalization, judge-layer test, negative result; GRPO is secondary. A claims-evidence table and a non-claims box are included.

W3 (RAVEL-Set and 29/546). RAVEL-Set does not stably outperform Naive (0.666 vs 0.635 at 128 steps; 0.619 vs 0.663 at 256; intervals overlap; two of eight checkpoints exceed the SFT AUROC of 0.656). The 6.41% is Gate-256 only; the other rows are 53.3/63.7/60.6. With 517:29, always predicting success gives 94.7%, so accuracy is dominated by the majority class. Gate-256's threshold is 0.805, with TPR 1.55/TNR 93.10, BAcc 0.473, and AUROC 0.646; this is threshold transfer, not an anti-correlated model. We added majority-class 94.69% and always-success 5.31/5.91 rows to Table 1.

The manuscript has been revised accordingly.

AuthorsOpenReview ↗

Follow-up to Reviewer 41ET: Takeaway and Terminology

We thank Reviewer 41ET for the detailed review and for the two clarifying questions. Below we answer them directly and restate the accuracy point.

Q1 (Core takeaway). The paper is an observational audit of the measurement layer in coding-agent evaluation. It does not claim that RAVEL-Set outperforms the baselines, and it does not evaluate Eq. (7) end-to-end. Eq. (8) is a formal extension, and the rule we do evaluate is version-interval certification (Section 4). This is the intended scope: a measurement study, not an end-to-end system paper.

Q2 (Terminology). Appendix A now defines conditionality, implementation uncertainty, recovery, alignment, default value, and applicability. Section 3 uses these terms consistently, and Section 1 states the 99.74% default explicitly: recovery and alignment are at their maximum on 99.74% of traces because almost no trace contains a correction episode.

W3 (29/546 accuracy). The 6.41% is Gate-256 only; the other rows are 53.3/63.7/60.6. Under 517:29, always predicting success gives 94.7%, so accuracy is dominated by the majority class. Gate-256's threshold is 0.805, with TPR 1.55/TNR 93.10, which is threshold transfer rather than an anti-correlated model.

If the revision addresses your concerns, we would be grateful if you would consider updating your assessment. If anything remains unclear, we would be glad to clarify.

Reviewer_rnV2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RAVEL, a framework for auditing coding-agent evaluations that treats deterministic rules as versioned, evidence-linked sensors rather than ground-truth labels. Each rule is associated with applicability conditions, implementation provenance, evidence requirements, and proxy-derived reliability sets. The paper audits 402K production trajectories and finds substantial measurement instability: a parser correction changes 36.9% of letter grades, only 58.0% of sampled pairwise rankings are certified across two detector versions, and several rules are dominated by default values or context-dependent applicability. The authors also train a compact evidence-citing judge using four matched reward variants. Supervised initialization substantially improves structured-output coverage, but the proposed robust RAVEL-Set reward does not consistently outperform simpler alternatives.

Critical strengths
  1. Compelling large-scale measurement audit. The production audit provides strong evidence that deterministic trajectory metrics can be highly sensitive to parser versions, applicability assumptions, and missing evidence. The analysis identifies several distinct failure modes rather than relying on a single anecdote.

  2. Careful and transparent claim calibration. The paper explicitly distinguishes proxy agreement from semantic accuracy, treats automatically extracted outcomes as diagnostic rather than human ground truth, reports the negative result for its proposed reward, and explains the limitations caused by class imbalance and weak metamorphic contracts.

  3. Practically useful sensor-contract abstraction. Recording applicability, implementation version, provenance, and supporting trace evidence could make deployed agent-evaluation pipelines considerably easier to audit.

  4. Well-controlled reward comparison. The reward variants share initialization, data, prompts, optimizer settings, decoding, and all reward terms except the treatment of rule uncertainty, reducing several important experimental confounders.

Critical weaknesses
  1. Questionable admissibility of the pre-fix detector. The main version-uncertainty result treats the pre- and post-correction detectors as equally admissible, even though Appendix C.5 describes the earlier implementation as incorrectly matching tool calls to results. If the earlier version is a known bug, including it in the admissible set may artificially inflate the reported uncertainty. If both versions are genuinely defensible, the paper needs validation explaining why.

  2. Proxy agreement is not validated reliability. The proposed reliability sets are derived from agreement between only two model proxies, while the paper acknowledges that this agreement does not estimate semantic correctness. Nevertheless, these values are used as reliability weights in the training objective and score fusion. Without validation against human judgments or verified rule semantics, it is unclear whether the construction preserves meaningful uncertainty or merely proxy disagreement.

  3. No end-to-end evaluation of the full RAVEL method. The experiments do not evaluate the fused judge-sensor intervals from Equation 7 or the interval/conflict selector from Equation 8. The reported interval analysis uses only two detector versions, while the selective-risk metric uses self-reported confidence rather than the proposed selector. Consequently, the evidence supports versioning and applicability audits but does not establish the effectiveness of RAVEL's central fusion and selection mechanisms.

  4. Weak evidence for semantic judgment quality. The evaluation contains only 29 natural test failures and seven development failures. The metamorphic relations permit unchanged predictions, synthetic markers can reveal the intervention, and only seven observed recovery pairs are available. These diagnostics therefore provide limited evidence about evidence faithfulness or general process-level judgment quality.

Questions for authors
  1. Admissibility of the old detector. Why is the pre-fix detector considered admissible after the ordered-matching behavior was identified as a parser error? Was either version manually or executably validated on the affected trajectories? This determines whether the reported ranking uncertainty is genuine or caused by retaining a known-bad implementation.

  2. Meaning and validation of reliability. What quantity is the proxy-derived reliability value q intended to estimate? Do you have any validation showing that it correlates with human-assessed applicability, evidence fidelity, or rule correctness?

  3. Evaluation of the proposed selector. Were the full fused intervals and the interval/conflict selector evaluated on the frozen test set? If so, could you report their coverage-risk performance relative to self-reported confidence and a simple detector-disagreement baseline?

  4. Sensitivity to training seed. How sensitive are the reward rankings to random seed? Since the 128- and 256-step results are correlated checkpoints from one run per method, the observed reversal could reflect optimization noise rather than a general training-budget effect.

  5. Reliability-set granularity. Is the reliability set Q_k global to each criterion or specific to each trajectory-criterion pair? The description says proxies judge each trace-rule pair and that reliability fields are stored for every target, whereas Equations 5-7 use criterion-level notation.

2 comments
AuthorsOpenReview ↗

Response to rnV2: Admissibility, Proxy Validation, and End-to-End Evaluation

We thank Reviewer rnV2 for a careful and detailed review. The point about TES and the question about the selector led to substantial changes in the paper.

W1 & Q1 (Pre-fix detector). You are right that the pre-fix detector is not admissible: TES is constant at 0.0000 (changed rate 38.4%). We removed the phrase "equally admissible" from Section 6.1, split the version set into V_obs and V_adm, and added the predicate A(v) (non-degeneracy, evidence coverage, anchor agreement). Under V_adm only the post-fix version passes, so the version-induced interval collapses. The 36.9% is now reported as bug impact and the 42.0% as a fragility bound under V_obs. We did not validate either version trace-by-trace; the per-trace records are not available, and Section 7 says so. The non-degeneracy check alone rules out the TES = 0 version. For the strict/broad loop pair we only have 100-trace aggregates (25 vs 33; 92.0%; κ = 0.807; ρ = 0.884; 19.4×), so we do not extrapolate a certified-order rate.

W2 & Q2 (Proxy reliability). Here q is the lower endpoint of the hull of the joint/GPT/Claude Beta(1,1) 95% posterior intervals for agreement-based support; it is a support weight, not an accuracy estimate. We now describe it as a support interval, report the [0.0168, 0.6611] range, and no longer suggest that it estimates rule error. Q_k is criterion-level and q_ik is per-target; Section 3 states this. No human audit is available under the data-use agreement, and Section 7 says that Q_k captures proxy ambiguity.

W3 & Q3 (Eq. 7-8). We agree that Eqs. (7)-(8) are not evaluated end to end. In the objective suite the rule is the execution outcome itself, so fusing it would leak the label; and the production artifacts have no per-criterion rule-judge join. We removed Eq. (8) from the contributions and abstract, marked it as a formal extension, and now state that the rule we actually evaluate is version-interval certification (58.0% certified, mean width 0.0618). Risk@90 is now explicitly confidence-based; we do not report selector coverage-risk.

W4 (29 failures). We added a power analysis. With 29 failures and 517 successes, the Hanley-McNeil SE is 0.057, the 95% CI width is 0.22, and the minimum detectable difference is 0.22; the observed spread is 0.031 at 128 steps and 0.045 at 256. The ablation is therefore descriptive. Strict response rates are now reported (RAVEL-Set-256: 0.150/0.546; SFT: 0.139/0.555; Claude: 0.441/0.643; GPT-5.5: 0.626/0.808). Section 7 states the 29/546 and 7/124 limits, the visible markers, and the seven recovery pairs. A failure-enriched suite is not possible under the data agreement.

Q4 (Seed). The 128/256 comparison uses two checkpoints from one run, so we relabeled it checkpoint sensitivity. We cannot add seeds under the data agreement; Section 7 notes the confound. The three SFT seeds in the 35B study (74/77 for seeds 29 and 43) are only indirect evidence.

Q5 (Granularity). Q_k is criterion-level and q_ik is per-target; Section 3 defines both, and Section 4 gives the instance-level reductions.

The manuscript has been revised accordingly.

Thanks for author's detiled response. The paper introduces too many specialized terms for relatively simple concepts, and several terms are used inconsistently across sections. The paper indicated the defacts of the measurement instrument in the work of coding-agent evaluation, but did not present the end-to-end evaluation of proposed system, RAVEL. So, I can only give the rating of 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Reviewers and Area Chair: Summary of Revisions

We thank all three reviewers for their detailed reviews, and the Area Chair and Senior Area Chair for overseeing the discussion. The revision addresses the two main criticisms and narrows the paper's claims.

For the Area Chair: the main change is a scope correction. The paper no longer claims that the proposed reward or the selector improves evaluation. It reports (i) the audit of measurement instability, (ii) the admissibility formalization that separates bug impact from uncertainty among admissible versions, and (iii) a descriptive negative result about reward design. Eq. (8) is a formal extension, not a contribution.

First, we separate bug impact from admissible-version uncertainty: the pre-fix TES detector is degenerate and fails the admissibility predicate, so only the post-fix version is admissible; the 42.0% uncertified fraction is reported as a fragility bound under the deployed implementations (Sections 3 and 6.1). Second, Eq. (8) is a formal extension and is not evaluated; the rule we do evaluate is version-interval certification (Sections 4 and 7).

We also add a power analysis and a descriptive label for the reward ablation, strict response rates, an always-success baseline, and the GPT-5.5 event-ID invariance result. The revised PDF is included; point-by-point responses are under each review. We hope the revised scope is a better match for what the study can establish.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.