acceptodds
Under review as a conference paper at ICLR 2027

Raw Evaluator Disagreement Can Misrank Trajectories: Interaction-Targeted Agent RL

Abstract

Raw evaluator disagreement can be an unstable policy-learning target in multi-evaluator agent RL. It mixes evaluator-wide strictness with trajectory-specific evaluator dependence, so additive evaluator shifts can change trajectory-penalty rankings even when the interaction profiles are unchanged. We trace this instability to a row-specific cross term and show that, in a fully crossed block with unclipped scores, unrestricted additive shifts can reverse the raw ordering of any two distinct interaction profiles while leaving their interaction-energy ordering unchanged. Interaction-Suppressing Advantage (ISA) addresses this mismatch, relative to an interaction-targeted objective, by double centering the score block before assigning trajectory-level disagreement penalties. With a fixed common baseline and no clipping, its fixed-block advantage is invariant to evaluator-wide additive offsets. In 40 frozen-policy pilot blocks, raw and interaction-energy rankings disagree on 41.98% of trajectory pairs. Matched Raw-Energy/DC-Energy training under the same adaptive energy construction tests whether this target change affects learned policies: double centering lowers rollout discrepancy by 0.0241 (95% CI [-0.0272,-0.0210]), and a subsequent official-success evaluation of the frozen checkpoints yields a +3.64 percentage-point gain. In a separate 40-task Qwen3-8B study with ten seeds per arm, ISA reduces primary held-out global discrepancy by 27.5% relative to mean-minus-SD while improving held-out reward; with the interaction target fixed, a tuned comparison supports the adaptive penalty rule with a +4.36 percentage-point success gain. These results support a conditional design principle: when trajectory-specific evaluator dependence is the intended regularization target and a reliable fully crossed panel is available, remove shared evaluator shifts before forming trajectory-level disagreement penalties.

Primary area:
reinforcement learning
Keywords:
agent reinforcement learning, evaluator disagreement, interaction regularization, double centering, reward aggregation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8C3POpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the limitations of raw evaluator disagreement as a regularization signal in multi-evaluator agent RL. The authors show that evaluator-wide shifts can distort trajectory-level disagreement rankings and propose Interaction-Suppressing Advantage (ISA), which uses double centering to remove evaluator main effects and penalizes trajectory-specific interaction energy. Experiments show improvements in held-out evaluator discrepancy, reward, and task success.

Critical strengths
  1. The paper identifies an interesting issue with raw evaluator disagreement and provides a mathematical explanation of how evaluator-wide shifts can affect trajectory-penalty rankings.

  2. The experimental evaluation is fairly comprehensive, including multiple agent benchmarks, matched ablations, held-out evaluator evaluations, and official task-success measurements. The Raw-Energy vs. DC-Energy comparison is particularly useful for examining the effect of double centering.

Critical weaknesses
  1. While the paper explains why raw disagreement can be affected by evaluator-wide shifts, it is less clear why the remaining trajectory-specific disagreement should necessarily be penalized. Different evaluators may have different areas of expertise, and disagreement on a particular trajectory could reflect meaningful evaluator specialization rather than an undesirable signal.

  2. I found the paper difficult to follow due to the dense writing, extensive notation and abbreviations, and the large amount of supporting information deferred to the appendix. Some figures also appear visually crowded, with minor overlapping elements. These issues make it harder to understand and evaluate the main contributions from the main text alone.

Questions for authors

Could the authors further justify why interaction energy is an appropriate regularization target? In particular, how does ISA distinguish undesirable trajectory-specific evaluator disagreement from meaningful evaluator specialization?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 8C3P

Thank you for recognizing the raw-disagreement issue and the matched ablations. We have clarified the distinction between correcting a regularization target and deciding when interaction regularization is useful, and improved the main-text presentation.

1. Interaction energy and specialization

ISA measures panel-relative evaluator dependence; it cannot classify exploitation versus useful specialization. Double centering removes row and evaluator main effects, not all bias. Our rationale applies when evaluators assess a common task objective: a finite penalty trades mean reward against evaluator dependence. Appendix B.1 bounds residual sensitivity to reweighting the same panel, without guaranteeing task utility or transfer. Distinct valuable competencies may require task-specific weighting or multi-objective treatment.

Matched Raw-Energy/DC-Energy training gives a rollout-discrepancy difference of −0.0241 [−0.0272, −0.0210]; its official-success follow-up gives +3.639 pp [1.027, 6.292] (Table 1; Appendix I.8, Table 30). These results support the target intervention in the tested setting. The human-label gaming AUC of 0.6517 and overlap with proxy-specialization labels remain construct-validity limitations.

Tables S1–S3 below report the specialization-focused evaluation (Appendix I.6.4, Tables 26–28). This is the same study discussed with Reviewer 4Qpw, not an independent replication. Ten paired seeds, three equally weighted strata, 30 tasks per stratum, and eight rollouts per seed–task cell give 7,200 outcomes per method. Tasks and weights were fixed before training. This cohort uses different tasks, seeds, and checkpoints from the generic direct-success study.

The two primary comparisons use 10,000 crossed seed- and within-stratum task-bootstrap draws with common indices across methods. Individual two-sided 97.5% percentile intervals use Bonferroni adjustment to target joint nominal 95% coverage; bootstrap coverage is approximate. Both lower bounds exceed the prespecified −2 pp aggregate margin. Aggregate non-inferiority does not establish it in every stratum or guarantee preservation of every specialized behavior. Task-based specialization relevance is not human validation of each trajectory.

The earlier auxiliary flags are randomized failure-subsampling probes, not harm measures, and have been removed. ISA's training rule is unchanged.

2. Readability and figures

The main text distinguishes:

  • Raw-Energy/DC-Energy: the residual-target intervention under a matched adaptive rule.
  • ISA/DC-SD: complete penalty rules at a fixed residual target.
  • ISA/MMSD: complete methods.

Section 3.1 now has a compact notation box; Section 3.2 explains the conditional rationale. Section 4.5/Table 3 reports direct ISA–MMSD success, and Section 4.7 summarizes specialization evidence and scope. Figures 1–2 were redrawn with larger labels and fewer annotations. Figure 1 separates the schematic decomposition from direct-success evidence; Figure 2 shows row-centered scores, residuals, and penalties. We checked the PDF for legibility. Theorem statements remain in Section 3.3; proofs and bounded-shift derivations are in Appendix D.6–D.7. The main narrative ends on page 9.

Our claim is that this target correction helps in the evaluated conditions, not that all trajectory-specific disagreement is harmful.

Specialization-study results

Table S1. Aggregate outcomes

MethodSuccesses / outcomesSuccess rate
DR4296/720059.67%
MMSD4231/720058.76%
ISA4550/720063.19%

Pairing is by seed–task, not identical generated trajectories.

Table S2. Paired aggregate success differences

ContrastSuccess difference, ppAdjusted success intervalAggregate NI at −2 pp
ISA − DR+3.53[0.43, 6.49]Meets criterion
ISA − MMSD+4.43[1.15, 7.54]Meets criterion

The −2 pp margin is a prespecified aggregate tolerance, not a domain-standard threshold.

Table S3. Exploratory task-stratum results

StratumDRMMSDISAISA − DR, pp [95%]ISA − MMSD, pp [95%]
tau1509/2400 (62.88%)1462/2400 (60.92%)1567/2400 (65.29%)+2.42 [−2.75, 7.42]+4.38 [−0.17, 8.50]
SWE1319/2400 (54.96%)1269/2400 (52.88%)1414/2400 (58.92%)+3.96 [−0.46, 8.29]+6.04 [1.00, 11.00]
Toolathlon1468/2400 (61.17%)1500/2400 (62.50%)1569/2400 (65.38%)+4.21 [−0.42, 8.71]+2.88 [−1.38, 7.08]

Each stratum has 2,400 outcomes per method. Intervals are pointwise, unadjusted 95% percentile intervals and describe exploratory heterogeneity.

Reviewer_4QpwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper argues that raw evaluator disagreement can misrank trajectories relative to an interaction-specific regularization objective because evaluator main effects enter row-wise penalties through a cross term. ISA double-centers a crossed reward matrix and penalizes interaction energy using an adaptive coefficient. Matched training and official task-success evaluations support this intervention in the studied agent-RL setting.

Critical strengths
  1. ISA uses classical double centering to remove evaluator main effects before computing trajectory-level penalties. The method is simple, interpretable, and easy to implement given a fully crossed score matrix.
Critical weaknesses
  1. The main operation—two-way double centering—is classical. The novelty primarily lies in identifying the RL objective mismatch and validating this target choice rather than introducing a fundamentally new statistical technique.
  2. Interaction energy is only a panel-relative dependence signal: it may penalize legitimate specialization while failing to detect errors shared by all evaluators. Its human-label gaming AUC of 0.6517 indicates limited construct validity.
Questions for authors
  1. What evidence supports treating evaluator-specific interactions as nuisance variation rather than valid signals of task-specific competence?
  2. How can ISA distinguish harmful evaluator exploitation from legitimate specialization, given that both may produce large interaction energy?
  3. The invariance theorem covers additive evaluator shifts in a fixed, fully crossed block; what evidence supports its relevance under nonlinear, task-dependent evaluator differences?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 4Qpw

Thank you for recognizing the method's simplicity and interpretability. Double centering is classical. Our contribution is a policy-regularization target: identifying its mismatch with raw disagreement, deriving the correction, and testing it through matched policy training.

1. Novelty and the regularization target

We do not claim a new statistical decomposition. Raw row energy satisfies

piraw=B+pi+2c⟨β^,γ^i⟩.p_i^{\mathrm{raw}} = B + p_i + \frac{2}{c}\langle\hat{\beta},\hat{\gamma}_i\rangle.

Although B is common to trajectories, the cross term is not. Raw disagreement is therefore not simply interaction energy plus a constant. ISA changes this target while retaining the centered mean-reward term.

The matched Raw-Energy/DC-Energy comparison tests the complete target intervention within the same adaptive energy family, with the coefficient recomputed for each residual input. It reduces rollout discrepancy by 0.0241, with a 95% interval for DC minus Raw of [−0.0272, −0.0210]. The subsequent official-success evaluation gives +3.639 percentage points [1.027, 6.292] (main-text Table 1 and Appendix I.8, Table 30). These results support the target correction in the studied setting.

2. Why interactions, and can ISA distinguish exploitation from specialization?

ISA cannot distinguish harmful exploitation from legitimate specialization using interaction energy alone. The residual is a panel-relative dependence signal, not an estimate of pure nuisance. Legitimate competence can produce heterogeneous assessments, and errors shared by all evaluators can remain in the row effect.

The rationale is conditional: when the panel supplies alternative assessments of the same task objective, a finite penalty on evaluator dependence can improve the trade-off with mean reward. There is also a limited sensitivity interpretation. Proposition 3 in Appendix B.1 shows, for two weight vectors on the same evaluator support,

\leq \sqrt{c}\,\|w-w'\|_2\sqrt{p_i}.$$ This bounds same-panel residual sensitivity, without identifying exploitation or guaranteeing task utility or transfer. Distinct valuable competencies may require task-specific weighting or multi-objective treatment. Retaining mean reward does not guarantee preservation of every specialized behavior. We retain the unfavorable construct-validity evidence. On 300 human-labeled trajectories, the penalty's gaming AUC is 0.6517. A threshold attaining 0.70 gaming recall also flags 18.4%–27.3% of proxy-labeled specialization trajectories. The evaluator-shared pattern receives only 1.09× the reference median penalty, compared with 5.10× for the evaluator-specific pattern (Appendices I.6.1–I.6.2). These are limitations, not evidence of reliable gaming detection. We have completed a specialization-focused task evaluation reported in Appendix I.6.4, Tables 26–28. Each method has 7,200 outcomes: ten seeds, three equally weighted task strata, 30 tasks per stratum, and eight rollouts per seed–task cell. Strata and weights were fixed before training; outcomes were not selected or excluded according to results. Specialization relevance follows the task design, rather than human validation of every trajectory. ISA has 4550/7200 successes (63.19%), versus 4296/7200 for DR (59.67%) and 4231/7200 for MMSD (58.76%). Its paired gains are +3.53 pp [0.43, 6.49] and +4.43 pp [1.15, 7.54]. The two primary comparisons use 10,000 crossed seed- and within-stratum task-bootstrap draws with common resampling indices across methods. Individual two-sided 97.5% percentile intervals target nominal joint 95% coverage through Bonferroni adjustment; bootstrap coverage is approximate. Both lower bounds exceed zero and the prespecified −2 pp aggregate non-inferiority margin. This is evidence for aggregate success in the evaluated tasks. It does not establish non-inferiority in every stratum: the exploratory ISA–DR interval for tau-like tasks is [−2.75, 7.42] pp. Nor does it directly establish preservation of every independently validated specialized behavior. The auxiliary counts 317/701/519 are randomized failure-subsampling probes with method-specific probabilities and provide no harm or exploitation evidence. They are omitted from the revised manuscript; the specialization analysis uses observed benchmark success. Part 2 of this response addresses additive invariance and the nonlinear retraining experiments.

3. Additive invariance and nonlinear, task-dependent differences

Proposition 2 concerns the score-to-advantage map for a fixed fully crossed block, with a fixed common baseline and no clipping or nonlinear score transformation. Task dependence alone does not invalidate it: an offset δ(t,j) can vary by task and evaluator if it is constant across trajectories within that task's block. Trajectory-dependent shifts and evaluator-specific nonlinear transformations generally do not cancel. We do not claim whole-training invariance.

The theorem identifies one avoidable source of sensitivity. Heterogeneous-panel and external-evaluator evaluations give empirical context but are not proofs of nonlinear robustness. Appendix K.3 also reports adverse training sensitivity: ISA's discrepancy gap rises from 0.046 to 0.087 and 0.126 under one- and two-corrupted scoring-rule conditions. These are discrepancy gaps, not official task-success differences.

We retrained ISA, Raw-Energy, and DR under identity, evaluator- and task-family-specific power and sigmoid transformations, including strong saturation. Initial policies, tasks, paired seeds, sampling/update budgets, and other update settings were matched. ISA and Raw-Energy recomputed coefficients from their respective residuals under the same adaptive energy rule. Evaluation used official-success criteria.

The evaluation has four regimes × three methods × ten seeds × 90 tasks × eight rollouts, or 86,400 outcomes. Table 2A reports the ISA–Raw-Energy comparisons.

Table 2A. Nonlinear retraining stress test

RegimeISA successRaw-Energy successISA − Raw-Energy, pp [adjusted interval]
Identity64.51%58.79%+5.72 [1.64, 9.99]
Moderate power57.81%51.39%+6.42 [3.01, 9.81]
Moderate sigmoid55.21%48.65%+6.56 [3.02, 10.07]
Severe sigmoid36.13%36.65%−0.53 [−4.24, 3.17]

Each regime has 7,200 outcomes per method. The four primary contrasts use individual two-sided 98.75% crossed-bootstrap percentile intervals, targeting nominal joint 95% coverage through Bonferroni adjustment. Resampling indices are shared across methods and regimes.

Relative gains under moderate transformations do not imply absolute invariance. Compared with ISA's own identity condition, success falls by −6.71 pp [−9.07, −4.22] under moderate power, −9.31 pp [−12.15, −6.43] under moderate sigmoid, and −28.39 pp [−30.96, −25.75] under severe sigmoid; these are exploratory, unadjusted 95% intervals. The severe-regime comparison crosses zero and establishes neither superiority nor equivalence.

The supplement specifies all 12 evaluator-by-task transformations below. Indices j=0–3 map to A1–A4 and g=0–2 to tau-bench, SWE-bench-lite, and Toolathlon. Power uses f(x)=x^a; sigmoid uses f(x)=1/(1+exp(−k(x−b))). Each native [0,1] score replication is transformed before averaging four replications. There is no post-transform normalization, quantile calibration, or score/advantage clipping; full-block residuals and adaptive coefficients are recomputed. Data-error blocks containing NaN or native out-of-range scores are excluded rather than silently clipped. Optimizer gradient clipping is a separate operation.

Table 2B. Full nonlinear parameter grid

Task familyEvaluatorPower aModerate kModerate bSevere kSevere b
tau-benchA10.6753.00.375240.650
tau-benchA20.9004.00.425320.690
tau-benchA31.1255.00.475400.730
tau-benchA41.3506.00.525480.770
SWE-bench-liteA10.7503.30.400240.670
SWE-bench-liteA21.0004.40.450320.710
SWE-bench-liteA31.2505.50.500400.750
SWE-bench-liteA41.5006.60.550480.790
ToolathlonA10.8253.60.425240.690
ToolathlonA21.1004.80.475320.730
ToolathlonA31.3756.00.525400.770
ToolathlonA41.6507.20.575480.810

Appendix K.5, Table 46, specifies the parameters and transformation order; Tables 47–48 report relative and absolute contrasts. These results do not establish nonlinear or whole-training invariance. The generic and specialization studies use different tasks, seeds, and checkpoints: 57.26% versus 63.19% is not a causal estimate of specialization or task-set change. Appendix L, Table 49, maps these studies.

Thank you for the additional experiments and clarifications. The nonlinear stress tests and specialization-focused evaluation strengthen the empirical evidence, but my main concern remains the justification for penalizing evaluator-specific interactions. The sensitivity bound does not establish that such dependence is harmful, and aggregate success gains do not resolve whether ISA suppresses legitimate specialized behaviors or clarify when interaction penalization is appropriate. I appreciate the explicit acknowledgment of these limitations, but the applicability of the proposed regularization target remains insufficiently established. I will therefore maintain my original score.

AuthorsOpenReview ↗

Clarification of the interaction regularization target

Thank you for clarifying the remaining concern. We would like to make the rationale for the penalty and its applicability conditions more explicit.

Rationale for the penalty. ISA expresses a preference for task performance that is less dependent on the particular scoring implementation. When evaluator implementations are alternative measurements of fixed task requirements, sensitivity to their incidental scoring differences can be an unwanted property of the learned policy. After shared evaluator strictness is removed, interaction energy measures the remaining trajectory-specific variation across the panel. Within a block, at equal mean reward, ISA assigns a higher advantage to the trajectory with lower interaction energy; at unequal mean reward, its finite penalty trades off reward against this dependence. Double centering ensures that the dependence penalty is not distorted by shared strictness offsets. The sensitivity bound characterizes the quantity being controlled, rather than proving that individual interactions are harmful. The value of this preference must be tested against independent task outcomes.

Applicability conditions. This rationale requires (i) common task requirements across evaluators, (ii) scoring-implementation uncertainty against which robustness is desired, and (iii) evidence that the resulting reward–dependence trade-off is useful on the intended task distribution. A shared task objective alone does not establish that every interaction should be suppressed. If evaluators intentionally reward different valuable competencies, their differentiated assessments may be part of the objective; task-conditioned weighting or multi-objective treatment is then more appropriate than treating those differences as nuisance.

Our evaluator construction motivates the shared-objective setting: A1 aggregates auditable benchmark subchecks, A2 independently recomputes those components, A3 normalizes representation before applying A1's components and weights, and A4 provides a semantic completion score from the task requirements and evidence (Appendix F.3, Table 7). Their auxiliary scores can differ in calibration and semantics, so this design motivates the target choice without certifying evaluator equivalence.

Empirical support and specialization. The matched Raw-Energy/DC-Energy study tests the target intervention within the same adaptive energy-penalty family. Its subsequent benchmark-native completion evaluation gives +3.639 percentage points [1.027, 6.292] (Table 1; Appendix I.8, Table 30). This is evidence that the trade-off improves task completion in the tested setting, evaluated independently of interaction-derived labels. The specialization-focused study adds task-level evidence, while retention of particular specialized behaviors remains a separate endpoint. Where those behaviors are deployment requirements, explicit retention checks are needed. Our contribution is the explicit regularization target, its shift-corrected construction, and matched policy-learning evidence under the specified evaluator and task configurations.

Reviewer_7fZbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates penalty construction in multi-evaluator reinforcement learning. It distinguishes evaluator-wide differences in scoring strictness from interactions between trajectories and evaluators, analyzes their effects on trajectory penalty rankings, and proposes Interaction-Suppressing Advantage (ISA). ISA applies double centering to the complete score matrix, constructs an adaptive penalty from the mean squared interaction residuals, and subtracts this penalty from the centered mean reward to obtain the advantages used for policy updates. Experiments primarily use Qwen3-8B on three benchmarks, tau-bench, SWE-bench-lite, and Toolathlon, and support the method’s effectiveness in the evaluated settings.

Critical strengths
  1. The paper provides a useful perspective on how evaluator disagreement should influence policy learning. Distinguishing differences in evaluator strictness from interactions between trajectories and evaluators gives the proposed method a clear motivation.

  2. The paper is straightforward and supported by extensive experiments. The evaluation considers both scoring consistency and actual task success, providing evidence of the method’s practical value.

Critical weaknesses
  1. Appendix J.3 shows that when trajectories exhibiting the same evaluator-dependent behavior form a majority within a group, ISA may assign smaller penalties to these trajectories, further increasing their relative advantage. However, Section 4.7 and Appendix J.4 mainly examine the extreme case in which 13 of 14 trajectories exhibit the same pattern, with limited results reported for less extreme majority compositions. The low frequency of this extreme case therefore does not establish that the risk is negligible during actual training.

  2. The paper uses MMSD as its primary comparator, but the reported advantages of ISA over MMSD are mainly measured by scoring discrepancy and held-out reward. The appendix does include evaluations of official task success: Appendix I.8 compares ISA with Raw-Energy, while Appendix J.2 compares ISA with DC-SD. These results support the respective design choices, but they do not directly establish whether the scoring improvements of ISA over MMSD also translate into higher task success rates.

Questions for authors
  1. Could you use the trajectory pools already collected in Appendix J.4 to report the frequencies, conditional advantage gaps, and uncertainty estimates for n = 8, 9, 10, 11, and 12 at each training stage, separately from statistics on actual training score blocks? Please also clarify how the E/H patterns are identified in real trajectories. Evidence that amplification in these configurations is also rare or does not affect official task success would alleviate my first concern.

  2. Could you report official task success rates for ISA and MMSD on the same held-out tasks, together with paired differences and confidence intervals? This would help determine whether the improvements in scoring discrepancy and held-out reward also translate into better task completion.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7fZb

Thank you for recognizing the motivation and practical value of separating evaluator-wide strictness from trajectory-specific interactions. We have completed additional composition analyses and a direct ISA–MMSD task-success evaluation to address your two concerns.

1. Majority compositions beyond n = 13

We agree that the low frequency of the 13-of-14 case does not establish that n = 8–12 compositions are rare or harmless. We re-analyzed the trajectory pools at updates 0, 60, 120, 180, and 240, reporting each n separately. The accompanying composition-audit table gives frequencies, conditional advantage gaps, and pointwise 95% intervals for all 25 checkpoint–composition combinations.

The conditional ISA gap is the mean E advantage minus the mean H advantage within a block. Amplification is this gap minus the corresponding DR gap on the same score matrix; a positive ISA gap alone is not sufficient to identify amplification.

Combined n = 8–12 pool frequencies at the five checkpoints were 2.15%, 2.23%, 2.11%, 2.03%, and 2.06%. The final-minus-initial difference was −0.095 percentage points (95% CI [−0.393, 0.210]). Amplification relative to DR was positive in these categories. Thus, the failure mode occurs with low but nonzero observed frequency, and we do not detect an increase over checkpoints. This does not establish that it is harmless or cannot increase.

We separately examined actual score blocks used for training updates:

Update windown = 8–12 events / blocksFrequency [95% CI]
1–6014/6002.33% [0.54, 4.89]
61–12014/6002.33% [0.63, 4.49]
121–18013/6002.17% [0.66, 4.03]
181–2408/6001.33% [0.16, 3.01]

These are training-block frequencies, distinct from pool resampling; updates within a seed are not independent training runs.

How E/H are identified. The fixed signature is a completion assertion contradicted by execution evidence. E requires an unambiguous completion claim explicitly contradicted by reliable evidence for the same object, execution version, and time. H means no such claim, or claims verifiably consistent with evidence; it includes unsuccessful trajectories that honestly report failure. Unresolved cases remain U and are neither recoded as H nor deleted from score blocks. For example, a failed cancellation followed by an active-order terminal state and “the order has been cancelled” is E; “the cancellation failed” is H. For software tasks, contradictions are tied to the claimed command and patch version, rather than inferred from hidden-test failure.

The codebook was frozen before inspecting the new audit outcomes. Two annotators independently labeled evidence bundles while blinded to method, checkpoint, evaluator outputs, and companion composition; disagreements were adjudicated under the same blinding. Labels were fixed across companion draws. This operationalization covers one concrete class of evaluator-dependent behavior, not all exploitation or specialization.

2. Direct official task success: ISA versus MMSD

We agree that ISA–Raw-Energy and ISA–DC-SD comparisons cannot replace this comparison. We evaluated frozen ISA and MMSD checkpoints on the same held-out tasks using benchmark-native binary completion rules. The layout was 30 tasks per benchmark × 10 paired seeds × 8 rollouts, giving 7,200 outcomes per method.

BenchmarkMMSDISAISA − MMSD, pp [95% CI]
tau-bench1272/2400 (53.00%)1419/2400 (59.13%)+6.13 [1.29, 10.67]
SWE-bench-lite1087/2400 (45.29%)1190/2400 (49.58%)+4.29 [0.50, 7.88]
Toolathlon1395/2400 (58.13%)1514/2400 (63.08%)+4.96 [0.25, 9.33]
Overall3754/7200 (52.14%)4123/7200 (57.26%)+5.13 [2.42, 7.82]

Intervals use 10,000 paired seed- and benchmark-stratified task-bootstrap draws, retaining method pairing within seed–task units. Benchmark-specific intervals are pointwise, not multiplicity-adjusted confirmations.

This direct evaluation supports higher task completion for ISA relative to MMSD in the tested setting. It does not establish causal mediation by discrepancy reduction or show that majority-composition amplification has no effect on success.

We will incorporate the full composition audit and operational definitions into Appendix J.4, add the direct ISA–MMSD success table to the main experiments, and revise Section 4.7 to state “no detected increase” while retaining majority-composition amplification as a limitation.

AuthorsOpenReview ↗

Composition audit: n = 8–12 across training checkpoints

This table provides the n-by-checkpoint trajectory-pool results accompanying our response. The ISA conditional gap is the mean E advantage minus the mean H advantage within the same block. Amplification relative to DR is the ISA gap minus the DR gap computed on that score matrix. Frequencies are percentages; gaps are in reward/advantage units.

UpdatenFrequency (%) [95% CI]ISA conditional gap [95% CI]Amplification relative to DR [95% CI]
080.910 [0.685, 1.150]0.0163 [0.0158, 0.0167]0.0013 [0.0013, 0.0013]
090.593 [0.455, 0.748]0.0182 [0.0173, 0.0190]0.0031 [0.0030, 0.0033]
0100.373 [0.255, 0.513]0.0206 [0.0195, 0.0216]0.0058 [0.0055, 0.0061]
0110.198 [0.133, 0.273]0.0255 [0.0236, 0.0272]0.0104 [0.0097, 0.0111]
0120.078 [0.038, 0.123]0.0321 [0.0288, 0.0355]0.0181 [0.0163, 0.0200]
6080.948 [0.760, 1.163]0.0163 [0.0158, 0.0170]0.0013 [0.0013, 0.0014]
6090.563 [0.432, 0.695]0.0178 [0.0171, 0.0184]0.0031 [0.0029, 0.0032]
60100.390 [0.280, 0.515]0.0211 [0.0202, 0.0222]0.0059 [0.0057, 0.0062]
60110.225 [0.145, 0.308]0.0258 [0.0242, 0.0276]0.0105 [0.0099, 0.0113]
60120.100 [0.058, 0.150]0.0350 [0.0318, 0.0382]0.0198 [0.0180, 0.0216]
12080.885 [0.708, 1.085]0.0163 [0.0158, 0.0169]0.0013 [0.0013, 0.0013]
12090.593 [0.450, 0.750]0.0183 [0.0175, 0.0191]0.0032 [0.0030, 0.0033]
120100.348 [0.240, 0.465]0.0209 [0.0197, 0.0220]0.0059 [0.0055, 0.0062]
120110.223 [0.148, 0.313]0.0253 [0.0234, 0.0269]0.0103 [0.0096, 0.0110]
120120.063 [0.028, 0.105]0.0351 [0.0313, 0.0388]0.0198 [0.0177, 0.0219]
18080.895 [0.705, 1.088]0.0164 [0.0159, 0.0169]0.0013 [0.0013, 0.0013]
18090.515 [0.397, 0.663]0.0181 [0.0172, 0.0189]0.0031 [0.0030, 0.0033]
180100.333 [0.227, 0.445]0.0204 [0.0189, 0.0218]0.0057 [0.0053, 0.0061]
180110.193 [0.133, 0.263]0.0258 [0.0236, 0.0278]0.0105 [0.0097, 0.0114]
180120.090 [0.045, 0.148]0.0351 [0.0305, 0.0394]0.0198 [0.0172, 0.0223]
24080.880 [0.690, 1.080]0.0163 [0.0157, 0.0167]0.0013 [0.0013, 0.0013]
24090.548 [0.405, 0.703]0.0181 [0.0175, 0.0188]0.0031 [0.0030, 0.0032]
240100.358 [0.250, 0.470]0.0212 [0.0200, 0.0225]0.0059 [0.0056, 0.0063]
240110.195 [0.123, 0.275]0.0253 [0.0238, 0.0269]0.0103 [0.0098, 0.0110]
240120.075 [0.030, 0.130]0.0340 [0.0296, 0.0384]0.0192 [0.0167, 0.0217]

The nominal design is 10 seeds × 40 tasks × 100 companion draws per checkpoint. Draws reuse anchors and trajectories and are not 40,000 independent observations. The analysis uses 2,000 percentile-bootstrap draws with shared seed and stratum resampling indices across checkpoints. Intervals are pointwise 95% intervals, not a simultaneous confidence band or a formal no-worsening test. Values retain the reported rounding.

Positive amplification is observed in these categories. Their low frequency and the absence of a detected increase do not establish that the mechanism is harmless; actual training-block frequencies are reported separately in the main response.

Thank you for the additional experiments and clarifications. The rebuttal has helped address my concerns to some extent. After carefully considering the authors' responses, the other reviewers' comments, and the overall merits of the paper, I have decided to maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.