- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the limitations of raw evaluator disagreement as a regularization signal in multi-evaluator agent RL. The authors show that evaluator-wide shifts can distort trajectory-level disagreement rankings and propose Interaction-Suppressing Advantage (ISA), which uses double centering to remove evaluator main effects and penalizes trajectory-specific interaction energy. Experiments show improvements in held-out evaluator discrepancy, reward, and task success.
Critical strengths
-
The paper identifies an interesting issue with raw evaluator disagreement and provides a mathematical explanation of how evaluator-wide shifts can affect trajectory-penalty rankings.
-
The experimental evaluation is fairly comprehensive, including multiple agent benchmarks, matched ablations, held-out evaluator evaluations, and official task-success measurements. The Raw-Energy vs. DC-Energy comparison is particularly useful for examining the effect of double centering.
Critical weaknesses
-
While the paper explains why raw disagreement can be affected by evaluator-wide shifts, it is less clear why the remaining trajectory-specific disagreement should necessarily be penalized. Different evaluators may have different areas of expertise, and disagreement on a particular trajectory could reflect meaningful evaluator specialization rather than an undesirable signal.
-
I found the paper difficult to follow due to the dense writing, extensive notation and abbreviations, and the large amount of supporting information deferred to the appendix. Some figures also appear visually crowded, with minor overlapping elements. These issues make it harder to understand and evaluate the main contributions from the main text alone.
Questions for authors
Could the authors further justify why interaction energy is an appropriate regularization target? In particular, how does ISA distinguish undesirable trajectory-specific evaluator disagreement from meaningful evaluator specialization?
Response to Reviewer 8C3P
Thank you for recognizing the raw-disagreement issue and the matched ablations. We have clarified the distinction between correcting a regularization target and deciding when interaction regularization is useful, and improved the main-text presentation.
1. Interaction energy and specialization
ISA measures panel-relative evaluator dependence; it cannot classify exploitation versus useful specialization. Double centering removes row and evaluator main effects, not all bias. Our rationale applies when evaluators assess a common task objective: a finite penalty trades mean reward against evaluator dependence. Appendix B.1 bounds residual sensitivity to reweighting the same panel, without guaranteeing task utility or transfer. Distinct valuable competencies may require task-specific weighting or multi-objective treatment.
Matched Raw-Energy/DC-Energy training gives a rollout-discrepancy difference of −0.0241 [−0.0272, −0.0210]; its official-success follow-up gives +3.639 pp [1.027, 6.292] (Table 1; Appendix I.8, Table 30). These results support the target intervention in the tested setting. The human-label gaming AUC of 0.6517 and overlap with proxy-specialization labels remain construct-validity limitations.
Tables S1–S3 below report the specialization-focused evaluation (Appendix I.6.4, Tables 26–28). This is the same study discussed with Reviewer 4Qpw, not an independent replication. Ten paired seeds, three equally weighted strata, 30 tasks per stratum, and eight rollouts per seed–task cell give 7,200 outcomes per method. Tasks and weights were fixed before training. This cohort uses different tasks, seeds, and checkpoints from the generic direct-success study.
The two primary comparisons use 10,000 crossed seed- and within-stratum task-bootstrap draws with common indices across methods. Individual two-sided 97.5% percentile intervals use Bonferroni adjustment to target joint nominal 95% coverage; bootstrap coverage is approximate. Both lower bounds exceed the prespecified −2 pp aggregate margin. Aggregate non-inferiority does not establish it in every stratum or guarantee preservation of every specialized behavior. Task-based specialization relevance is not human validation of each trajectory.
The earlier auxiliary flags are randomized failure-subsampling probes, not harm measures, and have been removed. ISA's training rule is unchanged.
2. Readability and figures
The main text distinguishes:
Section 3.1 now has a compact notation box; Section 3.2 explains the conditional rationale. Section 4.5/Table 3 reports direct ISA–MMSD success, and Section 4.7 summarizes specialization evidence and scope. Figures 1–2 were redrawn with larger labels and fewer annotations. Figure 1 separates the schematic decomposition from direct-success evidence; Figure 2 shows row-centered scores, residuals, and penalties. We checked the PDF for legibility. Theorem statements remain in Section 3.3; proofs and bounded-shift derivations are in Appendix D.6–D.7. The main narrative ends on page 9.
Our claim is that this target correction helps in the evaluated conditions, not that all trajectory-specific disagreement is harmful.
Specialization-study results
Table S1. Aggregate outcomes
Pairing is by seed–task, not identical generated trajectories.
Table S2. Paired aggregate success differences
The −2 pp margin is a prespecified aggregate tolerance, not a domain-standard threshold.
Table S3. Exploratory task-stratum results
Each stratum has 2,400 outcomes per method. Intervals are pointwise, unadjusted 95% percentile intervals and describe exploratory heterogeneity.