- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies a mismatch between reward-model prediction quality and downstream decision quality. It shows that increasing feedback data can improve prediction of a proxy target while reducing the utility of the selected candidate. The paper introduces a symmetrization-based diagnostic that decomposes utility changes into top-set entry and within-label allocation effects, followed by a local repair formulation. These ideas are evaluated using several frozen candidate pools and two industrial online experiments.
Critical strengths
- The paper studies a practical problem: improving prediction of behavioral feedback does not necessarily improve the decisions induced by the reward model. The large-scale online experiment provides valuable evidence that this mismatch can persist throughout the entire RM and RL pipeline.
- The paper includes contrasting cases, permutation controls, repair experiments, and extensive appendices.
Critical weaknesses
- The presentation is excessively compressed and difficult to follow. Key concepts such as feedback-top sets, entry mass, conditional allocation advantage, symmetrization, and local repair are introduced in rapid succession without a concrete running example. Multiple datasets, candidate pools, signals, selectors, and experimental protocols are interleaved before their relationships are clearly established. As a result, it is difficult to determine exactly which hypothesis each experiment tests.
- The claimed evidence chain combines substantially different experimental settings. The quantity-scaling study, alpha-interpolation protocol, local repair experiments, learned-selector analysis, and online experiments use different populations, models, signals, and interventions. Appendix Table 5 explicitly acknowledges that these are complementary tests rather than replications of one chain. Nevertheless, the main text often presents them as establishing a single shared mechanism. A unified held-out experiment connecting prediction scaling, allocation diagnosis, repair, and downstream utility is missing.
- The online mechanism claim is not fully supported. In Experiment 2, the critical conversion comparison between semantic filtering and random downweighting is only marginal (p=0.076), while the semantic-filtering comparison against the baseline does not survive Bonferroni correction. Therefore, the claim that the random control “confirms” or “isolates” semantic filtering as the mechanism is too strong. The proprietary data and incomplete model details further limit independent verification.
Questions for authors
- Can the paper be reorganized around one concrete example and a table that maps every experiment to its dataset, feedback target, downstream outcome, selector, intervention, and intended conclusion?
- Can the authors evaluate the full sequence of scaling, diagnosis, and repair on the same untouched task population rather than connecting heterogeneous protocols retrospectively?
- What practical or theoretical advantage does local repair provide over directly training an outcome predictor using the same downstream labels and information?
- Given that the B-vs-C conversion comparison has p=0.076, on what statistical basis can semantic filtering be claimed as the isolated mechanism?