acceptodds
Under review as a conference paper at ICLR 2027

Scaling Feedback Alone Is Not Scaling Supervision: From Prediction Quality to Decision Quality in Reward Learning

Abstract

Scaling behavioral feedback data improves reward-model prediction but can reduce downstream decision quality. On an industrial dialogue agent serving over 100,000 users per group, increasing visit-event training data from 50% to 100% raised prediction AUC by 0.018 yet lowered order conversion by 0.15 percentage points (p<0.0001) through the full RM to RL to deployment pipeline. This prediction–decision mismatch extends to frozen-pool settings: across five conditions on tau^2-bench and VisualWebArena, better feedback fit accompanies worse candidate selection, with utility losses from -0.04 to -0.25. We propose a loss-preserving allocation diagnostic that exactly decomposes utility change into entry-mass and within-label ordering components, and derive an exact ceiling C-D on local repair gain within feedback-equivalent top sets. The ceiling holds universally—zero violations across 4,000+ task-configuration instances in three tau^2-bench domains—and significant positive repair appears under weak base selectors, confirming the diagnostic is actionable. A second online experiment with a random-down weight control isolates the mechanism: semantic filtering of ambiguous positive labels—not regularization from reduced volume—drives the improvement.

Primary area:
reinforcement learning
Keywords:
reward learning, deployment feedback, agentic systems, reward modeling, verifier optimization, reinforcement learning, optimizer robustness, AI agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_PEz6OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a mismatch between reward-model prediction quality and downstream decision quality. It shows that increasing feedback data can improve prediction of a proxy target while reducing the utility of the selected candidate. The paper introduces a symmetrization-based diagnostic that decomposes utility changes into top-set entry and within-label allocation effects, followed by a local repair formulation. These ideas are evaluated using several frozen candidate pools and two industrial online experiments.

Critical strengths
  1. The paper studies a practical problem: improving prediction of behavioral feedback does not necessarily improve the decisions induced by the reward model. The large-scale online experiment provides valuable evidence that this mismatch can persist throughout the entire RM and RL pipeline.
  2. The paper includes contrasting cases, permutation controls, repair experiments, and extensive appendices.
Critical weaknesses
  1. The presentation is excessively compressed and difficult to follow. Key concepts such as feedback-top sets, entry mass, conditional allocation advantage, symmetrization, and local repair are introduced in rapid succession without a concrete running example. Multiple datasets, candidate pools, signals, selectors, and experimental protocols are interleaved before their relationships are clearly established. As a result, it is difficult to determine exactly which hypothesis each experiment tests.
  2. The claimed evidence chain combines substantially different experimental settings. The quantity-scaling study, alpha-interpolation protocol, local repair experiments, learned-selector analysis, and online experiments use different populations, models, signals, and interventions. Appendix Table 5 explicitly acknowledges that these are complementary tests rather than replications of one chain. Nevertheless, the main text often presents them as establishing a single shared mechanism. A unified held-out experiment connecting prediction scaling, allocation diagnosis, repair, and downstream utility is missing.
  3. The online mechanism claim is not fully supported. In Experiment 2, the critical conversion comparison between semantic filtering and random downweighting is only marginal (p=0.076), while the semantic-filtering comparison against the baseline does not survive Bonferroni correction. Therefore, the claim that the random control “confirms” or “isolates” semantic filtering as the mechanism is too strong. The proprietary data and incomplete model details further limit independent verification.
Questions for authors
  1. Can the paper be reorganized around one concrete example and a table that maps every experiment to its dataset, feedback target, downstream outcome, selector, intervention, and intended conclusion?
  2. Can the authors evaluate the full sequence of scaling, diagnosis, and repair on the same untouched task population rather than connecting heterogeneous protocols retrospectively?
  3. What practical or theoretical advantage does local repair provide over directly training an outcome predictor using the same downstream labels and information?
  4. Given that the B-vs-C conversion comparison has p=0.076, on what statistical basis can semantic filtering be claimed as the isolated mechanism?
Reviewer_AZLiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies an issue in behavioral feedback extension: increasing the amount of feedback training data can improve the prediction accuracy of reward models, but does not necessarily enhance the quality of final decisions and may even lead to worse downstream performance. The authors propose a diagnostic method that maintains prediction loss constant: swapping predicted scores between candidates with identical feedback labels does not change the per-sample loss, yet may alter the final selection and task outcome. Based on this observation, the paper analyzes utility changes through probability uniformization within sets and derives an upper bound on the benefit of local repairs. Experiments include feedback learning and repair on a fixed candidate pool, as well as two online experiments in real dialogue systems.

Critical strengths

The paper addresses a practical concern in reward learning. The first online experiment provides useful evidence that improved feedback prediction can coincide with worse downstream outcomes through a complete reward-model training, policy optimization, and deployment pipeline.

The diagnostic makes useful distinctions between entering a candidate set, choosing within that set, and the remaining opportunity for improvement. In particular, the paper clearly explains why a decline in advantage over uniform selection does not imply that uniform selection would improve the model.

Critical weaknesses

There is still a missing direct connection between the repair experiment and the previous feedback quantity experiment. The current repair primarily targets weaker selectors, introducing additional features and direct supervision on downstream outcomes; therefore, it remains unclear to what extent these gains stem from the proposed diagnosis.

The proposed decomposition helps explain existing results, but its value as a practical diagnostic tool has not yet been fully established. Upper-bound zero violation is a property directly guaranteed by construction, yet the paper has not demonstrated whether this diagnosis can predict new failure configurations or aid in selecting more effective repairs. Experiments that directly measure the utility of the diagnosis are recommended to support its claim as a pre-deployment inspection tool.

Questions for authors

Can the previously high-budget model, which performed worse due to added feedback data, be directly repaired? The current repair experiments mainly focus on weaker selectors.

Besides explaining utility changes after the fact, can this diagnosis help predict which configurations will fail or assist in selecting effective repair methods?

Reviewer_wGYkOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Unfortunately I had an extremely difficult time following this paper. I will try my best to summarize the paper. The authors argue that sometimes more feedback alone does not transfer to better model outcomes. That is, more data can be used to train a more accurate reward model, but optimizing against that reward model may not always lead to better downstream models. I think this core argument makes sense. The authors attempt to showcase this across five different settings of tau² bench and VisualWebArena. They provide a mathematical formulation of their setting and back this with an experiment. The experiment has three conditions, a baseline where the reward model is trained on all visit-event labels (for a web navigation task), a condition where ambiguous visit events are removed, and a random control. They find despite using less data the second condition was best. Overall they argue more feedback doesn't always lead to better models.

Critical strengths

The main argument of the paper seems to be reasonable (that sometimes collecting better data or the right data is more important than collecting more data).

Critical weaknesses

The paper is missing the required AI-disclosure statement. I also checked parts of the paper for AI authorship and they were flagged as 100% AI generated. I think it would be useful to include the AI disclosure statement and comment on how much of the paper and experiments were AI produced.

I found the paper extremely challenging to follow. The abstract/intro dive directly into result without motivation or context. For example the very first line of the paper is "On tau² bench retail, scaling training feedback from 5% to 100% of the training fold reduces prediction error by 0.0043 while reducing selection utility by 0.0440." By this point in the paper we have not been introduced to the problem yet, and we certainly have not been introduced to the way we are measuring prediction error or selection utility and what is meant by feedback on a percentage of the training fold. The rest of the paper reads quite similarly to me.

I found the mathematical derivations to be a bit confusing. Also some terms are never defined, for example "M" in equation (1) is used without definition.

Some of the statistical significance levels reported are inconsistent given the described experiment setting. For instance lines 305-306 say "Within-set permutation tests (500 permutations) confirm genuine discrimination in all five domain x coarse-definition combinations (p <0.001)." The lowest p-value measurable with Monte Carlo permutation calculation and 500 permutations is p = 1/501 = 0.001996 which is above p<0.001. This p<0.001 figure is used several times throughout the paper.

The paper could benefit from some visual figures, right now everything is presented in tables.

Questions for authors

Can you please add an AI-Usage Disclosure to the paper?

Can you help clarify to me the primary motivation for your paper? I was struggling to understand a bit and believe that if I understand better I can provide more substantive feedback on the experimental setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.