acceptodds
Under review as a conference paper at ICLR 2027

Winning Comparisons, Losing Predictions: Comparison Feedback Misleads World Models

Abstract

Held-out comparison loss can select checkpoints that predict observed outcomes less well. In a separately frozen Delta-IRIS/Crafter study with correct nearest-candidate labels, feedback-selected checkpoints have worse covered-target likelihood than fixed-500 endpoints in both principal pairwise arms. A retrospective reading of all twelve pairwise validation trajectories finds their feedback minima at the covered-NLL grid maxima. The larger relative benefit of balancing at the selected endpoint accompanies larger sampled-model error, not restored calibration. We explain why feedback fitting and outcome prediction are distinct through specified acquisition and scoring laws. Exact single-row analyses retain multiplicities, ties and finite-batch normalization, separating the pairwise stationary target from winner-channel calibration and outcome identification. An identifying control links objective mismatch to an action error. Assigned multiplicity interventions shift a majority of active covered-target probabilities in the predicted direction across released IRIS and Delta-IRIS checkpoints, while normalization sharply attenuates the Breakout magnitude and both fixed-500 Delta tests fail. Channel's unchanged checkpoint has better held-out feedback loss than every fitted candidate on its recorded grid. These results show how comparison-interface choices and checkpoint selection can diverge from predictive quality, with explicit boundaries between population laws and finite neural training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
World Models, Comparison Feedback, Prediction Bias

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ii5FOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the gap between the pairwise Bradley–Terry (BT) objective and the goal of recovering the true outcome distribution, and shows that lower feedback loss need not mean better prediction or better actions.

Critical strengths

The authors link world-model learning with pairwise Bradley–Terry training. This paper is well organized.

Critical weaknesses

As I understand it, regardless of the world models, this seems to be a problem of estimating the outcome distribution when only a set containing the realized outcome is observed. Further explanations of what the methods and theory add to existing work are needed.

Questions for authors

Could the authors explain what their methods and theory add to closely related work, such as Jin and Ghahramani (2002) and Ma and Collins (2018), and how these contributions apply to world-model learning and planning?

Reviewer_oHQdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when you use comparison feedback to optimize world models. Even though the model is getting better at comparison feedback, can the world model be worse at predicting what happens?

Critical strengths

The paper gives a solid theory analysis of why comparison feedback can degrade world-model predictions. The theoretical findings are supported by well-designed experiments across multiple environments, and the use of an exact oracle effectively isolates the issues in the learning objective.

Critical weaknesses

The results for all-token NLL are mixed. For example, Sampled BT improves all-token NLL on Pong and Seaquest despite worsening covered-target NLL. The authors could further clarify how these conflicting results affect the paper's main conclusions. Additionally, one critical problem is how this finding could lead to any practical solutions and downstream validation. Balanced BT and channel likelihood cannot reliably resolve the problem.

Questions for authors

See weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer oHQd

W1. Our conclusions rest on covered targets, where the actual outcome is among the candidates. The mixed all-token results do not change them. On covered targets, the oracle names the true outcome itself. If feedback helps prediction anywhere, it should help on covered targets, yet BT feedback makes prediction worse in every world model. All-token NLL also includes targets that feedback cannot identify. On Pong and Seaquest, the original models start on data that differs from their training data, because the public recordings we use lack the intermediate frames that the models' preprocessing merges into each input frame. Any adaptation therefore improves these targets and masks the damage on covered ones. Even so, Sampled BT ends with the worst all-token NLL among the feedback objectives on both games.

W2. We agree that neither Balanced BT nor the set likelihood solves the problem, and we do not propose them as solutions. They are controls that locate the cause. Balanced BT removes duplicate counting and shows that it is a main source of the damage. The set likelihood removes the bias of BT and exposes what remains: an answer does not say how probability should be split inside the winner's cell (the set of outcomes for which the winner would win). Improving the feedback loss alone therefore need not improve this part of prediction. We have not yet found an effective method. Our findings indicate that effective comparison feedback should tell not only who wins, but also under which outcomes a candidate wins and by how much. More broadly, finding a reasonable rubric or verifier is a promising direction. We would be happy to discuss possible solutions with the reviewer.

We sincerely thank the reviewer for the review. We hope our answers resolve these concerns, and we look forward to the reviewer's reply.

Thank you for the reply. I will maintain my score. I am not an full expert that keep chasing the latest work in this field, so my assessment to the novelty may be limited.

Reviewer_uhWTOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper discusses the limitations of using pairwise comparison methods for tuning pre-trained world models. Comparison feedback methods use the Bradley-Terry loss (BTL) to optimize the probability of an outcome that wins the most head-to-head comparisons. The authors show that this has three main limitations: (1) BTL can be biased if more than one copy of an possible outcome is observed (multiplicity bias), (2) a loss minimized at the truth doesn't mean it will reliably compare checkpoints and (3) a biased law can alter the calculation of the reward value of actions, which can change the optimal action (one that maximizes the reward). The authors conduct an empirical study to demonstrate their results.

Critical strengths
  • Paper is organized pretty well
  • Takeaway blocks seems intentional (and I appreciated it)
Critical weaknesses

Significance and Novelty I see quite a few issues regarding the significance and novelty of the results in the paper.

Theorem 4.1 is not a new result. [1] shows that BTL implicitly aggregates pairwise preferences according to the Borda count voting method in social choice theory. It is well known in social choice theory that Borda count is not independent of clones (see [2]). Hence it is not surprising nor new that BTL is susceptible to "multiplicity bias."

Moreover, the other theoretical results seem like they are existence proofs with hand constructed toy examples. How do these generalize?

Most importantly, I am not sure if comparison feedback methods are prevalent in the literature. As far as I could tell [3] (RENEW) is the only one. In fact, this paper almost seems like a direct response/criticism of [3]. This was quite odd given how recent (very) and significant (yet to be seen) the work is in fine tuning world models.

Also, [3] doesn't assume that we have access to an oracle. It proposes to use human judges to compare two outputs (arguing that humans are good at identifying if a given scene is not coherent/realistic). So the setting and motivation that this work draws also seems different.

Clarity I found the paper difficult to read. A lot of vague phrases and sentences throughout the paper. Just to name a few:

  • "correctly specified feedback loss" (line 73) -> what is this?
  • "true law" (line 71) -> what is this true law?
  • "cell" (line 76) -> what is this exactly? only defined in Def 3.1
  • What does "fitting gap" exactly mean?
  • What is a "central outcome"? (line 371)

[1] Siththaranjan, Anand, Cassidy Laidlaw, and Dylan Hadfield-Menell. "Distributional preference learning: Understanding and accounting for hidden context in RLHF." In International Conference on Learning Representations, vol. 2024, pp. 27448-27471. 2024.

[2] Tideman, T. N. “Independence of Clones as a Criterion for Voting Rules.” Social Choice and Welfare 4, no. 3 (1987): 185–206. http://www.jstor.org/stable/41105866.

[3] Bhamidipaty, L. M., Kochenderfer, M., & Ramamoorthy, S. (2026). RENEW: Towards Learning World Models and Repairing Model Exploitation from Preferences. arXiv preprint arXiv:2607.14180.

Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.