Significance and Novelty
I see quite a few issues regarding the significance and novelty of the results in the paper.
Theorem 4.1 is not a new result. [1] shows that BTL implicitly aggregates pairwise preferences according to the Borda count voting method in social choice theory. It is well known in social choice theory that Borda count is not independent of clones (see [2]). Hence it is not surprising nor new that BTL is susceptible to "multiplicity bias."
Moreover, the other theoretical results seem like they are existence proofs with hand constructed toy examples. How do these generalize?
Most importantly, I am not sure if comparison feedback methods are prevalent in the literature. As far as I could tell [3] (RENEW) is the only one. In fact, this paper almost seems like a direct response/criticism of [3]. This was quite odd given how recent (very) and significant (yet to be seen) the work is in fine tuning world models.
Also, [3] doesn't assume that we have access to an oracle. It proposes to use human judges to compare two outputs (arguing that humans are good at identifying if a given scene is not coherent/realistic). So the setting and motivation that this work draws also seems different.
Clarity
I found the paper difficult to read. A lot of vague phrases and sentences throughout the paper. Just to name a few:
- "correctly specified feedback loss" (line 73) -> what is this?
- "true law" (line 71) -> what is this true law?
- "cell" (line 76) -> what is this exactly? only defined in Def 3.1
- What does "fitting gap" exactly mean?
- What is a "central outcome"? (line 371)
[1] Siththaranjan, Anand, Cassidy Laidlaw, and Dylan Hadfield-Menell. "Distributional preference learning: Understanding and accounting for hidden context in RLHF." In International Conference on Learning Representations, vol. 2024, pp. 27448-27471. 2024.
[2] Tideman, T. N. “Independence of Clones as a Criterion for Voting Rules.” Social Choice and Welfare 4, no. 3 (1987): 185–206. http://www.jstor.org/stable/41105866.
[3] Bhamidipaty, L. M., Kochenderfer, M., & Ramamoorthy, S. (2026). RENEW: Towards Learning World Models and Repairing Model Exploitation from Preferences. arXiv preprint arXiv:2607.14180.
Response to Reviewer oHQd
W1. Our conclusions rest on covered targets, where the actual outcome is among the candidates. The mixed all-token results do not change them. On covered targets, the oracle names the true outcome itself. If feedback helps prediction anywhere, it should help on covered targets, yet BT feedback makes prediction worse in every world model. All-token NLL also includes targets that feedback cannot identify. On Pong and Seaquest, the original models start on data that differs from their training data, because the public recordings we use lack the intermediate frames that the models' preprocessing merges into each input frame. Any adaptation therefore improves these targets and masks the damage on covered ones. Even so, Sampled BT ends with the worst all-token NLL among the feedback objectives on both games.
W2. We agree that neither Balanced BT nor the set likelihood solves the problem, and we do not propose them as solutions. They are controls that locate the cause. Balanced BT removes duplicate counting and shows that it is a main source of the damage. The set likelihood removes the bias of BT and exposes what remains: an answer does not say how probability should be split inside the winner's cell (the set of outcomes for which the winner would win). Improving the feedback loss alone therefore need not improve this part of prediction. We have not yet found an effective method. Our findings indicate that effective comparison feedback should tell not only who wins, but also under which outcomes a candidate wins and by how much. More broadly, finding a reasonable rubric or verifier is a promising direction. We would be happy to discuss possible solutions with the reviewer.
We sincerely thank the reviewer for the review. We hope our answers resolve these concerns, and we look forward to the reviewer's reply.