acceptodds
Under review as a conference paper at ICLR 2027

CO-OIPO: Coverage-Optimized Online IPO for Process Rewards under Intransitive Preferences

Abstract

Process reward models (PRMs) provide signals for evaluating and guiding intermediate steps in multi-step reasoning. Existing PRMs typically adopt a Bradley–Terry (BT) model trained on preference pairs, with the training objective of distinguishing the better from the worse within a pair of samples. However, in multi-hop knowledge graph question answering, multiple paths may correspond to the correct answer, leaving some intermediate steps without a clear distinction between better and worse. Furthermore, multiple preference pairs may form long-chain preference relations, thereby causing a sample to appear in multiple positions within the preference relations, which gives rise to contradictions. To address the above problem, we propose formulating preference learning as a game over reasoning candidates, with a regularized Nash equilibrium as the learning objective. This formulation accommodates a broader class of pairwise preference relations, allowing two samples without clear preference to receive similar exploration probabilities. Furthermore, to make more effective use of a limited comparison budget, we propose Coverage-Optimized Online Identity Preference Optimization (CO-OIPO), which optimizes pair allocation within a sampled candidate pool. We prove that, with sufficient sampling coverage, an expressive enough policy model, and full optimization in each round, CO-OIPO converges at a fixed point corresponding to a regularized Nash equilibrium of the preference game.Experiments show that CO-OIPO achieves the highest performance on most metrics; the controlled experiments further demonstrate more effective utilization of samples without clear preference, and correctness of theoretical analysis.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Implicit Process Reward Models, Online Preference Optimization, Regularized Nash Equilibrium, Multi-Hop Knowledge Graph Question Answering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_j8H7OpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes CO-OIPO, an implicit process-reward framework for multi-hop knowledge-graph question answering that replaces Bradley–Terry preference modeling with a preference-game objective to accommodate potentially intransitive preferences over reasoning trajectories. It generates candidate pools from a current-reference mixture, allocates a limited pairwise-comparison budget toward undercovered pairs, and applies importance corrections for both candidate generation and pair selection. The paper provides theoretical analysis of finite-budget target preservation, Nash correspondence, and coverage-optimal pair allocation, and reports experiments on WebQSP and Complex WebQuestions along with controlled preference-game and fixed-pool analyses.

Critical strengths
  1. Novelty, clear problem framing and targeted modeling: The paper identifies a specific limitation of existing preference modeling and motivates an alternative formulation. The modeling choice follows directly from that limitation. The motivation and formulation are internally consistent. The idea is really interesting.
  2. Complete method pipeline: The method integrates candidate generation, comparison selection, sampling correction, and policy update into one procedure. It addresses the full chain. The design is therefore more complete than a single-component modification.
  3. Rich theoretical analysis: The paper provides analysis for key components of the method. And the theoretical parts are not purely heuristic.
  4. Experimental coverage: The evaluation covers end-to-end performance, component ablations, controlled preference settings, and fixed-pool diagnostics.
Critical weaknesses
  1. Limited dataset scope: The evaluation uses only 2 of multi-hop QA datasets. The paper does not clearly explain why these datasets were chosen as the primary evaluation settings. What motivated the choice of these datasets?

  2. High online collection and hyperparameter cost: The method requires iterative online candidate generation and repeated preference queries. It also involves multiple sampling and optimization hyperparameters, which increases engineering and computational cost. The paper does not provide a systematic cost comparison under matched compute budgets.

Questions for authors

Please see Weaknesses.

Reviewer_FHQJOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposed CO-OIPO, which formulates preference learning as a game over reasoning. The designed formulation helps to distinguish unclear comparison pairs and enables reward signals. Compared with multiple baselines, the evaluation has been done on the WebQSP and CWQ datasets.

Critical strengths
  1. The formulation is definitely novel and interesting, which enables a lot of potential in PRMs.
  2. The linkage to the knowledge graph is natural. However, I did not find it completely convincing that it is necessary (please see weaknesses).
Critical weaknesses
  1. I am not convinced why such a game should be motivated by a knowledge graph, since the limitations mentioned by the authors are general to multi-hop question-answering. Also, the pain point raised by the authors--unclear preference--could be even universal and not limited to multi-hop QA.

  2. In the abstract, the authors claimed that the proposed method can enable "two samples without clear preference to receive similar exploration probabilities". However, such a claim has not been mentioned or justified in the later content. In addition to this limitation, I was also wondering about the implications of them receiving similar exploration probabilities and how that could potentially help PRMs.

  3. The main experiments are limited to a single backbone model Qwen3-8B, and on two datasets, which can be insufficient to validate the proposed method.

  4. The performance gain of the proposed method compared to KG-enhanced methods is marginal. Since the proposed method also builds upon knowledge graph-based techniques, it might be suspicious regarding the attribution of the performance gain. A more rigorous and clearer ablation study could be useful to justify this.

Questions for authors

Please refer to the weaknesses.

1 comment

Thanks for the responses. I will raise my soundness score based on the additional experimental results. I still have doubts about the technique itself. Evaluating a generic methodology on a specific task and one specific instantiation cannot capture the randomness or other effects of the claimed methodology. Also, simply evaluating on one task, one cannot actually understand the real merit and scientific contribution of the claimed methodology.

Reviewer_21iNOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses common problem in multi-hop question answering where the lack of consistency of preference relation (non-transitivity) makes standard Direct Preference Optimization (DPO) and Bradley–Terry (BT) model fail, and suggested CO-OIPO (Coverage-Optimized Online Identity Preference Optimization) that learns an implicit process reward from pairwise preferences over complete reasoning trajectories, enabling handling of intransitive preferences over reasoning trajectories and improving finite-budget preference-sampling efficiency through coverage-optimized pair selection. CO-OIPO is theoretically grounded in game theory where preference game objective is used in place of BT-based preference modeling. Through Through its fixed point under exact online optimization corresponding to a KL-regularized Nash equilibrium, CO-OIPO is shown to learn from preferences beyond the BT model, while consuming a reduced preference budget. The model has been validated on WebQSP and Complex WebQuestions datasets as well as on as well as controlled cyclic game environments, with testing through controlled preference-game experiments and fixed-pool analyses to examine equilibrium learning behavior, pair selection coverage, and objective-estimation error. The model exhibited higher perforrnance over offline DPO on synthetic cyclic preference games, and top hit and F1 score on WebQSP and Complex WebQuestions datasets in overall.

The paper is well structured with strong theoretical development and ablation analysis.

Critical strengths

Very strong mathematical development that grounds the proposal CO-OIPO on game-theory perspective, for instance, demonstrating that the expected corrected estimator and the round-wise population objective have the same context-wise minimizers, providing direct relationship between the finite-budget estimator and the population objective.

Critical weaknesses

The model relies heavily on external LLM-as-a judge paradigm -DeepSeek-V4-Flash- to provide real-time pairwise preferences, this is subject to possible errors, hallucination and bias.

The model incorporates several multiple interacting parameters controlling regularization, bounds, mixing (see appendix section). The study did not include any sensitivity analysis or another approach to back up the choice of the hyperparameter values.

Several abbreviations were not defined in the document, e.g., DPRM, IPO, IPO-MD, WPO, ...

Questions for authors

How to guarantee that the outcomes of LLM-as-judge for real-time pairwise preference is reliable enough and does not include any noise or hallucinated date? A small scale experiment with human annotation can provide answers to this.

How to motivate the choice of specific values for the multiple interacting parameters, e.g., alpha, Lambda, gamma, K, B ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.