- Rating
- 3
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes CO-OIPO, an implicit process-reward framework for multi-hop knowledge-graph question answering that replaces Bradley–Terry preference modeling with a preference-game objective to accommodate potentially intransitive preferences over reasoning trajectories. It generates candidate pools from a current-reference mixture, allocates a limited pairwise-comparison budget toward undercovered pairs, and applies importance corrections for both candidate generation and pair selection. The paper provides theoretical analysis of finite-budget target preservation, Nash correspondence, and coverage-optimal pair allocation, and reports experiments on WebQSP and Complex WebQuestions along with controlled preference-game and fixed-pool analyses.
Critical strengths
- Novelty, clear problem framing and targeted modeling: The paper identifies a specific limitation of existing preference modeling and motivates an alternative formulation. The modeling choice follows directly from that limitation. The motivation and formulation are internally consistent. The idea is really interesting.
- Complete method pipeline: The method integrates candidate generation, comparison selection, sampling correction, and policy update into one procedure. It addresses the full chain. The design is therefore more complete than a single-component modification.
- Rich theoretical analysis: The paper provides analysis for key components of the method. And the theoretical parts are not purely heuristic.
- Experimental coverage: The evaluation covers end-to-end performance, component ablations, controlled preference settings, and fixed-pool diagnostics.
Critical weaknesses
-
Limited dataset scope: The evaluation uses only 2 of multi-hop QA datasets. The paper does not clearly explain why these datasets were chosen as the primary evaluation settings. What motivated the choice of these datasets?
-
High online collection and hyperparameter cost: The method requires iterative online candidate generation and repeated preference queries. It also involves multiple sampling and optimization hyperparameters, which increases engineering and computational cost. The paper does not provide a systematic cost comparison under matched compute budgets.
Questions for authors
Please see Weaknesses.
Thanks for the responses. I will raise my soundness score based on the additional experimental results. I still have doubts about the technique itself. Evaluating a generic methodology on a specific task and one specific instantiation cannot capture the randomness or other effects of the claimed methodology. Also, simply evaluating on one task, one cannot actually understand the real merit and scientific contribution of the claimed methodology.