- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PITA, an inference-time alignment method that learns a decoding guide directly from pairwise preferences while keeping the base LLM frozen. The guide estimates preference probabilities for candidate continuations and uses them to modify the model's next-token distribution, avoiding a separately trained reward model. The paper derives a connection between preference probabilities and a KL-regularized optimal policy and presents theoretical analysis under linear function approximation. Experiments on GSM8K, MATH-500, synthetic graph reasoning, and AlpacaEval evaluate its effectiveness.
Critical strengths
- PITA provides a direct approach to learning inference-time guidance from pairwise preferences without modifying the base LLM or fitting a separate reward model.
- The method achieves substantial improvements on GSM8K and synthetic graph reasoning tasks.
Critical weaknesses
- The methodological contribution appears incremental relative to existing value-guided decoding approaches. PITA primarily changes the supervision used to learn the guidance signal while retaining a similar inference-time guidance framework. In addition, the theoretical guarantees concern the exact preference-guided policy, whereas the implemented algorithm uses an approximation whose error is not analyzed.
- The empirical results do not establish convincing generalization. Although PITA achieves strong gains on GSM8K and synthetic reasoning tasks, improvements on MATH-500 and AlpacaEval are much smaller. Its performance is also sensitive to the amount of preference data, leaving uncertainty about its effectiveness across different tasks and preference distributions.
Questions for authors
- Beyond replacing reward supervision with preference supervision, what is the main methodological advantage of PITA over existing value-guided decoding methods? Can the authors also quantify the discrepancy between the theoretical policy in Equation 11 and the practical approximation in Equation 13?
- Why are the improvements substantially smaller on MATH-500 and AlpacaEval? Can the authors provide stronger evidence of generalization to more challenging tasks or different preference distributions?
- There are several reference inaccuracies. Auer et al. (2014) on page 10 refers to an edited conference proceedings volume rather than a specific research paper. Lightman et al. is listed as a 2023 paper at the 12th ICLR on page 11, although the conference took place in 2024. The Shah et al. (2016) reference on page 12 also incorrectly abbreviates Kannan Ramchandran as Kannan Ramch. These references should be checked and corrected.