acceptodds
Under review as a conference paper at ICLR 2027

PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training

Abstract

Inference-time alignment enables large language models (LLMs) to generate outputs aligned with end-user preferences while keeping the base model fixed. Recent methods use small guidance models to modify token generation under a KL-regularized reward objective. When supervision consists of pairwise preferences, a common approach first estimates a reward model and then learns a decoding guide from its outputs. We introduce PITA, a framework that learns the guidance model directly from preference feedback. PITA uses predicted preference probabilities to reweight the base model's next-token distribution, with iterative data collection and refinement of the guide. We connect preference probabilities to the optimal tilted output distribution and establish performance bounds under linear function approximation. Our experiments cover mathematical reasoning, graph reasoning, and alignment from noisy human preferences. On reasoning tasks, PITA attains accuracy comparable to inference-time alignment methods with access to the true reward function, demonstrating the value of learning guidance directly from preferences.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
post training, inference time alignment, preference learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_izHYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PITA, an inference-time alignment method that learns a decoding guide directly from pairwise preferences while keeping the base LLM frozen. The guide estimates preference probabilities for candidate continuations and uses them to modify the model's next-token distribution, avoiding a separately trained reward model. The paper derives a connection between preference probabilities and a KL-regularized optimal policy and presents theoretical analysis under linear function approximation. Experiments on GSM8K, MATH-500, synthetic graph reasoning, and AlpacaEval evaluate its effectiveness.

Critical strengths
  1. PITA provides a direct approach to learning inference-time guidance from pairwise preferences without modifying the base LLM or fitting a separate reward model.
  2. The method achieves substantial improvements on GSM8K and synthetic graph reasoning tasks.
Critical weaknesses
  1. The methodological contribution appears incremental relative to existing value-guided decoding approaches. PITA primarily changes the supervision used to learn the guidance signal while retaining a similar inference-time guidance framework. In addition, the theoretical guarantees concern the exact preference-guided policy, whereas the implemented algorithm uses an approximation whose error is not analyzed.
  2. The empirical results do not establish convincing generalization. Although PITA achieves strong gains on GSM8K and synthetic reasoning tasks, improvements on MATH-500 and AlpacaEval are much smaller. Its performance is also sensitive to the amount of preference data, leaving uncertainty about its effectiveness across different tasks and preference distributions.
Questions for authors
  1. Beyond replacing reward supervision with preference supervision, what is the main methodological advantage of PITA over existing value-guided decoding methods? Can the authors also quantify the discrepancy between the theoretical policy in Equation 11 and the practical approximation in Equation 13?
  2. Why are the improvements substantially smaller on MATH-500 and AlpacaEval? Can the authors provide stronger evidence of generalization to more challenging tasks or different preference distributions?
  3. There are several reference inaccuracies. Auer et al. (2014) on page 10 refers to an edited conference proceedings volume rather than a specific research paper. Lightman et al. is listed as a 2023 paper at the 12th ICLR on page 11, although the conference took place in 2024. The Shah et al. (2016) reference on page 12 also incorrectly abbreviates Kannan Ramchandran as Kannan Ramch. These references should be checked and corrected.
Reviewer_XnutOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces PITA, a new method for inference-time alignment in LLMs that influences the next-token distribution of a frozen LLM through a small guidance model learned from pairwise preference labels. Exploiting the reward differences from the Bradley-Terry model, PITA trains a model that predicts the probability that a reference model continuation would be preferred over a greedy completion. These predictions are then converted into token-level “corrections” for the decoding process. Experiments on widely used benchmarks like GSM8K, as well as synthetic ones, show that PITA can improve the performance of frozen reference models.

Critical strengths
  • The problem setting is particularly interesting as PITA aims to learn inference-time alignment from pairwise preferences without an explicit reward model or a re-training of the underlying model;
  • The reported empirical performances are potentially encouraging, as on tasks like the ones in GSM8K the improvement over the frozen model is tangible;
  • The connection between the BT preference probabilities and the decoding corrections is interesting and worthy of attention.
Critical weaknesses
  • Novelty is not partiuclarly strong, as PITA follows Q# inference-time guidance whereas using pairwise preferences. I think the benefits of this “change” are not properly isolated and would provide greater novelty to PITA.
  • The main results, despite strong, seem to be confounded and to favor PITA. As for the first point, for instance, the strong GSM8K results might be confounded by very different budgets, since PITA has a larger KL value than competing approaches, i.e., is more distant from the reference distribution. This is further supported by Table 2, where at a similar KL (2.61 vs. 2.58) Q# and PITA achieve a comparable pass@1. Similar patterns hold in Table 8 in the Appendix. Eliminating such a potential confound is key to appreciate the proper performance of PITA when compared under a shared setting with other approaches. As for the second point, it seems that PITA’s preferences are inferred/derived from ground-truth correctness, and this might imply it has a better guidance than competing methods.
  • Evidence of strong performance beyond the in-domain GSM8K are very limited, as for OOD evaluations the gain over Q# is mostly absent or particularly limited. This is particularly evident for AlpacaEval, where one would expect more real-world-like noisy preference signals.
Questions for authors
  • Can you clarify the practical discrepancy between the exact policy in Eq 11 and Eq 13? Is such an approximation properly preserving rankings?
  • Can you provide some tradeoffs between performance and KL?
  • Related to my W1, can you please better highlight the concrete novelty introduced by PITA over related approaches?
Reviewer_v5spOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PITA trains a lightweight preference guide to steer a frozen language model during decoding, avoiding a separate reward-model fitting stage. The paper derives an exact preference-based policy, analyzes it under linear reward assumptions, and evaluates a practical approximation on reasoning, sentiment, and instruction-following tasks.

Critical strengths
  1. PITA improves GSM8K pass@1 from 69.10% to 77.11% while keeping the base model frozen, approaching reward-supervised Q#.
  2. The experiments include guidance-strength sweeps, data and capacity ablations, multiple evaluation seeds, and computational measurements.
Critical weaknesses
  1. More challenging reasoning benchmarks and broader human-preference evaluations would strengthen the claims.
  2. Mathematical reasoning experiments use only a Llama-3-8B backbone. Although AlpacaEval includes Qwen, the gains are modest. Evaluating the same reasoning benchmarks across additional model families would clarify whether PITA’s benefits generalize beyond the selected backbone.
Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.