acceptodds
Under review as a conference paper at ICLR 2027

OVAL: Online Value Adaptation from Lookahead for Test-Time Alignment

Abstract

Test-time alignment (TTA) increasingly relies on inference-time search to steer frozen language models toward desired preferences. Yet existing planning methods under-utilize the information generated by search. Lookahead rollouts expose how candidate segments differ in their downstream preference outcomes, but these comparisons are typically consumed only by the current decision and then discarded, leaving subsequent decisions to the same critic. We introduce OVAL, a novel TTA framework that turns lookahead rollouts from a transient search signal into a source of online value adaptation. For each prompt, OVAL propagates rollout-induced rankings back to candidate segments and accumulates them as prompt-specific supervision. It then employs Follow-the-Regularized-Leader (FTRL) to learn a lightweight residual correction to an offline critic, enabling the critic to progressively incorporate evidence collected along the generation trajectory. Theoretically, under informative prompt-level signals, cross-step evidence accumulation changes the decision-regret regime from linear to bounded, while OVAL’s FTRL update achieves predictive regret. Experiments across multiple LLM backbones, benchmarks, and preferences demonstrate consistent gains over strong alignment baselines, with ablations confirming the importance of prompt-specific critic adaptation and cross-step evidence retention. This work highlights online value adaptation as a general principle for TTA.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Model Predictive Control, Online Learning, Preference Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iF8vOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OVAL, a test-time alignment method that continuously updates the critic using information obtained from Lookahead search. OVAL generates multiple candidate segments, simulates their future continuations, and then uses a critic to score them. The resulting preference information is subsequently used to update the critic for later generation steps. Although OVAL does not require modifying the backbone, it does require pretraining a critic. Experiments across different backbones show that both Lookahead and critic updates contribute to performance.

Critical strengths

The idea of reusing information obtained from Lookahead search is interesting and is the most valuable aspect of this paper. The experiments cover different models and validate the effectiveness of OVAL.

Critical weaknesses
  • The core contribution and novelty lie in accumulating preference information obtained from Lookahead search and using it to update the critic. However, based on Table 2, OVAL's performance seems to come primarily from Lookahead, while the additional value brought by critic update, which is the main focus of the paper, appears relatively limited. This weakens the experimental evidence supporting the core innovation.

  • In the current experiments, the sources of preference supervision are not entirely consistent across different methods. For example, Best-of-N uses UltraRM-13B, GenARM, according to its paper, directly uses chosen–rejected preference pairs to obtain a specific RM, while OVAL's offline Critic is trained based on ArmoRM scores. However, the main evaluation ultimately uses ArmoRM for all methods. Even, the authors additionally use GLM-5.2 to evaluate preference dimensions, but there is currently no validation of the consistency between ArmoRM or GLM-5.2 and the original preference annotations in the datasets.

Questions for authors
  1. My understanding is that ArmoRM represents the preferences being optimized, and using the same RM for evaluation can demonstrate the efficiency of test-time optimization, while GLM-5.2 is used for preference-dimension evaluation to avoid reward hacking issue (previous works such as ARGS and TMPC also adopt a similar approach). However, it seems somewhat strange that Best-of-N uses UltraRM-13B to select responses, and I would appreciate clarification from the authors regarding the rationale behind this choice. Could Best-of-N also use ArmoRM for maximization?

  2. The reward used for optimization and the final GLM-5.2 evaluation do not seem to guarantee consistency. I wonder whether it would be necessary to use the original preference annotations from HelpSteer2 and UltraFeedback to examine the ranking reliability of these evaluators. This would help confirm that OVAL's improvements are not merely due to better alignment with ArmoRM's scoring criteria, but actually reflect improvements in response quality under the target preferences.

  3. The current efficiency analysis mainly reports wall-clock latency and final output length. However, since OVAL generates a large number of candidates and lookahead during search that are ultimately discarded, providing an analysis of the total number of generated tokens, including these discarded outputs, would better support the claims regarding alignment efficiency.

  4. How are the boundaries of response segments determined? The paper sets the length of each candidate segment to 32–128 tokens, but does not clearly explain how the stopping position of each segment is determined (is it possible for a segment to be truncated in the middle of a sentence?). Since OVAL needs to score these segments, segment boundaries may affect both the quality of preference comparisons and computational cost.

Reviewer_DJZwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OVAL uses comparisons from lookahead search to improve later generation decisions. It keeps the language model and the original critic fixed, and learns a small adjustment to the critic's scores for each prompt. An FTRL update combines information from earlier steps. Experiments on two models and nine preference dimensions show higher average reward than the tested baselines. The paper also analyzes when accumulating these comparisons can help.

Critical strengths
  1. OVAL reuses information from search. Comparisons collected at one step can help choose better continuations at later steps. The update is small and does not require changing the language model.
  2. The overall results are encouraging. OVAL improves average ArmoRM scores on both models and ranks first in 14 of 18 settings (Table 1). Evaluation with a separate judge also supports the method, and it runs faster than the reported TMPC and TPO implementations (Figure 2; Table 11).
Critical weaknesses
  1. The benefit of keeping information across steps needs stronger evidence. Compared with resetting the adjustment at every step, the gains are 0.34 and 0.67 reward units on the two models, and 0.04 and 0.02 judge-score units (Table 2). No uncertainty estimates are given. Figure 3 checks agreement with the same rollout labels used for updates, so it does not independently confirm better decisions. The theory also relies on a simplified model or bounds ranking loss, rather than final response quality.
  2. Differences in reward signals and compute make the comparisons hard to judge. BoN uses UltraRM, while OVAL's critic is trained with ArmoRM, which is also the main evaluator (Appendix C.3). This makes the source of the improvement less clear. Table 11 omits BoN and the fixed-critic variants, leaving the extra cost and benefit of the online update unclear.
Questions for authors
  1. How accurate are the lookahead labels under an independent evaluator, and how much new information do later steps add?
  2. Does OVAL still improve over BoN and fixed-critic search when they use the same reward signal and a similar time or token budget?
Reviewer_tdz9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes OVAL, a test-time alignment method that picks one text segment at a time. At each step, it samples candidate segments, runs short lookahead rollouts for each one, and scores them with an offline critic (a linear probe on the LLM hidden states). The rankings of the rollouts are used to learn a small residual on the critic with FTRL. The residual is kept across steps and reset for each new prompt. OVAL has the highest average ArmoRM score on Llama-3-8B and Qwen3-8B across 9 preferences.

Critical strengths
  1. A new way to use lookahead. Lookahead results are usually thrown away after one use. OVAL reuses them to correct the critic for the current prompt, without changing the LLM.

  2. Consistent gains. OVAL has the highest average score on both models (66.41 vs. 64.97 for TPO on Llama-3-8B; 68.48 vs. 65.32 on Qwen3-8B, Table 1).

Critical weaknesses
  1. The main metric is also the training signal. The critic is trained on ArmoRM scores, and the results are also ArmoRM scores, so the gains may only come from fitting ArmoRM. The LLM judge shows almost no gap (7.50 vs. 7.43, Table 2).

  2. It is slow to serve, and the compute is not matched. OVAL takes 113.8 s per response vs. 20.4 s for the base model (Table 11), and uses about 16K lookahead tokens per prompt vs. about 3.7K for BoN (N = 8).

  3. The gain from accumulation is small. OVAL is 74.51 vs. 74.17 without accumulation (Table 2). All results come from only one run, and α, γ, m are tuned on the test prompts (Table 9).

Questions for authors

1/ It would help to evaluate with a reward model that is not used in training, and to run BoN with ArmoRM as the selector. 2. How does OVAL compare with BoN at the same number of generated tokens? 3. Results over several seeds would be useful. Were α, γ and m chosen on a separate split? 4. Without accumulation, OVAL fits a residual on the current-step pairs and then picks by the segment score. Why does this beat picking by the lookahead value Û directly (74.17 vs. 73.37)? 5. In Figure 3, are the pre- and post-update accuracies measured on the same pairs used for the update? 6. GenARM on Llama-3-8B UltraFeedback (44.84 on instruction following) is below SFT (56.87). Is this a setup issue? 7. Do the gains hold on newer or larger models?

2 comments

Thanks for the responses. The new experiments address most of my concerns. I am still curious whether the 27B gains hold under an evaluator other than ArmoRM, since the margin over TPO is quite small there.

Thanks for the additional experiments. They addressed my remaining concern. I have raised my score from 2 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.