- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes OVAL, a test-time alignment method that continuously updates the critic using information obtained from Lookahead search. OVAL generates multiple candidate segments, simulates their future continuations, and then uses a critic to score them. The resulting preference information is subsequently used to update the critic for later generation steps. Although OVAL does not require modifying the backbone, it does require pretraining a critic. Experiments across different backbones show that both Lookahead and critic updates contribute to performance.
Critical strengths
The idea of reusing information obtained from Lookahead search is interesting and is the most valuable aspect of this paper. The experiments cover different models and validate the effectiveness of OVAL.
Critical weaknesses
-
The core contribution and novelty lie in accumulating preference information obtained from Lookahead search and using it to update the critic. However, based on Table 2, OVAL's performance seems to come primarily from Lookahead, while the additional value brought by critic update, which is the main focus of the paper, appears relatively limited. This weakens the experimental evidence supporting the core innovation.
-
In the current experiments, the sources of preference supervision are not entirely consistent across different methods. For example, Best-of-N uses UltraRM-13B, GenARM, according to its paper, directly uses chosen–rejected preference pairs to obtain a specific RM, while OVAL's offline Critic is trained based on ArmoRM scores. However, the main evaluation ultimately uses ArmoRM for all methods. Even, the authors additionally use GLM-5.2 to evaluate preference dimensions, but there is currently no validation of the consistency between ArmoRM or GLM-5.2 and the original preference annotations in the datasets.
Questions for authors
-
My understanding is that ArmoRM represents the preferences being optimized, and using the same RM for evaluation can demonstrate the efficiency of test-time optimization, while GLM-5.2 is used for preference-dimension evaluation to avoid reward hacking issue (previous works such as ARGS and TMPC also adopt a similar approach). However, it seems somewhat strange that Best-of-N uses UltraRM-13B to select responses, and I would appreciate clarification from the authors regarding the rationale behind this choice. Could Best-of-N also use ArmoRM for maximization?
-
The reward used for optimization and the final GLM-5.2 evaluation do not seem to guarantee consistency. I wonder whether it would be necessary to use the original preference annotations from HelpSteer2 and UltraFeedback to examine the ranking reliability of these evaluators. This would help confirm that OVAL's improvements are not merely due to better alignment with ArmoRM's scoring criteria, but actually reflect improvements in response quality under the target preferences.
-
The current efficiency analysis mainly reports wall-clock latency and final output length. However, since OVAL generates a large number of candidates and lookahead during search that are ultimately discarded, providing an analysis of the total number of generated tokens, including these discarded outputs, would better support the claims regarding alignment efficiency.
-
How are the boundaries of response segments determined? The paper sets the length of each candidate segment to 32–128 tokens, but does not clearly explain how the stopping position of each segment is determined (is it possible for a segment to be truncated in the middle of a sentence?). Since OVAL needs to score these segments, segment boundaries may affect both the quality of preference comparisons and computational cost.
Thanks for the responses. The new experiments address most of my concerns. I am still curious whether the 27B gains hold under an evaluator other than ArmoRM, since the margin over TPO is quite small there.