- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper extends LTPO by replacing its latent update rule with an ensemble Kalman filter that moves the confidence proxy toward a specified target. A small policy selects the target increase for each question and is trained online using answer agreement along the optimization trajectory as its reward. The final answer is selected by majority vote over the answers generated along that trajectory.
Critical strengths
- Addresses a practical problem. The paper examines whether excessive optimization of a confidence proxy can harm reasoning accuracy, a meaningful concern for test-time latent optimization.
- Evaluation across multiple models and tasks. Experiments cover four LLMs and five mathematical reasoning benchmarks. GoATT-R achieves the highest average accuracy among the evaluated methods for each backbone.
Critical weaknesses
-
Agreement is not a reliable measure of correctness. The policy uses answer agreement to select the optimization pressure, but the paper does not establish that higher agreement indicates better answers. Nearby states along the same optimization trajectory can repeatedly produce the same wrong answer, so this reward may encourage preserving existing errors.
-
The need for problem-specific optimization limits is insufficiently supported. Section 3.1 assumes that each problem has a critical optimization threshold. Figure 3 only shows dataset-level accuracy as the target pressure changes, without reporting the achieved proxy reward. This does not establish the proposed relationship between confidence and correctness or show why different problems require different limits.
-
The policy update has an unclear justification. Equation 11 pushes the policy away from pressure values associated with below-average agreement, which Section 3.4 interprets as overoptimization. However, low agreement could also reflect a difficult question, insufficient optimization, or a transition toward a correct answer. The update does not distinguish these cases or identify whether the pressure should increase or decrease.
-
The source of the improvements remains unclear. Compared with LTPO-MV, GoATT-R changes the optimizer, the optimization target, and the pressure-selection policy together. Their contributions are not sufficiently separated, so the results do not establish whether the gains come from controlling overoptimization or from the other changes to the optimization procedure.
-
Substantial reference errors and policy concerns. On page 10, the references to Hao et al. (2024), lines 500–502, and Li et al. (2025), lines 514–517, contain ten incorrect author names. Multiple authors are replaced with entirely different names, going well beyond minor typos and strongly suggesting hallucinated bibliographic information. These errors raise a potential violation of the ICLR 2027 AI Policy for Authors, which holds authors responsible for their submissions and explicitly treats substantial LLM-generated falsehoods or misrepresentations as Code of Ethics violations.
Questions for authors
See the weaknesses above.
7 comments
3 - We believe there may be a misunderstanding regarding the role of the optimization pressure . The policy does not use to directly maximize agreement. Instead, sets the target of the confidence-based proxy reward (Eq.9), which the EnKF uses to steer the latent embeddings. The policy observes the resulting majority-vote agreement across the decoded answers given an , and adjusts accordingly to maximize the agreement.
Hence, Section 3.4 does not interpret moving away from low agreement as overoptimization. Rather, it identifies overoptimization as moving away from the initial confidence reward through excessively large optimization pressure. The agreement signal serves as a reward to detect when pressure has pushed the model too far from its initial operating point, not as a direct measure of overoptimization itself. Crucially, high agreement does not correspond to high optimization pressure. Excessive drives the EnKF to overoptimize the embeddings, producing inconsistent answers across the trajectory and therefore low agreement, as illustrated by the decreasing mean agreement at high in Figure8 (Appendix F). Conversely, moderate values keep the embeddings in a region where the model produces high agreement. The policy is thus incentivized toward the moderate pressures that maximize agreement, not toward the extreme pressures that maximize the confidence-based proxy reward but reduce agreement. This is also empirically observed in Figure 3.
Regarding the concern that low agreement could reflect difficult questions or insufficient optimization rather than excessive pressure, we agree that these cases exist. However, as we demonstrate in Figures 6 and 7, agreement is correlated with accuracy, so maximizing agreement is beneficial in all of these cases. Furthermore, as we show in Appendix F, cases with low agreement and high accuracy are outliers. Therefore, even in cases where low agreement reflects question difficulty or insufficient optimization rather than overoptimization, steering the policy toward higher agreement still leads to higher accuracy. This is distinct from cases with high agreement and low accuracy, which we did not observe in our experiments. Since maximizing agreement is correlated with higher accuracy, the policy update does not need to distinguish or identify these individual cases to be effective.
4 - We wish to clarify that LTPO-MV does not have an optimization target or a pressure-selection policy. It simply maximizes the confidence-based reward indefinitely without any bound. This is precisely the motivation for introducing the EnKF optimizer, which can drive the proxy reward toward a prescribed target rather than maximizing it without limit.
Regarding the separation of contributions, we note that Figure 3 already isolates the individual components. First, when , this is nearly equivalent to LTPO-MV since it maximizes the reward with very large pressure, compared to the indefinite pressure in LTPO. The clear performance degradation at in Figure 3 isolates the benefit of having a bounded optimization target rather than unbounded maximization. Second, Figure 3 demonstrates that there exists a beneficial optimization region around moderate values. A randomly selected is unlikely to land in this narrow region, as we discuss in our response to second weakness. The learned policy consistently drives into this beneficial region, which isolates the contribution of the pressure-selection policy that maximizes accuracy per model-dataset pair. Furthermore, as mentioned in our response to other reviewers, we can run additional experiments with a fixed random threshold across all models and datasets to further isolate the contribution of the learned policy from a fixed random target, since at inference we do not have access to ground-truth data. We are happy to include these results if the reviewer considers them necessary.
5 - We sincerely thank the reviewer for catching these errors and we deeply apologize for this mistake. We have corrected all author names for Hao et al (2024) and Li et al (2025) in the revised manuscript.
Regarding the ICLR 2027 AI Policy, our AI usage statement (Lines 474--481) explicitly states that generative AI tools were used for "text editing'' among other auxiliary tasks, and that "all AI-assisted work was reviewed and verified by the authors.'' These incorrect author names likely originated during the text editing process, where we initially used AI tools to look up references and reference citations before later verifying them individually. Unfortunately, the author names in these two references were not caught during our verification pass. This was a human oversight in our verification process, not an intentional misrepresentation. We acknowledge that reference lookup is not strictly the same as text editing, however no reference search option is present in the (https://iclr.cc/Conferences/2027/AIPolicyForAuthors)(ICLR 2027 AI Policy for Authors). For further clarity, we have updated our AI usage statement in the revised manuscript to explicitly reflect this use case.
Finally, we hope we have addressed the reviewer’s concerns. We welcome any further feedback or suggestions that would help strengthen the paper, and we are happy to run additional experiments during the discussion period if further required by the reviewer.
References :
[1] Zuo, Yuxin, et al. "Ttrl: Test-time reinforcement learning." Advances in Neural Information Processing Systems 38 (2026): 131459-131483.
[2] Zhang, Kongcheng, et al. "Consistent paths lead to truth: Self-rewarding reinforcement learning for llm reasoning." Advances in Neural Information Processing Systems 38 (2026): 59849-59887.
[3] Zhang, Qingyang, et al. "Right question is already half the answer: Fully unsupervised llm reasoning incentivization." Advances in neural information processing systems 38 (2026): 67345-67372.
We sincerely thank the reviewer for their careful reading of our paper and appreciate the time and effort invested in the review. We address each concern below and provide additional supporting evidence with detailed analysis that we invite the reviewer to read in Appendix F:
Weaknesses: 1 - The reviewer argues that incorrect answers can remain consistent across nearby latent states, receiving high agreement scores. While this is possible in principle, we provide empirical evidence in Appendix F that agreement is in fact a well-calibrated measure for correctness. Across all 12 model-dataset combinations, accuracy increases monotonically with agreement (Figures 6, 7). Importantly, none of these experiments show an instance where incorrect answers consistently receive high agreement, which contradicts the hypothesis that agreement rewards incorrect but consistent outputs. Furthermore, we show in Figure 8 that the mean agreement rˉ tracks accuracy across optimization pressures, peaking at ϵ=0.2 then decreasing, following the same pattern as the Goodhart curve in Figure 3.
Moreover, using majority-vote agreement as a label-free reward signal is not a novel assumption of our work. It is the established approach across multiple published and concurrent methods. TTRL(Zuo et al., 2025)[1](lines 434-437 in manuscript) uses the majority-vote reward to fine-tune model weights on unlabeled test data. CoVo (Zhang et al., 2025a)[2] extends TTRL and uses consistency across reasoning trajectories instead of final answers as a reward signal to fine-tune model weights. EMPO [3] uses entropy-based signals as intrinsic rewards for weight-level RL, and, according to the authors, "optimizes the model to favor reasoning traces yielding consistent answers, enhancing output reliability."
This means that the reviewer's concern, if taken to its conclusion, would equally invalidate all of these works. Crucially, our use of agreement is lighter than theirs, where we use it only to select optimization pressure on input embeddings, while the above methods use it to update model weights.
Furthermore, while we do not argue that majority voting is a substitute for actual correctness, however in the absence of access to ground truth data at inference, it is a substantially better measure of correctness than the confidence-based reward used by LTPO, which, as we demonstrate empirically, rapidly degrades accuracy when maximized without bound.
2 - We thank the reviewer for raising this point. We wish to clarify that Section 3.1 establishes, as a theoretical motivation, that optimization pressure depends on the prompt and problem, which are dependent on the dataset. It does not assume the method learns a per-problem optimal pressure. Rather, this theoretical framing motivates learning a suitable dataset-level pressure from majority-voting reward without access to ground-truth labels. In the current work, we demonstrate dataset-dependent pressure selection through the agreement-based policy. We also note that our stated contributions (Lines 70--82) do not claim the ability to find per-problem optimal pressure, but rather that the policy learns "a suitable optimization pressure from label-free reward signals.'' We further revised Section 3.1 (Lines 160--161) in the updated manuscript to clarify that the policy learns a suitable dataset-level optimization pressure rather than per-problem ϵ∗(s).
Furthermore, we collected per-question correctness across all 16 ϵ values and found that, for example on GSM8K with Llama-3.2-3B (the same setting as Figure 3), 738 out of 1319 questions were answered correctly regardless of the ϵ value. This demonstrates that verifying a per-problem optimal ϵ is inherently challenging, as a big chunk of questions are insensitive to the threshold, and a dataset-level average ϵ value is a more practical and meaningful selection criterion. Additionally, the policy starts from a random initialization and has a limited number of optimization steps to converge, which limits its ability to learn per-problem optimal thresholds. However, as our experiments demonstrate, the policy is effective at learning ϵ values that remain within beneficial optimization regions.
Regarding achieved confidence reward reporting, we plot the relationship between mean achieved confidence reward and accuracy in Figure9 (Appendix F.4), which shows the same Goodhart effect where correctness peaks at a moderate confidence reward and degrades under higher optimization.
Finally, as discussed in our response to first weakness, we present additional experiments establishing the proposed relationship between confidence and correctness. Figures 6, 7, and combining Figure 8 with Figure 3 confirms that the policy drives the threshold toward regions of both high agreement and high accuracy, providing a principled reward for pressure selection without ground-truth data.