acceptodds
Under review as a conference paper at ICLR 2027

Goodhart-Aware Test-Time Reasoning

Abstract

Recent work has shown that injecting learnable latent tokens into frozen large language models (LLMs) and optimizing them at test time by maximizing a confidence-based proxy reward improves reasoning performance. However, we show that optimizing such rewards without bound can lead to Goodhart overoptimization, where accuracy peaks at a problem-dependent threshold and degrades beyond it. We propose GoATT-R (GoodhartAware Test-Time Reasoning), which treats the frozen LLM as an uncertain system and steers its output using an Ensemble Kalman Filter (EnKF) that tracks a prescribed proxy target reward rather than maximizing the proxy indefinitely. To set this target adaptively, GoATT-R employs a lightweight online policy network, trained via SGD during evaluation, that predicts a suitable optimization target from prompt features and initial ensemble statistics using majority-vote agreement as a label-free reward signal. GoATT-R consistently outperforms state of the art baselines across multiple language models and reasoning benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reasoning, reinforcement learning.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DCQzOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper extends LTPO by replacing its latent update rule with an ensemble Kalman filter that moves the confidence proxy toward a specified target. A small policy selects the target increase for each question and is trained online using answer agreement along the optimization trajectory as its reward. The final answer is selected by majority vote over the answers generated along that trajectory.

Critical strengths
  • Addresses a practical problem. The paper examines whether excessive optimization of a confidence proxy can harm reasoning accuracy, a meaningful concern for test-time latent optimization.
  • Evaluation across multiple models and tasks. Experiments cover four LLMs and five mathematical reasoning benchmarks. GoATT-R achieves the highest average accuracy among the evaluated methods for each backbone.
Critical weaknesses
  1. Agreement is not a reliable measure of correctness. The policy uses answer agreement to select the optimization pressure, but the paper does not establish that higher agreement indicates better answers. Nearby states along the same optimization trajectory can repeatedly produce the same wrong answer, so this reward may encourage preserving existing errors.

  2. The need for problem-specific optimization limits is insufficiently supported. Section 3.1 assumes that each problem has a critical optimization threshold. Figure 3 only shows dataset-level accuracy as the target pressure changes, without reporting the achieved proxy reward. This does not establish the proposed relationship between confidence and correctness or show why different problems require different limits.

  3. The policy update has an unclear justification. Equation 11 pushes the policy away from pressure values associated with below-average agreement, which Section 3.4 interprets as overoptimization. However, low agreement could also reflect a difficult question, insufficient optimization, or a transition toward a correct answer. The update does not distinguish these cases or identify whether the pressure should increase or decrease.

  4. The source of the improvements remains unclear. Compared with LTPO-MV, GoATT-R changes the optimizer, the optimization target, and the pressure-selection policy together. Their contributions are not sufficiently separated, so the results do not establish whether the gains come from controlling overoptimization or from the other changes to the optimization procedure.

  5. Substantial reference errors and policy concerns. On page 10, the references to Hao et al. (2024), lines 500–502, and Li et al. (2025), lines 514–517, contain ten incorrect author names. Multiple authors are replaced with entirely different names, going well beyond minor typos and strongly suggesting hallucinated bibliographic information. These errors raise a potential violation of the ICLR 2027 AI Policy for Authors, which holds authors responsible for their submissions and explicitly treats substantial LLM-generated falsehoods or misrepresentations as Code of Ethics violations.

Questions for authors

See the weaknesses above.

7 comments

We sincerely thank the reviewer for their careful reading of our paper and appreciate the time and effort invested in the review. We address each concern below and provide additional supporting evidence with detailed analysis that we invite the reviewer to read in Appendix F:

Weaknesses: 1 - The reviewer argues that incorrect answers can remain consistent across nearby latent states, receiving high agreement scores. While this is possible in principle, we provide empirical evidence in Appendix F that agreement is in fact a well-calibrated measure for correctness. Across all 12 model-dataset combinations, accuracy increases monotonically with agreement (Figures 6, 7). Importantly, none of these experiments show an instance where incorrect answers consistently receive high agreement, which contradicts the hypothesis that agreement rewards incorrect but consistent outputs. Furthermore, we show in Figure 8 that the mean agreement rˉ\bar{r} tracks accuracy across optimization pressures, peaking at ϵ=0.2\epsilon=0.2 then decreasing, following the same pattern as the Goodhart curve in Figure 3.

Moreover, using majority-vote agreement as a label-free reward signal is not a novel assumption of our work. It is the established approach across multiple published and concurrent methods. TTRL(Zuo et al., 2025)[1](lines 434-437 in manuscript) uses the majority-vote reward to fine-tune model weights on unlabeled test data. CoVo (Zhang et al., 2025a)[2] extends TTRL and uses consistency across reasoning trajectories instead of final answers as a reward signal to fine-tune model weights. EMPO [3] uses entropy-based signals as intrinsic rewards for weight-level RL, and, according to the authors, "optimizes the model to favor reasoning traces yielding consistent answers, enhancing output reliability."

This means that the reviewer's concern, if taken to its conclusion, would equally invalidate all of these works. Crucially, our use of agreement is lighter than theirs, where we use it only to select optimization pressure on input embeddings, while the above methods use it to update model weights.

Furthermore, while we do not argue that majority voting is a substitute for actual correctness, however in the absence of access to ground truth data at inference, it is a substantially better measure of correctness than the confidence-based reward used by LTPO, which, as we demonstrate empirically, rapidly degrades accuracy when maximized without bound.

2 - We thank the reviewer for raising this point. We wish to clarify that Section 3.1 establishes, as a theoretical motivation, that optimization pressure depends on the prompt and problem, which are dependent on the dataset. It does not assume the method learns a per-problem optimal pressure. Rather, this theoretical framing motivates learning a suitable dataset-level pressure from majority-voting reward without access to ground-truth labels. In the current work, we demonstrate dataset-dependent pressure selection through the agreement-based policy. We also note that our stated contributions (Lines 70--82) do not claim the ability to find per-problem optimal pressure, but rather that the policy learns "a suitable optimization pressure from label-free reward signals.'' We further revised Section 3.1 (Lines 160--161) in the updated manuscript to clarify that the policy learns a suitable dataset-level optimization pressure rather than per-problem ϵ∗(s)\epsilon^*(s).

Furthermore, we collected per-question correctness across all 16 ϵ\epsilon values and found that, for example on GSM8K with Llama-3.2-3B (the same setting as Figure 3), 738 out of 1319 questions were answered correctly regardless of the ϵ\epsilon value. This demonstrates that verifying a per-problem optimal ϵ\epsilon is inherently challenging, as a big chunk of questions are insensitive to the threshold, and a dataset-level average ϵ\epsilon value is a more practical and meaningful selection criterion. Additionally, the policy starts from a random initialization and has a limited number of optimization steps to converge, which limits its ability to learn per-problem optimal thresholds. However, as our experiments demonstrate, the policy is effective at learning ϵ\epsilon values that remain within beneficial optimization regions.

Regarding achieved confidence reward reporting, we plot the relationship between mean achieved confidence reward and accuracy in Figure9 (Appendix F.4), which shows the same Goodhart effect where correctness peaks at a moderate confidence reward and degrades under higher optimization.

Finally, as discussed in our response to first weakness, we present additional experiments establishing the proposed relationship between confidence and correctness. Figures 6, 7, and combining Figure 8 with Figure 3 confirms that the policy drives the threshold toward regions of both high agreement and high accuracy, providing a principled reward for pressure selection without ground-truth data.

3 - We believe there may be a misunderstanding regarding the role of the optimization pressure ε\varepsilon. The policy does not use ε\varepsilon to directly maximize agreement. Instead, ε\varepsilon sets the target R~=R0+ε\tilde{R} = R_0 + \varepsilon of the confidence-based proxy reward (Eq.9), which the EnKF uses to steer the latent embeddings. The policy observes the resulting majority-vote agreement rnr_n across the decoded answers given an ε\varepsilon, and adjusts ε\varepsilon accordingly to maximize the agreement.

Hence, Section 3.4 does not interpret moving away from low agreement as overoptimization. Rather, it identifies overoptimization as moving away from the initial confidence reward R0R_0 through excessively large optimization pressure. The agreement signal serves as a reward to detect when pressure has pushed the model too far from its initial operating point, not as a direct measure of overoptimization itself. Crucially, high agreement does not correspond to high optimization pressure. Excessive ε\varepsilon drives the EnKF to overoptimize the embeddings, producing inconsistent answers across the trajectory and therefore low agreement, as illustrated by the decreasing mean agreement at high ε\varepsilon in Figure8 (Appendix F). Conversely, moderate ε\varepsilon values keep the embeddings in a region where the model produces high agreement. The policy is thus incentivized toward the moderate pressures that maximize agreement, not toward the extreme pressures that maximize the confidence-based proxy reward but reduce agreement. This is also empirically observed in Figure 3.

Regarding the concern that low agreement could reflect difficult questions or insufficient optimization rather than excessive pressure, we agree that these cases exist. However, as we demonstrate in Figures 6 and 7, agreement is correlated with accuracy, so maximizing agreement is beneficial in all of these cases. Furthermore, as we show in Appendix F, cases with low agreement and high accuracy are outliers. Therefore, even in cases where low agreement reflects question difficulty or insufficient optimization rather than overoptimization, steering the policy toward higher agreement still leads to higher accuracy. This is distinct from cases with high agreement and low accuracy, which we did not observe in our experiments. Since maximizing agreement is correlated with higher accuracy, the policy update does not need to distinguish or identify these individual cases to be effective.

4 - We wish to clarify that LTPO-MV does not have an optimization target or a pressure-selection policy. It simply maximizes the confidence-based reward indefinitely without any bound. This is precisely the motivation for introducing the EnKF optimizer, which can drive the proxy reward toward a prescribed target rather than maximizing it without limit.

Regarding the separation of contributions, we note that Figure 3 already isolates the individual components. First, when ε=1.0\varepsilon=1.0, this is nearly equivalent to LTPO-MV since it maximizes the reward with very large pressure, compared to the indefinite pressure in LTPO. The clear performance degradation at ε=1.0\varepsilon=1.0 in Figure 3 isolates the benefit of having a bounded optimization target rather than unbounded maximization. Second, Figure 3 demonstrates that there exists a beneficial optimization region around moderate ε\varepsilon values. A randomly selected ε\varepsilon is unlikely to land in this narrow region, as we discuss in our response to second weakness. The learned policy consistently drives ε\varepsilon into this beneficial region, which isolates the contribution of the pressure-selection policy that maximizes accuracy per model-dataset pair. Furthermore, as mentioned in our response to other reviewers, we can run additional experiments with a fixed random threshold across all models and datasets to further isolate the contribution of the learned policy from a fixed random target, since at inference we do not have access to ground-truth data. We are happy to include these results if the reviewer considers them necessary.

5 - We sincerely thank the reviewer for catching these errors and we deeply apologize for this mistake. We have corrected all author names for Hao et al (2024) and Li et al (2025) in the revised manuscript.

Regarding the ICLR 2027 AI Policy, our AI usage statement (Lines 474--481) explicitly states that generative AI tools were used for "text editing'' among other auxiliary tasks, and that "all AI-assisted work was reviewed and verified by the authors.'' These incorrect author names likely originated during the text editing process, where we initially used AI tools to look up references and reference citations before later verifying them individually. Unfortunately, the author names in these two references were not caught during our verification pass. This was a human oversight in our verification process, not an intentional misrepresentation. We acknowledge that reference lookup is not strictly the same as text editing, however no reference search option is present in the (https://iclr.cc/Conferences/2027/AIPolicyForAuthors)(ICLR 2027 AI Policy for Authors). For further clarity, we have updated our AI usage statement in the revised manuscript to explicitly reflect this use case.

Finally, we hope we have addressed the reviewer’s concerns. We welcome any further feedback or suggestions that would help strengthen the paper, and we are happy to run additional experiments during the discussion period if further required by the reviewer.

References :

[1] Zuo, Yuxin, et al. "Ttrl: Test-time reinforcement learning." Advances in Neural Information Processing Systems 38 (2026): 131459-131483.

[2] Zhang, Kongcheng, et al. "Consistent paths lead to truth: Self-rewarding reinforcement learning for llm reasoning." Advances in Neural Information Processing Systems 38 (2026): 59849-59887.

[3] Zhang, Qingyang, et al. "Right question is already half the answer: Fully unsupervised llm reasoning incentivization." Advances in neural information processing systems 38 (2026): 67345-67372.

Reviewer_t51AOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes GoATT-R, a test-time latent reasoning method that can control how far to optimize in latent space. From the current literature, the authors show that Latent Thought Policy Optimization does not improve accuracy when increasing a confidence-based reward. They then define optimization pressure as the desired increase in confidence reward relative to the unoptimized state and use this observation to motivate controlled rather than unbounded latent optimization. Specifically, GoATT-R uses an Ensemble Kalman Filter to move the latent thought tokens toward a prescribed target confidence reward, while a lightweight policy network is trained online to select the pressure for each problem using majority vote agreement across optimization steps as a label-free reward. In terms of performance, the method shows improvement across four instruction-tuned LLMs and five mathematical reasoning benchmarks compared with the baselines.

Critical strengths

For me the paper finds an important problem in test-time latent optimization, where maximizing a confidence-based proxy reward without bound can hurt accuracy, and shows it directly with the curve in Figure 3, where accuracy peaks at a moderate optimization pressure and drops by several points under high pressure. This is a useful observation for the line of work on test-time latent reasoning, since methods like LTPO optimize such proxy rewards without any mechanism to stop. The second strength is the proposed adaptive target-based optimization framework. GoATT-R uses Ensemble Kalman Filter to move latent thought tokens toward a chosen confidence target and an online policy network to select the optimization pressure for each problem without using ground-truth labels. The method is evaluated across four instruction-tuned LLMs and five mathematical reasoning benchmarks, where it consistently achieves the best average performance among the compared methods.

Critical weaknesses

The adaptive policy relies on majority-vote agreement as a label-free proxy for answer quality. However, I have concerns about whether the paper fully validates its central adaptive mechanism. In particular, the online policy learns the optimization pressure using majority vote agreement as a label-free reward, but the paper does not directly establish how well this signal tracks true answer correctness, because the model can repeatedly return the same wrong answer on a topic it might not have enough knowledge about.

The second thing is about the online policy network learning that is used to get the optimization pressure, which is the main new component, and it is not supported by sufficient experimental analysis since there is no ablation against a fixed or random value across models and benchmarks, and its reward based on majority vote agreement is biased toward low pressure because the answers agree most when the embeddings barely change.

Some minor comments: The paper should be more polished since it still contains a lot of typos, such as in the paragraph "Goodhart’s Law in latent space". The experiment in Figure 3 can be used as motivation for the method, which helps improve the coherence of the paper.

Questions for authors

It would be good if the authors could show how well the majority agreement correlates with ground-truth answer correctness across different problems and pressure levels, and whether the method's effectiveness correlates with the base model's performance on the given tasks. Additional thing is about evaluating on non-math tasks such as coding or reasoning benchmark, since the confidence reward may behave differently there.

2 comments

We sincerely thank the reviewer for their careful reading of our paper and their constructive feedback. We address each concern below and provide additional supporting evidence with detailed analysis that we invite the reviewer to read in Appendix F:

Weaknesses :

1 - The reviewer argues that incorrect answers can remain consistent across nearby latent states, receiving high agreement scores. While this is possible in principle, we provide empirical evidence in Appendix F that agreement is in fact a well-calibrated measure for correctness. Across all 12 model-dataset combinations, accuracy increases monotonically with agreement (Figures 6, 7). Importantly, none of these experiments show an instance where incorrect answers consistently receive high agreement, which contradicts the hypothesis that agreement rewards incorrect but consistent outputs. Furthermore, we show in Figure 8 that the mean agreement rˉ\bar{r} tracks accuracy across optimization pressures, peaking at ϵ=0.2\epsilon=0.2 then decreasing, following the same pattern as the Goodhart curve in Figure 3.

Moreover, using majority-vote agreement as a label-free reward signal is not a novel assumption of our work. It is the established approach across multiple published and concurrent methods. TTRL(Zuo et al., 2025)[1] uses the majority-vote reward to fine-tune model weights on unlabeled test data. CoVo (Zhang et al., 2025a)[2] extends TTRL and uses consistency across reasoning trajectories instead of final answers as a reward signal to fine-tune model weights. EMPO (Zhang et al., 2025b)[3] uses entropy-based signals as intrinsic rewards for weight-level RL, and, according to the authors, "optimizes the model to favor reasoning traces yielding consistent answers, enhancing output reliability."

This means that the reviewer's concern, if taken to its conclusion, would equally invalidate all of these works. Crucially, our use of agreement is lighter than theirs, where we use it only to select optimization pressure on input embeddings, while the above methods use it to update model weights.

Furthermore, while we do not argue that majority voting is a substitute for actual correctness, however in the absence of access to ground truth data at inference, it is a substantially better measure of correctness than the confidence-based reward used by LTPO, which, as we demonstrate empirically, rapidly degrades accuracy when maximized without bound.

2 - Regarding the first part of the weakness, we emphasize that the primary contribution of this work is the Goodhart-aware optimization pressure mechanism itself, not solely the adaptive policy. Even with a fixed ε\varepsilon, our framework provides an approach to controlling optimization pressure during test-time reasoning via the EnKF, a contribution that, to the best of our knowledge, does not exist in the literature. The adaptive policy is an additional component that selects ε\varepsilon in a label-free, principled manner.

We note that the fixed-ε\varepsilon sweep in Figure 3 already demonstrates that the beneficial optimization region is very narrow, and a randomly selected ε\varepsilon would with high probability fall outside this region, degrading performance. Furthermore, combining Figure 8 with Figure 3 confirms that the policy drives the threshold toward high-agreement regions (Figure 8) and high-accuracy regions (Figure 3), which correspond since agreement and accuracy are correlated. This provides a principled way to select the optimization pressure without access to ground-truth labels.

While one could in principle sweep all fixed ε\varepsilon values across all models and datasets and report the best-performing value, this would require access to ground-truth labels to select the optimal ε\varepsilon, information that is unavailable at inference time. Reporting such oracle-selected results as a baseline would misrepresent the method's practical capabilities. Our adaptive policy is designed precisely to approximate this selection without label access.

If the reviewer still requires these experiments, we are happy to run and include them during the discussion period and we can present results comparing the adaptive policy against fixed-ε\varepsilon, random-ε\varepsilon-per-question, and random-ε\varepsilon-drawn-once baselines across all 4 models and 5 datasets in the rebuttal and revised manuscript once experiments are complete.

2 (Continued) - Regarding the second part of the weakness, the concern that agreement is biased toward low pressure assumes that high agreement can mislead the policy. However, as we show empirically in Appendix F (Figures 6, 7), agreement is well-correlated with correctness across all 12 model-dataset combinations: accuracy increases monotonically with agreement. Therefore, regardless of any bias, wherever agreement is high, accuracy is correspondingly high. The policy is driven toward high-agreement regions, which are precisely the high-accuracy regions. Furthermore, if agreement were simply biased toward low pressure, the optimal pressure would be at $\epsilon = 0 $ (no optimization). Instead, both Figure 3 and Figure 8 show that peak accuracy and peak agreement occur at moderate ϵ\epsilon (approximately 0.20.2), confirming that the agreement signal does not collapse to trivially favoring no optimization, but rather identifies the beneficial optimization region. We also note that accuracy itself is biased against high optimization pressure. For example LTPO-MV , which maximizes the confidence reward without bound, degrades accuracy below competing methods. Agreement follows this same pattern precisely because the two are correlated. Rather than being a flaw, this behavior reflects agreement correctly tracking the accuracy landscape, validating it as a reliable label-free reward.

Regarding typos, we sincerely thank the reviewer for carefully identifying them. We appreciate the attention to detail and will ensure the final manuscript is thoroughly polished. Regarding Figure 3, we agree with the reviewer’s suggestion and will restructure the paper to use the fixed-epsilon sweep as early motivation for the adaptive policy. We will upload a revised version before the discussion period ends

Questions:

We believe the first point regarding agreement correlation with correctness has been addressed in our response above and in Appendix F.

Regarding the relationship between the method's effectiveness and the base model's performance, this is already reflected in our results. As shown in Tables 1 and 2, the improvement from GoATT-R varies with model strength and dataset difficulty. Smaller and weaker models on challenging datasets (e.g., Llama-3.2-8B on MATH-500) show larger gains, while stronger models on easier datasets (e.g., Qwen-2.5-14B on GSM8K) show diminishing improvements as the base accuracy is already high. This is consistent with the expectation that test-time optimization provides the most benefit when the model has room to improve.

Regarding evaluation on non-math tasks such as coding or reasoning benchmarks, we agree this is a valuable direction and will include additional benchmarks in the final manuscript once experiments are done.

Finally, we hope we have addressed the reviewer’s concerns. We welcome any further feedback or suggestions that would help strengthen the paper, and we are happy to run additional experiments during the discussion period if further required by the reviewer.

References:

[1] Zuo, Yuxin, et al. "Ttrl: Test-time reinforcement learning." Advances in Neural Information Processing Systems 38 (2026): 131459-131483.

[2] Zhang, Kongcheng, et al. "Consistent paths lead to truth: Self-rewarding reinforcement learning for llm reasoning." Advances in Neural Information Processing Systems 38 (2026): 59849-59887.

[3] Zhang, Qingyang, et al. "Right question is already half the answer: Fully unsupervised llm reasoning incentivization." Advances in neural information processing systems 38 (2026): 67345-67372.

Reviewer_ZLw9OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GoATT-R to mitigate overoptimization of confidence-based rewards in test-time latent reasoning. It uses an Ensemble Kalman Filter to steer latent embeddings toward a prescribed confidence target, an online policy to select the optimization pressure, and trajectory majority voting to produce the final answer. The policy is trained using voting agreement rather than ground-truth labels.

Critical strengths

The paper states a problem: improving a confidence proxy does not necessarily improve answer correctness. Target-controlled latent optimization is an interesting direction, and the experiments show improvements on some benchmarks.

Critical weaknesses
  1. The policy-learning objective lacks sufficient justification.

    Equation (11) uses signed advantage-weighted squared error to train the policy. However, pushing the policy away from low-reward pressures does not necessarily lead to better pressure values. The objective may even favor extreme values rather than the optimal ones.

  2. Majority-vote agreement may not reflect correctness.

    The paper argues that confidence is not a reliable indicator of correctness, but instead uses majority-vote agreement to guide the policy. However, incorrect answers can also remain consistent across nearby latent states and receive high agreement scores.

  3. The contribution of the adaptive policy remains unclear.

    Figure 3 shows that fixed pressure achieves higher accuracy than the learned policy (78.1% vs. 76.6%), although the fixed value was selected using the test set. A fair comparison with validation-tuned fixed pressure and simple stopping strategies is therefore needed. Moreover, the gains over LTPO-MV are small on the Qwen models (0.1 and 0.5 percentage points), while Llama-3.2-3B shows a 4.8-point drop on MATH-500. These results raise concerns about the effectiveness and generalizability of the adaptive policy.

Questions for authors
  1. What is the justification for the policy-learning objective in Eq. (11)? Can the authors provide theoretical analysis or controlled experiments showing that it learns appropriate pressure values?
  2. How well does majority-vote agreement correlate with correctness across different pressures? Can the authors demonstrate that optimizing agreement improves correctness rather than simply reinforcing consistent but incorrect answers?
  3. Does the adaptive policy outperform validation-tuned fixed-pressure EnKF and simple stopping strategies under matched inference-time compute budgets?
4 comments

We thank the reviewer for their detailed feedback. We address each concern below and provide additional supporting experiments with detailed analysis that we invite the reviewer to read in Appendix F:

Weaknesses:

1 - We believe there may be a misunderstanding regarding the role of the optimization pressure ϵ\epsilon. The policy does not use ϵ\epsilon to directly maximize agreement. Instead, ϵ\epsilon sets the target R~=R0+ϵ\tilde{R} = R_0 + \epsilon of the confidence-based proxy reward (Eq.9), which the EnKF uses to steer the latent embeddings. The policy observes the resulting majority-vote agreement rnr_n across the decoded answers given an ϵ\epsilon, and adjusts ϵ\epsilon accordingly to maximize the agreement. For example, if the model generates 55 answers and 33 of them match, the agreement is 3/5=0.63/5 = 0.6, and the policy adjusts ϵ\epsilon to maximize this agreement.

This distinction is critical to addressing the concern about extreme values. Crucially, high agreement does not correspond to high optimization pressure. Excessive ϵ\epsilon drives the EnKF to overoptimize the embeddings, producing inconsistent answers across the trajectory and therefore low agreement, as illustrated by the decreasing mean agreement rˉ\bar{r} at high ϵ\epsilon in Figure.8 (Appendix.F) in the updated manuscript. Conversely, moderate ϵ\epsilon values keep the embeddings in a region where the model produces high agreement. The policy is thus incentivized toward the moderate pressures that maximize agreement, not toward the extreme pressures that maximize the confidence-based proxy reward but reduce agreement. This is also empirically observed in Figure 3.

2 - The reviewer argues that incorrect answers can remain consistent across nearby latent states, receiving high agreement scores. While this is possible in principle, we provide additional empirical evidence in Appendix.F that agreement is in fact a well-calibrated measure for correctness. Across all 12 model-dataset combinations, accuracy increases monotonically with agreement (Figures 6, 7). Importantly, none of these experiments show an instance where incorrect answers consistently receive high agreement, which contradicts the hypothesis that agreement rewards incorrect but consistent outputs. Furthermore, we show in Figure 8 that the mean agreement rˉ\bar{r} tracks accuracy across optimization pressures, peaking at ϵ=0.2\epsilon=0.2 then decreasing, following the same pattern as the Goodhart curve in Figure 3.

Moreover, using majority-vote agreement as a label-free reward signal is not a novel assumption of our work. It is the established approach across multiple published and concurrent methods. TTRL(Zuo et al., 2025)[1] uses the majority-vote reward to fine-tune model weights on unlabeled test data. CoVo (Zhang et al., 2025a)[2] extends TTRL and uses consistency across reasoning trajectories instead of final answers as a reward signal to fine-tune model weights. EMPO (Zhang et al., 2025b)[3] uses entropy-based signals as intrinsic rewards for weight-level RL, and, according to the authors, "optimizes the model to favor reasoning traces yielding consistent answers, enhancing output reliability."

This means that the reviewer's concern, if taken to its conclusion, would equally invalidate all of these works. Crucially, our use of agreement is lighter than theirs, where we use it only to select optimization pressure on input embeddings, while the above methods use it to update model weights.

Furthermore, while we do not argue that majority voting is a substitute for actual correctness, however in the absence of access to ground truth data at inference, it is a substantially better measure of correctness than the confidence-based reward used by LTPO, which, as we demonstrate empirically, rapidly degrades accuracy when maximized without bound.

3 - The reviewer notes that fixed ϵ=0.20\epsilon = 0.20 achieves 78.1% on GSM8K while the learned policy achieves 76.6% (ϵˉ=0.11\bar{\epsilon} = 0.11). We want to clarify the nature of this comparison:

  • The reviewer suggests that a fixed ϵ=0.20\epsilon = 0.20 could be selected via validation tuning. However, this is not how the sweep was conducted. The sweep simply evaluates all 16 pressure levels on all problems to verify that the Goodhart curve exists. No ϵ\epsilon was selected' using test or validation data. More fundamentally, at inference we have no access to ground-truth labels or validation sets. Therefore, tuning a fixed ϵ\epsilon on a validation set falls outside the scope of this work, as we argue in Section~1 (lines 62--67): "identifying [the optimal pressure] would require sweeping over all possible pressures and evaluating accuracy at each, which requires ground-truth labels and is infeasible at inference.'' The sweep in Figure 3 is a diagnostic to verify the Goodhart curve exists, not a practical method. Moreover, even if one were to use validation data to tune a fixed pressure, such a method does not exist in the literature, and proposing it would itself be a contribution of this work, not a competing baseline.

  • The reviewer notes that the fixed ϵ=0.20\epsilon = 0.20 achieves higher accuracy than the learned policy (78.1% vs. 76.6%). However, identifying ϵ=0.20\epsilon = 0.20 as optimal requires evaluating all 16 pressure levels with ground-truth labels, which is infeasible at inference. The policy learns ϵˉ=0.11\bar{\epsilon} = 0.11 in a single online pass using only majority-vote agreement, with no access to ground truth. Despite this, it achieves 76.6%, well above competing methods such as LTPO-MV (69.6%) and slightly below the optimal achievable accuracy (78.1%), while staying in the beneficial region.

  • Regarding the narrow gains on Qwen models: as model capacity increases, the baselines strengthen and leave less room for improvement. GoATT-R still achieves the highest Avg$_3$ and Avg$_5$ for every model in Table 1, including Qwen-2.5-14B. the avg5 improvement on Qwen7B is still 2 percent and 0.9 percent on qwen14B.

  • Regarding the MATH-500 drop on Llama-3.2-3B: Figure 4 shows that K=8K = 8 thought tokens is suboptimal for this specific model-task pair (accuracy peaks at K=1K = 1 and declines monotonically). This is a thought-token count issue, not a policy or method failure. The Qwen models and Llama-3.1-8B show stable MATH-500 performance across all token counts. Furthermore, performance sensitivity to thought-token count is a well-known issue, documented in LTPO [4], which reports only the best-performing token count per model-dataset pair after sweeping over all thought token counts number, and in other works such as SoftCoT[5](figure 2) and UniVLR[6] (figure 5); it is common in reasoning methods that use thought tokens and is not related to our contribution. We also note that MATH-500 contains only 500 problems, which is a relatively short horizon for an online policy to converge. Finally, this is one model-dataset pair out of 12; the reviewer's concern singles out the weakest case while the aggregate results across all combinations consistently show improvement.

Questions :

1 - We refer the reviewer to our response to W1 and W2 above, which directly address this question. The justification for the policy objective in Eq.(11) is that the policy maximizes majority-vote agreement, which we empirically demonstrate is well-correlated with correctness across all 12 model-dataset combinations (Appendix F, Figures 6--8). As discussed in W2, using agreement as a label-free reward signal is an established approach in the literature [1][2][3]. The controlled experiments the reviewer requests already exist in Figure 3, which shows the Goodhart curve across 16 pressure levels, and Figure 8, which shows the policy learns appropriate pressure values.

2 - We address this question in detail in the new Appendix F (Figures 6--8). Across all 12 model-dataset combinations, accuracy increases monotonically with agreement (Figures 6, 7 ). Importantly, none of these experiments show an instance where incorrect answers consistently receive high agreement, directly contradicting the hypothesis that agreement merely reinforces consistent but incorrect answers. Furthermore, our experiment (Figure 8 ) sweeping fixed ϵ\epsilon values on GSM8K with Llama-3.2-3B shows that mean agreement rˉ\bar{r} tracks accuracy across optimization pressures, peaking at ϵ=0.2\epsilon = 0.2 then decreasing, following the same pattern as the Goodhart curve in Figure 3, confirming that optimizing agreement does improve correctness.

3 - As discussed above, validation-tuned fixed pressure requires access to ground-truth labels, which is infeasible at inference and falls outside the scope of this work. Regarding simple stopping strategies, we are unaware of such methods in the literature for mitigating Goodhart's law in latent reasoning. While one might be able to design ad-hoc heuristics, such strategies lack a principled foundation. Therefore, we use an adaptive policy precisely because it finds the optimal pressure without access to a validation ground-truth set. We note that proposing and evaluating fixed stopping strategies would itself constitute a contribution, as we are unaware of such methods in the literature for latent reasoning. If the reviewer has specific stopping strategies in mind, we would be happy to evaluate them in a revision.

Finally, we hope we have addressed the reviewer’s concerns. We welcome any further feedback or suggestions that would help strengthen the paper, and we are happy to run additional experiments during the discussion period.

References:

[1] Zuo, Yuxin, et al. "Ttrl: Test-time reinforcement learning." Advances in Neural Information Processing Systems 38 (2026): 131459-131483.

[2] Zhang, Kongcheng, et al. "Consistent paths lead to truth: Self-rewarding reinforcement learning for llm reasoning." Advances in Neural Information Processing Systems 38 (2026): 59849-59887.

[3] Zhang, Qingyang, et al. "Right question is already half the answer: Fully unsupervised llm reasoning incentivization." Advances in neural information processing systems 38 (2026): 67345-67372.

[4] Ye, Wengao, Yan Liang, and Lianlei Shan. "Thinking on the fly: Test-time reasoning enhancement via latent thought policy optimization." International Conference on Learning Representations. Vol. 2026. 2026.

[5] Xu, Yige, et al. "Softcot: Soft chain-of-thought for efficient reasoning with llms." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

[6] Jiang, Houcheng, et al. "UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMs." arXiv preprint arXiv:2605.11856 (2026).

Thank you for the response and the additional experiments. My concern is that the observed positive association between agreement and correctness does not establish that optimizing the policy for higher agreement improves majority-vote accuracy. Could you compare the learned policy against two non-learned baselines: (1) using a fixed ε (e.g., 0.2), and (2) sampling ε at random?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.