- Rating
- 2
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a method for repairing functional coherence and estimating uncertainty in frozen pretrained forecasters. The authors formulate coherence repair as a KL-minimal reweighting problem, enforcing zero expected changes in a chosen target functional, plus extensions to covariance constraints and approximate repair. The authors further introduce Amortized Covariance Estimation (ACE), which predicts finite-horizon uncertainty from current candidate responses without requiring future sequential rollouts at inference time.
The approach is evaluated with TabPFN classifiers and small reasoning LLMs. The reported results demonstrate substantial reductions in defined self-induced drift and gains in predicting future uncertainty and computational efficiency relative to explicit resampling.
Critical strengths
-
The paper identifies a motivated issue in predictive resampling with pretrained models, where repeated self-generated imputations may systematically shift the quantity of interest. The distinction between the forecaster's original predictive distribution and the induced continuation law provides a clear motivation for coherence repair.
-
he analysis connects martingale coherence with accumulated finite-horizon covariance, motivating the estimation of future uncertainty beyond the current local covariance. ACE provides a practical approach to amortizing this otherwise expensive computation.
-
The evaluations include tabular classifiers and reasoning LLMs, with reported improvements in drift reduction, future-variation estimation, and inference efficiency support.
Critical weaknesses
-
The main coherence-repair formulation is essentially a classical KL information projection under moment constraints, whose exponential-tilting solution and convex-hull feasibility conditions are well established. Similarly, the finite-horizon covariance decomposition and Gaussian limit largely follow standard martingale theory under the stated assumptions.
-
The proposed repair enforces a martingale condition for a selected readout under the modified continuation law. However, this property does not establish predictive coherence for the full distribution or calibration with respect to the underlying data-generating process. Although the paper acknowledges this distinction, the nonlinear synthetic comparisons in Appendix F.2 do not show uniformly improved ground-truth coverage or prediction error after repair.
Questions for authors
-
As stated in weakness point 1, the reviewer is more familiar with optimization; the principal appears to lie in the problem formulation and integration with amortized uncertainty estimation rather than in the underlying optimization machinery. Personally, I think the most interesting takeaway is rather the problem definition: frozen forecasters can systematically alter a selected prediction when repeatedly conditioned on self-generated observations.
-
Empirical generality remains somewhat limited; e.g., the LLM uncertainty experiments consider relatively short continuation horizons. Since repairability and computational cost depend on candidate support and augmented-context evaluations, it remains unclear how well the proposed framework extends to other model families or more complex continuation settings.
-
How would the authors explain the gap between asymptotic guarantees and finite-horizon uncertainty estimation? The Gaussian approximation relies on martingale limit conditions and sufficiently accurate covariance estimation. However, these assumptions are not directly established for the evaluated finite-horizon settings, particularly when approximate repair introduces residual drift. And perhaps undercoverage issues in some setups?
6 comments
[Q2] Longer Horizons, Candidate Support, and Computational Scope
Thank you for raising the questions about longer continuation horizons, candidate support, and computational scalability. To extend our LLM evaluation, we conducted an additional evaluation, comparing native and repaired continuations.
Table R.7. Additional LLM evaluation at .
| Model | Squared drift reduction | Share of from steps 5–8 | Accuracy (pp) [95% CI] |
|---|---|---|---|
| Llama | 54% | 7.4% | +0.3 [−2.9, +3.5] |
| OLMo | 24% | 15.2% | −2.2 [−5.2, +0.7] |
At , repair reduces squared conditional mean shifts without a statistically significant accuracy difference. We also found out that the ACE predictor transfers to via the horizon rescaling in Appendix G.2 without retraining. On independent repaired continuations, ACE outperforms the calibrated local-covariance baseline for one-pool Llama (squared-error difference , 95% CI ), with significant gains also in the evaluated OLMo comparisons. These results extend the empirical evidence for drift reduction and amortized uncertainty estimation over different horizons, without retraining the ACE predictor.
Candidate support and computational cost.
We further evaluated the feasibility of exact mean repair at fixed reasoning prefixes while varying the candidate pool size from 8 to 96 blocks. At initial prefixes, feasibility increases from 78% to 98% for Llama and from 65% to 91% for OLMo. After two reasoning blocks, the corresponding rates increase from 26% to 49% and from 27% to 46%, respectively. These results show that larger candidate pools improve repairability.
As analyzed in Appendix G.4, ACE requires current augmented-context readouts, whereas explicit estimation through repaired rollout paths of horizon requires up to candidate generations and readouts. Thus, increasing candidate support incurs additional probing cost, while ACE avoids future sequential rollouts at inference without eliminating current probing or offline supervision. We will incorporate these additional findings and clarify the computational trade-offs in the final manuscript.
[Q3] Finite-Horizon Uncertainty, Gaussian Approximation, and Undercoverage
Thank you for raising the important questions. We clarify that our finite-horizon covariance analysis and Gaussian approximation rely on different levels of assumptions.
Finite-horizon covariance and Gaussian limits.
As established in Proposition D.3, the decomposition under approximate repair remains valid without invoking a CLT.
\mathbb{E}[M_{n,H}\mid\mathcal F_0]=0, \qquad \operatorname{Var}(M_{n,H}\mid\mathcal F_0)=S_{n,H}.$$ Our additional diagnostics support this finite-horizon identity, with ratios of realized centered second moments to accumulated covariance ranging from 0.963 to 1.032 across the evaluated horizons up to $H=128$. Thus, $S_{n,H}$ characterizes the accumulated centered variation even when approximate repair introduces residual drift. In contrast, Theorem 4.1 and Proposition D.1 require exact mean repair and additional martingale CLT conditions. Since exact repair is **not always feasible under finite candidate supports**, practical continuations may require approximate repair. Consequently, these asymptotic conditions cannot be assumed to hold. Therefore, we further assess the endpoint Gaussian directly as an empirical finite-horizon approximation. **Understanding the observed undercoverage.** To investigate the 84.1% coverage of nominal 90% intervals, we separately replaced the learned endpoint mean and variance with independently estimated reference moments. **Table R.8.** Coverage of nominal 90% intervals under endpoint-moment substitutions. | Gaussian approximation | Coverage | |-|-| | Learned mean and variance | 0.841 | | Reference mean, learned variance | 0.852 | | Learned mean, reference variance | 0.850 | | Reference mean and variance | **0.903** | Replacing both learned moments with reference estimates increases coverage from 84.1% to 90.3%, close to the nominal level. The individual substitutions also improve coverage, indicating that **errors in estimating both the endpoint mean and variance contribute to the observed undercoverage**. Further analysis of independent rollouts finds 90.3% and 94.1% coverage for nominal 90% and 95% intervals with reference moments, comparable to Gaussian finite-sample benchmarks of 88.5% and 93.7% using 32 paths for moment estimation and 128 for evaluation. These findings support the **finite-horizon interval adequacy of a moment-matched Gaussian**. We will clarify how the finite-horizon covariance identities and conditional Gaussian limits provide a theoretical foundation for the empirically supported finite-horizon approximation.Re: Official Comment by Authors
Thank you for the response and additional experiments.
The additional H=8 experiments provide useful evidence of longer-horizon applicability, although the gains remain relatively modest. Two limitations nevertheless remain: (1) exact repairability degrades substantially along LLM continuations, even with larger candidate pools, making the frequency and impact of fallback steps under the actual 16-candidate setting important to clarify; and (2) the learned endpoint Gaussian still achieves 84.1% coverage against a nominal 90%, despite the useful diagnosis provided by the reference-moment experiments.
So, I am therefore inclined to maintain my rating.
Thank you for evaluating our work from an optimization perspective and for highlighting the importance of the problem formulation. While our approach builds on classical information projection, its central contribution is to formulate how predictive resampling can induce uncertainty without systematically shifting the selected readout of a frozen forecaster, and to connect this repair to efficient future uncertainty estimation.
Specifically, we reweight the imputation law while preserving the forecaster's responses to augmented contexts. Exact repair enforces functional coherence when feasible, while our approximate repair formulation provides a means to control residual drift when exact constraints are infeasible. Using in-context influences computed from exact augmented-context responses, we construct the required moment constraints without retraining the forecaster. Our conditional repair formulation further allows these constraints to be enforced while preserving the prescribed covariate marginal.
Importantly, enforcing functional coherence does not make future uncertainty directly available from the current state. Even when the current local covariance Γ0 is known, the finite-horizon centered covariance Sn,H depends on how the forecaster responds to subsequent imputations. This motivates ACE, which learns to estimate future centered variation from current candidate responses without requiring sequential future rollouts at inference time. Together, coherence repair and ACE provide a unified approach to constructing and efficiently estimating uncertainty under a frozen forecaster.
Our additional experiments further support these contributions. Direct evaluation against Sn,H yields a Spearman correlation of 0.901 for ACE, compared with 0.826 for Local covariance and 0.854 for Hidden + Local. Moreover, controlled comparisons with direct endpoint regression show improved CRPS across the evaluated horizons, including an 8.2% reduction at H=8 under limited rollout supervision, confirmed on independent evaluation states (Holm-adjusted p=0.0004). These results demonstrate the practical value of the proposed covariance-based uncertainty estimation beyond the underlying information-projection formulation.
We will clarify these contributions in the final manuscript, emphasizing the formulation of functional coherence for frozen forecasters, the response-preserving repair mechanism, and its integration with amortized future uncertainty estimation.
Thank you for highlighting the distinction between functional coherence and calibration to the underlying data-generating process. Our approach enforces coherence for a selected readout while preserving the frozen forecaster, not imposing coherence on its entire predictive distribution. Under exact repair, the readout has zero conditional mean drift, providing a martingale-based uncertainty construction without guaranteeing external calibration.
We acknowledge that the nonlinear comparisons in Table 7 do not establish a statistically significant improvement in ground-truth coverage or probability error (Repaired − Raw: coverage −0.006 [−0.081, +0.069], interval width +0.028 [−0.010, +0.050], and probability error +0.025 [−0.004, +0.054]). This is consistent with the scope of functional coherence. Under exact repair, the expected future readout remains equal to the current prediction, so any initial bias relative to the true probability is not automatically corrected by self-imputation.
Nevertheless, our additional experiments provide evidence of inferential benefits in specific settings. On three logistic data-generating tasks with known ground-truth probabilities, repair improves coverage from 0.722 to 0.778 while reducing interval width by 0.090 and CRPS by 0.012, with paired confidence intervals excluding zero, as reported in Reviewer Cf7r [W1, Q1]. In a separate evaluation on nonlinear tasks, repair also reduces conditional drift and achieves lower CRPS than both raw continuation and post-hoc re-centering, as shown in Table R.3. On LLM reasoning tasks, repaired continuations also achieve lower NLL and ECE for continuation-averaged answer probabilities across both models at H=2 and H=4, with paired confidence intervals excluding zero and no statistically detectable change in final-answer accuracy (Table R.6, Reviewer RABn [W2, Q2]).
These findings demonstrate that functional coherence can lead to improved uncertainty quality, while not implying universal calibration guarantees. We will clarify this distinction and incorporate the additional evaluations into the final manuscript.