acceptodds
Under review as a conference paper at ICLR 2027

From Smoothing to Noise Cancellation: Memory in Sequential Error Detection

Abstract

Language models often solve mathematical problems through a sequence of calculations and deductions. An evaluator, called a verifier, assigns each step a score intended to indicate whether the reasoning is correct. These scores are imperfect, so a detector must decide how to use earlier scores alongside the current one. Earlier scores may provide evidence, but they may also repeat the same scoring error. We study when these scores should be added as evidence, subtracted to cancel persistent noise, or ignored. We develop a theory of this choice for causal linear detectors under correlated noise, optimizing a class-weighted signal-to-noise ratio (SNR). In a stationary binary Markov model, all historical weights are positive when the state persists longer than the noise, negative when the noise persists longer, and zero when the two persistences match. After normalizing the current weight to one, nonzero total absolute historical weight strictly decreases as observation quality improves. We characterize when this law extends beyond binary states. For finite traces, initialization can reverse the stationary rule or produce mixed signs. A study of 27 methods on 2,000 MATH and GSM8K problems finds that the model-derived kernel improves within-position SNR over current-only scores in all six settings, with simultaneous intervals excluding zero. Matched-SNR comparisons and conditional-moment diagnostics assess when the model-prescribed weights are accurate. The results explain both the role of history and the conditions under which these memory rules apply.

Primary area:
interpretability and explainable AI
Keywords:
process reward models, step-level verification, correlated observation noise

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eSiPOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies when a linear detector should assign positive or negative weights to historical verifier scores. Under a binary Markov state model with correlated observation residuals, the paper characterizes the signs of the optimal coefficients and shows that the normalized total weight placed on history decreases as observation quality improves.

The extensions give a necessary-and-sufficient innovation condition for the sign law and show that finite initialization can reverse signs or produce kernels with both positive and negative weights. The evaluation combines controlled simulations, numerical covariance checks, and comparisons of 27 aggregation methods across 6 observation settings on 2000 MATH/GSM8K problems.

Critical strengths
  1. The theory explains the structure of the optimal weights beyond the standard covariance-inverse solution. Thm 1 determines the sign of every historical coefficient and establishes monotonicity of the total historical weight for arbitrary fixed class weights. Thm 2 identifies exactly which innovation profiles preserve this sign law across observation qualities. Thm 3 then explains why simply truncating a stationary filter cannot reproduce finite-start sign reversals. These results provide a substantive understanding of how the optimal weights behave.

  2. The experiments distinguish the benefits of using history from the accuracy of the theoretical weight prescriptions. Problem-level nested fitting, EWMA controls tuned to the corresponding metric, and position-aware comparisons address important attribution questions. Table 2 shows improvements over scores based only on the current observation, while Table 8 and Appendix N.4 report cases where the fitted model’s prescribed weights fall short.

Critical weaknesses

my main concern: The practical value of the distinctive cancellation results remains unestablished. Every fitted stationary setting satisfies α^<ν^\hat{\alpha}<\hat{\nu}, placing these kernels in the positive-smoothing regime (Appndix M.6). Appendix N.6 finds no consistently supported subtraction direction under its stated descriptive rule. Moreover, the colored kernel falls below SNR-tuned EWMA in 5 of 6 within-position point estimates (Table 2), and substantial higher-lag residual discrepancies remain (Table 9). Thus, the experiments support beneficial history aggregation more strongly than practical use of cancellation or the exact prescribed weights. This limits demonstrated significance without contradicting the theoretical results.

Questions for authors
  1. Do any fitted finite-start Fisher kernels assign negative weights at supported positions? If so, how do those kernels compare with nonnegative kernels optimized under the same fitted moments? Appendix N.6 examines individual perturbations around scoring based only on the current observation. Those comparisons do not determine the signs or benefits of the jointly optimal kernel.

  2. Does relaxing the parametric moment model improve the empirical Fisher prescription? Appendix N.5 documents departures from the assumed covariance and conditional-mean structure, while Appendix N.4 shows that matching the evaluation objective does not eliminate the shortfall relative to tuned EWMA. Within the existing evaluation protocol, could a regularized Fisher rule estimated directly from training-set means and covariances over a short lag window clarify this discrepancy?

Reviewer_LxArOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work studies causal linear aggregation of noisy verifier scores for distinguishing the current error of a reasoning trace. Under a binary Markov model with geometrically correlated residuals, the authors characterize when optimal historical weights are positive, negative, or zero. They go on to prove that normalized total absolute history weight decreases as observation quality improves.

The analysis extends to more general innovation profiles and finite-start traces, where initialization can reverse the stationary sign prescription. Experiments on MATH and GSM8K demonstrate gains over current-only scoring, while also documenting departures from the assumed model and inconsistent advantages over tuned exponential smoothing.

Critical strengths
  1. Interesting theoretical results. The paper characterizes when historical verifier scores should be added, subtracted, or ignored. The analysis of initialization effects and the conditions under which these results hold are also useful.

  2. Careful experimental analysis. The authors evaluate several baselines and explicitly examine where their theoretical assumptions fail on real data.

Critical weaknesses
  1. Limited connection between theory and practice. The main theoretical distinction between smoothing and noise cancellation is not demonstrated on real verifier scores. Furthermore, the observed noise correlations and state transitions differ substantially from the theoretical assumptions, making it unclear how applicable the results are.

  2. Unclear practical benefit. The proposed weighting scheme does not consistently outperform simple exponential smoothing. The evaluation also relies on rollout-derived labels rather than actual reasoning errors, limiting the conclusions about error detection.

Questions for authors
  1. Can the authors demonstrate a realistic setting where subtracting historical scores improves error detection? The current experiments only provide evidence for smoothing.

  2. How well do the theoretical results generalize beyond the assumed noise and state-transition models, given the departures observed in the experiments?

  3. What practical advantage does the proposed framework offer over simply tuning exponential smoothing? Stronger results on actual reasoning-error detection would help establish its significance.

Im happy to revise my score if the above are addressed, and also acknowledge that this isn't my area of expertise

Reviewer_oGDDOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper asks how a streaming error detector should combine the current step-verifier score with earlier scores.

Model. The latent clean/error state is a stationary two-state Markov chain with persistence ν = 1 − λ − ε (Eq. 1). Normalized scores follow X_t = D·S_t + ξ_t, where the path-conditional residual covariance is 2Dα^{|t−s|} (Eq. 2). The detector is a causal linear kernel that maximizes a class-weighted SNR, which is a Fisher discriminant with direction N⁻¹g (Eqs. 6–8).

Theoretical results.

  • Theorem 1 (p. 4; Theorem 4 in App. A). For every class weight w, all historical coefficients have sign(ν − α). For α ≠ ν, the normalized total absolute history H_abs strictly decreases in D, with limits 1 + |ν − α|/(1 − ν) and 1 (Eq. 10). Individual coefficients can still grow as D increases (p. 5; A.10).
  • Theorem 2 (p. 6; Theorem 6). For general positive innovations, the condition q_{j+1} ≥ αν q_j is necessary and sufficient for this sign law. A relative-perturbation margin follows (Eq. 17; Corollary 1, Eq. 68 in B.1).
  • Proposition 1 (p. 5; Theorem 5). A residual-column criterion, r_j ≥ g_j, decides when current-only scoring is optimal among nonnegative kernels.
  • Theorem 3 (p. 6; Theorem 7). An exact finite-start kernel, computable in O(t). Its boundary α_crit(t) depends on the initial law and can reverse or mix the signs (Eqs. 88–90).

Validation.

  • Exact numerical checks and AR(1) simulations (Sec. 5.1, Fig. 2; App. L, M.1; Figs. 3–5).
  • Real-data study (Sec. 5.2–5.3; App. M). It uses one Math-Shepherd trajectory for each of 1,000 MATH and 1,000 GSM8K problems (7,538 and 3,558 steps), with three score streams: two frozen-LLM linear probes and the frozen Qwen2.5-Math-PRM-7B. It compares 24 aggregators on pooled and within-position AUROC (Tables 1, 5–7).
  • Follow-up SNR study (App. N). It adds three SNR-tuned or finite-start rules (27 methods in total) and conditional-moment diagnostics (Tables 2, 8–11).
  • Both studies use paired, problem-clustered simultaneous bootstrap intervals (M.5; N.4).
Critical strengths
  1. Substantial theoretical development. The paper puts real effort into the theory, and [the derivations / the moment model / specific result] seem careful. My one complaint is that the results aren't tied together by a clear narrative, so it's hard to see how each piece supports the main claim. A short roadmap at the start of the theory section would help a lot.

  2. Extensive experimental evaluation. The evaluation covers six settings and reports both AUROC and SNR, along with within-position versions of each (Tables 1, 2, 6, 7 and 11, App. N). The main baseline, EWMA, is tuned separately on each metric being reported, so the comparison is a fair one. This setup also lets the authors separate two possible causes of failure: a mismatched objective, and a wrong moment model.

Critical weaknesses
  1. The paper's distinctive predictions get no empirical support.

    • Cancellation never occurs on real data.
      • In all six streams, mean ν̂ − α̂ lies between 0.329 and 0.431, and every outer fold has α̂ < ν̂ (M.6, p. 50).
      • The model-free first-column diagnostic finds 580 of 600 cells positive, and all 91 supported position–lag groups favor adding history (N.6, p. 56). The paper concedes this is "without establishing a real-data cancellation effect" (p. 9).
      • "Noise cancellation" (the title) is therefore shown only on synthetic AR(1) data (Fig. 2a; M.1). No figure shows any real-data result.
    • The new refinements do not improve on the white-noise prescription or on tuned EWMA.
      • α-correction vs white kernel. On within-position SNR, the balanced colored kernel's point estimates fall below the white kernel in 5 of 6 settings (Table 11; e.g., 0.9949 vs 1.0106 on MATH Gen-7B, 2.0416 vs 2.0776 on GSM8K PRM; the exception is MATH PRM). They also fall below the SNR-tuned EWMA in 5 of 6 (Table 2). The same 5-of-6 pattern holds for within-position AUROC against the white kernel (Table 6). The intervals for "colored minus white" and for the within-position "stationary colored minus metric-matched SNR-tuned EWMA" contrasts are listed in N.4 (p. 55) but never reported, and Table 2 omits the white kernel altogether. These gaps (about 0.014–0.045 SNR) therefore remain untested.
      • The model predicts the wrong ordering. Under the fitted AR(1) working model, the colored kernel is predicted to beat SNR-tuned EWMA in all six settings. Empirically the order reverses in 5 of 6 (Table 8, p. 57; e.g., MATH Gen-7B predicted 0.983 vs 0.973, observed 0.995 vs 1.012).
      • Significant losses on pooled SNR. The only colored-vs-EWMA SNR differences reported as significant are on pooled SNR for the two MATH local probes, and both go against the colored kernel: −0.0539 [−0.0959, −0.0118] and −0.0574 [−0.1040, −0.0109] (Sec. 5.3; N.4).
      • Significant losses on pooled AUROC. Against pooled-tuned EWMA, two intervals exclude zero, both against the colored kernel (Table 7): MATH Math-1.5B −0.0148 [−0.0285, −0.0010] and GSM8K PRM −0.0049 [−0.0095, −0.0002]. Main-text Table 1 shows bounds only for the within-position contrast. The main text does not discuss either pooled loss; its GSM8K PRM sentence on "pooled-tuned EWMA" (p. 9) concerns within-position AUROC (M.6, p. 50).
      • Within-position AUROC vs metric-matched EWMA. Point estimates are negative in 5 of 6 settings (+0.001 on MATH PRM), and no interval excludes zero (Table 1; Table 7).
      • Finite-start correction (Theorem 3). It helps in one setting (GSM8K PRM +0.0084 [+0.0001, +0.0166]) and hurts in another (MATH Gen-7B −0.0051 [−0.0099, −0.0003]) (Table 2).
    • The headline claim uses the weakest comparator.
      • The abstract (p. 1) and Sec. 5.3 (p. 8) lead with improvement "over current-only scores in all six settings, with simultaneous intervals excluding zero", and Sec. 6 (p. 9) repeats the improvement. Nearly every history-using row in Table 11 also beats current-only. The exception is continuous CUSUM, which is below per-step in 4 of 6 settings (e.g., 0.9641 vs 1.0244 on MATH PRM).
      • The intervals behind this claim are not tabulated. Sec. 5.3 cites Table 2 for them, but Table 2 gives intervals only for finite minus stationary.
      • The claim also comes from the App. N follow-up, which reuses the App. M predictions. The paper says this follow-up's intervals do not "turn an adaptive follow-up study into a confirmatory experiment" (N.4, p. 55).
  2. The novelty over classical colored-noise filtering is limited, and the stationary model fits the evaluated traces poorly.

    • Novelty.
      • At w = π₁ (or λ = ε), Eq. 13 is the classical measurement-differencing filter for AR(1) measurement noise: h_t = τh_{t−1} + X_t − αX_{t−1}, with κ_j = (τ − α)τ^{j−1} and τ strictly between α and ν by Eq. 37. It already implies the sign law.
      • The paper calls this case classical (p. 5; A.2), but Sec. 2 cites only a textbook (Särkkä & Svensson) and a Gaussian-process measurement-noise preprint (Kurtz & Lin, 2019). It does not cite the classical treatments of AR(1)-correlated measurement noise (e.g., Bryson & Henrikson's measurement differencing).
      • What is new is the w ≠ π₁ extension, total-mass monotonicity, and the finite-start boundary. In the paper's own measurements, w ≠ π₁ non-geometricity is worth at most 0.017% SNR at the L.10 grid point and 0.118% over a 5,000-point scan. Both figures are for the white-noise model (L.10, M.1). Fig. 5a shows the same thing visually: the w-dependent ratios merge into a common geometric tail within about 15–20 lags.
      • Empirically, the balanced and stationary colored kernels differ by less than 0.01 AUROC throughout Tables 5–6.
    • Model mismatch.
      • MATH has no observed recoveries (none among 2,749 error-source transitions; Sec. 5.3, p. 9; M.7). The estimate ε̂ ≈ 2.2×10⁻⁴ therefore comes entirely from the Jeffreys pseudocount (L.3). The chain is effectively absorbing, and the λ, ε > 0 assumption of Theorems 1–2 holds only through smoothing.
      • Residual correlations at lags 2 and 5 far exceed the AR(1) fit (Table 9; MATH Gen-7B at lag 5: 0.330 vs 0.023).
      • Residual means depend on the transition, contrary to E[ξ_t | z] = 0 (N.5).
      • GSM8K labels violate first-order Markov dynamics (M.7).
      • Traces average about 7.5 (MATH) and 3.6 (GSM8K) steps, and the within-position evaluation keeps only positions ≤ 10 and ≤ 4 (Sec. 5.2). The infinite-history results (Eq. 10; Fig. 1c, Fig. 5b) therefore have little direct bearing.

    On pooled AUROC, the finite LMMSE, Gaussian HMM, tuned fixed gain and position-aware aggregators do better (Table 5; e.g., MATH Gen-7B 0.7722 for the HMM vs 0.7458 for the colored kernel). This advantage is mostly a position/prior effect (ω = 0.083 on MATH, M.5; p. 9). On within-position SNR the HMM is below the balanced colored kernel in 4 of 6 settings (Table 11).

Questions for authors

see above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.