acceptodds
Under review as a conference paper at ICLR 2027

Learning to Correct Overlapping Forecasts: Accuracy, Queries, and Local Errors

Abstract

When several models predict the same outcome from partly overlapping inputs, combining their outputs can count shared evidence repeatedly or lose useful information. Residual correction offers an alternative: each model predicts the error remaining after earlier predictions, using only its own inputs, and its output is added as a correction. Learning these corrections from finite data introduces errors that affect later prediction targets. We develop a theory of the accuracy attainable with imperfect corrections and the conditions under which learning them reduces rather than amplifies error. In an independent additive-signal model, we identify which local errors later experts can remove and which survive, and show that residual queries can outperform ordinary forecasts even with imperfect responses. When response error scales with the predictable residual, we determine the optimal worst-case risk for unequal expert precisions and every repeated-query budget. An optimal allocation and ordering rule attains it without knowing the information overlap. We then analyze corrections learned by local ridge regression on fresh residual labels. For iid symmetric features with finite variance, we give a necessary and sufficient stability condition for fixed covering sweeps with common training parameters. Gaussian least-squares correction requires more samples per fit than the total feature dimension plus one to converge, even with correlated features. Regularization can stabilize smaller batches. Matched-data and fixed-budget experiments test how regularization and sample allocation turn these stability conditions into useful correction.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
minimax risk, forecast combination, noisy queries, sequential experimental design

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cbeDOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article discusses how to sequentially correct the errors of expert models and provides a theory applicable to additive signal models and linear models. It also provides a lower-bound understanding of this model from the perspective of minimax optimality and min-max optimization.

Critical strengths
  1. Provided a theory for the sequential error correction model and conducted experimental verification.

  2. The exact finite-sample energy recursion and the necessary-and-sufficient stability criterion (Theorems 8 and 9) are mathematically clean and technically useful.

Critical weaknesses
  1. The technical contribution of the entire study is mainly to establish precise error and stability characterization of specific models under existing residual queries (such as the cited study Frongillo et al. (2025)) and the local projection framework (a commonly used theoretical analysis method in statistics), rather than proposing a new information aggregation principle or universal learning algorithm. In addition, the discussion in the related work section of this article is not sufficient.

  2. Section 5 only considers the noiseless form when using linear regression. However, in statistical research on linear models, it is often assumed that Y=θ⊤Z+ξY=\theta^\top Z+\xi, which means there exists white noise. Therefore, the impact of the model's inherent randomness on expert aggregation has not been studied yet.

Questions for authors
  1. Equation (13) given in Section 4 often does not hold in the linear model in Sec 5. In fact, Sec 5 uses equation (18). Therefore, the analysis of these two parts is relatively disjointed, and the connection between the previous minimax conclusion and subsequent analysis is unclear.

  2. Why is the sum of Y considered as a learning objective in Section 2? Can we assume that $X_ {\ell} \sim N (\beta_ {\ell}, 1) $ and discuss the joint estimation of βℓ\beta_ {\ell}, would it be more intuitive?

  3. Can the processing in Section 5 achieve minimax optimization for models with the (high-dimensional) ℓ0\ell_0 sparse structure (a common statistical model)? And correspondingly, what would be the assumption on sample size at this point?

  4. Is this study related to online learning?

Reviewer_yJx1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper considers forecast aggregation for linear regression. In the model, each forecaster (expert) privately observes a set of linear features. The regression result is deterministically identified by these features. Moreover, each expert's forecast is their local linear combination of the features. The elicitation mechanism in the paper elicits the residual prediction from each expert, i.e., the forecast minus current aggregation, in a sequential order. I view the main result as Theorem 8-10, showing that, when experts do not know the residual prediction from others, by dynamically learning the residual weights, they converge to perfect prediction and thus perfect aggregation.

Critical strengths

The paper analyzes an interesting instantiation of linear regression of prior sequential elicitation literature. However, I found the paper very hard to read. I'll explain in weaknesses below.

Critical weaknesses
  • The main problem is the writing quality. Many terms and mathematical notations were used before being formally introduced or defined. I'll list some examples below.

    • "energy / total energy"; page 2, page 4
    • "communting orthogonal projections"
    • "full sweep" / "covering sweep"
    • protocol Π\Pi and class PB\mathcal{P}_B in equation 10.
    • PiP_i, the mathematical definition was never made clear.
    • ...
  • Besides the undefined terms/math above, many assumptions lack clarification / explanation. For example, the assumption that Pitet=etP_{i_t}e_t = e_t, which means that ete_t is simply a deterministic function of the features, which doesn't even permit noise. That assumption later made more sense when it was justified for the linear regression parameters being frozen.

  • The novelty is limited beyond prior work that proposes residual elicitation. The analyses are standard. The main result is for a noiseless linear regression model, which feels very restrictive and unrealistic.

  • The first half of the paper assumes that experts have sufficient knowledge of the underlying distribution and other experts’ prediction functions to answer residual queries accurately. This assumption seems unrealistic. The convergence analysis in the second half is also confusing: experts now train new local linear regressions on residual labels, without reusing their existing regressors. It is therefore unclear how this retraining procedure connects to the original motivation of aggregating predictions from existing models.

Questions for authors

Is it hard to generalize the results to a standard noisy regression model?

Reviewer_doxPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper considers a set-up for predicting a scalar response based on m explanatory variables/features. It considers n linear experts (linear predictive models) each of which uses d features. It considers fitting the experts for predicting the response in a iterative way where each expert fits the residual based on N independents observations. The process is repeated for K sweeps. The paper provides theoretical results to understand the performance of the above algorithm for combining experts. It establishes the minimax mean square error when the experts have bounded local errors. The study provides a series of results for OLS and ridge-based linear experts. An interesting result that the paper establish is that Gaussian least-squares correction requires more samples per fit than the total feature dimension plus one to converge.

Critical strengths

The results look very interesting and quite strong. Unfortunately, they are not written in a human-readable way. Otherwise, I liked the results in Section 3 & 5.

Critical weaknesses

I had a very hard time reading the paper. I did not know the literature before but I'd spent some time to understand the results. While I liked its topic I felt the problem should be better motivated. Also, a real data demonstration would really help to motivate the problem. The graphic is Figure 1 contains symbols that are not defined in that section. So, the graphic aggravates readability. Also, \sigma in (1) is undefined. I am assuming that it represents a sigma-algebra. It will be better to motivate equation (2) better. After this equation, the examples of OLS and ridge can be provided to showcase why the last two equations in (2) holds for popular linear experts. Also, the set-up is unnecessarily too complex compared to the set-up in Frongillo et al. (2025) where a=1 and b=-1. I do not see any practical benefit for this complexity. The paper's writing should be made better.

Questions for authors
  1. The theory suggests that in Gaussian case, the algorithm would need at least (m+1)*n sample size. If m is large and n is large, such that this condition is not met what would be a way out? Would a screening rule for weeding out bad experts before this algorithm is applied work?
  2. I did not get the sporadic proofs that were provided in the paper. I found them entirely unreadable. I wonder why those proofs are presented in the main paper. It would be better to contextualize them and present them in the appendix along with other proofs & derivations.
  3. What happens if Y is a function of only a few (sparse) features and the number of features m is large? Can the algorithm take advantage of this simple sparse additive set-up.
  4. What happens with non-linear experts?
  5. A real data demonstration will greatly help to motivate the method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.