- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper is situated in the field of forecasting. Rephrased in ML terminolgy, the paper tackes the issue of OOD prediction; that is, one may query the prediction model for datapoints that lie outside the observed distribution and there the model may be inaccurate. The paper quantifies the ambiguity of the OOD predictions using a measure they call Support Indiced Composition Radius (something like model multiplicity). Then the authors continue to show that when the underlying true function is additive we might be able to extrapolate well when each individual feature has full support (although some combinations of features may not have full support). This can be exploited to query data points that reduce faulty OOD prediction.
Critical strengths
OOD prediction is an important problem, I suppose especially so in forecasting.
Critical weaknesses
- The main weakness for me is the writing. For example, the paper introduces and uses lots of terminology that I am not aware of and also had trouble understanding when asking LLMs about it. Some examples from the introduction:
- Many queries "[...] specify a combination of factors, regimes, or control inputs ". What are factors, regimes, and control outputs? Are factors features as in machine learning? Are control inputs interventional queries ? What are regimes?
- What is a "marginal law" and what is a "target law"?
- What are "atoms" in this context and what are "atomic effects"? Even reading Section 2 did not clarify these terms for me.
- I am unsure about the novely of the findings in the paper. OOD generalization, model multiplicity, rashomon sets, causal interpretation of purely predictive models are all large literatures that this work directly connects to (although without using their terminology).
- The application on the bike sharing dataset is pretty limited and fails to convince me that the approach is generally unseful.
Questions for authors
How does the contribution compare to work in Rashomon sets, model multiplicity, OOD generalization, causal interpretation of predictive models? Could the authors try to rephrase the central findings in the machine learning terminology that is common at ICLR?
4 comments
Thank you for asking about the practical value of the framework. The revised experiments connect the identification question to observable prediction outcomes, using controlled support changes, multiple acquisition choices, and the Bike Sharing evaluation.
Changing the information available to the learner. The controlled experiments show why sample count and support answer different questions. At 200 trajectories per represented composition, the forcing estimator has target regret 0.2247 on ambiguous support and 0.0051 with bridge 011 (Appendix C.1). Marginal regret falls from 2.1675 to 0.0095 while the target law remains unidentified (Appendix C.2). When total observation budgets are matched, targeted support expansion reduces target regret from 0.2286 to 0.0263 at 384 added trajectories, in close agreement with the analytic comparison (Appendix C.3; Figure 2). This isolates the contribution of acquiring a missing interaction direction.
Choosing among multiple repairs and allocating for precision. Across 200 six-factor tasks, three TSA additions reduce mean normalized squared target residual to 0.0023, compared with 0.5057 for D-optimal design, 0.5047 for maximum-residual selection, and 0.6257 for random selection. TSA covers 98% of targets after three additions and all targets after four (Appendix C.4).
The new noisy study compares seven policies at equal observation budgets with the same minimum-norm predictor. TSA again completes target coverage by four observations. At noise standard deviation 0.5 and 48 added observations, following coverage with c-optimal allocation gives target-mean MSE 0.0227, compared with 0.2021 for uniform continuation and 0.0226 for middle-regularization c-optimal alone (Appendix F.1; Table 2). The full curves quantify how estimation variance and allocation affect prediction alongside structural coverage. These MSEs are noise-integrated averages over coefficient draws and tasks; SICR measures worst-case target-law ambiguity.
Predictive gains and practical checks on Bike Sharing. In the original full-feature retrospective comparison, TSA achieves mean target RMSLE 0.619 versus 0.699 for random-pool acquisition at 200 added records, an 11.4% reduction. At 400 added records, the reduction is 10.2% relative to the strongest non-targeted policy. At budget 200, improvements over same-span acquisition are 15.2% for Ridge, 19.6% for random forest, 5.6% for HistGradientBoosting, and 1.3% for MLP. The predictor-specific curves, including the MLP reversal at budget 400, remain reported in Figure 7.
The gain over same-span acquisition also holds at budget 200 under all three rush-hour definitions (Appendix E.3). In the tested three-dimensional setup, the geometric audit takes 57.8 microseconds, compared with 0.012-0.808 seconds for model fitting (Table 1).
A separate common-context audit uses four temperature/humidity cells to examine the overlap issue raised by the full predictor features. Bridge acquisition raises the full design rank from six to seven but does not improve either predictor under this coarser context definition (Appendix F.2; Table 3). This result distinguishes empirical prediction from a structural certificate, whose application requires population overlap and a correctly specified interaction family. The real-data evidence comes from one retrospective dataset; the controlled multi-candidate study and the shared-context audit address complementary questions within that scope.
Thank you for the clarifications which helped me to understand the paper‘s contribution.
Based on my understanding the novelty of the contribution is relatively small, although it is sound. Moreover my main criticism is the writing; I believe that its assessment deserves another round of reviews.
As such, I leave my score unchanged.
Thank you for revisiting our response and for recognizing the technical soundness of the work. We appreciate your feedback on novelty and writing.
Regarding novelty, Section 8 and Appendix A.8 clarify how our contribution differs from classical estimability, sequential c-optimal design, and prior compositional identification results: we connect a necessary-and-sufficient criterion for identifying a specified target predictive law to the Support-Induced Composition Radius (SICR) and its sample-size-independent minimax risk consequence.
As writing was your main concern, we would also like to highlight the changes in the revised PDF uploaded to OpenReview. We rewrote the Introduction around an intuitive three-factor example, clarified key terminology at first use, improved the explanation accompanying Figure 1, organized the theorem assumptions by proof direction, and revised the related work discussion to make the contribution and its relationship to prior work clearer.If any sections remain unclear, we would welcome specific pointers and will address them in the final version.
We appreciate your time and consideration of these revisions.
Thank you for the comments on clarity and the connection to familiar machine-learning concepts. We have rewritten the opening around a concrete example and defined the terminology at its first use. The central question is: does the training distribution uniquely determine the predictive distribution at a specified, possibly unseen combination of conditions?
The question in ordinary prediction terms. H is the observed context, Y is the outcome, and Z records a declared combination of factors. A factor is one coordinate of Z; a composition is their joint setting. For example, let z=(year 2012, rush hour, adverse weather), with target 111. These are observed operating conditions. The prediction target is an observational conditional distribution; causal effects would require additional assumptions.
Marginal forecasting predicts p(Y∣H), averaging over the observed mixture of conditions. Mechanism-resolved forecasting predicts p(Y∣H,z⋆) for a specified composition. Accurate estimation of the first need not determine the second.
Figure 1 illustrates the distinction in a pairwise interaction family. Training contains 000, 100, 010, 001, 110, and 101, so the interaction z2z3 is never activated. Two models can agree on all these training laws, the baseline at 000, and every single-factor distributional change, yet disagree at 111. Those single-factor changes are the atomic effects. Holding them and the factor meanings fixed defines the atom-preserving comparison. Under the stated reference and overlap assumptions, adding 011 to the training support supplies the missing interaction direction and identifies the target law while 111 remains unobserved.
What the theory contributes. SICR quantifies the disagreement among target distributions compatible with the same population training experiment and semantic anchors. Theorem 1 gives its worst-case excess-log-loss consequence for every training-only learner at every sample size. Appendix A.5 establishes an exact minimax value for a fixed equivalence class. Theorem 2 gives a necessary and sufficient interaction-span condition within the anchored family, and TSA turns the uncovered target direction into a support-selection rule.
The model permits interactions rather than only additive raw-factor effects. The pairwise example uses χ(z)=(z1z2,z1z3,z2z3); the theorem also permits higher-order known finite-dimensional maps satisfying its assumptions. Section 5 and Appendix A.6 state the reference, context-overlap, and local-admissibility conditions together.
Relation to model multiplicity, OOD prediction, and prior identification results. Rashomon sets and predictive multiplicity study disagreement among predictors with similar loss (Fisher et al., 2019; Marx et al., 2020). Here, the compared models agree on the entire observable training law, so additional data from that unchanged experiment cannot distinguish them. The question is partial identification of an OOD conditional distribution under a declared model family. This asks what information the training experiment supplies before a prediction architecture is chosen.
Classical estimability provides the row-space algebra. Bravo-Hermsdorff et al. (2023) study full-distribution reconstruction through PR transformations and discuss training-condition design. Our anchored-family construction connects explicit training-equivalent target disagreement to minimax prediction risk and target-specific support repair. Section 8 and Appendix A.8 set out these relationships and the respective modeling assumptions.
The controlled experiments show the predicted error plateau and the equal-budget benefit of support expansion. The 200-task study examines multiple repair choices, and the original Bike comparison reports 11.4% and 10.2% mean target-RMSLE improvements at 200 and 400 added records relative to the strongest non-targeted policy. The accompanying update summarizes these findings and the new precision and context-support analyses.