acceptodds
Under review as a conference paper at ICLR 2027

**StatOrch: Evidence-Conditioned Stagewise Inference Under Unknown Distribution Shift**

Abstract

Inference on an unlabeled target population requires deciding how to respond to distribution shift: correct class proportions, adapt a predictor, acquire measurements, or retain the current procedure. We present StatOrch, a framework connecting observable statistical evidence to these choices through stage-specific effect prediction and fixed fallbacks. It organizes established tools around population targets, dependent contexts, and resource constraints. Controlled evaluations show routing gains over validation-selected fixed actions. An executed adaptation–quantification–classification chain improves the joint population-and-prediction objective on seen mechanisms across four fitting seeds on a fixed cohort, with no corresponding advantage established on unseen mechanisms. A matched one-shot complete-pipeline selector attains similar losses; these results do not establish a stagewise-selection advantage. Separately, an archival measurement/resource instance with fixed downstream heads records 21.1% lower known-class conditional prevalence error, 5.26 percentage points higher Macro-F1, and 10.5% lower recorded measurement cost than a near-cost fixed policy on 1,232 held-out units, under its preserved ledger. Acquisition, re-entry, and real-sensor stress tests characterize operational behavior under their respective protocols. The controlled results support conditional selection and dependent execution within the evaluated settings.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
distribution shift, statistical decision-making, class prevalence estimation, active measurement acquisition, uncertainty quantification, cost-aware learning, adaptive inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_QzozOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StatOrch trains selectors on offline tasks to choose statistical methods from summaries of unlabeled target data. The executed pipeline adapts a predictor, estimates class proportions, and selects a classification rule, with a fixed fallback when expected gains are small. Other experiments separately evaluate measurement choices and a broader method pool.

Critical strengths
  1. The problem is stated clearly. Inference on an unlabeled target becomes a choice among prior correction, adaptation and acquisition, each tied to the evidence that supports it, with the current procedure kept as a fallback.
  2. The order of execution handles the dependency between stages well. Adaptation replaces the predictor and the reference a quantifier needs changes with it, so the proportions are recomputed afterwards and the joint objective is reported for the whole chain.
Critical weaknesses
  1. The added value of refreshing the evidence between stages remains unestablished. Tables 3 and 34 do not separate the two: the seen joint loss is 0.3309 for stagewise and 0.3300 for one-shot, the paired interval spans zero, and on unseen mechanisms stagewise shows no advantage. The reported comparisons therefore do not demonstrate a gain over one-shot selection that would justify the additional stages and evidence refreshes.
  2. The adaptation method is chosen by a criterion that the paper does not show to be aligned with the final objective. Equation (6) ranks candidates by a stage-local proxy, EMQ proportions plus known-class Macro-F1, which the paper states is not the terminal objective, while the later stages choose their own proportion estimator and classification rule. Whether the proxy's ranking agrees with the ranking under the final joint loss is not reported.
Questions for authors
  1. In Tables 3 and 34, is there a split or task group where stagewise selection gains enough over one-shot to justify refreshing between stages? The seen joint losses are 0.3309 and 0.3300.
  2. Do the recorded pipelines rank the adaptation candidates the same way under Equation (6) and under the final joint loss? Where the two differ, how large is the final gap?
  3. In Table 3 the greedy variants give 0.3234 for stagewise and 0.3237 for one-shot. Does that difference fall inside the paired interval, and does the ordering hold across the four fitting seeds?
Reviewer_4qA5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StatOrch, an evidence-conditioned framework for statistical inference under unknown distribution shift. It uses observable statistical features to predict the performance of candidate methods and select actions across measurement, adaptation, quantification, classification, and open-world analysis. The framework also incorporates stage dependencies, fixed fallbacks, and resource constraints. Experiments show that conditional selection can outperform validation-selected fixed actions, particularly on seen shift mechanisms, while stagewise and one-shot pipeline selection achieve comparable results.

Critical strengths
  • The paper provides a comprehensive integration of statistical tools into a structured workflow, with explicit consideration of observation permissions, dependencies between stages, fallback mechanisms, and resource constraints.

  • The experimental evaluation covers multiple statistical tasks and shift scenarios. The controlled comparisons show meaningful improvements from conditional selection over fixed actions, and the inclusion of a matched one-shot selector helps clarify the scope of the findings.

Critical weaknesses
  • The research motivation and methodological novelty are insufficiently established. The core mechanism relies on predicting candidate performance from observable task features and selecting the corresponding algorithm, which is closely related to established performance-based algorithm selection and meta-learning approaches. Although the paper discusses ASML (Gómez-Casares et al., 2025) and MetaML (Maia et al., 2025), it does not sufficiently establish what important limitation of existing approaches necessitates a separate framework. The distinction between the proposed evidence-conditioned inference and conventional feature-based algorithm selection also remains unclear. Consequently, the significance of the proposed framework beyond integrating established tools is difficult to assess.

  • The experimental results do not sufficiently establish the necessity of the proposed framework. The controlled experiments support the effectiveness of conditional selection over fixed actions, but this finding is largely consistent with the motivation of existing algorithm selection methods. More importantly, the matched one-shot selector achieves comparable performance to stagewise selection (Table 3), while improvements on unseen shift mechanisms are not established for the executed chain. Furthermore, no experiment jointly evaluates all stages of the proposed framework. These results demonstrate the usefulness of conditional selection in the evaluated settings, but provide limited evidence for the additional value of the proposed stagewise architecture.

Questions for authors
  1. What specific methodological limitations of existing algorithm selection and pipeline recommendation approaches, particularly ASML (Gómez-Casares et al., 2025) and MetaML (Maia et al., 2025), does StatOrch overcome?

  2. What additional predictive or decision-making capability is provided by the proposed evidence representation compared with using the original observable statistical features directly? Can the contribution of the learned diagnostic scores be isolated?

  3. Under what conditions does stagewise context updating provide an advantage over one-shot pipeline selection? Can this contribution be isolated under comparable supervision and computational budgets?

  4. What evidence supports the general applicability of the integrated framework beyond the evaluated seen shift mechanisms, particularly given the limited gains on unseen mechanisms and the absence of a complete joint evaluation of all stages?

Reviewer_EZh3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tackle the selection of high level strategies one could employ (e.g. active learning strategies, label shift estimator) when unlabeled data is present. The contribution seems to be the proposal of a pipeline that selects appropriate strategy. The main results seem to be on synthetic data in Table 1 and 2.

Critical strengths

None

Critical weaknesses
  • The paper misses critical details needed to assess the paper, e.g. what is the data behind Table 1, or what is EMQ and other baseline method the paper compares against
  • The paper aims to tackle a very complex meta-learning problem, but lack a careful discussion on appropriate dataset or control baselines for such studies. For example, it may be possible to study a subset of the stages on well known dataset like the adult census income dataset. But it's unclear why the specific synthetic dataset that the paper considers in Table 1 serves as a good testbed for all aspects of the pipeline.
  • As written, there's very little evidence that the entire pipeline that the paper proposes is actually effective. Again, due to the multiple stages involved, a careful ablation study with appropriate baseline methods must be considered to isolate the effects from different components of the pipeline.
Questions for authors
  • How is the synthetic data behind table 1 generated?
  • What are the isolated effects of each component in the StatOrch pipeline?
  • Most of the baselines appear to be weak baseline. Are there no related work that present an appropriate baseline to compare the method against?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.