acceptodds
Under review as a conference paper at ICLR 2027

Returned-Predictor Audit for Early Model Selection

Abstract

Early model selection briefly trains several candidates, continues training on a subset, and uses validation to select a saved state for prediction. An improvement over random choices can come from favoring stronger candidates overall (usage), or assigning them to contexts where they help more (matching). We introduce Returned-Predictor Audit (RPA), an offline evaluation that separates these gains and traces them to the states actually returned. From complete records for the compared training plans, RPA evaluates random assignment at the policy's frequencies, computes the best reassignment at those frequencies using recorded test losses, and varies final validation while keeping early choices fixed. In an external two-task partial differential equation (PDE) study, a preselected pair yields the same states as the six-model pool in all 120 cases, with 41% less reconstructed training and validation work. In a six-task PDE study, positive usage and negative matching estimates nearly cancel each other out. On LCBench, pairing one fixed default with a second model chosen for each dataset reduces classification error by 0.250 percentage points compared to random assignment at the same frequencies. Broader final selection changes both measured matching and the opportunity left among returned predictors: matching can decrease while the policy's loss gap to the best reassignment under the new rule narrows.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
PDE foundation models, early model selection, multi-fidelity optimization, validation, model portfolios, instance matching, learning curves

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_j4wkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Returned-Predictor Audit seems to look at early model selection after the training plans have already been run. It tries to separate “stronger models were picked more often” from “they were matched to the right dataset,” and it scores the checkpoint validation actually returns. On the held-out PDE tasks, a pre-chosen Poseidon pair appears to match a six-model pool in all 120 cases, with about 41% less reconstructed work. On LCBench the matching gain looks like 0.25 percentage points of error.

Critical strengths

A pair chosen in advance seems to return the same predictor as searching the whole pool. Changing only the final validation rule also seems to move both the policy and the benchmark it is scored against.

Critical weaknesses

The audit seems to need every route trained before it can say which routes were unnecessary. The remaining gains look hard to separate from noise, and always training the strong model appears to beat the adaptive rule.

Questions for authors
  1. Why run the audit only after every route has already been trained, rather than a sequential rule that can drop a route once validation has ruled it out?
  2. Why score the adaptive rule against the full pool, rather than against always training the single strong model?
Reviewer_yYD7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces the Returned-Predictor Audit (RPA), an offline evaluation framework for analysing early model selection under limited training budgets. It distinguishes gains from usage (selecting stronger candidates more often) and matching (assigning candidates to the tasks where they are most useful), while accounting for the final predictor actually returned. Experiments on partial differential equation (PDE) models and LCBench show that these effects can differ substantially, with matching providing a modest improvement on LCBench but negative gains in one PDE study. The authors also identify opportunities to reduce computational costs by eliminating redundant training plans. Overall, the paper offers a useful analytical framework, although the generality and practical significance of its findings remain uncertain.

Critical strengths
  • Interesting analytical framework: the distinction between usage and matching provides a useful perspective on understanding the sources of gains in early model selection, showing that improved performance does not necessarily imply effective task-specific matching.
  • Extensive empirical evaluation: experiments across PDE modelling and LCBench show that the relative contributions of usage and matching vary across settings; matching can even have a negative effect, while redundant training plans can sometimes be eliminated without changing the returned predictor.
Critical weaknesses
  • the distinction between usage and matching is intuitively appealing, but its novelty relative to existing work on model selection, model portfolios and resource allocation is not fully established.
  • RPA is an offline audit requiring complete records for the candidate training plans; the oracle reassignment additionally relies on test losses and is therefore not deployable in practice.
  • the 41% reduction in training and validation work is reconstructed for a specific setting, and it remains unclear how reliably redundant training plans can be identified a priori.
Questions for authors

How does RPA advance beyond existing approaches to model selection and resource allocation, and to what extent can its offline estimates of matching gains and computational savings be translated into practical benefits when evaluating new tasks?

Reviewer_XVUiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Instead of choosing models randomly, they take an approach that is based on classical controls results (without mentioning that this is an old controls technique) and shows that it could pick good models on a PDE, though using PDEs here is really just to get SciML brownie points because the PDE's properties aren't used in any way, the dataset/benchmarks could've been anything.

Critical strengths

The benchmarks are pretty favorable in the exact case they were looking at.

Critical weaknesses

The general point of the whole topic is rather trivial. You're just predicting which models would do well... doesn't everyone do that? That's just an MoE. It's both simple but also hard to read.

Questions for authors

Why is this better than every MoE method that came before? Benchmark the 50 of them and show the full result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.