- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Returned-Predictor Audit seems to look at early model selection after the training plans have already been run. It tries to separate “stronger models were picked more often” from “they were matched to the right dataset,” and it scores the checkpoint validation actually returns. On the held-out PDE tasks, a pre-chosen Poseidon pair appears to match a six-model pool in all 120 cases, with about 41% less reconstructed work. On LCBench the matching gain looks like 0.25 percentage points of error.
Critical strengths
A pair chosen in advance seems to return the same predictor as searching the whole pool. Changing only the final validation rule also seems to move both the policy and the benchmark it is scored against.
Critical weaknesses
The audit seems to need every route trained before it can say which routes were unnecessary. The remaining gains look hard to separate from noise, and always training the strong model appears to beat the adaptive rule.
Questions for authors
- Why run the audit only after every route has already been trained, rather than a sequential rule that can drop a route once validation has ruled it out?
- Why score the adaptive rule against the full pool, rather than against always training the single strong model?