acceptodds
Withdrawn from ICLR 2027

EvidLoop: Staged Verification for Agentic Model Search in Aerodrome Weather Forecasting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language-model agents increasingly turn research hypotheses into executable model edits, but improvements observed during adaptive search may not persist across training seeds or evaluation conditions. We introduce EvidLoop, a framework that links each evaluated implementation to staged acceptance criteria and separates development feedback from final assessment. Candidates advance through single-seed screening, paired five-seed confirmation, and one-time assessment of frozen models on temporal and spatial holdouts, including a predefined weather-event subset termed HAZARD. On the AeroWF aerodrome-weather benchmark, three language models explore four intervention streams, yielding 70 scored attempts, 24 screening survivors, and 11 confirmed candidates. The confirmed edits reduce development RMSE by 1.22%–3.76% relative to the parent's mean development RMSE. Seven satisfy a joint mean-based acceptance rule, fixed before held-out evaluation, that requires temporal improvement and limits spatial and HAZARD-conditioned degradation. Five improve all three held-out means. Four of the eleven confirmed candidates improve temporal and spatial means but exceed the HAZARD margin, including the candidate with the largest temporal gain. Accepted edits span error-dependent reweighting, horizon-conditioned robust penalties, and adaptive residual scaling. These results identify useful forecasting modifications and show that cross-seed development gains and aggregate held-out improvements can coexist with conditional regressions. EvidLoop makes the resulting acceptance decisions explicit and traceable to the implementations evaluated.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Agentic AI, Large Language Models, Machine Learning Agents, Automated Research, Model Improvement, Scientific Discovery, Reliable Machine Learning, Weather Forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xc8ZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper starts by questioning actual automated agentic workflows which generally tries to validate any hypothesis by single seed validation. The authors argue that this verification is not trustable at all and can change for different seeds. They propose instead to perform as in cross validation, different seed verification (i.e. in the case of automated data science for example train the model with different seed to evaluate its efficiency instead of one seed). Besides this seed instability the authors propose a method for avoiding test leakage by only using what they called development data during the choices. On a specific setting (HAZARD), they define a joint acceptance rule (a list of rules to accept the proposal of the agents).

Critical strengths
  1. The problem of variability of agent is a real problem worth solving.
Critical weaknesses
  1. The solution to solve the variability of the agentic workflow is brute force. The agentic workflow is already heavy and adding a layer where we need to make several executions make the running time not accountable. The compute time is discussed in appendix and reveal a strong drawback of the approach which is heavy to compute and mainly a brute force approach. One may wonder why not use other efficient search like Bayesian optimization, genetic algorithm and even more modern ones. This approach is not scalable for large models that needs to be trained 2) The experiment scope is surprisingly low. Why to choose Aerodrome Weather forecasting which is not that standard in the agentic benchmarking. MLE Bench and other benchmarks are much more adapted and may suit the idea. Also it will be easier to see how the proposed approach can enhance existing workflow pipelines like AIDE, ...
Questions for authors

1- How do you think this method is scalable. For example, is the method can be added on top of existing agentic workflow as a supplementary verification layer for example AIDE. What is the supplementary cost needed and can they run on existing benchmarks (MLE-Bench)?

2- Did you compare the brute force which consists to validate a step with other more efficient approach which wouldn't sample too many edits across different and will use efficient one (like in the style of Bayesian optimization or genetic algorithm or efficient verification layer instead of the brute force one) ?

Reviewer_tw6WOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a framework for staged verification based agentic search. The paper applied the proposed method on a real-world dataset AeroWF to demonstrate its performance.

Critical strengths
  • The paper tested the proposed framework on a real-world dataset AeroWF
Critical weaknesses
  • The paper does not include strong baseline methods to compare with the proposed framework. So I am not sure how significant the reported RMSE reduction (1.22%–3.76%) are.
  • The paper only focused on one specific dataset. It is not convincing how generalizable this proposed framework is.
  • The proposed framework itself does not introduce any groundbreaking ideas or techniques, rather the proposed methods are incremental.
Questions for authors
  • I noticed the three AI models used in the experiments are all from China. Did the authors also try other AI models?
Reviewer_SedfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EvidLoop, a framework for agentic model search that integrates staged verification with traceable candidate evaluation. It employs single-seed screening, five-seed confirmation, and held-out assessment across temporal, spatial, and hazardous weather conditions. Experiments on the AeroWF benchmark yield 11 confirmed candidates from 70 attempts, of which seven satisfy the final acceptance criteria. The results demonstrate that improvements across training seeds and aggregate evaluation conditions do not necessarily guarantee robust performance under hazardous weather conditions.

Critical strengths

S1. The paper proposes EvidLoop, a structured framework that integrates development feedback with staged verification for agentic model search.

S2. The paper evaluates EvidLoop on the AeroWF benchmark, which contains real-world weather observations from four airports.

S3. The paper evaluates model performance across temporal, spatial, and hazardous weather conditions, providing insights into generalization under different evaluation settings.

Critical weaknesses

W1. Figure 1 and Sec. 3.2 specify what the development feedback contains, but not how it is rendered into the prompt: how history is selected, truncated, or summarized across streams. Table 24 suggests these choices matter. The rendered prompts were also not preserved (App. F.3, Table 28), so the feedback the agents actually received cannot be verified. Moreover, the lineage control shows no consistent benefit from history (Table 23), leaving the effectiveness of the feedback loop unsubstantiated.

W2. Key components lack ablations. It is unclear whether staged verification outperforms simpler selection strategies under a matched compute budget, especially since the refuted candidates are never evaluated on held-out data. Beyond the HAZARD-margin analysis, sensitivity to the screening threshold, the confirmation criteria, and the number of seeds is not explored.

W3. The configurations of DeepSeek, GLM, and Kimi are not fully documented. Temperature and reasoning settings change across campaign segments, and retry and timeout settings are unavailable (Table 27). This limits reproducibility and confounds comparisons across lineages.

W4. The robustness of the search to inaccurate or misleading feedback is not investigated. For example, #873, #883, and #885 were proposed after #871’s promising screening result but before its refutation (Table 22). A feedback-corruption experiment would help assess the reliability of the closed-loop search.

W5. The HAZARD slice is thinly supported: ZSPD and ZSSS each contribute only one event block, yet each receives one quarter of the aggregate weight (Table 6). Among the four rejected candidates, only #982 has an interval excluding zero (Table 17), so the central claim about conditional regressions is weakly supported.

W6. The fixed-weight baseline (3,1,1) already outperforms two confirmed candidates (#741, #939) on development RMSE, yet it is not evaluated on the held-out sets. It is therefore unclear whether the discovered objectives offer benefits beyond simple reweighting.

Questions for authors

Please see details in Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.