acceptodds
Under review as a conference paper at ICLR 2027

EffectPreconception: Guiding ML Research Agents through Effect Prediction and Observation

Abstract

Autonomous machine-learning research agents optimize models through iterative code modifications, relying on scalar validation scores to guide exploration. A validation score reveals whether performance changed, but not whether the edit achieved its intended effect, leaving agents unable to separate flawed ideas from uncalibrated implementations. We introduce EffectPreconception, a framework that separates the predicted effect of an edit from its performance outcome. Before execution, the agent states an explicit, testable prediction about the observable behavior the edit should induce. When domain decompositions or accumulated observations support it, an outcome relation maps this predicted effect to an expected change in validation score. After execution, EffectPreconception checks the measured effect against the prediction and records the result in a persistent evidence ledger that informs every subsequent proposal, including evidence from discarded candidates. We evaluate EffectPreconception on Autoresearch, MLE-bench Lite, and PostTrainBench. On Autoresearch, it reduces mean best validation bits per byte from 0.9501 to 0.9404 across three runs with Sonnet 5. Applied to MLEvolve, it increases the mean gold-medal rate on MLE-bench Lite from 56.1% to 65.2% with Opus 5. These results demonstrate that evaluating targeted structural effects alongside validation performance provides substantially better guidance for autonomous research agents than score-only feedback or direct score prediction. Code: https://anonymous.4open.science/r/EffectPreconception-8022.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, AutoML, Self-improvement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_f9A2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EffectPreconception, a framework that augments autonomous ML research with explicit predictions about the observable effects of proposed edits. Before execution, the agent records an effect prediction and an associated measurement procedure. After execution, the framework compares the prediction with measurements, separately assesses any supported performance prediction, and stores the evidence in a persistent ledger. This distinguishes whether an edit behaved as intended from whether its behavior improved the final validation score. The framework is evaluated on Autoresearch, MLE-bench Lite, and PostTrainBench. It improves mean best Autoresearch validation BPB and MLE-bench gold-medal rates across repeated runs. An Autoresearch ablation also shows an advantage over retaining measured history without explicit effect predictions. The contribution is interesting, but the validation of diagnostic correctness and the independence of the PostTrainBench evaluation remain insufficiently convincing.

Critical strengths

1.The distinction between intended effects and performance outcomes is useful and well motivated. Scalar scores alone do not reveal whether an implementation failed to produce its intended behavior or whether the intended behavior was unhelpful. Recording predictions before execution and preserving evidence from discarded candidates gives agents a more informative basis for subsequent edits.

2.The paper provides meaningful empirical evidence beyond a score-only comparison. Autoresearch and MLE-bench Lite include three independent runs, and Figure 4(c) compares the full framework with a variant that retains measured history and outcome relations while removing effect predictions. The resulting BPB difference, 0.9404 versus 0.9465, supports an incremental benefit from explicitly recording expectations rather than merely collecting additional measurements.

Critical weaknesses

1.The diagnostic evaluation does not convincingly establish reliable scientific diagnosis. Figure 4(b) measures agreement with a predefined diagnosis/action rule rather than independently verified causal explanations or successful recovery. Appendix B.2 explains that the prediction-plus-score condition requests measurements because the prompt instructs it to do so when evidence is missing, making the 3% baseline difficult to interpret as diagnostic failure. The appendix also explicitly reports removing one case because reconciliation failed, introducing outcome-dependent selection. These limitations do not invalidate the online search gains, but they weaken the claim that reconciliation produces accurate diagnosis and actionable failure recovery.

2.PostTrainBench appears to use test scores as adaptive search feedback and for candidate selection. Appendix A.6.1 states that every candidate is evaluated on the full test set. Appendix A.6.3 then supplies parent and child scores to lesson and recommendation calls and retains candidates when their scores improve. Although the effect probes use a separate held-out split, the benchmark test scores themselves therefore participate in optimization. Without an untouched final evaluation, Table 3 measures performance under adaptive access to the test evaluator rather than clearly demonstrating independent generalization. This matters particularly because several reported gains are modest.

Questions for authors

1.Diagnostic-evaluation clarification: What are the results when all recorded cases, including the excluded reconciliation failure, are retained? How should the 3% baseline be interpreted given the explicit instruction to request missing measurements?

2.Feedback-validity clarification: Have free-form probes and reconciliation verdicts been independently checked for whether they measure the intended effect and support the assigned diagnosis? What is the error rate of this feedback?

3.Evaluation-protocol clarification: Are the full-test scores in PostTrainBench returned during search and used for retention, as Appendix A.6 appears to describe? Do you have results from selecting candidates with validation feedback and evaluating the final selection on an untouched test set?

Reviewer_dWRCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EffectPreconception, a framework that guides ML research agents by predicting the measurable effects of code modifications before execution and comparing them with actual observations afterward. It separates whether an edit achieves its intended effect from whether it improves validation performance, and stores the resulting evidence from both successful and failed experiments to guide future modifications. The method is evaluated on Autoresearch, MLE-bench Lite, and PostTrainBench, showing improvements over existing baselines across different ML optimization tasks.

Critical strengths
  • The idea of checking whether a modification produces its intended effect, rather than relying only on validation scores, provides more useful feedback for future experiments, especially when an experiment fails.
  • The method shows improvements across three different benchmarks, and the ablation studies help isolate the contribution of the proposed mechanism.
Critical weaknesses
  • The reliability of effect predictions and measurements is not sufficiently evaluated. Free-form measurement probes are generated by LLMs, and their results are also assessed using LLM-based judgments. However, it is unclear whether these probes reliably capture the intended effects of code modifications. The authors can conduct a robustness ablation by introducing inaccurate feedback and measuring its impact on subsequent experiments.

  • The contributions of different components are not fully isolated. The framework involves multiple components, but the individual contributions of outcome relations, reconciliation, and the evidence ledger remain unclear. More detailed ablation studies would help identify which components drive the improvements and potentially lead to a simpler, equally effective system.

  • The PostTrainBench evaluation raises concerns about generalization. As described in Appendix A.6.1, the agent repeatedly evaluates candidates on the full benchmark test set during optimization and reports the best retained scores. This creates a risk of overfitting to the evaluation benchmarks. The authors should evaluate the final models on independent held-out tasks to verify whether the improvements generalize beyond the benchmarks used during search.

Questions for authors

Please see the weaknesses section above

Reviewer_P48HOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that ML research agents relying on scalar validation scores cannot tell whether an edit failed because it did not produce its intended effect or because the effect occurred but did not help. These two cases call for different next steps. The authors therefore propose that, before execution, the agent commits to a testable prediction about an observable effect of each edit. After execution, the prediction is checked against measurements, and the result is carried into later proposals, including evidence from discarded candidates.

The contributions are as follows:

  1. A formulation of experimental feedback that separates an edit’s intended effect from its expected performance benefit.
  2. Experiments showing improvements on Autoresearch, MLE-bench Lite and PostTrainBench, plus an ablation suggesting that effect predictions add value beyond measured history.

Validation:

  1. Autoresearch (Table 1). With Sonnet 5 and Opus 5, the authors compare against Autoresearch, ScorePred, LLAMBO and FOREAGENT over 100 nodes with three runs per arm. They report the lowest final BPB at both sizes (Sonnet 5: 0.9501 → 0.9404; Opus 5: 0.9483 → 0.9368).
  2. MLE-bench Lite (Table 2). Applied to MLEvolve over 22 competitions with three runs, the authors report two gains: DeepSeek V4 Flash: any-medal rate rises from 51.5% to 63.6%; Opus 5: gold-medal rate rises from 56.1% to 65.2%.
  3. PostTrainBench (Table 3). Across four base models and seven benchmarks, the weighted average rises from 38.26 to 39.29, with 14 wins, 5 ties and 1 loss over Autoresearch on the non-judge benchmarks.
  4. Ablations (Figure 4). The authors report three results: higher proposal retention in the two architecture starting cases; an offline diagnosis study in which agreement rises from 3% to 65% to 74% as observations and then reconciliation are added; and removing effect predictions while keeping measured history raises final validation BPB from 0.9404 to 0.9465.
Critical strengths
  1. The framework extracts value from experiments that greedy search discards. In Appendix E.1, measurements from a discarded attention-gate candidate lead to a revision that beats the parent. This matters because it identifies a real failure of score-only feedback that the framework addresses, independent of the benchmark gains.
Critical weaknesses
  1. The experiments cannot isolate the effect of the predictions (Tables 1-3, Figure 4). The method adds diagnostics, a ledger, outcome relations and an extra deliberation step at once, but the only search ablation removes predictions while keeping the rest (0.9404 vs 0.9465 BPB, three runs, no significance tests). Separately, Table 1 implies node-1 BPBs that differ by about 0.007 across methods that share one starting program, well beyond the 0.0003 BPB noise the authors report.
  2. Novelty relative to BCO is unclear, and many held predictions are implementation checks. BCO (Chen et al., 2026) already checks pre-registered behavioral expectations against execution but is not compared against. Many held predictions (scales move off initialization, momentum follows the schedule the edit sets) are nearly guaranteed if the code runs as written. This matters because the contribution may be edit verification rather than hypothesis testing, which is a narrower claim.
Questions for authors
  1. Starting-point variance. Table 1 implies that the starting program’s BPB varies by about 0.007, both within methods (e.g. Autoresearch recovery SD 0.0076 vs node-100 SD 0.0013) and across them (0.9751 for Autoresearch vs 0.9818 for EffectPreconception with Sonnet 5). This is far above the 0.0003 BPB noise in Appendix A.4.4. How was that noise measured?
  2. Component controls. Can you add the following arms, all with the same node budget and three or more runs, with paired confidence intervals or significance tests for all arms in Table 1?
  • an arm with the same measurements and ledger but no predictions, verdicts or outcome relations (extending Figure 4(c));
  • a compute-matched arm that spends the same auxiliary LLM calls on generic reflection before each edit.
  1. BCO comparison. Concurrent work, BCO (Chen et al., 2026), also separates behavioral from aggregate predictions and checks them after execution. Can you compare against it on Autoresearch, or explain what EffectPreconception adds beyond it?
  2. Prediction type and quality. Can you label predictions as implementation checks (the edit directly sets the predicted quantity) or behavioral hypotheses, and report held rates for each? Do held rates and magnitude errors improve as the ledger grows, and how accurate are the outcome relations?
2 comments

Dear authors,

Thank you for the thorough response.

The added controls (Arms A and B) and significance tests are helpful.

Two brief clarifications:

  1. The BCO response notes that EffectPreconception’s configuration includes a scripted design sweep and compute checks absent from BCO. Do the Table 1 baselines and Arms A/B include the same sweep and checks, and what exactly do the compute checks do?
  2. The component-removal variants use modified context handling and search-control policies, while the full-method reference is taken from Table 1. Was the full method also run under the modified harness, and if not, what exactly differs?

Thank you for the clear answers.

Since the design sweep is shared by EffectPreconception, ScorePred and Arms A/B, those comparisons fairly test the added value of effect predictions and their assessment. This addresses my main concern, and I have raised my score to 3 accordingly.

For the revised paper, I would ask that you:

  1. State in Table 1 which methods include the sweep. This is especially important, since the comparisons with Autoresearch, LLAMBO, FOREAGENT and BCO, including the headline result in the abstract, differ in both the sweep and the prediction machinery.
  2. Describe the design sweep in the method section, and the compute checks in the implementation details.
  3. Note that the component-removal ablations use a modified harness without a matched full-method rerun.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.