- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EffectPreconception, a framework that augments autonomous ML research with explicit predictions about the observable effects of proposed edits. Before execution, the agent records an effect prediction and an associated measurement procedure. After execution, the framework compares the prediction with measurements, separately assesses any supported performance prediction, and stores the evidence in a persistent ledger. This distinguishes whether an edit behaved as intended from whether its behavior improved the final validation score. The framework is evaluated on Autoresearch, MLE-bench Lite, and PostTrainBench. It improves mean best Autoresearch validation BPB and MLE-bench gold-medal rates across repeated runs. An Autoresearch ablation also shows an advantage over retaining measured history without explicit effect predictions. The contribution is interesting, but the validation of diagnostic correctness and the independence of the PostTrainBench evaluation remain insufficiently convincing.
Critical strengths
1.The distinction between intended effects and performance outcomes is useful and well motivated. Scalar scores alone do not reveal whether an implementation failed to produce its intended behavior or whether the intended behavior was unhelpful. Recording predictions before execution and preserving evidence from discarded candidates gives agents a more informative basis for subsequent edits.
2.The paper provides meaningful empirical evidence beyond a score-only comparison. Autoresearch and MLE-bench Lite include three independent runs, and Figure 4(c) compares the full framework with a variant that retains measured history and outcome relations while removing effect predictions. The resulting BPB difference, 0.9404 versus 0.9465, supports an incremental benefit from explicitly recording expectations rather than merely collecting additional measurements.
Critical weaknesses
1.The diagnostic evaluation does not convincingly establish reliable scientific diagnosis. Figure 4(b) measures agreement with a predefined diagnosis/action rule rather than independently verified causal explanations or successful recovery. Appendix B.2 explains that the prediction-plus-score condition requests measurements because the prompt instructs it to do so when evidence is missing, making the 3% baseline difficult to interpret as diagnostic failure. The appendix also explicitly reports removing one case because reconciliation failed, introducing outcome-dependent selection. These limitations do not invalidate the online search gains, but they weaken the claim that reconciliation produces accurate diagnosis and actionable failure recovery.
2.PostTrainBench appears to use test scores as adaptive search feedback and for candidate selection. Appendix A.6.1 states that every candidate is evaluated on the full test set. Appendix A.6.3 then supplies parent and child scores to lesson and recommendation calls and retains candidates when their scores improve. Although the effect probes use a separate held-out split, the benchmark test scores themselves therefore participate in optimization. Without an untouched final evaluation, Table 3 measures performance under adaptive access to the test evaluator rather than clearly demonstrating independent generalization. This matters particularly because several reported gains are modest.
Questions for authors
1.Diagnostic-evaluation clarification: What are the results when all recorded cases, including the excluded reconciliation failure, are retained? How should the 3% baseline be interpreted given the explicit instruction to request missing measurements?
2.Feedback-validity clarification: Have free-form probes and reconciliation verdicts been independently checked for whether they measure the intended effect and support the assigned diagnosis? What is the error rate of this feedback?
3.Evaluation-protocol clarification: Are the full-test scores in PostTrainBench returned during search and used for retention, as Appendix A.6 appears to describe? Do you have results from selecting candidates with validation feedback and evaluating the final selection on an untouched test set?
Dear authors,
Thank you for the thorough response.
The added controls (Arms A and B) and significance tests are helpful.
Two brief clarifications: