- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper starts by questioning actual automated agentic workflows which generally tries to validate any hypothesis by single seed validation. The authors argue that this verification is not trustable at all and can change for different seeds. They propose instead to perform as in cross validation, different seed verification (i.e. in the case of automated data science for example train the model with different seed to evaluate its efficiency instead of one seed). Besides this seed instability the authors propose a method for avoiding test leakage by only using what they called development data during the choices. On a specific setting (HAZARD), they define a joint acceptance rule (a list of rules to accept the proposal of the agents).
Critical strengths
- The problem of variability of agent is a real problem worth solving.
Critical weaknesses
- The solution to solve the variability of the agentic workflow is brute force. The agentic workflow is already heavy and adding a layer where we need to make several executions make the running time not accountable. The compute time is discussed in appendix and reveal a strong drawback of the approach which is heavy to compute and mainly a brute force approach. One may wonder why not use other efficient search like Bayesian optimization, genetic algorithm and even more modern ones. This approach is not scalable for large models that needs to be trained 2) The experiment scope is surprisingly low. Why to choose Aerodrome Weather forecasting which is not that standard in the agentic benchmarking. MLE Bench and other benchmarks are much more adapted and may suit the idea. Also it will be easier to see how the proposed approach can enhance existing workflow pipelines like AIDE, ...
Questions for authors
1- How do you think this method is scalable. For example, is the method can be added on top of existing agentic workflow as a supplementary verification layer for example AIDE. What is the supplementary cost needed and can they run on existing benchmarks (MLE-Bench)?
2- Did you compare the brute force which consists to validate a step with other more efficient approach which wouldn't sample too many edits across different and will use efficient one (like in the style of Bayesian optimization or genetic algorithm or efficient verification layer instead of the brute force one) ?