- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes FORGE, an evidence-governed framework for automated algorithm discovery. It maintains structured records of execution outcomes, failure causes, counterexamples, and resource usage to guide iterative modifications. The framework combines failure attribution, targeted interventions, and verification-based admission to improve the reliability of algorithm discovery.
Critical strengths
1.The research problem is meaningful, and using structured evidence and failure attribution to guide algorithm discovery is an interesting idea.
2.The framework provides a systematic workflow that integrates failure analysis, targeted interventions, and verification.
Critical weaknesses
1.Limited improvements over the strongest baseline. FORGE improves the average hidden score from 0.892 to 0.910 compared with AI-Scientist v1. Moreover, the improvements are primarily concentrated in empirical-rule discovery, while gains on other task families are marginal or negative. This raises concerns about the general effectiveness of the proposed framework.
2.Insufficient component-level ablations. Although the paper includes preliminary ablation experiments, they cover only two task families. The individual contributions of evidence memory, failure attribution, counterfactual probes, and bounded admission are not sufficiently established across the full benchmark.
3.Limited generalization to realistic algorithm discovery. The main evaluation focuses on relatively small synthetic tasks using a single base model. Although additional transfer studies are presented, they are exploratory and do not provide sufficient evidence for broader applicability to complex, real-world algorithm discovery.
4.Additional computational overhead. FORGE requires more public executions and longer wall-clock time than AI-Scientist v1 (165.7 vs. 128.2 seconds). Given the modest improvement in hidden scores, its computational cost-effectiveness remains unclear.
Questions for authors
1.How do the authors justify the general effectiveness of FORGE given that improvements are concentrated in one task family?
2.Can the authors provide more comprehensive ablation studies to isolate the contributions of the core components?
3.Can the authors validate FORGE on more complex real-world tasks and additional base models?
4.Can the authors provide a more detailed analysis of the performance–cost trade-off compared with existing baselines?