- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces BeliefOPT, a framework for autonomous research agents that converts experimental experience into structured, revisable beliefs and uses these beliefs to guide subsequent exploration. The authors instantiate the framework in GPU kernel optimization and evaluate it on 26 SOL-ExecBench kernels on NVIDIA B200, reporting improvements over several evolutionary coding agents under matched token budgets.
Critical strengths
1. Carefully controlled optimization evaluation.
The paper provides a relatively detailed experimental setup with matched backbone models, per-task token budgets, and full-workload validation, making the reported GPU kernel optimization results reasonably reproducible.
2. Detailed analysis of the optimization process.
The paper reports additional analyses beyond the final benchmark score, including belief lifecycle statistics and budget-dependent experiments.
Critical weaknesses
1. The effectiveness of the belief mechanism is not sufficiently validated.
The only direct ablation of the belief store is conducted in the small-budget setting (six tasks, 2.4M tokens), where BeliefOPT achieves 24.1 ± 3.1× compared with 24.5 ± 1.5× without the belief store (p ≈ 0.8), showing no statistically significant benefit. The larger-budget experiments compare different belief structures but do not include a corresponding no-belief baseline.
2. The experimental evidence is insufficient to support the broader claims.
The evaluation is limited to a single benchmark (SOL-ExecBench) and one domain (GPU kernel optimization), without demonstrating generalization to other research settings. The overall improvement over CodeEvolve is also modest (5.7%) and sensitive to specific workloads: excluding the two MLA tasks reduces the advantage to approximately 0.9% (calculated from Table 4). Moreover, no belief reaches the "resident" state during the evaluation, leaving the benefits of persistent, validated belief reuse unverified.
Questions for authors
-
The paper reports that none of the 92 beliefs reaches the "resident" state during the entire evaluation. Although the residency criteria are described, could the authors explain why no belief satisfies them and what this implies for the effectiveness of the proposed persistent belief mechanism?
-
Appendix B.1 states that the reference implementation initializes the incumbent and that promotion requires at least a 0.5% improvement. However, Table 4 reports only 0.80× for BeliefOPT on Task 005. How can this be explained? Furthermore, several baselines retain solutions with speedups below 1.00×. Although Appendix A attributes some OpenEvolve regressions to subset screening, why are reference fallback and candidate selection handled differently across methods? Are the comparison protocols consistent and fair?
-
Why does Table 2 omit the high-budget ShinkaEvolve result, especially given that ShinkaEvolve achieves the best performance in the small-budget setting?
Since I have not received the rebuttal, I will retain my original score for the time being.