- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes RelEvolve, a framework for automated LLM harness optimization that maintains a persistent Inquiry Graph alongside the harness evolution history. A Curator updates the inquiry state and selects a parent harness, while an Evolver generates new candidates based on the selected parent and active inquiries. The paper reports improvements over several prompt and harness optimization baselines across five benchmarks and two LLM backbones.
While the empirical results appear promising, I have substantial concerns regarding the clarity of the problem formulation, the technical novelty of the proposed approach, the fairness of the experimental comparisons, and the generalizability of the method.
Critical strengths
The paper tries to address an interesting problem in automated agent optimization.
Critical weaknesses
The paper is difficult to follow due to excessive, unclear definitions. What exactly are "unresolved improvement directions"? The motivation is also weak. What specific limitations of existing harness optimizers does RelEvolve address?
The technical novelty is unclear. How does RelEvolve fundamentally differ from existing feedback-driven optimization methods? The separation between Curator and Evolver also seems unnecessary. Why are two separate roles needed? The ablations do not fully justify the proposed complexity. Would a simpler memory structure achieve similar results?
The experimental comparison raises fairness concerns. What stopping criteria are used for each baseline, and are their total token budgets comparable? Finally, the generalizability of the method remains unclear. Can the same prompts and hyperparameters work on unseen benchmarks without task-specific tuning?
Questions for authors
See my weakness