- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SoL-Pi, which optimizes the agent harness through an automated research pipeline, reducing token consumption and operating costs without modifying the underlying model. The approach integrates four mechanisms—action fusion, context compaction, tool-output management, and evidence-preserving log compression—and evaluates its efficiency gains and generalization capability through experiments across multiple models and task settings, complemented by single-component analyses and a development case study.
Critical strengths
- The paper addresses a practically important problem and proposes concrete, reusable mechanisms.
- The experiments cover multiple models and task settings.
Critical weaknesses
My main interest is in the automated research process itself, but its contribution is not sufficiently established. Most experiments evaluate the final harness, whose mechanisms address optimization directions already explored in prior work. Although the Action Fusion case study provides some insight, the paper offers limited analysis of how candidate quality and cost evolve across iterations, how unsuccessful directions are discarded, and how budgets and stopping decisions are determined. It also lacks controlled comparisons that isolate the benefits of broader candidate exploration, more diverse environments, deeper refinement, or independent review. Without comparisons against simpler search procedures under comparable budgets, it remains unclear which aspects of the proposed loop drive the improvements and whether the additional complexity and search effort are justified.
The central contribution and the effectiveness of the auto-research loop need clearer justification. I am uncertain whether the paper’s primary contribution is a more effective automated research process or the four resulting harness optimization mechanisms. Action fusion, context compression, and tool-output management are already established directions in prior research and engineering practice, making it important to distinguish the novelty of the mechanisms from the contribution of the process used to discover them.
Questions for authors
- What is the main contribution: the auto-research loop, the four harness mechanisms, or their combination? How do these mechanisms differ from existing approaches? Beyond the model’s pretraining knowledge, did the search use external papers, code, or human guidance, and how did these sources influence candidate generation and selection?
- How does the overall search improve candidates over time? Beyond the Action Fusion case study, could existing logs show how candidate quality and cost evolve, why unsuccessful directions are discarded, and how budget allocation and stopping decisions are made?
- What evidence supports the benefits of the loop’s design? Are there comparisons or intermediate results showing that broader exploration, more diverse environments, deeper refinement, or independent review improve outcomes relative to a simpler procedure under comparable budgets?
- What was the total cost of discovering the final harness? Please clarify the computational and API costs of proposal generation, implementation, review, evaluation, and integration, so that the efficiency of the research process itself can be assessed.