acceptodds
Under review as a conference paper at ICLR 2027

SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness

Abstract

As coding agents move from supervised code completion to unattended, around-the-clock exploration, their work expands from isolated predictions into long trajectories of reasoning, tool use, and feedback. Token efficiency therefore becomes important for scaling recursive self-improvement. We take an RSI-inspired approach at the harness layer, scaling auto-research loops across increasingly numerous and diverse environments for harness rollouts. At this scale, the process yields reusable improvements that transfer beyond their development setting, moving automated harness discovery toward production-level outcomes. Four mechanisms survive selection and form SoL-Pi (SoL stands for Speed of Light), spanning action execution, context compaction, observation handling, and delegated reading. On the 51-task EdgeBench evaluation, SoL-Pi achieves performance comparable to Pi-baseline across GPT-5.6 Sol and Opus 5 while reducing recorded token traffic by 44.7-49.0% and API cost by about one third. In other words, estimated hourly savings are $8.75-13.50 relative to native Codex and Claude Code harnesses, and $4.36-5.71 versus Pi-baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Recursive Self-Improvement, Coding Agents, Agent Harness Optimization, Harness Efficiency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nvnWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SoL-Pi, which optimizes the agent harness through an automated research pipeline, reducing token consumption and operating costs without modifying the underlying model. The approach integrates four mechanisms—action fusion, context compaction, tool-output management, and evidence-preserving log compression—and evaluates its efficiency gains and generalization capability through experiments across multiple models and task settings, complemented by single-component analyses and a development case study.

Critical strengths
  1. The paper addresses a practically important problem and proposes concrete, reusable mechanisms.
  2. The experiments cover multiple models and task settings.
Critical weaknesses

My main interest is in the automated research process itself, but its contribution is not sufficiently established. Most experiments evaluate the final harness, whose mechanisms address optimization directions already explored in prior work. Although the Action Fusion case study provides some insight, the paper offers limited analysis of how candidate quality and cost evolve across iterations, how unsuccessful directions are discarded, and how budgets and stopping decisions are determined. It also lacks controlled comparisons that isolate the benefits of broader candidate exploration, more diverse environments, deeper refinement, or independent review. Without comparisons against simpler search procedures under comparable budgets, it remains unclear which aspects of the proposed loop drive the improvements and whether the additional complexity and search effort are justified.

The central contribution and the effectiveness of the auto-research loop need clearer justification. I am uncertain whether the paper’s primary contribution is a more effective automated research process or the four resulting harness optimization mechanisms. Action fusion, context compression, and tool-output management are already established directions in prior research and engineering practice, making it important to distinguish the novelty of the mechanisms from the contribution of the process used to discover them.

Questions for authors
  1. What is the main contribution: the auto-research loop, the four harness mechanisms, or their combination? How do these mechanisms differ from existing approaches? Beyond the model’s pretraining knowledge, did the search use external papers, code, or human guidance, and how did these sources influence candidate generation and selection?
  2. How does the overall search improve candidates over time? Beyond the Action Fusion case study, could existing logs show how candidate quality and cost evolve, why unsuccessful directions are discarded, and how budget allocation and stopping decisions are made?
  3. What evidence supports the benefits of the loop’s design? Are there comparisons or intermediate results showing that broader exploration, more diverse environments, deeper refinement, or independent review improve outcomes relative to a simpler procedure under comparable budgets?
  4. What was the total cost of discovering the final harness? Please clarify the computational and API costs of proposal generation, implementation, review, evaluation, and integration, so that the efficiency of the research process itself can be assessed.
Reviewer_zx3kOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies reducing agent execution costs through automated harness modification. Starting from Pi, research agents analyze development trajectories, implement candidate changes, and evaluate them under capability and efficiency constraints. The search produces four mechanisms that reduce model round trips and repeated processing of context and tool outputs. Evaluations cover multiple model backends, held-out benchmarks, and a multi-agent setting. The complete stack substantially lowers token usage and API costs with some performance degradation, while selected individual mechanisms improve both performance and efficiency.

Critical strengths
  1. The experiments cover multiple benchmarks, model backends, and a multi-agent setting. Component evaluations and token breakdowns also help explain how the mechanisms affect task performance and execution costs.

  2. The method builds on the mature Pi harness and achieves substantial cost reductions. The fixed mechanisms also reduce costs on Opus 5 without further search or adaptation, providing evidence that the improvements have practical value beyond the original development setting.

Critical weaknesses
  1. The motivation for using auto-research to improve agent efficiency is not sufficiently clear. Meta-Harness [1], AHE [2], and DGM [3] already improve harnesses or agents through task execution, feedback analysis, and code modification. DGM also explores multiple branches, while Meta-Harness considers both performance and context cost. This paper follows a similar feedback-driven process, with efficiency as its main objective. However, it does not clearly explain what difficulties existing harness evolution methods face when optimizing efficiency, or how the proposed auto-research approach addresses them.

  2. The paper lacks sufficient discussion and comparison with closely related work, particularly AHE [2], which it already cites. AHE likewise analyzes execution trajectories, identifies problems, modifies harness components, and evaluates transfer across tasks and models. The brief discussion does not explain how the two methods differ in their use of feedback, modification procedures, or candidate evaluation. The experiments mainly compare the resulting SoL-Pi harness with systems such as Codex and Pi, rather than with evolution methods such as AHE. It is therefore difficult to assess what additional benefit the auto-research process provides over existing approaches.

  3. The paper aims to substantially reduce agent execution costs while keeping performance degradation within an acceptable range. The authors conduct extensive auto-research at the harness level and obtain four mechanisms, but their advantage over directly adjusting the LLM’s thinking effort remains unclear. For example, reducing GPT’s reasoning effort from xhigh to medium may also trade some performance for lower costs, without additional search or development effort. Without this comparison, it is difficult to judge whether the improvements obtained through auto-research justify the effort involved. I would also appreciate a report of the total time and API costs of the auto-research process.

  4. The complete stack shows some performance degradation across both primary backends and evaluations such as Terminal-Bench 4. Although this degradation is acceptable relative to the substantial cost reductions, its causes remain unclear. Could the authors provide case studies comparing execution trajectories before and after introducing the mechanisms to help explain these declines? Such analysis would help readers understand how the mechanisms affect agent behavior and under what circumstances they may hurt task performance.

[1] Meta-Harness: End-to-End Optimization of Model Harnesses

[2] Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

[3] Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents

Questions for authors

See Weaknesses.

Reviewer_XQDsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies automated harness optimization for token efficiency. Starting from the open-source Pi harness, LLM research agents analyze execution trajectories in about 535 coding and synthetic environments (GitHub issue–PR tasks and synthetic verifier-based tasks), then propose, implement, and filter harness changes. A change is kept only if it stays within predefined capability tolerances and improves an efficiency metric. Four surviving mechanisms—Action Fusion, Online Context Compact, ObservationPack, and Evidence-Preserving Reducer—are integrated into vanilla Pi to form SoL-Pi. The harness is then frozen and evaluated on EdgeBench, a held-out long-horizon benchmark. In comparison to the Pi-baseline, the Efficiency setting reduces API cost by about one third on GPT-5.6 Sol and Opus 5 while retaining about 94% of the average score. Additional results cover Terminal-Bench 4, IMO 2026, and a multi-agent setting.

Critical strengths

S1. Optimizing long-horizon agents at the harness level needs no model training and is orthogonal to model- and serving-side efficiency work. For the Efficiency stack, the roughly 33% API-cost reduction holds on both GPT-5.6 Sol and Opus 5. S2: Metrics and capability tolerances are fixed before search; mechanisms are discovered on separate environments; and the harness is frozen before held-out evaluation. This protocol directly targets the known risk that evolved harnesses overfit their search tasks. S3: The token-category breakdown (input / cache read / cache write / output) and the discussion of compaction versus prompt-cache reuse make clear where the savings come from.

Critical weaknesses

W1: Efficiency trades accuracy for tokens: relative to Pi-baseline, EdgeBench average score is only about 94% retained, so the gains are not lossless. W2: Held-out generalization has a cost: on Terminal-Bench 4, solved tasks fall from 18 to 15, so the stack does not transfer without capability loss. W3: Reproducibility and transfer are limited: the search is expensive and hard to reproduce; mechanisms found mainly on GPT-5.6 Sol trajectories trigger less often and less strongly on Opus 5.

Questions for authors

Q1. What are the exact numerical capability tolerances used during search (metrics, thresholds, and how “comparable to Pi-baseline” was operationalized)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.