- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the generalization problem of harness evolution for LLM agents.
It proposes ModularRSI, a method that decomposes harness evolution into agent loop, tool use, observation management, context management, and task completion detection. It evolves each module independently and the independent evolved modules are integrated to construct the final harness.
Experiments evolve the harness on self-curated TerminalBench-like and SWE-like tasks and evaluate it on TerminalBench (TB) 2.0 and SWE-Bench-Verified, with DeepSeek-V4-Flash-Preview as the backbone. The evolved harness improves over the unevolved baseline and outperforms existing harness evolution methods under matched rollout budgets.
Critical strengths
- the generalization problem of harness evolution is important, and the proposed method is well-motivated.
- The ablations (Table 6) are comprehensive.
Critical weaknesses
- The method is complex and involves multiple rounds of judgment and verification. I suggest adding a cost analysis that accounts for these steps. Is this modular decomposition still necessary for stronger models? It would be helpful to evaluate stronger models and check whether the modular approach still performs best.
- The paper lacks details on the self-curated SWE-related and TB-related tasks. It is unclear whether these tasks are of high quality and to what extent they differ from TerminalBench 2.0 and SWE-Bench-Verified.
- The comparison with existing harness evolution methods is limited. The paper would be stronger if Meta-Harness and AHE were evaluated on the full set of settings in Table 1.
Questions for authors
- Why is the method called ModularRSI? A name like ModularHarnessEvolution seems more accurate.
- Were the evolution conducted with a single random seed?
4 comments
Response (2/N)
Q5: Why is the method called ModularRSI?
We use the term Recursive Self-Improvement (RSI) because our framework iteratively analyzes the agent's own execution trajectories, identifies deficiencies in its harness, and modifies the harness to improve subsequent executions. The updated harness is then used to generate new trajectories, forming a recursive feedback loop in which the agent continuously improves its own execution mechanisms.
Unlike parametric self-improvement, which updates the underlying model weights, our approach performs harness-level self-improvement without modifying the foundation model. The term Modular further emphasizes our core methodological contribution: decomposing the harness into functional modules and evolving them independently before cross-module integration. Therefore, we believe ModularRSI accurately reflects both the recursive self-improvement process and the modular evolution strategy underlying our framework.
Q6: Were the evolution experiments conducted with a single random seed?
The randomness in ModularRSI primarily arises from trajectory generation during evolution. Specifically, for each module, we generate three independent execution trajectories per task without enforcing a fixed or shared random seed. This stochastic sampling allows the evolution process to observe diverse execution behaviors and identify recurring harness deficiencies, rather than relying on a single trajectory. As a result, harness modifications are guided by patterns observed across independent rollouts, which helps reduce sensitivity to individual execution outcomes.
Due to the computational cost, we conduct one complete evolution run per experimental setting rather than repeating the entire evolution process with multiple random seeds. We will clarify this distinction in the revised paper.
Follow-up on Rebuttal Responses and Remaining Concerns
Dear Reviewer QqXL,
Thank you very much for your time and effort in reviewing our paper and for your valuable feedback.
As the rebuttal deadline is approaching, we would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns. If any questions or concerns remain unresolved, please feel free to let us know at your earliest convenience so that we can provide further clarification before the deadline.
Thank you again for your thoughtful comments and engagement in the review process.
Thank you so much for the detailed response. I’m leaning toward keeping my score.
Response (1/N)
We thank the reviewer for the constructive comments and questions. Our point-by-point responses are provided below. We will incorporate these clarifications and additional experimental results into the revised paper.
Q1. Cost analysis and evaluation using a strong model The LLM token consumption of ModularRSI primarily comes from six stages: Trajectory Collection, Contrastive Trajectory Analysis, Harness Modification, Diff Review, Function Library Management, and Cross-Module Integration.
We compare the total token consumption of ModularRSI with that of Meta-Harness and AHE, and provide a breakdown of ModularRSI's token costs below. All values are reported in millions of tokens (M).
ModularRSI consumes fewer tokens than both comparison methods. Trajectory Collection accounts for the majority of its computational cost, while the additional stages for trajectory analysis, harness modification, validation, and integration contribute a comparatively smaller overhead.
Q2. Is this modular decomposition still necessary for stronger models?
To investigate whether modular decomposition remains beneficial with a stronger foundation model, we use Claude Opus 4.6 as the backbone model for harness evolution and compare ModularRSI with joint all-module evolution. The results on Terminal-Bench 2.0 (TB2.0) are shown below.
ModularRSI improves accuracy by 2.99 percentage points over the non-evolution baseline and by 4.50 percentage points over joint all-module evolution. In contrast, jointly evolving all modules reduces performance below the non-evolution baseline. These results suggest that independent module evolution remains beneficial even with a stronger foundation model, as it helps avoid the performance degradation observed when all modules are optimized jointly.
Q3. Details of the self-curated SWE-related and TB-related tasks
Please refer to our response to Q1 from Reviewer a7Es, where we provide further details on the construction and characteristics of the self-curated evolution tasks.
Q4. Comparison with existing harness evolution methods
To further address the reviewer's concern, we conducted a more comprehensive comparison with Meta-Harness and AHE under the experimental settings used in Table 1. We evaluate both in-domain and out-of-domain transfer on SWE-Bench Verified and TB2.0. The results are presented below (all metrics are percentages).
ModularRSI consistently achieves the highest accuracy and Pass³ across all four benchmark/evolution-set combinations, outperforming Meta-Harness and AHE in both in-domain and out-of-domain settings. It also achieves the highest Pass@3 in three out of four settings, with the only exception being SWE-Bench Verified under TB-related evolution, where AHE achieves a marginally higher Pass@3 (84.80% vs. 84.60%). Overall, these results provide further evidence of the effectiveness of ModularRSI and its ability to generalize harness improvements across different task domains.