- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes PluginRSI, a recursive optimization framework for agent harnesses. It decomposes agent mechanisms into reusable plugins and enables continuous optimization through two stages: Plugin Mutation, which independently modifies and evaluates plugins while keeping other components and workflows fixed, and Harness Recomposition, which selects components from an accumulated plugin library and adjusts workflows to construct new harnesses. The framework is evaluated on SWE-bench Verified, Terminal-Bench 2.1, and several QA benchmarks.
Critical strengths
- The two-stage design is intuitive, separating plugin-level optimization from harness-level recomposition and providing a structured approach to iterative harness improvement.
- The paper is clearly written and easy to follow.
Critical weaknesses
- The evaluation of plugin reuse raises concerns about test-set independence. Appendix E indicates that the 100 held-out tasks are used in subsequent harness optimization, making it unclear whether the reported improvement from 65% to 69% generalizes to an independent test set.
- Plugin Mutation evaluates each plugin in isolation, potentially discarding variants that are only beneficial through cross-component interactions. It remains unclear how Harness Recomposition can effectively identify such synergies.
Questions for authors
See Weakness.
3 comments
Thank you for the detailed clarification. Your responses address most of my concerns, but I remain unconvinced by the evidence for the method’s effectiveness and novelty, so I maintain my rating.
Thanks for your reply! We are glad that our responses have addressed most of your concerns. Below, we reiterate our novelty relative to several concurrent preprints, as well as the effectiveness of our method.
We do notice that several concurrent works are exploring related directions. While their findings are also valuable to the community, our method has the following core novelty relative to recent research:
- Co-optimization of plugins and the workflow. A major part of our method design is tailored for the simultaneous optimization of both plugins and control flow , while recent works focus on refining local implementations within a predefined, fixed harness skeleton [1, 2, 3].
- Open-ended sub-module exploration. Our method enables searching over existing plugins and creating brand-new sub-modules on demand, while recent works concentrate more on refining components within predefined module slots [1, 2, 3].
- Transferability with plugins. Our core findings highlight the reuse value of accumulated plugins. In our experiments, evolved plugins can be reused as a search prior on unseen tasks (both in- and out-of-domain) to substantially accelerate convergence and raise the inference ceiling, whereas existing works primarily evaluate the direct deployment of a single final artifact (e.g., a frozen optimized harness [1, 3] or model [2]).
Our effectiveness:
- Compared with baseline methods, our method converges to higher scores using fewer rollouts and lower total cost.
- Across 10 benchmarks spanning software engineering, command-line interaction, and question answering, our method demonstrates stronger in-task held-out generalization, cross-task generalization, cross-model generalization, and plugin library reuse capability.
We truly appreciate your comments, which have helped improve the quality of this work, as well as your recognition of the soundness and informativeness of our paper. Thank you for your time and consideration.
Regards,
The Authors
[1] ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement
[2] JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution
[3] HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses
Authors' Rebuttal to Reviewer EUsE
We thank the reviewer for the comments on test-set independence and cross-component synergies. We respond point by point below. All new results are included in the revision.
Thank you for pointing this out. Our wording in Appendix E and the y-axis label of Figure 4 could be misleading. We provide clarification and additional experiments below.
[Clarification] The main results in Tables 1–4 never use the held-out tasks for optimization. Figure 4 is a separate experiment that tests whether the evolved plugin library helps optimization on a new task set. We use the 100 previously held-out tasks as this new set and run independent optimization on them, with and without the evolved library. The 65% (without reuse) and 69% (with reuse) after two steps are therefore scores on the tasks being optimized, not test scores. Only the step-0 points are measured before any optimization on these tasks: 51% for the initial harness and 65% for the evolved harness, the latter being its Table 1 held-out score.
[New Experiments] To test whether the benefit of library reuse also holds on tasks not used for optimization, we split the 100 tasks into 50 adaptation tasks and 50 test tasks. We optimize on the adaptation tasks for five steps, select the harness with the best adaptation score, and report its score on the test tasks. The test tasks are never used for optimization or selection. Other settings are the same as in Figure 4. The results are shown in Table R6.
Table R6. Plugin reuse with disjoint adaptation and test splits on SWE-bench Verified, reported as resolve rates in %.
We observe that:
In the revision, we have updated Section 4.5 and Appendix E, and added these results to Appendix P.
[How recomposition builds synergies]
finishcheck, and mutation #3 updates the skill to readanswer_modeandchoices. Recomposition then integrates both into one workflow.Table 4 of our paper shows that this step matters: removing recomposition reduces Kimi-K3 held-out from 69.00% to 59.00%.
A variant that only helps together with a second simultaneous change is still rejected at mutation, as noted in Appendix D. To measure how much this matters, we conduct additional experiments where each mutation modifies 2, 3, or 4 plugins jointly; the full results are in Table R4 of our response to Reviewer 9cn1. Joint mutation keeps the held-in average almost unchanged but lowers held-out accuracy as more plugins are modified together. With one shared minibatch score, an unhelpful edit can pass the gate together with a helpful one, e.g., an unused completion-check skill admitted alongside a helpful calculator tool.
In the revision, we have added this discussion to Section 3.2.2 and Appendix N.
We hope these responses address your concerns, and we would be grateful if you could take them into account in your final assessment.