acceptodds
Under review as a conference paper at ICLR 2027

PluginRSI: Recursive Improvement of Agent Harnesses with Reusable Plugins

Abstract

The harness surrounding a language model is a central determinant of agent performance. Recent methods optimize harnesses by searching over complete programs, where individual mechanisms are difficult to isolate and reuse. We introduce PluginRSI, which represents a harness as a composition of atomized plugins and organizes harness evolution around these plugins. Individual plugins are improved independently and accumulated in a shared library, then recombined into new harnesses at each iteration. PluginRSI improves over existing harness optimization methods across software engineering, command-line interaction, and question-answering tasks. The resulting harnesses retain their advantage when transferred to other solver models without further optimization. The evolved plugin library accelerates subsequent optimization from the initial harness, which helps faster and higher convergence on unseen tasks. These results show that accumulating reusable mechanisms provides an effective basis for continued harness improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent, Harness, Recursive Self-Improvement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_EUsEOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes PluginRSI, a recursive optimization framework for agent harnesses. It decomposes agent mechanisms into reusable plugins and enables continuous optimization through two stages: Plugin Mutation, which independently modifies and evaluates plugins while keeping other components and workflows fixed, and Harness Recomposition, which selects components from an accumulated plugin library and adjusts workflows to construct new harnesses. The framework is evaluated on SWE-bench Verified, Terminal-Bench 2.1, and several QA benchmarks.

Critical strengths
  1. The two-stage design is intuitive, separating plugin-level optimization from harness-level recomposition and providing a structured approach to iterative harness improvement.
  2. The paper is clearly written and easy to follow.
Critical weaknesses
  1. The evaluation of plugin reuse raises concerns about test-set independence. Appendix E indicates that the 100 held-out tasks are used in subsequent harness optimization, making it unclear whether the reported improvement from 65% to 69% generalizes to an independent test set.
  2. Plugin Mutation evaluates each plugin in isolation, potentially discarding variants that are only beneficial through cross-component interactions. It remains unclear how Harness Recomposition can effectively identify such synergies.
Questions for authors

See Weakness.

3 comments
AuthorsOpenReview ↗

Authors' Rebuttal to Reviewer EUsE

We thank the reviewer for the comments on test-set independence and cross-component synergies. We respond point by point below. All new results are included in the revision.

Weakness 1. The evaluation of plugin reuse raises concerns about test-set independence. Appendix E indicates that the 100 held-out tasks are used in subsequent harness optimization, making it unclear whether the reported improvement from 65% to 69% generalizes to an independent test set.

Thank you for pointing this out. Our wording in Appendix E and the y-axis label of Figure 4 could be misleading. We provide clarification and additional experiments below.

[Clarification] The main results in Tables 1–4 never use the held-out tasks for optimization. Figure 4 is a separate experiment that tests whether the evolved plugin library helps optimization on a new task set. We use the 100 previously held-out tasks as this new set and run independent optimization on them, with and without the evolved library. The 65% (without reuse) and 69% (with reuse) after two steps are therefore scores on the tasks being optimized, not test scores. Only the step-0 points are measured before any optimization on these tasks: 51% for the initial harness and 65% for the evolved harness, the latter being its Table 1 held-out score.

[New Experiments] To test whether the benefit of library reuse also holds on tasks not used for optimization, we split the 100 tasks into 50 adaptation tasks and 50 test tasks. We optimize on the adaptation tasks for five steps, select the harness with the best adaptation score, and report its score on the test tasks. The test tasks are never used for optimization or selection. Other settings are the same as in Figure 4. The results are shown in Table R6.

Table R6. Plugin reuse with disjoint adaptation and test splits on SWE-bench Verified, reported as resolve rates in %.

Adaptation (50), step 0Adaptation (50), bestTest (50), step 0Test (50), selected harness
Initial harness + initial library54644860
Initial harness + evolved library54724864
Evolved harness + evolved library68746268

We observe that:

  1. Library reuse improves accuracy on the independent test split. Starting from the same initial harness, reusing the evolved library raises test accuracy from 48 to 64, compared with 60 when using the initial library.
  2. With the evolved library, the initial harness approaches the evolved harness. After five steps, it reaches 72 on adaptation and 64 on test, compared with 74 and 68 for the evolved harness.

In the revision, we have updated Section 4.5 and Appendix E, and added these results to Appendix P.

Weakness 2. Plugin Mutation evaluates each plugin in isolation, potentially discarding variants that are only beneficial through cross-component interactions. It remains unclear how Harness Recomposition can effectively identify such synergies.

[How recomposition builds synergies]

  • Mutation is performed in context, not in isolation. Each plugin is mutated inside the current harness H(W,P)\mathcal{H}(W, P) using its execution traces, so a variant only needs to work with the components already in place.
  • Recomposition identifies complementary plugins from the mutation results. The recomposition proposer reads which plugins were admitted, their minibatch score changes, and the corresponding traces. It then selects complementary plugins discovered in different parallel mutation branches and revises the workflow W→W′W \to W' to connect them. For example, in iteration 1 of the QA run in Table 2, mutation #1 aligns the role's output format with the workflow's finish check, and mutation #3 updates the skill to read answer_mode and choices. Recomposition then integrates both into one workflow.

Table 4 of our paper shows that this step matters: removing recomposition reduces Kimi-K3 held-out from 69.00% to 59.00%.

A variant that only helps together with a second simultaneous change is still rejected at mutation, as noted in Appendix D. To measure how much this matters, we conduct additional experiments where each mutation modifies 2, 3, or 4 plugins jointly; the full results are in Table R4 of our response to Reviewer 9cn1. Joint mutation keeps the held-in average almost unchanged but lowers held-out accuracy as more plugins are modified together. With one shared minibatch score, an unhelpful edit can pass the gate together with a helpful one, e.g., an unused completion-check skill admitted alongside a helpful calculator tool.

In the revision, we have added this discussion to Section 3.2.2 and Appendix N.

We hope these responses address your concerns, and we would be grateful if you could take them into account in your final assessment.

Thank you for the detailed clarification. Your responses address most of my concerns, but I remain unconvinced by the evidence for the method’s effectiveness and novelty, so I maintain my rating.

Thanks for your reply! We are glad that our responses have addressed most of your concerns. Below, we reiterate our novelty relative to several concurrent preprints, as well as the effectiveness of our method.

We do notice that several concurrent works are exploring related directions. While their findings are also valuable to the community, our method has the following core novelty relative to recent research:

  • Co-optimization of plugins and the workflow. A major part of our method design is tailored for the simultaneous optimization of both plugins P\mathcal{P} and control flow WW, while recent works focus on refining local implementations within a predefined, fixed harness skeleton [1, 2, 3].
  • Open-ended sub-module exploration. Our method enables searching over existing plugins and creating brand-new sub-modules on demand, while recent works concentrate more on refining components within predefined module slots [1, 2, 3].
  • Transferability with plugins. Our core findings highlight the reuse value of accumulated plugins. In our experiments, evolved plugins can be reused as a search prior on unseen tasks (both in- and out-of-domain) to substantially accelerate convergence and raise the inference ceiling, whereas existing works primarily evaluate the direct deployment of a single final artifact (e.g., a frozen optimized harness [1, 3] or model [2]).

Our effectiveness:

  1. Compared with baseline methods, our method converges to higher scores using fewer rollouts and lower total cost.
  2. Across 10 benchmarks spanning software engineering, command-line interaction, and question answering, our method demonstrates stronger in-task held-out generalization, cross-task generalization, cross-model generalization, and plugin library reuse capability.

We truly appreciate your comments, which have helped improve the quality of this work, as well as your recognition of the soundness and informativeness of our paper. Thank you for your time and consideration.

Regards,

The Authors

[1] ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement
[2] JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution
[3] HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

Reviewer_X8LqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PluginRSI, a recursive harness optimization framework that decomposes an agent harness into reusable plugins such as tools, roles, memory modules, and skills. The optimization process alternates between two stages: plugin mutation, which locally edits individual plugins based on execution feedback, and harness recomposition, which rebuilds the overall workflow using the evolved plugin library. The method is evaluated on SWE-bench Verified, Terminal-Bench, and several QA benchmarks, where the authors report consistent improvements over baselines such as Meta-Harness. The paper also studies plugin reuse, cross-model transfer, and optimization dynamics, arguing that reusable mechanisms can accumulate over time and accelerate future harness optimization.

Critical strengths
  1. The paper addresses an important and timely problem in agent harness optimization.

  2. The empirical section is fairly comprehensive. Experiments cover coding, terminal interaction, and QA tasks, and the paper includes ablations, transfer experiments, and optimization trajectory analyses.

  3. The paper is well written and easy to follow.

Critical weaknesses
  1. The efficiency analysis is also incomplete. While the paper reports solver rollout efficiency, the proposer-side optimization overhead introduced by plugin mutation is much less clear. The method repeatedly generates and evaluates multiple candidate plugins in parallel, yet the paper does not provide wall-clock runtime, total compute usage, or scalability analysis as the plugin library grows.

  2. The novelty is somewhat incremental. Many of the underlying ideas — iterative harness refinement, modular skills, reusable components, and reflective optimization — have already appeared in recent agent optimization literature.

  3. Another concern is the baseline comparison. Some closely related approaches, particularly AHE which show great resullts for Harness Evolution [1], are not prominently compared in the main results.

[1]. Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

Questions for authors

Please refer to weaknesses.

3 comments
AuthorsOpenReview ↗

Authors' Rebuttal to Reviewer X8Lq (1/3)

We thank the reviewer for the constructive comments. We respond point by point below to the concerns on efficiency, novelty, and baselines. All new results are included in the revision.

Weakness 1. The efficiency analysis is also incomplete ... the paper does not provide wall-clock runtime, total compute usage, or scalability analysis as the plugin library grows.

We address total compute, wall-clock runtime, and scalability in turn.

[Total compute] We kindly suggest the reviewer to also check Figure 13 in Appendix L of our original paper, where we report tokens per solver rollout, including proposer tokens, for the GPT-5.6 Terra setting of Table 1. At the reviewer's suggestion, we further report the end-to-end cost of the full search, counting both solver and proposer. The results are shown in Table R7.

Table R7. End-to-end search cost on SWE-bench Verified in the GPT-5.6 Terra setting, using the same runs as Figure 13.

MethodIterationsSolver rolloutsSolver tokens / rollout (in / out)Proposer tokens (in / out)Total tokens (in / out)Total tokensHeld-out
Meta-Harness2510,000128.0K / 1.59K12.1M / 0.2M1,292.0M / 16.1M1,308.1M55.00
PluginRSI158,400134.2K / 1.73K39.2M / 1.7M1,166.7M / 16.2M1,182.9M65.00

We observe that:

  1. PluginRSI reaches a higher held-out score with fewer total input tokens. Each solver rollout uses 7.5% more input tokens than in Meta-Harness (Figure 13), but the search needs 16% fewer rollouts, so the total cost is lower while the held-out resolve rate is higher.
  2. Plugin mutation adds little token cost. Proposer calls, including all mutation and recomposition steps, account for only 3.5% of total tokens, and solver rollouts dominate the compute.

[Wall-clock runtime] We did not report wall-clock time in the submission because it varies with time-of-day API load and the resulting number of retried requests, whereas token cost is stable across runs.

At the reviewer's suggestion, we conduct new speed tests that run PluginRSI and Meta-Harness concurrently on the same API endpoints and infrastructure. We optimize with the same setup as Table 1 for five iterations. The results are shown in Table R8.

Table R8. Wall-clock runtime on SWE-bench Verified.

MethodMutation per iterationRecomposition / rewrite per iterationHeld-in eval per iterationTotal per iterationHeld-out eval
Meta-Harness–6.3 min60.5 min66.8 min25.4 min
PluginRSI31.1 min4.1 min61.2 min96.4 min25.4 min

We observe that:

  1. The extra per-iteration runtime comes from plugin mutation, while the full search is shorter. Recomposition and evaluation take about as long as in Meta-Harness, and PluginRSI runs fewer iterations, so the estimated full search takes about 24.1 hours (15 iterations) against 27.8 hours (25 iterations) for Meta-Harness.
  2. Mutation candidates run in parallel. The 8 mutation candidates are proposed and evaluated concurrently on separate 20-task minibatches, so the mutation stage costs one proposer session plus one minibatch evaluation, regardless of the number of candidates.

[Scalability] We record recomposition cost during the search, where the library holds 80 plugins at step 1 and 132 at step 15. To stress-test larger libraries, we further let the proposer synthesize up to 1,000 plugins offline through one-shot proposals without validation, with a LoC distribution similar to the plugins developed during search. The results are shown in Table R9.

Table R9. Recomposition cost vs. library size.

Library sizeInitial contextModel callsFiles readTotal tokensWall-clock
80 (step 1)9.4K1662632K242s
132 (step 15)11.0K1765661K237s
200 (synthetic)12.6K1668649K275s
500 (synthetic)20.5K19771,067K337s
1,000 (synthetic)33.6K18641,379K332s

We observe that:

  1. The initial context grows slowly with library size. The proposer receives a manifest index of plugin.yaml paths instead of the plugin source code, which adds about 26 tokens per plugin, so even a 1,000-plugin library needs 33.6K tokens of initial context.
  2. Recomposition cost grows much more slowly than the library. The proposer reads plugins on demand, mainly those admitted in the current iteration, so the numbers of file reads and model calls stay flat. As the library grows 12.5 times from 80 to 1,000 plugins, total tokens grow 2.2 times and wall-clock time 1.4 times.

In the revision, we have added these analyses to Section 4.3 and Appendix L.

AuthorsOpenReview ↗

Authors' Rebuttal to Reviewer X8Lq (2/3)

Weakness 2. The novelty is somewhat incremental. Many of the underlying ideas, iterative harness refinement, modular skills, reusable components, and reflective optimization, have already appeared in recent agent optimization literature.

We agree that iterative refinement, modular skills, and reusable components have each been studied in prior work. Our contribution is not modularity itself, but a search procedure that assigns credit to individual executable mechanisms while still allowing the workflow to change globally.

Table R10. Comparison between prior agent optimization paradigms and PluginRSI. HH: harness, cc: prompt or context, SS: skills, PP: plugins or modules, WW: workflow, W0,P0W_0, P_0: fixed templates.

Prompt / contextModular skillsIterative harness refinementSearch w/ reusable modulesPluginRSI (Ours)
Representative worksGEPA, ACESkillOPT, VoyagerDGM, Meta-Harness, AHEAgentSquare [1], AFlow [2]Ours
Unit of editPrompt textTextual skillsEntire harness codeModules under fixed wiring, or graph over fixed opsExecutable plugins + workflow
Credit assignmentWhole promptWhole candidateWhole harnessSurrogate / whole graphPer plugin, rest-fixed
Cross-task reuseNoVia skillsNoNoVia plugins
What to optimizearg⁡max⁡cJ(c)\arg\max_c J(c)arg⁡max⁡SJ(S)\arg\max_S J(S)arg⁡max⁡HJ(H)\arg\max_H J(H)arg⁡max⁡PJ(H(W0,P))\arg\max_P J(\mathcal{H}(W_0, P)) or arg⁡max⁡WJ(H(W,P0))\arg\max_W J(\mathcal{H}(W, P_0))arg⁡max⁡pJM(H(W,P−i∪{p}))\arg\max_p J_M(\mathcal{H}(W, P_{-i} \cup \{p\})) and arg⁡max⁡W′,P′J(H(W′,P′))\arg\max_{W', P'} J(\mathcal{H}(W', P'))

Our design addresses two limitations of existing paradigms:

  1. Whole-harness rewriting cannot evaluate or preserve individual mechanisms. DGM, Meta-Harness, and AHE rewrite the entire harness at each iteration, so coupled mechanisms cannot be credited separately and effective ones can be overwritten later. PluginRSI instead evaluates each plugin with the rest of the harness fixed and keeps beneficial variants in a shared library, making "mechanism-level reuse a first-class object", as noted by Reviewer 9cn1. The gain comes from this mechanism-level credit assignment rather than from the library alone: adding our plugin library to Meta-Harness raises its Kimi-K3 held-out score only from 63.00% to 64.00% (Table 4).
  2. Fixed-template module search cannot co-evolve mechanisms and their coordination. AgentSquare [1] evolves modules under a fixed wiring template, AFlow [2] searches graphs over predefined operators, and concurrent work such as ModularRSI [3] focuses on contrastive trajectory analysis to improve a modularized harness. In contrast, PluginRSI grows an open-ended plugin library and recomposes plugins and workflow at every iteration. Compared with these works, our paper offers a fundamentally different framework design, alternating plugin mutation and harness recomposition, and a new empirical finding on the transferability of evolved plugins.

In the revision, we have added this discussion to Section 5 and Appendix R.

[1] Shang et al., AgentSquare: Automatic LLM Agent Search in Modular Design Space, ICLR 2025.

[2] Zhang et al., AFlow: Automating Agentic Workflow Generation, ICLR 2025.

[3] Wu et al., ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement, arXiv:2609.14857, 2026.

AuthorsOpenReview ↗

Authors' Rebuttal to Reviewer X8Lq (3/3)

Weakness 3. Some closely related approaches, particularly AHE ..., are not prominently compared in the main results.

We kindly suggest the reviewer to also check Table 7 in Appendix F of our original paper, where we compare PluginRSI with AHE, SkillOPT, and A-Evolve on SWE-bench Verified under the same 400/100 protocol with Kimi-K3. PluginRSI outperforms AHE by 6.50 points on held-in, 66.75% vs. 60.25%, and by 6.00 points on held-out, 69.00% vs. 63.00%.

At the reviewer's suggestion, we conduct additional experiments that extend these baselines, together with DGM, to the QA benchmarks. The results are shown in Table R11.

Table R11. Comparison with additional baselines on QA tasks with Kimi-K3, reported as accuracies.

MethodMathSciCiteEcoHeld-in Avg. (150)FinFSPhysicsLawMedHeld-out Avg. (250)
ReAct0.4800.5400.4200.4800.2800.6200.1400.3200.3000.332
GEPA0.6800.6400.5800.6330.3800.6600.2000.6200.5000.472
DGM0.7000.5800.7400.6730.3600.7000.1000.7000.4600.464
SkillOPT0.7200.6800.7200.7070.5000.6800.2400.7200.4000.508
A-Evolve0.6600.6200.8000.6930.4200.5800.1800.7400.4600.476
AHE0.7400.6000.7800.7070.4000.6200.2200.7600.5400.508
Meta-Harness0.7800.6800.8000.7530.4800.7400.2800.8000.5400.568
PluginRSI0.8800.7800.8000.8200.5600.7200.2800.8200.5600.588

We observe that PluginRSI achieves the best held-in and held-out averages. It outperforms AHE by 0.113 on held-in and 0.080 on held-out, consistent with the SWE-bench comparison in Table 7.

In the revision, we have added these comparisons to Section 4.2 and Appendix Q.

We hope these responses address your concerns, and we would be grateful if you could take them into account in your final assessment.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Authors' Rebuttal to Reviewer 9cn1 (1/2)

We thank the reviewer for the detailed reading of our paper and for recognizing our contribution. We respond point by point below to the concerns on statistical significance, plugin lifecycle statistics, multi-plugin mutation, and within-family transfer. All new results are included in the revision.

Weakness 1 / Question 1. Could you provide task-level bootstrap confidence intervals (or repeated-run variances) for at least the main SWE-bench Verified and Terminal-Bench held-out comparisons in Tables 1 and 3?

At the reviewer's suggestion, we compute task-level paired bootstrap CIs on the per-task outcomes behind Tables 1 and 3, and conduct three repeated evaluation runs of the selected harnesses. The results are shown in Tables R1.a, R1.b, and R2.

Table R1.a. Task-level 95% bootstrap CIs on SWE-bench Verified held-out tasks, with n=100n = 100 and 10,000 resamples. Rows marked Δ report paired CIs of PluginRSI minus Meta-Harness.

MethodGPT-5.6 TerraKimi-K3GLM-5.2
Optimized w/ GPT-5.6 Terra
Meta-Harness55.00 [45.00, 65.00]53.00 [43.00, 63.00]44.00 [34.00, 54.00]
PluginRSI (Ours)65.00 [56.00, 74.00]60.00 [50.00, 69.00]52.00 [42.00, 62.00]
Δ (paired)+10.00 [+3.00, +17.00]+7.00 [+2.00, +13.00]+8.00 [+2.00, +15.00]
Optimized w/ Kimi-K3
Meta-Harness60.00 [50.00, 69.00]63.00 [53.00, 72.00]59.00 [49.00, 69.00]
PluginRSI (Ours)64.00 [54.00, 73.00]69.00 [60.00, 78.00]64.00 [54.00, 73.00]
Δ (paired)+4.00 [-1.00, +10.00]+6.00 [-1.00, +13.00]+5.00 [-1.00, +11.00]

Table R1.b. Task-level 95% bootstrap CIs on Terminal-Bench-2.1 held-out tasks with Kimi-K3, n=30n = 30 and 10,000 resamples.

MethodHeld-Out (%)
Meta-Harness66.7 [50.0, 83.3]
PluginRSI (Ours)73.3 [56.7, 86.7]
Δ (paired)+6.7 [-6.7, +20.0]

Table R2. Three repeated evaluation runs of the selected Kimi-K3-optimized harnesses on SWE-bench Verified held-out tasks with the Kimi-K3 solver. Seed 42 is the run reported in Table 1.

MethodHeld-in (Table 1)Held-out, Seed 41Held-out, Seed 42Held-out, Seed 43Held-out, Mean ± Std
Meta-Harness64.2564.0063.0063.0063.33 ± 0.58
PluginRSI (Ours)66.7568.0069.0068.0068.33 ± 0.58

We observe that:

  1. With GPT-5.6 Terra optimization, all three paired CIs exclude zero. With Kimi-K3 optimization, the CIs include zero because the held-out sets are small: the SWE-bench lower bounds of −1.00 correspond to one task out of 100, and the 6.7-point Terminal-Bench gain corresponds to two of 30 tasks.
  2. The held-out gain is statistically significant across repeated runs. PluginRSI scores 68.33 ± 0.58 against 63.33 ± 0.58 for Meta-Harness, and a two-sided two-sample t-test gives p=4.5×10−4<0.001p = 4.5 \times 10^{-4} < 0.001.

In the revision, we have added these results to Section 4.2 and Appendix M.

Question 2. In the mutation stage, what fraction of proposed plugin variants are retained in L, and of the plugins ultimately selected by recomposition in the final harness, how many were added during evolution vs. drawn from the initial 75-plugin library?

[Retention in library] In the run shown in Figure 5, the proposer generates 120 plugin variants over 15 iterations, 8 per iteration. 111 of them complete evaluation, and the other 9 fail due to infrastructure timeouts or a proposer refusal. 57 variants improve their minibatch score and enter L, a retention rate of 47.5% (51.4% over completed evaluations), consistent with the library growth from 75 to 132 plugins in Figure 5(a).

[Origin of plugins in the final harnesses] For each plugin in the final harnesses, we trace its version history back to the seed library and count the mutation rounds along the way. The results are shown in Table R3.

Table R3. Number of mutation rounds of the 26 plugins in the final harnesses of PluginRSI on SWE-bench Verified runs.

0 (Seed)1 round2 rounds3 rounds4+ roundsEvolved shareMean rounds
3.8%19.2%30.8%34.6%11.5%96.2%2.31

We observe that:

  1. The final harnesses consist mostly of evolved plugins. Only one of the 26 plugins comes unchanged from the initial library, so the final harnesses are built from mechanisms developed during the search rather than from the seed library.
  2. Most plugins are refined over several iterations. 76.9% of them go through at least two mutation rounds, so plugins accumulate improvements across iterations rather than being one-shot edits.

This agrees with the "w/o Initial Plugins" ablation in Table 4 of our paper: starting from only the four basic plugins, PluginRSI still reaches 67.00% Kimi-K3 held-out, compared with 69.00% with the full seed library.

In the revision, we have added these results to Section 4.5 and Appendix N.

AuthorsOpenReview ↗

Authors' Rebuttal to Reviewer 9cn1 (2/2)

Weakness 2. Plugin mutation's rest-fixed evaluation may systematically miss coordinated multi-plugin mechanisms ... A small controlled study (e.g., allowing paired mutations ...) would clarify how binding this constraint is in practice.

Rest-fixed evaluation gives up joint search in exchange for clean per-plugin credit assignment. Regarding the second suggested measurement, a variant rejected at mutation never enters L, so recomposition cannot select it by construction. We therefore test the constraint directly by relaxing it.

At the reviewer's suggestion, we conduct three additional controlled runs on the QA benchmarks with Kimi-K3. In each mutation, the proposer selects and modifies 2, 3, or 4 plugins of the current harness jointly, and the joint change is admitted only if it improves the minibatch score. Other settings follow Table 2. The results are shown in Table R4.

Table R4. Variants of the mutation stage on QA benchmarks with Kimi-K3, reported as accuracies.

Mutation variantMathSciCiteEcoHeld-in Avg. (150)FinFSPhysicsLawMedHeld-out Avg. (250)
PluginRSI, 1 plugin / mutation (original)0.8800.7800.8000.8200.5600.7200.2800.8200.5600.588
PluginRSI, 2 plugins / mutation0.8600.7600.8200.8130.4400.7800.2400.8000.5600.564
PluginRSI, 3 plugins / mutation0.9200.7600.7600.8130.4000.6200.1400.6800.5200.472
PluginRSI, 4 plugins / mutation0.9000.7600.7800.8130.2600.4400.2200.8400.5400.460

We observe that:

  1. Joint mutation lowers held-out accuracy. The held-in average stays almost unchanged, while the held-out average decreases steadily as more plugins are modified per mutation, from 0.588 to 0.460.
  2. Joint admission lets unhelpful edits into the library. When several plugins share one minibatch score, an edit can pass the gate because of its partner. In mutation #6 of iteration 3 (2 plugins per mutation), an unused completion-check skill is admitted together with a helpful calculator tool: the pair solves 5 minibatch tasks that the incumbent fails and fails 3 that it solves, although the trajectories show the skill is never loaded.

Changes that require a coordinated workflow edit are handled by recomposition, which revises WW together with the plugin selection. In the revision, we have added this analysis to Section 3.2.2, Appendix D, and Appendix N.

Question 3. Have you measured held-out transfer within a domain family (e.g., math→physics) ...?

We kindly suggest the reviewer to also check Table 2 in our original paper, where we provide a partial answer: FinanceQA is the held-out domain closest to the held-in Economics domain, and PluginRSI improves it by 0.080 over Meta-Harness (0.480 → 0.560), compared with 0.020 on Law and Medicine.

At the reviewer's suggestion, we conduct additional optimization runs that test this directly. We optimize PluginRSI and Meta-Harness with Kimi-K3 only on Olympiad-Math for 5 steps, and then evaluate on a within-family set of Olympiad-Physics and FrontierScience, and a cross-family set of SuperGPQA Law and Medicine. Each set has 100 tasks. The results are shown in Table R5.

Table R5. Within-family vs. cross-family transfer from Math-only optimization.

Math (held-in)Within-family Avg.PhysicsFrontierScienceCross-family Avg.
Meta-Harness0.7200.4300.1800.6800.750
PluginRSI0.7600.4800.2400.7200.720

We observe that:

  1. Plugins transfer within the domain family. PluginRSI outperforms Meta-Harness on the within-family set, 0.480 vs. 0.430, with gains on both Physics and FrontierScience.
  2. Transfer to distant domains is limited. On Law and Medicine, Meta-Harness reaches 0.750 against 0.720 for PluginRSI, so we do not claim an advantage in cross-family transfer.

These results support the explanation that plugins do not transfer well across dissimilar domains, rather than that they do not transfer to held-out tasks at all. In the revision, we have added these results to Section 4.2 and Appendix O.

We thank the reviewer again for the constructive suggestions, which helped us strengthen the paper.

Thank you for your timely reply! We are glad that our responses have resolved most of your concerns. Below, we would like to briefly reiterate our technical contributions, as well as our core novelty relative to several concurrent preprints.

Our main contributions:

  • Formulation of plugin-oriented harness search. We formulate the harness optimization problem as a search problem over both plugins and the workflow, H=H(W,P)H = \mathcal{H}(W, \mathcal{P}). By this design, we represent individual sub-mechanisms as plugins to store and transfer experience across harness optimization stages.
  • Comprehensive empirical validation. We conduct evaluations across 10 benchmarks spanning software engineering, command-line interaction, and question answering, thoroughly validating our method's optimization ceiling, cross-task generalization, cross-model generalization, and plugin library reuse capability. Our ablation studies also provide supporting evidence for the cumulative gains of the plugin system.

We do notice that several concurrent works are exploring related directions. While their findings are also valuable to the community, our method has the following core novelty relative to recent research:

  • Co-optimization of plugins and the workflow. A major part of our method design is tailored for the simultaneous optimization of both plugins P\mathcal{P} and control flow WW, while recent works focus on refining local implementations within a predefined, fixed harness skeleton [1, 2, 3].
  • Open-ended sub-module exploration. Our method enables searching over existing plugins and creating brand-new sub-modules on demand, while recent works concentrate more on refining components within predefined module slots [1, 2, 3].
  • Transferability with plugins. Our core findings highlight the reuse value of accumulated plugins. In our experiments, evolved plugins can be reused as a search prior on unseen tasks (both in- and out-of-domain) to substantially accelerate convergence and raise the inference ceiling, whereas existing works primarily evaluate the direct deployment of a single final artifact (e.g., a frozen optimized harness [1, 3] or model [2]).

Thank you for your time and consideration, and for being willing to discuss our paper in the next discussion phase. We also appreciate your comments on our efficiency analysis and baseline comparison, which have helped us strengthen our work.

Regards,

The Authors

[1] ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement
[2] JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution
[3] HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.