acceptodds
Under review as a conference paper at ICLR 2027

FedRSI: Collective Self-Improvement through Harness Evolution

Abstract

Language-model agents increasingly improve themselves by rewriting their own harness, the program that manages the context, memory, tools, and calls of a frozen base model. However, because each agent evolves within a private deployment, its improvements remain unavailable to other agents that could benefit from them. Existing agent federations do not close this gap. Almost all of them share a single kind of harness component, such as prompts, knowledge, or skills; the only one that shares the entire harness requires its server to read client code and to use labelled validation data; and all of them aggregate once or in synchronous rounds, which proceed at the pace of the slowest client. Therefore, we introduce FedRSI, to our knowledge the first protocol that converts private improvements to any part of the harness into shared executable modules while every client retains its data, its code, and its own pace. Clients export only prose mechanism cards. The server writes a work order for each proposed mechanism, reviews it for task-specific assumptions, and re-implements the mechanism from scratch within a shared harness. Each client adopts the shared modules for its own task only if they raise accuracy on local validation data, and a task-aware priority gives the reviewed shared modules precedence on all other tasks. Because a card refers to no particular version of the shared harness, the server aggregates as soon as a fixed number of new cards have arrived, so that clients whose items take from seconds to an hour never wait for one another. In a federation of three text clients whose tasks differ in domain, language, and output space, FedRSI exceeds independent evolution for every client on its own task and raises the macro accuracy by 4.9 points on the clients' own tasks and by 6.7 points on the tasks of the other clients; in a federation of three coding clients, it also exceeds independent evolution on the tasks of the other clients. Moreover, FedRSI achieves these gains without exposing client data or making any client wait: none of the exchanged cards or shared modules contains such data, whereas synchronous rounds would have kept the faster clients idle for up to 59% of a run.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Recursive Self-Improvement, LLM Agents, Harness Evolution, Federated Learning, Cross-Task Transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TUGGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a federated agentic framework to enable the cross-task harness transfer without revealing the private data or code that reside on the local agents. Further experiments show that this federation design outperforms the standalone agents, and this transferred harness particularly benefits out-of-domain tasks.

Critical strengths

In this work, the authors focus on transfer executable harness improvements across agents without exposing their private data or source code. The proposed framework is reasonably well structured, and the experiments provide some evidence that sharing these improvements can benefit agents across different tasks. Overall the paper is well-organized.

Critical weaknesses
  1. The authors emphasize asynchronous harness aggregation as a key contribution, but it is unclear why synchronization would be necessary in the first place. Unlike traditional federated learning, where asynchronous updates to shared model weights can introduce staleness and consistency issues, each agent here independently evolves its own harness. The server could simply process incoming improvements in a first-come-first-served manner without blocking other clients, so a more meaningful baseline would be straightforward incremental integration, instead a synchronous setup that unnecessarily forces clients to wait.

  2. Privacy concerns. In FL, local training data remain on clients, but model updates are not automatically encrypted in a way that hides them from the server. Differ from numerical FL updates, arbitrary natural-language mechanism cards cannot simply be encrypted and summed, but this is insufficient to establish comprehensive confidentiality.

  3. The paper evaluates two federations, each containing only three clients, the experimental evidence is too limited for the generality of the claims

  4. In Appendix F.3, the coding experiment is much less convincing than the text experiment, only 1.53% performance gain compared to independent evolution

  5. Despite the authors report 100% routing accuracy, the task-aware routing is tested on highly distinct domains, this does not establish robustness when tasks have similar inputs or gradually changing distributions

  6. FedSI uses different rules when it comes to competing harness module pairs, and the priority is determined by whether the task is local or external. However, why should either be assumed to perform better? The server may reconstruct a better implementation of the same functionality, but local-first does not appear to consider this possibility.

  7. Although the overall narrative is reasonable, I found Section 3 unnecessarily difficult to follow. Many of the components are conceptually straightforward, but their descriptions are lengthy, the writing feels somewhat over-polished with LLMs, which does not help with readability.

Questions for authors

See the weaknesses listed above

9 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer TUGG (1/7)

W1. The authors emphasize asynchronous harness aggregation as a key contribution, but it is unclear why synchronization would be necessary in the first place. Unlike traditional federated learning, where asynchronous updates to shared model weights can introduce staleness and consistency issues, each agent here independently evolves its own harness. The server could simply process incoming improvements in a first-come-first-served manner without blocking other clients, so a more meaningful baseline would be straightforward incremental integration, instead a synchronous setup that unnecessarily forces clients to wait.

Re: (1) Why would synchronization be necessary? FedRSI does not need synchronization (Section 3.2). The paper compares FedRSI with synchronous rounds because every prior agent federation aggregates once or in synchronous rounds. A synchronous round ends only when its slowest client has finished. With the durations that we measured for each client, synchronous rounds would have kept the faster clients idle for up to 59% of a run. FedRSI's clients never wait. FedRSI needs no synchronization because its clients share mechanism cards. A card describes a behavior and refers to no version of the shared harness, so the server can integrate a card whenever it arrives (Appendix F.11). In contrast, a parameter update or a code diff refers to a version of the shared state. Such an update must therefore be rebased or weighted for staleness when it arrives late.

(2) First-come-first-served integration as the baseline. This baseline is FedRSI with a buffer of one card (B=1B = 1), and FedRSI's default buffer of two cards (B=2B = 2) transfers better. We believe the paper's buffered aggregation already contains this baseline. The server opens a federation round as soon as BB new cards have arrived (Section 3.2). With B=1B = 1, the server therefore integrates every card on arrival. Specifically, we conducted three new runs of FedRSI with B=1B = 1 on the text federation. The results show that other-task accuracy is 3.1 points higher with B=2B = 2 than with B=1B = 1 [+1.1, +5.2], and that own-task accuracy is unchanged (−0.5 points [−3.1, +2.1]).

AuthorsOpenReview ↗

Authors' Response to Reviewer TUGG (2/7)

W2. Privacy concerns. In FL, local training data remain on clients, but model updates are not automatically encrypted in a way that hides them from the server. Differ from numerical FL updates, arbitrary natural-language mechanism cards cannot simply be encrypted and summed, but this is insufficient to establish comprehensive confidentiality.

Re: FedRSI's confidentiality claim is that no client record, no label and no line of code leaves a client. The protocol establishes this claim by construction, and the paper verifies this claim with an audit and an attack. Specifically, the paper scans every artifact of two runs of the text federation and attacks the reported run with a simulated client (Appendix F.10).

(1) What leaves a client. By construction, only the card leaves a client. A card is plain prose, and the client screens it before upload for literals and for the canary strings planted in the served inputs. The server writes work orders from the cards, and the server's coding agent (the implementer) reads the work orders and never the cards.

(2) The audit. The results show that none of the 17 cards, 16 work orders and 15 shared modules contains a file path of our deployment, a dataset name, a label value, a verbatim span of a served input or the canary. In contrast, 7 of the 18 private modules contain true labels, which a federation that exchanged code would disclose. A judge model that reads each card without context identifies the domain of the card's author for only 1 of the 17 cards.

(3) The attack. The results show that, given the shared modules, the cards and the work orders, the simulated attacker quotes no canary and names no label of another client. Given one client's private modules, the attacker names 8 to 9 of the 23 labels that this client served. Given the served records, the attacker quotes the canary in every attack.

The revised paper states this threat model explicitly in Appendix F.10.

AuthorsOpenReview ↗

Authors' Response to Reviewer TUGG (3/7)

W3. The paper evaluates two federations, each containing only three clients, the experimental evidence is too limited for the generality of the claims.

Re: The transfer holds in a larger federation and with a stronger base model. Specifically, we conducted two runs of a federation of five clients with GLM-5.3-Flash as the base model. The five clients serve the three text tasks, FiNER (tagging financial entities) and Aegis2 (detecting unsafe user messages), one task each, with the same schedule for every client. The results show that FedRSI exceeds independent evolution on other tasks by 4.2 points [+2.8, +5.6] (176 items won, 58 lost; p<0.0001p < 0.0001), and that own-task accuracy is unchanged (+1.4 points [−0.9, +3.8]).

The text federation also has three new runs with GLM-5.3-Flash, in which FedRSI exceeds independent evolution on other tasks by 5.5 points [+3.4, +7.6]. The coding federation has three new runs as well (see our response to W4).

Reviewer_v54NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FedRSI transfers harness improvements between agents built around frozen language models. Clients export prose mechanism cards; a server reviews and reimplements their mechanisms as shared modules without receiving private client source code or records. Clients validate adoption locally, and subsequent local searches start from the adopted harness. Buffered asynchronous aggregation avoids waiting for other clients, while deployment priority distinguishes the client's own task from other clients' tasks. Evaluation uses three text clients drawn from Gretel symptom-to-diagnosis, LawBench/CAIL2018 charge prediction, and USPTO-50K retrosynthesis, plus three coding clients constructed from BigCodeBench. Comparisons include independent and centralized evolution, protocol ablations, and transfer to other base models.

Critical strengths
  1. A concrete protocol for transferring executable mechanisms under restricted sharing. Mechanism cards, server-side reimplementation, local adoption tests, and composition priorities address practical problems beyond exchanging prompts. The worked example traces a retrieval mechanism from Symptom to Crime, where adoption improves validation accuracy from 64.6% to 66.7% (Section 4.4). This establishes that useful cross-client mechanism transfer is possible.

  2. The paper reports replicated text results and informative negative cases. Across three text runs, ID and OOD macro accuracy exceed independent evolution by 4.30 and 4.60 percentage points, respectively (Table 11), with paired item-level analyses in Table 12. Reporting adverse client-task pairs and the smaller coding gain makes the empirical picture more informative than the headline alone.

Critical weaknesses
  1. The incremental value of ongoing collective evolution is not isolated from simpler sharing and additional resources. The motivation for transferring private discoveries is reasonable, but the main comparison with independent evolution does not establish why repeated interaction during evolution is needed. There is no control in which clients evolve independently and then exchange/reimplement their discoveries once, with comparable optimization and server resources. FedRSI also adds implementation, review, merging, and validation activity; Table 6 reports substantially higher total base-model token usage, although these totals do not separately identify optimization and evaluation costs. A resource-matched independent-search control is therefore important. The observed gains support the implemented protocol, but do not isolate the benefit of ongoing collective evolution over simpler alternatives. The evidence is also uneven: in the reported run, Crime is the only client whose final serving harness contains modules rebuilt from other clients' cards (Appendix F.4), and four of the six OOD client-task pairs are worse than independent evolution in the three-run means (Table 11). These facts do not negate the positive macro gains or possible intermediate search effects, but they limit a broad interpretation of mutual improvement.

  2. The recursive-improvement interpretation is supported more by the workflow than by a demonstrated benefit of recursive feedback. The introduction explicitly fixes both the base model and the local search procedure; Section 3.3 calls the process recursive because a recipient starts its next search from an adopted harness. Harness evolution can legitimately instantiate this broad definition, so the concern is not that model weights must change. Rather, the paper does not clearly trace and test whether an imported improvement enables a recipient to discover a better subsequent mechanism that then benefits another client, or whether this feedback produces gains beyond ordinary iterative harness optimization and module transfer. The detailed case establishes a one-way transfer, not the added value of a multi-hop feedback loop. The conclusion itself leaves improving the search procedure to future work. The evaluation provides limited support for a broader claim: the text tasks are small benchmark subsets, the BigCodeBench federation has only one reported evolution run and a 1.53-point OOD gain, and OOD evaluation includes replay of 24 labeled target-task history items and access to public example retrieval (Appendix D). Thus the evidence more directly supports federated harness transfer than a distinctive, experimentally established recursive self-improvement mechanism.

Questions for authors
  1. Does allowing exchange during evolution outperform a single post-hoc exchange/reimplementation after independent evolution, when local search, server calls, and validation resources are matched? A resource-matched independent-search comparison would also help establish the incremental value of the collective protocol.
  2. Can the recorded runs identify a concrete multi-hop lineage in which an imported mechanism enables a new local discovery that is subsequently exported and benefits another client? Is there an ablation of the imported mechanism at that search stage that attributes the later discovery or performance gain to this feedback?
  3. How much of the OOD improvement remains without replaying labeled target-task history, and which findings are stable across repeated coding evolution runs? Please clarify the evaluation scope and present the three-run text means alongside the main results.
7 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer v54N (1/5)

W1. The incremental value of ongoing collective evolution is not isolated from simpler sharing and additional resources. The motivation for transferring private discoveries is reasonable, but the main comparison with independent evolution does not establish why repeated interaction during evolution is needed. [...] The evidence is also uneven: in the reported run, Crime is the only client whose final serving harness contains modules rebuilt from other clients' cards (Appendix F.4), and four of the six OOD client-task pairs are worse than independent evolution in the three-run means (Table 11). These facts do not negate the positive macro gains or possible intermediate search effects, but they limit a broad interpretation of mutual improvement.

Re: FedRSI's gain comes from its exchange protocol and not from additional resources. The transfer reaches most clients and tasks.

(1) Why is exchange during evolution needed, rather than a single exchange after independent evolution? Because FedRSI's clients evolve online. With exchange during evolution, no client waits for the others to finish, and a client's next search can build on an imported improvement. Exchange during evolution also transfers more when the clients search more often. The protocol itself produces the transfer in both cases, during evolution and in a single exchange after it. Specifically, we conducted the control that the reviewer describes, with three new runs per configuration (base model: GLM-5.3-Flash). In this control, the clients first evolve independently: they run the same local search as in FedRSI and exchange nothing. After the last search of every client, the server processes the cards of all clients in a single federation round. This round uses the same work-order, review and implementation steps as FedRSI and builds at most eight mechanisms. Each client then runs the same adoption test as in FedRSI. The results show that this control raises other-task accuracy above independent evolution in every run, by 7.8, 3.5 and 3.8 points. The control also reaches the same accuracy as FedRSI with the paper's schedule, in which a client searches after every 6 items:

Configuration (mean of three new runs)Own tasks (ID), %Other tasks (OOD), %
Independent evolution61.2348.38
Independent evolution, then one federation round60.8253.40
FedRSI60.6353.87

The control is FedRSI's own protocol applied once. This result therefore confirms that the protocol produces the transfer. FedRSI applies the protocol during evolution for two reasons. First, FedRSI's clients evolve online, each at its own pace (Section 3). An exchange after evolution needs a moment at which every client has finished, and waiting for this moment is the synchronization that FedRSI avoids. Second, only an exchange during evolution lets a client's next search build on an imported improvement. The run records show such searches (see our response to W2).

Additionally, exchange during evolution transfers more when the clients search more often. Specifically, we conducted two runs of FedRSI with a search after every 3 items instead of 6. This schedule gives FedRSI's clients 48 proposer sessions and more federation rounds. The results show that, in both runs, FedRSI's other-task accuracy is higher with a search after every 3 items than with a search after every 6 items, by 5.9 points [+3.2, +8.5]. With a search after every 3 items, FedRSI also exceeds independent evolution with the same 48 proposer sessions on other tasks by 9.0 points [+6.6, +11.4]. More local search without exchange has the opposite effect: independent evolution's other-task accuracy falls from 48.38% with 24 proposer sessions to 43.85% with 48.

(2) Does a resource-matched independent search close the gap? No: an independent baseline with more resources than FedRSI stays behind FedRSI on other tasks. Specifically, we conducted a control in which independent evolution scores four proposals per search instead of two, with three new runs. This baseline has twice FedRSI's proposer sessions and costs 55% more than FedRSI. The results show that this baseline is 10.0 points behind FedRSI on other tasks [+8.0, +12.1]. Resources per run, averaged over the three new runs of each configuration:

ConfigurationProposer sessionsImplementer sessionsBase-model tokensCost (USD)
FedRSI243.083M16.6
Independent evolution, then one federation round241.0108M17.4
Independent evolution, four proposals per search480135M25.7

(3) Is the improvement mutual? Yes. In the new runs, FedRSI is above independent evolution on 5 of the 6 OOD pairs of client and task. In these runs, 6 of the 9 final serving harnesses (three clients in three runs) contain modules rebuilt from other clients' cards.

The revised paper includes the single-round control and the resource table in Appendix F.13.

AuthorsOpenReview ↗

Authors' Response to Reviewer v54N (2/5)

*W2. The recursive-improvement interpretation is supported more by the workflow than by a demonstrated benefit of recursive feedback. The introduction explicitly fixes both the base model and the local search procedure; Section 3.3 calls the process recursive because a recipient starts its next search from an adopted harness. [...] The conclusion itself leaves improving the search procedure to future work. The evaluation provides limited support for a broader claim: the text tasks are small benchmark subsets, the BigCodeBench federation has only one reported evolution run and a 1.53-point OOD gain, and OOD evaluation includes replay of 24 labeled target-task history items and access to public example retrieval (Appendix D). Thus the evidence more directly supports federated harness transfer than a distinctive, experimentally established recursive self-improvement mechanism.

Re: The run records contain the multi-hop feedback that the reviewer describes. A client adopts a shared module. A later search of this client starts from the adopted harness and keeps a new module. The server rebuilds the new module from its card, and a third client adopts the rebuilt module. Specifically, in the records of the new runs, we traced three things: the cards from which the server rebuilt every shared module, every adoption of a shared module, and the harness from which every search started. The results show that such a lineage occurs in 5 of the new runs.

(1) The recursive feedback. Such a lineage is the recursion that Section 3.3 defines. The next local search of a client starts from the adopted harness. An improvement found by one client therefore shapes the search of a second client, and the next improvement of the second client then reaches a third client. Our response to Q2 follows one lineage step by step. In that lineage, the second client's proposer writes the new module with code taken from the imported module. The imported module thus becomes material for the second client's next improvement, which in turn benefits a third client.

(2) The evaluation. FedRSI's other-task gain remains without the replayed history. The search over public labelled examples is available to every configuration, independent evolution included. Specifically, we evaluated the three new text runs again without replaying the target task's labelled history, and we conducted three new runs of the coding federation. Our response to Q3 gives the numbers of both. Additionally, we conducted two runs of a new federation of five clients. The five clients serve the three text tasks, FiNER (tagging financial entities) and Aegis2 (detecting unsafe user messages). The results show the same transfer in this federation: FedRSI exceeds independent evolution on other tasks by 4.2 points [+2.8, +5.6] (176 items won, 58 lost).

AuthorsOpenReview ↗

Authors' Response to Reviewer v54N (3/5)

Q1. Does allowing exchange during evolution outperform a single post-hoc exchange/reimplementation after independent evolution, when local search, server calls, and validation resources are matched? A resource-matched independent-search comparison would also help establish the incremental value of the collective protocol.

Re: (1) Does exchange during evolution outperform a single post-hoc exchange? The single post-hoc exchange is FedRSI's own protocol run once. With the paper's schedule, in which a client searches after every 6 items, the single post-hoc exchange transfers as much as exchange during evolution, and both exceed independent evolution on other tasks. Specifically, we conducted the single post-hoc exchange as a control, with three new runs per configuration. In this control, independent evolution is followed by one federation round (see our response to W1). The results show that own-task accuracy is 60.63% for FedRSI and 60.82% for this control, and that other-task accuracy is 53.87% for FedRSI and 53.40% for this control. FedRSI and this control run the same protocol with the same local search. The equal accuracy therefore shows that the protocol, not additional resources, produces the transfer. FedRSI applies the protocol during evolution for two reasons: FedRSI's clients evolve online at their own pace, and only an exchange during evolution lets a client's next search build on an imported improvement. Exchange during evolution also transfers more when the clients search more often: with a search after every 3 items instead of 6, FedRSI's other-task accuracy is 5.9 points higher (see our response to W1).

(2) A resource-matched independent-search comparison. Additional resources do not account for FedRSI's gain. Specifically, we conducted a control in which independent evolution gets twice FedRSI's proposer sessions (three new runs; the resource table is in our response to W1). The results show that this control costs 55% more than FedRSI and is 10.0 points behind FedRSI on other tasks [+8.0, +12.1].

Reviewer_6ms4OpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers a federation of agents that share one frozen language model but each evolves a private harness, the program that handles memory, context, tools, and model calls. Each owner improves its harness by a local search that adds modules and keeps one only if accuracy on private validation items rises. The search procedure itself is fixed. Owners will not share data, traces, or code. The thesis is that a prose description of what a kept module does and why is enough for another party to rebuild it. Improvements can then move between owners without their raw data or private source code.

The paper's protocol, FedRSI, has four steps. A client writes a card (the paper calls it a mechanism card) for each kept module, screened for code, literals, and planted canaries. When enough cards arrive, the server writes a build specification per mechanism (the paper calls it a work order). A language-model judge checks the assumptions behind each mechanism and accepts it, revises it with conditions, or defers it. A coding agent then implements it from scratch into a shared harness, without seeing the cards or any client code. Each client combines the shared harness with its private modules. Where a shared module and a private module do the same job, the client keeps its private one on its own task and the shared one on other tasks. On its own task, the client switches to the new combination only if it scores higher than the current harness on its private validation items. On other tasks it serves a combination built from the final shared harness, with shared modules winning, and that combination passes no such test.

For this setting the authors build their own testbed. The datasets and the online task format are existing ones; the federation, the harness runtime, and the seed harness are theirs. Three text clients (English diagnosis, Chinese charges, reactant SMILES) start from a seed harness with memory disabled and the environment's example search unregistered, and every client's environment offers the same tools. The comparator is the same local search without exchange (the paper calls it independent evolution), run three times. Averaged over runs, own-task macro accuracy rises 4.3 points and other-task accuracy 4.6 points. The main text reports one run selected on FedRSI's own evolution-time accuracy, in which only one client adopted any shared module, and compares single-run ablations against it. FedRSI also adds server-side building and client-side adoption tests on top of the shared local search, and no configuration gives the comparator the same extra building and selection.

Three further results address generalization and two of the five properties the protocol promises, private code and asynchrony. Generalization beyond text rests on a second federation of three coding clients, run once. It gains 1.5 points on other tasks. Its own-task accuracy, the number that leads the abstract for the text federation, is not reported. The property that no client data leaves is checked by an audit. In the two audited text runs, a scan detects no path, label value, input span, or planted canary in any exchanged card, specification, or module. A simulated attacker, given the shared modules and the cards, recovers no canary or label value, and its membership accuracy does not differ significantly from chance. The property that no client waits holds by construction. Its quantification is a projection of measured durations under a hypothetical synchronous schedule.

Critical strengths
  1. The problem is real and the protocol makes the information boundary checkable. Owners that will not share code or data are the constraint the paper starts from, and prior federations either share one component or let the server read code. Fixing that only screened prose leaves a client, and that the implementer sees work orders rather than cards, turns the privacy claim into something that can be audited.

  2. A harness rebuilt from prose alone reaches the same reported macro accuracy as the harnesses its owners evolved with their own data. The shared harness scores 59.43% on the clients' own tasks, the macro accuracy of the independently evolved harnesses and twice that of the seed harness.

  3. Unfavorable results are reported where they occur. For example, removing the adoption test raises accuracy on other tasks from 58.86% to 64.13%, and the main text reports it. The value is above all three FedRSI runs (Table 13). The pilot in which one client fell from 55% to 0% is described in Section 4.3. The limitation on task-membership judgment is stated.

Critical weaknesses
  1. No configuration separates the information carried by other clients' cards from the additional candidate modules that the server's coding agent adds to FedRSI. Without it, how much of the gain the cards account for cannot be determined. Both configurations run 24 proposer sessions (Table 6). FedRSI's server adds four implementer sessions committing seven shared modules, and clients run six adoption tests on compositions including them (Tables 14 and 15). The modules that transferred register the corpus search the environment offers to every client and align answers with retrieved examples. Whether a builder that sees the runtime interface would reach these without a card is untested.

  2. The two protocol ablations are single runs compared against a run selected on FedRSI's own evolution accuracy, so their effects cannot be separated from run-to-run variation. The reported run has the highest evolution-time accuracy of the three (lines 346–347). The ablations of the review step and the adoption test run once each (line 1193), and the main text compares them with that run (lines 407–409). The three FedRSI runs differ among themselves in outcome (Table 13) and in which modules were adopted (Table 14), and nothing in the design estimates this variation for the ablated configurations. The main text notes this control only for the composition-order comparison (lines 375–376). This is concerning because the quantitative trade-off claimed in Section 4.3 rests on these two single-run differences alone.

Questions for authors
  1. If the baseline were given extra proposer sessions to try as many candidate modules as FedRSI does, how much of the gain on own and other tasks would remain?
  2. Did any independently evolved client find the mechanisms that the shared modules implement, such as the corpus search or the alignment of answers with retrieved examples?
  3. What is the run-to-run spread of the own-task and other-task macros for FedRSI and for the comparator?
8 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer 6ms4 (1/5)

W1. No configuration separates the information carried by other clients' cards from the additional candidate modules that the server's coding agent adds to FedRSI. Without it, how much of the gain the cards account for cannot be determined. Both configurations run 24 proposer sessions (Table 6). FedRSI's server adds four implementer sessions committing seven shared modules, and clients run six adoption tests on compositions including them (Tables 14 and 15). The modules that transferred register the corpus search the environment offers to every client and align answers with retrieved examples. Whether a builder that sees the runtime interface would reach these without a card is untested.

Re: FedRSI's gain on other tasks is produced by its protocol. In this protocol, the server re-implements mechanisms on the shared harness, each recipient runs its adoption test, and each client gives the shared modules priority on other tasks. Additional proposals (candidate modules) do not produce this gain, and this gain needs nothing that a card reveals about its client. Specifically, we conducted two controls that separate the information in the cards from the additional proposals, with three new runs each (base model: GLM-5.3-Flash).

(1) Do the additional candidate modules account for the gain? No. Specifically, we conducted a control in which independent evolution scores four proposals per search instead of two. This control has 48 proposer sessions per run, against 24 for FedRSI. The results show that the control's other-task accuracy is 43.85%, which is below that of plain independent evolution (48.38%). FedRSI exceeds the control on other tasks by 10.0 points [+8.0, +12.1]. Our response to Q1 gives the full comparison.

(2) Would a builder that sees the runtime interface reach the transferred mechanisms without a card? Yes. Specifically, we conducted a control in which FedRSI's server reads no card text. The server runs the same work-order, review and implementation steps, and the clients run the same adoption tests. The server decides what to build from the runtime interface, the shared source code and its own record of earlier decisions. The results show that in every run the server builds modules that put the environment's labelled examples into the context or expose a search over these examples. In every run, the server also builds a module that stores and retrieves experience. The control's other-task accuracy is above that of independent evolution in every run, by 6.0, 4.8 and 15.2 points (57.02% against 48.38% on average).

The card is the part of the protocol that lets a client contribute what its local search found without disclosing data or code. None of the 17 cards, 16 work orders and 15 shared modules of the audited runs contains a label value, a file path, a span of a served input or the planted canary. In contrast, 7 of the 18 private modules contain true labels (Appendix F.10).

The revised paper includes both controls in Appendix F.13.

AuthorsOpenReview ↗

Authors' Response to Reviewer 6ms4 (2/5)

W2. The two protocol ablations are single runs compared against a run selected on FedRSI's own evolution accuracy, so their effects cannot be separated from run-to-run variation. The reported run has the highest evolution-time accuracy of the three (lines 346–347). The ablations of the review step and the adoption test run once each (line 1193), and the main text compares them with that run (lines 407–409). The three FedRSI runs differ among themselves in outcome (Table 13) and in which modules were adopted (Table 14), and nothing in the design estimates this variation for the ablated configurations. The main text notes this control only for the composition-order comparison (lines 375–376). This is concerning because the quantitative trade-off claimed in Section 4.3 rests on these two single-run differences alone.

Re: Transfer review and local validation are justified independently of run-to-run variation, because both checks prevent the failure of our pilot study (Section 4.3). With three runs of each ablation, transfer review shows the trade-off that Section 4.3 reports, and local validation also protects other-task accuracy. In the pilot study, which ran without either check, USPTO's validation accuracy fell from 55% to 0% (Appendix F.9). We thank the reviewer for the suggestion to separate the effects of the ablations from run-to-run variation. Specifically, we conducted each ablation three times with GLM-5.3-Flash as the base model (new runs). We compared each ablation with the three new runs of FedRSI, item by item over the held-out items. The results show that:

  • Transfer review. Without transfer review, other-task accuracy is 3.1 points higher than with it [+0.7, +5.5] (p=0.04p = 0.04), and own-task accuracy is 0.8 points lower [−3.8, +2.3]. Transfer review thus restricts what transfers, as Section 4.3 reports.
  • Local validation (the client's adoption test). With local validation, other-task accuracy is 2.7 points higher than without it [+0.5, +4.8], and own-task accuracy differs by −1.9 points [−4.4, +0.5].

Accuracy (%), mean ± s.d. over the three new runs of each configuration:

ConfigurationOwn tasks (ID)Other tasks (OOD)
FedRSI60.63 ± 2.3753.87 ± 4.17
FedRSI without transfer review59.81 ± 2.7956.97 ± 4.81
FedRSI without local validation62.56 ± 4.6151.19 ± 7.12

The revised paper includes these three-run ablations in Section 4.3 and Appendix F.14.

AuthorsOpenReview ↗

Authors' Response to Reviewer 6ms4 (3/5)

Q1. If the baseline were given extra proposer sessions to try as many candidate modules as FedRSI does, how much of the gain on own and other tasks would remain?

Re: On other tasks, all of FedRSI's gain remains against a baseline with twice FedRSI's proposer sessions. The gain grows from 5.5 points against plain independent evolution to 10.0 points [+8.0, +12.1] against this baseline. On own tasks, FedRSI reaches this baseline's accuracy when FedRSI is given the same number of proposer sessions. Specifically, we conducted a control in which independent evolution scores four proposals (candidate modules) per search instead of two (three new runs with GLM-5.3-Flash as the base model). This baseline has 48 proposer sessions per run, against 24 for FedRSI. A run of this baseline costs 25.7 USD, against 16.6 USD for a run of FedRSI. The results show that:

Configuration (three new runs)Proposer sessionsOwn tasks (ID), %Other tasks (OOD), %
Independent evolution2461.2348.38
Independent evolution, four proposals per search4864.2043.85
FedRSI2460.6353.87
  • Other tasks. The additional proposals specialize each harness further to its own task. Independent evolution's other-task accuracy is 43.85% with four proposals per search and 48.38% with two. FedRSI exceeds the baseline with four proposals per search by 10.0 points on other tasks [+8.0, +12.1] (164 items won, 22 lost).
  • Own tasks. The additional proposals raise independent evolution's own-task accuracy to 64.20%. FedRSI has the same 48 proposer sessions when its clients search after every 3 items instead of 6 (two runs). With these 48 proposer sessions, FedRSI reaches the same own-task accuracy (64.03%). On these two runs, FedRSI also exceeds the baseline on other tasks by 9.0 points [+6.6, +11.4].

The revised paper includes this control in Appendix F.13.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank all reviewers for their careful and constructive reviews.

Strengths recognized by the reviewers

  • A real problem and a concrete protocol. "The problem is real and the protocol makes the information boundary checkable" (6ms4). FedRSI is "a concrete protocol for transferring executable mechanisms under restricted sharing", and its "mechanism cards, server-side reimplementation, local adoption tests, and composition priorities address practical problems beyond exchanging prompts" (v54N). "The proposed framework is reasonably well structured" and "Overall the paper is well-organized" (TUGG).
  • An auditable privacy boundary. "Fixing that only screened prose leaves a client, and that the implementer sees work orders rather than cards, turns the privacy claim into something that can be audited" (6ms4).
  • Transfer that works. The case study "establishes that useful cross-client mechanism transfer is possible" (v54N), and "the experiments provide some evidence that sharing these improvements can benefit agents across different tasks" (TUGG).
  • A replicated and transparent evaluation. "The paper reports replicated text results and informative negative cases", "with paired item-level analyses" (v54N). "Unfavorable results are reported where they occur" (6ms4).

New evidence

The new evidence confirms the paper's central finding that an improvement found at one client benefits the others. Specifically, following the reviewers' suggestions, we conducted new runs and controls and traced the run records:

New evidenceResult
Text federation, three runs with a stronger base modelFedRSI exceeds independent evolution on other clients' tasks by 5.5 points [+3.4, +7.6] (82 items won, 30 lost, p<0.0001p<0.0001)
Federation of five clients, two runsFedRSI exceeds independent evolution on other tasks by 4.2 points [+2.8, +5.6] (176 won, 58 lost, p<0.0001p<0.0001)
Independent evolution with twice FedRSI's proposer sessions, at 55% higher cost than FedRSIFedRSI exceeds this baseline on other tasks by 10.0 points [+8.0, +12.1] (164 won, 22 lost)
Evaluation without replaying the target task's labelled historyFedRSI exceeds independent evolution on other tasks by 5.0 points [+3.2, +6.8] (79 won, 24 lost)
Coding federation, three new runsFedRSI is above independent evolution in every run on own tasks and is ahead on other tasks (112 won, 80 lost, p=0.025p=0.025)
Lineages in the run recordsIn 5 of the new runs, a client adopts a shared module, its later search keeps a new module, the server rebuilds this module from its card, and a third client adopts the rebuilt module

All new runs use GLM-5.3-Flash, the stronger base model on which Appendix F.4 evaluates the evolved harnesses, and the submission's seed harness, schedule, validation sets and evaluation protocol. The seed harness scores 31.45% with GLM-5.3-Flash over three new runs, against 28.62% with Qwen3.8-Flash in the submission's runs. Differences are paired by held-out item, with 95% bootstrap confidence intervals, items won and lost, and two-sided sign tests.

Questions raised by several reviewers

Does the gain come from additional resources rather than from the exchange? (6ms4 W1 and Q1, v54N W1 and Q1) No. The exchange produces the gain, and additional resources do not. Specifically, we conducted two controls with three runs each: independent evolution with twice FedRSI's proposer sessions, and independent evolution followed by a single federation round of FedRSI. The results show that the first control reaches 43.85% on other tasks, below plain independent evolution (48.38%) and 10.0 points below FedRSI. The second control raises other-task accuracy above independent evolution in every run.

Are the results stable across runs, including the coding federation? (6ms4 W2 and Q3, v54N Q3, TUGG W4) Yes. Specifically, FedRSI, independent evolution and every control of the text federation have three runs, and the coding federation has three new runs. The results show that in the text federation FedRSI's run-to-run standard deviation is smaller than that of independent evolution on both macros, in the submission's runs and in the new runs. In the coding federation, FedRSI is above independent evolution in every run on own tasks and is ahead on other tasks.

What the revised paper adds

  • Section 4.2: the three-run means with their standard deviations next to the reported run.
  • Section 4.3: the three-run ablations.
  • Appendix F: the new controls with their resource table, the five-client federation, and the evaluation without replaying the labelled history.
  • The coding federation over three runs, on own and other tasks.
  • The traced lineages, the privacy threat model, the measurement of the task-membership judgment, and the test of the global-first composition on own tasks.
  • Section 3: each component presented by what it takes in, what it does and what it produces.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.