- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a federated agentic framework to enable the cross-task harness transfer without revealing the private data or code that reside on the local agents. Further experiments show that this federation design outperforms the standalone agents, and this transferred harness particularly benefits out-of-domain tasks.
Critical strengths
In this work, the authors focus on transfer executable harness improvements across agents without exposing their private data or source code. The proposed framework is reasonably well structured, and the experiments provide some evidence that sharing these improvements can benefit agents across different tasks. Overall the paper is well-organized.
Critical weaknesses
-
The authors emphasize asynchronous harness aggregation as a key contribution, but it is unclear why synchronization would be necessary in the first place. Unlike traditional federated learning, where asynchronous updates to shared model weights can introduce staleness and consistency issues, each agent here independently evolves its own harness. The server could simply process incoming improvements in a first-come-first-served manner without blocking other clients, so a more meaningful baseline would be straightforward incremental integration, instead a synchronous setup that unnecessarily forces clients to wait.
-
Privacy concerns. In FL, local training data remain on clients, but model updates are not automatically encrypted in a way that hides them from the server. Differ from numerical FL updates, arbitrary natural-language mechanism cards cannot simply be encrypted and summed, but this is insufficient to establish comprehensive confidentiality.
-
The paper evaluates two federations, each containing only three clients, the experimental evidence is too limited for the generality of the claims
-
In Appendix F.3, the coding experiment is much less convincing than the text experiment, only 1.53% performance gain compared to independent evolution
-
Despite the authors report 100% routing accuracy, the task-aware routing is tested on highly distinct domains, this does not establish robustness when tasks have similar inputs or gradually changing distributions
-
FedSI uses different rules when it comes to competing harness module pairs, and the priority is determined by whether the task is local or external. However, why should either be assumed to perform better? The server may reconstruct a better implementation of the same functionality, but local-first does not appear to consider this possibility.
-
Although the overall narrative is reasonable, I found Section 3 unnecessarily difficult to follow. Many of the components are conceptually straightforward, but their descriptions are lengthy, the writing feels somewhat over-polished with LLMs, which does not help with readability.
Questions for authors
See the weaknesses listed above
9 comments
Authors' Response to Reviewer TUGG (2/7)
W2. Privacy concerns. In FL, local training data remain on clients, but model updates are not automatically encrypted in a way that hides them from the server. Differ from numerical FL updates, arbitrary natural-language mechanism cards cannot simply be encrypted and summed, but this is insufficient to establish comprehensive confidentiality.
Re: FedRSI's confidentiality claim is that no client record, no label and no line of code leaves a client. The protocol establishes this claim by construction, and the paper verifies this claim with an audit and an attack. Specifically, the paper scans every artifact of two runs of the text federation and attacks the reported run with a simulated client (Appendix F.10).
(1) What leaves a client. By construction, only the card leaves a client. A card is plain prose, and the client screens it before upload for literals and for the canary strings planted in the served inputs. The server writes work orders from the cards, and the server's coding agent (the implementer) reads the work orders and never the cards.
(2) The audit. The results show that none of the 17 cards, 16 work orders and 15 shared modules contains a file path of our deployment, a dataset name, a label value, a verbatim span of a served input or the canary. In contrast, 7 of the 18 private modules contain true labels, which a federation that exchanged code would disclose. A judge model that reads each card without context identifies the domain of the card's author for only 1 of the 17 cards.
(3) The attack. The results show that, given the shared modules, the cards and the work orders, the simulated attacker quotes no canary and names no label of another client. Given one client's private modules, the attacker names 8 to 9 of the 23 labels that this client served. Given the served records, the attacker quotes the canary in every attack.
The revised paper states this threat model explicitly in Appendix F.10.
Authors' Response to Reviewer TUGG (3/7)
W3. The paper evaluates two federations, each containing only three clients, the experimental evidence is too limited for the generality of the claims.
Re: The transfer holds in a larger federation and with a stronger base model. Specifically, we conducted two runs of a federation of five clients with GLM-5.3-Flash as the base model. The five clients serve the three text tasks, FiNER (tagging financial entities) and Aegis2 (detecting unsafe user messages), one task each, with the same schedule for every client. The results show that FedRSI exceeds independent evolution on other tasks by 4.2 points [+2.8, +5.6] (176 items won, 58 lost; ), and that own-task accuracy is unchanged (+1.4 points [−0.9, +3.8]).
The text federation also has three new runs with GLM-5.3-Flash, in which FedRSI exceeds independent evolution on other tasks by 5.5 points [+3.4, +7.6]. The coding federation has three new runs as well (see our response to W4).
Authors' Response to Reviewer TUGG (1/7)
W1. The authors emphasize asynchronous harness aggregation as a key contribution, but it is unclear why synchronization would be necessary in the first place. Unlike traditional federated learning, where asynchronous updates to shared model weights can introduce staleness and consistency issues, each agent here independently evolves its own harness. The server could simply process incoming improvements in a first-come-first-served manner without blocking other clients, so a more meaningful baseline would be straightforward incremental integration, instead a synchronous setup that unnecessarily forces clients to wait.
Re: (1) Why would synchronization be necessary? FedRSI does not need synchronization (Section 3.2). The paper compares FedRSI with synchronous rounds because every prior agent federation aggregates once or in synchronous rounds. A synchronous round ends only when its slowest client has finished. With the durations that we measured for each client, synchronous rounds would have kept the faster clients idle for up to 59% of a run. FedRSI's clients never wait. FedRSI needs no synchronization because its clients share mechanism cards. A card describes a behavior and refers to no version of the shared harness, so the server can integrate a card whenever it arrives (Appendix F.11). In contrast, a parameter update or a code diff refers to a version of the shared state. Such an update must therefore be rebased or weighted for staleness when it arrives late.
(2) First-come-first-served integration as the baseline. This baseline is FedRSI with a buffer of one card (B=1), and FedRSI's default buffer of two cards (B=2) transfers better. We believe the paper's buffered aggregation already contains this baseline. The server opens a federation round as soon as B new cards have arrived (Section 3.2). With B=1, the server therefore integrates every card on arrival. Specifically, we conducted three new runs of FedRSI with B=1 on the text federation. The results show that other-task accuracy is 3.1 points higher with B=2 than with B=1 [+1.1, +5.2], and that own-task accuracy is unchanged (−0.5 points [−3.1, +2.1]).