- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
SkillGit lets several federated clients improve a shared agent skill through semantic commits, server-side merging, validation, and rollback. It performs better than independent client evolution and a one-shot merge baseline in the reported experiments.
Critical strengths
The paper studies a relevant setting where experience is spread across clients and raw traces may not be pooled. The use of semantic commits, provenance, rollback, and validation makes the workflow easy to follow and audit.
Critical weaknesses
The evaluation only partly matches the deployment motivation. The authors do construct deliberately non-IID client splits by question type, difficulty, and answer format, which is useful. Still, these clients are created from benchmark partitions rather than naturally separate institutions, and the paper does not test paraphrased or inferential leakage or measure network-level communication. The literal-value audit and task results therefore support a limited payload-sanitization claim, but not the broader privacy and real-world communication claims.
It is also hard to judge the novelty over prior federated skill evolution because FederatedSkill is discussed but not evaluated under the same conditions. I was also confused by the commit-ID description. If the hash includes the client, slice, round, and evidence, then two clients making the same correction should still get different IDs, unless the method uses a separate canonical key for deduplication.
Questions for authors
Please test the claimed deployment constraints more directly. Report total bytes or tokens sent and the number of rounds, evaluate paraphrased or inferential leakage, and add naturally occurring or cross-institutional heterogeneity beyond the constructed non-IID benchmark partitions already reported.
Please compare with FederatedSkill using the same partitions, validator, optimizer, and communication budget, or explain clearly why that baseline cannot be run. Also define exactly what goes into the commit-ID hash and whether another canonical key is used to merge semantically identical edits.
2 comments
Response to Reviewer ZmNz—W2 + Q2. FederatedSkill Comparison and Commit Identification (2/2)
W2. FederatedSkill is discussed but was not evaluated under comparable conditions, and a hash containing client identity, private slice, round, and evidence cannot also deduplicate identical cross-client edits.
Q2. Compare with FederatedSkill using aligned experimental conditions or explain why exact alignment is not possible; define the commit-ID inputs and whether a separate canonical key handles identical edits.
We thank the reviewer for identifying these two important points, and we apologize that the original presentation did not make the baseline comparison and the distinction between commit provenance and deduplication sufficiently clear.
Additional baseline experiment. We have added FederatedSkill to the main three-dataset, three-model evaluation, using the same controlled task partitions, model settings, participation schedule, and per-round exposure where applicable, while preserving its native personalized-library organization. The reported TSR values are averaged over three random seeds.
Response Table R3-2. FederatedSkill comparison (TSR, %).
| Model | Dataset | FederatedSkill | SkillGit | Gain (pp) |
|---|---|---|---|---|
| GPT-5.4-mini | SearchQA | 77.7 | 78.6 | +0.9 |
| LiveMath | 27.2 | 28.8 | +1.6 | |
| DocVQA | 85.3 | 88.5 | +3.2 | |
| Qwen3.5-flash | SearchQA | 80.3 | 82.4 | +2.1 |
| LiveMath | 24.8 | 29.6 | +4.8 | |
| DocVQA | 80.2 | 85.3 | +5.1 | |
| Gemini3-flash | SearchQA | 82.6 | 87.3 | +4.7 |
| LiveMath | 35.2 | 40.0 | +4.8 | |
| DocVQA | 87.7 | 90.1 | +2.4 |
SkillGit achieves higher TSR in all nine comparisons, with gains between 0.9 and 5.1 percentage points and an average improvement of 3.3 points. This evidence supports the effectiveness of integrating distributed updates into one validated shared skill in the studied setting. We also clarify the conceptual distinction: FederatedSkill evolves personalized multi-skill libraries, while SkillGit treats locally proposed rule edits as versioned contributions to a common skill. Consequently, their output structures and validation targets are not identical, even when the underlying tasks and training exposure are aligned.
Commit identification and deduplication. We now distinguish two identifiers with different purposes:
- Provenance identifier: . It records the parent version, client identity, private-slice hash, communication round, and commit payload hash. Thus, two clients that submit the same edit may correctly have different provenance identifiers.
- Canonical edit key: It is computed from the parent-document hash and the normalized edit operation, target rule, and revision content. It excludes client identity and the private-slice/round fields, enabling identical normalized edits from different clients to be recognized as duplicates. Similar but nonidentical revisions are left for the merge agent's semantic adjudication rather than automatically collapsed by hashing. The provenance ID therefore supports attribution and version lineage, while the canonical key supports cross-client deduplication. These mechanisms are complementary, not interchangeable.
Where revised: The FederatedSkill objective and architectural distinction are presented in Section 2.3 (Lines 105–120). Its baseline settings and results have been added to Section 4.1 (Lines 342–368) and Section 4.2, Table 1 (Lines 378–415), with protocol details in Appendix C (Lines 943–969). The two identifiers and their inputs are defined in Section 3.3.1, Eqs. (5)–(6) (Lines 235–243), and their role in selective merge is further described in Section 3.4.2 (Lines 298–310).
Response to Reviewer ZmNz—W1 + Q1. Deployment Constraints, Privacy Scope, and Heterogeneity (1/2)
We thank the reviewer for highlighting the importance of evaluating the stated deployment constraints. Following this feedback, we have expanded the revision with quantitative communication measurements, client-participation/convergence analyses, and a controlled sanitizer ablation. We have also clarified that the current experimental setting is controlled non-IID benchmark heterogeneity, rather than a deployment involving independently administered institutions.
We construct heterogeneous client partitions by classifying tasks and varying their proportions across clients, including differences in question type, difficulty, and answer format. This setting enables reproducible comparisons under nonidentical client distributions. The new measurements are summarized below.
Response Table R3-1. Communication, convergence, and privacy-filtering analyses.
The communication results show 67% lower mean bytes and 62% lower mean tokens per transmitted commit than for a complete document. The sensitivity analyses reveal the expected operating trade-off: increasing the client population from 10 to 100 lengthens convergence from five to nine rounds, whereas increasing participation from 10% to 50% shortens it from thirteen to three rounds. The sanitization ablation further shows a 48-percentage-point increase in first-pass audit clearance. Together, these measurements provide more direct evidence of compact update payloads, convergence behavior, and the effect of the privacy-filtering stage in the controlled setting.
We also appreciate the reviewer's distinction between semantic screening and residual privacy risk. The L2 auditor is designed to check direct and paraphrased disclosure against client-local private values, while the threat model allows an honest-but-curious server to associate commits across rounds. Nevertheless, the first-pass clearance experiment is not an independent measurement of paraphrase-specific false negatives, inferential leakage, or an adversary's reconstruction success. Likewise, the current communication results measure payloads rather than complete network-layer costs, and the non-IID partitions are benchmark-derived rather than naturally cross-institutional. We have made these limits explicit rather than claiming that the current experiments establish stronger guarantees or real-world deployment coverage.
Where revised: The controlled non-IID partition protocol is documented in Section 4.1 (Lines 342–350) and Appendix C (Lines 948–961). We clarify the semantic auditor and privacy threat model in Section 3.3.2 (Lines 245–276) and the inference limitation in Appendix B (Lines 924–930). The new sanitization results are in Section 4.3.3, Table 4 (Lines 458–473), communication payloads in Section 4.4, Table 5 (Lines 475–492), and convergence/participation results in Section 4.5, Figures 2–3 (Lines 494–517).