acceptodds
Under review as a conference paper at ICLR 2027

SkillGit: Federated Skill Evolution via Version Control

Abstract

Large language model agents can continuously improve their capabilities by evolving reusable natural-language skills from execution experience. In real-world deployments, however, such experience is often distributed across multiple data owners and cannot be directly pooled due to privacy constraints, which naturally motivates federated skill evolution. A straightforward solution is to let each client independently evolve the shared skill and upload the resulting document for server-side aggregation. However, this document-level update abstraction is poorly suited to skill evolution. Specifically, i) an evolved skill may still encode client-specific information distilled from private traces, ii) repeated transmission of the entire document incurs unnecessary communication when only a small portion is modified, and iii) the actual knowledge contributed by a client is buried within a largely unchanged skill document, making it difficult to identify and evaluate the useful revisions. To address this issue, we propose SkillGit, a federated skill evolution framework that reformulates cross-client collaboration from full-document sharing to semantic revision sharing. Instead of uploading complete evolved skills, each client submits compact semantic commits that explicitly describe the intended changes to a shared skill version. SkillGit further introduces a commit-aware aggregation strategy that evaluates and integrates client revisions according to their contribution to the shared skill, allowing complementary knowledge to accumulate while filtering redundant or harmful updates. To stabilize the evolution process, we design a version validation mechanism that evaluates newly merged skill versions with distributed client feedback and rolls back regressions when necessary. Experimental results across three datasets and three LLM families demonstrate that SkillGit outperforms state-of-the-art baselines by up to 22% and approaches centralized evolution without sharing raw client data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Federated Learning,Skill Evolution,LLM Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZmNzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SkillGit lets several federated clients improve a shared agent skill through semantic commits, server-side merging, validation, and rollback. It performs better than independent client evolution and a one-shot merge baseline in the reported experiments.

Critical strengths

The paper studies a relevant setting where experience is spread across clients and raw traces may not be pooled. The use of semantic commits, provenance, rollback, and validation makes the workflow easy to follow and audit.

Critical weaknesses

The evaluation only partly matches the deployment motivation. The authors do construct deliberately non-IID client splits by question type, difficulty, and answer format, which is useful. Still, these clients are created from benchmark partitions rather than naturally separate institutions, and the paper does not test paraphrased or inferential leakage or measure network-level communication. The literal-value audit and task results therefore support a limited payload-sanitization claim, but not the broader privacy and real-world communication claims.

It is also hard to judge the novelty over prior federated skill evolution because FederatedSkill is discussed but not evaluated under the same conditions. I was also confused by the commit-ID description. If the hash includes the client, slice, round, and evidence, then two clients making the same correction should still get different IDs, unless the method uses a separate canonical key for deduplication.

Questions for authors

Please test the claimed deployment constraints more directly. Report total bytes or tokens sent and the number of rounds, evaluate paraphrased or inferential leakage, and add naturally occurring or cross-institutional heterogeneity beyond the constructed non-IID benchmark partitions already reported.

Please compare with FederatedSkill using the same partitions, validator, optimizer, and communication budget, or explain clearly why that baseline cannot be run. Also define exactly what goes into the commit-ID hash and whether another canonical key is used to merge semantically identical edits.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZmNz—W1 + Q1. Deployment Constraints, Privacy Scope, and Heterogeneity (1/2)

W1. The deployment motivation is supported only in part: benchmark-constructed non-IID clients do not establish cross-institutional applicability, and direct paraphrased/inferential leakage and network-level communication have not been fully evaluated.

Q1. Please report communication volume and rounds, evaluate paraphrased or inferential leakage, and add naturally occurring or cross-institutional heterogeneity beyond constructed benchmark partitions.

We thank the reviewer for highlighting the importance of evaluating the stated deployment constraints. Following this feedback, we have expanded the revision with quantitative communication measurements, client-participation/convergence analyses, and a controlled sanitizer ablation. We have also clarified that the current experimental setting is controlled non-IID benchmark heterogeneity, rather than a deployment involving independently administered institutions.

We construct heterogeneous client partitions by classifying tasks and varying their proportions across clients, including differences in question type, difficulty, and answer format. This setting enables reproducible comparisons under nonidentical client distributions. The new measurements are summarized below.

Response Table R3-1. Communication, convergence, and privacy-filtering analyses.

AspectConfigurationObserved result
CommunicationComplete skill document5.1 KB / 1.3k tokens
Single semantic commit1.7 KB / 0.5k tokens
Three commits in one round5.1 KB / 1.5k tokens
Convergence10 → 100 clients at 30% participation5 → 9 rounds
10% → 50% participation with 10 clients13 → 3 rounds
Privacy filteringFirst-pass L2 clearance without sanitizer33%
First-pass L2 clearance with sanitizer81%

The communication results show 67% lower mean bytes and 62% lower mean tokens per transmitted commit than for a complete document. The sensitivity analyses reveal the expected operating trade-off: increasing the client population from 10 to 100 lengthens convergence from five to nine rounds, whereas increasing participation from 10% to 50% shortens it from thirteen to three rounds. The sanitization ablation further shows a 48-percentage-point increase in first-pass audit clearance. Together, these measurements provide more direct evidence of compact update payloads, convergence behavior, and the effect of the privacy-filtering stage in the controlled setting.

We also appreciate the reviewer's distinction between semantic screening and residual privacy risk. The L2 auditor is designed to check direct and paraphrased disclosure against client-local private values, while the threat model allows an honest-but-curious server to associate commits across rounds. Nevertheless, the first-pass clearance experiment is not an independent measurement of paraphrase-specific false negatives, inferential leakage, or an adversary's reconstruction success. Likewise, the current communication results measure payloads rather than complete network-layer costs, and the non-IID partitions are benchmark-derived rather than naturally cross-institutional. We have made these limits explicit rather than claiming that the current experiments establish stronger guarantees or real-world deployment coverage.

Where revised: The controlled non-IID partition protocol is documented in Section 4.1 (Lines 342–350) and Appendix C (Lines 948–961). We clarify the semantic auditor and privacy threat model in Section 3.3.2 (Lines 245–276) and the inference limitation in Appendix B (Lines 924–930). The new sanitization results are in Section 4.3.3, Table 4 (Lines 458–473), communication payloads in Section 4.4, Table 5 (Lines 475–492), and convergence/participation results in Section 4.5, Figures 2–3 (Lines 494–517).

AuthorsOpenReview ↗

Response to Reviewer ZmNz—W2 + Q2. FederatedSkill Comparison and Commit Identification (2/2)

W2. FederatedSkill is discussed but was not evaluated under comparable conditions, and a hash containing client identity, private slice, round, and evidence cannot also deduplicate identical cross-client edits.

Q2. Compare with FederatedSkill using aligned experimental conditions or explain why exact alignment is not possible; define the commit-ID inputs and whether a separate canonical key handles identical edits.

We thank the reviewer for identifying these two important points, and we apologize that the original presentation did not make the baseline comparison and the distinction between commit provenance and deduplication sufficiently clear.

Additional baseline experiment. We have added FederatedSkill to the main three-dataset, three-model evaluation, using the same controlled task partitions, model settings, participation schedule, and per-round exposure where applicable, while preserving its native personalized-library organization. The reported TSR values are averaged over three random seeds.

Response Table R3-2. FederatedSkill comparison (TSR, %).

ModelDatasetFederatedSkillSkillGitGain (pp)
GPT-5.4-miniSearchQA77.778.6+0.9
LiveMath27.228.8+1.6
DocVQA85.388.5+3.2
Qwen3.5-flashSearchQA80.382.4+2.1
LiveMath24.829.6+4.8
DocVQA80.285.3+5.1
Gemini3-flashSearchQA82.687.3+4.7
LiveMath35.240.0+4.8
DocVQA87.790.1+2.4

SkillGit achieves higher TSR in all nine comparisons, with gains between 0.9 and 5.1 percentage points and an average improvement of 3.3 points. This evidence supports the effectiveness of integrating distributed updates into one validated shared skill in the studied setting. We also clarify the conceptual distinction: FederatedSkill evolves personalized multi-skill libraries, while SkillGit treats locally proposed rule edits as versioned contributions to a common skill. Consequently, their output structures and validation targets are not identical, even when the underlying tasks and training exposure are aligned.

Commit identification and deduplication. We now distinguish two identifiers with different purposes:

  • Provenance identifier: πir=H(dr−1,i,i,hash(Di),r,hash(cir))\pi_i^r=H(d_{r-1,i}, i, hash(D_i), r, hash(c_i^r)). It records the parent version, client identity, private-slice hash, communication round, and commit payload hash. Thus, two clients that submit the same edit may correctly have different provenance identifiers.
  • Canonical edit key: It is computed from the parent-document hash and the normalized edit operation, target rule, and revision content. It excludes client identity and the private-slice/round fields, enabling identical normalized edits from different clients to be recognized as duplicates. Similar but nonidentical revisions are left for the merge agent's semantic adjudication rather than automatically collapsed by hashing. The provenance ID therefore supports attribution and version lineage, while the canonical key supports cross-client deduplication. These mechanisms are complementary, not interchangeable.

Where revised: The FederatedSkill objective and architectural distinction are presented in Section 2.3 (Lines 105–120). Its baseline settings and results have been added to Section 4.1 (Lines 342–368) and Section 4.2, Table 1 (Lines 378–415), with protocol details in Appendix C (Lines 943–969). The two identifiers and their inputs are defined in Section 3.3.1, Eqs. (5)–(6) (Lines 235–243), and their role in selective merge is further described in Section 3.4.2 (Lines 298–310).

Reviewer_yCMYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies federated skill evolution of a shared skill document for the frozen LLM agent.

Each selected client runs the current skill document on private data and extracts a compact semantic commit. The commit contains rules and experience. The server merges commits by disposition. A candidate document is published only if its validation score does not decrease. Two rollbacks stop the process.

The contributions are semantic commits as the unit of collaboration, a commit-aware merge, and a validation-based rollback rule. Experiments use three datasets and models and compare against no skill, independent evolution, centralized evolution, and a single-shot LLM merge.

Critical strengths

Real deployments often can not pool execution traces, and frozen API models make skill documents a practical adaptation interface. The commit-based abstraction is useful. It exposes individual revisions for inspection. and it avoids the meaningless arithmetic of averaging rule sets.

Critical weaknesses
  1. The missing comparison with FederatedSkill.

  2. The privacy argument is also weak. L1 sanitization and L2 auditing reduce obvious leakage. They do not provide a formal guarantee. The paper does not state what the server or other clients may infer from repeated commits. Privacy is one of the two central motivations for federation.

  3. the lack of statistical evidence. Single-run results on three datasets and three model families are not enough to support, especially when several margins are small. The paper does not report variance across seeds, client partitions, or scheduler choices.

Questions for authors
  1. What are the results against FederatedSkill under the same client partition, participation ratio, budget, and acceptance rule? Why does SkillGit exceed the centralized reference in several cells?

  2. What is the communication cost in tokens or bytes for commits versus full-document uploads across rounds?

  3. What threat model is assumed for the privacy analysis and what is the false negative rate of the L2 audit?

  4. How is the merge disposition implemented and what is its agreement rate across repeated runs? Are the reported numbers averaged over multiple seeds? If not, can you provide variance and significance tests?

  5. Does the validation-union acceptance rule overfit on the selection set? Is the final evaluation split used only once?

  6. What is the concrete edit budget and commit size limit used in the experiments?How does the scheduler select clients and does the selection depend on past scores?

4 comments
AuthorsOpenReview ↗

Reponse to Reviewer yCMY—W1 + Q1. FederatedSkill Comparison and the Centralized Reference (1/4)

W1. The missing comparison with FederatedSkill.

Q1. What are the results against FederatedSkill under the same client partition, participation ratio, budget, and acceptance rule? Why does SkillGit exceed the centralized reference in several cells?

We thank the reviewer for identifying FederatedSkill as an important closest baseline. Following this suggestion, we have added FederatedSkill to the main evaluation across the same three datasets and three model families. The experimental protocol uses controlled client partitions, participation schedules, and per-round data exposure; comparable budget and publication settings are aligned where applicable, while FederatedSkill retains its native personalized multi-skill-library design. The results below are averages over three random seeds.

Response Table R2-1. Comparison with FederatedSkill (TSR, %).

ModelDatasetFederatedSkillSkillGitGain (pp)
GPT-5.4-miniSearchQA77.778.6+0.9
LiveMath27.228.8+1.6
DocVQA85.388.5+3.2
Qwen3.5-flashSearchQA80.382.4+2.1
LiveMath24.829.6+4.8
DocVQA80.285.3+5.1
Gemini3-flashSearchQA82.687.3+4.7
LiveMath35.240.0+4.8
DocVQA87.790.1+2.4

SkillGit outperforms FederatedSkill in all nine model–dataset combinations, with improvements of 0.9–5.1 percentage points and a mean gain of 3.3 points. The strongest gains occur on Qwen3.5-flash/DocVQA (+5.1 points), Qwen3.5-flash/LiveMath (+4.8), and Gemini3-flash/LiveMath (+4.8). These results provide direct evidence that SkillGit's shared-skill integration is effective under the evaluated heterogeneous-client partitions. The two approaches do, however, target different shared artifacts: FederatedSkill maintains personalized libraries, whereas SkillGit aggregates contributions into a common, version-controlled skill with population-level publication checks.

We also appreciate the question about the centralized reference. Centralized optimization is an empirical comparison, not a theoretical upper bound on test TSR. SkillGit exceeds it in two settings: GPT-5.4-mini/DocVQA (88.5% vs. 86.4%) and Gemini3-flash/SearchQA (87.3% vs. 85.9%); it ties on GPT-5.4-mini/LiveMath (28.8%). One plausible explanation is that locally generated, targeted rule revisions preserve complementary failure-specific improvements that may compete when pooled evidence is optimized in one context. This is an interpretation of the observed results rather than a separately isolated causal effect.

Where revised: The distinction between personalized-library evolution and a shared, validated skill is explained in Section 2.3 (Lines 105–120). We have added the FederatedSkill baseline description in Section 4.1 (Lines 342–368) and its results and centralized-reference discussion in Section 4.2, Table 1 (Lines 370–415). The partition, scheduling, and budget protocol is further documented in Appendix C (Lines 943–969).

AuthorsOpenReview ↗

Reponse to Reviewer yCMY—W2 + Q3. Privacy Threat Model and L2 Audit Reliability (2/4)

W2. The privacy argument is also weak. L1 sanitization and L2 auditing reduce obvious leakage. They do not provide a formal guarantee. The paper does not state what the server or other clients may infer from repeated commits.

Q3. What threat model is assumed for the privacy analysis and what is the false negative rate of the L2 audit?

We thank the reviewer for this important question, and we apologize that the original presentation did not sufficiently distinguish the privacy-filtering procedure from a formal privacy guarantee. The intended setting is an honest-but-curious server: it follows the aggregation protocol but can inspect sanitized commits and associate submissions across rounds. Other clients do not directly receive individual commits, although they can observe published global skill versions. Raw task instances and execution trajectories remain with their respective clients.

The two filtering stages serve different roles. L1 sanitization replaces client-specific entities, identifiers, and environment details in free-text patch/experience fields with typed placeholders while preserving the edit operation and target rule. L2 auditing is an LLM-based semantic judgment grounded in a fixed, client-local private-value set. It checks both direct disclosure and disclosures by paraphrase, and failed commits return for revision rather than being uploaded.

To assess the effectiveness of the L1 stage, we have added the following controlled comparison.

Response Table R2-2. First-pass semantic audit clearance (GPT-5.4-mini, SearchQA).

ConfigurationFirst-pass audit clearance (%) ↑
Without L1 sanitization33
With L1 sanitization81

The 48-point increase in first-pass clearance indicates that L1 substantially reduces content flagged by L2 and makes privacy-screened commits easier to transmit without additional rewriting. However, 81% is not the L2 false-negative rate: a false-negative estimate requires an independent set of labeled leakage cases, whereas this experiment reports the auditor's own clearance decisions. We therefore do not infer an independently validated missed-leakage probability from this table. We also state explicitly that repeated accepted revisions may reveal information by inference and that neither stage guarantees differential privacy or resistance to adversarial inference.

Where revised: The L1/L2 operations, fixed private-value set, and honest-but-curious threat model are specified in Section 3.3.2 (Lines 245–276), with the auditor prompt in Appendix A.5 (Lines 849–862). The added sanitization experiment appears in Section 4.3.3, Table 4 (Lines 458–473). The boundary between payload screening and inferential privacy is clarified in Appendix B (Lines 924–930).

AuthorsOpenReview ↗

Reponse to Reviewer yCMY—W3 + Q4. Multi-Seed Evaluation and Merge Consistency & Q2. Communication Cost of Commits Versus Full Documents (3/4)

W3. The paper does not report variance across seeds, client partitions, or scheduler choices.

Q4. How is the merge disposition implemented and what is its agreement rate across repeated runs? Are the reported numbers averaged over multiple seeds? If not, can you provide variance and significance tests?

We thank the reviewer for stressing statistical reliability and the reproducibility of merge decisions. We apologize that the original manuscript did not sufficiently describe the evaluation repetitions and the disposition procedure. We have clarified that every main TSR cell is averaged over three random seeds, with paired method comparisons evaluated on the same test instances within each run. The client-selection schedule is generated from a fixed seed and is shared across the compared methods.

The merge operation uses structural checks and an LLM-based semantic adjudication step. Each submitted edit receives one of four dispositions: keep (new applicable edit), deduplicate (repeat of a retained edit), promote (incorporate into a shared rule), or reject (structurally invalid). Canonical keys identify identical normalized edits; potentially overlapping, nonidentical edits are handled by the merge agent. The implementation and repeatability evidence are as follows.

Response Table R2-3. Evaluation and merge-repeatability protocol.

Evaluation aspectSetting / observed result
Main TSR evaluationMean over 3 random seeds
Merge repeatability10 commit pools, each merged 10 times
Exact final patch-set agreement100 / 100 runs (100%)

The 100% exact agreement shows that the resulting skill-patch sets were reproducible across repeated merges of the evaluated fixed pools. Together with the three-seed TSR protocol, this strengthens evidence for implementation consistency and reduces reliance on a single observed run. We distinguish this exact merge agreement from statistical significance of TSR differences: the present revision reports seed means but does not supply per-seed standard deviations, partition-wise variance, or formal significance tests, so we do not claim that small TSR gaps are statistically significant.

Where revised: The three-seed protocol is documented in Section 4.1 (Lines 344–355) and Appendix C (Lines 956–961, 976–987). The disposition definitions are given in Section 3.4.2 (Lines 298–310) and Appendix A.6 (Lines 884–903). The 10 × 10 merge-repeatability experiment and its 100% agreement are reported in Appendix C (Lines 985–987).

Q2. What is the communication cost in tokens or bytes for commits versus full-document uploads across rounds?

We thank the reviewer for requesting a direct, quantitative communication-cost comparison. We have added measurements of the transmitted payload sizes of rule-level commits and complete skill documents, including the combined payload from three participating clients in a round.

Response Table R2-4. Measured transmission payloads.

Transmission unitMean bytes ↓Mean tokens ↓
Complete skill document5.1 KB1.3k
Single semantic commit1.7 KB0.5k
Three semantic commits in one round5.1 KB1.5k

A single semantic commit uses approximately 67% fewer bytes and 62% fewer tokens than a complete skill-document payload. Under the default three-client participation setting, the combined commit payload is approximately 5.1 KB / 1.5k tokens per round. The reduction comes from sending the current rule-level revisions and associated experience rather than retransmitting the entire accumulated skill document. These results directly establish per-payload efficiency; they should not be read as a measurement of full network-layer traffic or a cumulative sum over all rounds.

Where revised: We have added Section 4.4, Table 5 (Lines 475–492) for the byte/token comparison. The default three-client participation setting is stated in Section 4.1 (Lines 342–345), and the observed number of evolution rounds across configurations is reported in Section 4.5, Figures 2–3 (Lines 494–517).

AuthorsOpenReview ↗

Reponse to Reviewer yCMY—Q5. Validation-Union Overfitting and Final Test Split & Q6. Edit Budget, Commit Size Limit, and Client Scheduling (4/4)

Q5. Does the validation-union acceptance rule overfit on the selection set? Is the final evaluation split used only once?

We thank the reviewer for asking us to clarify the distinction between version selection and final evaluation, and we apologize that this separation was not sufficiently explicit in the original manuscript. SkillGit uses three distinct sources of evidence. The private client training partitions are used to generate execution trajectories and candidate edits, and the fixed union of client validation sets is used exclusively to decide whether a candidate version is published. Additionally, an independent, sealed test split is used for the final evaluation after the published version has been frozen.

In particular, the validation union is fixed before evolution and is not expanded with examples generated during optimization. The test split does not influence prompt selection, patch merging, candidate publication, or early stopping. This separation prevents direct test-feedback contamination of the evolution process and allows the final test results to assess the selected version independently. At the same time, we do not equate a fixed validation union with a proof of zero selection bias: repeated decisions on the same validation set can still produce validation-set overfitting. The purpose of the held-out test evaluation is to assess generalization separately from that selection process.

Where revised: The publication rule and validation/test separation are clarified in Section 3.4.3 (Lines 314–333). The non-overlapping dataset construction and sealed final-evaluation protocol are documented in Appendix C (Lines 948–954, 976–983).

Q6. What is the concrete edit budget and commit size limit used in the experiments? How does the scheduler select clients and does the selection depend on past scores?

We thank the reviewer for pointing out the need to specify these implementation details, and we apologize that the original description did not make the constraints and sampling procedure explicit. The main experiments use 10 clients with a 30% participation ratio, selecting three clients per round. A seeded random permutation is sampled without replacement within each round; the participation schedule is shared across methods and does not depend on previous validation or task scores.

For each participating client, the local optimizer may propose at most four rule-level edits per round, with the allowable edit budget following a cosine schedule from four to two. The shared skill document is kept below 4,000 tokens. An individual semantic commit has no separate hard token cap beyond the structural edit budget and the shared-document ceiling; accordingly, the measured mean commit size should not be mistaken for a maximum allowed commit size. These settings define both the optimization scope and the reproducible client-selection procedure.

Where revised: The number of clients, participation ratio, and evaluation setup are specified in Section 4.1 (Lines 342–355). The seeded schedule, independence from previous scores, edit budget, and document ceiling are documented in Appendix C (Lines 956–966).

Reviewer_sxPHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

For this paper, the authors propose SkillGit to address collaborative evolution of natural-language skills across clients that retain their raw execution data locally. Clients submit rule-level edits and experience summaries, which undergo sanitization and auditing. On the other hand, a server combines these updates with accumulated experience, constructs a candidate skill, and uses distributed validation to decide whether to publish it. Extensive experiments and additional analyses examine experience reuse, sanitization, and participation settings to show the superiority of the proposed SkillGit.

Critical strengths
  1. Practically, the proposed update abstraction is well matched to collaborative editing of natural-language skills. Rule-level revisions provide a potentially useful interface between local learning and global knowledge integration. They make it possible to inspect a potential change without treating an entire skill document as an indivisible update. This could support more understandable conflict handling, provenance, and debugging.
  2. From the evaluation perspective, the evaluation provides encouraging evidence that collaborative skill evolution can be useful. The authors conduct extensive experiments on it.
Critical weaknesses
  1. The main comparisons do not isolate the benefit of commit-based evolution. We can have sufficient ablation studies to know the contribution or benefit of each part for a wholistic view.
  2. For some experiments, I find that the privacy and communication-efficiency-related motivations are not adequately validated by the experiments. For example, for privacy, we can add the assessment about the residual leakage.
Questions for authors
  1. The isolation or ablation studies of various components and settings, as mentioned in the weakness point #1.
  2. The the privacy and communication-efficiency-related experiments to enhance the paper, as mentioned in the weakness point #2.
5 comments
AuthorsOpenReview ↗

Response to Reviewer sxPH—W1 + Q1. Component Isolation and Ablation (1/2)

W1. The main comparisons do not isolate the benefit of commit-based evolution. We can have sufficient ablation studies to know the contribution or benefit of each part for a wholistic view.

Q1. The isolation or ablation studies of various components and settings, as mentioned in weakness #1.

We thank the reviewer for this valuable suggestion. To better isolate the contributions of the proposed framework, we have conducted additional controlled analyses of cross-round experience reuse, server-side deduplication, and commit-based communication. These experiments use GPT-5.4-mini on SearchQA and examine task success and the efficiency of the corresponding update operations. The results are summarized below.

Response Table R1-1. Component-level evaluation of SkillGit.

ComponentConfigurationMetricResult
Experience reuseFull SkillGitTSR (%) ↑78.6
Without experience in next-round optimizationTSR (%) ↑76.9
Without experience in server-side mergingTSR (%) ↑77.2
DeduplicationWith deduplicationMerge input (KB / k tokens) ↓3.6 / 1.0
Without deduplicationMerge input (KB / k tokens) ↓5.1 / 1.5
CommunicationSemantic commitMean payload (KB / k tokens) ↓1.7 / 0.5
Complete skill documentMean payload (KB / k tokens) ↓5.1 / 1.3

The results show that the two uses of global experience make complementary contributions: removing experience from subsequent local optimization reduces TSR by 1.7 percentage points (78.6% → 76.9%), while removing it from server-side merging reduces TSR by 1.4 points (78.6% → 77.2%). Deduplication reduces the merge input by approximately 29% in bytes and 33% in tokens, indicating that redundant edits can be filtered before aggregation. At the communication level, semantic commits are approximately 67% smaller in bytes and 62% smaller in tokens than complete skill-document payloads on average. Together, these results support the value of experience-guided refinement and compact rule-level updates, with each measurement assessing the component it directly isolates.

Where revised: We have incorporated the deduplication ablation in Section 4.3.1, Table 2 (Lines 419–437), the experience-reuse ablation in Section 4.3.2, Table 3 (Lines 439–455), and the transmission-size analysis in Section 4.4, Table 5 (Lines 475–492). The latter measures communication payload size rather than an independent TSR effect of the commit representation.

Reviewer_sxPHOpenReview ↗

Response to the author response

Thanks the reviewers for addressing some concern. But, from my point of view, some issue still remains (e.g., lack of experiments to justify the main points and the novelty is still limited to some degree)

AuthorsOpenReview ↗

Response to Reviewer sxPH—W2 + Q2. Privacy Filtering and Communication-Efficiency Evidence (2/2)

W2. For some experiments, I find that the privacy and communication-efficiency-related motivations are not adequately validated by the experiments. For example, for privacy, we can add the assessment about the residual leakage.

Q2. The privacy and communication-efficiency-related experiments to enhance the paper, as mentioned in weakness #2.

We thank the reviewer for emphasizing the need for direct evidence behind the privacy and communication motivations. In response, we have added a sanitization ablation and quantitative payload-size measurements, and we have made the scope of the privacy claim more precise.

Privacy-filtering experiment. SkillGit first sanitizes client-specific information in the patch and experience fields, then applies an LLM-based semantic audit grounded in a client-local private-value set. We measure how frequently payloads pass this audit on the first attempt, with and without sanitization.

Response Table R1-2. Sanitization ablation (GPT-5.4-mini, SearchQA).

ConfigurationFirst-pass L2 audit clearance (%) ↑
Without sanitization33
With sanitization81

Adding sanitization improves first-pass audit clearance from 33% to 81%, a gain of 48 percentage points. Equivalently, the share of payloads rejected by the auditor on the first attempt falls from 67% to 19%. This provides empirical evidence that sanitization removes material flagged by the semantic auditor and reduces the need for repeated rewriting. Importantly, this metric is an audit-clearance rate, not an independently measured residual-leakage rate or L2 false-negative rate. The revision explicitly limits the resulting privacy claim to payload-level filtering and does not assert a formal inference guarantee.

Communication experiment. We additionally compare the mean transmitted payload sizes of one semantic commit and one complete skill document.

Response Table R1-3. Communication payloads.

PayloadMean bytes ↓Mean tokens ↓
Complete skill document5.1 KB1.3k
Semantic commit1.7 KB0.5k
Three transmitted commits in one round5.1 KB1.5k

An individual commit reduces mean uplink payload by approximately 67% in bytes and 62% in tokens relative to a complete document. With three participating clients, the measured commit payloads total approximately 5.1 KB / 1.5k tokens per round. These measurements directly substantiate the compactness of rule-level transmission without conflating payload size with end-to-end network traffic.

Where revised: We clarify the sanitization and semantic-auditing procedure, the honest-but-curious threat model, and its limitations in Section 3.3.2 (Lines 245–276) and Appendix B (Lines 924–930). The corresponding sanitization results are added in Section 4.3.3, Table 4 (Lines 458–473), and the communication measurements in Section 4.4, Table 5 (Lines 475–492).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.