acceptodds
Under review as a conference paper at ICLR 2027

MASK-GATE: Measuring and Mitigating Cross-User Leakage in Evolving Agent Skills

Abstract

Agent skills increasingly distill execution experience into reusable instructions that improve future agent behavior. Reusing these evolved skills across users introduces a new risk: information from one user’s trajectories may persist in a shared Skill and influence another user’s decisions even when model weights remain fixed. We treat the trajectory-to-shared-Skill interface as a controlled information-release boundary that preserves task-general evidence while excluding user-scoped signals. We introduce MASK-GATE, an evolution-time framework that enforces this boundary through typed semantic masks, sanitized Evidence records, Evidence auditing, and fail-closed privacy, integrity, and capability checks. These controls operate offline, add no model calls at inference time, and retain the Original Skill when an update is rejected. On CrossUserSkillLeak, a controlled benchmark for cross-user leakage in location-based tool use, we compare the no-evolution Original Skill, SkillOpt, SkillClaw, and MASK-GATE under the same registered evolution groups, frozen base model, tool snapshot, probe inputs, and evaluation protocol. The 100-probe test set is held out throughout method development and is opened only after all artifacts, release decisions, and evaluation rules are frozen. Across 165 registered evolution groups and 100 held-out probes, MASK-GATE achieves the highest observed grounded task correctness, reaching 74.0%, compared with 68.9% for SkillOpt, 65.7% for SkillClaw, and 51.0% for the no-evolution Original Skill. It also attains the lowest behavioral contamination rate (BCR), at 0.9%, compared with 2.8% for SkillOpt and 5.7% for SkillClaw. Our method therefore attains the lowest observed leakage and highest observed correctness among the evaluated methods, with gains concentrated in capability-generalization probes.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, agent skill evolution, agent adaptation, continual learning, privacy-preserving learning, cross-user leakage, agent evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vuDcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MASK-GATE aims to limit cross-user leakage when execution trajectories update shared textual Skills. It combines typed masking, outcome-graded sanitized Evidence, deterministic release checks, and fallback to the Original Skill. In controlled map-task experiments, it reports lower observed behavioral contamination and higher average correctness than SkillOpt. Correctness averages per-turn scores rather than measuring end-to-end success.

Critical strengths
  • The paper identifies an important risk in sharing learned Skills. Its approach is sensible: remove private details before generating an update, check the update, and keep the original Skill if the checks fail. It also limits what the checker can reveal during retries.

  • The comparison is carefully set up. Methods use the same source trajectories and test conditions, and test probes stay hidden until updates are finalized. Rejected updates remain in the results. The analysis preserves paired comparisons, and the task and turn breakdowns show where the gains do and do not hold.

Critical weaknesses
  • The privacy results need clearer controls. BCR counts cases where the updated Skill selects the source user’s preferred option but the original Skill does not, among valid comparisons where that option is available. Such a change could reflect preference transfer, but it could also come from ordinary changes in ranking or randomness. The paper says it repeats runs with the original Skill to check randomness, but does not show those results. The underlying BCR counts and a paired comparison with SkillOpt would also help; separate confidence intervals do not settle that comparison. As the authors acknowledge, the text scanner checks the same vocabulary as the release gate, so it cannot establish protection against unfamiliar forms of leakage. Testing only one location workflow with synthetic preferences also limits the conclusions.

  • It is hard to tell which parts of MASK-GATE drive the gains. Several changes are introduced together, and most correctness gains come from tasks the original Skill handles poorly. Appendix D does test outcome grading separately, but on different tasks with a different scorer. That leaves its role in the main results unclear.

Questions for authors
  1. Can you report the repeated-original-run results, BCR event and opportunity counts, and a confidence interval for the paired difference from SkillOpt? A matched test that removes or swaps the source preference could help explain the behavioral changes. Please define which preference is tracked in that test.

  2. Can you test masking, outcome grading, and release checks separately on the main benchmark, with comparable task guidance and compute? Please report both privacy and correctness, including how often updates are rejected.

1 comment

I thank the authors for the additional experiments and clarifications. However, the missing repeated baseline and limited evidence from the preference-swap experiment leave my main concern about the source of the observed effect unresolved. Therefore, I will maintain my current score.

Reviewer_LavdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cross-user leakage in evolving agent skills and proposes MASK-GATE, which uses typed semantic masking, sanitized and outcome-graded Evidence, Evidence auditing, and fail-closed privacy, integrity, and capability checks before releasing an evolved Skill.

The paper also introduces CrossUserSkillLeak, a controlled benchmark for cross-user Skill transfer in location-based tool use. Across 165 registered evolution groups and 100 held-out probes, MASK-GATE achieves a behavioral contamination rate of 0.9%, compared with 2.8% for SkillOpt and 5.7% for SkillClaw, while also achieving the highest observed grounded task correctness among the evaluated methods, 74.0% versus 68.9% and 65.7%, respectively.

Critical strengths

The identified privacy boundary in experience-driven agent evolution is important. The possibility that user-specific information can be propagated through persistent Skills is practically relevant and is not captured by conventional model-weight privacy analysis. Treating Skill evolution as a controlled release problem is a useful systems perspective.

The proposed evaluation that carefully separates privacy from downstream utility is also interesting. MASK-GATE is evaluated not only for direct artifact leakage but also for behavioral contamination and grounded task correctness. Within the controlled benchmark, it achieves both the lowest observed behavioral contamination and the highest observed correctness among the evaluated methods.

Critical weaknesses

The current evaluation does not sufficiently establish robustness or generality beyond the controlled CrossUserSkillLeak setting. The use of synthetic users and trajectories is reasonable for controlled privacy evaluation, but it has limitation in coverage. The experiments involve a single location-based workflow, four source profiles, a fixed model/tool environment, and predefined forms of leakage, and it therefore remains unclear whether MASK-GATE is robust across other application domains, agent backbones, Skill types, privacy attributes, or previously unseen forms of semantic leakage.

The paper does not demonstrate whether MASK-GATE is broadly competitive as a general Skill-evolution method, nor does it isolate the contributions of its individual components. MASK-GATE achieves better grounded correctness than SkillOpt and SkillClaw within CrossUserSkillLeak, but this comparison is limited to the privacy-focused setting, and the improvement is concentrated in capability-gap probes. There is no evaluation on a general Skill-evolution benchmark that would show whether the privacy controls preserve or improve evolution quality more broadly.

In addition, the paper does not provide component ablations for typed masking, outcome grading, Evidence auditing, and the release gates, making it difficult to determine which mechanisms are responsible for privacy protection and which drive the utility improvement.

Questions for authors

Please provide stronger evidence that MASK-GATE generalizes beyond the specific location-based workflow and predefined leakage patterns in CrossUserSkillLeak?

How does MASK-GATE perform as a general Skill-evolution method outside the privacy-focused benchmark, and which components are responsible for the observed improvements?

One minor presentation issue is Figure 1, where the text is too small and difficult to read comfortably.

Reviewer_WPTZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies cross-user information leakage in evolving agent skills, where information from one user’s execution trajectories may be incorporated into reusable instructions subsequently deployed for other users. The authors propose MASK-GATE, an evolution-time framework that sanitizes trajectories using typed semantic masks, produces outcome-graded evidence, and applies privacy, integrity, and capability checks before releasing updated skills.

Critical strengths
  1. Important and timely problem. Cross-user information transfer through evolved agent skills is a meaningful security and privacy concern. Unlike conventional prompt injection or memory poisoning, the paper examines a specific information-flow boundary through which private execution histories become shared instructions.
  2. Thoughtful experimental protocol. The authors explicitly separate induction data from held-out evaluation, use matched evolution groups across methods, and retain failed release attempts in the reported results. The paired evaluation design is useful for distinguishing changes introduced by evolved skills from behavior under the Original Skill.
  3. Separation of privacy and utility outcomes. The paper distinguishes direct artifact leakage, behavioral contamination, and grounded task correctness instead of treating privacy as a single binary property. This distinction is conceptually useful, particularly because a skill could influence downstream behavior without explicitly reproducing protected text.
Critical weaknesses

Insufficient causal attribution. The paper combines masking, outcome grading, evidence auditing, and capability-based release gates without showing which component produces the improvement. In particular, the large correctness gain may be driven by explicit capability checks rather than privacy-preserving evolution.

Privacy evaluation is too closely coupled to the defense. The artifact scanner uses the same protected-term vocabulary as the release gate. Consequently, achieving zero detected matches is partly a consequence of how the metric is constructed, not independent evidence of privacy protection.

Limited evidence of generalization. The evaluation uses one location-based workflow, four source profiles, and 100 unique held-out probes reused across 165 evolution groups. This does not yet establish that the approach generalizes to broader agent skills or realistic private user information.

Questions for authors
  1. How much of the correctness improvement remains when the same capability checks and public task specifications are applied to an otherwise unmodified or minimally evolved skill?
  2. Can the authors provide controlled ablations separating masking, outcome grading, evidence auditing, privacy gating, and capability gating?
  3. How does MASK-GATE compare with simple trajectory redaction followed by SkillOpt/SkillClaw, or with applying the release gates directly to their generated skills?
  4. What is the behavioral contamination rate for independently repeated Original-versus-Original executions? Can the authors also report the opportunity denominators and direct paired confidence intervals between evolution methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.