- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MASK-GATE aims to limit cross-user leakage when execution trajectories update shared textual Skills. It combines typed masking, outcome-graded sanitized Evidence, deterministic release checks, and fallback to the Original Skill. In controlled map-task experiments, it reports lower observed behavioral contamination and higher average correctness than SkillOpt. Correctness averages per-turn scores rather than measuring end-to-end success.
Critical strengths
-
The paper identifies an important risk in sharing learned Skills. Its approach is sensible: remove private details before generating an update, check the update, and keep the original Skill if the checks fail. It also limits what the checker can reveal during retries.
-
The comparison is carefully set up. Methods use the same source trajectories and test conditions, and test probes stay hidden until updates are finalized. Rejected updates remain in the results. The analysis preserves paired comparisons, and the task and turn breakdowns show where the gains do and do not hold.
Critical weaknesses
-
The privacy results need clearer controls. BCR counts cases where the updated Skill selects the source user’s preferred option but the original Skill does not, among valid comparisons where that option is available. Such a change could reflect preference transfer, but it could also come from ordinary changes in ranking or randomness. The paper says it repeats runs with the original Skill to check randomness, but does not show those results. The underlying BCR counts and a paired comparison with SkillOpt would also help; separate confidence intervals do not settle that comparison. As the authors acknowledge, the text scanner checks the same vocabulary as the release gate, so it cannot establish protection against unfamiliar forms of leakage. Testing only one location workflow with synthetic preferences also limits the conclusions.
-
It is hard to tell which parts of MASK-GATE drive the gains. Several changes are introduced together, and most correctness gains come from tasks the original Skill handles poorly. Appendix D does test outcome grading separately, but on different tasks with a different scorer. That leaves its role in the main results unclear.
Questions for authors
-
Can you report the repeated-original-run results, BCR event and opportunity counts, and a confidence interval for the paired difference from SkillOpt? A matched test that removes or swaps the source preference could help explain the behavioral changes. Please define which preference is tracked in that test.
-
Can you test masking, outcome grading, and release checks separately on the main benchmark, with comparable task guidance and compute? Please report both privacy and correctness, including how often updates are rejected.
I thank the authors for the additional experiments and clarifications. However, the missing repeated baseline and limited evidence from the preference-swap experiment leave my main concern about the source of the observed effect unresolved. Therefore, I will maintain my current score.