- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper tackles the problem of agents learning appropriate permissions for a given user in an online fashion. Modern agentic systems often make tool calls that are potentially unsafe (e.g. vulnerable to prompt injections or exfiltrating user secrets). As a layer of safe-guards, these systems can ask the user for permissions to call certain tools, at a usability cost. The present paper aims to learn a context-dependent policy specific to the given user which minimizes the usability cost as well as the cost of incorrectly allowing or denying a tool call in an online fashion.
Critical strengths
The formuation of the problem as an online learning problem is nice.
Critical weaknesses
- One of my major concerns is with the writing of the work. The paper is extremely hard to read as a human. The phrasing is awkward, the sentence structure is hard to follow, some symbols seem undefined (e.g. in Eq (1), IPW in the results) . The method section is also very poorly written, e.g. what is the channel , what is in computing the propensity, how are determined. The system design does not seem to be well motivated, or at least the motivating principles are not well articulated. I also do not understand when one asks, the expected costs seem to imply that p_{ask}=1 ?
- The method is also extremely complicated with a lot of moving parts, and it would be good for the paper to work through an example to clarify how the method works. I also do not understand why each component of the method is useful, e.g. why is policy distillation done every few steps? What happens if this were not incorporated in the system? These are also not empirically demonstrated through ablations.
- The objective in Eq (1) is not well motivated. Notably, the cost should be assymetric (i.e. a false allow is often catastrophically bad from a security perspective, despite ), the cost for asking needs to change with the number of asks
- Critical prior work is missing -- https://arxiv.org/pdf/2511.17959 proposes using a similar learning based approach to asking for user permissions.
- There is no human evaluation. This seems to be crucial.
Questions for authors
Apart from the weaknesses above,
- Could the authors provide a motivation (intuitive or empirical) for all the moving parts of the system?
2 comments
Thank you for your response. Given that the entire paper is rewritten it needs another round of review to assess its merits. I hence maintain my current recommendation.
Response to Reviewer Eb7D
Thank you for the specific criticisms on writing, symbols, motivation, and the missing citation. We address each below.
Reviewer's point: The paper is extremely hard to read; some symbols seem undefined (Eq. (1), IPW in the results); the method section is poorly written (what is the channel, how is the propensity computed, how are the thresholds determined). Our response: A full readability rewrite was done. Every quantity of Eq. (1) is now defined in place (cost-matrix semantics and magnitudes, delta, the conditional meaning of FAR); the four feedback channels and their propensities are tabulated (Appendix A); IPW is expanded at first use; the thresholds theta_f/theta_a/n_min/kappa_h/k_o are defined where they are introduced; and the IPW weights are given exactly (1.18/1.03/2.86).
Reviewer's point: The expected costs seem to imply p_ask = 1 when the system asks. Our response: Clarified: E[cost | ask] = c_ask + (1 - pi_a) c_ig, where pi_a = 0.85 is the probability the user answers. The second term is round-independent, so it never changes which action minimizes the cost, but it belongs in the cumulative interaction-cost accounting; an answered ask incurs no decision-error cost, and utility counts a task only if it completes autonomously or after an answered ask. The original text wrote only c_ask, which created the confusion — thank you for pointing it out.
Reviewer's point: A worked example would clarify the many moving parts; the usefulness of each component (e.g., distillation every few steps) is not demonstrated.
Our response: A worked example was added (Appendix B): one concrete request (a 21:00 critical send) is traced through p~ = 0.55, the three arm costs (5.4/0.55/0.37), the gate test (1 - p_lo = 0.10 > 0.01, hence ASK), the update weight 1.18 after the user's answer, and the next distillation cycle with the fit/calibration fidelity and false-allow arithmetic, admission, and user confirmation. A new paragraph maps each component to the failure mode it removes, with ablation evidence. Please also note that Table 2 already contained four distillation ablations (w/o distillation, cell-constant, w/o calibration validation, w/o demotion); we additionally swept the period T_d: {100, 250, 500} are indistinguishable; 1,000 degrades; never distilling gives 61.2% coverage and 3.8% FAR.
Reviewer's point: The objective is not well motivated; the cost should be asymmetric, and the ask cost should change with the number of asks.
Our response: The cost matrix semantics and magnitudes are now stated (1/2/6/12; fd = 1; ask = 0.35; ig = 0.15), with the asymmetry motivated (a critical false allow is an irreversible external effect: a wire, a public post, credential disclosure); the budgets delta follow from cost-normalized risk (0.05 x 6 = 0.30 vs. 0.01 x 12 = 0.12). New ask-fatigue experiment: the per-ask cost is multiplied by (1 + gamma x asks so far) for gamma in {0, 1e-4, 5e-4, 1e-3}. Measured: asks per 100 requests fall from 9.3 to 5.4, coverage 90.7 -> 94.6, SelAcc 97.0 -> 94.6, utility 97.0 -> 92.7, FAR 2.5 -> 2.8 — the learner front-loads its interruption budget and then guesses more; the flat cost (gamma = 0) is the best operating point.
Reviewer's point: Critical prior work is missing: arXiv:2511.17959 proposes a similar learning-based approach to asking for permissions. Our response: Thank you for the pointer. We verified and now cite Wu et al., Towards Automating Data Access Permissions in AI Agents (IEEE S&P 2026; arXiv:2511.17959). Their user study (205 participants; 85.1% accuracy, 94.4% at high confidence) actually strengthens our motivation (preferences vary with context but are consistent within a context). Our four deltas: online MNAR learning (vs. a static history); asymmetric, per-tier false-allow budgets; distillation into enforced symbolic rules; and the interruption budget as a first-class objective (enforcement is explicitly out of scope in their work).
Reviewer's point: There is no human evaluation, which seems crucial.
Our response: we cannot complete a user study within the rebuttal window (institutional review and recruitment take weeks), and we will not fabricate one. We have pre-registered a two-stage protocol in the paper (Appendix J): Stage 1, a usability pilot (N ~ 20; rule readability, time to locate a rule, error discovery on three seeded mis-distilled rules, SUS); Stage 2, personalization vs. interruption (N ~ 100, three randomized conditions, mixed-effects analysis). We commit to completing Stage 1 before camera-ready. All current claims are scoped to synthetic personas.