- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper formalizes context-dependent authority resolution by selecting the highest-priority applicable principal and adopting that principal's judgment. MPAGuard compares direct selection with sequential or exhaustive principal evaluation, and joint versus separate applicability/polarity outputs. A synthetic benchmark shows benefits from explicit decomposition. On AgentDojo, PU-Two reduces average attack success from 0.31 to 0.10 while keeping average utility under attack near 0.42. The contribution is mainly an empirical study of structured authority prompting and its use as an action filter.
Critical strengths
-
The separation of inapplicability, prohibition, and conflict is clear and useful. Proposition 1 is correct under its stated assumptions, and the paper appropriately distinguishes a unique formal decision from an LLM's ability to compute it. Appendix B also explains how selection errors accumulate without claiming that its simplified model proves PU superior to UP.
-
The experiments compare several model families and expose specific errors rather than only reporting accuracy. The OneStep/TwoStep comparisons, all-matching control, and token/cost results show when explicit applicability helps. The paper also reports cases where CoT is competitive or better, which makes the practical comparison more balanced.
Critical weaknesses
-
The practical action-control evidence is incomplete. AgentDojo uses one attack family and compares mainly variants of the proposed guard. Appendix E.3 says the acting agent is instructed to call the guard and selects the authority sources itself, but does not establish whether every side-effecting call is forcibly gated or how source identity and priority are protected. Moreover, PU-Two reduces benign utility from 59.8% to 36.1% for Llama3.3-70B (Table 12). Thus, preserved average utility under attack is not preserved utility generally. A trace audit of guard coverage and a matched established defense would make this claim much more convincing.
-
The benchmark establishes a narrow prompting result, leaving the added value of learned authority resolution unclear. Each principal has one binary rule, contexts match by exact attribute equality, and the natural-language variant changes only the query surface. Local conflicts from the formal framework are therefore not tested. Proposition 1 is a straightforward consequence of the selection rule. A deterministic resolver, or extraction followed by deterministic resolution where language interpretation is needed, is an important reference point.
Questions for authors
-
Is guard invocation enforced outside the acting model? Can you report the fraction of side-effecting calls checked, the treatment of missing or malformed guard calls, and how source attribution and priority are obtained? How does the guard compare with an established defense on the same attacked and benign tasks?
-
Which parts of the evaluated authority task require learned interpretation rather than deterministic rule matching? A deterministic or extraction-plus-resolution baseline would clarify the practical role of MPAGuard and the source of its remaining errors.
3 comments
Response to Weaknesses: Action Control and Benchmark Scope
W1. Action-control evidence, defense comparisons, and utility
We have expanded AgentDojo evaluation to Important Instructions, Ignore Previous, and Tool Knowledge, with comparisons against Tool Filter, Spotlighting, and Repeat User Prompt. Please see our General Rebuttal Comment for the experimental settings, baseline comparisons, and detailed results.
MPAGuard consistently achieves lower ASR than Repeat User Prompt and Spotlighting, and generally lower ASR than Tool Filter. Utility under attack is slightly lower than Repeat User Prompt for Qwen but higher than all three defenses for Gemma. Compared with Tool Filter, benign utility is slightly lower for Qwen but substantially higher for Gemma.
We also clarify that authorization checks are enforced by the executor before non-read-only actions. In the audited conditions, every valid target call reaching the executor was checked; invalid tool arguments were blocked before execution. The runtime distinguishes original user requests from tool responses and fixes the original user’s priority over tool-provided instructions. Please see our response to Q1 for the audit counts and implementation details.
We agree that preserving utility under attack does not establish general utility preservation. The reported Llama3.3-70B benign-utility decrease is a genuine trade-off, and we will revise the manuscript to distinguish benign utility from utility under attack and qualify our claims accordingly.
W2. Benchmark scope and the role of learned interpretation
Our paper focuses on (1) formalizing the structure of authorization from explicitly specified multiple principals and (2) analyzing LLM authorization tendencies under these controlled conditions. The MPA Dataset provides verifiable labels to isolate errors in applicability, polarity, and governing-principal selection. Proposition 1 characterizes the conditions for a unique authority decision; it does not imply that computing that decision requires a learned resolver.
We agree that the structured setting admits deterministic resolution and therefore add an LLM-extraction-plus-deterministic-resolution baseline. It achieves near-perfect accuracy on structured queries, while its advantages are less consistent on natural-language queries, where contextual interpretation remains a source of error. Please see our response to Q2 for the detailed comparison and discussion.
We additionally evaluate MUSES-IF Selection and OrgAccess, where MPAGuard achieves the highest overall performance among the evaluated methods for both models. These results show that the approach also benefits authorization decisions beyond the controlled MPA setting. Please see our General Rebuttal Comment for the experimental settings, baseline comparisons, and detailed results.
We consider contextual ambiguity and within-principal conflicts important problems, but their systematic treatment lies beyond this paper’s scope. Overlapping rules require explicit conflict-resolution criteria, while ambiguous contexts require determining how uncertainty affects applicability and authorization. Please see our response to Reviewer Xb7a, under “Scope 1: Overlapping rules within a single principal” and “Scope 2: Ambiguous authority scopes,” for further discussion. Establishing clear LLM authorization decisions under such conditions is an important direction for future research.
We will revise the manuscript to explicitly state these limitations and opportunities for extending the framework.
Gentle Follow-Up Before the Discussion Deadline
Dear Reviewer d4F2,
We apologize for reaching out again as the discussion period draws to a close, and sincerely appreciate your time and thoughtful review.
We have updated the revised manuscript with additional experiments and clarifications addressing your concerns.
We would be grateful if you could review these updates and share any remaining feedback.
Thank you again for your time and consideration.
Thank you for taking the time to review our paper and for highlighting the need to clarify our evaluation and implementation.
Response to Questions
Q1. Guard enforcement, coverage, and authority-source inputs
Yes. Guard invocation is enforced by the executor rather than left to the acting model. Our wording “instruct the agent to invoke” was imprecise: MPAGuard is exposed through a tool-compatible interface, but the executor automatically intercepts non-read-only tool calls and requires a permit decision before execution.
The audit logs for Multi-turn + MPAGuard show the following action-level counts, including repeated attempts:
All target calls with valid arguments that reached the executor were checked: 73/73 and 609/609 for Qwen, and 53/53 and 467/467 for Gemma. The remaining Qwen attempts were blocked before guard invocation by tool-schema validation—for example, passing
{"content": []}whencontentmust be a string, or omitting a requireddateargument. These calls were not executed.Q2. Learned interpretation versus deterministic resolution
Extraction + Resolution achieves near-perfect accuracy across all three models on structured queries, outperforming the prompting baselines. On natural-language queries, it remains strong for Llama3.3-70B but reaches 89.3% for Gemma3-27B and falls four percentage points below PU-Two for Qwen3-32B. Thus, deterministic resolution is effective when the required information is reliably structured, while contextual interpretation remains a source of error.
Structured tool query
Natural-language tool query
Authorization is context-dependent: users delegate authority for particular situations, and the agent must determine whether a proposed action falls within that scope. Keywords or categorical representations must therefore preserve the relevant contextual boundaries.
For example, a user may authorize
rm -rf *within a specific directory. Matching the command alone is insufficient; authorization also depends on the working directory and the scope of the delegation. If these conditions are precisely specified and reliably available to the execution system, deterministic checks can be accurate and efficient. If an LLM must first interpret the instruction and determine whether the current situation satisfies those conditions, however, that interpretation burden remains. Generating a structured representation adds cost and may introduce errors, so an end-to-end efficiency advantage is not guaranteed.MPAGuard structures these contextual judgments by explicitly distinguishing whether delegated authority applies from whether it permits or prohibits the action. Its role is complementary to deterministic resolution, which can compute the final decision once those judgments are reliably established.