- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
LLM agents use a “skill layer”: reusable skill modules (documented in natural language, for instance a SKILL.md file) that say what kind of task to perform and which tools may be used. The idea of the authors is that this layer can be also seen as privilege boundary. For this reason, the paper proposes FORTIS, a benchmark with two tasks. In Task 1 (skill selection) the model sees a user request and short descriptions of all 20 skills in a domain and must find a skill. If this skill is appropriate (minimally sufficient/lowest-privilege skill or uses to much privileges) is one question researched by the authors. A second task (Task 2 skill-grounded tool selection) the model is aware of the skill it uses is shown the full skill document and the whole tool inventory and has to output a list of tool call, The authors investigate if it stays inside the tools that the skill and the request needs to. Ten frontier models are evaluated on 600 Task 1 queries and 1,543 Task 2 queries in three domains (email, e-commerce, filesystem). The main conclusion is that skill boundaries cannot be delegated to the model’s own reading of instructions and must be mechanically enforced at the skill or tool invocation layer.
Critical strengths
The paper addresses a timely issue for security researchers working on least-privilege, permission models and runtime enforcement for LLM agents. It's a pity that the paper did not delve into more access control mechanisms and security mechanisms for AI agents, and provide (besides the theoretical/formal model) a concrete instantiation for the security use case. Similalry, other researchers interested in building benchmarks for in selection-time rather than completion-time evaluation. Another target audience is made of researchers having to chose between relying on model self-restraint or on mechanical checks. The idea of the paper is original and the state of the art summary is sufficient. The main results are consistent in the sense that no model reaches reliable minimal-privilege selection in any domain, and the pattern repeats across model families. The idea of the benchmark itself poses a well-framed problem and measuring restraint under benign ambiguity, with a two-stage structure (skill selection, then skill-grounded tool selection) and an end-to-end funnel, is conceptually different from previous known research work.
Critical weaknesses
There is no condition with an explicit least-privilege instruction, with privilege labels shown, or with a filtered tool inventory, so the claim that the fix lies at the architecture, or training-objective level is not tested against these cheap mitigations. The audit reviewed 200 of 2,143 queries by two authors. This was not done by independent annotators, and reports raw agreement (193/200) without kappa or the label-invalid rate. Agreement between reviewers is not the same as the share of labels that are valid. There are no confidence intervals, a single run per model at temperature 0, and the per-setting sample sizes are not stated. The EM1 × EM2 assumes independence between tasks on different query sets. The reviewer is not convinced that this assumption holds on the general case. The paper does not propose a solution to the identified problem. The authors argue that the model should not be the only enforcement point for scope, but they do not test alternative approaches.
Questions for authors
Task 1 over-privilege is defined by privilege level, ℓS(ŝ) > ℓS(s*). Can we assume that choosing a higher-level skill can always do the task ? How was the number of privilege levels decided to the set {0,1,2,3,4} for tools and skils ? What happens if the number of levels goes up ? For task1, the prompt deliberately contains no least-privilege instruction (Appendix B.2), and higher-level skills are described with convenience language. What happens if the prompt includes explicit least-privilege instructions ? How can an agent be forced/incentivized to use the adequate level of skills/tools? Why is only over-privilege treated as the main outcome? What are the rates of incorrect-but-lower-privilege picks? The ordinal levels are used for ranking and gaps. What justifies treating the distance from L1 to L2 as comparable to L3 to L4? Skills and tools are authored and constructed for three domains. How would you expect the results to change on real, independently written skill registries?
2 comments
Rebuttal
Q7: Can we assume that choosing a higher-level skill can always do the task?
Higher privilege does not automatically imply functional suitability. FORTIS constructs overlapping skills so that a task admits both a minimally sufficient skill and a broader, higher-privilege alternative. The evaluation tests whether the model chooses the minimally sufficient option among these alternatives.
Q8: How was the number of privilege levels decided to the set {0,1,2,3,4} for tools and skils ? What happens if the number of levels goes up?
The five levels capture distinct operational scopes shared across the three domains: observation, content access, annotation or modification, execution, and bulk or administrative operations. This provides a consistent hierarchy for comparing narrower and broader capabilities.
The formulation does not require exactly five levels. Additional levels can represent finer distinctions in authority while retaining the same minimum-privilege objective. If a refinement preserves the ordering and minimum sufficient solutions for existing queries, their classifications remain unchanged; refinements that change those relationships require corresponding ground-truth updates.
Q9: For task1, the prompt deliberately contains no least-privilege instruction (Appendix B.2), and higher-level skills are described with convenience language. What happens if the prompt includes explicit least-privilege instructions? How can an agent be forced/incentivized to use the adequate level of skills/tools? Why is only over-privilege treated as the main outcome? What are the rates of incorrect-but-lower-privilege picks?
Over-privilege is the primary safety outcome because it captures the selection of authority beyond what the task requires. Selection correctness is also evaluated through exact match.
Separating strictly lower-privilege errors from incorrect selections at the same privilege level gives the following Task 1 breakdown:
| Model | Exact match | Higher-privilege selections | Incorrect lower-privilege selections | Incorrect same-level selections |
|---|---|---|---|---|
| GPT-5.4-mini | 43.50% | 45.50% | 0.50% | 10.50% |
| Claude Opus 4.7 | 54.83% | 35.50% | 1.17% | 8.50% |
For these models, higher-privilege selections substantially outnumber lower-privilege errors, supporting the focus on unnecessary authority expansion.
Q10: The ordinal levels are used for ranking and gaps. What justifies treating the distance from L1 to L2 as comparable to L3 to L4?
The privilege levels are ordinal categories. The over-privilege criterion uses their ordering to determine whether the selected capability exceeds the minimum required level; it does not require equal distances between adjacent levels. A transition from L1 to L2 and one from L3 to L4 both indicate an increase in authority, but are not assumed to represent equal increases in risk. Level gaps describe the number of hierarchy boundaries crossed, rather than a calibrated measure of harm.
Q11: Skills and tools are authored and constructed for three domains. How would you expect the results to change on real, independently written skill registries?
Independently authored skill registries may introduce greater variation in documentation, overlapping functionality, and implicit permission boundaries, potentially making minimum-privilege selection more difficult. Conversely, narrower tool exposure or stronger enforcement could reduce over-privilege. We therefore expect performance to depend on the registry’s documentation and access-control design. FORTIS provides a controlled reference for studying these factors systematically; the direction and magnitude of changes on real registries require empirical evaluation.
Rebuttal
We evaluated explicit least-privilege instructions across all three domains and both tasks:
Explicit instructions reduce failure rates by 6.0–9.5 percentage points on Task 1 and approximately 4.2–5.9 points on Task 2. These results show that a simple prompting intervention helps, while substantial failures remain even when least-privilege behavior is explicitly requested. They motivate further investigation of training and architectural approaches, rather than establishing that such approaches are the only possible remedies.
Ground-truth validity in FORTIS is supported by both controlled construction and manual auditing. The benchmark is designed to exclude equally valid alternatives at the same or lower privilege level, and Appendix A.6 explicitly treats the existence of such alternatives as grounds for invalidating an annotation.
The two-author audit checked whether these construction constraints held in the sampled queries. The reported 193/200 agreement measures consistency between the auditors, rather than the proportion of valid labels. The seven disagreements were resolved using the stated minimum-privilege criterion, with annotations refined where necessary.
We used a fixed temperature of 0 to limit sampling variability and maintain a consistent evaluation protocol across models. The main finding is supported by recurring patterns across ten models, three domains, and multiple evaluation settings, rather than a single model comparison. Our central conclusion concerns the substantial failure rates observed across these settings, rather than small differences between individual models.
The per-setting sample sizes are provided below. Task 1 uses the same distribution in each domain, while Task 2 counts vary by domain.
These sum to 600 queries for Task 1 and 1,543 queries for Task 2.
The product EM₁ × EM₂ is intended as an illustrative estimate of how errors at two stages can compound, rather than a directly measured end-to-end success rate. We agree that independence is not guaranteed because the two tasks use different query sets. Accordingly, the product should be interpreted under this explicit assumption, rather than as a general upper bound. Our main findings rely on the separately evaluated results for skill selection and skill-grounded tool selection, and do not depend on this estimate.
Our least-privilege prompting ablation evaluates a simple intervention across all three domains and both tasks:
These results show that explicit guidance improves performance, while substantial failures remain. Beyond prompting, repeated human-in-the-loop approvals can introduce workflow friction, motivating methods that train least-privilege restraint directly into models. Promising directions include preference training that favors minimally sufficient capabilities and skill-aware planning. FORTIS provides a testbed for evaluating these approaches jointly on privilege restraint and selection correctness.