- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper is about permission boundary inference. With task and read-only access to its environment, an authorisation agent predicts the file read, write, and execute permissions that a different agent would need. AuthBench contains 120 tasks in 10 domains. The authors combine agreement with a human-reviewed reference policy, task success under generated permissions, sensitive permission exposure, and realised attack outcomes. They report simultaneous under- and over-authorisation, investigate how different models respond to increasing reasoning effort and introduce Sufficiency-Tightness (S-T) Decomposition. This is generating a policy, prioritising coverage, and then auditing and narrowing it.
Critical strengths
The problem is interesting and seems relevant. A successful agent can still have been given unnecessary authority. Combining static permissions agreement, functional validators, and separate attack endpoint validators is a well-appearing evaluation design. They authors also note that several different valid workflows can need different permissions, and an appendix don't treat all failed executions as proof of an authorisation error.
This submission also provides great experimental controls and substantial supporting material. The full access and golden permissions references help to interpret the tradeoff. In the appendix, they check alternative executors and also include neat/avoid and generic revision baselines and phase-wise results. Those comparisons are stronger than evaluating S - T only against direct. For example, Table 2 reports Claude's sensitive task success increasing from 61.7% to 75.0%, while attack success decreases from 25.8% to 15.0%.
The provided task environments, validators, prompts, and enforcement code also make the evaluation well-inspectable.
Critical weaknesses
Section 2.1 writes about a whitelist comprising the generator policy and fixed runtime permissions, but in policy_enforcement.py, _expand_write_patterns converts an exact write, such as /workdir/report.json, into /workdir/**. In authbench_log_launch.c, add_rule_from_arg also gives read access to an existing ancestor where an exact retarget does not yet exist. That could be important for tasks that create an output and read it afterwards. The policy guard plugin checks the read, write, and edit tools, but I did not find an equivalent restriction there on file access performed within exec. That means a policy can look narrow in static scoring but allow more broad access while execution. This could affect task success and security outcomes and make the connection between the inferred policy and measured behaviour not as it seemed.
Figure 5 and the phase-wise results make it seem like model-dependent tendencies, but they do not show that one past generation has an inherent cognitive conflict or that the particular ordering of s - t solves its original cause. The direct procedure itself allows several inspection turns. The prompting baselines are useful, but token/tool budgets and uncertainty are not reported with the comparisons, and the derivation of Figure 5's vector field is not specified enough. The supplement gives configurable repeat counts, which is helpful, but does not show the run configurations behind figures and tables.
Questions for authors
-
What is the supplied enforcement implementation used for the reported experiments? What prevents shell subprocesses from accessing siblings covered by the expanded parent directory rules? It would be great if you could please explain the handling of writes and reads of not-yet-created outputs and provide a check that the intended file is accessible while an unlisted sibling is denied. If the effective permissions are broader than the score policy, which reported results are affected?
-
Which SER definitions create Tables 1, 2, and 8? The per-axis average in Equation 8 or the pooled implementation? How are empty sensitive accesses handled?
-
Could you maybe provide the observed metrics at each reasoning effort setting and explain how they produce Figure 5, including any fitting or interpolation? What were the generation/replay repeat counts and influence budget for the compared methods? Task-level uncertainty estimates for the main differences would help.
3 comments
Response to Reviewer afQM (Part 2)
Q3 & W2. Vector field, repeats, compute, and uncertainty
Interpretation and S-T design. Increasing reasoning effort does not consistently improve authorization quality. We interpret this pattern as a tension between discovering necessary permissions and pruning unnecessary grants, which motivates sufficiency reasoning followed by a tightness audit. Compared with Needed/Avoid and Generic Revision, S-T achieves lower SER and ASR across all three models, supporting this design.
Here, “attractor” describes a model-specific tendency in authorization decisions rather than a physical dynamical mechanism. Such tendencies may reflect knowledge and decision biases acquired during pretraining and post-training.
Vector-field construction. For each task and reasoning-effort setting, we average its metrics and compute:
\qquad B_{\mathrm{over}}=R_{\mathrm{macro}} \left(\frac{1}{P_{\mathrm{adjusted}}}-1\right).$$ Here P\_adjusted is P\_macro for standard tasks and P\_macro × (1 − SER) for sensitive tasks. We clip B\_over to \[0, 3] and display x = log(1 + B\_over) / log(4), y = B\_under; zero adjusted precision maps to B\_over = 3. Displacements pair the same task at **adjacent effort settings**: Claude low→medium→high→max; Gemini low→medium→high; GPT-5.4 none→low→medium→high→xhigh. We discard displacement magnitudes below 0.006. On a **20 × 14 grid**, we apply distance-based Gaussian averaging with bandwidth 0.12 and radius 0.30. Grid points with total weight below 0.36 or smoothed magnitude below 0.006 are omitted. Arrows have fixed length; direction is the averaged displacement, and color shows its magnitude normalized within each panel. **Observed metrics by reasoning effort.** P and R are macro-averaged over read, write, and execute; all values are percentages. \* marks the configuration used in the main evaluation. | Model | Effort | Standard P | Standard R | Sensitive P | Sensitive R | SER ↓ | | --------------- | ------- | ---------- | ---------- | ----------- | ----------- | ----- | | Claude Opus 4.6 | low | 81.8 | 77.7 | 77.1 | 88.0 | 51.7 | | | medium | 82.2 | 77.6 | 78.2 | 87.2 | 48.8 | | | high | 82.2 | 79.5 | 78.6 | 88.1 | 50.3 | | | max\* | 80.0 | 76.4 | 71.3 | 85.1 | 47.0 | | GPT-5.4 | none | 71.2 | 70.4 | 74.0 | 85.7 | 45.2 | | | low | 76.4 | 70.5 | 82.8 | 86.7 | 41.6 | | | medium | 76.3 | 69.8 | 84.0 | 87.1 | 42.9 | | | high | 74.5 | 66.1 | 84.4 | 83.5 | 28.7 | | | xhigh\* | 78.4 | 69.0 | 79.7 | 79.1 | 21.1 | | Gemini 3.1 Pro | low | 74.3 | 81.2 | 78.4 | 89.0 | 46.8 | | | medium | 72.7 | 84.7 | 77.8 | 92.9 | 34.3 | | | high\* | 69.7 | 87.2 | 76.1 | 94.6 | 34.8 | Across the evaluated settings, **increasing reasoning effort does not consistently improve authorization quality**. **Repeats and budgets.** Each task has **three policy generations, each followed by one execution replay**. The four methods share the following effort settings and per-stage turn limits: | Model | Reasoning effort (all four methods) | Turn limit per stage | | --------------- | ----------------------------------- | -------------------- | | Claude Opus 4.6 | max | 50 | | GPT-5.4 | xhigh | 50 | | Gemini 3.1 Pro | high | 50 |Response to Reviewer afQM (Part 3)
Observed average usage is:
| Model | Method | Mean input tokens | Mean output tokens | Mean tool calls | Mean model turns |
|---|---|---|---|---|---|
| Claude Opus 4.6 | Direct | 35,848 | 22,690 | 17.12 | 6.97 |
| Claude Opus 4.6 | Needed/Avoid | 70,456 | 21,506 | 14.39 | 14.12 |
| Claude Opus 4.6 | Generic Revision | 104,220 | 58,260 | 30.72 | 20.54 |
| Claude Opus 4.6 | S-T | 83,462 | 33,232 | 20.16 | 15.17 |
| GPT-5.4 | Direct | 22,010 | 29,424 | 11.43 | 4.92 |
| GPT-5.4 | Needed/Avoid | 32,522 | 25,128 | 6.72 | 6.41 |
| GPT-5.4 | Generic Revision | 56,352 | 55,236 | 22.66 | 11.39 |
| GPT-5.4 | S-T | 45,428 | 27,626 | 13.45 | 7.34 |
| Gemini 3.1 Pro | Direct | 33,024 | 17,867 | 13.32 | 7.50 |
| Gemini 3.1 Pro | Needed/Avoid | 26,791 | 11,696 | 9.89 | 6.72 |
| Gemini 3.1 Pro | Generic Revision | 65,184 | 29,890 | 24.85 | 14.53 |
| Gemini 3.1 Pro | S-T | 32,790 | 14,573 | 13.78 | 10.25 |
S-T uses fewer input tokens, output tokens, tool calls, and model turns than Generic Revision for all three models, indicating that its gains are not merely due to test-time compute scaling.
Task-level uncertainty. We compute 95% confidence intervals using paired bootstrap over the 40 sensitive tasks under the main GPT-5 executor, retaining each task’s three runs together. We use 10,000 resamples and the 2.5th and 97.5th percentiles of the bootstrap difference distribution. Differences are S-T minus Direct, in percentage points.
| Model | ΔASR, pp (95% CI) | ΔSUR, pp (95% CI) |
|---|---|---|
| Claude Opus 4.6 | −10.8 [−18.3, −3.3] | +11.7 [+4.2, +19.2] |
| GPT-5.4 | −4.2 [−7.2, −0.8] | +15.8 [+7.0, +24.0] |
| Gemini 3.1 Pro | −15.8 [−24.2, −7.8] | +5.0 [+0.8, +9.2] |
Response to Reviewer afQM (Part 1)
Thank you for the careful review and detailed feedback on permission enforcement and evaluation methodology.
We have revised the manuscript: Appendix E.2 details enforcement and metric aggregation; Appendix F.2 reports run configurations and compute usage (Table 10); Appendix H.1 specifies the vector-field construction. The main text now uses pooled SER and describes adjacent-effort displacements.
Q1 & W1. Permission enforcement
Thank you for examining the enforcement implementation so closely. You are the first reviewer to engage with these concrete implementation details; the design involves subtle technical details that required careful consideration during implementation, and we appreciate your thoughtful discussion of them.
The supplement mistakenly included an earlier implementation. That version broadened exact write paths and some nonexistent read paths to parent or ancestor directories. The reported experiments used the updated implementation, so none of the reported results are affected. It combines Landlock with the policy guard: Landlock constrains the execution process and its descendants, including file access inside exec; the guard additionally checks read/write/edit tool paths.
Landlock rules attach to existing filesystem objects. We enforce exact permissions on existing inputs and pre-create task-required output files explicitly granted write access by the generated policy, then assign read and write permissions separately. Intermediate files and directories use /tmp, a fixed read/write runtime workspace; final results are written into the authorized output files. Other directory or file-set creation requires explicit policy grants. The final implementation enforces exact-path grants without broadening them to parent directories, so it does not allow access to unlisted sibling files through parent-directory expansion.
In a manual inspection of 120 sampled trajectories across executors, all agents discovered that /tmp was writable through their attempts, without an explicit reminder. None of these runs failed because temporary-file writes were denied.
Q2. SER definition
Tables 1, 2, and the original Table 8 all use the same implementation, with pooled read/write/execute counts within each policy evaluation:
\frac{\sum_{a\in\\{r,w,x\\}} |\pi_{i,a}\cap S_{\mathrm{sens},i,a}|} {\sum_{a\in\\{r,w,x\\}} |S_{\mathrm{sens},i,a}|}.$$ We then take the **arithmetic mean across runs with a sensitive-exposure metric**, rather than pooling counts across tasks. An empty axis contributes zero to both sums. An entirely empty sensitive set returns 0.0, which is included in the mean; only runs without the sensitive-exposure metric field are excluded. SER is designed for sensitive tasks, and all 40 sensitive tasks have nonempty sensitive-permission annotations. Different operations on the same file count separately. **We have corrected the paper’s formula to this pooled definition; the reported SER values are unchanged.**