- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents PERMDELTA, a benchmark on whether an agent's authorization changes with its task. Each test is a base and mutation pair. An oracle gives the permissions to add, remove, and keep.
Contributions:
- An atomic track (150 pairs) and a multistep track (25 pairs) over five software domains.
- Metrics for task utility (UCR), exact relation (RDE), and end-to-end exactness (E2E).
- Results for 9 models: atomic UCR is 63.7%, but RDE is only 24.2%.
Critical strengths
-
An important question with a reusable paired design. Most agent benchmarks only check task success. This one checks whether requested authority tracks task scope.
-
Layered diagnostics. The multistep track shows where authorization breaks. Only 60 of 204 exact declarations stay exact on all five layers.
Critical weaknesses
-
The benchmark does not measure errors caused by a task change. Each endpoint starts from a fresh reset, so the agent never sees the base task. Identity pairs, where nothing changes, score 24.4% RDE, close to real mutations (23.0% to 26.1%).
-
The headline metric is too strict. RDE needs exact JSON equality, so claude-sonnet-5 completes 69.7% of tasks but gets 0.1% RDE. The promised evidence package is also missing from the supplementary material.
Questions for authors
- What RDE does a baseline that copies the visible parameters reach?
- How many inexact policies authorize the same resources as the reference?
- Can you test the base and the mutation in one session?