acceptodds
Under review as a conference paper at ICLR 2027

PERMDELTA: Can AI Agents Keep Authorization Aligned with Task Changes?

Abstract

AI-agent benchmarks usually ask whether work succeeds; they rarely test whether authorization changes with the task. To our knowledge, PERMDELTA is the first replayable benchmark for AI agents that uses a directed base–mutation pair as the primary unit for evaluating authorization alignment. For each pair, a typed oracle specifies the authority to preserve, add, or remove, and the benchmark tracks that change from declaration through execution, with workflow-impact evidence for multistep tasks. Across evaluated systems, 63.7% of atomic task conditions complete successfully, but only 24.2% of paired authorization changes are exact and 8.0% are exact end to end. This gap shows that task success can coexist with residual authority or downstream realization failure. PERMDELTA contributes deterministic reset and replay, fail-closed execution, and layered evidence that expose where authorization alignment is lost. It reframes authorization from a one-time policy string into a measurable relation between changing work and authority. Because the oracle records what must be preserved, changed, and protected, the same evidence can be replayed for model evaluation, policy regression testing, and broker validation.

Primary area:
datasets and benchmarks
Keywords:
AI Agents, Agent Benchmarks, Authorization Alignment, Task Mutations, Relational Evaluation, Temporal Authorization, Least Privilege

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kmDAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents PERMDELTA, a benchmark on whether an agent's authorization changes with its task. Each test is a base and mutation pair. An oracle gives the permissions to add, remove, and keep.

Contributions:

  1. An atomic track (150 pairs) and a multistep track (25 pairs) over five software domains.
  2. Metrics for task utility (UCR), exact relation (RDE), and end-to-end exactness (E2E).
  3. Results for 9 models: atomic UCR is 63.7%, but RDE is only 24.2%.
Critical strengths
  1. An important question with a reusable paired design. Most agent benchmarks only check task success. This one checks whether requested authority tracks task scope.

  2. Layered diagnostics. The multistep track shows where authorization breaks. Only 60 of 204 exact declarations stay exact on all five layers.

Critical weaknesses
  1. The benchmark does not measure errors caused by a task change. Each endpoint starts from a fresh reset, so the agent never sees the base task. Identity pairs, where nothing changes, score 24.4% RDE, close to real mutations (23.0% to 26.1%).

  2. The headline metric is too strict. RDE needs exact JSON equality, so claude-sonnet-5 completes 69.7% of tasks but gets 0.1% RDE. The promised evidence package is also missing from the supplementary material.

Questions for authors
  1. What RDE does a baseline that copies the visible parameters reach?
  2. How many inexact policies authorize the same resources as the reference?
  3. Can you test the base and the mutation in one session?
Reviewer_Am3bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tests whether agents choose the right permissions when tasks change. Its main value is showing which permissions are wrongly added, removed, or kept. However, for the "Strengths" part, I used AI to polish the writing.

Before I polish: "The paper addresses an important and relevant problem practically: whether agents approppriatelly manage permission as task requirements change. Its explicit Add/Renove/Preserve decomposition provides an interpretable way to distinguish different types of authorization errors, while deterministic replay & layered execution evidence facilitates systematic failure analysis and reproducibility. Lastly, the authors clearly and thoughtfully acknowledge the limitations of fresh-session evaluation and articulate the distinction between delta exactness and endpoint correctness"

Prompt: check grammar & polish

After I polish: The paper addresses an important and practically relevant problem: whether agents use permissions appropriately as task requirements change. Its explicit Add/Remove/Preserve decomposition provides an interpretable way to distinguish different authorization errors, while deterministic replay and layered execution evidence make failures easier to inspect and reproduce. Lastly, the authors also clearly acknowledge the limitations of the fresh-session evaluation and explain the relationship between delta exactness and endpoint correctness.

Critical strengths

The paper addresses an important and practically relevant problem: whether agents use permissions appropriately as task requirements change. Its explicit Add/Remove/Preserve decomposition provides an interpretable way to distinguish different authorization errors, while deterministic replay and layered execution evidence make failures easier to inspect and reproduce. Lastly, the authors also clearly acknowledge the limitations of the fresh-session evaluation and explain the relationship between delta exactness and endpoint correctness.

Critical weaknesses

However, I remain a bit unconvinced that the current evaluation sufficiently supports its broader task-change motivation. Independently reset base and mutation runs measure counterfactual authorization consistency, not adaptation of inherited permissions during an actual task transition, right?

Moreover, since complete delta exactness is equivalent to correctness at both endpoints, the main added value lies in the diagnostic decomposition. So a controlled comparison showing its benefits for failure analysis or targeted repair would strengthen the contribution.

Questions for authors
  1. Do delta diagnostics improve targeted repair compared with endpoint diagnostics?

  2. What happens after correcting conflicting prompts and increasing execution budgets?

  3. The presentation is nice, and the idea is interesting. Could you supplement more experiments for the core claim? What would be some you would consider?

Reviewer_ySjoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates authorization through matched base–mutation task pairs across five infrastructure domains. Independently reset runs produce typed policies and execution evidence, which are compared against add/remove/preserve oracles. An atomic track evaluates policy declarations, while a multistep track adds grant, execution, impact, and revocation evidence across model–harness configurations.

Critical strengths
  • I aggree with the gap this benchmark address in comparison with other relevant work, that task completion alone does not establish appropriately scoped authorization. Controlled mutations and separate evidence layers provide useful diagnostics for policy generation and workflow execution.

  • The measurement protocol is carefully specified, including deterministic replay, typed oracles, execution-path controls, family-cluster uncertainty estimates, and explicit missing-data accounting.

Critical weaknesses
  • Unclear about the distinctive value of the paired evaluation: complete delta equality is equivalent to exactness at both endpoints, whereas the reported expansion/contraction contrasts do not clearly differ from identity controls. What mutation-specific insight this provides beyond static permission inference? particularly relative to AuthBench.

  • The headline scores are insufficiently separated from declaration-contract sensitivity. Canonicalization handles some equivalent encodings, but the main results do not quantify how much inexactness reflects genuine authority errors versus contract or representation mismatches. The retained effect and representation audits are important for interpreting the reported model differences.

  • The practical scope seems narrower than the task-change framing: both endpoints start with clean authority, so contraction does not test whether an agent revokes permissions inherited from an earlier task. The within-workflow lifecycle track is useful but does not close this gap.

  • Multistep failures should be attributed more clearly: e.g., for four-to-six-operation workflows, the relationship between the stated interaction/tool-attempt limits and actual execution is unclear. Without termination-reason counts, low workflow scores cannot be confidently attributed specifically to authorization-lifecycle management.

Questions for authors

please see the weakness above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.