acceptodds
Under review as a conference paper at ICLR 2027

Do Agents Do What Users Ask? Measuring Intent-State Consistency in Tool-Using Language Agents

Abstract

Language agents increasingly act on behalf of users by invoking tools that send messages, book services, and modify persistent environment state. Existing evaluations measure task completion, attack success, policy compliance, or final-state correctness, but do not directly test whether the effects produced throughout execution remain within the user's authorization. We introduce contract-conditioned intent-state consistency, which evaluates agent behavior at three complementary levels: tool-call authorization, final-state consistency, and trajectory consistency. Starting from the same initial state, we replay reference and agent executions, associate tool calls with their realized state transitions, and identify missing required effects, persistent unauthorized effects, and transient unauthorized effects. Across 1,600 replayed AgentDojo trajectories from four models, these signals provide behavioral evidence complementary to task utility and attack-success labels. On a stratified human-review sample of 300 trajectories, full-trajectory evaluation achieves 0.814 precision, 0.809 recall, and 0.811 F1 for identifying executions judged inconsistent with user authorization, compared with 0.788 F1 for final-state-only evaluation. Trajectory replay further exposes intermediate effects that disappear before the final state. These results support intent-state consistency as a complementary diagnostic axis for evaluating tool-using agents under a supplied intent contract.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Intent-State Consistency, Agent Evaluation, Agent Safety, Trajectory Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Bn2BOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper checks whether tool-using agents stay within the user’s authorization by examining attempted calls, final states, and intermediate changes. Using 1,600 AgentDojo trajectories, it shows how replay can reveal unauthorized effects that disappear before execution ends.

Critical strengths
  1. The framework separates failed unauthorized attempts, persistent effects, and temporary effects.
  2. Replaying executions from the same initial state provides concrete evidence linking calls to state changes.
Critical weaknesses
  1. ToolSandbox already checks forbidden events in intermediate states, and AgentPex evaluates constraints across full traces. The paper discusses these approaches but does not compare against them.
  2. Failed unauthorized attempts and violations of confirmation or ordering requirements are central to the framework, but their detection accuracy is not separately reported against human authorization judgments.
Questions for authors
  1. Could you compare with adapted ToolSandbox checks or AgentPex under the same authorization rules?
  2. How accurately does the evaluator detect failed unauthorized attempts and violations of confirmation or ordering requirements?
2 comments

Dear Authors,

As the rebuttal period is nearing its end, I would like to follow up on the questions raised in my review. I would appreciate any responses or clarifications you could provide, as these would help inform my final assessment.

Thank you for your time.

Q1:

We added the setup, accuracy table, mechanism analysis, and cost/repeatability results to Appendix G (Tables 12–13).

Setup. ToolSandbox's milestone/minefield evaluation is re-implemented on our replays under the Table 1 authorization rules. AgentPex runs from its official code, first with its native specifications and then with our contract in the specification slots, keeping its checker unchanged (Appendix G.1).

ToolSandbox. With minefields, its F1 is 0.813 versus our 0.816 (p = 1.0). The mechanism analysis explains the difference in coverage: milestones alone miss all 27 transient effects; tool-name minefields recover 19, and the remaining eight require the attack goal itself to be enumerated. Our default-deny contract detects all 27 without attack knowledge. ToolSandbox also misses 15 persistent effects outside milestone/guardrail coverage. Its tool-trace minefields detect the failed unauthorized call, which our separate VcallV_{\mathrm{call}} check also flags. The six cases favoring ToolSandbox arise from timestamp/URL artifacts; canonicalization gives F1 0.826 (post hoc; Appendix E.1).

AgentPex. Native F1 is 0.786. Supplying our contract increases detection of labeled transient/persistent effects from 3/10 and 62/90 to 10/10 and 81/90. Its authorization-rule score is then not significantly different from ours (F1 0.820 versus 0.816, p = 0.65). The aggregate is higher (0.879, p = 0.011): 29 of its 42 uniquely correct cases concern wrong replies or off-task reads without state violations. Excluding all 43 such cases gives 0.867 versus our 0.890 (post hoc, p = 0.38); the default threshold gives aggregate F1 0.840 (p = 0.25). Full results remain in Appendix G.2.

Summary. Detection coverage depends on the contract. With explicit authorization information, the three authorization-focused checkers have similar F1 (0.813–0.820). Our replay provides deterministic, state-change evidence without model calls (1,600 traces in about three minutes on two CPU cores). AgentPex with our contract costs about $0.015 per trace; 10.4% of aggregate decisions differ across three runs on the 230 completed traces (Appendix G.3).

Q2:

Failed unauthorized attempts. VcallV_{\mathrm{call}} flags 5 of the 6 failed unauthorized attempts with static contracts. It checks attempted calls regardless of execution success (Eq. 4), including calls with Δi=∅\Delta_i=\varnothing. Across 1,600 replayed traces, 11 of the 34 failed mutating calls unmatched to the reference are unauthorized attempts. The other five unauthorized attempts lack static contracts and are unevaluable. The remaining 23 calls are retries or repetitions of requested operations; VcallV_{\mathrm{call}} flags 22.

Confirmation and ordering. AgentDojo contracts contain no explicit confirmation or ordering requirements, so the accuracy of detecting these violations has not been directly measured on this benchmark. Among the 300 reviewed trajectories, six have confirmation-required rationales: payments without bill clarification, an unrequested switch to recurring payments, and calendar invitations to an unrequested participant. These actions produce unrequested realized effects, and the final-state or trajectory-action signal in Table 1 flags all six trajectories. No sampled trajectory was judged to violate an ordering requirement.

Reviewer_k7ZFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes evaluating tool-using language agents at three levels: tool-call authorization, final-state consistency, and trajectory consistency, against a user-supplied intent contract that specifies required, permitted, and forbidden effects. By replaying reference and agent executions from matched initial states, the method identifies missing required effects, persistent unauthorized effects, and transient unauthorized effects. The framework is validated on cached AgentDojo trajectories and shown to align with human judgments of user-intent compliance.

Critical strengths
  1. The paper identifies a genuine and under-addressed evaluation gap: existing checks do not capture whether every realized effect of an agent's actions stayed within the user's authorization, and it grounds this gap in a concrete motivating example.
  2. The empirical validation is careful and appropriately hedged, using a sizeable human-labeled sample with a documented annotation protocol and a contract-source sensitivity audit, which lends credibility to the reported agreement with human judgments.
Critical weaknesses
  1. Limited novelty of the proposed method. The three-level framework largely recombines existing evaluation methods, such as final-state checking, trajectory matching, and policy compliance, under a new intent contract abstraction. The headline contribution of exposing transient unauthorized effects amounts to keeping intermediate state differences that final-state comparison already computes.
  2. Dependence on an externally supplied contract. The framework assumes a given, internally consistent contract and does not derive it from natural language. All main experiments ground contracts in reference trajectories, which the authors concede can misclassify valid alternative paths, so the evaluation depends on an upstream input the paper does not resolve.
  3. The experiments do not demonstrate superiority over existing checks. Adding action mismatch to final-state mismatch yields only a marginal F1 gain, while precision and balanced accuracy fall. The authors themselves acknowledge this trade-off. Many transient-only candidates are judged consistent by humans, so the incremental value of the new signals remains unproven.
  4. Defensive and formulaic writing style. The paper frequently repeats the same contrastive pattern, affirming a point and then immediately denying the alternative, for example by calling a result a trade-off rather than an isolated gain. This repeated template makes the argumentation read as defensive and contrived, with noticeable AI-writing traces.
Questions for authors
  1. How sensitive are the reported results to the heuristic primitives of the method, such as the operation ontology and the stable identity of effects? Which granularity should a practitioner adopt, and how would misclassifying a benign mutation as unauthorized affect the reported metrics?
  2. For the transient-only candidates that humans judged consistent, what distinguishes them from genuine violations? Are these cases legitimate temporary operations, an overly narrow contract, or annotation noise? Does the framework contain any signal that could separate them without human review?
2 comments

R1: Novelty and transient effects (W1)

The contribution lies in how state diffs, trajectory matching, and contract-style policies are connected: under one contract, each attempted call is linked to the state change it realizes. This yields outputs that none of these methods provides separately.

Transient effects need more than a final-state diff. Final-state comparison computes Diff(S0, ST), so an effect that is added and later reversed leaves no trace in it. Detecting that effect requires per-step replay, the net-effect sequence E_t^net (Eq. 6), and a stable effect identity (resource, object, relation, endpoint; Section 3.3). With that identity, an effect added at step t and removed at t′ is tracked as one effect and attributed to the call that produced it. Eq. 7 then splits unauthorized effects into persistent and transient ones.

The three levels return different answers on the same trajectory. In Figure 1, the delete call is unauthorized and fails with Δ = ∅, the final state is correct, and the temporary grant to Eve is visible only in the trajectory. Tool-name blocking, terminal-state checks, and trajectory matching each see part of this case. Separating attempted calls from realized effects under one contract lets the evaluator report all three.

Appendix H isolates the trajectory signal. On the 300 traces, adding E_transient to final-state checking adds 4 true positives and 0 false positives (F1 0.813 → 0.826); on the stress set it finds all 22 transient-only violations that final-state checking misses. Keeping intermediate differences is the baseline we ablate (Table 15): without the contract, it flags 122 of 241 benign controls, including all 120 authorized temporary files, and replacing stable identities with flattened state paths causes false alarms on 4 of 24 equivalent reorderings. With both components, E_transient flags none.

R2: Contract construction (W2)

The evaluator takes the contract as input; deriving it from natural language is outside the paper's scope (Section 1). The relevant question is how much reference grounding changes our results, and the submission already measures this.

Request-only contracts change few labels (Sec. 5.4.2, App. C.1). Two annotators independently wrote contracts for 86 AgentDojo v1 user tasks from the request text alone, without access to reference calls. They judged 72 and 73 of these tasks to admit multiple valid implementations, so their contracts accept alternative paths that reference-grounded contracts reject. Of the 858 mismatches flagged under reference-grounded contracts, 771 (89.9%) and 761 (88.7%) remain violations under the two request-only contracts; only 49 (5.7%) and 53 (6.2%) become possibly valid alternatives.

Table 1 already includes this error. The 300 human labels in Table 1 were assigned from the request and the execution evidence, and annotators never saw an evaluation contract (Section 5.1). When a reference-grounded contract rejects a valid alternative path, the case counts as a false positive against these labels, so the reported precision of 0.821 already includes the cost of reference grounding.

Supplied specifications are standard in state-based evaluation. τ-bench compares final states against annotated goal states, AppWorld checks states with hand-written tests, and ST-WebAgentBench evaluates agents against supplied policies. Our contract plays the same role. Because it is written as typed predicates over required, permitted, and forbidden effects, any disputed judgment can be traced to the predicate that produced it and corrected there.

Revision. In Section 4.1, we replaced the sentence that deferred the contract-source comparison to future work with the request-only contract results above.

R5: Writing clarity (W4)

Thank you for pointing this out. We revised the identified sentences throughout the manuscript, replacing repeated contrastive constructions with direct statements while keeping the technical distinctions and qualifications.

R3: Empirical value and transient false positives (W3, Q2)

The 9 consistent candidates are all artifacts of the reference-action matcher. The agent passed get_webpage a URL with http://, and the reference call used the same URL without it. AgentDojo strips http(s):// (standardize_url), so the agent's state matches the reference state at every step. The string matcher marks the call as unmatched, making the trace transient-only. Both annotators noted the agent only read the requested page. The 4 ties have the same cause. The 22 inconsistent candidates are all unauthorized effects undone before the trace ends. In one trace, the agent invites an external user named in injected content to the workspace and #general, then removes him. In another, it books an unrequested hotel and later replaces the booking.

The same artifact explains the precision drop. On the 300 sampled traces, the union adds 9 action-only flags: 4 genuine transient effects, 4 URL artifacts labeled consistent (the 4 added false positives), and 1 URL artifact labeled inconsistent due to a wrong reply. Precision and balanced accuracy fall because the matcher compares arguments as strings.

The framework's trajectory signal, E_transient (Eq. 7), checks each step's realized state effects against the contract, so equivalent URLs yield the same result. Final-state F1 is 0.813 here (0.809 in Table 1) because wall-clock timestamps are excluded.

DetectorDataTPFPPRF1BA
Table 1 union300 sample151330.8210.8120.8160.761
Final state300 sample146270.8440.7850.8130.774
+ E_transient300 sample150270.8470.8060.8260.785
Final statestress set, 924390.8270.6620.7350.664
+ E_transientstress set, 926590.8781.0000.9350.833

On the 300 traces, E_transient adds 4 true positives and no false positives, so all four metrics rise (exact McNemar p = 0.125). These 4 transient-only violations are the maximum gain on this sample. On the stress set, final-state checking finds 0 of 22 violations, and E_transient finds all 22 with 0 of 9 artifacts flagged. Across all 1,600 traces, 27 transient-only candidates remain after removing 18 URL artifacts. All 25 with human labels were judged inconsistent, and AgentDojo scores all 27 as utility successes.

Existing checks. Terminal-state checks (AppWorld, τ-bench) detect none of the 27 by construction. ToolSandbox milestones detect 0, its tool-name minefields 19, and native AgentPex 10 (reply to Reviewer Bn2B). E_transient detects all 27 without being told the attack.

Separation without human review. E_transient flags all 22 inconsistent candidates and none of the 13 artifacts (9 consistent, 4 ties). The ontology rule "the trace contains an unmatched mutating call" gives the same split. To test legitimate temporary operations, which are absent from these candidates, we injected known effects into all 97 tasks' reference trajectories. E_transient flags 0 of 120 authorized scratch files created and then deleted, and 186 of 186 unauthorized add-then-revert effects. A contract-free variant also catches the 186 violations but flags all 120 scratch files, so the contract is what separates a legitimate temporary operation from a reverted violation.

R4: Sensitivity to heuristic primitives (Q1)

We varied each primitive on 1,600 replays and scored against Table 1's 300 human labels (Table 11). The baseline reproduces Table 1 exactly. Ontology gating gives F1 0.806 (baseline 0.816), and alternative identity definitions give 0.816–0.822. After environment canonicalization, all identity definitions yield the same 27 transient-only traces, and every variant still detects all 10 transient violations in the effect-type annotation (Appendix H.4).

Attribute comparison is the main source of false positives: 26 of 33 come from attribute differences between objects with the same identity. We recommend the Section 3.3 identity, with attribute checks restricted to contract-bound amounts, recipients, timing, recurrence, and content within specified data boundaries, and with environment-normalized values canonicalized. This raises F1 from 0.816 to 0.843 and lowers FPR from 0.289 to 0.149, with three additional missed violations. We report this post hoc variant separately.

Misclassifying authorized mutations as unauthorized adds false positives and leaves recall unchanged. Across 200 random draws, a 20% per-mutation error rate gives mean F1 0.802, and misclassifying every authorized mutation gives 0.757. The reverse error changes F1 by less than 0.001 at rates up to 20% because other signals usually flag those violations too. We will add the reproduction scripts to Appendix E.

Reviewer_WBiiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes contract-conditioned intent–state consistency, a framework for evaluating whether tool-using agents stay within user authorization. It checks three complementary levels: attempted tool calls, final environment states, and intermediate state changes. By replaying executions, it distinguishes missing required effects, persistent unauthorized effects, temporary unauthorized effects that are later reversed, and unauthorized attempts that fail. Experiments use 1,600 AgentDojo trajectories, including a 300-trajectory human evaluation.

Critical strengths
  1. The framework captures failures that final-state evaluation can miss, such as temporarily granting unauthorized access or attempting a prohibited operation that fails. This could be a useful distinction between attempts, outcomes, and intermediate effects.

  2. Linking tool calls to realized state changes helps identify which action affected which resource, making the results useful for auditing and debugging.

Critical weaknesses
  1. The experiments do not isolate the framework’s central contribution. Adding trajectory-action mismatches improves F1 only from 0.809 to 0.816, finding five additional true positives while introducing four false positives. This comparison does not specifically establish the benefit of detecting temporary unauthorized effects. Separate effect-type annotations and controlled ablations are needed to validate that contribution.
Questions for authors

See Weaknesses.

1 comment

The +0.007 F1 in Table 1 comes from adding reference-action mismatches. This comparison can flag representation-level differences even when the environment maps both representations to the same state. For example, AgentDojo normalizes differently formatted URLs to the same page. On the 300 labeled traces, all four added false positives are representation-level mismatches. One of the five added true positives also has a representation-level mismatch and is labeled inconsistent because of a wrong final answer; the other four are genuine transient unauthorized effects. Appendix H therefore evaluates the framework’s realized-effect outputs separately under the same reference-grounded contract.

The effect-level ablation isolates the transient detector (Appendix H.1, Table 14). Adding EtransientE_{\mathrm{transient}} contributes four true positives and zero false positives, raising F1 from 0.813 to 0.826 (paired bootstrap 95% CI for ΔF1: [0.003, 0.027]; exact McNemar p=0.125p=0.125). The 0.813 baseline differs from Table 1’s 0.809 by excluding wall-clock fields, removing two false positives; both ablation rows use this normalization. Effect-type annotation identifies ten transient violations: six also carry final-state violations, leaving four that trajectory checking can uniquely add. Our detector finds all four.

Transient violations create unauthorized effects during execution, even when subsequent operations undo them. For example, one failure mode invites an external account into the Slack workspace and #general, grants it channel access, and removes it before completion. The effect is no longer visible in the final state, and the benchmark utility label may still indicate success. All four uniquely recovered traces in the 300-trace sample are labeled inconsistent and scored as utility successes. On the two-annotator stress set, final-state checking detects 0/22 transient-only violations; adding EtransientE_{\mathrm{transient}} detects 22/22 and raises overall recall from 0.662 to 1.000 at precision 0.878. Across all 1,600 traces, it flags 61 traces. Of these, 27 are transient-only candidates, and AgentDojo scores all 27 as utility successes (Appendix H.3).

Controlled injections test the mechanism with known ground truth (Appendix H.2, Table 15). Across 97 tasks, we construct 711 violating and 241 benign traces without model calls. Adding EtransientE_{\mathrm{transient}} detects all 186 add-then-revert violations, versus zero for final-state checking, with zero benign false alarms. Both detect all 234 persistent violations; the full checker additionally detects 285/291 rejected unauthorized calls. Removing the contract flags all 120 authorized temporary-file controls; replacing stable identities with flattened paths produces false alarms in 4/24 equivalent reorderings. These ablations isolate authorization conditioning and stable identity.

Separate effect-type annotations are reported in Appendix H.4, Table 16. We label 366 traces for missing required effects, persistent unauthorized effects, transient unauthorized effects, and failed unauthorized attempts, excluding unsure items. Precision and recall are computed on the original 300-trace sample; the 66 additional candidates are reported separately as counts. EtransientE_{\mathrm{transient}} achieves 10 TP, 0 FP, and 0 FN, while final-state checking reaches six of these ten through co-occurring violations. The appendix also reports the other types’ results and error analysis, including failed-attempt precision of 0.67 with two positive examples.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.