- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem of Web Agents combining information from the DOM, visual inputs, and tool outputs to perform complete operations that lack user authorization—such as pairing a legitimate object with a legitimate recipient in a way that was not explicitly authorized. The authors propose CAF, a framework that constructs task contracts from trusted task requests and pre-registered policies, normalizes candidate tool calls into complete semantic relationships, and grants authorization based on provenance evidence and compositional permissions; it then constrains actual dispatch and side-effect submission through parameter binding and a CapabilityBroker. The paper also introduces C3 Bench, covering eight task categories and twelve channel distributions. In replay tests using fixed call pairings across five models, CAF reduced the observed rate of unauthorized effects from a range of 7.7%–21.5% to 0%, while preserving all 1,987 authorized effects found in the baseline. Supplementary experiments included contract compilation, new schema validation, closed-loop execution in AgentDojo, pre-execution intent checking in WebArena, and attack tests involving fixed and limited feedback. These results demonstrate the framework's effectiveness across registered policies, schemas, and protected execution paths.
Critical strengths
- The paper's most valuable contribution is the insight that a set of valid values does not define a set of valid operations. When users authorize specific pairings between two objects and two recipients, simply checking the validity of the object, recipient, and tool individually can still allow erroneous pairings to slip through. CAF explicitly incorporates these pairings into task contracts and composition permissions, enabling security policies to express the operations actually authorized by the user. This advantage is supported by specific experimental evidence:
- Table J2 categorizes samples into 133 pairs distinguishable at the field level and 59 pairs that pass individual field checks but differ in their full relational authorization. This classification helps identify the conditions under which relational checks are truly effective.
- Even more compelling is Table E7: when domain constraints are applied to four newly registered schemas, independent field policies still result in 19/48 unauthorized effects, whereas schema-aware CAF results in 0/48; both successfully preserve all 32/32 authorized effects. This latter comparison rules out a significant alternative explanation: that CAF’s benefits stem merely from stricter schema checks or the rejection of more operations. It provides substantive evidence that checking combined relationships is genuinely necessary. Rationale for acceptance: This insight has concrete implications for the design of agent security systems and has been implemented as an executable, auditable authorization mechanism.
- The paper goes beyond merely determining whether a candidate invocation is safe; it binds approved parameters and verifies actual side effects along the protected execution path. The experiment in Table H4 is particularly valuable: when unmodeled parameters are permitted, both approaches achieve 1,600/1,600 handler reachability; however, with the introduction of the Broker, unauthorized effects drop from 1,600/1,600 to 0/1,600, while authorized control is maintained. Table J3 further demonstrates the versatility of this execution-binding layer by applying it to different authorization backends: in all cases, the 48 instances of parameter modification following approval were eliminated. This demonstrates the author's ability to decouple the benefits of execution binding from those of backend authorization, rather than attributing all improvements solely to the relational representation of the CAF. The contract compilation experiments also employed a corresponding isolation design: the compiler runs prior to the emergence of candidate calls—without accessing the calls or result labels themselves—and verification takes place within a new task context after the state is frozen. These design choices mitigate the risk of the experiments relying on information regarding the expected answers. The paper presents a comprehensive method for failure decomposition, enabling readers to pinpoint exactly which stage of the process yields the security benefits. Its value lies in both the system implementation and the evaluation methodology.
Critical weaknesses
W1. Compared to backends with relational constraints, CAF's advantages have not fully ruled out differences arising from policy encoding versus compilation coverage.
In Table J2, CAF-Compiled-Rule resolves 44 out of 59 pairs within the strict relational difference subset, whereas the two baselines incorporating relational constraints resolve 20 out of 59. Table J3 reports a discrepancy of 15/144 versus 39/144 regarding unauthorized effects, even after sharing execution bindings.
However, the equivalence claimed in Section J.4 is primarily based on the fact that the backends construct representations from the same canonical manifest and process identical instances. Sharing inputs and expected labels is not equivalent to actually encoding semantically equivalent, complete authorization predicates. The current PDF lacks representative rules and failure instances, making it impossible to determine whether the additional 24 errors stem from limitations in relational representation or from the adapter overlooking specific joint constraints, provenance contexts, or channel rules.
My confusion arises because Section A.1 explicitly acknowledges that existing systems are not theoretically incapable of expressing equivalent relational constraints. On the controlled subset in Table D2, both Source/Field and Task Shield-style approaches successfully prevent all observed unauthorized effects under full role integrity conditions.
W2. Unresolved ambiguities remain regarding execution security endpoints, particularly the state anomalies in Appendix J.
First, Tables J5–J7 report 5 out of 576 end-to-end unauthorized state discrepancies for Full CAF, while simultaneously stating that no unauthorized protected path submissions occurred. Based on the content, we can infer the following:
- Four cases occurred after a
blockor unconfirmedconfirmoperation and were attributed to parser fallback or unprotected harness routing. - In the fifth case, a parameter modification was rejected by the Broker yet persisted in the end-to-end UER (Unauthorized Effect Report).
The current documentation fails to clarify: What specific environmental state change occurred in the fifth case that qualified as a UER-relevant event? Are the first four cases actual, exploitable integration gaps, or are they tests specifically constructed to probe boundaries outside the protection scope?
This does not directly invalidate the conclusions regarding the local security of protected paths. However, if these routes belong to the complete Agent system under evaluation, merely excluding them from the path analysis is insufficient to establish the system's end-to-end security.
Furthermore, while J.6 claims that the four variants merely add mechanisms incrementally without altering prior logic, Table J5 shows pre-gateway error counts of 3, 2, 4, and 0, and the authorization retention rate rose from 142/144 to 144/144. These changes require case-by-case explanation to confirm that the ablation did not simultaneously alter parsing or execution behavior.
It remains difficult to determine the security outcomes of the Full CAF when deployed as a complete system, or to fully accept the attribution of effects to specific mechanisms in the staged ablation.
W3. Automated contract construction is validated but primarily covers explicit field extraction; it has not yet been proven capable of supporting tasks where the authorization target depends on environmental information.
The compiler in I.3 requires the action to match the tool mapping precisely and mandates that object, destination, and scope appear verbatim in the task text; it returns confirm in cases of omission or ambiguity. The successful recovery of context in 31 out of 32 cases after freezing is a valuable result, yet it primarily validates this constrained input scenario.
Meanwhile, §4.2 allows runtime observation to instantiate parameters within the contract's authorized value range. A sufficiently clear link between the two is still missing: for tasks where object identity or destination must be determined via environmental queries, how does the contract express a constrained yet uninstantiated authorization and subsequently securely bind the query results?
Table E8 reports the configuration scale for the new schema and indicates rule reuse; while this addresses some concerns regarding configuration costs, it does not substitute for the validation of dynamic task binding.
W4. Binding rules based on tracing parameter values back to their sources lack clear semantics for scenarios involving identical values from multiple sources or derived values.
I.4 explains that the normalizer binds parameters to a registered SourceRef based on parameter values, current observation records, and the schema, while verifying consistency across value, locator, and channel. Refusing to trust self-reported sources is a sound design choice, yet parameter value matching may still yield non-unique results. This does not require the system to infer the model's internal reasoning process; rather, it requires specifying: what verifiable conditions are sufficient for a source to serve as authorized evidence for that parameter. Without this, the practical implications of channel authorization and witness independence remain insufficiently defined.
Tables E5 and H3 have tested erroneous channels, missing sources, and duplicate witnesses; however, the current PDF does not present the corresponding rules and results for cases where source ambiguity exists.
W5. A more specific, record-level explanation is needed regarding the relationship between "strict cross-channel success" and "incomplete-role conditions."
Section 3.3 requires participating channels to jointly provide the necessary roles for the target relationship, whereas B.1–B.2 define certain masks as lacking at least one core role. The paper explicitly states that these construction labels are not equivalent to authorization labels; thus, it is not inherently contradictory for incomplete-role conditions to yield authorized outcomes.
However, the specific tallying of "strict cross-channel success" requires further detailed explanation. Taking GPT-5.5 as an example:
- The combined count of target effects in Tables D1 and D9 is 37;
- The total number of target effects under the five complete-role conditions is 26;
- Yet, Table D10 shows 35 effects satisfying the strict cross-channel condition.
This implies that at least 9 instances of strict success originated from incomplete-role conditions. The authors need to present the values, the roles provided by each channel, and the actual relationships for these instances, explaining why the missing core roles did not prevent the coalition conditions from being met.
Questions for authors
-
For the 24 cases in Table J3 where CAF succeeded but the relationship-augmented baseline failed, could you provide representative, complete authorization predicates and the three types of backend encodings? Do these errors stem from limitations in expressive power, compilation omissions, or implementation discrepancies? Do the discrepancies persist after applying manual equivalence verification?
-
Please detail the actual state transitions (before and after) for the five cases in J7. Specifically regarding J7-005: given that the submission was rejected, what state change caused it to still be counted as a UER? Additionally, which experiments utilized the same final CAF configuration, and which relied on older compiler versions or different execution paths?
-
Regarding contract applicability: How does the current compiler generate contracts for authorization operations where the task text does not explicitly specify the final object ID or destination, requiring tool queries to resolve them? Among the reported AgentDojo successes, how many involve such dynamic binding, and how are the confirmations processed and factored into utility?
-
How does SourceRef make a selection when the same parameter value appears across different trust levels or channels? When multiple consistent sources exist, what rules prevent satisfying composite permissions by selectively choosing a favorable witness?
-
Could you provide at least nine records of strict success from GPT-5.5 involving "incomplete role" conditions, listing the actual relationship, , and ? This would directly clarify the correspondence between Table D10 and the definitions.
-
Did the held-out runs in Table G3 record task success rates, confirmation counts, and privilege reduction outcomes? If so, please report these by defense condition to determine whether the "zero unauthorized effects" result is accompanied by a significant loss in task completion performance.
Dear reviewer p3Rm,
We appreciate your valuable comments and have addressed your concerns one by one. Since the rebuttal deadline is approaching, we would be very grateful if you could kindly let us know where further improvements are needed.
Best regards,
The authors