- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how purchasing agents can resist misleading seller messages while still responding to real improvements in a deal. It proposes a three-way framework for deciding which types of manipulation can be blocked using the available information. The method removes irrelevant information, evaluates uncertain contract terms using their worst possible outcomes, and uses conservative estimates for unverified item quality.
Critical strengths
The paper addresses a practical problem and provides a useful framework for organizing manipulation risks. Its paired tests keep the underlying transaction fixed while changing its presentation, helping isolate the effects of manipulation. Tests with genuine concessions also check whether the defense remains responsive to better offers. The experiments cover bargaining datasets and AgentDojo.
Critical weaknesses
The existing propositions separately address invariance, monotonicity under set inclusion, and non-identifiability. However, it is unclear how they jointly establish a complete three-way classification. The authors should clarify this connection, since the classification is the paper’s main claimed theoretical contribution.
The AgentDojo experiments lack a comparison with existing structural defenses such as CaMeL. Including such baselines under the same attack assumptions and information access would help assess the proposed method’s benefits for safety and task performance.
Questions for authors
See the Critical Weaknesses.
3 comments
Thank you for the substantial effort during the rebuttal. The rebuttal has addressed most of my concerns, especially through additional experiments and clarifications. The proposed criterion is valuable, but the demonstrated protection remains limited to well-specified channels under particular information assumptions. After considering the rebuttal together with the other reviewers’ comments, I maintain my original score. The initial submission also has room for improvement in presentation. I believe this work has potential and encourage the authors to further develop it. Thank you again.
We thank the reviewer for the careful reading and for acknowledging that the rebuttal addressed most of the concerns.
We agree with the remaining point, and would add one observation about how we read it. Exactness is always relative to a specified set of admissible actions; that is precisely what makes it checkable in advance rather than after a failure. The alternative is a defense that appears unrestricted, and our added probe measures what that buys: a model that refuses every injection-shaped payload at 0.000 still transfers to the attacker in 0.567 [0.392, 0.726] of episodes once the same account is presented as a correction to the document’s own payee, while the structural operator refuses that transfer under all six framings because it never reads the framing. We therefore regard the specification requirement as a property of the problem that the criterion makes explicit, rather than a restriction peculiar to this method.
On presentation, the revision states the scope condition in the abstract and reorganises the introduction around this distinction, with the supporting competence analysis moved out of the main text.
We thank the reviewer again for the encouragement.
We revised Section 4.1 and the proofs to clarify how the three propositions relate to the classification. We also added a direct CaMeL comparison.
1. How do the propositions support the three-way classification?
We added Corollary 1 and clarified its scope.
For a specified channel and observable representation, let Δk(m) be the admissible displacement set and let g be the local utility gradient. At each observation, exactly one of three conditions holds:
(a) Every displacement has zero inner product with g.
(b) All inner products are nonpositive, with at least one negative.
(c) At least one inner product is positive.
These conditions form an exhaustive partition of local directional signs. They do not, by themselves, prove that every channel admits one of three defense implementations.
Each operational guarantee requires further assumptions. Proposition 1 concerns exact representation invariance across arbitrary downstream decision functions. Proposition 2 gives one-sided closure under valuation dominance. Proposition 3 establishes an identification boundary when indistinguishable latent states require different decisions.
Case (a) requires invariance over admissible finite changes. Case (b) requires finite completion sets and valuation dominance. Case (c) gives an impossibility result only when latent states share an observation but require conflicting decisions.
For mixed channels, guarantees apply only to components satisfying the relevant assumptions. Assigning an inseparable mixed channel to the residual regime is a conservative convention. It does not follow from Corollary 1.
We made these conditions explicit in Section 4.1 and Appendix A.2.
We also corrected Proposition 2.
The submitted version used ordinary completion-set inclusion. This is insufficient when a manipulation also changes the valuation input.
The revised proposition uses valuation dominance. Every clean completion must have a manipulated completion that is no better for the buyer.
For nonempty finite completion sets, dominance is equivalent to the conservative value not increasing. It prevents rejection-to-acceptance flips for every threshold. If dominance fails, some threshold produces a flip, although the deployed threshold need not.
Ordinary set inclusion is a sufficient special case when shared completions retain their valuation.
We checked dominance by enumerating clean completions and finding corresponding manipulated completions under the specified buyer valuation. This check uses the rendered signals and the buyer's preference parameters. It does not use the hidden test oracle.
Both contract manipulation families satisfy dominance on all 2,605 held-out tasks. Ordinary set inclusion holds on only 486 tasks because the manipulations can also change the headline price.
The revised proof is in Appendix A.1. The experimental verification is reported in Appendix K.
The guarantee is one-sided. It is independent of the learned buyer model only when the deployed decision follows the specified conservative threshold rule. It is not the arbitrary-function invariance of Proposition 1.
2. Comparison with CaMeL.
We ran the official CaMeL implementation in the same AgentDojo benchmark. The compared arms use matched tasks, attacks, model snapshots, and evaluation verdicts (Appendix Q, Table 17).
On closable banking goals, canonicalization and CaMeL both achieve zero measured ASR. Their task utility is 0.414 and 0.422, respectively.
On closable travel goals, both also achieve zero measured ASR. Task utility is 0.475 for canonicalization and 0.025 for CaMeL under gpt-4o-mini. A gpt-4o replication gives 0.600 and 0.250, respectively, with zero measured ASR for both methods.
Slack is different because its security verdict uses attempted calls. Canonicalization blocks the guarded operation but retains a measured ASR of 0.333.
CaMeL also blocks goals outside our selected operator's scope. Its architecture uses additional trusted control information and a restricted execution environment. Some runs fail to complete the legitimate task. Zero measured ASR therefore does not always imply successful task execution.
The comparison shows different coverage and task-utility costs. It does not establish that either method is universally preferable.
We have clarified these assumptions and limits in the revised manuscript.