- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Semantic Behavioral Watermarking (SBW), which maps agent actions into semantic clusters and embeds behavioral watermarks using history-conditioned encoding to improve robustness against tool renaming and semantic rewriting. In addition, the authors employ keyed random projection and binning to mitigate watermark forgery attacks and provide theoretical analyses of watermark stability and capacity. Experiments on ToolBench and ALFWorld demonstrate the effectiveness of the proposed approach.
Critical strengths
-
The paper investigates semantic rewriting robustness and forgery resistance in agent behavioral watermarking, addressing research problems of potential significance.
-
The proposed method requires no additional model training, making it relatively straightforward to implement.
-
The experiments cover multiple models, semantic encoders, and datasets, with additional analyses of the trade-offs among watermark capacity, robustness, and forgery resistance.
Critical weaknesses
-
Limited realism of the threat model. The proposed method assumes that the verifier can access the agent's execution trajectories. However, commercial agent APIs typically only need to return final results. An adversary could simply hide internal tool calls and execution traces, making behavioral watermark detection impossible. The paper does not adequately justify the practical scenarios in which this assumption holds.
-
Insufficient justification for behavioral watermarking. If the objective is to verify agent ownership, why not embed the watermark directly into the final output? Compared with output-level watermarking, behavioral trajectory watermarking introduces additional complexity and potential performance degradation, yet the paper lacks sufficient comparisons or justification for this design choice.
-
Limited methodological novelty. Semantic clustering-based watermarking, history-conditioned encoding, and keyed random mapping have all been explored in prior research. The proposed method primarily combines and adapts these existing techniques to the agent setting, without demonstrating substantial new methodological insights.
Questions for authors
-
In practical commercial agent API scenarios, why is it reasonable to assume that the verifier can access internal execution trajectories? How would SBW function if an adversary only returned the final execution results?
-
What are the irreplaceable advantages of behavioral trajectory watermarking over directly embedding watermarks into the agent's final outputs? Could the authors provide comparative experiments or concrete application scenarios to justify this choice?
2 comments
Response to Reviewer NEec — Part 2/2
3. What is technically new beyond combining existing components?
We agree that semantic clustering, keyed pseudorandom functions, and history-conditioned watermarking are not individually new, and we do not rely on novelty of these primitives. The contribution is the construction and characterization of a semantic action-level watermark for multi-step agent trajectories, including the limitations that emerge in this setting.
First, Section 4.1 changes the watermark symbol and its conditioning history from exact action strings to semantic-cluster symbols. This addresses a specific synchronization failure under tool renaming: both the current symbol and the history used for verification can change under a semantics-preserving rewrite. Section 6.1, Table 2 shows paraphrase detection of 0.49–0.66 for CLUSTER versus 0.05–0.17 for the exact-symbol baseline on ToolBench across five agent models; Section 6.4, Table 5 shows 0.92–0.97 versus 0.00–0.01 on ALFWorld. These results are accompanied by measurable costs: clean detection and deletion robustness can be lower than the exact-symbol baseline (Table 2).
Second, Section 5 studies two constraints of this agent setting: synchronization stability under semantic displacement and watermark capacity under the elicited action distribution. The latter matters because watermarking cannot assume a freely selectable output token at each agent step. Section 6.1 reports that clustering at k=100 retains only 41.9–50.1% of tool-level capacity across the five models; this is a limitation, not an unqualified improvement.
Third, Sections 4.2 and 6.2 examine a forgery surface arising when the semantic representation is public. Keyed binning reduces the evaluated adaptive fresh-action forgery success, while the paper also reports attacks that remain effective: Table 3 gives chained-replay success of 0.763–0.980 across models. The fresh-bucket guarantee is explicitly narrower than general resistance to all forms of stealing or replay. Moreover, the robustness and forgery results use different operating configurations; we do not claim that a single high-dimensional keyed configuration has been jointly validated for both properties.
We therefore view the contribution as an agent-specific design, quantitative analysis of its robustness–capacity trade-offs, and a transparent examination of both successful and unsuccessful forgery defenses—not as the invention of clustering or PRFs.
We appreciate the reviewer's recognition of the problem's significance and the empirical breadth of the study. We agree that trajectory visibility limits where SBW can be deployed, and that output-level watermarking is preferable when only final responses are available. Within the explicitly stated trajectory-observable setting, however, we hope the reviewer will consider the demonstrated semantic-rewriting robustness, the analysis of agent-action capacity and synchronization, and the measured forgery boundaries as substantive contributions.
Response to Reviewer NEec — Part 1/2
We thank the reviewer for the careful assessment and for recognizing the importance of studying both semantic rewriting and forgery, the training-free design, and the evaluation across models, encoders, and benchmarks. We appreciate the concerns about deployment assumptions, the motivation for behavioral watermarking, and methodological novelty. We clarify the scope and evidence already provided in the submission below.
1. When are execution trajectories available?
We agree that SBW cannot verify an agent from its final answer alone. Our threat model in Section 3 explicitly assumes that the verifier observes an action trajectory. This is an access requirement, not a claim that all commercial agent APIs expose their internal traces.
The intended use cases are settings where tool-call or workflow traces are retained and made available to an authorized verifier, for example: (i) enterprise deployments with audit logs; (ii) agent platforms where tool invocations are logged for monitoring or compliance; and (iii) shared or transferred agent workflows where the trajectory itself is the object whose provenance is checked. These are examples of deployment settings, not settings we claim to have evaluated end-to-end.
If a third-party service returns only a final response and neither the verifier nor a trusted logging service can access the underlying actions, SBW is not applicable. We agree that this limits deployment coverage. Our claim concerns provenance of observable agent behavior, not ownership verification for arbitrary black-box APIs.
2. Why watermark behavior rather than only the final output?
The two approaches verify different artifacts. An output watermark can provide evidence about a marked final response; it does not, by itself, identify the sequence of tool choices and intermediate actions that produced that response. Different agents can produce similar final answers while following different tool-use trajectories, and the same agent may express a final answer in many ways. For applications where the execution trace is retained and its provenance matters, behavioral watermarking targets information not represented by the final answer alone.
We do not argue that behavioral watermarking universally replaces output watermarking. They are complementary: output watermarking is more appropriate when only the final text is observable; behavioral watermarking is relevant when the action sequence is available and is itself the provenance target. This distinction is also why the introduction (Section 1) motivates ownership signals in discrete agent behavior rather than surface tokens.
We acknowledge that the current paper does not provide a matched experimental comparison with final-output watermarking. Such a comparison would address a broader deployment question, but it would not make the two methods interchangeable: they mark different objects and require different observations. The experiments in this submission establish robustness properties of the trajectory-level signal, not superiority over output-level methods.