- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AgentMarkArena, a unified evaluation framework for behavioral watermarking of LLM agents. The paper is motivated by the observation that existing watermarking methods are difficult to compare because they rely on different watermark carriers, verifier-side evidence, execution environments, and robustness assumptions. AgentMarkArena evaluates six watermark mechanisms on 249 graded tasks from three task suites using three agent backbones under a shared runtime. A particularly useful aspect of the framework is its explicit hierarchy of verifier access (A0–A4), which separates improvements due to the watermark mechanism from those enabled by richer verification evidence. The framework jointly measures detectability, utility, robustness to post-execution record edits, behavioral distortion, and computational overhead.
Critical strengths
The main strength is the explicit treatment of verifier access. I find the distinction between A0–A4 conceptually clean and practically useful: the experiments convincingly demonstrate that apparently stronger detection can result from stronger verifier assumptions rather than an intrinsically stronger watermark.
The paper also evaluates watermarks through actual agent execution rather than only manipulating offline trajectories. This is important because changing an action can change its arguments, future observations, task completion, and computational cost. Evaluating marked and unmarked agents under a common runtime and retaining the original task graders makes the utility evaluation substantially more informative than purely synthetic comparisons.
I also appreciate the multidimensional evaluation. In addition to TPR/FPR, the framework exposes utility loss, action substitution, token/call overhead, and robustness to several types of record editing. The results therefore reveal trade-offs that would be hidden by a detectability-only benchmark.
Finally, the paper is well organized and generally clear, and the appendix provides substantial implementation and configuration detail. The framework appears potentially useful to the agent-security and watermarking communities.
Critical weaknesses
My main concern is the fairness and interpretation of comparisons across heterogeneous methods. Some entries are not native implementations of the corresponding original detection setting. In particular, AgentWM is evaluated with a trajectory-only adapter rather than its native detector that queries a suspect model, while RedGreen is an action-level adaptation of a token watermark. The paper appropriately discloses these adaptations, but this makes it unclear to what extent the resulting leaderboard should be interpreted as comparing the original methods versus comparing AgentMarkArena-specific instantiations of them.
Relatedly, each method appears to be evaluated using one configuration. Watermark strength often controls an explicit detection–utility/distortion trade-off, so relative rankings can depend heavily on the selected operating point. I would find the benchmark substantially more convincing if several watermark strengths or detection–utility Pareto frontiers were reported.
A second concern is that A0, described as the lowest verifier-access level, still assumes access to the ordered candidate menu at each action. These menus are provided by the runtime rather than reconstructed from the surviving log. This is already fairly strong information in many provenance/forensics settings, where a verifier may only observe executed tool calls. The claim of strong performance under minimal evidence should therefore be interpreted relative to the proposed hierarchy rather than as log-only attribution.
Third, the robustness evaluation is limited to post-execution edits of exported records. This is a useful dimension, and the paper clearly states this restriction, but it does not test adversaries that alter or re-execute the underlying workflow. This distinction is especially important for the paper's motivation around protecting reusable agent workflows. Behavior-level transformations, semantic re-planning, or imitation by another agent may lead to different conclusions. Finally, the statistical comparison could be strengthened. The achieved FPR varies from approximately 0.8% to 3.0% despite the nominal 1% target, and uncertainty for the main detection rankings is not prominently reported. Because the contribution is specifically a fair evaluation platform, tighter operating-point matching or confidence intervals for TPR/rankings would be valuable.
Questions for authors
-
How were the single configurations for each watermark chosen? Could the authors report multiple watermark strengths or detection–utility Pareto curves to show that the main ranking conclusions do not depend on hyperparameter selection?
-
For methods that are adapted from their native setting, particularly AgentWM, can the authors quantify how closely the adapted evaluation tracks the native detector where both are applicable?
-
How realistic is the assumption that the verifier has access to the ordered candidate menus used at A0? Would it be possible to add a log-only access level using only actually executed actions, or actions plus arguments?
-
The current robustness attacks modify completed records but not agent behavior. Can the authors clarify the intended scope of the robustness claim and discuss behavior-level attacks such as semantic workflow rewriting or re-execution by another agent?
-
Can the authors provide uncertainty estimates for the main TPR results and leaderboard positions, especially given that the achieved FPR differs across methods at the nominal 1% target?
4 comments
# Response to Reviewer XPre (2/2)
W2 / Q3 — Practicality of A0 and Log-Only Verification
A0 matches real attribution settings. In the July 2026 Hugging Face intrusion, about 17,600 agent actions were recovered from logs, yet the model was unknown until its provider disclosed it (Hugging Face, 2026; Larcher et al., 2026; OpenAI, 2026). Public trajectory datasets such as NVIDIA's Open-SWE-Traces (huggingface.co/datasets/nvidia/Open-SWE-Traces) likewise store tool calls with the agent's tool definitions. A0 therefore requires only the executed action names plus the tool registry, as we state in the revised definition; the reviewer's "actions plus arguments" level is our A1.
Prior selection-based watermarks rely on the same assumption. SeqWM's deletion theorem (Thm. 4.1) states: "Assume the verifier can reconstruct the candidate action set used for each scored transition." TRACE lists among its limitations that "detection requires the verifier to access the candidate sets." AgentWM relies on the owner's equivalence sets mined from its tool library.
Required evidence and B=10 TPR:
- Executed actions only (A0): SeqWM 0.771, TraceWM-S 0.334, AgentGuide 0.235, RedGreen 0.207
- Executed actions or added arguments (A0/A1): AgentWM 0.117/0.262
- Adds P_t (A3): AgentMark 0.371
- Adds a trusted run identifier (A4): TraceWM-T 1.000
The platform also supports the stricter log-only level the reviewer suggests. We added A0⁻, where the verifier has only the executed actions and not the registry; this required only a new evidence view in the access configuration. Under conformal calibration, RedGreen and TraceWM-S keep their detection (B=10 TPR 0.304 and 0.343) because they draw keyed randomness per candidate, while SeqWM and AgentGuide, which draw it by enumerating the menu, drop to 0.022 and 0.078 (achieved FPR 1.3–2.6%); under the raw wrong-key test, SeqWM, AgentGuide, and RedGreen reach at most 0.08 (F3, follow-up to Reviewer jkVr). Exposing such evidence dependence is what the access hierarchy is for.
W3 / Q4 — Robustness Beyond Post-Execution Record Edits
Our robustness evaluation targets attributing a record after it has been edited, the setting in which per-run watermarks (SeqWM, AgentMark, TRACE) evaluate robustness. A keyless adversary may delete or truncate decision groups, collapse duplicate observations, rewrite calls into same-effect forms, or drop planning records. These edits leave the executed workflow and its outcome unchanged, so detection changes isolate the verifier. Re-execution by an independent agent removes any behavioral watermark by construction, since the mark lives in the marked agent's choices; TRACE notes that a reseller who "re-runs the task on a different model produces a genuinely unwatermarked trajectory."
A behavior-level attack, run end to end. Attacks are modular plug-ins graded with the same task graders, so we added a runtime Equiv as an example: the attacker controls tool execution and swaps each call that has a same-effect form with probability 0.5; the swapped call executes, the agent continues from its output, and the task is re-graded (Nemotron-3.5, first 50 DRBench tasks). Utility is unchanged (0.93–1.01 of unattacked) at 1.3–1.8× tokens, and detection drops no more than when the same rewrite is applied to the record. SeqWM keeps a B=10 TPR of 0.99 (0.72 on the record), since its windows follow the executed actions; TraceWM-S, whose selection channel is keyed on call content, drops to 0.00 under both; neither detector flags an attacked unmarked run.
Re-planning and imitation require re-execution or a second trained model and remain outside our evaluation. In the revised paper, we state this scope in the threat model and report the runtime attack in Appendix G.
W3 / Q5 — Detection Uncertainty and Leaderboard Reliability
At an identical realized FPR, SeqWM ranks first on every backbone, and bootstrap intervals confirm both ends of the ranking. The 0.8–3.0% spread in achieved FPR is mostly granularity: each held-out half holds 117–131 of the 248 unmarked runs per backbone, so one run moves achieved FPR by about 0.8% at B=1, and at B=10 a few extreme runs drive it. B=10 TPR at a realized FPR of 1%:
| Backbone | SeqWM | TraceWM-S | AgentGuide | RedGreen | AgentWM |
|---|---|---|---|---|---|
| Qwen3.6 | 1.000 | 0.523 | 0.554 | 0.293 | 0.268 |
| Gemma-4 | 0.328 | 0.131 | 0.049 | 0.232 | 0.133 |
| Nemotron-3.5 | 1.000 | 0.419 | 0.140 | 0.247 | 0.057 |
TraceWM-T reaches 1.000 at A4 on all three backbones; second place at A0 varies by backbone. A task-level bootstrap with recalibration (200 replicates per backbone) agrees: SeqWM is first in 100% of replicates (0.775 [0.747, 0.827]) and TraceWM-S second in 97% (0.332 [0.294, 0.448]), with AgentGuide and RedGreen within noise. SeqWM also stays first under additional keys and on newly generated unmarked runs.
Thank you for the detailed response and the additional experiments. The new sensitivity analysis, A0⁻ setting, runtime attack, and uncertainty estimates are helpful.
I have one remaining question: for RedGreen on Nemotron-3.5 DRBench at δ=4, the conformal TPR is 0.91, while the raw wrong-key TPR is at most 0.015. Could you briefly clarify how to interpret this difference and what it implies about the strength of the keyed watermark signal?
# Follow-up Response to Reviewer XPre
We thank the reviewer for the follow-up question and are glad the additional analyses were helpful.
The difference shows that RedGreen at δ = 4 produces a behavioral shift that is not key-specific, so the keyed signal is weak. The conformal TPR is the provider's test: the provider holds the key and its own unmarked runs, and checks whether runs depart from the unmarked agent at a common FPR. The raw wrong-key test isolates the key: it checks whether the true key scores higher than wrong keys.
AgentMarkArena reveals why: the P_t it reads from the agent shows that decisions are concentrated (Section 6.5), so biasing alone changes few actions and the key cannot be distinguished from wrong keys. The floor makes low-probability candidates available for the keyed bias to act on, which is why floored RedGreen gains a keyed signal and SeqWM scores on both tests. Reporting the wrong-key test alongside the leaderboard on the same execution is what makes this observable.
# Response to Reviewer XPre (1/2)
We thank the reviewer for recognizing AgentMarkArena's verifier-access hierarchy, real agent execution, and multidimensional evaluation. We address your constructive advice below.
W1 / Q1 — Configuration Sensitivity and Pareto Trade-offs
Varying the configuration supports our findings: new runs varying δ and the floor show the trade-off reported in the paper, with detection rising alongside substituted actions and cost.
Each method follows its published configuration, adapted to our 14-tool menu where the original setting differs, and every parameter is a configuration entry:
SeqWM's original absolute floor of 0.2 is 2.8× uniform on our 14-tool menu, so we scale it as 1/|B|; in our sweep, 1/|B| already reaches a TPR of 1.00, and 2/|B| only adds substitution (63%) and tokens (5.1×).
We raised δ from 2 to 4 for AgentGuide and RedGreen, varied SeqWM's floor from 0 to 2/|B|, and added the floor to the other methods. Whether a parameter is effective depends on how concentrated the agent's decisions are (substitution and B=10 TPR):
New configurations or methods can be run on the platform and submitted with their execution logs to a public leaderboard, following open evaluation frameworks such as RouterArena (github.com/RouteWorks/RouterArena). We add these frontiers to Section 6.7 and Appendix F in the revised paper.
W1 / Q2 — Native Methods versus Unified Adaptations
The leaderboard compares AgentMarkArena instantiations under one trajectory-only audit protocol. RedGreen is an intentional baseline that applies a token watermark to actions with TRACE's RG parameters. For the methods with released code, we match their released code on identical inputs (AgentGuide 20,000, SeqWM 5,000, AgentMark 20,000 decisions). The platform then gives every method the same P_t, read from the agent, and executes every substituted action. Self-reported and read distributions can differ (total-variation distance 0.54–0.74 under SeqWM's prompt on 680 probed decisions; Section 6.6), so this shared basis keeps all methods on the same agent.
AgentWM. Our adapter applies its native statistic and parameters to recorded segments instead of probe responses, so it can be verified from records like the other methods. When the audited agent is the marked one, both detectors apply to the same trajectories, and they agree on 93–98% of single trajectories. They differ only in aggregation: the native rule pools segments before testing, which detects more where runs are short (WorkspaceBench, K=20: 0.50–0.54 vs. at most 0.03 on Qwen3.6 and Nemotron-3.5), while both reach 1.00 on RoadmapBench. The native rule has no null distribution, so the leaderboard applies one calibrated aggregation protocol to every method, so that rankings compare watermarks rather than aggregation rules.