acceptodds
Under review as a conference paper at ICLR 2027

AgentMarkArena: A Unified Platform for Evaluating Agent Behavior Watermarking

13 views

Abstract

LLM agents increasingly execute real-world tasks autonomously, making it important to identify which agent produced an observed sequence of actions. Their execution trajectories also contain reusable workflows, so attribution helps protect agent intellectual property. Behavioral watermarking addresses these needs by embedding keyed signals into agent decisions. However, comparison is difficult because watermarks alter tool choices and task outcomes, detectors require different evidence, and record edits differ from behavioral changes. We introduce AgentMarkArena, a unified and extensible platform for developing and evaluating agent behavioral watermarks under matched conditions. It combines 249 graded tasks across diverse domains and integrates methods into a shared agent runtime through common selection and verification interfaces. The platform distinguishes watermark carriers from verifier evidence, separating mechanism improvements from advantages of richer verification access. Its automated pipeline handles execution, grading, attacks, and aggregation, producing access-specific leaderboards covering detectability, task utility, robustness, behavioral distortion, and overhead. Across six watermark mechanisms and three agent backbones, we compare methods at a target 1% false-positive rate and stated verifier access, and evaluate robustness under post-execution record edits. With minimal verifier evidence, SeqWM reaches a 77.1% true-positive rate while retaining 92.0% utility. In contrast, TraceWM-T reaches 100% only with a trusted run identifier, and removing duplicate observations reduces it to 1%. Our results motivate future designs that exploit semantic redundancy, accommodate realistic verifier access, and explore equivalent argument choices as additional carriers. Our framework is available at https://anonymous.4open.science/r/AgentMarkArena-336D/.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Behavioral Watermarking, Agent Attribution, Trajectory Watermarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XPreOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AgentMarkArena, a unified evaluation framework for behavioral watermarking of LLM agents. The paper is motivated by the observation that existing watermarking methods are difficult to compare because they rely on different watermark carriers, verifier-side evidence, execution environments, and robustness assumptions. AgentMarkArena evaluates six watermark mechanisms on 249 graded tasks from three task suites using three agent backbones under a shared runtime. A particularly useful aspect of the framework is its explicit hierarchy of verifier access (A0–A4), which separates improvements due to the watermark mechanism from those enabled by richer verification evidence. The framework jointly measures detectability, utility, robustness to post-execution record edits, behavioral distortion, and computational overhead.

Critical strengths

The main strength is the explicit treatment of verifier access. I find the distinction between A0–A4 conceptually clean and practically useful: the experiments convincingly demonstrate that apparently stronger detection can result from stronger verifier assumptions rather than an intrinsically stronger watermark.

The paper also evaluates watermarks through actual agent execution rather than only manipulating offline trajectories. This is important because changing an action can change its arguments, future observations, task completion, and computational cost. Evaluating marked and unmarked agents under a common runtime and retaining the original task graders makes the utility evaluation substantially more informative than purely synthetic comparisons.

I also appreciate the multidimensional evaluation. In addition to TPR/FPR, the framework exposes utility loss, action substitution, token/call overhead, and robustness to several types of record editing. The results therefore reveal trade-offs that would be hidden by a detectability-only benchmark.

Finally, the paper is well organized and generally clear, and the appendix provides substantial implementation and configuration detail. The framework appears potentially useful to the agent-security and watermarking communities.

Critical weaknesses

My main concern is the fairness and interpretation of comparisons across heterogeneous methods. Some entries are not native implementations of the corresponding original detection setting. In particular, AgentWM is evaluated with a trajectory-only adapter rather than its native detector that queries a suspect model, while RedGreen is an action-level adaptation of a token watermark. The paper appropriately discloses these adaptations, but this makes it unclear to what extent the resulting leaderboard should be interpreted as comparing the original methods versus comparing AgentMarkArena-specific instantiations of them.

Relatedly, each method appears to be evaluated using one configuration. Watermark strength often controls an explicit detection–utility/distortion trade-off, so relative rankings can depend heavily on the selected operating point. I would find the benchmark substantially more convincing if several watermark strengths or detection–utility Pareto frontiers were reported.

A second concern is that A0, described as the lowest verifier-access level, still assumes access to the ordered candidate menu at each action. These menus are provided by the runtime rather than reconstructed from the surviving log. This is already fairly strong information in many provenance/forensics settings, where a verifier may only observe executed tool calls. The claim of strong performance under minimal evidence should therefore be interpreted relative to the proposed hierarchy rather than as log-only attribution.

Third, the robustness evaluation is limited to post-execution edits of exported records. This is a useful dimension, and the paper clearly states this restriction, but it does not test adversaries that alter or re-execute the underlying workflow. This distinction is especially important for the paper's motivation around protecting reusable agent workflows. Behavior-level transformations, semantic re-planning, or imitation by another agent may lead to different conclusions. Finally, the statistical comparison could be strengthened. The achieved FPR varies from approximately 0.8% to 3.0% despite the nominal 1% target, and uncertainty for the main detection rankings is not prominently reported. Because the contribution is specifically a fair evaluation platform, tighter operating-point matching or confidence intervals for TPR/rankings would be valuable.

Questions for authors
  1. How were the single configurations for each watermark chosen? Could the authors report multiple watermark strengths or detection–utility Pareto curves to show that the main ranking conclusions do not depend on hyperparameter selection?

  2. For methods that are adapted from their native setting, particularly AgentWM, can the authors quantify how closely the adapted evaluation tracks the native detector where both are applicable?

  3. How realistic is the assumption that the verifier has access to the ordered candidate menus used at A0? Would it be possible to add a log-only access level using only actually executed actions, or actions plus arguments?

  4. The current robustness attacks modify completed records but not agent behavior. Can the authors clarify the intended scope of the robustness claim and discuss behavior-level attacks such as semantic workflow rewriting or re-execution by another agent?

  5. Can the authors provide uncertainty estimates for the main TPR results and leaderboard positions, especially given that the achieved FPR differs across methods at the nominal 1% target?

4 comments
AuthorsOpenReview ↗

# Response to Reviewer XPre (1/2)

We thank the reviewer for recognizing AgentMarkArena's verifier-access hierarchy, real agent execution, and multidimensional evaluation. We address your constructive advice below.

W1 / Q1 — Configuration Sensitivity and Pareto Trade-offs


Varying the configuration supports our findings: new runs varying δ and the floor show the trade-off reported in the paper, with detection rising alongside substituted actions and cost.

Each method follows its published configuration, adapted to our 14-tool menu where the original setting differs, and every parameter is a configuration entry:

  • AgentGuide: γ=0.5, δ=2.0
  • RedGreen (TRACE's RG baseline): γ=0.5, δ=2.0
  • SeqWM: w=3, m=8, δ=2.0; floor 1/|B|
  • AgentWM: δ=2, θ_J=0.05, θ_N=3
  • AgentMark: 8-bit payload; TRACE has no strength parameter

SeqWM's original absolute floor of 0.2 is 2.8× uniform on our 14-tool menu, so we scale it as 1/|B|; in our sweep, 1/|B| already reaches a TPR of 1.00, and 2/|B| only adds substitution (63%) and tokens (5.1×).

We raised δ from 2 to 4 for AgentGuide and RedGreen, varied SeqWM's floor from 0 to 2/|B|, and added the floor to the other methods. Whether a parameter is effective depends on how concentrated the agent's decisions are (substitution and B=10 TPR):

SettingDecisions with entropy < 0.1Methodδ=4: subst.δ=4: TPR+ floor: subst.+ floor: TPR
Nemotron-3.5, DRBench76%AgentGuide11%0.3959%1.00
RedGreen15%0.9136%1.00
Nemotron-3.5, RoadmapBench92%AgentGuide3%0.4558%1.00
RedGreen3%0.1517%1.00
Qwen3.6, DRBench (first 30 tasks)74%AgentGuide12%0.8564%1.00
RedGreen14%0.8449%1.00
  • δ alone: on concentrated decisions a larger δ changes few actions (3% at δ=4 on Nemotron-3.5 RoadmapBench), because a near-zero probability stays near zero after biasing, and detection stays below 0.5.
  • The floor: on concentrated decisions only the floor brings every method to a TPR of 1.00, at up to 2.8× tokens and 0.46–0.69 utility retention on Nemotron-3.5 RoadmapBench. On less concentrated decisions (DRBench), a stronger δ shows the same trend at lower cost: with δ=4, RedGreen reaches 0.91 on Nemotron-3.5 and AgentGuide 0.85 on Qwen3.6, at 12–15% substitution. These TPRs are conformal; on Nemotron-3.5 DRBench, δ=4 gives a raw wrong-key TPR of at most 0.015 (Table R1, follow-up to Reviewer jkVr).
  • Keyed signal: raw wrong-key tests confirm a keyed signal for floored AgentGuide, and a weaker one for floored RedGreen.

New configurations or methods can be run on the platform and submitted with their execution logs to a public leaderboard, following open evaluation frameworks such as RouterArena (github.com/RouteWorks/RouterArena). We add these frontiers to Section 6.7 and Appendix F in the revised paper.

W1 / Q2 — Native Methods versus Unified Adaptations


The leaderboard compares AgentMarkArena instantiations under one trajectory-only audit protocol. RedGreen is an intentional baseline that applies a token watermark to actions with TRACE's RG parameters. For the methods with released code, we match their released code on identical inputs (AgentGuide 20,000, SeqWM 5,000, AgentMark 20,000 decisions). The platform then gives every method the same P_t, read from the agent, and executes every substituted action. Self-reported and read distributions can differ (total-variation distance 0.54–0.74 under SeqWM's prompt on 680 probed decisions; Section 6.6), so this shared basis keeps all methods on the same agent.

AgentWM. Our adapter applies its native statistic and parameters to recorded segments instead of probe responses, so it can be verified from records like the other methods. When the audited agent is the marked one, both detectors apply to the same trajectories, and they agree on 93–98% of single trajectories. They differ only in aggregation: the native rule pools segments before testing, which detects more where runs are short (WorkspaceBench, K=20: 0.50–0.54 vs. at most 0.03 on Qwen3.6 and Nemotron-3.5), while both reach 1.00 on RoadmapBench. The native rule has no null distribution, so the leaderboard applies one calibrated aggregation protocol to every method, so that rankings compare watermarks rather than aggregation rules.

AuthorsOpenReview ↗

# Response to Reviewer XPre (2/2)

W2 / Q3 — Practicality of A0 and Log-Only Verification


A0 matches real attribution settings. In the July 2026 Hugging Face intrusion, about 17,600 agent actions were recovered from logs, yet the model was unknown until its provider disclosed it (Hugging Face, 2026; Larcher et al., 2026; OpenAI, 2026). Public trajectory datasets such as NVIDIA's Open-SWE-Traces (huggingface.co/datasets/nvidia/Open-SWE-Traces) likewise store tool calls with the agent's tool definitions. A0 therefore requires only the executed action names plus the tool registry, as we state in the revised definition; the reviewer's "actions plus arguments" level is our A1.

Prior selection-based watermarks rely on the same assumption. SeqWM's deletion theorem (Thm. 4.1) states: "Assume the verifier can reconstruct the candidate action set used for each scored transition." TRACE lists among its limitations that "detection requires the verifier to access the candidate sets." AgentWM relies on the owner's equivalence sets mined from its tool library.

Required evidence and B=10 TPR:

  • Executed actions only (A0): SeqWM 0.771, TraceWM-S 0.334, AgentGuide 0.235, RedGreen 0.207
  • Executed actions or added arguments (A0/A1): AgentWM 0.117/0.262
  • Adds P_t (A3): AgentMark 0.371
  • Adds a trusted run identifier (A4): TraceWM-T 1.000

The platform also supports the stricter log-only level the reviewer suggests. We added A0⁻, where the verifier has only the executed actions and not the registry; this required only a new evidence view in the access configuration. Under conformal calibration, RedGreen and TraceWM-S keep their detection (B=10 TPR 0.304 and 0.343) because they draw keyed randomness per candidate, while SeqWM and AgentGuide, which draw it by enumerating the menu, drop to 0.022 and 0.078 (achieved FPR 1.3–2.6%); under the raw wrong-key test, SeqWM, AgentGuide, and RedGreen reach at most 0.08 (F3, follow-up to Reviewer jkVr). Exposing such evidence dependence is what the access hierarchy is for.

W3 / Q4 — Robustness Beyond Post-Execution Record Edits


Our robustness evaluation targets attributing a record after it has been edited, the setting in which per-run watermarks (SeqWM, AgentMark, TRACE) evaluate robustness. A keyless adversary may delete or truncate decision groups, collapse duplicate observations, rewrite calls into same-effect forms, or drop planning records. These edits leave the executed workflow and its outcome unchanged, so detection changes isolate the verifier. Re-execution by an independent agent removes any behavioral watermark by construction, since the mark lives in the marked agent's choices; TRACE notes that a reseller who "re-runs the task on a different model produces a genuinely unwatermarked trajectory."

A behavior-level attack, run end to end. Attacks are modular plug-ins graded with the same task graders, so we added a runtime Equiv as an example: the attacker controls tool execution and swaps each call that has a same-effect form with probability 0.5; the swapped call executes, the agent continues from its output, and the task is re-graded (Nemotron-3.5, first 50 DRBench tasks). Utility is unchanged (0.93–1.01 of unattacked) at 1.3–1.8× tokens, and detection drops no more than when the same rewrite is applied to the record. SeqWM keeps a B=10 TPR of 0.99 (0.72 on the record), since its windows follow the executed actions; TraceWM-S, whose selection channel is keyed on call content, drops to 0.00 under both; neither detector flags an attacked unmarked run.

Re-planning and imitation require re-execution or a second trained model and remain outside our evaluation. In the revised paper, we state this scope in the threat model and report the runtime attack in Appendix G.

W3 / Q5 — Detection Uncertainty and Leaderboard Reliability


At an identical realized FPR, SeqWM ranks first on every backbone, and bootstrap intervals confirm both ends of the ranking. The 0.8–3.0% spread in achieved FPR is mostly granularity: each held-out half holds 117–131 of the 248 unmarked runs per backbone, so one run moves achieved FPR by about 0.8% at B=1, and at B=10 a few extreme runs drive it. B=10 TPR at a realized FPR of 1%:

BackboneSeqWMTraceWM-SAgentGuideRedGreenAgentWM
Qwen3.61.0000.5230.5540.2930.268
Gemma-40.3280.1310.0490.2320.133
Nemotron-3.51.0000.4190.1400.2470.057

TraceWM-T reaches 1.000 at A4 on all three backbones; second place at A0 varies by backbone. A task-level bootstrap with recalibration (200 replicates per backbone) agrees: SeqWM is first in 100% of replicates (0.775 [0.747, 0.827]) and TraceWM-S second in 97% (0.332 [0.294, 0.448]), with AgentGuide and RedGreen within noise. SeqWM also stays first under additional keys and on newly generated unmarked runs.

Thank you for the detailed response and the additional experiments. The new sensitivity analysis, A0⁻ setting, runtime attack, and uncertainty estimates are helpful.

I have one remaining question: for RedGreen on Nemotron-3.5 DRBench at δ=4, the conformal TPR is 0.91, while the raw wrong-key TPR is at most 0.015. Could you briefly clarify how to interpret this difference and what it implies about the strength of the keyed watermark signal?

AuthorsOpenReview ↗

# Follow-up Response to Reviewer XPre

We thank the reviewer for the follow-up question and are glad the additional analyses were helpful.

The difference shows that RedGreen at δ = 4 produces a behavioral shift that is not key-specific, so the keyed signal is weak. The conformal TPR is the provider's test: the provider holds the key and its own unmarked runs, and checks whether runs depart from the unmarked agent at a common FPR. The raw wrong-key test isolates the key: it checks whether the true key scores higher than wrong keys.

AgentMarkArena reveals why: the P_t it reads from the agent shows that decisions are concentrated (Section 6.5), so biasing alone changes few actions and the key cannot be distinguished from wrong keys. The floor makes low-probability candidates available for the keyed bias to act on, which is why floored RedGreen gains a keyed signal and SeqWM scores on both tests. Reporting the wrong-key test alongside the leaderboard on the same execution is what makes this observable.

Reviewer_KquNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

AgentMarkArena is a unified evaluation platform for agent behavioral watermarking rather than a new watermarking algorithm. This paper curates 249 tasks across three domains, research, office tasks, and software development, to evaluate six watermarking mechanisms on three models under a standardized pipeline for execution environments, task scoring, attacks, and detection. Its key feature is decoupling where the watermark is embedded from what information the verifier can access, establishing five levels of verification access to systematically compare detection rate, task utility, robustness, behavioral shift, and computational overhead.

Critical strengths
  • Evaluating watermark costs during real task execution rather than solely through offline detection.

  • By separating access to actions, arguments, observations, action probabilities, and trusted run IDs, and distinguishing unverifiable from negative detection, the benchmark avoids unfairly favoring methods that assume stronger white-box assumptions.

  • Concrete failure modes offer much deeper insight than raw leaderboard rankings.

Critical weaknesses
  • Realized FPRs vary across methods (0.8%–3.0% in Table 2) despite a shared 1% target. This shows the comparability of TPRs and leaderboard rankings. Moreover, the 4,000 trajectory groups per fold in the appendix overlap and are not independent audit samples; uncertainty for TPR, FPR, and rankings should be estimated at the task or trajectory level.

  • The paper does not explain how multi-token tool names or missing candidate probabilities are handled when constructing PtP_t from token log-probabilities, affecting both distribution-preserving methods and the action-concentration analysis.

  • Attacks like deletion, truncation, and tool renaming only modify post-execution logs without re-running the agent or re-evaluating task utility. This tests sensitivity to log tampering, but neither verifies whether altered workflows still succeed nor covers replanning, imitation training, or runtime interventions.

Questions for authors
  • Does SeqWM's advantage rely on the probability floor? Could you compare SeqWM with the floor removed or lowered against other methods equipped with the same floor, and report detection rates under matched utility loss or token budgets?

  • Do the rankings hold under a strictly matched realized FPR? Could you provide ROC curves, TPRs at a uniform empirical FPR, and confidence intervals obtained by resampling at the raw task or trajectory level?

  • Have the action probability construction and method adaptations been independently validated? Specifically, how is PtP_t derived from token log-probabilities? Could you benchmark your unified implementation against the official implementations in their native testbeds to confirm that key mechanisms remain intact, and quantify the gap between AgentWM's native detector and your trace-adapted version?

4 comments
AuthorsOpenReview ↗

# Response to Reviewer KquN (1/2)

We thank the reviewer for recognizing the importance of real-execution evaluation and explicit verifier-access levels. We address your constructive advice below.

W1 / Q2 — Matched FPR and Statistical Reliability


The rankings hold under a matched realized FPR: from 0.5% to 5%, SeqWM and TraceWM-S stay first and second and AgentWM last, while AgentGuide and RedGreen remain within noise. The 0.8–3.0% spread in achieved FPR is largely granularity: with 117–131 held-out unmarked runs per half, one run moves achieved FPR by about 0.8% at B=1, and B=10 groups reuse the same runs. We set each threshold on held-out unmarked runs so that all methods share the same realized FPR, and resample tasks (200 bootstrap replicates per backbone, thresholds recomputed each time). A0, B=10:

MethodFPR 0.5%FPR 1%FPR 2%FPR 5%Bootstrap TPR95% CIRank (share of replicates)
SeqWM0.7600.7760.7930.8250.775[0.747, 0.827]1st (100%)
TraceWM-S0.3200.3570.4070.4870.332[0.294, 0.448]2nd (97%)
AgentGuide0.2100.2480.2840.3570.230[0.193, 0.326]3rd (56%)
RedGreen0.2000.2570.3230.4070.195[0.179, 0.310]3rd (39%)
AgentWM0.1370.1530.2090.2500.116[0.097, 0.281]5th (87%)
  • At B=1: SeqWM and TraceWM-S rank first and second; AgentGuide, RedGreen, and AgentWM stay below 0.06 at a realized FPR of 2% or less.
  • New negatives: on 496 newly generated unmarked runs on Nemotron-3.5 (two per task), the paper's thresholds give a realized FPR at B=10 of at most 0.6% for every A0 method except TraceWM-S (3.9%), which keeps its native analytical null.
  • Additional keys: SeqWM stays first under one or two additional keys per method.

In the revised Appendix I, we report the counts of runs, keys, and groups, describe this task-level procedure, and add ROC curves at matched realized FPR up to 20% for B=1 and B=10.

W2 / Q3 — Construction of P_t and Implementation Validation


P_t is read exactly from the model's log-probabilities, and it matches the agent's tool choices. The platform reads the log-probability tree where the tool name begins, under the agent's temperature and the engine's top-k/top-p truncation. A multi-token prefix is credited to the unique tool it begins, with ambiguous prefixes resolved one token at a time; candidates outside the truncated support get zero, and a decision whose executed tool gets no mass is left unmarked.

Validation. On 680 probed decisions, P_t is within a total-variation distance of 0.069 of the tool choices in 16 responses regenerated from the same prefix, while self-reported distributions, which several prior methods ask the model for, are 0.54–0.74 away under SeqWM's prompt. We add the procedure to Section 3 and the validation to Section 6.6 in the revised paper.

W3 — Scope of the Post-Execution Robustness Evaluation


Our robustness evaluation targets the record-level threat model of watermarks that attribute individual runs. SeqWM, AgentMark, and TRACE evaluate robustness by editing the logged trajectory. Our six edits model a keyless adversary editing an exported trajectory, with matched seeds on marked and unmarked records: deletion, truncation, Dedup, Equiv (same-effect tool forms), Canon (shell form), and No-op (dropping planning records). The task outcome stays fixed, so detection changes are attributable to the verifier.

A behavior-level attack, run end to end. Attacks are modular plug-ins graded with the same task graders, so we added a runtime Equiv as an example: the attacker controls tool execution and swaps each call that has a same-effect form with probability 0.5; the swapped call executes, the agent continues from its output, and the task is re-graded (Nemotron-3.5, first 50 DRBench tasks). Utility is unchanged (0.93–1.01 of unattacked) at 1.3–1.8× tokens, and detection drops no more than when the same rewrite is applied to the record. SeqWM keeps a B=10 TPR of 0.99 (0.72 on the record), since its windows follow the executed actions; TraceWM-S, whose selection channel is keyed on call content, drops to 0.00 under both; neither detector flags an attacked unmarked run.

Re-planning the workflow or imitating the marked agent requires re-execution or a second trained model and remains outside our evaluation. In the revised paper, we state this scope in the threat model and report the runtime attack in Appendix G.

AuthorsOpenReview ↗

# Response to Reviewer KquN (2/2)

Q1 — SeqWM Floor Ablation and Matched Watermark Strength


The floor ablation explains SeqWM's lead and supports the leaderboard. The floor is part of SeqWM's published design, so the leaderboard evaluates SeqWM as designed; the ablation identifies the floor as the source of its detection and, consistent with Section 6.5, of its cost.

  • Floor ablation (DRBench): removing the floor lowers SeqWM's B=10 TPR from 1.00 to 0.09 on Nemotron-3.5 and 0.11 on Qwen3.6; adding it to AgentGuide, TraceWM-S, and RedGreen raises them to 1.00 on Nemotron-3.5, and AgentGuide and RedGreen also on Qwen3.6.
  • Keyed signal: part of this gain may reflect behavioral shift under conformal calibration, but raw wrong-key tests confirm a keyed signal: on Nemotron-3.5 and Qwen3.6, 92–100% of floored AgentGuide runs reach p<0.01 (floor-only: 0–4%), and RedGreen a weaker 8–53% (0%).

With every substituted tool executed, detection can be set against utility and tokens (RoadmapBench; Qwen3.6: first 15 tasks):

ConfigurationBackboneSubstitutionB=10 TPRUtility [95% CI]Tokens
SeqWM, no floorNemotron-3.52%0.010.96 [0.61, 1.46]0.98×
Qwen3.67%0.001.08 [0.84, 1.51]1.10×
SeqWM, floor (paper)Nemotron-3.552%1.000.69 [0.41, 1.09]2.78×
Qwen3.660%1.000.47 [0.27, 0.65]1.74×
AgentGuide + floorNemotron-3.558%1.000.46 [0.24, 0.75]2.63×
Qwen3.664%1.000.61 [0.38, 0.82]2.40×
TraceWM-S + floorNemotron-3.530%1.000.46 [0.23, 0.76]1.76×
RedGreen + floorNemotron-3.517%1.000.59 [0.34, 0.91]1.32×
Qwen3.633%1.000.81 [0.49, 1.24]1.81×
AgentGuide, δ=4Nemotron-3.53%0.451.07 [0.75, 1.56]1.15×
RedGreen, δ=4Nemotron-3.53%0.150.81 [0.53, 1.21]0.99×

Utility is relative to a contemporaneous unmarked run (for Qwen3.6, the main unmarked run on the same engine configuration), with paired 95% intervals. On Nemotron-3.5 RoadmapBench, only the floor brings every method to a TPR of 1.00. On DRBench, where decisions are less concentrated, a stronger bias shows the same trend at lower cost: RedGreen with δ=4 reaches 0.91 at 15% substitution and 1.9× tokens, versus SeqWM at floor 0.5/|B| with 33% and 2.4×. These TPRs are conformal; RedGreen with δ=4 has a raw wrong-key TPR of 0.000 (Table R1, follow-up to Reviewer jkVr).

Q3 — Implementation Fidelity and the AgentWM Adaptation


On identical inputs, we match the released code of AgentGuide, SeqWM, and AgentMark. The released SeqWM and AgentMark code has the model self-report P_t as JSON and marks in simulation or replay, whereas our platform reads P_t and marks during real execution. We therefore fed identical inputs (P_t, menu, history, key) to both:

MethodCompared outputsAgreement
AgentGuideContext, green list, selected action20,000 / 20,000
SeqWMKeyed green subsets; watermarked distribution5,000 / 5,000
SeqWMTrajectory detection statistic2,000 / 2,000
AgentMarkEncoding and decoding20,000 / 20,000
AgentMarkRLNC bits and payload decoding2,000 / 2,000

The platform gives every method one P_t source, read from the agent. Self-reported and read distributions can differ (TV 0.54–0.74, above), so this keeps the comparison on the same agent. We checked TRACE and AgentWM against their papers.

AgentWM: adapter versus native detector. On the same K trajectories of the marked agent, the native rule and our adapter use the same statistic, agree on 93–98% of single trajectories, and differ only in aggregation: the native rule pools segments before testing. Where runs contain few segments, pooling detects more (WorkspaceBench, K=20: 0.50–0.54 vs. at most 0.03 on Qwen3.6 and Nemotron-3.5), while both reach 1.00 on RoadmapBench. Since the native rule is a fixed threshold without a null distribution, the leaderboard applies one calibrated protocol to every method, so that rankings compare watermarks rather than aggregation rules.

Extensibility. The floor and δ sweeps and the runtime attack above were added as configuration entries and a plug-in; new watermarks and attacks enter the arena the same way and are evaluated under the same execution, access levels, and metrics.

Thanks for addressing my concerns. I keep my score.

AuthorsOpenReview ↗

# Response to Reviewer KquN

We thank the reviewer for taking the time to read our response and are glad that the concerns have been addressed.

Reviewer_jkVrOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

AgentMarkArena integrates behavior watermarks into a shared agent runtime and evaluates actual execution on 249 graded tasks across research, workspace, and software domains. The platform distinguishes watermark carriers from verifier evidence, defines access levels A0-A4, and reports detection, task utility, robustness to record edits, action substitution, and computational overhead. It evaluates six mechanisms as seven entries on three backbones, targeting a nominal 1% false-positive rate and aggregating ten trajectories by default. SeqWM reaches 77.1% TPR with 92.0% mean utility retention at A0. TraceWM-T reaches 100% TPR with a trusted run identifier, but detection falls to around 1% after duplicate observations are removed. The platform provides modular interfaces for methods, tasks, and attacks, together with access-specific leaderboards.

Critical strengths
  1. Making verifier access part of the protocol is an important contribution. The same detection rate has different meaning when the verifier has only action records, access to action probabilities, or a trusted run identifier. Table 2 separates AgentMark's A3 requirement and TraceWM-T's A4 requirement, and marks unavailable evidence as N/A. This improves the interpretation of comparisons. TraceWM-T's strong clean detection and its collapse after deduplication also diagnose its dependence on observation counts, giving a useful direction for future methods.

  2. Actual task execution makes utility and cost measurements more meaningful. When a watermark changes a tool, the platform regenerates the arguments, executes the tool, and scores the result with the source benchmark's grader. Figures 8-9 and Appendix J.2 show the resulting costs and failures: strong detection can involve changing around half of the actions, using more tokens, or replacing a file-reading operation with a URL request that fails. These observations make the platform a valuable resource for studying the practical effects of behavioral watermarks.

Critical weaknesses
  1. A single configuration per method leaves important questions about the source and stability of the leading result. Table 4 gives SeqWM a probability floor of ϵ=1/∣Bt∣\epsilon=1/|B_t|. Its selection law first replaces Pt(b)P_t(b) with max⁡(Pt(b),ϵ)\max(P_t(b),\epsilon) and then applies the keyed bias. The paper discloses this floor and its effect on unlikely actions. Its scale matters: for a deterministic distribution over MM candidates, flooring alone changes the dominant action's probability to M/(2M−1)M/(2M-1), before any watermark bias. With 14 candidates, this is about 0.519, implying an action-change probability of about 0.481. Since 73%-87% of unmarked decisions have entropy below 0.1 nats, this is a plausible contributor to the observed substitution rate and deserves a direct ablation. SeqWM also uses about 1.9 times and 2.4 times as many tokens on two backbones. The reported utility and overhead make these trade-offs visible, but equal numbers of trajectories need not provide equal numbers of usable verification points. A floor-only control, a floor ablation, and a few matched substitution or token-budget operating points would make the method comparison more informative. The floor is part of the evaluated method; the concern is the strength of the conclusions drawn from this one operating point.

  2. The statistical precision and scope of the leaderboard need clearer limits. Table 2 reports achieved FPRs from 0.8% to 3.0%. The authors explicitly describe the target as nominal, which is appropriate. For a ranking presented at 1% FPR, however, held-out uncertainty and comparisons at a common operating rate would help. SeqWM's own achieved FPR is 1.3%, below several baselines, so the effect of recalibration on its lead remains an empirical question. Appendix I samples 4,000 aggregation groups that may overlap. This can estimate the grouped-null distribution of the available trajectory pool, but it does not create 4,000 independent newly generated negative trajectories. The underlying task, trajectory, and key counts should be reported, with uncertainty that reflects those sources of variation. Separately, Section 4 supplies candidate menus from the runtime and preserves original indices during attacks. These are explicit and defensible audit assumptions, but the resulting ranking applies to that setting. A sensitivity check using reindexed or less informative exported records would clarify how well it transfers to other attribution settings.

Questions for authors
  1. Intervention and evidence budgets. How do the detection, utility, and overhead trade-offs change with a floor-only control, removal of the floor from SeqWM, or comparable smoothing in other methods? Could you report a small number of operating points matched by action substitution, token cost, or usable verification decisions? These checks would help explain the results without requiring a full parameter sweep.

  2. Calibration and uncertainty. Please report the number of distinct unmarked tasks and trajectories, the number of keys, and the aggregation-group sampling procedure. Could you give uncertainty estimates by resampling the underlying data and recalibrating thresholds, and show whether the main ordering persists near a common held-out 1% FPR? Please distinguish variation from group sampling within a fixed pool from variation across independently generated trajectories.

  3. Available evidence and action probabilities. Who supplies the candidate menus and original indices at A0, and what establishes their integrity? Which methods remain verifiable if records are reindexed or only exported fields are available? Please also explain how Section 3 and Appendix B construct the complete PtP_t from token log-probabilities, including probabilities for tools that were not selected. This detail matters for both the entropy measurements and distribution-preserving methods.

I recognize the value of the platform, especially its access levels and actual utility evaluation. My main hesitation is whether the current single-configuration comparison supports sufficiently stable conclusions about the methods' trade-offs. A small set of targeted controls, clearer uncertainty, and a common-FPR comparison would substantially strengthen the case for acceptance.

2 comments
AuthorsOpenReview ↗

# Response to Reviewer jkVr (1/2)

We thank the reviewer for recognizing the value of AgentMarkArena as a platform. We address your constructive advice below.

W1 / Q1 — SeqWM Probability Floor and Matched Operating Points


The configuration sweeps support our findings and the leaderboard. The leaderboard evaluates each method at its published configuration, and the floor is part of SeqWM's own design. New runs that vary the floor and bias on two backbones and three task sets confirm the paper's finding that stronger detection comes with more substituted actions, more tokens, and lower utility (Sections 6.2 and 6.5), and they trace SeqWM's lead at published settings to its floor.

(a) Where SeqWM's detection comes from

The probability floor drives both SeqWM's detection and its overhead. The original SeqWM applies P^wm ∝ max(P, 0.2)·exp(γ·s/m) (§5.1, App. E.3), an absolute floor of 0.2, about 2.8× uniform on our 14-tool menu; we use the milder 1/|B|. On our unmarked decisions, the floor alone would change 50–53% of actions, matching SeqWM's substitution rate. Sweeping the floor on DRBench (B=10 TPR at nominal 1% FPR):

SeqWM configurationNemotron-3.5 subst.Nemotron-3.5 TPRQwen3.6 subst.Qwen3.6 TPR
Floor 0 (bias only)6%0.098%0.11
Floor 1/|B| (paper config)47%1.0053%1.00
Floor 2/|B|63%1.0069%1.00

Floor-only control. We ran the floor without the key (46% and 51% substitution). Its raw wrong-key test matches unmarked runs (at most 1% of runs reach p<0.01), so it carries no keyed signal. Conformal calibration still flags it at 0.40 on Nemotron-3.5 and 0.27 on Qwen3.6 (first 30 DRBench tasks): it tests departure from the unmarked agent's behavior, the null the leaderboard is defined on, and the floor alone changes that behavior. Attribution needs both tests, so in the revised paper we report the raw wrong-key test next to every ablation as a key-specificity check.

The same floor on other methods. AgentGuide, TraceWM-S, and RedGreen rise from a B=10 TPR of 0.09–0.78 to 1.00 on Nemotron-3.5, with the same pattern on WorkspaceBench and RoadmapBench; on Qwen3.6, AgentGuide and RedGreen also reach 1.00. Raw wrong-key tests confirm a keyed signal beyond the behavioral shift: on Nemotron-3.5 and Qwen3.6, 92–100% of floored AgentGuide runs reach p<0.01 (floor-only: 0–4%), and RedGreen a weaker 8–53% (0%).

(b) What the floor costs

Only real execution reveals the cost. On Nemotron-3.5, tokens grow to 2.8× at the paper's floor on RoadmapBench and 5.1× at floor 2/|B| on DRBench. On RoadmapBench, floored configurations retain 0.46–0.69 of unmarked utility on Nemotron-3.5 (floor 0: 0.96) and 0.47–0.81 on Qwen3.6 (floor 0: 1.08). Floored runs are also longer (median 22 → 33 scored decisions on Nemotron-3.5), but SeqWM's lead does not come only from length. Among the methods that score each decision rather than each distinct context, SeqWM has the highest B=1 TPR at each number of scored decisions per run (paper runs; TraceWM-S scores each keyed context once, so its counts are not comparable):

Method<1010–2020–4040–100≥100
SeqWM0.010.170.390.610.89
AgentGuide0.000.000.020.070.06
RedGreen0.000.000.000.010.01

(c) Operating points and an open arena

The platform makes these trade-offs explorable. On Nemotron-3.5 DRBench, RedGreen with δ=4 reaches 0.91 at 15% substitution, while SeqWM needs a floor of 0.5/|B| and 33% substitution to reach 1.00. The reported 77.1% remains the result for SeqWM with its floor, and the ablation shows what each component contributes and costs. We add the floor ablation to Section 6.7 and Appendix F in the revised paper.

An open arena for new watermarks. Every analysis in this response (the floor and δ sweeps here, and the A0⁻ evidence view and renumbering attack below) is a configuration entry or a plug-in. A new watermark implements the Watermark and Selector interfaces and is then evaluated under the same execution, verifier access, attacks, and metrics. The arena is built to accept such submissions with their execution logs, so future watermark research can be compared on equal footing.

AuthorsOpenReview ↗

# Response to Reviewer jkVr (2/2)

W2 / Q2 — FPR Calibration and Statistical Uncertainty


At a common realized FPR of 1%, the A0 ordering is unchanged apart from AgentGuide and RedGreen, which are within noise; task-level resampling and newly generated negatives confirm it.

  • Unmarked runs: 248 per backbone, one per task (DRBench 100, WorkspaceBench 99, RoadmapBench 49). WorkspaceBench task 42 is excluded: its grading prompt exceeds the LLM judge's context, so no run can be scored, and we keep each benchmark's grader unmodified. We state this in the revised Appendix A.
  • Split: a run-identifier hash gives two halves (131/117 for Qwen3.6, 125/123 for Gemma-4, 120/128 for Nemotron-3.5). Each half sets the threshold while the other reports achieved FPR, and the two folds are averaged.
  • Keys and groups: each method uses one key, with p-values against 1,000 wrong keys. For B=10, each fold draws 4,000 overlapping groups of 10 runs, which estimate the grouped null of this pool rather than independent negatives.

Why achieved FPR deviates from 1%. The B=1 threshold is the smallest calibration p-value, so one held-out run moves achieved FPR by about 0.8%; at B=10 the groups reuse the same runs, so a few extreme runs move it in similar steps. We therefore set each threshold on held-out runs for a realized FPR of exactly 1%, and resample tasks with recomputed thresholds (200 bootstrap replicates per backbone). A0, B=10:

MethodTable 2 TPRTable 2 FPRTPR at realized FPR 1%Bootstrap TPR95% CI
SeqWM0.7711.3%0.7760.775[0.747, 0.827]
TraceWM-S0.3342.2%0.3570.332[0.294, 0.448]
AgentGuide0.2350.8%0.2480.230[0.193, 0.326]
RedGreen0.2073.0%0.2570.195[0.179, 0.310]
AgentWM0.1172.2%0.1530.116[0.097, 0.281]
  • Bootstrap: SeqWM ranks first in 100% of replicates and TraceWM-S second in 97%; AgentGuide and RedGreen are within noise.
  • New negatives: on 496 newly generated unmarked runs on Nemotron-3.5 (two per task), the paper's thresholds give a realized FPR at B=10 of at most 0.6% for every A0 method except TraceWM-S (3.9%), which keeps its native Gamma null.
  • Reruns and keys: two SeqWM reruns on Nemotron-3.5 reproduce a B=10 TPR of 1.00, and SeqWM stays first under one or two additional keys per method.

W2 / Q3 — Verifier Evidence Availability and Integrity


The verifier takes the candidate menu from the agent's tool registry, not from the record. It holds the owner's key and the registry; the runtime also logs the menu, but verification does not read it, as we state in the revised Section 4. A0 thus needs only the executed action names, whose integrity our record edits perturb.

Original indices are not required. Under a renumbering attack (delete 20% of decision groups, renumber the rest consecutively), p-values are unchanged at every access level for SeqWM, TraceWM-S, AgentGuide, RedGreen, and AgentMark (0 of 75 sampled records each) and AgentWM (0 of 1,495). Only TraceWM-T changes (4 of 75): it keys observation-count targets on the run identifier and group position, the dependence its A4 requirement already reflects.

A stricter log-only level (A0⁻). Without the registry, RedGreen and TraceWM-S keep their detection (B=10 TPR 0.304 and 0.343), while SeqWM and AgentGuide, whose marks are defined over the candidate set, drop to 0.022 and 0.078.

Q3 (cont.) — Construction of Action Probability Distributions


P_t is read exactly from the model's log-probabilities. At the position where the tool name begins, the platform reads the log-probability tree under the agent's temperature and the engine's top-k/top-p truncation (from the request, else the model's generation config). This is the distribution the agent samples from. A multi-token prefix is credited to the unique tool it begins, with ambiguous prefixes resolved one token at a time; tools outside the truncated support get zero, and a decision whose executed tool gets no mass is left unmarked.

Validation. On 680 probed decisions, P_t is within a total-variation distance of 0.069 of the tool choices in 16 responses regenerated from the same prefix. Self-reported distributions, which several prior methods ask the model for, are 0.54–0.74 away under SeqWM's prompt. The entropy statistics in Section 6.5 therefore describe the agent's real decisions. We add the procedure to Section 3 and the validation to Section 6.6 in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.