acceptodds
Under review as a conference paper at ICLR 2027

AutoMark: Enabling Autoresearch to Discover Better LLM Watermarks

14 views

Abstract

With LLM watermarking being deployed commercially and now required by regulations, improving its reliability and effectiveness has become crucial. Yet, recent progress in the field of LLM watermarking has increasingly been driven by improving details of existing methods, an effort fundamentally limited by the pace of human researchers. In this work, we enable for the first time the autonomous discovery of new distortion-free state-of-the-art watermarking schemes. To enable this, we (i) establish strict criteria to ensure that watermarks are reliable (e.g., they do not have an unexpectedly high false positive rate), (ii) propose rigorous statistical tests to automatically evaluate whether a watermarking scheme satisfies our criteria, and (iii) design an evaluation suite to rank watermarks along three key dimensions: detectability, quality, and robustness. By running our framework with 3 frontier models (GPT-6 Astra, Opus 5, Gemini-3.8 Flash), we discover over 50 different watermarking schemes, including several that outperform prior works along all key dimensions. We complement this by a manual study of the discovered schemes, distilling the key ideas into smaller components, and individually studying the impact of each component across dimensions (detectability, quality, robustness) to better understand how the proposed schemes operate. Importantly, we find that the agents, on top of improving existing ideas, also discover fundamentally new ideas (e.g., aligning watermark scores with random per-request direction). Overall, our work establishes the first steps of fully autonomous watermarking research, enabling the discovery of more reliable and effective watermarks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, watermarks, autoresearch

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Nj2zOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper presents AutoMark, an automated research harness for discovering LLM watermarking schemes.  Using said harness with frontier LLMs, the researchers generated 52 watermarking schemes, some of which beat existing watermarks on their benchmarks. They also analyzed schemes designed by their AutoMark agents to identify and categorize novelties and discoveries.

Critical strengths
  1. Formalizing what makes a watermarking scheme valid is interesting.
  2. The agents find novel watermarking schemes that improve on existing work.
Critical weaknesses
  1. The paper does not stand on its own without the very lengthy Appendix. In particular, Section 4 is incredibly dense and vague, making it hard to parse and difficult to assess the genuine improvements/novelty.
  2. You should add a reference to the part of the Appendix where you explain what a Lexical group is when you first mention it. Ideally, since it seems to be one of the core innovations of your agents, you should explain it in the main paper.
Questions for authors
  1. In Section 3.3, for Algorithm 3, you account for the allowed fraction of bad keys and the probability that a sound key screens positive. But in that case, shouldn’t you also account for the probability that a bad key screens negative?
  2. Does each agent only get what it has submitted, or does it have access to every submission by every agent? For example, does the GPT-6-ASTRA (xhigh) agent see what the OPUS-5 agent is doing?
  3. In Section 4.2, context seeding, you mention assigning less frequent tokens a lower kk. Do you mean preceding tokens, the candidate token, or both? Additionally, you argue that this method preserves the robustness of short contexts, but that is not necessarily true for any attacker aware that you are doing this, since they could focus their efforts on the later part of the text, which is more likely to have longer contexts. Likewise, assigning tokens to categories is also vulnerable to an adaptive adversary.
2 comments

We are happy to see that the reviewer found our formalization of valid watermarks interesting and that our agents find novel schemes improving on existing work. We address their remaining concerns below, and have updated our submission. All changes in the submission are colored in blue for clarity.

Q1: Can you elaborate on why you chose to structure the paper this way?

We agree that our work has a high information density and includes many novel ideas (52 new watermarking schemes organized in 39 components), and as such requires a lengthy appendix. We therefore made the explicit decision to tailor our main paper (and its evaluation) around 3 equally important takeaways:

  • Section 4.1 (dynamics of the autoresearch harness): show that autoresearch for watermarking is feasible, understand each model's capabilities, and guide future works applying autoresearch to other fields.
  • Section 4.2 (component ablations): highlight the components we believe are relevant for future works in watermarking. We believe this is the part the reviewer was most interested in.
  • Section 4.3 (held-out evaluation): propose to practitioners fully-fledged schemes that have been rigorously inspected by human researchers and evaluated on a held-out set.

Given the space constraints, we had to cut down the main-paper description of individual components and only keep pointers to the relevant appendix sections. We believe that clear communication is essential for impactful work, and would be happy to adjust our writing if the reviewer has concrete suggestions. Leveraging the additional page given for the rebuttal, we expanded Section 4.2 with detailed explanations of Private geometries and of the Clustering and Excess detectors, which are relevant novel components proposed in our work.

Q2: What are the main components proposed in this work?

In Section 4.2, we focus on the components we believe are the most interesting for future works: Lexical grouping, adaptive context seeding, the Gaussian copula, ResidualRace, and clustering-based detection. In particular, Gaussian copula and ResidualRace have already been adopted by industry, highlighting their practical relevance (we avoid naming the adopting scheme during the review process to preserve anonymity, but will cite it in our final revision). Thanks to the reviewer's comment, we now include more cross-references to ease navigation, including a pointer to the definition of lexical groups at their first mention.

Q3: Are the AutoMark runs of different models independent?

Yes, the runs of each model (GPT-6 Astra (xhigh), GPT-6 Astra (low), Opus 5 (high) and Gemini-3.8 Flash (high)) are independent: each agent only sees its own prior submissions (e.g. GPT-6 Astra (xhigh) only sees prior schemes submitted by GPT-6 Astra (xhigh)). We now clarify this point in Section 4.1.

Q4: When assigning less frequent tokens a lower context length kk, does the frequency refer to preceding tokens, the candidate token, or both?

We thank the reviewer for highlighting this ambiguity. In the frequency-based variant, kk depends on the immediately preceding token: if its frequency in a fixed reference corpus is below 0.1%0.1\%, we use k=1k=1, and otherwise k=2k=2. The candidate token does not determine kk in this variant. We now clarify this rule in Section 4.2.

Q5: Can an adaptive attacker exploit longer contexts at later positions, undermining the claim that the method preserves the robustness of short contexts?

We agree that an attacker aware of this mechanism could target positions with longer contexts to remove the watermark. Nonetheless, this remains more robust than prior works with a fixed long context, as positions with short contexts remain detectable (i.e. even if the signal at long-context positions is removed, the short-context positions still carry watermark signal).

Q6: Is context seeding based on token categories also vulnerable to an adaptive adversary?

We agree that an adaptive attacker aware of Lexical grouping could reduce its robustness gains. Nonetheless, it remains more robust than prior works without grouping against non-adaptive attackers (e.g. students who simply replace a few words in the hope of removing the watermark).

Q7: A genuinely bad key might escape detection. Shouldn't Algorithm 3 account for those missed bad keys?

We thank the reviewer for raising this point. Algorithm 3 uses an upper bound on the probability that a key screens positive: a sound key contributes at most rr, while a bad key contributes at most one, yielding β+(1−β)r\beta + (1-\beta)r. A bad key screening negative (i.e. a bad key that is not rejected) only decreases this probability, so no additional correction is needed for the validity of the rejection. However, such false negatives might reduce power (and accounting for them requires assumptions on how far a bad key's FPR exceeds α\alpha).

I thank the reviewers for addressing my concerns. I will be increasing my score accordingly.

Reviewer_fyMYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies autoresearch for LLM text watermarking. Instead of designing a new watermark by hand, it builds a framework that lets coding agents autonomously propose, verify, and evaluate watermarking schemes. Its core contribution is formalizing criteria for a valid watermark, especially the stricter requirement of controlled false positives under a given key, and using automated statistical tests to filter out invalid schemes. The authors let multiple frontier agents iteratively generate over 50 schemes, decompose and ablate their components, and finally select two schemes, ConcordMark with strong robustness and RotationFirst with high quality, which outperform or are comparable to existing baselines on held out data. Overall, it is a relatively interesting paper, but some issues need to be discussed.

Critical strengths
  1. This paper makes a concrete attempt to use agents to solve LLM watermarking. It moves watermark design from manual proposal and tuning toward an automated search process in which coding agents propose, verify, and evaluate new schemes.

  2. The paper formalizes validity criteria and builds automated statistical tests to screen agent proposals, especially the stricter soundness given a key requirement. This reduces the risk of agents exploiting weak evaluation and makes automated watermark discovery more trustworthy.

  3. The agent search produces over 50 schemes, and the component decomposition and ablation identify reusable new ideas. The selected schemes, ConcordMark and RotationFirst, show competitive robustness and quality on held out data, supporting the practical value of the agent based exploration.

Critical weaknesses
  1. The claim of autonomous discovery is weaker than the framing suggests. Agents propose schemes, but humans still decompose schemes into components, design ablations, and interpret which ideas matter. It remains unclear how much of the scientific insight comes from the agents and how much from post hoc human analysis.

  2. The search process may overfit the private evaluator. Agents can repeatedly submit, observe leaderboard feedback, and adjust their directions, so the selected schemes may partly reflect selection bias rather than genuinely generalizable watermarking advances. The held out evaluation uses only two selected schemes and a limited set of attacks, which does not fully rule out this concern.

  3. The verification harness, while carefully designed, may still be exploitable, and its coverage is limited. The paper covers only text watermarking, a small number of models and languages, and a narrow attack suite. Security, radioactivity, open weight settings, and multi bit watermarks are not addressed, so the generality of the autoresearch framework remains unproven.

Questions for authors
  1. Can the authors quantify the boundary between agent and human contributions? Please list which of the final five component families were first proposed by agents and which became clear only after human decomposition and ablation, and trace at least one component from agent proposal to final ablation conclusion. An additional baseline where agents directly output final schemes without human post processing would strengthen this point.

  2. How do the authors rule out overfitting to the private evaluator? Please report the held out performance of all 52 submissions, not just the two selected schemes, compute the correlation between private grader rank and held out rank, for example Spearman correlation, and compare the held out gap between the top1 and top5 submissions.

  3. Is the verification harness still reliable against adversarial agents? Please design stress tests that explicitly encourage agents to bypass the three statistical checks and report whether any false passes occur. Also explain whether the sample sizes, significance levels, and multiple testing corrections still control the overall false positive rate when agents submit adaptively and repeatedly.

  4. Which parts of the framework need to change when extending to open weight models or multi bit watermarks? Please provide at least one small scale extension experiment showing how the detector threshold, sound given key correction, context seeding, and evaluation protocol would need to be adjusted, and how much the computational cost would increase.

2 comments

We thank the reviewer for their positive review and detailed suggestions. We address their remaining concerns below, and have updated our submission. All changes in the submission are colored in blue for clarity.

Q1: Do you evaluate schemes proposed by the agents directly, without human intervention?

We believe there is a slight misunderstanding. Both RotationFirst and ConcordMark are entirely generated by the agents, and we evaluate them in Section 4.3. More generally, all components evaluated in our work were generated by our agents without any human intervention. Human intervention is entirely post hoc and restricted to classifying and structuring the schemes proposed by the agents.

Q2: More broadly, which parts of watermarking research still require human contributions?

Our goal is to show that research agents can autonomously discover new watermarking schemes. Thus, researchers should focus more on evaluation (i.e. what we want) than on designing schemes (i.e. how to achieve it). Yet, we find in Appendix B that, with an incorrect harness, agents can overoptimize certain metrics. Thus, we do not recommend deploying autoresearch schemes in production without inspecting them.

The agents only produced .py files implementing fully-fledged schemes; no agent proposed "components" directly. We then manually inspected the code, split it into atomic components, ablated them, and studied them. We now clarify this scope in Section 4.2.

Q3: Do the agents overfit the private evaluator because of repeated submissions?

Agents could in theory use leaderboard feedback to "guess" the private evaluation and overfit it. In practice, inspecting the traces and schemes, we find that this is not the case. Also, agents run for few iterations (~15) and the leaderboard signal is sparse, making overfitting hard.

We nonetheless ran the suggested experiment (new Appendix E): we evaluate all 52 schemes with a new model, Qwen3.8-27B, on English WildChat prompts (a new held-out dataset), on clean text and our four attacks (word deletion, synonym substitution, back-translation, paraphrasing), with GPT-6 Luna replacing the private GPT-5.4-mini. Held-out and private results agree: Spearman's ρ\rho is 0.840.84 for quality, 0.810.81 for clean detectability, and 0.740.74 for robustness. The top-5 schemes are also similar.

Q4: Are the statistical tests used in verification still valid for repeated submissions?

Yes: multiple testing can only increase the rate of false rejections (i.e. rejections of valid schemes), so repeated submissions do not increase the rate of invalid schemes being accepted.

Q5: Is the verification harness reliable against adversarial agents?

We inspect all schemes produced by our agents to ensure that they are valid. The verification harness guides the autoresearch loop towards valid schemes but does not replace human understanding of the generated schemes. We therefore argue that robustness against adversarial agents is out of scope for our work.

Q6: How do we ensure that the schemes proposed by the agents generalize well?

See our answer to Q4 of Reviewer 3ZAv.

Q7: Is focusing on text watermarking detectability too narrow? Can the autoresearch harness be used in other watermarking fields (e.g. multibit or radioactivity)?

We respectfully disagree that focusing on detectability is narrow: many influential works only study zero-bit watermark detection, which deployed schemes also target [1-3].

Nonetheless, our harness generalizes to other properties by adjusting the metrics and the verification: we ran a small-scale multibit extension (new Appendix F). The scheme embeds a message m∈0,1Lm\in\\{0,1\\}^L (L∈8,16,32L\in\\{8,16,32\\}), and a decoder, given the text, key and LL, returns m^\hat{m} and a detection p-value. We keep the p-value metrics of our main experiment and add the message accuracy (MA) of m^\hat{m}. We use MPAC [4] and two binomial schemes [5] as baselines.

In 5 submissions, GPT-6 Astra raised clean MA from 74% to 94% (16 bits) and from 26% to 71% (32 bits). On held-out data (English/Chinese), it outperforms all baselines in robustness (8-bit MA after 50% deletion: 24/37% vs. ≤4%) and 32-bit MA (47/53% vs. ≤42/47%).

[1] Dathathri, Sumanth, et al. "Scalable Watermarking for Identifying Large Language Model Outputs." Nature (2024).
[2] Li, Xiang, et al. "textGrain: Entropy-Calibrated Watermarking for Language Model Text." OpenAI technical report (2026).
[3] Sander, Tom, et al. "TextSeal: A Localized LLM Watermark for Provenance & Distillation Protection." arXiv:2605.12456 (2026).
[4] Yoo, KiYoon, et al. "Advancing Beyond Identification: Multi-bit Watermark for Large Language Models." NAACL (2024).
[5] Gloaguen, Thibaud, et al. "Every Bit, Everywhere, All At Once: A Binomial Multibit LLM Watermark." NeurIPS (2026).

Reviewer_fyMYOpenReview ↗

Interesting work

This is an interesting work on LLM watermarking. I thank the authors for resolving my questions.

Reviewer_3ZAvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the AI research agent's independent design, verification and improvement of large language model watermarking algorithm. An automation research framework AutoMark is proposed, which combines coding proxy, modular watermark database, automatic statistical verification and private evaluation pipeline to propose and improve the watermark algorithm iteratively. The paper uses multiple research agent configurations, including GPT-6 Astra, Opus 5, and Gemini-3.8 Flash, and accumulatively evaluates 52 watermarking schemes.

Critical strengths

This paper proposes an automated research framework capable of algorithm design, verification, and optimization, which is very interesting.

The authors address shortcomings in the statistical validity of traditional watermark detection when applied in practice.

Critical weaknesses

Although the paper distinguishes the formally proven scheme from the empirically proven scheme, it will be more helpful to distinguish the theoretical guarantee from the limited sample screening results in terms of the broader proposition of finding reliable and credible watermarks. The author should describe the statistical capabilities and limitations of validation testing, including which categories of validity violations can not be found.

For an automated watermarking research platform intended for eventual deployment, an evaluation that omits active-attack security risks enabling agents to identify methods that excel in the present test setting yet remain vulnerable to targeted compromise.

Questions for authors

Q1: I am curious about what types of invalid watermarks the automated verifier can actually detect. Also, which algorithms that violate soundness might still pass verification?

Q2: How do the authors distinguish genuine generalizable improvements from adaptive optimization that targets the evaluator?

1 comment

We thank the reviewer for their positive review and are happy that they found our automated research framework interesting. We address their remaining concerns below, and have updated our submission. All changes in the submission are colored in blue for clarity.

Q1: Can the authors give a scheme flagged as invalid by the verification?

Certainly! We kindly refer the reviewer to Appendix B, where we list schemes generated by our agents that failed our verification:

  • ConcordMark-v0 combines a Gumbel-race watermark detector with an n-gram classifier trained to distinguish watermarked outputs from human texts. The classifier also flags unwatermarked LLM replies (up to 31% at a nominal 1% FPR). Because the classifier is independent of the key, any text it flags is flagged under every key, violating soundness over the key.
  • MeridianMark fits a null distribution using decoy keys, then artificially shrinks small p-values using a threshold tuned to the public human texts. This calibration fits the training distribution and does not generalize to our private test distribution.

In Appendix C, we show that some of our baselines also fail the soundness given a key test. Additionally, for other schemes from the literature:

  • Unigram [1] (i.e. Red-Green watermarks [2] with a context size of 0) without de-duplication fails both soundness over and given a key. This is because repeated occurrences of a token share the same randomness and are no longer independent.
  • Red-Green watermarks with a Z-score test: the null distribution is binomial and not normal [3]. Our tests can detect this approximation error, which leads to a slightly incorrect FPR.

Q2: Can the authors provide an invalid scheme that passes the verification?

It is possible to construct a scheme designed to trick our tests, though we did not observe such behavior in practice. For instance, one limitation of our tests is that we only use a finite set of α\alpha, whereas the tested properties (Eq. 2 and 3) should hold for all α\alpha. Concretely, take the TextSeal watermark [4] and its detector, which returns the p-value Pξ(ω)P_\xi(\omega) of a text ω\omega. Now replace it with Pξ′(ω)=min⁡Pξ(ω),0.1.P'_\xi(\omega)=\min\\{P_\xi(\omega),0.1\\}. Because we only test α∈0.001,0.01,0.05\alpha \in \\{0.001,0.01,0.05\\}, this scheme passes our tests. Yet, it is neither sound over the key nor given a key. We now include this example in Appendix H to clarify the limitations of our tests. We highlight that, in practice, agents are unaware of the private verification parameters, and are thus unlikely to cheat in this manner.

Q3: More generally, what are the limitations of the proposed validation testing?

Our tests are statistical tests: they can only detect violations of the tested property with a predetermined confidence level. Their goal is to guide the autoresearch loop towards interesting schemes, which, we believe, our experimental results and industry adoption [5] show is achieved. Thanks to the reviewer's comments, we now expand on their limitations in Section 3.3.

Q4: How do we ensure that the schemes proposed by the agents generalize well?

We believe there is a slight misunderstanding. Our goal is to show that researchers should focus more on evaluation (i.e. what we want) than on exploring a large set of scheme variations (i.e. how to achieve it), as agents have become incredibly effective at the latter.
Yet, we find in Appendix B that, with an incorrect harness, agents can be prone to overoptimizing certain metrics. Thus, we do not recommend directly deploying autoresearch schemes in production without inspecting what has been generated.

Instead, we advocate, as we did in this work, for manual investigation of the generated schemes (i.e. understanding how they work) and additional held-out evaluation (e.g. Section 4.3 for the selected schemes, and now Appendix E for all 52 schemes).

In practice, among the components we highlight in our work, Gaussian copula and ResidualRace have already been adopted by industry [5], which suggests that the schemes proposed by our agents indeed generalize well and are suitable for large-scale watermarking deployment.

[1] Zhao, Xuandong, et al. "Provable Robust Watermarking for AI-Generated Text." International Conference on Learning Representations (2024).
[2] Kirchenbauer, John, et al. "A Watermark for Large Language Models." International Conference on Machine Learning (2023).
[3] Fernandez, Pierre, et al. "Three Bricks to Consolidate Watermarks for Large Language Models." IEEE International Workshop on Information Forensics and Security (2023).
[4] Sander, Tom, et al. "TextSeal: A Localized LLM Watermark for Provenance & Distillation Protection." arXiv:2605.12456 (2026).
[5] We intentionally avoid naming the adopting scheme during the review process to avoid de-anonymization, but are happy to include the relevant citation in our final revision.

Comments on the paper

1 comment
Area_Chair_6qVuOpenReview ↗

Please, start a discussion

Dear reviewers,

The discussion period is about to end and the authors have responded to your reviews. This is a friendly reminder to check these responses and respond if necessary.

@ Reviewer fyMY: thanks for acknowledging the rebuttal.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.