- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper presents AutoMark, an automated research harness for discovering LLM watermarking schemes. Using said harness with frontier LLMs, the researchers generated 52 watermarking schemes, some of which beat existing watermarks on their benchmarks. They also analyzed schemes designed by their AutoMark agents to identify and categorize novelties and discoveries.
Critical strengths
- Formalizing what makes a watermarking scheme valid is interesting.
- The agents find novel watermarking schemes that improve on existing work.
Critical weaknesses
- The paper does not stand on its own without the very lengthy Appendix. In particular, Section 4 is incredibly dense and vague, making it hard to parse and difficult to assess the genuine improvements/novelty.
- You should add a reference to the part of the Appendix where you explain what a Lexical group is when you first mention it. Ideally, since it seems to be one of the core innovations of your agents, you should explain it in the main paper.
Questions for authors
- In Section 3.3, for Algorithm 3, you account for the allowed fraction of bad keys and the probability that a sound key screens positive. But in that case, shouldn’t you also account for the probability that a bad key screens negative?
- Does each agent only get what it has submitted, or does it have access to every submission by every agent? For example, does the GPT-6-ASTRA (xhigh) agent see what the OPUS-5 agent is doing?
- In Section 4.2, context seeding, you mention assigning less frequent tokens a lower . Do you mean preceding tokens, the candidate token, or both? Additionally, you argue that this method preserves the robustness of short contexts, but that is not necessarily true for any attacker aware that you are doing this, since they could focus their efforts on the later part of the text, which is more likely to have longer contexts. Likewise, assigning tokens to categories is also vulnerable to an adaptive adversary.
2 comments
I thank the reviewers for addressing my concerns. I will be increasing my score accordingly.
We are happy to see that the reviewer found our formalization of valid watermarks interesting and that our agents find novel schemes improving on existing work. We address their remaining concerns below, and have updated our submission. All changes in the submission are colored in blue for clarity.
Q1: Can you elaborate on why you chose to structure the paper this way?
We agree that our work has a high information density and includes many novel ideas (52 new watermarking schemes organized in 39 components), and as such requires a lengthy appendix. We therefore made the explicit decision to tailor our main paper (and its evaluation) around 3 equally important takeaways:
Given the space constraints, we had to cut down the main-paper description of individual components and only keep pointers to the relevant appendix sections. We believe that clear communication is essential for impactful work, and would be happy to adjust our writing if the reviewer has concrete suggestions. Leveraging the additional page given for the rebuttal, we expanded Section 4.2 with detailed explanations of Private geometries and of the Clustering and Excess detectors, which are relevant novel components proposed in our work.
Q2: What are the main components proposed in this work?
In Section 4.2, we focus on the components we believe are the most interesting for future works: Lexical grouping, adaptive context seeding, the Gaussian copula, ResidualRace, and clustering-based detection. In particular, Gaussian copula and ResidualRace have already been adopted by industry, highlighting their practical relevance (we avoid naming the adopting scheme during the review process to preserve anonymity, but will cite it in our final revision). Thanks to the reviewer's comment, we now include more cross-references to ease navigation, including a pointer to the definition of lexical groups at their first mention.
Q3: Are the AutoMark runs of different models independent?
Yes, the runs of each model (GPT-6 Astra (xhigh), GPT-6 Astra (low), Opus 5 (high) and Gemini-3.8 Flash (high)) are independent: each agent only sees its own prior submissions (e.g. GPT-6 Astra (xhigh) only sees prior schemes submitted by GPT-6 Astra (xhigh)). We now clarify this point in Section 4.1.
Q4: When assigning less frequent tokens a lower context length k, does the frequency refer to preceding tokens, the candidate token, or both?
We thank the reviewer for highlighting this ambiguity. In the frequency-based variant, k depends on the immediately preceding token: if its frequency in a fixed reference corpus is below 0.1%, we use k=1, and otherwise k=2. The candidate token does not determine k in this variant. We now clarify this rule in Section 4.2.
Q5: Can an adaptive attacker exploit longer contexts at later positions, undermining the claim that the method preserves the robustness of short contexts?
We agree that an attacker aware of this mechanism could target positions with longer contexts to remove the watermark. Nonetheless, this remains more robust than prior works with a fixed long context, as positions with short contexts remain detectable (i.e. even if the signal at long-context positions is removed, the short-context positions still carry watermark signal).
Q6: Is context seeding based on token categories also vulnerable to an adaptive adversary?
We agree that an adaptive attacker aware of Lexical grouping could reduce its robustness gains. Nonetheless, it remains more robust than prior works without grouping against non-adaptive attackers (e.g. students who simply replace a few words in the hope of removing the watermark).
Q7: A genuinely bad key might escape detection. Shouldn't Algorithm 3 account for those missed bad keys?
We thank the reviewer for raising this point. Algorithm 3 uses an upper bound on the probability that a key screens positive: a sound key contributes at most r, while a bad key contributes at most one, yielding β+(1−β)r. A bad key screening negative (i.e. a bad key that is not rejected) only decreases this probability, so no additional correction is needed for the validity of the rejection. However, such false negatives might reduce power (and accounting for them requires assumptions on how far a bad key's FPR exceeds α).