acceptodds
Under review as a conference paper at ICLR 2027

PROVENANCELAB: AN AUDITABLE LONGITUDINAL CORPUS FOR LLM WATERMARK TRANSITION ANALYSIS

Abstract

Major providers began embedding machine-readable watermarks in large language model (LLM) text output during 2026 amid the implementation of the EU AI Act's synthetic-content marking requirements, but for many pre-existing commercial models the exact date at which outputs on a particular serving surface begin carrying a watermark is not publicly disclosed, and marking is being rolled out gradually. This creates a narrow and irreproducible window: outputs a model produced before it was marked cannot be regenerated afterward. We present ProvenanceLab, a longitudinal, externally auditable corpus of Claude, GPT-5.6, and Gemini outputs collected across this deployment boundary. Each record carries cryptographic integrity hashing and, for the strongest cohort, service-generated cloud evidence (invocation logs, provider request identifiers), retention-protected storage, and external RFC 3161 timestamps, so that the record of what a model emitted during the transition survives in a form a third party can audit. We further validate the statistical instrument that a future analysis will use: a controlled watermark on/off calibration experiment with the SynthID-Text logits processor on Gemma-2B-IT (1,460 generations) shows, under prompt-clustered inference and prompt-disjoint held-out evaluation, that pre-specified lexical and bigram diversity features register a known watermark with reproducible on/off effects (Cliff's ; 95% cluster-bootstrap CI excluding zero) that strengthen in longer outputs ( for 250–500-word outputs) and persist on held-out prompts. The corpus, the provenance architecture, and the calibrated instrument are the contribution and are complete in this paper; a provider-supported verifier, when one becomes available, supplies the final retrospective label. We do not claim watermark detection or attribution for any commercial model from text statistics.

Primary area:
datasets and benchmarks
Keywords:
LLM watermarking, SynthID, content provenance, longitudinal corpus, EU AI Act, dataset, auditability, RFC 3161, cluster bootstrap, AI-generated text detection

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ejToOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ProvenanceLab, a longitudinal, externally auditable corpus of top LLM outputs (Claude, GPT-5.6, and Gemini ) collected across EU AI Act deployment boundary. It shows each record carries cryptographic integrity hashing. They also validate on a controlled watermark on/off calibration experiment and it shows predefined features related to vocabulary diversity and bigram diversity constitute of watermarking, which offers a reproducible "on/off" effect.

Critical strengths
  1. Given 2026 amid the implementation of the EU AI Act’s synthetic-content marking requirements, Paper study an pratical time-sensitive problem.

  2. ProvenanceLab shows a carefully designed, multi-layered structure, encompassing logs from service providers or the cloud, cryptographic hashes, RFC 3161-compliant timestamps, and "write-once, immutable" storage.

  3. It designed the validation experiments to show reproducible on/off effect.

Critical weaknesses
  1. Although the watermark is related to EU requirement, but not every country has this requirement. The motivation is not enough.

  2. it uses a small open model (Gemma-2B-IT). Not sure scalability. It seems that paper's analysis of commercial models is limited purely to registry and collection. No direct watermark testing was performed on them.

  3. Since watermark have been extensive researched, there are many anti-watermark strategy. We don't see many strong baseline.

  4. With the addition of watermarking, the diversity of bigrams increases from 0.831 to 0.856; this does not imply that the model’s performance has worsened or its reliability has decreased.

  5. For the datasets. We need to know more statistics about prompts, including question category and prompt length.

Questions for authors
  1. Given the randomness of LLM, do we need to worry about the gap with and without watermark if the qualities are similar?

  2. If EU requirement is a motivation, does it consdier different langugage?

Reviewer_VWDGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents ProvenanceLab, a longitudinal corpus and provenance framework for preserving model outputs across potential watermark deployment transitions. It combines fixed-prompt collection, tiered provenance evidence, cryptographic hashes, and trusted timestamps to support future retrospective verification. A controlled SynthID-Text experiment on Gemma evaluates whether selected text statistics respond to watermarking. The collected service-model outputs currently lack verified watermark labels.

Critical strengths

The paper proactively preserves historical model outputs that may become unavailable after watermark deployment, with explicit provenance tiers and external timestamping.

Critical weaknesses
  1. The paper explains the value of preserving outputs generated before watermark deployment for tracking rollout timing, commercial auditing, and compliance checks. However, it does not identify the specific scientific research that this dataset can support or explain its unique scientific value compared with controlled experiments using open-source models, which can readily generate outputs with and without watermarking. Thus, the paper motivates the commercial and compliance value of data collection but does not establish its scientific research value.
  2. The discussion of how chain-of-thought (CoT) affects model outputs in the related work section has no clear connection to this study. The paper does not explain how these studies inform its motivation, methodology, or analysis. This discussion should be clarified or removed.
  3. Figure 1 appears to be AI-generated. Using AI to create a workflow illustration is not inherently problematic, but the authors should carefully check its accuracy and relevance. Some elements appear unnecessary and should be removed to improve clarity. The label “Figure 1” should appear only in the caption, not within the image itself.
  4. The provenance mechanism does not establish the authenticity of records before they are committed. Rather than obtaining provider-signed records at the source, the requester submits the collected files for subsequent timestamping, leaving an opportunity to modify them beforehand. Consequently, the hashes and timestamps can reveal changes made after commitment but cannot establish that the corpus was unaltered during collection.
  5. The calibration experiment evaluates only one watermarking method, SynthID-Text. Different watermarking mechanisms, such as logit perturbation, random-number-based sampling, and text rewriting, may affect output statistics differently. The paper should include experiments with additional watermarking methods to assess whether the selected features generalize across mechanisms.
Questions for authors

Beyond tracking watermark rollout timing and supporting commercial auditing and compliance checks, what specific scientific questions can this dataset answer? Which of these questions cannot be studied through controlled watermarking experiments using open-source models?

Reviewer_W6KPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents ProvenanceLab, a longitudinal corpus designed to preserve commercial LLM outputs during the transition to text watermark deployment. The paper propose a provenance framework combining cryptographic hashing, cloud service logs, write-once storage, and external timestamps to support future auditing. It also conducts a controlled watermark on/off experiment using SynthID-Text on Gemma-2B-IT to evaluate whether text-level statistical features are sensitive to watermarking.

Critical strengths
  1. The paper addresses a timely problem: preserving commercial LLM outputs before potentially irreversible watermark deployment changes. Such historical observations could be valuable for future research.

  2. The provenance framework considers multiple evidence sources, and the controlled calibration uses prompt-clustered statistical testing and held-out evaluation to assess watermark-related features.

Critical weaknesses
  1. The core watermark transition analysis remains incomplete. The paper does not identify or verify any actual watermark deployment transition in commercial LLMs. Section 3.3 reports no coherent watermark-signature shift, but provides no detailed quantitative results to support this finding. Consequently, the paper offers little empirical insight into the watermark deployment changes that motivate the study.
  2. The longitudinal corpus is insufficiently characterized and evaluated. Despite presenting the corpus as a primary contribution, the paper does not report essential dataset statistics, such as the total number of commercial model generations, per-model sample counts, collection frequency, or detailed temporal coverage. It also lacks meaningful longitudinal visualizations and concrete demonstrations of independent end-to-end provenance verification. These omissions make it difficult to assess the scale, quality, auditability, and research utility of the proposed corpus.
  3. The controlled calibration experiment provides limited support for the intended application. The only substantial quantitative experiment uses Gemma-2B-IT with a known SynthID-Text configuration rather than the commercial models studied in the corpus. The observed effects are relatively weak, with a held-out ROC-AUC of only 0.591. Although the authors do not claim to build a practical detector, the experiment does not establish whether these statistical features can reliably reveal watermark-related changes in commercial LLMs under unknown decoding configurations and potential serving updates.
  4. The technical novelty and independent scientific contribution are limited. The provenance architecture primarily combines established techniques, including SHA-256 hashing, cloud invocation logs, write-once storage, and trusted timestamps. The paper does not clearly demonstrate new technical guarantees or advantages over simpler existing provenance mechanisms. Moreover, its primary intended application depends on future provider-supported watermark verification, leaving the practical value of the current framework insufficiently demonstrated.
Questions for authors
  1. Could the authors provide complete statistics for the commercial LLM corpus, including the number of generations per model, collection periods, sampling frequency, and longitudinal coverage? Could they also demonstrate an actual end-to-end independent audit of representative records?
  2. Section 3.3 states that no coherent watermark-signature shift was observed. Could the authors provide the corresponding quantitative results, temporal feature distributions, and statistical tests supporting this conclusion?
  3. Given the weak calibration performance on Gemma-2B-IT and the potential confounding effects of commercial model updates, what evidence supports the applicability of these statistical features to real-world watermark transition analysis?
  4. What specific technical or scientific contributions does the proposed provenance framework provide beyond existing logging, hashing, and trusted-timestamp mechanisms? How much of its intended research utility can be demonstrated without a future provider-supported verifier?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.