- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper extends BREW’s block-wise designated-codeword watermarking framework with constant-hit-rate embedding, distortion-aware erasures, and Chase-style list decoding. At generation time, an adaptive logit offset targets a prescribed probability of sampling from the token partition corresponding to the embedded bit. At detection time, constant-magnitude log-likelihood scores, erasure handling, and shifted block searches are combined to verify designated BCH codewords. Experiments on OPT-1.3B and Mistral-7B cover clean generation, token edits, and paraphrasing, with additional analyses of payload size, text length, quality, and computational cost. The reported results show improvements over BREW in several settings, although the extent to which these improvements arise from calibrated soft information than other changes to the detector or operating point, remains unclear.
Critical strengths
-
A coherent connection between embedding and decoding.
The paper explicitly connects generation-time probability shaping with detection-time error correction. The constant-hit-rate identity is valid when the intended bias is applied without clipping or subsequent distribution changes. This provides a reasonable starting point for investigating how the watermark channel and decoder should be designed together.
-
Meaningful empirical improvements in some settings.
Several results support the practical promise of the construction. For example, under 10% substitution on OPT-1.3B/C4, Table 9 reports an increase in TPR from 0.6550 for BREW to 0.7017, with no observed false positives for either method. The evaluation also goes beyond clean detection to include paraphrasing, short texts, payload scaling, and runtime.
Critical weaknesses
My concerns affect the interpretation of the central mechanism and the comparability of the main results. Addressing them would require additional calibration diagnostics, matched detector baselines, and comparisons at controlled false-positive and quality levels, instead of only textual clarification.
-
The implemented channel does not consistently satisfy the calibration used to interpret its scores and the soft-decoding contribution is not isolated.
In Section 4.1 and Algorithm 1, clipping can leave a position non-erased whenever its target-list mass remains above 1/2, even if it falls substantially below the target hit rate. Appendix E.5 additionally states that calibration precedes top-p sampling with . Nevertheless, the detector assigns the same nominal LLR magnitude to every non-erased position. The paper acknowledges some of these qualifications, but does not quantify how often the default implementation operates outside the calibrated regime or how much this affects detection. This gap is consequential because calibrated reliability is central to the proposed explanation of the method.
All non-erased positions have the same reliability magnitude. Consequently, for a candidate block with non-erased positions and mismatches, the designated-codeword score is proportional to . It therefore contains no reliability ordering among non-erased positions. Table 2 simultaneously changes erasure treatment, candidate search, and score filtering: the hard baseline fills erased positions with PRF bits, whereas the proposed detector preserves zero reliability. This does not isolate the benefit of calibrated soft information. A BCH errors-and-erasures baseline and a matched candidate-budget baseline are needed to determine whether the improvement primarily comes from erasure handling or a larger search region.
-
The main comparisons do not establish an advantage at matched false-positive and quality levels and the incremental contribution and payload-recovery protocol require clearer positioning.
Table 1 evaluates methods at their default or recommended thresholds, with very high FPRs for MPAC and Qu et al. This does not establish their relative performance under a common low-FPR requirement. Some improvements over BREW also accompany higher FPR: under OPT/C4 insertion, Table 11 reports TPR increasing from 0.4567 to 0.5250 while FPR increases from 0 to 0.0083. Quality comparisons are similarly difficult to interpret jointly: Table 6 reports PPL on OPT/C4 but BLEU and BERTScore on NLLB/WMT16 translation. The latter cannot establish quality preservation in the former setting, where PPL rises from 8.454 without watermarking to 15.306. Comparisons at matched FPR and matched quality are necessary to establish the claimed trade-off improvement.
BREW (Kim et al., 2026) already provides the relationship between target-list probability and logit bias in Appendix C.5, from which the constant-hit-rate offset follows directly. Its public description also includes blind message estimation, whereas the submitted CORE-BREW detector takes designated codewords as input. The paper should clarify what is newly contributed beyond applying this relationship with erasures and list decoding, and whether its payload-recovery results are obtained without access to the true payload. In addition, the stated lack of a public implementation for Chao et al. should be reconciled with the currently available official GitHub repository (Chao et al.), with attention to version and release timing.
Questions for authors
-
Calibration under default sampling settings.
Under the default clipping and top-p settings, what fractions of positions are fully calibrated, clipped but retained, and erased? What are their realized hit rates, and does the improvement persist when using scores consistent with the actual sampling distribution?
-
Comparison with matched decoding baselines.
How does CORE-BREW compare with errors-and-erasures BCH decoding and a baseline using the same number of candidate trials, on identical generated texts and at matched FPR? These comparisons would materially affect whether I attribute the gain to soft reliability.
-
Evaluation at matched FPR and generation quality.
Can the authors provide comparisons at a common FPR and comparable generation quality, together with the number of independent null samples and confidence intervals? For the translation quality results, what detection performance and effective embedding rate are achieved on those same translations?
-
Blind payload extraction and recovery.
Is the true payload available to any part of the evaluated extraction procedure, including erasure construction or designated-codeword selection? If not, what is the complete blind extraction procedure and its message-level exact recovery rate?
3 comments
Response to Reviewer R2Hh (2/2)
Q3 — FPR, independent samples and quality. The main tables contain 600 null trials per setting. If these trials are independent, zero false positives gives a 95% two-sided Clopper-Pearson upper endpoint of 0.613%. We will specify the source/prompt/seed structure when reporting uncertainty; pooling attacks that reuse null sources does not create independent samples.
- Substitution: zero observed false positives for both methods is equal observed FPR, not equal population FPR.
- OPT/C4 insertion: TPR rises from 45.67% to 52.50% while FPR rises from 0% to 0.83%, so we claim no matched-FPR advantage at this default-threshold point.
- Table 1: likewise does not rank MPAC, Qu et al. and CORE-BREW at low FPR. In the revision we will select thresholds on calibration nulls, freeze them, and evaluate on disjoint test nulls, reporting counts and intervals.
On OPT/C4, PPL is 8.454 unwatermarked, 13.643 for BREW and 15.306 for CORE-BREW. The safeguards reduce PPL from 18.631 to 15.306 (17.85%). This is a safeguard benefit, not a matched-quality advantage over BREW. Detector thresholds cannot change PPL on fixed text, so we will add an embedding-strength sweep for matched-PPL comparison.
NLLB/WMT16 BLEU and BERTScore come from a separate translation run that logged only reference-quality metrics without block-detection or erasure outputs, and we will remove the inference that those cross-model scores show quality preservation for OPT/C4 detection. In the revision we will instead evaluate quality and detection jointly on the same watermarked continuations, using the embedding tokenizer and block schedule.
Q4 — Targeted verification vs. blind payload extraction. In the submitted experiments, yes: as documented in Algorithm 3 and App. F.1, our evaluated pipeline implements targeted provenance verification of a hypothesized payload rather than blind decoding. Specifically, Algorithm 3 takes the candidate payload's designated codewords as an explicit input, and observation construction also uses them in the feasibility erasure check. Because we evaluated this verifier at the embedded payload, the reported metric is verification-conditioned: an accepted designated chunk is known by construction, so crediting it does not measure unknown-message recovery. Calling this "payload recovery" obscured that distinction; we will relabel the metric and state its exact accounting rule.
Blind search also carries a multiple-testing cost that the per-hypothesis analysis does not cover. Under an ideal null with independent fair bits, accepting any of the 128 BCH(63,7) codewords within radius 15 has per-block/shift probability exactly for hard decoding (the radius-15 balls are disjoint because ), and at most for the ungated two-position list, by a union bound with payload-independent selected positions. Blind thresholds must therefore be recalibrated, not inherited from targeted verification. These are ideal-null values, not natural-text FPRs.
Neither the submission nor this response reports blind-extraction numbers; the blind evaluation we will add in the revision uses the following protocol.
- The detector receives the text, the key and the scoring model, without the payload or designated codewords.
- The message is sampled independently of the key; key-dependent placement or permutation is allowed.
- Erasures use only payload-independent conditions, or .
- For each block and shift, the full codebook is searched, and decoded chunks are combined by the block schedule.
- Thresholds are selected on calibration nulls and frozen before evaluation on disjoint test nulls.
- We will report exact-message success over all positives, counting abstentions as failures, together with the abstention rate.
Positioning and corrections. BREW already provides the logit-bias relationship and a blind estimation procedure. Our extension is the target-mass/safeguard configuration coupled to model-aware erasure handling and bounded verification. Chao et al.'s official repository is public; we will correct Section 6.1 and include an empirical comparison under our matched low-FPR protocol in the revision (not yet completed here).
The diagnostic quantifies how far realized reliability departs from nominal and shows that a homogeneous correction leaves the default decisions unchanged. The same-text results add E&E and four-trial comparators. The matched-FPR, matched-quality and blind evaluations described above will be reported separately in the revision.
Thank you for the candid response and additional measurements. While the clarifications resolve several ambiguities regarding nominal calibration, detector operating points, and verification-conditioned recovery, they also confirm that central claims in the submission are not supported by the evaluated protocol. I therefore maintain my recommendation of clear rejection (1).
1. The reported “payload recovery” evaluates known-payload verification, not unknown-message extraction. The response confirms that the detector receives the designated codewords corresponding to the true embedded payload, which are also used in the feasibility-based erasure check. Crediting an accepted designated chunk therefore measures verification conditioned on a known payload, as opposed to recovering an unknown message. While targeted provenance verification is a valid task, it represents a substantially narrower contribution than the payload-recovery claims in the paper, and comparing it directly against baselines that must blindly extract the payload creates an unequal evaluation. Relabeling the metric does not supply evidence for blind recovery, and the proposed blind-extraction protocol remains an unexecuted plan with no reported results.
2. The new baselines show no clear gain from the proposed soft-decoding machinery. Under insertion, CORE-BREW detects 200/371 positives compared with 197/371 for simple errors-and-erasures decoding, with an identical 7/1,800 false positives. A three-sample difference without paired uncertainty analysis does not establish a meaningful advantage. Under substitution, CORE-BREW yields both lower TPR and lower FPR than the added baselines, leaving the trade-off inconclusive. Moreover, the score-only verifier matches the aggregate rates of the errors-and-erasures baseline and is analytically equivalent over the stated threshold range. Thus, while explicitly handling erasures is beneficial, the core algorithmic additions—calibrated LLR scoring and Chase-style list decoding—demonstrate no clear improvement over simpler erasure-aware alternatives.
3. Homogeneous-scale invariance underscores the limited role of calibration. The observation that replacing the nominal homogeneous scale with the realized scale leaves default decisions unchanged addresses the scalar-replacement question, but it also highlights that all non-erased positions share a uniform magnitude. Because the detector does not differentiate per-position reliability, this invariance does not show that calibrated reliability drives the observed gains. Furthermore, nominally calibrated, unclipped positions achieve a hit rate of 0.873 (whose confidence interval excludes the 0.9 target), showing that clipping alone does not explain the calibration gap, while position-specific two-hypothesis scores remain untested. Consequently, the evidence supports only a nominal-score verifier with erasure handling, leaving the claimed benefit of calibrated soft information unsubstantiated.
4. Principal comparisons at matched FPR and matched quality remain incomplete. The response acknowledges that the default-threshold insertion results do not represent a matched-FPR advantage, and that low-FPR rankings against MPAC and Qu et al. are not established; the proposed independently calibrated evaluations have not been conducted. Similarly, CORE-BREW’s PPL on OPT/C4 remains worse than BREW’s, with the embedding-strength sweep needed for a matched-quality comparison deferred to future work. Meanwhile, the NLLB/WMT16 experiment reports quality metrics without corresponding detection or erasure measurements, leaving the joint quality–detection trade-off unsupported across both setups.
Overall, while the rebuttal improves transparency, it substantially shrinks the demonstrated contribution: the actual task is targeted verification, the complex soft-decoding machinery shows no clear edge over basic erasure handling, and the critical evaluations—blind extraction, matched-FPR detection, and matched-quality comparisons—remain prospective plans rather than realized evidence. Accordingly, the empirical support is insufficient to warrant acceptance.
Response to Reviewer R2Hh (1/2)
Thank you for the detailed questions. We report the additional measurements below, keeping three things distinct: nominal calibration, detector operating points, and verification-conditioned recovery.
Q1 — Calibration and realized reliability. The target-mass identity holds before clipping and the subsequent sampling transforms. Our additional OPT-1.3B/C4 diagnostic reports these generation-time fractions and hit rates:
Clipped-but-retained positions are rare: 99.4% of non-erased positions are unclipped. Their hit rate nonetheless shows that a retained position need not reach p∗=0.9. The table separates the nominal target from the realized hit frequency, but it does not isolate the causal effect of each sampling transform. With this diagnostic we will report the token/text counts, interval construction, sampling order and safeguard defaults, and reconcile Appendix E.5 with the corresponding run configuration.
The implemented ±ln9 values are therefore nominal scores. An exact bit LLR requires the transition probabilities under both hypotheses, log[P(Y=y∣z=1)/P(Y=y∣z=0)].
One replacement can be settled exactly. Since 63−f−2e is an integer, the default gate τ=5 accepts iff 63−f−2e≥3, for every homogeneous scale λ∈[5/3,,5/2). That range corresponds to hit rates p∈[σ(5/3),,σ(5/2))≈[0.8411,,0.9241), where σ(x)=1/(1+e−x). Any positive scale also leaves the Chase ordering unchanged. Both p∗=0.9 and the realized 0.873 (λ^=ln(0.873/0.127)≈1.93) lie in this range, so replacing the nominal scale by the realized homogeneous one changes no decision. Position-specific two-hypothesis scores would define a different detector, and we will evaluate them in the revision.
Q2 — Additional decoding baselines on identical texts. The General Response compares separately calibrated configurations on the same 371 positives and 1,800 held-out nulls, with calibration nulls disjoint from the test nulls.
These rows are separately calibrated configurations without a paired uncertainty analysis. They are therefore neither single-factor nor matched-FPR comparisons, and none relies on graded non-erased reliability. Within that limit, both configurations that treat erasures explicitly detect more insertion positives than four PRF fills (197 and 200 vs. 179 of 371) at a similar false-positive count (7 vs. 6 of 1,800). This is consistent with a benefit from erasure handling, but it does not isolate one. A gate-only ablation with all other settings fixed can only remove detections (AChase+gate⊆AChase); we will report it in the revision.