acceptodds
Under review as a conference paper at ICLR 2027

SpecPF: Speculative Permute-Flip Decoding with Improved Watermark

Abstract

We study speculative decoding for the permute-and-flip (PF) decoder, proposed by Zhao et al. (ICLR 2025) while preserving its native watermark detectability. Our method, SpecPF, speculatively accelerates the recently proposed permute-and-flip decoder while preserving the correct output distribution, and hence all of its desirable properties, including optimal stability, natural watermarkability, and Pareto-optimality with respect to the stability–perplexity trade-off. We design a novel watermark scheme, namely a cross-fitted bigram filter followed by the exact PF Fisher/Gamma test. Our watermark detection method is tailored to the permute-and-flip distribution and is model-agnostic, consistently outperforming the original watermarking scheme. While keeping the target distribution exactly unchanged and hence inheriting all desirable properties of the original PF decoder, our speculative decoding method achieves up to wall-clock speedup, while longer lookahead increases the number of emitted tokens per target batch to . We also investigate the recently discovered sampling-watermark trade-off frontier for the SpecPF decoder. Experimental results across multiple model pairs and datasets validate both the improved sampling efficiency and stronger watermark detectability of our method.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLM Watermarking, Speculative Decoding, Speculative Sampling, Permute-and-Flip, information theory.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YPZdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies speculative acceleration of Permute-and-Flip (PF) decoding while preserving its native watermark. SpecPF couples the draft and target models through a shared exponential random field, proposes multiple candidates at the root, extends each branch along a single path, and lets the target PF map traverse the precomputed tree. Theorem 2 establishes pathwise equivalence to direct PF under identical target logits, randomness, and the other stated conditions. For detection, the paper derives the alternative density of the PF pivot, proves single-pivot optimality of the native score, and applies five-fold cross-fitted bigram-signature filtering before Fisher/Gamma testing. Across Qwen and Vicuna configurations on three datasets, shallow SpecPF reports up to 1.32x measured speedup over direct PF. Longer lookahead raises aggregate accepted tokens per verification from 1.787 to 2.745. Signature filtering improves aggregate detection rates at 64 and 128 scored tokens. The approach is promising, but implementation-level exactness and gains under fully aligned comparisons remain insufficiently established.

Critical strengths
  1. A simple construction with a clear exactness objective. The target PF map always determines the output, while the draft only schedules contexts for advance computation. This supports a direct pathwise-equivalence argument and offers a principled route to retaining the native watermark. Root branching followed by single-path extensions limits tree growth. The same-run results in Table 1 show approximately 1.27-1.32x speedups on Vicuna, demonstrating practical value for the shallow configuration.
  2. A theoretically motivated detector that does not require target-model replay. The monotone alternative density motivates retaining the native single-pivot score and improving detection by excluding weak-evidence positions identified through observable bigrams. Cross-fitting, shrinkage, and minimum-retention safeguards make the procedure reasonably well specified. The aggregate TPR increase from 0.554 to 0.600 at 128 scored tokens in Table 2 is a useful empirical signal that merits further controlled evaluation.
Critical weaknesses
  1. The gap between theoretical pathwise exactness and implementation agreement remains unresolved. Table 21 reports whole-sequence agreement with L=1 of 0.909, 0.903, and 0.903 for L=2, 3, and 4, respectively, with the same pivot-exactness rates. Table 10 explicitly uses different context-label domains for Direct and Fused generation. These comparisons therefore do not directly validate the same-random-field, same-target-logits equivalence required by Theorem 2. The appendix appropriately acknowledges that similar detection rates do not prove pathwise preservation. Nevertheless, exact sampling and watermark preservation are central claims, so the discrepancies require a concrete explanation and a targeted implementation check. The reported mismatches do not by themselves refute the idealized theorem, but they limit its support for the current implementation.
  2. Efficiency and detection gains are not yet isolated under aligned operating conditions. At M=64 in Table 2, TPR rises from 0.455 to 0.481 while empirical FPR rises from 0.009 to 0.013; the gain at strictly matched empirical FPR is therefore unclear. For efficiency, a width-three SpecPF tree is compared with methods using one draft sequence and different target output distributions. Higher accepted-token counts do not establish better throughput at equal target-computation cost. Table 21's L=4 configuration achieves only 0.903x the geometric-mean throughput of the new L=1 baseline, and Table 17 also shows slowdowns for Qwen7. The manuscript discloses these limitations, but stronger evidence at matched computational budgets and empirical FPR is needed to substantiate the practical advantage.
Questions for authors
  1. What causes the approximately 9-10% sequence disagreement in Table 21? Please compare direct PF, shallow SpecPF, and longer-lookahead SpecPF token by token using identical context labels, randomness, and target-processing rules. A trace of the first divergence, including logits, support, noise, and cache execution, would help distinguish random-field addressing or execution differences from numerical effects. Can complete agreement be recovered when the theorem's conditions are enforced?
  2. How much does SigFisher improve detection on the same test population at matched empirical FPR? Please provide low-FPR ROC results or matched-FPR TPR estimates with uncertainty intervals, and clarify whether cross-fitting groups samples by prompt and key. Do the conclusions hold when filter-training and threshold-calibration uncertainty, excluded from Table 6's intervals, is included?
  3. Do the efficiency gains remain when target-context computation or measured runtime budgets are matched? A breakdown of drafting, target verification, and tree/cache management would clarify when longer lookahead improves token yield and when it improves actual throughput.
  4. Please specify the context-label function and treatment of repeated labels so that the Fisher/Gamma assumptions can be checked against the implementation. Which constructions or theoretical results are specifically new to PF relative to the cited shared-randomness methods and Hong et al.'s signature filtering?
4 comments
AuthorsOpenReview ↗

Responses to the questions (part 1).

Thank you so much for a precise evaluation of the main outstanding soundness and empirical-comparison issues! We have substantially revised the paper according to your comments. All the changed are highlighted in blue.

We performed additional numerical and statistical analyses specifically targeted at these questions. We answer the questions one by one as follows.

Q1. What causes approximately 9–10% L1/L>1 sequence disagreement? Can complete agreement be recovered?

Mathematical statement. Define a single deterministic processed-target operator UP(c)=(uc,SP(c))\mathcal U_P(c)=(u_c,\mathcal S_P(c)). With the same operator, labeler Λ\Lambda, keyed field EΛ(c)(y)E_{\Lambda(c)}(y), support, tie-breaking and stopping rule at every reached context, both direct PF and SpecPF emit Y(c)=arg⁡max⁡y∈SP(c){uc(y)T+EΛ(c)(y)}.Y(c)=\arg\max_{y\in\mathcal S_P(c)}\{\frac{u_c(y)}{T}+E_{\Lambda(c)}(y)\}.

Induction over tokens gives exact pathwise equality, for any draft, BB and LL. This is a fixed-operator theorem; bitwise agreement between separately batched/cached FP16 forward passes is not assumed.

Physical experiment. The completed 6-cell lookahead sweep reports whole-sequence agreement with production L1 of 0.909, 0.903 and 0.903 for L2, L3 and L4. We extracted first disagreements and replayed six selected examples on the installed production shallow decoder and rank-tree executor. Each implementation independently reproduced its saved output in all six cases. Context labels, the processed top-50 support and keyed PF uniforms agreed; the numerical PF selection matched independent score calculations. The physical processed target logits nevertheless differed sufficiently to reverse the PF winner:

Model/datasetPrompt IDLFirst positionProduction L1 → rank-tree tokenMaximum top-50 ∣Δu∣\lvert \Delta u \rvert
Qwen14 / ELI521210 → 110.031250
Qwen14 / ELI5102313594 → 5250.031250
Qwen14 / ELI52824039814 → 323130.046875
Vicuna / ELI552521221577 → 14220.015625
Vicuna / ELI56474020419 → 56740.015625
Vicuna / ELI5658205674 → 4500.015625

For example, on Qwen14 prompt 102, the processed logit of competing token 525 is 39.78125 in L1 versus 39.81250 in the rank tree, reversing the PF winner under the same keyed field. The two later-position traces were completed after correcting our diagnostic tracer's post-forward cache-length assumption; this was not evidence of a decoder cache bug.

These six examples identify a reproducible selected-case mechanism consistent with execution-geometry-dependent numerical differences. They do not establish that every population mismatch has this cause, that FP16 rounding alone is responsible, or that the current physical executors have achieved 100% agreement. Table 10's historical Direct/Fused experiments use distinct label domains, and we now explicitly label that comparison as a statistical rather than pathwise same-field check. A common-operator bitwise replay of the entire generation campaign has not been performed; we will not claim otherwise.

AuthorsOpenReview ↗

Responses to the questions (part 2).

Q2. Matched FPR, low-FPR sensitivity, prompt/key grouping, and bootstrap uncertainty

Scoring correction. A CPU audit reconstructed the frozen leave-one-dataset-out results and verified legacy_v1_reproduction = True. The legacy program thresholded a raw retained sum, although the retained number of pivots varies by text. The correct canonical score uses SA=∑t∈A−log⁡Vt,pA=Pr⁡{Γ(∣A∣,1)≥SA}.S_A=\sum_{t\in A}-\log V_t,\qquad p_A=\Pr\{\Gamma(|A|,1)\ge S_A\}. With unchanged frozen tables and the original empirical calibration convention, Original PF and corrected SigFisher attain TPR 0.5508 and 0.5920 at M=128M=128 (empirical FPR 0.0129 and 0.0139). This is a frozen cross-dataset transfer audit and does not independently establish matched-FPR improvement.

Main population reanalysis. We use the same held-out positive and null sample sets for Original PF and SigFisher within each model–dataset–budget cell. Five-fold splits group by prompt; each prompt has its own derived key. For each of 100 bootstrap repetitions per eligible cell, we resample paired prompt records, refit the five-fold bigram tables and recalibrate null thresholds. The stored generator outputs, original folds and keyed-field realizations remain fixed. All ten eligible cells reproduce the earlier canonical detector-suite output before resampling.

BudgetProtocolTPR Original → SigFisherMeasured FPR Original / SigFisherTPR difference (pp)Descriptive pooled 95% CI (pp)
64Common analytical p≤.01p\le .0143.30% → 45.65%0.75% / 0.99%+2.35[+0.95, +3.08]
64Independent null calibration46.67% → 48.23%1.09% / 1.19%+1.56[−1.35, +5.98]
64Descriptive matched-FPR ROC45.96% → 47.38%1.12% / 1.12%+1.43[−3.37, +5.16]
128Common analytical p≤.01p\le .0159.41% → 62.89%1.40% / 1.34%+3.47[+1.37, +4.39]
128Independent null calibration56.12% → 60.88%1.34% / 1.04%+4.75[−4.34, +13.14]
128Descriptive matched-FPR ROC56.19% → 60.21%1.10% / 1.10%+4.02[−1.66, +9.39]

All six cells qualify at M=64M=64 (n+=2,942n_+=2{,}942 test), but at M=128M=128 only four cells qualify (n+=1,641n_+=1{,}641). Qwen14/CNN has just 14 calibration and 2 test texts; Vicuna/CNN has 94 and 45; both are excluded by the prespecified minimum of 100 in each split. The 128-token figures thus cannot be directly compared with the paper's original nine-cell or main six-cell aggregate. The intervals are descriptive test-count-weighted pooled bootstrap intervals, not hierarchical population-wide confidence intervals. Analytical cutoffs do not ensure identical empirical FPR, and matched-ROC thresholds were selected using evaluation nulls rather than an independent calibration split. Crucially, both independently calibrated 95% intervals include zero. We therefore do not claim statistically conclusive SigFisher superiority at strictly matched realized FPR, and we now provide low-FPR sensitivity plots with these qualifications.

Q3. Matched target work and runtime budget; drafting, verification and cache-management costs

The same-run direct-PF comparisons show the wall-clock regimes in which shallow SpecPF accelerates the same target law (Qwen14: 1.036–1.142×, Vicuna: 1.274–1.320×) and the Qwen7 slowdown (0.870–0.947×). The completed L1–L4 sweep also reports target contexts, draft/target calls, block latency, peak memory, and token rate; the average represented target contexts/block grows from about 4.0 at L1 to about 12.9 at L4. Although AATPS rises from 1.787 to 2.745, the geometric-mean L4/L1 throughput is only 0.903×. The L>1 rank-tree verifier is not optimized to precisely the same degree as the production L1 path, so this comparison includes implementation maturity as well as algorithmic overhead.

We now show actual token rate alongside AATPS and discuss represented target contexts as a work proxy, not measured FLOPs. We do not claim a controlled equal-compute, same-device comparison against all softmax-based baselines or separate per-component draft/verification/cache latencies; those measurements have not been completed. We also distinguish the U-score-only MSE-PSEUDO result from the unvalidated acceptance-aware detector. These revisions narrow the cross-method interpretation to observed operating points rather than a compute-matched Pareto-optimality claim.

AuthorsOpenReview ↗

Responses to the questions (part 3) and conclusion

Q4. Context labeler, repeated labels, Gamma-null assumptions, and novelty

The installed production shared-noise generator constructs Λ(c)=domain∥BE8(∣c∣)∥BE8(c1)∥⋯∥BE8(ck).\Lambda(c)=\mathtt{domain}\Vert\mathrm{BE}_{8} (|c|)\Vert\mathrm{BE}_8(c_1)\Vert\cdots \Vert\mathrm{BE}_8 (c_k). where k=∣c∣k=|c|. using the domain separator specwater-shared-noise-portfolio-pf-v1|, an 8-byte unsigned big-endian length, and signed 8-byte big-endian token IDs. Prefixes at different output positions have different lengths and therefore cannot reuse the same label within a sequence. Keys are derived separately for different prompts. The null requires a text independent of the held-out key field, unique scored keyed context–token pairs, and a frozen text-only filter fitted without access to held-out-key randomness. Conditional on these assumptions in the ideal random field, retained pivots are i.i.d. uniform, giving the exact Γ(∣A∣,1)\Gamma(|A|,1) null. The implemented PRF is computationally pseudorandom rather than an ideal independent field; finite-sample empirical FPRs remain necessary.

We now explicitly credit generic shared-randomness decoding and Hong et al.'s signature filtering; the original contributions are the PF-specific bounded-width shared-exponential executor, the conditional fixed-operator pathwise theorem, the selected-PF-pivot density/monotonicity and single-pivot lower-tail optimality, and their application to the PF retained-count Fisher/Gamma test.

We remain confident in the value and contributions of our work, and would sincerely appreciate it if you could kindly reconsider your evaluation in light of the additional clarifications, corrections, and new experimental results. Thank you again for your time and consideration!

Reviewer_YPZdOpenReview ↗

Thank you for the detailed responses

Thank you for the detailed responses and the effort devoted to the revisions and additional analyses. I have carefully considered the clarifications, corrections, and new results, which help clarify the scope of the claims and the interpretation of the experiments. However, after considering the response as a whole, my overall assessment remains unchanged. I therefore maintain my original score and recommendation of weak rejection. I appreciate the authors’ engagement throughout the discussion. score:2.

Reviewer_WzbPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper accelerates the permute-and-flip decoder while preserving the correct output distribution and its native watermark detectability. It provides a cross-fitted bigram filter for watermarking, achieving up to 1.32× wall-clock speedup.

Critical strengths
  1. The research problem is clear and potentially benefits practical applications.
  2. The construction of the paper is good.
Critical weaknesses
  1. The problem statement could be improved to help evaluate the innovation this paper may contribute.
  2. This paper should clarify the scope of the Signature-filtered PF, SpecPF, and original PF scores reported in the experimental results.
Questions for authors
  1. There is a typo on line 145: "tbhe" should be "the".
  2. Why is Theorem 1 named "Drafter invariance"? It says "does change" and "can change". What is the purpose of introducing Theorem 1?
  3. In Figure 1, why does SpecPF have Average tokens per verification step (AATPS) higher than other baselines? Why do do the other baselines show a decline in TPR as AATPS grows,, while SpecPF stays flat?
  4. What does "Retained@64" in Table 2 mean? Why is Signature-filtered PF's performance lower in Retained@64 but higher in other metrics compared with baselines?
  5. When watermark detection results are reported on SpecPF, does it mean speculative permute-decoder + Signature-filtered detection, or the speculative permute-decoder score? Does watermark detection require LLM inference to obtain PF scores? Do PF scores equal the speculative PF score?
  6. Does the use of a lightweight draft model to predict the realized PF random map introduce any concerns or challenges?
2 comments
AuthorsOpenReview ↗

Responses to the questions.

We thank the reviewer for the detailed questions. We have substantially revised the paper according to your comment. We have clarified that SpecPF is the generator, while Original PF/Fisher and SigFisher are post-hoc detectors; the latter does not change the target PF sampler and neither requires model inference at verification time.

We answer the questions one by one as follows.

Q1. Typo on line 145

Thank you. We corrected “tbhe” to “the”, as well as “Gor” to “For” and “Concludion” to “Conclusion”. All changes are marked in blue.

Q2. Why is Theorem 1 called drafter invariance? “Does change” versus “can change”

The phrase “does change the fixed-length output sequence” is a typographical error and has been corrected to “does not change.” The intended statement is that changing the draft model or its parameters can alter accepted paths, verification blocks, cache traces and runtime, but cannot alter the emitted sequence when the target-logit operator and random field are fixed. Drafter invariance is a direct consequence of Theorem 2; we state this dependency explicitly while retaining the original theorem numbering to avoid disrupting manuscript cross-references.

Q3. Higher AATPS in Figure 1; why does SpecPF TPR remain flat?

SpecPF retains up to three distinct root alternatives and longer rank-one continuations, whereas the compared baselines propose only one draft sequence. This improves the number of tokens covered by a target-verification step, but incurs more target contexts and draft/tree work. Accordingly, higher AATPS does not prove superior equal-compute efficiency. The measured within-PF L1–L4 throughput actually declines in the geometric mean despite increasing AATPS.

Theoretically, for a common fixed target operator and random field, altering speculative lookahead does not change the emitted target-PF sequence or its watermark pivots. In our physical L1/L>1 experiments, however, differing FP16 target execution geometries can change selected tokens. The nearly flat observed PF detection rates across L are an empirical result, not proof of bitwise pathwise invariance. The other methods' weaker or declining U-score TPR reflects their method-specific acceptance/watermark policies and is not evidence of a common-law, common-detector comparison.

Q4. Meaning of Retained@64

Retained@64 = 0.834 means SigFisher retains 83.4% of the first 64 scored evidence positions on average, after dropping observed bigram signatures associated with weaker training evidence. It is not a TPR; filtering may increase signal density while reducing sample count. The retained-count Gamma tail uses m=∣A∣m=|A|, not a fixed 64, and the detector falls back to all positions if fewer than 30 remain. We clarified this and the distinction between a nominal 1% FPR threshold and the empirical FPR.

Q5. What does “SpecPF detection” mean? Need target-model inference?

SpecPF emits PF-generated text; the target's native pivot for emitted token wtw_t is Vt=RΛ(ct)(wt)V_t=R_{\Lambda(c_t)}(w_t). Original PF combines −log⁡Vt-\log V_t over all scored tokens, while SigFisher retains a text-only selected subset and applies the corresponding Gamma-tail p-value. There is no separately defined “SpecPF score”: the speculative executor should use the same target PF field, conditional on the exactness theorem's operator assumptions. A detector needs the candidate text, tokenizer, initial prompt or saved context labels, and secret key—but no target-model replay.

Q6. Does a lightweight draft introduce difficulties?

The draft's approximation quality affects coverage, AATPS, drafting cost and the number of verification blocks. It does not enter the mathematical target PF winner. In practice, low target/draft agreement or high draft/tree overhead can eliminate the speed benefit; this is observed on Qwen7. Physical batch/cache numerical differences also warrant care: our new first-divergence replays diagnose a concrete way that two physical target executions can disagree even with matching context labels and keyed uniforms. We make these limitations explicit.

Based on the clarifications, corrections, and new results, we would be grateful if the reviewer could kindly reconsider the evaluation. Thank you for your time!

Thank you to the authors for their clarifications. Most of my concerns have been addressed. I consider this paper to be technically sound. However, I still have reservations about the practical applicability of this work, and It seems to me that the new knowledge it contributes to the field is relatively limited. Therefore, I have raised my soundness score while maintaining my overall rating.

Reviewer_unekOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose SpecPF, a speculative decoder for permute-and-flip sampling. The draft and target models share exponential randomness, and a bounded-width draft tree predicts the target’s realized continuation. They prove pathwise equivalence to pure PF decoding under the stated assumptions. They also introduce a bigram filter that removes weak watermark evidence before applying the PF Fisher test. Experiments on different LLMs across three datasets report the speedup over pure PF and improved watermark detection.

Critical strengths
  1. The shared-noise construction provides a clean way to preserve the target PF output under speculative execution.

  2. The analysis of the selected PF pivot is useful. The explicit density and monotonicity result explain why small pivots provide watermark evidence and give a principled basis for single-token testing.

  3. The method has a working implementation with measurable gains. Detection also avoids running the target model again, which is useful when generation and verification are handled separately.

Critical weaknesses
  1. The novelty seems limited. Shared-randomness decoding, watermarkable multi-draft coupling, and signature filtering already exist. The contribution appears mainly to be a PF-specific adaptation of these ideas.

  2. Speedups are modest and model-dependent: 1.036–1.142× on Qwen-14B, but 0.870–0.947× on Qwen-7B. These gains require additional draft and tree computation, with much more peak memory.

  3. The efficiency comparison is not compute-matched. SpecPF uses a width-3 tree, while the main baselines use one draft sequence. Higher tokens per verification step therefore do not establish better efficiency, especially when the methods also use different output distributions.

Questions for authors

Does SpecPF still improve the wall-clock speed–detection trade-off with comparable compute budgets, matched FPR, and each baseline’s native detector?

1 comment
AuthorsOpenReview ↗

Responses on novelty, speedup, memory, fair computation, and wall-clock speed–detection trade-off

Thank you for recognizing the value of the shared-exponential coupling, the PF-pivot density analysis, and the implementation. We agree that a fair comparison must separate same-law acceleration of PF, improved detection for PF-generated texts, and cross-law operating points, which currently use different speculative widths and detectors.

We have substantially revised the paper according to your comment. We answer the questions as follows.

1. Novelty relative to shared-randomness decoding and signature filtering

We do not claim to have introduced the general ideas of shared-randomness speculative sampling or signature filtering. We now make the PF-specific contributions explicit:

  1. PF-targeted bounded-width executor. Draft and target use the same exponential random field for the PF report-noisy-max map. The draft selects the top-BB distinct root alternatives under that field and extends each branch along one rank-one continuation; the target alone determines emitted tokens. The tree has at most BB leaves and 1+BL1+BL represented target contexts, not BLB^L leaves.
  2. Fixed-operator pathwise identity and drafter invariance. For the same deterministic target operator UP:c↦(uc,SP(c))\mathcal U_P:c\mapsto(u_c,\mathcal S_P(c)), context labels, keyed field, stopping rules, and tie-breaking, direct PF and the speculative executor make the same target-PF decision at each reached context. The draft only determines which contexts are precomputed. We have clarified the common-operator assumption and separated it from physical FP16 execution.
  3. PF-specific selected-pivot theorem. Writing ay=exp⁡((uc(y)−uc⋆)/T)a_y=\exp((u_c(y)-u_c^\star)/T) and Y=arg⁡min⁡yRy/ayY=\arg\min_y R_y/a_y, the selected uniform pivot V=RYV=R_Y has density gc(v)=∑y∈SP(c)∏z≠y[1−vazay]+,0<v<1.g_c(v)=\sum_{y\in\mathcal S_P(c)}\prod_{z\ne y}\left[1-v\frac{a_z}{a_y}\right]_+,\qquad 0<v<1. Its monotonicity makes the lower-tail test most powerful for each fixed-logit, single-pivot testing problem. This does not assert unique numerical optimality of −log⁡V-\log V or multi-token UMP optimality.
  4. PF-specific signature-filtered Fisher/Gamma detection. We credit Hong et al. for signature filtering. Our contribution is its principled application to PF's heterogeneous watermark evidence, retaining the native score and normalizing the retained sum by its actual Γ(m,1)\Gamma(m,1) null, m=m= number of retained positions.

The manuscript now explicitly distinguishes these contributions from earlier coupling and filtering methods.

2. Speedup, memory and fair computation

We agree that reported speedups are model-dependent. The original same-run, 1,000-prompt-per-cell SpecPF versus vanilla PF experiment reports 1.036–1.142× on Qwen2.5-14B/0.5B and 1.274–1.320× on Vicuna-7B/68M, with additional peak memory up to approximately 1.7 GiB in these six settings. On Qwen2.5-7B/0.5B, the system is below break-even (0.870–0.947×). We retain these negative results and do not imply uniform speedup.

The lookahead study further demonstrates that AATPS is not equal target work: from L=1L=1 to L=4L=4, AATPS improves from 1.787 to 2.745 while geometric-mean within-cell throughput falls to 0.903× the measured new L1 baseline. The rank tree represents up to 1+3L1+3L target contexts per block, with drafting, batching and cache overhead. Its L>1 executor is also separately implemented from the optimized shallow path. We revised the interpretation and supply the completed per-cell timing results rather than claiming higher AATPS proves higher throughput.

3. Wall-clock speed–detection trade-off at matched compute, matched FPR and native detector

The current evidence does not establish all three requirements simultaneously. SpecPF watermarks the PF distribution; MWS, MSE and MSE-PSEUDO benchmark softmax-based output laws and mostly use one draft sequence, whereas SpecPF uses width 3. In the existing baseline study, MSE-PSEUDO is evaluated using the U-score, not the additional acceptance-aware native detector of He et al. We have explicitly labelled this and no longer describe the frontier as an equal-compute, fully native-detector comparison.

For a within-PF, paired detector comparison, we added an analytical-cutoff test and a refitting bootstrap. At common p≤0.01p\le0.01, Original/SigFisher TPR is 43.30%/45.65% (M=64M=64; empirical FPR 0.75%/0.99%) and 59.41%/62.89% (M=128M=128 on four eligible cells; FPR 1.40%/1.34%). At descriptively matched realized FPR, point estimates favor SigFisher, but its refitting-bootstrap confidence intervals include zero. Independent null calibration likewise yields uncertain gains. We report these limitations rather than claiming statistically established, general strict-FPR superiority.

We believe the revised contributions and new audits clarify both the positive results and the regimes where additional optimization or evaluation is needed. We would be grateful if you could kindly reconsider the evaluation in light of these clarifications, corrections, and new results.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Thank you and summary of revision.

We sincerely thank the Area Chair and all reviewers for their careful assessment and constructive feedback.

We have substantially revised the manuscript using the permitted additional page, with changes highlighted in blue.

In particular, we have:

  • corrected and clarified the fixed-target-operator exactness and drafter-invariance statements;

  • performed production-code first-divergence GPU replays that identify a concrete mechanism behind six selected physical L1/L>1 mismatches;

  • repaired the retained-count Gamma scoring audit and added paired, low-FPR analyses with prompt-level refitting/calibration uncertainty;

  • separated AATPS, measured throughput, and target-work fairness, explicitly acknowledging the limits of cross-law/native-detector comparisons.

We believe these additions substantially clarify and strengthen the paper and address the reviewers' main technical concerns. We warmly welcome any further questions, comments, or discussion from the reviewers and the Area Chair.

If our clarifications, corrections, and new results are satisfactory, we would be grateful if the reviewers could reconsider their assessments in light of the updated evidence.

Thank you again for your time and consideration.

AuthorsOpenReview ↗

Follow-up on Author Responses and Revisions

Dear Reviewers and Area Chairs,

As the discussion period approaches its end in less than 35 hours, we would like to kindly follow up on our responses.

We have provided detailed responses to the reviewers' concerns and substantially revised the manuscript based on your constructive feedback, with all changes highlighted in blue. Our main additions include:

  1. Theoretical clarifications: Strengthened the novelty and theoretical contributions of SpecPF, clarified the fixed-target exactness and drafter-invariance guarantees, and distinguished theoretical guarantees from physical implementation behavior.
  2. Implementation: Conducted GPU replays of six first-divergence cases, identifying numerical differences that explain the mismatches in these cases between different lookahead implementations.
  3. Watermark detection: Corrected the Gamma scoring procedure and conducted 100-bootstrap paired analyses. SigFisher improves TPR from 43.30% to 45.65% at 64 tokens and 59.41% to 62.89% at 128 tokens.
  4. Efficiency and fair comparisons: Reported up to 1.32× measured speedup over direct PF, investigated the trade-off between AATPS and actual throughput, and clarified the limitations of comparisons.

We sincerely hope these revisions have adequately addressed your concerns. We would greatly appreciate any further feedback from the reviewers and Area Chairs, especially if any concerns remain or additional clarification would be helpful.

We would also be grateful if you could kindly reconsider your assessments in light of the clarifications, corrections, and new experimental evidence.

Thank you again for your time, thoughtful feedback, and consideration!

The Authors

AuthorsOpenReview ↗

A Gentle Follow-up on Our Responses and Revisions

Dear Reviewers and Area Chairs,

As the discussion period approaches its end (in 9 hours), we would like to kindly follow up on our responses.

We have provided detailed responses to all reviewers' concerns and substantially revised the manuscript based on your constructive feedback, with the initial revisions highlighted in blue.

We sincerely hope these revisions have adequately addressed your concerns. We would be grateful if you could kindly reconsider your assessments in light of the strengthened theoretical results, additional experiments, and revised presentation.

Thank you again for your time, thoughtful feedback, and consideration!

The Authors

AuthorsOpenReview ↗

Summary of Contributions and Rebuttal Revisions for the Area Chair

Dear Area Chair,

As the discussion period about to conclude, we here briefly summarize our main contributions and the additional evidence provided during the rebuttal to facilitate your assessment.

1. Main Contributions and Novelty

Our work goes beyond directly applying existing speculative decoding and watermark detection techniques.

  • Speculative PF: We develop a novel PF-specific bounded-width speculative decoding algorithm based on shared exponential randomness. Our theoretical results establish pathwise equivalence to direct PF. Keeping to align with PF is exactly our purpose: we hence inherit all PF's desirable stability, utility, and watermarking properties. We use novel speculation to accelerate it.

  • Novel PF-specific watermark detection: We theoretically derive the exact density of the PF pivot and establish its monotonicity and single-token most powerful testing property (Theorem 3). This provides a principled theoretical foundation for PF watermark detection.

  • Signature-filtered PF detection: Building on existing signature filtering, we develop a PF-specific Fisher/Gamma detector that exploits heterogeneous watermark evidence, requires no model inference at detection time, and preserves the exact analytical null distribution under the stated assumptions.

2. Main Additions and Clarifications During Rebuttal

We have substantially revised the manuscript, with all changes highlighted in blue.

  • Theoretical correctness: Clarified the exactness and drafter-invariance guarantees, their assumptions, and the distinction between mathematical guarantees and physical numerical precision.
  • Implementation validation: Conducted production-code GPU replays of six first-divergence cases, identifying execution-dependent numerical differences that explain the observed mismatches in these cases.
  • Watermark detection: Corrected the retained-count Gamma scoring procedure and conducted paired analyses with 100 bootstrap repetitions. At a common analytical threshold, SigFisher improves TPR from 43.30% to 45.65% at 64 tokens and 59.41% to 62.89% at 128 tokens, while strictly matched empirical-FPR improvements remain statistically uncertain.
  • Practical efficiency: Demonstrated up to 1.32× measured wall-clock speedup over direct PF decoding, extended the lookahead evaluation, and provided detailed computational and memory measurements. We also explicitly acknowledged the model-dependent speedups and limitations of cross-method comparisons.

We appreciate Reviewer WzbP's acknowledgment that most of their concerns have been addressed and that our paper is technically sound.

While acknowledging the remaining practical limitations, we believe that the exact speculative acceleration of PF decoding and the novel PF-specific detection theory constitute meaningful and complementary contributions to efficient and watermarkable LLM decoding.

We sincerely thank the Area Chair and all reviewers for their time, constructive feedback, and consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.