- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a DPO method to fine-tune an LLM and train a detector simultaneously, enabling the detector to detect text generated by the LLM and improving the LLM's downstream performance. It offers a potential new approach to machine-generated text detection.
Critical strengths
- Introducing DPO as a solution to achieve a better trade-off between detectability and downstream task performance of machine-generated text.
- Experiments cover many aspects, providing a comprehensive assessment of the method's applicability and potential limitations.
Critical weaknesses
- The potential use of the proposed method is limited. It requires fine-tuning an LLM on a specific task.
- The metrics for task performance evaluation and the DPO training appear to use the same metric. This may introduce a systematic bias in evaluation and be unfair to baseline methods.
- Using the OpenAI detector on the base model without PUPPET fine-tuning as Vanilla may exaggerate the improvement brought by PUPPET proposed by this paper. The authors may consider comparing other fine-tuning baselines such as [1],
- The trade-off between detectability and downstream task performance is a challenge for watermarking. But the method proposed in this paper is not watermarking. So, the paper should identify its own potential trade-off or limitation rather than compare it with those challenges of watermarking.
[1] Xiaojun Xu, Yuanshun Yao, Yang Liu. “Learning to Watermark LLM-generated Text via Reinforcement Learning.” 2024.
Questions for authors
- Are there any other task-irrelevant metrics that can be used to inspect the influence of fine-tuning an LLM with PUPPET, or compare it with an LLM without PUPPET?
- Do different tasks have different difficulty levels to become detectable via DPO? Can the authors provide any insight?
- How long of a text is used to evaluate the detectability and downstream performance in this paper?
- How do the authors calibrate the false positive rate of detection during evaluation?
4 comments
Response to Reviewer jzTt (2/2)
This is the second part of our response, covering the questions.
Q3.2
Yes. Across base LLMs and detectors, the detectability of Vanilla on IELTS is consistently lower than on the other tasks, which makes joint optimization more challenging (App. I). IELTS thus demands the largest AUROC gains (up to +28.5 points for Qwen3; §5.2) and more training samples to approach its ceiling (App. E). These results suggest that, when Vanilla is hard to detect, raising detectability is harder and tends to limit the gain in task performance.
Q3.3
All methods generate under the same limit on new tokens: 300 for ELI5 and 512 for Multi-News, following WaterBench [1], and 2,500 for IELTS, which covers 1.5 times the longest human essay. Detectability and task performance are measured on the full generated text.
The average length in words is as follows.
| Method | ELI5 | Multi-News | IELTS |
|---|---|---|---|
| Vanilla | 237 | 281 | 400 |
| KGW | 244 | 291 | 406 |
| PUPPET | 253 | 334 | 411 |
Q3.4
In the submission, the thresholds for 1% and 0.1% FPR were determined on the evaluation set itself, with its human references as negatives (200–600 texts; §5.3).
We increased the human negatives to 15,000 texts for ELI5 and 5,000 for Multi-News and recomputed TPR for every generated text. Brackets show 95% Wilson confidence intervals.
| Method | ELI5 TPR@1% | ELI5 [email protected]% | Multi-News TPR@1% | Multi-News [email protected]% |
|---|---|---|---|---|
| Vanilla | 48.5 | 33.3 | 47.0 | 16.5 |
| KGW | 94.8 [92.8, 96.3] | 86.3 [83.4, 88.9] | 93.5 [89.2, 96.2] | 73.5 [67.0, 79.1] |
| PUPPET | 95.5 [93.5, 96.9] | 89.3 [86.6, 91.6] | 93.0 [88.6, 95.8] | 75.0 [68.6, 80.5] |
PUPPET and KGW lie within each other's confidence intervals at both FPR levels on both tasks. We will replace Tab. 5 with this table.
References
[1] Tu et al. "WaterBench: Towards Holistic Evaluation of Watermarks for Large Language Models." ACL 2024.
Thank the authors for their response. My questions have been partially addressed, and I now have a better understanding of the underlying principles and mechanisms of PUPPET. I agree that the task-specific fine-tuning approach proposed by PUPPET has the potential to simultaneously improve both detectability and downstream task performance. However, I still feel that the current work lacks sufficient depth in analyzing the underlying sources of these gains as well as the potential trade-offs or costs involved. Therefore, I have decided to maintain my original score.
Response to Reviewer jzTt follow-up comment
Thank you for the follow-up. We would be glad to address your interest in PUPPET as fully as we can, and we respond to the two remaining points below.
Underlying sources of the gains
We read this point as the following question: why does applying PUPPET (DPO with a detection reward and a task reward) make the outputs of a model both more detectable and better on the task? If you have a specific analysis in mind, please let us know.
Changes in the outputs. We compared the outputs of Llama-3 on ELI5 before and after PUPPET (Fig. 2, §4; setup in App. A.6). After PUPPET, the coverage of the spans that the OpenAI Detector relies on increases by 1.8%, and the coverage of the spans that ROUGE-L rewards increases by 13%. This suggests that the model learns expressions that serve each objective, which makes its outputs more detectable and better on the task.
Compatibility of the two rewards. The reward ablation shows that both rewards are needed (Fig. 4, §5.4). We also analyzed the detector scores and the task scores of the candidate responses in the training data. Within each prompt, the two scores are nearly uncorrelated (Pearson r between −0.04 and +0.09 across four base LLMs and three tasks), so the two rewards do not systematically conflict. As a result, in about 90% of the preference pairs built with the combined score (§2), the chosen response is higher than the rejected one in both scores. DPO therefore steers the model toward outputs that satisfy both objectives.
Potential trade-offs or costs
The paper already discusses these points as summarized below. If any of them remains unclear, please tell us which one.
Trade-off between detectability and task performance. This trade-off is a central challenge in machine-generated text detection (§1). With joint optimization, PUPPET improves both detection and task performance over Vanilla on the three tasks (Tab. 1, §4). A trade-off remains along the reward weight α, but PUPPET occupies a more favorable region than Vanilla (App. I; W3.4 in our first response).
Task specificity: performance on other tasks and computational cost. PUPPET optimizes an explicitly given target task. On tasks not used in training, the gain in detection transfers without degrading task performance (Tab. 2, §5.1). We also observed no significant degradation on math, coding, and agent benchmarks after applying PUPPET (see our response to Reviewer fv4T on 19 Nov). PUPPET requires training for each target task, but the cost in data and compute is small (Fig. 5, §5.5), and a single model can also be trained on a mixture of tasks (App. J).
Response to Reviewer jzTt (1/2)
We thank you for recognizing PUPPET as a stable way to improve both detectability and task performance, and the coverage of our experiments. We would like to clarify one point in the summary: PUPPET does not train a detector. The detector is fixed, and only the LLM is fine-tuned (§2). We respond to each comment below. To address each point fully, we reply in two parts.
W3.1
We respectfully disagree and consider PUPPET sufficiently practical. PUPPET is designed to be fine-tuned per task, but the cost is small: the whole pipeline, from building the preference data to fine-tuning, fits within 12 hours, and fine-tuning itself takes 1–2 GPU hours (§5.5). Because the preference data are built offline, ablating α for a given use case requires no regeneration. Moreover, the gain in detectability generalizes to tasks not used in training (§5.1). Our results also indicate that a single model can be trained on a mixture of tasks (App. J).
W3.2, Q3.1
Watermarking methods do not optimize a task metric. Optimizing it explicitly, together with detectability, is what PUPPET contributes. The intended use case is therefore one where the metric to optimize is given.
As an additional analysis, to check that the gain is not specific to the reward metric, we evaluated with metrics that were not used as rewards.
Perplexity and BARTScore also show the improvement in task performance over Vanilla (BARTScore: Tab. 7, App. A). The model tuned with PUPPET is thus not overfitted to the given metric.
W3.3
Vanilla as the reference. PUPPET does not compete with the detector. Whether the detector is already strong or still weak, PUPPET boosts its detection performance while improving task performance. For Llama-3 on ELI5, Vanilla already shows high detectability, and PUPPET improves task performance without lowering it (Tab. 1). For Qwen3-8B on Multi-News, Vanilla shows relatively low detectability, and PUPPET largely improves it together with task performance (§5.2, Tab. 3). Fine-tuning alone does not explain this: the task reward alone provides no reliable gain in detection (§5.4).
Other fine-tuning baselines. As a training-based baseline, we added Wong et al. [2] on Llama-3. Each cell shows AUROC / task metric (ROUGE-L for ELI5 and Multi-News, judge score for IELTS).
PUPPET is above Wong et al. in both AUROC and task performance on the three tasks. Wong et al. also train with a detection objective and a quality objective, yet their task performance stays close to that of Vanilla. The results of PUPPET therefore do not follow trivially from existing training-based methods.
We did not run Xu et al. [3] because its 2023 implementation targets OPT-1.3B and Llama-2, and stabilizing its RL-based training on our models was not feasible within the rebuttal period.
W3.4
In machine-generated text detection, watermarking is the de facto standard, and its trade-off with task performance is the central challenge (§1). Our claim is that optimizing the LLM for both detectability and task performance can resolve this challenge and may open a new paradigm for machine-generated text detection.
As for the trade-off of PUPPET itself, the reward weight α balances the two objectives (§5.4, App. I): a trade-off exists along α, but PUPPET occupies a more favorable region than Vanilla, and it exceeds the watermarking baselines in task performance while matching their detection (Tab. 1). We will also state the limitations of PUPPET explicitly: its detectability is established empirically for a target detector, not guaranteed statistically as in watermarking, and robustness beyond naive paraphrasing is left for future work.
References
[1] Yuan et al. "BARTScore: Evaluating Generated Text as Text Generation." NeurIPS 2021.
[2] Wong et al. "An End-to-End Model for Logits-Based Large Language Models Watermarking." ICML 2025.
[3] Xu et al. "Learning to Watermark LLM-generated Text via Reinforcement Learning." arXiv 2024.