- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies content-source detection under changes in expression and recurring differences between human and LLM content choices. Author propose converting documents into detailed content points restated in a neutral register, then using these as inputs to existing detectors, which show improved detection robustness to paragraph and voice changes. Meanwhile, they propose Content Repertoire Occupancy (CRO) learns overlapping content categories from topic-centered embeddings, which reveals population-level patterns that human writing more often includes concrete details, while LLM writing more often emphasizes significance and shared values.
Critical strengths
-
Clear problem framing. The distinction between “content source” and “expression source” is insightful, and the proposed content-point strategy offers an intuitive approach to improving robustness.
-
Thoughtful CRO design. Separately validating whether population-level differences replicate and whether category descriptions are meaningful is more convincing than simply presenting a few plausible cluster names.
Critical weaknesses
-
The incremental contribution of content-point decomposition to detection robustness is not fully established. Appendix C.2 provides a useful comparison with detailed plain rewriting, but this ablation has a narrower scope than the main experiments. In Table 10, plain rewriting achieves nearly identical mean AUROC to content points across the 12 test-only expression-change settings (0.965 versus 0.964), suggesting that neutral restatement already recovers much of the observed robustness. Content points do reduce mean FPR (0.045 versus 0.067), although this accompanies lower TPR (0.852 versus 0.890). However, this comparison is restricted to RoBERTa on HART and does not cover training-side changes or joint train–test changes,. It therefore remains unclear how much additional detection robustness point decomposition provides over detailed neutral rewriting across the broader experimental setting.
-
Sensitivity to the choice of content extractor remains unclear. Although the detection experiments cover multiple detectors and expression transformations, all content-point representations are produced by the same GPT-4o extractor. The results therefore leave open how sensitive the reported robustness gains are to this model’s particular content-selection and restatement behavior.
Questions for authors
See weakness.
3 comments
Robustness Across Content-Point Extraction Models
2. Robustness across extraction models (new Appendix K).
Strong content-source discrimination persists when different models construct the content points. We vary the extraction model while holding the extraction instruction and remaining experimental setup fixed, using the same four domains, three detectors, and 36 expression-change settings.
| Extraction model | Mean AUROC | Mean TPR | Mean FPR |
|---|---|---|---|
| GPT-4o | 0.956 | 0.855 | 0.064 |
| Llama 3.3 70B Instruct | 0.948 | 0.824 | 0.069 |
| Qwen3.8-27B | 0.943 | 0.808 | 0.065 |
All three extractors support strong ranking performance and similarly low false-positive rates, with some variation in recall. Appendix K presents the detector-specific setting comparisons and examples of the extracted content.
A matched video-prediction example makes the extraction behavior concrete. The source passage explains why predicting leg movement requires distinguishing walking from running. The corresponding content point from each extractor is reproduced verbatim below.
| Extractor | Corresponding content point |
|---|---|
| GPT-4o | For example, predicting future leg movements requires knowing whether the partial motion corresponds to walking or running. |
| Llama 3.3 70B Instruct | For instance, to predict the future of leg movement, it is necessary to know whether the partial leg movement belongs to walking or running. |
| Qwen3.8-27B | Predicting future leg movement requires determining whether the observed partial movement is part of walking or running. |
Each extractor retains the walking/running distinction despite differences in wording. The example illustrates preservation of the same substantive information; the broader performance comparison shows that strong detection persists across the three tested extractors.
We hope these additional results address your concerns about the role of decomposition and demonstrate the robustness of our findings across extraction models.
Thank You and Evaluations for Your Final Consideration
We especially appreciate your recognition of the distinction between content and expression sources, as well as CRO's careful validation of population differences and feature descriptions.
We believe our substantial additional experiments further strengthen the evidence for our content-source detection approach. Our expanded comparisons across detectors and expression-change settings demonstrate that both our detailed plain rewriting and content points achieve strong voice robustness by preserving substantive detail and neutralizing expression, while content-point decomposition provides substantial additional robustness to paragraph changes. Together, these results support our overall representation design, beyond just the performance difference between the two rewrites. Furthermore, our experiments with three different extraction models show consistently strong detection performance, supporting the robustness of our approach beyond a single extractor.
We sincerely hope this additional evidence has strengthened your assessment of our work. If you feel it has addressed your concerns, we would be very grateful if you would take these into account in your final assessment.
Thank you again for your time and consideration!
Detailed Plain Rewriting and the Additional Benefits of Content-Point Decomposition
Thank you for recognizing the distinction between content and expression sources and the value of separately validating CRO's population differences and descriptions.
1. Detailed restatement and decomposition across the broader evaluation (new Appendix I).
Our detailed plain rewrite inherits the content-point extractor's specific design: retain every substantive claim, detail, and example while replacing distinctive expression with a common, neutral register. Both formulations were developed in this work, with instructions distinct from the prior rewriting prompts evaluated in Appendix J. Their key difference is structural: the plain rewrite retains connected prose, while the extractor decomposes the content into self-contained points.
Following your suggestion, we extend the comparison to four HART domains, all three detectors, and 36 expression-change settings covering training-only, test-only, and joint changes. The broader evaluation shows that both formulations provide strong voice robustness, while decomposition adds substantial robustness to paragraph changes.
The results below average RoBERTa, RACE, and DETree. TPR and FPR use thresholds calibrated to 5% human-content validation FPR and held fixed for the associated test evaluations.
The benefit of decomposition is especially visible under joint paragraph changes. Reversing the paragraph–source assignment between training and testing can reverse plain-rewrite rankings, while content points retain strong discrimination. Across the broader evaluation, the reduction in mean FPR accompanies similar mean TPR, alongside improved AUROC. In Appendix I, we report the full detector-specific results across settings.
The plain rewrite's strong voice performance provides further support for the shared content-preservation and restatement design. Under the matched RoBERTa evaluation, it achieves mean voice-change AUROC of 0.956, compared with 0.858 for TruthOverTricks Summary and 0.823 for HART Simplify. We discuss these comparisons with prior prompts in our response to Reviewer wT2Z and Appendix J.