- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents an empirical study of text generated through token-level blending across multiple LLMs. It introduces a benchmark with generation logs to evaluate detection, blend identification, and token-level attribution. Experiments, over two four-model pools and three domains, show good detection performance, but limited ability to distinguish blended from single-model text or attribute words to their generating models.
Critical strengths
The matched generation setup and detailed source logs help assess whether accurate text detection is accompanied by reliable source attribution.
Critical weaknesses
- The paper does not clearly establish what new findings its token-level benchmark adds beyond related multi-LLM authorship research, such as the uncited CollabStory (NAACL Findings 2025).
- The findings are based on only two fixed pools of older, small models with full probability access. Their relevance to modern post-trained models and less restrictive access settings remains unverified.
Questions for authors
- In Contribution (3), the paper presents 41.3% as a bound on word-level provenance recovery. Does the bound refer to the best observed attribution accuracy, or an upper limit on recoverable provenance? If the former, what new insight does this result provide, and why is it highlighted as a main contribution? If the latter, what evidence supports this interpretation?
- Attribution results confound model identity with decoding settings. How much of Llama-2’s higher attribution accuracy comes from its different decoding settings? Does the gap persist when all models use the same setting?
2 comments
Thanks for the response. For Weakness1, the response addresses my concerns. But for Question1, my main concern about the recovery bound remains unsolved. It is unclear wheather the 40.50% accuracy at TL1 setting is an estimate of the best available accuracy or due to the method’s limitation? This point is important to the Contribution (3). Thus, I keep my original rating.
Response to Reviewer 5KLn
Thank you for noting that the matched setup and source logs let us test whether accurate detection comes with reliable attribution.
Thank you for pointing to CollabStory [1], which we will cite and discuss. Its stories are written by up to five instruction-tuned LLMs, one sequential part of 180 to 900 words each, for authorship tasks such as single- versus multi-author classification. Our setting differs in granularity (the generator is redrawn for every chunk of one to five NLTK tokens and changes at 74.8% of adjacent word pairs at TL1), in control (the fixed-model arm shares prompts, loop, formatting, decoding, and stopping rule with the blended arm; Section 3.4), and in ground truth (every step's generator is logged, and white-box likelihoods give the per-chunk Bayes rule at TL1, up to NLTK re-spacing). The findings are complementary: CollabStory reports at least 0.8 F1 for single- versus multi-author classification at segment level, whereas at token level with matched pipelines the tested detectors separate blended from single-model text with AUROC of at most 0.58.
Full access is the setting most favorable to the defender (Section 1): for attribution, an attributor with less access cannot exceed, in expectation, the Bayes accuracy of the full-access attributor for the same process. For blend specificity, detectors that require no pool likelihoods remain low, at 0.5443 for RoBERTa, 0.5357 for Binoculars, and 0.5179 for Fast-DetectGPT with the outside-pool OLMo-2-7B scorer (Tables 22 and 27), whereas the white-box likelihood-ratio test in our response to Reviewer gvSy separates the arms; detection itself requires no likelihood access (text-only RoBERTa: median TPR 0.975 and 0.953 at a 5% FPR threshold; Table 21). We will add a matched run with an instruction-tuned pool (Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, Gemma-2-9B-it), and we will include those experiment results in the camera-ready version. Model scale remains a stated limit (nominal size 7B or less; Section 7); the instruction-tuned pool extends it to 9B, and the Limitations will keep scale as a boundary of the claims.
At TL1 it is both, at the chunk level. Because each step's generator is drawn uniformly and independently of the text and of earlier draws, the exact-context attributor is the Bayes rule for this process up to NLTK re-spacing (see our response to Reviewer gvSy), so its chunk accuracy, 40.50% against 25% chance, estimates the upper limit for this pool, these decoding settings, and one-token chunks. The paper's 41.3% is the word-level accuracy of this rule (41.25% [40.65%, 41.84%]) and of the sampling-aware attributor (41.31%); the revision will state the bound at the chunk level and drop TL3 from it. With every pool model, its decoding settings, and the true chunk boundaries known, the optimal rule still misattributes most chunks: provenance is lost in the generation process itself rather than through a weak tool. It is a main contribution because it separates the two problems on the same texts: the frozen reader detects these blends with AUROC 0.9506 (Table 4), yet even the Bayes rule recovers 40.50% of chunks, so a detection alone says little about which models wrote a passage (Section 6).
Most of it. Llama-2 alone was sampled with its shipped temperature 0.6 and top-p 0.9 (Appendix B). Without the decoding transform it is the hardest member to attribute (31.9%, against 78.8% with it; Table 26), and the net gain from 36.53% to 41.31% comes entirely from Llama-generated words. In the classic pool, whose members share one configuration, the transform leaves accuracy unchanged (38.20% versus 38.19%; Table 28), so we do not expect the gap to persist under matched settings. We will run the advanced pool with matched decoding and include those experiment results in the camera-ready version.
References
[1] Venkatraman, S., Tripto, N. I., & Lee, D. (2025). CollabStory: Multi-LLM collaborative story generation and authorship analysis. In Findings of the Association for Computational Linguistics: NAACL 2025 (pp. 3665–3679). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.findings-naacl.203