- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines whether the apparent detectability reduction of token-level multi-model generation arises from switching between models or from the accompanying generation procedure. The authors introduce pool-of-one controls, in which each model independently generates through the same chunked loop, and compare an original NLTK-re-spaced loop with a native-string loop. Experiments cover three English domains, classic and advanced model pools, an additional instruction-tuned pool, and multiple detector configurations. Under the primary GPT-Neo-scored Fast-DetectGPT setting, the classic pool shows no measurable switching advantage relative to same-loop controls. In the matched native loop, switching increases detectability for the advanced and instruction-tuned pools. Additional analyses examine score inversion, formatting, paraphrasing, quality, and computational cost. The principal contribution is a controlled evaluation showing that apparent multi-model evasion cannot automatically be attributed to model switching.
Critical strengths
-
The pool-of-one control is consequential and directly addresses the principal confound. Running each constituent model through the same generation loop is the appropriate control for asking whether switching itself changes detectability. This design separates switching from effects introduced by chunking, repeated tokenization, and string handling. Under the primary Fast-DetectGPT configuration, the control changes the interpretation from an apparent blending advantage to a near-zero effect in the classic pool and an increase in detectability for the advanced native-loop pool. This is a substantive result that supports acceptance as a measurement study.
-
The evaluation explicitly investigates score inversion and sensitivity to analysis choices. The paper recognizes that AUROC below 0.5 can indicate reversed separation rather than disappearance of information. It considers held-out direction selection, reports realized false-positive rates, and evaluates multiple scorers and formatting treatments. These diagnostics help prevent score inversion or formatting artifacts from being mischaracterized as general detector failure. Although the notation needs correction, the underlying attention to these evaluation issues is valuable.
Critical weaknesses
-
The original-loop attribution is not fully identified, and the broad concluding claim exceeds the cross-configuration evidence. The historical original-loop blends were generated in fp32, while the corresponding pool-of-one controls were regenerated later using half precision and batched sampling. The authors acknowledge this difference, but disclosure does not make the conditions equivalent. Consequently, the reversal between the original and native loops cannot be attributed exclusively to string handling without separating loop effects from execution-configuration effects. The broader suggestion that switching alone is not a route around current detectors also needs qualification. For example, Table 7 reports a native-loop advanced-SQuAD difference of −0.0312, with a reported 95% interval of [−0.0615, −0.0014], under the OLMo-2-7B Fast-DetectGPT scorer. This does not overturn the primary GPT-Neo and Binoculars findings, but it demonstrates that not all evaluated configurations follow the headline direction. The supported conclusion is that switching provides no consistent, configuration-independent evasion advantage, rather than that switching never reduces detectability in the evaluated setting.
-
The detectability statistic and the effective comparator need clearer definitions. The manuscript repeatedly combines the definition with a procedure that fits score direction on 20% of prompts and evaluates the remaining 80%. These are different statistics. The former is necessarily at least 0.5; the latter can fall below 0.5 because of direction-selection error or sampling variation. Table 7 includes a value of 0.4975 labeled as detectability for native classic XSum under OLMo-2-7B. This value need not be erroneous, but it cannot literally equal evaluated on that test set. The implementation, estimand, and notation should be reconciled.
Questions for authors
- Clarification of experimental matching. Are there results in which original-loop blends and pool-of-one controls were generated together using the same precision, software environment, sampling configuration, and matched seed protocol? If not, which conclusions are supported solely by the fully matched native-loop experiments, and which depend on the historical-versus-regenerated comparison?
- Clarification of the primary statistic. Do Tables 4, 5, and 7 report or held-out oriented AUROC? Please explain the values below 0.5 and provide a pooled pool-of-one comparison with a common score direction. Does this change the estimated switching effects or the advanced-versus-classic contrast?
- Clarification of null effects and conclusion scope. For the classic pool, what practically meaningful reduction can the confidence intervals exclude? Please distinguish failure to detect a difference from evidence of approximate equivalence. How should the overall conclusion be revised to incorporate the native-loop reductions observed under alternative scorers?
2 comments
Response to Reviewer tPBd (2/2)
Weakness 2: The detectability statistic and the effective comparator need clearer definitions.
Question 2: Do Tables 4, 5, and 7 report or held-out oriented AUROC? Please explain the values below 0.5 and provide a pooled pool-of-one comparison with a common score direction.
Question 2: Does this change the estimated switching effects or the advanced-versus-classic contrast?
They report the held-out oriented AUROC: the direction is fitted on 20% of prompts and the AUROC computed on the other 80%, so it can fall below 0.5. The 0.4975 cell is such a case: its full-sample AUROC is 0.5042 (Table 23) with the declared direction, a sampling effect near chance rather than a direction error. The only other value below 0.5 in Table 7, 0.4591 for the GPT-2 detector on the original-loop advanced SQuAD blend, also reflects direction selection near chance: its full-sample AUROC in the calibrated direction is 0.5221 (Table 3), and the direction fitted on the 20% split is the reverse one. The revision will give this statistic its own name and reserve D(A) = max(A, 1 − A) for the full-sample value, reporting both. The pool-of-one comparator in Tables 4, 5, and 7 is the mean of the four arms' held-out values, each arm with its own fitted direction (the P1 columns of Table 7), and the revision will define it next to the statistic. Under the GPT-Neo scorer, the blend and all four pool-of-one arms share one orientation in every native-loop cell, and the p = 0 rung of the redraw sweep is a pooled single-model reference (Table 22); orientations differ within 6 of the 48 zero-shot cells (four scorers, two loops, two pools, three domains). We computed pooled common-direction comparisons with intervals for every cell and for the RQ3 contrast. Under the GPT-Neo scorer, the classic null and the advanced native-loop increases keep their sign and significance under every estimand, and the RQ3 primary contrast stays positive (0.0548 [0.0306, 0.0788] with a pooled reference; 0.1050 [0.0859, 0.1246] with max(A, 1 − A) per arm). The original-loop advanced XSum reduction becomes 0.0283 [−0.0218, 0.0740] with a pooled reference, and two of the six native-loop exceptions do not hold with a pooled reference, as listed in Part 1. The revision will report every estimand beside the printed one.
Question 3: For the classic pool, what practically meaningful reduction can the confidence intervals exclude?
Under the GPT-Neo scorer, the classic-pool intervals exclude switching reductions larger than 0.0276 in the original loop and 0.0168 in the native loop (Table 4), whereas the 2024 classic blends have 0.0624 to 0.2190 lower Fast-DetectGPT AUROC than data-builder single-model text (Table 13). Relative to that drop, the intervals indicate approximate equivalence rather than only a failure to detect a difference. A two-one-sided equivalence test at a margin of 0.03, below the smallest of these reductions, supports equivalence in all six GPT-Neo classic cells (bootstrap TOST p ≤ 0.047), also with a pooled reference; the 90% intervals exclude effects larger than 0.0177 to 0.0297 in the original loop and 0.0123 to 0.0158 in the native loop. We set the margin after seeing the intervals, so these excluded margins are the primary readout. Under the outside-pool scorers and the supervised detectors, the excluded margins are wider in several classic cells (up to 0.0691 for Fast-DetectGPT with the Falcon-7B scorer), consistent with the listed exceptions. The revised conclusion will read: under the primary scorer, ToBlend's uniform switching adds no measurable evasion in the classic pool and raises detectability in the matched native loop for the advanced and instruction-tuned pools; across scorers and detectors, it provides no consistent, configuration-independent evasion advantage, with the listed exceptions.
Response to Reviewer tPBd (1/2)
Thank you for the careful reading and for regarding the evidence from the pool-of-one control as a substantive result that supports acceptance as a measurement study. This part answers Weakness 1 and Question 1; Part 2 answers Weakness 2 and Questions 2 and 3.
We adopt this formulation for the abstract and conclusion, restricted to ToBlend's uniform per-chunk scheme. On the original loop, the historical blends and the regenerated pool-of-one arms differ in execution configuration beyond precision (software, seeding, batching, and possibly the Llama-2 sampling configuration, unrecorded in 2024 and set to temperature 0.6 and top-p 0.9 in the regenerated arms), so the revision will associate the original-loop reversal with the loop and its execution configuration rather than with string handling alone. We will also run the Llama-2 pool-of-one arms at temperature 1.0 in both loops, which separates its sampling configuration from the loop effect, and we will include those experiment results in the camera-ready version. In the matched native loop, switching lowers detectability with the interval excluding zero in 6 of the 48 native-loop cells of Table 7, all under scorers or detectors other than GPT-Neo (largest −0.0397), in Table 16 (−0.0310), and under the Hello-SimpleAI formatting diagnostic in all three instruction-tuned cells (Table 11; −0.0532 to −0.1846), while it raises detectability in 17 of these 48 cells. With a pooled pool-of-one reference under the held-out estimand, two of the six Table 7 exceptions no longer hold (OLMo-2-7B advanced SQuAD, 0.0864 [0.0600, 0.1125]; Hello-SimpleAI advanced XSum, 0.0030 [−0.0297, 0.0336]), and the Table 16 SQuAD cell becomes a reduction under the full-sample estimands (−0.0230 [−0.0446, −0.0030]); the revision will list the exceptions under each estimand next to the conclusion.
Not in the submission. We will generate original-loop blends and pool-of-one arms of both pools together under one configuration, and we will include those experiment results in the camera-ready version. The Limitations section will carry the following map. Supported by the matched native loop: the classic null under GPT-Neo (0.0016 to 0.0050); the increases from switching in the advanced (0.0411 to 0.1049) and instruction-tuned (0.0271 to 0.0655) pools; RADAR scoring native blends and same-loop single-model text alike (differences of at most 0.0098); seed stability (Table 10); the redraw sweep (Table 22); and the classic pool without GPT-Neo (Table 16). Dependent on the historical-versus-regenerated comparison: the original-loop advanced reductions, the primary RQ3 contrast (0.0965), and the original-loop half of the classic null.