- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The work tests whether an AI-text detector can identify LLM output with simple numerical outputs. Using Pangram, FastDetectGPT and other methods, they identify this domain to contain large distributional differences from how humans traditionally choose numbers.
Critical strengths
- Number sequences is a pretty interesting domain to consider; the number of experiments and breadth of experiments are very much appreciated.
Critical weaknesses
- The paper is pretty descriptive. It discusses what changes causes the detector's score to change - very much like a brute force adversarial search. Most experiments are based on next-digit fit rather than on which mechanism drives detection. I think this is a pretty good empirical study but it does not help the reader understand why exactly these distributional difference arises in LLMs.
Questions for authors
- Do the authors have an intuition on why do spaces or newlines remove about 90% of the signal?
- I'd also love to see which phase of training / data causes this distributional difference in number sequences.
2 comments
Response to Authors
Thank you for the detailed rebuttal. I think something worth adding to this paper would be some sort of mechanistic study on these biases and why exactly distribution differences arose, as stated in my weaknesses.
As a result, I will keep my score.
Hello there, thank you for taking the time to consider our paper. We would like to sincerely apologize for the late response within the discussion period.
Q1: Intuition on New Lines and Spaces
We thank the reviewer for raising this question. While we observe that spaces and newlines substantially reduce the detection signal, we do not currently have a clear mechanistic explanation for this effect.
Q2: OLMo Phase Experiment
Thank you for your suggestions! We conducted an experiment based on numbers generated by different checkpoints of publicly released OLMo 2, covering intermediate pretraining checkpoints where available, final base models, SFT and DPO checkpoints, and final instruction-tuned models.
We held sampling fixed at temperature 0.7 and top-p 0.8, evaluated continuation and chat prompts separately, and measured eight sequence statistics alongside Pangram 3.3.2 scores. We collected 100 complete 100-digit sequences per successful condition, totaling 1,000 sequences.
The training stages are as follows:
The results are as below:
Table 1: Detectability and digram entropy
\textbf{Training stage} & \textbf{Pangram mean} & \textbf{Digram entropy (nats)} \\\\ \hline \text{Pretraining (1.0T)} & 0.041 / 0.094 & 1.65 / 1.66 \\\\ \text{Pretraining (3.9T)} & 0.195 / 0.223 & 1.56 / 1.53 \\\\ \text{Final Base} & 0.168 / 0.038 & 1.57 / 1.68 \\\\ \text{SFT} & 0.908 / 0.889 & 0.43 / 0.97 \\\\ \text{DPO} & 0.867 / \text{---} & 0.17 / \text{---} \\\\ \text{Final Instruct} & 0.923 / \text{---} & 0.12 / \text{---} \end{array}$$ **Table 2: Step behavior and autocorrelation** $$\begin{array}{l|c|c} \textbf{Training stage} & \textbf{Mean step} & \textbf{Lag-1 correlation} \\\\ \hline \text{Pretraining (1.0T)} & 3.09 / 3.30 & -0.101 / -0.090 \\\\ \text{Pretraining (3.9T)} & 3.48 / 3.55 & -0.127 / -0.116 \\\\ \text{Final Base} & 2.99 / 2.74 & -0.079 / -0.046 \\\\ \text{SFT} & 3.77 / 3.71 & -0.168 / -0.142 \\\\ \text{DPO} & 3.88 / \text{---} & -0.183 / \text{---} \\\\ \text{Final Instruct} & 3.92 / \text{---} & -0.196 / \text{---} \end{array}$$ **Table 3: Coverage and digit preference** $$\begin{array}{l|c|c} \textbf{Training stage} & \textbf{Coverage (draws)} & \textbf{Favorite digit (\\%)} \\\\ \hline \text{Pretraining (1.0T)} & 83.82 / 37.37 & 18.11 / 15.41 \\\\ \text{Pretraining (3.9T)} & 10.38 / 19.72 & 16.16 / 19.16 \\\\ \text{Final Base} & 42.66 / 73.74 & 18.38 / 19.53 \\\\ \text{SFT} & 10.00 / 10.00 & 12.02 / 11.99 \\\\ \text{DPO} & 10.00 / \text{---} & 10.59 / \text{---} \\\\ \text{Final Instruct} & 10.00 / \text{---} & 10.89 / \text{---} \end{array}$$ **Table 4: Repetition and step behavior** $$\begin{array}{l|c|c} \text{Training stage} & \pm 1\\ \text{steps }(\\%) & \text{Immediate repeats }(\\%) \\\\ \hline \text{Pretraining (1.0T)} & 21.03 / 21.49 & 4.33 / 1.08 \\\\ \text{Pretraining (3.9T)} & 19.45 / 18.93 & 2.81 / 4.60 \\\\ \text{Final Base} & 21.68 / 21.08 & 6.90 / 11.88 \\\\ \text{SFT} & 14.62 / 18.95 & 0.28 / 0.01 \\\\ \text{DPO} & 11.32 / \text{---} & 0.19 / \text{---} \\\\ \text{Final Instruct} & 11.25 / \text{---} & 0.22 / \text{---} \end{array}$$ **Table 5: Complete ten-digit block structure** $$\begin{array}{l|c} \textbf{Training stage} & \textbf{Complete 10-digit blocks (\\%)} \\\\ \hline \text{Pretraining (1.0T)} & 1.10 / 7.10 \\\\ \text{Pretraining (3.9T)} & 9.20 / 7.80 \\\\ \text{Final Base} & 5.40 / 0.30 \\\\ \text{SFT} & 61.70 / 53.00 \\\\ \text{DPO} & 87.70 / \text{---} \\\\ \text{Final Instruct} & 87.50 / \text{---} \end{array}$$ *Each cell reports continuation / chat. “—” indicates that no complete 100-digit sequence was obtained within 500 generation attempts.* From this OLMo 2 experiment, we can see that distributional biases are already present during pretraining and become substantially stronger during the Base-to-SFT transition, alongside a sharp increase in detectability. These biases persist through DPO and RLVR, suggesting that SFT is a major stage of amplification.