- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper generalizes Turing’s imitation game to arbitrary interactive agents. In the Generalized Turing Test GTT(A, B), an instance of B must decide whether its interlocutor is another B or an A instructed to imitate B. A ≥_ε B holds if B’s success probability is at most 1/2 + ε. The authors study when this comparator is transitive and introduce three variants: the imitator may first query the target (GTTQ), the distinguisher is a third agent (UTT), or the number of interaction rounds is bounded. They also define Turing Scores that aggregate pairwise outcomes within a finite set of agents. Experiments on nine LLMs from early 2026 (3,105 GTT and GTTQ trials) show three things: the resulting scores broadly agree with AAII, LiveBench and Arena rankings; the empirical relation is only partly transitive; and distinguishers rely on both identity/style cues and substantive capability probes.
Critical strengths
-
The framing is original and conceptually clean. Turning the Turing test into an indistinguishability-based relation between agents, and asking whether that relation is transitive, yields a simple formal object with a natural family of variants. The separation between the GTT and the GTTQ is a good example. The test needs no fixed task set and becomes harder as both imitators and judges improve, which speaks directly to the saturation and contamination of static benchmarks. This is the main reason I think the paper deserves serious consideration.
-
The empirical study is carefully documented. The authors release prompts, transcripts and an interactive explorer. They test prompt paraphrases and style-controlled instructions, with a manipulation check and Newcombe–Wilson intervals, and they report results that work against their own thesis. That a pairwise game without any task set produces an ordering close to that of AAII is a useful observation in its own right, even though its interpretation is less clear than the paper suggests.
Critical weaknesses
-
The claim that Turing Scores recover a capability ordering is not adequately established, and it is unclear what the GTT measures in practice. The GTT T scores agree well with AAII (Kendall’s τ ≈ 0.89). However, the paper reports no rank-correlation statistic, Table 1 gives no intervals for T, F or D, and the top four T scores lie within about 0.06 of each other. F, moreover, agrees with AAII only at τ ≈ 0.61 and ranks DeepSeek V3.2 second. Several distinguishers are degenerate. GPT-5.4 accepts its own copy only about one time in five, so even Ministral 8B passes the GTT against it at ε = 0.06, and only Gemini, Opus and Sonnet distinguish clearly better than chance. Because the scores are raw averages rather than estimates from a model of judge bias, such judges distort them: more than half of GPT-5.4’s F lead over DeepSeek V3.2 comes from GPT-5.4 never having to face its own, very strict column. The protocol adds further confounds. Footnote 1 and the self-control rows of Table 2 indicate that the self-control interlocutor is itself given the imitation prompt, unlike the plain instance of B in Definition 1. Distinguishers are also not told the 1/2 prior, and the imitator is told the target’s name whereas the distinguisher is not told its own. The style-control evidence is mixed. Ministral 8B is still detected without identity and style cues. Opus’s accuracy against a Qwen3-32B imitator, however, drops from 0.96 to 0.28 under the style-control prompt and recovers only to 0.60 with hard probes. Gemini’s accuracy against the same imitator is not significantly affected by style control (0.88) but falls to 0.60 with hard probes. Without non-imitating or name-only baselines, it remains unclear how much of the ordering reflects capability rather than identity fingerprints and judge bias.
-
The theory is thin and does not deliver the ordering it motivates. Propositions 1, 4, 5 and 7 each follow in a line or two. The Turing classes of Proposition 2 require transitivity at a fixed threshold, which the transitivity theorems provide only under degenerate conditions. For ε > 0, neither bound recovers the original threshold, since ε′ ≥ α + β even when ζ = 1. At the ε = 0 threshold used in the main figures, Theorem 3 requires γ = 0 (statistically exact imitation of C’s distinguisher by B) and δ = 0, while Theorem 6 requires ζ = 1, which can hold for at most one third agent C. Theorem 3 additionally degrades with the size of the universe. The recursion probabilities over different C sum to at most one, so some C has ζ ≤ 1/(n−1), and the bound becomes ε′ ≥ (n−1)α, i.e., 0.4 for nine models with α = 0.05. Moreover, the statement of Theorem 3 omits an assumption that its proof uses (footnote 3): that the distinguisher is no worse than chance when it does not recurse. Without it, the same argument yields the ½(1/ζ − 1) term of Theorem 6. Separately, the proof applies per-message statistical closeness (Definitions 4 and 6) to the outcome of an entire multi-round interaction. That step requires either a factor equal to the number of rounds or a transcript-level assumption. The authors acknowledge that statistical indistinguishability is not directly testable, and they do not claim that the models studied satisfy these assumptions. Finally, the password attack of Proposition 5 extends to a challenge–response scheme based on a pseudorandom function, which no polynomial number of specimen queries can defeat. A developer could therefore cheaply make a model inimitable in the GTT and GTTQ. A UTT whose distinguisher has no access to the target is immune to this attack, but such a judge must then rely on its prior knowledge of each target, which is the identity channel discussed in Weakness 1.
Questions for authors
-
Suppose the existing pairwise data are re-analyzed with a model that separates imitator ability, distinguisher discriminability and distinguisher bias, such as a Bradley–Terry or IRT model. Does the agreement with AAII, LiveBench and Arena persist, and how stable is the within-tier ordering? Please report Kendall’s τ with bootstrap intervals for T, F and D separately, together with the MOVER intervals described in Appendix B.3, and compare against simple proxies such as price and release date. Eliciting confidences in a rerun would additionally allow AUC-based scores that do not depend on each distinguisher’s decision threshold.
-
Does the ordering in Table 1 survive if the Gemini, Opus and Sonnet distinguisher columns of the Figure 2 matrix are rerun in full under the style-controlled prompts (Variants 2 and 3), and the Turing Scores are recomputed? The current style-control results cover only a few cells and point in different directions. A known-answer validation would address the same question more directly. It could use a model family whose capability ordering is known in advance, such as different sizes of one architecture or checkpoints along one training run, since such models share a style.
-
How do the results change if the self-control interlocutor is an uninstructed instance of B, as in Definition 1, and distinguishers are told the 1/2 prior? Opus and Gemini accept their instructed copies at 0.975 and 0.95, so the instruction by itself cannot explain GPT-5.4’s rate of about 0.18; only such an ablation can separate the possible causes. It would also help to add two baselines: an actor that answers as itself without imitation instructions, and an actor that only claims the target’s identity.
-
Can the transitivity results be stated at a fixed threshold, in particular at ε = 0 as used in Figure 2, without requiring γ = δ = 0 or ζ = 1? Relatedly, does Theorem 3 hold for multi-round interactions under the per-message closeness of Definition 6, and can the assumption in footnote 3 be moved into the theorem statement? The transcripts could also show whether any distinguisher ever initiates a nested GTT, which would indicate whether Turing-recursivity is more than a formal device.
The idea is original and the paper is well written. However, the evidence does not yet separate capability from judge bias and identity fingerprints, and the theory does not deliver transitivity at the thresholds used in the experiments. Convincing answers to Questions 1–3 would move me to "3: Weak acceptance",
4 comments
Thank you for the rebuttal and the additional experiments, which address my questions directly: intervals added (full-universe T–AAII 0.89 [0.70, 0.94]; frontier-only intervals include zero); full-column style controls preserving the top-six ordering; the genuine-control/prior ablation quantifying the protocol confound (GPT-5.4 T 0.716 → 0.506, frontier order reversed); actor baselines separating imitation from identity (44.2%/24.2%/11.3%); and a corrected Theorem 3.
Two concerns remain: no decomposition of imitator ability, distinguisher discriminability, and judge bias was performed, and the primary 9×9 table is still produced under the prompted-self protocol; the within-family null limits capability claims to coarse stratification.
Given the novelty of the formulation, the released artifacts, and claims recalibrated to the evidence, I will raise my rating by one point, from 2 to 3, conditional on the revised paper implementing the stated scoping, dual-protocol reporting, and corrected Theorem 3.
Dear reviewer, thank you for responding to our rebuttal so quickly, and for increasing our score. We have taken the last day + night to run a few additional experiments in line with your remaining concerns: the first was easy as it re-used existing data, but the last 2 are new experiments. Please take a look, as we believe these findings sharpen our claims. In any case, we are deeply thankful to you, as your input led us to refinements and additional results that have significantly improved our paper.
Experiment 1: Separating imitation ability, judge discrimination, and acceptance bias.
As an additional analysis of the 9x9 table, to try to explain the results using the 3 separate characteristics of each AIs, we fit regularized logistic models that distinguish: actor imitation effects, judge discrimination, and tendency to accept B as the same model. Allowing judge-specific sensitivity to actor ability improved mean held-out log loss from 0.412 to 0.372 across 100 splits of trials within each cell.
The fitted effects differed across roles. In the flexible model, GPT-5.4 had the largest standardized imitation effect (1.40; 95% bootstrap interval [1.10, 1.69]), alongside a negative acceptance-bias estimate (−2.01 [−2.95, −1.53]), indicating a tendency to reject same-model claims. Gemini and Opus had the largest estimated separations between genuine controls and imitators.
Setting the fitted acceptance-bias term to zero yielded positive rank correlations with AAII, LiveBench, and Arena Text: respectively, Kendall’s τ = 0.667, 0.200, and 0.500 under the additive model, and 0.778, 0.467, and 0.643 under the flexible model. However, both models retain significant lack of fit (parametric-bootstrap p = 0.002), showing actor performance can approximate actor/judge estimates, but particular actor-judge pairings may behave differently from what general estimates predicts.
Experiment 2: Replacing prompted self-controls with unprompted genuine controls. We collected 20 fresh self-control trials for each of the nine models, removing the actor’s imitation instruction:
\hline \textbf{Model} & \textbf{Genuine-self acceptance} & \textbf{Original T} & \textbf{Revised T [95\\% CI]} \\\\ \hline \text{Gemini 3.1 Pro} & 20/20 & 0.769 & 0.781\\;[0.731,\\,0.802] \\\\ \text{Claude Opus 4.6} & 20/20 & 0.747 & 0.753\\;[0.703,\\,0.772] \\\\ \text{Claude Sonnet 4.6} & 20/20 & 0.708 & 0.718\\;[0.666,\\,0.742] \\\\ \text{GPT-5.4} & 3/20 & 0.729 & 0.716\\;[0.677,\\,0.771] \\\\ \text{DeepSeek V3.2} & 12/20 & 0.656 & 0.606\\;[0.544,\\,0.658] \\\\ \text{Grok 4.20} & 18/20 & 0.571 & 0.566\\;[0.509,\\,0.601] \\\\ \text{Mistral Large} & 16/20 & 0.471 & 0.471\\;[0.411,\\,0.513] \\\\ \text{Qwen3 32B} & 18/20 & 0.454 & 0.449\\;[0.393,\\,0.484] \\\\ \text{Ministral 8B} & 16/20 & 0.445 & 0.395\\;[0.337,\\,0.436] \\\\ \hline \end{array}$$ Intervals are 95% confidence intervals from Wilson intervals using MOVER. Main finding: T scores retained positive correlations with AAII (τ = 0.833), LiveBench (0.467), and Arena Text (0.643), over nine, six, and eight matched models, respectively. In fact, only GPT-5.4 and Sonnet exchanged positions in the point-estimate ordering; their confidence intervals overlapped substantially. **Experiment 3: Another foray into fixed-family** The previous experiment we wrote for the first rebuttal, running the standard GTT between 3 very closely matched Qwen models, was inconclusive. However, we realized in responding to point 1 above (logistic model separating actor/distinguisher ability) that for such close models, a version of the GTT focusing on *one* sub-ability may be better. We used Gemini as the fixed external judge to compare the three Qwen models *only as actors/imitators*. This gives the three actors a common reference and focuses the comparison on their imitation performance. Excitingly, we find acceptance increased with model size: 30% for Qwen 35B, 32% for Qwen 122B, and 35% for Qwen 397B. Recall that these models had *very close* benchmark performance (85.3/86.7/87.8, obtained with extensive fixed-dataset questioning). The observed order matches the expected capability order in this fixed-judge test. Overall we see this result as an *initial foray* into running generalized Turing Tests (and importantly its variants, which may be better suited versions) on fixed families – we think a deeper investigation is a great line for future work, as our paper opens these questions.Thanks, please update the submission and I will raise the score
Thank you for recognizing the originality of our framework and the value of its empirical exploration. We are pleased to report that over the past week, we completed the requested full-column prompt controls, the two actor baselines, and the joint genuine-control/prior ablation, alongside additional uncertainty analyses. These substantially extend the evidence while identifying limits that we state more precisely.
This rebuttal is a summary, and in several places we say that we rewrote something in the paper – see this anonymized Google doc for the full line-by-line changes: https://docs.google.com/document/d/1f_qoHb20qOj-sMoIs0hclsNU2Dxv_KnVYKzGXjxFSA4/edit?usp=sharing
1. Score uncertainty and judge bias. The original GTT’s Kendall correlations and 95% trial-resampling intervals are:
$ \begin{array}{lccc} \hline \textbf{Leaderboard} & \mathbf{T} & \mathbf{F} & \mathbf{D} \\ \hline \text{AAII (9)} & \mathbf{0.89\;[0.70,\,0.94]} & 0.61\;[0.54,\,0.83] & 0.67\;[0.44,\,0.78] \\ \text{LiveBench (6)} & \mathbf{0.60\;[0.33,\,0.87]} & 0.20\;[0.07,\,0.73] & 0.33\;[0.14,\,0.60] \\ \text{Arena Text (8)} & \mathbf{0.71\;[0.50,\,0.79]} & 0.21\;[0.14,\,0.57] & 0.71\;[0.43,\,0.79] \\ \hline \end{array} $
Agreement is strongest for T, but frontier-*only* T intervals include zero for all three leaderboards, so they do not establish a fine-grained ordering (see “Leaderboard agreement” in our response to Reviewer 2). We added the score-level MOVER intervals to Appendix (see google doc).
2. Full-column style controls. We reran all nine actor rows for each requested judge—Gemini, Opus, and Sonnet—under both V2 and V3; the first six positions remain unchanged in both variants, with Qwen moving ahead of Mistral. Please see “Style/identity controls” in our Reviewer 2 response (who had the same request).
Next we ran the requested within-family test on Qwen3.5-35B/122B/397B, which Reviewer 2 requested too. Please Control-F for “Family of similar models”. This was less decisive: all six pairwise advantage intervals include zero under both the original and V2 prompts. We report this limitation rather than claiming a resolved ordering of extremely closely matched models.
3. Genuine controls, prior, and actor baselines. We reran the frontier block with uninstructed genuine actors and an explicit 50–50 prior, 40 trials per cell. Recomputing scores within this three-model universe gives:
$ \begin{array}{lcc} \hline \textbf{Model} & \textbf{Original F/D/T} & \textbf{Revised F/D/T} \\ \hline \text{Opus} & 0.225\;/\;0.625\;/\;0.425 & 0.188\;/\;0.719\;/\;0.453 \\ \text{Gemini} & 0.250\;/\;0.694\;/\;0.472 & 0.100\;/\;0.563\;/\;0.331 \\ \text{GPT-5.4} & 0.850\;/\;0.581\;/\;0.716 & 0.300\;/\;0.713\;/\;0.506 \\ \hline \end{array} $
The point ordering changes from GPT-5.4 > Gemini > Opus to GPT-5.4 > Opus > Gemini. These results demonstrate protocol sensitivity. We will distinguish the original prompted-self implementation from Definition 1’s uninstructed genuine branch and report both protocols explicitly.
Finally, your suggestion for actor baselines provides a clearer intervention result:
\hline \textbf{Actor instruction} & \textbf{Accepted / trials} & \textbf{Acceptance} & \textbf{Decrease from full imitation} \\\\ \hline \text{Full imitation} & 106/240 & \mathbf{44.2\\%} & \text{—} \\\\ \text{No imitation prompt} & 27/240 & 11.3\\% & \mathbf{32.9}\\;\textbf{pp}\\;\mathbf{[26.4,\\,38.5]} \\\\ \text{Identity claim only} & 58/240 & 24.2\\% & \mathbf{20.0}\\;\textbf{pp}\\;\mathbf{[13.4,\\,26.1]} \\\\ \hline \end{array}$$ Thus, explicitly attempting to imitate the target has a substantial effect beyond answering normally or merely claiming its identity. This establishes a contribution from the imitation instruction. **4. Theory.** Thank you for identifying the interaction-level gap. We replaced per-message closeness in the relevant assumption with statistical closeness of complete interaction transcripts, and moved the nonrecursive “at least chance” assumption into Theorem 3’s statement. We will also state explicitly that the results are **error-accumulating composition bounds**, not proofs of fixed-threshold transitivity for the empirical graphs. The theoretical contribution is to identify conditions under which imitation composes when both the judge and the actor’s target change—an implication that is not automatic from fixed-test indistinguishability. These conditions are sufficient, not necessary. We reiterate that our paper introduces these theoretical questions for the first time, and finds the first non-trivial condition set under which transitivity holds. Our paper leaves open the theoretical CS question of other conditions under which the Turing test is transitive.