acceptodds
Under review as a conference paper at ICLR 2027

Generalizing the Turing Test to Interactive Agents

Abstract

We initiate the study of the *Generalized Turing Test* (GTT), a formal generalization of Turing’s imitation game from humans to arbitrary interactive *agents*. For agents and , passes the GTT against if an instance of , acting as a distinguisher, cannot reliably distinguish an instructed to imitate from another instance of ; if so, we write . We study the theoretical and empirical consequences of this idea. On the theory side, we prove sufficient conditions under which this "Turing Comparator" is transitive. We introduce natural variants with *querying* (the imitator can first interact with a specimen of the target), a *Universal Turing Test* with arbitrary distinguishers and targets, and complexity-theoretic variants that control interaction length. As a proof of concept, we evaluate the GTT and its variants across nine large language models. Remarkably, Turing Scores recover a clear model stratification consistent with standard external benchmarks despite being derived entirely from pairwise imitation games. Transcript analysis reveals that models use both stylistic signatures and substantive STEM and logic-based probes. Together, these results suggest indistinguishability could provide a meaningful signal for comparing agents, yielding an inherently adaptive form of evaluation that does not rely on fixed benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Turing Test, imitation game, imitation, indistinguishability, complexity theory, distinguisher, theory, agent comparison, interactive agents, AI evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DS7GOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper generalizes Turing’s imitation game to arbitrary interactive agents. In the Generalized Turing Test GTT(A, B), an instance of B must decide whether its interlocutor is another B or an A instructed to imitate B. A ≥_ε B holds if B’s success probability is at most 1/2 + ε. The authors study when this comparator is transitive and introduce three variants: the imitator may first query the target (GTTQ), the distinguisher is a third agent (UTT), or the number of interaction rounds is bounded. They also define Turing Scores that aggregate pairwise outcomes within a finite set of agents. Experiments on nine LLMs from early 2026 (3,105 GTT and GTTQ trials) show three things: the resulting scores broadly agree with AAII, LiveBench and Arena rankings; the empirical relation is only partly transitive; and distinguishers rely on both identity/style cues and substantive capability probes.

Critical strengths
  1. The framing is original and conceptually clean. Turning the Turing test into an indistinguishability-based relation between agents, and asking whether that relation is transitive, yields a simple formal object with a natural family of variants. The separation between the GTT and the GTTQ is a good example. The test needs no fixed task set and becomes harder as both imitators and judges improve, which speaks directly to the saturation and contamination of static benchmarks. This is the main reason I think the paper deserves serious consideration.

  2. The empirical study is carefully documented. The authors release prompts, transcripts and an interactive explorer. They test prompt paraphrases and style-controlled instructions, with a manipulation check and Newcombe–Wilson intervals, and they report results that work against their own thesis. That a pairwise game without any task set produces an ordering close to that of AAII is a useful observation in its own right, even though its interpretation is less clear than the paper suggests.

Critical weaknesses
  1. The claim that Turing Scores recover a capability ordering is not adequately established, and it is unclear what the GTT measures in practice. The GTT T scores agree well with AAII (Kendall’s τ ≈ 0.89). However, the paper reports no rank-correlation statistic, Table 1 gives no intervals for T, F or D, and the top four T scores lie within about 0.06 of each other. F, moreover, agrees with AAII only at τ ≈ 0.61 and ranks DeepSeek V3.2 second. Several distinguishers are degenerate. GPT-5.4 accepts its own copy only about one time in five, so even Ministral 8B passes the GTT against it at ε = 0.06, and only Gemini, Opus and Sonnet distinguish clearly better than chance. Because the scores are raw averages rather than estimates from a model of judge bias, such judges distort them: more than half of GPT-5.4’s F lead over DeepSeek V3.2 comes from GPT-5.4 never having to face its own, very strict column. The protocol adds further confounds. Footnote 1 and the self-control rows of Table 2 indicate that the self-control interlocutor is itself given the imitation prompt, unlike the plain instance of B in Definition 1. Distinguishers are also not told the 1/2 prior, and the imitator is told the target’s name whereas the distinguisher is not told its own. The style-control evidence is mixed. Ministral 8B is still detected without identity and style cues. Opus’s accuracy against a Qwen3-32B imitator, however, drops from 0.96 to 0.28 under the style-control prompt and recovers only to 0.60 with hard probes. Gemini’s accuracy against the same imitator is not significantly affected by style control (0.88) but falls to 0.60 with hard probes. Without non-imitating or name-only baselines, it remains unclear how much of the ordering reflects capability rather than identity fingerprints and judge bias.

  2. The theory is thin and does not deliver the ordering it motivates. Propositions 1, 4, 5 and 7 each follow in a line or two. The Turing classes of Proposition 2 require transitivity at a fixed threshold, which the transitivity theorems provide only under degenerate conditions. For ε > 0, neither bound recovers the original threshold, since ε′ ≥ α + β even when ζ = 1. At the ε = 0 threshold used in the main figures, Theorem 3 requires γ = 0 (statistically exact imitation of C’s distinguisher by B) and δ = 0, while Theorem 6 requires ζ = 1, which can hold for at most one third agent C. Theorem 3 additionally degrades with the size of the universe. The recursion probabilities over different C sum to at most one, so some C has ζ ≤ 1/(n−1), and the bound becomes ε′ ≥ (n−1)α, i.e., 0.4 for nine models with α = 0.05. Moreover, the statement of Theorem 3 omits an assumption that its proof uses (footnote 3): that the distinguisher is no worse than chance when it does not recurse. Without it, the same argument yields the ½(1/ζ − 1) term of Theorem 6. Separately, the proof applies per-message statistical closeness (Definitions 4 and 6) to the outcome of an entire multi-round interaction. That step requires either a factor equal to the number of rounds or a transcript-level assumption. The authors acknowledge that statistical indistinguishability is not directly testable, and they do not claim that the models studied satisfy these assumptions. Finally, the password attack of Proposition 5 extends to a challenge–response scheme based on a pseudorandom function, which no polynomial number of specimen queries can defeat. A developer could therefore cheaply make a model inimitable in the GTT and GTTQ. A UTT whose distinguisher has no access to the target is immune to this attack, but such a judge must then rely on its prior knowledge of each target, which is the identity channel discussed in Weakness 1.

Questions for authors
  1. Suppose the existing pairwise data are re-analyzed with a model that separates imitator ability, distinguisher discriminability and distinguisher bias, such as a Bradley–Terry or IRT model. Does the agreement with AAII, LiveBench and Arena persist, and how stable is the within-tier ordering? Please report Kendall’s τ with bootstrap intervals for T, F and D separately, together with the MOVER intervals described in Appendix B.3, and compare against simple proxies such as price and release date. Eliciting confidences in a rerun would additionally allow AUC-based scores that do not depend on each distinguisher’s decision threshold.

  2. Does the ordering in Table 1 survive if the Gemini, Opus and Sonnet distinguisher columns of the Figure 2 matrix are rerun in full under the style-controlled prompts (Variants 2 and 3), and the Turing Scores are recomputed? The current style-control results cover only a few cells and point in different directions. A known-answer validation would address the same question more directly. It could use a model family whose capability ordering is known in advance, such as different sizes of one architecture or checkpoints along one training run, since such models share a style.

  3. How do the results change if the self-control interlocutor is an uninstructed instance of B, as in Definition 1, and distinguishers are told the 1/2 prior? Opus and Gemini accept their instructed copies at 0.975 and 0.95, so the instruction by itself cannot explain GPT-5.4’s rate of about 0.18; only such an ablation can separate the possible causes. It would also help to add two baselines: an actor that answers as itself without imitation instructions, and an actor that only claims the target’s identity.

  4. Can the transitivity results be stated at a fixed threshold, in particular at ε = 0 as used in Figure 2, without requiring γ = δ = 0 or ζ = 1? Relatedly, does Theorem 3 hold for multi-round interactions under the per-message closeness of Definition 6, and can the assumption in footnote 3 be moved into the theorem statement? The transcripts could also show whether any distinguisher ever initiates a nested GTT, which would indicate whether Turing-recursivity is more than a formal device.

The idea is original and the paper is well written. However, the evidence does not yet separate capability from judge bias and identity fingerprints, and the theory does not deliver transitivity at the thresholds used in the experiments. Convincing answers to Questions 1–3 would move me to "3: Weak acceptance",

4 comments

Thank you for recognizing the originality of our framework and the value of its empirical exploration. We are pleased to report that over the past week, we completed the requested full-column prompt controls, the two actor baselines, and the joint genuine-control/prior ablation, alongside additional uncertainty analyses. These substantially extend the evidence while identifying limits that we state more precisely.

This rebuttal is a summary, and in several places we say that we rewrote something in the paper – see this anonymized Google doc for the full line-by-line changes: https://docs.google.com/document/d/1f_qoHb20qOj-sMoIs0hclsNU2Dxv_KnVYKzGXjxFSA4/edit?usp=sharing

1. Score uncertainty and judge bias. The original GTT’s Kendall correlations and 95% trial-resampling intervals are:

$ \begin{array}{lccc} \hline \textbf{Leaderboard} & \mathbf{T} & \mathbf{F} & \mathbf{D} \\ \hline \text{AAII (9)} & \mathbf{0.89\;[0.70,\,0.94]} & 0.61\;[0.54,\,0.83] & 0.67\;[0.44,\,0.78] \\ \text{LiveBench (6)} & \mathbf{0.60\;[0.33,\,0.87]} & 0.20\;[0.07,\,0.73] & 0.33\;[0.14,\,0.60] \\ \text{Arena Text (8)} & \mathbf{0.71\;[0.50,\,0.79]} & 0.21\;[0.14,\,0.57] & 0.71\;[0.43,\,0.79] \\ \hline \end{array} $

Agreement is strongest for T, but frontier-*only* T intervals include zero for all three leaderboards, so they do not establish a fine-grained ordering (see “Leaderboard agreement” in our response to Reviewer 2). We added the score-level MOVER intervals to Appendix (see google doc).

2. Full-column style controls. We reran all nine actor rows for each requested judge—Gemini, Opus, and Sonnet—under both V2 and V3; the first six positions remain unchanged in both variants, with Qwen moving ahead of Mistral. Please see “Style/identity controls” in our Reviewer 2 response (who had the same request).

Next we ran the requested within-family test on Qwen3.5-35B/122B/397B, which Reviewer 2 requested too. Please Control-F for “Family of similar models”. This was less decisive: all six pairwise advantage intervals include zero under both the original and V2 prompts. We report this limitation rather than claiming a resolved ordering of extremely closely matched models.

3. Genuine controls, prior, and actor baselines. We reran the frontier block with uninstructed genuine actors and an explicit 50–50 prior, 40 trials per cell. Recomputing scores within this three-model universe gives:

$ \begin{array}{lcc} \hline \textbf{Model} & \textbf{Original F/D/T} & \textbf{Revised F/D/T} \\ \hline \text{Opus} & 0.225\;/\;0.625\;/\;0.425 & 0.188\;/\;0.719\;/\;0.453 \\ \text{Gemini} & 0.250\;/\;0.694\;/\;0.472 & 0.100\;/\;0.563\;/\;0.331 \\ \text{GPT-5.4} & 0.850\;/\;0.581\;/\;0.716 & 0.300\;/\;0.713\;/\;0.506 \\ \hline \end{array} $

The point ordering changes from GPT-5.4 > Gemini > Opus to GPT-5.4 > Opus > Gemini. These results demonstrate protocol sensitivity. We will distinguish the original prompted-self implementation from Definition 1’s uninstructed genuine branch and report both protocols explicitly.

Finally, your suggestion for actor baselines provides a clearer intervention result:

\hline \textbf{Actor instruction} & \textbf{Accepted / trials} & \textbf{Acceptance} & \textbf{Decrease from full imitation} \\\\ \hline \text{Full imitation} & 106/240 & \mathbf{44.2\\%} & \text{—} \\\\ \text{No imitation prompt} & 27/240 & 11.3\\% & \mathbf{32.9}\\;\textbf{pp}\\;\mathbf{[26.4,\\,38.5]} \\\\ \text{Identity claim only} & 58/240 & 24.2\\% & \mathbf{20.0}\\;\textbf{pp}\\;\mathbf{[13.4,\\,26.1]} \\\\ \hline \end{array}$$ Thus, explicitly attempting to imitate the target has a substantial effect beyond answering normally or merely claiming its identity. This establishes a contribution from the imitation instruction. **4. Theory.** Thank you for identifying the interaction-level gap. We replaced per-message closeness in the relevant assumption with statistical closeness of complete interaction transcripts, and moved the nonrecursive “at least chance” assumption into Theorem 3’s statement. We will also state explicitly that the results are **error-accumulating composition bounds**, not proofs of fixed-threshold transitivity for the empirical graphs. The theoretical contribution is to identify conditions under which imitation composes when both the judge and the actor’s target change—an implication that is not automatic from fixed-test indistinguishability. These conditions are sufficient, not necessary. We reiterate that our paper introduces these theoretical questions for the first time, and finds the first non-trivial condition set under which transitivity holds. Our paper leaves open the theoretical CS question of other conditions under which the Turing test is transitive.

Thank you for the rebuttal and the additional experiments, which address my questions directly: intervals added (full-universe T–AAII 0.89 [0.70, 0.94]; frontier-only intervals include zero); full-column style controls preserving the top-six ordering; the genuine-control/prior ablation quantifying the protocol confound (GPT-5.4 T 0.716 → 0.506, frontier order reversed); actor baselines separating imitation from identity (44.2%/24.2%/11.3%); and a corrected Theorem 3.

Two concerns remain: no decomposition of imitator ability, distinguisher discriminability, and judge bias was performed, and the primary 9×9 table is still produced under the prompted-self protocol; the within-family null limits capability claims to coarse stratification.

Given the novelty of the formulation, the released artifacts, and claims recalibrated to the evidence, I will raise my rating by one point, from 2 to 3, conditional on the revised paper implementing the stated scoping, dual-protocol reporting, and corrected Theorem 3.

Dear reviewer, thank you for responding to our rebuttal so quickly, and for increasing our score. We have taken the last day + night to run a few additional experiments in line with your remaining concerns: the first was easy as it re-used existing data, but the last 2 are new experiments. Please take a look, as we believe these findings sharpen our claims. In any case, we are deeply thankful to you, as your input led us to refinements and additional results that have significantly improved our paper.

Experiment 1: Separating imitation ability, judge discrimination, and acceptance bias.

As an additional analysis of the 9x9 table, to try to explain the results using the 3 separate characteristics of each AIs, we fit regularized logistic models that distinguish: actor imitation effects, judge discrimination, and tendency to accept B as the same model. Allowing judge-specific sensitivity to actor ability improved mean held-out log loss from 0.412 to 0.372 across 100 splits of trials within each cell.

The fitted effects differed across roles. In the flexible model, GPT-5.4 had the largest standardized imitation effect (1.40; 95% bootstrap interval [1.10, 1.69]), alongside a negative acceptance-bias estimate (−2.01 [−2.95, −1.53]), indicating a tendency to reject same-model claims. Gemini and Opus had the largest estimated separations between genuine controls and imitators.

Setting the fitted acceptance-bias term to zero yielded positive rank correlations with AAII, LiveBench, and Arena Text: respectively, Kendall’s τ = 0.667, 0.200, and 0.500 under the additive model, and 0.778, 0.467, and 0.643 under the flexible model. However, both models retain significant lack of fit (parametric-bootstrap p = 0.002), showing actor performance can approximate actor/judge estimates, but particular actor-judge pairings may behave differently from what general estimates predicts.

Experiment 2: Replacing prompted self-controls with unprompted genuine controls. We collected 20 fresh self-control trials for each of the nine models, removing the actor’s imitation instruction:

\hline \textbf{Model} & \textbf{Genuine-self acceptance} & \textbf{Original T} & \textbf{Revised T [95\\% CI]} \\\\ \hline \text{Gemini 3.1 Pro} & 20/20 & 0.769 & 0.781\\;[0.731,\\,0.802] \\\\ \text{Claude Opus 4.6} & 20/20 & 0.747 & 0.753\\;[0.703,\\,0.772] \\\\ \text{Claude Sonnet 4.6} & 20/20 & 0.708 & 0.718\\;[0.666,\\,0.742] \\\\ \text{GPT-5.4} & 3/20 & 0.729 & 0.716\\;[0.677,\\,0.771] \\\\ \text{DeepSeek V3.2} & 12/20 & 0.656 & 0.606\\;[0.544,\\,0.658] \\\\ \text{Grok 4.20} & 18/20 & 0.571 & 0.566\\;[0.509,\\,0.601] \\\\ \text{Mistral Large} & 16/20 & 0.471 & 0.471\\;[0.411,\\,0.513] \\\\ \text{Qwen3 32B} & 18/20 & 0.454 & 0.449\\;[0.393,\\,0.484] \\\\ \text{Ministral 8B} & 16/20 & 0.445 & 0.395\\;[0.337,\\,0.436] \\\\ \hline \end{array}$$ Intervals are 95% confidence intervals from Wilson intervals using MOVER. Main finding: T scores retained positive correlations with AAII (τ = 0.833), LiveBench (0.467), and Arena Text (0.643), over nine, six, and eight matched models, respectively. In fact, only GPT-5.4 and Sonnet exchanged positions in the point-estimate ordering; their confidence intervals overlapped substantially. **Experiment 3: Another foray into fixed-family** The previous experiment we wrote for the first rebuttal, running the standard GTT between 3 very closely matched Qwen models, was inconclusive. However, we realized in responding to point 1 above (logistic model separating actor/distinguisher ability) that for such close models, a version of the GTT focusing on *one* sub-ability may be better. We used Gemini as the fixed external judge to compare the three Qwen models *only as actors/imitators*. This gives the three actors a common reference and focuses the comparison on their imitation performance. Excitingly, we find acceptance increased with model size: 30% for Qwen 35B, 32% for Qwen 122B, and 35% for Qwen 397B. Recall that these models had *very close* benchmark performance (85.3/86.7/87.8, obtained with extensive fixed-dataset questioning). The observed order matches the expected capability order in this fixed-judge test. Overall we see this result as an *initial foray* into running generalized Turing Tests (and importantly its variants, which may be better suited versions) on fixed families – we think a deeper investigation is a great line for future work, as our paper opens these questions.

Thanks, please update the submission and I will raise the score

Reviewer_bMFyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines a Generalized Turing Test (GTT): agent A "passes" against agent B (written A ≥ B) if B, acting as distinguisher, cannot reliably tell an A instructed to imitate B apart from another instance of B. Indistinguishability is formalized with an ε-advantage as in cryptography. On the theory side, the paper gives sufficient conditions for this comparator to be transitive. The main result assumes agents can recursively start the game ("Turing-recursive"). It also introduces variants: GTTQ (the imitator can first query the target), the Universal Turing Test (UTT, with a third-party distinguisher), round-limited versions, and Elo-like Turing Scores. Empirically, it runs about 2,160 GTT trials across nine LLMs, from Ministral 8B to Gemini 3.1 Pro. The resulting Turing Scores give a ranking broadly consistent with external leaderboards, and transcript analysis shows that distinguishers use both style/identity probes and substantive STEM probes.

Critical strengths
  1. An interesting, benchmark-free way to compare models. Having models imitate and distinguish each other is a creative idea. It needs no fixed dataset, adapts automatically as models improve, and the transcript analysis and prompt-robustness variants (e.g., banning identity and style cues) are a useful first look at how models actually play. The paper is also open about its limitations, and the code is provided.
Critical weaknesses
  1. The theory doesn't bring much novelty, and closely related prior work is missing. Most theory results in this paper are standard tools from cryptographic indistinguishability. Prop. 1 is the triangle inequality on total variation. Theorem 6 is the usual hybrid argument (with ζ→1 it reduces to ε = α + β). Props. 4, 5 and 7 are close to definitional. The one non-trivial result, Theorem 3, relies on agents being "Turing-recursive", which the authors acknowledge the tested LLMs do not satisfy, and empirical transitivity is only about 60% at ε = 0. So the theory does not explain or support the experiments. On the conceptual side, LLMs acting as Turing-test judges have been studied before (e.g., the inverted Turing test [1]), and using LLMs as examiners that generate their own questions to rank other models without a fixed benchmark, including peer examination, was proposed in [2]. Neither is discussed. The genuinely new piece, one model imitating another specific model, is more incremental than the paper's framing suggests ("an idea that has never been considered before").

  2. The agreement with leaderboards may come from the choice of models. The nine models span very different tiers (8B to frontier), so almost any reasonable signal would separate them coarsely. Within tiers, the GTT ranking disagrees with external rankings: the top three are ordered Gemini > Opus > GPT, versus Gemini > GPT > Opus on AAII and Opus first on Arena, and Grok is 6th on the GTT but 3rd on Arena. The GTTQ reorders the top three again. Differences among top models are 0.02–0.04 in T, from 25–40 trials per cell, and Table 1 reports no rank correlation or uncertainty. In addition, 88.5% of trials use style/identity probes, and prior work shows LLMs can be identified from style alone with very high accuracy [3]. So the GTT may largely measure style imitation and self-recognition, which also correlate with model tier. The claim that the GTT recovers capability rankings needs a more controlled test.

[1] Rathi et al. GPT-4 is Judged More Human than Humans in Displaced and Inverted Turing Tests. https://arxiv.org/abs/2407.08853

[2] Bai et al. Benchmarking Foundation Models with Language-Model-as-an-Examiner. https://arxiv.org/abs/2306.04181

[3] Sun et al. Idiosyncrasies in Large Language Models. https://arxiv.org/abs/2502.12150

Questions for authors
  1. Can you report rank correlations (e.g., Kendall's τ) with confidence intervals against each external leaderboard, both over all nine models and within the top tier? Does the GTT still agree with leaderboards on a universe of closely matched models, e.g., models with similar AAII scores or checkpoints from the same family?
  2. How much of the ranking is driven by style and identity cues rather than capability? For example, does the full-universe ranking hold under your Variant 2/3 prompts (no identity or style cues), not just within the frontier block?
  3. What does Theorem 3 offer beyond a standard hybrid argument, given that its assumptions don't hold for current LLMs? Can any of its assumptions be measured empirically?
2 comments

Thank you for the specific suggestions – we are excited to report that we spent the last week running the new experiments you proposed. These results strengthen the evidence for broad agreement while clarifying the limits of fine-grained ranking.

This rebuttal is a summary, and in several places we say that we rewrote something in the paper – see this anonymized Google doc for the full line-by-line changes: https://docs.google.com/document/d/1f_qoHb20qOj-sMoIs0hclsNU2Dxv_KnVYKzGXjxFSA4/edit?usp=sharing

Leaderboard agreement. For the original GTT T scores, Kendall correlations and 95% trial-resampling intervals are:

$ \begin{array}{lcc} \hline \textbf{Leaderboard} & \textbf{All matched models} & \textbf{Frontier only (3)} \\ \hline \text{AAII} & 0.89\;[0.70,\,0.94]\quad(n=9) & 0.33\;[-0.33,\,1.00] \\ \text{LiveBench} & 0.60\;[0.33,\,0.87]\quad(n=6) & -0.33\;[-1.00,\,1.00] \\ \text{Arena Text} & 0.71\;[0.50,\,0.79]\quad(n=8) & 0.33\;[-1.00,\,1.00] \\ \hline \end{array} $

Thus, agreement across the tested universe is consistently positive, but the frontier intervals do not establish a reliable ordering.

Family of similar models: Next, based on your suggestion we tested Qwen3.5-35B/122B/397B (25 trials per cell). Their reported MMLU-Pro scores are 85.3/86.7/87.8. GTT T scores were 0.485/0.445/0.550 under the original prompt and 0.565/0.490/0.555 under the version 2 variant. All six distinguishing-advantage intervals include zero in both conditions. Therefore, this experiment (which is just a one-off for these 3 Qwen 3.5 models) does not establish a within-family ordering at this sample size. We have rewritten to distinguish broad model stratification from such fine-grained capability ranking in the paper.

Style/identity controls. Reviewer 3 also asks for this, so we expanded V2/V3 to nine actors × three judges (Gemini, Opus, Sonnet—the three strongest baseline distinguishers), with 25 trials per cell per variant: 1,350 trials. The resulting GTT T scores are below. These replace three judge columns; the other six remain unchanged.

$ \begin{array}{lccc} \hline \textbf{Model} & \textbf{Original T} & \textbf{V2 T} & \textbf{V3 T} \\ \hline \text{Gemini} & 0.769 & 0.808 & 0.787 \\ \text{GPT-5.4} & 0.729 & 0.718 & 0.715 \\ \text{Opus} & 0.747 & 0.683 & 0.715 \\ \text{Sonnet} & 0.707 & 0.650 & 0.665 \\ \text{DeepSeek} & 0.656 & 0.664 & 0.656 \\ \text{Grok} & 0.571 & 0.641 & 0.616 \\ \text{Mistral} & 0.471 & 0.536 & 0.504 \\ \text{Qwen3 32B} & 0.454 & 0.551 & 0.536 \\ \text{Ministral} & 0.445 & 0.460 & 0.452 \\ \hline \end{array} $

The frontier 3 remains the top 3 in both variants, while Qwen overtakes Mistral. We find that ordering by the mean of GTT and GTTQ, the first six retain their order in both variants. This substantially extends the controls beyond the frontier block and supports robustness of the broad structure. Of course, this does not establish a fully style-independent ranking.

Theoretical contribution. We agree that the triangle inequality and several supporting propositions use standard tools. However, transitivity of this comparator is not automatic: A ≥ B constrains B's distinguishing procedure, whereas B ≥ C constrains C's, and the actor's target-conditioning also changes. A fixed-test hybrid argument alone cannot transfer between these games. Theorem 3 identifies an explicit closure condition—B can recursively implement C's test—under which that transfer is possible, with quantitative loss α/ζ + β + γ + δ. Its contribution is the structural condition and resulting composition theorem, rather than a new hybrid technique.

These are sufficient, not necessary, conditions; finding weaker alternatives remains open, a question our work initiates. The theorem concerns the formal framework and is not asserted to explain the tested LLMs. The observed approximately 60% closure rate is therefore a separate diagnostic, not validation of its hypotheses. Uniform statistical closeness cannot be certified from finite black-box trials. We make these distinctions explicit in the paper.

Finally, we added all three suggested references and remove the absolute historical novelty claim. Rathi et al. study human/AI attribution from transcripts; Bai et al. already provide generated questions, follow-ups, and peer examination, but evaluate answer quality. Our distinguishing task concerns a specified target versus an actor deliberately imitating it. Sun et al.'s attribution results motivate the cue controls and the caution above. To reiterate, our contribution is the indistinguishability-based imitation comparator which generalizes the Turing Test, its structural analysis, and these initial empirical findings.

Dear Reviewer, thank you again for your thoughtful feedback. We would appreciate your assessment of our rebuttal and revisions, which we believe address your main concerns. We also invite you to look at our follow-up response to Reviewer 3, where we report additional experiments on judge bias, unprompted genuine controls, and within-family comparisons. Please let us know if any questions remain; we would be happy to address them.

Reviewer_h6z9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper claims to generalize Turing’s imitation game to arbitrary agents: A >= B if a B acting as judge can’t tell an A imitating B from another B. It proves sufficient conditions for transitivity, introduces variants and defines Elo-like Turing Scores. In a proof of concept on 9 LLMs, the resulting scores broadly match external leaderboards without using any benchmark tasks.

Critical strengths

The paper casts the imitation game as a cryptography style indistinguishability relation between agents. That gives a self adapting evaluation, plus useful theory: variants with querying, and interaction limits, and conditions under which the relation is transitive.

Critical weaknesses

I thank the authors for taking me to the seminal work. Based on my reading Turing considers an imitation game in which a digital computer (A) mimics an arbitrary discrete-state machine (B), such that an interrogator cannot distinguish between them. Thus, extending the imitation game beyond human targets, including machine machine imitation evaluated through behavioral indistinguishability, was already anticipated by Turing himself. This paper cannot claim that as a novelty.

If I understand correctly the paper's work differs in making the target agent B also serve as the distinguisher. This should be separated from the broader idea of generalizing the imitation game to non human agents, which is not really new. The assertion that the Turing Test had not previously been considered for arbitrary agents is historically too broad. I am not an expert in the domain but there are a lot of papers on the topic , also Turing's explicit machine machine construction and some follow ups.

Also the question posed in the paper deviates a lot from ‘Can machines think?’. Just because there is the distinguish used in the framework it seems not so reasonable to call it a generalized Turing test as the primary question investigated here is different. In the proposed game imitation game A >= B can hold just because B is a poor judge, not because A imitates well. Nearly every model “passes” against weak distinguishers: the columns for Mistral, Qwen and Ministral validate that. The popularized version of the human game is meaningful under the assumption that the humans are ‘sufficiently’ good at thinking.

Variations of turing test creating breaking asymmetry never been considered before, is not true at abstraction. The Meta-Turing Test, self-recognition work and LLM-as-judge are very near to the proposed work. I think there is a lot of overlap for this paper and the large volume of work on self recognition.

Questions for authors

Why not just call it a new framework, isnt it sufficiently different in objective from classical imitation game ?

3 comments

Thank you for recognizing the value of the formal framework and its theory, and for identifying a historical overstatement. This rebuttal is a summary, and in several places we say that we rewrote something in the paper – see this anonymized Google doc for the full line-by-line changes: https://docs.google.com/document/d/1f_qoHb20qOj-sMoIs0hclsNU2Dxv_KnVYKzGXjxFSA4/edit?usp=sharing

First of all, we agree: Turing explicitly describes a digital computer imitating a discrete-state machine as part of his universality argument. We have rewritten our manuscript in a few places, starting with crediting this precedent and correcting the overly broad claims in the Introduction (lines 43–45) and Discussion (lines 473–474).

Our contribution is to develop a formal framework for comparing interactive agents through imitation, and study the resulting comparator. This extends beyond assigning B the judge role: the UTT (§3.3) allows independently specified actor, target, and distinguisher, each an arbitrary interactive protocol; the base GTT is the special case with target and distinguisher drawn from the same agent. We study conditional transitivity (§§3.1, 3.3), access to an unfamiliar target through specimen queries (§3.2), and interaction budgets (§3.4), and instantiate the framework empirically (§4). These are the contributions for which we claim novelty, rather than the idea of machine–machine imitation or machine judges alone.

Passing against B does not establish full simulation of B or superior intelligence: it establishes success against B's particular distinguishing procedure. We state this explicitly beside Definition 2. The UTT makes this dependence examinable by holding the distinguisher D fixed while varying actor and target; our pilot experiments already compare the structures induced by different judges (§4.1; Fig. 17). Separately, Table 1 reports both fooling and distinguishing scores, so the empirical evidence concerns performance across both roles and an opponent population, rather than a pass against just one weak judge.

We also agree that the connections to prior work deserve more explicit treatment. We have expanded §2's existing comparisons and added your and other references, see the Google doc for the full text.

Finally, we agree that “a new framework” is an appropriate framing. We propose the following introductory sentence: “We introduce the Generalized Turing Test (GTT), a framework for comparing interactive agents through imitation games and behavioral indistinguishability relative to a specified distinguisher.” The name identifies the inherited imitation-game structure; our objective is to study this comparator and its consequences, without claiming to settle “whether machines think”.

Dear Reviewer, thank you again for your thoughtful feedback. We would appreciate your assessment of our rebuttal and revisions, which we believe address your main concerns. We also invite you to look at our follow-up response to Reviewer 3, where we report additional experiments that sharpen our claims. Please let us know if any questions remain; we would be happy to address them.

Reviewer_h6z9OpenReview ↗

Reply to rebuttal

Thank for the rebuttal. I am ready to update my score after going through the updated paper as most of the concerns are about the primary framing of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.