- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a Turing-test-inspired reward for reinforcement learning post-training of language models. A stronger teacher estimates the probability that a student-generated response could have been written by itself, and the student is optimized with GRPO using the log of this self-attribution probability as the reward. The proposed RL updates require neither ground-truth answers nor teacher-generated reference solutions.
Critical strengths
The idea of adapting the Turing Test into a training objective is conceptually interesting and connects modern LLM post-training with decades of research on the nature and evaluation of machine intelligence.
Critical weaknesses
- The advantage of self-attribution over existing feedback mechanisms remains unclear. The paper show that self-attribution is correlated with correctness, so it acts as an indirect proxy for response quality. Beyond correctness, LLM can also provide denser supervision via grading intermediate progress, or reasoning strategy. Teacher knowledge can also be transferred through SFT or on-policy distillation. The paper does not establish that self-attribution is more informative, reliable, or effective than these alternatives.
- The practical significance of the “ground-truth-free” claim needs stronger justification. Access to a strong teacher already offers several ways to obtain supervision without additional human annotation. Therefore, the absence of ground-truth labels alone does not establish a distinctive advantage.
- The empirical results do not establish a consistent advantage over existing methods, and robustness in the black-box setting remains uncertain.
Questions for authors
See weakness.
Thank you for your feedback – please see our responses below, and note that we have spent the last week running additional experiments to strengthen our results. Please take a look.
W1 · Comparison with feedback mechanisms
Our central aim is to test whether the signal used in the Turing test can also support learning. A distinguisher judges whether a response could be its own, and we use that judgment to train a student. What we call “quality” is another version of the same teacher-feedback setup: it asks the teacher to assess correctness instead of authorship, while keeping the rest of the setup fixed. It provides a natural comparison between learning from a correctness judgment and learning from an indistinguishability judgment.
The Turing-test-based attribution score can reflect both content and style. Quality targets correctness directly, whereas self-attribution can also reward features that make a response more teacher-like without making it more correct. We therefore expected Quality to be more closely aligned with task accuracy.
Note that in Section 6, we examine groups of four alternative code solutions generated by the student for the same KodCode task. among groups where all four solutions failed the full verifier, we compared pairs in which one solution passed more individual tests than the other, yielding 977 pairs. We checked whether the self-attribution and Quality rewards assigned a higher score to the solution that passed more tests. The teacher did not see the test results. On this metric, self-attribution scored 61.8% and quality 58.4%. This shows that self-attribution (Turing-test) can distinguish degrees of partial correctness that the binary verifier labels equally as failures.
SFT on teacher-generated solutions is already included in Table 2. On-policy distillation would supervise the student on its own generated responses and remains untested.
W2 · Practical significance of ground-truth-free training
Here, “ground-truth-free” means that gold answers and task verifiers do not enter reward computation or RL updates. The teacher sees only the task and the student’s response; answers and tests are used separately to evaluate outcomes, specifically to assess correctness and objectively measure whether the model’s performance on the task improves or worsens. The verifiers that make this evaluation possible do not enter reward computation or RL updates.
The practical benefit is for training without reference solutions or a task-specific verifier. This allows us to investigate and use the model’s internal verification capabilities, relying on the knowledge contained in its weights rather than the external oracles used by verifier-based RL, such as reference-answer checks or unit tests. This benefit is shared with Quality and other teacher-judge approaches. However, our specific scientific question is whether the teacher-Turing-test judgment can supply a learning signal even though the teacher is asked about authorship rather than correctness.
W3 · Empirical comparisons and black-box robustness
Again, our method is not intended as a state-of-the-art training method, but as a setting that allows us to distill the signal contained in the Turing test and use it to formulate a learning problem for a student. To corroborate this, we’ve conducted several new experiments:
(1) Training on additional benchmarks: MATH and BigCodeBench. Self-attribution / Turing test improves MATH accuracy from 40.7% to 46.6% and BigCodeBench pass@1 from 15.2% to 20.3%; Quality reaches 49.1% and 21.9% on the same tasks.
(2) Training additional student-teacher pairs. On CommonsenseQA, with DeepSeek V4 Flash as teacher, an Olmo-3-7B-Think-DPO improves from 71.1% to 74.0%, while Qwen3-1.7B with thinking enabled improves from 71.5% to 73.1%. With Qwen2.5-72B-Instruct as teacher, Qwen2.5-1.5B-Instruct improves from 57.5% to 68.5%, while Qwen3-1.7B with Thinking remained at 71.5% (the ability to transfer information depends on the original capability gap between teacher and student).
(3) Additional black-box teacher experiment. We ran an additional seed of training on CommonsenseQA run with GPT-5 Mini supplying sampled binary judgments; Qwen-2.5 1.5B improves from from 57.5% to 66.1% (+8.6 points, paired 95% CI 5.5 to 11.5). This adds evidence that the procedure can work without teacher log probabilities.