acceptodds
Under review as a conference paper at ICLR 2027

Uncovering the Turing Test's Latent Signal: Indistinguishability as Reward

Abstract

Can a model learn by trying to fool another model? We show that it can. Inspired by the Turing Test and its recent generalization to arbitrary distinguishers, we introduce a *domain-based teacher-Turing-Test*, in which a stronger teacher model serves as both the *target* of imitation and the *distinguisher*, and a weaker student model is trained to make the teacher attribute responses to itself. We use this self-attribution (or fooling) probability as a reinforcement learning reward, without ever providing the student ground-truth answers, nor requiring the teacher to generate responses. We test our approach independently in four domains with verifiable answers: CommonsenseQA (reasoning), GSM8K (mathematics), KodCode (programming), and Spider (SQL), training a Qwen2.5-1.5B student/imitator model using DeepSeek V4 Flash as teacher/distinguisher. Beyond improving at passing the Turing Test, the student improves on independent, held-out evaluations in all domains and runs. To probe *why* being mistaken for the teacher can improve capability, our analyses reveal that self-attribution is associated with correctness – substantially more than formatting changes – and that it rewards *partial* correctness. Our results demonstrate a new, simple teacher-student training paradigm with a private and/or high-cost teacher. They also suggest that indistinguishability need not merely measure behavior – it can also be a mechanism for learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Turing test, RL, Turing reward, imitation, indistinguishability, student-teacher, imitation game, distinguisher, reward modeling, knowledge distillation, black-box supervision, task generalization

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_17JpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a Turing-test-inspired reward for reinforcement learning post-training of language models. A stronger teacher estimates the probability that a student-generated response could have been written by itself, and the student is optimized with GRPO using the log of this self-attribution probability as the reward. The proposed RL updates require neither ground-truth answers nor teacher-generated reference solutions.

Critical strengths

The idea of adapting the Turing Test into a training objective is conceptually interesting and connects modern LLM post-training with decades of research on the nature and evaluation of machine intelligence.

Critical weaknesses
  • The advantage of self-attribution over existing feedback mechanisms remains unclear. The paper show that self-attribution is correlated with correctness, so it acts as an indirect proxy for response quality. Beyond correctness, LLM can also provide denser supervision via grading intermediate progress, or reasoning strategy. Teacher knowledge can also be transferred through SFT or on-policy distillation. The paper does not establish that self-attribution is more informative, reliable, or effective than these alternatives.
  • The practical significance of the “ground-truth-free” claim needs stronger justification. Access to a strong teacher already offers several ways to obtain supervision without additional human annotation. Therefore, the absence of ground-truth labels alone does not establish a distinctive advantage.
  • The empirical results do not establish a consistent advantage over existing methods, and robustness in the black-box setting remains uncertain.
Questions for authors

See weakness.

1 comment

Thank you for your feedback – please see our responses below, and note that we have spent the last week running additional experiments to strengthen our results. Please take a look.

W1 · Comparison with feedback mechanisms

Our central aim is to test whether the signal used in the Turing test can also support learning. A distinguisher judges whether a response could be its own, and we use that judgment to train a student. What we call “quality” is another version of the same teacher-feedback setup: it asks the teacher to assess correctness instead of authorship, while keeping the rest of the setup fixed. It provides a natural comparison between learning from a correctness judgment and learning from an indistinguishability judgment.

The Turing-test-based attribution score can reflect both content and style. Quality targets correctness directly, whereas self-attribution can also reward features that make a response more teacher-like without making it more correct. We therefore expected Quality to be more closely aligned with task accuracy.

Note that in Section 6, we examine groups of four alternative code solutions generated by the student for the same KodCode task. among groups where all four solutions failed the full verifier, we compared pairs in which one solution passed more individual tests than the other, yielding 977 pairs. We checked whether the self-attribution and Quality rewards assigned a higher score to the solution that passed more tests. The teacher did not see the test results. On this metric, self-attribution scored 61.8% and quality 58.4%. This shows that self-attribution (Turing-test) can distinguish degrees of partial correctness that the binary verifier labels equally as failures.

SFT on teacher-generated solutions is already included in Table 2. On-policy distillation would supervise the student on its own generated responses and remains untested.

W2 · Practical significance of ground-truth-free training

Here, “ground-truth-free” means that gold answers and task verifiers do not enter reward computation or RL updates. The teacher sees only the task and the student’s response; answers and tests are used separately to evaluate outcomes, specifically to assess correctness and objectively measure whether the model’s performance on the task improves or worsens. The verifiers that make this evaluation possible do not enter reward computation or RL updates.

The practical benefit is for training without reference solutions or a task-specific verifier. This allows us to investigate and use the model’s internal verification capabilities, relying on the knowledge contained in its weights rather than the external oracles used by verifier-based RL, such as reference-answer checks or unit tests. This benefit is shared with Quality and other teacher-judge approaches. However, our specific scientific question is whether the teacher-Turing-test judgment can supply a learning signal even though the teacher is asked about authorship rather than correctness.

W3 · Empirical comparisons and black-box robustness

Again, our method is not intended as a state-of-the-art training method, but as a setting that allows us to distill the signal contained in the Turing test and use it to formulate a learning problem for a student. To corroborate this, we’ve conducted several new experiments:

(1) Training on additional benchmarks: MATH and BigCodeBench. Self-attribution / Turing test improves MATH accuracy from 40.7% to 46.6% and BigCodeBench pass@1 from 15.2% to 20.3%; Quality reaches 49.1% and 21.9% on the same tasks.

(2) Training additional student-teacher pairs. On CommonsenseQA, with DeepSeek V4 Flash as teacher, an Olmo-3-7B-Think-DPO improves from 71.1% to 74.0%, while Qwen3-1.7B with thinking enabled improves from 71.5% to 73.1%. With Qwen2.5-72B-Instruct as teacher, Qwen2.5-1.5B-Instruct improves from 57.5% to 68.5%, while Qwen3-1.7B with Thinking remained at 71.5% (the ability to transfer information depends on the original capability gap between teacher and student).

(3) Additional black-box teacher experiment. We ran an additional seed of training on CommonsenseQA run with GPT-5 Mini supplying sampled binary judgments; Qwen-2.5 1.5B improves from from 57.5% to 66.1% (+8.6 points, paired 95% CI 5.5 to 11.5). This adds evidence that the procedure can work without teacher log probabilities.

Reviewer_aW5AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes using the self-attribution probability of a stronger teacher model as a GRPO reward, without requiring ground-truth labels or teacher-generated solutions during RL training. Experiments using Qwen2.5-1.5B-Instruct demonstrate improvements in held-out accuracy across four domains. The authors further analyze the relationship between self-attribution rewards and solution correctness.

Critical strengths

The proposed reward is simple and does not require ground-truth labels or teacher-generated solutions. Consistent improvements across four domains and three seeds demonstrate its feasibility. The additional analyses provide useful evidence that self-attribution captures correctness-related information rather than merely superficial response characteristics.

Critical weaknesses
  1. Limited evidence for generality across students.

All student experiments use Qwen2.5-1.5B-Instruct, with 1000 training questions per domain. The GPT-5 Mini extension changes the teacher, not the student, and uses only one seed. Although acknowledged in Appendix A, this limits the evidence to a small-scale proof of concept: applicability across student families and sizes remains untested. This does not invalidate the reported improvements, but leaves the broader scope of the proposed training paradigm insufficiently supported.

  1. The contribution beyond the matched Quality baseline needs clearer justification.

Both methods use the same teacher and GRPO setup, without ground-truth answers or teacher-generated solutions; the central difference lies in asking whether a response is the teacher's own rather than whether it is correct. Table 2 shows comparable performance on CommonsenseQA, GSM8K, and KodCode. The apparent Spider advantage is affected by one Quality run's format-drift failure; attribution training also experiences format drift with GPT-5 Mini on GSM8K, so these results do not establish a general robustness advantage.

The finding that self-attribution can support learning is interesting, but its additional scientific or practical significance remains insufficiently developed. The positioning should also distinguish this contribution from direct-RLAIF in Lee et al., RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024), which already obtains RL rewards directly from an LLM.

Questions for authors
  1. Generalizability: Can the effectiveness of the proposed reward be validated on students from different model families or scales?

  2. Clarification: Beyond demonstrating feasibility, what scientific insight or practical benefit is specific to self-attribution compared with the Quality baseline? In particular, do the partial-correctness analyses establish an advantage over Quality, or only over the binary verifier reward?

  3. Evaluation robustness: What proportion of responses is truncated by the 768-token limit? Would the conclusions remain consistent with longer completions and sampling-based evaluation?

2 comments

Thank you for your detailed feedback – we are excited to report that we have run many additional experiments based on your suggestions.

Student families, scale, and teacher choice (W1, Q1)

To test whether the indistinguishability reward can support knowledge transfer across models, we ran 5 additional experiments:

(1) On CommonsenseQA, using DeepSeek V4 Flash as teacher, an Olmo-3-7B-Think-DPO improved from 71.1% to 74.0% (+2.9 percentage points, paired 95% CI 0.5 to 5.2)

(2) Qwen3-1.7B with thinking enabled improves from 71.5% to 73.1% (+1.6 percentage points, CI −0.4 to 3.6)

(3) With Qwen2.5-72B-Instruct as teacher, the original Qwen2.5-1.5B-Instruct student improves from 57.5% to 68.5% (+11.0 percentage points, CI 8.0 to 14.0); with this teacher, Qwen3-1.7B with thinking remains at 71.5%.

(4) We ran an additional seed of black-box training with GPT-5 Mini on CommonsenseQA, improving the student from 57.5% to 66.1%.

(5) We extended the evaluation to 2 additional benchmarks using the original student–teacher pair. Self-attribution improves MATH accuracy from 40.7% to 46.6% and BigCodeBench-Instruct pass@1 from 15.2% to 20.3%, providing further evidence beyond the original datasets.

We believe these results significantly extend the test beyond the original student/teacher pair.

What self-attribution establishes (W2, Q2)

Our aim is to test whether the signal used in the teacher Turing test can support learning. The distinguisher judges only whether a response could be its own; we use that judgment to train the student.

“Quality” (the teacher judging if the student’s answer is correct) is the natural comparison because it directly proxies whether an answer is correct. We alter only the judgment requested in the prompt. Of course, we expected Quality to be more closely aligned with task accuracy. The Turing-test / self-attribution, on the other hand, could also reward stylistic features that make responses more teacher-like without making them more correct. The formatting failures illustrate why these outcomes must be distinguished. Our purpose is to study this learning signal – i.e. that of the Turing test – and show it can transfer a useful learning signal, rather than propose a method that must outperform correctness-based feedback.

Towards this, in Section 6 we examine groups of 4 alternative code solutions generated by the student for the same task. Among groups where all four solutions failed the full verifier, we compared pairs in which one solution passed more individual tests than the other, yielding 977 pairs. We checked whether the self-attribution and Quality rewards assigned a higher score to the solution that passed more tests. The teacher does not see test results. Self-attribution scored 61.8% and Quality 58.4% on this metric. This shows that self-attribution can distinguish degrees of partial correctness that the binary verifier labels equally as failures.

We also agree about the Spider outlier. Extracting the SQL query recovers 58.99% for Quality seed 23, giving a corrected three-seed mean of 60.28%, versus 59.77% for self-attribution. Overall, these results support learning from the Turing-test attribution signal, with performance close to Quality.

RLAIF methods also use reinforcement learning to learn from model feedback. In particular, Lee et al.'s direct-RLAIF already obtains rewards directly from an LLM. But our work adopts this general approach to study a different question: whether the teacher's attribution judgment can supply a learning signal.

Completion limit and sampled evaluation (Q3).

Good question - the original self-attribution runs reach the 768-token cap in 0% of responses on CommonsenseQA, 1.82–5.46% on GSM8K, 0–1.11% on KodCode, and 0–0.77% on Spider across seeds.

We also tested sampling in preliminary experiments and observed similar performance. We chose greedy decoding for the main evaluation so as to compare all methods using a deterministic decoding rule, without the variability or repeated evaluations needed to average over random samples.

Regarding the token limit, increasing it under unchanged greedy decoding can only affect responses that reached the original cap. Even if every such change favored the student before training, self-attribution would retain a positive gain in all twelve runs, with a minimum of 1.21 percentage points gained.

Thank you for the additional experiments and clarifications, which substantially address my concerns about student generality and the completion limit. I appreciate the feasibility goal, but remain unconvinced that using the teacher’s authorship judgment instead of its correctness judgment provides a sufficiently substantial scientific contribution. The partial-correctness comparison is useful, although uncertainty for the difference between self-attribution and Quality (61.8% versus 58.4%) remains unreported. I therefore maintain my rating of 2.

Reviewer_UakyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses a Turing-Test-style reward for RL: a weak student is trained with GRPO to increase the probability that a stronger teacher attributes the student's response to itself, without ground-truth labels or teacher generations. On 4 benchmarks, this improves the student's accuracy, comparable to correctness-based baselines. Their analysis shows that the teacher's self-attribution tracks response correctness, including partial correctness.

Critical strengths

This paper proposes an interesting approach. The authors conducted fairly thorough experiments across multiple setups and datasets. The experimental results support the effectiveness of the method proposed by the authors.

Critical weaknesses
  1. For the benchmarks used in the experiments, GSM8K is too easy for current math RL work, and MATH or harder benchmarks would be more appropriate. For code, KodCode is not a commonly used benchmark. Could the authors explain why it was chosen, and report results on more widely used code benchmarks?

  2. In the experiments, the student model, Qwen2.5-1.5B-Instruct, is not a reasoning model. Since reasoning models are now the standard base models for RL post-training, it is unclear whether the findings transfer to this setting. The authors should evaluate the method with a reasoning model as the student.

  3. It is unclear why one should use the proposed method in the experimental setting. The setup assumes access to the teacher and its log-probabilities, as well as oracle answers for the training questions. Given this access, simpler alternatives, including SFT/Oracle/Quality baselines, are available and perform better. Moreover, the paper does not discuss training compute, and the method likely costs substantially more compute for similar or worse results.

  4. Since the teacher in the main experiments is open-weight, a simpler generation-free reward is available by feeding the question and the student's response directly to the teacher and use the teacher's likelihood or perplexity of the response as the reward. This directly measures how teacher-like the response is. The authors should compare against this method.

  5. All training data in the experiments have oracle answers, so a verifier reward is readily available. Domains without a verifier, where the method would be most useful, are not tested.

  6. In terms of presentation, Table 2 reports a Gain column only for the proposed method and not for the baselines. The authors should either report gains for all methods or remove the column.

Questions for authors

The reward reads the probabilities of "0" and "1" from the teacher's top-5 first-token log-probabilities (Table 7). How is p_ind computed when one of these labels does not appear in the top-5? Does this situation ever come up?

2 comments

Thank you for your feedback – we have run many additional experiments in the past week based on your suggestions. Please take a look:

Harder math and a standard coding benchmark (W1).

We had chosen KodCode because its executable tests provide a verifier and its dataset size is comparable to the other domains. To test a harder math benchmark, we’ve run experiments on MATH. Accuracy is 40.7% for the untrained student (before our RL training), 46.6% after training with Turing test / self-attribution, and 49.1% for Quality. We also ran on BigCodeBench-Instruct as a coding benchmark. On the test set, pass@1 is 15.2% for the untrained student, 20.3% for self-attribution, and 21.9% for Quality. We believe these new results demonstrate the Turing Test provides a useful learning signal in a much broader range of datasets.

Reasoning students (W2).

Following your suggestion, we tested two new reasoning students. With DeepSeek V4 Flash as teacher on CommonsenseQA, an Olmo-3-7B-Think-DPO improves from 71.1% to 74.0% (+2.9 percentage points, paired 95% CI 0.5 to 5.2), and a Qwen3-1.7B with thinking enabled improves from 71.5% to 73.1% (+1.6 percentage points, CI −0.4 to 3.6). Both started at higher accuracy than the original student, which scored 57.5%. This leaves less room for improvement (less of an initial “capability gap”).

Purpose, alternatives, and compute (W3).

Our paper studies teacher–student knowledge transfer through RL with GRPO, with a reward built around the Turing test: can the teacher distinguish the student's answer from one it could have produced itself? Our aim is to test whether the signal used to make this distinction can also support learning, a fundamental question about Turing tests, rather than to propose a method that outperforms SFT, Oracle, or Quality.

“Quality” and self-attribution (Turing test) use the same training setup but ask the teacher to make different judgments. Quality asks whether the answer is correct; self-attribution asks whether the teacher could have produced it. We expected Quality to be more directly aligned with task accuracy because it explicitly asks about correctness. Self-attribution could also reflect stylistic features that make an answer more teacher-like without making it more correct (in fact the higher Quality scores on MATH and BigCodeBench are consistent with this expectation). The observed formatting failures also show why making an answer resemble the teacher's output should not be equated with improving its correctness. Our results demonstrate the Turing test’s signal is useful for learning.

We’ve added to the Related Work: “For comparison, Constitutional AI (Bai et al., 2022) and the standard RLAIF method (Lee et al., 2024) use AI preferences between candidate responses to train a separate reward model, which then supplies the RL reward. Here, the teacher directly supplies the judgment used to compute the reward. Direct-RLAIF also avoids a separate reward model; our specific contribution is to study learning from the authorship judgment.”

Again, our focus was on testing the Turing-test learning signal, rather than proposing a state-of-the-art training method, and we did not include a compute comparison. Recorded training time is 6.1–10.8 hours for self-attribution, 10.3–14.3 hours for Quality. SFT takes 0.9–3.0 hours to generate teacher solutions and 0.27–0.54 hours to train; Oracle takes 0.36–28.0 hours. These times include waiting, and SFT and RL use different amounts of supervision; we do not find a general efficiency advantage.

Teacher likelihood as a reward (W4).

Although DeepSeek is open-weight, we accessed it through a hosted OpenRouter endpoint that returns log probabilities for 20 next-token candidates; the proposed reward requires the probability of every token in the student’s answer which our endpoint does not support (Turing-test based RL needs only the top token prob).

Why use verifiable tasks? (W5).

We chose them to measure two things independently of the teacher: whether test-set performance improves, and what the attribution score reflects. Gold answers are used for those measurements, not for the self-attribution reward / RL updates. Their availability also enables the Oracle comparison. This demonstrates learning without a verifier contributing to the reward.

Presentation and missing labels (W6, Q1).

The Gain column was intended to highlight improvement over the untrained student and show that a reward based on the Turing-test judgment can support learning. Our aim was not to rank the training methods, which is why we did not include the column in every table. But we agree and have removed the Gain column.

In our OpenRouter setup, we initially request the log probabilities of the top five candidate tokens for the first output token. Both labels, 0 and 1, appeared in the top five in 95,995 of 96,000 evaluable initial responses; either 0 or 1 was missing in five cases (0.0052%).

Thank you for the additional experiments. On both new benchmarks, the Quality reward outperforms the Turing-test reward, which weakens the motivation for the proposed signal. I will keep my score.

Reviewer_n9D9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether indistinguishability in the Turing test can be directly used as a RL reward signal to improve the actual capabilities of weaker student models on objective tasks, without ground-truth labels, teacher generated reference answers, or teacher-based correctness evaluation. The authors propose a domain-based teacher-Turing-Test framework and a log self-attribution probability reward, �0��0�. For closed-source models, they further introduce a repeated sampling voting protocol. Experiments across four verifiable domains, including commonsense reasoning, mathematics, code generation, and SQL, show that making Qwen2.5-1.5 indistinguishable from DeepSeek V4 Flash leads to test accuracy gains of 5.4% to 13.0%.

Critical strengths
  1. The paper introduces identity self-attribution from the generalized Turing test into reinforcement learning and proposes a lightweight post training approach that requires neither ground truth answers, teacher generated long form responses, nor explicit correctness judgments. This offers an interesting perspective on capability transfer from black-box models.
  2. The paper provides a relatively detailed investigation into why imitating a teacher's identity can translate into improvements in objective task performance. Specifically, it examines the ranking of unit test assertion pass rates within all incorrect sampling groups, conducts residual regression analysis based on 22 dimensional surface features, and performs counterfactual intervention experiments to explore the underlying relationship.
Critical weaknesses
  1. Under the same label free setting and with the same single-token overhead, the self-attribution signal �0��0� achieves a lower or, at best, comparable AUC for distinguishing correct answers compared with directly asking the teacher to evaluate correctness using �1��1�. Its only lead in the average Spider score comes from a formatting drift in Quality Seed 23, which results in a zero score and artificially lowers the average. This raises the question of whether simply rephrasing the correctness judgment as "Did you write this?" serves as a noisy and suboptimal proxy for direct correctness evaluation.
  2. Despite substantial teacher-student capability gaps of 25.8% to 40.6%, the method achieves only a 5.4% improvement on KodCode. Moreover, most performance gains across domains are concentrated within the first 500 training steps. After 1,000 steps, the deception rate continues to increase, while test accuracy stagnates or even declines. The paper also evaluates the method using only a single 1.5B student model and 1,000 training samples, leaving its generalizability insufficiently established.
  3. The qwen2.5 model used in this paper is a bit old, making the benefit of the proposed method seems to be marginal. The qwen2.5 model family suffers from the Spurious Rewards issue: https://arxiv.org/abs/2506.10947. The performance on simple benchmarks improves as long as you train it, even with a wrong reward signal. The authors should add more discuss and analysis of this issue.
Questions for authors
  1. Both the Spider failure of Quality (Seed 23) and GPT-5 Mini's GSM8K performance drop result from formatting issues. After standard answer extraction, what is Quality's actual Spider score? Does Indist. still underperform Quality across all four domains?
  2. Does the method generalize to same-family teachers (e.g., Qwen2.5-72B) or larger students (e.g., 7B), beyond the tested Qwen2.5-1.5B setup?
  3. Why does Spider use a generic prompt naming the models, while the other domains explicitly mention problem-solving strategies? Would (log⁡pind\log p_{\text{ind}}) remain correlated with correctness without this guidance?
1 comment

Thank you for your useful feedback. We are excited to report that we’ve run multiple additional experiments, clarified the distinction between correctness vs. authorship, rechecked formatting failures, and added a shuffled-reward control. Please take a look.

W1/Q1

We agree that self-attribution is not better than correctness. Our paper’s core question is whether the signal of the Turing test can also support learning. A verification-based reward aims to improve accuracy by rewarding correct answers; we ask whether learning is also possible when the teacher is asked to distinguish its own authorship rather than verify.

“Quality” provides a comparison under the same training setup: it asks the teacher whether the answer is correct, rather than self-attribution. Neither uses a reference answer, and both require a single-token judgment. We expected attribution (the reward based on the Turing test) to reflect both content and style. Its association with correctness shows that it contains task-related information, even though it is not directly targeted at correctness.

For Spider Quality seed 23, extracting the SQL query recovers 610/1,034 correct answers (58.99%), compared with 0% under strict scoring. The corrected three-seed mean is 60.28%, versus 59.77% for self-attribution. With this correction and the published scores for the other domains, Quality has the higher mean on all four domains, but again this was expected.

For the original GPT-5 Mini GSM8K run, extracting the final numeric answer gives 858/1,319 correct answers (65.05%), compared with 33.06% under strict scoring. The frozen student reaches 56.79% with the same extraction rule, still leaving a gain of 8.26 percentage points using the Turing-test based reward.

W2

The initial teacher–student capability gap shows how much improvement is possible, but does not determine how much the reward will transfer. Indeed KodCode improves by 5.4 percentage points. Because attribution can reflect style as well as correctness, a student may become more teacher-like without solving more problems. This is why attribution keeps rising while test accuracy levels off – most gains occur within the first 500 steps. We believe this is because the small model overfits and makes its largest learning gains within a few steps, which are sufficient to transfer capabilities. In other words, improvements in task-solving ability occur early, while later training may also transfer stylistic features that lead to incorrect answers or no further gains in capability. We therefore attribute the slowdown both to slower capability transfer and to stylistic changes that make responses more teacher-like at the expense of correctness (that is, changes in response format caused by the indistinguishability reward can lead to correct answers being marked as incorrect; we run additional analysis extracting only the final answer before scoring, which reveals accuracy improvements across all checkpoints).

W3

We agree that spurious rewards are a concern. We report a new control that randomly permutes the teacher's scores among the student answers to the same question. Results: on CommonsenseQA, accuracy rises from 57.5% to 70.6% with the original reward but only 65.0% with shuffled scores. This may be because changing the assignment of rewards within a group can still produce a learning signal, just a less informative one. An interesting experiment that we have not run would be to exchange rewards between groups (shuffling rewards across groups rather than only permuting the 4 rewards within each group). We also tested new students, reported below, and observed improvements that lead us to rule out the possibility that the effectiveness of this method is due to spurious rewards.

New results with larger students (W2, Q2)

On CommonsenseQA, with Qwen2.5-72B-Instruct as teacher, a Qwen2.5-1.5B-Instruct improves from 57.5% to 68.5%. With DeepSeek V4 Flash as teacher, a Olmo-3-7B-Think-DPO improves from 71.1% to 74.0%, and Qwen3-1.7B with thinking enabled improves from 71.5% to 73.1%. Finally we ran new additional experiments with the original student–teacher paion 2 new benchmark datasets: MATH (improvement from 40.7% to 46.6% accuracy), and BigCodeBench-Instruct (from 15.2% to 20.3). Quality reaches 49.1% and 21.9%, respectively.

Spider prompt variation (Q3)

We used a shorter Spider wording in the paper simply as a prompt variation. With this original prompt, self-attribution training raises Spider accuracy from 47.9% to 59.8%. However, to address your concern we also repeated the experiment with SQL guidance following the structure used in the other domains. The accuracy gain is 12.5 percentage points, an additional 0.6 points compared with the original prompt. The small increase may reflect a more focused judgment due to the guidance provided in the prompt, while the accuracy gains remain similar with both prompts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.