acceptodds
Withdrawn from ICLR 2027

WHAT SHOULD A SELF-DISTILLATION TEACHER SEE? SOLVER FEEDBACK FOR LOGICAL REASONING

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A theorem prover can check a language model’s program and explain what went wrong. Self-distillation lets a teacher copy of the model see privileged context during training while the student never sees it at test time. What should the teacher see? We test three answers on ProverQA with Qwen2.5-7B-Instruct, using Self- Distillation Policy Optimization (SDPO) after a GRPO warm start. A sibling’s verified program ties GRPO in a matched 30-step pilot, +0.67 points [−3.17, +4.50] on a 600-item holdout, and trails it by 4.60 points after 150 steps at the same mini-batch size; moving the program between prompt slots changes accuracy by +0.67 points [−0.98, +2.31] in this seed. Prover diagnostics cost 18.07 points against the program and teach the defect they complain about. An uninformative context collapses program generation: a constant string in a 150-step run, and an empty slot in a fresh matched pilot that scored 0 of 600 on each of two datasets with everything but the payload fixed. A pre-run probe of the teacher’s preference margin ranks the verified program first, so a context could be screened before a run, though the ranking depends on showing the teacher the failed attempt. But every added context scores below the plain prompt, so the probe is a relative screen, not a certificate. Every arm has one seed and one model. The pilot holdout is the clean evaluation; the 1,500-item test set was reused in development.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Post-Training, Reinforcement Learning with Verifiable Rewards, Privileged Supervision, Self-Distillation Policy Optimization, Neurosymbolic Reasoning, Logical Reasoning, First-Order Logic, Reward Modeling, Verifier Feedback

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_c3iEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors compare different forms of privileged information (PI) used during self-distillation. The work is rooted in the logic domain with programs verifiable via logic solvers, which provide diagnostic feedback that can be used as PI for the teacher model during self-distillation. Comparing the different forms of diagnostics inserted as PI during self-distillation, the authors find that the models mostly collapse, and with their best setup, they find that self-distillation ties GRPO.

Critical strengths

None

Critical weaknesses

Overall, I find the paper very hard to follow and comprehend. The authors use terminology before defining it, making the text hardly comprehensible. E.g. what do you mean in the abstract with "A sibling’s verified program", "in a matched 30-step pilot", "moving the program between prompt slots changes", "Prover diagnostics cost 18.07 points against the program and teach the defect they complain about", "A pre-run probe of the teacher’s preference margin ranks the verified program first", etc. After reading the abstract, I have little to no understanding of what you are doing in the paper. Further, this continues throughout the paper.

To me, the overall method, contribution, and findings of the paper are unclear. After reading the paper, it remains unclear to me what its main message is.

Questions for authors

What do you want to show with this paper, and what is the main message? And what kind of solver feedback is most valuable for SDPO?

Reviewer_gNdWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work aims to investigate what type of privileged information a self-distillation teacher should be given and compares self-distillation with standard RL training (GRPO). The paper uses ProverQA and Qwen2.5-7B-Instruct as the testbed/training model. In the evaluated setting, the main findings appear to be (1) that self-distillation does not show a clear advantage over GRPO. And (2) that providing a sampled, verified solution to the problem as privileged information performs better than providing diagnostics or uninformative context.

Critical strengths
  • The research question of what privileged information a self-distillation teacher should receive is interesting and timely.
  • The authors acknowledge some of the limitations of the work themselves and discuss them at length.
Critical weaknesses
  • Presentation: The writing makes the paper difficult to follow. I found myself going back and forth multiple times to understand the explanations/sentences. Besides, an overview diagram would help guide the reader through. The conclusions are also overly compressed in terms of text, the paragraphs consist of short declarative sentences that look like bullet points spaced out in paragraphs. Some terminology like “sibling sample” is also introduced very early in the intro without being defined yet.

  • Experimental scope: While the question is timely and interesting, the experimental setting is too limited to support broad conclusions. Training is conducted on a single dataset using a single 7B instruction model. It remains unclear whether the same observations apply to “thinking” models, other model families, larger variants, training on other datasets or even training seeds.

  • Analysis and practical relevance: It is unclear what practitioners can reliably infer from the analysis or how they could use it to improve self-distillation/llm posttraining. In particular, the finding that diagnostic feedback underperforms a sampled verified solution could be specific to this model and setting. Stronger or more recent models could behave differently. Furthermore, the observed collapse could benefit from a deeper investigation and mitigation mechanism and it remains to be verified if the behaviour holds across model families/sizes.

Overall, I find that the writing needs significant improvement and the paper requires a major rewrite. Further, the current presentation and experimental evidence are not sufficiently convincing to establish a reliable practical insight. I recommend substantial revision. The paper would also benefit from a clearer explanation of what practitioners can reliably infer from its findings. I appreciate some of the main limitations are covered by the authors in the limitations sections, however acknowledging them does not resolve them.

Questions for authors

see above

Reviewer_sum9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies what the teacher should see in Self-Distillation Policy Optimization (SDPO). The task is logical reasoning: the model translates natural-language premises into a first-order logic program that is solved by the theorem prover Prover9. In SDPO, the teacher is a copy of the model that additionally sees a privileged context, and the student, without that context, is trained to match the teacher's next-token distribution. Using Qwen2.5-7B-Instruct on ProverQA from a GRPO warm start, the authors compare teacher contexts containing a verified sibling program, prover diagnostics, a constant string, or an empty payload. A verified program ties GRPO, and its placement does not matter; prover diagnostics score 18.07 points below the program, and the student makes more of the errors the diagnostics describe; an uninformative context collapses program generation, and in a matched 30-step pilot that changes only the payload, the empty payload scores 0/600 on both datasets. The authors also propose a pre-training probe to screen contexts. All experiments use a single model and a single seed.

Critical strengths

The controlled design is careful and the claims are restrained. In the 30-step pilot, the program and empty-payload arms share the same wrapper, eligibility rule, loss, optimizer and warm start, differing only in what the teacher sees; the authors separate the reused test set from a clean holdout and state the single-model, single-seed limitation in the abstract.

Critical weaknesses

There is no positive result, and the negative findings do not provide enough new information.

(a) The collapse under an empty context is expected: the teacher sees nothing informative yet still provides a target distribution under a fixed wrapper, so the student being pulled off course is a natural outcome. The limitations section of the SDPO paper (Hübotter et al., 2026) already notes that a model may be unable to learn from uninformative or misleading feedback.

(b) "A verified program ties GRPO" carries little information after 30 steps with a single seed and confidence intervals of about ±4 points.

(c) The only potentially surprising finding is that prover diagnostics hurt, which runs against the view in Section 4.6 of the SDPO paper that environment output is useful feedback, yet the paper does not discuss this difference; the finding comes from one model, one seed and one specific diagnostic format, and its proposed mechanism is described inconsistently (see Question 3), which is not enough to support a counter-intuitive conclusion.

This determines my recommendation: for a paper without positive results, I would expect its findings to substantially change readers' expectations; the current findings are either expected or not supported strongly enough to be convincing.

Questions for authors
  1. The authors note that the 30-step pilot is cheap to replicate. Could you repeat the PG, PP and PE arms with multiple seeds and on at least one other model?

  2. How do your conclusions compare with those in Section 4.6 of the SDPO paper? For example, why are prover diagnostics harmful in your setting, whereas the SDPO paper finds environment output to be useful feedback?

  3. Section 5.2 partly attributes the harm of diagnostics to "the distillation loss being masked to responses with a verified sibling, which diagnostics rarely yield"; however, Section 3.3 and Appendix B.12 state that eligibility in the diagnostic arm depends on whether feedback exists, not on a successful sibling. Please clarify which eligibility rule was actually used.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.