acceptodds
Under review as a conference paper at ICLR 2027

Instruct, Not Answer: Using Instruction Privileges in On-Policy Context Distillation

Abstract

On-Policy Context Distillation (OPCD) has recently emerged as a powerful technique for transferring context to student models and for self-improvement. In OPCD, the teacher is conditioned on privileged information, and the goal is to minimize the Kullback-Leibler (KL) divergence between the privileged teacher and the student, evaluated on student-generated tokens. Many existing studies show that using instance-specific gold answers or gold demonstrations as the default privilege can hurt training performance, especially out-of-distribution (OOD). In this work, we instead design general instructions that target common student mistakes observed on the training samples, and show that such simple instructions can outperform gold as the OPCD privilege. In autoformalization tasks, using a matched formatting instruction as the privilege could outperform gold in OOD accuracy by a large margin. In 7 out of 8 experiments using ProverQA, ProofWriter, and ProntoQA as datasets, and Qwen3-Thinking and Olmo3-Thinking families as models, matched instruction privileges outperform gold in OOD by roughly 4 to 17 points, while remaining on par with gold in-domain. Each instruction is only a few sentences (and thus contains much less information compared to all instance-specific gold) and is applied uniformly to every training sample. These results indicate that a general instruction, one that applies equally to source and target domain examples, can be substantially more transferable than instance-specific gold in OPCD while maintaining in-domain performance.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
On-policy distillation, context distillation, autoformalization, neuro-symbolic AI, post-training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RbAzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies an interesting question in on-policy context distillation: whether simple, reusable instructions can be more effective teacher privileges than instance-specific gold answers. Using autoformalization as a testbed, the authors show that targeted formatting instructions often preserve in-domain performance while substantially outperforming gold privileges under cross-dataset transfer.

Critical strengths
  • The paper challenges the assumption that more privileged information necessarily leads to better distillation. The distinction between instance-specific supervision and transferable behavioral guidance is interesting and potentially useful beyond autoformalization.

  • The experiments cover multiple datasets and two model families. The error decomposition helps connect the observed improvements to concrete failure modes rather than treating instruction design as a black box.

Critical weaknesses
  • The strongest OOD results rely on "matched" instructions. It is unclear how these instructions would be selected without knowing the target domain or evaluating multiple candidates. This matters for whether the reported gains reflect genuine transfer or favorable instruction selection.

  • Table 8 suggests that much of the advantage comes from avoiding the degradation caused by gold privileges, rather than consistently improving over the no-privilege baseline. These are different conclusions, and the paper should distinguish them more clearly.

  • The weaker results on ProntoQA suggest that formatting instructions mainly help when syntactic errors dominate. The paper would benefit from a clearer characterization of when instruction privileges are preferable to gold, rather than presenting them as broadly superior.

  • Solver verdict accuracy does not necessarily guarantee faithful FOL translation. Moreover, experiments use only one training seed, making it difficult to assess the stability of the reported improvements.

Questions for authors
  • How are matched instructions selected in a genuinely unseen target domain? Can the authors demonstrate a target-blind selection strategy?

  • Would combining gold privileges with targeted instructions outperform either alone? This could help determine whether the problem is the gold information itself or how it influences the teacher.

  • Can the authors evaluate the semantic fidelity of generated FOL independently of solver verdict accuracy?

Reviewer_USS2OpenReview ↗
Rating
1
Confidence
2(−1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses how to provide a teacher model with short instructions instead of ground-truth answers in On-Policy Context Distillation (OPCD). The authors design five instructions by analyzing common errors and use these instructions as context. Across three logical reasoning datasets (ProverQA, ProofWriter, and ProntoQA), the proposed approach outperforms the approach of providing ground-truth answers to the teacher model.

Critical strengths

Practical research question. This paper addresses a practical research question. Instead of designing context for each individual example, the authors explore a method that uses a common set of instructions across all examples.

Useful additional analysis. The authors categorize the student's errors by type and report that using ground-truth answers as context is more effective than the proposed approach for certain error types.

Critical weaknesses

The novely of the idea and lack of comparison with relevant baselines. RGSD [1] and RuPI [2] are similar OPCD methods that use rubrics instead of exact answers. However, this paper does not compare the proposed method with these approaches, making it difficult to determine where the performance gains come from.

Limited experimental scope. The experimental setting is restricted to logical reasoning datasets. All experiments involve translating sentences into first-order logic, including the out-of-distribution (OOD) experiments. Therefore, additional experiments in other domains, such as mathematics, are needed to demonstrate the generalizability of the proposed approach.

Insufficient qualitative analysis of performance gains. The qualitative analysis does not sufficiently explain the observed performance gains. In ProverQA, the most effective instruction, "Coref-split," does not reduce the corresponding coreference-splitting errors. Therefore, further analysis is needed to identify which components contribute most to the performance gains.

[1] https://arxiv.org/abs/2606.12507

[2] https://arxiv.org/abs/2608.02948

Questions for authors

Which data split was used to select the final instructions? The instruction design is based on error analysis of the training split, but it is unclear whether the instructions were further refined or selected using the validation or test splits. Please clarify the process used to refine and select the final instructions.

What happens if both ground-truth answers and instructions are provided to the teacher model? Since providing ground-truth answers and instructions separately improves performance, a natural question is whether these two types of context provide complementary information that could lead to further performance gains when combined.

Reviewer_nBgFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the privilege information design in context-distillation for LLMs. They first argue that using gold answers as privileged information, while improving in-domain performance, can actually hurt OOD performance. To fix this, they propose using general instructions that target the student's common mistakes as the privileged information. By doing controlled experiments on two model families at multiple sizes on AutoFormalization benchmarks, they show the efficacy of their method in improving in-domain performance while causing less degradation, or even improving, OOD performance.

Critical strengths
  • The topic of the paper, On-policy context-distillation, is very important, and the design of the privileged information is a key aspect of it. The paper does a good job of motivating and illustrating the challenge. I find the observation that less informative but more general privileges can lead to better OOD performance interesting.

  • The experiments are reasonably comprehensive within autoformalization, covering three datasets, different teacher sizes, and two model families. The OOD results in Tables 7 and 8 are particularly interesting, showing that instruction privileges can avoid the performance degradation caused by gold privileges.

Critical weaknesses
  • The question that the paper claims to study is broader than the evaluation setting. In particular, it poses the use of instructions instead of gold answers in context distillation, but it only tests autoformalization benchmarks with a focus on formatting errors. I'd like to see how this method would generalize to longer reasoning/agentic tasks such as math, code, and terminal.

  • It seems that the authors have manually designed the instructions by observing the failure cases of the student. This doesn't seem to be a scalable approach and would require substantial human supervision. I'm curious to see how the instruction design can be automated and hopefully show larger gains by scaling the compute.

  • A key motivation behind context-distillation and, in particular, self-distillation is self-improvement, which means the same base model should be used as its own teacher. However, most of the main experiments in the paper seem to be done using a larger teacher.

Questions for authors
  • Have you considered self-distillation with instruction privileges? I'd like to see some ablations on the effect of using larger teachers vs. better-privileged information.

  • Have you considered a baseline where the teacher is given the gold answer but instructed to learn general reasoning rules from it rather than directly using the answer? I wonder if combining gold answers with instructions would give similar or better results.

  • Did you evaluate teacher performance when generating conditioned on their instructions? I would like to know the upper bound of the teacher that they were learning from? And maybe even ablating on different instructions.

  • Do you have a recipe for how to make (or tell) a good set of instructions before doing the training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.