acceptodds
Under review as a conference paper at ICLR 2027

SE-OPSD: Self-Elicited On-Policy Self-Distillation For Open-Ended Task

Abstract

On-policy distillation provides dense token-level supervision on model-generated trajectories. Recent on-policy self-distillation (OPSD) removes the need for external teachers by supervising a model with a privileged copy of itself. In verifiable domains such as mathematics and code, this privilege can be supplied by reference answers or verified reasoning traces. In open-ended QA, however, response quality depends not only on factual content but also on recognizing missing context, calibrating uncertainty, respecting task-specific boundaries, and communicating appropriately. A reference answer may leave these response-framing requirements implicit. A model may identify them when explicitly prompted to analyze a query, yet fail to apply them consistently when answering directly. These elicited judgments constitute natural privileged information: a teacher that has already formed them can supervise a student that has not. We propose SE-OPSD, a reference-free self-distillation method that uses query-specific response context as training-time privileged information. For each query, a frozen teacher copy first elicits a compact, structured checklist of response requirements through complementary views, forming Self-Elicited Privileged Context (SEPC). The student observes only the original query, while the teacher conditions on both the query and SEPC to provide token-level supervision on the student's on-policy trajectory. SE-OPSD thus uses additional inference computation to construct its own supervision and distills these response-framing judgments into single-call answer generation. On HealthBench Consensus, SE-OPSD improves Qwen3-1.7B from 67.86 to 72.88 and Qwen3-4B from 82.15 to 86.58, outperforming reference-answer SFT and reference-conditioned OPSD baselines. Gains in context awareness, communication quality, and completeness, together with improvements on general-domain WildBench, support self-elicited response context as an effective source of supervision for open-ended QA. Our code is available at https://anonymous.4open.science/r/SE-OPSD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post Training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sCPmOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SE-OPSD, a reference-free on-policy self-distillation method for open-ended QA. It introduces Self-Elicited Privileged Context (SEPC), which captures query-specific response requirements through five structured views and guides a privileged teacher during distillation. The student is trained to incorporate this guidance into direct-answer generation without requiring SEPC at inference time. Experiments on HealthBench and WildBench show consistent improvements over base models and reference-based baselines.

Critical strengths
  • The paper extends on-policy self-distillation to open-ended QA, where reliable reference answers or verifiable rewards are difficult to obtain. The proposed approach achieves consistent improvements across multiple models and benchmarks without relying on external teachers.
Critical weaknesses
  • The paper provides limited insight into why this works, when it is effective, or what fundamentally distinguishes it from existing context-distillation methods. Test-time SEPC already achieves better performance, raising questions about the necessity of additional training. The comparison is also somewhat limited, as simpler alternatives, such as generating all five SEPC views in a single call or combining planning and answer generation in one pass, are not explored. Furthermore, the efficiency analysis focuses on inference token counts without accounting for training costs or measuring actual wall-clock latency. Given the relatively short generated responses and the possibility of parallelizing SEPC elicitation, the practical latency benefits may be less significant. Overall, it remains unclear whether the efficiency gains are substantial enough to justify the additional training effort.
Questions for authors
  • How does SE-OPSD compare to a test-time baseline that generates all five SEPC views in a single call (or a single-pass approach that combines SEPC elicitation and answer generation), in terms of performance and inference latency?

  • How much training compute does SE-OPSD require, and how many inference requests are needed to offset this cost?

  • Is there evidence that SE-OPSD learns query-specific response-framing decisions, rather than simply adopting general response patterns?

Reviewer_Y5m2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes SE-OPSD for open-ended QA. A frozen copy of the base model generates query-specific guidance through five prompting views. This guidance conditions the teacher during standard on-policy self-distillation, while the student receives only the original query. Evaluation includes HealthBench, knowledge-retention benchmarks, and WildBench.

Critical strengths

The reported results validate the effectiveness of the implementation well, but do not establish a meaningful advance over existing context- and rubric-conditioned self-distillation.

Critical weaknesses
  1. The contribution beyond existing context- and rubric-conditioned self-distillation is limited.

    The objective in Eq. (2) is standard OPSD [1]. Then, [2] already applies rubric-conditioned OPSD to open-ended generation. Besides, [3] also use generated rubrics as teacher context.

    These works are acknowledged, but the paper does not establish a substantial advance over them. Its remaining distinction appears to be the particular five-view checklist construction. Demonstrating that prompted guidance can be distilled, or that doing so reduces inference cost, does not establish a new contribution when these principles are already demonstrated by prior work.

  2. The experiments do not establish the benefit of the five-view construction over simpler query-specific guidance.

    Appendix C.1 compares single-view and multi-view elicitation using JSON validity and schema adherence, without comparing downstream performance after distillation. Table 6 removes individual views, which also removes their information; it therefore does not isolate the benefit of generating the guidance through separate calls. Generic medical instructions and shuffled checklists are useful controls for query alignment, but neither represents a strong alternative for generating query-specific guidance.

    The prompting and SEPC-SFT comparisons address inference cost and the choice of training procedure. They do not establish an advantage over ordinary prompted rubrics or a single-call checklist used with the same OPSD objective. Consequently, the experiments leave the main remaining methodological distinction insufficiently supported. The additional model and domain evaluations mostly compare against the untrained base model and do not resolve this gap.

[1] On-Policy Context Distillation for Language Models

[2] Rubrics as Privileged Information for Open-Ended Generation

[3] Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation

Questions for authors

I have no further questions.

Reviewer_w7dnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SE-OPSD uses a frozen copy of a model to elicit query-specific response requirements through five complementary views, forming Self-Elicited Privileged Context (SEPC). During on-policy self-distillation, the teacher receives this context while the student sees only the query. The aim is to transfer the benefits of extra response-framing computation into single-call answer generation without reference answers. Experiments on HealthBench and additional tasks compare reference-based training, matched SEPC-SFT, test-time SEPC, and several guidance ablations. The core distillation effect is supported, while the added value of the specific five-view elicitation design remains insufficiently isolated.

Critical strengths
  1. The paper supports a useful reference-free training effect with more than base-model comparisons. It includes reference-answer SFT/OPSD, matched SEPC-SFT, test-time SEPC, and generic/shuffled-context controls. Table 4 provides direct evidence that some benefit of explicit response guidance can be transferred into the student’s parameters, which is relevant to deployment without repeated guidance generation.

  2. The evaluation is comparatively thorough. It reports gains across model sizes, a second grader, paired uncertainty estimates, length checks, and three SE-OPSD training seeds. The view-removal experiments also help characterize the guidance. These results support improvements in response organization, context awareness, and requirement satisfaction, rather than relying on a single aggregate score.

Critical weaknesses
  1. The closest simple alternative is missing from the downstream comparison: ask the same frozen model once to generate a query-specific rubric or checklist, then use it in the same OPSD procedure. Reference-answer, generic, and shuffled-context baselines do not answer whether five-view elicitation improves over this simpler query-aligned guidance. Table 12 compares format/schema validity, not post-training response quality; leave-one-view-out ablations also do not replace the single-call control. Given the closely related rubric-conditioned self-distillation methods discussed in the paper, this missing comparison is central to establishing the method’s incremental contribution and is the main reason for my current weak-reject recommendation.

  2. The stability and scope of the distinctive gains remain uncertain. Three seeds for SE-OPSD are useful, but the matched SEPC-SFT and key component comparisons are not equivalently repeated. The strongest improvements concern response framing and requirement satisfaction, while factual-accuracy and difficult-setting gains are smaller or mixed. Thus the evidence currently supports a narrower behavioral improvement more strongly than a broad reasoning or factual-reliability improvement.

Questions for authors
  1. Decision-critical comparison: With the frozen model, query set, OPSD objective, and optimization budget held fixed, how does five-view SEPC compare with a single-call query-specific rubric/checklist? Please report downstream quality together with guidance-generation tokens and calls.

  2. Decision-relevant stability check: Can you report uncertainty or additional training repeats for the closest single-call baseline and matched SEPC-SFT, prioritizing these comparisons over repeating every small ablation?

  3. Clarification: Which quality dimensions improve consistently across graders and difficult/general-domain settings, and which do not? Please separate the supported response-framing contribution from stronger claims about factual accuracy or general reasoning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.