acceptodds
Under review as a conference paper at ICLR 2027

Plan What the Reply Needs: Requirement-Guided Synthetic Supervision

Abstract

High-quality synthetic supervision requires more than correct content: a response must also decide what to ask, explain, qualify, or correct for a particular user. We study whether making these response requirements explicit during data construction yields improvements that students can learn. We present a plan-then-revise approach built around ASMF, a planning interface defined without domain terms that records concrete requirements, relevant conversational states, response actions, and their expression. A planner reads only the conversation, and a reviser uses its plan to improve the teacher's draft. Construction uses neither evaluation rubrics nor candidate selection, and students learn only from the resulting conversation-response pairs. We test this on HealthBench, a medical benchmark whose physician-written rubrics grade what each reply needs. On 4,000 training conversations, plan-guided revision raises the teacher's rubric score by 6.8 percentage points (95% CI [6.2, 7.5]) over generic revision, which applies the same revision rules without a plan. Across three student backbones from two model families (Qwen3.5 4B and 9B, Ministral-3 8B), students trained on plan-guided revisions exceed those trained on generic revisions by 5.0 to 7.1 percentage points in all six comparisons on held-out HealthBench Hard and the independently released HealthBench Professional, with every 95% confidence interval excluding zero, while generic revision gives no reliable improvement over direct answers. The gains are largest in completeness and context awareness. These findings support explicit response planning as a practical way to construct synthetic supervision, with benefits that persist when students answer without any plan.

Primary area:
foundation or frontier models, including LLMs
Keywords:
synthetic data, supervised fine-tuning, response planning, plan-then-revise, knowledge distillation, rubric-based evaluation, HealthBench, post-training

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SnnUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a plan-then-revise method for synthetic data generation, aiming to capture more requirements in conversations than just answering questions correctly. The core of the method is the planning framework ASMF, a four-layer planning structure consisting of facts and requirements, states, modes, and attributes to guide response revision. The method is tested on HealthBench, showing better performance than generic revision, and the generated data are further used to train three student backbones to demonstrate its effectiveness. Further analysis shows that the largest improvements are in completeness and context awareness.

Critical strengths
  1. The motivation is meaningful and study a practical problem. The paper focus on an important limitation in synthetic supervision: a response can be factually correct while still failing to satisfy the specific requirements of the user's request.

  2. The improvements is consistent at both the teacher and student levels on HealthBench.

Critical weaknesses
  1. The novelty and contribution of the paper need to be further considered. Prior works like STICK have already explored generating task-specific requirements to guide response refinement, while modeling conversational states, response actions, and communication styles has also been studied in works like SFMSS. The paper does not clearly establish the advantages of the proposed ASMF framework over existing methods.

  2. Although ASMF is proposed as a general method, the experiments are conducted only on medical dialogues. Its effectiveness in other domains remains unclear. And he evaluation relies solely on LLM judge without human evaluation to further validate the results.

Questions for authors
  1. How does ASMF compare with existing planning-based revision methods, such as STICK?

  2. The evaluation relies on a single LLM judge without human evaluation. Can the authors provide human evaluation to further validate the results?

Reviewer_jwNfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The problem setup is as follows:

  • You have a tranining dataset of questions and some form of answer or answer rubric to judge against.
  • You wish to train a model of this dataset, by constructing a set of "good" answers to the questions, and then SFT'ing on them
  • You have access to a stronger model than the one you wish to train, which can aid you in answer construction.

The naive way to do this is to sample answers from a stronger model. However, these are off-policy for the weaker model - or as the paper puts it "they carry the generating model's defaults".

Alternatively, you may sample many outputs from the weaker model and use the stronger model to pick the best.

The paper suggests a different way: produce a plan for the answer and, independently a 'draft' answer. Compare the draft to the plan and rewrite it as the final answer that gets trained on. It is unclear whether the plan and draft are produced by the same model with access to the same information, but my assumption is that the plan must either be a) generated by the weak model but with access to the answer guidance or b) produced by the strong model or c) both.

They compare this to training on the draft, and training on a rewriting of the draft without access to the plan.

They do this for the HealthBench training set, and evaluate on held-out HealthBench Hard and Professional, seeing ~5-10% gains in accuracy on 3 different base models.

Critical strengths
  • The method is a nice extension of existing work on checklists and synthethic supervison
  • Results on healthbench are strong
  • The further analysis is modest but interesting: breaking down improvements vs rubric axis and showing where the improvements happen the most
Critical weaknesses
  • The writing is extremely difficult to follow
  • It is not clear where the strong and weak models are used
  • The related work section is overly long and does not provide much clarity into where this sits with existing work
  • No other methods for synthetic supervision are presented for comparison other than their own baselines
  • Results beyond HealthBench are required to demonstrate that ASMF is not overfited to that setting
Questions for authors

I recommend splitting up the work into clear background, problem setting, related work and method section. This would make it clearer where existing work, baselines and your work fits in. If the authors can show results beyond HealthBench, or can improve writing quality, i would consider revising my score.

Reviewer_bQrhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether explicit response planning improves synthetic supervision and whether students can learn the resulting revisions within a plan-then-revise framework.

A planner reads only the conversation and produces a plan containing facts and requirements, conversational states, response behaviors, and expression attributes. A reviser uses this plan to improve a shared teacher draft. The resulting conversation–response pairs train students through standard supervised fine-tuning.

On HealthBench, plan-guided revision improves teacher scores over generic revision by 6.8 percentage points. Across three student backbones and two held-out test sets, the corresponding student improvements range from 5.0 to 7.1 points, primarily in completeness and context awareness.

Critical strengths

1. The paper addresses a useful question and evaluates the full path from data construction to student learning. Improving teacher responses does not automatically improve a student trained on those responses. Testing this transfer across Qwen3.5 4B and 9B and Ministral-3 8B is a meaningful contribution. All six ASMF-versus-generic comparisons on HealthBench Hard and Professional have positive paired bootstrap intervals. Professional provides an additional test beyond the held-out subset of the original release, although both tests remain within medical rubric-based evaluation.

2. The experimental design controls several important sources of variation. The revision arms share the same initial drafts and core revision rules, all arms retain one response per task without quality-based selection, and students use the same training recipe. Evaluation rubrics are withheld from the planner and revisers, and the judge does not receive arm labels or plans. These choices make the improvement over the specified generic-revision pipeline reasonably convincing.

3. The reporting is detailed and appropriately qualified in several respects. The appendices provide the prompts, atom definitions, training configuration, model revisions, and cost information. The authors also examine negative-point criteria and alternative score aggregation. In particular, the improvements remain positive under per-task clipping and when negative-point criteria are removed. These analyses help establish that the result is not simply an artifact of the reported aggregation convention.

Critical weaknesses

1. ASMF's contribution beyond a self-generated checklist is unclear.

F is effectively a list of independently checkable requirements, similar to a self-generated rubric. The authors discuss checklist methods, but provide no direct comparison. The student gains establish the value of ASMF, but do not isolate the four-layer structure's benefit over a simpler checklist or F-only plan at a comparable budget.

2. Concrete requirements may overconstrain valid answers.

Many open-ended requests admit several good answers with different content. A planner that sees only the conversation may mistake content specific to one answer for a necessary requirement. The reviser is instructed to preserve reasonable draft choices while adding every applicable missing plan item. However, it is unclear how applicability is determined when a valid answer reasonably omits planned content. Testing only on medical benchmarks leaves this issue unresolved in other domains.

3. The evidence for the scope and quality of the learned gains is limited.

All trained students are 4B–9B Base models. It remains unclear whether ASMF also improves stronger students. The teacher's gains from additional inference-time planning do not establish gains from self-distillation. Moreover, all results rely on one judge, and improvements mainly concern completeness and context awareness, with smaller, inconsistently significant accuracy gains. Without response-length statistics or independent assessment, the value of the added content remains insufficiently validated.

Questions for authors
  1. How does ASMF compare with a conversation-only checklist or F-only planner using the same coverage rules and a comparable budget, at both teacher level and for one student?

  2. How do you distinguish necessary requirements from content specific to one valid answer? Can a small audit show whether revision preserves legitimate alternatives without unnecessary additions, ideally including nonmedical examples?

  3. Does the training advantage persist for a stronger, already post-trained open model? This would help assess applicability beyond small Base students without requiring fine-tuning the proprietary teacher.

  4. What are the response lengths by arm, and does a small independent evaluation confirm that the added content is relevant and correct?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.