- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a plan-then-revise method for synthetic data generation, aiming to capture more requirements in conversations than just answering questions correctly. The core of the method is the planning framework ASMF, a four-layer planning structure consisting of facts and requirements, states, modes, and attributes to guide response revision. The method is tested on HealthBench, showing better performance than generic revision, and the generated data are further used to train three student backbones to demonstrate its effectiveness. Further analysis shows that the largest improvements are in completeness and context awareness.
Critical strengths
-
The motivation is meaningful and study a practical problem. The paper focus on an important limitation in synthetic supervision: a response can be factually correct while still failing to satisfy the specific requirements of the user's request.
-
The improvements is consistent at both the teacher and student levels on HealthBench.
Critical weaknesses
-
The novelty and contribution of the paper need to be further considered. Prior works like STICK have already explored generating task-specific requirements to guide response refinement, while modeling conversational states, response actions, and communication styles has also been studied in works like SFMSS. The paper does not clearly establish the advantages of the proposed ASMF framework over existing methods.
-
Although ASMF is proposed as a general method, the experiments are conducted only on medical dialogues. Its effectiveness in other domains remains unclear. And he evaluation relies solely on LLM judge without human evaluation to further validate the results.
Questions for authors
-
How does ASMF compare with existing planning-based revision methods, such as STICK?
-
The evaluation relies on a single LLM judge without human evaluation. Can the authors provide human evaluation to further validate the results?