- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes C4-EDS, a framework for planning synthetic educational dialogue datasets before generation. The central idea is to represent Content and Task as independently addressable semantic objects, explicitly define their admissible relations, and compile legal Content–Task Pairs into executable Jobs with integer candidate Slot allocations. By preserving Pair, Job, and Slot identities throughout generation and quality filtering, the framework enables dataset composition to be planned beforehand and audited afterward.
The paper evaluates C4-EDS in K–12 Classical Chinese tutoring through three experiments. E1 demonstrates the construction of an enumerable planning space containing 2,807 legal Content–Task Pairs and 5,614 executable Jobs, along with exact materialization of integer allocation targets. E2 compares four progressively structured generation conditions under a matched candidate budget and reports that full Job-level planning (CTSP) achieves the highest automatic semantic–pedagogical hard-pass rate (88.28%) and the highest mean continuous human quality score. The experiments also reveal that explicit reasoning requirements can improve structural realization while reducing pedagogical quality. E3 investigates downstream supervised fine-tuning using different curated training views and observes a trade-off between Process and Result performance.
Overall, the paper presents a systematic approach to dataset-level planning and quality governance. Its main contribution is a structured representation and workflow for controlling and auditing synthetic-data composition rather than a new generation model or optimization algorithm.
Critical strengths
-
A useful dataset-level perspective on synthetic-data generation. The paper addresses a meaningful limitation of example-centric synthetic-data pipelines: specifying detailed conditions for individual examples does not necessarily provide an explicit representation of the feasible dataset space or its target composition. By separating Content and Task, defining admissible Pairs, and maintaining stable Job/Slot identities, C4-EDS offers a practical mechanism for planning and auditing dataset composition. This perspective is potentially valuable for applications that require systematic coverage and traceability.
-
A reasonably careful empirical analysis with human evaluation. The paper separates structural validity from semantic–pedagogical quality, compares four generation conditions using the same candidate identities and budget, and reports both automated and independent human evaluations. In particular, the finding that explicit reasoning improves early structural validity while substantially reducing semantic hard-pass rates is interesting and practically relevant. The authors also report inter-evaluator agreement, automatic–human calibration, and important limitations of their experimental design, which strengthens the credibility of the descriptive findings.
Critical weaknesses
-
Insufficient evidence for the benefits of explicit planning. Although the paper compares B0, B1, B1-CoT, and CTSP, these experiments mainly evaluate different generator-facing instructions under the same upstream planning space. It remains unclear what additional benefits the explicit Content–Task/Job/Slot planning framework provides over a simpler workflow with equivalent prompting instructions.
-
Lack of component-level ablation studies. CTSP simultaneously introduces role relations, turn/message planning, cognitive progression, pedagogical actions, and closing requirements. Without individual ablations, it is difficult to identify which components contribute to the observed improvements or whether all these design choices are necessary.
Overall Assessment: I find the proposed framework promising and have given the paper a relatively favorable initial rating. However, the above concerns are important for establishing the paper's core contributions. I would be particularly interested in seeing additional experimental evidence addressing these points, as it would help me better understand and assess the proposed method.
Questions for authors
Q1. Could the authors compare CTSP with a baseline that uses identical generation prompts and quality filtering but samples Content–Task combinations without explicit Pair/Job/Slot allocation, under the same candidate budget?
Q2. Could the authors provide component-level ablations of CTSP to identify which execution-planning components contribute most to the improvements over B1-CoT?
Q3. The prompts provided in the appendix are helpful for understanding the implementation. Could the authors also provide English translations alongside the original Chinese prompts? This would improve accessibility for non-Chinese-speaking readers and facilitate reproducibility. Whether or not the authors address this suggestion will not affect my overall rating.