acceptodds
Under review as a conference paper at ICLR 2027

Scaffold Builder: Diagnosis-Guided Data Synthesis from Intervenable Failures

Abstract

The bottleneck of post-training is shifting from insufficient data scale to the scarcity of high-learning-value data. Existing data synthesis methods are often decoupled from the model's current capability state: the generated samples may be more complex or higher quality, yet they do not necessarily target the model's actual capability gaps. To address this issue, we propose Scaffold Builder, a diagnosis-guided data synthesis framework that formulates data construction as a teaching process centered on the model's capability boundary. Inspired by educational scaffolding and the Zone of Proximal Development (ZPD), Scaffold Builder organizes synthesis into an iterative loop of boundary mining, scaffold construction, controlled synthesis, quality filtering, and feedback training. It first identifies samples near the student model's capability boundary, after which a teacher model diagnoses the corresponding failures into a structured RootCause–Fix–Scaffold triplet. The effectiveness of each scaffold is then verified through student-model re-evaluation. Guided by the validated scaffolds and diagnosed root causes, the framework synthesizes targeted training data along two complementary paths and uses a three-stage curriculum to shift from scaffolded learning toward independent answering. Experiments across multiple reasoning benchmarks show that, with only 300 seed examples, Scaffold Builder achieves higher average performance than strong data synthesis and distillation baselines. Further ablation, iterative, model-scale, training-paradigm, and cross-domain experiments demonstrate the effectiveness, scalability, and transferability of the proposed framework.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Post-Training, Data Synthesis, Capability Boundary, Instructional Scaffolding, Curriculum Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_baFhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Scaffold Builder, a diagnosis-guided data synthesis framework that targets the current capability gaps of student language models. Inspired by educational scaffolding and the Zone of Proximal Development, the framework first identifies capability-boundary problems through multi-view reasoning. A teacher model then diagnoses incorrect responses into structured RootCause–Fix–Scaffold triplets. Candidate scaffolds are validated by measuring whether they improve the student's performance on the original problems.

Critical strengths
  1. A well-motivated framework for student-adaptive data synthesis.
  2. Comprehensive experimental evaluation and component analysis.
Critical weaknesses
  1. The performance improvements over the strongest baselines are relatively limited, and the computational efficiency of the framework remains unclear.
  2. The reliability of scaffold validation and its unique contribution to diagnosis-guided synthesis are insufficiently established.
Questions for authors
  1. How does Scaffold Builder compare with existing synthesis methods under matched computational budgets?
  2. How stable are the measured scaffold intervention gains?
  3. To what extent do the diagnosed root causes provide additional value beyond effective hints or general error-driven data generation?
  4. Why not choosing hard examples for training? This is proved effective in existing works[1][2][3].

[1] Llm2llm: Boosting llms with novel iterative data enhancement

[2] Llms-as-instructors: Learning from errors toward automating model improvement

[3] Diagnosing and remedying knowledge deficiencies in llms via label-free curricular meaningful learning

[2] and [3] are missing citations in the related works.

Reviewer_KAjrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Scaffold Builder, a framework that synthesizes post-training data with scaffold hints aimed at the student model's current weaknesses. It frames data construction as teaching, inspired by educational scaffolding and the Zone of Proximal Development. Experiments on math benchmarks showed that Scaffold Builder beats the strongest baseline, LaMini-LM, by 0.17 and 1.64 points.

Critical strengths
  1. The paper addresses an important problem in LLM post-training.
  2. Broad experimental scope: The paper compares against eight baselines, uses two model families, and tests both SFT and GRPO training.
Critical weaknesses
  1. The improvements over baseline methods are small (mostly within 1.0 points). Without confidence intervals or significance tests, it's unclear whether these gaps are meaningful.
  2. The ablation study is hard to interpret. All ablations use a single run of Llama3-8B at iteration 1, and most differences are under 1.0 point, which makes me wonder if the proposed scaffold-guided synthesis, teacher self-verification, and scaffold curriculum are useful in the proposed method. The only component that makes a large difference is the multi-view prompts, without which the performance drops even below the initial model without post-training. Without further explanation, this may imply that the post-training process itself is unstable.
Questions for authors
  1. Table 1 reports "the observed peak at iteration 2," and both students decline at iteration 3. Does this mean the best round was picked using benchmark (test) scores?
Reviewer_axRMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes Scaffold Builder, a diagnosis-guided data synthesis framework that identifies student capability boundaries, diagnoses failures into RootCause–Fix–Scaffold triples, validates scaffolds through student re-evaluation, and synthesizes targeted training data. Experiments on mathematical reasoning benchmarks show improvements over several synthesis and distillation baselines, with additional ablations and scale/transfer studies.

Critical strengths
  • The student-side validation of generated scaffolds is a useful idea: interventions are tested before being used to guide synthetic data generation.

  • The paper is generally well organized and provides extensive implementation details, ablations, and analyses across model sizes and training settings.

Critical weaknesses
  • The main comparison reports the best observed iteration on the evaluation benchmarks. For Qwen3-4B, the claimed advantage over the strongest baseline is only 0.17 points and occurs at this selected iteration, making the result difficult to interpret without an independent validation criterion or uncertainty estimates.

  • Several experimental details weaken the causal conclusions. The stated temperature-0 evaluation is difficult to reconcile with eight apparently distinct generations, and some ablations change data quantity or training exposure together with the component being tested.

Questions for authors
  • How are eight distinct evaluation responses obtained with temperature 0? Please clarify the exact inference configuration.

  • Was iteration 2 selected after observing performance on the reported benchmarks? If so, can results be reported using a pre-specified or validation-selected checkpoint?

  • Can the authors provide uncertainty across training runs and clarify whether ablations and baselines are matched for training data size and compute?

1 comment

Thank you for the clarifications and additional experiments. The fixed-iteration, matched-training comparison strengthens the evidence. However, sustained benefits from iteration remain unestablished, the original headline result uses the benchmark-selected peak, and the repeated ablations lack a repeated Full reference. I therefore retain weak rejection while improving my soundness assessment. I have increased soundness from 1 to 2 based on new improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.