- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies compositional self-improvement: a model solves the subproblems of a harder problem, a composition rule combines those answers into a pseudo-label, and the model is trained on these labels over several rounds. On three algorithmic tasks, the paper shows this extends the range of solved instances, and identifies three failure modes: wrong component answers, invalid composition rules, and limited coverage of composed labels. It then shows gains over direct self-labeling on Bash pipelines and MuSiQue, and shows that composed labels can be used as an SFT warm start for RL.
Critical strengths
The experiments are clean and well controlled, and the error audit is careful. The paper is also honest about results that do not favor it, such as decomposed inference beating the trained model (Table 10).
Critical weaknesses
-
The approach is too heuristic, and most of the supervision comes from hand-designed rules. Every task needs its own decomposition and composition rule written by the authors: corridor moves in the grid, block concatenation plus a no-carry filter in addition, joining with | in Bash, a fixed wrapper in Python. In Bash, Python, and the string tasks, the steps are even listed in the input. So the hard part of each problem is solved by the rule, and the model only needs to solve the easy pieces. The three failure modes also follow directly from this setup: if a component is wrong the label is wrong, if the rule is wrong the label is wrong, and a rule cannot produce outputs it cannot express. These points are correct, but they do not give a general method or a new insight beyond “design good rules for your task.”
-
Training does not beat simply using the decomposition at test time, and the baselines are weak. In Table 10, the initial model with decomposed inference beats the final trained model on grid (88.6 vs 80.9), Bash (94.9 vs 88.1), and most MuSiQue settings. So the real benefit is distilling a known multi-step procedure into one pass.
Questions for authors
- Given that decomposed inference beats the trained model when the rule is exact (Table 10), what is new compared with System 2 Distillation or iterated amplification? Can you compare against them directly?
- Is there any setting where the model itself proposes the decomposition or the composition rule? Without this, what part of the improvement does the model provide on its own?
3 comments
Thank you for the response and the new experiments. The model-generated decomposition result and the matched comparison with single-stage distillation make the contribution clearer, although each task still relies on hand-designed composition rules. My score is already 3, and I will keep it.
Thank you for your thoughtful follow-up and for considering our new experiments! We would also like to clarify that, in our multi-hop reasoning experiments, the composition procedure is relatively generic: intermediate answers are substituted into dependent subquestions, and the final answer is still generated by the model. We appreciate your constructive feedback!
We thank the reviewer for recognizing the careful experimental controls and detailed error analysis. Below, we clarify the contributions of compositional self-improvement relative to prior work and decomposed inference, and present new evidence on learning from model-generated decompositions.
(Q1) On the contribution relative to System 2 distillation and iterated amplification.
Our main contribution is a systematic characterization of when compositional supervision can sustain self-improvement, rather than a new decomposition or distillation procedure. Building on ideas from System 2 Distillation [1] and iterated amplification [2], we identify three distinct bottlenecks—component reliability, composition validity, and supervision coverage—and investigate their effects through theoretical analysis and controlled experiments.
To examine the benefits of iterative training, we additionally compare CSI against single-stage compositional distillation and a frozen-teacher curriculum, using matched initializers, training budgets, and replay protocols.
CSI outperforms both alternatives under these matched conditions, suggesting that iterative training can provide benefits beyond distilling a fixed set of composed predictions. We view CSI as a concrete instantiation of the broader iterated-amplification paradigm, and do not claim that iteration is universally superior to fixed-teacher distillation.
Regarding Table 10, our objective is not necessarily to outperform decomposed inference, which uses additional computation at test time, but to learn direct prediction from composed supervision. Decomposed inference remains stronger on Grid and Bash, whereas the trained model outperforms decomposed inference on Addition and Max Run Length, where the composition rules are imperfect. One possible limitation of direct prediction is that models with finite capacity may struggle to internalize multi-step computations, particularly for tasks involving long contexts or many dependencies. Although we do not isolate this effect experimentally, these comparisons help characterize when distillation succeeds and when decomposed inference remains preferable.
(Q2) On model-generated decompositions.
We investigate this setting in a new MuSiQue experiment (Appendix C.10). Rather than using annotated decompositions, we prompt models to generate their own subquestions and dependencies from the original question and passages, without reference decompositions or answers. We then construct supervision using the same composition and filtering procedure, keeping the executor, student, and training budget fixed at 9B.
Kept measures the fraction of candidate parents passing the filter; correct-label yield is evaluated against reference answers. Accuracy measures direct prediction after training.
Model-generated decompositions achieve comparable improvements to supplied decompositions, despite yielding fewer correct labels. Smaller planners produce fewer usable plans, indicating that reliable decomposition generation remains important. These results show that CSI can obtain useful supervision without per-example annotated decompositions: the model generates the decomposition plan and predicts its components, while the supplied composition rule combines those predictions into training labels.
References
[1] Yu et al. (2024). Distilling System 2 into System 1. arXiv.
[2] Christiano et al. (2018). Supervising Strong Learners by Amplifying Weak Experts. arXiv.