acceptodds
Under review as a conference paper at ICLR 2027

Compositional Self-Improvement: Promises and Pitfalls

Abstract

Although self-improvement is becoming increasingly important as tasks become more complex, models struggle to generate reliable supervision for harder, out-of-distribution instances. One way to address this challenge is to exploit compositional structure: models often solve simpler subproblems more reliably than the full problem, so their subproblem predictions can be composed into supervision for harder ones. We systematically study when such composed supervision supports self-improvement and when it fails. Our controlled experiments identify three failure modes: unreliable subproblem predictions, errors introduced by composition, and limited supervision coverage. Beyond controlled settings, these findings carry over to more realistic coding and reasoning tasks: composed labels are more accurate than direct self-generated labels and improve performance even on compositions longer than those seen during training, while the failure modes remain visible. Further, although these failure modes often keep compositional self-improvement from fully succeeding on its own, a short SFT warmup with compositional self-improvement can bootstrap reinforcement learning on tasks that require composing skills the model already has, particularly when direct exploration rarely succeeds. Together, these findings characterize when compositional self-improvement can provide sufficiently reliable supervision to expand model capabilities.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
compositional self-improvement, pseudo-labeling, compositional generalization, self-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8hTkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies compositional self-improvement: a model solves the subproblems of a harder problem, a composition rule combines those answers into a pseudo-label, and the model is trained on these labels over several rounds. On three algorithmic tasks, the paper shows this extends the range of solved instances, and identifies three failure modes: wrong component answers, invalid composition rules, and limited coverage of composed labels. It then shows gains over direct self-labeling on Bash pipelines and MuSiQue, and shows that composed labels can be used as an SFT warm start for RL.

Critical strengths

The experiments are clean and well controlled, and the error audit is careful. The paper is also honest about results that do not favor it, such as decomposed inference beating the trained model (Table 10).

Critical weaknesses
  1. The approach is too heuristic, and most of the supervision comes from hand-designed rules. Every task needs its own decomposition and composition rule written by the authors: corridor moves in the grid, block concatenation plus a no-carry filter in addition, joining with | in Bash, a fixed wrapper in Python. In Bash, Python, and the string tasks, the steps are even listed in the input. So the hard part of each problem is solved by the rule, and the model only needs to solve the easy pieces. The three failure modes also follow directly from this setup: if a component is wrong the label is wrong, if the rule is wrong the label is wrong, and a rule cannot produce outputs it cannot express. These points are correct, but they do not give a general method or a new insight beyond “design good rules for your task.”

  2. Training does not beat simply using the decomposition at test time, and the baselines are weak. In Table 10, the initial model with decomposed inference beats the final trained model on grid (88.6 vs 80.9), Bash (94.9 vs 88.1), and most MuSiQue settings. So the real benefit is distilling a known multi-step procedure into one pass.

Questions for authors
  1. Given that decomposed inference beats the trained model when the rule is exact (Table 10), what is new compared with System 2 Distillation or iterated amplification? Can you compare against them directly?
  2. Is there any setting where the model itself proposes the decomposition or the composition rule? Without this, what part of the improvement does the model provide on its own?
3 comments

We thank the reviewer for recognizing the careful experimental controls and detailed error analysis. Below, we clarify the contributions of compositional self-improvement relative to prior work and decomposed inference, and present new evidence on learning from model-generated decompositions.

(Q1) On the contribution relative to System 2 distillation and iterated amplification.

Our main contribution is a systematic characterization of when compositional supervision can sustain self-improvement, rather than a new decomposition or distillation procedure. Building on ideas from System 2 Distillation [1] and iterated amplification [2], we identify three distinct bottlenecks—component reliability, composition validity, and supervision coverage—and investigate their effects through theoretical analysis and controlled experiments.

To examine the benefits of iterative training, we additionally compare CSI against single-stage compositional distillation and a frozen-teacher curriculum, using matched initializers, training budgets, and replay protocols.

Training strategyBash (accuracy)MuSiQue (EM)
Single-stage distillation72.23%51.10%
Frozen-teacher curriculum69.85%52.83%
Iterative CSI81.63%54.67%

CSI outperforms both alternatives under these matched conditions, suggesting that iterative training can provide benefits beyond distilling a fixed set of composed predictions. We view CSI as a concrete instantiation of the broader iterated-amplification paradigm, and do not claim that iteration is universally superior to fixed-teacher distillation.

Regarding Table 10, our objective is not necessarily to outperform decomposed inference, which uses additional computation at test time, but to learn direct prediction from composed supervision. Decomposed inference remains stronger on Grid and Bash, whereas the trained model outperforms decomposed inference on Addition and Max Run Length, where the composition rules are imperfect. One possible limitation of direct prediction is that models with finite capacity may struggle to internalize multi-step computations, particularly for tasks involving long contexts or many dependencies. Although we do not isolate this effect experimentally, these comparisons help characterize when distillation succeeds and when decomposed inference remains preferable.

(Q2) On model-generated decompositions.

We investigate this setting in a new MuSiQue experiment (Appendix C.10). Rather than using annotated decompositions, we prompt models to generate their own subquestions and dependencies from the original question and passages, without reference decompositions or answers. We then construct supervision using the same composition and filtering procedure, keeping the executor, student, and training budget fixed at 9B.

Decomposition sourceKept (%)Correct-label yield (%)Accuracy (%)
Initial model——43.7
Supplied99.265.550.7
Model-generated (9B)74.047.351.6
Model-generated (4B)28.416.446.5
Model-generated (2B)0.10.1—

Kept measures the fraction of candidate parents passing the filter; correct-label yield is evaluated against reference answers. Accuracy measures direct prediction after training.

Model-generated decompositions achieve comparable improvements to supplied decompositions, despite yielding fewer correct labels. Smaller planners produce fewer usable plans, indicating that reliable decomposition generation remains important. These results show that CSI can obtain useful supervision without per-example annotated decompositions: the model generates the decomposition plan and predicts its components, while the supplied composition rule combines those predictions into training labels.

References

[1] Yu et al. (2024). Distilling System 2 into System 1. arXiv.

[2] Christiano et al. (2018). Supervising Strong Learners by Amplifying Weak Experts. arXiv.

Thank you for the response and the new experiments. The model-generated decomposition result and the matched comparison with single-stage distillation make the contribution clearer, although each task still relies on hand-designed composition rules. My score is already 3, and I will keep it.

Thank you for your thoughtful follow-up and for considering our new experiments! We would also like to clarify that, in our multi-hop reasoning experiments, the composition procedure is relatively generic: intermediate answers are substituted into dependent subquestions, and the final answer is still generated by the model. We appreciate your constructive feedback!

Reviewer_3ujBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper discusses compositional self-improvement. Given a decomposition, the model predicts its parts, and a fixed rule combines them into a training label. This process repeats over rounds. The paper studies three bottleneck: unreliable parts, invalid rules, and limited supervision coverage. Composed labels beat direct self-labels on Bash and MuSiQue. A short SFT stage also helps RL learn when direct RL gets no reward.

Critical strengths
  1. Clear diagnosis. The three bottlenecks need different solutions. The coverage finding is interesting: filtering and more rounds cannot supervision coverage, and overall accuracy can hide this failure.
  2. Effective RL initialization. Just 24 SFT updates on labels with 77% accuracy improve direct RL from 0% to 19.2%.
  3. Careful experiments. Matched controls, no reference-answer leakage, negative results are reported.
Critical weaknesses
  1. The decomposition is given. The model only predicts the parts. The costly annotation is already provided, which weakens the claim of reducing human annotation. With exact rules, decomposed inference using the initial model already beats the final model: grid 88.6 vs 80.9. Bash 94.9 vs 88.1. This looks more like distillation than self-improvement.
  2. Diagnosis without a solution. Fixes for modes I and II are standard and need prior knowledge of failure conditions. Mode III, the new finding, has no fix in paper.
  3. Weak theory. A coupling bound, a definition, and a Hoeffding bound. There is no multi-round analysis, and the theory cannot predict when the student generalizes beyond coverage.
  4. Limited model scale. Sec 3 uses 14.2 M transformers. The largest LLM is Qwen3.5-9B with LoRA rank 16. No frontier-scale model is tested.
  5. Limited real-task evaluation. Mode II is tested only on toy tasks. Training lasts three rounds, with declining label accuracy. There are only 3-4 seeds, with no variance reported. Grid and Bash have test contamination. Baselines are limited to direct self-improvement.
Questions for authors
  1. What does training add beyond distilling decomposed inference? How is this different from System 2 distillation?
  2. Did you try splitting blocks again to avoid boundary carries, instead of discarding the parent example?
  3. Can you show mode II when the rule's failure conditions are not known in advance?
  4. Why can the addition model generalize to boundary carries, while the max-run-length model fails beyond its coverage?
  5. Do these bottlenecks remain at frontier scale or with full-finetuning?
4 comments

We thank the reviewer for highlighting the diagnostic value of our analysis, particularly the distinction between label errors and insufficient supervision coverage, as well as the RL initialization results and careful experimental controls. We provide additional experiments and analyses addressing the reviewer's concerns below.

(W1) On supplied decompositions and the role of training. Our main experiments use supplied composition interfaces to isolate CSI's limiting factors. We additionally investigate whether the observed gains extend to model-generated decompositions. In a new MuSiQue experiment, models generate their own subquestions and dependencies. We construct supervision using the same composition and filtering procedure, without using reference answers to select examples.

Decomposition sourceKept (%)Correct-label yield (%)Accuracy (%)
Initial model——43.7
Supplied99.265.550.7
Model-generated (9B)74.047.351.6
Model-generated (4B)28.416.446.5
Model-generated (2B)0.10.1—

All executors and students are 9B, with identical training budgets. Accuracy measures direct prediction after training.

Model-generated decompositions achieve comparable improvements to supplied decompositions, despite producing fewer correct labels. This demonstrates that CSI can obtain useful supervision without per-example annotated decompositions (Appendix C.10).

Regarding decomposed inference, we acknowledge that it can outperform the trained model when components are reliable and composition rules are valid, as in Grid and Bash. However, decomposed inference requires executing the decomposition at test time, whereas CSI learns to predict directly. Conversely, CSI outperforms decomposed inference in Addition and Max Run Length, where the composition rules are imperfect (Appendix C.7). These comparisons help characterize when training on composed supervision is beneficial and when decomposed inference remains preferable.

(W2) On mitigating limited output coverage.

Failure Mode III arises when composed supervision cannot represent outputs required by the task, even if its accepted labels are accurate. We test whether combining composition rules with complementary output coverage improves predictions on previously unsupported cases (Appendix C.13).

We revisit max run length on length-20 binary strings split into two length-10 components. The original maximum rule selects the longer predicted component run and accepts only when the boundary symbols differ; it therefore cannot produce labels for runs longer than 10.

We introduce an approximate crossing rule: when the boundary symbols and predicted winning symbols agree, and the predicted run lengths sum to more than 10, it outputs their shared symbol and summed run length; otherwise, it rejects the example. This rule can produce labels for longer runs but may be incorrect because the winning component runs need not reach the boundary. We compare both rules individually and a 50/50 mixture of their accepted labels.

Supervision strategyLabel precisionAccuracy on runs >10Overall accuracy
Original maximum rule100.00%0.00%95.49%
Approximate crossing rule77.71%65.30%0.61%
50/50 mixture89.61%64.06%89.20%

Despite perfect label precision, the original rule achieves 0% accuracy on runs longer than 10. Combining it with the less reliable crossing rule substantially improves accuracy on these previously unsupported cases while retaining much of the overall accuracy. These results illustrate how diversifying composition rules can mitigate Failure Mode III, although the reliability of additional labels remains important.

(W3) On multi-round analysis and generalization beyond coverage.

We have extended the theoretical analysis in Appendices B.4–B.5 to address both concerns.

First, we analyze how errors propagate across multiple CSI rounds, as each updated model generates supervision for the next. The analysis identifies conditions under which errors can diminish over repeated training, rather than treating each round independently.

Second, we establish a sufficient condition for generalization beyond accepted examples. Although some problems are excluded by the composition filter, learning from retained examples can still improve predictions on excluded problems when the model's predictions across these populations are sufficiently related. Our new result formalizes when improvements on accepted examples transfer to the broader candidate distribution.

(W4, Q5) On model scale and full fine-tuning.

Our main contribution is a systematic characterization of when compositional self-improvement succeeds and what limits it, identifying three distinct bottlenecks: component reliability, composition validity, and supervision coverage. These concern the mechanisms of compositional supervision and are not specific to a particular model scale or fine-tuning method.

Model scale can affect the severity of these bottlenecks. On MuSiQue, first-round composed-label accuracy increases from 44.8% (2B) to 60.5% (4B) and 65.2% (9B), suggesting that greater model capability alleviates Failure Mode I. However, the validity and output coverage of a fixed composition rule do not change with model scale. Stronger models may generalize beyond these supervision limitations, but scaling alone does not eliminate them.

We additionally examined sensitivity to the fine-tuning strategy on Bash (Appendix C.15). Starting from the same Qwen3.5-2B initial model, full fine-tuning achieves 81.75% direct execution accuracy, compared with 85.05% for rank-16 LoRA, averaged over three seeds. Thus, composed supervision also supports strong performance under full fine-tuning, with comparable mean accuracy across the two adaptation strategies.

(W5) On evaluation breadth and experimental controls.

We address the reviewer's concerns as follows:

  • Evaluation breadth and training duration. Our controlled experiments isolate individual failure mechanisms, including composition-rule errors (Mode II), while Bash and MuSiQue evaluate CSI on realistic programming and multi-hop reasoning tasks. Related structural failures have been documented in multi-hop QA: Min et al. [2] identify cases where decompositions omit necessary information or yield ambiguous intermediate answers, making composition unreliable even with accurate component predictions. The three-round schedule applies to our Bash and MuSiQue CSI experiments; our controlled experiments run for 8–12 rounds to study longer-term improvement and its limitations.

  • Train–test overlap and unseen compositions. Grid test chains and Bash test pipelines are disjoint from training examples. Reusing familiar components is intentional, although the original Bash evaluation permits shorter subcompositions to recur across training and test examples. To address this concern, we added a stricter Bash evaluation with held-out adjacent command pairs (Appendix C.11). No adjacent pair in any test pipeline appears in training, ensuring that all test subcompositions of length two or more are unseen. CSI achieves 86.9% accuracy, compared with 5.0% for the initial model and 2.9% for direct self-training, demonstrating generalization to unseen combinations of familiar components.

  • Declining pseudo-label accuracy. Later rounds train on increasingly complex compositions, where component errors can accumulate. For Bash, composed-label accuracy decreases from 90.6% to 83.6% and 80.6% across rounds, but remains substantially higher than direct self-training (7.9%, 2.0%, and 0.5%, respectively). This behavior is consistent with Failure Mode I, which characterizes how component unreliability limits continued self-improvement (Appendix C.8, Table 12).

  • Additional controls and variance reporting. We added matched comparisons against single-stage distillation and a frozen-teacher curriculum, as discussed in Q1. We also report standard deviations across seeds in Figures 6–8 and Tables 13–15 and 18 in the updated manuscript.

(Q1) On what training adds beyond decomposed inference and System 2 distillation.

Decomposed inference can achieve high accuracy but requires multiple component calls for each test problem. Training transfers this capability into direct prediction, reducing inference-time computation. The updated model can also serve as the component solver in subsequent rounds, enabling supervision for progressively harder problems.

System 2 distillation [1] studies the first benefit: transferring expensive inference procedures into direct predictions. Our focus is understanding when this process can sustain self-improvement across multiple rounds. We systematically investigate three bottlenecks—component reliability, composition validity, and supervision coverage—through theoretical analysis and controlled experiments.

To examine the benefit of iteration, we additionally compare CSI against single-stage compositional distillation and a frozen-teacher curriculum, using matched initializers, training budgets, and replay protocols.

Training strategyBashMuSiQue
Single-stage distillation72.23%51.10%
Frozen-teacher curriculum69.85%52.83%
Iterative CSI81.63%54.67%

These results show that updating the source of compositional supervision across rounds can provide additional gains beyond distilling predictions from a fixed model.

(Q2) On repartitioning addition examples instead of discarding them.

We thank the reviewer for suggesting repartitioning instead of discarding examples. We tested this approach when a proposed decomposition violates the no-carry condition required for composing component sums. Our original strategy rejects such examples, whereas repartitioning searches for alternative block boundaries that satisfy the condition. This validity check uses only the input digits, without verifying component predictions or pseudo-label correctness against reference answers.

We compare a matched rerun of the original rejection strategy with repartitioning over eight CSI rounds, using the same initializer and training budgets (3,000 updates per round).

Evaluation (accuracy)Original (reject)Repartition
8–23 digits95.0%96.5%
23 digits84.0%92.0%

Repartitioning increases the number of usable training examples from 63,254 to 150,386 out of 160,000 candidates in the final round, while maintaining comparable pseudo-label accuracy (97.0% versus 96.7%).

These results suggest that searching for alternative valid decompositions can recover useful supervision that would otherwise be discarded. More broadly, when composition validity can be checked, a failed decomposition need not imply that the entire example is unusable.

(Q3) On unknown composition failures. Our controlled experiments use known validity conditions to isolate the effects of invalid composition on self-generated supervision. In practice, failure conditions may not be known in advance, but post-hoc error analysis can help identify them. For example, examining cases where correct component answers produce incorrect composed answers can reveal previously unrecognized failure patterns and guide improvements to composition rules or validity checks. Developing general methods to automatically discover and address such failures remains an important direction for future work.

(Q4) On different generalization behavior in addition and max run length.

The key distinction is input coverage versus output coverage. In addition, filtering excludes certain input configurations with boundary carries, but the model still learns carry operations within individual blocks and can generalize them to new digit positions. In max run length, however, the original composition rule never produces labels for runs longer than its components, leaving those output patterns entirely unsupervised. Thus, generalization can mitigate gaps in input coverage, but output coverage must be explicitly considered when designing composition rules.

References

[1] Yu et al. (2024). Distilling System 2 into System 1. arXiv.

[2] Min et al. (2019). Multi-hop Reading Comprehension through Question Decomposition and Rescoring. ACL 2019.

Dear Reviewer 3ujB,

Thank you again for your time and effort in reviewing our paper. We believe our rebuttal addresses your main concerns through new experiments and analyses:

  • Supplied decompositions and iterative training: Model-generated decompositions achieve comparable performance to supplied decompositions on MuSiQue (51.6% vs. 50.7%). Iterative CSI also outperforms matched single-stage distillation and frozen-teacher baselines (Appendices C.10 and C.12).
  • Mitigating failure modes: Following your suggestion, repartitioning improves 23-digit addition accuracy from 84% to 92%. We also demonstrate that combining composition rules can improve accuracy on previously unsupported outputs from 0% to 64.1% (Appendices C.13–C.14).
  • Theoretical analysis: We extend our analysis to characterize error propagation across multiple rounds and conditions for generalization beyond accepted examples (Appendices B.4–B.5).
  • Experimental robustness: We add a stricter Bash evaluation with held-out command pairs, full fine-tuning experiments, and standard deviations across seeds (Appendices C.11 and C.15; updated figures and tables).

As the author–reviewer discussion period comes to a close, we would be happy to address any remaining concerns or questions.

We appreciate your thoughtful feedback and engagement!

Reviewer_C8kFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates compositional self-improvement, where a model constructs supervision for more difficult tasks by composing predictions on simpler subproblems. The authors identify three limiting factors: unreliable component predictions, invalid composition rules, and insufficient supervision coverage. Controlled experiments on algorithmic tasks characterize these failure modes, while additional experiments on Bash pipeline generation and multi-hop question answering demonstrate improvements over direct self-training. The paper also shows that composed pseudo-labels can provide a useful supervised warm start for reinforcement learning, particularly when direct RL rarely discovers successful compositions.

Critical strengths
  1. Systematic characterization of compositional self-improvement failures.

The paper provides a clear analysis of why compositional supervision can succeed or fail. The distinction between component errors, composition errors, and supervision coverage is useful, particularly because accurate composed labels do not necessarily provide supervision for all relevant output patterns.

  1. Consistent empirical improvements across different tasks.

The experiments demonstrate substantial improvements over direct self-training in controlled algorithmic settings and more realistic programming and multi-hop reasoning tasks. The authors also examine stronger direct self-training and decomposed-inference comparisons. The reinforcement-learning experiments are particularly interesting because a short compositional warmup enables learning in settings where direct RL struggles to obtain successful samples.

Critical weaknesses
  1. The approach relies heavily on externally specified compositional structure.

The proposed procedure assumes that useful decomposition plans and composition rules are available. In the controlled tasks, these rules are explicitly constructed, while the multi-hop reasoning experiments use annotated decompositions. In practice, discovering reliable decompositions and determining whether their outputs can be composed correctly may be among the most difficult parts of the problem. The experiments therefore establish the benefit of compositional supervision under favorable structural assumptions, but provide limited evidence that the approach can be applied to less structured tasks.

  1. The evidence for generalization to genuinely unseen compositions is not fully conclusive.

The Bash experiments show improvements on longer pipelines than those used for training. However, in Appendix, many longer test pipelines contain shorter compositions seen during training. This makes it difficult to distinguish generalization to genuinely novel compositions from the reuse of previously learned compositional patterns. The observed coverage gap for held-out commands reinforces the importance of this distinction.

Questions for authors
  • Dependence on decomposition quality. How does performance change when decomposition plans are generated by the model rather than supplied by the dataset or a task-specific construction procedure? How sensitive is the method to imperfect plans?

  • Compositional generalization. Could the authors report performance on test compositions that contain no previously seen shorter subcompositions?

  • Iterative improvement. How much additional improvement comes from iterative self-training compared with a single supervised distillation stage using a comparable budget of composed examples?

2 comments

We thank the reviewer for recognizing the characterization of failure modes, the empirical improvements, and the usefulness of compositional supervision for RL. We address the three main questions below.

(Q1) On dependence on supplied structure and decomposition quality.

We additionally compare supplied decompositions with model-generated plans to investigate how the source of decomposition affects CSI. In a new MuSiQue experiment, Qwen3.5 models generate subquestions and dependencies without reference decompositions or answers. We then construct pseudo-labels using the same composition and filtering procedure, discarding unusable examples without checking their ground-truth correctness. All students use the same 9B model and training budget.

Decomposition sourceKept (%)Correct-label yield (%)Accuracy (%)
Initial model——43.7
Supplied99.265.550.7
Model-generated (9B)74.047.351.6
Model-generated (4B)28.416.446.5
Model-generated (2B)0.10.1—

Results are averaged over three seeds. Kept denotes the percentage of candidate parents passing the filter; correct-label yield is measured against reference answers only for evaluation. Accuracy measures direct prediction after training.

Model-generated decompositions achieve comparable improvements to supplied decompositions, despite producing fewer correct labels. Smaller planners yield fewer usable examples, highlighting the importance of reliable plan generation. These results demonstrate that CSI can obtain useful supervision without per-example annotated decompositions. We have added this experiment to Appendix C.10.

(Q2) On generalization to unseen subcompositions.

Our original Bash experiments evaluate generalization to longer pipelines, where shorter subcompositions may have appeared during training. To directly evaluate unseen subcompositions, we conducted a new experiment that reserves command pairs before training.

Specifically, we partition ordered command pairs, rather than individual commands, into training, validation, and test sets (60%/20%/20%) before training. Every command in the test pipelines also appears in compositional training, so the individual skills are familiar. However, none of the adjacent command pairs in a test pipeline appears in any training input or pseudo-label target. For example, in a six-command test pipeline, all six commands have been seen during training, but none of its five adjacent pairs has been seen together. This also guarantees that no contiguous subpipeline of length two or more has appeared during training. Thus, the experiment tests whether CSI can combine familiar atomic skills into previously unseen command sequences, rather than merely extending previously learned compositions to greater lengths.

We evaluate 300 pipelines at each length from two to six, covering 881 distinct held-out command pairs:

Pipeline length23456Overall
CSI94.3%90.2%88.9%82.0%79.1%86.9%

CSI achieves 86.9% accuracy, compared with 2.9% for direct self-training and 5.0% for the initial model. The gains extend to five- and six-command pipelines, although compositional training uses only two- to four-command examples.

These results demonstrate that CSI can generalize to syntactically unseen subcompositions of familiar atomic skills, beyond simply reusing shorter compositions encountered during training. We have added the experimental setup and full results to Appendix C.11 of the revised manuscript.

(Q3) On the contribution of iterative training. We conducted a new experiment comparing CSI against two alternatives: (1) a frozen-teacher curriculum, which follows the same progression in difficulty but generates all pseudo-labels using the initial model, and (2) single-stage distillation, which trains on the same fixed composed labels with examples mixed across difficulty levels. All methods use matched training examples, update budgets, and component replay.

MethodBashMuSiQue
CSI81.6%54.7%
Frozen-teacher curriculum69.9%52.8%
Single-stage distillation72.2%51.1%

CSI outperforms single-stage distillation by 9.4 percentage points on Bash and 3.6 points on MuSiQue. These results suggest that iterative CSI can provide additional benefits beyond training on composed supervision generated by the initial model. We have added the experimental setup and full results to Appendix C.12 of the revised manuscript.

Dear Reviewer C8kF,

Thank you again for your time and effort in reviewing our paper. We believe our rebuttal addresses your three main concerns through the following new experiments:

  • Decomposition quality: Model-generated decompositions achieve comparable performance to supplied decompositions on MuSiQue (51.6% vs. 50.7%; Appendix C.10).
  • Unseen compositions: CSI achieves 86.9% accuracy on Bash pipelines with held-out adjacent command pairs, compared with 2.9% for direct self-training (Appendix C.11).
  • Iterative improvement: CSI outperforms both single-stage distillation and a frozen-teacher curriculum under matched training budgets (Appendix C.12).

As the author–reviewer discussion period comes to a close, we would be happy to address any remaining concerns or questions.

We appreciate your thoughtful feedback and engagement!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response

We thank all reviewers for their thoughtful comments and feedback. We appreciate their recognition of several strengths of our paper, which we briefly reiterate before addressing the individual concerns.

  • A systematic characterization of compositional self-improvement failures. We distinguish component errors, composition errors, and insufficient supervision coverage, showing that even accurate composed labels may fail to cover the patterns needed for further improvement. (Reviewers C8kF, 3ujB)

  • Empirical improvements and effective initialization for RL. Composed supervision outperforms direct self-training across algorithmic, programming, and reasoning tasks. A short compositional SFT warmup also enables RL to learn in settings where direct RL struggles to obtain successful samples. (Reviewers C8kF, 3ujB)

  • Careful experimental controls and transparent error analysis. Our experiments examine both the benefits and limitations of composed supervision, including cases where decomposed inference outperforms the trained model. (Reviewers C8kF, 3ujB, 8hTk)

We first clarify two common points raised across reviewers.

On the dependence on supplied decompositions (Reviewers C8kF, 3ujB, 8hTk). Our main experiments use supplied decompositions to isolate the factors governing compositional self-improvement (CSI). We additionally investigate whether similar results hold when no per-example decompositions are provided; this insight is independently valuable. Indeed, in a new MuSiQue experiment, models generate their own subquestions and dependencies without reference decompositions or answers. We construct pseudo-labels using the same composition and filtering procedure, discarding unusable examples without checking their correctness against reference answers. All students use the same 9B model and training budget.

Decomposition sourceKept (%)Correct-label yield (%)Accuracy (%)
Initial model——43.7
Supplied99.265.550.7
Model-generated (9B)74.047.351.6
Model-generated (4B)28.416.446.5
Model-generated (2B)0.10.1—

Model-generated decompositions achieve comparable performance to supplied decompositions, despite producing fewer correct labels. Smaller planners yield fewer usable examples, highlighting the importance of reliable plan generation. These results demonstrate that CSI can obtain useful supervision from model-generated decompositions (Appendix C.10).

On the contribution and its relationship to prior work (Reviewers C8kF, 3ujB, 8hTk). CSI studies how compositional structure in the task can support iterative self-training: predictions on simpler subproblems are combined to construct supervision for harder problems, and the process repeats as the model improves. This is related to System 2 distillation, which transfers the outputs of more elaborate inference into direct prediction, and to iterated amplification, which recursively constructs supervision through expert-guided decomposition. Our focus is on when imperfect model-generated compositions can support continued self-improvement without additional expert-provided labels for harder problems. We systematically characterize how component reliability, composition validity, and supervision coverage constrain this process, including why accurate pseudo-labels alone may not suffice for further improvement. Additional matched-budget experiments show that iterative CSI outperforms single-stage distillation on Bash and MuSiQue. These results are reported in Appendix C.12.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.