- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates how proven combinatorial trajectory supervision affects program ranking and combinatorial generalization of language models. The author built an environment containing five types of finite field polynomial operations, generated correct programs and intermediate states through exhaustive methods, and ranked and evaluated all 125 three-step programs. The Qwen3-0.6B pairing experiment of six groups of budget matching showed that combinatorial training increased the Hit@32 of familiar combinatorial tasks from 45.32% to 72.72%, and the gains could be transferred to the new finite domain, but performance declined on tasks involving withheld operation pairs, and it was accompanied by the forgetting of basic execution abilities. Manipulation-to-exclusion experiments further examined the impact of training coverage; search, GRPO, trajectory filtering, and entropy rewards yielded no clear gains under the settings tested. The main contributions of the paper are an evaluation framework for the full program space and a controlled empirical analysis of validation of supervised gains, combinatorial generalization bounds, and underlying skill retention.
Critical strengths
- The paper studies how verified composition training changes correct-program rankings on familiar compositions and tasks with withheld operation pairs.
- The main comparison uses six paired continuations with matched optimizer-step and loss-bearing target-token budgets under fixed exhaustive evaluation.
Critical weaknesses
-
The experiments provide limited explanatory insight into compositional learning. The composition branch changes task depth, output sequences, and intermediate-state supervision together, whereas the atomic control trains only single-step tasks. The comparison establishes a gain for the full recipe, as the authors acknowledge, but does not distinguish compositional learning from format adaptation or other supervision effects. Pair exclusion also changes operation frequencies, program signatures, and intermediate states. These limitations leave the central explanation of the ranking changes unresolved and reduce the scientific contribution beyond documenting exposure-dependent performance.
-
The gains remain narrow and come with substantial trade-offs. The main confirmatory evidence concerns a small synthetic environment, principally depth-three programs, and one shared Qwen3-0.6B atomic checkpoint. Familiar-family Hit@32 improves substantially, but family-A Hit@1 reaches only 10%, while withheld-pair performance and atomic execution accuracy decline. The developmental 8B comparison provides additional evidence but does not establish robustness across independent starting checkpoints. Consequently, the results support a local ranking improvement, while their significance for reliable program generation and broader compositional generalization remains insufficiently established.
Questions for authors
-
How do familiar and withheld-pair test families differ in operation frequencies, program structure, and solution multiplicity? How are these differences accounted for when interpreting their contrasting training effects?
-
What existing evidence separates the effects of multi-step format exposure, program-sequence supervision, and intermediate-state supervision in the main comparison?
-
Is withheld-pair degradation concentrated on tasks involving SH1? Given its initial APPLY accuracy of 57.5%, how can compositional failure be distinguished from weak atomic execution or forgetting?
-
Have the main findings been reproduced under the same evaluation protocol from independently trained atomic checkpoints?
-
Were alternative training recipes compared on the same multi-step tasks with the same scoring rule and comparable supervision budgets? If so, what advantage does full-trajectory supervision provide?
2 comments
Thanks for your response. I maintain my original score.
Family construction and frequency. Every shortest solution in A/C avoids the held pairs, and every shortest solution in B/D contains exactly one held-pair occurrence (Section 3). This directly changes the allowed program structure. Operation frequencies and solution multiplicity were not matched across families. The uniform reference accounts for each task’s solution count, and the family A gain appears in both the unique-solution and multiple-solution strata (Table 24). The A/B contrast remains unadjusted for operation frequencies and program structure.
Supervision components. The submitted main experiment has no matched comparison that isolates multi-step format exposure, program-sequence supervision and intermediate-state supervision. Appendix I varies objectives, masks and recipes in a separate development protocol. The evidence supports a gain for the full composition recipe.
SH1 APPLY accuracy was 57.5% at the shared start and reached 51.50% under the atomic control and 29.08% under composition training (Section 5). Weak SH1 execution is a plausible contributor to withheld-pair degradation. We do not have the main-series stratification comparing tasks for which every correct program omits SH1 with those whose solutions use it. The available aggregate scores cannot establish how much of the ranking decline is due to SH1 or other atomic failures.
Independent atomic starts. All six main pairs start from one atomic checkpoint. Appendix H uses one atomic checkpoint per model size. Reproducibility across independently trained atomic starts under the main protocol remains untested.
Alternative supervision recipes. The submitted experiments provide no comparison that isolates a full-trajectory advantage on the same multi-step tasks with the same scorer and comparable supervision budgets. The search, reward and Appendix I experiments address narrower questions under their respective protocols. A matched program-only versus trajectory comparison would directly address this question.