- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies whether a 3B language model can carry out longer sequences of 27 fixed word transformations after training on short sequences. Starting from training on correct solutions with one to three steps, it learns from corrections to its own attempts. Reference code either replaces the failed step’s work line and result, then completes the task (fix), or continues from the wrong result (recover). Correct solutions are also replayed during training. On unseen tasks with four or five steps, pooled accuracy rises from 73.8% to 81.5%. However, accuracy on combinations involving skills trained only individually falls from 53.0% to 45.9%
Critical strengths
- Useful checks and evaluation splits: Reference functions allow exact checking of parsed intermediate results. Tests separate longer sequences from new skill combinations, making the generalization results easier to interpret.
- Gains on longer tasks: Accuracy on longer chains improves in all three correction runs from the same initial model; the variant using a larger teacher also improves it.
Critical weaknesses
- Missing controls prevent attributing the gains to the proposed correction rules. Correction receives additional training and fresh tasks, but is not compared with an equal amount of further training on gold solutions. The 32B variant supervises only failed steps and lacks a comparison with the same teacher supervising all steps. These controls are important because GKD (https://arxiv.org/pdf/2306.13649) already supplies teacher feedback on sequences from the current student, and MOTAB (https://arxiv.org/pdf/2605.19433) trains on teacher corrections appended after a revision marker to flawed student prefixes. The missing comparisons would separate the value of the specific correction rules from extra training and established teacher supervision.
- The claimed depth benefit of seam weighting is not proven. In the same-seed comparison, uniform versus seam weighting scores 79.75% versus 81.0% on longer chains, and 40.5% versus 47.25% on held-out compositions (Table 9). Table 11 supports the held-out benefit but does not establish an additional depth benefit. Both remain below the initial model’s 53.0% held-out accuracy. Thus weighting has a measured benefit in reducing the transfer loss, but attributing the longer-chain improvement specifically to this weighting is not established by these results though.
- In Table 9, freezing attempts lowers pooled depth accuracy from 81.75% to 79.25% for fix-only, versus 81.0% to 73.25% for fix+recover. This supports refreshing attempts for the full method. Section 5 attributes its drop to corrections targeting outdated errors, but does not analyze why the drop is much larger when recover is included. Each frozen variant has one run, so the cause of this difference remains uncertain.
- The paper locates errors without distinguishing their possible causes. Appendix G shows a sorting error at step four after three correct steps. This does not establish whether the model could sort that particular input when asked to do only that operation. Earlier operations change the input and lengthen the context. Tests of the same operation on the same intermediate list, both alone and with the preceding trace, would help distinguish difficulty executing a skill from effects of composing it with other skills.
- The benefit over instructions loaded on demand is unclear. Documentation baselines receive all required documents together and generate the full chain in one response. There is no comparison with supplying one skill’s concise rules per call and passing predicted outputs forward. Skill order is given, so instruction lookup is straightforward. This leaves the practical accuracy/cost benefit of internalization unclear..
Questions for authors
- From the same starting checkpoint, how does the oracle method compare with further training on fresh gold solutions at depths one to three? For the 32B variant, what changes if teacher feedback covers all steps? Match task draws, training budgets, and gold-trace replay.
- Repeat uniform versus seam weighting across several training seeds, keeping other settings matched. Does weighting reliably improve accuracy on longer chains, or mainly reduce the loss on held-out compositions? Report uncertainty for both outcomes separately
- Why is the reported depth drop from freezing attempts much larger for fix+recover than for fix-only?
- Can the initial and corrected models solve the failing operations as standalone tasks on exactly the same input lists? Compare these results with the full-trace condition to test whether the difficulty lies in the intermediate inputs or their surrounding context.
- How does a baseline that supplies one skill’s concise rules per call and passes predicted outputs forward compare on the same tasks and model sizes? Report accuracy, total tokens and latency. When is learned execution preferable to invoking executable skills?