acceptodds
Under review as a conference paper at ICLR 2027

Composition Lives at the Seams: On-Policy Correction for Composing Internalized Skills

Abstract

A skill that lives in a prompt is paid for on every call, and two skill documents pasted into one prompt do not teach a model to chain the skills they describe. Distilling the documents into the weights looks like the cure, and at first it is. A 3B student fine-tuned on flawless demonstrations of 27 documented text skills executes them from memory more reliably than a model over ten times its size reading the documentation. The cure fails where nobody demonstrated anything, on chains one step deeper than the training data. The failure sits at the seams, the handoffs where one skill’s output becomes the next skill’s input; the states that need repairing arise only in the student’s own imperfect attempts, beyond off-policy reach. We therefore let the student attempt compositions, catch the first botched seam with a programmatic verifier, and train on a correct continuation spliced onto the student’s own prefix. This lifts deep chains to the level of a 32B model reading the documentation, at no in-distribution cost. Two controls show what the gain rests on. Freeze the rollouts and it collapses; take corrections from a teacher weaker than the student and it reverses. Composition lives at the seams, in the states where the student stumbles.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, skill composition, compositional generalization, on-policy learning, skill internalization, knowledge distillation, corrective imitation learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RFkiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether a 3B language model can carry out longer sequences of 27 fixed word transformations after training on short sequences. Starting from training on correct solutions with one to three steps, it learns from corrections to its own attempts. Reference code either replaces the failed step’s work line and result, then completes the task (fix), or continues from the wrong result (recover). Correct solutions are also replayed during training. On unseen tasks with four or five steps, pooled accuracy rises from 73.8% to 81.5%. However, accuracy on combinations involving skills trained only individually falls from 53.0% to 45.9%

Critical strengths
  1. Useful checks and evaluation splits: Reference functions allow exact checking of parsed intermediate results. Tests separate longer sequences from new skill combinations, making the generalization results easier to interpret.
  2. Gains on longer tasks: Accuracy on longer chains improves in all three correction runs from the same initial model; the variant using a larger teacher also improves it.
Critical weaknesses
  1. Missing controls prevent attributing the gains to the proposed correction rules. Correction receives additional training and fresh tasks, but is not compared with an equal amount of further training on gold solutions. The 32B variant supervises only failed steps and lacks a comparison with the same teacher supervising all steps. These controls are important because GKD (https://arxiv.org/pdf/2306.13649) already supplies teacher feedback on sequences from the current student, and MOTAB (https://arxiv.org/pdf/2605.19433) trains on teacher corrections appended after a revision marker to flawed student prefixes. The missing comparisons would separate the value of the specific correction rules from extra training and established teacher supervision.
  2. The claimed depth benefit of seam weighting is not proven. In the same-seed comparison, uniform versus seam weighting scores 79.75% versus 81.0% on longer chains, and 40.5% versus 47.25% on held-out compositions (Table 9). Table 11 supports the held-out benefit but does not establish an additional depth benefit. Both remain below the initial model’s 53.0% held-out accuracy. Thus weighting has a measured benefit in reducing the transfer loss, but attributing the longer-chain improvement specifically to this weighting is not established by these results though.
  3. In Table 9, freezing attempts lowers pooled depth accuracy from 81.75% to 79.25% for fix-only, versus 81.0% to 73.25% for fix+recover. This supports refreshing attempts for the full method. Section 5 attributes its drop to corrections targeting outdated errors, but does not analyze why the drop is much larger when recover is included. Each frozen variant has one run, so the cause of this difference remains uncertain.
  4. The paper locates errors without distinguishing their possible causes. Appendix G shows a sorting error at step four after three correct steps. This does not establish whether the model could sort that particular input when asked to do only that operation. Earlier operations change the input and lengthen the context. Tests of the same operation on the same intermediate list, both alone and with the preceding trace, would help distinguish difficulty executing a skill from effects of composing it with other skills.
  5. The benefit over instructions loaded on demand is unclear. Documentation baselines receive all required documents together and generate the full chain in one response. There is no comparison with supplying one skill’s concise rules per call and passing predicted outputs forward. Skill order is given, so instruction lookup is straightforward. This leaves the practical accuracy/cost benefit of internalization unclear..
Questions for authors
  1. From the same starting checkpoint, how does the oracle method compare with further training on fresh gold solutions at depths one to three? For the 32B variant, what changes if teacher feedback covers all steps? Match task draws, training budgets, and gold-trace replay.
  2. Repeat uniform versus seam weighting across several training seeds, keeping other settings matched. Does weighting reliably improve accuracy on longer chains, or mainly reduce the loss on held-out compositions? Report uncertainty for both outcomes separately
  3. Why is the reported depth drop from freezing attempts much larger for fix+recover than for fix-only?
  4. Can the initial and corrected models solve the failing operations as standalone tasks on exactly the same input lists? Compare these results with the full-trace condition to test whether the difficulty lies in the intermediate inputs or their surrounding context.
  5. How does a baseline that supplies one skill’s concise rules per call and passes predicted outputs forward compare on the same tasks and model sizes? Report accuracy, total tokens and latency. When is learned execution preferable to invoking executable skills?
Reviewer_hzGVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the composition of skills, when a small model internalized it or reading a skill doc.

The authors argue the failures happen at the seams, where one skill’s output becomes the next skill’s input. Their fix is to let the student attempt to use the chains of skills, and deterministic code is used to find the first failed step. The model is then trained on correct continuations with the student’s own attempt. This improves deeper chains (depth of 5) accuracy to 73.8.

Critical strengths

The recover splice is a cute idea. Gold traces never show what to do after a mistake, so the authors keep the student’s wrong step and train it to continue correctly from there. It helps where drift builds up: step-four accuracy in depth-five chains goes from 78.0 to 90.5 with recover splices alone.

Critical weaknesses

The significance is limited The skills are one-line operations on short word lists, like reversing, sorting, or dropping the first word. After plain SFT the student already gets every core skill right (100% in Table 7), so the only place left to fail is the handoff between skills. The main finding, that composition lives at the seams, seems to come from this setup rather than being a general finding. While it is possible the seam is the problem, with real skills the model would also fail inside a single skill, and it is not clear the seam story still holds.

The method also needs each intermediate step checked exactly by code. Even the distilled variant uses the programmatic verifier to find the failed step. The correction does come from the teacher. Real agent skills (writing, querying, editing code) would require different setups, and this paper does not provide a solution for those. Hence the paper is limited as a controlled study of simplified skills.

The motivation is thin too. The intro says agents chain skills, but there is no evidence or example of real tasks that need chains of four or five internalized skills. Chaining simplified skills like the ones in this paper can create probable meaningful tasks, but this paper does not motivate the scenarios that need chaining of multiple realistic skills.

Questions for authors

Q1: Do the results hold when the skills are not already mastered? In the current setup the student solves every core skill perfectly after SFT, so errors can only happen at the seams. An experiment with harder or more realistic skills, where single-skill errors also occur, would show whether seam correction still helps.

Q2: Would a cheaper off-policy version of the recover splice work as well? For example, corrupt one step of a gold trace at random and train on the correct continuation from there. This needs no student rollouts. If it matches the method, then the key ingredient is training on off-path states in general, not the student’s own mistakes.

Reviewer_sisBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how a small model (Qwen2.5-3B) can compose skills it learned by fine-tuning, using a synthetic testbed of 27 text-transformation skills. SFT on gold traces works well in distribution but drops a lot on chains longer than in training. They propose seam-corrective learning: the student does rollouts, a verifier finds the first wrong step, and they train on a correct continuation spliced onto the student’s own prefix, with more weight on the handoff tokens. This improves depth 4-5 accuracy (73.8 -> 81.5) without hurting in-distribution. There’s also a version that uses a 32B teacher instead of the oracle.

Critical strengths

The testbed is clean and every step can be checked by code, so the analysis is quite precise. Controls are good (frozen rollouts, different weightings, a weaker teacher), and they use multiple seeds and significance tests.

Critical weaknesses

The setting is very synthetic and the oracle version needs a reference implementation to make the corrections. So I’m not sure how much this transfers to real agent skills. The method is also quite close to on-policy distillation, just applied at the first error.

Also, the gain is only on k=4 and k=5, and held-out skills actually get worse (around 53 -> 46). Since there is a perfect verifier, I was expecting an RL baseline (e.g. GRPO with verifier reward), but it’s not there.

Questions for authors

How does it compare to RL with the verifier as the reward? Since skills are functions on any input, you could also make “recover” style data offline by putting random wrong states into gold traces. Does this give similar gains without on-policy rollouts? Any plan to test on more realistic skills, e.g. tool-use chains?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.