acceptodds
Under review as a conference paper at ICLR 2027

RL Unlocks Compositional Reasoning via Soft Tokens

Abstract

Large language models (LLMs) acquire reasoning capabilities in post-training via methods such as supervised fine-tuning (SFT) on demonstrations and reinforcement learning (RL) with verifiable rewards, but continuously extending these capabilities without forgetting previously acquired ones remains challenging. Skill neologisms (SNs) have recently been proposed as a way to learn new skills in modular fashion via semantically integrated soft tokens. However, prior work has demonstrated SNs only for direct input–output skills, whereas soft token methods are known to struggle with multi-step reasoning. It therefore remains unclear whether SNs can encode reasoning procedures that compose with other capabilities as effectively as skills internalized in model weights. We study this question by controlling which reasoning skills are available to a base model, training new skills via SNs and weight-based fine-tuning (FT), and evaluating their composition abilities with in- and out-of-distribution skills. Beyond preventing forgetting of existing skills by construction, we find that the effectiveness of reasoning SNs depends strongly on both model capability and training method. Under SFT, more capable models increasingly close the gap to internalized skills on in-distribution compositions, but remain weak on out-of-distribution composition. Training SNs with RL significantly improves performance, matching or exceeding internalized skills on in- and out-of-distribution compositions, and unlocking composition between independently learned SNs. We extend our findings to a medical reasoning task, where RL-trained neologisms significantly improve transfer to OOD compositions over SFT. Our results show that RL-trained skill neologisms are a promising path for continually extending the reasoning capabilities of strong LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reasoning skills, continual learning, neologisms, soft tokens, composition, generalization, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_owPyOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows that skill neologisms (training skill-specific embedding vectors while keeping model weights frozen) can acquire compositional reasoning capabilities through SFT and RL. The main conclusion is that SFT brings some compositional capability (but does not beat the baselines), whereas an additional RL training can match the baselines.

Critical strengths
  1. The experimental setup is carefully designed.
  • The authors distinguish between three types of composition: new-train (seen composition during training), new-train (heldout composition from training), new-new.

  • The authors also control for the models’ performance on the atomic skills when evaluating the compositional skills. The authors also report on the sensitivity of the chosen threshold.

  1. Strong result on compositional generalization
  • After RL training, Qwen3.5-9B reaches 61% accuracy on the new-new compositions, matching a weights-trained baseline.

  • The authors also extend to composition depths up to 4, showing that RL can unlock multiple layers of compositions with skill neologisms.

  1. Some evidence beyond a single synthetic task
  • While the extensions are preliminary, Section 7 shows promising directions for extending the analysis to domains other than a single synthetic task on string manipulation.
Critical weaknesses
  1. (Major) Extension to real-world settings is narrow. The authors already concede to this limitation, but the extension currently only covers 1 composition with 1 base model.

  2. (Minor) The authors report that RL training on skill neologism was unstable for Qwen2.5. This raises some minor concerns about the robustness of the method. While the fact that the more modern Qwen3 models showed a stable run partially addresses the concern, it would be good to verify that the method can be applied to other model families from the similar era as Qwen3.

Questions for authors
  1. Could the authors validate their findings on more atomic / more combinations on MedCalc- Bench / or on more compositional reasoning benchmarks?

  2. Could the authors validate their method on a different model family?

3 comments

We thank the reviewer for their feedback. We appreciate the recognition of our carefully designed experimental setup and the informativeness of our findings. We have updated our manuscript and address your comments below.

Extension to real-world settings [W1, Q1]

We agree that our real-world validation remains limited in scope. We would nevertheless like to clarify that our MedCalc evaluation includes four downstream calculation tasks spanning two independently learned skills: delta gap transfers to three untrained downstream calculations, while adjusted body weight transfers to Cockcroft–Gault.

The main focus of this work is to carefully study the composition abilities of skill neologisms for reasoning, which requires controlling the skills available to the base model and distinguishing atomic mastery from compositional generalization. We therefore center our investigation on a controlled setting where we can explicitly distinguish internalized and new skills and hold out specific compositions during training. In contrast, real-world settings usually involve fuzzy skill definitions and confounding pretrained knowledge, which makes it difficult to distinguish learning a new skill from eliciting an existing capability.

Our MedCalc experiments provide initial evidence that the benefits of RL-trained neologisms extend beyond this controlled setting, while we acknowledge that broader validation across real-world reasoning tasks remains an important next step.

Extension to other model families [W2, Q2]

We thank the reviewer for raising this point. Our initial experiments focused on the Qwen2.5 and Qwen3.5 families in order to study how the performance of skill neologisms for reasoning evolves over gradual improvements in model scale and capabilities. Nevertheless, we agree with the reviewer that our central claim should be validated with another model family.

We now extend our main RL training results to the DeepSeek-R1-Distill-Llama-8B model. We use the same training pipeline and datasets as for Qwen3.5-9B to train the pretrained model, the SN-SFT and SN-RL neologisms and the idealized FT-RL baseline. RL training succeeds for all 10 skills for this backbone. The results are consistent with those for Qwen3.5-9B: SN-RL performs significantly better than SN-SFT across compositions. For example, L2 new×new accuracy increases from 3.3% with SN-SFT to 31.3% with SN-RL. A gap does remain with the idealized FT-RL baseline, particularly for the new-only compositions. These results suggest that while RL training strongly benefits skill neologisms for models beyond the Qwen family, the extent to which RL closes the gap with the composition of internalized skills varies across models.

We have incorporated the new results and discussion in Section 6 and Appendix A.6 of the updated manuscript.


Thank you for your time and effort. We hope that our responses and additional experimental results address your concerns. We are happy to engage with any further comments you may have.

AuthorsOpenReview ↗

Follow-up on Rebuttal

As the discussion period is coming to a close, we wanted to follow up on our response to add an additional experiment. Following your concern that our medical calculations experiment covered only one base model, we have now repeated the MedCalc-Bench experiment with Qwen3.5-9B. We train neologisms on a frozen Qwen3.5-9B backbone for the same two learned skills, using the same training and evaluation protocol as our initial 4B experiment. The results are very similar to those for Qwen3.5-4B: RL improves over SFT on all six tasks. On the four untrained downstream calculations, RL exceeds SFT neologisms by between 32.5 and 42.5 percentage points. These results provide further support for our finding that the benefits of RL for skill neologisms extend beyond our synthetic setting.

We have incorporated these results in Section 7 and Appendix B of our updated manuscript.

Thank you again for your time and effort in reviewing our work. We look forward to your response.

The additional experiments mostly address my concerns. This paper presents a proof of concept that skill neologisms can also be trained via SFT/RL on multi step reasoning tasks offering an alternative approach to full-weights training that allows modular training without forgetting. I would be excited to see this presented at ICLR. Accordingly, I raise my score to a 4.

Reviewer_b9uROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether skill neologisms (SNs) — per-skill soft tokens trained with a frozen backbone — can encode composable reasoning skills, going beyond the direct input–output skills demonstrated in prior work. Using the controlled 25-function string-manipulation testbed of Yuan et al. (2026), the authors first pretrain Qwen2.5 (0.5B–7B) and Qwen3.5 (4B/9B) models on 15 internal skills via SFT+RL, then acquire the remaining 10 skills either as independently trained neologisms or via LoRA fine-tuning. Composition is evaluated at Level 2 in three slices: new×internal-train (ID), new×internal-holdout (OOD-ho), and new×new (OOD-new), with an additional mastery-controlled view filtering to function pairs where both constituents exceed p_L1 > 0.6.

Main findings: (1) SN-SFT acquires new atomic skills without forgetting internal ones, and the ID composition gap to weight-internalized skills shrinks with backbone capability, but OOD composition remains weak and does not improve monotonically with scale; (2) training neologisms with GRPO substantially improves composition (+30.6% ID, +57.2% OOD-ho, +48.5% OOD-new on Qwen3.5-9B), matching or exceeding an idealized FT-RL reference on new×internal pairs and unlocking zero-shot inter-neologism composition; (3) a failure-mode decomposition and an attention analysis suggest RL-trained neologisms receive more temporally concentrated attention, mitigating interference between multiple neologisms; (4) on MedCalc-Bench, RL-trained neologisms improve transfer to untrained downstream calculations over SFT. The authors conclude that RL-trained skill neologisms are a promising path for continually extending reasoning capabilities, with effectiveness depending on backbone capability and training algorithm.

Critical strengths
  1. A cleanly controlled answer to a question that is otherwise hard to isolate. Whether soft tokens can carry composable reasoning skills is confounded in real benchmarks by fuzzy skill definitions and pretraining contamination. The synthetic testbed (anonymized function IDs, exact-match rewards, explicit internal-train/holdout/new split) gives an unambiguous answer, and the mastery-filtered evaluation (p_L1 > 0.6, with threshold sensitivity in A.3) isolates composition ability from atomic mastery — a confound most compositionality papers ignore. This is consequential because the resulting finding is clear and testable: SFT-trained SNs fail at OOD composition; RL-trained SNs succeed.

  2. The paper explains, not just reports. The inner/follow-through decomposition (Table 1) localizes where composition fails — a second neologism interferes with both intermediate computation and its downstream use — and the attention analysis (Fig. 6) connects RL's benefit to temporally concentrated soft-token attention, consistent with Fan et al. (2025)'s account of soft prompts derailing reasoning. Together with the MedCalc transfer experiment, these sections turn a synthetic result into a reasonably complete scientific story, and they increase my confidence that the effect is real rather than an artifact of one metric.

Critical weaknesses
  1. Generality of the central claim is established on a single model family — effectively a single backbone. All RL conclusions come from Qwen3.5-9B; the paper itself reports reward collapse on Qwen2.5 ≤ 3B and only 6/10 usable neologisms on Qwen2.5-7B (footnote 1, §8), and no non-Qwen model (e.g., Llama, Mistral) is studied at all. Since the title-level claim is that RL — not a Qwen-specific property — unlocks soft-token composition, this is a significant gap. Qwen3.5's hybrid architecture (24 linear-attention layers, themselves excluded from the attention analysis in A.5) makes the architecture question concrete rather than hypothetical. Absent cross-family replication, the claims should be explicitly scoped to capable Qwen-class backbones.

  2. The value proposition over plain fine-tuning is not fully convincing, especially for new-only compositions. In Figure 5's bottom row ("only new functions"), FT-RL is the best method across depths and sampling budgets; SN-RL merely narrows the gap, and SN-SFT is near zero at Levels 3–4. The authors frame FT-RL as an idealized, not-strictly-comparable reference — but this argument applies in both directions: whenever joint training over all skills is feasible, simple FT-RL dominates the proposed method. The paper's real selling point is modularity, yet the most direct modular weight-space competitor — per-skill LoRA adapters trained with the same GRPO protocol — is never evaluated (Table A1 covers only per-skill LoRA with SFT). Without that baseline, "SN-RL rivals internalized skills" is shown only against a baseline the paper itself declares not comparable, leaving the practical niche of SN-RL unclear.

Minor (fixable): (a) inference prepends a header listing which neologisms occur in the query, i.e., oracle skill routing is assumed and never stress-tested; (b) MedCalc evidence is thin (2 skills, 20 questions per row, one 4B backbone, SEMs as large as ±7.8 points) — fine as a proof of concept, but it should be framed as suggestive; (c) the attention analysis is correlational (35 prompts, a heuristic concentration metric C_10); (d) neologism length is fixed at l = 50 with no ablation.

Questions for authors
  1. Cross-model generality. Did you attempt RL training of SNs on any non-Qwen backbone (e.g., Llama-3-8B-Instruct)? Even one additional family would substantially strengthen the central claim. Relatedly, are the Qwen2.5 collapses a property of soft-token RL specifically, or of RL on small backbones in general — e.g., would a KL penalty (β > 0) or a lower learning rate stabilize them?

  2. Modular weight-space baseline. How does SN-RL compare to per-skill LoRA adapters trained with the same GRPO protocol (independently trained, frozen backbone, composed at inference)? This is the most direct modular competitor, and it seems essential to justify soft tokens over adapters.

  3. Figure 5, "only new functions". FT-RL leads across all depths and sampling budgets there. Could you state precisely the deployment regime in which SN-RL is the recommended method — is the advantage strictly the no-forgetting / no-joint-data setting? If so, I would suggest saying so explicitly in the abstract and takeaways rather than emphasizing "matching or exceeding" FT-RL.

  4. Skill routing at inference. The header lists neologisms for functions present in the query. How sensitive is SN-RL composition to routing errors — including an irrelevant neologism, or omitting a relevant one? Would a simple learned router suffice?

  5. (Clarification) Is the RL benefit robust to shorter neologisms (e.g., l = 10 or 20), or does it depend on the 50-token capacity?

4 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for their detailed feedback and for recognizing our controlled study design and the insights provided by our analyses. We appreciate the reviewer's comments and suggestions, which have helped us significantly strengthen our paper. We answer each point below:

  1. Cross-model generality [W1, Q1]
  2. Value proposition over full fine-tuning [W2, Q2, Q3]
  3. Skill routing [Q4]
  4. Neologism length [Q5]

We specify the changes to the manuscript as Action and any new experiments as New Results. Please see the updated manuscript with changes marked in blue.

Cross-model generality [W1, Q1]

Our initial experiments focused on the Qwen2.5 and Qwen3.5 families in order to study how the performance of skill neologisms for reasoning skills evolves with model scale and capabilities. Nevertheless, we agree with the reviewer that our central claim would benefit from replication with another model family.

New results: We now extend our main RL training results to the DeepSeek-R1-Distill-Llama-8B model. We use the same training pipeline and datasets as for Qwen3.5-9B to train the pretrained model, the SN-SFT and SN-RL neologisms and the idealized FT-RL baseline. RL training succeeds for all 10 skills for this backbone. The results are consistent with those for Qwen3.5-9B: SN-RL performs significantly better than SN-SFT across compositions. For example, L2 new×new accuracy increases from 3.3% with SN-SFT to 31.3% with SN-RL. A gap does remain with the idealized FT-RL baseline, particularly for the new-only compositions. These results suggest that while RL training strongly benefits skill neologisms for models beyond the Qwen family, the extent to which RL closes the gap with the composition of internalized skills varies across models.

Action: We have incorporated the new results in Appendix A.6 and discuss them in Section 6 of the main text of the updated manuscript.

Value proposition over full fine-tuning [W2, Q2, Q3]

We thank the reviewer for raising this important point. Indeed, the main advantage of skill neologisms is the ability to combine multiple skills zero-shot at inference time. A core question explored by this work can be framed as: "What is the composition performance cost of this modularity?" The idealized FT-RL baseline provides a reference composition performance for a given model if the skills could be internalized simultaneously by the model. However, we agree that modular weight-space adapters provide an important baseline that was missing from our initial experiments.

New results: We now directly compare against modular weight-space baselines in a new experiment. We train one LoRA adapter for each individual new skill using the same SFT+RL pipeline and datasets used to train the SN-SFT and SN-RL neologisms. At inference time, we merge the adapters for the relevant skills (same skill routing as for neologisms) using seven different merging methods: linear (mean), concatenation (cat), SVD, TIES, TIES-SVD, DARE-linear and DARE-linear-SVD. We then compare the performance on new-only compositions with SN-RL. Note that given time constraints, we limit this new experiment to four out of the ten new skills (since each skill requires training LoRA adapters with SFT and RL), but would be happy to extend this result to the full 10 new skills if the reviewer thinks that would be helpful.

Results are shown Figure 7 in the updated manuscript, and show that SN-RL dominates all the merged adapter variants across the compositions of new skills. Notably, all the merged adapters collapse to near 0% accuracy on L3 and L4, while SN-RL still reaches 40.8% and 24.6% accuracy, respectively. These results show that skill neologisms enable significantly better zero-shot composition abilities for reasoning skills compared to modular weight-space adapters.

Action: We have added these results and discussion in Section 6.2 of the updated manuscript. We have also updated the abstract, introduction and takeaway in Section 6 to emphasize that RL-trained neologisms enable modular extension of a frozen backbone when joint training of new skills is not practical.

AuthorsOpenReview ↗

Rebuttal (2/2)

Skill routing [Q4]

We thank the reviewer for this suggestion. Our experiments indeed assume that the model is provided with exactly the relevant skills for each sample. While the question of how to accurately route skills at inference time is outside the scope of this work, we agree that investigating the sensitivity of the learned neologisms to routing errors is an interesting question. Omitting a relevant neologism in the context essentially reverts to the base model's performance for the corresponding skill, regardless of the quality of the neologism. However, the effect of including irrelevant neologisms in the context depends on how well neologisms can be "disregarded" by the model when irrelevant.

New results: We run a new experiment evaluating the atomic and composition performance under imperfect routing where 1, 3 or 5 irrelevant neologisms are inserted in the context, in addition to the relevant ones. Our results (Figure 8 in the updated manuscript) show that irrelevant neologisms generally degrade performance, but RL-trained neologisms are significantly more robust to this effect than SN-SFT ones. For example, performance on atomic internal skills (near 100% on the pretrained model) falls to 12% when 5 irrelevant SN-SFT skills are added in context, compared to 85% for SN-RL. This experiment suggests that SN-RL neologisms not only lead to stronger performance than SN-SFT ones but are also more robust to imperfect routing.

Action: We have incorporated this new experiment in Section 6.3 of the updated manuscript.

Neologism length [Q5]

Throughout our study, we keep the neologism length fixed at l=50l=50 to compare neologisms and backbones in the same setting. We expect the SFT+RL training pipeline would still work for shorter neologisms, up to a certain point where too few soft tokens would not leave enough representational capacity for successful SFT and RL training. Shorter neologisms might also affect optimization and make RL training more unstable. For this rebuttal period, we prioritized experiments on cross-family replication, modular LoRA adapters and sensitivity to imperfect routing. We would be happy to add a length ablation experiment in the camera-ready version if the reviewer thinks that would be helpful.


Thank you for your time and effort. We hope that our responses and additional experimental results address your concerns. We are happy to engage with any further comments you may have.

AuthorsOpenReview ↗

Follow-up on Rebuttal

As the discussion period is closing soon, we wanted to share some initial results on an ablation of the length of the neologisms, suggested by Question 5 in your initial review.

We train neologisms of length 10 on Qwen3.5-9B for the first two skills, func_0 and func_2, using the same SFT+RL training pipeline as the length-50 neologisms from the main paper. We evaluate l=10l=10 and l=50l=50 neologisms on the same test set involving only these two new skills. The results are in the table below and show that the l=10l=10 neologisms perform significantly better after RL than after SFT, with comparable performance with the length-50 SN-RL neologisms. This suggests that the benefits of RL training for skill neologisms are robust to shorter neologisms.

Evaluationl=10l=10, SN-SFTl=10l=10, SN-RLl=50l=50, SN-SFTl=50l=50, SN-RL
Atomic85.7594.2589.2598.25
New × internal-train68.2381.5165.6890.40
New × internal-holdout37.3579.3526.4580.97
New × new9.2565.757.7564.50

We have incorporated these results in Appendix A.10 of the updated manuscript.

Once again, we would like to thank you for your time and for your constructive feedback, which has helped us significantly improve our paper. We look forward to your response.

Thank you for the clarification from the author; my decision is to keep the score as is.

Reviewer_JS5BOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the influence of the model capability and the learning algorithm on skill neologisms. By using some skill combinations for training and others for OOD evaluation, the authors show that RL could improve the generalization of the learned skills, and different skills can be composed after RL training. These conclusions provide insights for utilizing skill neologisms.

Critical strengths
  1. Skill neologisms are a promising direction for learning new skills. This paper studies how different training methods affect the reasoning ability of SNs, which is interesting and meaningful.
  2. Although the experiment scale and scope are limited, the authors provide relatively comprehensive evaluations on the datasets. They compare different training methods, model sizes, and skill composition settings, and provide analyses of forgetting and attention patterns.
Critical weaknesses
  1. The experiments are limited. In the main experiments, most skills are simple string operations. The experiments on MedCalc-Bench also only include two learned skills. Thus, the conclusions cannot be easily generalized to more complex reasoning skills.
  2. The contribution of the paper is limited. Previous works have already shown that RL can achieve better OOD generalization than SFT. And stronger models benefit more from parameter-efficient Fine-tuning. Although the paper provides valuable insight that RL soft tokens can be composed, most of the conclusions are already in previous works.
Questions for authors
  1. Please see the weaknesses.
  2. Could the authors compare SN-RL with SN-SFT using similar training costs? It is unclear whether the improvements come from RL or additional training.
3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for their feedback and for recognizing the value of our research question and the breadth of our evaluations. We address their concerns below:

  • Limited experimental setting [W1]
  • Contribution of the paper [W2]
  • Comparing SN-SFT and SN-RL under similar training costs [Q2]

Please see the updated manuscript with changes marked in blue.

Limited experimental setting [W1]

We agree that broader validation across real-world reasoning tasks is an important next step. Our focus on a controlled setting is deliberate: we ask how well reasoning skills learned through neologisms compose, compared with skills internalized in model weights. To answer this, we need to control which skills are available to the base model, distinguish atomic mastery from compositional abilities, and hold out specific compositions during training. Our controlled setting allows us to make these distinctions cleanly. In contrast, real-world settings often involve fuzzy skill definitions and confounding pretrained knowledge, which makes it difficult to distinguish learning a new skill from eliciting an existing capability.

We would also like to point out that although the atomic operations have simple definitions, composing multiple skills together leads to complex queries requiring multi-step reasoning, and composition remains challenging even for internalized skills. The idealized Qwen3.5-9B FT-RL baseline’s pass@1 accuracy falls from 97.5% on atomic tasks to 72.0%, 41.0%, and 9.0% on compositions of two, three, and four new skills, respectively (Figure 5).

This combination of well-defined atomic skills and challenging compositions makes the setting useful for isolating the effects of backbone capability and training method on the composition abilities of skill neologisms. We believe this controlled evidence is a valuable contribution toward understanding reasoning with modular adapters like skill neologisms, while recognizing that the generality of our findings to a wide range of real-world settings remains to be established.

Contribution of the paper [W2]

We agree that prior work has shown RL’s benefits for OOD generalization and that parameter-efficient adaptation improves with stronger models. Our work asks a more specific question: whether reasoning skills learned independently as separate soft tokens can compose at inference time, without joint training or updates to the shared backbone.

Our original results show that stronger backbones and SFT-based skill acquisition do not by themselves ensure composition between independently learned neologisms. RL substantially improves this capability: on Qwen3.5-9B, new×new accuracy increases from 12.5% with SFT to 61.0% with RL. We now replicate this finding on DeepSeek-R1-Distill-Llama-8B, where accuracy increases from 3.3% to 31.3% (Appendix A.6).

Our new comparison with modular LoRA adapters further tests whether this compositional capability follows simply from learning skills independently with RL (Section 6.2). We train LoRA adapters for individual new skills using the same SFT+RL pipeline and datasets as the skill neologisms, and combine them at inference time using seven merging methods (e.g. mean, SVD, TIES, DARE). Both approaches achieve near-perfect atomic accuracy, but SN-RL outperforms every tested merging method on compositions. On three- and four-skill compositions, SN-RL achieves 40.8% and 24.6%, respectively, while all the merged adapters have near zero accuracy.

Together, these results show that RL substantially improves zero-shot composition between independently learned reasoning neologisms, and that this does not simply follow from learning skills with RL, since RL-trained modular LoRA adapters do not show the same composition abilities.

AuthorsOpenReview ↗

Rebuttal (2/2)

Comparing SN-SFT and SN-RL under similar training costs [Q2]

We thank the reviewer for raising this point. Since the SN-RL neologisms are initialized from the SN-SFT neologisms, we agree that our initial experiments do not by themselves establish whether the gains come from RL training or additional training compute.

We have added an experiment testing this (Section 6.3 and Appendix A.7 in the updated manuscript), finding that the performance gains from RL are not explained simply by additional training compute. Following standard FLOPs estimates [1,2,3], we approximate a forward pass as 2ND2ND FLOPs and a forward+backward pass for full fine-tuning as 6ND6ND FLOPs, where NN is the number of model parameters and DD is the number of processed tokens. Training neologisms requires gradients for the frozen model's activations but not for its weights, so we approximate the forward+backward cost as 4ND4ND FLOPs. We neglect the additional computation associated with the trainable embeddings (50×4096≈2×10550 \times 4096 \approx 2 \times 10^5 parameters per neologism vs 9×1099 \times 10^9 model parameters).

We therefore estimate the total FLOPs for SFT and RL as:

XSFT=4N(Dinit+DSFT)X_{\mathrm{SFT}} = 4N(D_{\mathrm{init}} + D_{\mathrm{SFT}})

XRL=4N(Dinit+DRL)+2NDrolloutsX_{\mathrm{RL}} = 4N(D_{\mathrm{init}} + D_{\mathrm{RL}}) + 2ND_{\mathrm{rollouts}}

where DinitD_{\mathrm{init}} is the number of tokens processed during training of the common SN-SFT initialization, DSFTD_{\mathrm{SFT}}/DRLD_{\mathrm{RL}} are the numbers of tokens processed with backpropagation during downstream SFT/RL training, and DrolloutsD_{\mathrm{rollouts}} is the number of tokens processed during forward-only GRPO rollout generation.

To limit the risk of overfitting, we generate a fresh training set of 5,000 demonstrations per function (2,500 L1 and 2,500 L2). Summed across all ten skills, the additional compute used by RL through step 150 corresponds to approximately 4.90 SFT epochs on this dataset (details in Appendix A.7). We therefore train each neologism for five additional SFT epochs, starting from the same initial checkpoint as SN-RL. We evaluate the SN-SFT neologisms after each additional epoch and compare them with SN-RL neologisms at training steps 50, 100 and 150 on the same test sets.

Figure A7 shows L1, L2 and L3 performance against estimated total training compute, including the common initialization. SN-RL neologisms improve substantially over training, while SN-SFT neologisms show little improvement from additional training. These results show that the performance gains from RL are not attributable simply to more training compute.


Thank you for your time and effort. We hope that our responses and additional experimental results address your concerns. We are happy to engage with any further comments you may have.

References:

[1] Hoffmann, Jordan, et al. "Training compute-optimal large language models." NeurIPS 2022.

[2] Snell, Charlie, et al. "Scaling LLM test-time compute optimally can be more effective than scaling model parameters." ICLR 2025.

[3] Chu, Tianzhe, et al. "SFT memorizes, RL generalizes: A comparative study of foundation model post-training." ICML 2025.

AuthorsOpenReview ↗

Follow-up on Rebuttal

As the discussion period is coming to a close, we wanted to follow up on our response to add an additional experiment, extending the MedCalc-Bench experiment to Qwen3.5-9B. We train neologisms on a frozen Qwen3.5-9B backbone for the same two learned skills, using the same training and evaluation protocol as our initial 4B experiment. The results are very similar to those for Qwen3.5-4B: RL improves over SFT on all six tasks. On the four untrained downstream calculations, RL exceeds SFT neologisms by between 32.5 and 42.5 percentage points. These results provide further support for our finding that the benefits of RL for skill neologisms extend beyond our synthetic setting, although validation across a wider range of real-world settings remains an important next step.

We have incorporated these results in Section 7 and Appendix B of our updated manuscript.

Thank you again for your time and effort in reviewing our work. We look forward to your response.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal

Dear Area Chair and Reviewers,

We are grateful for the detailed feedback and constructive suggestions. Reviewers highlighted the carefully controlled experimental setup (b9uR, owPy), the systematic comparisons across training methods, model sizes, and composition settings (JS5B), and the strong compositional generalization achieved through RL (owPy).

Below, we summarize the main additions and clarifications made in response to your feedback, with references to our rebuttal responses and revised manuscript. Changes are highlighted in blue in the revised PDF.

ConcernReviewerChangeRelevant Section
Generality beyond Qwen modelsb9uR W1 & Q1; owPy W2 & Q2We replicated the full training pipeline on DeepSeek-R1-Distill-Llama-8B for all 10 new skills. RL improves compositional generalization over SFT; for example, accuracy on compositions of two new skills increases from 3.3% to 31.3%.Rebuttal: b9uR (1/2), owPy. Manuscript: Sec. 6; App. A.6, Figs. A5–A6.
Comparison with modular weight-space adaptersb9uR W2 & Q2; JS5B W2We added independently trained per-skill LoRA adapters using the same training data, SFT epochs, and RL updates, evaluated with seven merging methods. Despite similarly near-perfect atomic accuracy, SN-RL outperforms all tested merges on compositions of two to four skills, reaching 40.8% and 24.6% at depths three and four, where merged adapters perform near zero.Rebuttal: b9uR (1/2), JS5B (1/2). Manuscript: Sec. 6.2, Fig. 7; App. A.8.
RL gains versus additional training computeJS5B Q2We added five further SFT epochs on fresh demonstrations, comparing SFT and RL at matched estimated total training FLOPs. Additional SFT yields little improvement, whereas RL substantially improves composition, supporting gains beyond additional compute alone.Rebuttal: JS5B (2/2). Manuscript: Sec. 6.3; App. A.7, Fig. A7.
Evidence beyond synthetic tasksJS5B W1; owPy W1 & Q1We repeated the MedCalc experiment on Qwen3.5-9B, covering the same two learned skills and four untrained downstream calculations. RL improves over SFT on all six tasks, with downstream gains of 32.5–42.5 percentage points. This strengthens the initial evidence beyond synthetic tasks while retaining the study’s proof-of-concept scope.Rebuttal: JS5B and owPy, November 19 follow-ups. Manuscript: Sec. 7; App. B, Table B1.
Sensitivity to imperfect skill routingb9uR Q4We tested adding 1, 3, or 5 irrelevant neologisms alongside the required modules. Both methods degrade, but SN-RL is substantially more robust: with five irrelevant modules, internal atomic accuracy remains 85%, versus 12% for SN-SFT.Rebuttal: b9uR (2/2). Manuscript: Sec. 6.3, Fig. 8; App. A.9.
Dependence on neologism lengthb9uR Q5We added a preliminary token length ablation on two new skills. With 10 tokens, new–new composition accuracy rises from 9.25% with SFT to 65.75% with RL, comparable to 64.50% for 50-token SN-RL on the same evaluation, showing that the RL benefit persists with shorter neologisms.Rebuttal: b9uR, November 19 follow-up. Manuscript: App. A.10, Table A2.
Contribution and intended use relative to joint fine-tuningJS5B W2; b9uR W2 & Q3We clarified that our contribution concerns zero-shot composition of independently learned reasoning modules, and positioned SN-RL as a method for extending a frozen backbone when joint training of new skills is impractical. We explicitly acknowledge the remaining gap to joint FT-RL on compositions involving only new skills.Rebuttal: JS5B (1/2), b9uR (1/2). Manuscript: Abstract; Secs. 1 and 6.

We thank the reviewers for their feedback and suggestions which helped us significantly improve the manuscript, with six additional experiments and clearer positioning of our contribution. We are pleased that owPy increased their score from 3 to 4 (clear acceptance), noting that the additional experiments mostly address their concerns, while b9uR retained their score of 3 (weak acceptance). We hope the new results and clarifications also address JS5B’s concerns. For further details, please see our full responses.

Thank you once again for your time and consideration.

Best regards,

The Authors of Submission #47992

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.