- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper shows that skill neologisms (training skill-specific embedding vectors while keeping model weights frozen) can acquire compositional reasoning capabilities through SFT and RL. The main conclusion is that SFT brings some compositional capability (but does not beat the baselines), whereas an additional RL training can match the baselines.
Critical strengths
- The experimental setup is carefully designed.
-
The authors distinguish between three types of composition: new-train (seen composition during training), new-train (heldout composition from training), new-new.
-
The authors also control for the models’ performance on the atomic skills when evaluating the compositional skills. The authors also report on the sensitivity of the chosen threshold.
- Strong result on compositional generalization
-
After RL training, Qwen3.5-9B reaches 61% accuracy on the new-new compositions, matching a weights-trained baseline.
-
The authors also extend to composition depths up to 4, showing that RL can unlock multiple layers of compositions with skill neologisms.
- Some evidence beyond a single synthetic task
- While the extensions are preliminary, Section 7 shows promising directions for extending the analysis to domains other than a single synthetic task on string manipulation.
Critical weaknesses
-
(Major) Extension to real-world settings is narrow. The authors already concede to this limitation, but the extension currently only covers 1 composition with 1 base model.
-
(Minor) The authors report that RL training on skill neologism was unstable for Qwen2.5. This raises some minor concerns about the robustness of the method. While the fact that the more modern Qwen3 models showed a stable run partially addresses the concern, it would be good to verify that the method can be applied to other model families from the similar era as Qwen3.
Questions for authors
-
Could the authors validate their findings on more atomic / more combinations on MedCalc- Bench / or on more compositional reasoning benchmarks?
-
Could the authors validate their method on a different model family?
3 comments
Follow-up on Rebuttal
As the discussion period is coming to a close, we wanted to follow up on our response to add an additional experiment. Following your concern that our medical calculations experiment covered only one base model, we have now repeated the MedCalc-Bench experiment with Qwen3.5-9B. We train neologisms on a frozen Qwen3.5-9B backbone for the same two learned skills, using the same training and evaluation protocol as our initial 4B experiment. The results are very similar to those for Qwen3.5-4B: RL improves over SFT on all six tasks. On the four untrained downstream calculations, RL exceeds SFT neologisms by between 32.5 and 42.5 percentage points. These results provide further support for our finding that the benefits of RL for skill neologisms extend beyond our synthetic setting.
We have incorporated these results in Section 7 and Appendix B of our updated manuscript.
Thank you again for your time and effort in reviewing our work. We look forward to your response.
The additional experiments mostly address my concerns. This paper presents a proof of concept that skill neologisms can also be trained via SFT/RL on multi step reasoning tasks offering an alternative approach to full-weights training that allows modular training without forgetting. I would be excited to see this presented at ICLR. Accordingly, I raise my score to a 4.
Rebuttal
We thank the reviewer for their feedback. We appreciate the recognition of our carefully designed experimental setup and the informativeness of our findings. We have updated our manuscript and address your comments below.
Extension to real-world settings [W1, Q1]
We agree that our real-world validation remains limited in scope. We would nevertheless like to clarify that our MedCalc evaluation includes four downstream calculation tasks spanning two independently learned skills: delta gap transfers to three untrained downstream calculations, while adjusted body weight transfers to Cockcroft–Gault.
The main focus of this work is to carefully study the composition abilities of skill neologisms for reasoning, which requires controlling the skills available to the base model and distinguishing atomic mastery from compositional generalization. We therefore center our investigation on a controlled setting where we can explicitly distinguish internalized and new skills and hold out specific compositions during training. In contrast, real-world settings usually involve fuzzy skill definitions and confounding pretrained knowledge, which makes it difficult to distinguish learning a new skill from eliciting an existing capability.
Our MedCalc experiments provide initial evidence that the benefits of RL-trained neologisms extend beyond this controlled setting, while we acknowledge that broader validation across real-world reasoning tasks remains an important next step.
Extension to other model families [W2, Q2]
We thank the reviewer for raising this point. Our initial experiments focused on the Qwen2.5 and Qwen3.5 families in order to study how the performance of skill neologisms for reasoning evolves over gradual improvements in model scale and capabilities. Nevertheless, we agree with the reviewer that our central claim should be validated with another model family.
We now extend our main RL training results to the DeepSeek-R1-Distill-Llama-8B model. We use the same training pipeline and datasets as for Qwen3.5-9B to train the pretrained model, the SN-SFT and SN-RL neologisms and the idealized FT-RL baseline. RL training succeeds for all 10 skills for this backbone. The results are consistent with those for Qwen3.5-9B: SN-RL performs significantly better than SN-SFT across compositions. For example, L2 new×new accuracy increases from 3.3% with SN-SFT to 31.3% with SN-RL. A gap does remain with the idealized FT-RL baseline, particularly for the new-only compositions. These results suggest that while RL training strongly benefits skill neologisms for models beyond the Qwen family, the extent to which RL closes the gap with the composition of internalized skills varies across models.
We have incorporated the new results and discussion in Section 6 and Appendix A.6 of the updated manuscript.
Thank you for your time and effort. We hope that our responses and additional experimental results address your concerns. We are happy to engage with any further comments you may have.