- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
GrowRL freezes the pretrained backbone and adds a small trainable pathway to each Transformer layer. Only this pathway is trained with Dr. GRPO. The learned pathway can then be inserted into a structurally compatible target model. Across five backbones, GrowRL is competitive with LoRA, DoRA, Parallel Adapter, and often Full RL.
Critical strengths
- The paper is well-written and easy to follow. The motivation behind GrowRL is clearly presented, and the proposed architecture is relatively simple and intuitive. The methodology and training procedures are explained in sufficient detail.
- The paper provides a fairly comprehensive empirical comparison. The authors compare GrowRL with several representative baselines, including full-parameter RL, LoRA, DoRA, and Parallel Adapter. Moreover, the experiments are conducted on five backbones and multiple mathematical reasoning and out-of-distribution benchmarks, which helps demonstrate the effectiveness of the proposed approach.
Critical weaknesses
- The code is not publicly available. Although the paper provides implementation details, releasing the training and transfer code would significantly improve reproducibility, particularly because GrowRL requires modifications to the standard Transformer architecture.
- The models used in this paper are still Qwen2.5 and Llama 3.2. Validation on newer models would strengthen the conclusions.
- The experiments only include transfers from the same model structure, which is less general than on-policy distillation. The result would be much stronger if GrowRL can be applied to different model sizes: e.g., 7B -> 1.5B.
Questions for authors
- Could the authors add a comparison to on-policy distillation (OPD)? To make the comparison fair, I would like to see an -step RL + -step OPD, compared with -step GrowRL.
- Is it possible to transfer GrowRL pathways between models of different sizes? For example, could a pathway trained on Qwen2.5-7B be transferred to Qwen2.5-1.5B?
6 comments
We sincerely thank you for your constructive comments — Experiments and analyses on newer model architectures, on-policy distillation, and cross-scale and cross-family transfer (Part 2/2)
(Question 1)
Comparison with RL + On-Policy Distillation
Following your suggestion, we use x=100, 200, 300, 400 to ensure a fair comparison, with Qwen2.5-Math-7B as the source model and Qwen2.5-Math-1.5B as the target model. The experimental results are shown in Table 4.
Table 4. Experimental results for Qwen2.5-Math-7B → Qwen2.5-Math-1.5B. The metrics are MATH500 Pass@1 / Pass@8 (%), with 8 samples per problem.
| Setting | Pass@1 / Pass@8 |
|---|---|
| Target Base | 25.13 / 74.20 |
| 400-step RL + 100-step OPD | 56.10 / 79.80 |
| 300-step RL + 200-step OPD | 57.23 / 82.60 |
| 200-step RL + 300-step OPD | 59.00 / 83.60 |
| 100-step RL + 400-step OPD | 59.48 / 83.80 |
| 500-step GrowRL | 63.20 / 85.00 |
(Weakness 3 / Question 2)
Can GrowRL Transfer to Models of Different Scales?
Yes. We have the following new experiments, including transfer across models with different scales, different layer counts, and different model families.
While keeping the target backbone frozen, we address architectural differences through Grow pathway interface alignment, branch strength calibration, and final-layer output calibration. This converts the computations learned by the source pathway into a form that the target model can use. The experimental results are shown in Table 5.
Table 5. Cross-architecture transfer results for our methods. The metric is MATH500 Pass@1 (%), with 8 samples per problem. No RL updates to the target model are required.
| Source Model → Target Model | Architectural Differences | Original Target Model | Cross-Architecture Grow Pathway Transfer | Improvement | Calibration Time (Seconds) |
|---|---|---|---|---|---|
| Qwen2.5-Math-7B → Qwen2.5-Math-1.5B | Different scales | 25.13 | 38.58 | +13.45 | 171.35 |
| Qwen2.5-Math-1.5B → Qwen2.5-Math-7B | Different scales | 44.03 | 61.33 | +17.30 | 231.66 |
| Qwen2.5-Math-1.5B → Qwen2.5-3B | Different scales and layer counts | 33.15 | 48.50 | +15.35 | 133.30 |
| FineMath-Llama-3B → Qwen2.5-Math-1.5B | Different scales and model families | 25.13 | 33.13 | +8.00 | 126.48 |
| FineMath-Llama-3B → Qwen2.5-1.5B | Different scales and model families | 4.35 | 14.18 | +9.83 | 121.47 |
As shown in Table 5, the target models all achieve substantial improvements after transfer across model scales, layer counts, and model families, and calibration is extremely fast.
In addition, we validated a short-run distillation-based transfer approach: the change in the source model’s logits before and after adding Grow serves as the transfer signal, and only the newly added Grow pathway in the target model is updated. As shown in Table 6, just 10 steps quickly yield a substantial improvement.
Table 6. Short-run distillation-based transfer results for Qwen2.5-Math-7B → Qwen2.5-Math-1.5B. Evaluation is conducted on Math500; the metric is Pass@1 (%), with 8 samples per problem.
| Metric | 0 (Base) | 10 Steps | 100 Steps |
|---|---|---|---|
| Pass@1 | 25.13 | 45.68 | 45.48 |
| Pass@8 | 74.20 | 81.40 | 80.60 |
Thank you for the additional experiments and detailed clarifications!
I have one question regarding Tables 4 and 5. Why does the Pass@1 for Qwen2.5-Math-7B → Qwen2.5-Math-1.5B improve from 25.13 to 63.20 in Table 4, but only from 25.13 to 38.58 in Table 5?
From my understanding of the original paper (Table 1), 63.20% corresponds to directly training GrowRL on Qwen2.5-Math-1.5B, rather than transferring from 7B to 1.5B. If so, the comparison with RL + OPD may not be entirely fair.
Would the following be a more appropriate comparison for 7B → 1.5B transfer?
| Setting | Pass@1 / Pass@8 |
|---|---|
| Target Base | 25.13 / 74.20 |
| 400-step RL + 100-step OPD | 56.10 / 79.80 |
| 300-step RL + 200-step OPD | 57.23 / 82.60 |
| 200-step RL + 300-step OPD | 59.00 / 83.60 |
| 100-step RL + 400-step OPD | 59.48 / 83.80 |
| Grow Pathway Transfer (Table 5) | 38.58 / Not Reported |
This would suggest that Grow pathway transfer performs substantially worse than RL + OPD. I would appreciate it if the authors could clarify whether I have misunderstood the experimental setup. Thank you!
We sincerely thank you for your constructive comments — Experiments and analyses on newer model architectures, on-policy distillation, and cross-scale and cross-family transfer (Part 1/2)
The current manuscript provides details needed to reproduce the GrowRL method: Section 3.1 describes the Grow pathway architecture and initialization; Section 3.2 introduces transfer and sequential composition; Appendices A.1–A.3 specify the training, evaluation, and target adaptation details; and B.1 further explains the initialization and gradient properties, providing a clear basis for independently implementing and validating the method. We commit to publicly releasing the code for training, pathway transfer, integration with vLLM frameworks, and evaluation upon publication. The implementations for the newly added MoE and hybrid attention experiments will also be released, and the relevant implementation details and experimental results will be included in the revised manuscript.
The original paper uses the Qwen2.5 and Llama families to maintain comparability with commonly used settings in existing RLVR studies. We have further added the more recent Qwen3.5-4B-Base hybrid attention model and Qwen3-30B-A3B-Base MoE.
This model contains 24 GDN layers and 8 standard attention layers. For GDN layers, we add an independent recurrent state pathway, preserving native gating, convolution, and state updates; For standard attention layers, we add one additional attention head; For FFNs in every layer, we add a SwiGLU branch with a width of 128.
All added output projections are initialized to zero, and only approximately 97.15M parameters, accounting for 2.31% of the original backbone, are trained. On the vLLM side, we made adaptations for GDN state caching, GVA grouping, and divisibility requirements for parallel execution. The added LoRA baseline uses rank 48, with 97.39M trainable parameters, closely matching GrowRL’s budget. The results are shown in Table 1.
Table 1. Evaluation results on the Qwen3.5-4B-Base hybrid attention model. The metrics are Pass@1 / Pass@8 (%), with 8 samples per problem.
We also completed validation on Qwen3-30B-A3B-Base MoE, with representative results shown in Table 2 and Table 3.
Table 2. Results on math datasets for Qwen3-30B-A3B-Base MoE. The metrics are Pass@1 / Pass@8 (%), with 8 samples per problem.
Table 3. Results on OOD datasets for Qwen3-30B-A3B-Base MoE. The metrics are Pass@1 / Pass@8 (%), with 8 samples per problem.
As shown in Table 2 and Table 3, GrowRL also achieves substantial improvements on the MoE backbone. Together, the results on hybrid attention and MoE architectures support GrowRL’s architectural applicability.