acceptodds
Under review as a conference paper at ICLR 2027

Growing and Transferring Pathways for LLM Reinforcement Learning

Abstract

Reinforcement learning (RL) can substantially improve the reasoning capabilities of large language models, but an effective training run requires considerable resources. Reusing the computation acquired through RL across models would extend the value of that training. We propose GrowRL, an RL method that organizes learned updates into compact computational pathways through structural growth. GrowRL freezes the pretrained backbone and adds an independent causal attention branch and a feed-forward network branch at each Transformer layer. Zero-initialized output projections preserve the pretrained policy at initialization, and RL updates only the added pathways. The learned pathways remain explicit after training, allowing transfer to structurally compatible target models and subsequent adaptation with the target backbone frozen. The architecture also supports sequential transfer of additional pathways. By keeping the learned attention and feed-forward computation separate from the pretrained weights, GrowRL provides an explicit structure for preserving and reusing the outcome of an RL run. Experiments across five backbones show competitive mathematical reasoning and out-of-distribution performance with a small number of trainable parameters. Learned pathways provide immediate gains across all three transfer directions, with performance after only 10 target-side updates already close to that after 100 updates in every case. These results support computational pathways as parameter-efficient and reusable units of reinforcement learning adaptation.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Large Language Models, Structural Growth

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XaDZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

GrowRL freezes the pretrained backbone and adds a small trainable pathway to each Transformer layer. Only this pathway is trained with Dr. GRPO. The learned pathway can then be inserted into a structurally compatible target model. Across five backbones, GrowRL is competitive with LoRA, DoRA, Parallel Adapter, and often Full RL.

Critical strengths
  • The paper is well-written and easy to follow. The motivation behind GrowRL is clearly presented, and the proposed architecture is relatively simple and intuitive. The methodology and training procedures are explained in sufficient detail.
  • The paper provides a fairly comprehensive empirical comparison. The authors compare GrowRL with several representative baselines, including full-parameter RL, LoRA, DoRA, and Parallel Adapter. Moreover, the experiments are conducted on five backbones and multiple mathematical reasoning and out-of-distribution benchmarks, which helps demonstrate the effectiveness of the proposed approach.
Critical weaknesses
  • The code is not publicly available. Although the paper provides implementation details, releasing the training and transfer code would significantly improve reproducibility, particularly because GrowRL requires modifications to the standard Transformer architecture.
  • The models used in this paper are still Qwen2.5 and Llama 3.2. Validation on newer models would strengthen the conclusions.
  • The experiments only include transfers from the same model structure, which is less general than on-policy distillation. The result would be much stronger if GrowRL can be applied to different model sizes: e.g., 7B -> 1.5B.
Questions for authors
  • Could the authors add a comparison to on-policy distillation (OPD)? To make the comparison fair, I would like to see an xx-step RL + (500−x)(500-x)-step OPD, compared with 500500-step GrowRL.
  • Is it possible to transfer GrowRL pathways between models of different sizes? For example, could a pathway trained on Qwen2.5-7B be transferred to Qwen2.5-1.5B?
6 comments
AuthorsOpenReview ↗

We sincerely thank you for your constructive comments — Experiments and analyses on newer model architectures, on-policy distillation, and cross-scale and cross-family transfer (Part 1/2)

(Weakness 1) Code Availability and Reproducibility

The current manuscript provides details needed to reproduce the GrowRL method: Section 3.1 describes the Grow pathway architecture and initialization; Section 3.2 introduces transfer and sequential composition; Appendices A.1–A.3 specify the training, evaluation, and target adaptation details; and B.1 further explains the initialization and gradient properties, providing a clear basis for independently implementing and validating the method. We commit to publicly releasing the code for training, pathway transfer, integration with vLLM frameworks, and evaluation upon publication. The implementations for the newly added MoE and hybrid attention experiments will also be released, and the relevant implementation details and experimental results will be included in the revised manuscript.

(Weakness 2) Validation on Newer Models and Architectures

The original paper uses the Qwen2.5 and Llama families to maintain comparability with commonly used settings in existing RLVR studies. We have further added the more recent Qwen3.5-4B-Base hybrid attention model and Qwen3-30B-A3B-Base MoE.

This model contains 24 GDN layers and 8 standard attention layers. For GDN layers, we add an independent recurrent state pathway, preserving native gating, convolution, and state updates; For standard attention layers, we add one additional attention head; For FFNs in every layer, we add a SwiGLU branch with a width of 128.

All added output projections are initialized to zero, and only approximately 97.15M parameters, accounting for 2.31% of the original backbone, are trained. On the vLLM side, we made adaptations for GDN state caching, GVA grouping, and divisibility requirements for parallel execution. The added LoRA baseline uses rank 48, with 97.39M trainable parameters, closely matching GrowRL’s budget. The results are shown in Table 1.

Table 1. Evaluation results on the Qwen3.5-4B-Base hybrid attention model. The metrics are Pass@1 / Pass@8 (%), with 8 samples per problem.

DatasetBase (ref.)Full RL (ref.)LoRAGrowRL
MATH50076.93 / 91.2084.30 / 95.8081.35 / 93.2084.50 / 95.20
AMC202364.38 / 87.5074.38 / 95.0068.75 / 87.5072.19 / 92.50
AIME202420.00 / 40.0031.25 / 43.3327.92 / 50.0029.17 / 50.00
Minerva33.09 / 56.9939.89 / 61.0336.72 / 59.9339.20 / 58.82
OlympiadBench41.64 / 57.5750.91 / 66.4746.51 / 61.4250.33 / 67.51
GSM8K92.08 / 98.3394.08 / 97.9593.88 / 98.1094.71 / 98.10
HumanEval21.80 / 61.5928.58 / 78.0528.81 / 76.2229.65 / 75.61
MMLU-STEM52.47 / 85.7267.57 / 91.6260.55 / 89.1768.77 / 91.65
ARC-Challenge64.27 / 96.2578.38 / 98.2972.23 / 97.3579.87 / 98.46
MMLU-Humanities36.55 / 75.6448.01 / 84.0244.30 / 81.8152.33 / 86.40
HellaSwag16.30 / 53.9617.06 / 55.6716.55 / 54.3317.21 / 55.17

We also completed validation on Qwen3-30B-A3B-Base MoE, with representative results shown in Table 2 and Table 3.

Table 2. Results on math datasets for Qwen3-30B-A3B-Base MoE. The metrics are Pass@1 / Pass@8 (%), with 8 samples per problem.

DatasetBase (ref.)Full RL (ref.)LoRAGrowRL
MATH50054.08 / 88.6077.90 / 89.4060.15 / 89.8077.28 / 88.20
AMC202338.44 / 80.0068.13 / 92.5049.06 / 90.0062.81 / 82.50
AIME202410.00 / 36.6718.75 / 46.6712.08 / 43.3318.33 / 43.33
Minerva19.03 / 47.7933.82 / 47.0622.89 / 50.7434.15 / 50.00
OlympiadBench23.37 / 53.8638.48 / 54.7527.21 / 53.4139.35 / 55.49

Table 3. Results on OOD datasets for Qwen3-30B-A3B-Base MoE. The metrics are Pass@1 / Pass@8 (%), with 8 samples per problem.

DatasetBase (ref.)Full RL (ref.)LoRAGrowRL
GSM8K71.33 / 97.9593.84 / 98.1877.58 / 98.2693.61 / 97.95
HumanEval41.23 / 91.4665.02 / 96.3447.71 / 94.5169.28 / 95.73
MMLU-STEM51.13 / 95.0673.18 / 93.5454.85 / 95.2374.48 / 93.17
ARC-Challenge56.10 / 98.2183.32 / 98.4660.13 / 98.1286.97 / 98.55
MMLU-Humanities39.12 / 90.8856.38 / 92.9641.05 / 91.6358.38 / 92.71
HellaSwag24.14 / 80.0236.52 / 90.6025.42 / 81.0237.23 / 91.18

As shown in Table 2 and Table 3, GrowRL also achieves substantial improvements on the MoE backbone. Together, the results on hybrid attention and MoE architectures support GrowRL’s architectural applicability.

AuthorsOpenReview ↗

We sincerely thank you for your constructive comments — Experiments and analyses on newer model architectures, on-policy distillation, and cross-scale and cross-family transfer (Part 2/2)

(Question 1) Comparison with RL + On-Policy Distillation

Following your suggestion, we use x=100, 200, 300, 400 to ensure a fair comparison, with Qwen2.5-Math-7B as the source model and Qwen2.5-Math-1.5B as the target model. The experimental results are shown in Table 4.

Table 4. Experimental results for Qwen2.5-Math-7B → Qwen2.5-Math-1.5B. The metrics are MATH500 Pass@1 / Pass@8 (%), with 8 samples per problem.

SettingPass@1 / Pass@8
Target Base25.13 / 74.20
400-step RL + 100-step OPD56.10 / 79.80
300-step RL + 200-step OPD57.23 / 82.60
200-step RL + 300-step OPD59.00 / 83.60
100-step RL + 400-step OPD59.48 / 83.80
500-step GrowRL63.20 / 85.00

(Weakness 3 / Question 2) Can GrowRL Transfer to Models of Different Scales?

Yes. We have the following new experiments, including transfer across models with different scales, different layer counts, and different model families.

While keeping the target backbone frozen, we address architectural differences through Grow pathway interface alignment, branch strength calibration, and final-layer output calibration. This converts the computations learned by the source pathway into a form that the target model can use. The experimental results are shown in Table 5.

Table 5. Cross-architecture transfer results for our methods. The metric is MATH500 Pass@1 (%), with 8 samples per problem. No RL updates to the target model are required.

Source Model → Target ModelArchitectural DifferencesOriginal Target ModelCross-Architecture Grow Pathway TransferImprovementCalibration Time (Seconds)
Qwen2.5-Math-7B → Qwen2.5-Math-1.5BDifferent scales25.1338.58+13.45171.35
Qwen2.5-Math-1.5B → Qwen2.5-Math-7BDifferent scales44.0361.33+17.30231.66
Qwen2.5-Math-1.5B → Qwen2.5-3BDifferent scales and layer counts33.1548.50+15.35133.30
FineMath-Llama-3B → Qwen2.5-Math-1.5BDifferent scales and model families25.1333.13+8.00126.48
FineMath-Llama-3B → Qwen2.5-1.5BDifferent scales and model families4.3514.18+9.83121.47

As shown in Table 5, the target models all achieve substantial improvements after transfer across model scales, layer counts, and model families, and calibration is extremely fast.

In addition, we validated a short-run distillation-based transfer approach: the change in the source model’s logits before and after adding Grow serves as the transfer signal, and only the newly added Grow pathway in the target model is updated. As shown in Table 6, just 10 steps quickly yield a substantial improvement.

Table 6. Short-run distillation-based transfer results for Qwen2.5-Math-7B → Qwen2.5-Math-1.5B. Evaluation is conducted on Math500; the metric is Pass@1 (%), with 8 samples per problem.

Metric0 (Base)10 Steps100 Steps
Pass@125.1345.6845.48
Pass@874.2081.4080.60

Thank you for the additional experiments and detailed clarifications!

I have one question regarding Tables 4 and 5. Why does the Pass@1 for Qwen2.5-Math-7B → Qwen2.5-Math-1.5B improve from 25.13 to 63.20 in Table 4, but only from 25.13 to 38.58 in Table 5?

From my understanding of the original paper (Table 1), 63.20% corresponds to directly training GrowRL on Qwen2.5-Math-1.5B, rather than transferring from 7B to 1.5B. If so, the comparison with RL + OPD may not be entirely fair.

Would the following be a more appropriate comparison for 7B → 1.5B transfer?

SettingPass@1 / Pass@8
Target Base25.13 / 74.20
400-step RL + 100-step OPD56.10 / 79.80
300-step RL + 200-step OPD57.23 / 82.60
200-step RL + 300-step OPD59.00 / 83.60
100-step RL + 400-step OPD59.48 / 83.80
Grow Pathway Transfer (Table 5)38.58 / Not Reported

This would suggest that Grow pathway transfer performs substantially worse than RL + OPD. I would appreciate it if the authors could clarify whether I have misunderstood the experimental setup. Thank you!

Reviewer_XtxFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GrowRL, a method that makes the computation acquired through RL explicit and reusable by adding, at each layer of a frozen backbone, a new attention branch and a narrow SwiGLU FFN branch. Both output projections are zero-initialized to exactly preserve the initial policy, and only the added pathways are trained (about 38.5M parameters for the 1.5B models). The authors compare GrowRL with Full RL, LoRA, DoRA, and PA across five backbones and eleven datasets, and evaluate immediate transfer, post-transfer adaptation, and sequential transfer on three source–target pairs. Results show that GrowRL is competitive on its source models and approaches Full RL in several settings; immediate transfer improves MATH500 on all three pairs without target-side training; post-transfer adaptation yields further gains; and sequential transfer accumulates gains on Qwen2.5-1.5B and 7B but declines after one transfer on Llama-3.2-3B.

Critical strengths
  1. It makes the computation acquired through RL explicit and structurally reusable, and validates transfer on three source–target pairs. By organizing trainable parameters into complete attention and FFN pathways with zero-initialized output projections, GrowRL keeps the learned update as a structurally distinct component that can be inserted into compatible models. Table 3 shows immediate transfer improves MATH500 Pass@1 on all three pairs without any target-side training; Table 6 and Figure 2 show that 100 steps of target-side adaptation on the transferred pathway yield further gains; and Table 4 shows sequential transfer accumulates gains on Qwen2.5-1.5B and 7B. Together these results consistently support the claim that RL-acquired computation can be preserved and reused rather than remaining tied to a single model.

  2. Source-model RL is competitive across five backbones and eleven datasets, and approaches Full RL in several settings. Tables 1–2 cover FineMath-Llama-3B, Llama-3.2-3B, Qwen2.5-Math-1.5B/7B, and Qwen2.5-7B, spanning general-purpose and math-specialized models. GrowRL exceeds LoRA on all five mathematical benchmarks under Pass@1 on Qwen2.5-Math-7B, and remains close to PA on Qwen2.5-7B, while training only the added pathways. This indicates that structural growth is a viable alternative organization of trainable capacity for RL, beyond simply reducing the number of trainable parameters.

Critical weaknesses
  1. The conditions for "structural compatibility" in transfer are never made precise, and the experiments cover only three same-family, similar-scale pairs. The paper's central claim is that learned pathways are transferable, but it gives no testable definition of compatibility and tests only Qwen→Qwen and Llama→Llama at matched scales. If compatibility requires matching hidden dimension, head count, layer count, and positional convention, the scope of "transferable" may be far narrower than implied. This directly limits the core claim.

  2. The DoRA baseline may be systematically understated under the shared BF16 setup, yet it is still presented as a comparable baseline in the main table. Appendix A.1 states that DoRA's magnitude parameters "changed little or not at all" during BF16 conversion and that the results characterize DoRA "rather than its best performance with a separately optimized precision configuration." DoRA's uniformly low scores in Table 1 may therefore partly reflect the precision setup rather than the method itself.

Questions for authors
  1. Can the authors give a testable definition of "structural compatibility"? What exact conditions (hidden dimension, head count, layer count, positional convention, normalization type) must hold between source and target for transfer to work? Would transfer succeed across families (e.g., Qwen → Llama) or across scales (e.g., 1.5B → 7B)?

  2. Can the authors re-run DoRA under an appropriate precision configuration? Appendix A.1 states that DoRA's magnitude parameters "changed little or not at all" under the shared BF16 setup. Would DoRA's scores in Table 1 change materially if its magnitude parameters were kept in a precision that preserves their updates?

1 comment
AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analyses on structural compatibility, cross-scale and cross-family transfer, and corrected DoRA precision

(Weakness 1 / Question 1) What Architectural Conditions Allow Transfer? Can GrowRL Transfer across Model Scales and Families?

Yes. GrowRL can transfer across models with different scales, different layer counts, and different model families.

Table 1. Cross-architecture transfer results for GrowRL. The metric is MATH500 Pass@1 (%), with 8 samples per problem. No RL updates to the target model are required.

Source Model → Target ModelArchitectural DifferencesOriginal Target ModelCross-Architecture Grow Pathway TransferImprovementCalibration Time (Seconds)
Qwen2.5-Math-7B → Qwen2.5-Math-1.5BDifferent scales25.1338.58+13.45171.35
Qwen2.5-Math-1.5B → Qwen2.5-Math-7BDifferent scales44.0361.33+17.30231.66
Qwen2.5-Math-1.5B → Qwen2.5-3BDifferent scales and layer counts33.1548.50+15.35133.30
FineMath-Llama-3B → Qwen2.5-Math-1.5BDifferent scales and model families25.1333.13+8.00126.48
FineMath-Llama-3B → Qwen2.5-1.5BDifferent scales and model families4.3514.18+9.83121.47

As shown in Table 1, the target models all achieve substantial improvements after transfer across model scales, layer counts, and model families, and calibration is extremely fast.

We briefly summarize the requirements of direct transfer and how to handle architectural mismatches through interface calibration in Table 2, along the dimensions comprehensively suggested by the reviewer.

Table 2. Architectural requirements for direct transfer of Grow pathways and approaches to handling architectural mismatches.

Architectural ConditionDirect Transfer RequirementApproach to Handling Architectural Differences
Hidden dimensionThe source and target models have the same hidden dimension.Calibrate the input and output projections of the Grow pathway to map it into the target representation space.
Number of attention headsThe source and target backbones do not need to have the same numbers of Q heads or KV heads.Retain Grow’s independent Q/K/V projections and head structure, without changing the backbone’s head counts.
Number of layersThe source and target models have the same number of layers, with a one-to-one correspondence by layer index.Establish a mapping between source and target layers based on normalized depth.
Positional encodingBoth the source and target use RoPE, with the rotary dimension matching the Grow head dimension.Use the target model’s RoPE parameters, scaling settings, and position indices.
NormalizationThe branch insertion points are the same, and the normalization output dimensions match.Use the target model’s normalization outputs and calibrate the pathway projections and branch strengths.

In addition, we validated a short-run distillation-based transfer approach: the change in the source model’s logits before and after adding Grow serves as the transfer signal, and only the newly added Grow pathway in the target model is updated. As shown in Table 3, just 10 steps quickly yield a substantial improvement.

Table 3. Short-run distillation-based transfer results for Qwen2.5-Math-7B → Qwen2.5-Math-1.5B. Evaluation is conducted on Math500; the metric is Pass@1 (%), with 8 samples per problem.

Metric0 (Base)10 Steps100 Steps
Pass@125.1345.6845.48
Pass@874.2081.4080.60

(Weakness 2 / Question 2) Re-evaluating DoRA at Appropriate Precision

As suggested by the reviewer, the BF16 setting may underestimate DoRA’s performance. We therefore retrain and evaluate DoRA on FineMath-Llama-3B using a more appropriate precision configuration. The magnitude parameters mm, low-rank parameters AA/BB, norm and magnitude normalization computations, and the optimizer now use FP32. The results are shown in Table 4.

Table 4. DoRA results before and after the precision correction on FineMath-Llama-3B. The metrics are Pass@1 / Pass@8 (%), with 8 samples per problem; GrowRL is included as a reference.

DatasetOriginal BF16 DoRADoRA with Corrected PrecisionGrowRL
MATH5007.93 / 35.2011.10 / 41.4033.30 / 63.00
AMC20232.50 / 20.002.81 / 22.5012.50 / 40.00
AIME20240.00 / 0.000.42 / 3.331.67 / 3.33
Minerva3.63 / 18.754.27 / 19.4912.59 / 32.72
OlympiadBench1.85 / 11.422.13 / 11.428.92 / 26.26

As shown in Table 4, the precision correction does improve DoRA. GrowRL continues to perform better in the tested settings. Your suggestion helped us identify and quantify this effect. We will include the corrected configuration and results in the revised manuscript to avoid presenting the original BF16 results as DoRA’s best performance.

Reviewer_8J2VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This papper proposes GrowRL, which introduces additional computational pathways to an LLM and finetunes them with RL. Compared to baselines including LoRA, the method shows improved performance and transferability across the same model families.

Critical strengths

Generally, GrowRL shows improved performance across i.i.d. datasets. And the transferability within the same model family is better than LoRA.

Critical weaknesses

(1) Performance comparison in Table 2 seems inconsistent. And the authors do not provide any second-order metrices like standard deviation. RL training is known to have large variance, I enrourage the authors to consider including at least SD to better showcase the improvement.

(2) Computational overhead is not known. The authors do not provide end-to-end comparison between baselines in terms of additional computational/time consumption. Also, the number of trainable parameters of each run is not shown.

(3) Table 3 should include the PA baseline.

Questions for authors

(1) Is GrowRL applicable to MoE models, which is more popular for large scale LLMs?

(2) Introducing new structure to an LLM is not specifically tie to RL training. How does GrowRL perform if you consider SFT-style training compared to LoRA?

(3) How much work does it take to support such structural in popular LLM inference framework like vLLM/SGLang? Do you use customized version of vLLM/SGLang in your experiments? How much is the additiaon overhead?

(4) You only consider one Math dataset for training, which is limited. Can you provide more results on coding RL?

13 comments
AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analyses on statistical reliability, parameter budgets and training costs, PA transfer, MoE, SFT, system integration, and coding RL (Part 1/6)

(Weakness 1) RL Training Variability and Statistical Reliability

As suggested by the reviewer, we have supplemented our results with statistical analyses of both evaluation uncertainty and RL training variability. For uncertainty evaluation, we performed 10,000 paired bootstrap resamples at the problem level and report the results in Table 1

Table 1. Pass@1 and Pass@8 improvements of GrowRL over LoRA on GSM8K, with paired bootstrap 95% confidence intervals. Improvements are reported in percentage points.

BackbonePass@1 improvement95% confidence intervalPass@8 improvement95% confidence interval
FineMath-Llama-3B28.25[26.75, 29.75]11.90[9.70, 14.10]
Llama-3.2-3B8.14[7.03, 9.25]9.63[6.60, 12.66]
Qwen2.5-Math-1.5B2.07[1.14, 2.99]0.91[−0.08, 1.97]
Qwen2.5-Math-7B5.80[4.86, 6.72]0.15[−0.53, 0.83]
Qwen2.5-7B5.30[4.49, 6.08]0.15[−0.45, 0.83]

For RL training variability assessment, we additionally evaluated three independent training runs for each of GrowRL and LoRA on Qwen2.5-Math-1.5B. As shown in Table 2, we report the mean and sample standard deviation across training seeds on all 11 datasets under the same evaluation protocol.

Table 2. Results (accuracy %) from three independent RL training seeds on Qwen2.5-Math-1.5B. Each checkpoint is evaluated with 8 samples per problem; values are means ± sample standard deviations across training seeds.

DatasetGrowRL Pass@1LoRA Pass@1GrowRL Pass@8LoRA Pass@8
MATH50062.54 ± 0.5761.03 ± 0.4985.00 ± 1.2584.87 ± 1.92
AMC202345.67 ± 1.8343.75 ± 1.1373.33 ± 3.8270.00 ± 5.00
AIME202410.42 ± 0.009.03 ± 2.5137.78 ± 1.9230.00 ± 3.33
Minerva19.93 ± 0.5520.24 ± 0.1738.24 ± 0.9737.75 ± 0.93
OlympiadBench26.27 ± 0.4825.23 ± 0.5044.56 ± 0.6243.92 ± 0.77
GSM8K74.53 ± 0.3172.66 ± 0.5195.38 ± 0.3994.72 ± 0.56
HumanEval29.14 ± 0.6420.25 ± 0.3467.07 ± 1.6162.60 ± 0.70
MMLU-STEM43.21 ± 0.0643.61 ± 0.3281.79 ± 0.2585.19 ± 0.32
ARC-Challenge51.98 ± 0.2749.04 ± 0.4692.32 ± 0.3091.81 ± 0.39
MMLU-Humanities31.31 ± 0.1330.29 ± 0.3583.27 ± 0.1484.05 ± 0.54
HellaSwag11.83 ± 0.2814.04 ± 0.4747.02 ± 0.8253.72 ± 1.37
AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analyses on statistical reliability, parameter budgets and training costs, PA transfer, MoE, SFT, system integration, and coding RL (Part 2/6)

(Weakness 2) Trainable Parameter Counts and Computational Overhead

We show the effective trainable parameter counts for all backbones and methods in Table 3(a).

Table 3(a). Effective trainable parameter counts for each backbone and method in the original experiments, in millions of parameters (M).

BackboneFull RLLoRADoRAPAGrowRL
Qwen2.5-Math-1.5B1543.7136.9337.5716.5238.54
FineMath-Llama-3B / Llama-3.2-3B3212.7548.6349.4033.0377.07
Qwen2.5-Math-7B / Qwen2.5-7B7615.6280.7482.1338.5489.92

To further examine the effect of the parameter budget, we widened the PA bottleneck to 448 so that its trainable parameter count matches GrowRL on the two backbones, and conducted additional training and evaluation. The configurations are shown in Table 3(b).

Table 3(b). Configurations of the additional parameter-matched PA experiments.

Source BackbonePA Bottleneck WidthPA Parameter CountGrowRL Parameter Count
Qwen2.5-Math-1.5B44838.54M38.54M
FineMath-Llama-3B44877.07M77.07M

We present the results of parameter-matched PA (PA448) alongside the original PA (PA192) and GrowRL in Table 4(a) and Table 4(b).

Table 4(a). Results on Qwen2.5-Math-1.5B. The metrics are Pass@1 / Pass@8 (%), with 8 samples per problem.

DatasetPA192PA448GrowRL
MATH50046.80 / 82.2049.08 / 83.4063.20 / 85.00
AMC202332.81 / 75.0038.44 / 72.5045.94 / 75.00
AIME20245.00 / 23.335.00 / 26.679.17 / 26.67
Minerva13.10 / 37.8715.24 / 37.5019.44 / 36.76
OlympiadBench18.99 / 42.4321.70 / 43.4726.67 / 44.51
GSM8K55.93 / 93.6360.20 / 94.4774.72 / 95.22
HumanEval5.79 / 34.766.14 / 34.6331.10 / 70.73
MMLU-STEM32.67 / 86.7841.09 / 83.4743.18 / 81.58
ARC-Challenge32.35 / 89.3327.18 / 89.7251.75 / 91.89
MMLU-Humanities21.49 / 80.7020.24 / 80.8931.25 / 83.66
HellaSwag11.84 / 51.7611.41 / 52.0011.90 / 46.84

Table 4(b). Results on FineMath-Llama-3B. The metrics are Pass@1 / Pass@8 (%), with 8 samples per problem.

DatasetPA192PA448GrowRL
MATH50030.58 / 60.605.25 / 30.4033.30 / 63.00
AMC202311.25 / 40.003.44 / 20.0012.50 / 40.00
AIME20240.42 / 3.330.00 / 0.001.67 / 3.33
Minerva11.86 / 35.293.26 / 19.8512.59 / 32.72
OlympiadBench7.18 / 23.891.85 / 12.178.92 / 26.26
GSM8K54.43 / 86.889.51 / 50.0459.51 / 87.34
HumanEval0.00 / 0.000.00 / 0.000.00 / 0.00
MMLU-STEM12.27 / 60.8014.85 / 69.4515.12 / 66.20
ARC-Challenge15.18 / 68.7715.33 / 67.5819.64 / 75.26
MMLU-Humanities12.89 / 63.3413.87 / 66.9715.06 / 67.74
HellaSwag12.26 / 56.7011.86 / 58.1812.26 / 57.01

As shown in Table 4(a) and Table 4(b), GrowRL still demonstrates better overall performance on both backbones. We also evaluated the transfer capability of these PA448 pathways; the results are presented in Table 9(a) and Table 9(b).

We further summarized the training costs for the five backbones, including rollouts and policy updates, standardized to 500 RL steps, as shown in Table 5.

Table 5. End to end training-loop computational costs (in hours) for the five backbones, in A800 GPU-hours.

BackboneFull RLLoRADoRAPA192GrowRL
FineMath-Llama-3B117.27132.84179.47112.95123.92
Llama-3.2-3B85.02117.88184.06102.0381.61
Qwen2.5-Math-1.5B60.1867.4988.4461.8872.11
Qwen2.5-Math-7B235.95327.66319.22320.19319.51
Qwen2.5-7B163.96217.95288.65190.01180.36

As shown in Table 5, the training costs of GrowRL, PA192, and LoRA are of a similar order of magnitude.

AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analyses on statistical reliability, parameter budgets and training costs, PA transfer, MoE, SFT, system integration, and coding RL (Part 3/6)

(Weakness 3) Additional PA Immediate Transfer and Target-Side Adaptation

We have added the PA results for all three transfer directions in Table 3 of the original paper, as shown in Table 6, Table 7, and Table 8, respectively.

Table 6. Transfer results for Qwen2.5-Math-1.5B → Qwen2.5-1.5B. The metric is Pass@1 (%), with 8 samples per problem.

MethodMATH500AMC2023AIME2024MinervaOlympiadBenchGSM8KHumanEvalMMLU-STEMARC-CMMLU-H
Base4.632.500.001.101.136.123.286.587.606.25
LoRA6.903.440.002.251.699.046.337.308.446.71
PA10.253.130.002.713.1514.838.089.0610.478.56
GrowRL22.6310.941.674.787.0726.0812.279.6310.617.31

Table 7. Transfer results for FineMath-Llama-3B → Llama-3.2-3B. The metric is Pass@1 (%), with 8 samples per problem.

MethodMATH500AMC2023AIME2024MinervaOlympiadBenchGSM8KHumanEvalMMLU-STEMARC-CMMLU-H
Base3.882.190.002.071.114.0410.828.3410.1410.94
LoRA4.453.440.003.631.397.1514.639.3510.6311.85
PA9.352.810.005.842.4518.4322.199.2513.9913.74
GrowRL9.632.810.005.842.2615.5722.269.7216.2115.16

Table 8. Transfer results for Qwen2.5-Math-7B → Qwen2.5-7B. The metric is Pass@1 (%), with 8 samples per problem.

MethodMATH500AMC2023AIME2024MinervaOlympiadBenchGSM8KHumanEvalMMLU-STEMARC-CMMLU-H
Base32.0820.313.758.8710.8948.5424.9236.4446.7628.66
LoRA34.7820.313.3310.0210.9251.9222.4141.6752.2232.49
PA38.3521.252.0812.0913.4855.4133.3148.4660.1937.43
GrowRL40.8820.943.7513.5614.2254.5238.6448.6761.7739.30

In addition, we directly transferred the parameter-matched PA (PA448) pathways added in W2 to the corresponding Base models and compared them with the original PA (PA192) and GrowRL. The results are shown in Table 9(a) and Table 9(b).

Table 9(a). Transfer results for Qwen2.5-Math-1.5B → Qwen2.5-1.5B. The metric is Pass@1, with 8 samples per problem.

DatasetPA192PA448GrowRL
MATH50010.2511.8322.63
AMC20233.135.7510.94
AIME20240.000.001.67
Minerva2.713.494.78
OlympiadBench3.153.587.07
GSM8K14.8322.2126.08
HumanEval8.089.5312.27
MMLU-STEM9.069.579.63
ARC-Challenge10.4710.4710.61
MMLU-Humanities8.567.787.31

Table 9(b). Transfer results for FineMath-Llama-3B → Llama-3.2-3B. The metric is Pass@1, with 8 samples per problem.

DatasetPA192PA448GrowRL
MATH5009.354.409.63
AMC20232.812.192.81
AIME20240.000.000.00
Minerva5.842.485.84
OlympiadBench2.451.322.26
GSM8K18.435.3815.57
HumanEval22.1919.2822.26
MMLU-STEM9.259.529.72
ARC-Challenge13.9915.6116.21
MMLU-Humanities13.7414.3315.16

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.