- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper targets the core limitations of existing Supervised Reinforcement Learning (SRL) for Small Language Models (SLMs) as interactive decision-making agents—neglect of reasoning quality and over-reliance on costly expert LLMs—and proposes the RISE training framework. On one side, it devises reasoning-aware SRL that delivers fine-grained supervision through reasoning-induced information gain rewards, separate rewards for reasoning and action, and adaptive supervision correction, accompanied by a proof of stationary convergence for the optimization objective. On the other side, it builds an expert-collaborative training paradigm with a Decision Maker and a lightweight Verifier, where expert LLMs are invoked only when decisions are unreliable, and the two-phase iterative training forms a virtuous cycle. Validated across multiple environments (ALFWorld, BabyAI) and multiple expert models, RISE matches or even surpasses the performance of expert LLMs.
Critical strengths
- The problem is clearly positioned, and the motivation experiments are persuasive.
- The framework is complete. RISE integrates reasoning-aware SRL, adaptive supervision correction, decision maker/verifier/expert collaboration, and trajectory-level transition memory into an end-to-end pipeline.
- The theoretical completeness is sound. The authors provide the weak convexity proof of the optimization objective, stationary convergence analysis under fixed snapshots, and also verify the convergence of the weighted SRL for the verifier.
- The experimental system is comprehensive with full evaluation dimensions: it covers multiple environments (ALFWorld, BabyAI), multiple expert models (GLM-5, Kimi-2.6), and in-distribution / out-of-distribution generalization tests; ablation studies verify the effectiveness of framework components and algorithm modules respectively; besides task performance, deployment-related metrics such as API cost, training time, and number of trajectories are also systematically evaluated.
Critical weaknesses
- The measurement of reasoning quality in the method still relies on expert supervision. The reasoning-induced information gain is computed based on expert actions, which essentially takes expert trajectories as ground truth and does not break free from the constraint of expert signals. If the expert actions themselves are suboptimal, the reward mechanism may be misleading.
- The granularity of ablation studies is insufficient. For reasoning-aware SRL, only the adaptive correction module is ablated, and the individual contributions of "reasoning-action separated reward" and "cross-segment consistency bonus" are not disentangled.
- The statistical rigor of the main experiments is insufficient. The main experiments do not report mean±std, confidence intervals, or paired tests. It is recommended to report the number of tasks and random seeds for each setting; if multiple training runs are conducted, mean±std should be reported. In particular, the comparisons of 74.29 vs 72.86 pp on ID (only 1.43 pp higher) and 68.66 vs 76.87 pp on OOD (8.21 pp lower) should be statistically tested.
Questions for authors
- How many independent training runs were performed for each method in the main experiments, and were statistical significance tests conducted for the performance gaps between RISE and the Expert LLM on both ID and OOD sets?
- The reasoning-aware SRL ablation only removes adaptive supervision correction. Have you ablated the separated reasoning/action reward and the consistency-aware advantage separately to quantify their individual contributions?
2 comments
Response (2/2)
Weakness 3: The statistical rigor of the main experiments is insufficient. The main experiments do not report mean±std, confidence intervals, or paired tests. It is recommended to report the number of tasks and random seeds for each setting; if multiple training runs are conducted, mean±std should be reported. In particular, the comparisons of 74.29 vs 72.86 pp on ID (only 1.43 pp higher) and 68.66 vs 76.87 pp on OOD (8.21 pp lower) should be statistically tested.
Question 1: How many independent training runs were performed for each method in the main experiments, and were statistical significance tests conducted for the performance gaps between RISE and the Expert LLM on both ID and OOD sets?
Response:
-
To provide stronger statistical support for the performance comparison, we have updated
Table 1to report the mean and standard deviation across 4 independent training runs, with each run using a different random seed. In each run, the results are collected on 140 ID tasks and 134 OOD tasks.-
In
Section 4.1, atlines 351-352, we have added the details regarding the number of tasks:For ALFWorld, we report results on 140 in-distribution (ID) and 134 out-of-distribution (OOD) tasks.
-
In
Section 4.2, atlines 401-403, we have added the setting of random seeds:Table 1 summarizes the means and standard deviations of the results collected under the methods in different categories. For each method, we conduct 4 independent training runs, with each run using a different random seed.
-
Table 1has been updated to below:Table 1: Comparison of task performance and training efficiency
Category Method Success Rate ID (pp) Success Rate OOD (pp) IBC ID (pp/USD) IBC OOD (pp/USD) Expert Budget of Training ($) # of Reasoning Traj. Training Time (h) Standalone Models Base SLM 4.05 ± 1.44 0.5 ± 1.20 -- -- -- -- -- Expert LLM 74.64 ± 1.85 73.67 ± 1.86 45.97 ± 3.17 51.20 ± 2.12 -- -- -- RL-based Methods GRPO 9.04 ± 1.21 8.45 ± 0.93 -- -- -- -- 107.30 ± 0.48 GiGPO 11.67 ± 1.22 11.01 ± 1.20 -- -- -- -- 74.14 ± 0.43 Supervision-based Methods SFT 24.08 ± 1.37 23.39 ± 2.82 -- -- 0.94 51.50 ± 3.35 1.28 ± 0.19 SFT+GRPO 23.20 ± 3.03 17.15 ± 2.15 -- -- 0.94 51.50 ± 3.35 61.28 ± 0.27 SRL 23.37 ± 1.62 22.38 ± 1.39 -- -- 0.94 51.50 ± 3.35 3.08 ± 0.20 RSRL 39.99 ± 1.53 34.12 ± 0.97 -- -- 0.94 51.50 ± 3.35 3.57 ± 0.20 ReLIFT 11.77 ± 1.56 8.97 ± 1.59 -- -- 0.94 51.50 ± 3.35 4.47 ± 0.24 Expert-Collaborative Methods Rand (k=8) 37.85 ± 2.10 39.55 ± 1.61 47.43 ± 3.78 52.74 ± 2.90 -- -- -- Rand (k=10) 46.42 ± 0.58 44.75 ± 1.19 49.15 ± 1.70 50.54 ± 2.95 -- -- -- RSRL+SR 69.04 ± 1.46 66.86 ± 1.26 59.01 ± 1.91 63.96 ± 1.71 0.94 51.50 ± 3.35 3.57 ± 0.20 STEER 63.78 ± 2.37 65.18 ± 2.82 42.67 ± 3.16 52.23 ± 2.40 -- -- -- RISE 73.38 ± 1.84 69.96 ± 0.81 118.34 ± 2.95 99.82 ± 2.25 0.94 111.33 ± 3.39 6.19 ± 0.17 -
-
We performed two-sided Welch’s t-tests with the Holm correction for the performance gaps between RISE and the Expert LLM on both ID and OOD sets, finding no statistically significant performance differences between RISE and the Expert LLM on either the ID or OOD test set. The detailed results are listed as below.
Task Sets RISE (%) Expert LLM (%) p-value Significant ID 73.39 ± 1.85 74.64 ± 1.85 0.4389 No OOD 69.97 ± 0.81 73.69 ± 1.86 0.0649 No
Response (1/2)
Thank you for the valuable feedback.
Response: We acknowledge that RISE uses expert trajectories as ground truth. However, RISE can be extended to stronger expert models, allowing it to benefit from continuous advances in model capabilities. Furthermore, as discussed in
Appendix F (Limitations), we plan to develop more adaptive and multi-expert collaboration strategies to better balance supervision quality and expert cost, thereby mitigating the risk of misleading supervision from suboptimal actions and trajectories provided by a single expert.Response: To increase the granularity of the ablation studies, we have added two additional variants of reasoning-aware SRL in
Fig 6. Specifically,RSRL-ASC-Cis constructed by further removing the cross-segment consistency bonus fromRSRL-ASC, retaining only the separate reward design and their corresponding group-wise advantage normalization.RSRL-ASC-C-Sis constructed by removing the separate reward design fromRSRL-ASC-C, where the reasoning-induced information gain reward and the action reward are simply summed before computing advantages through a single group-wise normalization.Fig. 6shows that the success rates on both ID and OOD followRSRL-ASC-C-S<SRL<RSRL-ASC-C<RSRL-ASC. The performance degradation ofRSRL-ASC-C-Srelative toSRLsuggests that naively combining the reasoning-induced information gain reward with the action reward may introduce reward conflicts and degrade performance. Meanwhile,RSRL-ASC-Creverses this degradation and surpasses SRL, suggesting that the separate reward design helps mitigate reward conflicts and performance degradation by assigning separately normalized advantages to the corresponding tokens, thereby providing fine-grained supervision for each segment. The results also show thatRSRL-ASCachieves further improvements overRSRL-ASC-C, indicating that the cross-segment consistency bonus contributes to strengthening the learning signals for correct actions and informative reasoning, thereby further improving learning efficiency. InSection 4.3, we have added the detailed analysis of the new ablation study results.