acceptodds
Under review as a conference paper at ICLR 2027

RISE: Reasoning-Induced Supervised Reinforcement Learning with Expert-Collaborative Training

Abstract

For the post-training of Small Language Models (SLMs), Reinforcement Learning (RL) often struggles with limited successful exploration, while Supervised Fine-Tuning (SFT) can lead to rigid imitation and poor generalization. Supervised Reinforcement Learning (SRL) bridges this gap by providing step-wise supervision within RL. Nevertheless, when applied to SLMs operating in multi-turn interactive environments, existing SRL methods remain impractical due to their neglect of reasoning quality in reward construction and heavy reliance on costly expert LLMs. We propose RISE, a Reasoning-Induced Supervised Reinforcement Learning framework with Expert-Collaborative Training. RISE introduces a reasoning-aware SRL approach that incorporates a reasoning-induced information gain reward and an adaptive supervision correction mechanism for fine-grained supervision. Moreover, RISE establishes an expert-collaborative training framework that jointly trains a Decision Maker for making primary decisions and a lightweight Verifier for action reliability assessment, enabling selective expert invocation only for unreliable decisions. Using two 1.5B-scale SLMs as the Decision Maker and Verifier, and pairing them with different Expert LLMs, we conduct extensive experiments on the ALFWorld and BabyAI environments, showing that RISE matches or surpasses Expert LLMs' task performance while reducing Expert LLM invocation cost by >50% during training and >33.33% during inference.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Language Model Agents

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_QD9fOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper targets the core limitations of existing Supervised Reinforcement Learning (SRL) for Small Language Models (SLMs) as interactive decision-making agents—neglect of reasoning quality and over-reliance on costly expert LLMs—and proposes the RISE training framework. On one side, it devises reasoning-aware SRL that delivers fine-grained supervision through reasoning-induced information gain rewards, separate rewards for reasoning and action, and adaptive supervision correction, accompanied by a proof of stationary convergence for the optimization objective. On the other side, it builds an expert-collaborative training paradigm with a Decision Maker and a lightweight Verifier, where expert LLMs are invoked only when decisions are unreliable, and the two-phase iterative training forms a virtuous cycle. Validated across multiple environments (ALFWorld, BabyAI) and multiple expert models, RISE matches or even surpasses the performance of expert LLMs.

Critical strengths
  1. The problem is clearly positioned, and the motivation experiments are persuasive.
  2. The framework is complete. RISE integrates reasoning-aware SRL, adaptive supervision correction, decision maker/verifier/expert collaboration, and trajectory-level transition memory into an end-to-end pipeline.
  3. The theoretical completeness is sound. The authors provide the weak convexity proof of the optimization objective, stationary convergence analysis under fixed snapshots, and also verify the convergence of the weighted SRL for the verifier.
  4. The experimental system is comprehensive with full evaluation dimensions: it covers multiple environments (ALFWorld, BabyAI), multiple expert models (GLM-5, Kimi-2.6), and in-distribution / out-of-distribution generalization tests; ablation studies verify the effectiveness of framework components and algorithm modules respectively; besides task performance, deployment-related metrics such as API cost, training time, and number of trajectories are also systematically evaluated.
Critical weaknesses
  1. The measurement of reasoning quality in the method still relies on expert supervision. The reasoning-induced information gain is computed based on expert actions, which essentially takes expert trajectories as ground truth and does not break free from the constraint of expert signals. If the expert actions themselves are suboptimal, the reward mechanism may be misleading.
  2. The granularity of ablation studies is insufficient. For reasoning-aware SRL, only the adaptive correction module is ablated, and the individual contributions of "reasoning-action separated reward" and "cross-segment consistency bonus" are not disentangled.
  3. The statistical rigor of the main experiments is insufficient. The main experiments do not report mean±std, confidence intervals, or paired tests. It is recommended to report the number of tasks and random seeds for each setting; if multiple training runs are conducted, mean±std should be reported. In particular, the comparisons of 74.29 vs 72.86 pp on ID (only 1.43 pp higher) and 68.66 vs 76.87 pp on OOD (8.21 pp lower) should be statistically tested.
Questions for authors
  1. How many independent training runs were performed for each method in the main experiments, and were statistical significance tests conducted for the performance gaps between RISE and the Expert LLM on both ID and OOD sets?
  2. The reasoning-aware SRL ablation only removes adaptive supervision correction. Have you ablated the separated reasoning/action reward and the consistency-aware advantage separately to quantify their individual contributions?
2 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for the valuable feedback.

Weakness 1: The measurement of reasoning quality in the method still relies on expert supervision. The reasoning-induced information gain is computed based on expert actions, which essentially takes expert trajectories as ground truth and does not break free from the constraint of expert signals. If the expert actions themselves are suboptimal, the reward mechanism may be misleading.

Response: We acknowledge that RISE uses expert trajectories as ground truth. However, RISE can be extended to stronger expert models, allowing it to benefit from continuous advances in model capabilities. Furthermore, as discussed in Appendix F (Limitations), we plan to develop more adaptive and multi-expert collaboration strategies to better balance supervision quality and expert cost, thereby mitigating the risk of misleading supervision from suboptimal actions and trajectories provided by a single expert.

Weakness 2: The granularity of ablation studies is insufficient. For reasoning-aware SRL, only the adaptive correction module is ablated, and the individual contributions of "reasoning-action separated reward" and "cross-segment consistency bonus" are not disentangled.

Question 2: The reasoning-aware SRL ablation only removes adaptive supervision correction. Have you ablated the separated reasoning/action reward and the consistency-aware advantage separately to quantify their individual contributions?

Response: To increase the granularity of the ablation studies, we have added two additional variants of reasoning-aware SRL in Fig 6. Specifically, RSRL-ASC-C is constructed by further removing the cross-segment consistency bonus from RSRL-ASC, retaining only the separate reward design and their corresponding group-wise advantage normalization. RSRL-ASC-C-S is constructed by removing the separate reward design from RSRL-ASC-C, where the reasoning-induced information gain reward and the action reward are simply summed before computing advantages through a single group-wise normalization.

Fig. 6 shows that the success rates on both ID and OOD follow RSRL-ASC-C-S<SRL<RSRL-ASC-C<RSRL-ASC. The performance degradation of RSRL-ASC-C-S relative to SRL suggests that naively combining the reasoning-induced information gain reward with the action reward may introduce reward conflicts and degrade performance. Meanwhile, RSRL-ASC-C reverses this degradation and surpasses SRL, suggesting that the separate reward design helps mitigate reward conflicts and performance degradation by assigning separately normalized advantages to the corresponding tokens, thereby providing fine-grained supervision for each segment. The results also show that RSRL-ASC achieves further improvements over RSRL-ASC-C, indicating that the cross-segment consistency bonus contributes to strengthening the learning signals for correct actions and informative reasoning, thereby further improving learning efficiency. In Section 4.3, we have added the detailed analysis of the new ablation study results.

AuthorsOpenReview ↗

Response (2/2)

Weakness 3: The statistical rigor of the main experiments is insufficient. The main experiments do not report mean±std, confidence intervals, or paired tests. It is recommended to report the number of tasks and random seeds for each setting; if multiple training runs are conducted, mean±std should be reported. In particular, the comparisons of 74.29 vs 72.86 pp on ID (only 1.43 pp higher) and 68.66 vs 76.87 pp on OOD (8.21 pp lower) should be statistically tested.

Question 1: How many independent training runs were performed for each method in the main experiments, and were statistical significance tests conducted for the performance gaps between RISE and the Expert LLM on both ID and OOD sets?

Response:

  • To provide stronger statistical support for the performance comparison, we have updated Table 1 to report the mean and standard deviation across 4 independent training runs, with each run using a different random seed. In each run, the results are collected on 140 ID tasks and 134 OOD tasks.

    • In Section 4.1, at lines 351-352, we have added the details regarding the number of tasks:

      For ALFWorld, we report results on 140 in-distribution (ID) and 134 out-of-distribution (OOD) tasks.

    • In Section 4.2, at lines 401-403, we have added the setting of random seeds:

      Table 1 summarizes the means and standard deviations of the results collected under the methods in different categories. For each method, we conduct 4 independent training runs, with each run using a different random seed.

    • Table 1 has been updated to below:

      Table 1: Comparison of task performance and training efficiency

    CategoryMethodSuccess Rate ID (pp)Success Rate OOD (pp)IBC ID (pp/USD)IBC OOD (pp/USD)Expert Budget of Training ($)# of Reasoning Traj.Training Time (h)
    Standalone ModelsBase SLM4.05 ± 1.440.5 ± 1.20----------
    Expert LLM74.64 ± 1.8573.67 ± 1.8645.97 ± 3.1751.20 ± 2.12------
    RL-based MethodsGRPO9.04 ± 1.218.45 ± 0.93--------107.30 ± 0.48
    GiGPO11.67 ± 1.2211.01 ± 1.20--------74.14 ± 0.43
    Supervision-based MethodsSFT24.08 ± 1.3723.39 ± 2.82----0.9451.50 ± 3.351.28 ± 0.19
    SFT+GRPO23.20 ± 3.0317.15 ± 2.15----0.9451.50 ± 3.3561.28 ± 0.27
    SRL23.37 ± 1.6222.38 ± 1.39----0.9451.50 ± 3.353.08 ± 0.20
    RSRL39.99 ± 1.5334.12 ± 0.97----0.9451.50 ± 3.353.57 ± 0.20
    ReLIFT11.77 ± 1.568.97 ± 1.59----0.9451.50 ± 3.354.47 ± 0.24
    Expert-Collaborative MethodsRand (k=8)37.85 ± 2.1039.55 ± 1.6147.43 ± 3.7852.74 ± 2.90------
    Rand (k=10)46.42 ± 0.5844.75 ± 1.1949.15 ± 1.7050.54 ± 2.95------
    RSRL+SR69.04 ± 1.4666.86 ± 1.2659.01 ± 1.9163.96 ± 1.710.9451.50 ± 3.353.57 ± 0.20
    STEER63.78 ± 2.3765.18 ± 2.8242.67 ± 3.1652.23 ± 2.40------
    RISE73.38 ± 1.8469.96 ± 0.81118.34 ± 2.9599.82 ± 2.250.94111.33 ± 3.396.19 ± 0.17
  • We performed two-sided Welch’s t-tests with the Holm correction for the performance gaps between RISE and the Expert LLM on both ID and OOD sets, finding no statistically significant performance differences between RISE and the Expert LLM on either the ID or OOD test set. The detailed results are listed as below.

    Task SetsRISE (%)Expert LLM (%)p-valueSignificant
    ID73.39 ± 1.8574.64 ± 1.850.4389No
    OOD69.97 ± 0.8173.69 ± 1.860.0649No
Reviewer_RQvjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies post-training of small language models (1.5B) as multi-turn agents, a regime where outcome-only RL stalls due to sparse success and SFT generalizes poorly. The proposed RISE framework combines (i) a reasoning-aware variant of supervised RL, in which a reasoning segment is rewarded by the information gain it provides toward the expert action under the old policy, together with a consistency-aware segment-wise advantage and a DPO-style correction term, and (ii) an expert-collaborative loop where a learned verifier (a second LoRA adapter on the same backbone) screens candidate actions and an expert LLM is queried only for rejected ones. On ALFWorld and BabyAI the trained 1.5B model approaches expert-level success while cutting expert API cost substantially.

Critical strengths

The motivating analysis in Sec. 2 is unusually well done for this kind of paper: the reasoning-swap, trajectory-truncation, and random-intervention experiments give direct quantitative support for both design choices, and are informative beyond the paper itself.

The information-gain reward is a principled idea and it is actually validated — the CDF analysis in the appendix and the quadrant breakdown in Fig. 9 show it discriminates good from bad reasoning in practice. Combined with the cost results (improvement-per-dollar roughly 2–3x the expert, and consistent across two experts and two environments), the empirical case is reasonably strong.

Critical weaknesses

The headline claim is overstated. The abstract says the method "matches or surpasses" the expert, but OOD success is 68.66 vs 76.87 for the expert, and the margin over STEER OOD (~1.5pp on ~134 tasks) is within noise. With a single run and no error bars or significance tests anywhere, these close rankings are not established.

The ablations also raise doubts about where the gains come from: standalone RSRL in Fig. 6 (OOD 70.9) actually exceeds the full RISE pipeline (68.66), and the SRL numbers in Fig. 6 do not match Table 1 without explanation. There is no "RSRL + simple routing" baseline, so the contribution of the collaborative framework itself is unclear, and the supervision-guided RL methods discussed in the related work are never compared against.

Questions for authors
  1. Can you report mean and std over at least 3 seeds, and explain the SRL discrepancy between Fig. 6 and Table 1? Was standalone RSRL given the same trajectory budget as in Table 1?
  2. What does the full RISE pipeline gain over standalone RSRL, or over RSRL combined with a simple confidence-based router? This seems necessary to separate the algorithmic contribution from the framework contribution.
  3. The verifier relies on trial execution with rollback. How would the framework transfer to environments where actions are irreversible (e.g. web/OS tasks), and what is the verifier's false-accept rate before and after training?
2 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for the valuable feedback.

Weakness 1: The headline claim is overstated. The abstract says the method "matches or surpasses" the expert, but OOD success is 68.66 vs 76.87 for the expert, and the margin over STEER OOD (~1.5pp on ~134 tasks) is within noise. With a single run and no error bars or significance tests anywhere, these close rankings are not established.

Question 1: Can you report mean and std over at least 3 seeds, and explain the SRL discrepancy between Fig. 6 and Table 1? Was standalone RSRL given the same trajectory budget as in Table 1?

Response: To avoid overstating the performance of our method, we have revised the corresponding statement "matches or surpasses Expert LLMs' task performance" to "approaches Expert LLMs' task performance" in the abstract and contributions.

Moreover, to provide stronger statistical support for the performance comparison, we have updated Table 1 to report the mean and standard deviation across 4 independent training runs, with each run using a different random seed.

  • In Section 4.2, at lines 401-403, we have added the setting of random seeds:

    Table 1 summarizes the means and standard deviations of the results collected under the methods in different categories. For each method, we conduct 4 independent training runs, with each run using a different random seed.

  • Table 1 has been updated to below:

    Table 1: Comparison of task performance and training efficiency

CategoryMethodSuccess Rate ID (pp)Success Rate OOD (pp)IBC ID (pp/USD)IBC OOD (pp/USD)Expert Budget of Training ($)# of Reasoning Traj.Training Time (h)
Standalone ModelsBase SLM4.05 ± 1.440.5 ± 1.20----------
Expert LLM74.64 ± 1.8573.67 ± 1.8645.97 ± 3.1751.20 ± 2.12------
RL-based MethodsGRPO9.04 ± 1.218.45 ± 0.93--------107.30 ± 0.48
GiGPO11.67 ± 1.2211.01 ± 1.20--------74.14 ± 0.43
Supervision-based MethodsSFT24.08 ± 1.3723.39 ± 2.82----0.9451.50 ± 3.351.28 ± 0.19
SFT+GRPO23.20 ± 3.0317.15 ± 2.15----0.9451.50 ± 3.3561.28 ± 0.27
SRL23.37 ± 1.6222.38 ± 1.39----0.9451.50 ± 3.353.08 ± 0.20
RSRL39.99 ± 1.5334.12 ± 0.97----0.9451.50 ± 3.353.57 ± 0.20
ReLIFT11.77 ± 1.568.97 ± 1.59----0.9451.50 ± 3.354.47 ± 0.24
Expert-Collaborative MethodsRand (k=8)37.85 ± 2.1039.55 ± 1.6147.43 ± 3.7852.74 ± 2.90------
Rand (k=10)46.42 ± 0.5844.75 ± 1.1949.15 ± 1.7050.54 ± 2.95------
RSRL+SR69.04 ± 1.4666.86 ± 1.2659.01 ± 1.9163.96 ± 1.710.9451.50 ± 3.353.57 ± 0.20
STEER63.78 ± 2.3765.18 ± 2.8242.67 ± 3.1652.23 ± 2.40------
RISE73.38 ± 1.8469.96 ± 0.81118.34 ± 2.9599.82 ± 2.250.94111.33 ± 3.396.19 ± 0.17

The SRL discrepancy in Table 1 and Fig 6 primarily stems from their differences in the number of training trajectories. Specifically, Table 1 compares methods under the same Expert budget of training, which can yield different numbers of reasoning trajectories across experiments. In contrast, Fig 6 aims to avoid the impact of varying expert guidance, so we provide all methods with the same 200 pre-collected expert reasoning trajectories. As a result, RSRL enjoys more benefit of reasoning in Fig 6 than in Table 1, and hence exhibits higher success rates, as illustrated in the updated Table 1.

To clarify the difference in settings, in Section 4.2, at lines 407-409, we have added below sentence:

For supervision-based methods, the reasoning trajectories used for training are collected entirely by the Expert LLM under the same training-stage expert budget (i.e., $0.94).

In Section 4.3, at lines 493-495, we have added below sentence:

Then, to avoid the impact of varying expert guidance, we preclude the SLM-LLM collaboration and instead provide the same 200 pre-collected expert reasoning trajectories for decision-making.

AuthorsOpenReview ↗

Response (2/2)

Weakness 2: The ablations also raise doubts about where the gains come from: standalone RSRL in Fig. 6 (OOD 70.9) actually exceeds the full RISE pipeline (68.66), and the SRL numbers in Fig. 6 do not match Table 1 without explanation. There is no "RSRL + simple routing" baseline, so the contribution of the collaborative framework itself is unclear, and the supervision-guided RL methods discussed in the related work are never compared against.

Question 2: What does the full RISE pipeline gain over standalone RSRL, or over RSRL combined with a simple confidence-based router? This seems necessary to separate the algorithmic contribution from the framework contribution.

Response: The explanation to the mismatch between RSRL and SRL results in Fig 6 and Table 1 is provided in the Response to your Weakness 1 and Question 1.

Then, to better separate the algorithmic contribution from that of the collaborative framework, we have added the results of standalone RSRL, RSRL+simple router (denoted as RSRL+SR) and a representative supervision-guided RL method, ReLIFT (disscussed in the related work) in the updated Table 1.

Specifically, the results show that RSRL achieves the highest success rate among the supervision-based methods. However, its performance remains limited by its complete reliance on expert-collected training trajectories and the lack of expert collaboration, as suggested by the superior performance of RISE. Furthermore, RSRL+SR benefits from the RSRL-trained model and expert collaboration enabled by a simple confidence-based router, achieving the second-best performance after RISE among the expert-collaborative methods. However, under the same budget, RSRL+SR still relies entirely on expert-collected trajectories during training, and confidence-based routing may mistakenly accept high-confidence but incorrect actions, both of which RISE can mitigate. In addition, ReLIFT exhibits the lowest performance among the supervision-based methods, which is caused by the conflict between the optimization objectives of SFT and GRPO, and the frequent alternation between them, which aligns with the observation in ReLIFT. In Section 4.2, we have accordingly updated the analysis of the new experimental results.

Question 3: The verifier relies on trial execution with rollback. How would the framework transfer to environments where actions are irreversible (e.g. web/OS tasks), and what is the verifier's false-accept rate before and after training?

Response: We concur that the Verifier relies on trial execution with rollback, which aims to improve its decision reliability and sampling efficiency. To extend the framework to environments where actions are irreversible (e.g., web/OS tasks), we plan to integrate RISE with a world model to predict next-state features that the Verifier can use to better assess candidate actions, as discussed in Appendix F (Limitations).

Then, to evaluate the Verifier's false-accept rate (FAR) before and after training, we conducted additional experiments that compare the base SLM with the trained Verifier using the same verification prompt, as reported in Appendix E.6. The results show that the Verifier training reduces FAR and improves decision accuracy (ACC), thereby enhancing the reliability of expert intervention decisions.

Specifically, we selected 50 held-out ALFWorld tasks and constructed 957 evaluation instances, comprising 319 reference-correct actions and 638 manually identified harmful alternatives. As shown in Fig. 17, the trained Verifier achieves a higher ACC and a lower FAR, which can be attributed to the construction of the verifier training instances and improvements in the verifier optimization introduced in RISE.

Reviewer_GZ4nOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper targets post-training of small language models (SLMs) as multi-turn interactive agents, where outcome-based RL suffers from failed exploration and supervised fine-tuning (SFT) encourages rigid imitation. It builds on Supervised Reinforcement Learning (SRL), which converts expert trajectories into step-level action supervision, and identifies two limitations of SRL for SLMs: its action-level reward ignores the quality of the reasoning that produced the action, and it requires costly expert LLM calls to obtain correct-action supervision. The proposed framework, RISE, has two parts. First, a reasoning-aware SRL objective computes a reasoning-induced information gain, defined as the length-normalized log-likelihood of the supervision action conditioned on a sampled reasoning trace minus the same quantity conditioned on empty reasoning, and uses its positive part as a separate reasoning reward. Reasoning and action rewards are normalized separately within each group and converted into segment-wise advantages, with a consistency term that shares credit when both are above the group average. An adaptive supervision correction adds an auxiliary preference loss between the supervision action and the best sampled action when no rollout exactly matches the supervision action and the highest-information-gain reasoning is positive. Second, an expert-collaborative training loop alternates between interaction and optimization: a Decision Maker (SLM), a Verifier (a second LoRA adapter on the same frozen SLM), and an Expert LLM interact, where the Verifier accepts or rejects each candidate action and the Expert is invoked only on rejected steps; successful trajectories are then decomposed into policy-training and verifier-training instances. Experiments use Qwen2.5-1.5B-Instruct as the shared backbone, GLM-5 as the main expert, ALFWorld as the main environment, and report success rate, incremental benefit per cost (IBC), expert API cost, trajectory count, and training time. RISE reaches 74.29 ID and 68.66 OOD success rate against 72.86 and 76.87 for the standalone expert, with IBC of 120.68 and 99.86 against 43.12 and 51.08 for the expert, and collects 112 reasoning trajectories under the same $0.94 expert budget that yields 50 trajectories for the supervised baselines. The paper additionally reports ablations of both components, a second expert (Kimi-2.6), a second environment (BabyAI), and a weak-convexity-based stationary convergence analysis.

Critical strengths
  1. The motivating experiment isolates reasoning from action and is unusually persuasive.
  2. The reward construction addresses a concrete scale and credit-assignment problem rather than adding an unmotivated term.
  3. The submission is reproducible by conference standards. Code is linked, the full hyperparameter table is given, the interaction and verifier prompts are printed, and the AI-use and reproducibility statements are specific.
Critical weaknesses
  1. The headline performance comparison is not supported at the level of statistical resolution the claim requires, and it is overstated for OOD. The abstract and contributions state that RISE "matches or surpasses Expert LLMs' task performance", but Table 1 shows 74.29 versus 72.86 on ID — a 1.43-point difference with no reported seed count, standard deviation, or confidence interval — and 68.66 versus 76.87 on OOD, where RISE is clearly worse by 8.21 points (Table 1, p. 8; Introduction and contributions, p. 1; lines 86–89). With one run per method, the ID difference is well within plausible run-to-run variation for a 30-step embodied task, so the defensible claim is the IBC result, not parity of success rate. This is fixable with multiple seeds and paired intervals, and the paper already evaluates both splits, so the cost is tabulation rather than new method development.

  2. The Verifier's supervision signal is never defined precisely enough to rule out self-training on its own outputs. Section 3.3.1 says that the Verifier's assessments along successful trajectories are "retained as supervision for its subsequent training", and Section 3.3.2 says it is optimized with standard SRL on verifier-training instances (Page 6, §3.3.1; lines 306–333). Standard SRL requires a supervision action for each step; the paper does not state whether the label is the Verifier's own yes/no judgment, the Expert's implicit judgment when it overrides a rejection, or whether the executed action ultimately contributed to a successful episode. These are very different objectives: the first is self-training on the model's own labels, the second is imitation of the Expert's intervention policy, and the third is a delayed outcome label. The ambiguity matters because the Verifier is responsible for the paper's largest ablation effect — 60.71 to 74.29 ID points when it is trained (Table 2, p. 9) — and because the reliability claim in the abstract depends on it. This is a fundamental clarity issue for the central mechanism, though likely fixable by stating the label construction and reporting its agreement with Expert interventions.

  3. The reasoning-quality contribution is narrower than the paper presents it. The reasoning-induced information gain in Eq. (4) is the same quantity used by information-gain-based policy optimization, which the paper cites as Wang et al. (2026) and uses as the justification for the metric (Page 4, §3.2.1; lines 189–203; references at line 589). The remaining novelty is the segment-level use of that quantity with separate group normalization and consistency-aware advantages, layered on standard SRL, plus a Verifier trained on top of a frozen SLM in the style of confidence-guided routing (STEER) and rollback frameworks. Removing the naming, the paper is a well-engineered composition of SRL, information-gain rewards, LoRA-adapter routing, and a learned verifier, rather than a new mechanism. This is not fatal — the composition is non-trivial and the empirical gains are real — but it should temper the contribution claim, and the paper should state precisely what Eq. (4)–(5) add beyond IGPO plus SRL (pp. 4–5, §3.2; lines 189–238).

  4. The training-cost accounting is not detailed enough to verify the central efficiency claim. Table 1 reports a training-stage expert API cost of $0.94 for RISE and for the supervised baselines, but RISE runs eight rounds of interaction and optimization, each round invoking the Expert on rejected steps, and the same budget is said to yield 112 trajectories for RISE against 50 for the baselines (Table 1, p. 8; lines 429–433; Table 3, p. 21). A reader cannot reconstruct what "the same budget" controls: whether it is enforced per round or over the whole run, whether cached Expert responses are reused, and whether Verifier training or the 8 rounds of Decision Maker updates are included. Since the efficiency claim is the paper's strongest result, a cost breakdown by round and by component is needed. This is fixable with a table, not a new experiment.

Questions for authors
  1. What exactly is the supervision label for the Verifier? For each verifier-training instance, is the target the Verifier's own assessment collected during interaction, the Expert's action on the same step, or the eventual success of the trajectory? If it is the Verifier's own assessment, how is the resulting self-training prevented from reinforcing its initial errors, and what is the agreement rate between Verifier rejections and Expert interventions?
  2. Why does standard SRL underperform SFT in your reproduction (Table 1, 21.43 versus 25.00 ID)? If SRL is weaker than the simpler baseline under the same data budget, then the RSRL-versus-SRL gap in Figure 6 may partly reflect a weak SRL configuration rather than a specific benefit of reasoning-aware rewards. Can you report the SRL hyperparameters and confirm they were tuned comparably to RSRL?
  3. Is the information-gain reward in Eq. (4) computed against the supervision action only, and can it be gamed? A reasoning trace that increases the likelihood of the supervision action receives positive reward even if it is semantically irrelevant. Did you observe degenerate reasoning that maximizes this quantity without improving task behavior, and does the BabyAI result show the same reasoning-quality effect?
4 comments
AuthorsOpenReview ↗

Response (1/4)

Thank you for the valuable feedback.

Weakness 1: The headline performance comparison is not supported at the level of statistical resolution the claim requires, and it is overstated for OOD. The abstract and contributions state that RISE "matches or surpasses Expert LLMs' task performance", but Table 1 shows 74.29 versus 72.86 on ID — a 1.43-point difference with no reported seed count, standard deviation, or confidence interval — and 68.66 versus 76.87 on OOD, where RISE is clearly worse by 8.21 points (Table 1, p. 8; Introduction and contributions, p. 1; lines 86–89). With one run per method, the ID difference is well within plausible run-to-run variation for a 30-step embodied task, so the defensible claim is the IBC result, not parity of success rate. This is fixable with multiple seeds and paired intervals, and the paper already evaluates both splits, so the cost is tabulation rather than new method development.

Response: To avoid overstating the performance of our method, we have revised the corresponding statement "matches or surpasses Expert LLMs' task performance" to "approaches Expert LLMs' task performance" in the abstract and contributions.

Moreover, to provide stronger statistical support for the performance comparison, we have updated Table 1 to report the mean and standard deviation across 4 independent training runs, with each run using a different random seed.

  • In Section 4.2, at lines 401-403, we have added the setting of random seeds:

    Table 1 summarizes the means and standard deviations of the results collected under the methods in different categories. For each method, we conduct 4 independent training runs, with each run using a different random seed.

  • Table 1 has been updated to below:

    Table 1: Comparison of task performance and training efficiency

CategoryMethodSuccess Rate ID (pp)Success Rate OOD (pp)IBC ID (pp/USD)IBC OOD (pp/USD)Expert Budget of Training ($)# of Reasoning Traj.Training Time (h)
Standalone ModelsBase SLM4.05 ± 1.440.5 ± 1.20----------
Expert LLM74.64 ± 1.8573.67 ± 1.8645.97 ± 3.1751.20 ± 2.12------
RL-based MethodsGRPO9.04 ± 1.218.45 ± 0.93--------107.30 ± 0.48
GiGPO11.67 ± 1.2211.01 ± 1.20--------74.14 ± 0.43
Supervision-based MethodsSFT24.08 ± 1.3723.39 ± 2.82----0.9451.50 ± 3.351.28 ± 0.19
SFT+GRPO23.20 ± 3.0317.15 ± 2.15----0.9451.50 ± 3.3561.28 ± 0.27
SRL23.37 ± 1.6222.38 ± 1.39----0.9451.50 ± 3.353.08 ± 0.20
RSRL39.99 ± 1.5334.12 ± 0.97----0.9451.50 ± 3.353.57 ± 0.20
ReLIFT11.77 ± 1.568.97 ± 1.59----0.9451.50 ± 3.354.47 ± 0.24
Expert-Collaborative MethodsRand (k=8)37.85 ± 2.1039.55 ± 1.6147.43 ± 3.7852.74 ± 2.90------
Rand (k=10)46.42 ± 0.5844.75 ± 1.1949.15 ± 1.7050.54 ± 2.95------
RSRL+SR69.04 ± 1.4666.86 ± 1.2659.01 ± 1.9163.96 ± 1.710.9451.50 ± 3.353.57 ± 0.20
STEER63.78 ± 2.3765.18 ± 2.8242.67 ± 3.1652.23 ± 2.40------
RISE73.38 ± 1.8469.96 ± 0.81118.34 ± 2.9599.82 ± 2.250.94111.33 ± 3.396.19 ± 0.17
AuthorsOpenReview ↗

Response (2/4)

Weakness 2: The Verifier's supervision signal is never defined precisely enough to rule out self-training on its own outputs. Section 3.3.1 says that the Verifier's assessments along successful trajectories are "retained as supervision for its subsequent training", and Section 3.3.2 says it is optimized with standard SRL on verifier-training instances (Page 6, §3.3.1; lines 306–333). Standard SRL requires a supervision action for each step; the paper does not state whether the label is the Verifier's own yes/no judgment, the Expert's implicit judgment when it overrides a rejection, or whether the executed action ultimately contributed to a successful episode. These are very different objectives: the first is self-training on the model's own labels, the second is imitation of the Expert's intervention policy, and the third is a delayed outcome label. The ambiguity matters because the Verifier is responsible for the paper's largest ablation effect — 60.71 to 74.29 ID points when it is trained (Table 2, p. 9) — and because the reliability claim in the abstract depends on it. This is a fundamental clarity issue for the central mechanism, though likely fixable by stating the label construction and reporting its agreement with Expert interventions.

Question 1: What exactly is the supervision label for the Verifier? For each verifier-training instance, is the target the Verifier's own assessment collected during interaction, the Expert's action on the same step, or the eventual success of the trajectory? If it is the Verifier's own assessment, how is the resulting self-training prevented from reinforcing its initial errors, and what is the agreement rate between Verifier rejections and Expert interventions?

Response:

  • The Verifier's supervision signal (label) is its own yes/no judgment. The prevention of reinforcing initial errors in self-training is implemented as below:

    1. During interaction, the Verifier judges yes or no until task success or the Decision Maker reaches the maximum number of environment steps.
    2. The Verifier will retain the yes/no judgements (for later supervision) only when the trajectory successfully finishes the task.
    3. The retained judgement labels will be saved as Verifier-training Instances and used for the self-training of the Verifier.

    For clarification, in Section 3.3.1, at lines 310-313, we have added more details:

    Meanwhile, πδ\pi_\delta first judges \texttt{yes} or \texttt{no} during interaction until task success or it reaches the maximum number of environment steps, and then retains the \texttt{yes}/\texttt{no} judgements only when the task is executed successfully. Therefore, the instances for the self-training of πδ\pi_\delta do not contain erroneous labels.

  • Since the instances for supervising the Verifier do not contain erroneous labels, the Expert will intervene with 100% agreement rate in the decision-making process whenever the Verifier rejects the Decision Maker's proposed action.

AuthorsOpenReview ↗

Response (3/4)

Weakness 3: The reasoning-quality contribution is narrower than the paper presents it. The reasoning-induced information gain in Eq. (4) is the same quantity used by information-gain-based policy optimization, which the paper cites as Wang et al. (2026) and uses as the justification for the metric (Page 4, §3.2.1; lines 189–203; references at line 589). The remaining novelty is the segment-level use of that quantity with separate group normalization and consistency-aware advantages, layered on standard SRL, plus a Verifier trained on top of a frozen SLM in the style of confidence-guided routing (STEER) and rollback frameworks. Removing the naming, the paper is a well-engineered composition of SRL, information-gain rewards, LoRA-adapter routing, and a learned verifier, rather than a new mechanism. This is not fatal — the composition is non-trivial and the empirical gains are real — but it should temper the contribution claim, and the paper should state precisely what Eq. (4)–(5) add beyond IGPO plus SRL (pp. 4–5, §3.2; lines 189–238).

Response: To temper the claim of our reasoning-quality contribution and more precisely state the benefits brought by Eqs. (4)-(5), we have made the following revisions to the manuscript:

  1. In Section 1, at lines 76-79, we have modified the contribution claim to:

    We propose an SRL approach with a segment-level reasoning-induced information gain reward mechanism, which separately evaluates the contributions of reasoning and action to action correctness, and accordingly tailors fine-grained supervision.

  2. In Section 3.2.2, at lines 228-232, we have added the following sentence to precisely explain the benefits brought by Eqs. (4)-(5) over IGPO plus SRL:

    Compared with IGPO (Wang et al., 2026) plus SRL (Deng et al., 2026), the incorporation of the segment-level reasoning-induced information gain rewards (Eq. (4)), group-wise advantage normalization, and consistency-aware advantages (Eq. (5)) can more effectively promote beneficial reasoning and correct actions separately, thereby improving learning efficiency in multi-step interactive environments.

Weakness 4: The training-cost accounting is not detailed enough to verify the central efficiency claim. Table 1 reports a training-stage expert API cost of $0.94 for RISE and for the supervised baselines, but RISE runs eight rounds of interaction and optimization, each round invoking the Expert on rejected steps, and the same budget is said to yield 112 trajectories for RISE against 50 for the baselines (Table 1, p. 8; lines 429–433; Table 3, p. 21). A reader cannot reconstruct what "the same budget" controls: whether it is enforced per round or over the whole run, whether cached Expert responses are reused, and whether Verifier training or the 8 rounds of Decision Maker updates are included. Since the efficiency claim is the paper's strongest result, a cost breakdown by round and by component is needed. This is fixable with a table, not a new experiment.

Response: We clarify that the budget corresponds to the cumulative Expert API cost exclusively, and it covers the whole 8 rounds. The expense incurred by the other components, such as cached Expert responses, Verifier training or Decision Maker training, are not included in this budget.

To verify the central efficiency claim, we break down the training cost of RISE by round as below:

Round12345678Sum
RISE0.200.170.160.120.080.110.050.040.93

We can notice that the sum of the training costs is lower than the predefined budget ($0.94). This is because upon the exhaustion of the budget, some trajectory sampling attempts are unsuccessful, and hence do not incur Expert API cost. Despite some fluctuations, the expert API cost incurred by the collection of reasoning trajectories for the training of the Decision Maker and the Verifier generally decreases across training rounds. This can be attributed to the progressive improvement in the Decision Maker's performance and the reliability of the Verifier's assessments, which jointly reduce the reliance on expert assistance for successful trajectory collection. The table of results and the detailed analysis have been added in Appendix E.2.

AuthorsOpenReview ↗

Response (4/4)

Question 2: Why does standard SRL underperform SFT in your reproduction (Table 1, 21.43 versus 25.00 ID)? If SRL is weaker than the simpler baseline under the same data budget, then the RSRL-versus-SRL gap in Figure 6 may partly reflect a weak SRL configuration rather than a specific benefit of reasoning-aware rewards. Can you report the SRL hyperparameters and confirm they were tuned comparably to RSRL?

Response: To ensure a fair comparison, the hyperparameters of SRL were tuned comparably to those of RSRL, as detailed in Appendix D. Based on the updated Table 1, we observe that although standard SRL still performs slightly worse than SFT on average across multiple seeds, the difference is marginal, indicating that SRL is not significantly weaker than the simpler baseline (SFT).

However, the data budgets of the methods in Table 1 and Fig 6 are indeed different, which stems from their differences in the number of training trajectories. Specifically, Table 1 compares methods under the same Expert budget of training, which can yield different numbers of reasoning trajectories across experiments. In contrast, Fig 6 aims to avoid the impact of varying expert guidance, so we provide all methods with the same 200 pre-collected expert reasoning trajectories.

Under these settings, we find that the performance of SRL increases from 23.37 (ID) and 22.38 (OOD) in Table 1 to 27.9 (ID) and 26.1 (OOD) in Fig 6, while the performance of RSRL increases much more from 39.99 (ID) and 34.12 (OOD) in Table 1 to 58.6 (ID) and 70.9 (OOD) in Fig 6, indicating that RSRL can indeed benefit more via reasoning-aware rewards.

To clarify the difference in settings, in Section 4.2, at lines 407-409, we have added below sentence:

For supervision-based methods, the reasoning trajectories used for training are collected entirely by the Expert LLM under the same training-stage expert budget (i.e., $0.94).

In Section 4.3, at lines 493-495, we have added below sentence:

Then, to avoid the impact of varying expert guidance, we preclude the SLM-LLM collaboration and instead provide the same 200 pre-collected expert reasoning trajectories for decision-making.

Question 3: Is the information-gain reward in Eq. (4) computed against the supervision action only, and can it be gamed? A reasoning trace that increases the likelihood of the supervision action receives positive reward even if it is semantically irrelevant. Did you observe degenerate reasoning that maximizes this quantity without improving task behavior, and does the BabyAI result show the same reasoning-quality effect?

Response: The information-gain (IG) reward in Eq. (4) is indeed computed against the supervision action only, and does not directly evaluate the semantic validity of the generated reasoning. Therefore, it is possible for irrelevant or incorrect reasoning to receive positive IG (e.g., as shown in Fig. 8, even random or incorrect reasoning may result in positive IG). Through conducting an additional analysis in Appendix E.5 following our BabyAI experiments, we observe that degenerate reasoning with positive IG also exists in the BabyAI environment. However, the design of RISE helps encourage the correct reasoning with high IG, and thereby gradually suppress the adverse effects of degenerate reasoning.

Specifically, in the BabyAI environment, we collected valid rollouts generated during Decision Maker optimization across training rounds and examined their IG values, which are illustrated in Fig 15. We then selected the top 20% of rollouts ranked by IG in each round and evaluated whether they are semantically relevant by manual assessment. As shown in Fig. 16, some high-IG rollouts indeed contain semantically irrelevant reasoning, particularly in the early training rounds. With the progress of training, the proportion of such rollouts generally decreases in spite of some fluctuations. This is primarily because the combination of the consistency-aware advantages (Section 3.2.2) and the adaptive supervision correction (Section 3.2.3) keeps favoring correct reasoning with high IG, thereby suppressing the degenerate reasoning with positive IG.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Reply

We sincerely thank all the reviewers for their effort, feedback, and comments. We are pleased that the paper has been recognized for its insightful motivation experiments (GZ4n, RQvj, QD9f), reasoning-aware reward design for fine-grained credit assignment (GZ4n, RQvj, QD9f), and a favorable performance-cost trade-off, supported by extensive experimental evaluation (RQvj, QD9f).

We invite each reviewer to consult our individual responses, where we address all points raised in detail. In this general reply, we provide a summary of the primary changes made in light of the feedback:

  • Evaluation of statistical reliability (GZ4n, RQvj, QD9f)

    We updated Table 1 to report the mean and standard deviation across 4 random seeds on both ID and OOD tasks, and performed two-sided Welch's t-tests with Holm correction to evaluate the statistical significance of performance differences between RISE and the Expert LLM (Section 4.2).

  • Clarification of contribution and novelty (GZ4n, RQvj)

    We refined our performance claim from "matches or surpasses" to "approaches" Expert LLMs and clarified the benefits of reasoning-aware SRL over IGPO plus SRL (Abstract, Section 1, Section 3.2).

  • Clarification of Verifier supervision and reliability (GZ4n, RQvj)

    We clarified the Verifier's supervision signal and then demonstrated through additional experiments that Verifier training achieves a lower false-accept rate, thereby improving its reliability (Section 3.3.1, Appendix E.6).

  • Clarification of experimental results (GZ4n, RQvj)

    We clarified that the discrepancy between Table 1 and Fig. 6 primarily stems from differences in the number of training trajectories and explained why they are set differently (Section 4.2, Section 4.3).

  • Additional baseline comparisons and ablation studies (RQvj, QD9f)

    We added comparisons with standalone RSRL, RSRL+SR, and ReLIFT, along with two ablation variants to evaluate the separate reward design and cross-segment consistency bonus (Section 4.2, Section 4.3).

  • Analysis of degenerate reasoning (GZ4n)

    We added an analysis on BabyAI showing that degenerate reasoning with positive information gain exists but generally decreases during training, and clarified how RISE mitigates this issue (Appendix E.5).

  • Clarification of training cost (GZ4n)

    We clarified that the training budget corresponds to the cumulative Expert API cost across all 8 rounds and provided a breakdown of the expert API cost by round (Appendix E.2).

  • Limitations and future extensions (RQvj, QD9f)

    We further discussed integrating RISE with a world model for environments where actions are irreversible, as well as adopting multi-expert collaboration strategies to mitigate the risk of misleading supervision from suboptimal expert actions (Appendix F).

Given the extensive additional experiments and the detailed methodological clarifications for addressing the reviewers’ questions, we have substantially revised the manuscript and highlighted the changes in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.