- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies cross-task transfer in reinforcement learning through Role-Based Reinforcement Learning (RBRL). The framework trains a shared language model with a primary reasoning role and auxiliary roles that operate on different views of the same underlying data. The auxiliary roles are used during training, while evaluation is performed using the primary reasoning prompt alone. The authors study three auxiliary roles: a Detector that identifies irrelevant clauses in arithmetic word problems, an Abstractor that generates general solution methods from related problems, and an Understander that extracts information useful for answering multiple questions about a passage. Experiments on GSM-NoOp and reading comprehension benchmarks show improvements over standard RLVR training. The paper also examines how the number of related instances used to construct auxiliary rewards affects transfer.
Critical strengths
The paper makes a useful effort to isolate transfer through shared model parameters. The primary reasoning policy does not rely on auxiliary outputs in its reasoning prompt, and the auxiliary roles are not needed at inference time. This makes the observed improvements interesting, as the benefits must be reflected in the shared model rather than coming from explicitly supplying auxiliary information at test time. The ablations on auxiliary views and reward construction are also valuable. In particular, the prompt-only and shuffled-reward controls help distinguish the effects of auxiliary reward optimization from simply introducing additional prompts or training objectives. The experiments varying the number of related instances provide further insight into when auxiliary training is beneficial.
Critical weaknesses
-
Limited methodological novelty and generality. My main concern is whether RBRL introduces a sufficiently substantial methodological contribution beyond existing auxiliary-task RL approaches. The experimental settings also leave some questions about generality. GSM-NoOp targets a specific failure mode involving irrelevant clauses, while the reading comprehension experiments rely on multiple questions associated with the same passage. Both settings provide readily identifiable relationships among training examples, which makes it relatively straightforward to construct the auxiliary views and rewards. It is less clear how RBRL would apply when these relationships are unavailable or difficult to identify, or when the primary task involves more complex, long-horizon reasoning. An experiment beyond these relatively structured settings would help establish the broader applicability of the framework.
-
Missing compute-matched comparisons. The paper emphasizes that the methods use the same number of trainable rollouts. However, the auxiliary roles require additional model generations for reward evaluation, and these generations are not included in the trainable rollout budget.
-
Limited evidence for the stability of the reported gains. The improvements on the reading comprehension benchmarks appear relatively small, averaging roughly 1–1.5 percentage points. This makes it important to understand how sensitive the results are to training randomness.
Questions for authors
-
Compute comparison: Could the authors report the total computational cost of RBRL, including the generations used for auxiliary reward evaluation?
-
Methodological contribution: What are the main advantages of the role-based formulation over conventional auxiliary-task RL? In particular, beyond organizing different auxiliary views and rewards, what new training capabilities or settings does RBRL enable? How should its contribution be distinguished from abstraction-based approaches such as RLAD?
-
Training stability: How consistent are the reported improvements across independent training seeds? Given the relatively small gains on reading comprehension, even a limited multi-seed evaluation would help assess their robustness.
4 comments
[W2, Q1] “Missing compute-matched comparisons …”
We have two controls for computation.
- The per step training budget is exactly the same 128 trainable generations across all models in the paper. The additional reward generations of each role are costed in Appendix I.
- We train until convergence on the training set. This means that increasing the number of training steps and training longer will not help the model. This can be seen in Appendix F.3, Figure 2.
Additionally, we ran tests on GSM-Symbolic with more rollouts and more prompts. In Table 9, we keep the 8 questions and increase the number of rollouts to 32, and we also ran new experiments with 32 questions and 4, 8 rollouts, and a reasoner shown the questions of 3 related problems from the same template, without answers and rewarded on its own problem only. We find that there is no gain. This shows that the performance of the reasoner role has saturated.
| Qwen3-4B-Base, GSM-NoOp with distractors | Unseen | Seen |
|---|---|---|
| Reasoner, 8 problems x 16 samples (Table 9) | 72.6 | 75.3 |
| Reasoner, 8 problems x 32 samples (Table 9) | 71.6 | 74.4 |
| Reasoner, 32 problems x 4 samples | 70.4 | 69.5 |
| Reasoner, 32 problems x 8 samples | 72.7 | 74.3 |
| Reasoner, shown 3 related problems of its template | 70.2 | 74.5 |
| + Abstractor | 78.7 | 80.4 |
These runs give a sense of how much the reasoner moves from run to run, and the abstractor's gain is well outside that range [W3, Q3].
In wall-clock time the two largest reasoner runs match the abstractor model. The 8 x 32 and 32 x 8 runs take 12.8 and 12.2 hours, against 13.0 for the abstractor model (Appendix I), and neither is more than a point above the reasoner, while the abstractor model is five to six points above it.
For the total cost, Appendix I gives the wall-clock time of every run on one H200, reward generations included. The detector trains in less time than the reasoner, the abstractor takes 1.2 to 1.3 times as long, and the understander 1.6 to 1.7 times.
[W3, Q3] “Limited evidence for the stability of the reported gains …”
We trained the reasoner and the understander with three seeds each. On average the understander does better, by 1.5 points at both sizes. Its seeds also vary less than the reasoner's (standard deviation 0.3 against 1.4 at 4B). The gain is larger on the five datasets the model never trains on, 1.7 points at 4B and 1.6 at 8B, against 0.8 on the training source.
| RACE-C | QuAIL | CosmosQA | LSAT-RC | LSAT-LR | QuALITY | Mean | |
|---|---|---|---|---|---|---|---|
| 4B Reasoner | 83.6 ± 0.5 | 75.1 ± 2.4 | 75.7 ± 3.0 | 73.0 ± 1.2 | 61.8 ± 2.1 | 71.8 ± 0.0 | 73.5 ± 1.4 |
| 4B + Understander | 84.4 ± 0.2 | 77.7 ± 0.3 | 78.4 ± 0.8 | 74.4 ± 0.4 | 63.3 ± 0.4 | 72.1 ± 0.4 | 75.0 ± 0.3 |
| 8B Reasoner | 85.7 ± 0.2 | 76.2 ± 0.8 | 77.1 ± 0.8 | 76.4 ± 1.0 | 66.1 ± 0.4 | 75.7 ± 0.2 | 76.2 ± 0.4 |
| 8B + Understander | 86.5 ± 0.5 | 78.8 ± 1.0 | 79.7 ± 0.9 | 78.0 ± 0.5 | 67.1 ± 0.3 | 76.0 ± 0.4 | 77.7 ± 0.3 |
Mean ± standard deviation over training seeds 123, 234 and 345, each seed at the argmax of its own development curve.
We will include all of these results in the final paper.
Gentle Reminder: Discussion Period Ending Soon
Dear Reviewer qHkA,
Thank you again for your feedback on our submission. As the discussion period is coming to a close, we wanted to check whether our responses have addressed your concerns. For [W1], we have stated our contributions explicitly and clarified them against other work. For [W2], we have added a thorough set of compute-matched experiments. For [W3], we have reported 3 training seeds for our reading-comprehension section, which strengthen our arguments.
We have also run experiments with RLAD and Llama-3.1-8B-Instruct that further support these points.
With the discussion period ending tomorrow, we would appreciate it if you could let us know whether these address your concerns, or whether there is anything further we can clarify.
Best regards, The Authors
Thanks for running the extra experiments. The new GSM-NoOp comparison addresses my compute concern for that setting. For the 4B models trained on RACE-C, the Understander raises the three-seed average across six test sets from 73.5% to 75.0%, with training time increasing from 4.1 to 6.6 hours. I’d still like to see how the Reasoner does with the same time budget. On TextCraft, success rises from 37.3% to 39.7% in a single-seed comparison, so it’s hard to tell how reliably that gain would hold. Concerning on the whole paper experiment and representation, I will maintain my score.
We thank the reviewer for their review and for finding our work interesting
[W1, Q2] “Limited methodological novelty and generality …”
Beyond the methodology, our work makes three contributions.
This is also how we differ from RLAD, whose solver reads the abstraction in training and at test time. We ran RLAD on our data as two LoRA adapters trained together, on GSM-NoOp problems with distractors (Qwen3-4B-Base).
Without its abstraction RLAD falls about seven points below the reasoner, so the solver has offloaded part of the skill to the abstraction model. With the abstraction RLAD gains one to four points over the reasoner, but it needs a second model and a second pass at test time. Our model is evaluated exactly like the reasoner and is still higher.
On generality, both datasets have a grouping we knew in advance, as our Future Works paragraph states, but reading comprehension has no targeted failure mode. The understander is built from the passage and its questions alone, and Llama-3.1-8B-Instruct reproduces the reading-comprehension result on all six datasets.
Additionally, we argue that each domain has its own role. In long-horizon reasoning a planning role fits naturally, and on TextCraft it raises the model's success from 37.3 to 39.7.
We also argue that the simplicity of the framework is a feature and not a flaw. RBRL allows introduction of multiple roles in diverse settings rather than confining itself to one setting and points towards dynamic roles defined by the data itself (meta-roles).
References
[RLAD] Qu, Yuxiao, et al. "RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems." International Conference on Learning Representations (ICLR), 2026.
[CTRL] Xie, Zhihui, et al. "Teaching Language Models to Critique via Reinforcement Learning." arXiv, 2025, arxiv.org/abs/2502.03492.
[MPO] Xiong, Weimin, et al. "MPO: Boosting LLM Agents with Meta Plan Optimization." Findings of the Association for Computational Linguistics: EMNLP 2025, 2025.
[s3] Jiang, Pengcheng, et al. "s3: You Don't Need That Much Data to Train a Search Agent via RL." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025.
[TextCraft] Prasad, Archiki, et al. "ADaPT: As-Needed Decomposition and Planning with Language Models." Findings of the Association for Computational Linguistics: NAACL 2024, 2024.