acceptodds
Under review as a conference paper at ICLR 2027

Cross-Task Transfer via Multi-Role Reinforcement Learning

Abstract

Reinforcement learning with verifiable rewards is often used to train a language model for a single role that includes a problem-solving task and an objective based on answer correctness. Humans often learn differently; they decompose a problem or draw on relations that span examples, pursuing goals that are not directly tied to the correct answer to any single problem. Inspired by this, we ask whether training a language model on the same examples with auxiliary roles, alongside the primary one, can improve its test performance. We define each role by a task, what the model is shown and asked for, and an objective, how its output is rewarded. We formalize this setting as Role-Based Reinforcement Learning (RBRL), in which one model is trained in several roles. To evaluate our approach, we use GSM-Symbolic with a single distractor clause, which is known to substantially reduce LLM accuracy, and define two auxiliary roles: one that identifies the distractor clause within a problem and another that writes a method to solve every problem sharing a template. When the model is trained in these multiple roles, where the auxiliary roles include distractors but the primary role does not, the resulting model solves more problems with distractors at test time than when trained in the primary role alone. We interpret this gain in terms of cross-task transfer. Specifically, the training in the auxiliary tasks improves model performance on the primary task at test time. When distractors are also added to the primary role training, the model improves further, suggesting that the benefits of each role may be additive. We also observe the same additive pattern on reading comprehension tasks. An auxiliary role that states what a passage implies without actually asking any questions improves question answering on RACE-C and on five other datasets we did not train the model on.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning with verifiable rewards, auxiliary objectives, cross-objective transfer, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qHkAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies cross-task transfer in reinforcement learning through Role-Based Reinforcement Learning (RBRL). The framework trains a shared language model with a primary reasoning role and auxiliary roles that operate on different views of the same underlying data. The auxiliary roles are used during training, while evaluation is performed using the primary reasoning prompt alone. The authors study three auxiliary roles: a Detector that identifies irrelevant clauses in arithmetic word problems, an Abstractor that generates general solution methods from related problems, and an Understander that extracts information useful for answering multiple questions about a passage. Experiments on GSM-NoOp and reading comprehension benchmarks show improvements over standard RLVR training. The paper also examines how the number of related instances used to construct auxiliary rewards affects transfer.

Critical strengths

The paper makes a useful effort to isolate transfer through shared model parameters. The primary reasoning policy does not rely on auxiliary outputs in its reasoning prompt, and the auxiliary roles are not needed at inference time. This makes the observed improvements interesting, as the benefits must be reflected in the shared model rather than coming from explicitly supplying auxiliary information at test time. The ablations on auxiliary views and reward construction are also valuable. In particular, the prompt-only and shuffled-reward controls help distinguish the effects of auxiliary reward optimization from simply introducing additional prompts or training objectives. The experiments varying the number of related instances provide further insight into when auxiliary training is beneficial.

Critical weaknesses
  1. Limited methodological novelty and generality. My main concern is whether RBRL introduces a sufficiently substantial methodological contribution beyond existing auxiliary-task RL approaches. The experimental settings also leave some questions about generality. GSM-NoOp targets a specific failure mode involving irrelevant clauses, while the reading comprehension experiments rely on multiple questions associated with the same passage. Both settings provide readily identifiable relationships among training examples, which makes it relatively straightforward to construct the auxiliary views and rewards. It is less clear how RBRL would apply when these relationships are unavailable or difficult to identify, or when the primary task involves more complex, long-horizon reasoning. An experiment beyond these relatively structured settings would help establish the broader applicability of the framework.

  2. Missing compute-matched comparisons. The paper emphasizes that the methods use the same number of trainable rollouts. However, the auxiliary roles require additional model generations for reward evaluation, and these generations are not included in the trainable rollout budget.

  3. Limited evidence for the stability of the reported gains. The improvements on the reading comprehension benchmarks appear relatively small, averaging roughly 1–1.5 percentage points. This makes it important to understand how sensitive the results are to training randomness.

Questions for authors
  1. Compute comparison: Could the authors report the total computational cost of RBRL, including the generations used for auxiliary reward evaluation?

  2. Methodological contribution: What are the main advantages of the role-based formulation over conventional auxiliary-task RL? In particular, beyond organizing different auxiliary views and rewards, what new training capabilities or settings does RBRL enable? How should its contribution be distinguished from abstraction-based approaches such as RLAD?

  3. Training stability: How consistent are the reported improvements across independent training seeds? Given the relatively small gains on reading comprehension, even a limited multi-seed evaluation would help assess their robustness.

4 comments
AuthorsOpenReview ↗

We thank the reviewer for their review and for finding our work interesting

[W1, Q2] “Limited methodological novelty and generality …”

Beyond the methodology, our work makes three contributions.

  1. We show that cross-task transfer exists in LLMs through disjoint roles and define two properties of auxiliary roles, substitutivity and additivity. Additionally, we show this transfer can lead to better generalization on unseen datasets.
  2. We introduce the view and show that grouping related problems in one view is what lets a role extract signal that training on one problem at a time misses. As far as we know, work that trains an auxiliary role keeps its view fixed.
  3. We show that roles can be defined per dataset. Humans do not learn every subject the same way, so we argue for moving away from static roles, and other papers point the same way, each adding a role for its own domain, an abstraction generator (RLAD), a critic (CTRL), a planner (MPO) or a searcher (s3), played by a second model kept at test time. RBRL unifies these as disjoint roles of one model that are dropped at test time.

This is also how we differ from RLAD, whose solver reads the abstraction in training and at test time. We ran RLAD on our data as two LoRA adapters trained together, on GSM-NoOp problems with distractors (Qwen3-4B-Base).

UnseenSeen
Reasoner72.675.3
RLAD, solver alone66.167.6
RLAD, abstraction then solve75.776.6
Ours (abstractor model)78.780.4

Without its abstraction RLAD falls about seven points below the reasoner, so the solver has offloaded part of the skill to the abstraction model. With the abstraction RLAD gains one to four points over the reasoner, but it needs a second model and a second pass at test time. Our model is evaluated exactly like the reasoner and is still higher.

On generality, both datasets have a grouping we knew in advance, as our Future Works paragraph states, but reading comprehension has no targeted failure mode. The understander is built from the passage and its questions alone, and Llama-3.1-8B-Instruct reproduces the reading-comprehension result on all six datasets.

Llama-3.1-8B-InstructRACE-CQuAILCosmosQALSAT-RCLSAT-LRQuALITYMean
Reasoner80.874.281.870.453.674.472.5
+ Understander81.975.083.370.754.475.173.4

Additionally, we argue that each domain has its own role. In long-horizon reasoning a planning role fits naturally, and on TextCraft it raises the model's success from 37.3 to 39.7.

We also argue that the simplicity of the framework is a feature and not a flaw. RBRL allows introduction of multiple roles in diverse settings rather than confining itself to one setting and points towards dynamic roles defined by the data itself (meta-roles).

References

[RLAD] Qu, Yuxiao, et al. "RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems." International Conference on Learning Representations (ICLR), 2026.

[CTRL] Xie, Zhihui, et al. "Teaching Language Models to Critique via Reinforcement Learning." arXiv, 2025, arxiv.org/abs/2502.03492.

[MPO] Xiong, Weimin, et al. "MPO: Boosting LLM Agents with Meta Plan Optimization." Findings of the Association for Computational Linguistics: EMNLP 2025, 2025.

[s3] Jiang, Pengcheng, et al. "s3: You Don't Need That Much Data to Train a Search Agent via RL." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025.

[TextCraft] Prasad, Archiki, et al. "ADaPT: As-Needed Decomposition and Planning with Language Models." Findings of the Association for Computational Linguistics: NAACL 2024, 2024.

[W2, Q1] “Missing compute-matched comparisons …”

We have two controls for computation.

  1. The per step training budget is exactly the same 128 trainable generations across all models in the paper. The additional reward generations of each role are costed in Appendix I.
  2. We train until convergence on the training set. This means that increasing the number of training steps and training longer will not help the model. This can be seen in Appendix F.3, Figure 2.

Additionally, we ran tests on GSM-Symbolic with more rollouts and more prompts. In Table 9, we keep the 8 questions and increase the number of rollouts to 32, and we also ran new experiments with 32 questions and 4, 8 rollouts, and a reasoner shown the questions of 3 related problems from the same template, without answers and rewarded on its own problem only. We find that there is no gain. This shows that the performance of the reasoner role has saturated.

Qwen3-4B-Base, GSM-NoOp with distractorsUnseenSeen
Reasoner, 8 problems x 16 samples (Table 9)72.675.3
Reasoner, 8 problems x 32 samples (Table 9)71.674.4
Reasoner, 32 problems x 4 samples70.469.5
Reasoner, 32 problems x 8 samples72.774.3
Reasoner, shown 3 related problems of its template70.274.5
+ Abstractor78.780.4

These runs give a sense of how much the reasoner moves from run to run, and the abstractor's gain is well outside that range [W3, Q3].

In wall-clock time the two largest reasoner runs match the abstractor model. The 8 x 32 and 32 x 8 runs take 12.8 and 12.2 hours, against 13.0 for the abstractor model (Appendix I), and neither is more than a point above the reasoner, while the abstractor model is five to six points above it.

For the total cost, Appendix I gives the wall-clock time of every run on one H200, reward generations included. The detector trains in less time than the reasoner, the abstractor takes 1.2 to 1.3 times as long, and the understander 1.6 to 1.7 times.

[W3, Q3] “Limited evidence for the stability of the reported gains …”

We trained the reasoner and the understander with three seeds each. On average the understander does better, by 1.5 points at both sizes. Its seeds also vary less than the reasoner's (standard deviation 0.3 against 1.4 at 4B). The gain is larger on the five datasets the model never trains on, 1.7 points at 4B and 1.6 at 8B, against 0.8 on the training source.

RACE-CQuAILCosmosQALSAT-RCLSAT-LRQuALITYMean
4B Reasoner83.6 ± 0.575.1 ± 2.475.7 ± 3.073.0 ± 1.261.8 ± 2.171.8 ± 0.073.5 ± 1.4
4B + Understander84.4 ± 0.277.7 ± 0.378.4 ± 0.874.4 ± 0.463.3 ± 0.472.1 ± 0.475.0 ± 0.3
8B Reasoner85.7 ± 0.276.2 ± 0.877.1 ± 0.876.4 ± 1.066.1 ± 0.475.7 ± 0.276.2 ± 0.4
8B + Understander86.5 ± 0.578.8 ± 1.079.7 ± 0.978.0 ± 0.567.1 ± 0.376.0 ± 0.477.7 ± 0.3

Mean ± standard deviation over training seeds 123, 234 and 345, each seed at the argmax of its own development curve.

We will include all of these results in the final paper.

AuthorsOpenReview ↗

Gentle Reminder: Discussion Period Ending Soon

Dear Reviewer qHkA,

Thank you again for your feedback on our submission. As the discussion period is coming to a close, we wanted to check whether our responses have addressed your concerns. For [W1], we have stated our contributions explicitly and clarified them against other work. For [W2], we have added a thorough set of compute-matched experiments. For [W3], we have reported 3 training seeds for our reading-comprehension section, which strengthen our arguments.

We have also run experiments with RLAD and Llama-3.1-8B-Instruct that further support these points.

With the discussion period ending tomorrow, we would appreciate it if you could let us know whether these address your concerns, or whether there is anything further we can clarify.

Best regards, The Authors

Thanks for running the extra experiments. The new GSM-NoOp comparison addresses my compute concern for that setting. For the 4B models trained on RACE-C, the Understander raises the three-seed average across six test sets from 73.5% to 75.0%, with training time increasing from 4.1 to 6.6 hours. I’d still like to see how the Reasoner does with the same time budget. On TextCraft, success rises from 37.3% to 39.7% in a single-seed comparison, so it’s hard to tell how reliably that gain would hold. Concerning on the whole paper experiment and representation, I will maintain my score.

Reviewer_fDjWOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether training a language model with auxiliary roles can improve its primary-task performance without invoking those roles at inference. The proposed Role-Based Reinforcement Learning (RBRL) framework jointly optimizes a shared model using role-specific prompts and rewards. Experiments on arithmetic reasoning with distractors and reading comprehension show improvements over reasoner-only training.

Critical strengths
  1. Clear evaluation of transfer without auxiliary inference. This design provides meaningful evidence that auxiliary training can improve primary-task performance through shared parameters.

  2. Consistent empirical improvements across two task domains. The approach improves arithmetic reasoning under distraction and reading comprehension at both model sizes.

  3. Useful ablations and transparent reporting. The appendices also disclose training costs and the limitations of the reported standard errors, helping readers assess the evidence.

Critical weaknesses
  1. Limited differentiation from established auxiliary-task learning. The general principle of improving a primary task through auxiliary objectives and shared parameters is well established, including in UNREAL and multitask language-model training.

  2. The scope of the conclusions exceeds the current evidence. The experiments cover two model sizes from one family and two task domains with explicitly available grouping structures.

  3. The fixed rollout budget does not control total training computation. The budget counts only trainable rollouts and excludes the additional model generations used to evaluate auxiliary artifacts.

Questions for authors

Q1. What technical contribution remains beyond auxiliary-task RL instantiated with role-specific prompts and rewards?

Q2. Can they compare against a strong existing auxiliary-learning or internalization approach under the same data and inference conditions?

3 comments
AuthorsOpenReview ↗

We thank the reviewer for their review and for finding our ablatations useful and transparent.

[W1, Q1] “Limited differentiation from established auxiliary-task learning …” and “What technical contribution remains beyond auxiliary-task RL …”

We see our work as related to auxiliary-task RL, from Horde and UNREAL onward, and we share its core intuition. That line predates LLMs and uses the same auxiliary tasks, such as pixel control and reward prediction, for every environment. It says nothing about LLMs, where a role is the whole model under another prompt. In LLMs, a second role trained on the same data usually checks or critiques the model's own answers, and it is easy to see why that helps solving. Our roles never see or give an answer, and it is not obvious that training them would carry over to the model's reasoning. In multitask language-model training each task brings its own data and labels and produces an answer, while two of our three roles have no answer key.

Beyond the methodology, our work makes three contributions.

  1. We show that cross-task transfer exists in LLMs through disjoint roles and define two properties of auxiliary roles, substitutivity and additivity. Additionally, we show this transfer can lead to better generalization on unseen datasets.
  2. We introduce the view and show that grouping related problems in one view is what lets a role extract signal that training on one problem at a time misses. As far as we know, work that trains an auxiliary role keeps its view fixed.
  3. We show that roles can be defined per dataset. Humans do not learn every subject the same way, so we argue for moving away from static roles, and other papers point the same way, each adding a role for its own domain, an abstraction generator (RLAD), a critic (CTRL), a planner (MPO) or a searcher (s3), played by a second model kept at test time. RBRL unifies these as disjoint roles of one model that are dropped at test time.

We will add these distinctions to Related Work.

We also argue that the simplicity of the framework is a feature and not a flaw. RBRL allows introduction of multiple roles in diverse settings rather than confining itself to one setting and points towards dynamic roles defined by the data itself (meta-roles).

References

[Horde] Sutton, Richard S., et al. "Horde: A Scalable Real-Time Architecture for Learning Knowledge from Unsupervised Sensorimotor Interaction." Proceedings of the 10th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2011, pp. 761-768.

[UNREAL] Jaderberg, Max, et al. "Reinforcement Learning with Unsupervised Auxiliary Tasks." International Conference on Learning Representations (ICLR), 2017.

[RLAD] Qu, Yuxiao, et al. "RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems." International Conference on Learning Representations (ICLR), 2026.

[CTRL] Xie, Zhihui, et al. "Teaching Language Models to Critique via Reinforcement Learning." arXiv, 2025, arxiv.org/abs/2502.03492.

[MPO] Xiong, Weimin, et al. "MPO: Boosting LLM Agents with Meta Plan Optimization." Findings of the Association for Computational Linguistics: EMNLP 2025, 2025.

[s3] Jiang, Pengcheng, et al. "s3: You Don't Need That Much Data to Train a Search Agent via RL." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025.

[TextCraft] Prasad, Archiki, et al. "ADaPT: As-Needed Decomposition and Planning with Language Models." Findings of the Association for Computational Linguistics: NAACL 2024, 2024.

[W2] “The scope of the conclusions exceeds the current evidence …”

We add three things. First, the reading-comprehension result replicates on three training seeds at both sizes. On average the understander does better, by 1.5 points at both sizes.

RACE-CQuAILCosmosQALSAT-RCLSAT-LRQuALITYMean
4B Reasoner83.6 ± 0.575.1 ± 2.475.7 ± 3.073.0 ± 1.261.8 ± 2.171.8 ± 0.073.5 ± 1.4
4B + Understander84.4 ± 0.277.7 ± 0.378.4 ± 0.874.4 ± 0.463.3 ± 0.472.1 ± 0.475.0 ± 0.3
8B Reasoner85.7 ± 0.276.2 ± 0.877.1 ± 0.876.4 ± 1.066.1 ± 0.475.7 ± 0.276.2 ± 0.4
8B + Understander86.5 ± 0.578.8 ± 1.079.7 ± 0.978.0 ± 0.567.1 ± 0.376.0 ± 0.477.7 ± 0.3

Mean ± standard deviation over training seeds 123, 234 and 345, each seed at the argmax of its own development curve.

Second, it replicates on another model family, Llama-3.1-8B-Instruct, trained and selected as in the paper.

Llama-3.1-8B-InstructRACE-CQuAILCosmosQALSAT-RCLSAT-LRQuALITYMean
Reasoner80.874.281.870.453.674.472.5
+ Understander81.975.083.370.754.475.173.4

Third, in long-horizon reasoning a planning role fits naturally, and on TextCraft it raises the model's success from 37.3 to 39.7. We will put this result in the appendix.

[W3] “The fixed rollout budget does not control total training computation …”

We have two controls for computation.

  1. The per step training budget is exactly the same 128 trainable generations across all models in the paper. The additional reward generations of each role are costed in Appendix I.
  2. We train until convergence on the training set. This means that increasing the number of training steps and training longer will not help the model. This can be seen in Appendix F.3, Figure 2.

Additionally, we ran tests on GSM-Symbolic with more rollouts and more prompts. First, we point the reviewer to Table 9. We also ran new experiments with 32 questions and 4, 8 rollouts, and a reasoner shown the questions of 3 related problems from the same template, without answers and rewarded on its own problem only, all shown below. Similar to Table9, we find that there is no gain. This shows that the performance of the reasoner role has saturated and it is the view and the additional roles that induce the gains.

Qwen3-4B-Base, GSM-NoOp with distractorsUnseenSeen
Reasoner, 8 problems x 16 samples (main result)72.675.3
Reasoner, 8 problems x 32 samples (Table 9)71.674.4
Reasoner, 32 problems x 4 samples70.469.5
Reasoner, 32 problems x 8 samples72.774.3
Reasoner, shown 3 related problems of its template70.274.5
+ Abstractor78.780.4

These runs give a sense of how much the reasoner moves from run to run, and the abstractor's gain is well outside that range.

In wall-clock time the two largest reasoner runs match the abstractor model. The 8 x 32 and 32 x 8 runs take 12.8 and 12.2 hours, against 13.0 for the abstractor model (Appendix I).

In fact, on problems with distractors, the detector uses less compute and still ends up performing better (Appendix I).

[Q2] “Can they compare against a strong existing auxiliary-learning or internalization approach under the same data and inference conditions?”

To our knowledge, RLAD is the most direct comparison. RLAD has an abstraction generator and a solver, trained as two policies, and it needs both at test time. We ran it on our data as two LoRA adapters trained together with no warm start, selected on its development curve with the abstraction. The table reports GSM-NoOp problems with distractors at Qwen3-4B-Base.

UnseenSeen
Reasoner72.675.3
RLAD, solver alone66.167.6
RLAD, abstraction then solve75.776.6
Ours (abstractor model)78.780.4

Without its abstraction RLAD falls about seven points below the reasoner, so the solver has offloaded part of the skill to the abstraction model. With the abstraction RLAD gains one to four points over the reasoner, but it needs a second model and a second pass at test time. Our model is evaluated exactly like the reasoner and is still higher.

AuthorsOpenReview ↗

Gentle Reminder: Discussion Period Ending Soon

Dear Reviewer fDjW,

Thank you again for your feedback on our submission. As the discussion period is coming to a close, we wanted to check whether our responses have addressed your concerns. For [W1, Q1], we have stated our contributions explicitly and set them against auxiliary-task RL. For [W2], we have extended the evidence with Llama-3.1-8B-Instruct, a TextCraft planner with no grouping structure, and 3 training seeds. For [W3], we have added compute-matched reasoners (more rollouts, more problems, wall-clock matched), and none of them gains. For [Q2], we have compared against RLAD on our data, and our single model beats its two-model pipeline.

With the discussion period ending tomorrow, we would appreciate it if you could let us know whether these address your concerns, or whether there is anything further we can clarify.

Best regards, The Authors

Reviewer_qUWbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Role-Based Reinforcement Learning (RBRL), which jointly trains one model as a reasoner and in auxiliary roles such as identifying distractors, producing general solution methods, and extracting passage implications. Auxiliary outputs are never supplied to the primary reasoner, and evaluation uses the reasoner alone. Experiments on Qwen3-4B-Base and Qwen3-8B-Base show improved arithmetic robustness and reading comprehension under a fixed budget of trainable rollouts, with additional computation for auxiliary reward evaluation.

Critical strengths
  1. Auxiliary roles are absent at evaluation, and prompt-only, shuffled-reward, and rollout-budget controls strengthen the evidence that their training objectives contribute useful signal.
  2. View ablations connect the gains to learning across related problems, while reading-comprehension improvements extend to five datasets outside the training source.
Critical weaknesses
  1. The role and view definitions largely describe shared-parameter auxiliary-task RL, while the additional methodological insight provided by this abstraction remains unclear.
  2. Wider views also introduce more related examples and reward evaluations, leaving it unclear whether the gains come from cross-instance abstraction or richer, more stable feedback.
Questions for authors
  1. Do the main gains, particularly on reading comprehension, persist across independent training seeds?
  2. Can a reasoner-only baseline using grouped examples and a comparable downstream evaluation budget achieve similar improvements?
3 comments
AuthorsOpenReview ↗

We thank the reviewer for their review

[W1] “… largely describe shared parameter auxiliary RL … additional methodological insight provided by this abstraction remains unclear.”

Beyond the methodology, our work makes three contributions.

  1. We show that cross-task transfer exists in LLMs through disjoint roles and define two properties of auxiliary roles, substitutivity and additivity. Additionally, we show this transfer can lead to better generalization on unseen datasets.
  2. We introduce the view and show that grouping related problems in one view is what lets a role extract signal that training on one problem at a time misses. As far as we know, work that trains an auxiliary role keeps its view fixed.
  3. We show that roles can be defined per dataset. Humans do not learn every subject the same way, so we argue for moving away from static roles, and other papers point the same way, each adding a role for its own domain, an abstraction generator (RLAD), a critic (CTRL), a planner (MPO) or a searcher (s3), played by a second model kept at test time. RBRL unifies these as disjoint roles of one model that are dropped at test time.

We also argue that the simplicity of the framework is a feature and not a flaw. RBRL allows introduction of multiple roles in diverse settings rather than confining itself to one setting and points towards dynamic roles defined by the data itself (meta-roles).

References

[RLAD] Qu, Yuxiao, et al. "RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems." International Conference on Learning Representations (ICLR), 2026.

[CTRL] Xie, Zhihui, et al. "Teaching Language Models to Critique via Reinforcement Learning." arXiv, 2025, arxiv.org/abs/2502.03492.

[MPO] Xiong, Weimin, et al. "MPO: Boosting LLM Agents with Meta Plan Optimization." Findings of the Association for Computational Linguistics: EMNLP 2025, 2025.

[s3] Jiang, Pengcheng, et al. "s3: You Don't Need That Much Data to Train a Search Agent via RL." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025.

[W2, Q2] “… introduce more related examples and reward evaluations … unclear whether the gains come from cross-instance abstraction or richer, more stable feedback.” and “Can a reasoner-only baseline using grouped examples and a comparable downstream evaluation budget achieve similar improvements?”

We argue that the gain is not from more feedback or training budget (as you noted in the strengths, the evaluation budget is the same). First, we train the models till convergence so extra training time does not help there. Additionally, we trained the following reasoners -

  1. Reasoner given 32 rollouts per problem instead of 16
  2. Reasoner given 32 problems per step, with 4 or 8 rollouts each
  3. Reasoner shown the questions of 3 related problems from the same template, without answers, and rewarded on its own problem only

These runs give a sense of how much the reasoner moves from run to run, and the abstractor's gain is well outside that range.

Qwen3-4B-Base, GSM-NoOp with distractorsUnseenSeen
Reasoner, 8 problems x 16 samples (Table 9)72.675.3
Reasoner, 8 problems x 32 samples (Table 9)71.674.4
Reasoner, 32 problems x 4 samples70.469.5
Reasoner, 32 problems x 8 samples72.774.3
Reasoner, shown 3 related problems of its template70.274.5
+ Abstractor (Table 9)78.780.4

In wall-clock time the two largest reasoner runs match the abstractor model. The 8 x 32 and 32 x 8 runs take 12.8 and 12.2 hours, against 13.0 for the abstractor model (Appendix I), and neither is more than a point above the reasoner, while the abstractor model is five to six points above it.

None of these helps the reasoner. The reasoner's has saturated, and the gain needs a different role.

[Q1] “Do the main gains, particularly on reading comprehension, persist across independent training seeds?”

Yes! On average the understander does better than the reasoner, and the mean gain even increases slightly with more seeds, from 1.2 to 1.5 points at 8B (1.4 to 1.5 at 4B).

RACE-CQuAILCosmosQALSAT-RCLSAT-LRQuALITYMean
4B Reasoner83.6 ± 0.575.1 ± 2.475.7 ± 3.073.0 ± 1.261.8 ± 2.171.8 ± 0.073.5 ± 1.4
4B + Understander84.4 ± 0.277.7 ± 0.378.4 ± 0.874.4 ± 0.463.3 ± 0.472.1 ± 0.475.0 ± 0.3
8B Reasoner85.7 ± 0.276.2 ± 0.877.1 ± 0.876.4 ± 1.066.1 ± 0.475.7 ± 0.276.2 ± 0.4
8B + Understander86.5 ± 0.578.8 ± 1.079.7 ± 0.978.0 ± 0.567.1 ± 0.376.0 ± 0.477.7 ± 0.3

Mean ± standard deviation over training seeds 123, 234 and 345, each seed at the argmax of its own development curve, evaluated as in Appendix C.

We also ran the reading comprehension experiments on Llama-3.1-8B-Instruct, and the result reproduces with another model family.

Llama-3.1-8B-InstructRACE-CQuAILCosmosQALSAT-RCLSAT-LRQuALITYMean
Reasoner80.874.281.870.453.674.472.5
+ Understander81.975.083.370.754.475.173.4

We will include the discussion and all the results in the final paper.

AuthorsOpenReview ↗

Gentle Reminder: Discussion Period Ending Soon

Dear Reviewer qUWb,

Thank you again for your feedback on our submission. As the discussion period is coming to a close, we wanted to check whether our responses have addressed your concerns. For [W1], we have stated our contributions explicitly and set them against shared-parameter auxiliary RL. For [W2, Q2], we have shown that the number of reward evaluations does not explain the gains, and that reasoners given grouped examples or more compute do not improve. For [Q1], we have reported 3 training seeds for reading comprehension, where the understander is above the reasoner on all six datasets at both sizes.

We have also run experiments with RLAD and Llama-3.1-8B-Instruct that further support these points.

With the discussion period ending tomorrow, we would appreciate it if you could let us know whether these address your concerns, or whether there is anything further we can clarify.

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to the Area Chair and all reviewers (1/2)

We thank all reviewers. They found

  • the evaluation of transfer without invoking the auxiliary roles at test time clear and meaningful (fDjW), and the effort to isolate transfer through shared parameters useful (qHkA),
  • the prompt-only, shuffled-reward and rollout-budget controls (Tables 11 and 12) strong evidence that the role objectives contribute signal (qUWb, qHkA),
  • the view ablations (Table 2) a link between the gains and learning across related problems (qUWb, qHkA),
  • the improvements consistent across two domains and both model sizes, and on five datasets outside the training source (fDjW, qUWb), and
  • the reporting of training costs and standard errors transparent (fDjW).

Our paper is inspired by how humans get better at a task by practicing on the same material in more than one way, such as comparing related problems or naming the part of a problem that matters. We ask whether formalizing this lets a language model learn more from the same data. We formalize it as Role-Based Reinforcement Learning (RBRL), in which one model is trained in several roles over the same data. The auxiliary roles never produce the reasoner's answer and are dropped at test time. We find two kinds of transfer from the roles to the reasoner. A role is substitutive when it improves a reasoner that never trains on the target problems, and additive when it still improves one that does. The two can differ for the same role. The detector substitutes as well as the abstractor but adds far less, and roles with wider views transfer more.

The three reviews raise nearly the same concerns, and we answer each with a new experiment. Every change in the revised PDF is in blue, and table and appendix numbers refer to it.

1. What RBRL adds beyond auxiliary-task RL (qUWb W1, fDjW W1/Q1, qHkA W1/Q2)

Auxiliary-task RL, from Horde and UNREAL onward, predates LLMs, fixes the same auxiliary tasks for every environment, and computes them on the agent's own observations. It has no view and no per-dataset role. Beyond the methodology, our work makes three contributions.

  1. Transfer. We show that cross-task transfer exists in LLMs through disjoint roles, and we define two properties of auxiliary roles, substitutivity and additivity (Table 1). We also show that this transfer can lead to better generalization on unseen datasets (Table 3).
  2. Views. We introduce the view, the set of related examples a role sees and is scored on, and show that grouping related problems in one view is what lets a role extract signal that training on one problem at a time misses (Table 2 and Section 5). As far as we know, work that trains an auxiliary role keeps its view fixed.
  3. Roles per dataset. We show that roles can be defined per dataset. Humans do not learn every subject the same way, so we argue for moving away from static roles. Other papers point the same way, each adding a role for its own domain, an abstraction generator (RLAD), a critic (CTRL), a planner (MPO) or a searcher (s3), played by a second model kept at test time. RBRL unifies these as disjoint roles of one model that are dropped at test time (Section 2).

We also see the simplicity of the framework as a strength. RBRL lets roles be introduced in diverse settings, and it points toward dynamic roles defined by the data itself, through meta-roles that propose them. The revised Introduction lists these contributions, and the revised Related Work draws these distinctions.

AuthorsOpenReview ↗

General response (2/2)

2. Whether the gain comes from more feedback or more compute (qUWb W2/Q2, fDjW W3, qHkA W2/Q1)

Every model in the paper trains the same 128 trainable generations per step, and we train to convergence. We also trained the following reasoners, and no reasoner improves on the original by more than 0.1 points (Appendix H, Table 13).

Qwen3-4B-Base, GSM-NoOp with distractorsUnseenSeen
Reasoner, 8 problems x 16 samples72.675.3
Reasoner, 8 problems x 32 samples71.674.4
Reasoner, 32 problems x 4 samples70.469.5
Reasoner, 32 problems x 8 samples72.774.3
Reasoner, shown 3 related problems of its template70.274.5
+ Abstractor78.780.4

In wall-clock time the two largest reasoner runs match the abstractor model, 12.8 and 12.2 hours against 13.0 (Appendix I), and neither improves on the reasoner, while the abstractor model is 6.1 and 5.1 points above it. Showing the reasoner the grouped problems leaves it below the original reasoner. The gain comes from the abstractor's role on the group. We argue that the reasoner has saturated and adding more compute does not help.

3. Seeds and generality (qUWb Q1, fDjW W2, qHkA W1 and W3/Q3)

  • Three training seeds on reading comprehension. The understander is above the reasoner on all six datasets at both 4B and 8B, by 1.5 points on the mean. Its seeds vary less than the reasoner's, and its gain on the five datasets it never trains on is larger than on RACE-C (Appendix F.6, Table 8).
  • Another model family. On Llama-3.1-8B-Instruct the understander is above the reasoner on all six datasets, 72.5 to 73.4 on the mean (Appendix F.7, Table 9).
  • A long-horizon setting. A planner role on TextCraft raises the model's success from 37.3 to 39.7 with the plan removed at test time (Appendix K).

4. Comparison with RLAD (fDjW Q2, qHkA Q2)

We ran RLAD on our data as two LoRA adapters trained together (Appendix J, Table 14).

Qwen3-4B-Base, GSM-NoOp with distractorsUnseenSeen
Reasoner72.675.3
RLAD, solver alone66.167.6
RLAD, abstraction then solve75.776.6
Ours (abstractor model)78.780.4

Without its abstraction RLAD falls about seven points below the reasoner, so the solver has offloaded part of the skill to the abstraction model. With the abstraction RLAD gains one to four points over the reasoner, but it needs a second model and a second pass at test time. Our model uses one model and one pass, exactly like the reasoner, and beats RLAD's two-model pipeline by 2.0 and 3.8 points.

Revisions in the PDF (in blue)

  • Introduction, a list of the three contributions.
  • Related Work, the distinctions from auxiliary-task RL and from prior work that adds a domain role as a second model.
  • Appendix F.6 and F.7, three training seeds and Llama-3.1-8B-Instruct on reading comprehension.
  • Appendix H, the reasoner runs above.
  • Appendices J and K, the RLAD comparison and the TextCraft planner.
  • Abstract, Sections 4 to 6 and Limitations, short pointers to these results and a few corrections, including the Llama result in Limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.