acceptodds
Under review as a conference paper at ICLR 2027

Tandem Reinforcement Learning with Verifiable Rewards

Abstract

Reinforcement learning with verifiable rewards (RLVR) has driven substantial gains in language-model reasoning. However, models fine-tuned with RLVR need not remain compatible with weaker partners: gains achieved alone may erode under collaboration, while policy drift may place reasoning beyond a weaker model's predictive reach. To address both problems, we introduce Tandem RLVR (TRLVR), bringing the recently proposed tandem-training paradigm beyond proof-of-concept settings and into modern RLVR. In TRLVR, a senior learns from rewarded trajectories co-generated with a frozen junior, while the standard GRPO objective is applied only to senior-emitted tokens. Training Qwen3-4B-Instruct on competition math, TRLVR performs on par with a matched GRPO senior when reasoning alone. We characterize compatibility through the complementary lenses of handoff robustness and junior legibility. Under reasoning-step handoffs, the TRLVR senior retains nearly all of its solo performance with the junior, reducing the communication tax to a negligible loss. The senior's reasoning is also more legible to the junior: its token distribution stays more closely anchored to the junior's, and the junior predicts the senior's chain-of-thought more readily token by token. An ablation that explicitly regularizes GRPO toward the junior does not recover the same combination of capability and compatibility. These results provide initial evidence that tandem rollout structure is a promising direction for keeping RLVR reasoning gains accessible to weaker partners.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM reasoning, RLVR, tandem training, human-AI compatibility, multi-agent collaboration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PS8pOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper adapts tandem training to reinforcement learning with verifiable rewards. A trainable senior and a frozen copy of its initial policy jointly generate trajectories through stochastic handoffs near word boundaries. A verifier rewards the completed trajectory, and GRPO updates apply only to senior-generated tokens. The method can be understood as implicit regularization toward compatibility with the initial policy: the frozen partner participates in the rewarded trajectories, encouraging the senior to improve while remaining effective with that partner. This differs from explicitly penalizing policy divergence through KL regularization. Experiments with Qwen3-4B-Instruct on four mathematics benchmarks show similar solo performance to GRPO, improved handoff performance, and reasoning traces that are more predictable to the junior.

Critical strengths
  1. By placing a frozen junior inside reward-bearing trajectories, TRLVR encourages the senior to remain effective with its partner without prescribing a preferred reasoning style or adding an explicit divergence penalty. This adaptation of tandem training to RLVR provides an interesting alternative to conventional regularization.
  2. The paper evaluates both whether the junior can participate successfully through handoff accuracy and capability retention, and whether it can predict the senior’s reasoning through cross-entropy and distribution overlap. Measuring these properties alongside solo performance makes the capability–compatibility trade-off concrete. The stochastic-handoff and cross-entropy controls further strengthen the evaluation.
Critical weaknesses

The experimental scope is narrow relative to the potential generality of the approach. All reported training experiments use Qwen3-4B-Instruct, paired with a frozen copy of the same initial checkpoint. Although evaluation covers four mathematics benchmarks, it remains unclear whether the favorable capability–compatibility trade-off extends across model families and scales. Different initial reasoning styles and stronger starting capabilities may change both the compatibility loss induced by RLVR and the effectiveness of tandem training. Experiments with another model family and a larger model, each using its own frozen initialization as the junior, would substantially strengthen the contribution.

Questions for authors
  1. Have you evaluated TRLVR on another model family or a larger model?
  2. How sensitive is the benefit to the choice and capability of the frozen junior? In particular, does pairing the senior with a smaller junior, rather than its own pre-RL initialization, preserve the observed improvement?
2 comments
AuthorsOpenReview ↗

Authors' Rebuttal

We thank the reviewer for the careful review.

W & Q1: Have you evaluated TRLVR on a larger model?

Yes. On Qwen3-8B in thinking mode, TRLVR reaches higher handoff accuracy than GRPO, while solo accuracy stays on par.

We chose a long-CoT thinking model stronger than the model in the paper, Qwen3-8B with thinking enabled, and paired each senior with its own frozen initialization as the junior, as the reviewer suggests. Thinking models open a new design space for tandem rollouts: the senior and the junior write the reasoning inside the thinking block together, and once the block closes, the junior summarizes it and gives the answer. This boundary between reasoning and answer comes naturally with thinking models and is difficult to operationalize in non-thinking models.

We compare the two seniors at the same training step, with a 16k-token generation budget at inference:

Qwen3-8B, thinkingSolo pass@4 (%)Handoff pass@4 (%)Handoff robustness (%)Junior CE (nats)
GRPO82.4180.1697.30.153
TRLVR81.9781.2599.10.140

A long-CoT thinking model is strong on its own, and so is its junior on competition math, so handoff costs GRPO only a few points in this setting. TRLVR removes most of that loss, and its reasoning stays more legible to the junior. We see this run as a first exploration of the design space that tandem rollouts open, here the split of thinking and answering between the two models. We leave further design choices, other model families and harder tasks, including agentic tasks, to future work. We thank the reviewer for this suggestion, which extends the paper to a larger thinking model.

Q2: Does pairing the senior with a smaller junior preserve the improvement?

Yes. The improvement holds with smaller juniors swapped in at evaluation, and TRLVR does not require its own initialization as the training junior.

First, we paired the trained seniors, without further training, with Qwen3-0.6B and Qwen3-1.7B and evaluated them with the strict alternation of Section 4.3 next to the training junior, Qwen3-4B-Instruct. With a weaker junior, handoff accuracy stays below the senior's own solo accuracy, so the room for improvement is the accuracy the senior loses when the junior joins. The first row gives each senior's solo accuracy, and the last column reports how much less TRLVR loses than GRPO:

Pass@4 (%)GRPOTRLVRHandoff loss removed
Solo, no junior55.5654.85
Qwen3-0.6B48.3350.8945%
Qwen3-1.7B52.0752.7540%
Qwen3-4B-Instruct52.4054.7497%

The revised paper adds these results in the new Section 5.3, with per-benchmark numbers in Appendix E.5.

Second, we trained TRLVR with a junior other than its own initialization. Qwen3-0.6B solves few competition problems on its own, so instead of a smaller junior we used a different one, Qwen3-4B in non-thinking mode, and kept all other settings. Paired with that junior under the protocol of Table 2, the new senior is on par with the original TRLVR:

Handoff with Qwen3-4B non-thinking (%)pass@1pass@4
TRLVR, trained with Qwen3-4B-Instruct43.1053.93
TRLVR, trained with Qwen3-4B non-thinking43.2653.68

The new Appendix E.1 reports this run. Training with a pool of juniors that vary in capability is the first direction Section 6 names, and we see these results as support for it.

We thank the reviewer again for the constructive review. The revised paper adds the Qwen3-8B results in Section 5.4, the unseen juniors in Section 5.3 and the run with a different training junior in Appendix E.1.

We greatly value the reviewer's feedback, which has made the paper stronger, and we would be glad to address any further questions or suggestions.

Thank you for the additional experiments. The results on Qwen3-8B and Qwen3-4B non-thinking mode address my concerns. Retaining comparable performance with a changed training partner is a useful finding. I have no further substantive comments.

Reviewer_FS3cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work applies tandem training to RLVR: a trainable senior model and a frozen junior model jointly generate trajectories, with reward derived from final answer correctness. GRPO updates only the tokens produced by the senior. Both models are initialized from the same Qwen3-4B-Instruct checkpoint, and their capability gap emerges over the course of training. Experiments show that TRLVR achieves solo performance comparable to GRPO, while attaining higher accuracy when generating in alternation with the junior. The authors further measure compatibility via marginal token distributions, junior cross-entropy, and distribution overlap, and compare these metrics against explicit KL regularization.

Critical strengths
  1. By directly incorporating continuations from the junior into reward-bearing trajectories, senior optimization is conditioned on whether the pair can jointly solve the task. This provides a direct task-level signal for compatibility, without requiring extra process labels or predefined stylistic constraints. Experiments show that TRLVR yields a higher macro average than GRPO, supporting that the method improves the trade-off between standalone capability and handoff performance under the studied setup.
  2. TRLVR retains advantages across four benchmarks under random reasoning-step handoffs. The paper also examines the relationship between junior cross-entropy and correctness, with corrections using the senior’s own cross-entropy, which strengthens the reliability of the distribution compatibility analysis. Additionally, integrating dual-model joint generation into vLLM with separate KV caches constitutes important engineering work that enables the method to operate on long RLVR trajectories.
Critical weaknesses
  1. Prior work [1] has already proposed a paradigm with a frozen junior, stochastic token-level handoff, and senior training using shared task rewards; it also explicitly discussed combinations with algorithms such as GRPO and the possibility of masking junior tokens. Accordingly, the core contribution of this paper should be framed as validating this paradigm within competitive mathematical RLVR settings and supplying more comprehensive compatibility measurements. It remains unclear whether the observed gains reflect overfitting to this specific handoff scheme or a more transferable improvement in compatibility.

[1]. West, R., Anderson, A., Kamar, E., & Horvitz, E. (2026). Tandem Training for Language Models. EACL, 8265–8278.

  1. Only two KL-Reg coefficients separated by two orders of magnitude are tested: one barely improves compatibility, while the other incurs substantial capability degradation. The absence of intermediate strengths prevents the experiments from locating the best achievable capability–compatibility trade-off under KL regularization. Checkpoint selection also requires tighter controls: TRLVR checkpoints are selected based on joint generation validation performance, whereas GRPO checkpoints are selected on solo performance. The matched-step comparison in Table 2 is helpful but does not amount to a comparison of optimal trade-offs under a unified selection rule. In fact, GRPO achieves a junior CE of 0 at step 120, lower than TRLVR, indicating that training progress itself shifts this trade-off frontier.
Questions for authors
  1. Does the handoff advantage persist when replacing the trained junior with an unseen, held-out junior, without further senior fine-tuning?
  2. With intermediate KL coefficients added and a shared validation selection rule adopted, does TRLVR still occupy a better capability–compatibility trade-off frontier?
  3. What are the token attribution ratios, handoff counts, and segment length distributions for both methods in the main handoff experiments? Are conclusions consistent under fixed-progress one-shot continuation setups?
  4. How stable are the handoff gains, solo-performance gaps, and the reported 99.8% retention rate across independent training replicates?
3 comments
AuthorsOpenReview ↗

Authors' Rebuttal (Part 1)

We thank the reviewer for the careful review.

W1: Relation to tandem training and transfer beyond one handoff scheme

TRLVR brings tandem training into modern RLVR and shows that it keeps RLVR's gains accessible to weaker partners. RLVR rewards correct answers without requiring reasoning that weaker partners can follow. Tandem training addresses this through the reward, since the senior is rewarded on responses it writes together with a frozen junior. West et al. [1] showed on GSM8K that this teaches the senior to adapt its language while keeping its accuracy, but whether it holds up with the long chains of thought and policy optimization of modern RLVR remained unclear. On competition math, the TRLVR senior matches GRPO alone and keeps nearly all of that accuracy when the junior takes over half of the reasoning. KL regularization toward the same junior stays below TRLVR in handoff accuracy at every coefficient we tried.

TRLVR's lead also holds under handoff schemes it was not trained with. Training hands off at random words, whereas Section 4.3 alternates at every reasoning step, Appendix C.5 hands off at random steps, and the legibility results of Section 4.4 involve no handoff at all. It holds as well with juniors never seen in training and when the junior must finish alone, as the answer to Q1 reports.

W2 & Q2: With intermediate KL coefficients and a shared selection rule, does TRLVR still occupy a better trade-off frontier?

Yes. No intermediate KL coefficient reaches TRLVR's handoff accuracy or robustness, and TRLVR's handoff lead over GRPO also holds at the same training step. We trained KL-Reg at two intermediate coefficients with the same data and settings, and evaluated both at the same training step with the protocol of Table 2. Every coefficient stays at least 2.4 points below TRLVR in handoff accuracy, whatever its solo capability:

MethodSolo pass@4 (%)Handoff pass@4 (%)Handoff robustness (%)
KL-Reg, β = 10⁻³55.2452.0994.3
KL-Reg, β = 10⁻², new53.7552.2997.3
KL-Reg, β = 5×10⁻², new53.2251.7097.1
KL-Reg, β = 10⁻¹50.1549.1598.0
TRLVR54.8554.7499.8

Moreover, the added coefficients make the trade-off under KL regularization clear, and we thank the reviewer for suggesting them. As β grows, KL-Reg gains handoff robustness only by giving up solo capability, so its handoff accuracy stays flat and then falls. TRLVR keeps GRPO-level solo capability with nearly full robustness, which no coefficient reaches, so TRLVR lies beyond the capability-compatibility frontier that KL regularization traces. The revised paper adds these coefficients in the new Appendix E.2.

On checkpoint selection, TRLVR leads under handoff in both of the comparisons that Table 2 makes. Each run is selected at its best validation checkpoint under the rollouts it trains on, solo rollouts for GRPO and joint rollouts for TRLVR (Section 5.1), and Table 2 also evaluates GRPO at TRLVR's training step, which involves no validation at all. Of these two GRPO checkpoints, the stronger under handoff is the best-validation one, and it still trails TRLVR by 2.3 points. The matched step also explains the lower junior cross-entropy the reviewer notes: earlier in training, GRPO is slightly easier for the junior to predict but less capable alone and under handoff. TRLVR improves on that trade-off: it is more capable than GRPO at the matched step, more legible than GRPO at its best step, and more robust under handoff than either.

Q1: Does the handoff advantage persist with an unseen, held-out junior?

Yes. With both unseen juniors, TRLVR reaches higher handoff accuracy than GRPO. We paired both seniors, without further training, with Qwen3-0.6B and Qwen3-1.7B, which never appear in training, and evaluated them with the strict alternation of Section 4.3 next to the training junior, Qwen3-4B-Instruct. With a weaker junior, handoff accuracy stays below the senior's own solo accuracy, so the room for improvement is the accuracy the senior loses when the junior joins. The first row gives each senior's solo accuracy, and the last column reports how much less TRLVR loses than GRPO:

Pass@4 (%)GRPOTRLVRHandoff loss removed
Solo, no junior55.5654.85
Qwen3-0.6B48.3350.8945%
Qwen3-1.7B52.0752.7540%
Qwen3-4B-Instruct52.4054.7497%

The revised paper adds these results in the new Section 5.3, with per-benchmark numbers in Appendix E.5. We also tested one-shot takeover, a second common handoff setting: the senior writes a full chain-of-thought, and the junior takes over at its midpoint and finishes alone. Across the benchmarks, every junior finishes TRLVR prefixes correctly more often, by 2.66 points with Qwen3-0.6B, 3.71 with Qwen3-1.7B and 1.33 with the training junior, although TRLVR never trains for this setting.

AuthorsOpenReview ↗

Authors' Rebuttal (Part 2)

Q3: What are the handoff statistics, and do the conclusions hold under one-shot continuation?

The two methods hand off in nearly the same way under the strict alternation of Section 4.3. Averaged over the responses:

StatisticGRPOTRLVR
Senior token share (%)50.551.0
Handoffs per response58.559.0
Senior segment length (tokens)25.627.1
Junior segment length (tokens)25.625.1

Segment lengths are right-skewed for both methods, with a median of about 19 tokens, and the new Appendix E.4 adds these statistics. The takeover protocol of Q1 is the fixed-progress one-shot setup the reviewer describes, and it gives the same conclusion: taking over at the midpoint of the senior's chain-of-thought, juniors finish TRLVR prefixes correctly more often, by 1.33 to 3.71 points.

Q4: Stability of the handoff gain, the solo gap and the 99.8% retention across training runs

Our revised paper adds an independently trained run in a new setting, Qwen3-8B in thinking mode, where TRLVR again leads GRPO under handoff. The new Section 5.4 trains this model with GRPO and with TRLVR. TRLVR matches GRPO alone and retains 99.1% of its solo accuracy, against 97.3% for GRPO. Within the main run, Table 2 reports bootstrap standard errors over problems of about 2 points. The two seniors are on par alone well within these errors, and Appendix C.1 finds the handoff gain over GRPO significant at every k. TRLVR also trains stably, as its training reward closely tracks GRPO's throughout (Figure 4a).

We thank the reviewer again for the detailed review. The revised paper adds the unseen juniors in Section 5.3, the Qwen3-8B thinking run in Section 5.4, and the intermediate KL coefficients and the handoff statistics in Appendix E.

We greatly value the reviewer's feedback, which has made the paper stronger, and we would be glad to address any further questions or suggestions.

References

[1] R. West, A. Anderson, E. Kamar, E. Horvitz. Tandem Training for Language Models. EACL, 2026.

AuthorsOpenReview ↗

Authors Comment

Dear Reviewer FS3c,

Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.

Best regards,

The Authors

Reviewer_HWVLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Tandem RLVR to improve the compatibility between stronger and weaker language models during collaborative reasoning. The method allows a trainable senior model and a frozen junior model to jointly generate reasoning trajectories, while updating only the senior using GRPO. Experiments on four mathematical reasoning benchmarks show that TRLVR maintains comparable standalone performance to standard GRPO while improving handoff robustness and making the senior's reasoning more predictable to the junior. Additional experiments demonstrate that simple KL regularization cannot achieve the same balance between reasoning capability and compatibility.

Critical strengths
  1. The paper addresses an important challenge in RLVR: improving reasoning capabilities while maintaining compatibility with weaker models. This is a valuable direction for multi-agent collaboration.

  2. The proposed method improves handoff robustness without sacrificing standalone reasoning performance. The experimental results provide convincing evidence of its effectiveness in the evaluated setting.

Critical weaknesses
  1. Experiments focus on a single model and mathematical reasoning tasks, leaving generalization to heterogeneous multi-agent settings unclear.
  2. Handoff accuracy and token predictability do not fully establish whether the junior effectively understands and uses the senior's reasoning.
Questions for authors
  1. Does TRLVR generalize to collaboration between models with different architectures or capability levels?
  2. Can the authors provide stronger evidence that the junior model effectively understands and uses the senior's reasoning, rather than simply continuing its output?
  3. How sensitive is TRLVR's performance to the handoff frequency and the choice of the junior model?
  4. How does TRLVR compare with alternative collaboration strategies under the same computational budget?
2 comments
AuthorsOpenReview ↗

Authors' Rebuttal

We thank the reviewer for the careful review.

W1 & Q1: Does TRLVR generalize to partners of other capability levels and architectures?

Yes. TRLVR reaches higher handoff accuracy than GRPO with juniors of every capability level we tested, including juniors it never met in training.

We paired the two trained seniors, without further training, with Qwen3-0.6B and Qwen3-1.7B and evaluated them with the strict alternation of Section 4.3 next to the training junior, Qwen3-4B-Instruct. With a weaker junior, handoff accuracy stays below the senior's own solo accuracy, so the room for improvement is the accuracy the senior loses when the junior joins. The first row gives each senior's solo accuracy, and the last column reports how much less TRLVR loses than GRPO:

Pass@4 (%)GRPOTRLVRHandoff loss removed
Solo, no junior55.5654.85
Qwen3-0.6B48.3350.8945%
Qwen3-1.7B52.0752.7540%
Qwen3-4B-Instruct52.4054.7497%

The revised paper adds these results in the new Section 5.3, with per-benchmark numbers in Appendix E.5.

Juniors from other model families use different tokenizers, while our dual-decoder backend runs both models on the same token sequence (Appendix B.1). Supporting them means re-tokenizing the shared response at each handoff, an engineering extension that leaves the objective unchanged, so we leave other families, and domains beyond mathematics, to future work.

W2 & Q2: Can the authors provide stronger evidence that the junior uses the senior's reasoning?

Yes. Given the first half of the senior's chain-of-thought and left to finish alone, the junior solves more problems from TRLVR's half than from GRPO's. We agree that handoff accuracy alone does not establish understanding, as Section 3.1 notes. Truncating a chain-of-thought is a common probe of how much an answer relies on the reasoning before the cut [1], so we cut each senior's full chain-of-thought at its midpoint and let the junior finish alone. The junior and the cut are the same for both seniors, so a difference in accuracy comes from the reasoning the junior receives. Every junior solves more problems from TRLVR's half:

Takeover accuracy (%)GRPO prefixTRLVR prefix
Qwen3-0.6B24.4527.11
Qwen3-1.7B36.5440.25
Qwen3-4B-Instruct46.0147.34

The gain holds with juniors TRLVR never met in training, and TRLVR never trains with a single takeover. In the step-level case study of Appendix C.3, TRLVR's reduction in junior cross-entropy also concentrates on steps whose meaning depends on earlier text, while self-contained steps show almost no gap.

Q3: How sensitive is TRLVR to the handoff frequency and the choice of junior?

TRLVR's lead holds when evaluation hands off at a very different frequency from training, and with every junior we tested. Training redraws the active model at every word with p = 0.5, following the original tandem training, whereas evaluation hands off once per reasoning step, and Appendix C.5 also hands off at random steps. TRLVR leads under both evaluation schedules. W1 & Q1 covers the choice of junior.

Q4: How does TRLVR compare with alternative collaboration strategies under the same compute budget?

Neither regularizing GRPO toward the junior nor spending TRLVR's extra compute on more GRPO rollouts matches TRLVR's handoff accuracy. KL-Reg of Section 5.2 is the direct alternative for keeping the senior compatible with the junior, at close to GRPO's cost. It pulls the senior's token distribution toward the junior's, but its task reward comes only from the senior's solo rollouts, so training never tests whether the junior can work with the senior's reasoning (Appendix D.1). Across four coefficients, including two new ones in Appendix E.2, KL-Reg stays at least 2.4 points below TRLVR in handoff accuracy.

TRLVR adds a second forward pass per generated token, so we also trained GRPO with G = 16, which doubles the rollouts per prompt and costs at least as much as TRLVR. At its best validation checkpoint, evaluated with the protocol of Table 2, it recovers only about a quarter of TRLVR's handoff gain and makes the senior's reasoning harder for the junior to predict:

MethodHandoff pass@4 (%)Junior CE (nats)
GRPO, G = 852.400.129
GRPO, G = 1652.950.151
TRLVR54.740.118

We thank the reviewer again for the constructive review. The revised paper adds the unseen juniors in Section 5.3, the intermediate KL coefficients in Appendix E.2 and the G = 16 comparison in Appendix E.3.

We greatly value the reviewer's feedback, which has made the paper stronger, and we would be glad to address any further questions or suggestions.

References

[1] T. Lanham et al. Measuring Faithfulness in Chain-of-Thought Reasoning. arXiv:2307.13702, 2023.

Thank you for the detailed rebuttal and additional experiments. The new results adequately address my main concerns, particularly regarding generalization across junior capability levels, the junior's ability to leverage the senior's reasoning, and comparisons under similar computational budgets. Although cross-architecture generalization and handoff frequency sensitivity remain open questions, I am satisfied with the response and maintain my positive assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttals and Paper Revisions

We thank the three reviewers for their careful reviews. We have uploaded a revised PDF with the revisions marked in blue.

New results in the responses

  1. TRLVR reaches higher handoff accuracy than GRPO with juniors of every capability level we tested, including juniors it never met in training (all reviewers). Paired without further training with Qwen3-0.6B and Qwen3-1.7B, TRLVR loses 40% to 45% less accuracy than GRPO when the junior joins. The compatibility TRLVR learns with one junior therefore carries over to smaller partners it never met.
  2. Given the first half of the senior's chain-of-thought and left to finish alone, the junior solves more problems from TRLVR's half than from GRPO's (HWVL, FS3c). This holds with every junior, although TRLVR never trains with a single takeover. Since the junior and the cut are the same for both seniors, the difference comes from the reasoning the junior receives.
  3. No intermediate KL coefficient reaches TRLVR's handoff accuracy or robustness (FS3c, HWVL). Across four coefficients, KL-Reg stays at least 2.4 points below TRLVR in handoff accuracy and gains robustness only by giving up solo capability, so TRLVR lies beyond the trade-off frontier that KL regularization traces.
  4. Spending TRLVR's extra compute on more GRPO rollouts recovers little of TRLVR's compatibility (HWVL). GRPO with G = 16, which costs at least as much as TRLVR, recovers only about a quarter of TRLVR's handoff gain and makes the senior's reasoning harder for the junior to predict, so the gain does not come from extra compute.
  5. On Qwen3-8B in thinking mode, TRLVR reaches higher handoff accuracy than GRPO, while solo accuracy stays on par (PS8p). The senior and the junior write the reasoning inside the thinking block together, and the junior gives the answer once the block closes, so tandem rollouts extend to a larger long-CoT model.

Changes to the paper

  • The abstract and introduction add that TRLVR leads GRPO with smaller juniors it never met in training.
  • The new Section 5.3 reports the unseen juniors, the takeover and a senior trained with a different junior, with details in Appendices E.1 and E.5.
  • The new Section 5.4 and Table 3 report Qwen3-8B in thinking mode.
  • Sections 5.1 and 5.2 add GRPO with G = 16 and two intermediate KL coefficients, reported in Appendices E.3 and E.2.
  • Section 4.3 states how often evaluation hands off, and Appendix E.4 adds the handoff statistics.

Our responses cite tables by their numbers in the original submission. In the revision, the new Table 3 holds the Qwen3-8B results, so former Tables 3 to 8 are now Tables 4 to 9.

We thank the reviewers again and would be glad to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.