- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper adapts tandem training to reinforcement learning with verifiable rewards. A trainable senior and a frozen copy of its initial policy jointly generate trajectories through stochastic handoffs near word boundaries. A verifier rewards the completed trajectory, and GRPO updates apply only to senior-generated tokens. The method can be understood as implicit regularization toward compatibility with the initial policy: the frozen partner participates in the rewarded trajectories, encouraging the senior to improve while remaining effective with that partner. This differs from explicitly penalizing policy divergence through KL regularization. Experiments with Qwen3-4B-Instruct on four mathematics benchmarks show similar solo performance to GRPO, improved handoff performance, and reasoning traces that are more predictable to the junior.
Critical strengths
- By placing a frozen junior inside reward-bearing trajectories, TRLVR encourages the senior to remain effective with its partner without prescribing a preferred reasoning style or adding an explicit divergence penalty. This adaptation of tandem training to RLVR provides an interesting alternative to conventional regularization.
- The paper evaluates both whether the junior can participate successfully through handoff accuracy and capability retention, and whether it can predict the senior’s reasoning through cross-entropy and distribution overlap. Measuring these properties alongside solo performance makes the capability–compatibility trade-off concrete. The stochastic-handoff and cross-entropy controls further strengthen the evaluation.
Critical weaknesses
The experimental scope is narrow relative to the potential generality of the approach. All reported training experiments use Qwen3-4B-Instruct, paired with a frozen copy of the same initial checkpoint. Although evaluation covers four mathematics benchmarks, it remains unclear whether the favorable capability–compatibility trade-off extends across model families and scales. Different initial reasoning styles and stronger starting capabilities may change both the compatibility loss induced by RLVR and the effectiveness of tandem training. Experiments with another model family and a larger model, each using its own frozen initialization as the junior, would substantially strengthen the contribution.
Questions for authors
- Have you evaluated TRLVR on another model family or a larger model?
- How sensitive is the benefit to the choice and capability of the frozen junior? In particular, does pairing the senior with a smaller junior, rather than its own pre-RL initialization, preserve the observed improvement?
2 comments
Thank you for the additional experiments. The results on Qwen3-8B and Qwen3-4B non-thinking mode address my concerns. Retaining comparable performance with a changed training partner is a useful finding. I have no further substantive comments.
Authors' Rebuttal
We thank the reviewer for the careful review.
Yes. On Qwen3-8B in thinking mode, TRLVR reaches higher handoff accuracy than GRPO, while solo accuracy stays on par.
We chose a long-CoT thinking model stronger than the model in the paper, Qwen3-8B with thinking enabled, and paired each senior with its own frozen initialization as the junior, as the reviewer suggests. Thinking models open a new design space for tandem rollouts: the senior and the junior write the reasoning inside the thinking block together, and once the block closes, the junior summarizes it and gives the answer. This boundary between reasoning and answer comes naturally with thinking models and is difficult to operationalize in non-thinking models.
We compare the two seniors at the same training step, with a 16k-token generation budget at inference:
A long-CoT thinking model is strong on its own, and so is its junior on competition math, so handoff costs GRPO only a few points in this setting. TRLVR removes most of that loss, and its reasoning stays more legible to the junior. We see this run as a first exploration of the design space that tandem rollouts open, here the split of thinking and answering between the two models. We leave further design choices, other model families and harder tasks, including agentic tasks, to future work. We thank the reviewer for this suggestion, which extends the paper to a larger thinking model.
Yes. The improvement holds with smaller juniors swapped in at evaluation, and TRLVR does not require its own initialization as the training junior.
First, we paired the trained seniors, without further training, with Qwen3-0.6B and Qwen3-1.7B and evaluated them with the strict alternation of Section 4.3 next to the training junior, Qwen3-4B-Instruct. With a weaker junior, handoff accuracy stays below the senior's own solo accuracy, so the room for improvement is the accuracy the senior loses when the junior joins. The first row gives each senior's solo accuracy, and the last column reports how much less TRLVR loses than GRPO:
The revised paper adds these results in the new Section 5.3, with per-benchmark numbers in Appendix E.5.
Second, we trained TRLVR with a junior other than its own initialization. Qwen3-0.6B solves few competition problems on its own, so instead of a smaller junior we used a different one, Qwen3-4B in non-thinking mode, and kept all other settings. Paired with that junior under the protocol of Table 2, the new senior is on par with the original TRLVR:
The new Appendix E.1 reports this run. Training with a pool of juniors that vary in capability is the first direction Section 6 names, and we see these results as support for it.
We thank the reviewer again for the constructive review. The revised paper adds the Qwen3-8B results in Section 5.4, the unseen juniors in Section 5.3 and the run with a different training junior in Appendix E.1.
We greatly value the reviewer's feedback, which has made the paper stronger, and we would be glad to address any further questions or suggestions.