acceptodds
Withdrawn from ICLR 2027

Adapter Thickets: Splitting an RLVR Budget Beats Concentrating It

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Majority voting over sampled completions is the workhorse of test-time scaling, and reinforcement learning with verifiable rewards (RLVR) is the workhorse for making each completion better. The standard pipeline composes the two: train one policy with RLVR, then sample it many times and vote. We show that this composition is lossy. A vote can only overturn mistakes that its voters do not share, and RLVR sharpens a policy such that its samples increasingly make the same mistakes. With every method drawing exactly completions per problem, training a single LoRA adapter on the full RLVR budget raises single-sample accuracy on every model we test (B-B). Yet on three of four models it leaves the majority vote below that of the untrained base model, by up to points. The damage builds during training: voter errors grow steadily more correlated, and the majority vote accuracy peaks early before falling by up to points. The cause is concentration, not RLVR itself. We split the same data and training budget across LoRA adapters, each trained on its own random disjoint shard, and call the result an . Thickets out-vote the fully trained adapter in all (model, ) settings, and for they stay within points of the base model or above it. A single adapter stopped early, at a thicket member's step count, is a strong control that matches thickets for small . For , thickets keep more of RLVR's single-sample gain and out-vote this control in six of eight settings. The cost of concentration also grows with the number of votes: from to votes, the thicket's lead over the fully trained adapter widens from to points. When the plan is to sample and vote, an RLVR budget is better spent broad than deep.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Majority Vote, LoRA, Test-Time Scaling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XdsBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows that improving single-sample accuracy through RLVR does not necessarily improve majority-vote accuracy. It proposes adapter thickets, which split a fixed training budget across LoRA adapters trained on disjoint data shards and pool their responses under the same inference budget. Across four models, thickets consistently outperform fully trained single adapters in majority voting, with lower error correlation among voters. Their main benefit is preserving voting performance lost during concentrated training, while gains over the base model and early-stopped single adapters are smaller and less consistent.

Critical strengths

None

Critical weaknesses
  1. Limited conceptual novelty relative to prior work. The core recipe, partitioning training data, independently fine-tuning LoRA adapters, and ensembling their predictions, closely resembles existing adapter-ensemble approaches. Notably, ELREA [1] already includes a random, disjoint data-partitioning baseline, while Tang et al. [2] explicitly study the mismatch between single-sample training objectives and inference-time aggregation. The present paper’s distinct contribution is therefore primarily the controlled evaluation of this recipe under a fixed RLVR budget. It does not establish a substantially new mechanism or explain when and why this allocation should outperform existing alternatives, leaving the conceptual advance limited.
  2. The strongest improvements are measured against a fully trained single adapter whose voting performance has already deteriorated. At 160 samples, the average advantage falls to only 0.71 percentage points over the base model and 0.27 points over the early-stopped single-adapter control; thickets outperform the base model in only 9 of 16 settings. Moreover, the early-stopping control receives approximately 20% fewer updates than thicket members at (K=8) and (K=16), weakening even this small comparative advantage. These results primarily demonstrate recovery from training-induced degradation and do not yet establish a compelling practical benefit over properly selected single-policy baselines.

[1] Li, Y., Gao, V., Zhang, C., & Torkamani, M. (2025, May). Ensembles of low-rank expert adapters. In International Conference on Learning Representations (Vol. 2025, pp. 92940-92968).

[2] Tang, Y., Zheng, K., Synnaeve, G. & Munos, R.. (2025). Optimizing Language Models for Inference Time Objectives using Reinforcement Learning. <i>Proceedings of the 42nd International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 267:59066-59085 Available from https://proceedings.mlr.press/v267/tang25o.html.

Questions for authors
  1. What are the new insights and takeaways the authors bring? How do you think about the comparison between your work and previous works including two papers above?
  2. Beyond the empirical curves and correlations, can the authors provide a predictive explanation of when splitting the RLVR budget is beneficial, how the optimal number of adapters KK depends on the training and inference budgets, and why some models benefit while others do not?
Reviewer_hRCTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes adapter thickets. With both the RLVR training budget and the inference budget held fixed, the training data and rollouts are split into KK shards and KK independent LoRA adapters are trained, one per shard. The authors find that spending the whole budget on a single adapter makes the sampled errors highly correlated, to the point that its majority vote falls below the untrained base model, while splitting the budget beats the single adapter.

Critical strengths
  1. Training is parallel, since the KK members do not depend on each other. With KK GPUs the thicket finishes in 1/K1/K of the wall-clock time.
  2. Diversity collapse is measured through majority-vote accuracy under a fixed budget together with the error correlation among voters, which I found to be a sensible way to quantify the problem.
Critical weaknesses
  1. The macro mean gives the 150-problem Countdown and the 1,319-problem GSM8K equal weight, and it also mixes in-domain and OOD results into a single number.
  2. Compared to the single adapter, each thicket member only sees 1/K1/K of the data and is only trained for 1/K1/K of the steps. What is missing is a control where KK adapters each see the full data and each train for 500/K500/K steps.
Questions for authors
  1. What are the unparseable rate and the truncation rate on each test set? Since nulls are dropped before voting, is the effective number of votes still 160 for every method?
  2. How were the 300 Knights & Knaves problems selected, and how were the 150 Countdown problems chosen?
Reviewer_AWabOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to allocate a fixed RLVR training budget when inference uses majority voting. The authors find that training a single adapter can improve pass@1 while hurting voting accuracy. They propose splitting the data and training budget across multiple LoRA adapters and pooling their samples at inference. Under a fixed sampling budget, this improves voting over the fully trained single adapter, though gains over an early-stopped adapter are smaller.

Critical strengths
  1. The paper highlights a useful mismatch between the training objective and inference-time aggregation: improving pass@1 can hurt majority-vote accuracy. The checkpoint analysis makes this finding concrete and shows why pass@1 alone may be insufficient for selecting models intended for test-time scaling.

  2. The experiments include meaningful controls, particularly a fixed sampling budget and an early-stopped single adapter, across four models and three seeds. These comparisons help distinguish the benefit of splitting the training budget from simply avoiding overtraining, even though the gains over early stopping are modest.

Critical weaknesses
  1. The practical advantage is a bit unclear: in Table 1, the proposed methods do not always outperform simple baseliens like no training (base), and under-trained single lora (grpo-single matched).
  2. There's limited evidence that the author picked a reasonable strong baseline--in the limitaitons, they said they didn't tune the hyper-parameters (e.g., turn kl loss off by default), so it's unclear if it's really necessary to train separate policies if the things can be potentially fixed by adding a regulation term for the single adapter training to avoid degradation.
  3. The experiments for coding does not have a natural majority voting mech. The author used coverage--marked as correct as long as one of the answer passed the test--but in practice users have no info about how to choose the best one like in math.
  4. The benefits with a larger K seems to be decreasing.
Questions for authors
  1. In Table 1, thickets do not consistently outperform GRPO-single (matched). What do you see as the strongest evidence that splitting the budget offers a meaningful advantage over early stopping, given that the latter uses less total training compute?
  2. Have you tested using kl regularation or setting different sampling temperature for majority voting for the single-agent baseline?
  3. Any idea on how to extend the method to the coding domains practically where majority voting doesn't work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.