acceptodds
Under review as a conference paper at ICLR 2027

TSRM: Two-Sided Reliability Modeling for Collaborative Reinforcement Post-Training

Abstract

Collaborative reinforcement post-training allows large language models to learn from distributed small-model evaluations without collecting their local training data. Existing approaches turn client evaluations of generated reasoning into rewards, making both the reasoning candidates and their evaluators part of the training signal. However, without reference-answer rewards, the resulting supervision is subject to uncertainty in both candidate reasoning quality and the suitability of client evaluators for different subjects. We propose TSRM, a two-sided reliability modeling framework that addresses these uncertainties by coordinating candidate selection and evaluator routing within a shared feedback loop. To guide candidate selection without correctness labels, we design MultiGranularity Candidate Confidence Filtering (MG-CCF), which captures overall confidence and local low-confidence patterns through multi-granularity aggregation of probability- and entropy-based token scores. To adapt evaluator selection to subject-dependent feedback, Stability-Aware Subject-Profile Routing (SASPR) maintains client–subject profiles through smoothed agreement estimates and variability penalties. Full-group feedback updates the profiles for subsequent evaluator selection, whereas feedback on retained candidates drives policy learning and thus shapes the next round of candidates. Experiments across six mathematical-reasoning benchmarks show that TSRM achieves the highest average accuracy under both Non-IID and identity-homogenized evaluator settings, outperforming FedGRPO-U by 2.7 and 2.2 points, respectively, without referenceanswer rewards or predefined expert assignments.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Collaborative Reinforcement Post-Training, Group Relative Policy Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_2ChqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

TSRM integrates candidate filtering with evaluator routing in collaborative reinforcement post-training. MG-CCF ranks generated reasoning traces using global, tail, and local confidence statistics. SA-SPR selects clients based on subject-specific histories of answer agreement and its variability. Each candidate receives an agreement reward from a single assigned client. Complete candidate groups are used to update routing profiles, while retained candidates contribute to group-relative policy optimization. In the experiments, Qwen2.5-Math-1.5B is trained on 2,000 MATH problems using eight cached client-answer sources and evaluated on six mathematical benchmarks under persistent and redistributed client identities.

Critical strengths
  1. Considering candidate and evaluator selection together is a reasonable direction when reference-answer rewards are unavailable.

  2. The paper includes both single-module ablations and a diagnostic with a fixed generator. The latter provides useful evidence that confidence filtering retains more correct answers in the candidate pool examined.

Critical weaknesses
  1. The practical motivation remains insufficiently developed, and the experiments do not establish the broader applicability of the proposed setting. The formulation assumes that questions have known subject labels, clients can independently solve the shared questions, and rewards are obtained by checking the equivalence of final answers (Section 3). The paper does not adequately explain which deployment constraints make this feedback interface necessary, whether clients may share their answers or only scalar rewards, and how these constraints distinguish the setting from learning directly from the outputs of multiple teachers. This distinction determines which alternatives are feasible and what practical problem the method solves. The empirical study is limited to a single 1.5B mathematical server, 2,000 MATH training problems, and eight clients represented by fixed cached answers (Section 5.1; Appendix A). All six evaluation benchmarks concern mathematics. IID-control redistributes the same answer pool across client identities rather than testing a new population with different capabilities or error correlations. Furthermore, Appendix A identifies four families of client training subsets but does not sufficiently specify the clients' architectures, sizes, or training procedures to assess the motivating large-server/small-specialist relationship. The experiments thus provide a limited proof of concept, without establishing the conditions under which the method transfers to other model capability regimes or to tasks for which final-answer agreement is not a suitable reward.

  2. The methodological advance over existing selection mechanisms is limited, and the paper does not demonstrate a transferable reliability mechanism. FedGRPO already provides collaborative group-relative learning and client selection [1]. DeepConf introduces whole-trace, tail, minimum-window, and bottom-window confidence measures [2]; GFPO filters candidate groups before computing relative advantages [3]; and LASeR adapts reward-model selection during training [4]. The paper acknowledges these works, and TSRM's particular combination without labeled expert assignments is a concrete distinction. However, the GRPO objective remains unchanged (Section 4.3), and the experiments do not isolate the added value of the custom confidence combination, variability penalty, or full-group profile updates compared with simpler alternatives. The two single-module ablations establish the modules' effects in the chosen configuration, but do not explain why the proposed coordination should work more generally. More fundamentally, the routing profile favors clients whose answers consistently agree with those of the evolving server, potentially favoring shared systematic errors over correct dissent. Although the paper acknowledges this limitation, Figure 3 reports participation frequencies rather than showing whether selected clients are more accurate or provide fewer incorrect positive rewards. Without these diagnostics and comparisons under matched sampling and feedback budgets, the evidence supports an incremental training recipe rather than a sufficiently established general approach to two-sided reliability.

[1] FedGRPO: Privately Optimizing Foundation Models with Group-Relative Rewards from Domain Clients.

[2] Deep Think with Confidence.

[3] Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning.

[4] LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Bandits.

Questions for authors
  1. What application motivates this feedback interface, and are clients allowed to share generated answers or only scalar rewards? Which constraint rules out learning directly from client outputs?

  2. How does the method behave when the server becomes more accurate than most clients, or when clients exhibit shared systematic errors?

  3. With matched candidate and client-query budgets, what additional value do the custom confidence combination and full-group profile updates offer over simpler filtering-and-routing alternatives [1; 2]?

  4. In the full TSRM framework, are selected clients more accurate than uniformly sampled clients, and do they reduce incorrect positive rewards rather than simply increase server–client agreement?

[1] Deep Think with Confidence.

[2] Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning.

Reviewer_9p75OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TSRM studies collaborative reinforcement post-training in which distributed client solvers provide rewards through agreement between their cached answers and server-generated candidate answers. It combines Multi-Granularity Candidate Confidence Filtering (MG-CCF) with Stability-Aware Subject-Profile Routing (SA-SPR). MG-CCF scores full traces, tails, and low-confidence windows using top-token probability and entropy, then retains seven of eight candidates after an initial stage. SA-SPR tracks smoothed client-subject agreement and variability, combines the resulting profile with uniform exploration, and samples two evaluators per question. on Qwen2.5-Math-1.5B, the proposed method with eight cached clients, 2,000 MATH training problems, and three seeds, TSRM improves the average over six math benchmarks

Critical strengths
  • the paper discusses how candidate filtering and evaluator routing affect the same policy update. Profiles use full assigned-group feedback, so filtering does not mechanically change the profile observation for that group.
  • The reported gains hold across two evaluator constructions and multiple training runs. TSRM has the highest reported six-benchmark average in both settings, improving over FedGRPO-U by 2.2 points in the identity-homogenized control and 2.7 points with persistent identities. The fixed pool filter diagnostic also shows useful prioritization.
Critical weaknesses
  • The central claim that SA-SPR routes to "more reliable" evaluators is unproven. Client profiles are scored purely on agreement with solutions from the current server policy. High agreement does not imply client competence, it can simply reward clients that correlated wrong answers.
  • Section 5.4 shows that client selection frequencies vary by subject, but never checks if prioritized clients actually produce correct answers or cleaner reward signals than other clients.
  • Both components rely on elaborate heuristics, but the ablation only tests dropping entire modules. Does the multi-granularity score actually beat dropping 1 candidate at random? In GRPO-style objectives, simply reducing group size from 8 to 7 changes baseline variance and gradient.
Questions for authors

If evaluate client agreement against ground-truth labels post-training, do clients selected by SA-SPR actually have higher answer accuracy on those subjects? What happens on hard problems where multiple clients confidently share the same incorrect answer? lack of client details, What are the architectures and training recipes of the 8 client models, and how exactly are profile states initialized at cold start? if randomly dropping 1 of 8 candidates, how does TSRM and baselines behave? what if routing by global agreement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.