- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
TSRM integrates candidate filtering with evaluator routing in collaborative reinforcement post-training. MG-CCF ranks generated reasoning traces using global, tail, and local confidence statistics. SA-SPR selects clients based on subject-specific histories of answer agreement and its variability. Each candidate receives an agreement reward from a single assigned client. Complete candidate groups are used to update routing profiles, while retained candidates contribute to group-relative policy optimization. In the experiments, Qwen2.5-Math-1.5B is trained on 2,000 MATH problems using eight cached client-answer sources and evaluated on six mathematical benchmarks under persistent and redistributed client identities.
Critical strengths
-
Considering candidate and evaluator selection together is a reasonable direction when reference-answer rewards are unavailable.
-
The paper includes both single-module ablations and a diagnostic with a fixed generator. The latter provides useful evidence that confidence filtering retains more correct answers in the candidate pool examined.
Critical weaknesses
-
The practical motivation remains insufficiently developed, and the experiments do not establish the broader applicability of the proposed setting. The formulation assumes that questions have known subject labels, clients can independently solve the shared questions, and rewards are obtained by checking the equivalence of final answers (Section 3). The paper does not adequately explain which deployment constraints make this feedback interface necessary, whether clients may share their answers or only scalar rewards, and how these constraints distinguish the setting from learning directly from the outputs of multiple teachers. This distinction determines which alternatives are feasible and what practical problem the method solves. The empirical study is limited to a single 1.5B mathematical server, 2,000 MATH training problems, and eight clients represented by fixed cached answers (Section 5.1; Appendix A). All six evaluation benchmarks concern mathematics. IID-control redistributes the same answer pool across client identities rather than testing a new population with different capabilities or error correlations. Furthermore, Appendix A identifies four families of client training subsets but does not sufficiently specify the clients' architectures, sizes, or training procedures to assess the motivating large-server/small-specialist relationship. The experiments thus provide a limited proof of concept, without establishing the conditions under which the method transfers to other model capability regimes or to tasks for which final-answer agreement is not a suitable reward.
-
The methodological advance over existing selection mechanisms is limited, and the paper does not demonstrate a transferable reliability mechanism. FedGRPO already provides collaborative group-relative learning and client selection [1]. DeepConf introduces whole-trace, tail, minimum-window, and bottom-window confidence measures [2]; GFPO filters candidate groups before computing relative advantages [3]; and LASeR adapts reward-model selection during training [4]. The paper acknowledges these works, and TSRM's particular combination without labeled expert assignments is a concrete distinction. However, the GRPO objective remains unchanged (Section 4.3), and the experiments do not isolate the added value of the custom confidence combination, variability penalty, or full-group profile updates compared with simpler alternatives. The two single-module ablations establish the modules' effects in the chosen configuration, but do not explain why the proposed coordination should work more generally. More fundamentally, the routing profile favors clients whose answers consistently agree with those of the evolving server, potentially favoring shared systematic errors over correct dissent. Although the paper acknowledges this limitation, Figure 3 reports participation frequencies rather than showing whether selected clients are more accurate or provide fewer incorrect positive rewards. Without these diagnostics and comparisons under matched sampling and feedback budgets, the evidence supports an incremental training recipe rather than a sufficiently established general approach to two-sided reliability.
[1] FedGRPO: Privately Optimizing Foundation Models with Group-Relative Rewards from Domain Clients.
[2] Deep Think with Confidence.
[3] Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning.
[4] LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Bandits.
Questions for authors
-
What application motivates this feedback interface, and are clients allowed to share generated answers or only scalar rewards? Which constraint rules out learning directly from client outputs?
-
How does the method behave when the server becomes more accurate than most clients, or when clients exhibit shared systematic errors?
-
With matched candidate and client-query budgets, what additional value do the custom confidence combination and full-group profile updates offer over simpler filtering-and-routing alternatives [1; 2]?
-
In the full TSRM framework, are selected clients more accurate than uniformly sampled clients, and do they reduce incorrect positive rewards rather than simply increase server–client agreement?
[1] Deep Think with Confidence.
[2] Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning.