- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a two-stage training method that first solves for the optimal probability ratios and then distills them into the language model policy through training, thereby avoiding the need for behavior-policy probabilities. The experimental results are promising, and the proposed method outperforms existing baselines.
Critical strengths
-
The proposed method is relatively intuitive: it first solves for the optimal probability ratios at low computational cost and then performs training.
-
The proposed method is relatively clean and easy to reproduce.
-
The experimental results demonstrate that the proposed method is indeed effective.
Critical weaknesses
-
The methodological formulation is unreasonable and potentially misleading. First, Stage 1 is actually performed with finite samples, so the expectations in Equation 3 are effectively empirical averages. Although the authors explain this replacement, the expectation-based formulation in Equation 3 can easily mislead readers. Second, under finite sampling, the constraint that the average probability ratio equals 1 is confusing. It does not necessarily need to hold and may interfere with policy improvement for actions outside the sampled set.
-
The Stage 2 objective function is problematic. First, normalization by $ |y| $ may prevent $ \pi_\theta $ from converging to the target specified by $ \rho^* $. Furthermore, tricks such as clamping and negative subsampling further hinder this convergence. Therefore, TRIBE is not actually solving for the optimal policy in the traditional sense.
-
The experiments are not sufficiently convincing. First, all models are small models below 7B. Second, the scores of RLOO in Table 2 and SimPO in Table 3 are unusually low. The authors need to analyze the reasons and investigate whether incorrect hyperparameter settings caused these results. In addition, Table 1 reports different Pass@k metrics for the two datasets, which is somewhat strange.
Questions for authors
TRIBE is an offline-oriented RL algorithm, yet it achieves higher scores than DAPO in Table 1. What do you think explains this?
4 comments
Response by Authors [2/2]
Q1. Why above DAPO in Table 1? TRIBE is not offline-oriented by design. Table 1 runs it fully on-policy, recomputing Stage 1 from fresh responses at every update. Its slightly higher mean does not establish superiority over DAPO. The result is that it matches strong RLVR methods while the same ratio-space method works off-policy and offline.
Closing statement. We thank the reviewer for taking the time to review our paper and for their constructive feedback. While the soundness rating of 1 reflects substantial concerns, we would also like to highlight that the review notes TRIBE "outperforms existing baselines," is "indeed effective," and is "relatively clean and easy to reproduce." We have aimed to address the concerns underlying this rating, particularly the relationship between the practical Stage 2 objective and the theory, which is now formally characterized in App. F. Theorem 5 provides a conditional performance bound for any returned policy, with the fitting procedure entering through the projection error. Sec. 3.1 states the finite-sample formulation of Stage 1, and the revision adds ablations of the practical Stage 2 components (App. G) and reports complete baseline hyperparameter sweeps (Table 6). Given these additional analyses and experiments, we believe these concerns have been substantially addressed. We request that the reviewer reconsider their assessment and overall score in light of these revisions, and would appreciate any clarification on concerns that remain unaddressed.
Thank you for the response. However, my main concerns remain unresolved. The authors justify their approach by arguing that similar issues also exist in methods such as GRPO (primarily W1–W2). However, GRPO's pioneering contribution to RLVR outweighs its limitations in other aspects. Indeed, several limitations of GRPO, including length normalization (addressed by DAPO, Dr-GRPO) and biased advantage estimation (addressed by RLOO and Dr-GRPO), have been widely recognized and subsequently improved upon. More recent rigorous works generally seek to avoid these issues.
More importantly, while most of the authors' theoretical analysis is conducted at the level of expectations, the actual algorithm incorporates sample normalization, length normalization, and other operations that can substantially alter the resulting objective. Such discrepancies are generally acceptable if the practical objective is an unbiased Monte Carlo estimator of the theoretical one, as is common in sample-based RL methods. However, the authors have not sufficiently established this property.
Therefore, I am inclined to maintain my current score unless the authors can provide a rigorous theoretical proof that the proposed estimator is unbiased.
The concerns rest on a misreading of the method. The reviewer conflates two different things.
Stage 1 is solved on the batch itself. Sample normalization is not a post-hoc operation applied to an estimator of the population objective. The algorithm directly defines and solves the empirical constrained program on the sampled responses (Sec. 3.1). Its normalization constraint, which makes the ratios sum to over each prompt's sampled responses (Eq. 3c), is the empirical counterpart of the policy-normalization condition in Lemma 1 and makes the ratios define a normalized redistribution over the sampled support. The sample normalization is therefore required by the empirical problem, not an approximation artifact. With a per-batch budget, its solution centers advantages per group and scales them by a pooled standard deviation, yielding the Dr. GRPO-style normalization. It follows from the trust-region program rather than being added as a heuristic.
Stage 2 modifies the projection objective by design, and Appendix F derives each modification. Without length normalization, subtracting the baseline removes exactly the behavior-cloning term, a zero-mean finite-batch score component, and preserves the expected gradient at the behavior policy (Prop. 12). Clamping makes the objective bounded below (Prop. 9). Negative subsampling yields, in expectation, a mixture of the signed objective and a forward-KL projection (Prop. 10). Length normalization defines the per-token fitting objective, and its effect is ablated in Appendix G.
Unbiasedness is not a property TRIBE claims. The reviewer does not identify a theoretical claim in the paper that requires TRIBE's practical objective to be an unbiased estimator of the unmodified projection objective. The paper explicitly states the practical objectives it optimizes and derives the effect of each departure from the idealized projection. Thus, the requested unbiasedness condition is an additional criterion being imposed by the review, rather than a missing premise of TRIBE's analysis.
Response by Authors [1/2]
W1. Is finite-sample Stage 1 unreasonable or misleading? Eq. 3 gives the population expectation, as standard RL objectives do, while Sec. 3.1 describes the sampled problem actually solved. PPO and GRPO also present expectation objectives estimated from samples. Under the empirical group measure, the decision vector lies in RNK and normalization becomes K1∑kρik=1 for each prompt.
The normalization condition ensures that ρ defines a reweighted distribution over the sampled responses, analogous to the population condition in Lemma 1. True importance ratios average to one in expectation but not necessarily in a finite sample, so normalization is imposed as an explicit constraint on the empirical problem. It redistributes mass among the sampled responses rather than scaling all ratios together, and yields the finite ratio bounds in Prop. 4. Stage 1 specifies the desired update on the sampled responses, and Stage 2 fits the parameterized policy using these weights, allowing it to generalize beyond the sampled responses, as in other sample-based policy optimization.
W2. Does practical Stage 2 converge to the target?
1) Length normalization. It makes the practical objective a length-weighted analogue rather than an exact forward-KL projection, as App. F.1 now states. The same normalization has precedent, and the distinction is not specific to TRIBE. GRPO averages token losses per response, and SimPO uses length-normalized log-likelihood for its implicit reward. The purpose is to control gradient scaling across responses of different lengths. Removing it reduces accuracy by at most about two points and raises pre-clip gradient norms by two to three orders of magnitude (Tables 10–11).
2) Clamping and subsampling. Stage 2 is not claimed to minimize KL(π⋆∥πθ) to convergence. App. F analyzes each modification. The signed practical loss removes the behavior-cloning term from the ideal projection, clamping bounds the offline loss (Prop. 9), and subsampling mixes the signed loss with a forward-KL projection onto upweighted responses in expectation (Prop. 10, Cor. 11). App. G isolates each component. Baseline removal reduces accuracy by 29.4 GSM8K and 10.1 MATH points offline and collapses on-policy training (Tables 8–9, Fig. 4). Intermediate subsampling outperforms both endpoints (Fig. 5), clamping improves accuracy by 2.5–5.3 points in three of four settings (Table 12), and length normalization lowers pre-clip gradient norms by two to three orders of magnitude (Table 11). These changes improve empirical performance and training stability.
Importantly, Theorem 5 provides a conditional performance bound for any returned policy, without assuming a particular fitting procedure. Stage 2's fit enters only through ϵproj=E[KL(π⋆∥πθ)]. Unlike PPO and GRPO, which optimize surrogate objectives through parameterized gradient updates, TRIBE explicitly solves for a nonparametric policy-improvement target over sampled responses. TRIBE solves its finite-sample constrained improvement step to optimality, while the subsequent parameterized fitting step is approximate.
W3. Are the experiments convincing?
1) Scale. Stage 1 has one scalar variable per sampled response and no dependence on LM parameter count. We test 1B and 3B learners, including a weak 1B learner trained entirely from 3B-generated data. Larger models would test generality, and we acknowledge this as a limitation of the current evaluation rather than an issue with the mathematical formulation.
2) RLOO and SimPO tuning. Every baseline was swept, with grids and selections in Table 6 (App. C.3). RLOO's poor offline results are consistent with two issues beyond hyperparameter selection. It applies an on-policy gradient estimator to another policy's data without importance correction, and unbounded negative-weight updates can continue reducing the likelihood of already unlikely responses. Applying TRIBE's clamping and subsampling to RLOO raises its GSM8K-3B accuracy from 0.48 to 67.05, yet TRIBE outperforms stabilizer-matched RLOO and REINFORCE in all four settings (Tables 2, 7). SimPO was swept over 18 controlled-offline and 13 UltraFeedback configurations, without achieving competitive performance in our setting. Its published gains use UltraChat SFT initialization or regenerated on-policy data from instruction-tuned models, rather than base-model offline learning on another policy's completions.
3) Evaluation metrics. We report Pass@1 on GSM8K and Pass@4 on the harder MATH benchmark. Pass@k beyond k = 1 is a standard metric in reasoning post-training [1]. All methods use the same metric within each dataset, so comparisons share the evaluation criterion.
[1] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, Yue et al