acceptodds
Under review as a conference paper at ICLR 2027

TRIBE: Trust-Region Policy Improvement from Batch Experience

Abstract

Post-training language models from scalar rewards is commonly approached with policy-optimization methods developed for more general sequential decision problems. In sequence-level post-training, treating a complete response as a single action allows policy improvement to be written directly in terms of how the candidate policy reweights sampled responses. This motivates , a two-stage method that first solves for the desired policy improvement in importance-ratio space and then updates the language model to realize it. The resulting formulation provides a unified approach to on-policy, off-policy, and fully offline training, including settings where the policies that generated the offline data are unavailable, and it allows additional policy constraints to be incorporated directly into the improvement step. Across GSM8K and MATH, TRIBE is competitive with strong on-policy baselines and is the strongest behavior-policy-independent method in our controlled offline experiments. On UltraFeedback, TRIBE is the only evaluated method to improve over rejection fine-tuning on both win rate and reward advantage. We also show that an expected-cost constraint can be added without changing the model-fitting stage, yielding competitive safety-constrained alignment.

Primary area:
reinforcement learning
Keywords:
Alignment, Reinforcement Learning, Bandits

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CdhfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a two-stage training method that first solves for the optimal probability ratios and then distills them into the language model policy through training, thereby avoiding the need for behavior-policy probabilities. The experimental results are promising, and the proposed method outperforms existing baselines.

Critical strengths
  1. The proposed method is relatively intuitive: it first solves for the optimal probability ratios at low computational cost and then performs training.

  2. The proposed method is relatively clean and easy to reproduce.

  3. The experimental results demonstrate that the proposed method is indeed effective.

Critical weaknesses
  1. The methodological formulation is unreasonable and potentially misleading. First, Stage 1 is actually performed with finite samples, so the expectations in Equation 3 are effectively empirical averages. Although the authors explain this replacement, the expectation-based formulation in Equation 3 can easily mislead readers. Second, under finite sampling, the constraint that the average probability ratio equals 1 is confusing. It does not necessarily need to hold and may interfere with policy improvement for actions outside the sampled set.

  2. The Stage 2 objective function is problematic. First, normalization by $ |y| $ may prevent $ \pi_\theta $ from converging to the target specified by $ \rho^* $. Furthermore, tricks such as clamping and negative subsampling further hinder this convergence. Therefore, TRIBE is not actually solving for the optimal policy in the traditional sense.

  3. The experiments are not sufficiently convincing. First, all models are small models below 7B. Second, the scores of RLOO in Table 2 and SimPO in Table 3 are unusually low. The authors need to analyze the reasons and investigate whether incorrect hyperparameter settings caused these results. In addition, Table 1 reports different Pass@k metrics for the two datasets, which is somewhat strange.

Questions for authors

TRIBE is an offline-oriented RL algorithm, yet it achieves higher scores than DAPO in Table 1. What do you think explains this?

4 comments
AuthorsOpenReview ↗

Response by Authors [1/2]

W1. Is finite-sample Stage 1 unreasonable or misleading? Eq. 3 gives the population expectation, as standard RL objectives do, while Sec. 3.1 describes the sampled problem actually solved. PPO and GRPO also present expectation objectives estimated from samples. Under the empirical group measure, the decision vector lies in RNK\mathbb R^{NK} and normalization becomes 1K∑kρik=1\frac{1}{K}\sum_k\rho_{ik}=1 for each prompt.

The normalization condition ensures that ρ\rho defines a reweighted distribution over the sampled responses, analogous to the population condition in Lemma 1. True importance ratios average to one in expectation but not necessarily in a finite sample, so normalization is imposed as an explicit constraint on the empirical problem. It redistributes mass among the sampled responses rather than scaling all ratios together, and yields the finite ratio bounds in Prop. 4. Stage 1 specifies the desired update on the sampled responses, and Stage 2 fits the parameterized policy using these weights, allowing it to generalize beyond the sampled responses, as in other sample-based policy optimization.

W2. Does practical Stage 2 converge to the target?

1) Length normalization. It makes the practical objective a length-weighted analogue rather than an exact forward-KL projection, as App. F.1 now states. The same normalization has precedent, and the distinction is not specific to TRIBE. GRPO averages token losses per response, and SimPO uses length-normalized log-likelihood for its implicit reward. The purpose is to control gradient scaling across responses of different lengths. Removing it reduces accuracy by at most about two points and raises pre-clip gradient norms by two to three orders of magnitude (Tables 10–11).

2) Clamping and subsampling. Stage 2 is not claimed to minimize KL(π⋆∥πθ)\mathrm{KL}(\pi^\star\|\pi_\theta) to convergence. App. F analyzes each modification. The signed practical loss removes the behavior-cloning term from the ideal projection, clamping bounds the offline loss (Prop. 9), and subsampling mixes the signed loss with a forward-KL projection onto upweighted responses in expectation (Prop. 10, Cor. 11). App. G isolates each component. Baseline removal reduces accuracy by 29.4 GSM8K and 10.1 MATH points offline and collapses on-policy training (Tables 8–9, Fig. 4). Intermediate subsampling outperforms both endpoints (Fig. 5), clamping improves accuracy by 2.5–5.3 points in three of four settings (Table 12), and length normalization lowers pre-clip gradient norms by two to three orders of magnitude (Table 11). These changes improve empirical performance and training stability.

Importantly, Theorem 5 provides a conditional performance bound for any returned policy, without assuming a particular fitting procedure. Stage 2's fit enters only through ϵproj=E[KL(π⋆∥πθ)]\epsilon_{\mathrm{proj}}=\mathbb E[\mathrm{KL}(\pi^\star\|\pi_\theta)]. Unlike PPO and GRPO, which optimize surrogate objectives through parameterized gradient updates, TRIBE explicitly solves for a nonparametric policy-improvement target over sampled responses. TRIBE solves its finite-sample constrained improvement step to optimality, while the subsequent parameterized fitting step is approximate.

W3. Are the experiments convincing?

1) Scale. Stage 1 has one scalar variable per sampled response and no dependence on LM parameter count. We test 1B and 3B learners, including a weak 1B learner trained entirely from 3B-generated data. Larger models would test generality, and we acknowledge this as a limitation of the current evaluation rather than an issue with the mathematical formulation.

2) RLOO and SimPO tuning. Every baseline was swept, with grids and selections in Table 6 (App. C.3). RLOO's poor offline results are consistent with two issues beyond hyperparameter selection. It applies an on-policy gradient estimator to another policy's data without importance correction, and unbounded negative-weight updates can continue reducing the likelihood of already unlikely responses. Applying TRIBE's clamping and subsampling to RLOO raises its GSM8K-3B accuracy from 0.48 to 67.05, yet TRIBE outperforms stabilizer-matched RLOO and REINFORCE in all four settings (Tables 2, 7). SimPO was swept over 18 controlled-offline and 13 UltraFeedback configurations, without achieving competitive performance in our setting. Its published gains use UltraChat SFT initialization or regenerated on-policy data from instruction-tuned models, rather than base-model offline learning on another policy's completions.

3) Evaluation metrics. We report Pass@1 on GSM8K and Pass@4 on the harder MATH benchmark. Pass@k beyond k = 1 is a standard metric in reasoning post-training [1]. All methods use the same metric within each dataset, so comparisons share the evaluation criterion.

[1] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, Yue et al

AuthorsOpenReview ↗

Response by Authors [2/2]

Q1. Why above DAPO in Table 1? TRIBE is not offline-oriented by design. Table 1 runs it fully on-policy, recomputing Stage 1 from fresh responses at every update. Its slightly higher mean does not establish superiority over DAPO. The result is that it matches strong RLVR methods while the same ratio-space method works off-policy and offline.

Closing statement. We thank the reviewer for taking the time to review our paper and for their constructive feedback. While the soundness rating of 1 reflects substantial concerns, we would also like to highlight that the review notes TRIBE "outperforms existing baselines," is "indeed effective," and is "relatively clean and easy to reproduce." We have aimed to address the concerns underlying this rating, particularly the relationship between the practical Stage 2 objective and the theory, which is now formally characterized in App. F. Theorem 5 provides a conditional performance bound for any returned policy, with the fitting procedure entering through the projection error. Sec. 3.1 states the finite-sample formulation of Stage 1, and the revision adds ablations of the practical Stage 2 components (App. G) and reports complete baseline hyperparameter sweeps (Table 6). Given these additional analyses and experiments, we believe these concerns have been substantially addressed. We request that the reviewer reconsider their assessment and overall score in light of these revisions, and would appreciate any clarification on concerns that remain unaddressed.

Thank you for the response. However, my main concerns remain unresolved. The authors justify their approach by arguing that similar issues also exist in methods such as GRPO (primarily W1–W2). However, GRPO's pioneering contribution to RLVR outweighs its limitations in other aspects. Indeed, several limitations of GRPO, including length normalization (addressed by DAPO, Dr-GRPO) and biased advantage estimation (addressed by RLOO and Dr-GRPO), have been widely recognized and subsequently improved upon. More recent rigorous works generally seek to avoid these issues.

More importantly, while most of the authors' theoretical analysis is conducted at the level of expectations, the actual algorithm incorporates sample normalization, length normalization, and other operations that can substantially alter the resulting objective. Such discrepancies are generally acceptable if the practical objective is an unbiased Monte Carlo estimator of the theoretical one, as is common in sample-based RL methods. However, the authors have not sufficiently established this property.

Therefore, I am inclined to maintain my current score unless the authors can provide a rigorous theoretical proof that the proposed estimator is unbiased.

The concerns rest on a misreading of the method. The reviewer conflates two different things.

Stage 1 is solved on the batch itself. Sample normalization is not a post-hoc operation applied to an estimator of the population objective. The algorithm directly defines and solves the empirical constrained program on the sampled responses (Sec. 3.1). Its normalization constraint, which makes the ratios sum to KK over each prompt's sampled responses (Eq. 3c), is the empirical counterpart of the policy-normalization condition in Lemma 1 and makes the ratios define a normalized redistribution over the sampled support. The sample normalization is therefore required by the empirical problem, not an approximation artifact. With a per-batch χ2\chi^2 budget, its solution centers advantages per group and scales them by a pooled standard deviation, yielding the Dr. GRPO-style normalization. It follows from the trust-region program rather than being added as a heuristic.

Stage 2 modifies the projection objective by design, and Appendix F derives each modification. Without length normalization, subtracting the baseline removes exactly the behavior-cloning term, a zero-mean finite-batch score component, and preserves the expected gradient at the behavior policy (Prop. 12). Clamping makes the objective bounded below (Prop. 9). Negative subsampling yields, in expectation, a mixture of the signed objective and a forward-KL projection (Prop. 10). Length normalization defines the per-token fitting objective, and its effect is ablated in Appendix G.

Unbiasedness is not a property TRIBE claims. The reviewer does not identify a theoretical claim in the paper that requires TRIBE's practical objective to be an unbiased estimator of the unmodified projection objective. The paper explicitly states the practical objectives it optimizes and derives the effect of each departure from the idealized projection. Thus, the requested unbiasedness condition is an additional criterion being imposed by the review, rather than a missing premise of TRIBE's analysis.

Reviewer_WdYmOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a policy optimization method for the RL objective that operates in two stages: first computing optimal likelihood ratios under a trust-region constraint, and then updating the parameterized policy using weights derived from those ratios. The method is presented as applicable to on-policy, off-policy and offline regimes, and is evaluated across these regimes on mathematical reasoning and alignment datasets.

Critical strengths

The ratio-space formulation provides a common framework for different trust-region divergences and allows additional expected-cost constraints to be incorporated into the policy-improvement stage.

Critical weaknesses

The main concern is the disconnection between the practical algorithm and the developed theory.

First, subtracting 1 from the optimal likelihood ratio leaves the population gradient unchanged when πθ=πb\pi_\theta=\pi_b, but in other settings this modification significantly changes the objective. The new objective becomes −Eπb[(ρ∗−1)log⁡πθ]=DKL(q∥πθ)−DKL(πb∥πθ)+constant-\mathbb E_{\pi_b}[(\rho^*-1)\log\pi_\theta] = D_{\mathrm{KL}}(q\|\pi_\theta) - D_{\mathrm{KL}}(\pi_b\|\pi_\theta) +\text{constant} instead of DKL(q∥πθ)D_{\mathrm{KL}}(q\|\pi_\theta), therefor significantly changing the optimization. Moreover, the argument for that change, namely "every sampled response receives a nonnegative coefficient, so the objective can promote responses but provides no explicit signal to demote them", is not convincing since positive-weight maximum likelihood can already decrease response probabilities through the normalization of the model (consider a simple forward KL training, the optimal distribution matches the target one even though all the coefficients multiplying the log probabilities are nonnegative).

Second, dividing the logprob by the completion length further disconnects the optimized objective from the theoretical one. Moreover, given the length normalization, the previous issue with subtracting 1 from the optimal likelihood ratio becomes even more critical and makes that operation invalid even in the on-policy setting.

The two issues from above already make the practical algorithm largely disconnected from the theoretical idea behind the method, however, the off-policy and offline versions add even more arbitrary tricks, namely probabilities clamping and negative subsampling. Together, these modifications have two main consequences: (1) the method becomes mostly empirical (and thus requires a more careful and comprehensive empirical evaluation and justification), and (2) the empirical evaluation mixes the effect of all the modification without isolating the contribution of each one, thus limiting the amount of useful information obtained from the experimental results. The paper, on the contrary, has some significant issues with the baselines (that are described below) and does not isolate the effects of subtracting 1, length normalization, and probability clamping.

Baselines:

The main issue with the baselines is that DPO is considered to be behavior-policy-dependent, while it's inherently not. The paper mixes the properties of requiring a reference model and the behavior policy while these are distinct requirements. DPO does require a reference policy (which can be chosen as the starting checkpoint), but can be successfully applied in fully offline setting [1]. Given this correction, DPO appears to be in the same group as TRIBE in the off-policy experiments which contradicts the claim that TRIBE is the strongest behavior-independent method across all settings, and even more importantly, DPO is a missing baseline in the Ultrafeedback experiments that is a serious issue since DPO is considered to be a strong baseline (DPO on UltraFeedback is an established recipe [2]).

The second concern regarding the baselines is that they seem to be not tuned enough. DPO, SimPO (and potentially other methods) are known to be very sensitive to learning rates, beta and other hyperparameters [3,6], while the search ranges are not reported in the paper and the same selected hyperparameters for both 1B and 3B model suggest that there was no or very limited hyperparameter search for those baselines. Another evidence of the same concern is that SimPO performs below the untrained base model on UltraFeedback that conflicts with the significant improvement on the same dataset reported in the SimPO original paper [3]. If the lack of hyperparameters tuning is indeed the case, it's a serious limitation that calls the interpretation into question.

Inconsistencies:

Global versus per-prompt trust-region budgets. Section 3.1 explicitly specifies one global multiplier λ\lambda for the batch. However, Equation 5 and its derivation allocate the full budget δ\delta separately to each prompt. These are different optimization problems. The tightened ratio bounds in Proposition 4 and Appendix E likewise assume a per-prompt budget. This is a substantive mathematical inconsistency.

Use of the χ2\chi^2 closed form outside its guaranteed positivity regime. The developed theory requires δ<1/(K−1)\delta<1/(K-1), while the offline experiments use δ=0.5\delta=0.5 with K=4,16,32K=4,16,32.

Bottomline:

Given all the concerns above, the paper lacks justification of the contribution. Why would one use TRIBE if it's on par with other baselines in the on-policy setting, and given that one is not convinced by the off-policy and offline settings results? Also there exist other methods that can be applied across on-policy, off-policy and offline settings (e.g. DPO and iterative DPO) and show competitive results in those regimes [2,4,5,6], as well as allow for additional constraints to be incorporated into the objective (C-DPO[7]).

[1] Team Olmo et al. Olmo 3. arXiv:2512.13961, 2025.

[2] Lewis Tunstall et al. Zephyr: Direct Distillation of LM Alignment. COLM, 2024.

[3] Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple Preference Optimization with a Reference-Free Reward. NeurIPS, 2024.

[4] Wei Xiong et al. Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint. ICML, 2024.

[5] Songjun Tu et al. Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation. COLM, 2025.

[6] Jack Lanchantin et al. Bridging Offline and Online Reinforcement Learning for LLMs. arXiv:2506.21495, 2025.

[7] Zixuan Liu, Xiaolin Sun, and Zizhan Zheng. Enhancing LLM Safety via Constrained Direct Preference Optimization. arXiv:2403.02475, 2024.

Questions for authors
  1. Is the divergence budget enforced across the entire batch or separately for each prompt? How are Equation 5 and Proposition 4 reconciled with the global multiplier described in Section 3.1?
  2. How many policy updates are performed on each sampled batch?
  3. Can the authors provide ablations for subtracting 1, length normalization and clamping, alongside the existing negative-subsampling ablation?
  4. What hyperparameter ranges, tuning budgets and selection criteria were used for each baseline and model size? Which reference checkpoint was used for DPO?
  5. Can DPO be included in the UltraFeedback experiment, and online DPO in the online comparison?
4 comments
AuthorsOpenReview ↗

Response by Authors [1/2]

W1. Is the practical algorithm disconnected from the theory?

1) Subtracting the baseline. As the review notes, the practical loss is the projection minus behavior cloning, KL(π⋆∥πθ)−KL(πb∥πθ)\mathrm{KL}(\pi^\star\|\pi_\theta)-\mathrm{KL}(\pi_b\|\pi_\theta) (Eq. 19, App. F.1). Their expected gradients agree at πb\pi_b, where every on-policy update begins (M=1M=1). Positive-weight maximum likelihood can indirectly lower a probability through normalization, but gives no explicit negative coefficient to a sampled response with 0<ρ⋆<10<\rho^\star<1. Subtracting one directly demotes exactly those responses. Without the subtraction, offline accuracy drops 29.4 GSM8K and 10.1 MATH points, and on-policy training collapses (App. G.2, Tables 8–9, Fig. 4).

2) Length normalization. Like GRPO, TRIBE uses group-centered weights and response-level length normalization. While this changes the exact theoretical objective, the same theory–implementation distinction exists in GRPO and is not specific to TRIBE (App. F.1). Removing it reduces accuracy by at most about two points and raises pre-clip gradient norms two to three orders of magnitude (App. G.4, Tables 10–11). It primarily improves optimization conditioning.

3) Clamping and subsampling. TOPR also limits negative push-down, and Hong et al. rebalance negatives. App. F shows how clamping bounds the loss (Prop. 9) and, in expectation, subsampling interpolates between the signed loss and a positive-weight forward-KL projection toward upweighted responses (Prop. 10, Cor. 11). App. G isolates every component (Tables 8–12, Fig. 5). Intermediate fnegf_{\mathrm{neg}} outperforms both endpoints; removing clamping reduces accuracy by 2.5–5.3 points in three of four settings. These stabilizers lift RLOO from 0.48 to 67.05 on GSM8K-3B, yet TRIBE still outperforms stabilizer-matched RLOO and REINFORCE in all four settings (Tables 2, 7).

W2. Baseline grouping and tuning.

1) DPO and UltraFeedback. In DPO's original RLHF instantiation, the reference is the policy that generated the preference data. Rafailov et al. (NeurIPS 2023) sample response pairs from the SFT model and set πref\pi_{\text{ref}} to it (Sec. 3, Eq. 3, and Sec. 4, "DPO outline"), and when that policy is unavailable, they construct a proxy to reduce the shift between "the true reference distribution which is unavailable, and πref\pi_{\text{ref}} used by DPO" (Sec. 4, "DPO outline"). This matters because the choice of reference changes the implicit reward parameterization and the policy's KL anchor, potentially affecting performance outside the preference-data distribution. Accordingly, in Table 2, where the behavior policy μ\mu is known, DPO uses πref=μ\pi_{\text{ref}} = \mu and so has access to likelihoods under the data-generating policy, whereas offline TRIBE requires neither behavior-policy nor reference-policy likelihoods. Even under the reviewer's grouping, TRIBE outperforms DPO in three of the four settings.

As requested, Table 3 adds DPO on UltraFeedback with the starting checkpoint as reference and the same base model, data and one-epoch protocol as other methods. It is on par with RFT, above RLOO and SimPO, but below TRIBE on all four metrics, including 10.8% versus 14.5% win rate against Best-of-4. The cited works [1,2] use DPO within multistage pipelines. Zephyr's ablation shows that applying direct DPO to a base model produces substantially lower scores than the full pipeline, 4.76 versus 7.00 on MT-Bench and 30.76 versus 86.07 on AlpacaEval (Tunstall et al., Table 3).

2) Hyperparameter tuning. Every method was swept, with the search grids and selected hyperparameters reported in Table 6 (App. C.3). SimPO was tried in 18 controlled-offline configurations and 13 UltraFeedback configurations, but none achieved competitive performance in our setting. Its published UltraFeedback gains use prior UltraChat fine-tuning or regenerated on-policy data from instruction-tuned models (Meng et al., 2024), which differs from our fully offline setting, where a base model is trained on fixed completions generated by other models.

W3. Theory and implementation. The reviewer correctly identifies the budget distinction. Eq. 5 and Prop. 4 use per-prompt budgets, but implementation uses one per batch, giving a shared multiplier and pooled within-group advantage standard deviation. The revision states this after Cor. 3 and in App. D.3. The closed form is never used outside its positivity regime. When Eq. 15 fails, cvxpy solves Stage 1 under ρ≥0\rho\ge0, so no negative ratios occur.

AuthorsOpenReview ↗

Reponse by Authors [2/2]

W4. Why TRIBE rather than DPO or iterative DPO? TRIBE solves improvement in ratio space and then fits the resulting target to the model. This improve-then-project approach (REPS, MPO) has received comparatively little attention in LLM post-training. For sequence-level bandits, the policy-improvement objective is exact rather than a surrogate, and Stage 1 solves it exactly on the sampled responses. The χ2\chi^2 formulation also recovers GRPO-style advantage weighting as a special case (Cor. 3). One scalar-reward formulation covers on-policy, off-policy and fully offline training. Offline it needs neither preference pairs nor behavior- or reference-policy log-probabilities.

While the reviewer has expressed reservations about our offline results, the concerns raised about baseline grouping, hyperparameter tuning and missing comparisons are each addressed above (W2), with additional experiments where needed. TRIBE is the strongest evaluated method that requires neither behavior-policy nor reference-policy log-probabilities in all controlled offline settings (Table 2). Every method was swept (Table 6). TRIBE outperforms DPO in three of four controlled settings (Table 2), stabilizer-matched baselines in all four (Table 7), and DPO with the suggested reference on all four UltraFeedback metrics (Table 3). We would appreciate clarification on which specific concerns remain unaddressed by these results.

The main distinction is how these capabilities are incorporated into a single formulation. The point is not specific to DPO. Any optimizer, including DPO, PPO or GRPO, can be extended with online updates or constraints through additional mechanisms, as iterative DPO, C-DPO and Safe RLHF's PPO-Lagrangian do. TRIBE puts constraints directly in convex Stage 1, leaves Stage 2 unchanged, and needs no behavior- or reference-policy log-probabilities offline (Sec. 4.4). This provides a unified framework for these settings without requiring separate policy-improvement objectives. Matching GRPO on-policy is the intended behavior and should not be seen as a limitation.

Q1. Budget and Eq. 5/Prop. 4. See W3. The implementation uses one batch budget and pooled Dr. GRPO-style normalization; the paper now states this after Cor. 3 and in App. D.3.

Q2. Updates per batch. M=1M=1 (App. C.1).

Q3. Component ablations. App. G now includes baseline removal (Tables 8–9, Fig. 4), length-normalization removal (Tables 10–11), clamping removal (Table 12) and the subsampling sweep (Fig. 5).

Q4. Tuning and DPO reference. Table 6 and App. C.3 give all grids and selected values. δ\delta was fixed, not tuned, and fnegf_{\mathrm{neg}} came from held-out 10% validation. DPO uses known μ\mu in Table 2 and the starting checkpoint in Table 3.

Q5. DPO on UltraFeedback and online DPO. Table 3 now includes DPO on UltraFeedback (W2.1). Table 1 now includes online DPO in the on-policy setting, where TRIBE outperforms it by about 10 points on GSM8K and 6 on MATH.

Closing statement. The revision addresses the main concerns raised, with Stage 2 analysis (App. F), component ablations (App. G), matched stabilizers (Tables 2, 7), DPO on UltraFeedback (Table 3), sweeps (Table 6) and clarification of the implemented batch-level formulation. Given these analyses and experiments, we believe the concerns underlying the soundness rating of 1 have been substantially addressed. The informativeness rating of 1 also warrants reconsideration, as the review itself credits the common framework for trust-region divergences and the incorporation of expected-cost constraints into policy improvement, both central contributions of the paper. We request that the reviewer reconsider the ratings and overall score, and would appreciate clarification on any concerns that remain unaddressed.

Thank you for the detailed response and for providing some implementation clarification as well as adding some ablations and additional baselines.

However, I have to flag that hyperparameter search for DPO baseline is not acceptable since it uses the fixed β=0.1\beta=0.1 parameter, while this parameter is one of the most important for the performance of DPO and according to my experience, as well as according to prior works [1,2], β=0.1\beta=0.1 is far too conservative for a single-task training regime. Therefore, the reported results do not establish the performance of a sufficiently tuned DPO baseline.

Now regarding the rationale behind various empirical tweaks (const baseline subtraction, length normalization, clamping and subsampling) that the authors presented in their response. I understand that some tweaks may seem to be motivated and may lead to practical improvements. However, as I stated in my review, after the application of the mentioned tweaks, the method becomes mostly empirical. There is nothing wrong with theory inspired but mostly empirical methods, but such methods require way more meticulous empirical evaluation and justification, that is not the case here (especially given my concern above). And referring to the other empirical methods (such as GRPO) that adopt similar tweaks and show high performance, can not be counted as a justification due to the empirical nature of the considered tweaks.

I would maintain my scores.

[1] Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple Preference Optimization with a Reference-Free Reward. NeurIPS, 2024.

[2] Simon Matrenok, Skander Moalla, and Caglar Gulcehre. Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functions. NeurIPS, 2025.

AuthorsOpenReview ↗

Author Response

Q1. The DPO baseline uses a fixed β = 0.1, which is too conservative for a single-task training regime, so the results do not establish a sufficiently tuned DPO baseline.

We swept β ∈ {0.01, 0.05, 0.1, 0.3, 1.0}, selecting on a held-out validation split on GSM8K and MATH. In each of the five settings below, the selected value performs worse than the paper's β = 0.1.

SettingMetricSelected βSelected β resultPaper (β = 0.1)
Online, GSM8K-3BPass@10.0162.4771.0 ± 1.0
Offline, GSM8K-3BPass@10.0161.8779.18 ± 0.56
Offline, GSM8K-1BPass@10.0122.3741.67 ± 1.01
Offline, MATH-3BPass@40.0538.5255.11 ± 0.92
Offline, MATH-1BPass@40.0512.4233.73 ± 0.29

Tuning β therefore does not improve on the paper's configuration, at least in these settings.

Q2. After the practical components (baseline subtraction, length normalization, clamping and subsampling), the method becomes mostly empirical and requires more meticulous evaluation, and the use of similar components in GRPO is not a justification.

TRIBE is an improve-then-project algorithm derived from policy improvement in importance-ratio space. It first solves for the optimal importance ratios under a trust region, then projects the resulting target onto the model's parameters, an improve-then-project approach that is new in LLM post-training. Stage 1 does not deviate from the theory. It solves the constrained improvement problem exactly on the sampled batch. Only the Stage 2 projection departs from its idealized form, and it does so to address the instability of the ideal projection in practice. The paper states this explicitly, and none of the theory is invalidated by it. Theorem 5 bounds the performance of any returned fitted policy through its projection error, so these practical Stage 2 modifications do not invalidate the stated guarantee. We characterize exactly what the practical projection optimizes and how it differs from the ideal one (Appendix F, Props. 9, 10 and 12), and ablate every practical addition (Appendix G). We also applied the same clamping and subsampling to RLOO and REINFORCE, and TRIBE still outperforms them, so its gains come from the method itself rather than from the shared components. These components are standard in LLM post-training or have precedent in RL, including GRPO, DAPO and TOPR, where they are accepted on empirical grounds alone. Requiring TRIBE to justify them beyond what is required of these methods applies a double standard. The reviewer does not specify what more meticulous evaluation is missing, given that every component has been ablated, so we are unable to identify what it would add beyond these ablations and controls.

Reviewer_JrAdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TRIBE treats each full response as a contextual-bandit action. Stage 1 solves a small convex problem over the importance ratios of the sampled responses (normalization, an ff-divergence budget δ\delta, and an optional KL term to a reference); with χ2\chi^2 the solution is the GRPO-style standardized advantage times δ\sqrt\delta. Stage 2 fits the LM with a likelihood weighted by ρ⋆−1\rho^\star-1. When the reference is the behavior policy, Stage 1 only needs rewards, so the method works offline with unknown generators, and a cost constraint can be added. Experiments use Llama-3.2 1B/3B on GSM8K/MATH, UltraFeedback and a safety task.

Critical strengths

The ratio-space view is clean: the improvement step is a small convex program, and the weighting follows from the choice of divergence (χ2\chi^2 gives standardized advantages, reverse KL a softmax). It is a nice way to look at the many GRPO variants. The offline version, which doesn't need the generator's probabilities, is practically useful, and it is the best behavior-independent method in Table 2.

Critical weaknesses
  1. The theory doesn't match what is run. With signed weights (ρ⋆−1\rho^\star-1), Stage 2 is not a projection onto the Stage-1 target: in population form the loss is ∑y(πb−π⋆)log⁡πθ\sum_y(\pi_b-\pi^\star)\log\pi_\theta, which is unbounded below (the paper notes this) and has no minimum at π⋆\pi^\star. So Thm. 5, which relies on a small projection error, doesn't really apply. With χ2\chi^2 weights, δ\delta just rescales the loss, i.e. it acts like a learning rate, so there is no real trust region on the model. Cor. 3 also has problems. D.3 uses a per-prompt budget while Prop. 2 and Algorithm 1 use one budget per batch, which gives a pooled standard deviation (closer to Dr. GRPO than GRPO). And the corollary needs δ<1/(K−1)\delta<1/(K-1), while all offline runs use δ=0.5\delta=0.5; with binary rewards Eq. (5) then gives negative ratios

  2. Overall, it does not seem that TRIBE improves on GRPO. On-policy (Table 1), TRIBE, GRPO, DAPO and RLOO are all within about 1.3 points of each other, which is roughly the seed noise with 3 seeds. Offline (Table 2), GRPO, which uses the behavior probabilities, is as good or slightly better in three of the four settings, and TRIBE is ahead only on GSM8K with the 3B learner. So the practical advantage comes down to not needing the behavior policy, and even that depends a lot on the negative-subsampling fraction: keeping all negatives (fneg=1f_{neg}=1), TRIBE drops to about 44% on GSM8K and 42% on MATH, well below rejection fine-tuning.

Questions for authors
  1. At δ=0.5\delta=0.5, do you use Eq. (5) as written or enforce ρ≥0\rho\ge0? Is the budget per prompt or per batch?

  2. Were fnegf_{neg} and δ\delta tuned on the test sets? Did REINFORCE and RLOO get the same clamping and subsampling?

  3. Can you compare with an MPO/AWR-style update (normalized positive weights, no −1-1 baseline), and with DPO on UltraFeedback?

4 comments
AuthorsOpenReview ↗

Response by Authors [1/2]

W1. Does the theory describe the algorithm actually run?

1) Signed weights, projection and Theorem 5. The signed loss is not KL(π⋆∥πθ)\mathrm{KL}(\pi^\star\|\pi_\theta). Without length normalization, it is that projection minus the behavior-cloning term (Eq. 19, App. F.1)

Limpl(θ)=KL(π⋆∥πθ)−KL(πb∥πθ)+const.\mathcal L_{\mathrm{impl}}(\theta)=\mathrm{KL}(\pi^\star\|\pi_\theta)-\mathrm{KL}(\pi_b\|\pi_\theta)+\mathrm{const}.

On-policy, the removed term has zero expected gradient at πθ=πb\pi_\theta=\pi_b. With one update per fresh batch (M=1M=1), the initial expected gradient matches that of the KL projection. Finite-batch gradients differ by a zero-mean score-function term in Proposition 12 (App. F.4). Offline, negative subsampling in expectation interpolates between the signed loss and the forward-KL projection onto upweighted responses (Prop. 10, App. F.3). Centering explicitly demotes responses with 0<ρ⋆<10<\rho^\star<1, to which ρ⋆\rho^\star weighted maximum likelihood assigns no negative coefficient (App. F.1).

The signed loss is unbounded below, as are REINFORCE and RLOO. On-policy, each fresh batch receives only a few updates, far from that regime. Offline, token-level clamping bounds the loss below by −log⁡(1/Cmin⁡)≈−18.4-\log(1/C_{\min})\approx-18.4 (Prop. 9, App. F.2), and removing clamping reduces accuracy by 2.5–5.3 points in three of four settings (Table 12).

Theorem 5 applies to any fitted policy. Its proof does not assume a fitting procedure, and Stage 2 enters only through ϵproj=E[KL(π⋆∥πθ)]\epsilon_{\mathrm{proj}}=\mathbb E[\mathrm{KL}(\pi^\star\|\pi_\theta)]. The practical loss is not designed to minimize this error to convergence. Removing behavior cloning gives the correct initial projection direction and removes its zero-mean batch-gradient noise (Prop. 12). Empirically, the centered loss beats the idealized projection loss by 29.4 GSM8K and 10.1 MATH points at 3B offline, while the idealized loss collapses on-policy (Tables 8–9).

2) Does δ\delta constrain a target or merely rescale an update? The δ\delta-budget constrains Stage 1's target, not the model step. When χ2\chi^2 nonnegativity is inactive, ρ⋆−1∝A−Aˉ\rho^\star-1\propto A-\bar A, so δ\delta scales the weights like a step size, as in first-order TRPO-style updates. But in our offline setting at δ=0.5\delta=0.5, nonnegativity binds and δ\delta changes which responses get zero weight, altering the target's shape (App. D.3). On-policy under reverse KL, it sets the softmax temperature λ\lambda in ρ⋆∝exp⁡(A/λ)\rho^\star\propto\exp(A/\lambda) and thus the relative response weights (Eq. 6).

3) Per-prompt versus batch budget. The reviewer is correct. App. D.3's derivation uses a per-prompt budget and yields GRPO-style group standardization. The implementation uses one batch budget and multiplier, centering advantages per group but scaling by pooled standard deviation as in Dr. GRPO. This is now stated after Corollary 3 and in App. D.3.

4) Positivity at δ=0.5\delta=0.5. No negative ratios occur. If Eq. 15 fails and nonnegativity binds, Stage 1 uses cvxpy with ρ≥0\rho\ge0 rather than the closed form. The revision states this after Corollary 3 and in App. D.3.

W2. GRPO and negative subsampling. On-policy, matching GRPO, DAPO and RLOO is consistent with our goal; we do not claim a substantial on-policy advantage over these methods. The χ2\chi^2 derivation connects TRIBE to GRPO-style advantage weighting, while the framework also covers off-policy and offline training. In our offline experiments, GRPO uses behavior-policy log-probabilities, whereas TRIBE does not. Among the evaluated methods that need neither behavior-policy nor reference-model log-probabilities, TRIBE performs best in every setting, as the review notes.

The gains cannot be attributed to fnegf_{\mathrm{neg}} alone. With TRIBE's negative subsampling and clamping applied to the baselines, RLOO and REINFORCE improve substantially, including RLOO's rise from 0.48 to 67.05 on GSM8K-3B. TRIBE continues to outperform these stabilizer-matched baselines across all four settings, by 6.9 GSM8K and 17.1 MATH points at 3B relative to matched RLOO (Tables 2, 7). We chose fnegf_{\mathrm{neg}} on validation data (App. C.3). Proposition 10 characterizes its relationship to positive-weight likelihood fitting and the signed objective. Both endpoints perform worse in the reported ablation: fneg=0f_{\mathrm{neg}}=0 removes explicit negative contributions, while fneg=1f_{\mathrm{neg}}=1 retains all of them. Intermediate retention probabilities yield higher accuracy (Fig. 5, App. G.3).

AuthorsOpenReview ↗

Response by Authors [2/2]

Q1. δ=0.5\delta=0.5 and budget. See W1.3–4. The budget is per batch, and cvxpy enforces ρ≥0\rho\ge0 when Eq. 15 fails (after Corollary 3, App. D.3).

Q2. Test tuning and matched stabilizers. δ\delta was fixed per setting, not tuned. fnegf_{\mathrm{neg}} was chosen on a held-out 10% validation split, never test. All sweeps appear in Table 6 (App. C.3). RLOO and REINFORCE now receive the same clamping and subsampling, yet TRIBE outperforms both matched baselines across all four settings (Tables 2, 7).

Q3. MPO/AWR update and DPO on UltraFeedback. Removing the baseline gives a positive-weight likelihood-fitting objective, analogous to MPO/AWR-style updates. The variant reduces accuracy by 29.4 GSM8K and 10.1 MATH points at 3B offline (Table 8) and reduces on-policy accuracy from 81.17→0.081.17\to0.0 on GSM8K and 64.23→21.3264.23\to21.32 on MATH (Table 9, Fig. 4). DPO, using the starting checkpoint as reference, is now in Table 3. TRIBE achieves better numerical resutls on all four reported metrics, including 14.5% versus 10.8% win rate against Best-of-4.

Closing Statement: We thank the reviewer for the detailed review, which prompted App. F, App. G and stabilizer-matched baselines. We hope these additional analyses and experiments address the reviewer's concerns and request any further clarifications which might help the reviewer with revisting their score.

Thank you for the detailed response, the new appendices and the stabiliser-matched baselines. The clarifications on the per-batch budget and on the positivity constraint answer my questions on Cor. 3. The RLOO and REINFORCE runs with the same clamping and subsampling, the DPO comparison and the no-baseline ablation are useful additions. My main concerns remain, though. The link between Stage 2 and Stage 1 holds only for the expected gradient at the behaviour policy, and the ϵ\epsilon budget still acts mostly as a step size, so the trust-region analysis does not really cover the algorithm that is run. On significance, the response confirms that TRIBE matches GRPO on-policy without improving on it, and that offline GRPO with behaviour probabilities is as good, so the gain is limited to the offline setting without behaviour probabilities. Finally, the MPO/AWR-style comparison I asked for is answered with an ablation of TRIBE without the baseline rather than with these methods themselves, so the advantage over these existing methods is not fully established. Indeed I think MPO and AWR could be applied in exactly this setting, offline and without the behaviour policy's probabilities, so a proper MPO/AWR comparison would still be needed to demonstrate the benefit of TRIBE in that setting. I will maintain my score.

AWR comparison. We ran AWR in exactly the setting the reviewer describes, offline and without behavior-policy probabilities, on the same offline data and evaluation as Table 2. Since no value function is available offline, AWR's advantage uses a leave-one-out group baseline, and its learning rate and temperature were selected by a sweep on a held-out validation split. On GSM8K, AWR reaches 61.64 Pass@1 with the 3B learner and 45.19 with the 1B learner, against 73.99 ± 0.79 and 46.15 ± 0.89 for TRIBE. On MATH with the 1B learner, AWR reaches 31.26 Pass@4, against 40.17 ± 0.64 for TRIBE. TRIBE outperforms AWR by 12.4 points on GSM8K with the 3B learner and by 8.9 points on MATH with the 1B learner. AWR's weights are always positive, so it provides no explicit probability-decreasing coefficient for incorrect responses, whereas TRIBE's signed Stage 2 weights explicitly demote responses assigned ρ⋆<1\rho^\star < 1.

Budget as step size. This concern was already addressed explicitly in our response. The budget is not merely a learning-rate parameter. The rescaling interpretation holds only for the χ2\chi^2 closed form when nonnegativity is inactive, which requires δ<1/(K−1)\delta < 1/(K-1) (Cor. 3). Our offline experiments use δ=0.5\delta = 0.5, where this condition fails and the nonnegativity constraint can change the solution itself. Our on-policy experiments use reverse KL, where the budget controls the temperature of the exponential tilt (Eq. 6). Even where δ\delta scales a fixed improvement direction, this is standard trust-region behavior. TRPO's natural-gradient step is θk+1=θk+2δ/g⊤F−1g F−1g\theta_{k+1} = \theta_k + \sqrt{2\delta / g^\top F^{-1} g}\, F^{-1} g, which likewise scales a fixed direction by a factor set by the trust-region radius. That the budget rescales the update therefore does not imply that it is not a trust region.

Stage 2 as a projection. The reviewer refers to a link between Stage 1 and Stage 2 that holds only for the expected gradient at the behavior policy. There is no separate link to establish between Stage 1 and Stage 2. Stage 2 is a parametric projection of the Stage 1 target, ideally the KL projection of π⋆\pi^\star onto the model class, which is the weighted maximum-likelihood objective with weights ρ⋆\rho^\star. The practical update adds a baseline to this objective. At the behavior policy, the baseline preserves the expected gradient of the KL projection and removes the zero-mean behavior-cloning term (Prop. 12). Away from the behavior policy, the paper states the practical objective it optimizes and derives the effect of each adjustment (Appendix F). The practical modifications are crucial to address the shortcomings of the idealized projection, which is unstable in practice (Appendix G).

Significance of TRIBE. TRIBE formulates policy improvement in importance-ratio space, and under a χ2\chi^2 trust region, its exact solution is GRPO's normalized-advantage weighting (Cor. 3). Because the program is defined on the sampled batch, this weighting no longer requires the behavior policy's probabilities. TRIBE therefore extends GRPO-style updates to offline data whose generating policy is unknown, a setting where GRPO itself cannot be applied without behavior-policy log-probabilities. Matching GRPO on-policy is the expected consequence of recovering its weighting. The gain is that the same ratio-space policy-improvement construction extends offline, without requiring behavior-policy log-probabilities, where TRIBE is the strongest behavior-independent method in Table 2 and outperforms AWR.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response

The revision addresses the shared concerns, with changes marked in blue.

  1. Theory and practice (App. F). Without length normalization, the baseline-centered Stage 2 loss is the ideal KL projection minus the behavior-cloning term KL(πb∥∣πθ)\mathrm{KL}(\pi_b\||\pi_\theta), and their expected gradients agree at the behavior policy. Clamping bounds the offline loss below (Prop. 9). In expectation, negative subsampling mixes the signed loss with a forward-KL projection onto upweighted responses (Prop. 10, Cor. 11).
  2. Component ablations (App. G). Removing the baseline costs 29.4 GSM8K and 10.1 MATH points offline and collapses on-policy training (Tables 8–9, Fig. 4). Intermediate negative subsampling yields better performance than either endpoint (Fig. 5). Clamping improves accuracy by 2.5–5.3 points in three of four settings (Table 12). Length normalization slightly improves accuracy and reduces gradient norms by two to three orders of magnitude (Tables 10–11).
  3. Stabilizer-matched baselines (Tables 2, 7). TRIBE's subsampling and clamping lift RLOO from 0.48 to 67.05 on GSM8K-3B, yet TRIBE continues to outperform the stabilizer-matched RLOO baseline across all four settings.
  4. DPO on UltraFeedback (Table 3). With the starting checkpoint as substitute reference, TRIBE performs better than DPO on all four reported metrics.
  5. Implementation and tuning. Table 6 and App. C.3 discuss hyper-parameters sweeped for every method. The batch-level trust-region budget and cvxpy fallback when the closed-form χ2\chi^2 solution violates nonnegativity are now made explicit after Corollary 3 and in App. D.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.