acceptodds
Under review as a conference paper at ICLR 2027

Learning from Failures: When Contrastive Post-Training Beats Success Conditioning

Abstract

Post-training a language model from rollouts with binary outcomes has two standard recipes: supervised fine-tuning on the successful rollouts (success conditioning), and pairwise contrastive learning on success–failure pairs. In practice the two are combined, and the combination is often reported to work best, but there is little theory of when and why. We analyze the combined family: a margin-based contrastive term plus a weighted success conditioning term on successful rollouts using a contextual bandit setup with binary rewards. We characterize when adding the contrastive term improves on success conditioning, both in value and other metrics such as those used in retrieval (e.g., precision, recall). We also propose another combined family that contains a DPO-style contrastive loss term that accounts for the base policy. We compare it with the combined loss that contains the more traditionally used triplet loss under different settings. We validate our theoretical results via synthetic experiments and also in the real-world context of generative retrieval.

Primary area:
reinforcement learning
Keywords:
Contrastive Learning, Contextual Bandits, Post Training, Generative retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UAK4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This article studies the performance of contrastive post-training, i.e. fine-tuning a model with a pairwise hinge on success/failure rollouts plus SFT on successes. The article proves several results in this context, notably that:

  • the "re-weighted" policy never has lower value than π0\pi_0 in any finite-horizon MDP (Thm 3.1)
  • in contextual bandits, both "re-weighted" and "un-weighted" have a higher value than the success conditioned policy π0+\pi_0^+, for sufficiently large margin δ\delta and success multiplier λ\lambda (Thm 3.2)
  • the "re-weighted" policy has at least the value of the "un-weighted" one, for contextual bandits where π0\pi_0 is aligned with QQ (Thm. 3.6) Experiments use a BART generative retriever on MS MARCO with Qwen3-Reranker-0.6B rewards, starting from different checkpoints. A semi-synthetic bandit with controlled rank correlation between π0\pi_0 and QQ is also included.
Critical strengths
  1. The question is well motivated. Contrastive training is a relevant domain and there are few existing results in this field.

  2. The results proposed are significant, with pertinent setups.

Critical weaknesses
  1. The experiments don't test the article's own claims well enough. While it claims to explain why combined losses work in LM post-training, experiments use a generative retrieval task. Moreover, the conditions on which depend the theorems are not tested: empirical alignment isn't measured and failure on specific MDPs isn't demonstrated. It shows flawed experimental practices, with apparently results from only one single run reported and no error/confidence interval. The baseline evaluation metric is also the reranker that supplies the training reward, so the reward gains are measured by the same output that was optimized.

  2. Results only focus on performance. In an idealized bandit setting (exact π0+/−\pi_0^{+/-}, unrestricted tabular policy) the optimal policy is greedy on Q, so it is unsurprising that a more aggressive update beats SFT once it is well defined. The reason to prefer conservative updates such as success conditioning is robustness (e.g. considering estimation error or distribution shift), which is not modeled here.

  3. Assessing novelty. The abstract claims the article "propose[s] another combined family" with a DPO-style term. However, Lunw\mathcal L_{unw} is essentially the SLiC-HF objective (Zhao et al.,2023), and the contrastive part of Lrw\mathcal L_{rw} is the hinge member of the GPO family (Tang et al., 2024). The author should reframe their contribution.

Minors:

  • Figure 3 has a placeholder caption
  • The Ethics, Reproducibility, Author Contributions, and Acknowledgments sections contain template text
  • l.070, typo: "λ ≥ be the scalar"
  • l.231, syntax: "for contextual , both"
  • l.431, wrong reference: Fig. 5 should be Fig. 2.
  • l.409, grammar: "Let We draw..."
  • l.413, grammar: "and base policy's weightage on picking"
  • l.490, typo: "componet"
  • Section 2 defines online and offline bandit regret, which is never used
  • ll. 227 & 232, Example 3.3 should be Lemma 3.3
  • Definition 2.1 mixes AiA_i and AtA_t
  • the Midpoint checkpoint is reported in Table 1 but not in Figs. 1 & 2
  • l.1825: "πunw\pi_{unw} is the winner, but πunw\pi_{unw} trails close"
Questions for authors
  1. What are λ⋆\lambda^\star and δ⋆\delta^\star on the MS MARCO feedback pools? Are any grid points inside the regime of Thm 3.2?

  2. How exactly are states weighted in the pairwise term of the MDP objective? Does Lemma 3.7 hold under that definition with a single global λ\lambda?

  3. Can you report multiple seeds with confidence intervals, plus DPO and SimPO under the same pair sampling as the hinge losses?

6 comments
AuthorsOpenReview ↗

Response from Authors Part 1

We thank the reviewer for their questions that will improve our paper. Following their suggestions, we have made several changes in the updated draft, which are written in teal colour.

The baseline evaluation metric is also the reranker that supplies the training reward, so the reward gains are measured by the same output that was optimized. + The reason to prefer conservative updates such as success conditioning is robustness (e.g. considering estimation error or distribution shift), which is not modeled here.

We think there might be some misunderstanding here. The goal of finetuning using rewards coming from an environment is to ensure that the reward provided by the same environment is maximized at test time. This is the goal of RL and our work is based on this setup. Robustness under distribution shift is another interesting direction for future work. As suggested by the reviewer, we have done some preliminary experiments and we're sharing the results below.

To compare the robustness of contrastive learning versus SFT, we have taken our trained checkpoints for the BART model (trained using RW, UNW, SFT and other baselines), and evaluated them using a different reward model. For training, the feedback was provided by the Qwen3-Reranker-0.6B model and we have used the publicly available Qwen3-Reranker-4B model for evaluation on the test data. We report the Monte Carlo expected reward for the various methods on the Qwen 4B model in the following table, where we have the best configuration for each method.

MethodEarlyMidpointFinal
Base policy (π0\pi_0)0.463 ±\pm 0.0010.505 ±\pm 0.0010.554 ±\pm 0.001
Reward-SFT0.571 ±\pm 0.0010.593 ±\pm 0.0010.622 ±\pm 0.001
DPO0.431 ±\pm 0.0010.532 ±\pm 0.0010.576 ±\pm 0.001
RankNet0.352 ±\pm 0.0010.389 ±\pm 0.0010.448 ±\pm 0.001
SimPO0.311 ±\pm 0.0010.488 ±\pm 0.0010.547 ±\pm 0.001
DPO+SFT0.534 ±\pm 0.0010.579 ±\pm 0.0010.604 ±\pm 0.001
πrw\pi_{\mathrm{rw}}0.604±0.001\mathbf{0.604 \pm 0.001}0.621±0.001‾\underline{0.621 \pm 0.001}0.638±0.001‾\underline{0.638 \pm 0.001}
πunw\pi_{\mathrm{unw}}0.603±0.001‾\underline{0.603 \pm 0.001}0.623±0.001\mathbf{0.623 \pm 0.001}0.640±0.001\mathbf{0.640 \pm 0.001}

We observe: (1) πrw\pi_{rw} and πunw\pi_{unw} have higher expected reward on the different feedback, indicating higher robustness than baselines. (2) πunw\pi_{unw} is better than πrw\pi_{rw} for the mid, final checkpoints. We hypothesize that the more contrastive factor from the adaptive ln⁡π0\ln\pi_0 dependent margins might have optimized harder on the training reward, hurting out of distribution generalization.

We now analyze the variation in expected reward with the weightage of the SFT component in the overall loss. As seen in the following table, (1) an optimum SFT weight happens in the interior where both losses contribute and (2) a combined loss beats both pure contrastive and SFT endpoints in terms of distributional robustness as well.

λ\lambdaδ\deltaEarly (πrw\pi_{\mathrm{rw}})Early (πunw\pi_{\mathrm{unw}})Mid (πrw\pi_{\mathrm{rw}})Mid (πunw\pi_{\mathrm{unw}})Final (πrw\pi_{\mathrm{rw}})Final (πunw\pi_{\mathrm{unw}})
0010100.507 ±\pm 0.0010.412 ±\pm 0.0010.558 ±\pm 0.0020.550 ±\pm 0.0010.586 ±\pm 0.0010.583 ±\pm 0.001
1/81/810100.564 ±\pm 0.0020.553 ±\pm 0.0010.564 ±\pm 0.0010.597 ±\pm 0.0010.624 ±\pm 0.0010.589 ±\pm 0.001
2/72/710100.583 ±\pm 0.0010.576 ±\pm 0.0020.613 ±\pm 0.0010.610 ±\pm 0.0010.597 ±\pm 0.0010.633 ±\pm 0.002
4/54/510100.592 ±\pm 0.0010.588 ±\pm 0.0010.620 ±\pm 0.0020.623±0.001\mathbf{0.623 \pm 0.001}0.638 ±\pm 0.0010.640±0.001\mathbf{0.640 \pm 0.001}
2210100.603±0.001\mathbf{0.603 \pm 0.001}0.603±0.001‾\underline{0.603 \pm 0.001}0.616 ±\pm 0.0010.613 ±\pm 0.0020.636 ±\pm 0.0020.639±0.001‾\underline{0.639 \pm 0.001}
7/27/210100.600 ±\pm 0.0020.599 ±\pm 0.0010.621±0.001‾\underline{0.621 \pm 0.001}0.618 ±\pm 0.0010.634 ±\pm 0.0010.631 ±\pm 0.001
8810100.593 ±\pm 0.0010.589 ±\pm 0.0020.613 ±\pm 0.0010.605 ±\pm 0.0010.626 ±\pm 0.0010.624 ±\pm 0.001
∞\inftySFT0.571 ±\pm 0.001----0.593 ±\pm 0.001----0.622 ±\pm 0.001----
AuthorsOpenReview ↗

Response from Authors Part 2

the conditions on which depend the theorems are not tested: empirical alignment isn't measured

In the generative retrieval experiment, we have taken three checkpoints from a common training trajectory with increasing average rewards, which is an indicator of how aligned π0\pi_0 is with the rewards. We report the V(π0)V(\pi_0) and V(πrw)−V(πunw)V(\pi_{rw}) - V(\pi_{unw}) numbers here, taken from Table 1 of our draft.

EarlyFinal
V(π0)V(\pi_0)0.530.63
V(πrw)−V(πunw)V(\pi_{rw}) -V(\pi_{unw})0.0010.002

However, we agree with the reviewer that a more principled way is to quantify this reward difference between πrw\pi_{rw} and πunw\pi_{unw} in terms of Kendall's correlation based alignment between π0\pi_0 and QQ, which we do in the synthetic experiments of Appendix C.1.

Synthetic experiment to validate alignment-based theorem

Reward structure. We use frozen real-query features, fixed success probability strictly between zero and one, and a stochastic binary reward sampled using these probabilities.

Shared parametric policy: To demonstrate robustness of our theoretical predictions for real classes of parametrized policies, we do the experiments for shared MLP and linear softmax (i.e., log-linear) class of policies that we optimize.

Alignment measurement: Kendall's τ\tau between π0(⋅∣s)\pi_0(\cdot\mid s) and Q(s,⋅)Q(s, \cdot), with values in [−1,1][-1,1].

Observations: We measure the difference between the expected rewards of the two contrastive losses compared in Theorem 3.6. A positive difference favors RW; a negative difference favors UNW. We report the reward differences across 8 positive SFT weights and 5 margins, in aligned and anti-aligned settings. Its grid-level summary is:

Shared modelAlignment and expected orderingStrictly predicted directionTiesReversals
Log-linearAligned: RW above UNW34/406/400/40
Log-linearAnti-aligned: UNW above RW32/408/400/40
MLPAligned: RW above UNW37/403/400/40
MLPAnti-aligned: UNW above RW37/403/400/40

Thus, the predicted gain in reward based on empirical alignment holds on gradient based learning on parametrized policy classes as well, validating Theorem 3.6 in a robust setting.

While it claims to explain why combined losses work in LM post-training, experiments use a generative retrieval task.

We'd like to politely point out that generative retrieval is essentially aligning a language model to preference data. When done on a pretrained language model, this is essentially a version of post-training with preferential data which falls exactly under our theoretical framework, making it a natural choice for empirical assessment. However, during the rebuttal we have done post training on a Qwen 2.5 Math 1.5B model on math reasoning, which is a more direct instance of LM post training.

Per question, we sample responses without beam search, collected from the base model. On the training set, the reward is binary answer correctness: exact-rational verification for GSM8K. On the test dataset of MATH-500 questions, we have used Math-Verify 0.9.0. We have used a similar tuning budget for all baselines and contrastive losses. We used margins 376 (called low) and 3765 (called high) on the sequence-summed log-probabilities.

We first report the accuracy (expected binary reward) of the best configurations of each method in the following table.

MethodAccuracy
Base model (π0\pi_0)0.794±0.0010.794 \pm 0.001
SFT0.799±0.0010.799 \pm 0.001
DPO (β=0.1\beta=0.1)0.836±0.001\mathbf{0.836 \pm 0.001}
πrw\pi_{\mathrm{rw}}0.814±0.001‾\underline{0.814 \pm 0.001}
πunw\pi_{\mathrm{unw}}0.814±0.001‾\underline{0.814 \pm 0.001}

As observed, (1) DPO does the best, followed by the contrastive hinge losses. (2) The contrastive methods beat SFT, which improves over the base model.

Now we see the margin and SFT anchor contributions for the unweighted and reweighted contrastive losses.

λ\lambdaδ\deltaπrw\pi_{\mathrm{rw}} accuracyπunw\pi_{\mathrm{unw}} accuracy
00Low0.671±0.0010.671 \pm 0.0010.812±0.001‾\underline{0.812 \pm 0.001}
00High0.363±0.0020.363 \pm 0.0020.384±0.0010.384 \pm 0.001
11Low0.814±0.001\mathbf{0.814 \pm 0.001}0.814±0.001\mathbf{0.814 \pm 0.001}
11High0.774±0.0010.774 \pm 0.0010.796±0.0020.796 \pm 0.002
44Low0.807±0.001‾\underline{0.807 \pm 0.001}0.798±0.0010.798 \pm 0.001
44High0.777±0.0020.777 \pm 0.0020.811±0.0010.811 \pm 0.001
∞\inftySFT0.799±0.0010.799 \pm 0.001—

As observed, contribution from both losses is essential for better performance. For binary exact rewards (this setting), exact SFT is essentially greedy (hence optimal) but it happens so that the parametric model is unable to replicate exact success conditioning while the contrastive losses remain more robust.

AuthorsOpenReview ↗

Response from Authors Part 3

Results only focus on performance. In an idealized bandit setting (exact unrestricted tabular policy) the optimal policy is greedy on Q, so it is unsurprising that a more aggressive update beats SFT once it is well defined.

We agree that a policy greedy in QQ maximizes bandit reward. It does not, however, imply that giving the contrastive term more weight moves the learned policy toward that optimum.

In fact, Lemma 3.4 of our paper gives a two-action counterexample to this statement: if 1>Q1>Q2>01>Q_1>Q_2>0 and 0≤δ<log⁡(Q1/Q2)0\leq\delta<\log(Q_1/Q_2), then Vπrw<Vπ0+V_{\pi_{rw}}<V_{\pi_0^+} for every finite λ≥0\lambda\geq0 (where λ\lambda is the weight on the SFT loss and δ\delta is the margin). At π0+\pi^+_0, pairs favoring the better action already satisfy the margin and give no gradient. But the worse action sometimes succeeds while the better one fails, and these pairs still push probability toward the worse action.

Thus a stronger contrastive update can reduce reward. And this is what led us to analyze the existence of margin and contrastive weights in Theorem 3.2 that improve the contrastive loss optimizer over SFT.

The abstract claims the article "propose another combined family" with a DPO-style term. However, Lunw\mathcal L_{unw} is essentially the SLiC-HF objective (Zhao et al.,2023), and the contrastive part of Lrw\mathcal{L}_{rw} is the hinge member of the GPO family (Tang et al., 2024). The author should reframe their contribution.

The phrase "propose another combined family" was used mistakenly, and we have removed it. We do not propose either objective: the revised introduction credits SLiC-HF (Zhao et al., 2023) for the unweighted loss and GPO (Tang et al., 2024) for the reweighted hinge term. Our contribution is to show when the reweighted loss beats the unweighted loss and success-only SFT, both theoretically in an RL framework, by analyzing the SFT weight, the margin, and the base policy's alignment with the rewards (Thms. 3.2 and 3.6) as well as empirically on synthetic bandits and MS MARCO.

What are $\lambda^{*} and delta∗delta^{*} on the MS MARCO feedback pools? Are any grid points inside the regime of Thm 3.2?

We want to highlight the fact that Theorem 3.2 is an existential guarantee: it establishes that suitable finite SFT weights and margins can yield improvement over success conditioning. Its constructive bounds are sufficient conditions, rather than a characterization of the smallest parameters needed for improvement. Accordingly, the theorem provides theoretical guidance, but not a necessary prescription for selecting hyperparameters.

In our experiments, we minimize a combined loss of the form Lpair+λLSFT\mathcal L_{pair} + \lambda \mathcal L_{SFT}, where the pairwise loss is given by Lpair=∑a+,a−[δ+ln⁡π(a−)−ln⁡π(a+)]+\mathcal L_{pair} = \sum_{a_+, a_-} [\delta + \ln \pi(a_-) - \ln \pi(a_+)]_+, with baseline re-weighting for the re-weighted version. In our MS MARCO experiments, we have varied (λ,δ)∈[0,8]×[0.1,10](\lambda, \delta) \in [0, 8] \times [0.1, 10]. We now report the sufficient] bounds for Theorem 3.2 from the feedback pools of Early, Midpoint and Final checkpoints:

CheckpointSufficient λ\lambdaSufficient δrw\delta_{rw}Sufficient δunw\delta_{unw}Largest grid (λ,δ)(\lambda, \delta)
Early2×10112\times10^{11}1677(8, 10)
Midpoint9×10109\times 10^{10}1780(8, 10)
Final6×10106\times 10^{10}1688(8, 10)

Therefore, no tested grid point satisfies Theorem 3.2's sufficient conditions uniformly across a checkpoint’s tested queries.

The empirical grid nevertheless contains configurations that improve over trained SFT. In a matched comparison, several (λ,δ)(\lambda, \delta) configurations improve both of the learnt πrw\pi_{rw} and πunw\pi_{unw} over SFT. We report some of those grid coordinates and the improvements below:

Checkpoint(λ,δ)(\lambda,\delta)V(πunw)−V(πSFT)V(\pi_{unw}) - V(\pi_{SFT})V(πrw)−V(πSFT)V(\pi_{rw}) - V(\pi_{SFT})
Early(2,10)(2, 10)+0.0359+0.0359+0.0366+0.0366
Early(8,10)(8, 10)+0.0232+0.0232+0.0272+0.0272
Midpoint(2,10)(2, 10)+0.0237+0.0237+0.0263+0.0263
Midpoint(8,10)(8, 10)+0.0162+0.0162+0.0246+0.0246
Final(2,10)(2, 10)+0.0244+0.0244+0.0210+0.0210
Final(8,10)(8, 10)+0.0110+0.0110+0.0128+0.0128

Thus, Theorem 3.2's conditions are sufficient, not necessary: failing its sufficient certificate does not preclude the empirical improvements observed on MS MARCO.

Reviewer_oF9aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when combining contrastive learning with SFT on successful rollouts can be better than success-only SFT. The authors provide theoretical results for two contrastive losses and evaluate them on generative retrieval.

Critical strengths
  1. Interesting problem. The paper studies an question about how to combine SFT (like RS) and pairwise fine-tuning (like DPO) training signal together for post-training.
  2. Theoretical analysis. The authors provide theoretical conditions for improvement, compare two losses, and discuss cases where the guarantees do not hold.
  3. Useful comparisons. The paper studies different loss weights, margins, and base model checkpoints. The results show that the choice of objective can affect both reward and ranking performance.
Critical weaknesses

While the topic is interesting, the author needs extra time to polish the paper. I have numbers of concerns arised:

  1. Mismatch between theory and experiments. The theory studies expected policy value, but the experiments mainly report average Qwen reward over beam-search outputs. These are different metrics, so the experiments do not directly verify the theoretical results.
  2. Unfair baseline comparison. DPO, RankNet, and SimPO use different rules to construct training pairs. The proposed methods also have a larger hyperparameter search, while DPO uses a fixed beta of 0.1. It is unclear whether the improvements come from the loss or the training setup.
  3. Weak SFT baseline. The SFT results are almost the same as the starting model, such as 0.3939 versus 0.3940 for the Early checkpoint. It is unclear why SFT provides almost no improvement, which makes the comparison less convincing.
  4. Limited experimental scope. The real-world experiments only use BART-large on MS MARCO. The three checkpoints come from the same training process, so they do not provide enough evidence of generalization to other models, tasks, or modern LLM post-training.
  5. Limited connection to practical LLM training. The theory studies population-optimal policies, while practical LLM training uses finite data and imperfect optimization. For deterministic binary rewards, ideal success conditioning already achieves the maximum reward. It is unclear how the improvement guarantees explain practical RLVR training.
  6. Not enough experiments and ablations. Most experiments change loss weights and margins within the same setting. More studies on different reward models, candidate selection methods, model families, and datasets are needed to support the general claims. The experiments use only one training seed. It is difficult to know how reliable the observed improvements are.
  7. Figures lack explanation. Many figures contain several curves but have limited explaination of the context and discussion of why the results change. Some figures are also difficult to read, and the connection between the figures and theoretical results is not always clear.
  8. Poor writing and formatting. The paper is not easy to follow and contains many typos, grammar errors, and formatting issues. The authors often summarize results first and then repeat them in later sections, making the paper unnecessarily long. There are also too many definitions compared with the limited experimental evidence, and Figure 3 in the appendix still has a placeholder caption.
Questions for authors

Please see weaknesses above, and this extra question: How do the theoretical results apply to practical LLM reasoning with deterministic rewards and finite training data?

5 comments
AuthorsOpenReview ↗

Response from Authors Part 1

We thank the reviewer for their questions that will improve our paper. Following their suggestions, we have made several changes in the updated draft, which are written in teal colour.

(Weakness 2, 3) Unfair baseline comparison. DPO, RankNet, and SimPO use different rules to construct training pairs. + Weak SFT baseline. The SFT results are almost the same as the starting model, such as 0.3939 versus 0.3940 for the Early checkpoint.

In order to address the reviewer's concerns we have made updates in the latest draft as explained below:

(1) Same pairs. All pairwise baselines are now trained on exactly the same frozen positive–negative pairs as the hinge losses, drawn from π0+\pi_0^+ and π0−\pi_0^- on the same candidate support.

(2) Tuned baselines. Every baseline is tuned separately for each checkpoint: βDPO∈\beta_{\mathrm{DPO}}\in{0.01,0.1,0.3,1}, βRankNet∈\beta_{\mathrm{RankNet}}\in{0.1,0.3,1,3}, βSimPO∈\beta_{\mathrm{SimPO}}\in{2,5,7} with margin ratio in {0,0.5}\{0,0.5\}, and DPO+SFT jointly over β\beta and the SFT weight, each over three learning rates. DPO+SFT is searched over 48 configurations, comparable to the 45 (ω,δ)(\omega,\delta) configurations of each hinge loss.

(3) Three seeds for every method.

(4) Wider SFT sweep: Originally we used the exact same grid of hyperparameters for each of the RW, UNW and their shared SFT endpoint. However, it turns out that SFT is quite sensitive to optimization hyperparameters. During the rebuttal, we ran a wider hyperparameter search for SFT and were able to significantly improve over the base model in terms of expected reward.

Details of the tuning budget can be found in Appendix C.2 of our updated draft.

We report the expected Qwen reward (Monte Carlo expectation by sampling without beam search) in the following table, where we report the best configuration of each method, across the three MS MARCO checkpoints:

MethodEarlyMidpointFinal
Base policy (π0\pi_0)0.5296±0.00100.5296 \pm 0.00100.5761±0.00080.5761 \pm 0.00080.6308±0.00110.6308 \pm 0.0011
Reward-SFT0.6473±0.00080.6473 \pm 0.00080.6701±0.00110.6701 \pm 0.00110.6951±0.00090.6951 \pm 0.0009
DPO0.4902±0.00110.4902 \pm 0.00110.6031±0.00090.6031 \pm 0.00090.6510±0.00120.6510 \pm 0.0012
RankNet0.4033±0.00130.4033 \pm 0.00130.4482±0.00100.4482 \pm 0.00100.5148±0.00080.5148 \pm 0.0008
SimPO0.3575±0.00090.3575 \pm 0.00090.5554±0.00110.5554 \pm 0.00110.6198±0.00100.6198 \pm 0.0010
DPO+SFT0.5265±0.00120.5265 \pm 0.00120.6145±0.00070.6145 \pm 0.00070.6566±0.00110.6566 \pm 0.0011
πrw\pi_{\mathrm{rw}}0.6844±0.0010\mathbf{0.6844 \pm 0.0010}0.7060±0.0012\mathbf{0.7060 \pm 0.0012}0.7225±0.0010\mathbf{0.7225 \pm 0.0010}
πunw\pi_{\mathrm{unw}}0.6832±0.0009‾\underline{0.6832 \pm 0.0009}0.7019±0.0009‾\underline{0.7019 \pm 0.0009}0.7204±0.0012‾\underline{0.7204 \pm 0.0012}

As observed: (1) tuning the baselines improved performance, but πrw\pi_{rw} is still the table winner. (2) Tuned SFT beats π0\pi_0 significantly.

AuthorsOpenReview ↗

Response from Authors Part 2

(Weakness 1) Mismatch between theory and experiments. The theory studies expected policy value, but the experiments mainly report average Qwen reward over beam-search outputs. These are different metrics, so the experiments do not directly verify the theoretical results

We thank the reviewer for pointing this out. We agree that average reward over beam-search outputs is not an estimate of expected reward or value function. However, our original presentation emphasized on average beam reward because generative retrieval returns a ranked candidate list using beam search, and average reward over that candidate list determines the quality of the retrieved items, which is crucial for ranking and retrieval.

However, we now explicitly compute Monte Carlo expected reward, where we draw independent samples and report the average of their Qwen3-Reranker-0.6B rewards, as our main reward metric (as reported in the previous response). Table 1 of Section 4 of our updated main text now reports average beam reward and expected reward side-by-side, which we report here too, for the sake of completeness.

MethodAvg. beam rewardAvg. beam rewardAvg. beam rewardExpected rewardExpected rewardExpected reward
EarlyMidpointFinalEarlyMidpointFinal
Base policy (π0\pi_0)0.394±0.0010.394 \pm 0.0010.438±0.0010.438 \pm 0.0010.474±0.0010.474 \pm 0.0010.530±0.0010.530 \pm 0.0010.576±0.0010.576 \pm 0.0010.631±0.0010.631 \pm 0.001
Reward-SFT0.386±0.0010.386 \pm 0.0010.417±0.0010.417 \pm 0.0010.454±0.0010.454 \pm 0.0010.647±0.0010.647 \pm 0.0010.670±0.0010.670 \pm 0.0010.695±0.0010.695 \pm 0.001
DPO0.461±0.0010.461 \pm 0.0010.474±0.0010.474 \pm 0.0010.511±0.001‾\underline{0.511 \pm 0.001}0.490±0.0010.490 \pm 0.0010.603±0.0010.603 \pm 0.0010.651±0.0010.651 \pm 0.001
RankNet0.452±0.0010.452 \pm 0.0010.472±0.0010.472 \pm 0.0010.510±0.0010.510 \pm 0.0010.403±0.0010.403 \pm 0.0010.448±0.0010.448 \pm 0.0010.515±0.0010.515 \pm 0.001
SimPO0.460±0.0010.460 \pm 0.0010.474±0.0010.474 \pm 0.0010.510±0.0010.510 \pm 0.0010.358±0.0010.358 \pm 0.0010.555±0.0010.555 \pm 0.0010.620±0.0010.620 \pm 0.001
DPO+SFT0.461±0.0010.461 \pm 0.0010.474±0.0010.474 \pm 0.0010.510±0.0010.510 \pm 0.0010.527±0.0010.527 \pm 0.0010.615±0.0010.615 \pm 0.0010.657±0.0010.657 \pm 0.001
πrw\pi_{\mathrm{rw}}0.470±0.001\mathbf{0.470 \pm 0.001}0.486±0.001\mathbf{0.486 \pm 0.001}0.515±0.001\mathbf{0.515 \pm 0.001}0.684±0.001\mathbf{0.684 \pm 0.001}0.706±0.001\mathbf{0.706 \pm 0.001}0.723±0.001\mathbf{0.723 \pm 0.001}
πunw\pi_{\mathrm{unw}}0.465±0.001‾\underline{0.465 \pm 0.001}0.476±0.001‾\underline{0.476 \pm 0.001}0.507±0.0010.507 \pm 0.0010.683±0.001‾\underline{0.683 \pm 0.001}0.702±0.001‾\underline{0.702 \pm 0.001}0.720±0.001‾\underline{0.720 \pm 0.001}

We make the following observations: (1) Tuned SFT that significantly beats π0\pi_0 in expected reward loses in terms of beam reward, possibly indicating policy weights being concentrated on the top actions, which maximizes expected reward but hurts average beam reward for higher beam width (in our case, 100). (2) Tuned baselines (like DPO, SimPO etc.) improve significantly in both metrics. (3) πrw\pi_{\mathrm{rw}} wins across metrics and checkpoints, closely followed by πunw\pi_{\mathrm{unw}}. In expected reward they are significantly better than the rest.

The experiments use only one training seed. It is difficult to know how reliable the observed improvements are

In our original submission, the experiments were single seeded due to limited compute. But post submission we have three seeded every experiment and have updated the numbers (along with error bars) in the updated draft.

AuthorsOpenReview ↗

Response from Authors Part 3

Most experiments change loss weights and margins within the same setting. More studies on different reward models, candidate selection methods, model families, and datasets are needed to support the general claims.

Our experiments were designed mostly to validate the theory, in a real-world practical application (generative retrieval). However, we agree with the reviewer's suggestions and believe that a broader set of experiments can be helpful. It's important and interesting to analyze how robust or applicable these contrastive losses are in practical settings. To do that, we have performed experiments to see how πrw,πunw\pi_{rw}, \pi_{unw} trained on one reward model perform when the reward is changed at evaluation. This analyzes an interesting empirical question on distributional shift and compares how robust the contrastive methods are, compared to SFT.

For training, the feedback was provided by the Qwen3-Reranker-0.6B model and we have used the publicly available Qwen3-Reranker-4B model for evaluation on the test data. We report the Monte Carlo expected reward for the various methods on the Qwen 4B model in the following table, where we have the best configuration for each method.

MethodEarlyMidpointFinal
Base policy (π0\pi_0)0.463 ±\pm 0.0010.505 ±\pm 0.0010.554 ±\pm 0.001
Reward-SFT0.571 ±\pm 0.0010.593 ±\pm 0.0010.622 ±\pm 0.001
DPO0.431 ±\pm 0.0010.532 ±\pm 0.0010.576 ±\pm 0.001
RankNet0.352 ±\pm 0.0010.389 ±\pm 0.0010.448 ±\pm 0.001
SimPO0.311 ±\pm 0.0010.488 ±\pm 0.0010.547 ±\pm 0.001
DPO+SFT0.534 ±\pm 0.0010.579 ±\pm 0.0010.604 ±\pm 0.001
πrw\pi_{\mathrm{rw}}0.604±0.001\mathbf{0.604 \pm 0.001}0.621±0.001‾\underline{0.621 \pm 0.001}0.638±0.001‾\underline{0.638 \pm 0.001}
πunw\pi_{\mathrm{unw}}0.603±0.001‾\underline{0.603 \pm 0.001}0.623±0.001\mathbf{0.623 \pm 0.001}0.640±0.001\mathbf{0.640 \pm 0.001}

We observe: (1) πrw\pi_{rw} and πunw\pi_{unw} have higher expected reward on the different feedback, indicating higher robustness than baselines. (2) πunw\pi_{unw} is better than πrw\pi_{rw} for the mid, final checkpoints. We hypothesize that the more contrastive factor from the adaptive ln⁡π0\ln\pi_0 dependent margins might have optimized harder on the training reward, hurting out of distribution generalization.

We now analyze the variation in expected reward with the weightage of the SFT component in the overall loss. As seen in the following table, (1) an optimum SFT weight happens in the interior where both losses contribute and (2) a combined loss beats both pure contrastive and SFT endpoints in terms of distributional robustness as well.

λ\lambdaδ\deltaEarly (πrw\pi_{\mathrm{rw}})Early (πunw\pi_{\mathrm{unw}})Mid (πrw\pi_{\mathrm{rw}})Mid (πunw\pi_{\mathrm{unw}})Final (πrw\pi_{\mathrm{rw}})Final (πunw\pi_{\mathrm{unw}})
0010100.507 ±\pm 0.0010.412 ±\pm 0.0010.558 ±\pm 0.0020.550 ±\pm 0.0010.586 ±\pm 0.0010.583 ±\pm 0.001
1/81/810100.564 ±\pm 0.0020.553 ±\pm 0.0010.564 ±\pm 0.0010.597 ±\pm 0.0010.624 ±\pm 0.0010.589 ±\pm 0.001
2/72/710100.583 ±\pm 0.0010.576 ±\pm 0.0020.613 ±\pm 0.0010.610 ±\pm 0.0010.597 ±\pm 0.0010.633 ±\pm 0.002
4/54/510100.592 ±\pm 0.0010.588 ±\pm 0.0010.620 ±\pm 0.0020.623±0.001\mathbf{0.623 \pm 0.001}0.638 ±\pm 0.0010.640±0.001\mathbf{0.640 \pm 0.001}
2210100.603±0.001\mathbf{0.603 \pm 0.001}0.603±0.001‾\underline{0.603 \pm 0.001}0.616 ±\pm 0.0010.613 ±\pm 0.0020.636 ±\pm 0.0020.639±0.001‾\underline{0.639 \pm 0.001}
7/27/210100.600 ±\pm 0.0020.599 ±\pm 0.0010.621±0.001‾\underline{0.621 \pm 0.001}0.618 ±\pm 0.0010.634 ±\pm 0.0010.631 ±\pm 0.001
8810100.593 ±\pm 0.0010.589 ±\pm 0.0020.613 ±\pm 0.0010.605 ±\pm 0.0010.626 ±\pm 0.0010.624 ±\pm 0.001
∞\inftySFT0.571 ±\pm 0.001----0.593 ±\pm 0.001----0.622 ±\pm 0.001----
Reviewer_43X8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies when learning from failed rollouts improves upon success-only fine-tuning. It analyzes combined contrastive and likelihood objectives, providing a systematic analysis of improvement conditions, reference-policy effects, and limitations when extending bandit guarantees to sequential decisions.

Critical strengths
  1. The paper studies a meaningful and well-motivated theoretical question. The investigation on when incorporating failed rollouts improves upon success-only training is conducted, connecting practical post-training methods with a tractable theoretical framework. Comparing against success conditioning, rather than merely the base policy, makes the question particularly relevant.

  2. The paper provides systematic analysis with informative guarantees and counterexamples. Notably, Theorem 3.2 establishes conditions for strict improvement over success conditioning. Complementary results clarify the role of reference-policy reweighting, distinguish reward from ranking performance, and demonstrate limitations in sequential settings, providing a coherent account of when the benefits hold.

Critical weaknesses
  1. Even though the results in the paper are generally interesting, the connection between them and practical learning can still be limited. The analysis concerns exact population minimizers over unrestricted policies, leaving finite-sample, approximation, and optimization effects uncharacterized. This matters particularly for deterministic binary verification: exact success conditioning already achieves perfect reward, making strict improvement impossible. Explaining the practical benefit of failures in such settings therefore requires analysis beyond the current framework.

  2. The experimental design has room of improvement. Generative retrieval is utilized as the benchmark task, while it only loosely matches with the conditions in the theory. For example, I treat a major interesting result as Theorem 3.2, which establishes a strict improvement guarantee for the fused objective. The guarantee requires the positive Q-function variance condition, meaning that among the good actions, there should still be difference in Q-values. While in Line 422-427, even though Theorem 3.2 is said to be experimentally justified, the justification of this condition remains lacking. Overall, I suggest designing the experiments more rigorously to justify each theorem with its assumptions and results fulfill. If real data tasks are difficult for this, synthetic data results are also acceptable.

Questions for authors
  1. Regarding Weakness 1, I would like to suggest including a Limitation Section to discuss the current gap between established theories and actual practice perspectives.

  2. Regarding Weakness 2, I would like to suggest refining the experimental part to make the justification of the theoretical results more soundly.

2 comments
AuthorsOpenReview ↗

Response from Authors Part 1

We thank the reviewer for their questions that will improve our paper. Following their suggestions, we have made several changes in the updated draft, which are written in teal colour.

The analysis concerns exact population minimizers over unrestricted policies, leaving finite-sample, approximation, and optimization effects uncharacterized.

We thank the reviewer for highlighting this distinction. We agree that guarantees for exact population minimizers over unrestricted policies do not, by themselves, characterize finite-sample, approximation, or optimization effects.

Benefits of the tabular formulation: This is a tractable starting point for any theoretical analysis that allows us to separate the effects of the base policy π0\pi_0, the margin δ\delta, and the SFT weight λ\lambda from the choice of policy parametrization and specific optimization algorithm. The comparison is nontrivial even in a tabular stochastic-reward setting as seen in our theorems and counterexamples. Our analysis identifies conditions under which incorporating failures can improve upon positive-only training, and how this depends on reward variance and policy–reward alignment. We view this as a structural explanation of the objectives rather than an end-to-end learning guarantee.

Beyond tabular policies: Shared parameters couple the predictions at different states, so the state-wise optimum need not be representable by one shared policy. Obtaining a comparable guarantee therefore requires controlling both this approximation error and the errors introduced by finite data and optimization. This distinction also appears in policy-gradient theory, where meaningful theoretical analysis of even the policy gradient algorithm is cumbersome beyond log-linear policies without making specific assumptions like overparametrization.

New theoretical result for linear softmax policies: As a first step in this direction, we have also obtained new theoretical results for shared log-linear policies. For policies of the form πθ(a∣s)∝π0(a∣s)exp⁡(θ⊤ϕ(s,a)),\pi_\theta(a\mid s)\propto\pi_0(a\mid s)\exp(\theta^\top\phi(s,a)), we can show that both RW+SFT and UNW+SFT strictly improve over the population SFT minimizer. Under a structural condition on the rewards given by Q(s,a)=c(s)+w⊤ϕ(s,a),Q(s,a)=c(s)+w^\top\phi(s,a), and sufficiently large SFT weight and margin, we can now show that optimizers of both the contrastive losses strictly improve over the SFT minimizer. This result is non-trivial since neither exact success conditioning nor the unrestricted combined-loss optimum needs to be representable by the shared log-linear parametrization, unlike the tabular setting. We have included this result as Theorem 3.11 under the new Section 3.8 of our updated draft and have formally proved the theorem in Appendix B.7 of the same.

However, beyond such restrictive and specific assumptions, how to analyze the case of general shared parametric policies remains largely intractable and could be part of future work.

This matters particularly for deterministic binary verification: exact success conditioning already achieves perfect reward, making strict improvement impossible.

We'd like to highlight that our work focuses on stochastic binary rewards and not deterministic binary rewards since most practical settings involve stochastic reward. For example, in generative retrieval, the binary reward depends on the user's preference, which is stochastic. This is also true for most preference-based RLHF settings.

In stochastic settings, success conditioning is not optimal even for tabular bandits, and therefore making strict improvement over it is possible as achieved by our work.

Suboptimality of success conditioning for stochastic rewards: As explained by Russo (2026), this is evident even for multi-armed bandits. If we consider NN actions with expected rewards Q1,Q2,⋯ ,QN∈(0,1)Q_1, Q_2,\cdots, Q_N\in (0,1) then the success conditioned distribution over the actions is given by π0+(i)∝Qiπ0(i)\pi^+_0(i) \propto Q_i\pi_0(i) for i∈{1,2,⋯ ,N}i\in\{1,2, \cdots, N\} while the optimal policy would be selecting the best action with probability 11. Hence, one round of success conditioning stays far below the optimal policy.

AuthorsOpenReview ↗

Response from Authors Part 2

...designing the experiments more rigorously to justify each theorem with its assumptions and results fulfill.

This is a great suggestion. In fact, in Appendix C.1 of our submission, we had validated Theorems 3.2 and 3.6 for the class of shared MLP parametrized policy via synthetic experiments. Taking the reviewer's suggestions, we have designed some more experiments that quantify the dominance of the contrastive optimizer over SFT as the reward variance and the SFT weight change. We describe this experiment next.

Experimental protocol:

Reward structure. We use frozen real-query features, fixed success probability strictly between zero and one, and a stochastic binary reward sampled using these probabilities.

Controlling reward-variance. For a context ss, we have constructed the reward as Q(s,a)=Q(s)+ϵ⋅x(s,a)Q(s,a) = Q(s) + \epsilon \cdot x(s,a), where x(s,⋅)x(s, \cdot) is a vector such that ∑ax(s,a)π0(a∣s)=0\sum_a x(s, a)\pi_0(a\mid s) = 0. Thus, by changing ϵ\epsilon, we explicitly control the reward variance without changing the expectation.

Policy class: We consider shared MLP and log-linear policy classes.

Validating Thm 3.1

We validate Theorem 3.1 on the above experimental setup. It proves that V(πrw)≥V(π0)V(\pi_{rw})\geq V(\pi_0) for any hyperparameter. We test the corresponding prediction using learned shared-MLP and shared log-linear policies for the contextual bandit setting across different alignments between π0\pi_0 and QQ. We report the results over 2 alignments ({±\pm 1}), 5 margins, 9 contrastive mixture weights, and 3 seeds, for both MLP and log-linear settings.

Policy classV(πrw)>V(π0)V(\pi_{\text{rw}})>V(\pi_0)Mean reward improvement
MLP270/270+0.073086
Log-linear270/270+0.010907

Out of 270270 settings, we observe that V(πrw)>V(π0)V(\pi_{\text{rw}})>V(\pi_0) holds for all of them, and the mean reward improvement is shown in the table above.

Validating Thm 3.2

We compare each learned RW and UNW policy against a learned SFT policy in the same model class, and compare it to the empirical variance of action success probabilities under the success-conditioned distribution.

The table below reports the results separately at every tested SFT weight (λ\lambda) in the combined loss over both MLP and log-linear classes. Each entry in the following table is read as two entries: V(πrw)−V(πSFT)V(\pi_{rw}) - V(\pi_{SFT}), V(πunw)−V(πSFT)V(\pi_{unw}) - V(\pi_{SFT}), with both reward gains multiplied by 1,000 for readability. Variance is unscaled.

Policy classVar$_{\pi^+_0}(Q)$\lambda=8\λ=16\lambda=16λ=32\lambda=32λ=64\lambda=64
Log-linear0.00007180.0427, 0.04270.0213, 0.02130.0107, 0.01070.0053, 0.0053
Log-linear0.00064340.3350, 0.33500.1676, 0.16760.0838, 0.08380.0419, 0.0419
Log-linear0.00177640.9100, 0.91000.4538, 0.45380.2270, 0.22700.1135, 0.1135
MLP0.00060860.3871, 0.28890.2071, 0.16330.1838, 0.16110.1578, 0.1463
MLP0.00537663.4111, 2.84711.8844, 1.67670.6802, 0.59450.5299, 0.4796
MLP0.01433928.6731, 7.76264.5929, 4.16061.9270, 1.72251.0185, 0.8736

For both RW and UNW, reward improvement increases with variance for both model classes. In the above table, we also notice that the reward improvement decreases with increasing λ\lambda. This is consistent with the fact that in the proof of Theorem 3.2, we in fact derive a lower bound proportional to Varπ0+(Q)/λ\text{Var}_{\pi^+_0}(Q)/\lambda for large margin and above the threshold λ\lambda. Thus, the above table provides a robust empirical validation of the theorem.

Validating Thm 3.6

We now measure the difference between the expected rewards of the two contrastive losses compared in Theorem 3.6. A positive difference favors RW; a negative difference favors UNW. We report the reward differences across 8 positive SFT weights and 5 margins, in aligned and anti-aligned settings. Its grid-level summary is:

Shared modelAlignment and expected orderingStrictly predicted directionTiesReversals
Log-linearAligned: RW above UNW34/406/400/40
Log-linearAnti-aligned: UNW above RW32/408/400/40
MLPAligned: RW above UNW37/403/400/40
MLPAnti-aligned: UNW above RW37/403/400/40

Thus, the predicted weak ordering holds throughout the grid. These additional synthetic experiments rigorously validate Thms 3.1, 3.2 and 3.6 across two different classes of parametrized policies optimized by gradient descent on empirical data. We have added these additional results in Appendix C.1 of our updated draft.

Conclusion: We thank the reviewer again for their rigorous assessment of our paper. We hope that our answers and additional experiments addressed their concerns. We'd be happy to answer any follow-up questions they might have.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Remarks

Dear AC and reviewers,

As the discussion period draws to a close, we would like to summarize the situation as we see it.

In our paper, we analyze contrastive learning using a contextual bandit setup with binary rewards, a framework commonly used in LLM post-training and alignment. We characterize when adding the contrastive term improves on simple SFT, both in expected value and other metrics such as those used in retrieval (e.g., precision, recall). We compare when having a reference re-weighting (as used in DPO) compared to the more traditionally used triplet loss under different settings. We validate our theoretical results via synthetic experiments and also in the real-world context of generative retrieval.

The reviewers requested clarification on the scope of the method, to what extent theory and practice align, and some issues on the experimental protocol. We believe we have made the required clarifications in the rebuttal and in the revised manuscript, provided additional experiments on distribution shift and on LLM post-training beyond generative retrieval in RLVR and stochastic reward settings. We are happy to see that one of the reviewers recommends acceptance of our paper and the other two reviewers also said that our additional experiments clarified several of their concerns. We truly believe this is a good paper with strong theoretical and empirical results, and most of the concerns have been addressed in the rebuttal and the updated draft.

Thanks for your time and effort,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.