- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This article studies the performance of contrastive post-training, i.e. fine-tuning a model with a pairwise hinge on success/failure rollouts plus SFT on successes. The article proves several results in this context, notably that:
- the "re-weighted" policy never has lower value than in any finite-horizon MDP (Thm 3.1)
- in contextual bandits, both "re-weighted" and "un-weighted" have a higher value than the success conditioned policy , for sufficiently large margin and success multiplier (Thm 3.2)
- the "re-weighted" policy has at least the value of the "un-weighted" one, for contextual bandits where is aligned with (Thm. 3.6) Experiments use a BART generative retriever on MS MARCO with Qwen3-Reranker-0.6B rewards, starting from different checkpoints. A semi-synthetic bandit with controlled rank correlation between and is also included.
Critical strengths
-
The question is well motivated. Contrastive training is a relevant domain and there are few existing results in this field.
-
The results proposed are significant, with pertinent setups.
Critical weaknesses
-
The experiments don't test the article's own claims well enough. While it claims to explain why combined losses work in LM post-training, experiments use a generative retrieval task. Moreover, the conditions on which depend the theorems are not tested: empirical alignment isn't measured and failure on specific MDPs isn't demonstrated. It shows flawed experimental practices, with apparently results from only one single run reported and no error/confidence interval. The baseline evaluation metric is also the reranker that supplies the training reward, so the reward gains are measured by the same output that was optimized.
-
Results only focus on performance. In an idealized bandit setting (exact , unrestricted tabular policy) the optimal policy is greedy on Q, so it is unsurprising that a more aggressive update beats SFT once it is well defined. The reason to prefer conservative updates such as success conditioning is robustness (e.g. considering estimation error or distribution shift), which is not modeled here.
-
Assessing novelty. The abstract claims the article "propose[s] another combined family" with a DPO-style term. However, is essentially the SLiC-HF objective (Zhao et al.,2023), and the contrastive part of is the hinge member of the GPO family (Tang et al., 2024). The author should reframe their contribution.
Minors:
- Figure 3 has a placeholder caption
- The Ethics, Reproducibility, Author Contributions, and Acknowledgments sections contain template text
- l.070, typo: "λ ≥ be the scalar"
- l.231, syntax: "for contextual , both"
- l.431, wrong reference: Fig. 5 should be Fig. 2.
- l.409, grammar: "Let We draw..."
- l.413, grammar: "and base policy's weightage on picking"
- l.490, typo: "componet"
- Section 2 defines online and offline bandit regret, which is never used
- ll. 227 & 232, Example 3.3 should be Lemma 3.3
- Definition 2.1 mixes and
- the Midpoint checkpoint is reported in Table 1 but not in Figs. 1 & 2
- l.1825: " is the winner, but trails close"
Questions for authors
-
What are and on the MS MARCO feedback pools? Are any grid points inside the regime of Thm 3.2?
-
How exactly are states weighted in the pairwise term of the MDP objective? Does Lemma 3.7 hold under that definition with a single global ?
-
Can you report multiple seeds with confidence intervals, plus DPO and SimPO under the same pair sampling as the hinge losses?
6 comments
Response from Authors Part 2
the conditions on which depend the theorems are not tested: empirical alignment isn't measured
In the generative retrieval experiment, we have taken three checkpoints from a common training trajectory with increasing average rewards, which is an indicator of how aligned is with the rewards. We report the and numbers here, taken from Table 1 of our draft.
| Early | Final | |
|---|---|---|
| 0.53 | 0.63 | |
| 0.001 | 0.002 |
However, we agree with the reviewer that a more principled way is to quantify this reward difference between and in terms of Kendall's correlation based alignment between and , which we do in the synthetic experiments of Appendix C.1.
Synthetic experiment to validate alignment-based theorem
Reward structure. We use frozen real-query features, fixed success probability strictly between zero and one, and a stochastic binary reward sampled using these probabilities.
Shared parametric policy: To demonstrate robustness of our theoretical predictions for real classes of parametrized policies, we do the experiments for shared MLP and linear softmax (i.e., log-linear) class of policies that we optimize.
Alignment measurement: Kendall's between and , with values in .
Observations: We measure the difference between the expected rewards of the two contrastive losses compared in Theorem 3.6. A positive difference favors RW; a negative difference favors UNW. We report the reward differences across 8 positive SFT weights and 5 margins, in aligned and anti-aligned settings. Its grid-level summary is:
| Shared model | Alignment and expected ordering | Strictly predicted direction | Ties | Reversals |
|---|---|---|---|---|
| Log-linear | Aligned: RW above UNW | 34/40 | 6/40 | 0/40 |
| Log-linear | Anti-aligned: UNW above RW | 32/40 | 8/40 | 0/40 |
| MLP | Aligned: RW above UNW | 37/40 | 3/40 | 0/40 |
| MLP | Anti-aligned: UNW above RW | 37/40 | 3/40 | 0/40 |
Thus, the predicted gain in reward based on empirical alignment holds on gradient based learning on parametrized policy classes as well, validating Theorem 3.6 in a robust setting.
While it claims to explain why combined losses work in LM post-training, experiments use a generative retrieval task.
We'd like to politely point out that generative retrieval is essentially aligning a language model to preference data. When done on a pretrained language model, this is essentially a version of post-training with preferential data which falls exactly under our theoretical framework, making it a natural choice for empirical assessment. However, during the rebuttal we have done post training on a Qwen 2.5 Math 1.5B model on math reasoning, which is a more direct instance of LM post training.
Per question, we sample responses without beam search, collected from the base model. On the training set, the reward is binary answer correctness: exact-rational verification for GSM8K. On the test dataset of MATH-500 questions, we have used Math-Verify 0.9.0. We have used a similar tuning budget for all baselines and contrastive losses. We used margins 376 (called low) and 3765 (called high) on the sequence-summed log-probabilities.
We first report the accuracy (expected binary reward) of the best configurations of each method in the following table.
| Method | Accuracy |
|---|---|
| Base model () | |
| SFT | |
| DPO () | |
As observed, (1) DPO does the best, followed by the contrastive hinge losses. (2) The contrastive methods beat SFT, which improves over the base model.
Now we see the margin and SFT anchor contributions for the unweighted and reweighted contrastive losses.
| accuracy | accuracy | ||
|---|---|---|---|
| Low | |||
| High | |||
| Low | |||
| High | |||
| Low | |||
| High | |||
| SFT | — |
As observed, contribution from both losses is essential for better performance. For binary exact rewards (this setting), exact SFT is essentially greedy (hence optimal) but it happens so that the parametric model is unable to replicate exact success conditioning while the contrastive losses remain more robust.
Response from Authors Part 3
Results only focus on performance. In an idealized bandit setting (exact unrestricted tabular policy) the optimal policy is greedy on Q, so it is unsurprising that a more aggressive update beats SFT once it is well defined.
We agree that a policy greedy in maximizes bandit reward. It does not, however, imply that giving the contrastive term more weight moves the learned policy toward that optimum.
In fact, Lemma 3.4 of our paper gives a two-action counterexample to this statement: if and , then for every finite (where is the weight on the SFT loss and is the margin). At , pairs favoring the better action already satisfy the margin and give no gradient. But the worse action sometimes succeeds while the better one fails, and these pairs still push probability toward the worse action.
Thus a stronger contrastive update can reduce reward. And this is what led us to analyze the existence of margin and contrastive weights in Theorem 3.2 that improve the contrastive loss optimizer over SFT.
The abstract claims the article "propose another combined family" with a DPO-style term. However, is essentially the SLiC-HF objective (Zhao et al.,2023), and the contrastive part of is the hinge member of the GPO family (Tang et al., 2024). The author should reframe their contribution.
The phrase "propose another combined family" was used mistakenly, and we have removed it. We do not propose either objective: the revised introduction credits SLiC-HF (Zhao et al., 2023) for the unweighted loss and GPO (Tang et al., 2024) for the reweighted hinge term. Our contribution is to show when the reweighted loss beats the unweighted loss and success-only SFT, both theoretically in an RL framework, by analyzing the SFT weight, the margin, and the base policy's alignment with the rewards (Thms. 3.2 and 3.6) as well as empirically on synthetic bandits and MS MARCO.
What are $\lambda^{*} and on the MS MARCO feedback pools? Are any grid points inside the regime of Thm 3.2?
We want to highlight the fact that Theorem 3.2 is an existential guarantee: it establishes that suitable finite SFT weights and margins can yield improvement over success conditioning. Its constructive bounds are sufficient conditions, rather than a characterization of the smallest parameters needed for improvement. Accordingly, the theorem provides theoretical guidance, but not a necessary prescription for selecting hyperparameters.
In our experiments, we minimize a combined loss of the form , where the pairwise loss is given by , with baseline re-weighting for the re-weighted version. In our MS MARCO experiments, we have varied . We now report the sufficient] bounds for Theorem 3.2 from the feedback pools of Early, Midpoint and Final checkpoints:
| Checkpoint | Sufficient | Sufficient | Sufficient | Largest grid |
|---|---|---|---|---|
| Early | 16 | 77 | (8, 10) | |
| Midpoint | 17 | 80 | (8, 10) | |
| Final | 16 | 88 | (8, 10) |
Therefore, no tested grid point satisfies Theorem 3.2's sufficient conditions uniformly across a checkpoint’s tested queries.
The empirical grid nevertheless contains configurations that improve over trained SFT. In a matched comparison, several configurations improve both of the learnt and over SFT. We report some of those grid coordinates and the improvements below:
| Checkpoint | |||
|---|---|---|---|
| Early | |||
| Early | |||
| Midpoint | |||
| Midpoint | |||
| Final | |||
| Final |
Thus, Theorem 3.2's conditions are sufficient, not necessary: failing its sufficient certificate does not preclude the empirical improvements observed on MS MARCO.
Response from Authors Part 1
We thank the reviewer for their questions that will improve our paper. Following their suggestions, we have made several changes in the updated draft, which are written in teal colour.
We think there might be some misunderstanding here. The goal of finetuning using rewards coming from an environment is to ensure that the reward provided by the same environment is maximized at test time. This is the goal of RL and our work is based on this setup. Robustness under distribution shift is another interesting direction for future work. As suggested by the reviewer, we have done some preliminary experiments and we're sharing the results below.
To compare the robustness of contrastive learning versus SFT, we have taken our trained checkpoints for the BART model (trained using RW, UNW, SFT and other baselines), and evaluated them using a different reward model. For training, the feedback was provided by the Qwen3-Reranker-0.6B model and we have used the publicly available Qwen3-Reranker-4B model for evaluation on the test data. We report the Monte Carlo expected reward for the various methods on the Qwen 4B model in the following table, where we have the best configuration for each method.
We observe: (1) πrw and πunw have higher expected reward on the different feedback, indicating higher robustness than baselines. (2) πunw is better than πrw for the mid, final checkpoints. We hypothesize that the more contrastive factor from the adaptive lnπ0 dependent margins might have optimized harder on the training reward, hurting out of distribution generalization.
We now analyze the variation in expected reward with the weightage of the SFT component in the overall loss. As seen in the following table, (1) an optimum SFT weight happens in the interior where both losses contribute and (2) a combined loss beats both pure contrastive and SFT endpoints in terms of distributional robustness as well.