- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of robust LLM post-training, aiming at minimizing the maximal loss over all potential domains. The paper fomulate this problem as an online learning regret minimization problem, and then identified a regret decomposition that upper bound the regret with the regret of a online optimization, plus the regret of bandit with drifting reward. Algorithms for these problems, together, indicate an regret upper bound of regret upper bound, thus a sample complexity upper bound of . The paper further conducts experiment on three training objectives and two models.
Critical strengths
- This paper provide a pretty clean mathematical formulation of robust online post-training. This formulation naturally leads to the application of algorithms like online gradient descent and non-stationary bandit algorithm.
- The paper is generally cleanly written and easy to follow.
Critical weaknesses
- The problem setup introduces several (seemingly) quite strong assumptions. While it is reasonable to make some assumptions for technical simplicty, the following assumptions might be too strong in practice and lacks corresponding justification
- Assumption 1 requires that the algorithm output is restricted to a pretty narrow area. This might not be realistic in LLM practice.
- Similarily, Assumption 2 requires the loss function to be Lipschitz. This might be not sufficiently realistic, as the landscape of the loss induced by LLM might be very spiky
- The problem setup it self assume several separate data sources. However, in practice, given a dataset that mixes different domain, how should the learner reliably divide the training set so that different part represent different source?
- The experiment setup is not clear enough. One of the key problem setup is the assumption of separate data source. However, in the presentation of Section 4.1, it is unclear whether the adopted datasets have such explicit separation.
- Also, could the author explain why the dataset for DPO and GRPO differes?
- It looks like that DUCB sometimes leads to inferior performance compare to baselines.
- Could the author clarify if there is any additional advantage of DUCB over other non-uniform baselines?
- Could the author provide discussion on the performance of DUCB? If it is due to discrepancy between theory and practice, then which assumptions in theory might be violated?
Questions for authors
See weakness section.
2 comments
I thank the authors for the detailed responses. I have updated my rating accordingly.
Assumption Clarification, Source Construction, and Empirical Performance
We thank the reviewer for requesting clearer explanations of the assumptions, source construction, and evaluation metrics. We have also refined the assumption statements (page 4, lines 182 - 188), and further clarified the experimental setup (page 7, line 368 - 415) accordingly in the updated PDF.
Weakness 1: Clarification of Bounded-Domain and Lipschitz Assumptions
Assumption Relevance: Assumptions 1 and 2 are used only for the convex OGD analysis and do not claim that a transformer is globally Lipschitz over its full parameter space. It also does not require this region to be narrow or successive iterates to remain close to a fixed point. Contrarily, we state the bounded domain condition for the iterates visited by the training trajectory and require Lipschitzness only on a bounded region containing the iterates visited during model training.
Empirical Justification: We report empirical diagnostics in Appendix-D.5, which shows the maximum per-step source-loss variations are modest (0.06-0.17 across all tasks). These observations are consistent with the slowly-varying and trajectory-level Lipschitzness interpretation used by the analysis.
Regularization During Practical Training: Practical training mechanisms also help keep the optimization trajectory bounded through techniques such as gradient clipping. Moreover, LLM post-training commonly employs KL regularization with respect to a reference model. This mechanism discourages large deviations from the fixed reference model, which further supports the boundedness condition.
Weaknesses 2 & 3: Data Source Definition and Dataset Selection
Data Source Definition: Our framework assumes that each training example has a source label before training. The sampler first selects a source and then draws a mini-batch from the corresponding subset. In our experiments, these labels are obtained directly from dataset domains or task categories:
Datasets of DPO v.s. GRPO: DPO and GRPO use different datasets because they require different forms of supervision: DPO requires paired preferred-dispreferred responses, whereas GRPO requires prompts with task-compatible reward signals for on-policy rollouts. They use different datasets because they belong to fundamentally different task families..
Weakness 4: When DUCB Helps and Why It Can Underperform
The Robustness Objective and Minimum Source-Wise Performance: The robust learning problem studied in our framework targets minimum source-wise performances rather than average accuracy. In the six main settings, our proposed DUCB achieves the highest minimum accuracy in five settings and remains competitive with the top performing baseline (i.e. AMA-S) in the remaining setting. We note that some baselines obtain higher average accuracies due to the expected minimum-average trade-off, since DUCB tends to allocate more training budget to the data source with comparatively worse performances.
Advantages of DUCB Compared to Other Baselines: DUCB’s practical advantage is strongest when the worst source changes during training (as shown in Tab-1, Sec-4.3, and Appendix-D.5). It is also suitable when only instantaneous mini-batch feedback is available or affordable. For example, although AMA-S slightly exceeds DUCB in one of the DPO settings, it requires training K additional source-specific specialist models. In contrast, DUCB requires no auxiliary model training or historical re-evaluation, maintains only O(K) scalar sampler state, and has per-step runtime comparable to Uniform sampling.
Relation Between Theory and Empirical Performances: The proposed theory provides a regret guarantee for population training losses under convexity and regularity assumptions. The mismatch between training loss as a sampling signal and downstream evaluation metrics could also affect the performance of different algorithms, since minimizing the training loss is not always equivalent to minimizing the downstream worst-source accuracy.
Nevertheless, although the theoretical bounds does not guarantee that DUCB will dominate every specialized baseline in finite-horizon downstream task accuracy, the strong minimum-accuracy results of DUCB across five of the six main settings suggests that the proposed DUCB has considerable practical relevance.