- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies LLM fine-tuning under heterogeneous and imbalanced training signals, where optimizing average utility can mask regressions on underrepresented or intrinsically difficult prompt groups. The paper proposes an adversarial reweighting framework in which an auxiliary adversary dynamically assigns larger training weights to groups or examples on which the current policy performs poorly, while the LLM is updated using the resulting weighted reward-likelihood objective. The framework instantiates three notions of fairness: group-aware worst-group performance, a group-agnostic variant using instruction-dependent weights, and relative fairness that seeks to improve the worst group relative to a reference policy. The method is evaluated on an imbalanced recipe-generation task with logged scalar rewards and on BIGPATENT summarization formulated as instruction tuning. The experiments report worst-group, average, and relative-performance metrics and generally show improved worst-group performance over naive fine-tuning and inverse-frequency weighting. The appendix additionally provides an idealized connection between adversarial group reweighting and max-min optimization, together with a stationarity result under smoothness and adversary-tracking assumptions
Critical strengths
-
The paper formulates a practically meaningful fairness problem and provides a clean unified treatment of several objectives. In particular, the distinction between absolute worst-group performance, group-agnostic robustness, and worst-group improvement relative to a reference policy is useful. I find the relative-fairness objective especially meaningful for model fine-tuning, since it directly captures the practical concern that an updated model should not substantially regress on particular groups relative to an existing policy.
-
The empirical results exhibit a coherent objective-alignment pattern rather than merely showing uniform gains from all variants. On recipe generation, the cuisine-aware variant performs best when evaluated over cuisine groups, the rarity-aware variant performs best over ingredient-rarity groups, while the universal variant remains competitive under both groupings. This pattern supports the intended interpretation of the different weighting modes. The patent-summarization experiments further show corresponding worst-group improvements under both LLM-based evaluation and the independent SummaC factual-consistency metric.
-
The evaluation is reasonably careful in several respects. The recipe experiments average results over 10 independently regenerated train/validation/test logs, the summarization experiments cover a substantially different instruction-tuning regime, and the theoretical appendix clearly distinguishes its idealized population analysis from a global convergence guarantee for the practical neural-network implementation.
Critical weaknesses
- There is a substantial gap between the population policy-value objective used to motivate the method and the logged-data surrogate actually optimized by the practical algorithm.
The paper defines group performance as , where the action in the population objective is sampled from the target policy . In contrast, the observed responses in the training data are generated by a different logging policy . However, the practical policy update is proportional to , without an explicit correction for the mismatch between and .
In standard learning from logged bandit feedback, this distribution mismatch is typically handled using propensity or importance weighting, for example through a ratio such as . More importantly, Appendix G analyzes the idealized population quantities , whereas the practical algorithm replaces them with a logged-data likelihood surrogate. I therefore do not see a derivation showing that the practical objective is an estimator of, or a justified surrogate for, the stated worst-group policy value. This weakens the interpretation of the method as worst-group off-policy optimization, even though the resulting weighted-likelihood procedure may still work empirically.
- The empirical comparison is too weak to establish the benefit of the proposed adaptive adversarial optimization over closely related alternatives.
The main experiments compare primarily against naive fine-tuning and static inverse-frequency weighting, while most of the remaining rows are variants of the proposed method. These results establish that adaptive weighting can outperform the two reported baselines, but they do not establish whether the proposed approach improves over existing adaptive worst-group or adversarial reweighting methods.
For the known-group setting, Group DRO is a particularly direct comparison because it also adaptively optimizes worst-group performance. For the group-agnostic setting, adversarially reweighted learning methods are similarly relevant. Since adaptive reweighting is presented as a central methodological contribution, the absence of such comparisons makes it difficult to assess both the empirical significance and the incremental novelty of the proposed optimizer.
Questions for authors
- Could the authors clarify the relationship between the practical logged-data objective and the population policy-value objective?
The manuscript defines group performance through , where actions in this expectation are generated by the target policy , while the observed training responses are sampled from the logging policy . Why is the practical update an appropriate surrogate for improving without an explicit off-policy correction such as the propensity ratio ?
Relatedly, how is in the adversary objective computed in practice? Is it the raw sequence probability, summed token log-probability, mean token log-probability, or a length-normalized score?
- Do the authors have comparisons against adaptive worst-group or adversarial reweighting baselines?
In the known-group setting, Group DRO appears to be a direct comparison, while in the group-agnostic setting an ARL-style adversarial reweighting method appears closely related. Such comparisons would materially affect my assessment because the current Naive and Inverse Weighting baselines do not isolate the benefit of the proposed adaptive adversary relative to established adaptive robust/fairness optimization methods.
5 comments
2. On the Comparison with Existing Adaptive Reweighting Methods
The reviewer raises concerns about the lack of comparisons with established adaptive reweighting methods, such as Group DRO and adversarially reweighted learning (ARL). However, these methods were originally developed for different problem settings. Group DRO optimizes worst-group supervised loss under predefined group labels, whereas ARL learns adversarial sample weights without predefined groups in supervised learning. In contrast, our framework addresses autoregressive LLM fine-tuning from heterogeneous training signals, including logged scalar rewards, and supports multiple fairness objectives.
The following table highlights the differences in their original formulations.
| Feature | Group DRO | ARL | Ours |
|---|---|---|---|
| Autoregressive LLM fine-tuning | – | – | ✓ |
| Logged responses with scalar rewards | – | – | ✓ |
| Group-aware worst-group objective | ✓ | – | ✓ |
| Group-agnostic, input-dependent weighting | – | ✓ | ✓ |
| Reference-relative worst-group objective | – | – | ✓ |
| Unified treatment of the three objectives | – | – | ✓ |
“–” indicates that the feature is not explicitly addressed in the original formulation, rather than being incompatible with possible extensions.
To directly address the reviewer's concern, we conducted additional experiments adapting Group DRO to our group-aware fine-tuning setting. The results are summarized below.
Recipe Generation
| Metric | Group DRO | Ours |
|---|---|---|
| Cuisine | 12.17 ± 0.12 | 12.25 ± 0.07 |
| Cuisine | +0.005 ± 0.018 | +0.013 ± 0.007 |
| Rarity | 12.03 ± 0.13 | 12.28 ± 0.07 |
| Rarity | −0.008 ± 0.023 | +0.012 ± 0.008 |
Patent Summarization
| Metric | Group DRO | Ours |
|---|---|---|
| CPC, LLM-based | 12.63 ± 0.91 | 12.91 ± 0.72 |
| CPC, SummaC | 0.606 ± 0.032 | 0.619 ± 0.024 |
| Length, LLM-based | 12.91 ± 0.61 | 13.19 ± 0.42 |
| Length, SummaC | 0.694 ± 0.022 | 0.698 ± 0.015 |
Our method consistently achieves higher mean worst-group performance than the adapted Group DRO baseline across both tasks and all evaluated groupings, including improvements in reference-relative fairness and the independent SummaC factual-consistency metric.
Regarding ARL, when adapted to our training formulation, its adversarial weighting mechanism can be viewed as a special case of our group-agnostic, absolute-fairness formulation. Our framework further extends this principle to predefined groups, reference-relative fairness, and heterogeneous training signals within a unified approach. Importantly, the applicability of an existing optimization technique does not imply that the broader fairness formulations and empirical findings investigated in our work have already been established.
Thus, our contribution lies not in inventing adversarial reweighting itself, but in developing a more general framework for fairness-aware LLM fine-tuning that unifies multiple objectives and training settings. We believe these distinctions, together with the additional Group DRO results, directly address the reviewer's concerns regarding the incremental contribution and the adequacy of our baseline comparisons.
3. Contribution of the group-agnostic and relative-fairness evaluations
We appreciate the reviewer's observation that the empirical results show a coherent pattern aligned with the proposed fairness objectives. This is particularly evident in the group-agnostic variant.
Without using cuisine or ingredient-rarity labels during training, our universal method consistently improves worst-group performance over Naive across both evaluation groupings, demonstrating its effectiveness in enhancing robustness without requiring predefined group information. Moreover, the same universal variant yields positive mean worst-group relative improvements under both groupings with respect to the fixed reference policy. These results suggest that adaptive instruction-dependent weighting can improve performance across multiple evaluation groupings, including groupings unavailable to the training adversary.
The empirical contribution therefore goes beyond showing that a known group can be assigned a larger training weight. It examines the interaction between available group information, the chosen fairness objective, and the resulting subgroup performance.
4. Scope of the theoretical and empirical claims
We also wish to clarify that the theoretical analysis in Appendix G is explicitly idealized and conditional. It does not claim a practical convergence guarantee for the full neural implementation.
Likewise, our empirical conclusions are limited to the considered tasks, backbone, and evaluation protocols. Within this stated scope, we believe the combination of multiple fairness objectives, two different fine-tuning regimes, and evaluations based on both task-specific reward and factual consistency constitutes a substantive empirical investigation.
5. Clarification of Sequence-Level Score Computation
We clarify that the quantity denoted by in the adversary objective is implemented as the exponential of the mean token log-probability, i.e., a length-normalized sequence score. This normalization mitigates the dependence of sequence probability magnitudes on response length and improves numerical stability compared with directly multiplying token probabilities. We acknowledge that this score differs from the exact sequence probability in the mathematical formulation, and we have revised the manuscript to explicitly clarify this implementation detail and the distinction between the two quantities.
We thank the reviewer for the thoughtful review and for recognizing the practical relevance of the fairness objectives, the objective-aligned empirical results, and the careful evaluation across different tasks and metrics. We would like to address the two main concerns regarding the practical optimization objective and comparisons with related adaptive methods. We have revised the manuscript to clarify the implemented objective and add direct Group DRO comparisons, which we believe address these major concerns.
1. Clarification of the population objective and practical surrogate
We agree that the distinction between the population objective and the practical training surrogate was insufficiently clear in the original submission, and we have now made it explicit in the revised manuscript.
The population group value is defined under responses sampled from the target policy πθ, whereas the practical objective uses responses logged under π0. Estimating the target-policy value from logged samples generally requires importance weighting by pi=πθ(ai∣xi)/π0(ai∣xi), assuming sufficient support. Without this correction, the empirical objective is not necessarily an unbiased estimator of the population value.
However, an important feature of our framework is the flexibility of the learned adversarial weights, which can potentially incorporate the effects of importance weighting while simultaneously prioritizing poorly performing groups. Conceptually, a sufficiently expressive sample-level weighting function could represent a combined effective weight of the form wi=wi′pi, where wi′ captures adversarial emphasis and pi accounts for the logging-policy distribution. Since Eq. (2) already includes πθ(ai∣xi), the corresponding correction within wϕ would involve the inverse logging propensity rather than the full importance ratio.
Moreover, explicit importance weighting is well known to suffer from high variance, particularly for autoregressive sequences, potentially destabilizing training without necessarily improving empirical performance. Indeed, our experimental results demonstrate improvements in worst-group performance across multiple fairness criteria, supporting the practical effectiveness of our approach without explicit importance weighting. While our learned weights may implicitly capture aspects of propensity correction, we do not claim that the current parameterization guarantees recovery of exact importance ratios or unbiased population-value estimation.