acceptodds
Under review as a conference paper at ICLR 2027

Adversarial Learning for Fair LLM Fine-Tuning from Imbalanced Training Signals

Abstract

Fine-tuning large language models (LLMs) using logged reward data can inadvertently lead to unfair performance disparities across prompt groups, particularly when training data is imbalanced or groups differ in inherent difficulty. In the context of personalized recipe generation, we observe that naive fine-tuning to optimize towards the expected reward severely degrades performance for underrepresented or challenging prompts (e.g., specific cuisines or rare ingredients). This motivates the need for more principled approaches to fine-tuning LLMs in the presence of biased distributions and varying task difficulties. We propose a novel adversarial offline policy optimization method for LLM fine-tuning, aiming to improve worst-case group performance, thereby addressing performance disparities. Our adversarial method alternates between reweighting training samples and updating the policy, which supports a wide range of fairness goals such as enforcing fairness for specific vulnerable groups, achieving balanced performance across all possible groups, and optimizing for uniform relative group-level improvements compared to a reference policy. We empirically validate our proposed approach through controlled experiments in recipe generation and content summarization. Our method consistently improves worst-case group performance, narrowing across-group performance gaps observed in baseline methods.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM fine-tuning, instruction tuning, fairness, performance disparity, adversarial learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NQftOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies LLM fine-tuning under heterogeneous and imbalanced training signals, where optimizing average utility can mask regressions on underrepresented or intrinsically difficult prompt groups. The paper proposes an adversarial reweighting framework in which an auxiliary adversary dynamically assigns larger training weights to groups or examples on which the current policy performs poorly, while the LLM is updated using the resulting weighted reward-likelihood objective. The framework instantiates three notions of fairness: group-aware worst-group performance, a group-agnostic variant using instruction-dependent weights, and relative fairness that seeks to improve the worst group relative to a reference policy. The method is evaluated on an imbalanced recipe-generation task with logged scalar rewards and on BIGPATENT summarization formulated as instruction tuning. The experiments report worst-group, average, and relative-performance metrics and generally show improved worst-group performance over naive fine-tuning and inverse-frequency weighting. The appendix additionally provides an idealized connection between adversarial group reweighting and max-min optimization, together with a stationarity result under smoothness and adversary-tracking assumptions

Critical strengths
  1. The paper formulates a practically meaningful fairness problem and provides a clean unified treatment of several objectives. In particular, the distinction between absolute worst-group performance, group-agnostic robustness, and worst-group improvement relative to a reference policy is useful. I find the relative-fairness objective especially meaningful for model fine-tuning, since it directly captures the practical concern that an updated model should not substantially regress on particular groups relative to an existing policy.

  2. The empirical results exhibit a coherent objective-alignment pattern rather than merely showing uniform gains from all variants. On recipe generation, the cuisine-aware variant performs best when evaluated over cuisine groups, the rarity-aware variant performs best over ingredient-rarity groups, while the universal variant remains competitive under both groupings. This pattern supports the intended interpretation of the different weighting modes. The patent-summarization experiments further show corresponding worst-group improvements under both LLM-based evaluation and the independent SummaC factual-consistency metric.

  3. The evaluation is reasonably careful in several respects. The recipe experiments average results over 10 independently regenerated train/validation/test logs, the summarization experiments cover a substantially different instruction-tuning regime, and the theoretical appendix clearly distinguishes its idealized population analysis from a global convergence guarantee for the practical neural-network implementation.

Critical weaknesses
  1. There is a substantial gap between the population policy-value objective used to motivate the method and the logged-data surrogate actually optimized by the practical algorithm.

The paper defines group performance as Vg(πθ)=E[q(x,a)]V_g(\pi_\theta)=\mathbb{E}[q(x,a)], where the action aa in the population objective is sampled from the target policy πθ\pi_\theta. In contrast, the observed responses in the training data are generated by a different logging policy π0\pi_0. However, the practical policy update is proportional to wiri∇θlog⁡πθ(ai∣xi)w_i r_i \nabla_\theta \log \pi_\theta(a_i \mid x_i), without an explicit correction for the mismatch between πθ\pi_\theta and π0\pi_0.

In standard learning from logged bandit feedback, this distribution mismatch is typically handled using propensity or importance weighting, for example through a ratio such as πθ(ai∣xi)/π0(ai∣xi)\pi_\theta(a_i \mid x_i) / \pi_0(a_i \mid x_i). More importantly, Appendix G analyzes the idealized population quantities VgV_g, whereas the practical algorithm replaces them with a logged-data likelihood surrogate. I therefore do not see a derivation showing that the practical objective is an estimator of, or a justified surrogate for, the stated worst-group policy value. This weakens the interpretation of the method as worst-group off-policy optimization, even though the resulting weighted-likelihood procedure may still work empirically.

  1. The empirical comparison is too weak to establish the benefit of the proposed adaptive adversarial optimization over closely related alternatives.

The main experiments compare primarily against naive fine-tuning and static inverse-frequency weighting, while most of the remaining rows are variants of the proposed method. These results establish that adaptive weighting can outperform the two reported baselines, but they do not establish whether the proposed approach improves over existing adaptive worst-group or adversarial reweighting methods.

For the known-group setting, Group DRO is a particularly direct comparison because it also adaptively optimizes worst-group performance. For the group-agnostic setting, adversarially reweighted learning methods are similarly relevant. Since adaptive reweighting is presented as a central methodological contribution, the absence of such comparisons makes it difficult to assess both the empirical significance and the incremental novelty of the proposed optimizer.

Questions for authors
  1. Could the authors clarify the relationship between the practical logged-data objective and the population policy-value objective?

The manuscript defines group performance through Vg(πθ)=E[q(x,a)]V_g(\pi_\theta)=\mathbb{E}[q(x,a)], where actions in this expectation are generated by the target policy πθ\pi_\theta, while the observed training responses aia_i are sampled from the logging policy π0\pi_0. Why is the practical update ri∇θlog⁡πθ(ai∣xi)r_i \nabla_\theta \log \pi_\theta(a_i \mid x_i) an appropriate surrogate for improving Vg(πθ)V_g(\pi_\theta) without an explicit off-policy correction such as the propensity ratio πθ(ai∣xi)/π0(ai∣xi)\pi_\theta(a_i \mid x_i) / \pi_0(a_i \mid x_i)?

Relatedly, how is πθ(ai∣xi)\pi_\theta(a_i \mid x_i) in the adversary objective computed in practice? Is it the raw sequence probability, summed token log-probability, mean token log-probability, or a length-normalized score?

  1. Do the authors have comparisons against adaptive worst-group or adversarial reweighting baselines?

In the known-group setting, Group DRO appears to be a direct comparison, while in the group-agnostic setting an ARL-style adversarial reweighting method appears closely related. Such comparisons would materially affect my assessment because the current Naive and Inverse Weighting baselines do not isolate the benefit of the proposed adaptive adversary relative to established adaptive robust/fairness optimization methods.

5 comments

We thank the reviewer for the thoughtful review and for recognizing the practical relevance of the fairness objectives, the objective-aligned empirical results, and the careful evaluation across different tasks and metrics. We would like to address the two main concerns regarding the practical optimization objective and comparisons with related adaptive methods. We have revised the manuscript to clarify the implemented objective and add direct Group DRO comparisons, which we believe address these major concerns.

1. Clarification of the population objective and practical surrogate

We agree that the distinction between the population objective and the practical training surrogate was insufficiently clear in the original submission, and we have now made it explicit in the revised manuscript.

The population group value is defined under responses sampled from the target policy πθ\pi_\theta, whereas the practical objective uses responses logged under π0\pi_0. Estimating the target-policy value from logged samples generally requires importance weighting by pi=πθ(ai∣xi)/π0(ai∣xi)p_i=\pi_\theta(a_i\mid x_i)/\pi_0(a_i\mid x_i), assuming sufficient support. Without this correction, the empirical objective is not necessarily an unbiased estimator of the population value.

However, an important feature of our framework is the flexibility of the learned adversarial weights, which can potentially incorporate the effects of importance weighting while simultaneously prioritizing poorly performing groups. Conceptually, a sufficiently expressive sample-level weighting function could represent a combined effective weight of the form w~i=wi′pi\widetilde{w}_i=w_i' p_i, where wi′w_i' captures adversarial emphasis and pip_i accounts for the logging-policy distribution. Since Eq. (2) already includes πθ(ai∣xi)\pi_\theta(a_i\mid x_i), the corresponding correction within wϕw_\phi would involve the inverse logging propensity rather than the full importance ratio.

Moreover, explicit importance weighting is well known to suffer from high variance, particularly for autoregressive sequences, potentially destabilizing training without necessarily improving empirical performance. Indeed, our experimental results demonstrate improvements in worst-group performance across multiple fairness criteria, supporting the practical effectiveness of our approach without explicit importance weighting. While our learned weights may implicitly capture aspects of propensity correction, we do not claim that the current parameterization guarantees recovery of exact importance ratios or unbiased population-value estimation.

2. On the Comparison with Existing Adaptive Reweighting Methods

The reviewer raises concerns about the lack of comparisons with established adaptive reweighting methods, such as Group DRO and adversarially reweighted learning (ARL). However, these methods were originally developed for different problem settings. Group DRO optimizes worst-group supervised loss under predefined group labels, whereas ARL learns adversarial sample weights without predefined groups in supervised learning. In contrast, our framework addresses autoregressive LLM fine-tuning from heterogeneous training signals, including logged scalar rewards, and supports multiple fairness objectives.

The following table highlights the differences in their original formulations.

FeatureGroup DROARLOurs
Autoregressive LLM fine-tuning––✓
Logged responses with scalar rewards––✓
Group-aware worst-group objective✓–✓
Group-agnostic, input-dependent weighting–✓✓
Reference-relative worst-group objective––✓
Unified treatment of the three objectives––✓

“–” indicates that the feature is not explicitly addressed in the original formulation, rather than being incompatible with possible extensions.

To directly address the reviewer's concern, we conducted additional experiments adapting Group DRO to our group-aware fine-tuning setting. The results are summarized below.

Recipe Generation

MetricGroup DROOurs
Cuisine Vmin⁡V_{\min}12.17 ± 0.1212.25 ± 0.07
Cuisine ΔVmin⁡\Delta V_{\min}+0.005 ± 0.018+0.013 ± 0.007
Rarity Vmin⁡V_{\min}12.03 ± 0.1312.28 ± 0.07
Rarity ΔVmin⁡\Delta V_{\min}−0.008 ± 0.023+0.012 ± 0.008

Patent Summarization

MetricGroup DROOurs
CPC, LLM-based Vmin⁡V_{\min}12.63 ± 0.9112.91 ± 0.72
CPC, SummaC Vmin⁡V_{\min}0.606 ± 0.0320.619 ± 0.024
Length, LLM-based Vmin⁡V_{\min}12.91 ± 0.6113.19 ± 0.42
Length, SummaC Vmin⁡V_{\min}0.694 ± 0.0220.698 ± 0.015

Our method consistently achieves higher mean worst-group performance than the adapted Group DRO baseline across both tasks and all evaluated groupings, including improvements in reference-relative fairness and the independent SummaC factual-consistency metric.

Regarding ARL, when adapted to our training formulation, its adversarial weighting mechanism can be viewed as a special case of our group-agnostic, absolute-fairness formulation. Our framework further extends this principle to predefined groups, reference-relative fairness, and heterogeneous training signals within a unified approach. Importantly, the applicability of an existing optimization technique does not imply that the broader fairness formulations and empirical findings investigated in our work have already been established.

Thus, our contribution lies not in inventing adversarial reweighting itself, but in developing a more general framework for fairness-aware LLM fine-tuning that unifies multiple objectives and training settings. We believe these distinctions, together with the additional Group DRO results, directly address the reviewer's concerns regarding the incremental contribution and the adequacy of our baseline comparisons.

3. Contribution of the group-agnostic and relative-fairness evaluations

We appreciate the reviewer's observation that the empirical results show a coherent pattern aligned with the proposed fairness objectives. This is particularly evident in the group-agnostic variant.

Without using cuisine or ingredient-rarity labels during training, our universal method consistently improves worst-group performance over Naive across both evaluation groupings, demonstrating its effectiveness in enhancing robustness without requiring predefined group information. Moreover, the same universal variant yields positive mean worst-group relative improvements under both groupings with respect to the fixed reference policy. These results suggest that adaptive instruction-dependent weighting can improve performance across multiple evaluation groupings, including groupings unavailable to the training adversary.

The empirical contribution therefore goes beyond showing that a known group can be assigned a larger training weight. It examines the interaction between available group information, the chosen fairness objective, and the resulting subgroup performance.

4. Scope of the theoretical and empirical claims

We also wish to clarify that the theoretical analysis in Appendix G is explicitly idealized and conditional. It does not claim a practical convergence guarantee for the full neural implementation.

Likewise, our empirical conclusions are limited to the considered tasks, backbone, and evaluation protocols. Within this stated scope, we believe the combination of multiple fairness objectives, two different fine-tuning regimes, and evaluations based on both task-specific reward and factual consistency constitutes a substantive empirical investigation.

5. Clarification of Sequence-Level Score Computation

We clarify that the quantity denoted by πθ(ai∣xi)\pi_\theta(a_i\mid x_i) in the adversary objective is implemented as the exponential of the mean token log-probability, i.e., a length-normalized sequence score. This normalization mitigates the dependence of sequence probability magnitudes on response length and improves numerical stability compared with directly multiplying token probabilities. We acknowledge that this score differs from the exact sequence probability in the mathematical formulation, and we have revised the manuscript to explicitly clarify this implementation detail and the distinction between the two quantities.

Reviewer_DBCHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes adaptive adversarial reweighting for LLM fine-tuning on imbalanced data. It supports group-aware, group-agnostic, and reference-relative objectives by alternating adversary and reward-weighted likelihood updates. Experiments on recipe generation and patent summarization report improved performance on the worst group compared to naive fine-tuning and inverse weighting. An idealized analysis connects adversarial weighting to max–min optimization.

Critical strengths
  1. The paper addresses a meaningful limitation of average-reward fine-tuning: improvements can conceal regressions for particular prompt groups. Considering both absolute performance and reference-relative improvements is practically relevant.

  2. Evaluation covers reward-based recipe training and supervised patent summarization, with ten random seeds and an additional factual-consistency metric. This provides useful empirical evidence beyond a single task or evaluator.

Critical weaknesses
  1. The practical objectives are not adequately connected to the stated policy values. Equation (2) averages πθ(a∣x)r\pi_\theta(a\mid x)r over actions sampled from the logging policy π0\pi_0, without propensity correction. In general, Ea∼π0(⋅∣x)[πθ(a∣x)q(x,a)]≠Ea∼πθ(⋅∣x)[q(x,a)].\mathbb{E}{a\sim\pi_0(\cdot\mid x)}[\pi\theta(a\mid x)q(x,a)]\neq\mathbb{E}{a\sim\pi\theta(\cdot\mid x)}[q(x,a)]. Meanwhile, the policy optimizes reward-weighted log-likelihood, rlog⁡πθ(a∣x)r\log\pi_\theta(a\mid x), a different surrogate. Appendix G analyzes population values rather than these actual updates, so it does not establish that the implemented adversary identifies the worst-performing groups. Full-sequence probabilities also introduce potential length sensitivity and numerical issues.

  2. The comparisons do not establish an advantage over existing adaptive robust training: only naive and inverse-weighting baselines are evaluated, despite discussing Group DRO and adversarial reweighted learning. The average-performance tradeoff also deserves stronger qualification.

Questions for authors
  1. How does the method compare with Group DRO and adversarial reweighted learning under the same backbone and tuning budget?

  2. How are patent descriptions of up to 10,000 words handled with the stated 1,024-token training limit, and how does truncation affect the short/long-group comparison?

4 comments

We thank the reviewer for the detailed technical assessment. We would like to clarify an important distinction between the population-level fairness objectives, the practical training surrogate, and the empirical evaluation. We have revised the manuscript to make this distinction explicit and incorporate additional Group DRO results, which we believe address the major concerns raised in this review.

1. On the relationship between population policy values and logged-data optimization

We agree that the distinction between the population objective and the practical training surrogate could have been made clearer.

The population group value is defined under responses sampled from the target policy πθ\pi_\theta, whereas the practical objective uses responses logged under π0\pi_0. Estimating the target-policy value from logged samples generally requires importance weighting by pi=πθ(ai∣xi)/π0(ai∣xi)p_i=\pi_\theta(a_i\mid x_i)/\pi_0(a_i\mid x_i), assuming sufficient support. Without this correction, the empirical objective is not necessarily an unbiased estimator of the population value.

However, an important feature of our framework is the flexibility of the learned adversarial weights, which can potentially incorporate the effects of importance weighting while simultaneously prioritizing poorly performing groups. Conceptually, a sufficiently expressive sample-level weighting function could represent a combined effective weight of the form w~i=wi′pi\widetilde{w}_i=w_i' p_i, where wi′w_i' captures adversarial emphasis and pip_i accounts for the logging-policy distribution. Since Eq. (2) already includes πθ(ai∣xi)\pi_\theta(a_i\mid x_i), the corresponding correction within wϕw_\phi would involve the inverse logging propensity rather than the full importance ratio.

Moreover, explicit importance weighting is well known to suffer from high variance, particularly for autoregressive sequences, potentially destabilizing training without necessarily improving empirical performance. Indeed, our experimental results demonstrate improvements in worst-group performance across multiple fairness criteria, supporting the practical effectiveness of our approach without explicit importance weighting. While our learned weights may implicitly capture aspects of propensity correction, we do not claim that the current parameterization guarantees recovery of exact importance ratios or unbiased population-value estimation.

2. On the baseline comparison

The reviewer suggests comparisons with Group DRO and adversarially reweighted learning (ARL). We would like to emphasize that these methods were originally developed for different problem settings. Group DRO addresses worst-group optimization with predefined groups, whereas ARL learns adversarial weights without group labels, both originally in supervised learning settings. In contrast, our framework supports autoregressive LLM fine-tuning from logged scalar rewards and multiple fairness objectives, including reference-relative fairness.

The following table summarizes the differences in their original formulations.

FeatureGroup DROARLOurs
(Traditional) Supervised learning✓✓–
LLM fine-tuning––✓
Group-aware worst-group objective✓–✓
Group-agnostic, input-dependent weighting–✓✓
Reference-relative worst-group objective––✓
Unified treatment of the three objectives––✓

Here, "–" indicates that the feature is not explicitly addressed in the original formulation, rather than being incompatible with possible extensions.

To address the reviewer's concern, we conducted additional experiments adapting Group DRO to our group-aware fine-tuning setting. The following tables compare Group DRO with our method under the corresponding group definitions.

Recipe Generation

MetricGroup DROOurs
Cuisine Vmin⁡V_{\min}12.17 ± 0.1212.25 ± 0.07
Cuisine ΔVmin⁡\Delta V_{\min}+0.005 ± 0.018+0.013 ± 0.007
Rarity Vmin⁡V_{\min}12.03 ± 0.1312.28 ± 0.07
Rarity ΔVmin⁡\Delta V_{\min}−0.008 ± 0.023+0.012 ± 0.008

Patent Summarization

MetricGroup DROOurs
CPC, LLM-based Vmin⁡V_{\min}12.63 ± 0.9112.91 ± 0.72
CPC, SummaC Vmin⁡V_{\min}0.606 ± 0.0320.619 ± 0.024
Length, LLM-based Vmin⁡V_{\min}12.91 ± 0.6113.19 ± 0.42
Length, SummaC Vmin⁡V_{\min}0.694 ± 0.0220.698 ± 0.015

Our method achieves higher worst-group performance than the adapted Group DRO baseline across both tasks and all evaluated groupings, including improvements in reference-relative fairness and the independent SummaC metric.

Regarding ARL, when adapted to our training formulation, its adversarial weighting mechanism can be viewed as a special case of our group-agnostic, absolute-fairness formulation. Our framework further accommodates predefined groups, reference-relative fairness, and heterogeneous training signals within a unified approach.

Thus, our contribution goes beyond applying existing adversarial reweighting techniques: it provides a more general framework for fairness-aware LLM fine-tuning, with additional empirical evidence of its effectiveness over Group DRO. We believe these results and distinctions directly address the reviewer's concerns regarding the adequacy of our baseline comparisons.

3. On the theoretical analysis

Appendix G explicitly states that the analysis concerns an idealized population-level optimization principle and does not establish global convergence of the practical neural implementation.

Proposition 1 characterizes the equivalence between idealized adversarial group weighting and worst-group optimization. Theorem 1 establishes a conditional stationarity result under smoothness and adversary-tracking assumptions.

These results are intended to explain the optimization principle rather than provide a finite-sample guarantee for the logged-data training algorithm. Instead, the effectiveness of our method in finite-sample settings is empirically demonstrated through extensive experiments across two distinct fine-tuning tasks, multiple group definitions, and evaluation metrics.

We have now emphasized this distinction more prominently in the revised manuscript. We also retain the scope qualification in Appendix G, which we believe should be considered when assessing the theoretical results.

Thanks for adding the Group DRO comparison and clarifying the scope of the theory. The new results address my concern about the missing adaptive baseline.

I still have some concern about the connection between the practical training objective and the stated worst-group objective. The response suggests that the learned weights could account for the logging policy, but does not show that this happens in practice. I understand the motivation for avoiding explicit importance weighting, though the connection to the population objective remains unclear.

The handling of long patent descriptions under the 1,024-token training limit also remains unclear. Without knowing the truncation procedure, I find it difficult to interpret the comparison between short and long inputs.

We sincerely thank the reviewer for the constructive follow-up and for acknowledging that our additional Group DRO experiments and theoretical clarifications have addressed the concerns regarding adaptive baselines.

Regarding the population objective and practical surrogate, we agree that the practical objective should not be interpreted as an unbiased off-policy estimator of population worst-group performance. To clarify our earlier explanation, we do not claim that the learned adversarial weights necessarily recover the logging-policy importance ratios, nor do we rely on such recovery for our empirical claims. Our goal is to learn adaptive weights that improve the empirical fairness objective (rather than to achieve unbiased objective estimation) without imposing explicit sequence-level importance ratios, which can introduce substantial variance.

Importantly, while we do not directly demonstrate that the learned weights recover the logging-policy correction, our experiments do demonstrate its practically relevant consequence; optimizing the practical surrogate using data collected under the logging policy leads to improved worst-group performance on newly generated outputs for held-out inputs. The population max–min objective provides the motivation for this design, while its practical effectiveness is assessed directly on these newly generated outputs. The observed improvements across multiple fairness criteria and evaluation metrics therefore provide empirical support for our intended contribution, independently of any claim of unbiased population-value estimation or explicit recovery of the logging-policy importance ratios.

Regarding the handling of long patent descriptions, we clarify that the 1,024-token limit stated in the original manuscript was a typographical error; the training configuration uses a maximum sequence length of 4,096 tokens. This provides a practical trade-off between document coverage and computational efficiency, particularly in terms of GPU memory consumption and training costs. Koh et al. (2023, Table 1) report an average source-document length of 3,573 tokens for BIGPATENT, supporting the reasonableness of this budget for typical documents.

Specifically, our implementation concatenates the summarization instruction, patent Description, and reference Abstract, and applies tokenizer-based truncation when the combined sequence exceeds the effective length limit. Consequently, sufficiently long descriptions may be truncated, and the entire Description is not guaranteed to be retained. Our short/long categorization is based on the original Description length, rather than the number of tokens retained after truncation. Thus, our experiments evaluate worst-group robustness across original-document-length categories under a common finite-context setting. The same training-length constraint is applied across methods, ensuring a consistent computational setting for comparison.

We appreciate the reviewer's careful attention to these distinctions. We believe that the clarified interpretation of the surrogate and the finite-context evaluation protocol provides a more precise account of our method and its empirical contributions. We thank the reviewer again for recognizing the substantial improvements made through the rebuttal and revised manuscript.


(Koh et al., 2023) Koh, H. Y., Ju, J., Liu, M., & Pan, S. (2023). An Empirical Survey on Long Document Summarization: Datasets, Models, and Metrics. ACM Computing Surveys, 55(8), Article 154.

Reviewer_j5HfOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to improve the worst-group performance of large language models through fine-tuning on imbalanced training data. The authors are motivated by the observation that optimizing average reward tends to favor frequent or easier tasks. They therefore introduce an adversarial weighting network that dynamically adjusts sample weights based on a proxy for the model’s current performance. The weighting network and the LLM are updated alternately, directing training toward poorly performing groups. The proposed framework supports three objectives: protecting predefined groups, improving robustness without group labels, and maximizing the minimum group-level improvement over a reference model. Experiments show that the method outperforms naive fine-tuning and inverse-frequency weighting in terms of worst-group performance.

Critical strengths
  • The paper addresses a practically relevant problem: average metrics can mask performance regressions in specific domains. Evaluating worst-group performance provides a more comprehensive assessment of model performances.

  • The proposed method adjusts the training objective through sample reweighting, making it straightforward to incorporate into existing training pipelines.

Critical weaknesses
  • Limited novelty. The formulation in Equation (1) closely resembles that of [1], which uses $ \lambda_i=1+\frac{n f_\phi(x_i,y_i)}{\sum_j f_\phi(x_j,y_j)}. $ The authors should carefully explain the connections and substantive differences between the two methods. The proposed approach also shares its central idea with Group DRO [2]. Although the paper briefly discusses this connection, it lacks direct experimental comparisons with these closely related methods.

  • Limited experiments. The experiments evaluate the method using only one backbone model and two tasks. Evaluation across more models and tasks, and imbalance settings would strengthen the evidence for its generality.

[1] Fairness without Demographics through Adversarially Reweighted Learning. 2020.

[2] Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. 2020.

Questions for authors
  • The main concern is the limited technical novelty, given the closely related ideas in prior work [1, 2]. What specific technical challenges does the proposed method address beyond adapting existing reweighting approaches to LLM fine-tuning? Could the authors provide direct comparisons under matched experimental settings?

  • Could the sample or group weights be optimized adaptively without introducing an additional neural network, for example, through exponentiated-gradient updates based on group losses? How does the proposed weighting network compare with such alternatives in terms of performance and computational overhead?

[1] Fairness without Demographics through Adversarially Reweighted Learning. 2020.

[2] Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. 2020.

4 comments

We thank the reviewer for recognizing the practical relevance of worst-group performance in LLM fine-tuning. We would like to clarify several points regarding the novelty assessment and the scope of the requested comparisons. We have revised the manuscript to clarify our contributions and include additional baseline results, which we believe address the major concerns raised in the review.

1. On Novelty and Comparisons with Group DRO and ARL

We acknowledge the close relationship between Eq. (1) and the weighting mechanism of Lahoti et al. (2020). However, we respectfully disagree that similarity in the weighting parameterization alone implies a lack of substantive novelty. Our contribution is a unified framework for autoregressive LLM fine-tuning that addresses three complementary fairness objectives: group-aware worst-group performance, group-agnostic robustness, and reference-relative worst-group improvement.

Group DRO and ARL were originally developed for different problem settings. Group DRO optimizes worst-group supervised loss using predefined groups, whereas ARL learns adversarial example weights without predefined group labels in supervised learning. The following table summarizes the scope of their original formulations relative to ours.

FeatureGroup DROARLOurs
(Traditional) Supervised learning✓✓–
LLM fine-tuning––✓
Group-aware worst-group objective✓–✓
Group-agnostic, input-dependent weighting–✓✓
Reference-relative worst-group objective––✓
Unified treatment of the three objectives––✓

Here, “–” indicates a feature not explicitly addressed by the original formulation, rather than an inability to extend the method.

Group DRO is the closest baseline for our group-aware setting, whereas ARL is most closely related to our group-agnostic setting. Neither original formulation directly addresses reference-relative fairness under logged scalar-reward feedback. In particular, our relative objective maximizes the minimum group-wise improvement over a fixed reference policy, rather than solely optimizing absolute worst-group performance. A meaningful comparison therefore requires adapting these approaches to the relevant training signals and fairness objectives.

To provide such a comparison, we additionally evaluated Group DRO adapted to our group-aware fine-tuning setting. The tables below compare it with the corresponding group-aware variant of our method (mean ± standard deviation). Our method achieves higher worst-group scores across both tasks and all evaluated groupings, as well as larger relative worst-group improvements in recipe generation.

Recipe generation

Evaluation groupingMetricGroup DRO (Known Groups)Adversarial (Ours)
CuisineVmin⁡V_{\min}12.17 ± 0.1212.25 ± 0.07
CuisineΔVmin⁡\Delta V_{\min}+0.005 ± 0.018+0.013 ± 0.007
Ingredient rarityVmin⁡V_{\min}12.03 ± 0.1312.28 ± 0.07
Ingredient rarityΔVmin⁡\Delta V_{\min}−0.008 ± 0.023+0.012 ± 0.008

Patent summarization

Evaluation groupingMetricGroup DRO (Known Groups)Adversarial (Ours)
CPC sectionsLLM-based Vmin⁡V_{\min}12.63 ± 0.9112.91 ± 0.72
CPC sectionsSummaC Vmin⁡V_{\min}0.606 ± 0.0320.619 ± 0.024
Input lengthLLM-based Vmin⁡V_{\min}12.91 ± 0.6113.19 ± 0.42
Input lengthSummaC Vmin⁡V_{\min}0.694 ± 0.0220.698 ± 0.015

Regarding ARL, when its learned reweighting mechanism is adapted to our training surrogate, it can be viewed as a special case of our group-agnostic, absolute-fairness formulation. Our framework additionally accommodates predefined groups, reference-relative improvement, and different types of training signals, including logged scalar rewards and supervised demonstrations. This is a statement about the scope of the reweighting formulation, not a claim that the original ARL algorithm is identical to our implementation.

Thus, our novelty lies not in inventing adversarial reweighting itself, but in its unified formulation and empirical investigation across fairness objectives and autoregressive LLM fine-tuning settings. The additional Group DRO results provide direct empirical support for the effectiveness of our approach relative to an adapted adaptive-reweighting baseline.

3. Scope and strength of the empirical evidence

We also believe the characterization of the experiments as limited deserves some qualification. Although the experiments use one backbone, they cover two distinct fine-tuning regimes, multiple group definitions, and complementary evaluation measures. Recipe-generation experiments use 10 independently regenerated training, validation, and test logs. Patent summarization provides an additional task with both LLM-based evaluation and SummaC factual consistency.

For instance, the group-agnostic variant consistently improves worst-group performance across both cuisine and ingredient-rarity evaluations, despite using neither grouping during training. Furthermore, the patent summarization experiments demonstrate improvements in worst-group performance under an independent factual-consistency metric, providing additional evidence of the effectiveness of our approach across different tasks and evaluation criteria.

These results do not establish universal generalization, but they provide more than a single-task demonstration of the proposed approach.

We appreciate the reviewer's suggestions for further evaluation. However, we believe that our systematic evaluation across diverse fairness objectives, fine-tuning regimes, and evaluation criteria provides more meaningful insights into the effectiveness and behavior of our method than simply increasing the number of backbone models or datasets. Our experiments are deliberately designed to examine different aspects of worst-group robustness, rather than merely demonstrate performance across additional benchmarks. We therefore believe that our current evaluation provides sufficient evidence to support the paper's empirical contributions.

4. On the Necessity of a Neural Adversary

We thank the reviewer for this question. We agree that exponentiated-gradient (EG) updates may offer a simpler and computationally efficient alternative when predefined group labels are available. However, our neural adversary is particularly motivated by the group-agnostic setting, where such labels are unavailable. By learning an input-dependent weighting function wϕ(x)w_\phi(x), it can exploit shared instruction representations and generalize to unseen prompts. In contrast, group-level EG requires predefined groups, while instance-level EG does not naturally generalize across examples.

We do not claim that neural adversaries are inherently superior to EG in the known-group setting, nor do we claim a measured computational advantage. We would appreciate clarification on whether the reviewer suggests group-level or instance-level EG, as these represent fundamentally different comparisons. Our primary motivation is to provide a unified framework that accommodates both group-aware and group-agnostic fairness objectives.

Thank you for the detailed responses. I believe that evaluating the method across multiple models and datasets is crucial for demonstrating its effectiveness, and I recommend that the authors expand their experiments accordingly. I also encourage the authors to explore alternatives such as exponentiated-gradient (EG) updates.

Thank you again for the rebuttal. I have raised my score.

We sincerely thank the reviewer for the constructive follow-up and for raising the score. We greatly appreciate the recognition of our responses and revisions.

Regarding experimental breadth, we agree that evaluation across additional backbone models and datasets would provide valuable further evidence for the generalizability of our approach. At the same time, we would like to clarify that the current experiments were deliberately designed to test the methodological contribution across substantially different fairness formulations and evaluation conditions, rather than a single narrow setting.

Specifically, we evaluate group-aware, group-agnostic, and reference-relative fairness objectives across two substantially different tasks, together with multiple evaluation criteria. These settings probe different aspects of the proposed framework and provide evidence that the observed improvements are not specific to a single fairness definition or evaluation metric. We therefore believe that the current experiments already provide substantial evidence for the central methodological claims, while evaluation on additional backbone models and datasets would be valuable for further establishing broader generalizability.

Regarding exponentiated-gradient (EG) updates, we agree that EG is a valuable alternative to investigate. Standard EG updates naturally apply when optimizing weights over predefined groups, whereas our adversary learns input-dependent weights and also covers the group-agnostic setting where predefined group membership is unavailable. Thus, while an EG comparison is possible in some settings, a comparison covering the full scope of our framework would require adapting EG to the same input-dependent and group-agnostic objectives, rather than simply replacing the neural adversary. Importantly, our method has already demonstrated superior performance over the evaluated baselines without EG-based optimization. We appreciate this suggestion and view such an extension as a valuable direction for further investigation.

Given the reviewer's positive reassessment of our rebuttal, we would be grateful if the reviewer could further consider whether the remaining concerns primarily concern the breadth of future validation rather than the validity of the current methodological contribution.

We believe that the current experiments already provide substantial evidence for the central claims, while additional models, datasets, and EG-based alternatives would further strengthen the generalizability and scope of the work. If the central contribution is now sufficiently supported, we would sincerely appreciate further reconsideration of the overall score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary Response and Paper Updates

We sincerely thank all reviewers for their constructive feedback. We believe that we have substantively addressed the major concerns raised across the three reviews, particularly regarding novelty, baseline comparisons, theoretical soundness, and empirical validation.

First, regarding novelty and baseline comparisons, we clarified how our framework differs from existing adversarial reweighting methods, particularly Group DRO and ARL, in terms of learning settings, training signals, and supported fairness objectives. Importantly, we conducted additional experiments with an adapted Group DRO baseline, demonstrating consistently higher worst-group performance across both tasks and all evaluated groupings. These results provide direct empirical evidence beyond the original baseline comparisons.

Second, regarding theoretical soundness, we clarified the distinction between the idealized population fairness objectives and the practical logged-data training surrogate, including the scope of our theoretical analysis. We also explained that our flexible adversarial weights can implicitly incorporate the effects of importance weighting while enabling stable optimization toward our max–min fairness objective, without relying on potentially high-variance explicit importance ratios.

Third, regarding empirical validation, we highlighted the breadth of our existing experiments, covering distinct fine-tuning regimes, multiple fairness objectives and group definitions, and complementary evaluation metrics. We further clarified the motivation for the neural adversary and implementation details regarding sequence-level probabilities.

Overall, we believe our responses, particularly the new Group DRO comparisons and the clarification of our theoretical and methodological contributions, directly address the reviewers' central concerns. We have updated the manuscript to incorporate these clarifications and additional results, and believe that the major concerns raised in the reviews have now been addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.