- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies why parameter-efficient fine-tuning (PEFT) can affect different subgroups (e.g., skin tone, age, race) differently. The authors argue that an adapter decides which directions the model can move in. They therefore look at each subgroup's Fisher information restricted to these directions, called the "reachable Fisher" , where is the adapter Jacobian. The theory shows when every small update must widen the loss gap between two subgroups. It also shows that the commonly used Fisher trace cannot guarantee this, and that equal traces can still hide a large mismatch. For LoRA, many parameter directions do not change the model at all, so such statements only make sense on the directions that do.
Critical strengths
- It correctly shows that LoRA's redundant directions make raw-coordinate curvature statements meaningless, and restricts them properly.
- It clearly shows what a Fisher trace can and cannot tell, and reports unfavorable audit results honestly.
Critical weaknesses
- Limited novelty. Matching per-group gradient statistics or Fisher curvature has been done before. Fishr [1] matches per-domain gradient variances during training, explicitly motivated by their relation to the Fisher and the Hessian. [2] uses per-group gradient norms and Hessian traces to explain and reduce disparate impact. FairPrune [3] uses per-group second-order information for fair pruning on Fitzpatrick-17k. None of these is cited, and the theory mainly relies on standard tools (Taylor expansion, Weyl's inequality).
- Missing baselines. The trace-matching penalty is compared only with group-balanced training. There is no comparison with GroupDRO [4] or fairness-aware PEFT methods such as FairTune [5], FairLoRA [6] and FairSAM [7]. By Lemma 2, the penalty is essentially equalizing per-group gradient norms, so a simple gradient-norm penalty might work just as well.
- The theorems' conditions are never checked, and the trace gap falls in only 20/30 settings, yet performance improves in all.
[1] A. Rame, C. Dancette, and M. Cord. Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization. In ICML, PMLR 162:18347–18377, 2022.
[2] C. Tran, M. Dinh, and F. Fioretto. Differentially Private Empirical Risk Minimization under the Fairness Lens. In NeurIPS, 2021.
[3] Y. Wu, D. Zeng, X. Xu, Y. Shi, and J. Hu. FairPrune: Achieving Fairness Through Pruning for Dermatological Disease Diagnosis. In MICCAI, 2022.
[4] S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. In ICLR, 2020.
[5] R. Dutt, O. Bohdal, S. A. Tsaftaris, and T. Hospedales. FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness in Medical Image Analysis. In ICLR, 2024.
[6] R. Sukumaran, A. Feizi, A. Romero-Sorian, and G. Farnadi. FairLoRA: Unpacking Bias Mitigation in Vision Models with Fairness-Driven Low-Rank Adaptation. arXiv:2410.17358, 2024.
[7] Y. Dai, J. Ji, X. Ma, and Y. Wu. FairSAM: Fair Classification on Corrupted Image Data Through Sharpness-Aware Minimization. arXiv:2503.22934, 2025.
Questions for authors
- How does trace matching differ from Fishr [1], which also matches per-group gradient variances linked to the Fisher?
- How does trace matching compare with [2]–[5], and with a simple per-group gradient-norm penalty, at the same compute?
[1] A. Rame, C. Dancette, and M. Cord. Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization. In ICML, PMLR 162:18347–18377, 2022.
[2] S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. In ICLR, 2020.
[3] R. Dutt, O. Bohdal, S. A. Tsaftaris, and T. Hospedales. FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness in Medical Image Analysis. In ICLR, 2024.
[4] R. Sukumaran, A. Feizi, A. Romero-Sorian, and G. Farnadi. FairLoRA: Unpacking Bias Mitigation in Vision Models with Fairness-Driven Low-Rank Adaptation. arXiv:2410.17358, 2024.
[5] Y. Dai, J. Ji, X. Ma, and Y. Wu. FairSAM: Fair Classification on Corrupted Image Data Through Sharpness-Aware Minimization. arXiv:2503.22934, 2025.
2 comments
Response to Reviewer TL7H (2/2): baselines
Question 2 / Weakness 2: Missing baselines and a gradient-norm penalty
We agree that these baselines were missing. We are running GroupDRO, FairTune, a FairLoRA-inspired subgroup loss-variance penalty, FairSAM and mean-gradient-norm matching.
Protocol. Every method uses the same 30 cells (three image tasks crossed with three encoders and three adapters, plus one tabular task crossed with the three adapters), data splits, group-balanced sampler, seed, schedule and early stopping. Checkpoints are selected on the pooled validation metric.
Batch-size selection. Within each dataset-encoder-adapter cell, we selected batch size from {64,128,256,512,1024} using only the plain group-balanced baseline's pooled validation utility: balanced accuracy for classification and MAE for regression. Subgroup gaps and test-set outcomes were not used. The rule was fixed before test evaluation, and the selected batch size was then frozen for the trace arm and all comparison methods.
Tuning budgets. Each baseline uses the same tuning budget as ours.
Final results (R = % reduction of the test best–worst gap against the group-balanced baseline of the same cell):
| method | narrows gap | narrows gap, worst group improves | narrows gap, pooled not worse | median R | trace has smaller gap | median | Wilcoxon |
|---|---|---|---|---|---|---|---|
| GroupDRO | 22 | 20 | 16 | 14.0% | 17/30 | +7.5 pts | 0.055 |
| FairLoRA-inspired | 21 | 19 | 14 | 10.7% | 22/30 | +8.5 pts | 0.006 |
| FairTune (8 trials) | 18 | 17 | 14 | 8.5% | 17/30 | +7.7 pts | 0.040 |
| FairSAM | 23 | 19 | 10 | 6.7% | 21/30 | +6.4 pts | 0.021 |
| mean-gradient-norm | 19 | 18 | 15 | 10.6% | 21/30 | +9.4 pts | 0.002 |
| trace matching | 30 | 27 | 23 | 17.2% | – | – | – |
The last two columns are the median of the paired per-cell differences and a Wilcoxon signed-rank test on them (alternative: trace matching narrows the gap more). The cells share tasks, encoders and adapters and use one seed, so the -values are indicative.
Reading. We show, trace matching has the smaller gap in 22/30 against the FairLoRA-inspired penalty, 21/30 against FairSAM, 21/30 against mean-gradient-norm matching, 17/30 against GroupDRO and 17/30 against FairTune. The gap narrows with an improved worst group in 17–20 cells for the baselines and 27 for trace matching, and with pooled performance not worse in 10–16 cells for the baselines and 23 for trace matching.
Is the penalty just a gradient-norm penalty? Our statistic is the mean squared per-example gradient norm, which by Lemma 2 decomposes as
The control matches only the first term, the squared norm of the subgroup mean gradient, with the same multipliers and cap. Its median R is 10.6% against 17.2% for trace matching. Thus, including the within-group dispersion term changes the intervention and is associated with smaller gaps in 21 of 30 cells. Neither objective is guaranteed to narrow the gap: equal mean-gradient norms do not give , and by Theorem 4 equal traces do not give equal matrices.
Response to Reviewer TL7H (1/2): novelty and relation to Fishr
Weakness 1: Limited novelty
Thank you for these references. We will cite all three; we currently cite only the same group's later pruning analysis (Tran et al., 2022). We agree that per-group gradient and curvature statistics have been used before. What differs is the object, the goal and the claim.
Question 1: How does trace matching differ from Fishr?
Two objects in our paper. The theory concerns the model-expected reachable Fisher. The training penalty uses a different statistic: the observed-label, uncentered second moment of per-example gradients in adapter coordinates. We keep them separate in the paper, and the comparison with Fishr below concerns the training statistic.
What is shared. Both methods use second-moment statistics of per-example gradients. Our subgroup statistic is
na1∑i∈a∥gai∥2=∥gˉa∥2+trΣ^a,
and trΣ^a is the sum of the per-coordinate centered variances that Fishr matches element-wise. As a training statistic, ours is therefore coarser than Fishr's (one scalar per subgroup) and adds the mean-gradient term. We do not claim the statistic itself as the contribution.
What differs.
To our knowledge, no prior work characterizes the guarantees and failures of subgroup-curvature summaries specifically in adapter-reachable directions. We will add this comparison to the related-work section.
Weakness 3: Conditions unchecked; trace gap falls in 20/30
We agree. We do not use Theorem 1 to explain the empirical intervention, and we will remove language connecting the observed improvements to that certificate. The 30/30 result shows an empirical effect of the penalty on this grid; it does not show that curvature matrices became aligned, and our audit (trace gap 20/30, operator 23/30, Frobenius 16/30) is reported for that reason.