acceptodds
Under review as a conference paper at ICLR 2027

Reducing The Adaptation Gap Through Reachable Fisher Geometry

Abstract

Parameter-efficient fine-tuning (PEFT) determines not only how many parameters are trained but which local directions a model can move, so similar adapters can affect subgroup losses differently. The relevant curvature matrices are infeasible to form at adapter scale, so practice uses scalar summaries, such as the Fisher trace. What can it reveal, and what does it discard? We answer through the *reachable Fisher*: each subgroup's full-model Fisher pulled back through the adapter Jacobian, under likelihood losses the Gauss–Newton curvature the adapter can reach, whose trace is computable from score-gradient norms without forming the matrix. Under matched subgroup gradients, if the reachable-Fisher difference is positive definite on model-changing directions by a margin exceeding the Hessian–Fisher defect, every sufficiently small nonzero update along them increases the signed gap. The restricted operator norm of the Hessian difference instead determines worst-case quadratic change; a matrix-free Frobenius discrepancy upper-bounds its reachable-Fisher component, with defect and estimation errors kept separate. A trace cannot certify definiteness, nor do matched traces control magnitude: equality rules out a positive-definite difference yet permits the largest operator mismatch that trace allows. LoRA's factorization also precludes unrestricted definiteness in raw coordinates. Across 306 single-seed models, higher trace accompanies greater subgroup difficulty in 75.7% of 1,218 eligible evaluations, and trace matching during training narrows the best–worst subgroup gap in all 30 dataset–encoder–adapter combinations. Yet a held-out audit exposes what it leaves uncontrolled: the observed-label operator discrepancy falls in 23 of these 30, the primary unbiased squared-Frobenius statistic in only 16. The trace is therefore a scalable diagnostic and training heuristic, not a certificate of local gap behavior or matrix alignment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Fisher, Subgroup, Loss Landscape Curvature

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TL7HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why parameter-efficient fine-tuning (PEFT) can affect different subgroups (e.g., skin tone, age, race) differently. The authors argue that an adapter decides which directions the model can move in. They therefore look at each subgroup's Fisher information restricted to these directions, called the "reachable Fisher" FaR=J⊤FaJF_a^R = J^\top F_a J, where JJ is the adapter Jacobian. The theory shows when every small update must widen the loss gap between two subgroups. It also shows that the commonly used Fisher trace cannot guarantee this, and that equal traces can still hide a large mismatch. For LoRA, many parameter directions do not change the model at all, so such statements only make sense on the directions that do.

Critical strengths
  1. It correctly shows that LoRA's redundant directions make raw-coordinate curvature statements meaningless, and restricts them properly.
  2. It clearly shows what a Fisher trace can and cannot tell, and reports unfavorable audit results honestly.
Critical weaknesses
  1. Limited novelty. Matching per-group gradient statistics or Fisher curvature has been done before. Fishr [1] matches per-domain gradient variances during training, explicitly motivated by their relation to the Fisher and the Hessian. [2] uses per-group gradient norms and Hessian traces to explain and reduce disparate impact. FairPrune [3] uses per-group second-order information for fair pruning on Fitzpatrick-17k. None of these is cited, and the theory mainly relies on standard tools (Taylor expansion, Weyl's inequality).
  2. Missing baselines. The trace-matching penalty is compared only with group-balanced training. There is no comparison with GroupDRO [4] or fairness-aware PEFT methods such as FairTune [5], FairLoRA [6] and FairSAM [7]. By Lemma 2, the penalty is essentially equalizing per-group gradient norms, so a simple gradient-norm penalty might work just as well.
  3. The theorems' conditions are never checked, and the trace gap falls in only 20/30 settings, yet performance improves in all.

[1] A. Rame, C. Dancette, and M. Cord. Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization. In ICML, PMLR 162:18347–18377, 2022.

[2] C. Tran, M. Dinh, and F. Fioretto. Differentially Private Empirical Risk Minimization under the Fairness Lens. In NeurIPS, 2021.

[3] Y. Wu, D. Zeng, X. Xu, Y. Shi, and J. Hu. FairPrune: Achieving Fairness Through Pruning for Dermatological Disease Diagnosis. In MICCAI, 2022.

[4] S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. In ICLR, 2020.

[5] R. Dutt, O. Bohdal, S. A. Tsaftaris, and T. Hospedales. FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness in Medical Image Analysis. In ICLR, 2024.

[6] R. Sukumaran, A. Feizi, A. Romero-Sorian, and G. Farnadi. FairLoRA: Unpacking Bias Mitigation in Vision Models with Fairness-Driven Low-Rank Adaptation. arXiv:2410.17358, 2024.

[7] Y. Dai, J. Ji, X. Ma, and Y. Wu. FairSAM: Fair Classification on Corrupted Image Data Through Sharpness-Aware Minimization. arXiv:2503.22934, 2025.

Questions for authors
  1. How does trace matching differ from Fishr [1], which also matches per-group gradient variances linked to the Fisher?
  2. How does trace matching compare with [2]–[5], and with a simple per-group gradient-norm penalty, at the same compute?

[1] A. Rame, C. Dancette, and M. Cord. Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization. In ICML, PMLR 162:18347–18377, 2022.

[2] S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. In ICLR, 2020.

[3] R. Dutt, O. Bohdal, S. A. Tsaftaris, and T. Hospedales. FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness in Medical Image Analysis. In ICLR, 2024.

[4] R. Sukumaran, A. Feizi, A. Romero-Sorian, and G. Farnadi. FairLoRA: Unpacking Bias Mitigation in Vision Models with Fairness-Driven Low-Rank Adaptation. arXiv:2410.17358, 2024.

[5] Y. Dai, J. Ji, X. Ma, and Y. Wu. FairSAM: Fair Classification on Corrupted Image Data Through Sharpness-Aware Minimization. arXiv:2503.22934, 2025.

2 comments
AuthorsOpenReview ↗

Response to Reviewer TL7H (1/2): novelty and relation to Fishr

Weakness 1: Limited novelty

Thank you for these references. We will cite all three; we currently cite only the same group's later pruning analysis (Tran et al., 2022). We agree that per-group gradient and curvature statistics have been used before. What differs is the object, the goal and the claim.

FishrTran et al. 2021FairPruneOurs
goalinvariance across training domainsdisparate impact of DP-SGDfairness of pruningsubgroup gaps under PEFT
statisticper-coordinate centered gradient varianceper-group gradient norm, Hessian traceper-group diagonal-Hessian saliencytheory: reachable Fisher J⊤FaJJ^\top F_a J; training: scalar trace of the observed-label adapter-gradient second moment
parameter spacefull-network formulation; classifier-gradient approximation used for efficiencyfull modelfull modeladapter-reachable directions
role of curvaturematched as a training regularizerexplains the disparityranks weights to pruneanalysed: what it certifies and what it cannot

Question 1: How does trace matching differ from Fishr?

Two objects in our paper. The theory concerns the model-expected reachable Fisher. The training penalty uses a different statistic: the observed-label, uncentered second moment of per-example gradients in adapter coordinates. We keep them separate in the paper, and the comparison with Fishr below concerns the training statistic.

What is shared. Both methods use second-moment statistics of per-example gradients. Our subgroup statistic is

1na∑i∈a∥gai∥2=∥gˉa∥2+tr⁡Σ^a,\frac{1}{n_a}\sum_{i\in a}\Vert g_{ai}\Vert ^2 = \Vert \bar g_a\Vert ^2 + \operatorname{tr}\hat\Sigma_a ,

and tr⁡Σ^a\operatorname{tr}\hat\Sigma_a is the sum of the per-coordinate centered variances that Fishr matches element-wise. As a training statistic, ours is therefore coarser than Fishr's (one scalar per subgroup) and adds the mean-gradient term. We do not claim the statistic itself as the contribution.

What differs.

  1. Object and space. We pull the Fisher back through the adapter Jacobian. For factorized adapters this matters: LoRA's symmetry (B,A)↦(BG,G−1A)(B,A)\mapsto(BG,G^{-1}A) creates at least r2r^2 parameter directions that leave the model unchanged, so curvature statements in raw coordinates are not well defined. Our directional results are stated on Hϕ=(ker⁡J)⊥\mathcal H_\phi = (\ker J)^\perp, and Theorem 3 shows that summaries computed without JJ can create or hide a reachable difference.
  2. Goal. Fishr targets invariance across training domains for out-of-distribution generalization. We study subgroup disparity within one deployment distribution under PEFT.
  3. Claim. Fishr motivates matching through the closeness of gradient covariance, Fisher and Hessian. We ask what such matching controls. The Hessian difference decomposes into the reachable-Fisher difference and an explicitly retained Hessian–Fisher defect. Each subgroup Fisher is positive semi-definite and computable from first-order gradients, but their difference need not be positive semi-definite. Theorem 4 shows that equal traces allow an operator mismatch as large as the trace. Matching diagonals likewise does not control off-diagonal structure or the operator discrepancy. Proposition 2 makes this checkable on trained models.

To our knowledge, no prior work characterizes the guarantees and failures of subgroup-curvature summaries specifically in adapter-reachable directions. We will add this comparison to the related-work section.

Weakness 3: Conditions unchecked; trace gap falls in 20/30

We agree. We do not use Theorem 1 to explain the empirical intervention, and we will remove language connecting the observed improvements to that certificate. The 30/30 result shows an empirical effect of the penalty on this grid; it does not show that curvature matrices became aligned, and our audit (trace gap 20/30, operator 23/30, Frobenius 16/30) is reported for that reason.

AuthorsOpenReview ↗

Response to Reviewer TL7H (2/2): baselines

Question 2 / Weakness 2: Missing baselines and a gradient-norm penalty

We agree that these baselines were missing. We are running GroupDRO, FairTune, a FairLoRA-inspired subgroup loss-variance penalty, FairSAM and mean-gradient-norm matching.

Protocol. Every method uses the same 30 cells (three image tasks crossed with three encoders and three adapters, plus one tabular task crossed with the three adapters), data splits, group-balanced sampler, seed, schedule and early stopping. Checkpoints are selected on the pooled validation metric.

Batch-size selection. Within each dataset-encoder-adapter cell, we selected batch size from {64,128,256,512,1024} using only the plain group-balanced baseline's pooled validation utility: balanced accuracy for classification and MAE for regression. Subgroup gaps and test-set outcomes were not used. The rule was fixed before test evaluation, and the selected batch size was then frozen for the trace arm and all comparison methods.

Tuning budgets. Each baseline uses the same tuning budget as ours.

Final results (R = % reduction of the test best–worst gap against the group-balanced baseline of the same cell):

methodnarrows gapnarrows gap, worst group improvesnarrows gap, pooled not worsemedian Rtrace has smaller gapmedian Rtrace−RmethodR_{\text{trace}}-R_{\text{method}}Wilcoxon pp
GroupDRO22201614.0%17/30+7.5 pts0.055
FairLoRA-inspired21191410.7%22/30+8.5 pts0.006
FairTune (8 trials)1817148.5%17/30+7.7 pts0.040
FairSAM2319106.7%21/30+6.4 pts0.021
mean-gradient-norm19181510.6%21/30+9.4 pts0.002
trace matching30272317.2%–––

The last two columns are the median of the paired per-cell differences and a Wilcoxon signed-rank test on them (alternative: trace matching narrows the gap more). The cells share tasks, encoders and adapters and use one seed, so the pp-values are indicative.

Reading. We show, trace matching has the smaller gap in 22/30 against the FairLoRA-inspired penalty, 21/30 against FairSAM, 21/30 against mean-gradient-norm matching, 17/30 against GroupDRO and 17/30 against FairTune. The gap narrows with an improved worst group in 17–20 cells for the baselines and 27 for trace matching, and with pooled performance not worse in 10–16 cells for the baselines and 23 for trace matching.

Is the penalty just a gradient-norm penalty? Our statistic is the mean squared per-example gradient norm, which by Lemma 2 decomposes as

1na∑i∈a∥gai∥2=∥gˉa∥2+tr⁡Σ^a.\frac{1}{n_a}\sum_{i\in a}\Vert g_{ai}\Vert ^2 = \Vert \bar g_a\Vert ^2 + \operatorname{tr}\hat\Sigma_a .

The control matches only the first term, the squared norm of the subgroup mean gradient, with the same multipliers and cap. Its median R is 10.6% against 17.2% for trace matching. Thus, including the within-group dispersion term changes the intervention and is associated with smaller gaps in 21 of 30 cells. Neither objective is guaranteed to narrow the gap: equal mean-gradient norms do not give gˉa=gˉb\bar g_a=\bar g_b, and by Theorem 4 equal traces do not give equal matrices.

Reviewer_FDgtOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates subgroup performance gaps in parameter-efficient fine-tuning from the perspective of the reachable Fisher geometry and proposes a penalty term to reduce performance gaps across subgroups. Experiments across multiple datasets, model architectures, and PEFT methods demonstrate that the proposed method consistently reduces subgroup performance gaps.

Critical strengths
  1. Comprehensive experimental evaluation. The experiments cover diverse model architectures, datasets, and PEFT methods.
  2. Extensive theoretical analysis. The paper provides a detailed theoretical analysis of subgroup performance gaps from a Fisher geometric perspective, offering insights into the relationship between reachable Fisher information and subgroup performance gaps.
Critical weaknesses
  1. Substantial computational overhead. According to Table 4, the proposed penalty introduces significant computational overhead, substantially limiting the practicality of the proposed method.
  2. Unclear motivation. Minimizing the subgroup performance gap alone may not be an ideal optimization objective, as the gap can decrease even when overall predictive performance deteriorates. For instance, some configurations on Fitzpatrick17k achieve substantial gap reductions while exhibiting decreases in pooled accuracy. Directly improving worst-subgroup performance while preserving overall performance may be a more meaningful and practical objective.
  3. Insufficient comparison with existing optimization methods. The experiments primarily compare the proposed method against standard PEFT without the additional penalty. Comparisons with related optimization approaches, such as gradient balancing methods from multi-task learning and Sharpness-Aware Minimization (SAM), would help establish whether the proposed Fisher-based regularization offers advantages over existing alternatives.
  4. Limited hyperparameter sensitivity analysis. The proposed method introduces several additional hyperparameters. However, the paper does not provide sufficient sensitivity analyses to evaluate the robustness of the method to these hyperparameters.
Questions for authors
  1. Provide further justification for the motivation and discuss the practical implications of the proposed method.
  2. Include additional comparisons with existing optimization methods.
  3. Provide additional hyperparameter sensitivity analyses, if possible.
6 comments
AuthorsOpenReview ↗

Response to Reviewer FDgt (1/2): contributions, motivation, overhead and baselines

Thank you for the review.

What the paper contributes. The paper is primarily an analysis, with the penalty as one application. Under PEFT, the adapter fixes which directions the model can move. The reachable Fisher FaR=J⊤FaJF^{\mathrm R}_a = J^\top F_a J is each subgroup's curvature along those directions. It governs the second-order change of a subgroup loss gap up to an explicitly tracked defect, and full-model summaries can create or hide a reachable difference (Theorem 3). We also show what the commonly used Fisher trace can and cannot certify (Theorem 4).

Question 1 / Weakness 2: Motivation and practical implications

  • Why it matters. Foundation models are increasingly adapted with PEFT on small local datasets, where average accuracy can hide failures on underrepresented subgroups, as in dermatology, face analysis and clinical prediction.
  • Diagnosis before deployment. The subgroup trace is a mean squared per-example adapter-gradient norm. It needs no matrix and no retraining. A higher trace accompanies greater subgroup difficulty in 75.7% of 1,218 evaluations, so, within a fixed model and parameterization, it provides a diagnostic signal for identifying subgroups that may warrant closer evaluation.
  • Is the gap the right objective? We agree that a gap can shrink by levelling down, so we report utility alongside it. Worst-group performance improves in 27 of 30 cells, and pooled utility is not worse in 23. In 6 of the 7 cells with a pooled cost, the worst group still improves. The largest classification cost is 2.4 points (Fitzpatrick17k/ViT/PiSSA), where the gap falls from 0.260 to 0.139. The task loss is always retained, and checkpoints are selected on pooled validation performance, never on the gap.

Weakness 1: Computational overhead

We agree. Table 4 shows that a penalized training step costs significantly more than an unpenalized one, because differentiating per-example gradient norms needs second-order terms. We therefore present trace matching as a proof-of-concept intervention arising from the analysis, not as a definitive optimization method, and we will revise the abstract and Section 6 to say so. We do not claim it is training-time competitive with cheaper methods. Two parts of the paper do not pay this cost:

  • The post-training diagnostic and audit require only first-order per-example gradients; computing the trace statistic alone takes 1.6–3.0 s per batch.
  • Restricting to the adapter is the cheaper scope: full-model scope is 41–62% slower with 1.6–2.1 times the peak memory (Table 11).

Question 2 / Weakness 3: Comparisons with existing methods

We ran GroupDRO, a FairLoRA-inspired subgroup loss-variance penalty, FairTune, FairSAM and mean-gradient-norm matching, with the same cells, splits, sampler, seed, schedule and batch size. Final results (R = % reduction of the test best–worst gap against the group-balanced baseline):

methodnarrows gapnarrows gap, worst group improvesnarrows gap, pooled not worsemedian Rtrace has smaller gapWilcoxon pp
GroupDRO22201614.0%17/300.055
FairLoRA-inspired21191410.7%22/300.006
FairTune1817148.5%17/300.040
FairSAM2319106.7%21/300.021
mean-gradient-norm19181510.6%21/300.002
trace matching30272317.2%––

The new comparisons include both a direct mean-gradient-norm control and FairSAM as gradient- and sharpness-based alternatives; we have not added a separate multi-task method. Protocol details are in our reply to Reviewer TL7H.

We will add this framing to the introduction and release our code.

AuthorsOpenReview ↗

Response to Reviewer FDgt (2/2): hyperparameter sensitivity


Question 3 / Weakness 4: Hyperparameter sensitivity

  • The penalty's constants are fixed across all tasks (Huber threshold 1, gradient cap β=0.5\beta=0.5, multiplier settings in Appendix C). Fixing them does not show the method is insensitive to them.
  • The penalty form was compared in a fixed-batch development sweep (Huber, ℓ1\ell_1, squared ℓ2\ell_2, one-sided hinge, minimum reference, raw deviation) evaluated on the validation split. Huber was the only form that reduced the validation gap on every dataset.
  • We test sensitivity to batch size and seed on all 30 cells and three seeds (12345, 23456, 34567). Seeds 23456 and 34567 (newly added) cover all five batch sizes (64–1024) in every cell. Seed 12345 adds the runs we already had (1–4 per cell). That gives 349 matched plain/trace pairs (10–14 per cell). Each run is scored on the test split, with the checkpoint chosen on pooled validation performance. R=100,(1−gapmatched/gapplain)R=100\\,(1-\mathrm{gap}_{\text{matched}}/\mathrm{gap}_{\text{plain}}) is the % reduction of the best–worst gap.

Batch size. No batch size is systematically better or worse:

batch sizeruns where the gap narrowsmedian RR
6436/66+2.1%
12847/87+2.3%
25643/71+3.4%
51234/64+1.9%
102433/60+0.1%
all*194/349+2.5% (IQR −9% to +18%)

*includes one seed-12345 run at batch size 32.

Seed. Run-to-run noise is large. Two plain runs that differ only in seed differ in gap by a median of 15% (246 pairs), about as much as the effect varies across single runs. A single run can't establish the effect, but averaging can.

After averaging. The penalty narrows the gap in 194 of 349 runs (56%; Wilcoxon signed-rank p=0.010p=0.010). The mean reduction is 4.0% of the plain gap (SE 1.7%; Wilcoxon signed-rank p=0.007p=0.007). The effect is modest but consistent across batch sizes and seeds.

Thanks to the authors for their response, which has addressed most of my concerns. However, the following questions remain.

  1. I still believe that directly improving worst-subgroup performance while preserving overall performance may be a more meaningful and practical objective. Additional comparisons in this direction would provide valuable insights. Nevertheless, I do not consider the absence of such experiments to undermine the main contributions of this paper.
  2. The hyperparameter sensitivity of the proposed method remains insufficiently explored, as the additional experiments only examine the effects of batch size and random seed rather than the hyperparameters of the proposed method, such as κ\kappa.
  3. As the authors acknowledge the substantial additional training overhead, comparisons under comparable computational budgets would provide a fairer assessment of the proposed method against existing alternatives.
Reviewer_QnQrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper pulls each subgroup’s Fisher back through the adapter Jacobian (the “reachable Fisher”). It proves local statements about when curvature forces a subgroup loss gap up or bounds its worst-case change. It shows trace matching cannot certify definiteness or matrix alignment, and gives a matrix-free Gram identity for the Frobenius discrepancy.

Critical strengths
  1. The identification of the LoRA/DoRA gauge kernel (dim ≥ r²) and the restriction to the horizontal subspace is useful.
  2. The Gram identity (Proposition 2) is cheap, and makes the audit practical.
Critical weaknesses
  1. The mechanism is not supported by the paper’s own audit.
  • Held-out trace gap falls in only 20 of 30 cells, yet the performance gap narrows in all 30. For example, FHIBE/SigLIP 2/DoRA has trace gap +42% and gap −53.5%.
  1. The training statistic is the observed-label squared gradient norm, which tracks loss times curvature. Penalizing its disparity is close to equalizing group losses or gradient norms.
  2. Theorem 1 needs slope matching, horizontal updates, and a small defect. None is enforced or measured.
Questions for authors
  1. Trace gap falls in only 20 of 30 held-out audits, yet the performance gap narrows in all 30 (e.g. FHIBE/SigLIP 2/DoRA: trace gap +42%, gap −53.5%). What explains narrowing where the penalty’s own target did not move?
  2. Frobenius discrepancy falls in 16 of 30 and rises in 9 of the 20 cells where the trace gap fell. Why present the 30/30 as supporting a curvature-matching mechanism?
2 comments
AuthorsOpenReview ↗

Response to Reviewer QnQr (Part 1 of 2): what the results support, Questions 1 and 2

Thank you for identifying an important distinction between the effectiveness of the intervention and its mechanism. We agree that the 30/30 performance-gap result does not, by itself, establish that the gaps narrowed because curvature matrices became better aligned. Our held-out audit was meant to expose this limitation, but our wording did not make it clear enough.

What the results support

  1. Geometric object. The reachable Fisher J⊤FaJJ^\top F_a J isolates the curvature that a PEFT parameterization can act on. Full-model summaries may create or hide reachable subgroup differences (Theorem 3). For factorized adapters, definiteness is meaningful only off the gauge kernel. For factorized adapters, definiteness must be assessed on the horizontal subspace Hϕ=(ker⁡J)⊥\mathcal H_\phi=(\ker J)^\perp, because gauge directions lie in ker⁡J\ker J and are null directions of every reachable Fisher.
  2. Limits of scalar summaries. Trace matching cannot certify definiteness or matrix alignment (Theorem 4). Proposition 2 makes a matrix-level Frobenius audit feasible without constructing a dense p×pp\times p matrix.
  3. Empirical intervention. Against plain training (no intervention) sharing the group-balanced sampler, seed, initialization, schedule, and batch size, the observed-label trace penalty narrows the best–worst performance gap in 30/30 cells (median 17.2%). Worst-group utility improves in 27/30, and pooled utility is not worse in 23/30.

The third result shows an empirical effect within this single-seed grid; it does not identify the causal mediator. We will state this in Section 6 and rename "Mechanism correlation" to "Exploratory association" throughout.

Question 1: Why do gaps narrow when the held-out trace gap does not?

The trace penalty acts on training batches, and its effect on the model need not show up as more similar trace values on held-out data. What we do observe is that group errors become more balanced: the surrogate-loss gap narrows in 27 of 30 cells. The trace gap and the performance gap measure different things, so they do not have to move together. Trace matching is a heuristic, not a certificate, and we do not claim a guarantee that links the two.

Question 2: Frobenius discrepancy falls in 16 of 30 and rises in 9 of the 20 cells where the trace gap fell. Why present the 30/30 as supporting a curvature-matching mechanism?

We thank the reviewer for highlighting this important distinction. The 30/30 result shows that adding the trace penalty narrowed the performance gap in every cell of this grid; it is not evidence of matrix alignment. The mixed Frobenius result (16/30) in fact supports Theorem 4: matching a scalar trace does not give matrix-level matching. We will revise Section 6 to say that alignment is neither guaranteed nor required for the observed improvement, and to present the mechanism as an open question.

AuthorsOpenReview ↗

Response to Reviewer QnQr (Part 2 of 2): loss and gradient norms, conditions of Theorem 1

Weakness 2: Relationship to loss and gradient norms

This is a real alternative explanation. For Gaussian regression, ∥∇ϕℓi∥2=2,ℓi,ti\Vert \nabla_\phi \ell_i\Vert ^2 = 2\\,\ell_i\\,t_i, where tit_i is the per-example expected-Fisher trace contribution. The penalty therefore combines residual loss with adapter sensitivity. It reduces to a direct loss penalty only under restrictive conditions, such as tit_i being constant across examples. This does not hold in our experiments: the baseline subgroup expected-Fisher trace CV ranges from 0.08 to 0.74.

We have results from two direct controls with the same cells, seed, sampler, schedule and gradient safeguard. In all 30 cells, trace matching has a smaller test gap than a FairLoRA-inspired subgroup loss-variance penalty in 22 (median reduction 17.2% vs. 10.7%; Wilcoxon signed-rank p=0.006p=0.006), and than mean-gradient-norm matching, which penalizes ∥gˉa∥2\Vert \bar g_a\Vert ^2 only, in 21 (17.2% vs. 10.6%; p=0.002p=0.002). The gap narrows with an improved worst group in 19 and 18 cells for the two controls (27 for trace matching), and with pooled performance not worse in 14 and 15 (23 for trace matching). One mean-gradient-norm cell is scored at its current best checkpoint while training continues; we will post the complete tables in this thread when it finishes.

Weakness 3: Conditions of Theorem 1

Agreed. We do not use Theorem 1 to explain the empirical intervention, and we will remove language connecting the observed improvements to that certificate. Theorem 1 is a conditional local characterization: it requires slope matching, horizontal updates, and a Hessian–Fisher defect smaller than the curvature margin, and the penalty enforces none of these. In our small-network experiment, on the subspace S+S_+ where the Hessian difference is positive definite, the reachable-Fisher difference has λmin⁡=−0.100\lambda_{\min}=-0.100 against a defect norm of 0.171, so the reachable-Fisher certificate fails. On the full horizontal subspace the defect norm is 0.66 times the reachable-Fisher-difference norm, so the defect is not negligible.

The geometric object and the limits of scalar summaries do not depend on the intervention's mechanism; you also identify the gauge-kernel result and Gram identity as strengths. With the intervention framed as an empirical result whose mechanism remains open, we would be grateful if you would reconsider your score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Final Manuscript Update (Part 1 of 2)

We thank all three reviewers for their careful reviews and constructive follow-ups. Below we summarize the main changes in the revised PDF and clarify how they strengthen the paper's contributions and conclusions.

  1. Framing: analysis as the primary contribution, with trace matching as an analysis-driven intervention (Abstract, Section 6). We have clarified the framing of the paper so that the theoretical and diagnostic analysis is the primary contribution, while trace matching serves as a concrete proof-of-concept intervention motivated by that analysis. The central contributions are the reachable Fisher, the characterization of the LoRA gauge kernel and horizontal subspace, the limits of scalar trace information formalized in Theorem 4, and the matrix-free Gram identity.

  2. Interpretation of the 30/30 result (QnQr, TL7H; Section 6). Section 6 now gives a more precise interpretation of the 30/30 gap-reduction result. The experiment demonstrates a consistent empirical effect across the evaluated single-seed grid, while the accompanying matrix audit helps distinguish the effect of trace matching from stronger notions of matrix alignment. Specifically, the trace gap decreases in 20/30 cells, operator discrepancy in 23/30, and Frobenius discrepancy in 16/30. These observations align with Theorem 4, which shows that scalar trace agreement does not imply full matrix agreement. We therefore report both the intervention results and the matrix-level audit, and have renamed "Mechanism correlation" to the more precise "Exploratory association."

  3. Comparison with existing methods (TL7H, FDgt; Appendix G.1, Table 12). We added five comparison methods: GroupDRO, FairTune, a FairLoRA-inspired loss-variance penalty, FairSAM, and a mean-gradient-norm control. All methods are evaluated using the same cells, splits, sampler, seed, training schedule, and early-stopping protocol. Batch size is selected using only the plain baseline's validation utility. Across the 30 cells, these methods reduce the subgroup gap in 18–23 cells, with median reductions of 6.7–14.0%, compared with 30/30 cells and a median reduction of 17.2% for trace matching. Trace matching produces the smaller subgroup gap in 17–22 cells depending on the comparison method and achieves better worst-group performance in 19–23 cells. Table 12 now reports the complete 30-cell results for every method. Because cells share tasks and the main grid uses one seed, we report the associated statistical tests as complementary evidence rather than relying on them as the primary basis for comparison.

  4. Distinguishing trace matching from mean-gradient-norm matching (QnQr W2, TL7H). Lemma 2 shows that the trace statistic decomposes into the squared mean gradient and within-group gradient dispersion. To isolate the contribution of these components, we added a control that matches only the mean-gradient-norm term. The mean-gradient-norm control achieves a median gap reduction of 10.6%, compared with 17.2% for trace matching, and trace matching produces the smaller gap in 21 of 30 cells. These results show that incorporating the dispersion component leads to a meaningfully different intervention in practice.

(Continued in next comment...)

AuthorsOpenReview ↗

Final Manuscript Update (Part 2 of 2)

  1. Expanded related work and positioning (TL7H; Section 2, Appendix D.7). We expanded the related-work discussion to include Fishr, Tran et al. (2021), and FairPrune. We now position our contribution more precisely relative to these methods. Our statistic shares important ingredients with prior gradient-statistics approaches, while our focus differs in three respects: Object: curvature restricted to directions that the parameter-efficient adapter can actually move; Setting: subgroup performance gaps under parameter-efficient fine-tuning; Analysis: characterization of what these statistics can and cannot certify about the underlying reachable geometry. This revised positioning makes the relationship to prior work more explicit while highlighting the distinct geometric and PEFT-specific contribution of our analysis.

  2. Gap reduction and worst-group performance (FDgt). We added additional results showing how subgroup-gap reduction relates to utility. Trace matching improves the worst-performing group in 27 of 30 cells and maintains or improves pooled utility in 23 of 30 cells. Among the seven cells with a pooled-utility decrease, the worst group still improves in six. Importantly, checkpoint selection is always based on pooled validation performance rather than subgroup gap. We also added a short discussion in the introduction explaining why improving subgroup balance while preserving pooled performance is practically relevant.

  3. Training cost and corrected implementation-level timing (FDgt; Table 4, Appendix G.1, Table 13). We updated Table 4 so that the reported timing corresponds to the vectorized implementation used in the actual experiments. A penalized training step costs approximately 5.2× the unpenalized step for ViT–LoRA and 16.6× for ConvNeXt–DoRA. We additionally compare end-to-end cost with the newly added baselines. Among cell–method comparisons where trace matching costs no more than twice as much as the comparison method, trace matching produces the smaller gap in 27 of 40 cases: 9/11 against FairSAM, 12/16 against mean-gradient-norm matching, and 6/13 against FairTune. The post-training reachable-Fisher diagnostic and matrix audit remain comparatively lightweight: they require only first-order per-example gradients and do not require retraining.

  4. Hyperparameter sensitivity (FDgt; Appendix G.2, Table 14). We added a sensitivity study for the Huber threshold κ and gradient cap β across four representative cells. The effect is retained across κ values from 0.5 to 2 in the three cells where the default setting improves the gap by more than a few percent. The gradient cap β, which bounds the penalty-gradient norm relative to the task-gradient norm, has a stronger influence: β = 0.25 removes or reverses the effect in two of the four cells, while UTKFace is particularly sensitive to deviations from the default setting. Overall, the default configuration provides the most consistent behavior and is the only evaluated setting that narrows the subgroup gap across all four cells.

These experiments provide a broader estimate of the intervention's effect beyond the original fixed-grid evaluation and show that the average improvement persists across changes in initialization and batch size. They also help quantify the variability of the intervention and provide a more complete picture of its expected behavior.

What the revision establishes and motivates next. The revised paper separates three complementary contributions more clearly: (1) a geometric analysis of the Fisher information that is reachable under parameter-efficient adaptation; (2) practical diagnostics for measuring this geometry efficiently; and (3) trace matching as an analysis-driven intervention whose empirical behavior is evaluated against several fairness and optimization baselines.

The matrix-audit results also motivate two natural extensions: directly estimating the Hessian–Fisher discrepancy and developing interventions that optimize matrix-level reachable-geometry discrepancies rather than scalar summaries. A complementary direction, as suggested by Reviewer FDgt, is to combine these geometric objectives with methods that explicitly target worst-group performance.

We cordially than Reviewer FDgt for reconsidering the score in response to our updates thus far. We would be grateful if the reviewers could consider the revised manuscript and our responses in light of these clarifications and additional experiments, and reconsider their scores where appropriate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.