acceptodds
Under review as a conference paper at ICLR 2027

Fair Envelope Regression: Predictive-Sensitive Subspace Decomposition for Fairness-Utility Trade-offs

Abstract

Fair regression with multivariate sensitive attributes requires controlling sensitive association without unnecessarily discarding predictive signal. We study this problem in linear regression with a continuous response and vector-valued sensitive attributes, using the proportion of prediction variance attributable to the sensitive attributes as the fairness criterion. We introduce Fair Envelope Regression Models (FERM), which uses envelope methodology to organize feature variation according to its relevance to the response and its association with the sensitive attributes. Under a compatibility condition, this yields four orthogonal components: predictive-only, shared predictive-sensitive, sensitive-only, and residual variation. FERM excludes sensitive-associated variation that is immaterial for prediction and penalizes only the shared predictive-sensitive component, providing a tunable path between a sensitive-invariant predictor and an unconstrained predictive model. Under the envelope model and regularity conditions, we establish consistency, asymptotic fairness at the fully constrained endpoint, reduced asymptotic estimation variance from removing response-immaterial directions, and a closed-form characterization of the resulting fairness–utility interpolation. Simulations and real-data experiments show that FERM can improve predictive utility at comparable levels of the chosen unfairness measure relative to closely matched fair-regression baselines. Beyond prediction performance, the decomposition makes explicit which predictive directions also carry sensitive-associated information, providing subspace-level auditability of the fairness-utility trade-off.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
algorithmic fairness, subspace decomposition, interpretability, envelope regression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ewzwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes to use the concept of Envelope Regression for fair regression by (under a given condition) decomposing the space into 4 different components containing qualitatively different information about the regression task. The fairness regularization if then confined only to the subspace relevant for predictive-sensitive associated information where as the sensitive only component can be discarded.

Critical strengths
  • The paper addresses a central gap in the literature that previous fair regression models usually do not explicitly take the overlap of the sensitive and target variable(s) into account. For this they extend the idea of envelope regression to further separate the information into different subspaces: the idea is well motivated and the paper well written. In particular, this paper allows to make the fairness-utility tradeoff explicit by isolating the subspace relevant for this.

  • The theory support central claims of the paper: from variance reduction to how the penalized regressor interpolates between the extremes these statements support the derived methodology.

Critical weaknesses
  • The whole premise of the paper is built on the fact that the RS2R^2_S coefficient is a sensible proxy for an actual fairness score. While I know that this is in line with previous works, in the mean-time several extensions of fairness scores where developed that are also usable for regression and also continuous sensitive attributes, e.g. [3-4] (which arguably was not the case yet for the previous works). However, the empirical evaluation completely relies on measuring fairness with the same metric it optimizes itself (which is not great) so the results don't actually indicate an achieved level of "fairness".

  • The empirical evaluation does not (yet) support the paper claims. Next to the missing fairness scores (see point above), an indirect but central claim is that the decomposition is useful and more intelligent than other naive approaches not taking the predictive component into account; whilst true from the motivation there is only limited empirical evidence of this advantage as the FRRM baseline is also based on envelope regression - a comparison to fair kernel learning [2] with a linear kernel or any other baseline would be helpful to actually support this claim. Also one of the real world datasets is confined to the appendix which is an odd choice given only a single real-world dataset is considered in the main text. Visually, the plots are also difficult to read: the text is too small and does not match the paper font, and the labels/ticks seem not to be vector graphics at least they render blurry, for ICLR I would expect better quality figures

  • Rather minor: The extensions to the non-linear kernel or DL setting are discussed in the appendix: this is a bit odd as these are discussed as straight-forward extensions so one either expects results supporting that this is indeed works or by shortening that outlook and placing it in the main text as clear future work, currently it is not fitting to the main text.

Overall I believe this work to be very well fit for ICLR but mainly the empirical section needs significant improvements to support the paper claims, I am open to increase my score if these issues can be adequately addressed.

[2] Pérez-Suay, et al. Fair Kernel Learning (MLKDD 2017)

[3] Jiang et al., Generalized Demographic Parity for Group Fairness (ICLR 2022)

[4] Narasimhan, H.,et al, Pairwise fairness for ranking and regression. AAAI Conference (2020)

Questions for authors
  • The acronym "FERM" is already used by at least one other approach in fairness (see [1]) so I suggest to change the acronym here to reduce the possibility of confusion

  • Can you clarify how you find λ\lambda? I understand that it is computed based on rr as the smallest lambda satisfying the required level, but can you compute that analytically?

  • You assume what you call the "compatibility condition" on the projection matrices PYP_Y and PSP_S to commute: how sensible is that assumption ? Does the algorithm work even if this is violated?

[1] Donini, M., Oneto, L., Ben-David, S., Shawe-Taylor, J. S., & Pontil, M. (2018). Empirical Risk Minimization Under Fairness Constraints. In Advances in Neural Information Processing Systems

5 comments
AuthorsOpenReview ↗

Rebuttal 1

We thank the reviewer for recognizing the paper's conceptual and theoretical contributions and its relevance to ICLR. We have substantially expanded the empirical evaluation to address the concerns raised. Representative results, including larger-sample and non-healthcare experiments, additional methodological comparisons, and nonlinear dependence diagnostics, are reported in our response to Reviewer maSX. Below, we address the reviewer's specific comments and questions.

W1. Fairness criterion and independent evaluation:

We respectfully clarify that the central premise of our paper is not the choice of RS2R_S^2 as a fairness measure, but the interpretable four-way decomposition of predictive and sensitive information. This decomposition is established independently of the subsequent fairness regularization. We adopt the established multivariate RS2R_S^2 criterion in the second stage because it yields a tractable fairness-utility trade-off, characterized analytically in Corollary 5.6. Thus, the structural decomposition is not tied to this particular fairness criterion, and extending the regularization framework to nonlinear measures, including kernel-based approaches suggested by the reviewer, is a natural direction for future work.

We agree that controlling RS2R_S^2 does not guarantee nonlinear independence, and appreciate the reviewer's suggestion to evaluate fairness beyond the optimized criterion. Accordingly, we added held-out HGR and normalized HSIC fairness scores, which assess nonlinear dependence between predictions and sensitive attributes. The new results show that FERM achieves lower nonlinear dependence than the references in several settings, though not uniformly across sample sizes and fairness budgets. Detailed results are provided in our response to Reviewer maSX.

W2. Additional comparisons and empirical validation:

We respectfully clarify that FRRM is not an envelope-regression-based method. We selected FRRM because it employs the same multivariate RS2R_S^2 fairness criterion and ridge regularization, allowing us to isolate the contribution of our proposed structural decomposition. Beyond providing interpretable, subspace-level auditing of fairness regularization, our controlled simulations demonstrate statistical efficiency gains in small-to-moderate samples with relatively high-dimensional predictors.

To further strengthen this evidence, we added Komiyama et al. (2018) as an additional non-envelope reference and substantially expanded the real-data experiments. In the new demographic MEPS experiments (Section 4 of our response to Reviewer maSX), at n=1,000n=1{,}000 and r=0.20r=0.20, FERM achieves lower mean test MSE (0.905 versus 0.933 for both references) and HGR (0.341 versus 0.368). In the non-healthcare Communities and Crime experiments (Section 5 of our response to Reviewer maSX), at (n=150,p=70,r=0.10), FERM achieves lower MSE (1.344 versus 1.542 and 1.977), held-out RS2R_S^2 (0.096 versus 0.146), and normalized HSIC (0.128 versus 0.152 and 0.151). These findings provide additional evidence of the decomposition's benefits in data-limited, higher-dimensional settings. At larger sample sizes, the references can achieve lower MSE, highlighting that the advantage is setting-dependent rather than universal.

Finally, due to space constraints, we originally prioritized theory-validating simulations in the main text. With additional space in the revision, we will move the second real-data experiment into the main text and report the expanded results in Appendices F.3–F.5. To improve readability, we have also redesigned the figures to present test MSE–unfairness trade-off frontiers, with clear labels, consistent typography, and standard-error bars, using vector graphics for improved visual quality.

W3. Nonlinear extensions: We agree that the kernel and deep-learning extensions should be presented more clearly as future directions. The appendix currently outlines how the proposed structural ideas might extend beyond linear regression, but does not establish corresponding theoretical or empirical guarantees. We will revise this discussion accordingly, emphasizing that developing such guarantees requires further research.

AuthorsOpenReview ↗

Rebuttal 2

Q1. Method acronym: We appreciate the reviewer for identifying the naming overlap with Donini et al. (2018). To avoid confusion, we will rename our method Fair Subspace Envelope Regression (FSER) throughout the revised manuscript, better reflecting its predictive–sensitive subspace decomposition.

Q2. Selection of λ∗\lambda^*: We determine λ∗\lambda^* through a one-dimensional numerical search, selecting the smallest nonnegative λ\lambda satisfying the prescribed fairness constraint. Although the fairness measure can be expressed analytically as a function of λ\lambda, inverting this relationship generally does not yield a simple closed-form solution.

Q3. Compatibility condition:

We characterize the commutativity condition as a sufficient condition for the exact four-way orthogonal decomposition, rather than a requirement for implementing the algorithm. Without this condition, predictive and sensitive directions may be closely related without overlapping exactly, leaving some directions unaccounted for in the decomposition. The predictive envelope remains well-defined, and the regression algorithm can still be implemented, although the exact identification of shared and predictive-only directions and the associated theoretical guarantees may no longer hold.

To assess the assumption in practice, we tracked the commutator norm δ=∣∣PYPS−PSPY∣∣F\delta=||P_YP_S-P_SP_Y||_F in our experiments. Across all MEPS experiments, δ≈10−13\delta\approx10^{-13}-10−1410^{-14}, indicating compatibility up to numerical precision. In Communities and Crime, the mean δ\delta decreased from 0.609 to 0.329 with increasing sample size. Despite these departures, the algorithm remained operational and achieved favourable fairness-utility performance in selected settings. Thus, compatibility ensures the exact theoretical interpretation, while our experiments demonstrate that exact commutativity is not necessary for practical implementation. We will clarify this distinction and include the diagnostics in the revision.

We thank the reviewer again for their constructive suggestions, which have helped us strengthen the empirical evaluation, improve the presentation, and clarify the scope of our theoretical contributions. We believe these additions directly address the reviewer's concerns and substantially improve the manuscript.

Thank you for the detailed response and extensive set of additional evaluations. My main concerns regarding the choice of the fairness criterion as well as those regarding the scope of the empirical evaluation are now resolved. The same is true for my questions raised above.

To improve readability, we have also redesigned the figures to present test MSE–unfairness trade-off frontiers, with clear labels, consistent typography, and standard-error bars, using vector graphics for improved visual quality.

As the revised document is not uploaded to OpenReview I cannot yet confirm that these (and other) points are adequately incorporated in the revision. Once you have uploaded the revised document, please reply to let me know so I can review the updated PDF and raise my score accordingly.

Reviewer_1ag3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper decomposes the feature variation into four components according to its relevance to the response and its association with the sensitive attributes. The authors introduce Fair Envelope Regression Models (FERM), which excludes sensitive-associated variation that is immaterial for prediction and penalizes only the shared predictive-sensitive component. Theoretical analysis is provided to characterize the fairness-utility trade-off.

Critical strengths
  1. The predictive-sensitive decmposition is intuitive and interpretable.
  2. The paper provides theoretical analysis of the proposed fair regression estimator.
Critical weaknesses
  1. The fairness criterion and ridge-based fairness control are inherited from prior fair regression methods. The main contribution is to distinguish predictive sensitive-associated directions from response-immaterial sensitive-associated directions.
  2. The paper does not provide theoretical guarantees for finite-sample or asymptotic control of the unfairness measure at a prespecified level.
  3. The robustness of the proposed method to model misspecification is not analyzed, either theoretically or empirically.
Questions for authors
  1. The paper focuses on a linear regression model and a specific linear-association fairness criterion. Can the proposed framework be extended to more general regression models and alternative notions of fairness? Can the proposed method be applied when the sensitive attribute SS is discrete?
  2. Can the compatibility condition be assessed empirically, and how sensitive is the method to violations of this condition?
  3. How is the parametric convergence rate of P^Y\hat P_Y established? Are the dimensions of XX and SS assumed to be fixed and finite?
  4. The theoretical analysis does not fully explain the advantages of the proposed method. For example, it remains unclear why the proposed decomposition should lead to improved predictive performance, and whether the method can guarantee control of the unfairness measure at a prespecified level.
  5. How sensitive is the proposed method to model misspecification?
  6. It would be helpful to include a diagonal reference line in the unfairness plots to indicate agreement between the prescribed and attained unfairness levels.
3 comments
AuthorsOpenReview ↗

Rebuttal 1

We thank the reviewer for their constructive assessment and for recognizing the interpretability of our predictive–sensitive decomposition and its theoretical contributions. We have substantially strengthened the manuscript with new finite-sample and asymptotic guarantees, a more explicit characterization of predictive efficiency, and expanded empirical evaluations. These additions directly address the concerns raised, as detailed below.


Novelty and theoretical advantages (Weakness 1; Q4)

Our central contribution is precisely the predictive-sensitive subspace decomposition, rather than another fairness criterion or regularization penalty. Using an established fairness criterion allows us to isolate the advantages of this new statistical structure relative to existing fair regression methods.

The decomposition is not merely descriptive: it determines which predictor directions are removed, preserved, or regularized. FSER eliminates response-immaterial variation, retains predictive-only directions without fairness penalization, and regularizes only the shared predictive-sensitive component. Moreover, it provides subspace-level auditability, explicitly identifying where predictive information and sensitive association overlap and where the fairness-utility trade-off arises.

We have further strengthened the theoretical justification for these advantages. Beyond the original asymptotic efficiency results (Section 5.1), Appendix H establishes new finite-sample prediction-risk results. Under the oracle orthogonal decomposition, eliminating d0d_0 response-immaterial directions reduces prediction risk by σ2d0/n\sigma^2d_0/n relative to an otherwise identical estimator retaining them. We additionally derive an explicit bias-variance characterization of selective regularization and a uniform high-probability prediction-error bound for data-selected fairness penalties. The expanded real-data experiments (Section 7; Appendices F.3-F.5) provide complementary evidence of favorable fairness-utility performance.

These results demonstrate that the proposed decomposition is a substantive methodological contribution, providing quantifiable predictive efficiency gains, selective fairness regularization, and subspace-level auditability beyond what the fairness criterion or ridge penalty alone provides.


Theoretical fairness guarantees and empirical validation (Weakness 2; Q4; Q6)

We establish a finite-sample bound quantifying the discrepancy between prescribed and population unfairness, which further yields asymptotic population fairness control at any prescribed level rr, extending the original endpoint result (Theorem 5.4). Using the covariance-matrix representation of multiple-RS2R_S^2, uniform convergence, and matrix concentration inequalities, we establish that any empirically feasible FSER estimator satisfies RS2(β^r)≤r+rεB+εA,R_S^2(\widehat\beta_r)\leq r+r\varepsilon_B+\varepsilon_A, where εA\varepsilon_A and εB\varepsilon_B quantify covariance estimation errors. Under the stated regularity and concentration conditions, this yields RS2(β^r)≤r+Op(dX+dSn),R_S^2(\widehat\beta_r) \leq r+O_p\left(\sqrt{\frac{d_X+d_S}{n}}\right), establishing asymptotic population fairness control at arbitrary prescribed levels. Importantly, these guarantees do not require exact recovery of the predictive-sensitive subspaces or the commutativity condition. Complete theorem statements and proofs are provided in the new Appendix G.

Following the reviewer's suggestion, Figures 15, 16, 20, and 24 now plot attained test unfairness against prescribed levels, with diagonal reference lines (Q6). FSER generally maintains test unfairness at or below the prescribed thresholds, except in some small-sample settings with stringent fairness constraints. In contrast, FRRM and Komiyama exceed these thresholds more frequently, including at larger sample sizes (Figure 20).


Robustness to model misspecification (Weakness 3; Q5)

Our new theoretical results in Appendix G also establish that population fairness control does not require correct specification of the conditional mean E[Y∣X]E[Y\mid X]. Thus, FSER retains its fairness-generalization guarantees under nonlinear regression misspecification, provided the covariance regularity conditions hold.

We further provide a dedicated sensitivity analysis (Appendix I) examining nonlinear effects in predictive-only, shared predictive-sensitive, and linearly response-immaterial directions, nonlinear interactions, and heteroskedasticity. Across these departures, test unfairness remains largely stable as misspecification severity increases. Predictive performance also exhibits relatively limited deterioration under moderate fairness constraints, particularly in the larger-sample setting, while stringent constraints produce greater variability. These results directly characterize both fairness and predictive sensitivity beyond the correctly specified linear model.

AuthorsOpenReview ↗

Rebuttal 2

Q1: We have expanded Appendix D to outline potential extensions of FSER to nonlinear regression through kernel methods and deep learning approaches. Importantly, the proposed predictive-sensitive subspace decomposition is conceptually separate from the subsequent fairness-constrained regression step. This modular structure allows alternative fairness criteria, such as nonlinear dependence measures, to be incorporated into the optimization framework. However, establishing analogous theoretical guarantees requires additional analysis. In particular, the closed-form characterization of the fairness-utility trade-off in Section 5.3 does not immediately extend to general fairness constraints. We therefore leave these theoretical extensions for future work.

FSER already accommodates discrete sensitive attributes through one-hot encoding. Indeed, our expanded real-data experiments use categorical sensitive attributes, including gender and ethnicity in MEPS, demonstrating the applicability of FSER to mixed continuous and discrete sensitive variables.


Q2: We characterize the commutativity condition as a sufficient condition for the exact four-way orthogonal decomposition. Without this condition, predictive and sensitive directions may be closely related without overlapping exactly, leaving some directions unaccounted for in the decomposition. The predictive envelope remains well-defined, and the regression algorithm can still be implemented, although the exact identification of shared and predictive-only directions and the associated theoretical guarantees may no longer hold.

We assess compatibility through the Frobenius norm of the commutator, δ=∣∣PYPS−PSPY∣∣F\delta=||P_Y P_S-P_S P_Y||_F. Despite measurable departures from commutativity, FSER remains operational and achieves competitive fairness–utility performance in several settings (Figure 23). Figure 23 plots the mean estimated commutator norm against sample size for Communities and Crime, with dX=70d_X=70 fixed and error bars representing ±1\pm1 standard error. These results confirm that exact commutativity is not necessary for implementation and provide empirical evidence of useful performance when the compatibility condition is not exactly satisfied.


Q3: The parametric rate in Proposition 5.3 follows from established n\sqrt{n} consistency results for envelope estimators. Specifically, the estimated response-material envelope projector P^Y\widehat P_Y, obtained using the procedures described in Appendix B.1, satisfies ∣P^Y−PY∣=Op(n−1/2)|\widehat P_Y-P_Y|=O_p(n^{-1/2}) under standard envelope regularity conditions; see Cook and Zhang (2016, Proposition 6) for the 1D algorithm and Zhang et al. (2023, Theorem 1) for NIECE. Combining this with β^OLS−β=Op(n−1/2)\widehat\beta_{\mathrm{OLS}}-\beta=O_p(n^{-1/2}) and PYβ=βP_Y\beta=\beta, the decomposition $\widehat\beta_{\mathrm{env}}-\beta =(\widehat P_Y-P_Y)\widehat\beta_{\mathrm{OLS}} +P_Y(\widehat\beta_{\mathrm{OLS}}-\beta)$ immediately yields β^env−β=Op(n−1/2)\widehat\beta_{\mathrm{env}}-\beta=O_p(n^{-1/2}), as established in Appendix C.

Our theoretical results operate in the classical fixed-dimensional regime, with dXd_X and dSd_S finite and fixed as n→∞n\to\infty.


References

Cook, R. D., & Zhang, X. (2016). Algorithms for Envelope Estimation. Journal of Computational and Graphical Statistics, 25(1), 284–300.

Zhang, X., Deng, K., and Mai, Q. (2023). Envelopes and principal component regression. Electronic Journal of Statistics, 17(2):2447–2484.


We sincerely thank the reviewer again for their thoughtful comments and constructive suggestions, which have helped us substantially strengthen both the theoretical and empirical contributions of our work. We hope that the new results, expanded evaluations, and clarifications satisfactorily address all the concerns raised and demonstrate the significance of our proposed framework. We would greatly appreciate the reviewer’s reconsideration of their assessment in light of these substantial revisions.

We sincerely thank the reviewer again for their thoughtful comments and constructive suggestions. As the discussion period approaches its conclusion, we would greatly appreciate your assessment of our detailed rebuttal and revised manuscript.

We have directly addressed your principal concerns through new finite-sample and asymptotic population fairness guarantees at arbitrary prescribed levels (Appendix G), finite-sample prediction-risk results quantifying the benefits of our predictive-sensitive decomposition (Appendix H), and a dedicated robustness analysis under model misspecification (Appendix I). We have also clarified the compatibility condition and convergence assumptions, and added the requested attained-versus-prescribed unfairness plots.

We believe these substantial additions address the theoretical and empirical concerns raised in your review and considerably strengthen the novelty and significance of FSER.

Given these new theoretical contributions and expanded empirical evidence, we would sincerely appreciate it if you could reconsider your initial rating and consider increasing your assessment of our work.

Thank you again for your constructive feedback and consideration.

Reviewer_maSXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new method for enforcing fairness to linear regression tasks with multivariate sensitive features. It employs envelope methods for decomposing the features into four components: the predictive only, predictive and sensitive, sensitive only and residuals. It then removes the variation that does not help to predict the ground truth and apply the fairness penalty only to the predictive and sensitive component. On synthetic datasets and on MEPS real world data, the approach shows lower or comparable error than the Fair Ridge Regression baseline for similar fairness levels.

Critical strengths

I found the decomposition into four components intuitive and well motivated with the penalty applied only to the component that is both predictive and sensitive, rather than to all sensitive associated variation. The method is supported by theoretical results (consistency, asymptotic fairness and variance reduction).

Critical weaknesses

My main concern is on the experimental evaluation that I found insufficient, the real data experiments use only a small fraction of the data (<1250 samples) and the approach is evaluated only on two datasets both on the same healthcare domain (one in main paper and another in appendix). For the only real dataset in the main paper (MEPS) it used some payment variables as sensitive attributes which are not common [1]. Since total expenditure seems to include these payments, S is partly a component of Y, so the experiment may not reflect a realistic fairness setting. The comparisons to other baselines is also insufficient since it only compares to FRRM. I would have expected other comparisons such as Komiyama et al. (2018) which use the same R2R^2 fairness criterion with multivariate sensitive attributes. The approach only mitigates linear links, the prediction can still have non linear links with the sensitive attributes.

[1] Fabris, Alessandro, et al. "Algorithmic fairness datasets: the story so far:" Data Mining and Knowledge Discovery 36.6 (2022): 2074-2152

Questions for authors
  1. Can you measure the nonlinear dependence between the FERM prediction and the sensitive attributes (e.g., HGR)?
  2. Why are the experiments only limited to 1250 samples? How does the approach perform on the full dataset?
  3. Could you add real world datasets outside the healthcare domain?
  4. How do the MEPS results change with gender, ethnicity and age as sensitive attributes without the payment variables?
4 comments
AuthorsOpenReview ↗

Rebuttal 1

We thank the reviewer for recognizing FERM's four-way decomposition and theoretical contributions. We have substantially expanded the empirical evaluation to address all raised concerns, adding larger-sample and full-cohort MEPS experiments, demographic sensitive attributes, the non-healthcare Communities and Crime dataset, comparisons with Komiyama et al. (2018), and nonlinear dependence diagnostics (HGR and HSIC). The new results clarify FERM's fairness–utility performance across sample sizes, application domains, and deployment requirements.


1. Larger-sample and full-cohort in MEPS

Our original experiments deliberately focused on moderate sample sizes n≤1,250n\leq 1{,}250 to evaluate FERM in the regime where the statistical efficiency gains established by our envelope-based theory are most relevant. This choice allowed us to assess whether the proposed structural decomposition translates into improved fairness-utility trade-offs under limited training data. To further establish performance beyond this regime, we have now extended the experiments to n∈{750,1000,1250,1500,2000,3000,5000}n \in \{750,1000,1250,1500,2000,3000,5000\} and the full MEPS cohort n=15,329n=15{,}329.

Results: FERM achieves improved or comparable (differences are often within the reported standard errors) fairness-utility trade-offs relative to FRRM across several sample sizes and fairness levels. Across the full-cohort fairness-level grid, FERM achieves lower mean test MSE and held-out RS2R_S^2 than FRRM at every reported level, demonstrating that its predictive advantages are not restricted to small samples. HGR reveals a corresponding nonlinear dependence–utility trade-off, with FERM attaining lower nonlinear dependence under stricter constraints but not uniformly across budgets. Komiyama achieves lower MSE in this experiment, under different prediction-time information requirements.

Original MEPS: Full-cohort evaluation (Mean ± SE over 30 replications. Lower values are better for all metrics.)

Method / Metricr = 0.05r = 0.10r = 0.15r = 0.20r = 0.30
FERM
Test MSE0.961 ± 0.0440.950 ± 0.0440.939 ± 0.0440.927 ± 0.0440.890 ± 0.043
Test RS2R_S^20.051 ± 0.0020.101 ± 0.0030.153 ± 0.0030.204 ± 0.0030.308 ± 0.003
HGR0.269 ± 0.0140.322 ± 0.0120.371 ± 0.0110.419 ± 0.0100.523 ± 0.011
FRRM
Test MSE0.969 ± 0.0440.963 ± 0.0440.959 ± 0.0440.955 ± 0.0440.947 ± 0.043
Test RS2R_S^20.066 ± 0.0040.118 ± 0.0060.168 ± 0.0070.219 ± 0.0080.318 ± 0.010
HGR0.325 ± 0.0100.339 ± 0.0100.366 ± 0.0100.392 ± 0.0090.438 ± 0.009
Komiyama
Test MSE0.924 ± 0.0420.874 ± 0.0380.826 ± 0.0350.777 ± 0.0310.682 ± 0.025
Test RS2R_S^20.063 ± 0.0030.113 ± 0.0040.162 ± 0.0050.211 ± 0.0060.309 ± 0.008
HGR0.332 ± 0.0100.339 ± 0.0100.355 ± 0.0100.376 ± 0.0110.422 ± 0.011

2. Additional reference: Komiyama et al. (2018)

We initially prioritized FRRM to isolate the contribution of targeted subspace regularization relative to conventional ridge-based fairness penalization. We have now included Komiyama as a complementary constrained-optimization reference, whose formulation has documented numerical challenges near (r=0) (Scutari et al., 2022, Section 3). The methods address different practical objectives: Komiyama directly minimizes prediction error under fairness constraints, whereas FERM exploits low-dimensional predictive structure, enables subspace-level auditability, and predicts from XX alone. Both FRRM and Komiyama require sensitive attributes at prediction time in the evaluated formulations. Komiyama's lower MSE in some settings establishes its predictive advantage under those conditions, but not under FERM's XX-only deployment requirement. We will clarify these distinctions in the revised manuscript.


3. Nonlinear dependence

We agree that controlling linear dependence does not guarantee nonlinear independence. Accordingly, we added held-out empirical HGR and normalized HSIC diagnostics. The results demonstrate that FERM exhibits meaningful nonlinear dependence-utility trade-offs, achieving lower nonlinear dependence than the references in several settings, though not uniformly. These evaluations complement, rather than extend, our theoretical guarantees, which concern linear fairness.


AuthorsOpenReview ↗

Rebuttal 2

4. Demographic sensitive attributes in MEPS

The original MEPS experiment used socioeconomic and payment-related sensitive attributes to evaluate FERM under strong multivariate associations with the response, providing a challenging setting for assessing fairness-utility trade-offs. To address the reviewer's concern about payment-related sensitive attributes, we conducted additional experiments using gender, ethnicity, and age as sensitive attributes, excluding payment variables. This eliminates the direct overlap between payment-related sensitive attributes and the expenditure response, providing a conventional demographic fairness setting. We evaluated FERM, FRRM, and Komiyama et al. (2018) across sample sizes and fairness budgets, using 30 replications. The table reports held-out predictive MSE, multiple RS2R_S^2, and an HGR (mean ± SE; lower is better).

Sample sizeMethod / Metricr = 0.05r = 0.10r = 0.20r = 0.30
1,000FERM
Test MSE0.989 ± 0.0880.943 ± 0.0850.905 ± 0.0770.904 ± 0.077
Test RS2R_S^20.074 ± 0.0040.120 ± 0.0050.194 ± 0.0070.195 ± 0.007
HGR0.170 ± 0.0160.231 ± 0.0140.341 ± 0.0120.343 ± 0.012
FRRM
Test MSE0.950 ± 0.0780.938 ± 0.0770.933 ± 0.0760.933 ± 0.075
Test RS2R_S^20.074 ± 0.0040.119 ± 0.0060.206 ± 0.0070.210 ± 0.008
HGR0.207 ± 0.0180.245 ± 0.0160.368 ± 0.0160.374 ± 0.017
Komiyama
Test MSE0.961 ± 0.0770.948 ± 0.0760.933 ± 0.0750.933 ± 0.075
Test RS2R_S^20.075 ± 0.0050.120 ± 0.0060.206 ± 0.0070.210 ± 0.008
HGR0.207 ± 0.0180.244 ± 0.0170.368 ± 0.0160.374 ± 0.017
2,000FERM
Test MSE0.603 ± 0.1140.587 ± 0.1120.583 ± 0.1120.583 ± 0.112
Test RS2R_S^20.059 ± 0.0040.106 ± 0.0050.167 ± 0.0080.169 ± 0.008
HGR0.202 ± 0.0170.262 ± 0.0140.339 ± 0.0130.341 ± 0.013
FRRM
Test MSE0.604 ± 0.1140.602 ± 0.1140.603 ± 0.1140.603 ± 0.114
Test RS2R_S^20.061 ± 0.0030.107 ± 0.0040.153 ± 0.0050.153 ± 0.005
HGR0.177 ± 0.0100.247 ± 0.0120.328 ± 0.0120.328 ± 0.012
Komiyama
Test MSE0.610 ± 0.1140.606 ± 0.1140.603 ± 0.1140.603 ± 0.114
Test RS2R_S^20.060 ± 0.0030.106 ± 0.0040.153 ± 0.0050.153 ± 0.005
HGR0.177 ± 0.0110.247 ± 0.0120.328 ± 0.0120.328 ± 0.012
Full (15,329)FERM
Test MSE0.984 ± 0.0410.966 ± 0.0410.928 ± 0.0400.907 ± 0.039
Test RS2R_S^20.051 ± 0.0020.100 ± 0.0020.200 ± 0.0020.245 ± 0.003
HGR0.249 ± 0.0110.295 ± 0.0100.410 ± 0.0100.453 ± 0.010
FRRM
Test MSE0.915 ± 0.0390.909 ± 0.0390.903 ± 0.0390.902 ± 0.039
Test RS2R_S^20.050 ± 0.0010.100 ± 0.0020.199 ± 0.0020.252 ± 0.003
HGR0.248 ± 0.0110.282 ± 0.0110.406 ± 0.0100.467 ± 0.010
Komiyama
Test MSE0.915 ± 0.0390.909 ± 0.0390.903 ± 0.0390.902 ± 0.039
Test RS2R_S^20.051 ± 0.0010.100 ± 0.0020.199 ± 0.0020.252 ± 0.003
HGR0.251 ± 0.0110.281 ± 0.0110.404 ± 0.0100.467 ± 0.010

Results and comparison: With conventional demographic sensitive attributes, FERM achieves competitive predictive accuracy and favorable linear and nonlinear dependence in several moderate-sample settings. At larger sample sizes, FRRM and Komiyama perform better under stricter constraints, while FERM approaches their predictive accuracy as constraints relax. At the full cohort and (r=0.30), FERM achieves comparable MSE with slightly lower (R_S^2) and HGR. These findings establish that FERM's favorable empirical properties are not dependent on treating payment-related variables as sensitive attributes, and that its linear fairness regularization can also reduce nonlinear dependence in selected settings.


AuthorsOpenReview ↗

Rebuttal 3

5. Evaluation beyond healthcare: Communities and Crime

To address the reviewer's concern about domain diversity, we conducted additional experiments on the Communities and Crime dataset, with per-capita violent crime rate (ViolentCrimesPerPop) as the continuous response YY. We selected two continuous sensitive attributes: the percentage of Black residents (racepctblack) and the percentage of foreign-born residents (PctForeignBorn). The predictors XX comprise community-level socioeconomic and demographic characteristics, excluding the response, sensitive attributes, and identifiers.

We evaluated FERM, FRRM, and Komiyama et al. (2018) over an extensive grid of sample sizes (n=100n=100–1,9001{,}900), predictor dimensions (p=10p=10–9090), and fairness budgets, using 50 replications per configuration and identical 75%/25% training–test splits across methods. Performance was assessed using held-out predictive MSE, multiple RS2R_S^2, and normalized HSIC. We additionally recorded estimated subspace dimensions and commutator norms to examine FERM's structural behavior. Representative results spanning data-limited and larger-sample regimes are reported below; the complete experimental grid will be included in the revision.

Representative results: Communities and Crime.

Mean ± SE over 50 replications. Lower values are better. The fairness budget rr refers to the training constraint, while all reported metrics are evaluated on held-out data.

ConfigurationMethodMSE (r=0.10)Test R²_SHSICMSE (r=0.30)Test R²_SHSIC
n=150, p=70FERM1.344 ± 0.0730.0960.1281.218 ± 0.0680.1240.145
FRRM1.542 ± 0.1100.1460.1521.394 ± 0.1020.2320.187
Komiyama1.977 ± 0.1440.1460.1511.770 ± 0.1290.2320.187
n=200, p=70FERM1.137 ± 0.0610.1020.1151.023 ± 0.0560.1470.136
FRRM1.131 ± 0.0640.1220.1270.966 ± 0.0570.2260.174
Komiyama1.379 ± 0.0880.1220.1271.186 ± 0.0790.2270.174
n=300, p=70FERM0.931 ± 0.0330.0840.0840.850 ± 0.0310.1460.111
FRRM0.804 ± 0.0300.1190.1090.655 ± 0.0250.2870.183
Komiyama0.892 ± 0.0330.1200.1090.733 ± 0.0270.2880.184
n=1,000, p=70FERM0.934 ± 0.0180.0440.0440.886 ± 0.0170.1150.081
FRRM0.651 ± 0.0140.1040.0970.513 ± 0.0110.2890.172
Komiyama0.654 ± 0.0140.1050.0960.504 ± 0.0100.2890.172
n=1,900, p=70FERM0.957 ± 0.0130.0440.0360.912 ± 0.0120.1200.077
FRRM0.644 ± 0.0090.1050.0900.507 ± 0.0070.3010.167
Komiyama0.628 ± 0.0090.1060.0890.477 ± 0.0070.3010.168

Results and Comparison: The experiments reveal two distinct regimes. In data-limited, higher-dimensional settings, particularly (n=150,p=70), FERM achieves lower predictive MSE and lower linear and nonlinear dependence than both FRRM and Komiyama at the reported fairness budgets. At n=200,p=70n=200,p=70, FERM remains competitive in predictive accuracy while maintaining lower dependence. As sample size increases with p=70p=70 fixed, FRRM and Komiyama achieve lower predictive MSE, whereas FERM exhibits substantially lower held-out RS2R_S^2 and normalized HSIC, at a predictive cost. These results characterize complementary fairness–utility trade-offs rather than uniform superiority.

Structural diagnostics: FERM consistently identifies a substantially lower-dimensional predictive envelope: at n=1,900,p=70n=1{,}900,p=70, the estimated predictive dimension averages 16.22, representing approximately 77% dimension reduction. The mean commutator norm generally decreases from 0.61 at n=150n=150 to 0.33 at n=1,900n=1{,}900, indicating reduced incompatibility between the estimated predictive and sensitive projections as sample size increases. These diagnostics support the structural interpretation of FERM's decomposition without implying guaranteed subspace recovery.

Overall, the non-healthcare experiments demonstrate FERM's applicability across sample-size regimes, with predictive benefits in selected data-limited settings and stronger observed dependence reduction in larger-sample settings. FERM additionally enables prediction without sensitive attributes at deployment and provides subspace-level auditability of fairness regularization.


6. Due to space constraints, we originally prioritized controlled simulations validating our theoretical results in the main text and placed the second real-data experiment in the appendix. With the additional space available for revision, we will move the second real-data experiment into the main text and incorporate the expanded experimental results and diagnostics in Appendices F.3–F.5.


We sincerely thank the reviewer again for their constructive feedback. As the discussion period approaches its conclusion, we would greatly appreciate your assessment of our detailed rebuttal and revised manuscript.

We have made substantial additions directly addressing your concerns, including larger-sample and full-cohort MEPS experiments, conventional demographic sensitive attributes, the non-healthcare Communities and Crime benchmark, comparisons with Komiyama et al. (2018), and nonlinear dependence evaluations using HGR and HSIC. These results are now incorporated into Section 7 and Appendices F.3-F.5.

We believe these extensive additions satisfactorily address the concerns raised in your review and substantially strengthen the empirical validation of FSER.

In light of these revisions, we would be very grateful if you could reconsider your initial rating and consider increasing your assessment of our work.

Thank you again for your time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.