acceptodds
Under review as a conference paper at ICLR 2027

Attribute Inference on Differentially Private and Fair Learning

Abstract

Recent fairness-and-privacy frameworks claim to protect sensitive attributes by applying Differential Privacy (DP) during fair optimization. However, existing evaluations often overlook whether released models still leak sensitive information through proxy correlations or prediction outputs. We conduct a systematic evaluation of sensitive-attribute leakage in DP-based fair learning pipelines. Using attribute inference attacks across datasets, models, and privacy budgets, we show that perturbing sensitive attributes alone is often insufficient: structural correlations in non-sensitive features allow models to "re-learn" protected information that remains inferable from released outputs. Across eight datasets and multiple attacker models, our results reveal a gap between formal privacy guarantees and effective protection in downstream models, highlighting the need for evaluations that account for proxy correlations and output-level leakage.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Differential Privacy, Fairness, Attribute Inference Attack

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hwbsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper considers attribute inference for models trained with differential privacy and fairness guarantees. The paper runs an attribute inference attack for 3 different algorithms that train with attribute differential privacy and fairness guarantees, 8 tabular datasets, across different privacy budgets. The attack targets demographic attriibutes that are considered sensitive/protetcted. The paper finds that the relationship between attribute inference attack success and privacy budget need not be monotonic.

Critical strengths
  • The authors conduct a thorough experimental analysis across diverse datasets and algorithms.
Critical weaknesses
  • The contribution of the paper is rather limited as it does not introduce a new attack or algorithm, but evaluates an existing attack on prior methods.
  • The authors acknowledge that the success of the attribute inference attack can be based on prior correlations between the protected attribute and other features, thus the success of the attack is not an indication of the model's privacy leakage. However, the authors do not provide any baseline for attack success without access to model output. They also do not use attacks which take into account the attacker's prior probability of success. As such, the attack results are hard to interpret. In the end, there is no clear takeway, other than the fact that the privacy budget does not have a clear relationship with the attack success.
  • The authors suggest that empirical audits are important for understanding what information is "recoverable from deployed model outputs", however I find this language misleading because we have no baseline for what is recoverable about the sensitive attribute even before the model outputs are deployed.
Questions for authors

n/a

2 comments

We thank the reviewer for the constructive feedback and for recognizing the breadth of our experimental evaluation. We address the main concerns below.

1. Contribution and novelty. Our goal is not to introduce a new attack or algorithm, but to highlight an important gap in the evaluation of fair-learning methods that apply DP to protect sensitive attributes. Existing evaluations typically focus on formal privacy guarantees, utility, and fairness, without assessing whether protected attributes remain inferable from released predictions. Across three representative pipelines, we show that attribute inferability can remain substantial and varies with the privacy budget and learning procedure. Our findings motivate incorporating attribute inference auditing into existing privacy–fairness–utility evaluations.

2. Prior correlations and missing baselines. We agree that successful attribute inference may arise from correlations between sensitive and non-sensitive features. However, our threat model considers an adversary who observes only the released prediction score S=fD(X)S=f_D(X), without access to the target's features (X). Thus, our evaluation measures attribute inferability from model outputs, rather than additional disclosure beyond what an adversary already observing (X) could infer. Without model outputs or other target-specific information, a population-prior baseline yields AUC = 0.5. While feature-only imputation would provide a useful complementary comparison, it assumes a different adversarial access setting. We clarify this distinction in our added section of appendix C and the interpretation of our results.

3. Main takeaway. Our central finding is not that DP guarantees are violated, but that formally protecting sensitive attributes does not necessarily prevent their inference from released predictions, notably due to feature correlations and how privatized attributes are used within learning pipelines. This distinction is particularly important for practitioners, who may interpret sensitive-attribute DP guarantees as providing broader protection against attribute inference than they actually ensure. Our work highlights this gap between formal guarantees and practical inferability, motivating the integration of empirical attribute inference audits into existing privacy–fairness–utility evaluations. We thank once again the reviewer again for the suggestions, which will help us clarify the scope and contributions of our work. We hope that our responses have addressed your main concerns and would be grateful if you could reconsider your assessment in light of these clarifications.

Dear Reviewer,

We wanted to follow up and ask whether you have had a chance to review our response. We remain available to provide any additional information or clarification that might be helpful.

Thank you for your time and consideration.

Reviewer_86wFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the inferability of sensitive attributes from model prediction outputs when Differential Privacy (DP) is combined with fairness-aware learning. We evaluate three representative Private Fair Learning approaches: (i) Lagrangian Dual Training based on Central DP, (ii) Lagrangian Fair Learning based on Local DP, and (iii) a Post-processing method that applies local perturbation to sensitive attributes. Through black-box Attribute Inference Attacks conducted on multiple tabular datasets, we analyze the relationships among the Privacy Budget, Predictive Utility, Fairness Violation, and Attribute Inference Performance. Experimental results demonstrate that there is no uniform monotonic relationship between the privacy budget ε and the AUC of attribute inference attacks. In particular, Model II may exhibit lower attack AUC as ε increases, whereas Model III displays the opposite trend. These findings highlight that the formal guarantees of DP do not imply that sensitive attributes are protected from inference via model prediction outputs. Accordingly, we recommend incorporating Attribute Inference Auditing as a complement to standard DP Privacy Accounting.

Critical strengths

1. The research question is practically significant.

The paper clearly distinguishes between the formal privacy guarantees of differential privacy (DP) and output-level attribute inferability, emphasizing that the two are not equivalent. This distinction has practical relevance for deploying machine learning systems that combine differential privacy with fairness constraints.


2. The experiments cover diverse privacy mechanisms and fairness-aware learning pipelines.

The authors evaluate three fairness-aware learning pipelines involving central differential privacy, local differential privacy, and demographic-aware post-processing. They examine predictive utility, fairness violations, and attribute inferability across multiple datasets and privacy budgets, providing a relatively comprehensive set of empirical observations.


3. Some experimental phenomena warrant further investigation.

In particular, Model II exhibits decreasing attack AUC as ε increases on certain datasets, whereas Model III shows the opposite trend in several settings. This suggests that different learning pipelines may affect the sensitive-attribute information contained in model outputs in different ways.

Critical weaknesses

W1. Increamental novelty. Prior work has extensively studied the privacy–fairness–utility trade-off in private fair learning [1, 2, 4]. The paper mainly extends existing approaches with output-level attribute inference evaluations, offering limited explanations of the underlying mechanisms.

W2. Confounding fairness objectives. Models II and III use different fairness objectives (Demographic Parity vs. Equalized Odds) and prediction pipelines. Prior work [1] has shown that test-time access to sensitive attributes affects privacy–fairness trade-offs, making the opposite AUC trends difficult to interpret.

W3. Missing baselines. The paper lacks DP-only and fairness-only baselines, making it difficult to isolate their respective effects on attribute inferability. Such comparisons are particularly important given the established impact of DP on group fairness [3, 4].

W4. Misalignment between fairness and privacy metrics. Fairness violations are measured on binary predictions, while attribute inference relies on continuous scores. Thus, improved fairness does not necessarily imply reduced attribute inferability.

W5. Unexplained fairness violations. Model III exhibits fairness violations above 0.1 despite a specified tolerance of 0.001. This discrepancy requires further explanation.


References

[1] Jagielski et al. "Differentially Private Fair Learning." ICML, 2019.

[2] Mozannar et al. "Fair Learning with Private Demographic Data." ICML, 2020.

[3] Tran et al. "Differentially Private Empirical Risk Minimization under the Fairness Lens." NeurIPS, 2021.

[4] Mangold et al. "Differential Privacy has Bounded Impact on Fairness in Classification." ICML, 2023.

Questions for authors

see weakness

3 comments

We sincerely thank the reviewer for the detailed and constructive assessment. We appreciate the recognition of the practical significance of our research question and the breadth of the experimental evaluation. We address each concern below.

W1: We thank the reviewer for this observation. While prior work has extensively studied the privacy–fairness–utility trade-off, our main contribution is to highlight an important gap in the evaluation of fair-learning methods that apply DP specifically to sensitive attributes. Existing evaluations typically focus on formal privacy guarantees, predictive utility, and fairness, without assessing whether protected attributes remain inferable from released predictions. Through a systematic audit of three representative pipelines, we show that attribute inferability can remain substantial and varies across privacy budgets and learning methods, partly reflecting differences in how privatized demographic information is used. This is particularly relevant for practitioners, who may interpret sensitive-attribute DP guarantees as providing broader protection against attribute inference than they actually ensure. Our findings therefore motivate incorporating attribute inference auditing into existing privacy–fairness–utility evaluations.

W2 : We thank the reviewer for highlighting this potential confounding factor. We initially retained each method's original fairness objective to faithfully reproduce its implementation. To address this concern, we additionally evaluated all models under Equalized Odds (EO), obtaining similar AUC trends and preserving our main conclusions (all the plots in section 5.4-5.6 have been updated).

W3: Our primary goal is to audit output-level attribute inferability in existing DP-based fair-learning pipelines. Our Pareto analysis jointly evaluates utility, fairness, and inferability across privacy budgets, showing that protected attributes can remain inferable despite formal DP guarantees. We acknowledge that DP-only and fairness-only ablations would help isolate individual effects, but address a complementary question beyond our main scope.

W4: We use classical decision-level DP/EO fairness violations, defined on binary predictions Y^∈0,1\hat Y\in{0,1}: $ {\mathrm{DP}} =\max_a\left| P(\hat Y=1\mid A=a)-P(\hat Y=1) \right|, $ $ {\mathrm{EO}} =\max_{a,y}\left| P(\hat Y=1\mid A=a,Y=y)-P(\hat Y=1\mid Y=y) \right|. $ These criteria concern hard decisions, while our attacks evaluate information exposed through continuous scores. Strong demographic parity instead requires independence of the entire score distribution from the protected attribute, S⊥AS\perp A. This is stronger than label DP and is measured by SPDD in Wasserstein Fair Classification (Jiang et al., 2020). Such distribution level fairness criteria lie outside our evaluation scope. In some evaluated settings, increasing ε reduces both fairness violations and attack AUC. This is consistent with the hypothesis that more accurate demographic information enables fairness optimization to reduce group-dependent information in predictions. However, we acknowledge that decision-level fairness does not guarantee score-level attribute privacy, and these observations should not be interpreted as establishing a causal relationship.

W5: The tolerance of 0.001 is a training target, not a certified bound on evaluation-time fairness violation. Model III uses finite-round Lagrangian optimization with bounded multipliers, which does not guarantee exact constraint satisfaction. Its post-processing also relies on finite-sample statistics reconstructed from privatized attributes, so satisfying these estimated constraints need not ensure the same tolerance on evaluation data. Both optimization and statistical estimation errors can contribute to the observed discrepancy. The experiments in Fair Learning with Private Demographic Data also report fairness violations exceeding the prescribed tolerance (Section 5.3, Figure 1), illustrating the distinction between the configured fairness tolerance and the fairness actually achieved.

We thank the reviewer again for the constructive feedback, which has helped us clarify the scope and contributions of our work. We hope our responses have addressed the main concerns and would greatly appreciate a reconsideration of the assessment in light of these clarifications.

Dear Reviewer,

We wanted to follow up and ask whether you have had a chance to review our response. We remain available to provide any additional information or clarification that might be helpful.

Thank you for your time and consideration.

Thank you for your response; it addresses some of my concerns. After reviewing all reviews, I decided to maintain my score

Reviewer_CyGEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper conducts a systematic evaluation of sensitive-attribute leakage in DP-based fair learning pipelines. Using attribute inference attacks across datasets, models, and privacy budgets, we show that perturbing sensitive attributes alone is often insufficient: structural correlations in non-sensitive features allow models to "re-learn" protected information that remains inferable from released outputs.

Critical strengths
  • The paper addresses an interesting topic, focusing on attribute inference attacks.

  • The paper is conducted under multiple pipelines, settings, and datasets, covering enough breadth of the topic.

Critical weaknesses
  • There is no baseline comparison inside the paper.

  • All datasets in the paper are tabular datasets. For tabular datasets, attribute inference attacks could be just imputation. It is difficult to tell whether the evaluation results in the paper show the privacy leakage of attributes or just imputation of hidden attributes. Further experiments may need to be conducted for this. Or more discussion on this should be provided.

  • The paper says "attack AUC follows neither a monotonic increase nor the usual intuition that weaker privacy should necessarily yield greater inferability." But the paper did not give deep discussion why this happens. This could just be due to noise, not a particular trend.

Questions for authors

See comments.

2 comments

We thank the reviewer for the constructive feedback and for recognizing the breadth of our evaluation. We appreciate the opportunity to clarify our threat model and strengthen the interpretation of our findings.

1. Missing baselines and distinction between attribute inference and imputation. We agree that distinguishing information already predictable from non-sensitive features from information accessible through model outputs is important. By “output-level leakage,” we refer to sensitive attribute information exposed through released prediction scores to an adversary who cannot observe the non-sensitive features X. This scope includes both information encoded in the model through training and information about the protected attribute carried by X and preserved in the model’s output. Our evaluation does not distinguish these sources. Specifically, the adversary computes A^=h(fD(X))\hat A=h(f_D(X)), rather than A^=h(fD(X),X)\hat A=h(f_D(X),X), where fDf_D is the trained model. Even when inference relies on population-level correlations involving X, releasing fD(X)f_D(X) can make some of that information accessible to an adversary who otherwise lacks access to X. Such inference based on correlation therefore falls within our intended scope of output-level exposure. We do not claim that it necessarily reflects memorization or additional disclosure beyond what an adversary already observing X could infer.
Still, we acknowledge that further experiments comparing X→AX\to A, (X,fD(X))→A(X,f_D(X))\to A, combining with our existing fD(X)→Af_D(X)\to A attack, would help assess how much attribute predictability is available from X alone and whether access to model outputs provides additional inference benefit. We provide the assessment of this part in our added appendix D.

2. Interpretation of non-monotonic attack AUC. We thank the reviewer for this observation. Importantly, DP protects sensitive attributes under a specified adjacency relation, but does not eliminate correlations with non-sensitive features or guarantee decreasing attribute inferability monotonically as privacy becomes stronger. Our results illustrate how this distinction manifests across learning pipelines. In Model II, increasing ε provides more accurate demographic information for fairness optimization, potentially reducing attribute inferability from predictions. Conversely, Model III also uses privatized demographic information during prediction-time post-processing. With binary Randomized Response, the probability of retaining the true attribute already reaches approximately 88% at ε = 2, potentially explaining the increasing attack AUC. Moreover, some observed differences are substantial relative to the reported 95% confidence intervals across ten seeds (e.g., Bank Marketing in Models I and II). While these explanations do not establish causality, they suggest that attribute inferability depends on how protected information is used within the learning pipeline, rather than on ε alone. We will clarify this point in the discussion of results. We thank the reviewer again for the constructive feedback, which has helped us clarify our findings and contributions. We hope our responses have addressed your concerns and would greatly appreciate your reconsideration of the paper's overall assessment and score in light of these clarifications.

Dear Reviewer,

We wanted to follow up and ask whether you have had a chance to review our response. We remain available to provide any additional information or clarification that might be helpful.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.