acceptodds
Under review as a conference paper at ICLR 2027

Using Rashomon Sets to Reduce Identity Leakage in Data Collected from People

Abstract

Machine Learning models increasingly recognize gestures and activities from behavioral data. The same measurements can also reveal who produced them. We use Rashomon sets and Model Class Reliance to construct models that retain the features most essential to a prediction task while simultaneously reducing reliance on identity-rich features. The models in a Rashomon set (i.e., all near-optimal models) often rely on different combinations of features. Treating identity as a second prediction task, we use Model Class Reliance to find features on which every near-optimal identity model depends, then trace effects of removing those features. We evaluate this approach on 10 public datasets from six sources, covering touch gestures, activity sensing and hospital records. As an example, in 14 of 18 ShearSense leave-one-subject-out folds, every near-optimal gesture tree uses at least one identity-reliant feature, so lower identifiability cannot be obtained simply by choosing another near-optimal gesture model. We therefore remove identity-reliant features one at a time, trace the full accuracy-identifiability curve, and cut it where further removal would cost more target task accuracy than identity accuracy. On the 6 datasets where the cut lowers identity accuracy by at least 5 percentage points, identity accuracy falls by 8.7 to 58.0 points and target task accuracy by 0.4 to 14.1 points. A pre-trained tabular deep model shows a similar target task cost. The tradeoff gives designers explicit control over target task accuracy and empirical identifiability, while identity retained in raw sensor data remains a system-design consideration.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
model multiplicity, representation selection, feature selection, human activity recognition, behavioural data, identity prediction, privacy-aware Machine Learning, Rashomon sets

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xtBLOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a pre-deployment procedure for auditing identity leakage in data collected for behavioral or clinical prediction tasks. It uses approximate Rashomon sets and Model Class Reliance to identify features consistently used by sampled near-optimal identity models, tests whether near-optimal target models can avoid them, and measures the target-accuracy cost of removing them.

Across ten public datasets, identity is predictable above chance from target-task features. On six datasets, feature removal reduces identity accuracy by 8.7-58.0 percentage points at a target-task cost of 0.4-14.1 points; on four high-dimensional datasets with substitutable features, the effect is limited. The results are further evaluated with TabPFN, alternative feature-selection methods, differential privacy, different Rashomon-set constructions, and personalization. The method is an empirical privacy–utility audit, not an anonymization or formal privacy guarantee.

Critical strengths

The paper addresses a practically important and underexamined deployment risk. It provides an interpretable workflow for measuring identity leakage in task-oriented features and explicitly quantifying the trade-off between reduced identifiability and target-task accuracy.

Critical weaknesses

The necessity and incremental value of the Rashomon analysis are not established. A single identity model selects highly similar feature subsets (median Jaccard 0.90) and is statistically indistinguishable from the proposed method on both tasks in three of five testable datasets. Moreover, the target-task Rashomon step does not directly determine the subsequent XGBoost removal curve. Thus, the central contribution appears to be a costly combination of existing tools without clear evidence that the Rashomon component materially improves the audit.

Key methodological choices threaten the validity of the main conclusions. The paper claims to identify features required by “every near-optimal identity model,” but flags a feature when this holds for only one model class; nearly all flags come from GAMs while near-optimal trees can avoid them. In addition, zero-loss models are assigned MR=1, potentially suppressing reliance precisely on near-perfect identity datasets, and feature selection is not fully nested with identity evaluation. Correcting these issues would require substantial reanalysis and could change both the flagged features and reported identity reductions.

Questions for authors

Given the median Jaccard similarity of 0.90 with single-model identity importance, what measurable benefit does the Rashomon analysis provide over the simpler baseline?

Why does the paper describe features as required by “every near-optimal identity model” when a feature is flagged based on only one model class and near-optimal models from another class may avoid it?

Why is model reliance set to MR=1 when the intact loss is zero, and how do the results change under a loss difference or smoothed ratio?

How do the results change under a fully nested evaluation in which held-out samples and subjects are excluded from feature flagging, removal ordering, and cut selection?

Why does the method remove every flagged feature when no step achieves the required identity reduction or favorable accuracy trade-off?

Reviewer_N6ztOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses Rashomon sets and Model Class Reliance to identify features that near-optimal identity classifiers consistently rely on. It then examines whether accurate task models can avoid these features and measures the trade-off between task accuracy and identity accuracy after feature removal. Experiments show meaningful identity reductions on some datasets, but limited effects when identity information is spread across substitutable features.

Critical strengths
  • The paper highlights an important problem. Data collected for behavioral and health applications can also reveal user identity. I believe this deserves more attention from the community as AI systems become more common in everyday life.
Critical weaknesses
  • The main findings are not very surprising. Sensor features can reveal identity, and removing identity-related features can reduce task performance. The possibility of selecting features to balance these two goals is also expected. Although the analysis is thorough, I am not convinced that it provides enough new insight or methodological novelty to meet the bar for ICLR.
  • As acknowledged in the limitations, some samples used for feature selection are later used for identity evaluation. This makes it difficult to assess how well the reported identity reduction holds on independent data. A nested evaluation would provide stronger evidence.
  • The method operates on engineered features. It is unclear how the procedure would apply to end-to-end models that learn representations directly from raw sensor signals. This limits its relevance to such models.
  • The simpler method based on a single identity model selects very similar features and shows similar performance on several datasets. What additional insight or practical benefit does the Rashomon-set analysis provide, and does it justify the extra complexity?
Questions for authors

I wrote the comments in the weaknesses section.

Reviewer_1XaVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates identity leakage in datasets collected for generic tasks, such as gesture recognition and activity classification, where features intended for task prediction may inadvertently reveal individuals’ identities. The authors propose a feature selection framework that leverages sampled Rashomon sets of near-optimal models for both the target task and identity prediction. Using Model Class Reliance (MCR), the method identifies consistent and informative features for identity prediction that can be gradually removed while preserving target task performance. The key contribution is a joint analysis of near-optimal models across both tasks to facilitate identity-aware feature removal. Experiments show reductions in identity prediction accuracy ranging from 8.7 to 58.0 percentage points, with corresponding reductions in target task accuracy of 0.4 to 14.1 percentage points.

Critical strengths

S1. Practical relevance of the problem. Figure 1 demonstrates above-chance identity prediction across all ten datasets, establishing the practical significance of identity leakage in datasets collected for generic prediction tasks. The proposed analysis provides a useful framework for examining potentially identifying features before deciding which features to use or share.

S2. Principled use of Rashomon sets. The use of separate Rashomon sets for identity prediction and the target task provides a principled approach to identifying identity-related features and investigating whether near-optimal target-task models can avoid relying on them. This offers a systematic perspective on the trade-off between identity protection and task utility.

S3. Promising empirical results. Across the evaluated datasets, the method reduces identity prediction accuracy by 8.7–58.0 percentage points, with corresponding reductions in target-task accuracy of 0.4–14.1 percentage points. The results on the WISDM phone accelerometer dataset are particularly encouraging, with a 58.0-percentage-point reduction in identity prediction accuracy and only a 2.0-percentage-point reduction in target-task accuracy (Table 5).

S4. Broad experimental evaluation. The authors evaluate the approach across different sensing modalities and clinical settings. They also compare it against five feature-selection baselines, a pretrained deep tabular model (TabPFN), and a differentially private random forest (Sections 5.5–5.8). These comparisons help contextualize the proposed method and examine its strengths and limitations relative to alternative approaches.

S5. Additional statistical and sensitivity analyses. The paper reports per-subject evaluations, statistical significance tests with multiple-comparison correction, sensitivity analyses, and alternative approximations of Rashomon sets. These analyses strengthen the empirical evaluation and provide additional evidence regarding the robustness of the proposed approach.

Critical weaknesses

W1. Unclear incremental contribution of Rashomon sets. The proposed approach primarily combines existing techniques, including Rashomon sets, Model Class Reliance (MCR), and iterative feature removal. The paper should more clearly establish the specific benefit of using Rashomon sets over simpler alternatives. In several datasets, single-model selection produces similar feature subsets without significant differences in either task. On the watch accelerometer dataset, Rashomon sets reduce identity prediction accuracy by 3.7 percentage points but also reduce target-task accuracy by 4.1 percentage points (Table 10), suggesting a trade-off rather than a clear improvement. A comparison at matched target-task performance would help establish the incremental value of the proposed approach.

W2. Limited comparison with established privacy-preserving methods. Most evaluated baselines prioritize target-task performance rather than explicitly mitigating identity leakage. Established privacy-preserving approaches, including Wang et al. (2023), Iwasawa et al. (2017), and Malekzadeh et al. (2019), are discussed but not evaluated experimentally. Comparisons with these methods under matched privacy–utility evaluation protocols would provide a stronger assessment of the proposed method’s competitiveness.

W3. Potential data leakage during feature selection. As acknowledged in Section 6, samples subsequently used for identity evaluation also contribute to feature selection. The potential impact of this overlap on the reported identity-accuracy reductions is unclear and may lead to optimistic estimates. An evaluation in which feature selection is performed independently of the final test samples would provide stronger evidence of the method’s effectiveness and generalizability.

W4. Limited benefit over removing all identified features. The selected threshold results in the removal of all flagged features in most datasets. Consequently, the proposed optimization may offer limited practical advantage over the simpler strategy of removing every identified feature. An explicit comparison against this straightforward baseline would help establish whether the proposed procedure provides meaningful additional benefits.

W5. Clarity and accessibility of the presentation. Although the paper contains extensive analyses, the presentation is too articulated and often difficult to follow, particularly in the performance evaluation sections (seems highly AI-Driven). The motivation and conceptual development of the Rashomon-set approach also require clearer exposition. A more direct explanation of the central idea, its underlying assumptions, and its distinction from existing methods would improve accessibility. The extensive analyses would benefit from a more focused narrative emphasizing the most important findings.

Minor / Presentation Weaknesses

W6. Manuscript organization and readability. The writing requires improvement in several sections of the main manuscript and appendix. In particular, the leave-one-subject-out (LOSO) protocol and identity-based data splits should be explained consistently. A concise textual summary of the findings in Table 10 would also improve readability.

W7. Clarification of novelty relative to prior work. The paper should more explicitly distinguish its contributions from earlier approaches to privacy–utility trade-offs in feature selection, such as Xiao et al. (2018), to clarify its methodological novelty.

W8. Excessive material. The appendix is substantially longer than the main manuscript and contains an extensive amount of experimental analysis (seems highly AI-Driven). While additional experiments can strengthen a paper, the current presentation makes it difficult to identify the most consequential findings. The authors should prioritize the essential analyses and provide a more accessible organization of the supplementary material for humans to interpret.

Questions for authors

Q1. Incremental value of Rashomon-based feature selection: Given the high feature overlap (Jaccard similarity of 0.90) and comparable performance achieved by single-model identity importance, what specific advantages does the proposed Rashomon-based approach offer over the simpler single-model baseline? Furthermore, how does the target-task Rashomon set influence the final feature-selection decisions and improve the privacy–utility trade-off?

Q2. Comparison with privacy-preserving baselines: Could the authors provide experimental comparisons with directly relevant privacy-preserving methods, such as Wang et al. (2023), or a stronger identity-aware feature-selection baseline under matched evaluation protocols? Such comparisons would help establish the competitiveness of the proposed approach.

Q3. Effectiveness of the cut-selection mechanism: Since the selected cut removes all flagged features in nine of the ten datasets, what practical advantage does the proposed cut-selection mechanism offer over the simpler strategy of removing all flagged features? Could the authors quantify this advantage on the remaining dataset and clarify whether the optimization provides consistent benefits?

Q4. Practical impact of target-model avoidance analysis: Section 6 identifies the target-model avoidance check as an additional benefit of Rashomon analysis. Could the authors provide a concrete example in which this analysis changes the feature-selection or deployment decision compared with the single-model baseline?

Comments on the paper

1 comment

Thank you for your reviews. We are glad the problem came through clearly. For the ethics question, the release terms of all datasets we used permit this analysis, as documented in our ethics statement. We understand the issues we need to address, and your thoughtful comments point us toward what the next version of this work needs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.