acceptodds
Under review as a conference paper at ICLR 2027

Majority Is The Reference You Never Had

Abstract

Feature shifts across data sources are prevalent in applications involving healthcare, biomedicine, socioeconomic, financial, surveys, and multi-sensor data. Such shifts can arise from unharmonized heterogeneous data sources, noisy measurements, or inconsistent processing and standardization pipelines, resulting in erroneous features. Localizing shifted features is essential for identifying the underlying causes and correcting or filtering affected data before they degrade downstream analyses. Existing methods typically rely on a prespecified trusted reference to identify feature shifts in query data. In practice, however, a fully trustworthy reference source is often unavailable, making conventional pairwise comparisons unable to determine which data source is responsible for an observed shift. We propose Multi-source Reference-Free Feature Shift Localization (MuRF-FSL), an end-to-end neural network that simultaneously localizes shifted features across multiple data sources without requiring an external clean reference. Under the assumption that, for each feature, a strict majority of sources remain normal, MuRF-FSL treats the pattern shared by this majority as the missing reference and learns to attribute shifts by combining source-specific statistical representations with cross-source deviation patterns. Trained on a large and diverse collection of datasets with simulated shifts, MuRF-FSL generalizes to previously unseen datasets and shift scenarios without retraining. Experiments show that, despite operating in the more challenging reference-free setting, MuRF-FSL achieves state-of-the-art localization performance with highly efficient inference and outperforms reference-based methods provided with clean reference data.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
multi-source adaption, feature shift localization, free reference source

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TFk2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MuRF-FSL, a method for finding shifted features across multiple datasets without needing a clean reference dataset. The main idea is that if most data sources agree on a feature, their shared pattern can be used as a reference to identify which sources have shifted.

The authors use this idea in a neural network and test it on 11 datasets with different types of simulated shifts. The method achieves good results compared to the baselines, including methods that have access to a clean reference.

Critical strengths
  1. I like the main idea of the paper. In many real applications, we do not have a clean reference dataset. Using information from multiple sources to build an internal reference is a useful way to address this problem. The method is also flexible and can work with different numbers of sources and features.

  2. The experiments are extensive. The authors test the method on 11 datasets with different sizes and feature types. The results are strong, and the ablation study clearly shows that the source-relative component is important. I also appreciate that the method works without retraining on the test datasets.

Critical weaknesses
  1. My main concern is that all the shifts used for evaluation are artificially generated. Although the original datasets are real, the feature shifts are introduced through predefined operations. It is not clear how well the method would work on naturally occurring shifts, such as differences between hospitals, scanners, or data collection systems.

  2. The method assumes that most sources are normal for each feature. This is an important assumption, but it may not always hold in real applications. For example, several hospitals may use the same processing pipeline and introduce the same error. The experiments always follow this assumption, so we do not know how the method behaves when it is violated.

Questions for authors
  1. Have the authors tested the method on any dataset with naturally occurring feature shifts rather than simulated ones? I think such an experiment would help show how useful the method is in real applications.

  2. What happens when half or more of the sources have a shift in the same feature? Can the authors show how the performance changes as the number of shifted sources increases beyond the assumed limit?

Reviewer_7xd1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses source-wise and feature-wise shift localization when no clean reference source is available. It assumes that each feature is normal in a strict majority of sources, even though every source may contain shifted features. MuRF-FSL extracts per-source statistical descriptors, computes deviations from a leave-one-source-out median reference, and predicts shift probabilities for each source–feature pair.

Critical strengths

The problem is relevant: without a trusted clean source, a pairwise discrepancy does not establish which source is shifted. Forming an internal reference from multiple sources is therefore a well-motivated approach. The evaluation covers continuous and categorical datasets, including high-dimensional cases, and reports a dataset-equal mean F1 of 0.9192 on the constructed benchmark. The ablation is especially useful: adding the source-relative residual raises F1 from 0.6670 to 0.9074, identifying the component responsible for most of the gain. These results provide concrete evidence that the proposed comparison mechanism works under the tested simulated-shift conditions, even though they do not settle the broader identifiability and deployment questions below.

Critical weaknesses

The central concern is the paper's claim that within-source partial shifts and a strict normal majority make source-feature attribution identifiable. This does not hold for all dependence shifts covered by the task. Let clean binary features A and B be perfectly correlated and fair, and let independent C satisfy P(C=1)=0.8. In three sources, flip A, B, and C respectively. Each source changes only one feature and each feature is normal in two sources, yet flipping A or B produces the same observed joint distribution. Swapping those two shift labels leaves all observations unchanged. The proposed assumptions therefore do not uniquely determine the target labels in this case. Separately, the benchmark forms observable sources by randomly splitting one base dataset and injecting shifts. It does not establish reliability when sources differ naturally before any shift.

Questions for authors
  1. What additional condition makes the shifted-feature set unique for dependence shifts? If two clean binary features are perfectly correlated, flipping either can produce the same observed distribution. The strict-majority and partial-shift assumptions do not distinguish these explanations. Please provide a proof covering this case or state a narrower task definition.
  2. Can the method be evaluated when sources already have natural distributional differences before shifts are introduced? The present benchmark randomly partitions one base dataset and then injects artificial shifts. Testing genuinely heterogeneous sources would show whether the internal reference remains reliable in the setting motivating the paper.
1 comment

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end with about one day remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Reviewer_gbEFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses how to identify unreliable features across several datasets when none can be assumed completely trustworthy. In healthcare, several entities collecting the same measurements. One entity's equipment might distort one measurement, while another has problems with a different measurement. Comparing two reveals a disagreement, but does not tell you which one is wrong. This is the motivation of this paper. The proposed method, MuRF-FSL, compares all sources together. For each feature, it uses the pattern shared by the majority as an internal reference, then predicts which sources have unusual values or distributions. A neural network learns this process from datasets containing simulated shifts and applies it to new datasets without retraining. The paper reports strong detection performance across eleven unseen datasets.

Critical strengths
  1. The proposed method identifies shifted features without requiring a trusted clean dataset, which is often unavailable when combining heterogeneous sources.
  2. It constructs a reference separately for each feature using majority agreement, so no single source needs to be completely clean - this is really practical use case.
  3. Another strength is the model rather than merely detecting a difference between datasets, predicts which features are affected in each source, supporting targeted investigation.
  4. The trained model handles new datasets, varying numbers of sources, and different feature dimensions. Evaluation includes eleven unseen datasets and shift manipulations distinct from training. This is useful for generalization.
  5. The paper reports an average F1 of approximately 0.92 and seconds-level inference, outperforming the evaluated baselines, including reference-based methods given clean reference data.
Critical weaknesses
  1. The method assumes that each feature is normal in a strict majority of sources. If several sources share the same systematic error, their agreement could become a misleading reference. I can't think of a solution for this, but this should be addresses.
  2. Disagreement does not necessarily indicate corruption. Legitimate differences between populations or acquisition settings can also produce feature shifts. The method identifies deviations from consensus, but cannot by itself establish whether those deviations are errors requiring correction. This is something we see often in practice for protocol variations to human errors.
  3. The training uses artificially introduced shifts. Although the paper tests unseen manipulations, these experiments do not fully establish robustness to complex, naturally occurring errors involving multiple interacting features.
  4. Strong F1 performance does not establish that the output probabilities are calibrated or that false alarms are controlled. This matters when users must choose thresholds across many sources and features. So some form of calibration is needed.
Questions for authors
  1. How does performance change as the proportion of shifted sources approaches or exceeds half? Can the method detect when its majority-based reference is unreliable? Some sort of confidence/uncertainty is desired.
  2. How can users distinguish erroneous shifts from valid differences between populations or acquisition settings? Does the method risk flagging meaningful variation as corruption?
  3. Have the authors evaluated the method on naturally occurring shifts with independently verified ground truth, beyond artificially introduced perturbations?
  4. Are the predicted shift probabilities calibrated on unseen datasets? How should users select thresholds to control false positives across many source–feature pairs? Proper calibration is needed or at least discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.