acceptodds
Under review as a conference paper at ICLR 2027

RFWall: A Structural Diagnostic of Generative Data via the Augmentation Difficulty Index

Abstract

Generative data augmentation is widely used to reduce labeling cost, and RF fingerprint localization, where every label requires a site survey, has become a prominent application. Reported gains are hard to compare: evaluations rarely share data or splits, focus on dense Wi-Fi, and often fit the generator on data withheld from the low-label condition. We introduce RFWall, a benchmark of 31 augmentation methods on 20 partitions spanning Wi-Fi, cellular, and non-RF controls, under a single leakage-controlled protocol. Fitting generators on the full labeled pools inflates the measured gain at a label budget by . Under leakage control, augmentation recovers of the available accuracy headroom on Wi-Fi and none on cellular fingerprints, and its positive effects under a nearest-neighbor localizer reverse under an MLP. Grounded by the Data Processing Inequality (DPI) on information bounds, we conduct controlled sparsification experiments on high-density physical testbeds and identify representational feature vocabulary as the causal driver of this gap: shrinking the vocabulary from 520 columns to 17 elevates exact label collision from 0.004 to 0.853, collapsing manifold structure and reducing augmentation gains to zero. We distill these mechanisms into the Augmentation Difficulty Index (ADI), a data-inherent screen combining label collision, relational manifold sparsity, and noise level, which orders held-out dataset utility at Spearman under leave-one-dataset-out validation. Beyond the RF domain, spatial voxel experiments on non-RF controls show identical label collision dynamics. Controlled noise injection confirms ADI's theoretical bounds, with structure-aware generators outperforming baseline models across all test cells. Our findings demonstrate that simple non-parametric interpolation frequently matches or exceeds deep generative models, and that standard distributional fidelity metrics fail to predict downstream utility.

Primary area:
datasets and benchmarks
Keywords:
Data augmentation, synthetic data, indoor localization, RF fingerprinting, representation sparsity, evaluation leakage, generative models, dataset difficulty

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_1moFOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RFWALL, a leakage-controlled benchmark for evaluating data augmentation in RF fingerprint localization. The benchmark covers 31 augmentation methods and 20 partitions spanning Wi-Fi, cellular, and non-RF datasets under a unified protocol. The paper reports that augmentation benefits are strongly affected by evaluation leakage, representation structure, and downstream models, and that simple interpolation methods often match or outperform deep generative approaches. Based on controlled sparsification experiments, the authors further propose the Augmentation Difficulty Index (ADI), which combines label collision, relational manifold sparsity, and noise level to predict whether augmentation is likely to improve downstream performance.

Critical strengths
  1. The benchmark is unusually comprehensive and carefully controlled. RFWALL evaluates a broad range of augmentation methods under fixed splits and explicitly prevents generators from accessing data outside the downstream training subset. The leakage experiment is particularly valuable, showing that the apparent gain can be dramatically inflated when generators are trained using information unavailable under the intended low-label condition.

  2. The paper goes beyond method ranking and investigates why augmentation succeeds or fails. The controlled sparsification experiments and analyses of label collision, feature vocabulary, and manifold structure provide a useful structural perspective. The finding that simple methods such as SMOTE and mixup can outperform substantially more complex generators is also practically relevant.

Critical weaknesses
  1. The evidence is not yet strong enough to support ADI as a general diagnostic of augmentation difficulty. The final ADI is fitted and evaluated on only 33 dataset/sweep rows from 12 source datasets. Although its leave-one-dataset-out Spearman correlation is reasonably high (ρ=0.739\rho=0.739), its MAE is essentially no better than a constant predictor, and its sign accuracy is only 0.70. More importantly, high-collision regimes and dense low-rank regimes require extrapolation because they are sparsely represented in the training data. The authors also acknowledge that the precursor index failed on the pre-registered ModelNet10 test and that the adopted ADI is intended only as a directional screen. These limitations make the current evidence insufficient for the relatively broad framing of ADI as a data-inherent diagnostic across domains.

  2. The central structural conclusions are strongly dependent on the downstream model. The positive augmentation effect on Wi-Fi is mainly observed with kNN, whereas the mean effect is negative under the MLP in every structural regime. Additional experiments with random forests and logistic regression also show substantial instability, with only Tampere consistently benefiting across all four classifiers. This raises an important question about whether the identified “sparsity wall” primarily characterizes the data representation itself or an interaction between augmentation and particular decision rules. Since the paper aims to derive a data-inherent predictor independent of downstream models, this dependence weakens one of its central claims.

Questions for authors
  1. How should ADI be interpreted as a model-independent diagnostic given the strong dependence of augmentation utility on the downstream classifier? If the same dataset can show positive augmentation effects under kNN and negative effects under MLP, what exactly is the target quantity that ADI is intended to predict independently of the downstream model?

  2. Can the authors provide stronger evidence that ADI generalizes beyond the current set of highly related dataset variants? In particular, evaluation on additional genuinely independent datasets or modalities would help determine whether the reported leave-one-dataset-out correlation reflects a transferable structural relationship rather than the particular composition of the current benchmark.

Reviewer_vCytOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RFWALL, a benchmark for evaluating generative data augmentation in RF fingerprint localization. It evaluates 31 augmentation methods across 20 data partitions under a unified leakage-controlled protocol, revealing that augmentation benefits depend strongly on feature representation and downstream models.

Critical strengths
  • The paper addresses an important and timely question: under what conditions does generative data augmentation actually improve downstream performance?

  • The benchmark covers 31 augmentation methods, multiple RF and non-RF datasets, different label budgets, and downstream models.

  • The observation that simple interpolation methods can outperform deep generative models, and that data augmentation may harm MLP-based localization, is interesting.

Critical weaknesses
  • The Data Processing Inequality (DPI) does not directly explain the proposed sparsity wall or establish the effectiveness of ADI. ADI offers only marginal improvement over a constant baseline in prediction error, raising questions about its practical value.

  • Augmentation improves kNN performance in some settings but consistently degrades MLP performance. This challenges the claim that augmentation difficulty can be determined solely by dataset properties.

  • The conclusions are primarily based on RF fingerprint data and generators trained from scratch, making their applicability to broader generative learning settings unclear.

  • The quality of presentation and figures should be improved.

Questions for authors
  1. Can the authors provide stronger theoretical justification connecting DPI, feature sparsity, and augmentation effectiveness?
  2. How useful is ADI in practice, given its limited improvement over simple baselines?
  3. Why is ADI defined as a data-inherent metric when augmentation performance strongly depends on the downstream model?
  4. Can the authors validate their findings on more diverse datasets and modern pretrained generative models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.