acceptodds
Under review as a conference paper at ICLR 2027

The Lazy Adversary: Data-Efficient Universal Perturbations for Tabular Classifiers via Coresets

Abstract

Universal Adversarial Perturbations (UAPs) are single, input-agnostic perturbations that can induce misclassification in a frozen classifier. Gradient-based UAP methods, which frame the problem as loss maximization and solve it with the Fast Gradient Sign Method (FGSM) or Projected Gradient Descent (PGD) over the training set, have become the dominant approach due to their simplicity and effectiveness. However, these methods require aggregating gradients over the entire training set at every optimization epoch, making the per-epoch cost scale linearly with the dataset size. We observe that because the victim classifier is frozen, its penultimate feature embedding never changes. We propose UAPCore, which uses this frozen feature geometry as a practical proxy for gradient redundancy: before any gradient update, a coreset is selected in the feature space of the frozen victim model, and all subsequent UAP optimization is performed exclusively on that fixed subset. Whether this proxy preserves UAP quality is evaluated thoroughly with experiments across 44 tabular classification datasets from OpenML, which show that UAPCore with k-Center selection at only 10% of the training set recovers full-dataset fooling rates to within 2 percentage points on 72-91% of the datasets, while achieving a median wall-clock speedup of 8.5x under PGD. We further show that MLP-crafted coreset perturbations transfer to unseen tree-based and attention-based tabular architectures with fidelity comparable to full-data crafting.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial ML, Coresets, Tabular ML, UAPs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TANcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces UAPCore, a data-efficient framework for computing universal adversarial perturbations (UAPs) for tabular classifiers. The main observation is that, during UAP optimization, the victim classifier remains frozen, and therefore its penultimate-layer representation is stationary. The authors exploit this property by selecting a representative coreset once, before attack optimization, and computing all subsequent gradient updates only on this subset.

The proposed framework selects samples in the frozen feature space using k-Center, Facility Location, or random sampling, and then applies standard FGSM or PGD-based UAP optimization on the selected subset. The method is evaluated on 44 OpenML tabular classification datasets. Using only 10% of the training data, k-Center-based UAPCore recovers the full-data fooling rate within two percentage points on a large fraction of datasets while providing a median wall-clock speedup of approximately 8.5$\times$ for PGD. The authors additionally study different coreset ratios and perturbation budgets and evaluate transfer of MLP-crafted perturbations to XGBoost, TabNet, and FT-Transformer models.

The paper presents a simple and practically useful way to reduce the computational cost of existing gradient-based UAP methods rather than introducing a fundamentally new adversarial attack.

Critical strengths
  1. The paper identifies a simple but useful structural property of UAP optimization. Since the victim model is frozen, the representation used for sample selection does not evolve during optimization. This allows the coreset to be selected only once rather than repeatedly recomputed, making the approach considerably cheaper than standard coreset-based training procedures.

  2. The empirical evaluation is reasonably broad within the tabular setting. The authors evaluate 44 datasets, multiple perturbation budgets, FGSM and PGD, several coreset sizes, different selection strategies, and cross-architecture transfer. In particular, obtaining approximately 8.5$\times$ median PGD speedup with a 10% coreset while maintaining near-full-data fooling rates demonstrates that the approach can provide a meaningful practical benefit.

Critical weaknesses
  1. The central proxy underlying UAPCore is not directly validated. The method selects examples according to geometric coverage in the frozen penultimate feature space, with the intuition that this captures redundant optimization signals. However, the UAP update is determined by gradients, and the paper itself acknowledges that nearby feature representations do not necessarily imply similar gradients. A quantitative analysis relating feature-space similarity to gradient similarity or gradient alignment would substantially strengthen the methodological justification.

  2. The experimental comparison and scalability evidence are somewhat limited relative to the claims. The main baselines are full-data UAP optimization, random subset selection, k-Center, and Facility Location. Although related gradient-aware coreset approaches such as CRAIG and GradMatch are discussed, there is no empirical comparison with an adapted gradient-aware subset selection baseline. Moreover, all evaluated datasets contain only 500-2,500 examples. This is particularly relevant because exact k-Center has a quadratic selection component, and the authors themselves note that its computational advantage can eventually disappear for sufficiently large datasets. Consequently, the current experiments do not fully establish scalability in the larger-data regime that motivates the method.

Questions for authors
  1. Can the authors directly validate the main feature-space proxy? For example, across samples or local neighborhoods, how strongly does distance in the penultimate representation correlate with cosine similarity or alignment between the corresponding UAP gradients? A strong relationship would substantially strengthen the explanation for why a fixed geometric coreset preserves the optimization trajectory.

  2. Have the authors considered a gradient-aware subset selection baseline, for example an adaptation of GradMatch, CRAIG, or direct gradient clustering/matching? If such approaches are computationally impractical in this setting, it would be useful to quantify that cost. This comparison would help determine whether the benefit comes specifically from frozen feature geometry or more generally from selecting informative subsets.

  3. How does the end-to-end speedup behave on substantially larger datasets, for example tens or hundreds of thousands of samples? Since exact k-Center introduces an O(N2)O(N^2) component, evidence using the proposed approximate variant or larger-scale datasets would materially affect my confidence in the scalability motivation.

2 comments
AuthorsOpenReview ↗

Response to Reviewer TANc

We thank the reviewer for a careful and accurate summary, and for recognizing the one-shot selection property and the breadth of the evaluation. We address the three questions below.

Q1 / W1: Directly validating the feature-space proxy. We agree that this is the central question. As stated in Sec. 3.1, the paper does not assume that feature proximity implies gradient similarity. It treats feature geometry as a proxy and evaluates it by its end result, fooling-rate fidelity.

We have one supporting diagnostic, which we will add to the appendix. It measures the cosine similarity at epoch 0 between the mean gradient on the kk-Center coreset and the mean gradient on the full training set (PGD, ε=0.10\varepsilon=0.10, ρ=0.10\rho=0.10):

  • High alignment is the norm: 73% of datasets have a cosine above 0.5.
  • Within this group, Δ\DeltaFR concentrates near zero, and the datasets outside the 2-pp band are concentrated among the low-alignment ones.
  • The kk-Center coverage radius alone does not predict Δ\DeltaFR (Pearson r=0.023r=0.023, p=0.88p=0.88).

This indicates that gradient alignment is what mediates fidelity, and that the coreset selected from frozen feature geometry attains high alignment on most datasets.

Q2 / W2a: Gradient-aware baselines. The goal of UAPCore is to select the right samples once, which the frozen victim makes possible. Our Facility Location strategy (C2) is CRAIG's facility-location selection rule, applied in this one-shot setting. CRAIG and GradMatch re-select periodically because, in training, the model and therefore the gradients change. Each re-selection would cost a full-data gradient pass, which is exactly the per-epoch cost UAPCore is designed to remove. Iterative re-selection is therefore outside the setting we study. We will make the relation to CRAIG explicit in Sec. 3.2.

On whether the benefit comes from frozen feature geometry or from informative subsets in general, the existing comparisons separate the two:

  • Random vs. geometric selection under the same one-shot protocol isolates the contribution of geometry beyond data reduction. A single random draw can fall anywhere between the Best and Worst rows of Table 1. kk-Center gives one deterministic selection that avoids the Worst-seed tail (86.4% vs. 59.1% of datasets within 2 pp under PGD at ε=0.10\varepsilon=0.10).
  • kk-Center vs. Facility Location compares two geometric objectives. They reach similar fidelity, but at very different selection costs.
  • The epoch-0 alignment diagnostic (Q1) links the selected subsets to the full-data gradient.

Q3 / W2b: Scalability to tens or hundreds of thousands of samples. We agree this is the main open question, and we will state it more prominently as a limitation. Three points in the paper bear on it:

  1. Selection cost at our scale. Exact kk-Center selection is cheap at the benchmark sizes. Under PGD, random sampling, which has no selection cost, reaches a speedup of 8.9×, compared with 8.5× for kk-Center (Table 2).
  2. Large-NN behaviour. App. B characterizes the regime where this changes. It derives the selection crossover NselN_{\mathrm{sel}} and the break-even NbreakN_{\mathrm{break}}, beyond which exact kk-Center would erode the speedup.
  3. Linear-time alternative. Block-kk-Center (App. C) removes the quadratic term. It has O(NbD)O(NbD) selection cost and an NN-independent speedup plateau. At N≤2,500N\le2{,}500 it is already 4.18× faster at selection, with a mean Δ\DeltaFR of +0.003 relative to exact kk-Center. We will further highlight empirical validation at N≥105N\ge10^5 in our future work.

We thank the reviewer again for questions that will sharpen the paper.

I thank the authors for their detailed response.

The validation of the feature-space proxy indeed makes the contribution of the paper much clearer now, i would also suggest highlighting it on the main paper.

I'll read everything carefully with the revised updated version, and ill reconsider my assessment accordingly.

Reviewer_F8SKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

UAPCore speeds up universal adversarial perturbations for tabular classifiers. Because the victim model is frozen, its features never change. So it picks a small coreset once, using k-Center selection in the model’s feature space, and optimizes only on that subset. On 44 OpenML datasets, using 10% of the data came within 2 points of the full-data fooling rate on most datasets, with a median 8.5× speedup. The perturbations also transferred well to tree- and transformer-based models.

Critical strengths
  • The presentation is clear and easy to follow.
Critical weaknesses
  • Limited conceptual contribution. The attack framework and the coreset selection framework are both well established, and k-center and facility location selection are not new. The method is an incremental, direct combination of existing techniques, with no further optimization or insight.
  • Small-scale datasets. The datasets appear chosen to be small (500–2,500 instances, 5–70 features). Validating data efficiency calls for larger datasets.
  • Limited model scope. Only a small MLP is used, so it is unclear whether the findings hold for modern, larger architectures.
  • Missing baselines. The only comparison is against the full-training-set attack. There is no comparison to other coreset selection methods, or to existing data-efficient attacks from the vision domain. The paper claims those methods would not work but gives no empirical comparison.
  • Weak results. The coreset selection methods do not perform much better than random sampling.
  • Incomplete reporting. Only the fooling-rate gap is shown. Without the fooling rates themselves, it is unclear whether these gaps are large.
Questions for authors

How are categorical and free-text fields handled?

2 comments
AuthorsOpenReview ↗

Response to Reviewer F8SK

We thank the reviewer for the detailed review and constructive feedback.

W1: Limited conceptual contribution. We do not claim a new selection algorithm. kk-Center and facility location are standard approaches, so the effect we study is not confounded with a new selector. Our contributions are:

  1. The structural observation. In UAP optimization, unlike training, the victim is frozen. The selection representation is therefore stationary, and the coreset needs to be selected only once. Training coresets (CRAIG, GradMatch) must be reselected because the model changes.
  2. The first systematic study of data efficiency for UAPs on tabular data. It spans 44 datasets, 2 primitives, 3 budgets, a ρ\rho sweep, and transfer to 3 architectures.
  3. A cost model with explicit crossover and break-even points (App. B), and a linear-time Block-kk-Center that removes the quadratic selection term (App. C).

W2: Small-scale datasets. The 500≤N≤2,500500\le N\le2{,}500 range was an explicit filter (Sec. 4), chosen so full-data baselines could run for every configuration: 2 primitives × 3 budgets × 3 strategies × 30 random seeds × the ρ\rho sweep × 44 datasets. Within this range, speedup already rises with NN.

For larger NN, App. B gives the dataset sizes at which exact kk-Center's selection cost becomes significant (NselN_{\mathrm{sel}}) and at which UAPCore would stop being faster than full-UAP (NbreakN_{\mathrm{break}}). Block-kk-Center (App. C) removes the quadratic term responsible for both. We agree evaluation at N≥105N\ge10^5 is important and will list it as the first future-work item and in the Limitations.

W3: Only a small MLP. The paper evaluates coreset UAPs on XGBoost, TabNet, and FT-Transformer (Sec. 5.3, Tables 3–4), and they are as effective as full-data UAPs: At ε=0.10\varepsilon=0.10, mean Δ\DeltaFR is +0.003 / −0.003 / +0.000, respectively. Absolute fooling rates are 0.119 / 0.131 / 0.106, compared with 0.116 / 0.135 / 0.105 for the full-data UAP.

In these experiments, the perturbation is crafted on the MLP. UAPCore itself requires only a penultimate embedding, which FT-Transformer and TabNet both provide. Crafting white-box on these architectures is listed as future work.

W4: Missing baselines. (i) Other coreset methods. Our Facility Location strategy (C2) is CRAIG's selection rule (Mirzasoleiman et al., 2020): greedy maximization of F(C)=∑imax⁡j∈CsijF(\mathcal{C})=\sum_i\max_{j\in\mathcal{C}}s_{ij} with the same (1−1/e)(1-1/e) guarantee. CRAIG re-selects its subset every few epochs because the model parameters change during training, and GradMatch is likewise designed for training with periodic re-selection. In UAP generation the victim is frozen, so nothing requires re-selection, and UAPCore applies the facility-location rule once. Iterative re-selection schemes therefore lie outside the setting we study. Random selection (30 seeds, same one-shot protocol) is the data-agnostic reference. We will make the relation to CRAIG explicit in Sec. 3.2.

(ii) Vision data-efficient attacks. Data-free UAPs (Mopuri et al., 2017) maximize convolutional activations, and generative UAPs train a perturbation generator. Both rely on image-specific structure (Sec. 1). We will soften our wording in Sec. 1 accordingly.

W5: Coresets are not much better than random. The two random rows in Table 1 are not comparable to kk-Center as baselines. They bound what a single random draw can do:

  • Random-Best is an oracle. It is the best of 30 seeds, chosen in hindsight using the test fooling rate. Obtaining it requires all 30 runs, about 3.4× the cost of one full-UAP under PGD (Sec. 5).
  • Random-Worst shows the risk. Under PGD at ε=0.10\varepsilon=0.10, the worst seed falls to a mean Δ\DeltaFR of −0.031, with only 59.1% of datasets within 2 pp.

In practice, one gets a single draw and cannot know in advance where in this range it lands. kk-Center gives a single deterministic selection at a mean Δ\DeltaFR of −0.006 with 86.4% of datasets within 2 pp. It costs about the same as random selection, and it requires no seed search. Since UAPCore selects only once, this single-run reliability is the relevant benefit.

W6: Only gaps reported. Absolute fooling rates are reported in Sec. 5. Under PGD at ε\varepsilon = 0.05 / 0.10 / 0.20, the mean test fooling rate is:

  • 0.077 / 0.121 / 0.216 for full-UAP;
  • 0.076 / 0.115 / 0.184 for kk-Center;
  • 0.074 / 0.111 / 0.192 for Facility Location.

The mean clean error is 0.055, and transferred fooling rates are given in Sec. 5.3. We agree these should be easier to find and will move them into Table 1.

Q: Categorical and free-text fields. All features of the 44 benchmark datasets are numerical and standardized (Sec. 4); none contain categorical or free-text columns. Categorical handling is orthogonal to UAPCore because selection operates on the victim's frozen embedding of whatever encoding is used.

We hope these clarifications address the reviewer's concerns.

Thank you for the response. Some of my concerns remain:

  • The authors list evaluation on larger datasets and white-box crafting on other architectures as future work. I think both are essential for a comprehensive evaluation.

  • Coreset selection is a well-studied topic, and several other methods also select once and would apply even with a frozen victim, such as coverage-centric coreset selection (Zheng et al., “Coverage-centric coreset selection for high pruning rates,” ICLR 2023). I recommend the authors review this literature more thoroughly and include proper comparisons.

  • I still think the conceptual contribution is limited, since the work mainly applies standard selection methods to a new setting.

I will keep my current score.

Reviewer_48YzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UAPCore, a coreset-based framework for accelerating universal adversarial perturbation (UAP) generation on tabular classifiers. The method selects a small subset of training samples and performs UAP optimization using only the selected samples. The proposed approach is evaluated on 44 OpenML datasets, with experiments examining its attack effectiveness, computational efficiency, and cross-model transferability.

Critical strengths

Exploring UAP for tabular classifiers is relatively underexplored and has practical relevance.

Critical weaknesses
  1. Unlike images, tabular data often contain categorical, immutable, and interdependent features. However, the proposed method only imposes a simple L∞L_\infty constraint, which cannot guarantee the imperceptibility of adversarial samples.
  2. Using fewer samples degrades UAP attack performance. The paper empirically evaluates different coreset ratios, it lacks theoretical justification for how many samples are sufficient to maintain attack effectiveness.
  3. The method assumes that feature similarity implies gradient redundancy. However, similar feature representations do not necessarily yield similar input gradients; the latter is the determining factor of attack effectiveness.
  4. The experiments consider only FGSM and PGD. There are more algorithms to test, such as CAPGD and CAA [1].

[1] Constrained Adaptive Attack: Effective Adversarial Attack Against Deep Neural Networks for Tabular Data. NeurIPS 2024.

Questions for authors
  1. Is there any theoretical guidance for determining the minimum coreset size required to maintain attack effectiveness?
  2. Can the authors consider tabular attacks, such as CAPGD and CAA?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 48Yz

We thank the reviewer for the review and for noting that UAPs on tabular classifiers are underexplored and practically relevant. The core of UAPCore is simple: the victim is frozen throughout UAP optimization, so the selection representation never changes, and the coreset needs to be selected only once. We address each point below.

W1: Tabular constraints / imperceptibility. Our goal is to accelerate UAP computation under the standard gradient-based UAP threat model (Shafahi et al., 2020), which bounds a single universal perturbation in ℓ∞\ell_\infty. All features are standardized (Sec. 4), so our budgets ε∈{0.05,0.10,0.20}\varepsilon\in\{0.05, 0.10, 0.20\} amount to at most 0.2 standard deviations per feature, applied identically to every input. What counts as a feasible tabular input (categorical, immutable, or interdependent features) is domain-specific, and UAPCore is deliberately decoupled from it. The coreset is selected once, on the frozen features, before optimization begins. Any constraint-aware projection or repair can therefore be used in the attack phase without changing the selection step. We will make this explicit in the paper and list domain-constrained evaluation as future work.

W2 / Q1: Guidance on minimum coreset size. We do not provide a theoretical minimum. As stated in Sec. 3.2, the guarantees of our selectors concern the selection objective: the 2×2\times coverage bound for kk-Center and the (1−1/e)(1-1/e) bound for Facility Location. They do not bound the gradient error in Definition 1. We will state this explicitly as a limitation, and we list a principled bound on the required coreset size as future work.

Empirically, the ρ\rho ablation (Sec. 5.2, Fig. 4) provides practical guidance. The median Δ\DeltaFR is 0 for every ρ∈[0.05,0.30]\rho\in[0.05, 0.30]. Only at ρ=0.05\rho=0.05 does the lower quartile drop to −0.019, and the interquartile range collapses for ρ≥0.15\rho\ge0.15. ρ=0.10\rho=0.10 (8.5× speedup) is therefore a safe default, and ρ≥0.15\rho\ge0.15 tightens the tail further at a lower speedup (4.6× at ρ=0.20\rho=0.20).

W3: Feature similarity vs. gradient similarity. We agree, and the paper does not assume the two are equivalent. Sec. 3.1 states that, by Eq. (4), gradients depend on the local Jacobian, the label and the current perturbation, and that "we therefore do not assume that Euclidean proximity in the unperturbed feature space directly guarantees gradient similarity." We present frozen geometry as a proxy and evaluate it by its end result, fooling-rate fidelity.

We will add the following new results to the appendix. It measures the cosine similarity at epoch 0 between the mean gradient on the kk-Center coreset and the mean gradient on the full training set (PGD, ε=0.10\varepsilon=0.10, ρ=0.10\rho=0.10):

  • High alignment is the norm: 73% of datasets have a cosine above 0.5.
  • Within this group, Δ\DeltaFR concentrates near zero, and the datasets outside the 2-pp band are concentrated among the low-alignment ones.
  • Coverage radius alone does not predict Δ\DeltaFR (Pearson r=0.023r=0.023, p=0.88p=0.88).

Fidelity therefore follows gradient alignment, and the coreset selected from feature geometry attains high alignment on most datasets. A finer per-sample analysis relating feature distance to gradient similarity, and gradient alignment as an explicit selection objective, are listed as future work.

W4 / Q2: CAPGD and CAA. We thank the reviewer for the pointer. CAPGD and CAA are instance-wise constrained attacks. UAPCore is orthogonal to the choice of attack primitive: it only fixes, once, the samples over which the universal objective is averaged. A universal variant of CAPGD or CAA could therefore run on the coreset unchanged. Table 1 is consistent with this. FGSM and PGD give the same coreset fidelity: at ε=0.10\varepsilon=0.10, kk-Center is within 2 pp of full-UAP on 86.4% of datasets with mean Δ\DeltaFR −0.006 under both. We will mention this and list universal CAPGD/CAA as future work.

We hope these clarifications address the reviewer's concerns, and we would be glad to discuss further.

Thank you for the detailed response and the additional experiments. However, I still have some concerns regarding the theoretical justification and the general applicability of the proposed approach.

1. Feature Similarity vs. Gradient Similarity. The additional experiments do not sufficiently address this concern. A cosine similarity of only 0.5 indicates that the similarity between them is not high. The results show only an empirical observation on small dataset rather than sufficient justification.

2. Coreset Size Selection. The authors acknowledge that there is no theoretical guidance for determining the minimum coreset size. It remains unclear whether this ratio is suitable for datasets with different sizes and feature distributions.

The two main concerns remain insufficiently addressed. In particular, the additional gradient alignment experiment does not support the motivation of the proposed method. I therefore maintain my original rating of 2 (Weak Rejection).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.