acceptodds
Withdrawn from ICLR 2027

Interpreting Samplewise Function Perturbations: Signal, Noise, and Nonlinear Structure

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Samplewise adversarial perturbations have been motivated as stronger regularization, but a larger worst-case penalty does not specify how the learned predictor changes. We study exact samplewise logistic estimation in an RKHS with a unit-diagonal kernel and fixed positive ridge, also realized by per-example perturbations of normalized fixed-feature heads. A common operator governs weak-signal population response and balanced-label limiting covariance. Increasing the radius raises the gain of every observable signal mode and inflates the limiting covariance, although the balanced population minimizer remains zero. These leading responses are not specific to the perturbed objective: ordinary logistic regression with an analytically chosen ridge and output scale matches both exactly. The agreement leaves nonlinear structure undetermined. A general cubic interaction operator identifies the first mechanism-specific difference after matching, with a fifth-order remainder. In a periodic weak-signal model, its third-harmonic projection has opposite local responses: samplewise perturbation attenuates the harmonic while its matched control amplifies it. This attenuation coexists with norm growth, and the Bayes score itself contains the harmonic. Separate covariance-error bounds quantify nonlinear fitting and averaged random-design effects. Independent kernel and frozen ResNet/ViT fits test the predictions, while ordinary logistic controls reproduce the fixed-setting task gains. Thus, samplewise perturbation cannot be summarized as uniform shrinkage, and component attenuation alone does not establish predictive benefit.

Primary area:
learning theory
Keywords:
samplewise perturbations, RKHS, logistic regression, asymptotic covariance, fixed-feature models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_J61rOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how samplewise adversarial perturbations affect signal, sampling variability, and nonlinear structure in logistic regression, showing that they amplify both signal and noise, while their distinctive effects emerge at the cubic order.

Critical strengths

The theoretical results are correct.

Critical weaknesses

The paper presents several technically interesting results, but the overall narrative is difficult to follow. The main insights are not communicated effectively, and the connections between the theoretical findings and their statistical significance remain insufficiently developed. As a result, it is difficult to identify the paper's central takeaway and appreciate the broader implications of its contributions.

Questions for authors

What is the practical significance of the cubic-order separation?

Does samplewise perturbation provide any advantage that cannot be recovered by ordinary regularization and score scaling?

Reviewer_C4s8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper attempts to understand samplewise weight perturbations, in which each example has its own worst-case perturbation, but the intended benefit and overall objective remain unclear, unlike in the cited works on SAM and δ\delta-SAM. These two papers had a clear objective of improving generalization, whereas the manuscript under review does not. The authors propose an objective in RKHS coordinates, where RKHS, I believe, stands for reproducing kernel Hilbert space, although they never define this abbreviation. They find that matching leading signal and noise responses does not determine how a learning rule changes nonlinear structure or whether prediction improves. The experiments on CIFAR-10, introduced rather abruptly, seem to confirm their statement.

I am aware that my summary so far is not entirely coherent. However, I wrote it based on what I could understand from the manuscript, which, in my view, reflects the poor quality of the original manuscript's presentation. For this reason, I would assign a low confidence score to my review, although I remain firmly convinced that this paper should be rejected.

Critical strengths
  1. The authors attempt to provide a detailed theoretical characterization of samplewise perturbations from multiple perspectives, including signal response, sampling variability, and nonlinear structure. The comparison with analytically matched ordinary logistic regression is also a potentially useful component of the investigation.
Critical weaknesses
  1. Unclear research motivation. Although the authors introduce their work through adversarial training, SAM, and δ\delta-SAM, they fail to establish what specific problem their study aims to resolve. Unlike the cited works, which clearly target improved generalization, the motivation and significance of the present investigation remain unclear.

  2. Insufficient explanation of basic terminology. The authors never define RKHS (reproducing kernel Hilbert space), despite extensively relying on this concept. Introducing fundamental terminology and abbreviations is a basic requirement of scientific writing.

  3. Poorly justified theoretical focus. The authors investigate signal response, sampling noise, and nonlinear structure without sufficiently explaining why these particular properties are important or what meaningful insights they provide into learning behavior.

  4. Inadequate problem formulation and explanation. Section 3 introduces a mathematically dense formulation without sufficient conceptual explanation of its assumptions and relevance to the motivating problem. The subsequent theoretical sections further complicate the presentation without clearly connecting their results to a well-motivated research objective.

  5. Poor organization and excessive reliance on appendices. A substantial portion of the theoretical derivations and experimental results is deferred to the appendices, bringing the manuscript to 54 pages. This organization imposes an unnecessary burden on readers trying to understand and assess the paper's contributions.

  6. Unclear significance of the conclusions. The authors conclude that attenuating nonlinear components does not necessarily improve predictive performance. However, they fail to establish why this finding is consequential for understanding or improving generalization. The CIFAR-10 experiments further suggest that ordinary logistic regression controls can reproduce the observed task-level gains, leaving the practical significance of the theoretical contributions unclear.

Questions for authors

See weaknesses.

However, my judgement is unlikely to change no matter how authors respond.

Reviewer_DQMJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper analyzes samplewise function perturbation using logistic estimation in an RKHS. The analysis show that the adversarial samplewise function perturbation increases the signal and noise. It also change the nonlinear structure of the prediction. By the experiments, the authors confirm the theoretical results.

Critical strengths

By using logistic estimation in an RKHS, the authors shows the exact margin reduction of the samplewise function perturbation.

It is interesting that adversarial perturbation increases the signal, and noise. The analysis helps to understand the behavior of the adversarial samplewise function perturbation.

Critical weaknesses

The presentation can be improved. Without describing the motivation of worst-case perturbations, several methods have been explained. There is no explanation of variables of the equation in the first page, e.g.,ell, w, nu, n.

Some analysis in Section 5 relies on q(x)=cos(kx). It is unclear how it can be generalized in other cases.

It is unclear how much it is important to analyze the adversarial samplewise function perturbation.

Questions for authors

See the weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.