- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates the capabilities of a robust classifier to generate images by inverting the classifier, i.e., by examining the gradient of the loss with respect to the input. The paper discusses the Jacobian instead of the gradient when differentiating the network across all classes. The paper claims that the matrix is an implicit denoiser that suppresses noise in orthogonal directions. The article reports a proposition and a theorem that look almost obvious to a researcher who is an expert in this field. The theorem is of interest, but it is also almost a consequence of the fact that is of low rank. From them, at least in my opinion, it is hard to derive new information. Perhaps the most interesting part is Eqs 6-9, where a proxy is used to compute implicitly, but the approach still receives the gradient from the network, as in all previous literature. Also, that part works by linearizing the network. In terms of experimental results, I do not see anything new there: the experiments confirm the hypothesis of the paper, but do not add much to the literature. Their report numbers related to the implicit denoiser hypothesis and the spectral statistics, and how these models can generate images from random noise (known). The paper kind of tries to do it at multiple scale (CNN, Transofmers), different norms, different layers, but it does not offer a full ablation on these and a full study.
Critical strengths
-
The only interesting part to me is Eq. 6-9, where the is linked to take the gradient from the network providing an input image and its perturbed versions using stop gradient. That part (modulo the linearization) is cool, and I learned something.
-
Theorem 1 is minimally interesting, but it just derives from applying the low-rank structure of to a distribution as input ϵ ∼N(0,σ2IP). I would not count this as strong novelty.
Critical weaknesses
- The community has published a lot of papers on the mystery of robust classifiers acting as generators. The paper reports most of them, with some missing. see below [A, B, C, D]. The main problem is that by reading the paper, I was not able to get new knowledge on the topic. I believe this topic is highly saturated, so gaining new knowledge or achieving a breakthrough is very difficult. The reason is that everyone works either at the output level (logits) or the input level (Jacobian input, etc.), but no one dares to understand what happens to the nonlinear part. Bottom line: contributions are a bit too weak; more or less, it sounds the same, but with different words.
- On the claim that is an implicit denoiser, this is true even for a matrix where are the principal components of a dataset with decay spectrum (natural images). If you pass data into , this will reconstruct images by just passing the low-frequency part. So if is low-rank (prior work), I am not surprised by the finding that it acts as a denoiser.
- Some claims are not correct. for example,
sentences like this
structure without explicit denoising training. [about adv. training]
They are not entirely true. Diffusion Models (DM) inject noise and then propagate noise in the loss (equivariance) Robust Classifier also injects noise (which is a function of the network) and then applies invariance (due to classification needs) So they are doing some denoising training; the problem is that, given that this was approached by a classification/discriminative model, people did not realize it. Every time you perturb the data somehow, you go back and look at the input density (i.e., you do generative modeling).
Theorem 1 does not add much; the PSD matrix J.T@J is already low-rank and thus acts as a denoiser. The proof and Theorem 1 only make this considering the input to be Gaussian noise. does not add up much. In other words, Theorem 1 puts the projection/unprojection of a generic point input for a distribution. It is known that components ortogonal to the span of J do not survive after J.T@J.
Given this spectral structure, J⊤J acts as a denoiser.
Regarding your claim. The fact that J⊤J acts as a denoiser is known if as low rank-structure and decay spectra.
The probing objective reduces the linear network to a linear operator, which is oversimplistic. i.e., gradient over the input is the Gram operator only if the network is linearized! This is an important distinction.
Eq. (6) is nothing new; it is essentially the same as optimizing the input to increase the logit of the desired class k. In the case of increasing the logit of the desired class k, the reference embedding will be the k-vector in the classification layer. In equation (6), it is replaced with the embedding of natural data. So, even in this part, I do not see new things.
- On the paper quality part:
- title should contain a ? (question mark)
- A lot of figures are there, but I am already on page 6, NO figure is cited in the text! The first figure is cited on page 7. Figure 2 is not cited.
The article is missing several related works:
- [A, B] similar generation approach reported in this paper.
- Connection to EBM [C]
- [D] with other strong priors, we can generate from discriminative models even with non-robust encoders.
I strongly believe that if you send this paper to ICLR, you cannot miss papers like [A] and [B]
[A ] Towards understanding the generative capability of adversarially robust classifiers. ICCV 2021
[B] Shedding More Light on Robust Classifiers Under the Lens of Energy-Based Models. European Conference on Computer Vision (ECCV), 2024
[C] Understanding Adversarial Training with Energy-based Models. https://arxiv.org/pdf/2505.22486
[D] Implicit Inversion turns CLIP into a Decoder. ICLR 2026.
Questions for authors
Theorem 1 takes spectral concentration as a premise. We verify empirically that it holds in adversarially trained networks (Section 5);
Did prior work already prove this empirically?
matrix multiplication requires O(P^2*K)
Is this really a bottleneck? Can you approximate somehow Jk.T@Jk for one class, this is O(P^2) Then do this in parallel, and then sum across all classes? Maybe in CIFAR10 this could work.
I do not see why I should buy that L2 is better than L_inf just by looking at one sample in Fig. 3B