acceptodds
Under review as a conference paper at ICLR 2027

TLM-SVDD: Dual-Driven Deep Large-Margin -SVDD with Vision Transformers for Anomaly Detection

Abstract

Deep extensions of the Support Vector Data Description (SVDD) paradigm can jointly learn representations and decision boundaries for anomaly detection. Yet, they typically rely on simplified objectives or use a surrogate objective that does not coincide with the one used to fit the boundary. In this work, we introduce Transformer-based Large-Margin SVDD (TLM-SVDD), a framework that updates the encoder using the exact dual boundary problem to maximize the margin between normal and anomalous samples. By applying Danskin's theorem, we show that at the optimum of the convex dual function, the encoder's hypergradient can be directly derived from a kernel quadratic term evaluated at the optimal dual variables. As such, the encoder update requires no recovered decision thresholds, surrogate losses, or differentiation through a Frank-Wolfe-based solver. Furthermore, we show that this objective is bounded and naturally depends on support vectors. For batch-independent representation learning, we make use of a vision transformer and conduct a controlled ablation study to separately analyze the merits of the backbone architecture for anomaly detection from those of our optimization approach. Through extensive experiments across anomaly detection and long-tailed recognition benchmarks, the proposed TLM-SVDD approach is shown to improve upon convolutional and approximate primal-space predecessors, outperforming state-of-the-art methods in the majority of evaluation settings.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
anomaly detection, SVDD, large-margin learning, vision transformers, alternating dual optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_tnVjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TLM-SVDD, an extension of DLM-SVDD that replaces its primal surrogate encoder loss with a dual-based objective. Using Danskin's theorem, the authors show that the gradient of this objective matches that of the optimal primal value function without differentiating through the Frank-Wolfe solver. They further provide a support-vector contrast interpretation, establish boundedness, and analyze convergence under idealized conditions. The method also replaces the CNN backbone with a pretrained ViT-B/16. Experiments are conducted on Fashion-MNIST, CIFAR-10, CIFAR-100, CIFAR-10-LT, CIFAR-100-LT, and ImageNet-LT, using one-vs-rest anomaly detection protocols. TLM-SVDD reports strong performance compared with existing methods, and ablation studies examine the effects of backbone architecture, encoder optimization, anomaly exposure, and fine-tuning.

Critical strengths
  1. Principled coupling of representation learning and boundary fitting. The proposed encoder update provides a mathematically motivated alternative to the primal surrogate used in DLM-SVDD. Under the stated assumptions, Proposition 1 derives the encoder gradient of the optimal primal value function without differentiating through the Frank-Wolfe solver. Proposition 2 provides an interpretable support-vector-weighted contrast formulation, and the objective is shown to be bounded. This formulation connects boundary fitting and representation learning through a common primal-dual optimal-value relationship, addressing an important limitation of the predecessor.

  2. Strong reported performance and efforts to disentangle architectural and optimization effects. The proposed framework reports strong performance across standard anomaly detection and long-tailed benchmarks. The authors also make efforts to distinguish the effects of backbone architecture and encoder optimization. Table 3 compares fixed and fine-tuned representations across ResNet-50 and ViT-B/16 backbones, while Table 4 provides a matched ViT-based primal-versus-dual comparison with multiple random seeds and statistical testing. Furthermore, Table 5 reports cases where fine-tuning decreases performance, providing useful information about the limitations of the proposed method.

Critical weaknesses
  1. Insufficient isolation and validation of the proposed dual encoder update. The main contribution is replacing DLM-SVDD’s primal surrogate with a dual-based encoder objective, but evidence isolating its effect from the backbone change remains limited. Table 3 compares ResNet-50 + primal against ViT-B/16 + dual, so it does not support the claimed consistent superiority of the dual update across three datasets independently of architecture. Table 4 provides a matched ViT-based comparison, but only on CIFAR-10 with 10% anomaly exposure. Moreover, at 10% exposure, the fixed ViT already achieves AUROCs of 0.995 and 0.989 on CIFAR-10 and CIFAR-100, respectively, exceeding all baseline results in Table 1 on both datasets. Dual fine-tuning adds 0.002 and 0.003, respectively, although the gain is larger on Fashion-MNIST (0.944 to 0.989). The primal ViT result in Table 4 (0.982 on CIFAR-10) is below the fixed ViT result in Table 3, suggesting that part of the dual update’s advantage may reflect avoiding degradation of pretrained features. This interpretation requires verification under identical settings. Table 5 also shows that dual fine-tuning reduces performance on CIFAR-100-LT at all three imbalance ratios, despite improvements on CIFAR-10-LT and ImageNet-LT. Broader comparisons of fixed, primal-updated, and dual-updated models using the same backbone are therefore needed to establish the update rule’s contribution across datasets and long-tailed settings.

  2. Unclear comparability of the long-tailed evaluation and limited validation of the motivating setting. Table 2 compares conventional multi-class long-tailed classifiers with one-vs-rest SVDD-based detectors. However, the paper does not clearly explain how balanced accuracy is computed and aggregated for the SVDD methods or whether all methods are evaluated under equivalent prediction protocols. If the SVDD results represent averages of binary one-vs-rest balanced accuracies, they would not be directly comparable to standard multi-class balanced accuracy. This ambiguity makes it difficult to assess the reported state-of-the-art improvements in long-tailed recognition. Additionally, the introduction motivates anomaly detection under test-time anomalies that may differ from those observed during training. However, the experiments sample labeled training anomalies from the same set of remaining classes used to define test anomalies. Therefore, generalization to previously unseen anomaly classes is not explicitly evaluated.

Questions for authors
  1. [Additional experiment] Controlled evaluation of the dual update. Could the authors extend the matched ViT-B/16 comparison in Table 4 to other datasets, particularly CIFAR-100 and representative long-tailed settings, including fixed, primal-updated, and dual-updated ViT models under identical settings? Results with ResNet-50 using the dual update would further separate the contribution of the update rule from that of the backbone. This would help determine whether the proposed dual update consistently improves representation learning rather than primarily avoiding degradation caused by the primal surrogate.

  2. [Clarification] Long-tailed evaluation protocol. How exactly is balanced accuracy computed for the SVDD-based methods in Table 2? Are the results obtained by averaging binary balanced accuracy across one-vs-rest tasks, or are the scores combined into multi-class predictions? How are these results made directly comparable to conventional multi-class long-tailed classification baselines?

  3. [Additional experiment] Unseen anomaly classes. The introduction states that labeled anomalies are typically not fully representative of those encountered at test time, and that a detector should avoid becoming a closed-set classifier that only separates the anomalies observed during training. Could the authors evaluate a setting where labeled training anomalies come from a subset of classes and test anomalies from held-out classes? This would directly test the motivation of the paper, and positive results in this setting would provide stronger evidence for generalization to unseen anomaly classes.

3 comments

As there was no further author discussion, I will maintain my original rating based on the submitted response and review.

Thank you for the careful review, and sorry that our response comes this late. The experiments you asked for took most of the discussion period. The revised paper with these changes will be uploaded shortly.

Q1. Isolating the dual update. We now compare three versions of the same ViT-B/16: fixed (pretrained encoder, only the boundary is fitted), primal (the DLM-SVDD update), and dual (our update). We do this on Fashion-MNIST, CIFAR-10, and CIFAR-100 at 10%, 50%, and 75% anomaly exposure. All three use the same initialization, data splits, anomaly samples, training budget, hyperparameters, and seeds. Test AUROC at 10% exposure:

FixedPrimalDual
Fashion-MNIST0.982680.985280.98854
CIFAR-100.995340.996560.99693
CIFAR-1000.988690.990720.99162
  • Does the dual update only avoid damage done by the primal one? No. You noted that the primal ViT in the submitted Table 4 (0.982) was below the fixed ViT. We have updated these values based on the full matched experiments, and with them the primal update is above the fixed encoder in all nine settings, and the dual update is above the primal one in all nine. The table has been updated with these values and more details.
  • Effect size. We agree that the fixed ViT is already strong. Because AUROC is close to 1 here, we also report metrics that separate the methods better. FPR95 is the fraction of anomalies still accepted when 95% of normal samples are accepted (lower is better). Balanced accuracy is the average of the true-positive rate (TPR, normal samples accepted) and the true-negative rate (TNR, anomalies rejected). AUPR-In is the area under the precision–recall curve with normal samples as the positive class. The dual update is best in all nine settings for AUROC, FPR95, and balanced accuracy, and in 7 of 9 for AUPR-In. On CIFAR-10 it adds only 0.04 AUROC points over the primal update, but it is better in all five seeds (paired t-test over seeds: t(4)=6.33t(4)=6.33, p=0.003p=0.003 for AUROC; t(4)=−12.8t(4)=-12.8, p=2×10−4p=2\times10^{-4} for FPR95), and FPR95 drops by 19.2% compared with the primal update and by 43.6% compared with the fixed encoder. On CIFAR-100 the dual update has a higher AUROC than the primal one in 69 of the 100 one-vs-rest tasks (Wilcoxon signed-rank test, p=4×10−6p=4\times10^{-6}).
  • Fashion-MNIST. Based on the full experiments, with the single configuration used for all results, the fixed ViT obtains 0.983 at 10% and 0.984 at 50% and 75% (submitted: 0.944). So training the encoder with the dual update adds 0.59 AUROC points there, and 0.33 of them come from the dual update itself.
  • Long-tailed settings and ResNet-50. We could not finish the matched long-tailed runs or a ResNet-50 run with the dual update in time. The drop on CIFAR-100-LT after fine-tuning is stated in the paper, and we do not claim that training the encoder always helps. A dual update on ResNet-50 would need frozen batch-norm statistics (the features that fit the boundary must be the ones that are differentiated), which changes two things at once, so we compare the update rules on the same ViT instead.

Q2. Long-tailed protocol. For the SVDD-based methods, we compute balanced accuracy separately for each one-vs-rest task, 12(TPRc+TNRc)\frac{1}{2}(\mathrm{TPR}_c+\mathrm{TNR}_c) at the learned radius, and then average over the CC classes. The scores are not combined into one multi-class prediction. The revised paper defines both protocols in Sec. 4.1, separates the two groups in Table 2, and limits the claim to the one-vs-rest SVDD methods. Among those, TLM-SVDD improves over DLM-SVDD by 4.9 to 8.9 points on the long-tailed CIFAR sets and by 6.1 points on ImageNet-LT.

Q3. Unseen anomaly classes. For each CIFAR-10 normal class, we split the other nine classes into four seen and five held-out classes. Training, validation, and model selection use only the seen classes; the held-out classes appear only at test time. Five seeds (mean ± SD):

AUROCFPR95Balanced acc.
Fixed0.984 ± 0.0040.073 ± 0.0170.828 ± 0.023
Primal0.985 ± 0.0050.061 ± 0.0190.858 ± 0.031
Dual0.987 ± 0.0040.046 ± 0.0150.862 ± 0.029

AUROC is about one point lower than in the standard setting, but the unseen classes are still rejected well (FPR95 of 4.6%). The order fixed < primal < dual holds, and FPR95 is lower for the dual than for the primal update in every one of the five seeds (24.8% lower on average). The AUROC differences are within one standard deviation, so we do not claim an AUROC gain here. This evaluation is added to the paper as Appendix A.4, next to published CIFAR-10 results of the methods we cite.

Thank you again for these helpful questions.

Thank you for following up, and apologies for the silence. We understood that the discussion period ends on November 18 (AoE), so we thought we still had time, and we were using it to run the experiments you requested to further strengthen the paper: the matched fixed / primal / dual comparison on the same ViT-B/16 across all three datasets and exposure ratios, and the evaluation on held-out anomaly classes. Our full response is now posted under your review, and the revised paper will be uploaded as soon as possible. We would be grateful if you could take them into account in your final assessment.

Reviewer_9xTyOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Transformer-based Large-Margin SVDD, an anomaly detection and one-class classification framework that couples Vision Transformers (ViT) with the exact dual boundary problem of large-margin lpl_p-SVDD. By applying Danskin’s theorem to the convex dual objective, the authors derive the hypergradient for the encoder directly from the kernel quadratic term evaluated at the optimal dual variables α∗\alpha^*, bypassing surrogate loss functions, recovered threshold heuristics, and differentiation through Frank-Wolfe optimization steps. The authors also argue that ViTs provide batch-independent representations via layer normalization, mitigating the train/eval discrepancy observed in batch-normalized CNNs.

Critical strengths
  1. Using Danskin’s theorem to link the representation update directly to the exact optimal value of the dual boundary problem is theoretically well-motivated, eliminating the heuristic surrogate losses used in prior work like DLM-SVDD.
  2. The resulting encoder gradient naturally depends only on the active support vectors (SNS_N and SAS_A), providing an intuitive contrastive pull-push mechanism grounded in margin optimization.
  3. The manuscript is clearly written, and the transitions from primal lpl_p-SVDD to the dual formulation and Danskin envelope gradient are logically presented.
Critical weaknesses
  1. Proposition 1 relies on Danskin’s theorem, which requires obtaining the exact global minimizer of the dual objective Fω(α)F_\omega(\alpha). In practice, the inner boundary subproblem is solved iteratively via the Frank-Wolfe (FW) algorithm to a finite duality gap ϵFW\epsilon_{\text{FW}}. When evaluated at an approximate solution α^\hat{\alpha}, the outer gradient ∇ωFω(α^)\nabla_\omega F_\omega(\hat{\alpha}) becomes a biased hypergradient. The paper provides no theoretical error bounds on ∥∇ωFω(α^)−∇ωF∗(ω)∥\Vert{}\nabla_\omega F_\omega(\hat{\alpha}) - \nabla_\omega F^*(\omega)\Vert{} or its accumulation across alternating epochs.
  2. In Section 3.2, the authors claim that the optimal primal value sequence is nonincreasing and convergent, assuming P∗P^* has an LL-Lipschitz gradient and the step size satisfies η≤1/L\eta \le 1/L. P∗(ω)P^*(\omega) depends on the deep Transformer parameters ω\omega through multi-head self-attention, layer normalization, and a Gaussian RBF kernel. Assuming that such a complex, non-convex composite value function exhibits global or local LL-Lipschitz smoothness without formal justification or proof is unrealistic. The paper does not prove convergence to a first-order stationary point of the bilevel problem under finite FW stopping criteria.
  3. An inspection of Table 3 reveals that the fixed, un-finetuned ViT-B/16 backbone alone achieves an AUROC of 0.996 on CIFAR-10 and 0.991 on CIFAR-100. The fixed ViT-B/16 baseline already outperforms almost all compared methods in Table 1, including the previous SOTA (DLM-SVDD at 0.984 on CIFAR-10 and 0.972 on CIFAR-100). This demonstrates that the high performance in Table 1 is primarily driven by the representation quality of pre-trained ViT-B/16, rather than the dual hypergradient learning formulation.
  4. The ViT-B/16 backbone was pre-trained in a supervised manner on ImageNet-1k. Since ImageNet-1k features broad semantic overlap with CIFAR-10 and CIFAR-100 classes, the representation space has already separated these concepts prior to SVDD training. This makes distance-based discrimination trivial and compromises the validity of the anomaly detection evaluation.
  5. In Section 4.1, the Gaussian kernel bandwidth γ\gamma is computed once from pre-trained features using the median pairwise squared distance and held constant throughout training. In Section 4.4, the authors admit that dynamically re-estimating γ\gamma at each epoch destabilizes optimization. However, as the encoder parameters ω\omega update, the pairwise distances in feature space inevitably shift. Freezing γ\gamma based on pre-trained features creates an inconsistency between the metric scale of the RBF kernel and the evolving representation space, indicating sensitivity to hyperparameters.
Questions for authors
  1. How does the termination threshold (duality gap ϵFW\epsilon_{\text{FW}}) of the Frank-Wolfe solver affect the gradient bias ∥∇ωFω(α^)−∇ωF∗(ω)∥\Vert{}\nabla_\omega F_\omega(\hat{\alpha}) - \nabla_\omega F^*(\omega)\Vert{}? Can you provide formal convergence guarantees under inexact inner solutions?
  2. What is the root cause of the fine-tuning degradation observed on CIFAR-100-LT in Table 5 (e.g., a 3.0% drop at ρ=10\rho=10)? Does the dual loss overfit to the head classes or destabilize tail-class representations?
  3. Could you provide benchmark comparisons on CIFAR-100 and ImageNet-LT where all competing methods (e.g., DLM-SVDD, Deep-SAD, PANDA) utilize the identical pre-trained ViT-B/16 backbone?
  4. What are the actual wall-clock training times per epoch and peak GPU memory requirements for TLM-SVDD compared to DLM-SVDD on ImageNet-LT?
  5. Do you have preliminary experimental results on established industrial anomaly detection datasets such as MVTec AD or VisA?
6 comments

(Part 1/5)

We thank the reviewer for the careful and technically precise review. It identified real gaps, and addressing them has made the paper clearly stronger. In summary:

  • The two soundness concerns (W1, W2) are now answered with proofs (Propositions 3–5) and with measurements, and we state their limits openly.
  • On the empirical concerns (W3, W4), we agree with the reviewer's reading of the submitted tables and have revised both the experiments and the claims.
  • Every question (Q1–Q5) is answered below with new results.

1. Error of the hypergradient under an inexact Frank–Wolfe solution (W1, Q1).

The reviewer is correct. With an inexact solution α^\hat{\boldsymbol\alpha}, the practical gradient ∇ωFω(α^)\nabla_\omega F_\omega(\hat{\boldsymbol\alpha}) is a biased estimate of ∇ωF⋆(ω)\nabla_\omega F^\star(\omega), and the submission gave no bound. The revision adds one (Proposition 3, Appendix A.2). Let Gω=max⁡j,m∥∇ωk(zj,zm)∥G_\omega=\max_{j,m}\Vert\nabla_\omega k(\mathbf z_j,\mathbf z_m)\Vert. For any feasible α^\hat{\boldsymbol\alpha}, ∥∇ωFω(α^)−∇ωF⋆(ω)∥≤2ν Gω ∥α^−α⋆∥1.\big\Vert\nabla_\omega F_\omega(\hat{\boldsymbol\alpha})-\nabla_\omega F^\star(\omega)\big\Vert\le2\nu\,G_\omega\,\Vert\hat{\boldsymbol\alpha}-\boldsymbol\alpha^\star\Vert_1 . This holds because only the quadratic term depends on ω\omega and ∥α∥1=ν\Vert\boldsymbol\alpha\Vert_1=\nu on the feasible set. For p≥2p\ge2, the dual is μ\mu-strongly convex with μ=q(q−1)min⁡(cˉ1,cˉ2)(ν+12)q−2\mu=q(q-1)\min(\bar c_1,\bar c_2)\big(\tfrac{\nu+1}{2}\big)^{q-2}. The Frank–Wolfe gap gFWg_{\rm FW} then bounds the suboptimality, which gives ∥∇ωFω(α^)−∇ωF⋆(ω)∥≤2ν Gω2n gFW/μ.\big\Vert\nabla_\omega F_\omega(\hat{\boldsymbol\alpha})-\nabla_\omega F^\star(\omega)\big\Vert\le2\nu\,G_\omega\sqrt{2n\,g_{\rm FW}/\mu} . All main experiments use p=100p=100, so this case applies. The bias therefore vanishes with the gap, at rate gFW\sqrt{g_{\rm FW}}.

Accumulation across epochs: Proposition 5 covers this for a gradient-descent encoder step with step size 1/L1/L: 1T∑t=0T−1∥∇ωP⋆(ω(t))∥2≤2L(P⋆(ω(0))−min⁡ΩP⋆)T+8ν2n M12μ⋅1T∑t=0T−1gFW(t),\frac1T\sum_{t=0}^{T-1}\big\Vert\nabla_\omega P^\star(\omega^{(t)})\big\Vert^2\le\frac{2L\big(P^\star(\omega^{(0)})-\min_\Omega P^\star\big)}{T}+\frac{8\nu^2n\,M_1^2}{\mu}\cdot\frac1T\sum_{t=0}^{T-1}g_{\rm FW}^{(t)} , where gFW(t)g_{\rm FW}^{(t)} is the gap at epoch tt. The inner errors do not compound. They enter additively, through the average gap. With exact inner solves, the bound gives the usual O(1/T)\mathcal O(1/T) rate to a stationary point of P⋆P^\star. With a finite budget, the average squared gradient norm is bounded by that term plus a term proportional to the average gap.

Limits of these results: The constants are conservative in our setting, because μ\mu is small when q≈1.01q\approx1.01. The bounds establish the rate and the absence of compounding, not a tight error estimate. They are also stated for a gradient-descent encoder step. We make no formal claim for Adam.

Measured error: We therefore also measured the bias directly. We compared the practical gradient with that of a 500k-iteration solve (means over the ten CIFAR-10 tasks, Table 6 and the figure in Appendix A.1):

IterationsFW gapGrad. cosineGrad. rel. err.α\alpha rel. err.Test AUROCTime (s)
1k0.62930.84260.67530.52160.995420.39
5k0.23420.94160.43710.28590.995931.41
10k0.13820.96990.29790.19490.996082.68
50k0.01710.99890.05070.03500.9961812.9
250k0.00161.00000.00430.00330.9961564.2
500k0.00081.0000000.99615128.0

At our budget (1k), the bias is substantial since the relative gradient error is 0.680.68. The gradient is still positively aligned with the reference (cosine 0.840.84), and the error decreases with the gap, as the bound predicts. Test AUROC changes by less than 10−310^{-3} over a 500×500\times range of budgets, while the cost grows 328×328\times. We have removed every sentence that described the practical update as an exact hypergradient from the paper.

(Part 2/5)

2. The Lipschitz-gradient assumption and convergence (W2, Q1).

The reviewer objects that an LL-Lipschitz gradient for P⋆P^\star was assumed without justification. We agree that the submission only asserted it. The revision derives it (Proposition 4, Appendix A.2) from two conditions that can be checked.

First, the encoder is twice continuously differentiable in its parameters. A ViT with GELU activations, layer normalization with a positive stabilizing constant, softmax attention, and ℓ2\ell_2 normalization of a non-zero class token is a composition of smooth maps. So is the Gaussian kernel. On a compact convex parameter set Ω\Omega, the quantities M1=max⁡∥∇ωk∥M_1=\max\Vert\nabla_\omega k\Vert and M2=max⁡∥∇ω2k∥M_2=\max\Vert\nabla^2_\omega k\Vert are therefore finite.

Second, for p≥2p\ge2 the inner problem is μ\mu-strongly convex over a fixed compact convex set. Its solution map is then Lipschitz, ∥α⋆(ω)−α⋆(ω′)∥2≤2νn M1μ∥ω−ω′∥\Vert\boldsymbol\alpha^\star(\omega)-\boldsymbol\alpha^\star(\omega')\Vert_2\le\tfrac{2\nu\sqrt n\,M_1}{\mu}\Vert\omega-\omega'\Vert. Combined with Danskin's theorem, this gives ∥∇ωP⋆(ω)−∇ωP⋆(ω′)∥≤L∥ω−ω′∥,L=ν2M2+4ν2n M12μ.\Vert\nabla_\omega P^\star(\omega)-\nabla_\omega P^\star(\omega')\Vert\le L\Vert\omega-\omega'\Vert,\qquad L=\nu^2M_2+\frac{4\nu^2n\,M_1^2}{\mu}. So P⋆P^\star is not an arbitrary non-convex composite. It is the optimal value of a strongly convex program whose data depend smoothly on ω\omega, and this structure yields the smoothness.

What remains an assumption: The Frank–Wolfe iterates need none. Each update is a convex combination of the current iterate and a vertex of the compact feasible set, so every iterate is feasible. The encoder weights are different. We assume that they stay in a compact convex set Ω\Omega during training, which keeps M1M_1, M2M_2, and LL finite. Weight decay and gradient clipping keep the weights bounded in practice, but we do not prove it. Convergence under Adam with cosine decay and clipping is also not proved. Sec. 3.2 now states both points, presents monotonicity as a property of an idealized gradient-descent step, and refers to Proposition 5 for the inexact case.

3. The pretrained ViT accounts for most of the performance (W3).

We agree. The fixed ViT-B/16 already exceeds every prior method in Table 1 on CIFAR-10 and CIFAR-100. The margins over those methods are mostly because of the representation, and the paper now mentions this. The contribution we claim for the dual update rests only on comparisons with the backbone held fixed. Sec. 4.2 points to this isolated comparison (Sec. 4.4).

These comparisons use the same ViT-B/16, initialization, splits, anomaly samples, budget, hyperparameters, and seeds. They cover three variants: the fixed backbone (the fixed-feature large-margin ℓp\ell_p-SVDD on ViT features), the primal update (DLM-SVDD on the same ViT), and the proposed dual update. At 10% exposure:

DatasetUpdateAUROCFPR95
Fashion-MNISTFixed0.982680.982680.067860.06786
Primal0.985280.985280.060790.06079
Dual0.98844\mathbf{0.98844}0.04401\mathbf{0.04401}
CIFAR-10Fixed0.995340.995340.019490.01949
Primal0.996560.996560.013620.01362
Dual0.99693\mathbf{0.99693}0.01100\mathbf{0.01100}
CIFAR-100Fixed0.988690.988690.044200.04420
Primal0.990720.990720.034520.03452
Dual0.99162\mathbf{0.99162}0.02988\mathbf{0.02988}

Standard deviations across seeds are at most 3.1×10−43.1\times10^{-4} for AUROC and 1.6×10−31.6\times10^{-3} for FPR95. Based on the full experiments, the fixed ViT-B/16 on Fashion-MNIST obtains 0.9830.983 at 10% exposure and 0.9840.984 at 50% and 75%. This supports the reviewer's point that the pretrained ViT accounts for most of the performance.

(Part 3/5)

Size of the effect: We mention it plainly. In AUROC, the dual update adds 0.160.16 points over the fixed backbone on CIFAR-10 and 0.040.04 over the primal update. On Fashion-MNIST it adds 0.580.58 points over the fixed backbone. Of these, 0.260.26 are also obtained by the primal update and 0.320.32 are specific to the dual update. In the low-false-positive regime the effect is larger. On CIFAR-10, FPR95 falls by 43.6%43.6\% relative to the fixed backbone and by 19.2%19.2\% relative to the primal update, several times the seed standard deviation. The dual update is better than the primal update in all five seeds (paired tt-test: t(4)=6.33t(4)=6.33, p=0.003p=0.003 for AUROC, and t(4)=−12.8t(4)=-12.8, p=2×10−4p=2\times10^{-4} for FPR95). Across all nine dataset–exposure settings, it is at least as good as the primal update in AUROC, FPR95, and balanced accuracy. The gain is smallest on CIFAR-100 (FPR95 down 11–13%13\% relative to primal) and largest on Fashion-MNIST (2828–51%51\%). Our claim is limited to this: with the representation held fixed, updating the encoder through the dual optimal value gives a consistent additional improvement over both no update and the primal update.

4. Supervised ImageNet-1k pretraining (W4).

The concern is that ImageNet labels overlap semantically with the CIFAR classes. We ran the matched comparison again with a DINO-initialized ViT-B/16, which is trained on ImageNet images without any labels. On CIFAR-10 at 10% exposure:

FixedPrimalDual
AUROC0.994560.994560.997330.997330.99786\mathbf{0.99786}
FPR950.024140.024140.011200.011200.00812\mathbf{0.00812}

The ordering is unchanged. The dual update lowers FPR95 by 6666 percent relative to the fixed DINO backbone and by 27.5%27.5\% relative to the primal update. The gain of the proposed update therefore does not rely on label supervision during pretraining. We do not claim that this removes the effect of large-scale pretraining, since DINO still sees ImageNet images (but not the labels).

Two further points support the evaluation. First, pretrained backbones are standard in this benchmark. The strongest prior methods in Table 1, PANDA and MSC, also start from ImageNet-pretrained networks, so the protocol is the one under which they were evaluated. Second, if pretraining made the task trivial, the remaining errors would not respond to the training rule. They do, with both supervised and self-supervised pretraining. We also added a class-disjoint evaluation, in which five of the nine anomaly classes are never seen in training (Appendix A.4). There, AUROC is 0.9840.984, 0.9850.985, and 0.9870.987 and FPR95 is 0.0730.073, 0.0610.061, and 0.0460.046 for fixed, primal, and dual (more details in Appendix).

5. A fixed bandwidth while the representation changes (W5).

Keeping γ\gamma fixed is a deliberate choice, for three reasons. Note also that the RBF bandwidth is a free parameter of the kernel. Any positive value gives a valid kernel, and fixing it follows common practice.

  1. γ\gamma is part of the model. It defines the kernel and therefore the objective P⋆(ω)P^\star(\omega). If γ\gamma were re-estimated from the current features, the kernel would depend on ω\omega through γ\gamma as well. The gradient we use would then no longer be the gradient of the objective, and the median statistic is not differentiable. A fixed γ\gamma makes the boundary step and the encoder step optimize one function.
  2. The features are ℓ2\ell_2-normalized, which bounds every pairwise distance. For two unit vectors, ∥zi−zj∥22=2−2cos⁡θij∈[0,4]\Vert\mathbf z_i-\mathbf z_j\Vert_2^2=2-2\cos\theta_{ij}\in[0,4]. This holds at every epoch, so the kernel values stay in [e−4γ,1][e^{-4\gamma},1], and the scale of the representation cannot drift away from the kernel scale.
  3. The encoder adapts to the kernel, which is the purpose of training it against the boundary objective.
Reviewer_9dJwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Content Summary

This paper proposes TLM-SVDD, an SVDD-based margin metric learning framework for anomaly detection under limited abnormal class data. The authors claim that previous deep large-margin SVDD methods have two problems: (i) the network is not trained on the exact boundary objective, since its training target depends on volatile support vectors and its surrogate loss does not match the original ℓp\ell_p penalty, which makes the learned decision boundary unstable; (ii) CNN features are ill-suited to a kernel-space boundary, because post-ReLU features are skewed (non-negative and anisotropic) for a single RBF bandwidth, and batch normalization makes features depend on batch composition and train/eval mode.

Building on the fixed-feature large-margin ℓp\ell_p-SVDD (Rahimzadeh Arashloo, 2024) and its CNN-based extension DLM-SVDD, the authors (i) replace the CNN encoder with a ViT-B/16 backbone and (ii) replace the primal surrogate-based encoder update with a dual-driven update. Training alternates between two steps across epochs. In the α\alpha-step, the convex dual boundary problem is solved with Frank–Wolfe for a fixed encoder ω\omega. In the ω\omega-step, the dual variables α⋆\alpha^\star are held fixed and the encoder is updated by maximizing the kernel quadratic term a⋆⊤Kωa⋆a^{\star\top} K_\omega a^\star (with a⋆=α⋆⊙ta^\star = \alpha^\star \odot t and KωK_\omega the Gaussian kernel matrix of the encoder features). By Danskin's theorem, this update is claimed to equal the hypergradient of the optimal primal objective value, thereby avoiding recovered thresholds, surrogate losses, and differentiation through the solver. This paper further shows that the encoder gradient depends only on support vectors (normal ones are pulled toward the learned description and anomalous ones pushed away), argues that the encoder objective is bounded and that constant-representation collapse is not a global optimum (for ν>1\nu > 1), and gives a monotonic convergence argument under an exact-inner-solution idealization.

Contributions (Claimed)

  1. Dual-driven encoder update. The authors propose to update the encoder through the optimal value of the exact dual large-margin ℓp\ell_p-SVDD problem. By Danskin's theorem, the encoder hypergradient at the dual optimum reduces to the gradient of a kernel quadratic term, so no recovered thresholds, surrogate losses, or differentiation through the Frank–Wolfe solver are needed. The boundary step and encoder step thus share a single objective, and the update is shown to be bounded and to depend only on support vectors.

  2. Encoder design for the update. The authors identify desirable encoder properties for this update (batch-independent features and compatibility with a single Gaussian kernel bandwidth), which motivates using a Vision Transformer (layer normalization, self-attention) instead of a batch-normalized CNN.

  3. Use of additional anomaly exposure. The authors argue that, unlike a fixed representation where extra anomalies only refit the boundary, the proposed update lets additional anomalous samples improve the learned representation itself.

  4. Empirical results. TLM-SVDD outperforms the compared methods, including the prior convolutional DLM-SVDD, in most settings on standard anomaly detection and long-tailed recognition benchmarks. A controlled ablation separates the effect of the backbone from that of the update rule.

Validation Methods

  • Standard anomaly detection: One-vs-rest protocol on Fashion-MNIST, CIFAR-10, and CIFAR-100, evaluated by AUROC at negative-to-positive training ratios of 10%, 50%, and 75%.
  • Long-tailed recognition: CIFAR-10-LT and CIFAR-100-LT (imbalance ratios 100, 50, 10) and ImageNet-LT (Many/Medium/Few splits), evaluated by balanced accuracy at the recovered radius r⋆2r^{\star 2}.
  • Setup: ViT-B/16 with ImageNet-1k supervised initialization (DINO initialization for ImageNet-LT to avoid label leakage); hyperparameters selected on a stratified 20% validation split; the SVDD boundary is refitted on the selected encoder checkpoint before testing.
  • Ablations: (i) backbone × update rule × anomaly exposure (Table 3); (ii) matched ViT-B/16 comparison of primal vs. dual updates on CIFAR-10 with 10% exposure, 10 seeds per class, with a paired t-test (Table 4); (iii) fixed vs. jointly fine-tuned backbones under the long-tailed protocol (Table 5); (iv) a text-only discussion of three Gaussian kernel bandwidth selection strategies (Sec. 4.4).
Critical strengths
  1. Clear Problem Formulation and Intuitive Approach The idea itself is clean and intuitive: this paper points out the issue of DLM-SVDD's dual-structured objective function—which finds the boundary via the dual formulation while training the network using a primal surrogate—and leverages Danskin's theorem to directly update the encoder using only the quadratic form term of the dual optimal value (−a⋆⊤Kωa⋆-a^{\star\top} K_\omega a^\star).

  2. Strong Benchmark Performance

  • In various public benchmarks, the proposed approach shows high AUROC and balanced accuracies. Furthermore, the proposed method exhibits a substantially larger performance gap than other state-of-the-art methods in the long-tail setting.
Critical weaknesses
  1. Incomplete Notation Definitions Undermine the Readability and Verifiability of the Method Several symbols in Eq. (1) are never defined or explained. Only rr, ρ\rho, pp, and c1,c2c_1, c_2 are described, whereas the slack variables ϵi,ϵl\epsilon_i, \epsilon_l (not identified as slacks at their first appearance), the hypersphere center CC, the feature map Φ(⋅)\Phi(\cdot), the coefficient ν\nu, and the roles of the base constants c~1,c~2\tilde{c}_1, \tilde{c}_2 are left unexplained. These are not incidental symbols, as they propagate through the entire method:
  • the slacks ϵj\epsilon_j are later reconstructed from the dual variables in Sec. 3.1.1 and are required to recover the radius and margin in Eqs. (5)–(6);
  • the center CC defines the anomaly score fω(x)f_\omega(x) in Eq. (4) through its optimal counterpart C⋆C^\star;
  • ν\nu determines the dual feasible set D\mathcal{D}, the per-class dual mass split (ν±1)/2(\nu \pm 1)/2 in the α\alpha-step, and the condition ν>1\nu > 1 used in the boundedness and non-collapse argument in Sec. 3.1.2.

As a result, readers are forced to infer these meanings from prior work or reverse-engineer them from later equations. Combined with the omitted intermediate derivation steps (see Q1), this makes it difficult to follow the update procedure and to verify the logical validity of the subsequent theoretical claims (Propositions 1–2), which undermines confidence in the soundness of the claimed contributions. All variables should be explicitly defined at their first appearance, ideally with a notation table.

  1. Insufficient Analysis of Optimization and Training Stability This paper motivates the dual-driven update by arguing that, in the prior primal-driven approach, the network "chases a moving target" and "no single objective is guaranteed to decrease." However, the training dynamics of the proposed method itself are not empirically analyzed. This paper provides:
  • no epoch-wise curves of the optimal-value objective P⋆(ω)P^\star(\omega) (or Lωdual\mathcal{L}^{\text{dual}}_\omega), the radius r⋆2r^{\star 2}, the margin ρ⋆2\rho^{\star 2}, or the Frank–Wolfe duality gap;
  • no report of the number of Frank–Wolfe iterations or the stopping tolerance used in the α\alpha-step;
  • no clear evidence of seed-to-seed stability. The only stability evidence is the standard deviation reported with Table 4, but the Table 4 caption states that it is computed across the ten per-class performances, and the values quoted in the text as the "pooled run-to-run standard deviation" (0.012 vs. 0.003) differ from those in the table (0.011 vs. 0.002) without explanation. They appear to correspond to the spread across the ten class means rather than to variability across seeds.

This matters because the encoder gradient is computed from an approximately solved dual problem. The monotonicity guarantee in Sec. 3.2 holds only under an exact-solution idealization, and Frank–Wolfe is known to converge slowly and exhibit zigzagging behavior near the optimum, particularly when the solution is sparse as here. Although the Reproducibility Statement says that "additional optimization behavior" is detailed in Sec. 4.4, no such analysis appears there. A direct comparison of objective and boundary trajectories, as well as seed-level variance, between the primal and dual updates is needed to substantiate the claimed advantage.

  1. Limited Comparison with Alternative Bilevel Optimization Strategies Section 2 discusses differentiable optimization and hypergradient-based approaches (e.g., Amos & Kolter, 2017; Franceschi et al., 2018) and explains why the envelope (Danskin) theorem applies to the proposed optimal-value objective. However, this discussion remains purely conceptual. This paper provides no empirical or computational comparison against an alternative bilevel strategy, such as unrolled or implicit differentiation, in terms of gradient accuracy, stability, runtime, or memory. Since the proposed gradient is exact only at the exact dual optimum, such a comparison, or at least an analysis of gradient error under inexact Frank–Wolfe solutions, would clarify the practical advantages of the proposed scheme.

  2. Unexplained Empirical Anomalies and Performance Drops Two counter-intuitive results lack adequate explanation:

  • Table 4: On CIFAR-10 with 10% anomaly exposure, the ViT-B/16 model trained with the primal update obtains a mean AUROC of 0.982. This is lower than the 0.995 obtained by the fixed ViT-B/16 backbone in Table 3 for the same dataset and exposure level, which suggests that primal fine-tuning degrades the pretrained representation. No explanation is given, and the comparison is split across two tables, so it is unclear whether the settings and numbers of runs are identical.
  • Table 5: On CIFAR-100-LT, joint fine-tuning with the proposed dual update decreases balanced accuracy compared with the fixed backbone at every imbalance ratio (by 1.86, 2.61, and 3.00 points at ϱ=100,50,10\varrho = 100, 50, 10). The degradation is reported in the text, but no analysis is provided on why the encoder update harms performance in this setting.

These observations require further investigation and discussion.

  1. Insufficient Evidence That the Dual Encoder Update Itself Contributes: Metric, Statistical Reliability, and Dataset Coverage The proposed solution is logically aligned with the problems stated in the Introduction. However, whether the dual encoder update itself yields a genuine improvement is not convincingly established, owing to the choice of metric, the lack of statistical reliability, and the limited coverage of the controlled comparison.
  • (a) Metric. The standard anomaly detection experiments (Tables 1, 3, and 4) rely solely on AUROC. On CIFAR-10 and CIFAR-100, the fixed ViT-B/16 already reaches 0.995–0.996 and 0.989–0.992, respectively (Table 3), so the remaining headroom is very small, and a threshold-free metric such as AUROC has little resolution in this regime. In Table 3, the dual update improves over the fixed backbone in every setting, but the gain is clear only on Fashion-MNIST (about 0.045–0.049), whereas it is merely 0.002–0.003 on CIFAR-10 and 0.003 on CIFAR-100. Complementary metrics that are more sensitive at high performance, such as AUPR and FPR at 95% TPR, are needed to judge whether these small gains are meaningful.

  • (b) Statistical reliability. Beyond Fashion-MNIST, it is unclear whether the small gains over the fixed backbone reflect a real effect or experimental chance. The only significance test in this paper (Table 4, paired t(9)=5.02t(9)=5.02, p=7.2×10−4p=7.2\times10^{-4}) is computed over the ten per-class means, i.e., over classes rather than seeds, and compares the primal and dual updates. It does not test the fixed vs. dual comparison, which is the one that isolates the contribution of encoder training on top of boundary learning. For Tables 1, 2, 3, and 5, it is not stated whether multiple seeds were used and no variance is reported. For Tables 1, 2, 3, and 5, it is not stated whether multiple seeds were used and no variance is reported (see also the discussion of seed-to-seed stability in Weakness 2).

  • (c) The primal vs. dual comparison is shown on a single dataset. This paper's key evidence that the improvement over DLM-SVDD cannot be explained by the transformer architecture alone comes from the matched ViT-B/16 comparison of the primal and dual updates in Table 4, which is conducted only on CIFAR-10 at 10% anomaly exposure. The authors explain that the comparison is limited to a single representative dataset "due to space constraints," but it is not clear why CIFAR-10 was chosen as the representative. Fashion-MNIST, where the gain of the dual update over the fixed backbone is clearest, CIFAR-100, which is the more challenging benchmark, the 50% and 75% exposure levels, and the long-tailed benchmarks are all excluded, even though these are the settings where the headline results are reported (Tables 1 and 2). The space constraint is also not convincing since an appendix could host these results. As a consequence, the generality of the conclusion that the dual update, rather than the backbone, is responsible for the gains over DLM-SVDD cannot be assessed beyond this one setting, and the statistical test in Table 4 (computed over the ten CIFAR-10 class means) supports only that setting.

  1. Absence of Any Figure Hinders Intuitive Understanding of the Method
  • This paper contains no figure at all. Figures could have helped readers understand the proposed framework (e.g., a diagram of the alternating α\alpha-step and ω\omega-step) and, more importantly, could have shown qualitatively whether the feature space actually changes in the intended way, for example through feature-space visualizations before and after encoder updates, or the behavior of support vectors and kernel structure during training. I do not consider this an essential flaw by itself. However, the absence of figures that would support intuitive understanding makes it harder to assess the authors' claimed contributions.
Questions for authors

Q1. Omitted derivations and Eq. (2). Could the authors provide the omitted derivation from the primal problem (Eq. 1) to the dual problem (Eq. 2)? In my derivation, the ℓq\ell_q term has a per-sample coefficient (1−1p)(cjp)−1p−1αjq(1-\frac{1}{p})(c_j p)^{-\frac{1}{p-1}}\alpha_j^{q}, whereas Eq. (2), written with cˉ1,2∥α⊙(1±t)∥qq\bar{c}_{1,2}\Vert\alpha\odot(1\pm t)\Vert_q^q and cˉ1,2=12(c1,2p)−1p−1(1−1p)\bar{c}_{1,2}=\frac{1}{2}(c_{1,2}p)^{-\frac{1}{p-1}}(1-\frac{1}{p}), gives a coefficient larger by a factor of 2q−12^{q-1} for ti∈{+1,−1}t_i\in\lbrace +1,-1\rbrace. Is this a typo or a different convention, and does P⋆(ω)=1−F⋆(ω)P^\star(\omega)=1-F^\star(\omega) in Proposition 1 hold exactly with Eq. (2) as written? An explicit derivation of the bound −ν2+12≤Lωdual≤0-\frac{\nu^2+1}{2}\le\mathcal{L}^{\mathrm{dual}}_\omega\le 0 would also help readers verify the mathematical claims.

Q2. Selected hyperparameters and sensitivity. Only the search ranges for pp, ν\nu, and c~1,c~2\tilde{c}_1,\tilde{c}_2 are reported in Sec. 4.1, even though the Reproducibility Statement says that the selected hyperparameters are discussed there. The values actually selected for each dataset are not given in this paper, and no sensitivity analysis is provided. Two points in the search space are of particular concern:

  • p=1.01p=1.01 (q=101q=101), where the ℓq\ell_q penalty becomes extremely flat on [0,1][0,1] and may affect the conditioning of the dual problem and the accuracy of finite Frank–Wolfe solutions;
  • ν=1\nu=1, where the anomaly dual mass (ν−1)/2(\nu-1)/2 becomes zero. If selected in any setting, labeled anomalies would no longer influence either the boundary or the encoder update, and the non-collapse argument (which requires ν>1\nu>1) would no longer apply.

Which values were selected for each dataset and exposure level, and how sensitive is the final performance to pp and ν\nu?

Q3. Bandwidth study and computational cost. The "Bandwidth sensitivity" analysis in Sec. 4.4 compares three strategies for selecting the Gaussian kernel bandwidth γ\gamma but reports no quantitative results, although the Reproducibility Statement states that this analysis is "detailed in Sec. 4.4." Could the authors provide the actual numbers? Relatedly, what is the wall-clock training time, given that the kernel matrix must be recomputed and the dual re-solved at every epoch?

Q4. Generality and reliability of the primal vs. dual comparison. The matched ViT-B/16 comparison of the primal and dual updates (Table 4) is conducted only on CIFAR-10 at 10% exposure. Could the authors report the same comparison on Fashion-MNIST and CIFAR-100 (and at the other exposure levels), and clarify whether multiple seeds were used for Tables 1–3 and 5? In particular, could they provide a significance test for the fixed vs. dual comparison, together with complementary metrics such as AUPR and FPR at 95% TPR, since AUROC is close to saturation on CIFAR-10/100?

7 comments

(Part 1/5)

We thank the reviewer for a very careful review and feedback on our submission. Every weakness and question is addressed below.

1. Eq. (2), the omitted derivation, and the bound (W1, Q1).

The reviewer's derivation is correct: Minimizing cjϵp−αjϵc_j\epsilon^p-\alpha_j\epsilon over ϵ≥0\epsilon\ge0 gives ϵj=(αj/(cjp))1/(p−1)\epsilon_j=(\alpha_j/(c_jp))^{1/(p-1)} and the value −(1−1p)(cjp)−1/(p−1)αjq-\big(1-\tfrac1p\big)(c_jp)^{-1/(p-1)}\alpha_j^{q}, so the per-sample coefficient is exactly the one the reviewer states. The submitted Eq. (2) wrote this term as cˉs∥α⊙(1±t)∥qq\bar c_s\Vert \boldsymbol\alpha\odot(\mathbf 1\pm\mathbf t)\Vert_q^q with a prefactor 12\tfrac12. Every non-zero entry of 1±t\mathbf 1\pm\mathbf t equals 22, so the correct prefactor in that notation is 2−q2^{-q}, and the submitted form was too large by 2q−12^{q-1}.

Typo or convention? It is a slip in the compact class-selector notation, which we carried over from the fixed-feature formulation, and not a different convention. It has a simple interpretation: because (q−1)(p−1)=1(q-1)(p-1)=1, we have 2q−1(csp)−1/(p−1)=(cs2p)−1/(p−1)2^{q-1}(c_sp)^{-1/(p-1)}=\big(\tfrac{c_s}{2}p\big)^{-1/(p-1)}, so the submitted Eq. (2) is the exact dual of the same primal with slack penalties cs/2c_s/2.

Does P⋆(ω)=1−F⋆(ω)P^\star(\omega)=1-F^\star(\omega) hold with Eq. (2) as written? Not for the penalties csc_s stated in the submission. It holds exactly for cs/2c_s/2. In the revision:

  • Eq. (2) is written class-wise, Fω(α)=cˉ1∑i∈INαiq+cˉ2∑l∈IAαlq+(α⊙t)⊤Kω(α⊙t)F_\omega(\boldsymbol\alpha)=\bar c_1\sum_{i\in\mathcal I_N}\alpha_i^q +\bar c_2\sum_{l\in\mathcal I_A}\alpha_l^q +(\boldsymbol\alpha\odot\mathbf t)^\top K_\omega(\boldsymbol\alpha\odot\mathbf t) with cˉs=(1−1p)(csp)−1/(p−1)\bar c_s=\big(1-\tfrac1p\big)(c_sp)^{-1/(p-1)}, which removes the ambiguity.
  • Our implementation follows the compact form. It therefore solves the corrected dual with cs=c~s/(2∣Is∣)c_s=\tilde c_s/(2\vert \mathcal I_s\vert ), and the paper now defines csc_s in exactly this way. With this definition the stated primal, the stated dual, and the code agree, P⋆=1−F⋆P^\star=1-F^\star holds exactly, and no AUROC result changes.
  • Appendix A.2 gives the complete derivation: the Lagrangian, each stationarity block, the elimination of the slacks, the dual function, the relation to the compact form above, and a proof of strong duality (for a Gaussian kernel the substitution R=r2−∥C∥2R=r^2-\Vert C\Vert ^2 makes the primal a convex program with affine constraints). We also verified the identity numerically by solving the primal directly on small instances.

The bound: Appendix A.2 derives it explicitly. Positive semidefiniteness gives a⊤Kωa≥0\mathbf a^\top K_\omega\mathbf a\ge0. Since 0<k≤10<k\le1 and the cross-class terms enter with a negative sign, a⊤Kωa≤(∑INαi)2+(∑IAαl)2=(ν+1)2+(ν−1)24=ν2+12\mathbf a^\top K_\omega\mathbf a\le(\sum_{\mathcal I_N}\alpha_i)^2 +(\sum_{\mathcal I_A}\alpha_l)^2=\tfrac{(\nu+1)^2+(\nu-1)^2}{4} =\tfrac{\nu^2+1}{2}. Hence −ν2+12≤Lωdual≤0-\tfrac{\nu^2+1}{2}\le\mathcal L^{\rm dual}_\omega\le0.

Notation: CC, rr, ρ\rho, ϵj\epsilon_j, Φ\Phi, ν\nu, and c~s\tilde c_s are now defined where Eq. (1) first appears, and a notation table has been added to the appendix.

2. Optimization and training stability (W2).

Solver settings: Every boundary update uses a fixed budget of 1,0001{,}000 Frank–Wolfe iterations with step size 2/(τ+2)2/(\tau+2). There is no tolerance-based stopping. This is now in Sec. 4.1.

Solver accuracy: We solved the same boundary problem with budgets up to 500k iterations and compared each solution with the 500k reference (means over the ten CIFAR-10 tasks, full table and plot in Appendix A.1):

IterationsFW gapGrad. cosineGrad. rel. err.α\alpha rel. err.Test AUROCTime (s)
1k0.62930.84260.67530.52160.995420.39
5k0.23420.94160.43710.28590.995931.41
10k0.13820.96990.29790.19490.996082.68
50k0.01710.99890.05070.03500.9961812.9
250k0.00161.00000.00430.00330.9961564.2
500k0.00081.0000000.99615128.0

(Part 2/5)

The reviewer's expectation of slow Frank–Wolfe convergence on a sparse solution is true, and we now say so: all main experiments use p=100p=100, i.e. q≈1.01q\approx1.01, for which the regularizer gradient varies sharply near αj=0\alpha_j=0 and the standard O(1/T)\mathcal O(1/T) rate does not apply. The 1k solution is not accurate (gap 0.630.63). Its gradient is nevertheless positively aligned with the reference (cosine 0.840.84), and the detection metric is flat over the whole range, while the 500k budget is 328×328\times slower. Away-step or pairwise Frank–Wolfe variants are a natural way to close this gap and are noted as future work.

Training dynamics: Within the rebuttal period we gave priority to the matched comparison, the solver diagnostics, and the sensitivity study, and did not add epoch-wise training curves to the paper (also because of page limit). Our training code logs Lωdual\mathcal L^{\rm dual}_\omega, the validation AUROC, r⋆2r^{\star2}, and ρ⋆2\rho^{\star2} at every epoch, and the convergence statements of Sec. 3.2 are now tied to Propositions 4 and 5 rather than to an observed curve.

Seed-to-seed stability and the inconsistent standard deviations: The standard deviation in the submitted Table 4 was the spread across the ten class means and not across seeds. The two sets of values in the text and in the table should not both have appeared, and both have been removed. In the revision the one-vs-rest tasks are averaged within each seed, and mean, standard deviation, and paired tests are computed across matched seeds (five matched seeds at 10% exposure on Fashion-MNIST and CIFAR-10, two on CIFAR-100 at 10%, and one seed at 50% and 75%). With this unit, the seed-level standard deviations of the primal and dual updates are of the same order (at most 3.1×10−43.1\times10^{-4} AUROC), so we have also removed the claim that the dual update reduces run-to-run variance.

3. Alternative bilevel strategies (W3).

The reviewer asked for a comparison with unrolled or implicit differentiation, or, failing that, for an analysis of the gradient error under inexact inner solves. We did not implement unrolled or implicit differentiation, for the structural reason given in the next paragraph. We instead provide the second option the reviewer named: the gradient error under inexact solves is now bounded in theory (Propositions 3 and 5) and measured (the table in item 2).

The outer objective is the inner optimal value. Its gradient does not contain ∂α⋆/∂ω\partial\boldsymbol\alpha^\star/\partial\omega, which is the quantity that unrolling and implicit differentiation exist to compute. At the exact inner solution all three approaches return the same gradient. At an inexact solution, all three are inexact. Unrolling is also poorly suited to this solver: the Frank–Wolfe vertex selection is an arg⁡min⁡\arg\min and is piecewise constant, so the unrolled gradient ignores changes of the selected atoms, and storing 1,0001{,}000 kernel updates over all training features on top of the ViT graph is not feasible in memory. Implicit differentiation requires a linear system on the active set, which is not identified at an inexact solution.

The analysis of gradient error under inexact solves is as follows. Proposition 3 (Appendix A.2) bounds the error by 2νGω∥α^−α⋆∥12\nu G_\omega\Vert \hat{\boldsymbol\alpha}-\boldsymbol\alpha^\star\Vert_1 for any feasible α^\hat{\boldsymbol\alpha}, and by 2νGω2n gFW/μ2\nu G_\omega\sqrt{2n\,g_{\rm FW}/\mu} for p≥2p\ge2, which covers our setting. Proposition 5 turns this into a statement for the alternating scheme: with a gradient-descent encoder step, the average squared gradient norm of P⋆P^\star is bounded by an O(1/T)\mathcal O(1/T) term plus a term proportional to the average Frank–Wolfe gap. The table above is the empirical counterpart. We state clearly that these results concern a gradient-descent encoder step and are not claimed for Adam.

(Part 3/5)

4. The two empirical anomalies (W4).

Primal ViT below fixed ViT: With the full matched experiments, the primal ViT is not below the fixed ViT: on CIFAR-10 at 10% exposure the fixed, primal, and dual models obtain 0.995340.99534, 0.996560.99656, and 0.996930.99693. The submitted Table 4 (0.9820.982 for the primal entry) has been updated with these values and more details. Primal fine-tuning therefore does not fall below the fixed backbone, and the dual update improves on the primal one. With these values, the gain of dual over primal on CIFAR-10 is 0.00040.0004 AUROC (0.040.04 points), and the paper states the claim accordingly.

CIFAR-100-LT: The reviewer is right that we reported the drop without analysing it. We did not complete a per-class decomposition of the drop within the rebuttal period. The paper reports the drop explicitly (Sec. 4.4: on CIFAR-100-LT the fixed ViT is better than the jointly fine-tuned one at all three imbalance ratios), we read it as a limitation of joint fine-tuning when classes have very few normal samples (which might cause overfitting during training), and we do not claim that the encoder update is uniformly beneficial.

5. Evidence for the dual update: metric, reliability, coverage (W5, Q4).

(a) Metric: We agree that AUROC has little resolution here. The matched comparison now reports AUPR-In, FPR95, and balanced accuracy (Appendix A.3). At 10% exposure:

DatasetUpdateAUROCAUPR-InFPR95BAcc
Fashion-MNISTFixed0.982680.982680.878600.878600.067860.067860.911190.91119
Primal0.985280.985280.897930.897930.060790.060790.924070.92407
Dual0.98844\mathbf{0.98844}0.90853\mathbf{0.90853}0.04401\mathbf{0.04401}0.93003\mathbf{0.93003}
CIFAR-10Fixed0.995340.995340.965320.965320.019490.019490.938670.93867
Primal0.996560.996560.97069\mathbf{0.97069}0.013620.013620.958670.95867
Dual0.99693\mathbf{0.99693}0.970460.970460.01100\mathbf{0.01100}0.96057\mathbf{0.96057}
CIFAR-100Fixed0.988690.988690.664640.664640.044200.044200.915760.91576
Primal0.990720.990720.670270.670270.034520.034520.930730.93073
Dual0.99162\mathbf{0.99162}0.67643\mathbf{0.67643}0.02988\mathbf{0.02988}0.93708\mathbf{0.93708}

On CIFAR-10 the dual update lowers FPR95 by 43.6%43.6\% relative to the fixed backbone and by 19.2%19.2\% relative to the primal update. AUPR-In is unchanged relative to the primal update, which the paper states. CIFAR-100 shows the same ordering in all four metrics. (Its AUPR-In values are lower because normal samples are only 1% of each CIFAR-100 test set.)

On Fashion-MNIST the reviewer computes a gain of 0.0450.045–0.0490.049 from the fixed-ViT value of 0.9440.944 in the submitted Table 3. Based on the full experiments, with the single configuration used for all results, the fixed ViT obtains 0.9830.983 at 10% exposure and 0.9840.984 at 50% and 75%, and the revised Table 3 reports these values. The gain from training the encoder at 10% exposure on Fashion-MNIST is therefore 0.260.26 points with the primal update and 0.580.58 points with the dual update, so more than half of it is specific to the dual update.

(b) Statistical reliability: Seeds are now the statistical unit. For fixed vs. dual, which the reviewer identifies as the comparison that isolates encoder training, the dual update is better in every seed on CIFAR-10. A paired tt-test over the five seeds gives t(4)=44.3t(4)=44.3, p=2×10−6p=2\times10^{-6} for AUROC and t(4)=−25.5t(4)=-25.5, p=1×10−5p=1\times10^{-5} for FPR95. On Fashion-MNIST the dual update has a higher mean AUROC and a lower FPR95 than both the fixed backbone and the primal update at all three exposure ratios. For dual vs. primal on CIFAR-10: t(4)=6.33t(4)=6.33, p=0.003p=0.003 (AUROC) and t(4)=−12.8t(4)=-12.8, p=2×10−4p=2\times10^{-4} (FPR95).

(c) Coverage: The matched comparison now covers Fashion-MNIST, CIFAR-10, and CIFAR-100 at all three exposure ratios (Table 3 and Appendix A.3), and we no longer cite a space constraint. Over these nine settings the dual update is at least as good as the matched primal update in every setting for AUROC, FPR95, and balanced accuracy, and in seven of nine for AUPR-In. The advantage is largest on Fashion-MNIST, where FPR95 falls by 2828–51%51\% relative to the primal update, and smallest on CIFAR-100 (11–13%13\%). The same ordering is also valid in a new class-disjoint evaluation on CIFAR-10, in which five of the nine anomaly classes are never seen in training (Appendix A.4).

Reviewer_MSzsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies deep anomaly detection with limited labeled anomaly exposure. It addresses a limitation of prior deep large-margin SVDD methods, where the boundary is optimized using a dual objective but the encoder is updated with a different approximate primal surrogate. To address this mismatch, the authors propose TLM-SVDD, which uses a ViT-B/16 backbone and updates the encoder through the optimal value of the large-margin ℓp-SVDD dual problem. By applying Danskin’s theorem, the encoder gradient is obtained from the kernel quadratic term at the optimal dual variables, without differentiating through the Frank–Wolfe solver. The method is evaluated on Fashion-MNIST, CIFAR-10/100, their long-tailed variants, and ImageNet-LT, with comparisons against anomaly-detection and long-tailed recognition baselines.

Critical strengths
  1. The paper addresses a relevant problem in deep anomaly detection: how to jointly learn a representation and a large-margin SVDD boundary when a limited number of labeled anomalies are available. The authors identify a concrete mismatch in the preceding DLM-SVDD framework, where the boundary and encoder updates are based on different objectives.
  2. The proposed dual-driven encoder update has a clear mathematical motivation. Proposition 1 uses Danskin’s theorem to derive the encoder gradient from the optimal value of the dual problem, while Proposition 2 gives a support-vector contrast interpretation. The evaluation also covers several standard and long-tailed benchmarks, and Table 4 makes a useful attempt to compare primal and dual updates using the same ViT-B/16 backbone and initialization.
Critical weaknesses
  1. The independent contribution of the proposed dual encoder update is not established across a sufficiently broad range of settings. Table 3 compares fixed and jointly updated representations, but it does not provide a complete backbone-by-update comparison: the ViT-B/16 setting is evaluated with fixed and dual updates, whereas the matched primal-versus-dual comparison is provided only in Table 4. Table 4 is limited to CIFAR-10 with 10% anomaly exposure. Therefore, the evidence that the dual update itself consistently explains the reported gains across datasets and anomaly-exposure ratios remains limited. In addition, the statistical unit in Table 4 should be clarified. The text mentions ten seeds per class, while the table reports statistics across ten per-class performances and the paired test is conducted on ten class means. It is not clear how seed-level repetitions enter the reported standard deviation and significance test. A broader matched comparison with the same backbone, initialization, data splits, and training budget would make the central methodological claim more convincing.
  2. The theoretical exact-gradient result is not sufficiently connected to the practical optimization procedure. Proposition 1 derives the encoder hypergradient at an optimal solution of the inner dual problem. However, Section 3.2 acknowledges that the Frank–Wolfe subproblem is solved only to a finite gap in practice. The main paper does not clearly report the stopping tolerance, typical iteration counts, or optimality gaps obtained during training, nor does it quantify how the inner-solver accuracy affects the encoder gradient and final performance. Moreover, the convergence discussion uses a gradient-descent step-size condition, whereas the implementation in Section 4.1 uses Adam, cosine learning-rate decay, and gradient clipping. Thus, the stated monotonicity argument does not directly establish convergence behavior for the actual training procedure. Reporting solver tolerances and conducting a sensitivity analysis would help connect the theoretical result to the implemented method.
  3. The long-tailed evaluation protocol and the generalization claims require further clarification. Table 2 places conventional multi-class long-tailed classifiers and one-vs-rest SVDD-based detectors in the same comparison and reports balanced accuracy. However, the paper does not sufficiently specify how the outputs of multiple one-vs-rest detectors are converted into a single multi-class prediction, how cases in which multiple detectors accept or reject a sample are handled, or precisely how balanced accuracy is aggregated. Without these details, it is difficult to assess whether the reported results are directly comparable across the two groups of methods. In addition, the Introduction motivates the method by noting that test-time anomalies may differ from those observed during training, but the described one-vs-rest protocol draws training anomalies from the remaining classes and does not report a class-disjoint evaluation in which some anomaly categories are entirely excluded from training. The current experiments therefore do not fully establish generalization to previously unseen anomaly categories. Overall, the paper presents a plausible and potentially useful optimization idea. However, the current evidence does not yet establish the independent benefit of the dual update across a broad range of settings, fully connect the theoretical optimal-gradient result to the finite-accuracy implementation, or sufficiently clarify the long-tailed evaluation protocol.
Questions for authors
  1. Could the authors provide a more complete matched comparison of primal and dual encoder updates beyond CIFAR-10 with 10% anomaly exposure? In particular, could the comparison be extended to additional datasets and anomaly-exposure ratios using the same backbone, initialization, data split, and training budget? Please also clarify whether the standard deviations and paired test in Table 4 are computed over seeds, classes, or class-level averages, and how the ten seeds per class enter the statistical analysis.
  2. Could the authors report the Frank–Wolfe stopping criterion, iteration counts, and optimality gaps during training, and provide a sensitivity analysis using different inner-solver tolerances? How close is the practical encoder gradient to the gradient obtained from a more accurately solved dual problem? Please also clarify which convergence statements apply to the idealized gradient-descent procedure and which, if any, are supported for the Adam-based implementation.
  3. Could the authors precisely describe the long-tailed prediction and evaluation protocol used for the one-vs-rest SVDD methods in Table 2? How are the outputs or radius-based decisions from multiple one-vs-rest detectors converted into a single class prediction, how are conflicting decisions handled, and what exact balanced-accuracy formula is used? Were the multi-class baselines evaluated using the same final prediction and metric protocol?
  4. Could the authors provide a class-disjoint anomaly evaluation in which selected anomaly categories are absent from training and appear only at test time? Also, the hyperparameter search includes ν = 1. From the stated dual feasible set, the total dual mass assigned to anomalous samples is (ν − 1)/2, which is zero when ν = 1. Could the authors report the selected ν values and clarify how anomaly exposure affects the encoder update in cases where ν = 1 is selected?
4 comments

(Part 1/4)

We thank the reviewer for the careful reading and the helpful comments provided. Based on the mentioned three weaknesses of the submission we have done new experiments, new analyses, and improved the text: (i) the updated comparison now covers fixed, primal, and dual ViT-B/16 models on three datasets and three exposure ratios, (ii) the finite Frank–Wolfe solver is now reported, measured, and connected to the theory through additional experiments, (iii) the long-tailed protocol is clarified, including what it does and does not do, and (iv) a class-disjoint (unseen) evaluation is included in the experiments.

1. Matched comparison of the encoder updates and the statistical unit (W1, Q1).

The comparison now includes the fixed ViT-B/16, the primal-driven update of DLM-SVDD on the same ViT-B/16, and the proposed dual-driven update, on Fashion-MNIST, CIFAR-10, and CIFAR-100 at 10%, 50%, and 75% anomaly exposure ratios. Inside each setting the three variants share the exact same initialization, data split, anomaly samples, training budget, hyperparameters, and random seeds. Table 3 of the paper now also contains the “ViT-B/16, primal” row, so the backbone-by-update grid the reviewer asked for is complete for the transformer. Test AUROC, averaged over the one-vs-rest tasks and then over seeds:

DatasetUpdate10%50%75%
Fashion-MNISTFixed0.982680.982680.983810.983810.984410.98441
Primal0.985280.985280.986190.986190.987480.98748
Dual0.98844\mathbf{0.98844}0.99244\mathbf{0.99244}0.99293\mathbf{0.99293}
CIFAR-10Fixed0.995340.995340.995840.995840.995970.99597
Primal0.996560.996560.997340.997340.997440.99744
Dual0.99693\mathbf{0.99693}0.99829\mathbf{0.99829}0.99851\mathbf{0.99851}
CIFAR-100Fixed0.988690.988690.991410.991410.991700.99170
Primal0.990720.990720.993650.993650.994160.99416
Dual0.99162\mathbf{0.99162}0.99400\mathbf{0.99400}0.99543\mathbf{0.99543}

The standard deviation across seeds is at most 3.1×10−43.1\times10^{-4} AUROC. At 10% exposure the dual update exceeds the matched primal update by 0.00320.0032 AUROC on Fashion-MNIST and by 0.00040.0004 on CIFAR-10. The CIFAR-10 difference is small in absolute terms because AUROC is saturated there, although it appears in all five matched seeds (see the paired tests below). For that reason we now also report three metrics that are less saturated (Appendix A.3):

  • AUPR-In: the area under the precision–recall curve with the normal samples taken as the positive class.
  • FPR95: the fraction of anomalous test samples that are still accepted at the threshold which accepts 95% of the normal test samples. Lower is better.
  • Balanced accuracy: the mean of the true-positive rate on normal samples and the true-negative rate on anomalous samples, at the recovered radius r⋆2r^{\star2}.

FPR95 is the metric with the most resolution in this regime: the dual update lowers it by 27.6%27.6\% on Fashion-MNIST and 19.2%19.2\% on CIFAR-10 relative to the primal update, and by 43.6%43.6\% on CIFAR-10 relative to the fixed backbone. AUPR-In improves on Fashion-MNIST and is unchanged on CIFAR-10 (0.97050.9705 vs. 0.97070.9707), which we state in the paper. Over the complete 3×33\times3 grid, the dual update is at least as good as the matched primal update in all nine settings for AUROC, FPR95, and balanced accuracy, and in seven of nine for AUPR-In. Relative to the primal update, FPR95 falls by 27.627.6–51.1%51.1\% on Fashion-MNIST and 19.219.2–56.3%56.3\% on CIFAR-10, but only by 1.21.2–13.4%13.4\% on CIFAR-100, where the two updates are close.

(Part 2/4)

Statistical unit: The reported standard deviation was the spread across the ten class means, and the paired test was run on those ten class means. The ten seeds per class entered only through the class means. In the revision, the one-vs-rest tasks are first averaged within each seed, and mean, standard deviation, and paired tests are then computed across matched seeds. On CIFAR-10 at 10% exposure (five seeds), the dual update is better than the primal update in every seed, with a paired tt-test over seeds giving t(4)=6.33t(4)=6.33, p=0.003p=0.003 for AUROC and t(4)=−12.8t(4)=-12.8, p=2×10−4p=2\times10^{-4} for FPR95. Against the fixed backbone, the test gives t(4)=44.3t(4)=44.3, p=2×10−6p=2\times10^{-6} for AUROC. On Fashion-MNIST the dual update has a higher mean AUROC and a lower FPR95 than the primal update at all three exposure ratios. CIFAR-100 has two seeds at 10%, too few for a seed-level test. Over its 100 one-vs-rest tasks (seed 42), the dual update has a higher AUROC than the primal update in 69 tasks (Wilcoxon signed-rank p=4×10−6p=4\times10^{-6}) and a lower FPR95 in 77 (p=5×10−7p=5\times10^{-7}).

The primal entry of the submitted Table 4: That table reported 0.9820.982 for the primal ViT on CIFAR-10 only. This table has been updated with added new values and more details. With the full experiments, the dual-over-primal AUROC gain on CIFAR-10 is 0.00040.0004.

Metrics at an operating point: AUROC summarizes the ranking over all thresholds, so a small AUROC difference can hide a larger difference at the threshold that is actually used for decisions. We therefore also compare the three variants at a fixed threshold. Writing sensitivity for the fraction of normal test samples accepted and specificity for the fraction of anomalous test samples rejected, balanced accuracy is their arithmetic mean. At the recovered radius r⋆2r^{\star2} it is 0.93870.9387, 0.95870.9587, and 0.96060.9606 for the fixed, primal, and dual models on CIFAR-10, and 0.91120.9112, 0.92410.9241, and 0.93000.9300 on Fashion-MNIST. At a fixed sensitivity of 95%, the dual model also rejects more anomalies than the primal one on both datasets (the FPR95 values above).

2. Connecting the exact-gradient result to the implemented procedure (W2, Q2).

Stopping criterion and iteration count: Every boundary update in every experiment uses a fixed budget of 1,0001{,}000 Frank–Wolfe iterations with step size 2/(τ+2)2/(\tau+2). No tolerance-based stopping is used. This is now stated in Sec. 4.1.

Optimality gaps and sensitivity: For our feasible set the Frank–Wolfe gap has a closed-form expression that is available at every iteration at no extra cost (Appendix A.1), and it upper-bounds the suboptimality Fω(α^)−F⋆(ω)F_\omega(\hat{\boldsymbol\alpha})-F^\star(\omega). We solved the same boundary problem with budgets from 1k to 500k and compared each solution with the 500k reference (means over the ten CIFAR-10 tasks):

IterationsFW gapGrad. cosineGrad. rel. err.α\alpha rel. err.Test AUROCTime (s)
1k0.62930.84260.67530.52160.995420.39
5k0.23420.94160.43710.28590.995931.41
10k0.13820.96990.29790.19490.996082.68
50k0.01710.99890.05070.03500.9961812.9
250k0.00161.00000.00430.00330.9961564.2
500k0.00081.0000000.99615128.0

We do not describe the 1k solution as accurate: its gap is 0.630.63 and its gradient has a relative error of 0.680.68. What the table shows is that the gradient is already positively aligned with the reference (cosine 0.840.84) and that the detection metric is almost flat across a 500×500\times range of budgets.

How close is the practical gradient, in theory: The revision adds three results (Appendix A.2):

  • Proposition 3: for any feasible α^\hat{\boldsymbol\alpha}, the gradient error is at most 2νGω∥α^−α⋆∥12\nu G_\omega\Vert\hat{\boldsymbol\alpha}-\boldsymbol\alpha^\star\Vert_1, and for p≥2p\ge2 at most 2νGω2n gFW/μ2\nu G_\omega\sqrt{2n\,g_{\rm FW}/\mu}, where μ\mu is the strong-convexity modulus of the dual. All main experiments use p=100p=100.
  • Proposition 4: for p≥2p\ge2 and a twice continuously differentiable encoder on a compact convex parameter set, the optimal value P⋆P^\star has a Lipschitz gradient. The smoothness used in Sec. 3.2 is therefore derived, not assumed.
  • Proposition 5: with a gradient-descent encoder step and arbitrary inexact boundary steps, the average squared gradient norm of P⋆P^\star is bounded by an O(1/T)\mathcal O(1/T) term plus a term proportional to the average Frank–Wolfe gap.

(Part 3/4)

Which statements apply to which procedure: The envelope identity (Proposition 1) holds at the exact inner solution. Proposition 3 holds for any inner solution and any outer optimizer, but it is a per-step bound. Monotonicity of P⋆P^\star and Proposition 5 hold for a gradient-descent outer step with step size 1/L1/L. None of these is claimed for the Adam updates with cosine decay and clipping that we actually use. For that procedure, our evidence is empirical. Sec. 3.2 has been rewritten to say this explicitly.

Why 1k iterations: Adam normalizes each update by a running estimate of the gradient magnitude and the gradient is clipped, so the encoder step depends mainly on the gradient direction. The boundary problem is re-solved for the current encoder at every epoch, so errors in the dual variables are not carried over. The reported boundary is refitted on the selected checkpoint. Against this, the solve time grows linearly: the 500k budget is 328×328\times slower, which amounts to about nine hours of solver time per CIFAR-10 run instead of under two minutes. In proportion to training: one training epoch of a CIFAR-10 one-vs-rest task takes about 9090 s on one NVIDIA RTX 6000 Pro. The 1k-iteration solve takes 0.390.39 s per one-vs-rest task, a small fraction of that, whereas a single 500k-iteration solve (128128 s) would by itself take longer than the whole epoch. So, the selection of 1k iterations introduces a trade-off.

3. Long-tailed prediction and evaluation protocol (W3, Q3).

We answer the three questions directly.

Are the one-vs-rest outputs converted into a single class prediction? No. Each class cc defines its own binary task, in which class cc is normal and all other classes are anomalous, following the common procedure in the literature. Detector cc accepts a test sample when fc(x)≤rc⋆2f_c(\mathbf x)\le r_c^{\star2}. Because no multi-class prediction is formed, conflicting decisions between detectors do not enter the reported metric.

Exact formula: BAccc=12(TPRc+TNRc)\mathrm{BAcc}_c=\tfrac12(\mathrm{TPR}_c+\mathrm{TNR}_c), where TPRc\mathrm{TPR}_c is the fraction of class-cc test samples accepted and TNRc\mathrm{TNR}_c the fraction of all other test samples rejected. The reported value is 1C∑cBAccc\tfrac1C\sum_c\mathrm{BAcc}_c.

Were the multi-class baselines evaluated with the same protocol? No. Their numbers are taken from the original papers and are the standard multi-class top-1 accuracy on the balanced test sets, i.e., the macro-averaged class recall. The two groups are therefore not evaluated on the same task. For consistency with prior work, we followed the protocol of the fixed-feature method (Rahimzadeh Arashloo, 2024), which reports the two groups side by side, but the reviewer is right that the submitted text presented this as a like-for-like comparison. The revision defines both protocols in Sec. 4.1, separates the two groups in Table 2, and restricts the claim to the comparison among the one-vs-rest SVDD-based detectors.

4. Class-disjoint anomalies and the role of ν\nu (W3, Q4).

Class-disjoint evaluation: For each CIFAR-10 normal class, the remaining nine classes are split into four seen and five held-out anomaly classes. Training, validation, model selection, and early stopping use only the normal class and the seen anomaly classes. The held-out classes appear only at test time. The split is identical for the three variants within each seed. Five seeds:

MethodAUROC ↑\uparrowAUPR-In ↑\uparrowFPR95 ↓\downarrowBAcc ↑\uparrow
Fixed0.98393±0.003540.98393\pm0.003540.93707±0.01543\mathbf{0.93707\pm0.01543}0.07340±0.016920.07340\pm0.016920.82814±0.022530.82814\pm0.02253
Primal0.98540±0.005150.98540\pm0.005150.93607±0.023890.93607\pm0.023890.06145±0.018700.06145\pm0.018700.85807±0.030540.85807\pm0.03054
Dual0.98706±0.00445\mathbf{0.98706\pm0.00445}0.93566±0.023990.93566\pm0.023990.04622±0.01533\mathbf{0.04622\pm0.01533}0.86226±0.02886\mathbf{0.86226\pm0.02886}

(Part 4/4)

Three observations:

  1. Detection of unseen anomaly classes is harder than the standard protocol (AUROC 0.9870.987 instead of 0.9970.997), as expected.
  2. The ordering fixed << primal << dual is preserved in AUROC, FPR95, and balanced accuracy, and FPR95 is lower for the dual update than for the primal update in all five matched seeds (a relative reduction of 24.8%24.8\% on average, and 37.0%37.0\% relative to the fixed backbone). The mean AUROC differences are smaller than the across-seed standard deviation, so we do not claim an AUROC improvement here, and AUPR-In is essentially equal for the three variants.
  3. This evaluation is now Appendix A.4 of the paper, together with published CIFAR-10 results of cited methods whose test anomaly classes are absent from training. Under held-out classes, the dual model (0.9870.987) is above all of them except PANDA with outlier exposure (0.9890.989), which trains on an external outlier dataset. Those methods use other backbones and all nine anomaly classes at test time, so this is a reference point and not a matched comparison.

Selected ν\nu and the case ν=1\nu=1: A single configuration, p=100p=100, ν=10\nu=10, c~1=10\tilde c_1=10, c~2=0.01\tilde c_2=0.01, is used for all datasets, exposure ratios, and protocols. It was chosen in a preliminary search on the validation splits of all datasets at 10% exposure, where it was the best setting on most of them, and was then kept fixed instead of being re-tuned per dataset. The value ν=1\nu=1 was therefore not used for any reported result except as one point of the sensitivity study below. The only other runs at ν=1\nu=1 are the industrial one-class experiments requested by another reviewer, whose training sets contain no anomalies, and they are reported as such. The written statement is correct: at ν=1\nu=1 the anomalous dual mass (ν−1)/2(\nu-1)/2 is zero, anomalies influence neither the boundary nor the encoder update, and our non-collapse argument does not apply. The revision states this. At ν=10\nu=10 the class masses are 5.55.5 for normal and 4.54.5 for anomalous support vectors, so anomalous samples carry 45%45\% of the total dual mass in the encoder gradient at every exposure ratio. What the exposure ratio changes is the pool of anomalous samples from which the support vectors are selected. The new sensitivity study (Figure 2, CIFAR-10 at 10% exposure) confirms this role: test AUROC is 0.9750.975 at ν=1\nu=1, below the fixed backbone (0.9950.995), rises to 0.9970.997 at ν=10\nu=10, and stays there at ν=20\nu=20 and ν=50\nu=50.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.