- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes ProtoSeam, a training method for supervised classification that splits a network at the penultimate representation layer ("the seam") and introduces one learnable prototype per class. During training, no gradient crosses from into . Instead, and are trained via a quadratic consensus penalty (plus an exponential repulsion penalty on ), while and are trained on Gaussian samples using the standard classification loss, where is computed empirically once per epoch and detached. At inference time, the prototypes are dropped and the reconnected composition is evaluated directly. The authors evaluate ProtoSeam on CIFAR-10, CIFAR-100, and TinyImageNet using ResNet-8 and a small ViT-S, comparing against a standard baseline and an unlifted control (which shares the bottleneck linear layer of ProtoSeam). An appendix provides a /Bures-Wasserstein risk-transfer bound, a second-order Taylor expansion showing that sampling acts as a -weighted trace-Hessian regularizer on , and a 1D cubic toy example
Critical strengths
-
Clean formulation and zero inference overhead: Decoupling the backbone and classifier head at a single penultimate interface via class-conditional Gaussian sampling is simple to implement (Algorithm 1), and reconnecting at test time adds no deployment overhead over the unlifted bottleneck architecture.
-
Transparent risk-transfer accounting: The derivations in Appendix B.1 and B.2 (bounding the deployment risk mismatch via and Bures-Wasserstein distance, and linking the Gaussian sampling step to second-order trace-Hessian curvature regularization of the linear head) are clean, easy to verify, and explicitly clarify the interplay between the variance floor and weight decay on .
Critical weaknesses
Confounded experimental protocol, weak baselines, and missing ablations/comparators:
-
Asymmetric hyperparameter tuning: According to Remark 4.1 and Appendix A, the authors ran a 20-point grid search over for
BaselineandProtoSeam, selecting the best configuration by validation accuracy, but did not tune theUnliftedcontrol (simply copying theBaselinehyperparameters). BecauseUnliftedreplaces the final projection with an unregularized rank- linear factorization (, which is a severe rank bottleneck on CIFAR-100 with and TinyImageNet with ), its optimization dynamics differ substantially fromBaseline. Indeed, in Table 1,Unlifteddrops by 2.52% on CIFAR-100 and 4.86% on TinyImageNet (ResNet-8) relative toBaseline. Leaving the primary architectural control untuned invalidates the claim that the comparison is under a "shared tuning protocol". -
Missing component ablations and natural baselines: ProtoSeam simultaneously introduces (a) the stop-gradient seam, (b) the quadratic center penalty, (c) the inter-class repulsion , and (d) Gaussian neighborhood sampling. There is zero ablation disentangling these four pieces. If you keep the exact same objective (5) without cutting the gradient (or compare directly against Center Loss / L-GM loss / Fixed ETF head / penultimate Gaussian noise injection under the same tuning), how much of the gain comes from lifting versus simply adding a center + repulsion auxiliary loss to a sub-optimally trained ViT-S (which only reaches 46.96% on TinyImageNet and 65.91% on CIFAR-100)? Furthermore, in Section 3 (lines 303-307), the authors admit that replacing the per-epoch empirical covariance with a simple scaled identity matrix "produced similar results to the empirical covariance computation," which directly undermines the necessity of the latent GMM covariance estimation that headlines the paper.
Disconnect between the primary area (Learning Theory), the main text, and the actual architecture:
-
Although submitted under the Learning Theory primary area, the entire 9-page main text contains no theorems, propositions, or formal analysis, relegating all theoretical claims to Appendix B (which is also explicitly disclosed in the AI Use Statement as formulated and proved using generative AI).
-
More importantly, the theoretical mechanism in Appendix B does not explain why cutting the gradient across the seam improves over end-to-end training with the same loss. Theorem B.3/B.4 merely bounds the risk degradation from discarding the prototypes at test time (a bound that is vacuous as grows on CIFAR-100/TinyImageNet, as acknowledged in Eq. 7). Proposition B.5 shows that sampling around prototypes adds , which is standard Tikhonov/vicinal noise regularization on a convex linear head (Bishop, 1995). Finally, Proposition B.6 (the 1D solvable model) requires replacing the linear head with a degree-3 polynomial head under MSE loss so that the head can flatten its derivative at the prototypes while still interpolating . In the actual experiments, is strictly a single linear layer : a linear function has constant Jacobian everywhere and cannot locally flatten around without globally shrinking (which is just standard weight decay).
Questions for authors
-
Ablating the gradient cut and loss components: What happens if you train the exact same objective (Eq. 5) end-to-end (allowing gradients from to flow through back into ), or if you compare against standard Center Loss (Wen et al., 2016) + repulsion under the exact same hyperparameter search? Without this ablation, how can we know that lifting/decoupling is beneficial rather than the auxiliary clustering loss itself?
-
Empirical vs. isotropic covariance: In lines 303-307, you state that using a scaled identity matrix instead of the empirical covariance yields similar accuracy at virtually zero overhead. Can you report the exact numbers in Table 1 for isotropic covariance (or ) versus full empirical covariance ?
-
Fair tuning of
Unliftedand rank bottleneck : Why wasUnliftednot given the same 20-point grid search asBaselineandProtoSeam? Additionally, since forces a rank-32 bottleneck on the 100-class and 200-class logits inProtoSeamandUnlifted(whereasBaselinehas full rank), what happens when matches the native penultimate feature dimension of ResNet-8 / ViT-S? -
Missing implementation details: What is the exact value of the variance floor and used across the experiments in Table 1? They are required inputs to Algorithm 1 and central to Corollary B.4 / Proposition B.5, but appear to be omitted from Section 4 and Appendix A.
3 comments
-
W2.2 Replacing the empirical covariance by an isotropic one is a heuristic that works well for the selected models and datasets because they learn well-separated small clusters for every class. We believe that this is not a flaw in our approach but a special case where it can be simplified. As mentioned in the paper, a plausible idea would be to choose it as , which depends on choosing this hyperparameter correctly. For a fair comparison we chose an isotropic covariance given by and ran a hyperparameter search on CIFAR-10, CIFAR-100, and TinyImageNet for both models. The results are summarized in the following table:
Dataset Model Covariance Seed 42 Seed 43 Seed 44 Mean Spread CIFAR-10 ResNet-8 empirical 95.27 95.54 95.57 95.46 0.30 95.36 95.45 95.54 95.45 0.18 ViT-S empirical 90.64 90.34 90.17 90.38 0.47 92.22 92.06 92.29 92.19 0.23 CIFAR-100 ResNet-8 empirical 78.18 78.21 78.63 78.34 0.45 78.03 77.90 78.51 78.15 0.61 ViT-S empirical 70.48 71.00 70.12 70.53 0.88 70.66 70.81 70.08 70.52 0.73 TinyImageNet ResNet-8 empirical 65.64 65.48 65.69 65.60 0.21 65.30 65.68 65.23 65.40 0.45 ViT-S empirical 51.71 51.88 52.78 52.12 1.07 54.35 53.37 53.20 53.64 1.15 We observe that both variants perform almost identically. The only instances where the isotropic covariance wins is for ViT-S and CIFAR-10 / TinyImageNet, where it selects hyperparameters for which the empirical covariance crashed. These crashes of ProtoSeam occur only at the beginning of the optimization, where the clusters are close and the penalties are large. They could be avoided by using gradient clipping. This heuristic no longer works in our large-scale investigation (see our response to W4 of reviewer nLe1). There, we observe that some classes in latent space actually have multiple clusters.
-
W3 Theory in the appendix: The relegation of the theoretical part to the appendix is due to space constraints. As mentioned above, cutting the gradient helps against ill-conditioned derivatives. Since this is a numerical effect, it does not allow for a compact theoretical investigation. Out of the given categories, we believed learning theory to fit our submission best.
We found that proving the theoretical feasibility of our lifted approach (Appendix B.1) was the most important theoretical aspect. Consequently, we based our further investigation of the improved accuracy on the theory of Appendix B.1.
Our main reason for using AI in the appendix was to construct a minimal example where a lifted variant finds a different and better minimizer and then to show this theoretically. The authors have a background in multivariate statistics, carefully read the theoretical results, and take full responsibility for their correctness. Furthermore, we are manually rewriting the appendix to improve its clarity. We agree that the setting of the degree-three polynomial does not transfer to a single-layer . However, we found it valuable to have a minimal example for this effect, which transfers to larger multi-layer because they can approximate polynomials.
Questions:
-
Q1 Ablating the gradient cut and loss components: Please see our response to W2 above.
-
Q2 Empirical vs. isotropic covariance: Please see our response to W2 above.
-
Q3 Fair tuning of Unlifted and rank bottleneck 32: The hyperparameter search of the unlifted variant is discussed in our response to W1. Regarding the performance of the unlifted variant with respect to , we observed different patterns for ResNet-8 and ViT-S. For ResNet-8, there is a clear trend of improved performance when increasing up to the number of classes, where it essentially matches the baseline. Increasing further leads only to very minor improvements (as seen for CIFAR-10). The ViT-S, on the other hand, seems to benefit from this bottleneck. Here, performance actually drops to the baseline level or slightly above it when increasing to the number of classes and beyond.
-
Q4 Missing implementation details: Thank you for pointing out this oversight. For all experiments we used (except for our ResNet-34 investigation) and chose , increasing it further where needed so that the resulting covariance is positive definite (which is usually only necessary for bad hyperparameter choices).
Summary of changes to the paper
- Ran the full hyperparameter grid search for the unlifted variant and included the comparison with the previous results (W1, Q3).
- Added the "Unlifted + Penalty" ablation, i.e., ProtoSeam without cutting the gradient (W2.1, Q1).
- Comparison of the empirical and the isotropic covariance on all three datasets (W2.2, Q2).
- Rewriting Appendix B to improve its clarity (W3).
- Dependence of the unlifted variant on the embedding dimension (Q3).
- Missing implementation details: and (Q4).
We very much appreciate the time and effort you have put into reviewing our contribution. We hope that the responses below address all weaknesses and questions you raised.
Weaknesses:
W1 Asymmetric hyperparameter tuning: In a preliminary test, the optimal hyperparameters for the unlifted variant matched those of the baseline. We agree, however, that a fair comparison requires tuning them separately, so we have now run the full grid search for the unlifted variant. For CIFAR-10 (both backbones) and TinyImageNet with ViT-S, the optimal hyperparameters are unchanged. For the other three pairs, accuracy improves, but the unlifted variant still performs worse than ProtoSeam:
We have included this investigation in the paper.
W2.1 Missing component ablations and natural baselines: We added a configuration using the same class center variables and repulsion penalty as the ProtoSeam variant, but without the decoupling. Advantages of lifting also include the possibility of parallelization and cheaper backpropagation, but when it comes to accuracy, the main benefit of cutting the gradient becomes visible when N2 has an ill-conditioned Jacobian. For our one-layer N2 this effect is rather small, but we still observe an improvement over only using the penalty. We tested it on TinyImageNet with ResNet-8 (after a hyperparameter search), the largest of the three datasets and the better-known of the two architectures: