acceptodds
Under review as a conference paper at ICLR 2027

ProtoSeam: Lifting Classifier Training with Latent Gaussian Mixture Models

Abstract

We propose a lifted reformulation of supervised classification that improves the final accuracy of standard classifiers without changing the architecture at inference time. A network is split at a single semantic interface and one learnable prototype per class is inserted there. Training combines a quadratic consensus penalty that pulls toward the prototype of its class with a classification loss of evaluated on samples drawn around the prototypes, whereat no gradient crosses the interface. At inference the prototypes are discarded and the unmodified network is used. Across CIFAR-10, CIFAR-100, and TinyImageNet with ResNet and vision transformer backbones, lifted training improves test accuracy by up to five percentage points over variants without lifting under a shared tuning protocol. Moreover, we provide theoretical justification of those results.

Primary area:
learning theory
Keywords:
lifting, CV, CNN, Deep Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zqMfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposes ProtoSeam, a training method for supervised classification that splits a network N=N2∘N1N = N_2 \circ N_1 at the penultimate representation layer ("the seam") and introduces one learnable prototype si∈Rks_i \in \mathbb{R}^k per class. During training, no gradient crosses from N2N_2 into N1N_1. Instead, N1N_1 and SS are trained via a quadratic consensus penalty ρ2∥N1(x)−sy∥22\frac{\rho}{2}\Vert N_1(x) - s_y\Vert_2^2 (plus an exponential repulsion penalty on SS), while N2N_2 and SS are trained on Gaussian samples zi∼N(si,Σ~i)z_i \sim \mathcal{N}(s_i, \tilde{\Sigma}_i) using the standard classification loss, where Σ~i=Σ^i+σ02Ik\tilde{\Sigma}_i = \hat{\Sigma}_i + \sigma_0^2 I_k is computed empirically once per epoch and detached. At inference time, the prototypes are dropped and the reconnected composition N2∘N1N_2 \circ N_1 is evaluated directly. The authors evaluate ProtoSeam on CIFAR-10, CIFAR-100, and TinyImageNet using ResNet-8 and a small ViT-S, comparing against a standard baseline and an unlifted control (which shares the bottleneck linear layer Rk→Rn\mathbb{R}^k \to \mathbb{R}^n of ProtoSeam). An appendix provides a W1W_1/Bures-Wasserstein risk-transfer bound, a second-order Taylor expansion showing that sampling acts as a Σ~i\tilde{\Sigma}_i-weighted trace-Hessian regularizer on N2N_2, and a 1D cubic toy example

Critical strengths
  • Clean formulation and zero inference overhead: Decoupling the backbone and classifier head at a single penultimate interface via class-conditional Gaussian sampling is simple to implement (Algorithm 1), and reconnecting N2∘N1N_2 \circ N_1 at test time adds no deployment overhead over the unlifted bottleneck architecture.

  • Transparent risk-transfer accounting: The derivations in Appendix B.1 and B.2 (bounding the deployment risk mismatch via W1W_1 and Bures-Wasserstein distance, and linking the Gaussian sampling step to second-order trace-Hessian curvature regularization of the linear head) are clean, easy to verify, and explicitly clarify the interplay between the variance floor σ0\sigma_0 and weight decay on W2W_2.

Critical weaknesses

Confounded experimental protocol, weak baselines, and missing ablations/comparators:

  • Asymmetric hyperparameter tuning: According to Remark 4.1 and Appendix A, the authors ran a 20-point grid search over (η,λ)(\eta, \lambda) for Baseline and ProtoSeam, selecting the best configuration by validation accuracy, but did not tune the Unlifted control (simply copying the Baseline hyperparameters). Because Unlifted replaces the final projection with an unregularized rank-kk linear factorization (k=32k=32, which is a severe rank bottleneck on CIFAR-100 with n=100n=100 and TinyImageNet with n=200n=200), its optimization dynamics differ substantially from Baseline. Indeed, in Table 1, Unlifted drops by 2.52% on CIFAR-100 and 4.86% on TinyImageNet (ResNet-8) relative to Baseline. Leaving the primary architectural control untuned invalidates the claim that the comparison is under a "shared tuning protocol".

  • Missing component ablations and natural baselines: ProtoSeam simultaneously introduces (a) the stop-gradient seam, (b) the quadratic center penalty, (c) the inter-class repulsion P(S)\mathcal{P}(S), and (d) Gaussian neighborhood sampling. There is zero ablation disentangling these four pieces. If you keep the exact same objective (5) without cutting the gradient (or compare directly against Center Loss / L-GM loss / Fixed ETF head / penultimate Gaussian noise injection under the same tuning), how much of the gain comes from lifting versus simply adding a center + repulsion auxiliary loss to a sub-optimally trained ViT-S (which only reaches 46.96% on TinyImageNet and 65.91% on CIFAR-100)? Furthermore, in Section 3 (lines 303-307), the authors admit that replacing the per-epoch empirical covariance Σ^i\hat{\Sigma}_i with a simple scaled identity matrix ρ−1Ik\rho^{-1} I_k "produced similar results to the empirical covariance computation," which directly undermines the necessity of the latent GMM covariance estimation that headlines the paper.

Disconnect between the primary area (Learning Theory), the main text, and the actual architecture:

  • Although submitted under the Learning Theory primary area, the entire 9-page main text contains no theorems, propositions, or formal analysis, relegating all theoretical claims to Appendix B (which is also explicitly disclosed in the AI Use Statement as formulated and proved using generative AI).

  • More importantly, the theoretical mechanism in Appendix B does not explain why cutting the gradient across the seam improves over end-to-end training with the same loss. Theorem B.3/B.4 merely bounds the risk degradation from discarding the prototypes at test time (a bound that is vacuous as k/mik/m_i grows on CIFAR-100/TinyImageNet, as acknowledged in Eq. 7). Proposition B.5 shows that sampling around prototypes adds 12tr(Σ~iW2⊤H(si)W2)\frac{1}{2}\text{tr}(\tilde{\Sigma}_i W_2^\top H(s_i) W_2), which is standard Tikhonov/vicinal noise regularization on a convex linear head (Bishop, 1995). Finally, Proposition B.6 (the 1D solvable model) requires replacing the linear head with a degree-3 polynomial head ga(z)=z+a(z3−z)g_a(z) = z + a(z^3 - z) under MSE loss so that the head can flatten its derivative ga′(±1)→0g_a'(\pm 1) \to 0 at the prototypes while still interpolating ga(±1)=±1g_a(\pm 1) = \pm 1. In the actual experiments, N2N_2 is strictly a single linear layer W2z+bW_2 z + b: a linear function has constant Jacobian W2W_2 everywhere and cannot locally flatten around sis_i without globally shrinking W2W_2 (which is just standard L2L_2 weight decay).

Questions for authors
  1. Ablating the gradient cut and loss components: What happens if you train the exact same objective (Eq. 5) end-to-end (allowing gradients from Lcls(N2(zi),i)\mathcal{L}_{\text{cls}}(N_2(z_i), i) to flow through zi=N1(x)+Σ~i1/2ξz_i = N_1(x) + \tilde{\Sigma}_i^{1/2}\xi back into N1N_1), or if you compare against standard Center Loss (Wen et al., 2016) + repulsion under the exact same hyperparameter search? Without this ablation, how can we know that lifting/decoupling is beneficial rather than the auxiliary clustering loss itself?

  2. Empirical vs. isotropic covariance: In lines 303-307, you state that using a scaled identity matrix instead of the empirical covariance Σ^i\hat{\Sigma}_i yields similar accuracy at virtually zero overhead. Can you report the exact numbers in Table 1 for isotropic covariance Σ~i=σ02Ik\tilde{\Sigma}_i = \sigma_0^2 I_k (or ρ−1Ik\rho^{-1}I_k) versus full empirical covariance Σ^i+σ02Ik\hat{\Sigma}_i + \sigma_0^2 I_k?

  3. Fair tuning of Unlifted and rank bottleneck k=32k=32: Why was Unlifted not given the same 20-point (η,λ)(\eta, \lambda) grid search as Baseline and ProtoSeam? Additionally, since k=32<nk=32 < n forces a rank-32 bottleneck on the 100-class and 200-class logits in ProtoSeam and Unlifted (whereas Baseline has full rank), what happens when kk matches the native penultimate feature dimension of ResNet-8 / ViT-S?

  4. Missing implementation details: What is the exact value of the variance floor σ0\sigma_0 and ρmin⁡\rho_{\min} used across the experiments in Table 1? They are required inputs to Algorithm 1 and central to Corollary B.4 / Proposition B.5, but appear to be omitted from Section 4 and Appendix A.

3 comments

We very much appreciate the time and effort you have put into reviewing our contribution. We hope that the responses below address all weaknesses and questions you raised.

Weaknesses:

  • W1 Asymmetric hyperparameter tuning: In a preliminary test, the optimal hyperparameters for the unlifted variant matched those of the baseline. We agree, however, that a fair comparison requires tuning them separately, so we have now run the full grid search for the unlifted variant. For CIFAR-10 (both backbones) and TinyImageNet with ViT-S, the optimal hyperparameters are unchanged. For the other three pairs, accuracy improves, but the unlifted variant still performs worse than ProtoSeam:

    DatasetModelVariantSeed 42Seed 43Seed 44MeanSpread
    CIFAR-100ResNet-8Unlifted (baseline HPs)74.1274.0474.5574.240.51
    Unlifted (tuned)75.6175.4975.3075.470.31
    ProtoSeam78.1878.2178.6378.340.45
    ViT-SUnlifted (baseline HPs)65.4965.3864.0464.971.45
    Unlifted (tuned)66.5866.6366.5466.580.09
    ProtoSeam70.4871.0070.1270.530.88
    TinyImageNetResNet-8Unlifted (baseline HPs)57.7058.5158.7558.321.05
    Unlifted (tuned)58.4559.7159.1659.111.26
    ProtoSeam65.6465.4865.6965.600.21

    We have included this investigation in the paper.

  • W2.1 Missing component ablations and natural baselines: We added a configuration using the same class center variables and repulsion penalty as the ProtoSeam variant, but without the decoupling. Advantages of lifting also include the possibility of parallelization and cheaper backpropagation, but when it comes to accuracy, the main benefit of cutting the gradient becomes visible when N2N_2 has an ill-conditioned Jacobian. For our one-layer N2N_2 this effect is rather small, but we still observe an improvement over only using the penalty. We tested it on TinyImageNet with ResNet-8 (after a hyperparameter search), the largest of the three datasets and the better-known of the two architectures:

    DatasetModelVariantSeed 42Seed 43Seed 44MeanSpread
    TinyImageNetResNet-8Unlifted + Penalty65.2064.8865.2165.100.33
    ProtoSeam65.6465.4865.6965.600.21
  • W2.2 Replacing the empirical covariance by an isotropic one is a heuristic that works well for the selected models and datasets because they learn well-separated small clusters for every class. We believe that this is not a flaw in our approach but a special case where it can be simplified. As mentioned in the paper, a plausible idea would be to choose it as 1/ρmax⁡1/\rho_{\max}, which depends on choosing this hyperparameter correctly. For a fair comparison we chose an isotropic covariance given by I/16I/16 and ran a hyperparameter search on CIFAR-10, CIFAR-100, and TinyImageNet for both models. The results are summarized in the following table:

    DatasetModelCovarianceSeed 42Seed 43Seed 44MeanSpread
    CIFAR-10ResNet-8empirical95.2795.5495.5795.460.30
    I/16I/1695.3695.4595.5495.450.18
    ViT-Sempirical90.6490.3490.1790.380.47
    I/16I/1692.2292.0692.2992.190.23
    CIFAR-100ResNet-8empirical78.1878.2178.6378.340.45
    I/16I/1678.0377.9078.5178.150.61
    ViT-Sempirical70.4871.0070.1270.530.88
    I/16I/1670.6670.8170.0870.520.73
    TinyImageNetResNet-8empirical65.6465.4865.6965.600.21
    I/16I/1665.3065.6865.2365.400.45
    ViT-Sempirical51.7151.8852.7852.121.07
    I/16I/1654.3553.3753.2053.641.15

    We observe that both variants perform almost identically. The only instances where the isotropic covariance wins is for ViT-S and CIFAR-10 / TinyImageNet, where it selects hyperparameters for which the empirical covariance crashed. These crashes of ProtoSeam occur only at the beginning of the optimization, where the clusters are close and the penalties are large. They could be avoided by using gradient clipping. This heuristic no longer works in our large-scale investigation (see our response to W4 of reviewer nLe1). There, we observe that some classes in latent space actually have multiple clusters.

  • W3 Theory in the appendix: The relegation of the theoretical part to the appendix is due to space constraints. As mentioned above, cutting the gradient helps against ill-conditioned derivatives. Since this is a numerical effect, it does not allow for a compact theoretical investigation. Out of the given categories, we believed learning theory to fit our submission best.

    We found that proving the theoretical feasibility of our lifted approach (Appendix B.1) was the most important theoretical aspect. Consequently, we based our further investigation of the improved accuracy on the theory of Appendix B.1.

    Our main reason for using AI in the appendix was to construct a minimal example where a lifted variant finds a different and better minimizer and then to show this theoretically. The authors have a background in multivariate statistics, carefully read the theoretical results, and take full responsibility for their correctness. Furthermore, we are manually rewriting the appendix to improve its clarity. We agree that the setting of the degree-three polynomial does not transfer to a single-layer N2N_2. However, we found it valuable to have a minimal example for this effect, which transfers to larger multi-layer N2N_2 because they can approximate polynomials.

Questions:

  • Q1 Ablating the gradient cut and loss components: Please see our response to W2 above.

  • Q2 Empirical vs. isotropic covariance: Please see our response to W2 above.

  • Q3 Fair tuning of Unlifted and rank bottleneck 32: The hyperparameter search of the unlifted variant is discussed in our response to W1. Regarding the performance of the unlifted variant with respect to kk, we observed different patterns for ResNet-8 and ViT-S. For ResNet-8, there is a clear trend of improved performance when increasing kk up to the number of classes, where it essentially matches the baseline. Increasing kk further leads only to very minor improvements (as seen for CIFAR-10). The ViT-S, on the other hand, seems to benefit from this bottleneck. Here, performance actually drops to the baseline level or slightly above it when increasing kk to the number of classes and beyond.

  • Q4 Missing implementation details: Thank you for pointing out this oversight. For all experiments we used ρmin⁡=1\rho_{\min} = 1 (except for our ResNet-34 investigation) and chose σ0=10−6\sigma_0 = 10^{-6}, increasing it further where needed so that the resulting covariance is positive definite (which is usually only necessary for bad hyperparameter choices).

Summary of changes to the paper

  • Ran the full hyperparameter grid search for the unlifted variant and included the comparison with the previous results (W1, Q3).
  • Added the "Unlifted + Penalty" ablation, i.e., ProtoSeam without cutting the gradient (W2.1, Q1).
  • Comparison of the empirical and the isotropic covariance on all three datasets (W2.2, Q2).
  • Rewriting Appendix B to improve its clarity (W3).
  • Dependence of the unlifted variant on the embedding dimension kk (Q3).
  • Missing implementation details: ρmin⁡\rho_{\min} and σ0\sigma_0 (Q4).
Reviewer_nLe1OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an alternative pipeline to train a classification model. Instead of training a model end-to-end with CE loss, the authors proposed to train the backbone and the head separately. The backbone is trained with a ``consensus'' loss and the head is trained with classification losses. One design is to break the gradient between the head and the backbone. The authors also introduce a loss to regularize the cluster centers.

Critical strengths

The algorithm seems to be sound and adds nearly no cost compared to normal training, while achieving better performance. It also shows some potential for larger scale applications. The algorithm seems to be principled as well with designs justified by some theories. Overall the paper is also well written and overall the technicals seems sound.

Critical weaknesses
  • The robustness of the algorithm is not well studied. My biggest concern is that there is not much ambiguity in these images nor noise that might cause the cluster center to easily collapse. For example, can we try CIFAR-10-C, or some fine-grained classification datasets, that can stress-test the center loss?

  • Need to quantitatively demonstrate how far the cluster centers move during training to justify some designs. For example, distribution of the centers to show whether FSInit is useful; need to visualize/measure whether the centers are indeed pushed away after having the penalty term; need to visualize/measure whether the learned centers indeed represent the class centers, by maybe doing nearest neighbors as classifiers. These are helpful for authors to justify the design / components of their algorithms.

  • The motivation of the algorithm is partial due to Jacobian of N2, but it seems that the authors only mention it in the introduction but then never revisit this point, and instead more focus on changing the objective of N1 from CE to MSE loss.

  • It is not obvious whether this algorithm can be scaled up to larger models. The authors stop the experiments at ViT-S scale with datasets of smaller-to-medium sizes, so it'd be helpful if experiments with larger scales were available. Also the authors should also check whether ViT-S and ResNet-8 are underfit.

Questions for authors

Please address the weakness point above, esp. on the robustness and more measurements.

3 comments

We very much appreciate the time and effort you have put into reviewing our contribution. We hope that the responses below address all weaknesses you raised.

Weaknesses:

  • W1 Robustness: Thank you for bringing this dataset to our attention. We believe that this aspect is also covered by our investigation in response to W4 below. Nevertheless, we agree that CIFAR-10-C would be an interesting comparison and will run the computations if time permits.

  • W2 Visualization: This is an interesting question. We believe that it is best to showcase this for the same setting as Figure 1. We created an illustration and will include it in the paper: https://anonymous.4open.science/r/ProtoSeam-material-5A10/class_centers_mnist.png

    We use FSInit and observe that the clusters start very close to each other, owing to the random initialization of the network weights. The clusters then drift apart while keeping their semantic arrangement intact (e.g., "6" close to "0", or "3" next to "5").

  • W3 Jacobian of N2N_2: We agree that this point deserves more emphasis. The reason for not focusing further on the Jacobian of N2N_2 is that N2N_2 consists of only one layer in our examples, so this effect is rather small. We focused on these small examples to make sure we could cover a wide range of hyperparameters, and for those one layer was sufficient. Nevertheless, the effect is still visible when comparing the ProtoSeam variant with its unlifted equivalent with all the same penalties (i.e., the same setting but without cutting the gradient):

    DatasetModelVariantSeed 42Seed 43Seed 44MeanSpread
    TinyImageNetResNet-8Unlifted + Penalty65.2064.8865.2165.100.33
    ProtoSeam65.6465.4865.6965.600.21

    Table: Multi-seed confirmation (seeds 42/43/44): test accuracy (%) for the unlifted version with added penalties and the lifted version for ResNet-8 on TinyImageNet.

    The effect will only increase for larger N2N_2, which might be required for the extensions suggested in W4 below. We will include this discussion in the results section.

  • W4 Scaling to larger models: To address the concerns regarding scalability, we trained ResNet-34 on ImageNet. Due to the limited rebuttal period, we took the learning rate and weight decay commonly used for the baseline in the literature (lr = 0.1, wd = 1e-4) and used them for all variants.

    Our modeling assumption is that N1N_1 can map each class to a single cluster. On ImageNet, ResNet-34 does not achieve this for every class. For example, the output of N1N_1 for class 592 (hard disk) shows two distinct clusters: one for disks inside a computer and one for disks outside of it (see https://anonymous.4open.science/r/ProtoSeam-material-5A10/class_0592_pc1_hist.png).

    On full ImageNet, the lifted ResNet-34 falls behind the baseline for all values of kk and ρ\rho that we tried: ~73.2% for the baseline vs. ~71.3% for the best lifted variant. The picture reverses, however, when we restrict ImageNet to the 294 classes whose latent distribution exhibits a single cluster. (We examined a trained lifted network and excluded classes where a second cluster makes up at least 15% of the samples and the distance between the two cluster means exceeds a certain threshold; we added the exact procedure to the appendix.) We call this subset GaussianImageNet; it is still the largest dataset in our evaluation and has a higher resolution than the others. Training ResNet-34 on it without further tuning of lr and wd yields:

    Variantlrwdkkρ\rho (min → max)Seed 42Seed 43Seed 44Mean
    Baseline0.11e-4––85.9285.8785.6585.81
    Unlifted0.11e-464–85.5985.2285.4885.43
    ProtoSeam0.11e-4640.001 → 3286.6286.6286.6786.64

    We did not perform a full search over kk and ρ\rho; we only decreased ρmin⁡\rho_{\min} to avoid an exploding penalty term at the start of training.

    These results suggest a natural extension of our model: using the empirical covariance, one can detect classes with multiple clusters and introduce additional class means for them, i.e., model each class as a Gaussian mixture. This may also require a more expressive N2N_2. We consider this extension beyond the scope of the present paper.

    Potential underfitting: We agree that this is an important concern. For this reason, we doubled the length of our initial training runs from 100 to 200 epochs. We also ran some configurations for 400 epochs, which showed no further improvement in accuracy. We will include these remarks in the paper.

Summary of changes to the paper

  • Added the "Unlifted + Penalty" ablation, i.e., ProtoSeam without cutting the gradient (W3).
  • Added the selection procedure for the GaussianImageNet classes to the appendix (W4).
  • Illustration of the prototype trajectories during training (W2).
  • Discussion of the Jacobian of N2N_2 in the results section (W3).
  • ResNet-34 results on ImageNet and GaussianImageNet, including the multi-cluster observation (W4).
  • Remarks on potential underfitting (200 vs. 400 epochs) (W4).
AuthorsOpenReview ↗

CIFAR-10-C investigation

We investigated the test accuracy on CIFAR-10-C using our networks trained on CIFAR-10 with their respective best hyperparameters. We use the standard benchmark metric, C-15, which is the mean accuracy over the 15 main corruptions and all 5 severities:

  • noise: Gaussian, shot, impulse
  • blur: defocus, glass, motion, zoom
  • weather: snow, frost, fog, brightness
  • digital: contrast, elastic, pixelate, JPEG

The 4 remaining corruptions (speckle noise, Gaussian blur, spatter, saturate) are reported separately as C-extra.

We plot the accuracy over the severity levels (https://anonymous.4open.science/r/ProtoSeam-material-5A10/cifar10c_severity.png) and give a tabular summary below. The table reports the test accuracy in %, as the mean over 3 seeds with the spread (max − min) in parentheses. The column "rel." is the ratio of C-15 to clean accuracy, i.e., the fraction of clean accuracy retained under corruption. Please note that we examined all crashed (with NaN values) runs during our hyperparameter search again. We discovered that two ViT-S runs for ProtoSeam were run with sub-optimal hyperparameters (among them ProtoSeam with lr=0.01 and wd=1.e-4 below).

ModelVariant(lr, wd)CleanC-15Severity 1Severity 5C-extrarel.
ResNet-8Baseline(0.02, 1e-3)94.91 (0.32)78.61 (0.56)90.0162.1582.000.828
Unlifted(0.02, 1e-3)95.16 (0.38)78.76 (0.47)90.0862.3082.180.828
ProtoSeam(0.05, 5e-4)95.46 (0.30)79.58 (1.22)90.7563.6883.200.833
ViTBaseline(0.02, 1e-4)89.43 (0.16)73.23 (0.34)83.6060.7376.210.819
Unlifted(0.02, 1e-4)90.05 (0.89)74.06 (1.72)84.3461.4276.980.822
ProtoSeam(0.01, 1e-4)91.42 (0.38)76.20 (0.41)86.2464.0279.030.834

We observe that ProtoSeam is the most robust of all variants, both in absolute accuracy at every severity and in relative accuracy (last column). We will add this investigation to the paper.

Reviewer_8KmzOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes ProtoSeam, a training method for classifiers in which the network is split into a feature extractor and a linear classification head, with one learnable prototype per class inserted between them. The feature extractor is trained to map each sample to a region near its class prototype, while the head is trained on Gaussian samples drawn around the prototypes.

Critical strengths

I find the idea of viewing a classifier as the composition of two networks, trained jointly but with separate objectives (one building class-structured embeddings, the other classifying them), interesting, even if not entirely new (cf. [1]). Moreover, the method does not change the architecture at inference time, so it introduces no additional cost once training is completed.

[1] Kang, Bingyi, et al. "Decoupling representation and classifier for long-tailed recognition." arXiv preprint arXiv:1910.09217 (2019).

Critical weaknesses
  1. The paper appears to be incomplete: The experiments compare ProtoSeam only with a standard network and with an untuned, unlifted variant. The closest methods, discussed by the authors themselves in Related works, are not evaluated. The results section is very short (8 rows + 1 table), several references in the appendix point to theorems that do not exist (e.g., Theorem B1/Theorem B2), and the AI use statement is internally inconsistent (generative AI is used “to formulate and prove the mathematical claims”, but not for “developing theoretical models”). Finally, the method combines several ingredients (removal of the gradient path, Gaussian sampling, repulsion, penalty annealing), but none of them is ablated.

  2. Figure 3 reports test accuracy for sweeps over kk and ρmax\rho_{\text{max}}, which are then used to motivate the choice of k=32k=32 and ρmax=16\rho_{\text{max}}=16 for the remaining experiments. This raises concerns about possible test-set involvement in model selection. Moreover, while Appendix A states that the learning rate and weight decay are selected based on validation accuracy, the construction of the validation splits is not described.

  3. The extensive theoretical appendix provides limited support for the central empirical claim. The risk-transfer bound controls the discrepancy between the sampled objective and deployed risk, but does not establish why lifting should outperform end-to-end training.

Questions for authors
  1. The AI use statement reports that generative AI tools were used “to formulate and prove the mathematical claims in the appendix”, as well as for feedback on methodology, interpretation of results, and code. Since the theoretical results are presented as one of the three contributions of the paper, it is not clear to me which parts of this contribution were developed by the authors themselves. Could the authors clarify their own role in the theoretical part, explain how the AI-generated proofs were verified, and reconcile this with the statement that AI tools were not used for “developing theoretical models”?

  2. ProtoSeam combines two mechanisms: (A) removing the gradient path between N1 and N2, and (B) training N2 on Gaussian samples rather than the actual embeddings, which introduces a regularization effect. The paper attributes the improvements primarily to (A), but they could also arise solely from (B). Could the authors include a simpler baseline that retains end-to-end training while adding Gaussian noise to N1(x) before passing it to N2?

  3. The experiments use a ResNet-8 and a custom small ViT, whose architectural configurations are not described in sufficient detail. Moreover, ViT is trained from scratch on small datasets, a setting in which vision transformers can struggle without appropriate training recipes and regularization. Its baseline accuracy is relatively low, and the largest improvements are observed precisely with this model. This raises the question of whether ProtoSeam mainly benefits relatively weakly trained architectures. Could the authors provide the full architectural specifications and evaluate the method on stronger, standard backbones (e.g., ResNet-18/50 or Wide ResNet), to establish whether the improvements persist under more competitive settings?

  4. How much do the class prototypes move from their initialization, and does ProtoSeam improve the quality of the embeddings produced by N1 compared to the standard baseline, as measured by nearest-class-mean or kNN accuracy?

6 comments

We very much appreciate the time and effort you have put into reviewing our contribution and hope that the responses below address all weaknesses and questions you raised.

Thank you for pointing us to Kang et al., who first train the network end-to-end and only afterwards retrain the classifier on frozen features. ProtoSeam trains both parts simultaneously with separate objectives and no gradient across the interface. We will include this in the related work section.

Weaknesses:

  • W1.1 Untuned unlifted variant. In a preliminary test, the optimal hyperparameters for the unlifted variant matched those of the baseline. We agree, however, that a fair comparison requires tuning them separately, so we have now run the full grid search for the unlifted variant. For CIFAR-10 (both backbones) and TinyImageNet with ViT-S, the optimal hyperparameters are unchanged. For the other three pairs, accuracy improves, but the unlifted variant still performs worse than ProtoSeam:

    DatasetModelVariantSeed 42Seed 43Seed 44MeanSpread
    CIFAR-100ResNet-8Unlifted (baseline HPs)74.1274.0474.5574.240.51
    Unlifted (tuned)75.6175.4975.3075.470.31
    ProtoSeam78.1878.2178.6378.340.45
    ViT-SUnlifted (baseline HPs)65.4965.3864.0464.971.45
    Unlifted (tuned)66.5866.6366.5466.580.09
    ProtoSeam70.4871.0070.1270.530.88
    TinyImageNetResNet-8Unlifted (baseline HPs)57.7058.5158.7558.321.05
    Unlifted (tuned)58.4559.7159.1659.111.26
    ProtoSeam65.6465.4865.6965.600.21
  • W1.2 Missing comparisons with the closest related methods. We agree and have added the ablation you suggested (see Q2) and more (see W2 for reviewer zqMf).

  • W1.3 Results section. We have extended the discussion in the results section, including the ablations described below.

  • W1.4 Incorrect cross-references in the appendix. Thank you for catching these misleading references. They were generated with the LaTeX command \cref, which rendered them as "Theorem B.1" and "Theorem B.2" instead of "Assumption B.1" and "Remark B.2". We have corrected this in the revised version.

  • W1.5 Inconsistent AI use statement. We refer to our response to Q1.

  • W2 Sweep over kk and ρmax⁡\rho_{\max}: Thank you for pointing out this concern. All hyperparameters were tuned on validation accuracy only; the test accuracies in Figure 3 are shown purely for illustration. To avoid any misunderstanding and for full transparency, we have added the validation accuracies to the plot (updated plot: https://anonymous.4open.science/r/ProtoSeam-material-5A10/vit_c100_sweeps_val.png). We have also added the following sentence to Section 4.2: “For CIFAR-10, CIFAR-100, and TinyImageNet, we held out 10% of the training set as a validation set.”

  • W3 Theoretical part: We explain our AI usage in the response to Q1. We found that proving the theoretical feasibility of our lifted approach (Appendix B.1) was the most important theoretical aspect.

    Cutting the gradient helps against ill-conditioned derivatives. Since this is a numerical effect, it does not allow for a compact theoretical investigation. We therefore based our further investigation of the improved accuracy on the theory of Appendix B.1. We found it valuable to have a minimal example for the regularization effect of Appendix B.2. This example transfers to larger multi-layer N2N_2, which can approximate polynomials.

Questions:

  • Q1 AI use: We want to be completely transparent regarding the use of AI. All ideas in this paper, including the lifted formulation, the method, the experimental design, and the theoretical questions addressed in Appendix B, originate from the authors. In Appendix B, generative AI tools were used to formulate the mathematical statements and their proofs and to find a minimal example for the regularization effect. The authors have a background in multivariate statistics, carefully read the theoretical results, and take full responsibility for the final content of this work, including all text, claims, and artifacts.

    Usage statement: We were unsure whether the use outlined above can be described as "developing theoretical models". Naturally, we will change the AI statement if you think that it applies.

  • Q2 Two mechanisms: Thank you for this suggestion. We ran experiments adding noise to the unlifted variant at its interface for ResNet-8 on CIFAR-100 (on a single seed):

    σVal (%)Test (%)‖N_1(x)‖
    0 (plain unlifted)76.1075.6118.1
    1/1675.8075.2418.2
    1/875.7875.5218.5
    0.2576.3875.2319.9
    0.576.0675.4623.7
    175.7875.2331.4
    275.8075.9344.3
    474.8274.3465.6
    871.3471.3199.0

    There seems to be little to no benefit to this kind of regularization. The network simply compensates by scaling up the latent output (as illustrated by ‖N_1(x)‖). Moreover, this regularization is of a different kind from that of ProtoSeam. Instead of sampling from a Gaussian distribution with approximately the same covariance, here we can only add noise to every sample individually.

  • Q3 Small ViT architecture: We have added the architecture details of the ViT-S to Appendix A. We use the SimpleViT implementation from https://github.com/lucidrains/vit-pytorch/tree/db05a141a6e3886353a249e64dc9678c6fa30419 with the following settings:

    HyperparameterValue
    patch_size4
    dim512
    depth6
    heads8
    mlp_dim512

    To address the concerns regarding scalability, we trained ResNet-34 on ImageNet. Due to the limited rebuttal period, we took the learning rate and weight decay commonly used for the baseline in the literature (lr = 0.1, wd = 1e-4) and used them for all variants.

    Our modeling assumption is that N1N_1 can map each class to a single cluster. On ImageNet, ResNet-34 does not achieve this for every class. For example, the output of N1N_1 for class 592 (hard disk) shows two distinct clusters: one for disks inside a computer and one for disks outside of it (see https://anonymous.4open.science/r/ProtoSeam-material-5A10/class_0592_pc1_hist.png).

    On full ImageNet, the lifted ResNet-34 falls behind the baseline for all values of kk and ρ\rho that we tried: ~73.2% for the baseline vs. ~71.3% for the best lifted variant. The picture reverses, however, when we restrict ImageNet to the 294 classes whose latent distribution exhibits a single cluster. (We examined a trained lifted network and excluded classes where a second cluster makes up at least 15% of the samples and the distance between the two cluster means exceeds a certain threshold; we added the exact procedure to the appendix.) We call this subset GaussianImageNet; it is still the largest dataset in our evaluation and has a higher resolution than the others. Training ResNet-34 on it without further tuning of lr and wd yields:

    Variantlrwdkkρ\rho (min → max)Seed 42Seed 43Seed 44Mean
    Baseline0.11e-4––85.9285.8785.6585.81
    Unlifted0.11e-464–85.5985.2285.4885.43
    ProtoSeam0.11e-4640.001 → 3286.6286.6286.6786.64

    We did not perform a full search over kk and ρ\rho; we only decreased ρmin⁡\rho_{\min} to avoid an exploding penalty term at the start of training.

    These results suggest a natural extension of our model: using the empirical covariance, one can detect classes with multiple clusters and introduce additional class means for them, i.e., model each class as a Gaussian mixture. This may also require a more expressive N2N_2. We consider this extension beyond the scope of the present paper.

  • Q4 Prototype location and embeddings: This is an interesting question. We believe that it is best to showcase this for the same setting as Figure 1. We uploaded an illustration to an anonymous repository: https://anonymous.4open.science/r/ProtoSeam-material-5A10/class_centers_mnist.png

    We use FSInit and observe that the clusters start very close to each other, owing to the random initialization of the network weights. The clusters then drift apart while keeping their semantic arrangement intact (e.g., "6" close to "0", or "3" next to "5"). However, we agree that there is probably still headroom when it comes to different initialization methods.

    Next, we measure how well the classes are separated by the different variants before the classification layer (using ResNet-8 and CIFAR-100). For the lifted and unlifted variants we consider the 32-dimensional bottleneck output of the first network. The end-to-end baseline has no bottleneck, so we use its 512-d pooled feature. Each variant is evaluated at its own best hyperparameters, selected on held-out validation accuracy, with 3 seeds (42/43/44). We compare several metrics of class separation. The arrows next to the metric names indicate whether higher (↑) or lower (↓) values are better.

    VariantTest acc. (%)Fisher ratio ↑Silhouette ↑Davies–Bouldin ↓NN ratio ↑NCM error ↓
    ProtoSeam78.343.71 (0.03)0.272 (0.003)1.311 (0.017)1.570 (0.004)0.227 (0.001)
    Unlifted75.471.27 (0.02)0.073 (0.001)2.703 (0.013)0.795 (0.004)0.260 (0.003)
    Baseline (512-d feature)76.761.17 (0.01)0.115 (0.001)2.284 (0.015)0.947 (0.004)0.232 (0.005)

    Mean over 3 seeds; spread (max − min) in parentheses. The representation learned by ProtoSeam is much better separated than both the baseline and the unlifted variant on every measure and every seed. We will include these results with a description of every metric in the paper.

Summary of changes to the paper

  • Discussion of Kang et al. in the related work section.
  • Ran the full hyperparameter grid search for the unlifted variant and updated its results and hyperparameters (W1.1).
  • Extended the discussion in the results section, including the new ablations (W1.2, W1.3, Q2).
  • Corrected the cross-references in Appendix B (W1.4).
  • Added the validation accuracies to Figure 3 (W2).
  • Added the description of the validation split (W2).
  • Added the architecture details of the ViT-S to Appendix A (Q3).
  • ResNet-34 results on ImageNet and GaussianImageNet, including the multi-cluster observation (Q3).
  • Added the selection procedure for the GaussianImageNet classes to the appendix (Q3).
  • Illustration of the prototype trajectories during training (Q4).
  • Class-separation metrics for all three variants, with a description of every metric (Q4).

Comments on the paper

1 comment

Thank you for the detailed follow-up and for raising the Soundness score. We are glad to hear that we are on the right track, and we are optimistic that we can also address the remaining issues.

  1. Gradient cut vs. penalties. We have now finished the Unlifted + Penalty runs for all six dataset-model pairs that you asked for:

    DatasetModelVariant(lr, wd)Seed 42Seed 43Seed 44MeanSpread
    CIFAR-10ResNet-8Unlifted + Penalty(0.05, 5e-4)95.2795.5095.3295.360.23
    ProtoSeam(0.05, 5e-4)95.2795.5495.5795.460.30
    CIFAR-10ViT-SUnlifted + Penalty(0.01, 2e-4)91.7592.0691.0291.611.04
    ProtoSeam(0.01, 1e-4)91.3591.6491.2691.420.38
    CIFAR-100ResNet-8Unlifted + Penalty(0.02, 2e-3)78.3778.2178.3378.300.16
    ProtoSeam(0.02, 2e-3)78.1878.2178.6378.340.45
    CIFAR-100ViT-SUnlifted + Penalty(0.02, 5e-4)71.2571.8870.0571.061.83
    ProtoSeam(0.01, 5e-4)70.4871.0070.1270.530.88
    TinyImageNetResNet-8Unlifted + Penalty(0.02, 1e-3)65.2064.8865.2165.100.33
    ProtoSeam(0.02, 2e-3)65.6465.4865.6965.600.21
    TinyImageNetViT-SUnlifted + Penalty(0.01, 2e-4)51.5551.8051.8151.720.26
    ProtoSeam(0.02, 2e-4)53.9553.6554.0153.870.36

    On the four CIFAR pairs, ProtoSeam’s gains come mainly from the consensus and repulsion terms. On TinyImageNet, the gradient cut and the sampling strategy give further improvements of +0.50 and +2.15 points for the two models. Even without the gradient cut, the combination of a consensus penalty with inter-class repulsion is novel, as discussed in the Related Work section. We will add these ablation results to the paper and revise our claims about the impact of the gradient cut accordingly.

    We believe this ablation has improved the paper a lot, and we thank you for pushing us in this direction. In addition to the overall improvement of the new method, we can now quantify the impact of the gradient cut: no significant improvement, but also no reduction in accuracy for the investigated data-model pairs. Evaluating the improvement potential in other applications is left to future work.

  2. Empirical vs. isotropic covariance. We agree that a suitably scaled identity gives accuracy similar to the full estimate. Prompted by your comment, we re-examined all crashed (with NaN values) runs during our hyperparameter search again. We discovered that two ViT-S runs for ProtoSeam had used sub-optimal hyperparameters (CIFAR-10 and TinyImageNet, as used in the table above).

    We respectfully disagree, however, that this undermines the analysis. Theorem B.3 holds for any sampling covariance, and the isotropic variant is the special case Σ^i=0\hat\Sigma_i = 0 of Corollary B.4, with σ02\sigma_0^2 equal to the chosen scale. The only change is how the covariance term is read: for the empirical variant it measures the estimation error; for the isotropic variant, it measures the deliberate misspecification BW(Σi,σ02Ik)B_W(\Sigma_i, \sigma_0^2 I_k).

    Section 3 already discusses the isotropic alternative. We chose the empirical estimate for interpretability and to avoid an additional hyperparameter. As mentioned before, the reason for the similar performance is that both backbones learn well-separated clusters with small covariances. We will present the isotropic variant as the cheaper default for this regime.

  3. ImageNet. You are completely right. In general, performing worse on a larger benchmark and then cherry-picking a subset on which the method excels is not scientifically sound. Here, though, the situation is different. We will report performance on both sets in the paper, together with an explanation of why the new method cannot be expected to perform well on datasets involving multiple distributions within one class (such as the aforementioned hard-disk example or images of crabs in the wild versus crabs in a cooking pot). The additional study adds value to our paper: a possible practical shortcoming is transparently reported, also in the spirit of good scientific behavior to avoid bias towards positive results. A simple exclusion rule leading to GaussianImageNet resolves the drop in accuracy and is directly linked to straightforward future extensions of the new method to train models also on ill-classified data.

    Theory. Only the illustrative example in Appendix B.3 uses a cubic head. Theorem B.3, Corollary B.4 and Proposition B.5 are stated for the single linear layer with cross-entropy used in all our experiments (Remark B.2).

Please let us know if there is anything else we can do to improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.