acceptodds
Under review as a conference paper at ICLR 2027

Preserving Pretrained Intra-Class Structure in Fine-Grained Visual Adaptation

Abstract

Vision foundation models often organize samples from the same category into distinct structures in feature space, reflecting variations in viewpoint, pose, appearance, and local details. Modeling such structure is crucial for fine-grained recognition, where subtle inter-class differences coexist with large intra-class variation. Yet downstream fine-tuning is typically driven by category-level supervision and does not explicitly account for this pretrained intra-class organization. We further observe that such structure is category-dependent: some categories form a single compact mode, while others contain several distinct modes. This motivates us to exploit category-specific structure already present in pretrained features. We propose Adaptive Proxy Modeling (APM), which models pretrained intra-class structure with a category-specific set of proxies. For each category, APM estimates the number of visual modes from the spectral structure of its DINOv2 feature graph and constructs corresponding proxies through expectation-maximization. During fine-tuning, these proxies are dynamically updated with the evolving representation and provide structural guidance for downstream learning. Unlike conventional proxy-based methods using a fixed number of freely learned proxies, APM adapts both proxy number and representation to each category without introducing additional trainable parameters. Across fine-grained classification, person re-identification, and vehicle re-identification, APM consistently improves downstream performance without introducing additional trainable parameters. Beyond accuracy, APM also better preserves pretrained representations after fine-tuning, suggesting that effective adaptation can benefit from retaining the category-specific structure already learned by foundation models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
fine-grained visual recognition, prototype-based representation learning, intra-class diversity, vision foundation models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4x5kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper observes category-dependent visual modes in pretrained feature spaces and proposes adaptive proxy modeling to estimate their number via graph Laplacian spectra, constructing adaptive proxies without additional trainable parameters.

Critical strengths

The paper is well motivated, with a novel perspective on category-dependent intra-class structures and a clear, logical narrative.

Critical weaknesses
  1. The experimental comparisons are not entirely fair, particularly for ReID, where the reported gains may largely stem from the stronger backbone. The ablation results also suggest that APM provides limited improvements, while the baseline already outperforms most competing methods.
  2. The experiments lack comparisons with related methods that also use DINOv2 features and proxy-based learning, as discussed in the related work.
  3. Table 4 lacks sufficient baselines. Although the ImageNet-1K results appear promising, the only comparison is with naive full fine-tuning. Methods such as Proxy-FDA, specifically designed to preserve linear-probing performance, are missing, weakening the claim of better representation preservation.
Questions for authors

My main concern is the fairness of the comparisons, as many experimental comparisons are not sufficiently meaningful or clearly justified. I would like to see evaluations on more datasets and more comprehensive, clearly structured ablation studies.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4x5k, W1

Response to W1:

We thank the reviewer for raising this concern. We agree that comparisons across methods with different backbone architectures should be interpreted cautiously. Our main evidence for the contribution of APM therefore comes from controlled comparisons using the same DINOv2 backbone and training setup, as discussed below:

Baseline Strength and Fairness of Comparison. The 2-cls baseline is indeed strong and competitive with several existing ReID methods, whereas the 1-cls baseline remains below many of them. Importantly, APM is evaluated on top of both settings. Under the 1-cls setting, APM improves MSMT17 from 70.9/86.9 to 79.5/91.5 in mAP/Rank-1 and Market-1501 from 85.0/93.0 to 93.3/96.4. Since the backbone and training configuration remain unchanged, these gains isolate the contribution of APM rather than the effect of using a stronger backbone.

APM also improves the stronger 2-cls baseline. On MSMT17, performance increases from 73.3/89.0 to 80.3/91.6, corresponding to gains of +7.0 mAP and +2.6 Rank-1. On Market-1501, where the baseline already reaches 92.0/96.0, APM further improves performance to 93.5/96.9. Thus, APM provides clear gains both when the baseline is relatively weak and when it is already competitive.

MethodMSMT17 mAPMSMT17 Rank-1Market mAPMarket Rank-1
DINOv2 Full FT (1cls)70.986.985.093.0
Ours (1cls)79.5 (+8.6)91.5 (+4.6)93.3 (+8.3)96.4 (+3.4)
DINOv2 Full FT (2cls)73.389.092.096.0
Ours (2cls)80.3 (+7.0)91.6 (+2.6)93.5 (+1.5)96.9 (+0.9)

These controlled comparisons also complement our ablation study, where adaptive proxy modeling outperforms both single-proxy and fixed multi-proxy alternatives under the same backbone and optimization setting. This further supports that the gains come from the proposed adaptive modeling rather than from backbone strength alone.

Further Comparison with DRFormer. To provide a broader comparison with methods using different architectures, we additionally include DRFormer in Table 2 of the revised manuscript. DRFormer combines CLIP and DINOv2 encoders with multiple learnable tokens, while our model uses only a single DINOv2 encoder. Despite this simpler backbone configuration, our 2-cls model achieves higher mAP on both Market-1501 (93.5% vs. 92.9%) and MSMT17 (80.3% vs. 78.7%).

AuthorsOpenReview ↗

Response to Reviewer 4x5k, W2

Response to W2:

Comparison with Methods in Related Work. The table below summarizes which DINOv2-based FGVR methods and proxy-based methods discussed in Related Work we compare with and explains the exclusions.

TypeMethodCompared?Reason
DINOv2-based FGVR methodPDiscoFormer (Aniraj et al., 2024)yes-
DINOv2-based FGVR methodIVPT (Wang et al., 2026b)yes-
DINOv2-based FGVR methodPrompt-CAM (Chowdhury et al., 2025)noUses a frozen DINOv2 backbone for interpretable trait localization, which differs from our full fine-tuning setting
proxy-based methodProxy-FDA (Huang et al., 2025)noUses a different evaluation protocol, and no official public implementation is available for reproducing it under our setting
proxy-based methodOther methods cited in Related WorknoTheir published evaluations use different backbones or task settings
AuthorsOpenReview ↗

Response to Reviewer 4x5k, W3

Response to W3:

Comparison with Existing Preservation Methods on CUB-200-2011. We agree that Table 4 should include stronger preservation-oriented baselines. We therefore add L2-SP and LP-FT, two methods designed to better retain pretrained representations, using DINOv2-B/14 under the same CUB-200-2011 fine-tuning and ImageNet-1K linear-probing protocol.

ConfigurationCUB Acc. (%)ImageNet-1K Linear Probe Acc. (%)Total Gain (pp)
Pretrained-83.6-
Full FT89.2 (+0.0)79.7 (+0.0)0.0
L2-SP90.1 (+0.9)82.4 (+2.7)+3.6
LP-FT90.2 (+1.0)82.6 (+2.9)+3.9
Ours91.4 (+2.2)81.5 (+1.8)+4.0

L2-SP and LP-FT retain more ImageNet-1K linear-probe accuracy, while APM achieves the highest CUB accuracy. Compared with full fine-tuning, APM improves CUB by +2.2 points and ImageNet linear probing by +1.8 points, yielding a total gain of +4.0 points. This indicates that APM does not simply maximize preservation of the pretrained representation; rather, it uses category-specific pretrained structure to improve downstream adaptation while retaining more pretrained knowledge than standard full fine-tuning.

Moreover, thank you for suggesting Proxy-FDA. Proxy-FDA is relevant, but no official public implementation is available for reproduction under the same setting for a fair comparison. We therefore use L2-SP and LP-FT as controlled preservation baselines, while Proxy-FDA is already cited in our manuscript.

Reviewer_aydCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Adaptive Proxy Modeling (APM) to adapt DINOv2 to fine-grained recognition while preserving pretrained intra-class structure. It estimates category-specific proxy numbers through graph eigengaps, constructs proxies using an EM-style procedure, and updates them through batch-to-category matching and EMA. Proxy- and boundary-based losses supplement the task objective. Experiments cover fine-grained classification, person and vehicle re-identification, multiple fine-tuning strategies, and representation-retention evaluation.

Critical strengths
  1. Well-motivated adaptive design. Allocating different numbers of proxies to different categories reasonably addresses heterogeneous intra-class variation. Constructing proxies from pretrained features gives the approach a concrete connection to the representation being adapted.
  2. Consistent downstream improvements. Under full fine-tuning, APM improves CUB, Cars, Dogs, and NABirds by 2.2, 1.8, 0.7, and 2.2 percentage points, respectively. Tables 2 and 5 additionally report competitive person and vehicle ReID results.
  3. Broad compatibility. Table 7 reports improvements across all 20 paired configurations covering five fine-tuning strategies and four datasets, without increasing their trainable-parameter counts. Table 6 also demonstrates gains across DINOv2-S/B/L.
  4. Useful ablations and retention evidence. Adaptive proxies achieve 93.5% mAP versus 92.5% for both single and fixed-three-proxy alternatives (Table 3). ImageNet probing improves from 19.8% to 52.9% after adaptation, providing preliminary evidence of better retained transferability (Table 4).
Critical weaknesses
  1. Inconsistent proxy optimization. Equation (16) uses an unnormalized mean, whereas the stated vMF model requires a unit mean direction. As written, the update can increase the Eq. (14) loss, while normalizing it generally invalidates the stated convex-hull claim.
  2. Insufficient justification of proxy-count estimation. Theorem 2 establishes a positive gap at RR, but does not establish that it is the largest gap selected by Eq. (3). Its converse also requires a spectral-separation assumption stated in Appendix A.2 but omitted from the theorem statement.
  3. The preservation mechanism is not isolated. Selected t-SNE examples and ImageNet probing do not establish retention of specific intra-class relations. Matched initialization/allocation controls and a direct relation-retention metric would help distinguish the proposed explanation from generic regularization.
  4. Experimental protocols need clarification. Tables 2 and 3 use different CLS baselines with 85.0% and 92.0% mAP, but their complete training configurations are unclear. Hyperparameter-selection splits, probing details, key-result variability, and additional computational costs are also insufficiently specified.
Questions for authors
  1. What exact objective and normalization operations are implemented in the proxy updates? Please reconcile Eqs. (13)–(16) and clarify whether correcting the formulation changes the reported results.
  2. Is the largest-eigengap rule intended as a heuristic? Please clarify its search range, stability under resampling, and handling of batches containing very few samples per category.
  3. Does the improvement persist when pretrained initialization and adaptive allocation are separately controlled, including a validation-tuned fixed-RR baseline? Can within-class relation retention be measured directly?
  4. What explains the baseline difference between Tables 2 and 3? Please specify the matched training configurations, model-selection and probing protocols, and additional training cost.
4 comments
AuthorsOpenReview ↗

Response to Reviewer aydC, Q1

Response to Q1/W1:

Implemented Objective and Normalization Operations. The objective is to optimize the vMF mixture objective in Eq. (14), with temperature parameter τp\tau_p. The normalization operations are L2-normalizing the input features and the responsibility-weighted feature mean in each M-step to obtain unit-norm proxies. Consequently, the convex-hull property applies to the mean before normalization, while the final proxy lies on the unit sphere.

Reconciliation of Eqs. (13) – (16). Eq. (13) defines each proxy as the L2-normalized weighted mean of the category features. Eq. (14) specifies the vMF mixture objective with unit-norm proxies and a fixed temperature. Eq. (15) computes the E-step responsibilities. In the M-step, we use these responsibilities to compute the weighted feature mean and L2-normalize it, yielding the optimal unit-norm proxy update in Eq. (16). The convex-hull property applies to the weighted mean before normalization.

Effect of the Correction on the Reported Results. The reported results remain unchanged because the implementation used in our experiments already L2-normalized the responsibility-weighted feature mean in each M-step. The revised equations and Algorithm 1 make this operation explicit.

AuthorsOpenReview ↗

Response to Reviewer aydC, Q2

Response to Q2/W2:

Clarification of the spectral-separation condition in Appendix A.2. The spectral-separation condition in Appendix A.2 is a property of the ideal Laplacian L∗L^*, rather than an additional assumption on the observed graph, and γ\gamma denotes the smallest nonzero eigenvalue of L∗L^*. To clarify this point, we have added an explicit definition of γ\gamma to Theorem 2 and revised Appendix A.2 accordingly.

The Largest-Eigengap Rule is a Heuristic. Theorem 2 establishes a positive gap at R and does not establish that it is the largest gap. The positive gap in Theorem 2 motivates our heuristic of selecting the largest eigengap to estimate the proxy number. And our experiments further support the effectiveness of this heuristic.

Proxy Number Search Range. For the search range, we consider proxy counts from $ 1 $ to $ n-1 $ when the number of category samples n≥2n\ge 2. If n=1n=1, the proxy number is set to $ 1 $.

Stability of Proxy Number Estimation under Resampling. we conduct five random subsampling runs per sampling ratio on CUB-200-2011. The table below reports means$\pm$standard deviations, using proxy number estimated from the full data as the reference.

Sampling ratioCategory agreement with reference (%)Mean absolute differenceCategories below reference (%)Categories above reference (%)
25%62.4 ±\pm 2.50.63 ±\pm 0.0514.4 ±\pm 1.023.2 ±\pm 3.1
50%75.5 ±\pm 2.60.42 ±\pm 0.0512.4 ±\pm 1.112.1 ±\pm 2.1
75%85.7 ±\pm 1.40.25 ±\pm 0.057.6 ±\pm 1.26.7 ±\pm 0.6
90%93.6 ±\pm 1.20.11 ±\pm 0.033.9 ±\pm 1.02.5 ±\pm 0.5

The estimates are stable at higher sampling ratios and closely match the full-data results. Differences at lower sampling ratios are expected, as fewer samples may miss some clusters or change the observed local structure.

Handling of Single-Sample Cases. For any category containing only one sample in a mini-batch, we set the batch proxy number to 1 and use the normalized feature as the proxy, skipping eigen decomposition and EM.

AuthorsOpenReview ↗

Response to Reviewer aydC, Q3

Response to Q3/W3:

Does the improvement persist when initialization and allocation are separately controlled? Yes. In our controlled comparison, both fixed-R and adaptive-R variants use the same pretrained DINOv2 features and the same EM-based proxy construction. The only difference is the proxy allocation strategy: fixed-R assigns the same number of proxies to every category, whereas APM determines R separately for each category. Therefore, the performance difference directly isolates the effect of adaptive allocation while keeping pretrained initialization and proxy construction unchanged.

How does APM compare with different fixed-R choices? We further sweep all feasible fixed values R∈{1,2,3}R\in \{1,2,3\} on CUB-200-2011, where R=3R=3 is the maximum proxy number estimated by APM. The best fixed setting reaches 90.1% accuracy at R=2R=2, whereas adaptive allocation achieves 91.4%.

MetricBaselineFixed (R=1)Fixed (R=2)Fixed (R=3)Adaptive(Ours)
Acc. (%)89.289.990.189.991.4

Direct Measurement of Intra-Class Relation Retention. We add a direct proxy-assignment retention analysis in Section 4.3.2 of the revised manuscript. For each sample, we identify its nearest pretrained proxy and test whether the corresponding EMA-updated proxy remains its nearest proxy after fine-tuning. We compute the retention percentage within each category and then average across categories. APM retains the original proxy assignment for 98.59% of samples across all CUB-200-2011 categories and 94.02% across categories with multiple proxies (R>1R>1). These results provide direct evidence that the intra-class mode assignments identified from pretrained features remain largely stable during APM fine-tuning.

AuthorsOpenReview ↗

Response to Reviewer aydC, Q4

Response to Q4/W4:

We thank the reviewer for pointing out that several experimental details were insufficiently specified. We have clarified the baseline configurations, training and checkpoint protocols, hyperparameter selection, linear probing, result variability, and computational overhead in the revision.

Why do Tables 2 and 3 use different baselines? The difference comes from the CLS configuration. The 1-cls baseline uses a single pretrained CLS feature, while the 2-cls baseline concatenates two CLS features and additionally uses the self-diverse constraint. Both use the same pretrained DINOv2-B/14 backbone; for the 2-cls setting, both CLS tokens are initialized by copying the pretrained CLS token. This explains the different Market-1501 baselines (approximately 85.0% vs. 92.0% mAP), rather than a difference in backbone or training budget.

For matched comparisons, we keep the remaining training settings unchanged: 100 epochs, learning rate $ 3.5 \times 10^{-4}$, input resolution $ 256 \times128$, PK sampling with 8 images per identity, and batch sizes of 64 for Market-1501 and 256 for MSMT17. All reported results use the final-epoch checkpoint.

Hyperparameter Settings. We use a shared hyperparameter configuration within each task family and keep it fixed across datasets. Specifically, we set λRP=3\lambda_{\mathrm{RP}}=3 for FGVC, λRP=4\lambda_{\mathrm{RP}}=4 for ReID, and β=0.2\beta=0.2, τg=0.3\tau_g=0.3, and τp=0.05\tau_p=0.05 throughout the experiments.

ImageNet-1K Linear Probing. All configurations use the same probing protocol. After downstream fine-tuning, we freeze the backbone and train only a linear classifier on the ImageNet-1K training set for 50 epochs, then evaluate on the official ImageNet-1K validation set.

Key-Result Variability Across Three Seeds on Market-1501. We additionally repeat the key Market-1501 comparisons over three seeds. The results show small variation across all four configurations:

ConfigurationmAPRank-1
DINOv2 Full FT - 1cls84.9 ±\pm 0.393.0 ±\pm 0.1
DINOv2 Full FT - 2cls92.1 ±\pm 0.196.2 ±\pm 0.2
Ours - 1cls93.3 ±\pm 0.196.5 ±\pm 0.1
Ours - 2cls93.5 ±\pm 0.196.9 ±\pm 0.1

These results indicate that the improvements are consistent across runs and are not driven by a particular random seed.

Model Checkpoint Selection. We use the final-epoch checkpoint for evaluation.

Computational Cost Evaluation on Market-1501. APM introduces no additional trainable parameters or peak GPU-memory overhead relative to the corresponding 1-cls or 2-cls baseline. Its additional computation comes from one-time proxy initialization and the eigendecomposition and EM-based proxy construction during training:

InitializationEigendecompositionEM, 50 iterationsTraining timePeak GPU memory
Baseline - 1cls---48.5 s/epoch8.28 GB
Baseline - 2cls---59.2 s/epoch9.95 GB
Ours - 1cls23.4 s2.7 ms/iter24.9 ms/iter73.3 s/epoch8.28 GB
Ours - 2cls25.2 s5.5 ms/iter49.7 ms/iter119.0 s/epoch9.95 GB
Reviewer_2EwQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

APM is a method that starts from the observation that pre-trained DINOv2 features form a different number of intra-class visual modes per category, models this structure as a set of category-specific proxies, and preserves it during fine-tuning. For each class token's feature space, the number of proxies is determined by the maximum eigengap of the normalized Laplacian on the RBF graph of all category features (spectral properties supporting this are presented as Theorems 1 and 2), and proxies are constructed as convex combinations of features by fitting a vMF mixture to an EM. During training, the number of proxies and representations are re-estimated for each mini-batch, matched to the existing category proxies using cosine similarity, and updated using an EMA. The loss is calculated by adding the average (weighted by λ_RP) of the CE(L_proxy) for cosine similarity with the nearest proxy and the CE(L_boundary) for the boundary representative updated with the most dissimilar sample within the batch to the task loss; in multi-CLS, the self-diverse constraint of the DC-Former is also used. There are no additional training parameters. CUB·Cars·Dogs·NABirds reported an average of 91.0% (Full FT 89.3), Market-1501 93.5 mAP, MSMT17 80.3 mAP, and VeRi-776 88.3 mAP, showing consistent improvement across five fine-tuning strategies and DINOv2 S/B/L, with adaptive R being superior to single·fixed (R=3) proxy (Table 3a), and Market-1501 maintaining an ImageNet linear-probe of 52.9% (Full FT 19.8%) after fine-tuning.

Critical strengths

It reports consistent improvements across various settings. The five fine-tuning strategies in Table 7 show improvement across all four FGVC datasets, and in Table 6, DINOv2-S/B/L on CUB improves by 1.1/2.2/1.0 pp, respectively. The fact that there are no additional trainable parameters in the same fine-tuning settings is also appealing. In the controlled ablation of Market-1501, the adaptive proxy is 1.0 mAP higher than the fixed R=3, supporting the practical value of categorical proxy allocation.

In addition to downstream performance, it investigates the retention ability of pre-trained representations. In Table 4, ImageNet linear-probe accuracy increases from 19.8% to 52.9% for Full FT. While this does not reach the 83.6% of the pre-trained model, it provides interesting evidence that downstream adaptation and the retention of transferable information can coexist. However, this should be distinguished from results obtained by directly measuring the preservation of specific intra-class structures themselves.

Critical weaknesses

There is a gap between the mathematical description of the core algorithm and the actual rules. Theorem 1 is a standard connected-component result, and the forward direction of Theorem 2 is an application of the Weyl perturbation bound. More importantly, this result merely shows that a positive eigengap is maintained at a specific R, but it does not guarantee that the maximum eigengap location in Eq. (3) is R. For example, two separated 4-node path graphs have a normalized Laplacian spectrum of ((0,0,0.5,0.5,1.5,1.5,2,2)), so they have 2 connected components, but the maximum eigengap is at (r=4). Therefore, maximum-eigengap recovery cannot be derived from the presented theorem. The additional spectral-separation assumption for the inverse direction of Theorem 2 should be included in the explicit conditions of the theorem as well as in the Appendix Remark. Also, Eq. in Appendix B (16) and Algorithm 1 do not describe the L2 renormalization step following the responsibility-weighted arithmetic mean. This differs from the standard vMF mean-direction M-step with fixed concentration and is generally not justified as an M-step that minimizes Eq. (14) even under the convex-hull constraint of Eq. (13). It is necessary to explain whether there is missing normalization in the actual implementation, whether a different objective function is being optimized, or whether heuristic soft clustering is being used.

There is a lack of comparison to attribute improvements and representation preservation to the unique design of APM. The cited Proxy-FDA is a relevant prior study that aligns the neighborhood structure of pretrained/fine-tuned features. Although it is not identical to APM's categorical mode proxy, it is worth comparing directly. It is necessary to compare downstream performance and representation retention under the same conditions with Proxy-FDA or FDA, and simple frozen-teacher feature distillation. Based solely on the linear probing and selected t-SNE cases in Table 4, it is difficult to distinguish whether the improvement is due to the adaptive intra-class structure or general normalization. In ReID as well, since the conditions differ between the single-CLS Full FT of 85.0 in Table 2 and the two-CLS and self-diverse loss baseline of 92.0 in Table 3, the entire increase from 85.0 to 93.5 cannot be interpreted as the contribution of APM. The two-CLS matched gain identified in Table 3 is +1.5 mAP. The presence or absence of APM for each single-/two-CLS setting, identical to the training recipes of the two baselines, must be presented together. This concern is intended to accurately separate the magnitude of the effect from its cause, rather than to deny the reported improvement.

Questions for authors
  1. Explain the rationale that Eq. (16) is the M-step of Eq. (14). Is the proxy L2 normalized in the implementation? If normalized, the vector is generally outside the convex hull of Eq. (13), so it is incompatible with the convex-hull argument; if not normalized, Eq. (14) is linear with respect to the proxy, so the optimal solution under the convex hull constraint is a single feature rather than a weighted average. Which is the case, and what is the actual purpose of optimization?

  2. What additional assumptions are required between the positive gap guaranteed by Theorem 2 and the maximum eigengap selection in Eq. (3)? (Two separate 4-node path graphs have a connection component of 2, but the maximum gap is at r=4.) Also, since RBF affinity assigns positive weights to all pairs, the unconnected graphs of Theorem 1 do not occur in this method. Please clarify whether the maximum eigengap rule should be limited to a heuristic or if additional conditions and verifications should be presented.

  3. Can Proxy-FDA (Huang et al., 2025), L2-SP, LP-FT, and simple pretrained-feature distillation be compared on both the downstream and ImageNet linear-probe sides using the same backbone recipe? Without this comparison, it is impossible to determine whether the retention effect and +1.5 mAP in Table 4 are unique to the adaptive proxy. (Proposal: Separation of contributions from pretrained proxy initialization and EMA updates, and direct measurements such as the nearest pretrained proxy retention rate per sample after fine-tuning.)

  4. What conditions, other than the number of CLS, differ between the Full FT of 85.0 in Table 2 and the baseline of 92.0 in Table 3? Please provide the four cells regarding the presence or absence of single-/two-CLS × APM for the same recipe, along with the standard deviation of at least three seed cycles. A row for 92.0 must be included in Table 2 for the reader to understand the actual gain.

  5. What are the number of mini-batch samples per category and the sampler (ReID is K in PK sampling)? If there is only one sample, W is a zero matrix, so D^{-1/2} and eigengap are undefined; how is this handled? Please report the R^cat distribution for each dataset, the variation of R^cat according to the sample size, and the increase in training time and memory including initialization, eigendecomposition for each iteration, and 50 EM iterations.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2EwQ, Q1

Response to Q1:

Rationale for Eq. (16) as the M-step of Eq. (14). In the M-step, the responsibilities$\gamma_{ir}^k$obtained from the E-step are fixed. The proxy-dependent part of the expected complete-data log-likelihood is then maximized: max⁡{py,rk}∑i=1n∑r=1Rykγirk(py,rk)⊤fik,∥py,rk∥2=1.\max_{\{p_{y,r}^k\}} \sum_{i=1}^n \sum_{r=1}^{R_y^k} \gamma_{ir}^k\left(p_{y,r}^k\right)^\top f_i^k, \left\Vert p_{y,r}^k\right\Vert_2=1. Therefore, the optimal proxy is the L2-normalized responsibility-weighted resultant vector, which gives Eq. (16).

L2 Normalization of Proxies in Implementation. In our implementation, proxies are L2-normalized within every EM iteration and then used to compute the next E-step responsibilities.

Does the normalized proxy lie in the convex hull? In general, no. Before normalization, the responsibility-weighted average lies in the convex hull of the category features. L2 normalization then maps this vector onto the unit sphere while preserving its direction. Therefore, the final proxy satisfies a unit-norm constraint rather than a convex-hull constraint.

The Actual Purpose of Optimization. The optimization fits a vMF mixture to the normalized category features so that different proxies capture distinct intra-class visual modes. Each proxy represents the mean direction of one mixture component rather than a point constrained to remain in the convex hull.

We have revised Eq. (13), explicitly added L2 normalization to Eq. (16) and Algorithm 1, and removed the misleading convex-hull interpretation of the final proxy.

AuthorsOpenReview ↗

Response to Reviewer 2EwQ, Q2

Response to Q2:

Condition for Guaranteed Proxy Number Selection. The eigengap at R must exceed all other gaps in the search range to guarantee that Eq. (3) uniquely selects R. Theorem 2 guarantees a positive gap at R, but not the largest one. Therefore, Eq. (3) serves as a heuristic for estimating the proxy number rather than providing a guaranteed selection.

The Role of Theorem 1. We agree that the disconnected graph in Theorem 1 does not directly occur in our implementation because RBF affinities are positive. Theorem 1 instead defines an ideal reference case in which the underlying modes form disconnected components. Theorem 2 then considers a fully connected graph whose Laplacian is sufficiently close to this ideal reference and shows that the spectral separation at R remains positive under bounded perturbation.

The Maximum Eigengap Rule is a Heuristic. The positive gap in Theorem 2 motivates our heuristic of selecting the largest eigengap to estimate the proxy number. And our experiments further support the effectiveness of this heuristic.

Clarification of the Spectral-Separation Condition in Appendix A.2. The spectral-separation condition in Appendix A.2 is a property of the ideal Laplacian $ L^{*} $, rather than an additional assumption on the observed graph, and γ\gamma denotes the smallest nonzero eigenvalue of L∗L^{*}. To clarify this point, we have added an explicit definition of γ\gamma to Theorem 2 and revised Appendix A.2 accordingly.

AuthorsOpenReview ↗

Response to Reviewer 2EwQ, Q3

Response to Q3:

Comparison with Existing Representation Preservation Methods. As suggested, we compare L2-SP, LP-FT, and simple pretrained-feature distillation using the same backbone recipe, evaluating both CUB-200-2011 accuracy and ImageNet-1K linear-probe accuracy. The results are presented below. Proxy-FDA is excluded due to the lack of an official public implementation for a fair comparison in the same setting.

ConfigurationCUB Acc. (%)ImageNet-1K Linear Probe Acc. (%)Total Gain (pp)
Pretrained-83.6-
Full FT89.2 (+0.0)79.7 (+0.0)0.0
L2-SP90.1 (+0.9)82.4 (+2.7)+3.6
LP-FT90.2 (+1.0)82.6 (+2.9)+3.9
Pretrain-Feature Distillation89.8 (+0.6)82.0 (+2.3)+2.9
Ours w/o EMA Update90.4 (+1.2)81.9 (+2.2)+3.4
Ours91.4 (+2.2)81.5 (+1.8)+4.0

Our method (APM) retains less ImageNet accuracy than L2-SP, LP-FT and simple pretrained-feature distillation but achieves higher CUB accuracy. This is expected, because APM aims not merely to preserve pretrained representations, but to leverage their category-specific intra-class structures to improve downstream discriminative ability. In addition, APM achieves a total gain comparable to that of LP-FT and yields a larger improvement on CUB, consistent with our focus on downstream performance.

The Effect of EMA. As shown in the table above, we separately evaluate the contributions of pretrained proxy initialization and EMA updates. Without EMA, proxies remain fixed in the pretrained feature space and may become misaligned with evolving features during fine-tuning. Although this leads to slightly better feature retention (81.9%), it limits downstream adaptation, resulting in lower CUB accuracy (90.4%).

Direct Measurement of Intra-Class Structure Retention. We directly measure intra-class structure retention after fine-tuning and report the results in Section 4.3.2 of the revised manuscript. For each sample, we identify its nearest pretrained proxy and test whether the corresponding EMA-updated proxy remains its nearest proxy after fine-tuning. We compute the retention percentage within each category and then average across categories. APM retains the original proxy assignment for 98.59% of samples across all CUB-200-2011 categories and 94.02% across categories with multiple proxies ($R>1$). These results provide direct evidence that the intra-class mode assignments identified from pretrained features remain largely stable during APM fine-tuning.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Kind Request for Feedback on Our Rebuttal for Paper 24759

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end with less than two days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.