- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates how to make learned representations of vision models more suitable for low-dimensional projection. The authors propose Projection-Aware Fine-Tuning (PAFT), which introduces an auxiliary branch consisting of a linear bottleneck and a classification head. In addition to the original classification objective, PAFT employs three regularization terms, including prediction consistency, CKA alignment, and neighborhood preservation, to jointly optimize the high-dimensional representations and their low-dimensional projections.
Critical strengths
- Simple and flexible framework. PAFT introduces a lightweight auxiliary bottleneck and classification head without requiring substantial modifications to the original classifier architecture.
Critical weaknesses
-
Unclear motivation and practical significance. The paper does not sufficiently explain why a vision model needs to be explicitly optimized for projectability. If the purpose is to visualize and understand an existing model (Model A), applying PAFT modifies its learned representations and produces a different model (Model B). Consequently, the resulting visualization no longer directly reflects what Model A originally learned. Alternatively, if the goal is to train a new model whose representations are inherently easier to visualize, the authors should clarify what practical applications require this property and demonstrate its benefits beyond improved projection-quality metrics. Neither scenario is adequately justified, which raises concerns about the significance of the proposed research problem.
-
Limited methodological novelty. The proposed method mainly combines three established techniques: knowledge distillation for prediction consistency, CKA for representation alignment, and the neighborhood-preservation objective from parametric t-SNE. Although jointly optimizing the high-dimensional representation and auxiliary bottleneck is a reasonable design, the paper does not introduce a fundamentally new mechanism for modeling or preserving high-dimensional and low-dimensional relationships. The methodological contribution therefore appears incremental.
-
Questionable evaluation of projection fidelity. The projection-quality metrics use each method's own high-dimensional representation as the reference, while PAFT explicitly modifies this representation during training. Therefore, improved projection fidelity may result partly from simplifying or compressing the high-dimensional geometry rather than preserving more meaningful information from the original representation. The spectral analyses in Appendix I also demonstrate substantial changes in the representation's rank and variance distribution. Although classification accuracy is largely maintained, this does not necessarily imply that other useful information in the original representation is preserved. Consequently, the current evaluation does not sufficiently establish whether PAFT produces genuinely more informative visualizations.
-
Insufficient comparisons with relevant methods. The experimental comparisons mainly involve standard fine-tuning and three rank-reduction alternatives. However, if PAFT is intended as a visualization or dimensionality-reduction approach, comparisons with existing learning-based visualization methods, such as parametric t-SNE, parametric UMAP, and other deep dimensionality-reduction approaches, are necessary. Conversely, if the contribution is a projection-aware representation-learning strategy, comparisons with existing geometry-preserving regularization and joint representation-learning methods would be more relevant. Although PCA, t-SNE, UMAP, and PaCMAP are included in the experiments, they are primarily used as post-hoc projection operators rather than competing learning-based approaches. It is also worth to compare other existing deep visialization methods, such as “Navigating the Effect of Parametrization for Dimensionality Reduction” (NIPS 2024), and "AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings" (ICML 2025). The current comparisons are therefore insufficient to establish the advantage of PAFT over closely related methods.
-
Limited validation of broader applicability. All experiments focus on supervised image classification. Although the geometric regularization terms do not directly require class labels, the overall framework relies on a supervised classification objective, while prediction consistency transfers label-related information through the main classifier's predictions. It remains unclear how the proposed approach can be adapted to unsupervised or self-supervised representation learning, where visualization is also commonly used. Furthermore, maintaining classification accuracy alone does not establish that the learned representations preserve other capabilities or information useful for downstream tasks. The applicability of PAFT beyond supervised classification therefore remains insufficiently validated.
Questions for authors
Effectiveness of the directly visualizable bottleneck. The paper emphasizes that setting the bottleneck dimension to enables direct visualization without additional dimensionality reduction. However, Table 21 shows that the auxiliary classifier performs poorly on several datasets with many classes, including CIFAR-100, TinyImageNet, and CUB-200. How should these results be reconciled with the claim that the bottleneck preserves predictive information? To what extent can such a low-dimensional representation faithfully reflect the task-relevant structure learned by the original classifier?
6 comments
Response to Reviewer HDoW (2/3)
W3: Compression and information retained in the representation. We agree that compression contributes to PAFT’s projection quality, but it does not fully account for PAFT’s advantage. The random-projection control in Appendix I (revised Table 18) isolates this effect. A random 2D map is not fitted to the data, so its gains come mainly from low rank: when the variance is concentrated in fewer directions, randomly chosen directions are more likely to capture it. Accordingly, these gains over the untreated baseline closely track the effective-rank ratio (Spearman ρ = −0.89). Compared directly with each rank-reduction alternative, PAFT’s advantage is larger under each of PCA, t-SNE, UMAP, and PaCMAP, which are fitted to the data, than under the random map: it rises from +4.97/20 to between +8.60 and +12.93 against the nuclear-norm penalty, and from +1.69 to between +7.20 and +11.07 against the top-d variance penalty. The narrow-penultimate baseline, whose selected widths are realistic penultimate-layer sizes for an ordinary model, compresses the spectral tail far more than PAFT (k99 ratio 0.03 vs. 0.88, Table 17) and is ahead of PAFT under the random map, yet PAFT is ahead of it on average under each fitted projection.
The spectral analysis also indicates that PAFT concentrates the representation in fewer leading dimensions without truncating the remaining ones. Relative to the untreated baseline, PAFT needs 0.43× as many principal components to explain 50% of the variance, but it still needs 0.88× as many components to explain 99% (Table 17).
To assess what PAFT preserves beyond accuracy, we compared PAFT with untreated models, using the variation between untreated models trained with different seeds as a reference (new Appendix K, Table 23). Mean prediction agreement is 0.889 between PAFT and untreated models, compared with 0.891 between untreated models. Their class-confusion structures have a mean Spearman correlation of 0.877, compared with 0.890 between untreated models. Untreated penultimate representations can also be linearly recovered from PAFT representations of a different seed on held-out data with mean R² = 0.594, compared with 0.589 from another untreated model. Despite the change in geometry, PAFT models therefore remain nearly as close to untreated models as two untreated training runs are to each other.
[Revision] We revised Table 18 to compare PAFT directly with each rank-reduction alternative, and added these analyses as the new Appendix K (Table 23).
Response to Reviewer HDoW (3/3)
W4: Comparisons with learned visualization and joint-learning methods. A useful distinction here is what each method learns. ParamRepulsor and AKRMap learn visualization mappings from fixed representations, whereas PAFT shapes the classifier representation itself. These methods can therefore serve as projection tools when comparing PAFT with other representation-learning methods, much as we use PCA, t-SNE, UMAP, and PaCMAP.
For this evaluation, we preferred the more widely used nonparametric methods. Huang et al. (2024) show that parametric methods tend to lose local structure relative to their nonparametric counterparts, and we observed the same on frozen untreated baseline representations: deep parametric t-SNE and UMAP score −2.53/20 and −1.47/20 against their nonparametric counterparts (new Appendix L, Table 24). We have now also trained learned projectors separately on PAFT and untreated representations; AKRMap requires a cross-modal metric, such as CLIPScore, as its regression target, which our single-modality classification setting does not provide. Each of the three performs better on PAFT representations, in all 15 backbone–dataset combinations: ParamRepulsor by +14.73/20, deep parametric t-SNE by +9.60/20, and deep parametric UMAP by +12.73/20 (Table 25).
In this context, the ablation experiment in Appendix H can be read as a variant of linear parametric t-SNE: it learns a linear projection of frozen representations using the t-SNE objective, with PAFT’s prediction-consistency and CKA terms added. Applying this procedure to PAFT representations gives a +14.1/20 advantage over applying it to untreated representations. We also tested the version with only the t-SNE objective on frozen PAFT and untreated baseline representations. Linear parametric t-SNE on PAFT representations outperforms its counterpart on untreated baseline representations by +15.13/20.
A similar connection can be made to joint learning. The neighborhood-only ablation in Appendix G can be viewed as a jointly trained linear parametric t-SNE variant, since classification loss and the neighborhood objective together update the backbone and its linear bottleneck. Full PAFT outperforms this variant by +5.58/20 (Table 14).
We have also considered other joint approaches. Parametric UMAP includes a joint-training variant aimed at improving classifier accuracy, in which a nonlinear UMAP head is trained jointly with the classifier. We adapted it to our setting (new Appendix M, Tables 26–28). Although it also shapes the representation during training, and improves on the untreated baseline by +9.30/20 while keeping main-head accuracy, PAFT’s penultimate representation outperforms its penultimate representation by +6.47/20, in 13 of 15 backbone–dataset combinations, and remains ahead even under nonparametric UMAP projection (+4.67/20). Comparing the low-dimensional outputs directly, PAFT’s linear bottleneck is ahead of the nonlinear UMAP head under t-SNE, UMAP, and PaCMAP (+7.00, +5.87, and +4.93/20); only under PCA is the UMAP head ahead (+3.67/20), since a nonlinear head can unfold clusters that a linear projection cannot. The UMAP head also adds up to 5.7% of the backbone’s parameters, compared with at most 0.62% for PAFT’s auxiliary branch. t-SimCNE takes a different direction: it learns a direct 2D view through augmentation-based contrastive learning, without a supervised classifier objective. Comparing it under our protocol would therefore require changing the training task, rather than simply replacing PAFT’s geometric regularization.
[Revision] We added ParamRepulsor (Huang et al., 2024) and AKRMap to Related Work, the parametric and learned-projector comparisons as the new Appendix L (Tables 24–25), the linear parametric t-SNE result as a new paragraph at the end of Appendix H, and the joint parametric UMAP comparison as the new Appendix M (Tables 26–28).
W5: Applicability beyond supervised classification. We agree that our experiments cover only supervised image classification. The two geometric terms, CKA alignment and neighborhood preservation, do not use labels, and removing the classifier-dependent prediction-consistency term reduces the win–loss difference by only 1.73/20 (Table 14). That variant is still fine-tuned with cross-entropy, however. In a self-supervised or label-free setting, we expect the classification loss to be replaced by the self-supervised objective, or by a distillation anchor to the pretrained model, with the geometric terms unchanged. We have not tested this. The predictive-behavior and representation-recovery analyses in our response to W3 also go beyond classification accuracy in assessing what PAFT preserves.
[Revision] We added this limitation to the Scope paragraph of the Limitations (Section 5).
Response to Reviewer HDoW (1/3)
We thank the reviewer for the detailed assessment. We first address the question about the d=2 bottleneck, followed by the motivation, novelty, information retention, and comparison choices.
Q: Predictive information in the d=2 bottleneck. The d=2 bottleneck is a special configuration that provides a direct visualization; PAFT’s main goal is to make the high-dimensional representation more faithfully projectable. To assess how much task-relevant structure this view retains, the appropriate comparison is with other 2D visualizations, rather than expecting it to match the high-dimensional main classifier’s accuracy.
Information loss is a common limitation in dimensionality-reduction methods, and we do not claim to preserve all predictive information. Our claim is that the 2D bottleneck can preserve structure better than post-hoc PCA views. Since it is a linear map of the penultimate representation, PCA provides a natural comparison. Across the 15 backbone–dataset combinations, the d=2 view outperforms PCA of the untreated representation by +15.2/20 on average (Table 4). The low auxiliary accuracy partly reflects the limits of the linear auxiliary head rather than the view itself, since a linear classifier must divide the plane into 100–200 class regions using linear boundaries. On ResNet-50/CIFAR-100, for example, the auxiliary head reaches 9.4% (Table 21), whereas 15-NN accuracy in the same 2D view is 39.1%, compared with 15.8% to 23.2% across the PCA views in Table 2.
[Revision] We revised the phrase “preserves predictive and geometric structure” in the Abstract, Introduction, and Section 3 to state that the bottleneck is trained to preserve this structure, rather than implying a guarantee of complete preservation.
W1: Motivation and practical value. PAFT deliberately changes representation learning so that the resulting classifier, the model we would actually use, can be inspected more faithfully.
Training a model to make it easier to inspect has precedents. Tree regularization (Wu et al., 2018) trains neural networks to be more faithfully approximated by simple decision trees. More directly related to visualization, t-SimCNE (Böhm et al., 2023) jointly trains an image encoder and a 2D projection head. These works pursue different goals, but share the choice to improve inspection by changing how representations or models are learned.
For a visualization to help us understand a model, we need to trust that the patterns we see reflect its high-dimensional representation. Well-separated classes or clear clusters can make a plot easier to read, but they tell us little about the model if they arise mainly from the projection. This is why we focus on fidelity: preserving the model’s neighborhoods, distances, and class relationships makes the visualization a more reliable basis for interpretation.
Section 4.3 and Appendix F illustrate the practical value of this fidelity through radial structure and class confusion. Distances from a representation center underlie anomaly-detection methods such as Deep SVDD (Ruff et al., 2018), and feature-norm-based scores are used for out-of-distribution detection (Dhamija et al., 2018; Park et al., 2023). Our radial analysis shows that PAFT’s views better preserve samples’ distances from the overall representation center. The class-confusion analysis also shows that visible class relationships better reflect which classes the classifier confuses (Table 13). Together, these results help us read model diagnostics from the visualization more reliably.
[Revision] We added tree regularization and t-SimCNE to Related Work, and the references on center distances and feature norms to the radial-structure paragraph of Section 4.3.
W2: Methodological novelty. We agree that the individual terms are established, and we do not claim a new loss. Our contribution is to shape the classifier representation during fine-tuning so that it remains faithful under standard projections, and the evidence indicates that this joint shaping, together with how the terms are combined, drives the gains. The same auxiliary branch fitted to frozen representations performs better on PAFT representations than on untreated ones by +14.1/20 (Appendix H). Full PAFT outperforms the jointly trained neighborhood-only variant by +5.58/20 (Table 14) and joint parametric UMAP by +6.47/20 (new Appendix M), and the new stop-gradient ablations in Appendix G show that how the terms are coupled also matters.
[Revision] We added the stop-gradient ablations as a new paragraph at the end of Appendix G; the joint parametric UMAP comparison (new Appendix M) is detailed under W4.