- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CRED (Cross-Layer Participation Regularization), a vision-language embedding distillation framework designed to mitigate dimensional collapse in compact student models. The method introduces a teacher-independent participation-ratio regularizer that preserves the effective dimensionality of intermediate image-token representations, together with relational embedding distillation and a randomized estimator for efficient computation. Experiments on MMEB classification, visual question answering, and out-of-distribution benchmarks demonstrate improvements over existing distillation approaches across two compact student architectures.
Critical strengths
-
Relevant and well-motivated problem. The paper investigates intermediate representation collapse in compact vision-language embedding models, addressing an important aspect of knowledge distillation that receives less attention than output-level semantic alignment. The analysis provides a useful perspective on how internal representations evolve across network depth.
-
Simple and practical regularization mechanism. The proposed cross-layer participation regularizer constrains intermediate effective dimensionality without requiring teacher hidden states or explicit teacher-student layer correspondence. This makes it applicable to teacher-student pairs with different architectures and embedding dimensions.
-
Strong quantitative improvements. CRED improves average classification and VQA performance over the evaluated distillation baselines. For example, on FastVLM-0.5B, classification performance increases from 67.11% for HieRD to 71.48%, while VQA performance improves from 52.76% to 55.68%.
-
Useful theoretical and computational contributions. The randomized participation-ratio estimator avoids explicit covariance eigendecomposition and provides favorable computational scaling. The theoretical analysis establishes a finite-sample approximation bound, while the experiments demonstrate approximately 19% lower training-step time compared with exact PR computation.
-
Comprehensive component ablations. Table 5 evaluates the contributions of embedding distillation, structural preservation, and participation regularization. Adding CPR to the two teacher-driven objectives improves OOD classification performance from 38.21% to 40.70%, supporting its complementary role in the proposed framework.
-
Evidence of transferability. The experiments cover FastVLM-0.5B and LLaVA-OneVision-0.5B, with additional evaluations involving a larger teacher. Applying CPR to CKD and HieRD also produces average classification improvements, suggesting that the regularizer is not restricted to the proposed distillation framework.
-
Evaluation under distribution shift. The paper includes held-out classification and VQA benchmarks, providing evidence that the benefits extend beyond the training distribution. The improvements on ImageNet-R, Places365, and several OOD VQA benchmarks are particularly encouraging.
Critical weaknesses
-
Insufficient evidence that dimensional collapse is specifically caused by distillation. The paper motivates CRED through substantial reductions in intermediate effective rank, but Figure 4 indicates that dimensional collapse is already present in pretrained FastVLM and persists under standard supervised fine-tuning. Several distillation baselines also exhibit trajectories similar to the non-distilled student. Consequently, the evidence does not clearly establish whether the phenomenon is primarily caused by knowledge distillation, compact model architecture, or general transformer representation processing.
-
Limited causal evidence connecting effective rank to semantic representation quality. Although CRED increases intermediate effective rank and improves downstream performance, these observations do not establish that dimensional collapse directly causes degraded semantic understanding. The paper does not sufficiently investigate whether the additional preserved dimensions encode meaningful task-relevant information or simply increase representation variance.
-
Insufficient justification for the input-layer reference. CPR assumes that the participation ratio of the initial image-token representation provides an appropriate lower bound for deeper layers. However, dimensionality reduction across transformer layers may reflect useful feature abstraction or removal of redundant information rather than harmful collapse. The paper does not establish why deeper representations should retain at least the same effective dimensionality as the input representation.
-
Moderate methodological novelty. Participation ratio is an established spectral statistic, and the proposed objective applies a relatively straightforward one-sided penalty across selected student layers. The randomized estimator also builds on standard stochastic spectral-moment estimation principles. The paper should more clearly establish its methodological advantages over existing effective-rank and representation-diversity regularization techniques.
-
Incomplete comparison with alternative dimensionality-preserving regularizers. The experiments primarily compare CRED against existing knowledge distillation approaches but do not sufficiently evaluate alternatives such as covariance decorrelation, variance regularization, orthogonality penalties, or other effective-rank-based objectives. These comparisons are necessary to establish whether the proposed participation-ratio formulation offers benefits beyond generic representation-diversity regularization.
-
Limited investigation of modality-specific collapse. The central analysis focuses on image-token representations, although the framework targets multimodal embedding learning. The paper does not systematically examine whether text-token representations exhibit comparable collapse or whether preserving visual effective dimensionality is particularly important for cross-modal alignment. This limits understanding of the modality-specific nature of the proposed mechanism.
-
Insufficient discussion of Country211 performance degradation. Despite strong average out-of-distribution performance, CRED underperforms the strongest competing methods on Country211 for both student architectures. Table 11 reports 8.1% accuracy for CRED compared with 8.6% for HieRD on FastVLM-0.5B, and 10.23% for CRED compared with 11.10% for the non-distilled student on LLaVA-OneVision-0.5B. This repeated exception is particularly important because improved generalization under distribution shift is a central claim of the paper. Although the authors briefly acknowledge these weaker results, they do not provide a substantive explanation of the underlying causes. The absence of such an analysis makes it difficult to determine whether the regularizer introduces task-dependent trade-offs or fails to preserve semantic information relevant to geographic scene recognition.
-
Performance improvements are not uniform across benchmarks. Beyond Country211, CRED also exhibits substantial degradation on HatefulMemes with LLaVA-OneVision-0.5B. Its accuracy decreases to 53.10%, compared with 59.27% for HieRD and 57.17% for the non-distilled student. These regressions suggest that increasing intermediate effective dimensionality does not uniformly improve semantic discrimination and warrant a more systematic failure-case analysis.
-
Incomplete separation of the regularizer from the overall distillation framework. CRED combines embedding-level distillation, structural preservation, and CPR. Although the ablations demonstrate the contribution of each component, the full method's improvements should not be attributed entirely to preventing representation collapse. In Table 5, the two teacher-driven objectives without CPR already improve in-distribution classification performance from 64.93% to 70.14%, while adding CPR raises it to 71.48%.
-
Limited analysis of the relationship between participation ratio and Shannon effective rank. The paper uses Shannon effective rank to identify the motivating phenomenon but optimizes participation ratio during training. Although these measures are theoretically related, they respond differently to spectral distributions. A more direct analysis would help determine whether optimizing PR reliably preserves the effective dimensionality measured in the initial experiments.
-
Limited coverage of student model scales. The principal experiments focus on two 0.5B student architectures using the same default 2B teacher. Although additional experiments consider a larger teacher, the results do not fully establish how CPR behaves across substantially different student capacities or compression ratios.
Questions for authors
-
Origin of dimensional collapse. Figure 4 suggests that substantial effective-rank reduction already occurs in pretrained FastVLM and persists under supervised fine-tuning. Can the authors clarify whether the observed collapse is primarily induced by distillation or is an inherent characteristic of compact transformer representations? A controlled comparison across pretrained, fine-tuned, and distilled models would strengthen the motivation.
-
Collapse versus useful dimensionality reduction. How do the authors distinguish harmful representation collapse from beneficial compression of redundant features? Is there evidence that preserving higher intermediate participation ratios consistently improves semantic discriminability rather than simply maintaining greater representation variance?
-
Justification of the input-layer reference. Why should the participation ratio at the initial hidden state serve as a lower bound for deeper representations? Could the authors compare this choice against fixed thresholds, teacher-derived references, or layer-dependent targets to determine whether the input-level reference is particularly appropriate?
-
Comparison with alternative regularizers. How does CPR compare with variance regularization, covariance decorrelation, orthogonality constraints, or other effective-rank-preserving objectives under the same distillation framework? These comparisons would clarify whether participation-ratio regularization provides a distinct advantage.
-
Causal relationship between effective rank and performance. Can the authors provide a quantitative analysis relating layer-wise effective rank to downstream performance across different baselines, regularization strengths, and training configurations? This would help determine whether effective dimensionality is a reliable indicator of semantic representation quality.
-
Modality-specific behavior. Do text-token representations exhibit similar layer-wise dimensionality collapse? If so, why is CPR applied only to image tokens? An ablation comparing image-only, text-only, and joint regularization would help clarify the importance of preserving dimensionality in each modality.
-
Country211 performance degradation. Why does CRED fail to outperform the strongest competing methods on Country211 for both student architectures? Given that improved OOD generalization is a central claim, can the authors investigate whether this behavior arises from the participation-ratio constraint, properties of geographic scene recognition, or limitations of the learned multimodal embeddings? In particular, a comparison of CRED with and without CPR on Country211 would help determine whether the regularizer itself contributes to the degradation.
-
HatefulMemes performance regression. What explains the substantial decrease in HatefulMemes accuracy for LLaVA-OneVision-0.5B? Could the authors investigate whether preserving greater intermediate spectral diversity affects tasks that require precise interactions between visual and linguistic information?
-
Contribution of CPR versus relational distillation. Table 5 shows that the teacher-driven distillation objectives already produce substantial improvements without CPR. Can the authors more clearly distinguish which downstream gains arise specifically from the regularizer and whether its relative contribution is consistent across both student architectures?
-
Choice of spectral statistic. The motivation uses Shannon effective rank, whereas the proposed objective optimizes participation ratio. Beyond computational efficiency, why is PR preferred? Could the authors compare the effectiveness of PR with alternative spectral measures and examine whether the resulting representations exhibit similar effective-rank improvements?
-
Randomized estimator reliability. How stable is the stochastic PR estimator across different token counts, layer depths, and representation spectra? A more detailed comparison against exact PR across both student architectures would help establish its robustness.
-
Regularized layer selection. The default objective regularizes layers between 50% and 80% of network depth, whereas the motivating FastVLM results show that collapse begins within the first 20%. Why are earlier layers excluded? Could the authors clarify whether this choice is motivated by optimization stability, computational efficiency, or empirical performance?
-
Generalization across model scales. How does CPR behave when applied to larger student models or when the teacher-student capacity gap decreases? Such experiments would strengthen the claims of general applicability.
-
Task-dependent trade-offs. Given the regressions on Country211 and HatefulMemes, can the authors identify characteristics of tasks where participation-ratio regularization is beneficial versus potentially harmful? This analysis would provide a more nuanced understanding of the method's generalization behavior.
13 comments
Q3 & W3 · Why use the student's input-level PR as the reference?
Q3 & W3 · Why use the student's input-level PR as the reference?
The representation immediately after visual projection provides a sample-dependent reference before language-backbone processing. Images differ in their available spectral diversity, so this reference adapts to each image rather than imposing one target rank. An early hidden-layer reference can already include the dimensionality reduction visible within the first 20% of FastVLM's depth.
A teacher-derived reference would require collecting additional teacher token statistics and defining comparisons across different tokenizations, hidden dimensions, and layers. CPR uses the student's own representations and requires neither those statistics nor a teacher–student layer mapping. The reference is measured at layer 0, immediately after visual projection and before the Transformer blocks; this is distinct from the deeper layers at which the penalty is evaluated.
Table R1. Reference ablations within CRED on FastVLM-0.5B. The remaining objectives and selected regularization layers are held fixed. Avg. is the mean of IND and OOD classification P@1. The fixed threshold is applied to unnormalized PR.
| PR reference | Time (s/step) | Peak memory (GB) | IND P@1 (%) | OOD P@1 (%) | Avg. (%) |
|---|---|---|---|---|---|
| No CPR | 1.35 | 28.65 | 70.14 | 38.21 | 54.18 |
| Fixed threshold, | 1.41 | 30.67 | 70.98 | 38.64 | 54.81 |
| Layer 5, approximately 20% depth | 1.41 | 30.67 | 70.88 | 38.24 | 54.56 |
| Input layer, default CPR | 1.41 | 30.67 | 71.48 | 40.70 | 56.09 |
The input reference performs best on IND, OOD, and the overall average. Relative to the early-layer reference, it improves OOD P@1 by 2.46 points and the average by 1.53 points, at the same reported training cost. These comparisons directly support the reference choice.
Stability of the adaptive reference. We additionally evaluate the same fixed image panel before and after training. During the one-epoch run, successive minibatches contain different images, so raw step-wise PR values combine batch-composition effects with parameter changes. The matched panel provides a controlled comparison using identical per-image centering and rank normalization.
Table R1a. Reference stability and intermediate-layer recovery on FastVLM-0.5B. PR is computed exactly per image, divided by , and then averaged over the same panel.
| Checkpoint | Input PR / cap (L0) | First-layer PR / cap (L1) | Intermediate PR / cap (L12 / L19) |
|---|---|---|---|
| Pretrained, before training | 0.049997 | 0.060203 | 0.004965 / 0.006483 |
| SFT, after training | 0.049581 | 0.060657 | 0.004805 / 0.005949 |
| CRED, after training | 0.050043 | 0.062341 | 0.064396 / 0.070416 |
The mean input reference changes by only +0.092%, while the selected intermediate layers recover substantially. Thus, the recovery is accompanied by a stable mean reference rather than a net reduction of the target between the evaluated checkpoints. This supports using the student's own input representation as an adaptive anchor.
Q4 & W4–W5 · Methodological contribution and alternative regularizers
Q4 & W4–W5 · Methodological contribution and alternative regularizers
The methodological contribution is the combination of sample-adaptive, one-sided cross-layer supervision with an efficient matrix-free implementation for visual-token distillation. It accommodates heterogeneous teachers and students while avoiding intermediate teacher activations and feature matching.
The constraint also differs from coordinate-wise variance and covariance objectives, for which we include a matched control inspired by VICRegL (Bardes et al., 2022). A variance floor can be satisfied by increasing feature magnitudes without increasing directional diversity, whereas exact PR is invariant to uniform rescaling and orthogonal rotations. It permits different covariance spectra above the reference, without requiring an identity covariance. In particular, a centered token covariance has rank at most ; requiring both positive variance in all coordinates and zero off-diagonal covariance can conflict with this rank limit when .
For the Shannon control, we replace PR in the cross-layer hinge with Shannon effective rank computed on the same per-image centered covariances and detached input-reference convention. The comparison therefore changes the spectral statistic while retaining the rest of the framework.
Table R2. Regularization mechanisms within the same CRED framework, FastVLM-0.5B. All controls retain InfoNCE, embedding distillation, structural preservation, and the same selected layers. Complexity is per sample and regularized layer; is token count, hidden dimension, and for the matrix-free estimator.
| Regularizer | Complexity | Time (s/step) | Peak memory (GB) | IND P@1 (%) | OOD P@1 (%) | Avg. (%) |
|---|---|---|---|---|---|---|
| No CPR | — | 1.35 | 28.65 | 70.14 | 38.21 | 54.18 |
| VICRegL-style variance + covariance | 1.81 | 30.95 | 69.10 | 39.98 | 54.54 | |
| Exact Shannon effective-rank hinge | 3.14 | 30.63 | 71.50 | 39.94 | 55.72 | |
| Exact PR hinge | 1.74 | 30.97 | 71.42 | 40.12 | 55.77 | |
| Matrix-free CPR | 1.41 | 30.67 | 71.48 | 40.70 | 56.09 |
Matrix-free CPR achieves the highest OOD and overall average P@1 among the evaluated variants. It improves average P@1 over the VICRegL-style and Shannon eRank controls by 1.55 and 0.37 percentage points, respectively, while reducing their training-step times by 22.1% and 55.1%. Compared with the matched no-CPR baseline, CPR improves average P@1 by 1.91 points with an additional 0.06 s/step (4.4%). These comparisons support the proposed sample-adaptive PR constraint as an effective and computationally efficient regularizer within the same distillation framework.
Official Response to Reviewer NZkr
Thank you for your thoughtful review and constructive feedback. We address your questions and corresponding weaknesses below.
Q1 & W1 · Origin of the observed dimensional collapse
The collapse observed in our experiments predates distillation: Figure 4 includes pretrained FastVLM, and the intermediate-layer reduction persists after supervised fine-tuning. Our motivation is therefore a student-side representation bottleneck that remains unresolved under output-level distillation, rather than a phenomenon introduced exclusively by knowledge distillation.
Figures 2 and 4 compare pretrained, fine-tuned, and distilled representations. Several distillation objectives leave the student trajectory close to SFT, while HieRD on LLaVA-OneVision exhibits a stronger decline at deeper layers. These observations motivate explicitly supervising intermediate dimensionality during distillation. Adding CPR improves CKD and HieRD classification by 2.03 and 1.05 percentage points, and improves CRED by 1.34 points IND / 2.49 points OOD. Thus, addressing the bottleneck benefits several transfer objectives. We will clarify the motivation to distinguish pre-existing student collapse from its persistence or intensification under distillation.
Q2 & W2 · Distinguishing harmful collapse from useful compression
We evaluate the usefulness of preserving intermediate dimensionality through downstream semantic discrimination under matched training conditions. Holding the architecture and other objectives fixed, adding CPR improves FastVLM IND/OOD classification P@1 from 70.14/38.21 to 71.48/40.70, and LLaVA-OneVision from 71.06/40.28 to 72.21/45.71. The effect also appears without teacher supervision: adding CPR to SFT improves FastVLM from 64.93/33.22 to 68.92/37.54. These controlled comparisons support the task relevance of the regularized representations across both architectures and training configurations (Table R7, Q9).
PR measures spectral concentration independently of overall variance magnitude. For any a>0, PR(aΣ)=PR(Σ), so uniformly scaling feature magnitudes cannot satisfy the exact PR objective. The matched diagnostics in Q5 additionally show that CRED's recovery persists under both centered PR and centered Shannon effective rank. We thus assess the spectral change together with its measured downstream benefit.
The objective also permits task-driven compression. Equation (9) applies a soft, one-sided penalty relative to each image's input reference:
\operatorname{sg}\left[\widehat{\operatorname{PR}}(\Sigma_i^{(0)})\right] -\widehat{\operatorname{PR}}(\Sigma_i^{(l)})\right).$$ The penalty becomes zero once the reference is reached; CPR therefore does not continually reward higher PR. It is evaluated at two selected intermediate layers near 50% and 80% depth, while the final layer is not directly regularized. The contrastive and teacher-guided objectives continue to learn task-relevant features and their output embedding. The depth ablations in Q12 (Table R9) support this placement: the default achieves an IND/OOD average of **56.09**, compared with **55.86** and **55.75** for the two later-layer settings. Together, the matched spectral diagnostics, controlled performance gains, and task-specific comparisons in Q8 and Q14 support CPR as complementary intermediate supervision. Our explanation concerns the usefulness of a targeted dimensionality constraint alongside semantic learning, rather than a universal rule that every reduction is harmful or every increase in PR improves semantic quality.