acceptodds
Under review as a conference paper at ICLR 2027

CRED: Cross-Layer Participation Regularization Against Token Collapse in Vision-Language Embedding Distillation

Abstract

Vision-language embedding models map multimodal inputs into a shared semantic representation space. Embedding distillation transfers multimodal representation knowledge from larger teacher models to more compact students, but can lead to token representation collapse in the student. We find that the effective rank of image-token representations in the student undergoes a nearly order-of-magnitude reduction relative to its initial hidden state, whereas the teacher exhibits substantially milder changes across its depth. To address this discrepancy without requiring intermediate teacher activations, we introduce CRED, a teacher-independent regularizer based on the participation ratio (PR). CRED constrains the PR of selected student hidden layers to remain no lower than that of the initial hidden state, thereby preserving effective dimensionality throughout the network. Unlike layer-wise feature or covariance matching, CRED operates on a scalar spectral statistic that is invariant to representation rotations and requires no additional teacher inference. Moreover, PR can be efficiently approximated using random projections, avoiding eigendecomposition and scaling more favorably to inputs with large token counts. Combined with output-level distillation, CRED reduces layer-wise token collapse and improves performance on downstream multimodal tasks in MMEB over competitive baseline methods.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
vlm2vec distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NZkrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CRED (Cross-Layer Participation Regularization), a vision-language embedding distillation framework designed to mitigate dimensional collapse in compact student models. The method introduces a teacher-independent participation-ratio regularizer that preserves the effective dimensionality of intermediate image-token representations, together with relational embedding distillation and a randomized estimator for efficient computation. Experiments on MMEB classification, visual question answering, and out-of-distribution benchmarks demonstrate improvements over existing distillation approaches across two compact student architectures.

Critical strengths
  1. Relevant and well-motivated problem. The paper investigates intermediate representation collapse in compact vision-language embedding models, addressing an important aspect of knowledge distillation that receives less attention than output-level semantic alignment. The analysis provides a useful perspective on how internal representations evolve across network depth.

  2. Simple and practical regularization mechanism. The proposed cross-layer participation regularizer constrains intermediate effective dimensionality without requiring teacher hidden states or explicit teacher-student layer correspondence. This makes it applicable to teacher-student pairs with different architectures and embedding dimensions.

  3. Strong quantitative improvements. CRED improves average classification and VQA performance over the evaluated distillation baselines. For example, on FastVLM-0.5B, classification performance increases from 67.11% for HieRD to 71.48%, while VQA performance improves from 52.76% to 55.68%.

  4. Useful theoretical and computational contributions. The randomized participation-ratio estimator avoids explicit covariance eigendecomposition and provides favorable computational scaling. The theoretical analysis establishes a finite-sample approximation bound, while the experiments demonstrate approximately 19% lower training-step time compared with exact PR computation.

  5. Comprehensive component ablations. Table 5 evaluates the contributions of embedding distillation, structural preservation, and participation regularization. Adding CPR to the two teacher-driven objectives improves OOD classification performance from 38.21% to 40.70%, supporting its complementary role in the proposed framework.

  6. Evidence of transferability. The experiments cover FastVLM-0.5B and LLaVA-OneVision-0.5B, with additional evaluations involving a larger teacher. Applying CPR to CKD and HieRD also produces average classification improvements, suggesting that the regularizer is not restricted to the proposed distillation framework.

  7. Evaluation under distribution shift. The paper includes held-out classification and VQA benchmarks, providing evidence that the benefits extend beyond the training distribution. The improvements on ImageNet-R, Places365, and several OOD VQA benchmarks are particularly encouraging.

Critical weaknesses
  1. Insufficient evidence that dimensional collapse is specifically caused by distillation. The paper motivates CRED through substantial reductions in intermediate effective rank, but Figure 4 indicates that dimensional collapse is already present in pretrained FastVLM and persists under standard supervised fine-tuning. Several distillation baselines also exhibit trajectories similar to the non-distilled student. Consequently, the evidence does not clearly establish whether the phenomenon is primarily caused by knowledge distillation, compact model architecture, or general transformer representation processing.

  2. Limited causal evidence connecting effective rank to semantic representation quality. Although CRED increases intermediate effective rank and improves downstream performance, these observations do not establish that dimensional collapse directly causes degraded semantic understanding. The paper does not sufficiently investigate whether the additional preserved dimensions encode meaningful task-relevant information or simply increase representation variance.

  3. Insufficient justification for the input-layer reference. CPR assumes that the participation ratio of the initial image-token representation provides an appropriate lower bound for deeper layers. However, dimensionality reduction across transformer layers may reflect useful feature abstraction or removal of redundant information rather than harmful collapse. The paper does not establish why deeper representations should retain at least the same effective dimensionality as the input representation.

  4. Moderate methodological novelty. Participation ratio is an established spectral statistic, and the proposed objective applies a relatively straightforward one-sided penalty across selected student layers. The randomized estimator also builds on standard stochastic spectral-moment estimation principles. The paper should more clearly establish its methodological advantages over existing effective-rank and representation-diversity regularization techniques.

  5. Incomplete comparison with alternative dimensionality-preserving regularizers. The experiments primarily compare CRED against existing knowledge distillation approaches but do not sufficiently evaluate alternatives such as covariance decorrelation, variance regularization, orthogonality penalties, or other effective-rank-based objectives. These comparisons are necessary to establish whether the proposed participation-ratio formulation offers benefits beyond generic representation-diversity regularization.

  6. Limited investigation of modality-specific collapse. The central analysis focuses on image-token representations, although the framework targets multimodal embedding learning. The paper does not systematically examine whether text-token representations exhibit comparable collapse or whether preserving visual effective dimensionality is particularly important for cross-modal alignment. This limits understanding of the modality-specific nature of the proposed mechanism.

  7. Insufficient discussion of Country211 performance degradation. Despite strong average out-of-distribution performance, CRED underperforms the strongest competing methods on Country211 for both student architectures. Table 11 reports 8.1% accuracy for CRED compared with 8.6% for HieRD on FastVLM-0.5B, and 10.23% for CRED compared with 11.10% for the non-distilled student on LLaVA-OneVision-0.5B. This repeated exception is particularly important because improved generalization under distribution shift is a central claim of the paper. Although the authors briefly acknowledge these weaker results, they do not provide a substantive explanation of the underlying causes. The absence of such an analysis makes it difficult to determine whether the regularizer introduces task-dependent trade-offs or fails to preserve semantic information relevant to geographic scene recognition.

  8. Performance improvements are not uniform across benchmarks. Beyond Country211, CRED also exhibits substantial degradation on HatefulMemes with LLaVA-OneVision-0.5B. Its accuracy decreases to 53.10%, compared with 59.27% for HieRD and 57.17% for the non-distilled student. These regressions suggest that increasing intermediate effective dimensionality does not uniformly improve semantic discrimination and warrant a more systematic failure-case analysis.

  9. Incomplete separation of the regularizer from the overall distillation framework. CRED combines embedding-level distillation, structural preservation, and CPR. Although the ablations demonstrate the contribution of each component, the full method's improvements should not be attributed entirely to preventing representation collapse. In Table 5, the two teacher-driven objectives without CPR already improve in-distribution classification performance from 64.93% to 70.14%, while adding CPR raises it to 71.48%.

  10. Limited analysis of the relationship between participation ratio and Shannon effective rank. The paper uses Shannon effective rank to identify the motivating phenomenon but optimizes participation ratio during training. Although these measures are theoretically related, they respond differently to spectral distributions. A more direct analysis would help determine whether optimizing PR reliably preserves the effective dimensionality measured in the initial experiments.

  11. Limited coverage of student model scales. The principal experiments focus on two 0.5B student architectures using the same default 2B teacher. Although additional experiments consider a larger teacher, the results do not fully establish how CPR behaves across substantially different student capacities or compression ratios.

Questions for authors
  1. Origin of dimensional collapse. Figure 4 suggests that substantial effective-rank reduction already occurs in pretrained FastVLM and persists under supervised fine-tuning. Can the authors clarify whether the observed collapse is primarily induced by distillation or is an inherent characteristic of compact transformer representations? A controlled comparison across pretrained, fine-tuned, and distilled models would strengthen the motivation.

  2. Collapse versus useful dimensionality reduction. How do the authors distinguish harmful representation collapse from beneficial compression of redundant features? Is there evidence that preserving higher intermediate participation ratios consistently improves semantic discriminability rather than simply maintaining greater representation variance?

  3. Justification of the input-layer reference. Why should the participation ratio at the initial hidden state serve as a lower bound for deeper representations? Could the authors compare this choice against fixed thresholds, teacher-derived references, or layer-dependent targets to determine whether the input-level reference is particularly appropriate?

  4. Comparison with alternative regularizers. How does CPR compare with variance regularization, covariance decorrelation, orthogonality constraints, or other effective-rank-preserving objectives under the same distillation framework? These comparisons would clarify whether participation-ratio regularization provides a distinct advantage.

  5. Causal relationship between effective rank and performance. Can the authors provide a quantitative analysis relating layer-wise effective rank to downstream performance across different baselines, regularization strengths, and training configurations? This would help determine whether effective dimensionality is a reliable indicator of semantic representation quality.

  6. Modality-specific behavior. Do text-token representations exhibit similar layer-wise dimensionality collapse? If so, why is CPR applied only to image tokens? An ablation comparing image-only, text-only, and joint regularization would help clarify the importance of preserving dimensionality in each modality.

  7. Country211 performance degradation. Why does CRED fail to outperform the strongest competing methods on Country211 for both student architectures? Given that improved OOD generalization is a central claim, can the authors investigate whether this behavior arises from the participation-ratio constraint, properties of geographic scene recognition, or limitations of the learned multimodal embeddings? In particular, a comparison of CRED with and without CPR on Country211 would help determine whether the regularizer itself contributes to the degradation.

  8. HatefulMemes performance regression. What explains the substantial decrease in HatefulMemes accuracy for LLaVA-OneVision-0.5B? Could the authors investigate whether preserving greater intermediate spectral diversity affects tasks that require precise interactions between visual and linguistic information?

  9. Contribution of CPR versus relational distillation. Table 5 shows that the teacher-driven distillation objectives already produce substantial improvements without CPR. Can the authors more clearly distinguish which downstream gains arise specifically from the regularizer and whether its relative contribution is consistent across both student architectures?

  10. Choice of spectral statistic. The motivation uses Shannon effective rank, whereas the proposed objective optimizes participation ratio. Beyond computational efficiency, why is PR preferred? Could the authors compare the effectiveness of PR with alternative spectral measures and examine whether the resulting representations exhibit similar effective-rank improvements?

  11. Randomized estimator reliability. How stable is the stochastic PR estimator across different token counts, layer depths, and representation spectra? A more detailed comparison against exact PR across both student architectures would help establish its robustness.

  12. Regularized layer selection. The default objective regularizes layers between 50% and 80% of network depth, whereas the motivating FastVLM results show that collapse begins within the first 20%. Why are earlier layers excluded? Could the authors clarify whether this choice is motivated by optimization stability, computational efficiency, or empirical performance?

  13. Generalization across model scales. How does CPR behave when applied to larger student models or when the teacher-student capacity gap decreases? Such experiments would strengthen the claims of general applicability.

  14. Task-dependent trade-offs. Given the regressions on Country211 and HatefulMemes, can the authors identify characteristics of tasks where participation-ratio regularization is beneficial versus potentially harmful? This analysis would provide a more nuanced understanding of the method's generalization behavior.

13 comments
AuthorsOpenReview ↗

Official Response to Reviewer NZkr

Thank you for your thoughtful review and constructive feedback. We address your questions and corresponding weaknesses below.

 

Q1 & W1 · Origin of the observed dimensional collapse

The collapse observed in our experiments predates distillation: Figure 4 includes pretrained FastVLM, and the intermediate-layer reduction persists after supervised fine-tuning. Our motivation is therefore a student-side representation bottleneck that remains unresolved under output-level distillation, rather than a phenomenon introduced exclusively by knowledge distillation.

Figures 2 and 4 compare pretrained, fine-tuned, and distilled representations. Several distillation objectives leave the student trajectory close to SFT, while HieRD on LLaVA-OneVision exhibits a stronger decline at deeper layers. These observations motivate explicitly supervising intermediate dimensionality during distillation. Adding CPR improves CKD and HieRD classification by 2.03 and 1.05 percentage points, and improves CRED by 1.34 points IND / 2.49 points OOD. Thus, addressing the bottleneck benefits several transfer objectives. We will clarify the motivation to distinguish pre-existing student collapse from its persistence or intensification under distillation.


 

Q2 & W2 · Distinguishing harmful collapse from useful compression

We evaluate the usefulness of preserving intermediate dimensionality through downstream semantic discrimination under matched training conditions. Holding the architecture and other objectives fixed, adding CPR improves FastVLM IND/OOD classification P@1 from 70.14/38.21 to 71.48/40.70, and LLaVA-OneVision from 71.06/40.28 to 72.21/45.71. The effect also appears without teacher supervision: adding CPR to SFT improves FastVLM from 64.93/33.22 to 68.92/37.54. These controlled comparisons support the task relevance of the regularized representations across both architectures and training configurations (Table R7, Q9).

PR measures spectral concentration independently of overall variance magnitude. For any a>0a>0, PR⁡(aΣ)=PR⁡(Σ)\operatorname{PR}(a\Sigma)=\operatorname{PR}(\Sigma), so uniformly scaling feature magnitudes cannot satisfy the exact PR objective. The matched diagnostics in Q5 additionally show that CRED's recovery persists under both centered PR and centered Shannon effective rank. We thus assess the spectral change together with its measured downstream benefit.

The objective also permits task-driven compression. Equation (9) applies a soft, one-sided penalty relative to each image's input reference:

\operatorname{sg}\left[\widehat{\operatorname{PR}}(\Sigma_i^{(0)})\right] -\widehat{\operatorname{PR}}(\Sigma_i^{(l)})\right).$$ The penalty becomes zero once the reference is reached; CPR therefore does not continually reward higher PR. It is evaluated at two selected intermediate layers near 50% and 80% depth, while the final layer is not directly regularized. The contrastive and teacher-guided objectives continue to learn task-relevant features and their output embedding. The depth ablations in Q12 (Table R9) support this placement: the default achieves an IND/OOD average of **56.09**, compared with **55.86** and **55.75** for the two later-layer settings. Together, the matched spectral diagnostics, controlled performance gains, and task-specific comparisons in Q8 and Q14 support CPR as complementary intermediate supervision. Our explanation concerns the usefulness of a targeted dimensionality constraint alongside semantic learning, rather than a universal rule that every reduction is harmful or every increase in PR improves semantic quality.
AuthorsOpenReview ↗

Q3 & W3 · Why use the student's input-level PR as the reference?

Q3 & W3 · Why use the student's input-level PR as the reference?

The representation immediately after visual projection provides a sample-dependent reference before language-backbone processing. Images differ in their available spectral diversity, so this reference adapts to each image rather than imposing one target rank. An early hidden-layer reference can already include the dimensionality reduction visible within the first 20% of FastVLM's depth.

A teacher-derived reference would require collecting additional teacher token statistics and defining comparisons across different tokenizations, hidden dimensions, and layers. CPR uses the student's own representations and requires neither those statistics nor a teacher–student layer mapping. The reference is measured at layer 0, immediately after visual projection and before the Transformer blocks; this is distinct from the deeper layers at which the penalty is evaluated.

Table R1. Reference ablations within CRED on FastVLM-0.5B. The remaining objectives and selected regularization layers are held fixed. Avg. is the mean of IND and OOD classification P@1. The fixed threshold is applied to unnormalized PR.

PR referenceTime (s/step)Peak memory (GB)IND P@1 (%)OOD P@1 (%)Avg. (%)
No CPR1.3528.6570.1438.2154.18
Fixed threshold, τ=12\tau=121.4130.6770.9838.6454.81
Layer 5, approximately 20% depth1.4130.6770.8838.2454.56
Input layer, default CPR1.4130.6771.4840.7056.09

The input reference performs best on IND, OOD, and the overall average. Relative to the early-layer reference, it improves OOD P@1 by 2.46 points and the average by 1.53 points, at the same reported training cost. These comparisons directly support the reference choice.

Stability of the adaptive reference. We additionally evaluate the same fixed image panel before and after training. During the one-epoch run, successive minibatches contain different images, so raw step-wise PR values combine batch-composition effects with parameter changes. The matched panel provides a controlled comparison using identical per-image centering and rank normalization.

Table R1a. Reference stability and intermediate-layer recovery on FastVLM-0.5B. PR is computed exactly per image, divided by min⁡(Niimg−1,D)\min(N_i^{\mathrm{img}}-1,D), and then averaged over the same panel.

CheckpointInput PR / cap (L0)First-layer PR / cap (L1)Intermediate PR / cap (L12 / L19)
Pretrained, before training0.0499970.0602030.004965 / 0.006483
SFT, after training0.0495810.0606570.004805 / 0.005949
CRED, after training0.0500430.0623410.064396 / 0.070416

The mean input reference changes by only +0.092%, while the selected intermediate layers recover substantially. Thus, the recovery is accompanied by a stable mean reference rather than a net reduction of the target between the evaluated checkpoints. This supports using the student's own input representation as an adaptive anchor.

AuthorsOpenReview ↗

Q4 & W4–W5 · Methodological contribution and alternative regularizers

Q4 & W4–W5 · Methodological contribution and alternative regularizers

The methodological contribution is the combination of sample-adaptive, one-sided cross-layer supervision with an efficient matrix-free implementation for visual-token distillation. It accommodates heterogeneous teachers and students while avoiding intermediate teacher activations and feature matching.

The constraint also differs from coordinate-wise variance and covariance objectives, for which we include a matched control inspired by VICRegL (Bardes et al., 2022). A variance floor can be satisfied by increasing feature magnitudes without increasing directional diversity, whereas exact PR is invariant to uniform rescaling and orthogonal rotations. It permits different covariance spectra above the reference, without requiring an identity covariance. In particular, a centered token covariance has rank at most min⁡(N−1,D)\min(N-1,D); requiring both positive variance in all DD coordinates and zero off-diagonal covariance can conflict with this rank limit when N<DN<D.

For the Shannon control, we replace PR in the cross-layer hinge with Shannon effective rank computed on the same per-image centered covariances and detached input-reference convention. The comparison therefore changes the spectral statistic while retaining the rest of the framework.

Table R2. Regularization mechanisms within the same CRED framework, FastVLM-0.5B. All controls retain InfoNCE, embedding distillation, structural preservation, and the same selected layers. Complexity is per sample and regularized layer; NN is token count, DD hidden dimension, and M=256M=256 for the matrix-free estimator.

RegularizerComplexityTime (s/step)Peak memory (GB)IND P@1 (%)OOD P@1 (%)Avg. (%)
No CPR—1.3528.6570.1438.2154.18
VICRegL-style variance + covarianceO(ND2)O(ND^2)1.8130.9569.1039.9854.54
Exact Shannon effective-rank hingeO(NDmin⁡(N,D))O(ND\min(N,D))3.1430.6371.5039.9455.72
Exact PR hingeO(NDmin⁡(N,D))O(ND\min(N,D))1.7430.9771.4240.1255.77
Matrix-free CPRO(NDM)O(NDM)1.4130.6771.4840.7056.09

Matrix-free CPR achieves the highest OOD and overall average P@1 among the evaluated variants. It improves average P@1 over the VICRegL-style and Shannon eRank controls by 1.55 and 0.37 percentage points, respectively, while reducing their training-step times by 22.1% and 55.1%. Compared with the matched no-CPR baseline, CPR improves average P@1 by 1.91 points with an additional 0.06 s/step (4.4%). These comparisons support the proposed sample-adaptive PR constraint as an effective and computationally efficient regularizer within the same distillation framework.

Reviewer_jfTLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies dimensional collapse of image tokens during vision-language embedding distillation. Cross-layer participation regularization (CPR) uses the student's current input representation as a stop-gradient reference and penalizes intermediate layers whose participation ratio falls below it. Random projections reduce the computational cost. The full CRED framework also combines contrastive learning, distillation of the teacher's similarity distributions, and relational distillation of pooled representations. Experiments use two compact vision-language students and cover classification, candidate-based VQA, and out-of-distribution tasks, with analyses of the regularizer, projection budget, and choice of layers.

Critical strengths
  1. The regularizer is simple and accommodates heterogeneous teachers and students. CPR compares scalar spectral statistics within the student, without requiring intermediate teacher features or layer correspondence. Randomized estimation reduces the training step time from 1.74 seconds for exact PR to 1.41 seconds, with similar task performance. This supports its practical value as a low-cost distillation component.
  2. There is direct evidence for the component's benefit. In Table 4, adding CPR improves average classification scores from 66.01 to 68.04 for CKD and from 67.11 to 68.16 for HieRD. In Table 5, adding CPR to the full set of distillation losses improves in-distribution and out-of-distribution classification from 70.14 and 38.21 to 71.48 and 40.70. The evidence therefore goes beyond comparing the full system against weaker baselines.
Critical weaknesses
  1. The connection between the training proxy and dimensionality preservation remains incomplete. Section 3.1 measures Shannon effective rank on an uncentered token matrix concatenated across the batch, whereas Section 3.2 regularizes PR of per-image centered covariances. These statistics differ in whether they include between-sample variation. Even on the same statistical object, preserving PR does not guarantee preservation of the input's Shannon effective rank. For example, changing a normalized spectrum from (0.7,0.1,0.1,0.1)(0.7,0.1,0.1,0.1) to (0.5,0.5,0,0)(0.5,0.5,0,0) increases PR from about 1.9231.923 to 22, while Shannon effective rank falls from about 2.5612.561 to 22. This does not invalidate Proposition 1, but its lower bound alone cannot establish cross-layer information preservation. Consistent spectral diagnostics and evidence linking them to task performance are needed.
  2. It is unclear how much the benefit depends on the particular reference and PR constraint. The input projector remains trainable, so stopping the gradient does not fix the reference value. A decline in reference PR would relax the constraint, but its training trajectory is not reported. The existing ablations show that CPR helps, yet do not compare it with variance or decorrelation penalties applied at the same layers within the same training framework. VICRegL applies such constraints to local visual features [1] and offers a basis for a targeted control. The relevant comparison is between regularization mechanisms, rather than unmodified methods designed for different tasks. This would help establish CPR's value over simpler alternatives. [1] VICRegL: Self-Supervised Learning of Local Visual Features, Advances in Neural Information Processing Systems (NeurIPS), 2022.
Questions for authors
  1. Consistency of the diagnostics: With the same per-image centering convention, do PR, Shannon effective rank, and task performance still support the proposed explanation? Are there cases where the spectrum improves but task performance declines?
  2. Reference stability: How do input PR, regularized-layer PR, and the fraction of active penalties evolve during training? Can existing logs or a frozen-reference control rule out substantial relaxation caused by a falling reference?
  3. Contribution of the design: How do variance or decorrelation penalties compare with CPR in performance and cost at the same layers and within the same distillation framework? Does this comparison support the choice of the current PR constraint?
  4. Model selection: Which validation sets were used to select the regularization weight, projection budget, and layer range? The appendix uses out-of-distribution averages to motivate the defaults; did the final out-of-distribution test sets inform these choices? Also, which losses are used in the zero-weight row of Table 8 and the CRED-without-CPR row of Table 5?
6 comments
AuthorsOpenReview ↗

Q1 & Weakness 1 - Consistent spectral diagnostics and their relationship to task performance

Thank you for your thoughtful review and constructive feedback. We address your questions below with matched spectral diagnostics, reference measurements, and comparisons of regularization mechanisms.

We follow the standard spectral definitions of Shannon effective rank (Roy & Vetterli, 2007) and participation ratio (Gao et al., 2017). The additional analysis evaluates both metrics on the same per-image centered covariances. Table R1 shows that CRED's intermediate-layer dimensionality recovery persists under both standard metrics, alongside improved downstream performance.

Standard definitions and Proposition 1. For image-token representations Hi,img(l)H_{i,\mathrm{img}}^{(l)}, we compute:

=H_{i,\mathrm{img}}^{(l)}-\mathbf 1\mu_i^{(l)\top}, \qquad \Sigma_{i,\mathrm{img}}^{(l)} =\frac{\widetilde H_{i,\mathrm{img}}^{(l)\top} \widetilde H_{i,\mathrm{img}}^{(l)}}{N_i^{\mathrm{img}}-1}.$$ For a nonzero covariance matrix $\Sigma$, let $\{\lambda_j\}$ be its eigenvalues and $p_j=\lambda_j/\sum_k\lambda_k$. The standard formulas are $$\operatorname{erank}_{\mathrm{Shannon}}(\Sigma) =\exp\left(-\sum_jp_j\log p_j\right), \qquad \operatorname{PR}(\Sigma) =\frac{\left(\sum_j\lambda_j\right)^2}{\sum_j\lambda_j^2} =\frac{1}{\sum_jp_j^2}.$$ For a positive semidefinite matrix, singular values equal eigenvalues, so the Shannon formula follows the standard normalized-singular-value definition. Likewise, Eq. (3) uses $\sigma_j(H^{(l)})^2$, the eigenvalues of the uncentered second-moment matrix $H^{(l)\top}H^{(l)}$, and normalizes the resulting effective rank by $\min(N_l,D)$. Per-image centering removes each image's token mean, so the matched comparison focuses on within-image variation and excludes the contribution of differences between image means. **Proposition 1** establishes, for any nonzero positive semidefinite $\Sigma$, $$1\leq\operatorname{PR}(\Sigma) \leq\operatorname{erank}_{\mathrm{Shannon}}(\Sigma) \leq\operatorname{rank}(\Sigma).$$ Proposition 1 relates the two metrics on the same covariance. We use it to motivate PR as a tractable spectral surrogate; the evidence for cross-layer dimensionality recovery comes from the matched measurements in Table R1. **Table R1. Matched spectral diagnostics for FastVLM-0.5B and reported classification P@1 (%).** Each pair lists **layer 12 (50% depth) / layer 19 (~80% depth)**, the selected intermediate layers. Centered PR and Shannon effective rank are computed exactly on the same fixed image panel, divided by $c_i=\min(N_i^{\mathrm{img}}-1,D)$ per image, and then averaged across images. The original batch-concatenated, uncentered diagnostic is reported separately for comparison. | Method | Centered PR / cap (L12 / L19) | Centered Shannon eRank / cap (L12 / L19) | Original uncentered nERank (L12 / L19) | IND CLS P@1 | OOD CLS P@1 | | :--- | :--- | :--- | :--- | ---: | ---: | | SFT | 0.004805 / 0.005949 | 0.007691 / 0.012796 | 0.008861 / 0.013465 | 64.93 | 33.22 | | CKD | 0.004899 / 0.006282 | 0.008097 / 0.014359 | 0.009389 / 0.014676 | 66.01 | 32.84 | | RKD | 0.004884 / 0.006208 | 0.008026 / 0.013978 | 0.009295 / 0.014360 | 66.17 | 32.40 | | HieRD | 0.004781 / 0.005955 | 0.007593 / 0.012859 | 0.008763 / 0.013865 | 67.11 | 35.10 | | **CRED** | **0.064396 / 0.070416** | **0.130008 / 0.146036** | **0.058950 / 0.027283** | **71.48** | **40.70** | Relative to SFT, CRED's centered PR is approximately **13.4× / 11.8×** higher at layers 12 / 19, and its centered Shannon effective rank is **16.9× / 11.4×** higher. The recovery is therefore consistent across both matched centered metrics and the original visualization. **Connection to task performance.** Controlled ablations isolate the contribution of adding CPR: with both teacher-driven losses retained, IND/OOD classification P@1 improves from **70.14/38.21 to 71.48/40.70** (Table 5 of the submitted paper). Adding CPR to CKD and HieRD also improves IND classification by **2.03** and **1.05 percentage points**, respectively (Table 4). These results support the downstream usefulness of intermediate dimensionality regularization across multiple transfer objectives. **Task-specific tradeoffs.** CPR uses a one-sided hinge relative to each sample's input PR (Eq. (9)); the penalty becomes zero once the reference is reached. It discourages intermediate dimensionality reduction rather than continually maximizing PR. On FastVLM, adding CPR decreases HatefulMemes P@1 from **64.10 to 61.73**, alongside the improved classification averages above. Its binary decision may favor more selective compression of task-relevant variation, providing a plausible explanation for this task-specific tradeoff. The matched diagnostics and controlled ablations support CPR as complementary supervision for semantic learning.
AuthorsOpenReview ↗

Q2 & Weakness 2 (Part 1) - Reference stability and choice of reference

During our one-epoch training, successive steps process different images. Both PR and the fraction of active hinge penalties depend on image content and token count, so raw statistics from different minibatches are not directly comparable as a trajectory of model change: their variation combines changes in batch composition with changes in model parameters. We therefore address reference stability through a controlled comparison on the same fixed image panel before and after training, using exact per-image centered PR and identical normalization throughout.

Table R2. Input and first-layer normalized PR for FastVLM-0.5B. Layer 0 is measured after visual projection and before the Transformer blocks. Each per-image PR is divided by min⁡(Niimg−1,D)\min(N_i^{\mathrm{img}}-1,D) before averaging over the panel.

CheckpointLayer 0 PR / cap (Input Reference)Layer 1 PR / capIntermediate Layers 12 / 19 PR / cap
Pretrained (Step 0, before training)0.0499970.0602030.004965 / 0.006483
SFT (after training, w/o CPR)0.0495810.0606570.004805 / 0.005949
CRED (after training, w/ CPR)0.0500430.0623410.064396 / 0.070416

Evolution of input PR, regularized-layer PR, and active penalties. Together, Tables R1 and R2 address all three quantities:

  1. Input reference stability: Because the vision encoder is frozen and only the lightweight projector is updated alongside the detached sg[PR(0)]\mathrm{sg}[\mathrm{PR}^{(0)}], the mean Layer 0 reference changes only from 0.049997 to 0.050043 over training (+0.092%), indicating negligible reference drift.
  2. Regularized-layer recovery and active penalties: Without CPR (SFT), intermediate PR at layers 12 / 19 collapses to 0.004805 / 0.005949, approximately an order of magnitude below the Layer 0 reference (0.049997). This indicates that the one-sided hinge max⁡(0,sg[PR⁡^(0)]−PR⁡^(l))\max(0, \mathrm{sg}[\widehat{\operatorname{PR}}^{(0)}] - \widehat{\operatorname{PR}}^{(l)}) is active for most samples. After CRED training, while Layer 0 remains nearly unchanged (0.050043), layers 12 / 19 recover to 0.064396 / 0.070416, exceeding the mean Layer 0 reference. The negligible reference drift supports interpreting sg[PR(0)]\mathrm{sg}[\mathrm{PR}^{(0)}] as an effectively stable, sample-adaptive anchor without requiring a separate frozen-projector copy. Accordingly, we assess the concern through these controlled before/after measurements while retaining the sample-adaptive reference, rather than introducing an additional frozen-reference variant.

Dependence on the reference. We additionally compare the sample-adaptive input reference with a fixed PR threshold and an early hidden-layer reference. The selected regularization layers and remaining CRED objectives are held fixed.

Table R2b. Reference ablations within CRED on FastVLM-0.5B. Avg. is the mean of IND and OOD classification P@1. The fixed threshold is applied to unnormalized PR.

PR referenceIND P@1 (%)OOD P@1 (%)Avg. (%)
No CPR70.1438.2154.18
Fixed threshold, τ=12\tau=1270.9838.6454.81
Layer 5, approximately 20% depth70.8838.2454.56
Input layer, default CPR71.4840.7056.09

The input reference achieves the highest IND, OOD, and overall average among all tested references. Compared with the early-layer reference, it improves IND P@1 by 0.60 percentage points, OOD P@1 by 2.46 points, and the overall average by 1.53 points; it also outperforms the fixed threshold across both IND (+0.50 points) and OOD (+2.06 points). This supports using the sample-dependent diversity available before backbone processing as the reference. Table R3 separately tests the choice of regularization mechanism under the same distillation framework.

AuthorsOpenReview ↗

Q3 & Weakness 2 (Part 2) - Matched comparisons with variance, covariance, and effective-rank regularization

We compare regularization mechanisms within the same CRED framework. The InfoNCE, embedding-distillation, and structural-preservation objectives are retained, together with the same per-image token representations and selected intermediate layers. We replace CPR with either (i) VICRegL-style variance and covariance penalties, or (ii) a Shannon effective-rank hinge using the same per-image centering and detached input-reference convention.

Table R3. Alternative regularizers within CRED on FastVLM-0.5B. Avg. is the mean of IND and OOD classification P@1. Regularizer complexity is per sample and layer, with NN image tokens, hidden dimension DD, and M=256M=256 projections for CPR. Training time and peak memory use the same batch size and hardware.

RegularizerRegularizer complexityIND P@1 (%)OOD P@1 (%)Avg. (%)Training time (s/step)Peak GPU memory (GB)
No CPR (CRED w/o CPR)—70.1438.2154.181.3528.65
VICRegL-style variance + covarianceO(ND2)O(ND^2)69.1039.9854.541.8130.95
Shannon effective-rank hingeO(NDmin⁡(N,D))O(ND\min(N,D))71.5039.9455.723.1430.63
Exact PR hinge (Tables 3 & 9)O(NDmin⁡(N,D))O(ND\min(N,D))71.4240.1255.771.7430.97
Matrix-Free CPR (Ours, M=256M=256)O(NDM)O(NDM)71.4840.7056.091.4130.67

CPR achieves the highest OOD P@1 (40.70) and overall average (56.09). Compared with the VICRegL-style control, it improves the average by 1.55 percentage points and reduces training time. Compared with the Shannon hinge, it improves the average by 0.37 points and reduces training time by 55.1%, with nearly identical IND performance (71.48 versus 71.50).

These results support the choice of PR on both computational and empirical grounds. Its underlying spectral objective is invariant to uniform feature rescaling and orthogonal rotations, and its two spectral moments admit matrix-free estimation. Together with the sample-adaptive, one-sided cross-layer reference, this yields effective dimensionality supervision at substantially lower cost than the exact Shannon control, while achieving the strongest OOD and overall performance in this comparison.

Reviewer_HoxEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the degradation of intermediate image representations during vision-language model distillation and proposes CRED. The method augments standard distillation objectives with cross-layer participation-ratio regularization (CPR). It uses the student’s input-level image-token participation ratio as a reference and penalizes selected deeper layers when their value falls below it. The authors also introduce a randomized estimator for the participation ratio and evaluate CRED on two compact student models across classification, visual question answering, and visual grounding tasks.

Critical strengths
  • The paper examines changes in intermediate representations during distillation, rather than focusing solely on final outputs.
  • CPR does not require teacher activations or layer-wise teacher–student matching, making it relatively straightforward to apply.
  • The evaluation includes two student architectures, out-of-distribution results, and several ablations.
Critical weaknesses
  • The central assumption is insufficiently justified. CPR treats the input-level participation ratio as a lower bound for deeper layers, although reducing or reorganizing information across layers may be useful. The paper does not establish why maintaining this value should generally improve representations.
  • The link between participation ratio and representation quality is unclear. A higher PR indicates a more distributed feature spectrum, but does not necessarily imply more useful semantic information. The results show that PR and downstream performance improve together, but do not establish that the former causes the latter.
  • The technical novelty appears limited. The main contribution is a participation-ratio penalty added to existing distillation objectives, together with a randomized estimator. The paper should clarify how this differs in substance from more general variance or rank regularization.
  • The evaluation remains narrow. The main results use two 0.5B student models and one default teacher. The larger-teacher experiment covers only classification, leaving generalization across model scales and task types insufficiently established.
  • Gains are not uniform. CRED underperforms the non-distilled student on Hateful Memes with LLaVA-OneVision and is slightly worse on RefCOCO-Matching. The paper provides limited analysis of these regressions and of when CPR may harm task-specific representations.
  • The efficiency claims need broader support. The randomized estimator reduces computation in the reported setting, but the method still requires choices of projection budget, layer range, and regularization weight. Its cost at larger model sizes or longer image-token sequences is unclear.
  • The main results cover only two MMEB task types. MMEB includes additional task categories, but the main tables report fine-grained scores only for image classification and VQA. The paper should explain this choice and provide results for the full benchmark to show whether gains extend across task types.
Questions for authors
  • Why is the input-level participation ratio an appropriate lower bound for deeper layers? Have the authors compared it with references from other layers or with alternative statistics?

  • Can the authors compare PR regularization with standard feature-variance, covariance, or effective-rank regularizers? Such comparisons would help determine whether the gains are specific to the proposed design.

  • How do the authors establish that higher PR reflects more useful representations rather than simply a more dispersed feature spectrum? Can they provide analyses that connect PR changes to semantic performance?

  • Can the authors analyze the regressions on Hateful Memes and RefCOCO-Matching? Do these cases indicate that CPR can overconstrain representations?

  • Can the authors report sensitivity to projection budget, regularized layer range, and regularization weight across different architectures or token counts, along with the associated training costs?

  • Why do the main results cover only image classification and VQA, rather than all four MMEB task categories? Please clarify this choice and provide fine-grained results for the remaining categories.

9 comments
AuthorsOpenReview ↗

Official Response to Reviewer HoxE

Thank you for your thoughtful review and constructive feedback. We address your six questions and the corresponding weaknesses below.

 

Q1 & Weakness 1 · Input-level reference and alternative references

The input PR provides a sample-adaptive reference for visual diversity before the observed intermediate-layer reduction. Figures 2 and 4 show that this reduction already occurs in the pretrained student and persists under SFT and several distillation objectives. A reference taken from a deeper layer may therefore incorporate the reduction that CPR aims to counteract. The input representation, measured immediately after visual projection and before the Transformer blocks, instead anchors supervision to the visual-token spectrum available before backbone processing.

This reference enters a soft, one-sided regularizer. Equation (9) penalizes a selected layer only when its estimated PR falls below the same image's detached input PR. The penalty becomes zero once the reference is reached. CPR does not match feature coordinates or continually maximize PR; the contrastive and teacher-guided objectives determine task relevance. The default applies the penalty at two intermediate positions, ⌊0.5L⌋\lfloor0.5L\rfloor and ⌊0.8L⌋\lfloor0.8L\rfloor, with no direct CPR penalty on the final layer. For the 24-layer FastVLM backbone, these are L12 and L19.

Matched reference ablations support this choice. Table R1 compares input PR with a fixed threshold and an early-layer reference. The Shannon control changes the spectral statistic while retaining the per-image centered covariance, detached input-reference convention, selected layers, and remaining objectives.

Table R1. Reference and statistic ablations within CRED on FastVLM-0.5B. Avg. is the mean of IND and OOD classification P@1. The fixed threshold uses unnormalized PR.

RegularizerReferenceIND P@1 (%)OOD P@1 (%)Avg. (%)
None (CRED w/o CPR)—70.1438.2154.18
PR hingeFixed threshold τ=12\tau=1270.9838.6454.81
PR hingeLayer 5 (approximately 20% depth)70.8838.2454.56
Shannon eRank hingeInput layer71.5039.9455.72
CPR (default)Input layer71.4840.7056.09

The input-reference PR formulation achieves the highest OOD and overall average. Relative to the early-layer reference, it improves OOD P@1 by 2.46 points and the average by 1.53 points; relative to the fixed threshold, the corresponding gains are 2.06 and 1.28 points. Together with the controlled CPR additions in Q3, these comparisons provide an empirical justification for preserving intermediate dimensionality relative to the input, rather than treating the reference as a universally optimal rank.

AuthorsOpenReview ↗

Q2 & Weakness 3 · Differences from variance, covariance, and effective-rank regularization

Q2 & Weakness 3 · Differences from variance, covariance, and effective-rank regularization

The contribution is sample-adaptive cross-layer supervision of intermediate visual tokens, implemented efficiently without intermediate teacher activations or teacher–student layer matching. Its formulation differs from generic variance or decorrelation objectives in three ways.

First, exact PR measures relative spectral concentration:

=\frac{\left(\operatorname{tr}\Sigma\right)^2} {\operatorname{tr}\left(\Sigma^2\right)}.$$ It is invariant to uniform feature rescaling and orthogonal changes of feature coordinates. A coordinate-wise variance floor can be satisfied by increasing magnitudes without adding independent directions; coordinate-wise variance and off-diagonal covariance penalties also depend on the feature basis. These invariance statements concern the underlying exact spectral objective. Second, CPR adapts the target to each image and stops penalizing once that target is met. It permits different spectra above the reference without requiring identity covariance. A centered token covariance has rank at most $\min(N-1,D)$; simultaneously requiring positive variance in all $D$ coordinates and zero off-diagonal covariance can conflict with this cap when $N<D$. Third, PR depends on two spectral moments, enabling an $O(NDM)$ matrix-free estimator. The exact Shannon control requires recovering the full spectrum. Table R2 compares these mechanisms within the same distillation framework, including a variance–covariance control inspired by VICRegL (Bardes et al., 2022). **Table R2. Alternative regularizers within CRED, FastVLM-0.5B.** All variants retain InfoNCE, embedding distillation, structural preservation, and the same selected layers. Complexity is per sample and regularized layer for the benchmarked implementations. | Regularizer applied at the selected intermediate layers | Per-layer complexity | Time (s/step) | Peak memory (GB) | IND CLS P@1 (%) | OOD CLS P@1 (%) | Avg. CLS P@1 (%) | | :--- | :---: | ---: | ---: | ---: | ---: | ---: | | None (CRED w/o CPR) | — | 1.35 | 28.65 | 70.14 | 38.21 | 54.18 | | VICRegL-style variance + covariance decorrelation | $O(ND^2)$ | 1.81 | 30.95 | 69.10 | 39.98 | 54.54 | | Exact Shannon effective-rank hinge | $O(ND\min(N,D))$ | 3.14 | **30.63** | **71.50** | 39.94 | 55.72 | | Exact PR hinge | $O(ND\min(N,D))$ | 1.74 | 30.97 | 71.42 | 40.12 | 55.77 | | **Matrix-free CPR ($M=256$)** | **$O(NDM)$** | **1.41** | 30.67 | 71.48 | **40.70** | **56.09** | CPR achieves the highest OOD and overall average among the evaluated variants. It improves the average over the VICRegL-style and Shannon controls by **1.55** and **0.37 points**, while reducing their step times, respectively. Against the matched no-CPR baseline, it adds **0.06 seconds per step (4.4%)** and improves the average by **1.91 points**. These controls support the specific cross-layer PR formulation beyond the benefit of adding a generic diversity regularizer.
AuthorsOpenReview ↗

Q3 & Weakness 2 · Connecting spectral changes to semantic performance

Q3 & Weakness 2 · Connecting spectral changes to semantic performance

Our evidence combines matched spectral diagnostics with controlled additions of CPR. The latter hold the architecture and other objectives fixed: CPR improves classification P@1 for CKD by 2.03 points, HieRD by 1.05 points, and CRED by 1.34 IND / 2.49 OOD points on FastVLM. On LLaVA-OneVision, adding CPR to CRED improves 71.06/40.28 to 72.21/45.71, a gain of 1.15/5.43 points. Its usefulness also extends beyond distillation: adding CPR to InfoNCE alone improves FastVLM from 64.93/33.22 to 68.92/37.54.

For the spectral comparison, we follow the standard definitions of Shannon effective rank (Roy & Vetterli, 2007) and PR (Gao et al., 2017). For a nonzero per-image centered covariance Σ\Sigma, let pj=λj/∑kλkp_j=\lambda_j/\sum_k\lambda_k. Then

\qquad \operatorname{erank}_{\mathrm{Shannon}}(\Sigma) =\exp\left(-\sum_jp_j\log p_j\right).$$ For a positive semidefinite covariance, eigenvalues equal singular values, giving the standard normalized-singular-value Shannon definition on that matrix. Both metrics below use the **same fixed image panel, per-image centering, rank-cap normalization, and aggregation**. This aligns the diagnostic with CPR's statistical object. **Table R3. Matched per-image centered spectral metrics paired with downstream performance on FastVLM-0.5B.** Spectral entries report **L12 (50% depth) / L19 (~80% depth)**, normalized per image by $\min(N_i^{\mathrm{img}}-1,D)$. | Method | Centered PR / cap (L12 / L19) | Centered Shannon eRank / cap (L12 / L19) | IND CLS P@1 (%) | OOD CLS P@1 (%) | IND VQA P@1 (%) | OOD VQA P@1 (%) | | :--- | :--- | :--- | ---: | ---: | ---: | ---: | | SFT | 0.004805 / 0.005949 | 0.007691 / 0.012796 | 64.93 | 33.22 | 51.03 | 49.68 | | CKD | 0.004899 / 0.006282 | 0.008097 / 0.014359 | 66.01 | 32.84 | 51.38 | 50.03 | | RKD | 0.004884 / 0.006208 | 0.008026 / 0.013978 | 66.17 | 32.40 | 51.32 | 49.58 | | HieRD | 0.004781 / 0.005955 | 0.007593 / 0.012859 | 67.11 | 35.10 | 52.76 | 53.12 | | **CRED (Ours)** | **0.064396 / 0.070416** | **0.130008 / 0.146036** | **71.48** | **40.70** | **55.68** | **56.72** | Relative to SFT, CRED's centered PR increases by **13.4× / 11.8×** and centered Shannon effective rank by **16.9× / 11.4×** at L12 / L19. Proposition 1 relates the two statistics on the same covariance; the matched measurements establish their joint recovery, while the controlled ablations establish the downstream usefulness of adding CPR. PR is also invariant to uniform scaling, so this effect cannot be explained solely as an increase in total variance magnitude. The motivation is consistent with prior empirical work: RankMe (Garrido et al., 2023) uses effective rank to assess downstream representation quality, while Xia et al. (2026) report increases in PR rank alongside improved neighborhood purity and multimodal task performance. For CRED, the central evidence is the combination of its own matched diagnostics and interventions. The interpretation is a dimensionality constraint that complements semantic learning, rather than an assumption that greater PR alone always implies better semantics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.