acceptodds
Withdrawn from ICLR 2027

Same Score, Same Capabilities? Regularizing Factorized Representations for Transfer

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Similar scores need not imply equally reusable components. We ask whether the selection readout should match the representation interface used downstream. A population construction shows that identical full-representation validation, transfer and spectra can conceal different component utility. FACTOR-MAE controls separate weight-decay strength from same-strength dependence regularization. Utility-Constrained Regularization Selection (UCRS) screens strength, then maximizes backbone transfer subject to unique-code non-decrease. Separate selection and evaluation cohorts test whether this constraint improves downstream component reuse. Exchanging code and trunk guards reverses preference across student targets, with a 0.250-point interaction. A second, prediction-preserving interface intervention yields reciprocal preferences in both ViT-Ti and ResNet18, with interactions 0.313/0.312 points. These results support matching the selection readout to the consumer's interface, rather than simply adding a probe. Unique-only tradeoffs and retained student losses delimit the benefit: UCRS is neither student-optimal nor a guarantee of loss-free reuse. The anonymized code, data registry and experiment records are available at https://anonymous.4open.science/r/ID20229.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, representation factorization, representation transfer, regularization selection, component utility, weight decay

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ZniCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that two models with the same validation score may not be equally useful downstream, especially when a downstream model uses only one part of the representation rather than the whole backbone. A simple construction (Proposition 1) shows that swapping coordinates can keep validation and transfer accuracy unchanged while changing the usefulness of a fixed coordinate. To study this, the authors train a ViT from scratch (FACTOR-MAE) with a reconstruction branch and an understanding branch, and split its features into shared and unique codes. They then propose UCRS, a two-step rule for choosing regularization. First, the weight-decay strength is picked by a short kNN run. Then MI regularization is added only if backbone transfer improves and the unique code does not get worse.

Critical strengths
  1. Weight decay is compared against a calibrated same-strength reference, a control most regularization papers omit.
  2. Negative results are reported candidly, including the never-binding constraint, suite reversals and a false rejection.
Critical weaknesses
  1. Gains are tiny (about 0.12 points at 75% accuracy), and UCRS underperforms default weight decay.
  2. The constraint rarely acts: it never changes a decision in six settings and only three of nine.
  3. Reported data are hard to trust. For ViT-S/IN-100, Table 1/A1 select λ∗=0.4\lambda^* = 0.4, while Table A5 selects λ=0.2\lambda = 0.2, and the same transfer value 45.70 is attributed to both. For ViT-B/IN-100, the shared-kNN screening scores are 62–65 in Table A5 but 16.84–17.64 in App. C. In Table 4, all four directional contrasts have the same SD (0.120), and two different UCRS-vs-unique-only quantities share identical statistics (+0.022, 0.030, [.008, .037]).
  4. The paper is very hard to read.
Questions for authors
  1. Please explain the data problems listed in Weakness 3. Why does the selected λ\lambda for ViT-S/IN-100 differ between Table 1/A1 and Table A5? Why are the ViT-B/IN-100 screening scores so different between Table A5 and App. C? And why are the statistics in Table 4 so regular? Also, are the models in the nine-condition study the same ones reported in Table 1?
  2. Could you release the raw per-block results behind Tables 4 and A11?
  3. How much does the distilled ViT-Ti vary across random seeds? Is a 0.12-point difference outside that range?
Reviewer_XBmTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the limitations of validation performance in predicting downstream transfer performance and proposes UCRS, a two-stage hyperparameter selection strategy that combines early weight-decay screening with utility-constrained regularization selection. Experiments evaluate its effectiveness across different ViT architectures and downstream tasks.

Critical strengths
  1. The analysis in this paper provides some meaningful observations.
  2. The component analysis experiments in this paper are comprehensive.
Critical weaknesses
  1. Clarity of presentation. The writing should be further improved. The experimental setups and analyses are not clearly presented. Frequent defensive statements also affect readability and obscure the main findings.
  2. Additional computational overhead. UCRS introduces substantial computational overhead, while the trade-off between additional costs and performance gains remains insufficiently analyzed.
  3. Limited analysis of intermediate weight-decay strengths. Figure 6 mainly examines a few WD λ\lambda values (0.05, 0.4, and 0.8). Further analysis of how transfer performance, effective rank, image spectra, and patch alignment evolve between WD = 0.4 and 0.8 would strengthen the findings.
  4. Lack of comparisons with existing hyperparameter optimization methods.
Questions for authors
  1. A more comprehensive evaluation of the computational overhead and associated trade-offs between performance and efficiency is needed.

  2. Additional analyses of the intermediate WD λ\lambda would strengthen the findings.

  3. Comparisons with established hyperparameter optimization or selection methods under comparable computational budgets are needed to better demonstrate the effectiveness of UCRS.

Reviewer_grjfOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Utility-Constrained Regularization Selection (UCRS), a method that controls weight decay and mutual information regularisation with the goal of improving the transfer capabilities of a given backbone. The authors first show there can be a mismatch between transfer accuracy and common metrics such as in-distribution accuracy or singular values rank. They then test their method on various ViT backbones.

Critical strengths
  1. S1 - Problem and approach: Exploring differences in pre-training accuracy and transfer-learning capabilities is an interesting research topic. Proposition 1, although difficult to understand in its current formulation, is a good toy example to show there can be a mismatch between the two, while common metrics such as singular-value rank stay constant.
Critical weaknesses
  1. W1 - Clarity: This paper is critically unclear. Writing is very dense, and many terms are never introduced. Mathematical formulations use symbols that were never defined. Overall, the paper is very difficult to follow.
  2. W2 - Results: The results in the paper are not very convincing. For example, in Table 1, in all cases but one, changes in both empirical and code probe accuracies are in the same direction, undermining the argument that there is an empirical difference between in-distribution and transfer capabilities of the backbones. In Tables 3 and 5, UCRS is rarely achieving the best results.
  3. W3 - Significance: My understanding is that UCRS is, in the end, essentially a grid search over weight decay and mutual information regularisation. Taking into account W2, the potential significance of the method appears to be limited.
Questions for authors
  1. Q1: The weight decay parameters explored in this paper are quite high for ViT-based models. Why is that the case?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.