acceptodds
Under review as a conference paper at ICLR 2027

No Single Point Suffices: Reporting Regularization Landscapes in Frozen-Representation Evaluation

Abstract

Frozen-representation evaluation compares pretrained representations through probe performance, but the measured advantage depends on the evaluation protocol. We study signed comparison gaps with ridge probes on three music benchmarks and an exploratory NLP/vision extension, distinguishing shared-penalty paths from independent validation-based selection. In the primary MTG-Jamendo configuration the sampled ROC-AUC gap has range 0.04432 and endpoint contrast 0.04156 (nominal 95% artist-cluster-bootstrap CI [+0.00765, +0.07980], 10,000 resamples), remaining positive at all ten sampled coefficients; simultaneous bands admit a constant gap curve. Both inherit the declared grid's extent: trimming the upper bound to the two arms' sampled peak reduces the endpoint contrast to +0.0060 and the range to 0.00877. Within the gap is not distinguishable from a constant gap under the paired cluster bootstrap (); the movement the bootstrap resolves is the post-peak rise from to . Restricting each arm to validation-near-optimal coefficients reveals a different reporting limitation: at , two of 22 cross-domain comparisons — each with a singleton common-coefficient set of zero range — admit independently acceptable test gaps of both signs, while the other twenty remain sign-consistent; matching effective complexity leaves four of 22 comparisons sign-changing. These exploratory envelopes are descriptive, not uncertainty-adjusted ranking claims. Different evaluation questions are therefore not interchangeable: a narrow shared-coefficient summary can omit variation under independent choices.

Primary area:
datasets and benchmarks
Keywords:
representation learning, evaluation methodology, linear probing, regularization, reproducibility, music representation evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_GcM7OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors study the problem of probe gap between pretrained models. They compare the performance of CLAP and MERT-95M on music benchmarks with ridge probes.

Critical strengths

None

Critical weaknesses
  • The paper is poorly-written
    • It is not clear what is the problem of interest exactly, the limitations of existing methods, and the contributions of this work.
    • The experiment settings are not clearly stated.
    • Some notations are used before they are defined.
  • The conclusion of this paper is not clear.
Questions for authors
  • What is the problem of interest?
  • What is the contribution of this work, compared with previous works?
  • What is the experiment setting, and what can we learn from the results?
  • What is the conclusion this paper?
Reviewer_nnfrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how regularization choices affect the evaluation of frozen pretrained representations using linear probes. Instead of reporting a single performance score, the authors analyze how performance gaps between models change across different regularization strengths and hyper-parameter selection strategies. Experiments across music, audio, NLP, and vision tasks show that regularization can affect the magnitude of performance differences. The paper suggests reporting regularization sensitivity alongside standard evaluation scores to provide a more complete comparison of pretrained representations.

Critical strengths

The paper raises a relevant concern about the sensitivity of representation comparisons for pretrained backbone.

Critical weaknesses
  1. The paper is difficult to follow. The manuscript appears heavily AI-assisted, and lacks sufficient human editing and a clear logical structure.
  2. The main observation that regularization affects Linear Probe performance and model comparisons is already well known. Moreover, the results in Sections 5, 7, and 8 provide limited evidence of statistically significant ranking changes under practical hyperparameter selection. The paper therefore is not novel and convincing.
Questions for authors

Could the authors clarify what practical limitations of standard linear probing their proposed reporting approach addresses? Is there evidence that it leads to more reliable model comparisons?

Reviewer_7jvcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how the comparison between two frozen pre-trained encoders, measured through ridge probes, depends on the probe’s regularization. The main experiments compare CLAP and MERT-95M on MTG-Jamendo. The authors report that the performance gap varies along the grid while keeping its sign. They also report that restricting the comparison to coefficients near-optimal for both encoders can hide sign changes that appear when each encoder chooses independently. Finally, they propose reporting practices for tuned comparisons.

Critical strengths

N/A

Critical weaknesses
  • While the research question of assessing the best way to compare two different pre-trained encoders can in principle be interesting, the paper is very hard to understand, and the overall message is very unclear since it is never stated plainly.
  • The primary evidence comes from the music domain, using a single encoder pair and one probe family. Such experiments cannot support general recommendation to the community. I would suggest a significant expansion of the experimental setup, with a broader set of benchmarks, models and probe families.
  • I find it very hard to interpret the results reported in Sections 5 to 8. The paper does not clearly include a discussion of their results, making it extremely difficult to understand the message they are trying to convey.
  • The authors use a high precision to report their numerical results, which interrupts the text flow. I would recommend using a different scale and fewer digits in the text
  • The notation is unclear. Most of the time, the symbols used are not previously defined.
Questions for authors

● What is the single, most important takeaway of the paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.