acceptodds
Under review as a conference paper at ICLR 2027

Towards the Scaling Law of Neural Model Probes

Abstract

Probing, the practice of training a lightweight classifier on frozen model representations and reading off its accuracy, has become a popular tool for studying what neural network models encode. Yet probes themselves have received little systematic study: design choices such as architecture, hidden width, and initialization are often made arbitrarily, and probing accuracy alone cannot separate information present in the representation from what the probe learns on its own. In this paper, we study the science of probes, and ask whether their behaviors follow scaling laws analogous to those of model pretraining. We use minimum description length (MDL; Voita & Titov, 2020) as our measure of probe faithfulness, and study its relationship with complexity of MLP probes (i.e., hidden width) and their training data size across data modalities, analyzed models, tasks, layers, and initialization schemes. We find that (1) faithfulness is non-monotone in probe width, exhibiting a characteristic U-shape with an optimum typically between widths of 64 and 256; (2) this optimum can be predicted from simple representation-level features, namely the layer depth and the accuracy of a linear probe; (3) dataset size affects faithfulness monotonically and exponentially; and (4) the scale of probe initialization shifts the optimum in a log-linear fashion, while maximal update parametrization flattens the faithfulness–complexity curve. Together, these findings set the first steps toward establishing a scaling law of probing and offer practical guidance for designing and configuring probes.

Primary area:
interpretability and explainable AI
Keywords:
language model, probing, faithfulness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1LTPOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the probe itself, not the model behind it. Authors see probing as a kind of compression and use description length to measure how faithful a probe is. They ask if probe faithfulness follows scaling law like model pretraining, with respect to probe size, data size and initialization. Main findings are that there is a sweet spot in probe size, this sweet spot can be predicted from simple features of the representation, more data gives better faithfulness, and initialization changes the whole curve. All results are empirical, from many sweeps over text, vision and audio models.

Critical strengths
  • Probe width is usually chosen by habit, and this paper gives a systematic sweep over three modalities, seven models and many tasks with multiple seeds. Even if one does not fully buy the scaling law framing, the observation that very wide probes rarely improve MDL is useful practical evidence.
  • The initialization and muP part is the most interesting for me. Under muP the optimal base learning rate is almost constant across probe widths (Fig. 8b), and the default init scale often falls outside the near-optimal gamma range (Fig. 7). This gives actionable advice for people who need to train many probes.
Critical weaknesses
  • I think the central U-shape is not an intrinsic property of probe faithfulness but rather of the specific training recipe, and the paper itself partly shows this. All standard probes in Sec. 4 use a single learning rate 1e-3 for every width, which under standard parametrization gives width-dependent effective update size for the output layer (this is exactly the motivation of muP). In Sec. 6.2 the right branch disappears under muP. Moreover, Init-B (W2 = 0) with gamma = 0.5 still shows a clear U-shape (MDL 9179 at width 32 vs 9259 at width 2048, Fig. 18), although its W1 init seems the same as muP, so the main remaining difference looks like the output-layer learning rate scaling. Since there is no per-width learning rate tuning for standard probes and no sanity baseline such as a randomly initialized model or shuffled labels, it is not clear for me whether the optimal width, and Eq. 5 built on it, reflects the representation or just the optimizer setting. This directly affects the main claim and the practical guidance of the paper.
  • Several quantitative laws look fragile when checked against the numbers in the appendix. For Init-A, the log-linear fit in Table 3 depends mostly on the four points with gamma >= 1.7. When refit the values of Fig. 17 using only gamma <= 1.6, the slope drops to about 0.59 with R^2 about 0.11. In the gamma >= 1.7 region, 14 to 22 widths are within 10 bits of the minimum, so the argmin is basically noise. The 31 extra widths are sampled uniformly in linear scale (68% of candidates are above 256), so a noisy argmin is naturally pushed toward large widths. The qualitative trend that larger gamma reduces the penalty of wide probes looks real, but the log-linear law itself is not well supported. Also, Fig. 9 and 10 use data sizes up to 21,000, while factuality and SV agreement have only 16,436 and 15,830 training samples (still below 21,000 even with the test set added). In Table 5 several fitted parameters collapse to boundary values (e.g. a1 about 1e-251, c about 1e-11), which makes Eq. 6 hard to read as a law.
Questions for authors
  • Which data is used for data sizes 18,000 and 21,000 in Fig. 9 and 10? If test data or unfiltered data is included, please clarify.
  • Which optimizer is used in Sec. 4, and is the learning rate tuned per width?
  • In Eq. 4 the inner sum runs over j from 1 to t_i, which is the data already used to train the stage-i probe. Is this a typo for the next block (j from t_i + 1 to t_(i+1))? Please also define d_1 in Eq. 7, the activation function, the block schedule of online coding, and the split used for early stopping.
  • Is the hit rate of 53.4% and 75.0% in Sec. 7 computed on held-out models and tasks, or in-sample?
  • A sanity check with randomly initialized model representations or shuffled labels would show whether the U-shape and its optimum come from the representation or from the probe training itself.
3 comments
AuthorsOpenReview ↗

Response to reviewer 1LTP (1/2)

We thank the reviewer for their constructive feedback.

W1 (intrinsic property)

the central U-shape is not an intrinsic property of probe faithfulness

We believe that the μP\mu\text{P} finding does not contradict the U-shape finding; rather, both jointly support our main claim: different initialization strategies induce distinct, lawful scaling behaviors in probes, which is consistent with our title. We do not claim that the U-shape is an intrinsic property independent of the training recipe. In this paper, we investigate 3 initialization strategies, not limited to the standard initialization that most existing literature uses:

  • Standard (Sec 4--5): Under this regime, the faithfulness--complexity curve is typically U-shaped with an optimal width w∗w^*.
  • Scaled (Sec 6.1): Varying the initialization scale γ\gamma shifts the optimum in a log-linear fashion (Eq. 7), where smaller initialization scale (larger γ\gamma) requires larger probes for optimal faithfulness.
  • μP\mu\text{P} (Sec 6.2): μP\mu\text{P} flattens the U-shape and achieves comparable or even lower MDL (with learning rates 2−112^{-11}--2−72^{-7}) than standard initialization. The optimal learning rate becomes width-invariant, achieving zero-shot hyperparameter transfer.

Init-B (W2 = 0) with gamma = 0.5 still shows a clear U-shape ..., although its W1 init seems the same as muP, so the main remaining difference looks like the output-layer learning rate scaling.

The reviewer is correct that W1W_1 has the same initialization under μ\muP and Init-B with γ=0.5\gamma=0.5, but we would like to respectfully point out that the initialization of W2W_2 is different: Init-B uses W2=0W_2=0, whereas in μP\mu\text{P}, W2∼N(0,1/w2)W_2\sim\mathcal N(0,1/w^2), where ww is the probe hidden width. We follow Table 3 and Eq. 3 of Yang et al. (2022) for initialization and learning rate scaling.

whether the optimal width, and Eq. 5 built on it, reflects the representation or just the optimizer setting.

As outlined above, the reviewer's observation that training choices affect the scaling curve does not contradict our main claim. Sec 4--5 characterize the widely used standard configuration, so understanding its scaling behavior is directly relevant to how probes are currently used, while Sec 6 investigates how alternative configurations change probes' behavior. The two alternative configurations we examined in Sec 6 are motivated by prior work that demonstrates the effect of initialization on the scaling behavior of LLMs. Per-width learning-rate tuning for standard probes is a useful extension, but in this paper, we do not aim to exhaustively explore all possible configurations.

Together, our findings take the first steps toward establishing a scaling law of probing and offer practical guidance for designing and configuring probes.

W2 (robustness of quantitative relationships)

We thank the reviewer for this detailed analysis.

For Init-A, the log-linear fit in Table 3 depends mostly on the four points with gamma >= 1.7. When refit the values of Fig. 17 (new Fig. 16 in the revised manuscript) using only gamma <= 1.6, the slope drops to about 0.59 with R^2 about 0.11.

γ\gamma rangennSlope bγb_\gammaR2R^2
[0.3, 0.8]64.7590.775
[0.3, 0.9]72.6810.389
[0.3, 1.0]82.0020.321
[0.3, 1.1]91.8900.375
[0.3, 1.2]101.4530.304
[0.3, 1.3]111.1490.253
[0.3, 1.4]121.1240.296
[0.3, 1.5]131.2400.389
[0.3, 1.6]140.7100.142

We thank the reviewer for mentioning this important point. We repeated the regression over smaller ranges [0.3,γ^\hat{\gamma}], with γ^\hat{\gamma} varying from 0.8 to 1.6. All fitted slopes remain positive, indicating that the positive fitted association is not solely attributable to the points with γ≥1.7\gamma\geq1.7. We acknowledge that the slope magnitude and R2R^2 depend on the fitting interval, which is exactly why we report the overall slope and R2R^2 as a summary of the full range.

AuthorsOpenReview ↗

Response to reviewer 1LTP (2/2)

W2 (robustness of quantitative relationships), continue

In the gamma >= 1.7 region, 14 to 22 widths are within 10 bits of the minimum, so the argmin is basically noise.

All reported MDL results are averaged over 5 random seeds, which could reduce the variance of the estimator. In the future version, we will also analyze near-optimal width ranges.

The 31 extra widths are sampled uniformly in linear scale (68% of candidates are above 256), so a noisy argmin is naturally pushed toward large widths.

The additional widths were included to improve resolution between the increasingly widely spaced candidates on the logarithmic grid (e.g., 4096 -> 8192).

Fig. 9 and 10 (new Fig. 8 and 9 in the revised manuscript) use data sizes up to 21,000, while factuality and SV agreement have fewer training samples.

This was an error in the manuscript. Thank you very much for pointing it out.

We found more English treebank corpora (LinES and Atis) to extend the SV agreement dataset size beyond what we initially reported in Sec. 4.1. For the factuality task, our initial dataset is imbalanced (64% positive, 36% negative, 22,797 total), and the number 16,436 is the dataset size after we randomly dropped some positive examples to balance the dataset. The balanced dataset is used in Sec. 4.1, but we use the full version in Sec. 5.2 (updated Fig. 8 and 9).

In total, we have 22,604 training sentences for SV agreement and 22,797 training sentences for factuality task, which are sufficient for the analysis in Sec. 5.2. We have updated the paper in Sec. 4.1 to reflect this.

Several fitted parameters collapse to boundary values […] which makes Eq. 6 hard to read as a law.

Near-zero coefficients indicate that some components contribute negligibly in a particular setting; for example, the constant term (c) may be close to zero. Importantly, a1≈0a_1\approx0 does not eliminate the first term, since its coefficient is a1+dx2a_1+d x_2. These values (even near 0) do not invalidate the fitted relationship by themselves.

Questions

Which data is used for data sizes 18,000 and 21,000 in Fig. 9 and 10? If test data or unfiltered data is included, please clarify.

Test data were not included. All three task datasets have more than 21,000 data entries in their training splits. For factuality, we use the full, unbalanced training dataset (22,797 examples) in Sec. 5.2, rather than the balanced subset used in Sec. 4.1.

Which optimizer is used in Sec. 4, and is the learning rate tuned per width?

We use AdamW as the optimizer, and the learning rate is fixed for all widths (1e-3). It was not tuned per width.

In Eq. 4 the inner sum runs over j from 1 to t_i, which is the data already used to train the stage-i probe. Is this a typo for the next block (j from t_i + 1 to t_(i+1))? Please also define d_1 in Eq. 7, the activation function, the block schedule of online coding, and the split used for early stopping.

Yes, this is a typo. Thank you very much for the careful review. We have updated our manuscript. The formula should be

= \sum_{i=1}^{S-1} \sum_{t_i < j \leq t_{i+1}} \bigl(y_j - f_{\theta_i}(x_j)\bigr)^2.$$ Here, $\theta_i$ is the probe trained on the first $i$ blocks $\{x^\ell_{1:t_{i}}, y_{1:t_{i}}\}$. > Is the hit rate of 53.4% and 75.0% in Sec. 7 computed on held-out models and tasks, or in-sample? It is in-sample. However, the width-based guessing baselines (45.0\% and 49.2\%, respectively) are also in-sample, so the comparison is fair. > A sanity check with randomly initialized model representations or shuffled labels would show whether the U-shape and its optimum come from the representation or from the probe training itself. This is a good idea, and we tested the SOMO task on randomly initialized llama-3.1-8b. We observed the following width-MDL values: `[10080.51 10072.17 10062.05 10061.14 10061.80 10062.39 10062.08 10061.71 10061.72 10061.52 10062.00 10060.86]`. All these MDL values are close to a random guess (random guess baseline: $15000\ln 2=10397$), and there is no U-shape behavior, meaning that the U-shape and its optimum mainly come from the representation.

Thank you for the detailed response and the new control. My main concerns remain.

1. The U-shape is not shown to come from the representation. The random-initialization control shows a flat curve when there is nothing to learn. Any training recipe would give that result, so it cannot tell whether the optimum comes from the representation or from the optimizer. The paper's own results point to the optimizer. Init-B (γ = 0.5) and µP start from nearly the same probe. Yet the U-shape survives in Init-B and disappears under µP, and the main difference left is that µP shrinks the output-layer learning rate as width grows. The rebuttal says the U-shape is not claimed to be recipe-independent, and later says it "mainly comes from the representation". Both cannot be true. Standard probes are never trained with a learning rate tuned for each width. Without that, the "characteristic U-shape" and the "representation-level" predictors in Eq. 5 cannot be separated from this particular training recipe.

2. Eq. 5 is only evaluated on the data used to fit it. The hit rates are in-sample. Nothing shows that Eq. 5 works on models or tasks it was not fitted on, which is what "avoiding hyperparameter sweeping" requires.

3. The log-linear trend in Sec. 6.1 comes from the two ends of the γ range. Every sub-interval in the rebuttal starts at the smallest γ. Once the two smallest values are dropped, the middle range shows no trend. At large γ, two almost identical widths (e.g., 362 and 364) differ in MDL by about as much as the gaps that decide the argmin, so the chosen optimum there is mostly noise. The justification given for the random widths (4096 → 8192) is outside the width range of this section, and those widths are still concentrated at large values.

4. The "exponential" data-size law is not tested against simple alternatives. On the paper's own Fig. 9 values, a power law with the same number of parameters fits most curves better. An exponential fitted on the smaller sizes predicts the largest size no better than simply reusing the last observed value. The near-zero values in Table 5 are fits stuck at the constraint boundary. Removing the constraint makes NLL increase with more data at small widths. That points to a model that is not identifiable, not to terms that are negligible. Faithfulness is also defined as total MDL elsewhere but measured as per-sample NLL here.

5. The revised data description does not match the results. Sec. 4.1 now says all tasks are label-balanced but gives the unbalanced factuality count. The paper never states that Sec. 5.2 uses unbalanced data. For SV agreement, Fig. 1 and the same data size in Fig. 9 give different optimal widths. SOMO, whose data did not change, gives matching results. It is unclear which data each figure actually uses.

Minor. My question about the early-stopping split was not answered. The released code picks the epoch using the block that is being encoded, which is not a valid online code unless that selection is accounted for. The effect is small. The activation function and the block schedule are still not reported.

Reviewer_ABwcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary
  • the authors studied how the quality of two-layer MLP probes evolves as the hidden width and initialization schemes change
  • they use minimum description length (MDL) as a measure of such probe quality, motivated by the fact that simpler probes should be favored over complex ones. Lower MDL means better quality.
  • They take different existing text, vision and audio models. They show that, as the hidden width is increased, the MDL first decreases, then either plateaus or goes back up, depending on the model.
  • They fit a linear regression model to predict the optimal hidden width based on layer index and accuracy of a linear probe, separately for text-based and vision-based models (R^2 = 0.29 and 0.56, respectively).
  • They show that the weight initialization scheme impact the optimal hidden size, and the muP initialization completely suppresses the U-shape.
Critical strengths

Probing is an interesting topic to study. We value the fact that information is easily extractable from representations, raising the need for a different measure than probe accuracy.

Critical weaknesses
  • Poor analyses of the effect of dataset size, which would probably invalidate/weaken two of the four main claims.
    • one of the main claim of the paper is the presence of U-shaped curves. In the main text, these curves are shown only for a fixed dataset size. In supplementary figure 10, we see that the U-shapes disappear as the dataset size increases. Hence, the conditions under which these U-shapes appear remain unclear, and the suggested guidelines may not apply in other scenarios.
    • Another main claim: MDL decreases "exponentially" with dataset size. This claim seems to be made based on observation of decreasing curves (e.g. figure 4). There seem to be two problems with this claim:
      • The plots it refers to show a decrease of NLL (=MDL/dataset size) with increasing dataset size, not MDL.
      • The decrease doesn't seem to be exponential, and could instead be explained by the dataset size in the denominator. In fact, as the dataset size grows, the model codelength (in the computation of MDL) become negligible, and only the average data codelength matters, which could explain the saturation of the curves in fig 4 and fig 9. This doesn't need anything exponential. Also, the exponential fit doesn't match the data well in some plots of fig 9 (e.g. subplot (e), h=2048).
  • The claim "The optimum is predictable from cheap, representation-level features" is a bit strong. The R^2 is rather low for text-based models, and the fitted coefficients vary a lot between different models. It is also not clear to me how this can be used to guide probe design (see my questions for authors below).
Questions for authors
  • the conclusion says "practitioners can fit a customized scaling law with layer depth and linear-probe accuracy as predictors to guide complexity selection (Eq. 5), avoiding hyperparameter sweeping". How do you fit the scaling law without computing the optimal width? It seems difficult to use parameters fitted for another model/task as they appear to vary a lot (table 2).
  • Do you have an explanation on why the fitted coefficients for the layer index (b_l) have opposite signs between text and vision models? (table 2)
1 comment
AuthorsOpenReview ↗

Response to Reviewer ABwc

We thank the reviewer for their feedbacks.

W1 (dataset size analysis)

We would like to respectfully clarify some points which we believe are a misunderstanding on the reviewer’s side:

In supplementary Fig 10 (note: it's the new Fig 9 in the revised manuscript), we see that the U-shapes disappear as the dataset size increases

This is not the case in the paper. Fig 10 (new Fig 9) is a heatmap that shows the NLL given both probe hidden size and dataset size. The optimal hidden size (lowest NLL for each row) is highlighted in red. It is clear that even with large dataset size, the optimal hidden size is still between 4 and 2048, meaning that U-shapes still exist regardless of the dataset size. In fact, we can conclude that as long as the dataset size is reasonably large (≥\geq 3000), the optimal hidden size is stable. Thus, one can use a small dataset to determine the minimum point of the U-shape curve and then use a larger dataset to achieve the best probing performance.

The plots it refers to show a decrease of NLL (=MDL/dataset size) with increasing dataset size, not MDL.

Using NLL is the intended way for the analysis when dataset size is a factor. According to Eq. 2 in our original manuscript,

Lonline(y1:n∣x1:nℓ)=t1log⁡2K−∑i=1S−1log⁡2pθi(yti+1:ti+1∣xti+1:ti+1ℓ)L _{\text{online}}(y _{1:n} \mid x^\ell _{1:n}) = t _1 \log _2 K - \sum _{i=1}^{S-1} \log _2 p _{\boldsymbol{\theta} _i}(y _{t _i+1:t _{i+1}} \mid x^\ell _{t _i+1:t _{i+1}})

The total online MDL is strictly increasing with the increase of dataset size nn. In fact, MDL measures the codelength required to encode the entire dataset. So, when we compare the probe faithfulness across different dataset sizes, the fairer approach is to calculate the average MDL (NLL).

The decrease doesn't seem to be exponential, and could instead be explained by the dataset size in the denominator.

Again, NLL measures the probe faithfulness with respect to the codelength for a unit length of dataset. Although dataset size nn appears in the formula NLL=MDLn\text{NLL} = \frac{\text{MDL}}{n}, that alone can't explain the full behavior of the NLL curve.

In fact, as the dataset size grows, the model codelength (in the computation of MDL) become negligible, and only the average data codelength matters, which could explain the saturation of the curves in Fig 4 and Fig 9 (note: new Fig 8 in the revised manuscript).

We don't think there is evidence showing that the model codelength becomes negligible, and only the average data codelength matters.

Also, the exponential fit doesn't match the data well in some plots of Fig 9 (e.g. subplot (e), h=2048).

Outliers and large variance occur when the probe size is very large (2048) while the dataset size is very small (1500). Nevertheless, all curves have an R2R^2 over 0.9.

W2 & Q1 (low r2)

The claim "The optimum is predictable from cheap, representation-level features" is a bit strong. The R^2 is rather low for text-based models, and the fitted coefficients vary a lot between different models. It is also not clear to me how this can be used to guide probe design

Despite the lower R2R^2, Eq. 5 is practically useful for narrowing down a width choice. Please refer to the 3rd paragraph of section 7 in our manuscript for details.

Q2 (opposite sign)

Do you have an explanation on why the fitted coefficients for the layer index (b_l) have opposite signs between text and vision models? (table 2)

We can share our hypothesis, though we have not investigated it yet: the image models we test are all image encoders, meaning that deeper layers encode more information and are easier to probe. This is not necessarily true in text models, as they are decoder models whose jobs are to predict the next token, and different layers contain different information.

Reviewer_qenAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In their manuscript, titled ‘Towards the scaling law of neural model probes’, the authors investigate various properties of probes: small auxiliary decoders designed to track hidden representations in the main neural models. Their work is motivated by a fundamental question: how can the quality of the hidden representation being probed be disentangled from the computations performed by the probe during decoding? To address this question, the authors conduct a numerical study exploring the effects of probe size, weight initialisation, and training dataset size. They also propose empirical relationships between optimal probe size and other hyperparameters.

Unfortunately, both the novelty of the key finding and the depth of its interpretation seems insufficient for a main-track ICLR paper. Furthermore, the applicability of the empirical formula to practical problems has not been convincingly demonstrated.

Critical strengths

-The manuscript provides useful illustrations of the sizing effects that may be of interest to the broader machine learning community.

-The authors cover a wide range of modalities, including language, vision, and speech, and the literature review appears comprehensive and well written.

Critical weaknesses
  1. The novelty relative to the key reference, Voita & Titov (2020), is not clearly articulated. The definition of minimum description length (MDL) in Voita & Titov (2020) makes it clear that a trade-off must exist between probe loss and the description penalty: beyond a certain probe size, the penalty will outweigh the benefit of further increases in size. It is therefore unclear why the U-shaped MDL curve can be claimed as a non-trivial or novel finding.

The authors’ claims regarding the sweet spot of this U-shaped curve, in the width range 64–256, remain anecdotal and have neither been systematically verified nor analytically supported. Moreover, in cases such as Figures 1a and 2c, the U-shaped curve is highly skewed and difficult to distinguish from a plateau.

  1. The fitting formulae are not adequately explored. In particular, Equations 5 and 8 are neither compared with alternative rules nor motivated by a clear analytical rationale. It is therefore unclear how these results should be interpreted. For example, how should one judge whether an R2R^2 of approximately 0.25–0.6 (Tables 2 and 3) represents a good or a trivial result?

    In Table 3 and Figure 6, the fit is performed over an excessively broad range, 0.25≤γ≤2.00.25 \le \gamma \le 2.0. It is unclear how this range was selected, given that the original paper by Yao et al. (2025) uses a much narrower interval, 0.3≤γ≤0.80.3 \le \gamma \le 0.8. Furthermore, for γ≠0.5\gamma \ne 0.5, the result is expected to be sensitive to the data width d1d_1.

Questions for authors

Minor comments

  1. The exponential dependence of NLL on dataset size requires stronger justification. In many panels of Figure 10, discrepancies are clearly visible at the first two or three data points, where most of the decay occurs. To demonstrate the usefulness of the proposed relationship, it would be helpful to assess whether the asymptotic NLL for a large dataset can be reasonably estimated from small dataset sizes.

  2. The motivation for introducing the term ‘faithfulness’, rather than retaining the notion of ‘MDL’ from Voita & Titov (2020), is unclear. Moreover, the two terms are sometimes used interchangeably within a single paragraph (e.g., lines 461–464), potentially causing confusion.

  3. The learning rate result seems poorly connected to the paper’s other results. Could the authors better motivate this connection?

  4. Using xx as a variable in the regression fits (Equations 5 and 6) is confusing. I recommend using zz instead.

  5. d1d_1 should be defined before it is used.

  6. Many figures report unnormalised losses of several thousand nats. Normalising these values, for example by data length, would make the manuscript easier to read.

  7. There is a gap in the figure numbering: Figure 4 is followed by Figure 6, with no Figure 5.

3 comments
AuthorsOpenReview ↗

Response to reviewer qenA (1/2)

We thank the reviewer for their helpful feedbacks.

W1.1 (Novelty)

It is therefore unclear why the U-shaped MDL curve can be claimed as a non-trivial or novel finding.

Our contribution is not the invention of MDL itself, but applying it to answer a fundamental, unstudied question: how does a probe's behavior depend on its own design, and do scaling laws like those of model pretraining exist for probes? Developing a rigorous science of probes requires using theoretically grounded tools, and MDL is the established information-theoretic metric for measuring probe faithfulness.

Furthermore, our analysis of predicting the optimal width and how initialization strategies (e.g., μP\mu\text{P}) impact probing behaviours under scaling is novel. As an analytical study, our core contribution lies in uncovering these phenomena and empirical connections, rather than inventing new evaluation metrics.

W1.2 (Evidence of the sweet spot)

In Fig 1a and 2c, the U-shaped curve is highly skewed and difficult to distinguish from a plateau

As we unified the y-axis for all figures under the same setting, some curves might seem flat. When each curve is plotted with its own y-axis, the U-shape is clear. Also, the reported optimal widths are selected from MDL averaged over five seeds.

claims regarding the sweet spot of this U-shaped curve, in the width range 64–256, remain anecdotal and have neither been systematically verified nor analytically supported

The statement that optimal widths "generally fall between 64 and 256" summarizes our empirical results. To make it explicit, across nine text settings (3 models ×\times 3 tasks), four have an optimal width (w∗w^{*}) of 64, four of 256, and one of 128. All 9 settings have an optimal width within the reported range.

W2 (Fitting formula)

Equations 5 and 8 are neither compared with alternative rules nor motivated by a clear analytical rationale.

Any fitting should begin with the simplest, most straightforward form: a linear model. We deliberately make Eq. 5 and 8 simple and interpretable. For Eq. 5, we first identify all possible factors that could predict w∗w^*, and then test whether they are significant predictors (line304--315). Layer index and linear-probe accuracy are retained because they are consistently significant across both modalities. The same rationale applies to Eq. 8: since we aim to predict the optimal probe width under different values of γ\gamma, γ\gamma is the natural predictor.

In other words, our linear predictions are already "baselines". It is natural to expect that a more complex form would give stronger prediction.

How should one judge whether an R2R^2 of approximately 0.25–0.6 (Tables 2 and 3) represents a good or a trivial result?

First, in our Table 2--3, the R2R^2 range is 0.29--0.6.

Second, there is no universal R2R^2 threshold that indicates whether a prediction is useful. Here, the relevant question is whether the predictions help narrow the width search.

Even for Eq. 5 with relatively lower R2R^2, it is practically useful for narrowing down a width choice, as shown in line 474--481 of our paper. These results provide a practical interpretation of the regression performance beyond R2R^2.

In Table 3 and Figure 6 (note: it's new figure 5 in the revised manuscript), the fit is performed over an excessively broad range, 0.25≤γ≤2.00.25 \leq \gamma \leq 2.0. How this range was selected?

First, we want to respectfully clarify that γ\gamma is varied "from 0.3 to 2.0 in steps of 0.1" (line 401). We will update Fig. 5 to better reflect this. We select a broader range than Yao et al. (2025) to investigate whether initialization effects extend beyond the values previously studied.

for γ≠0.5\gamma \neq 0.5, the result is expected to be sensitive to the data width d1d_1.

Variance-preserving initialization is an empirical rule of thumb, particularly for stabilizing variance through deep networks, but it is not a requirement for every architecture or experimental setting, especially given that our probes are shallow MLPs. Dimension-dependent initialization scale is therefore a property that we want to investigate.

AuthorsOpenReview ↗

Response to reviewer qenA (2/2)

Q1 (dependence of NLL on dataset size)

it would be helpful to assess whether the asymptotic NLL for a large dataset can be reasonably estimated from small dataset sizes.

Fig. 10 (new Fig. 9) already shows that the optimal width is highly consistent across dataset sizes. Our current results suggest a practical possibility: a practitioner could use a smaller dataset (e.g., 3,000 or 6,000 examples) to identify the optimal width, then use that width with a larger dataset.

Q2 ("faithful" vs "MDL")

As stated in the introduction, "faithful" names the property we want to assess: a probe should "achieve high performance while relying minimally on its own capacity", so any valid measure must reflect both parts: the quality of the probe's performance, and the cost to achieve it. We believe MDL is the most suitable metric to operationalize this property---it is theoretically grounded, and widely adopted in interpretability (Perez et al., 2021; Du et al., 2023, inter alia).

Q3 (connection between learning rate results and other parts)

μ\muP (Maximal Update Parametrization; Yang et al., 2022) is designed to facilitate hyperparameter transfer. The authors show that under μ\muP, the optimal hyperparameters (especially the learning rate) remain stable as width increases, unlike under standard parametrization. Therefore, a practitioner could parametrize the target model in μ\muP, tune its hyperparameters on a smaller model, and zero-shot transfer them to the full-sized model.

This original motivation for μ\muP leads us to examine whether the optimal learning rate also remains stable across probe widths when probes are parameterized using μP\mu\text{P}.

Q4 & 5 & 7 (suggestions on writing)

Thank you for your suggestions. We will use a better notation in the future version. We have defined d1d_1 (which denotes the input dimension) and unified the numbering in the updated manuscript.

Q6 (normalize loss)

Many figures report unnormalised losses of several thousand nats.

We report total codelength because this is the quantity by definition. It is also reported in unnormalized form in Voita & Titov (2020). Within each fixed-dataset comparison in Sec. 4, normalization by dataset size only rescales the values by the same constant, without changing the curve's shape and optimal width.

References

Ethan Perez et al., "True few-shot learning with language models." In NeurIPS, 2021.

Kevin Du et al., "Generalizing backpropagation for gradient-based interpretability." In ACL, 2023.

I would like to thank the Authors for their rebuttal. Unfortunately many of the initially voiced concerns remain unaddressed.

(W1) For example, my question about the nature of U-shape: whether it is a straightforward consequence of the penalty term or not, has not been addressed.

(W1.2) Furthermore, the scaling argument and multiplicity of seeds seem unrelated to my concern about skew.

(W2) Much of the reply seems unclear. In particular, I am still puzzled how sweep over γ\gamma with a single and fixed value of d1d_1 addresses the ‘dimension-dependent’ initialisation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Update in Manuscript

We thank all reviewers for their helpful feedbacks. We have updated our manuscript to address several questions raised by the reviewers:

  • Section 4.1: In the “Tasks” paragraph, we clarify the dataset size for 3 tasks.
  • Section 4.2: In the “Camera Parameter Regression” paragraph, we corrected a typo in Eq. 4.
  • Section 6.1: We added the definition of d1d_1 (input dimension).
  • We fixed the gap in figure numbering (originally Figure 4 is followed by Figure 6, with no Figure 5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.