- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the probe itself, not the model behind it. Authors see probing as a kind of compression and use description length to measure how faithful a probe is. They ask if probe faithfulness follows scaling law like model pretraining, with respect to probe size, data size and initialization. Main findings are that there is a sweet spot in probe size, this sweet spot can be predicted from simple features of the representation, more data gives better faithfulness, and initialization changes the whole curve. All results are empirical, from many sweeps over text, vision and audio models.
Critical strengths
- Probe width is usually chosen by habit, and this paper gives a systematic sweep over three modalities, seven models and many tasks with multiple seeds. Even if one does not fully buy the scaling law framing, the observation that very wide probes rarely improve MDL is useful practical evidence.
- The initialization and muP part is the most interesting for me. Under muP the optimal base learning rate is almost constant across probe widths (Fig. 8b), and the default init scale often falls outside the near-optimal gamma range (Fig. 7). This gives actionable advice for people who need to train many probes.
Critical weaknesses
- I think the central U-shape is not an intrinsic property of probe faithfulness but rather of the specific training recipe, and the paper itself partly shows this. All standard probes in Sec. 4 use a single learning rate 1e-3 for every width, which under standard parametrization gives width-dependent effective update size for the output layer (this is exactly the motivation of muP). In Sec. 6.2 the right branch disappears under muP. Moreover, Init-B (W2 = 0) with gamma = 0.5 still shows a clear U-shape (MDL 9179 at width 32 vs 9259 at width 2048, Fig. 18), although its W1 init seems the same as muP, so the main remaining difference looks like the output-layer learning rate scaling. Since there is no per-width learning rate tuning for standard probes and no sanity baseline such as a randomly initialized model or shuffled labels, it is not clear for me whether the optimal width, and Eq. 5 built on it, reflects the representation or just the optimizer setting. This directly affects the main claim and the practical guidance of the paper.
- Several quantitative laws look fragile when checked against the numbers in the appendix. For Init-A, the log-linear fit in Table 3 depends mostly on the four points with gamma >= 1.7. When refit the values of Fig. 17 using only gamma <= 1.6, the slope drops to about 0.59 with R^2 about 0.11. In the gamma >= 1.7 region, 14 to 22 widths are within 10 bits of the minimum, so the argmin is basically noise. The 31 extra widths are sampled uniformly in linear scale (68% of candidates are above 256), so a noisy argmin is naturally pushed toward large widths. The qualitative trend that larger gamma reduces the penalty of wide probes looks real, but the log-linear law itself is not well supported. Also, Fig. 9 and 10 use data sizes up to 21,000, while factuality and SV agreement have only 16,436 and 15,830 training samples (still below 21,000 even with the test set added). In Table 5 several fitted parameters collapse to boundary values (e.g. a1 about 1e-251, c about 1e-11), which makes Eq. 6 hard to read as a law.
Questions for authors
- Which data is used for data sizes 18,000 and 21,000 in Fig. 9 and 10? If test data or unfiltered data is included, please clarify.
- Which optimizer is used in Sec. 4, and is the learning rate tuned per width?
- In Eq. 4 the inner sum runs over j from 1 to t_i, which is the data already used to train the stage-i probe. Is this a typo for the next block (j from t_i + 1 to t_(i+1))? Please also define d_1 in Eq. 7, the activation function, the block schedule of online coding, and the split used for early stopping.
- Is the hit rate of 53.4% and 75.0% in Sec. 7 computed on held-out models and tasks, or in-sample?
- A sanity check with randomly initialized model representations or shuffled labels would show whether the U-shape and its optimum come from the representation or from the probe training itself.
3 comments
Response to reviewer 1LTP (2/2)
W2 (robustness of quantitative relationships), continue
In the gamma >= 1.7 region, 14 to 22 widths are within 10 bits of the minimum, so the argmin is basically noise.
All reported MDL results are averaged over 5 random seeds, which could reduce the variance of the estimator. In the future version, we will also analyze near-optimal width ranges.
The 31 extra widths are sampled uniformly in linear scale (68% of candidates are above 256), so a noisy argmin is naturally pushed toward large widths.
The additional widths were included to improve resolution between the increasingly widely spaced candidates on the logarithmic grid (e.g., 4096 -> 8192).
Fig. 9 and 10 (new Fig. 8 and 9 in the revised manuscript) use data sizes up to 21,000, while factuality and SV agreement have fewer training samples.
This was an error in the manuscript. Thank you very much for pointing it out.
We found more English treebank corpora (LinES and Atis) to extend the SV agreement dataset size beyond what we initially reported in Sec. 4.1. For the factuality task, our initial dataset is imbalanced (64% positive, 36% negative, 22,797 total), and the number 16,436 is the dataset size after we randomly dropped some positive examples to balance the dataset. The balanced dataset is used in Sec. 4.1, but we use the full version in Sec. 5.2 (updated Fig. 8 and 9).
In total, we have 22,604 training sentences for SV agreement and 22,797 training sentences for factuality task, which are sufficient for the analysis in Sec. 5.2. We have updated the paper in Sec. 4.1 to reflect this.
Several fitted parameters collapse to boundary values […] which makes Eq. 6 hard to read as a law.
Near-zero coefficients indicate that some components contribute negligibly in a particular setting; for example, the constant term (c) may be close to zero. Importantly, does not eliminate the first term, since its coefficient is . These values (even near 0) do not invalidate the fitted relationship by themselves.
Questions
Which data is used for data sizes 18,000 and 21,000 in Fig. 9 and 10? If test data or unfiltered data is included, please clarify.
Test data were not included. All three task datasets have more than 21,000 data entries in their training splits. For factuality, we use the full, unbalanced training dataset (22,797 examples) in Sec. 5.2, rather than the balanced subset used in Sec. 4.1.
Which optimizer is used in Sec. 4, and is the learning rate tuned per width?
We use AdamW as the optimizer, and the learning rate is fixed for all widths (1e-3). It was not tuned per width.
In Eq. 4 the inner sum runs over j from 1 to t_i, which is the data already used to train the stage-i probe. Is this a typo for the next block (j from t_i + 1 to t_(i+1))? Please also define d_1 in Eq. 7, the activation function, the block schedule of online coding, and the split used for early stopping.
Yes, this is a typo. Thank you very much for the careful review. We have updated our manuscript. The formula should be
= \sum_{i=1}^{S-1} \sum_{t_i < j \leq t_{i+1}} \bigl(y_j - f_{\theta_i}(x_j)\bigr)^2.$$ Here, $\theta_i$ is the probe trained on the first $i$ blocks $\{x^\ell_{1:t_{i}}, y_{1:t_{i}}\}$. > Is the hit rate of 53.4% and 75.0% in Sec. 7 computed on held-out models and tasks, or in-sample? It is in-sample. However, the width-based guessing baselines (45.0\% and 49.2\%, respectively) are also in-sample, so the comparison is fair. > A sanity check with randomly initialized model representations or shuffled labels would show whether the U-shape and its optimum come from the representation or from the probe training itself. This is a good idea, and we tested the SOMO task on randomly initialized llama-3.1-8b. We observed the following width-MDL values: `[10080.51 10072.17 10062.05 10061.14 10061.80 10062.39 10062.08 10061.71 10061.72 10061.52 10062.00 10060.86]`. All these MDL values are close to a random guess (random guess baseline: $15000\ln 2=10397$), and there is no U-shape behavior, meaning that the U-shape and its optimum mainly come from the representation.Thank you for the detailed response and the new control. My main concerns remain.
1. The U-shape is not shown to come from the representation. The random-initialization control shows a flat curve when there is nothing to learn. Any training recipe would give that result, so it cannot tell whether the optimum comes from the representation or from the optimizer. The paper's own results point to the optimizer. Init-B (γ = 0.5) and µP start from nearly the same probe. Yet the U-shape survives in Init-B and disappears under µP, and the main difference left is that µP shrinks the output-layer learning rate as width grows. The rebuttal says the U-shape is not claimed to be recipe-independent, and later says it "mainly comes from the representation". Both cannot be true. Standard probes are never trained with a learning rate tuned for each width. Without that, the "characteristic U-shape" and the "representation-level" predictors in Eq. 5 cannot be separated from this particular training recipe.
2. Eq. 5 is only evaluated on the data used to fit it. The hit rates are in-sample. Nothing shows that Eq. 5 works on models or tasks it was not fitted on, which is what "avoiding hyperparameter sweeping" requires.
3. The log-linear trend in Sec. 6.1 comes from the two ends of the γ range. Every sub-interval in the rebuttal starts at the smallest γ. Once the two smallest values are dropped, the middle range shows no trend. At large γ, two almost identical widths (e.g., 362 and 364) differ in MDL by about as much as the gaps that decide the argmin, so the chosen optimum there is mostly noise. The justification given for the random widths (4096 → 8192) is outside the width range of this section, and those widths are still concentrated at large values.
4. The "exponential" data-size law is not tested against simple alternatives. On the paper's own Fig. 9 values, a power law with the same number of parameters fits most curves better. An exponential fitted on the smaller sizes predicts the largest size no better than simply reusing the last observed value. The near-zero values in Table 5 are fits stuck at the constraint boundary. Removing the constraint makes NLL increase with more data at small widths. That points to a model that is not identifiable, not to terms that are negligible. Faithfulness is also defined as total MDL elsewhere but measured as per-sample NLL here.
5. The revised data description does not match the results. Sec. 4.1 now says all tasks are label-balanced but gives the unbalanced factuality count. The paper never states that Sec. 5.2 uses unbalanced data. For SV agreement, Fig. 1 and the same data size in Fig. 9 give different optimal widths. SOMO, whose data did not change, gives matching results. It is unclear which data each figure actually uses.
Minor. My question about the early-stopping split was not answered. The released code picks the epoch using the block that is being encoded, which is not a valid online code unless that selection is accounted for. The effect is small. The activation function and the block schedule are still not reported.
Response to reviewer 1LTP (1/2)
We thank the reviewer for their constructive feedback.
W1 (intrinsic property)
We believe that the μP finding does not contradict the U-shape finding; rather, both jointly support our main claim: different initialization strategies induce distinct, lawful scaling behaviors in probes, which is consistent with our title. We do not claim that the U-shape is an intrinsic property independent of the training recipe. In this paper, we investigate 3 initialization strategies, not limited to the standard initialization that most existing literature uses:
The reviewer is correct that W1 has the same initialization under μP and Init-B with γ=0.5, but we would like to respectfully point out that the initialization of W2 is different: Init-B uses W2=0, whereas in μP, W2∼N(0,1/w2), where w is the probe hidden width. We follow Table 3 and Eq. 3 of Yang et al. (2022) for initialization and learning rate scaling.
As outlined above, the reviewer's observation that training choices affect the scaling curve does not contradict our main claim. Sec 4--5 characterize the widely used standard configuration, so understanding its scaling behavior is directly relevant to how probes are currently used, while Sec 6 investigates how alternative configurations change probes' behavior. The two alternative configurations we examined in Sec 6 are motivated by prior work that demonstrates the effect of initialization on the scaling behavior of LLMs. Per-width learning-rate tuning for standard probes is a useful extension, but in this paper, we do not aim to exhaustively explore all possible configurations.
Together, our findings take the first steps toward establishing a scaling law of probing and offer practical guidance for designing and configuring probes.
W2 (robustness of quantitative relationships)
We thank the reviewer for this detailed analysis.
We thank the reviewer for mentioning this important point. We repeated the regression over smaller ranges [0.3,γ^], with γ^ varying from 0.8 to 1.6. All fitted slopes remain positive, indicating that the positive fitted association is not solely attributable to the points with γ≥1.7. We acknowledge that the slope magnitude and R2 depend on the fitting interval, which is exactly why we report the overall slope and R2 as a summary of the full range.