- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper provides a theory-backed separation between information which "exists" in a presentation, information that a restricted probe of that representation can actually access, and information that a model could learn from limited samples.
They very neatly define the differences, prove claims about the scope of those definitions, as well as a witness condition based on xor's in parity.
Empirical evidence supports all the theorems and claims, which hold tightly for the parity regime.
The mathematical details often exceed my direct area of expertise, particularly the denser coding-theoretic portions. My evaluation therefore focuses primarily on the conceptual contribution, internal consistency of the claims, clarity of assumptions, and alignment between the theoretical and empirical components.
Critical strengths
- Primary concepts of the paper are hyper concise, and focused on narrowly. The focus around witness complexity is tight and effective, and any sprawling from the main message are exact and relevant.
- None of the results seem to disagree either in favor or against any of the theoretical claims and bounds
- Scope being tightly connected to parity is honest and constructive
Critical weaknesses
The only weakness I can even remotely conceive of is practical usefulness. I thought this may have been a quantum paper when I first started reading, perhaps this is feedback more than a weakness.
Questions for authors
- The paper convincingly separates recoverability, accessibility, and learnability. What practical conclusion should a researcher designing probes or evaluating representations draw from these results?
- Witness complexity appears to be the central quantity throughout the paper. Are there existing representation-learning settings where witness complexity can be estimated or approximated in practice, or is it primarily a theoretical diagnostic?
- The paper argues that probe failure need not imply absence of information. How should the results change how readers interpret negative probing results in modern representation-learning settings?
3 comments
I appreciate the clarifications and the time put into the response period. There is, however, not much I can do to improve the score given I already like the paper and cannot see any revisions as of yet.
Thank you for following up. The revised PDF is on the submission page as a clean version, so the changes are incorporated into the text rather than highlighted. We should have pointed you to the relevant sections more clearly.
For practical probing (Q1), Sections 6.1 and 6.3 (pp. 7, 9) specify the evaluation metrics and probe-training scope. Section 6.4 and Appendix I.1 (pp. 9, 25–26) illustrate the gap between information in a fixed representation and what a trained decoder can extract. Section 6.5 and Appendix H (pp. 9, 22–24) extend this distinction to a continuous neural setting.
For witness complexity (Q2), Sections 3–4 (pp. 4–5) give the exact characterization for parity representations, while Appendix F.1 (p. 18) also treats grouped interfaces. The Limitations (p. 10) clarify the scope of these results and their applicability beyond parity.
For negative probing (Q3), Section 6.3 (p. 9) distinguishes low-degree score correlation from threshold-classification accuracy and clarifies the scope of the chance-level result. Appendix I.1 (pp. 25–26) further illustrates why a failed trained decoder need not imply missing information.
We hope these pointers make the relevant material easier to locate. Thank you again for your feedback and positive assessment.
Response to Reviewer gbHS
Thank you for your positive feedback! The practical implication is that a probe result should be interpreted relative to the interface and learning procedure that produced it.
1. Practical probing. A probe study should specify (i) the observed interface (all coordinates, a subset, or allowed interactions), (ii) the function class and training objective, and (iii) the sample and compute budgets. This separates missing global information from interface-level unreadability and failure of a finite training procedure. Positive controls, learning curves, and stronger-interface references are useful when feasible. Our decoder-refitting experiment illustrates the distinction: on the same frozen parity matrices, fresh decoder training raises bit accuracy from about 0.52 to 0.86, while exact Bayes block accuracy and Mean Rec remain unchanged. A failed trained probe therefore need not imply that the representation lacks information.
2. Witness complexity. For a known parity matrix, rq(A)=minA⊤v=q∣v∣ can be computed exactly by bounded-order enumeration or syndrome search. This gives an exact unreadability certificate below the witness order. For general neural embeddings, the parity matrix and Walsh support are unknown, and a practical analogue is an accessibility profile over nested coordinate subsets, interaction orders, or feature groups. Such a profile is an empirical diagnostic, not an exact witness certificate.
3. Negative probing results. Under the uniform-prior parity assumptions, a balanced target is independent of every sub-threshold coordinate subset, so even a Bayes classifier is at chance. Zero correlation with low-degree polynomial scores is weaker: the three-coordinate BSC example reaches 0.68225 threshold accuracy despite zero degree-at-most-two score correlation. Thus a negative probe result is evidence about the tested interface, function class, loss, and training budget—not automatically about all information in the representation.
The continuous experiment illustrates the same distinction beyond parity. A rotation of the final normalized codebook preserves full Bayes information, while changing which signals are exposed through one- or two-coordinate interfaces. Across 20 encoders, access-aware mixing reduces best-two-coordinate Bayes accuracy relative to three prespecified Haar rotations by 4.48, 4.49, and 1.73 percentage points for triangle, left-half, and XOR (paired pointwise 95% intervals exclude zero). Unrestricted full-coordinate linear, quadratic, and MLP probe classes are invariant under orthogonal changes of basis, although finite-training accuracy can differ. A negative local probe and a successful global probe can therefore both be correct: they test different interfaces.