- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents data on linear-probe decodability, output discrimination, and probe-direction interventions across six sizes of Pythia, eight training checkpoints, and four binary tasks. It reports near-ceiling decodability very early on, intervention scores below reference early on and then moving towards a score for random-direction reference later on. Two propositions explore how decodability and intervention efficacy change under various assumptions.
Critical strengths
-
The experimental design addresses an important question that is typically not addressed by final-checkpoint analyses, particularly when probe accuracy is useful for steering.
-
The propositions establish a conditional distinction between decodability and intervention efficacy, providing a possible geometric explanation for their separation.
Critical weaknesses
-
The reported aggregates do not establish consistency across tasks. Each cell is a median of four signed effects (one for each task), which could conceal opposing task-level effects. In addition, random-direction spread needs to be distinguished from sampling uncertainty. The null-distribution and dependence assumptions underlying Section 5.2's sign calculation and independent-normal median argument need to be justified.
-
The intervention target lacks an explicit token and sign convention. Although Equation 3 suggests that a positive-class token is fixed, the prompt and label mapping are not specified. If each example's correct token is used instead, opposing class effects could cancel out in the signed average. This is a conditional concern that could be resolved by providing an explicit fixed-class definition.
Questions for authors
- [Clarification] Do the task-level effects and sampling uncertainty support the same trajectory, and what justifies the null assumptions in Section 5.2?
- [Clarification] Is the answer token fixed across classes, and how is the probe direction oriented? An example from each class would resolve the ambiguity.