- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper measures changes in the final-layer hidden state at the last prompt token under formatting perturbations. It reports that separator and enumeration edits produce larger activation distances, that activation distance correlates with changes in restricted next-token label distributions, and that formatting metadata predicts activation distance.
Critical strengths
The paper uses matched prompt pairs, multiple open-weight models, task-held-out evaluation, and several robustness analyses. The experimental pipeline is extensive and clearly documented.
Critical weaknesses
-
The central activation measure is strongly confounded by tokenization and token identity. Formatting perturbations can change the final input token and its position, yet the paper compares final-layer states at that token. The reported separator hierarchy may therefore primarily reflect comparing representations of different terminal tokens rather than a meaningful prompt-level representational shift.
-
The activation–output association is largely expected because the measured hidden state is directly projected into the next-token logits. The paper does not demonstrate that activation distance provides useful information beyond output-side quantities. Its incremental predictive value is very small: JSD MAE does not improve, while answer-flip AUC increases only from 0.8082 to 0.8128.
-
PFSS generalizes across held-out tasks but not unseen formatting operations. Training and test sets share the same small perturbation vocabulary, making high R^2 compatible with memorizing transition-specific distance scales. The evaluation therefore does not establish general prediction of prompt sensitivity.
Questions for authors
- Do the results hold when base and perturbed prompts have identical terminal tokens, identical token counts, and aligned measurement positions?
- Does activation distance outperform logit distance, margin change, or output entropy for predicting answer instability?
- Can PFSS generalize to formatting transitions entirely excluded during training?