- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper formalizes the idea that models have a shared linear structure of representations. The metric they introduce, which they term the “transport”, is the correlation of the scores of the same linear probe transferred across models using a “stitch”: an unsupervised method which simply finds the best-fit linear transformation from one space to the other. They decompose this correlation metric into a within-class and between-class component, where it turns out that the within-class component dominates for a probe that flags rarely. They also show that the more obvious metric, the AUC transfer gap, splits into a label transfer and geometry transfer component, and that the geometry part is negligible in practice.
They empirically validate their formula for the transport using the case of a probe which predicts whether a jailbreak will lead to an unsafe response. Moreover, they find that probes transport better from small to large models than large to small.
Critical strengths
The small-to-large transfer result could be useful, as you could imagine training activation monitors more cheaply on jailbroken transcripts from smaller models and then just applying a stitch to transfer to a larger model. Larger models can be harder to jailbreak and more expensive to generate from, so these would be significant savings.
Critical weaknesses
If you just read the main text, it seems like the main contribution is Theorem 4: the abstract says “we derive a transport law”, it is the only theorem (everything else is a lemma or corollary) and it is the subject of the headline figure. However, in B.7 the authors admit “The transport law is not new: canonical analysis goes back to Hotelling (1936), and the fact that the regression of one model’s representation on another’s decomposes into separate canonical components is set out in textbooks of multivariate statistics.” This reads to me as very disingenuous and causes me to conclude that a large fraction of the presented work is not novel.
The narrative for rejecting the AUC transfer gap is also unclear. Observation 3 says that the AUC transfer gap is zero “for every label”, but I think what it really mean is every label shared by both models? In that case, if two models have the same labels and the same geometry, what is wrong with the AUC transfer gap being zero? Why would you call that degenerate? The text following the observation only adds to the confusion, stating “a measured nonzero gap must then be explained not by a difference of spaces” but the the contrapositive of Observation 3 is that a non-zero gap implies that the two spaces are not linearly equivalent, so it actually does imply “a difference of spaces”? This is key to the story of the paper, where the AUC transfer gap is rejected in favor of the transport metric of Theorem 4, but is very poorly explained. In fact, the transport metric seems to suffer from the same issue as the AUC gap “function of the triple (representation, representation, label), not of the pair of representations” since it depends on the label via the probe w. Figure 2 presents an empirical argument against using the AUC transfer gap but this is discussed much later.
Questions for authors
- Can you clearly explain why you reject the AUC gap in favor of the transport statistic? Is it for theoretical or empirical reasons?
- How do you imagine these jailbreak-input probes being used in practice?
- Figure 3B: Why is tau decreasing with depth (vs. r peaking ~.25), and why is it greater than 1 in early layers?
- Could you please expand on the threat modeling? Why is the probe applied to the prompt rather than the response? If I wanted to actually use this to detect jailbreak inputs, I would want to know what kind of classification (AUC or recall or FPR) am I going to get depending on whether I train a probe directly or carry over from a smaller model. In Appendix I.3 I think it is saying the AUCs that you get even without transfer are <80%, which seems way too low to be useful? Having a small AUC gap doesn’t mean anything if the baseline AUC is so low. Moreover, is it not a major problem that the transfer “drops by half” on a different prompt source?
- Could you please work to improve the clarity of the writing? There are lots of unnecessary clauses which appear LLM-generated. Lots of “it’s not X, it’s Y”, lots of writing out numbers rather than just referring to plots. The text could be improved structurally for instance by starting with a threat model which motivates the design choices in the paper. The phrase “guards’ verdicts” in particular was very confusing to me, it sounds like two guards disagreeing with each other, but in fact it is the same guards giving a verdict on two different responses.
- The conclusion drops a bombshell that the transfer “drops by half” with distribution shift. What does this mean exactly? This should have been discussed properly in the results.
We thank you again for the references.
We have attached a revised version that adds the references you listed. It also includes the new results and a reworked narrative, since we agree that the pre-revision version was messy.
We would be grateful if you could take the time to look at the revised manuscript and consider revisiting your score. If anything in your assessment remains open, we would be glad to address it in the remaining window.