- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ProtoT V2, it build upon the original architecture (ProtoT, ICML 2026) by improving its computational efficiency, numerical stability, and language modeling performance. Additionally, it introduces a synthetic-prototype-based editing mechanism to enable more flexible and targeted model interventions.
Critical strengths
This paper builds upon the original ProtoT architecture and introduces several improvements in computational efficiency and model editability. These improvements make the architecture more practical and flexible.
Critical weaknesses
Although the proposed method shows promising results, the current experiments do not demonstrate a clear advantage over existing baselines in terms of editing performance, particularly activation steering methods. This raises questions about the unique benefits of the proposed editing mechanism.
Questions for authors
Given that conventional Transformers can already achieve comparable or even better behavioral editing results through activation steering, what unique advantages does ProtoEdit offer over these existing approaches?
We thank the reviewer for their time and suggestions! We will add any new comments and presentation improvements to the camera-ready version of the paper upon acceptance.
We agree that ProtoEdit does not outperform the strongest activation-steering methods on raw edit success, and we do not claim that it does: it is competitive with them (e.g., 0.973 vs. 0.964 for DiffMean on Neg→Pos).
ProtoEdit's distinct contribution is that it is architecture-native: rather than adding a fixed direction to the residual stream, it inserts the edit as an explicit synthetic prototype that competes with the learned prototypes in the model's own write-gate routing, so the edit is applied in proportion to how strongly each token routes to it. This shows that the prototype mechanism is a practical place to edit model behavior, not only to inspect it. The routing competition matters: with the same records, a variant that scores the same gate outside the write-gate softmax ends at twice ProtoEdit's toxicity (0.095 vs. 0.047) and at higher disfluency (14.34 vs. 11.85) (Appendix A.7), and ProtoEdit has the lowest perplexity among all arms above 0.85 on Pos→Neg. Unlike the subtractive prototype editing of prior work, it is also not limited to concepts the model has already isolated in a prototype.
Our main finding from the editability results is that: ProtoT v2 is a more editable backbone than LLaMA-3. To test this, we applied the same steering methods (ActAdd and the four AxBench recipes), with an identical tuning and selection protocol, to ProtoT v2 and to a parameter-matched LLaMA-3 trained on the same data with the same tokenizer (Section 5.2, Table 4). Measured as gain over each backbone's own unedited model (Appendix Table 22), the same method gives a larger gain on ProtoT v2 in 10 of 15 (method, condition) comparisons, a smaller one in 3, with 2 indistinguishable. Using non-overlapping 95% prompt-level bootstrap CIs on these gains as a conservative criterion, ProtoT v2 is significantly better for ActAdd on all three conditions, for PCA on both sentiment directions, and for DiffMean and LAT on detoxification. The gaps can be large: ActAdd flips 0.745 vs. 0.472 of reviews on Pos→Neg, and reaches 0.038 vs. 0.102 toxicity. On detoxification, DiffMean, LAT and ActAdd reach 40–63% lower toxicity on ProtoT v2, even though its unsteered baseline is slightly more toxic (0.2315 vs. 0.2157). The advantage is not uniform (LLaMA-3 is better for DiffMean on Neg→Pos and for Probe and LAT on Pos→Neg), so we read it as method-dependent rather than universal. Since none of these baselines access the prototypes, this suggests that part of ProtoT v2's editability lies in its representations themselves, and conventional steering methods benefit from it directly.