- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper extends the Univeral SAE, which trained an SAE across three different vision models sharing a transformer backbone, and proposes the X-SAE, a cross-model SAE for different vision architectures trained across architectures. They train the X-SAE across DINOv2 and the PixArt diffusion model, and report cross-architecture and per token evaluations.
Critical strengths
The paper is very well written -- it was super easy to follow and did not read of AI slop! I also really appreciate the research direction of feature universality. I think it is timely and an important direction to push on in both vision and language.
Critical weaknesses
The contribution is extremely limited in my opinion. The original USAE paper applies a cross-model SAE across three vision models, and [a] (a key missing citation) does it across language models. The application of a USAE to a diffusion model is indeed a novel application but requires limited novel architectural or training techniques. The main section describing the novel details is 2.3 which is less than half a page and does not even write out the equation of a new loss term they added. Overall, I think only the first of their three contributions are real contributions (USAE extended to diffusion model) and the later two are just validation of the method (which is also has several weakness in outcomes and rigor, see below)
Furthermore, the results themselves are not strong enough to raise my score either. The cross model reconstruction of 0.139 R^2 is surprisingly low, and the visualizations of the features, while somewhat interesting, do not improve our understanding or control of these models further (the figure quality is extremely low as well, poor resolution, cropped or screenshotted straight from a visualization tool, and should be improved). It would also have been great to visualize shared and unique features across models
Given that both technical novelty (simply applying USAEs or [a] to include diffusion models with limited architectural/training improvements) and scientific findings are limited, I believe this paper would be a much better fit at a workshop style venue as the authors develop the work further (scaling it up, more features, applications of the tool, more models, more findings, etc).
[a] Jiralerspong et al. Cross-Architecture Model Diffing with Crosscoders: Unsupervised Discovery of Differences Between LLMs, Mech Interp Workshop (NeurIPS 2025). https://arxiv.org/abs/2602.11729
Questions for authors
What is the exact equation for the 1-cosine loss you mention in section 2.3? It would be great to actually write out each novel architectural/loss component you introduce in the methods section.
What affordances does the X-SAE provide you that normal SAEs trained on each model does not? For example, could you steer the generative model in an interpretable way using rich DINO features through the joint dictionary?