acceptodds
Under review as a conference paper at ICLR 2027

Cross-Architecture Universal SAEs: A Shared Sparse Dictionary for Vision and Diffusion Transformers

Abstract

Universal Sparse Autoencoders (USAEs) learn a single sparse dictionary whose indices denote the same concept across several models, but the original USAE was evaluated only on vision transformers. We extend the USAE to a vision transformer (DINOv2) and a diffusion transformer (PixArt-). We train a TopK dictionary with a separate encoder and decoder for each model, map both models' tokens to a common patch grid, and add a per-token alignment loss that encourages the two models to activate the same indices at corresponding patches. On held-out COCO images, a code computed from either model explains variance in the other model's activations: from DINOv2 to PixArt- and in the reverse direction. At the average patch the two models share 43 of their 128 active indices, 38.7 times the rate expected under independent selection. Of the 12,288 dictionary indices, 74.7% are used by both models. Because the training objective rewards per-token agreement, the co-firing rate measures what the objective achieves, not whether the two architectures converge without it. We also show that agreement measures that pool codes over an image's patches are invariant to any permutation of one model's patches, so they cannot detect whether the models agree at corresponding locations. On our checkpoint, the pooled bag-of-features cosine is 0.9955, while the per-token Jaccard index is 0.203. We therefore report cross-model agreement per token and relative to a chance baseline.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoders, universal sparse autoencoders, mechanistic interpretability, computer vision, dictionary learning, vision models, diffusion models, vision transformers, diffusion transformers, DINOv2, PixArt, feature extraction, semantic features, superposition, monosemanticity, cross-model representation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WUL2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper extends the Univeral SAE, which trained an SAE across three different vision models sharing a transformer backbone, and proposes the X-SAE, a cross-model SAE for different vision architectures trained across architectures. They train the X-SAE across DINOv2 and the PixArt diffusion model, and report cross-architecture and per token evaluations.

Critical strengths

The paper is very well written -- it was super easy to follow and did not read of AI slop! I also really appreciate the research direction of feature universality. I think it is timely and an important direction to push on in both vision and language.

Critical weaknesses

The contribution is extremely limited in my opinion. The original USAE paper applies a cross-model SAE across three vision models, and [a] (a key missing citation) does it across language models. The application of a USAE to a diffusion model is indeed a novel application but requires limited novel architectural or training techniques. The main section describing the novel details is 2.3 which is less than half a page and does not even write out the equation of a new loss term they added. Overall, I think only the first of their three contributions are real contributions (USAE extended to diffusion model) and the later two are just validation of the method (which is also has several weakness in outcomes and rigor, see below)

Furthermore, the results themselves are not strong enough to raise my score either. The cross model reconstruction of 0.139 R^2 is surprisingly low, and the visualizations of the features, while somewhat interesting, do not improve our understanding or control of these models further (the figure quality is extremely low as well, poor resolution, cropped or screenshotted straight from a visualization tool, and should be improved). It would also have been great to visualize shared and unique features across models

Given that both technical novelty (simply applying USAEs or [a] to include diffusion models with limited architectural/training improvements) and scientific findings are limited, I believe this paper would be a much better fit at a workshop style venue as the authors develop the work further (scaling it up, more features, applications of the tool, more models, more findings, etc).

[a] Jiralerspong et al. Cross-Architecture Model Diffing with Crosscoders: Unsupervised Discovery of Differences Between LLMs, Mech Interp Workshop (NeurIPS 2025). https://arxiv.org/abs/2602.11729

Questions for authors

What is the exact equation for the 1-cosine loss you mention in section 2.3? It would be great to actually write out each novel architectural/loss component you introduce in the methods section.

What affordances does the X-SAE provide you that normal SAEs trained on each model does not? For example, could you steer the generative model in an interpretable way using rich DINO features through the joint dictionary?

Reviewer_5WetOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Cross-Architecture Universal Sparse Autoencoders (X-SAE) as an extension to Universal Sparse Autoencoders (USAE) by introducing a shared sparse dictionary between a Vision Transformer model (DINOv2) and a Diffusion Transformer model (PixArt-α\alpha). The method learns the shared dictionary with separate encoder and decoder for each model, and by applying a per-token alignment loss, thus pushing the two models to activate the same indices in the shared dictionary at the same token by construction. This setup results in 74.7%74.7\% of the dictionary being shared by the two models, and on the token level the two models share 43 of the 128 active indices on average. Furthermore, the cross-model reconstruction reaches R2=0.32R^2 = 0.32 and R2=0.14R^2 = 0.14 from DINOv2 to PixArt-α\alpha and vice versa respectively. The authors also demonstrate that pooled agreement measures are permutation invariant, which might give a weaker interpretability capability compared to per-token measures.

Critical strengths
  • The paper is clear and well written, with honest limitations section and an easy to follow approach.
  • The Cross-Architecture Univesal SAEs are benefitial in interpretability and explainability, and maybe could be utililzed in other directions such as finetuning, and concept deletions.
Critical weaknesses
  • The paper misses multiple baselines and ablations:
    • measuring whether the two architectures co-firing rate converge without the per-token alignment objective (using the reconstruction objective only)
    • the baseline of random chance is a low bar when it comes to per-token co-firing, which makes the 38.738.7x improvement look inflated.
  • the experiments in general are not extensive, with only 2 models, and without rigorous theoretical analysis or a mention of multi-seed runs and the error ranges.
Questions for authors
  • How does the co-firing rate per-patch with reconstruction objective only compare to the co-firing rate with the per-token alignment on the TopK codes active?
  • How are the self and cross reconstruction performances affected by deactivating the per-token alignment objective?
  • The pooled measures are permutation invariant in comparison to the per-token measures, which means that the co-firing of indices could result from different patches in the image, how does this in practice affect explainability and interpretability methods? An experiment where one model's tokens are shuffled would be helpful in showing the invariance to permutations of pooled measures compared to per-token measures decreasing.
Reviewer_NRpeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a cross sparse autoencoder (X-SAE) that learns shared features across a vision transformer and a diffusion transformer. The goal is to decompose visual representations into human-interpretable features shared across models, providing a foundation for jointly analyzing and interpreting vision models.

Critical strengths
  • The paper investigates shared features between a ViT and a DiT trained for substantially different objectives, proposing X-SAE as a new approach to identifying and interpreting these features.
Critical weaknesses
  • The paper appears to remain at the stage of an initial idea, with insufficient experimental design and validation. Although its main contribution is an X-SAE applicable across architectures, the approach lacks a systematic investigation of what the ViT and DiT represent at different layers and token positions, how the DiT representations vary across timesteps, and how these differences should inform the construction of X-SAE. The layer and timestep choices rely substantially on settings from prior work, with limited analysis of how these choices affect reconstruction performance and the learned features. Regarding SAE training, the method adds a per-token alignment loss to the existing USAE, but the ablations do not isolate the necessity or contribution of this component. More comprehensive ablation studies and analyses are needed.

  • The reported R2R^2 values for both self-reconstruction and cross-reconstruction are not particularly high. These results provide limited evidence that the method successfully identifies shared features that can faithfully reconstruct the corresponding representations. Furthermore, there are no direct comparisons with independently trained SAEs or the original USAE under matched conditions, making it difficult to assess the strength of these results and the benefits of the proposed method.

Questions for authors
  • Have you compared X-SAE with SAEs trained independently on the ViT and DiT, as well as with the original USAE, using the same backbone models and matched training conditions?

  • What roles do the features activated exclusively in DINOv2 or exclusively in PixArt-α play in their respective models?

  • Have you considered models other than DINOv2 and PixArt-α? How readily can X-SAE be extended to other model pairs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.