acceptodds
Under review as a conference paper at ICLR 2027

SAE-STM: A Sparse Autoencoder-Augmented Structural Topic Model for Interpretable Pairwise Preference Prediction

Abstract

Reward models trained on pairwise human judgments drive RLHF and DPO, but they reveal little about what in a response makes it preferred. Neural topic models offer a compact, readable vocabulary but are too coarse to capture fine-grained preference factors; sparse autoencoders (SAEs) recover fine-grained features but are fit post hoc, with no link to the prediction task or to any structured summary. We introduce SAE-STM, which couples the two: a Top- SAE over the embedding difference of a response pair produces a sparse code that, following the Structural Topic Model, shifts the prior of a ProdLDA topic model, and a classifier predicts the human preference from the resulting topic proportions alone. Across six pairwise-preference datasets and four topic granularities, topic proportions and SAE features are complementary: a probe on both outperforms a probe on either alone. Finally, we check that the explanations are usable by someone outside the model: an LLM judge that sees only the two responses and the validated natural-language traits of their dominant topic recovers the human label with 53–74% accuracy, above the 50% chance level in all 24 dataset–granularity settings. SAE-STM thus provides preference predictions that come with a validated topic-and-feature explanation.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoders, structural topic models, neural topic models, interpretability, preference modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8SDvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SAE-STM, a model for interpretable pairwise preference prediction that combines a sparse autoencoder with a structural topic model. The SAE extracts sparse features from response embedding differences, which condition both the topic encoder and the prior of a ProdLDA model. The topic model reconstructs differences in word usage between responses, while a classifier predicts preferences using only the inferred topic proportions. To interpret these topics, the authors generate natural-language descriptions of associated SAE features and validate them against separate LLM annotations. Experiments cover six preference datasets and four topic granularities, evaluating predictive information through linear probes and the usefulness of topic traits through an external LLM judge. The paper reports complementary information in topic and SAE representations and trait-based preference prediction accuracies of approximately 53–74%, with substantial variation in feature interpretability across datasets

Critical strengths
  1. The model combines sparse features with a topic bottleneck, providing both topic keywords and associated feature descriptions for inspecting preference predictions. This offers a useful way to organize otherwise unstructured SAE features.

  2. The evaluation covers six preference datasets and four topic granularities, examining both predictive information and the usability of natural-language descriptions. The variation across domains provides useful evidence about where the approach works and where its explanations become less informative.

  3. Feature descriptions are checked against separate examples rather than supported only by selected illustrations. The paper also acknowledges cases where validated features capture surface style rather than the substantive preference factors of interest.

Critical weaknesses
  1. The explanation evaluation does not establish fidelity to the model’s predictions. Recovering human preferences with an external LLM differs from explaining the classifier, which uses all topic proportions rather than only the dominant topic. Comparisons with generic or shuffled traits are also missing.

  2. The reported complementarity gains are difficult to reconcile across evaluation settings. The main text emphasizes substantial gains, while Appendix B reports only a small average improvement on held-out test data, and Appendix H acknowledges probe overfitting. The main results’ evaluation split and configuration need clarification.

  3. The contribution of SAE conditioning is not cleanly isolated. The no-SAE baseline also removes preference supervision, so its weaker performance cannot establish the benefit of the proposed coupling. A supervised topic model without SAE conditioning would provide a more informative comparison.

Questions for authors
  1. How faithfully do the dominant-topic traits explain the classifier’s predictions? Can you evaluate agreement with the classifier and compare against generic or shuffled traits?

  2. Which splits and SAE configurations produce the gains in Tables 2–3? How do these results relate to the smaller held-out gains in Appendix B and the probe overfitting noted in Appendix H?

  3. Does SAE conditioning improve prediction or interpretability over the same supervised topic model without the SAE, while retaining the preference classification loss?

Reviewer_Zp38OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SAE-STM, a framework designed to provide structured explanations while predicting pairwise preferences between responses. SAE-STM first trains a sparse autoencoder (SAE) on the signed difference between the two response embeddings to extract fine-grained sparse features that characterize how the responses differ. These SAE features are then used as learned covariates to shift the topic prior of a structural topic model (STM). The resulting topic proportions serve as a bottleneck: the preference classifier predicts the pairwise label using only the topic proportions. Finally, each topic is described using both its most representative words and the natural-language descriptions of its associated SAE features.

Critical strengths
  1. Reasonable motivation. SAE-STM naturally combines the fine-grained features of SAEs with the higher-level semantic structure of topic models.

  2. Extensive evaluation. The paper evaluates six datasets and four topic granularities, and provides additional analyses of SAE dictionary size, representation probing, feature validity, and redundancy.

Critical weaknesses
  1. Complementarity is not sufficiently validated. The joint probe shows that topics and SAE features contain complementary decodable information, but does not demonstrate improved end-to-end preference prediction. More controlled ablations of the SAE conditioning, alignment loss, and topic bottleneck are needed.

  2. Insufficient comparison with existing methods. The paper lacks unified comparisons with state-of-the-art interpretable preference models in terms of prediction accuracy, explanation quality, and computational cost.

  3. The interpretability claim is somewhat overstated. SAE-STM is a proxy classifier built on frozen external embeddings rather than an explanation of an existing reward model. It also omits prompt information, although preference is often prompt-dependent. The contribution may be more accurately described as learning complementary topic and sparse representations with domain-dependent language descriptions.

  4. Pairwise consistency is not guaranteed. Swapping response order during data augmentation does not guarantee consistency between the predicted probabilities for the sample based on these two preferences (i.e., the sum of the predicted probabilities from the two preferences should be close to 1). The paper should report antisymmetry errors and compare against architectures that enforce this property by design.

Questions for authors

See Weaknesses.

Reviewer_38e1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends Movva et al (2025), which uses SAEs to interpret preference pairs in human feedback, by incorporating structured topic modeling in addition to SAEs.

Critical strengths

None so far because I have some skepticism about the evaluations.

Critical weaknesses

The paper largely follows that of "What's in My Human Feedback?" (WIMHF) by Movva et al. (2025) which uses SAEs to interpret human feedback data. What this paper does as an extension is to incorporate structured topic modeling in addition to the SAE features. Given its close derivation from Movva et al. (2025), one would expect them to compare directly against the WIMHF method, but they do not. They also do far fewer evaluations than the original paper does: e.g., WIMHF compares to ICAI baseline, shows that features recover held-out natural language explanations on Community Alignment, shows that WIMHF can be used for data curation to improve RewardBench2 safety scores without reducing general scores. Also, it seems a little bit sketchy that this paper uses 6 out of the 7 datasets in the original WIMHF paper, while excluding Community Alignment for some reason.

RQ1

Their RQ1 "DO TOPICS AND SAE FEATURES CARRY COMPLEMENTARY INFORMATION?" is a confounded evaluation. Why is it confounded? Interestingly, the authors say themselves that although they find that the SAE + topic features combined carries most signal, this seems to be because the SAE features are fixed at M=32 dimensions while the number of topics they range from 20 to 300, with gains higher when number of topics is higher. This is simply a confounded evaluation for whether topic modeling helps; the authors should be evaluating the SAE embeddings against topic vectors with the same dimension. In fact, when they compare an M=32 SAE embedding with the K=50 topic embedding, the benefit of topic embeddings is only +0.008 compared to the vanilla SAE embedding. Moreover, they do not provide statistical significance testing in the reported tables of results (Table 2/6).

Another thing that makes it difficult to interpret these results is that they are not comparing directly against WIMHF, they compare to the SAE embeddings which are learned jointly with their topic embeddings. However, it is difficult to predict how this affects the final learned SAE embeddings (which are affected both by the SAE reconstruction loss and the alignment loss). Moreover, the exact way they implement their SAE is different from WIMHF (see Appendix A.1 of WIMHF paper where they replace ReLU with identity activations to avoid learning redundant features, whereas this paper uses ReLUs on mirrored inputs [e_a - e_b; e_b- e_a] which would seemingly incentivize such redundancy). It would be best if the authors compared to the direct WIMHF results reproduced directly from the WIMHF code. WIMHF provides an AUC figure in Figure 4 of their paper, while this paper uses accuracy directly, so it is difficult to know the true comparison between methods. Also unlike WIMHF, their logistic regression probe does not include length as a control variate.

RQ2

Again main issue with the evaluations in this RQ2 is no clear comparison to WIMHF. Also why is Table 7 reported with a different (M, k) than the main results in RQ1? And again there is also a lack of error bars and CIs

Questions for authors

The paper extends the WIMHF method of Movva et al. (2025), but fails to properly evaluate against WIMHF. Because of this, I recommend reject for now. However, if the authors were able to improve their evaluations in the rebuttal, I would consider raising my score, although apriori I'm skeptical that their results would hold up under more rigorous evaluation. This would require:

  1. Proper evaluation against WIMHF in all their experiments. The WIMHF reproduction should be based directly on the WIMHF code and should reproduce the numbers in the WIMHF paper.
  2. In RQ1, the benefit of their method should be evaluated against SAE baselines where M (number of SAE features) = K (number of topics), to avoid confound of greater dimension size. They should also have an ablation where they remove label loss on their method (but keep all other losses) to understand how much the supervised loss is helping in their method. Length should be controlled for in the logistic regression probe, similar to in the WIMHF paper.
  3. Adding error bars / CIs to results
  4. Consistent hyperparameter use or methodology for selection of hyperparameters. Why does RQ2 use different (M. k) than RQ1?
  5. Add the Community Alignment dataset. Why does this paper use only 6/7 datasets that WIMHF does? It just seems a bit sketchy to drop just one dataset without any justification.
  6. Robustness checks: WIMHF checks that features reproduce across random seeds & different embedding models. The same should be done in this paper.
  7. The paper does not have a limitations section; one should be added. Since their method relies on labels (through the label loss), it cannot be used to understand "measurable preferences" prior to dataset collection like WIMHF can, it can only be used for "realized preferences". This should be mentioned explicitly in the paper as a limitation. Another one is that they have no human validation (unlike the WIMHF paper). They also provide no evidence that their topic features can be used for downstream use cases (unlike WIMHF which shows SAE features can be used for data curation / personalization). Of course, better than just adding these to the limitations would be to do a human evaluation and to add evaluations of downstream use cases.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.