- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MAxBench evaluates multinomial concept representations through sampled activation interventions. It compares directional, linear, affine, SAE-based and spline representations using concept, fluency, instruction-following and newly introduced diversity scores. Experiments cover six concepts, four instruction-tuned models and multiple training subsets. The main empirical findings are that centering strongly affects performance, affine variants generally outperform origin-centered variants under this protocol, and prompting remains a strong baseline for larger models.
Critical strengths
-
The paper addresses a useful evaluation gap: recovering a concept with multiple values requires assessing coverage as well as successful concept injection. The proposed sampling-based framework makes this distinction explicit.
-
The experiments separate several relevant factors, including rank, offset and basis, and include prompting, isotropic sampling, held-out concept instances/categories and supplementary layer sweeps. These controls provide informative evidence about when simple representations can compete with more complex methods.
-
The evaluation spans four models and six structured concepts, with ten concept-training splits. The paper provides metric breakdowns, judge prompts and human agreement results for concept, fluency and instruction scores, supporting inspection of the reported findings.
Critical weaknesses
-
The diversity normalization does not guarantee the claimed range. Appendix B.2 divides instance entropy by log(N), where N = 150 is the number of outputs, despite allowing multiple entities per output. For example, 300 distinct equally frequent entities give log(300)/log(150) = 1.138, exceeding the stated instance-score bound. The appendix acknowledges this possibility but supplies no fixed correction or evidence that every evaluated setting satisfies the assumed sparsity. Since diversity enters the benchmark ranking, its normalization needs a well-defined, consistently bounded rule.
-
Diversity extraction does not consistently measure valid concept coverage. Appendix E.2 explicitly maps non-animal uses such as programming “Python” to an animal and “train your model” to a vehicle, while the concept judge distinguishes intended meanings from such uses. Generic references also count as instances. Thus lexical ambiguity can increase diversity without recovering additional valid concept values. Table 10 validates only concept/fluency/instruction judgments, not entity extraction or taxonomy assignment; an audit of the novel diversity component is missing.
-
Sampling can depend on parameterization rather than the recovered subspace. Section 3.2 uses s = Lz + b with isotropic z, so sample covariance is sigma^2 L L^T. Factor-analysis loadings are not generally orthonormal, and Appendix A.2 explicitly leaves SAE directions non-orthogonal. Two bases spanning the same subspace can therefore receive different scores. Moreover, Appendix B.1 restricts sigma to [r_min, r_max], although the RMS norm for an origin-centered orthonormal rank-k sampler is sigma sqrt(k). Orthonormalization and reported norm-acceptance diagnostics are needed to disentangle geometric quality from sampling artifacts.
-
The directional-versus-affine comparison changes the intervention semantics. Rank-1 directions are sampled with a zero-centered Gaussian, including both signs, and all methods use h′ = (1 - alpha)h + alpha s. This can push a signed concept-presence direction toward its negative side and suppress unrelated activation components. Footnote 3 reports diversity collapse under additive steering but provides no corresponding quantitative comparison. Positive-direction sampling, a mean-only baseline and context-preserving interventions would test whether the headline offset advantage persists beyond this specific operator.
-
Prompt-level robustness is weakly established. Section 4 uses only five validation instructions and five test instructions, with 30 samples per instruction. The ten training splits measure sensitivity to concept-data selection, not uncertainty over new instructions. Consequently, repeated samples do not establish broad prompt generalization for the method rankings or prompting comparison. Appendix B.1 also omits the actual alpha grid and the selected decay parameters needed to reproduce this sensitive part of the protocol.
Questions for authors
Clarifications that could materially affect the assessment:
-
What bounded normalization and semantic validity rules are actually implemented for diversity? Does a human-audited entity/taxonomy evaluation and corrected scoring preserve the main rankings?
-
Do the offset and geometry conclusions persist with orthonormalized bases, rank-aware norm calibration, positive-only directional sampling and a mean-only control? Please report sampling norm-acceptance rates.
-
How stable are the rankings across a larger held-out instruction set with prompt-level uncertainty? Please also specify the alpha grid and the initial/final injection strengths and decay duration.
4 comments
Do the offset and geometry conclusions persist with orthonormalized bases, rank-aware norm calibration, positive-only directional sampling and a mean-only control? Please report sampling norm-acceptance rates.
For all methods except for SAEs, the bases are orthonormal. We also tune the standard deviation selection for each rank separately (see L.1296 - L.1299). For directional methods, we also now show affine variant results (in App.H, Tab. 12). For mean-only control, we report isotropic results, where there is no notion of what the bases are (i.e., the only set parameter is the mean/centroid). We unfortunately did not understand what is meant by “norm-acceptance rates”, but we are happy to discuss further. We believe that sampling outside of orthonormal space is only an issue for SAEs, especially given our tuning of the hyperparameters of the sampling to optimize for good scores. We are happy to discuss further!
How stable are the rankings across a larger held-out instruction set
We observe very small variations in performance across ranks for most methods (seen in Fig. 2). Based on this observation, we expect our findings to generalize to additional held-out prompts as well. However, we acknowledge that evaluating on a larger set of held-out instructions would further strengthen our findings, and we have noted this as a direction for future work. We hope this convinces you that with our experiments we do establish a rather broad prompt generalization for the methods we analyse.
Alpha grid and the initial/final injection strengths
We now added these details in App. B.1 L.1135-L.1137.
Thank you for the revisions. Several aspects of my original concerns remain insufficiently addressed:
- The human–GPT agreement analysis does not establish whether correcting semantically invalid mentions and taxonomy assignments would preserve the main rankings.
- Per-rank tuning does not resolve the concern about the sigma search range, and norm-acceptance rates remain unreported. Positive-only directional sampling is also missing, while isotropic sampling is not equivalent to the requested mean-only control.
- Evaluation still uses only five test instructions. Small variation across training splits does not establish generalization across new prompts. My overall rating currently remains 2 (Weak rejection).
Semantically invalid mentions
We thank the reviewer for this suggestion. To directly address this, we conduct an additional experiment where we recompute diversity scores using human-annotated concept entities instead of GPT-extracted entities. Due to time constraints we select four methods from different geometry classes in our main results and annotate a subset of generated sentences for each method and concept. We report the resulting MAxBench scores in the new Tab. 16. We observe that affine version of DiffMean remains the highest-performing method and that the rankings across geometry classes remain highly similar. We also compare affine and linear variants using human annotations in Fig. 19; we observe that affine methods still outperform linear methods. These results provide additional evidence that our main findings are robust to these implementation choices.
sigma search range and norm-acceptance rates
We are unfortunately still unsure what these terms refer to; with clarification, we would be happy to engage with these points further.
positive only directional sampling and mean only control
We report these results now in Fig. 20 and Tab. 18. We find that positive-only sampling improves DiffMean-r1 performance but still performs worse than affine DiffMean (Pairs) (Tab. 18), while the mean-only control performs worse than the affine methods (Fig. 20).
Evaluation still uses only five test instructions.
We now conduct an additional experiment with 20 new test prompts. Due to time and compute constraints, we select four methods from different geometry classes: DiffMean-r1, DiffMean (Pairs), MFA, and SAE (Top-K). We see that ranking across methods remains mostly unchanged (Tab. 17) and affine methods still outperform linear methods (Fig. 20).
We thank the reviewer for the thorough feedback; we’re glad they found our main questions and findings useful and interesting.
This would be a good point to clarify. We clip all normalized diversity scores at 1.0, ensuring that the final scores remain within the range of [0, 1]. We now also measure the distribution of diversity scores; we report their distributions in Tab. 15 and Fig. 18 of Appendix K. The results demonstrate that only a small number of ranks naturally yield scores marginally above 1.0, so few values are clipped.
We agree that entity extraction can often be limited; there is indeed quite a bit of fuzziness in deciding what counts as a valid concept instance. This is particularly true for broad concepts, where lexical ambiguity can make entity extraction less straightforward. We now discuss this limitation in App. J in the revised PDF. We have also added an experiment to App. J, which reports the Jaccard similarity of extracted entities from a human (on 100 samples) vs. the GPT LLM judge. The scores are reported in the table below and in the appendix.
We observe good agreement between GPT and human annotations for COUNTRIES, DAYS, and YEARS, and reasonable (far better than random) agreement for ANIMALS, VEHICLES, and PLANTS. We also compare human and GPT entity extraction to regular expressions, which do not distinguish semantically relevant vs. irrelevant instances. We have added this analysis and a discussion to the revised manuscript in App. J.
Thanks for pointing out the subtle influence of the L-(ortho)normalization on sampling in identical subspaces. We would like to clarify that we orthonormalize the basis vectors we obtain after factor analysis method using QR decomposition. We have added this L.1132 - L. 1135 in the PDF. For SAEs, we indeed keep the unit norm vectors as they are. Thus, since we used a single sampling method for z uniformly over all affine methods, we do not make isolated claims about the effectiveness of the sampling subroutine. Indeed, if we were to make such specific claims we would need to control for the orthonormalization. To keep the scope manageable, we leave such investigations for future work.
We now report additive steering results for additive methods including DiffMean-r1 and REFT-r1 in Table 13. We observe that both additive and convex combination steering strategies achieve similar diversity scores but vary in concept scores with replacement methods getting better overall MAxbench scores than additive methods. We have updated the paper to reflect this finding (L. 254 - L.256) and footnote update in page 5) and added all the results to Table 13 and App. I.
We use a similar number of prompts during evaluation as in AxBench, and generally observe very small variations in performance across ranks for most methods (seen in Fig.2, which reports the variance across 5 test prompts as well as data splits), as well as sufficiently small variances to make claims about the importance of offsets and geometry classes (where confidence intervals often do not overlap). Based on these observations, we expect our findings to generalize to additional held-out prompts as well, as adding more prompts from a similar distribution would be likely to further reduce variance. We acknowledge that evaluating on a larger set of held-out instructions would further increase trustworthiness of the results, however, and we have noted this as a current limitation and direction for future work.
We tune α0 and α1 in [0, 1], with the constraint that α0 > α1. To find the best alphas, we grid search over for both values (following the above constraint) in [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]. We inject it throughout the length of generation and our max_tokens are set to 50. We have added these details in App. B.1 L.1135-L.1137.
See responses to weaknesses 1 and 2 for details, as well as App. K, Fig. 18 and Tab. 15. As requested, we now report Jaccard similarities of human annotations vs. GPT, as well as GPT vs. regular expressions, in App. J. Agreements are generally well above chance.