acceptodds
Under review as a conference paper at ICLR 2027

SAEScientist-Bench: Evaluating LLM Agents on Concept-Driven SAE Feature Discovery

Abstract

Beyond improving model performance through training and data curation, enabling agents to study internal representations is a vital step toward closed-loop autonomous AI research and development (R&D). We introduce SAEScientist-Bench, a benchmark that evaluates whether large language model (LLM) agents can conduct interpretability research to discover concept features in sparse autoencoders (SAEs). Given a target concept, an agent investigates a 131K-feature dictionary for Gemma-2-9B-IT by designing contrastive probes, interpreting activation patterns across candidates, and submitting the feature that best captures the concept. To evaluate each submission, our benchmark provides held-out test suites containing positive texts, confusable negative texts, and prompts for text generation. Submissions are scored on three dimensions: prominence within the dictionary (Rank), selectivity in separating positive texts from negative texts (Activation), and the ability to steer generation toward the concept (Steering). We find that agents reliably isolate features that distinguish texts, reaching 92.9 Activation against a reference feature's 98.9, but these features struggle to steer generation, reaching at most 31.5 Steering against 57.8. This gap stems from candidate selection: agents usually uncover and test strong candidates during their investigation, yet end up choosing features that appear selective on probe texts but fail during generation. Moreover, language concepts prove the hardest category due to confusion across related languages. These findings suggest that conducting rigorous interpretability research remains a critical capability that agents have yet to master. Code is available at https://anonymous.4open.science/status/SAEScientist.

Primary area:
interpretability and explainable AI
Keywords:
automated AI R&D, LLM agents, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2tnTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SAEScientist-Bench evaluates whether agents can discover features for specified concepts in fixed SAE dictionaries. It contains 20 tasks across two layers of Gemma 2 9B IT, with hidden text and generation tests measuring dictionary prominence, activation selectivity, and steering. Across ten agent configurations, selected features often distinguish relevant texts well but provide substantially weaker generative control than curated references.

Critical strengths

I appreciate the separation of discovery quality into distinct dimensions. RANK measures prominence, ACTIVATION tests selectivity against confusable controls, and STEERING measures an actual generative effect. This is more informative than asking whether highly activating examples merely appear consistent with a concept, and it makes the distinction between input detection and output control explicit.

Critical weaknesses
  1. STEERING does not yet establish usable generative control. Equation (7) rewards increased target relevance without including task preservation or degeneration in the final score. The coefficient calibration does impose a nondegeneration constraint, but Table 16 reports instruction preservation of approximately 0.95 to 1.12 out of 4 and degeneration rates of roughly 21% to 32% for alternative selections across the 13 analysed tasks. Thus, calibration does not fully resolve quality failures at evaluation. A joint measure requiring both concept induction and preserved task completion would show whether the rankings reflect reliable control.

  2. The benchmark lacks a simple causal selection baseline that calibrates its difficulty. How well would activation based candidate retrieval followed by a small number of steering tests perform under a comparable query budget? Given prior evidence that input selectivity and output effects differ, this is an important reference point. Without it, I cannot tell how much of the agent gap reflects difficult experimental reasoning and how much reflects omission of an established validation step.

Questions for authors
  1. Although your paper describes a process for discovering features—resembling the pipeline researchers typically follow—I still fail to see how this serves as a valid metric for evaluating an agent. In other words, I am not convinced that this benchmark adds any real value to the assessment of an agent’s capabilities; it is difficult for me to interpret what a high or low score actually signifies. Could you provide a reliable, well-founded reference to support this approach?

  2. Could you provide the relevant original task instructions? Were agents explicitly told that the objective included generative steering, rather than only identifying a feature that detects the concept? Section 4.1 states that steering tests were available, but I could not determine how this capability was exposed or how often it was used. Please include the relevant tool instructions and the fraction of investigations that performed steering tests. These details materially affect how I interpret the reported neglect of causal validation.

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer 2tnT (1/n)

We appreciate your careful reading of our manuscript and your constructive feedback. Your comments helped guide our revisions, which we describe point by point below.

Response to Weakness #1

A joint measure requiring both concept induction and preserved task completion would show whether the rankings reflect reliable control.

We appreciate the reviewer’s suggestion to assess whether steering can induce the target concept while preserving task completion and response quality. Some agent-selected features produced degenerate responses on held-out prompts despite the non-degeneration constraint used during calibration. We added Joint Steering to account for concept induction, task preservation, and degeneration in the final score.

For each response, we take the harmonic mean of the target relevance and task preservation ratings and rescale it to 0–100. Degenerate responses receive a score of zero. Within each task, we average the response scores separately for feature steering, no intervention, and random-direction steering. Following Eq. (7), we compute Joint Steering = max(0, Q_feature − max(Q_unsteered, Q_random)), where each Q is the mean response score for the corresponding condition. The final score is the average across tasks.

Expanded evaluation. We apologize for the limited coverage in the original submission, as limited API resources constrained the range of tasks we could evaluate. During the rebuttal period, we added 100 tasks, bringing the evaluation to 120 tasks across two Gemma-2 models and five SAE configurations.

Source modelSAE layerFeatures per SAEOriginal tasksAdded tasksTotal
Gemma-2-2B-IT1016,38402020
Gemma-2-2B-IT2016,38403030
Gemma-2-9B-IT9131,07212012
Gemma-2-9B-IT20131,07282836
Gemma-2-9B-IT31131,07202222
Total20100120

We compared Steering and Joint Steering on the expanded evaluation.

Model / MethodSteeringAgent RankJoint SteeringAgent Rank
Expert50.13—34.27—
Gemini 3.8 Flash37.97125.441
Claude Opus 533.98223.033
Claude Sonnet 533.98221.268
Kimi K333.36424.382
GPT-5.532.98522.166
GPT-5.6 Sol31.95621.577
GLM-5.231.81722.285
Grok 4.631.56822.894
GPT-5.6 Luna29.55921.159

The Expert remains highest overall, and Gemini 3.8 Flash remains the top agent. Accounting for response quality changes several other positions. Claude Sonnet 5 falls from tied second to eighth, while Kimi K3 rises from fourth to second and Grok 4.6 rises from eighth to fourth.

We further analyzed how agents used steering during exploration. Claude Sonnet 5 made the fewest steering calls, averaging 2.94 per task, while Gemini 3.8 Flash made the most, averaging 15.03 per task (Figure 11). More extensive steering tests may help agents identify features that induce the target concept while preserving response quality. These examples illustrate how agents differ in the extent of their causal testing and in the quality of their final selections.

We appreciate your suggestion to evaluate concept induction and response quality jointly. We will include this comparison in the main ablation analysis.

AuthorsOpenReview ↗

Author Response to Reviewer 2tnT (2/n)

Response to Weakness #2

Given prior evidence that input selectivity and output effects differ, this is an important reference point. Without it, I cannot tell how much of the agent gap reflects difficult experimental reasoning and how much reflects omission of an established validation step.

We appreciate the reviewer’s suggestion to include a baseline that tests steering before selecting a feature. We added four non-agent baselines and compared them with the Expert reference and all nine agents on the expanded evaluation described in our response to Weakness #1. For each task, the activation-based baselines use a common sample of positive and negative texts drawn from the probes submitted by agents.

  • Activation Top-1 selects the feature with the largest activation difference between positive and negative probes.
  • Activation Top-3 takes the three features with the largest activation differences, tests each on the steering calibration prompts, and selects the feature with the highest calibration Steering score. We evaluate the selected feature on separate held-out prompts.
  • FeatureFinder applies its feature selection procedure to the same sampled probes.
  • Random Feature samples a feature uniformly from the task’s SAE.
Model / MethodRankActivationSteering
Expert53.5688.7950.13
Activation Top-167.2078.2030.92
Activation Top-362.2680.7733.31
FeatureFinder36.7772.7610.63
Random Feature5.7411.090.44
Claude Opus 561.6592.0633.98
Gemini 3.8 Flash57.7589.9837.97
GLM-5.260.2189.0931.81
Claude Sonnet 558.7186.7533.98
GPT-5.6 Sol58.2190.1431.95
Grok 4.658.9189.6231.56
Kimi K358.3888.3233.36
GPT-5.558.8987.9532.98
GPT-5.6 Luna56.9585.5829.55

Comparison within Non-Agent Baselines. Activation Top-3 improved held-out Steering from 30.92 to 33.31 and Activation from 78.20 to 80.77 compared with Activation Top-1, while its Rank fell from 67.20 to 62.26. This trade-off shows how testing the three candidates during calibration can favor a feature that steers better on held-out prompts. FeatureFinder used the same probe sample and scored below both activation baselines on all three metrics, with a Steering score of 10.63.

Comparison of Non-Agent Baselines and Agents. All nine agents exceeded Activation Top-3 on Activation, and four also exceeded it on held-out Steering. These four agents found features through exploration that separated positive and negative examples more clearly and steered generation more effectively than the heuristic. The other five scored below Activation Top-3 on Steering, showing that selecting features that perform well on both metrics remains challenging. We will include all four baselines in the main results table to make these differences across agents visible.

AuthorsOpenReview ↗

Author Response to Reviewer 2tnT (3/n)

Response to Question #1

Q1. What capability does the benchmark measure, and how should its scores be interpreted?

We appreciate the reviewer’s question about how the benchmark measures an agent’s ability to conduct interpretability research.

What capability the benchmark measures. Each task asks the agent to find an SAE feature for a given concept. The agent decides how to design probes, which candidates to compare, and which feature to submit. We evaluate the submitted feature on held-out data. This follows the outcome-based approach used in RE-Bench, which scores the solutions agents produce for ML research engineering tasks [1], and MLE-bench, which scores agents’ Kaggle submissions on held-out test sets [2]. Here, the outcome measures how well the agent’s investigation identifies a feature for the target concept.

Why the scores are valid measures of a concept feature. AxBench evaluates concept representations by their ability to detect a concept in inputs and to steer generation toward it [3]. In our benchmark, Rank measures how prominently the submitted feature activates on concept-relevant inputs relative to other features in the SAE, and Activation measures how well it separates positive and negative examples. Steering tests whether intervening on the feature increases expression of the concept in generated responses. Arad et al. show that features with strong input selectivity can have weak effects on generation and that selecting features by their output effects improves steering [4]. Together, these metrics assess what the feature detects and what it can change.

What a high or low score means. We interpret the scores alongside the Random Feature and Expert results on the same tasks. RE-Bench similarly uses starting and expert reference solutions to contextualize agent performance [1]. In our expanded evaluation, agent Activation scores ranged from 85.58 to 92.06, compared with 88.79 for the Expert. Agent Steering scores ranged from 29.55 to 37.97, compared with 0.44 for Random Feature and 50.13 for the Expert. As discussed in Weakness #2, four of the nine agents outperformed Activation Top-3 on both Activation and Steering. This shows that stronger agents can use exploration to find more selective features that also steer generation more effectively than a heuristic that already tests steering.

Hence, the benchmark provides a verifiable way to compare agents on a task researchers already perform. Agents that can reliably find and verify these features could reduce the manual work involved in SAE analysis.

[1] Wijk H, Lin T R, Becker J, et al. RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts. Proceedings of ICML, 2025.

[2] Chan J S, Chowdhury N, Jaffe O, et al. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. Proceedings of ICLR, 2025.

[3] Wu Z, Arora A, Geiger A, et al. AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders. Proceedings of ICML, 2025.

[4] Arad D, Mueller A, Belinkov Y. SAEs Are Good for Steering – If You Select the Right Features. Proceedings of EMNLP, 2025.

Response to Question #2

Q2. Could you provide the original task and tool instructions, clarify whether agents were explicitly asked to test generative steering, and report how often they did so?

We appreciate the reviewer’s question about how steering was presented to agents. We provide the task instructions for the expanded evaluation in Figure 12 (Appendix C.5). Agents were asked to identify the SAE feature that best matched the concept description. We provided two tools. probe_sae measures feature activations on input texts, and steer_sae tests a candidate’s effect on generation. The tool takes a feature ID, an intervention strength, and test prompts, and returns the generated text and stop reason. To keep the comparison fair and assess how agents design their investigations, we withheld the evaluation prompts and scoring criteria, including the use of Steering as a final metric.

All reported investigations in the expanded evaluation successfully used steer_sae at least once. Under the same instructions and tool access, Gemini 3.8 Flash averaged 15.03 steering calls per investigation, compared with 5.34 for Grok 4.6 High and 2.94 for Claude Sonnet 5 (Figure 11). The differences in call frequency show how much emphasis each agent placed on testing generative effects before selecting a feature. Gemini 3.8 Flash tested steering most extensively and achieved the highest held-out Steering score among agents (37.97).

Agents need to interpret the target concept, decide which tests will distinguish promising candidates, and use the results from both tools to support their final choice. The tools give agents a practical way to carry out that analysis, and the common task setup lets us compare how effectively they do it.

Reviewer_uXBDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SAEScientist-Bench, a benchmark that tests whether LLM agents can find sparse autoencoder (SAE) features related to a given concept. Agents search through a large set of SAE features by creating test examples, examining activations, and choosing the most relevant feature.

The benchmark includes 20 tasks across four categories and uses three metrics: Rank, Activation, and Steering. Experiments with 10 agent configurations show that agents can often find features that respond strongly to a target concept, but struggle to find features that can effectively change model outputs. The authors also study how agents search for features and find that choosing the final feature remains a major challenge.

Critical strengths
  1. The paper studies whether LLM agents can explore and understand internal model features through their own experiments. This is an important direction that existing agent benchmarks have not fully explored.

  2. The benchmark uses three metrics to evaluate feature discovery from different perspectives, including feature ranking, activation, and steering. This provides a more complete evaluation than relying only on activation scores.

  3. The paper goes beyond comparing agent performance and examines why agents fail. An interesting finding is that agents often discover good candidate features but fail to select the best one. This helps identify an important limitation of current agents.

Critical weaknesses
  1. The benchmark has only 20 tasks using one model and two SAE layers. It is unclear whether the findings generalize to other models, SAE dictionaries, or more complex concepts.

  2. The steering metric relies mainly on an LLM judge, but quantitative evidence of judge reliability is missing.

  3. The paper identifies feature selection as a key bottleneck but lacks simple baselines based on activation scores or steering tests.

Questions for authors
  1. Do the benchmark findings generalize to other language models, SAE dictionaries, or concepts involving multiple features?

  2. Can the authors report agreement between the LLM judge and human evaluations? How do the choice of judge and steering strength affect the results?

  3. How do simple baselines, such as selecting features based on activation scores or steering effects, compare with the evaluated agents?

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer uXBD (1/n)

Thank you for taking the time to review our manuscript. Your feedback guided our revisions, and we explain how we addressed each comment below.

Response to Weakness #1

The benchmark has only 20 tasks using one model and two SAE layers. It is unclear whether the findings generalize to other models, SAE dictionaries, or more complex concepts.

We appreciate the reviewer’s concern about the scope of the original evaluation. We apologize for the limited coverage in the original submission, as limited API resources constrained the range of tasks we could evaluate. During the rebuttal period, we expanded the evaluation to 120 tasks across two Gemma-2 models and five SAE configurations. We added Gemma-2-2B-IT at layers 10 and 20 and extended Gemma-2-9B-IT coverage to layer 31.

Source modelSAE layerFeatures per SAEOriginal tasksAdded tasksTotal
Gemma-2-2B-IT1016,38402020
Gemma-2-2B-IT2016,38403030
Gemma-2-9B-IT9131,07212012
Gemma-2-9B-IT20131,07282836
Gemma-2-9B-IT31131,07202222
Total20100120

The added tasks also cover more complex concepts. One task targets language about primary or first occurrences in medical contexts. Another targets specific cooking actions and food preparation techniques, such as dicing and blanching. These tasks require agents to design more challenging positive and negative probes that better assess whether agents can design informative probes and discover features for complex concepts.

The expanded evaluation shows how consistently agents perform across SAE settings. Figure 9 breaks down Rank and Activation by configuration. Claude Opus 5 leads the agents on Rank and Activation overall, while Gemini 3.8 Flash leads on Steering. The table below reports the expanded evaluation results.

Model / MethodRankActivationSteering
Expert53.5688.7950.13
Activation Top-167.2078.2030.92
Activation Top-362.2680.7733.31
FeatureFinder36.7772.7610.63
Random Feature5.7411.090.44
Claude Opus 561.6592.0633.98
Gemini 3.8 Flash57.7589.9837.97
GLM-5.260.2189.0931.81
Claude Sonnet 558.7186.7533.98
GPT-5.6 Sol58.2190.1431.95
Grok 4.658.9189.6231.56
Kimi K358.3888.3233.36
GPT-5.558.8987.9532.98
GPT-5.6 Luna56.9585.5829.55

Response to Weakness #2

The steering metric relies mainly on an LLM judge, but quantitative evidence of judge reliability is missing.

We appreciate the reviewer’s request for quantitative evidence on judge reliability. Our blinded human verification shows strong agreement between GPT-4o and human ratings, with a Pearson correlation of 0.936 and a mean squared error of 0.522.

We randomly sampled 100 steering prompts, each paired with an unsteered response, a response steered with the selected SAE feature, and a response steered in a random direction. The annotator saw the original prompt, the target concept, and its scoring criteria, then used the same 0–4 rubric to rate how well each response expressed the concept. We shuffled the responses and hid their generation conditions. These results indicate high scoring fidelity for LLM judge.

AuthorsOpenReview ↗

Author Response to Reviewer uXBD (2/n)

Response to Weakness #3

The paper identifies feature selection as a key bottleneck but lacks simple baselines based on activation scores or steering tests.

We appreciate the reviewer’s suggestion to include simple baselines for feature selection. We added four non-agent baselines. The activation-based baselines use positive and negative examples sampled from the texts agents used in their probe calls for each task.

  • Activation Top-1 selects the feature with the largest difference in mean activation between positive and negative probe texts.
  • Activation Top-3 tests the three highest-ranked activation candidates on the steering calibration prompts and selects the one with the highest calibration Steering score.
  • FeatureFinder averages token activations within each text, compares positive and negative texts using a Wilcoxon rank-sum statistic, and selects the highest-ranked positive marker.
  • Random Feature samples one feature uniformly from the task’s SAE dictionary.

The table in our response to Weakness #1 reports all four baselines alongside the agent results. Activation Top-1 leads on Rank, all nine agents outperform every baseline on Activation, and four agents surpass Activation Top-3 on Steering. These results show that stronger agents can use exploration to find features that better separate positives from controls and more effectively steer generation. For several agents, the heuristic of testing just three candidates still yields better Steering.

Response to Question #1

Do the benchmark findings generalize to other language models, SAE dictionaries, or concepts involving multiple features?

As described in our response to Weakness #1, we expanded the evaluation to 120 tasks across two Gemma-2 models and five SAE configurations. Figure 9 shows how agent performance varies across these settings. The added tasks also cover more complex concepts that require more challenging positive and negative probes.

Our focus on single-feature discovery follows a central goal of SAE research: extracting relatively monosemantic features. In future work, the same framework could be extended to evaluate how agents discover and test combinations of features that jointly represent a concept.

Response to Question #2

Can the authors report agreement between the LLM judge and human evaluations? How do the choice of judge and steering strength affect the results?

We appreciate the reviewer’s question. We compared GPT-4o with a stronger judge, GPT-5.5, on the same responses. GPT-5.5 agreed more closely with human ratings, with a Pearson correlation of 0.976 and an MSE of 0.136, compared with 0.936 and 0.522 for GPT-4o. GPT-5.5 gave higher relevance ratings to feature-steered responses, yielding higher Steering scores. The agent rankings were identical across the two judges.

Steering strength affects both concept induction and response quality. A weak intervention may fail to elicit the target concept, while an overly strong one can disrupt task completion or cause degeneration. We calibrated the strength for each submitted feature on separate prompts using the same non-degeneration criterion. This protocol puts features on a common footing for held-out evaluation while allowing each feature to be tested at a strength suited to it.

Response to Question #3

How do simple baselines, such as selecting features based on activation scores or steering effects, compare with the evaluated agents?

We describe the four non-agent baselines in our response to Weakness #3 and report their scores alongside the agents in the table under Weakness #1. The strongest agents outperform all four heuristic baselines on Activation and Steering. Activation Top-1 achieves the highest Rank, and several agents score below Activation Top-3 on Steering.

Dear Reviewer uXBD,

We appreciate the time you took to review our paper. Your feedback guided the additional experiments and analyses in our rebuttal. We hope these additions answer your questions, and we’d be happy to discuss anything that remains unclear.

Best regards.

Reviewer_nTwcOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SAEScientist-Bench, which is a benchmark for evaluating whether LLM agents can autonomously perform SAE interpretability. They show their results on 2 layers of Gemma-2-9B, with a 131k dictionary. The authors introduce three evaluation metrics, which are rank, activation and steering. 10 frontier agents are tested on 20 tasks in a sandboxed environment, and the results show high activation but low steering. The authors argue that the failure mode is candidate selection, rather than retrieval.

Critical strengths

Clear benchmark definition with behavioural evaluation. Figure 2 shows clearly that the focus of the benchmark is the interactive discovery by the agents, and that the results are evaluated with metrics separately. The separation of the metrics (rank, activation and steering) allows for testing distinct desiderata and supports evaluation of examples where they differ. The idea of automated feature interpretability for SAEs is relevant, given the time the process usually consumes

Critical weaknesses

The steering score depends heavily on LLM as a judge, with limited validation. This is mentioned in the appendix section B6. I do not understand what “human verification” is referring to, which significantly impacts the conclusions that can be drawn from the steering results. I also miss a reference to the appendix in 4.2, for readers who have questions about the steering metric. I found the additional information with ctrl-F, which should not be the case.

Missing is a non-agent baseline. This would show the improvement and allow for the interpretation of the agent results. A baseline could be just to select a feature with the best AUC on the agent's probe, or a random feature or random top-k.

It seems that a single reference feature is treated as the expert ground truth per task. The story of the retrieval-selection failure relies on finding a particular feature, which then relies on using this reference feature. Not selecting the reference is not necessarily equal to a bad selection. Additionally, it is unclear how many features are tested in total from the main text. Is there just one feature per task? Which would indicate that the results are based on 20 features only.

The scope is quite narrow for a benchmark, with claiming autonomous interpretability research. Only one model, only two layers, only jumprelu SAEs in combination with single feature localisation make the stated benchmark claim too strong for what is shown.

Questions for authors
  1. Are there baseline comparisons with no-agent settings? How do simpler heuristics perform?
  2. What is the human verification process of gpt-4o mini? And what could be the influence of picking this model as a judge for steering? 3. Would the scores change for another model?
  3. How would the results change on choosing a different SAE or a different source model?
  4. In Table 4 there is an overview where the reference feature is selected/lost. Are there any more logs on why features might not be selected? Which criterion dominates? And are there relations if activation differences on probes mispredict steering?
  5. Is one reference feature tested per task? So that would mean that the scores depend on 20 selected features? Could you add a table in the appendix or a sentence in the main text with this information?
7 comments
AuthorsOpenReview ↗

Author Response to Reviewer nTwc (1/n)

We sincerely appreciate the time and care you took in reviewing our manuscript. Your feedback helped us improve the paper, and we address your comments point by point below.

Response to Weakness #1

The steering score depends heavily on LLM as a judge, with limited validation.

We appreciate the reviewer’s concern about the reliability of the LLM judge for Steering. Our human verification shows strong agreement between GPT-4o and human ratings, with a Pearson correlation of 0.936 and a mean squared error of 0.522.

Human verification procedure. We provide the detailed human evaluation procedure below. We randomly sampled 100 steering prompts, each paired with an unsteered response, a response steered with the selected SAE feature, and a response steered in a random direction. For each prompt, the annotator saw the original user prompt, the target concept, and the criteria for strong and insufficient evidence. The annotator used the same 0–4 rubric as GPT-4o to rate how well each response expressed the target concept. The responses were shown in shuffled order with their generation conditions hidden.

Relative Steering score. As defined in Eq. (7), we compute Steering as the gain in mean relevance to the target concept over the higher of the unsteered and random-direction steering scores. This relative comparison helps preserve scoring fidelity.

We have added the full procedure and results to the updated Appendix C.3 and will highlight them in the main text of the final manuscript.

Response to Weakness #2

Missing is a non-agent baseline.

We appreciate the reviewer’s suggestion to include heuristic baselines for feature selection. We added four non-agent baselines and evaluated them on the original 20 tasks from Table 1. The activation-based baselines use positive and negative examples sampled from the texts agents used in their probe calls for each task.

  • Activation Top-1 selects the feature with the largest difference in mean activation between positive and negative probe texts.
  • Activation Top-3 tests the three highest-ranked candidates from Activation Top-1 on the steering calibration prompts and selects the candidate with the highest calibration Steering score.
  • FeatureFinder averages token activations within each text, uses a Wilcoxon rank-sum statistic to compare positive and negative texts, and selects the highest-ranked positive marker.
  • Random Feature samples one feature uniformly from the task’s SAE dictionary.
Model / MethodRankActivationSteering
Expert73.8298.9257.75
Activation Top-161.2986.1026.13
Activation Top-363.2986.7329.50
FeatureFinder25.5761.304.31
Random Feature2.986.820.00
Kimi K370.4892.9130.26
Claude Sonnet 570.6492.0129.66
Grok 4.668.4190.1131.47
Claude Opus 568.9391.8928.99
Claude Opus 4.868.0790.9026.55
GPT-5.6 Sol64.5590.0330.16
Gemini 3.8 Flash66.6086.7822.43
GLM-5.261.9084.8524.28
GPT-5.564.2284.1923.19
GPT-5.6 Luna63.4283.3620.54

Agents can discover better features through exploration. Four of the ten agents outperformed Activation Top-3 on all three metrics. Activation Top-3 is a competitive heuristic baseline because it already tests the steering effects of three candidates before selection. The strongest agents improved on this fixed procedure, while several others fell below it on Activation or Steering. This variation explains what the benchmark measures: agents must turn a concept description into informative probes, interpret the experimental evidence, and select a feature whose behavior holds up under independent evaluation.

AuthorsOpenReview ↗

Author Response to Reviewer nTwc (2/n)

Response to Weakness #3

It seems that a single reference feature is treated as the expert ground truth per task. Not selecting the reference is not necessarily equal to a bad selection. Additionally, it is unclear how many features are tested in total from the main text.

We appreciate the reviewer’s concern about the role of the reference feature and the size of the original evaluation.

The Expert feature provides a human-curated point of comparison alongside the agent results and non-agent baselines. As described in Section 4.1, we curated one verified reference feature per task from Neuronpedia’s public steering presets for Gemma Scope. The retrieval and selection analysis uses this feature to trace how agents handle a known candidate during their investigations. Notably, selecting a different feature does not constitute a failure. Instead, we assess the submitted feature by its measured performance on Rank, Activation, and Steering. In our experiments, agents sometimes discovered features that outperformed the Expert reference.

Scope of the original evaluation. The reviewer is correct that the original submission evaluated 20 tasks. We apologize that limited API access during the initial experiments restricted the coverage of the evaluation. During the rebuttal period, we expanded the evaluation as described below.

Expanded evaluation. We expanded the evaluation to 120 tasks across five SAE configurations. Claude Opus 5 leads the agents on Rank and Activation, and Gemini 3.8 Flash leads on Steering. Kimi K3 ranked first on Activation in the original 20-task evaluation. In the expanded evaluation, Claude Opus 5 scores higher than Kimi K3 on four of the five SAE configurations, including the newly added Gemma-2-9B-IT layer 31, where it scores 88.1 versus 83.1 (Figure 9). The per-configuration results show which agents consistently select features that separate positive examples from controls across SAE settings.

The tables below report the expanded task distribution and results. We will revisit our analysis using the broader evaluation.

Source modelSAE layerFeatures per SAEOriginal tasksAdded tasksTotal
Gemma-2-2B-IT1016,38402020
Gemma-2-2B-IT2016,38403030
Gemma-2-9B-IT9131,07212012
Gemma-2-9B-IT20131,07282836
Gemma-2-9B-IT31131,07202222
Total20100120
Model / MethodRankActivationSteering
Expert53.5688.7950.13
Activation Top-167.2078.2030.92
Activation Top-362.2680.7733.31
FeatureFinder36.7772.7610.63
Random Feature5.7411.090.44
Claude Opus 561.6592.0633.98
Gemini 3.8 Flash57.7589.9837.97
GLM-5.260.2189.0931.81
Claude Sonnet 558.7186.7533.98
GPT-5.6 Sol58.2190.1431.95
Grok 4.658.9189.6231.56
Kimi K358.3888.3233.36
GPT-5.558.8987.9532.98
GPT-5.6 Luna56.9585.5829.55

Response to Weakness #4

The scope is quite narrow for a benchmark, with claiming autonomous interpretability research.

We appreciate the reviewer’s concern about the scope of our claims. SAEScientist-Bench evaluates concept-driven SAE feature discovery as one step toward fully autonomous interpretability research. Finding a feature for a specified concept is a meaningful research task. AxBench demonstrates the value of locating representations for concept detection and steering [1], and Arad et al. show that choosing the right SAE feature can substantially improve steering [2]. These results motivate evaluating whether agents can make that choice through their own investigations.

We turn this research task into a verifiable benchmark. Given a concept description and a fixed SAE, agents design probes, compare candidates, and submit a feature. We evaluate every submission on the same held-out suite. Rank measures prominence on concept-positive texts, Activation measures separation from controls, and Steering measures the feature’s effect on generation. Together, the three metrics let us compare agents under controlled conditions and identify where their feature choices succeed.

As described in our response to Weakness #3, we expanded the evaluation to 120 tasks across five SAE configurations, covering two Gemma-2 models, multiple layers, and two dictionary sizes. The expanded results let us examine how consistently agents perform this specific research task across SAE settings.

[1] Wu Z, Arora A, Geiger A, et al. AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders[C]//Proceedings of ICML. 2025: 67035–67080.

[2] Arad D, Mueller A, Belinkov Y. SAEs Are Good for Steering – If You Select the Right Features[C]//Proceedings of EMNLP. 2025: 10241–10259.

AuthorsOpenReview ↗

Author Response to Reviewer nTwc (3/n)

Response to Question #1

Are there baseline comparisons with no-agent settings? How do simpler heuristics perform?

As reported in our responses to Weaknesses #2 and #3, we compared agents with four non-agent baselines on the original and expanded evaluations. On the original 20 tasks, four of the ten agents outperformed Activation Top-3 on all three metrics. In the expanded evaluation, Activation Top-1 achieved the highest Rank, while the best agents exceeded all four baselines on Activation and Steering. These results show that agents with stronger concept-driven feature discovery skills can use their own exploration to find features that outperform simple heuristic baselines.

Response to Question #2

What is the human verification process of gpt-4o mini? And what could be the influence of picking this model as a judge for steering? 3. Would the scores change for another model?

We used GPT-4o instead of GPT-4o mini as the judge for Steering. We describe the human verification procedure in our response to Weakness #1. To assess the effect of judge choice, we compared GPT-4o and GPT-5.5 on the same responses. GPT-5.5 agreed more closely with human ratings, with a Pearson correlation of 0.976 and an MSE of 0.136, compared with 0.936 and 0.522 for GPT-4o.

GPT-5.5 tended to assign higher relevance ratings to feature-steered responses, yielding higher Steering scores. The two judges produced identical Steering rankings across all ten agents on the original evaluation, supporting the main conclusions under GPT-4o scoring. We will include this comparison in the judge ablation analysis in the final manuscript.

Response to Question #3

How would the results change on choosing a different SAE or a different source model?

As described in our response to Weakness #3, we expanded the evaluation to 120 tasks across two Gemma-2 models and five SAE configurations. Figure 9 shows that agent performance varies across SAE configurations. In the expanded evaluation, Claude Opus 5 leads the agents on Rank and Activation, and Gemini 3.8 Flash leads on Steering.

Response to Question #4

Are there any more logs on why features might not be selected? Which criterion dominates? And are there relations if activation differences on probes mispredict steering?

We understand your reference to be to Table 2. The main criterion visible in the investigation logs was activation selectivity on agent-written probes. Agents favored candidates that activated strongly across positive examples and weakly on negative controls. When another candidate showed a clearer contrast on those probes, agents could favor it over the reference feature. In the supplementary runs, most agents also called the probe tool more often than the steering tool (Figure 11).

A favorable activation contrast did not always predict effective steering. On the Portuguese task, several agents selected features that responded to Portuguese text in their probes, yet steering with those features left the model answering in English. The traces suggest that agents sometimes committed to a candidate after limited testing, before checking its activation on a wider range of texts or examining its steering effects across different prompts.

Response to Question #5

Is one reference feature tested per task? So that would mean that the scores depend on 20 selected features? Could you add a table in the appendix or a sentence in the main text with this information?

Each of the original 20 tasks has one Expert reference feature, and Table 4 lists all 20 reference IDs. Each agent run also submits its own feature, which we evaluate on the task’s held-out suite. The scores in Table 1 therefore reflect the measured performance of each agent’s submissions. In the supplementary evaluation, we added 100 tasks, each with one Expert reference feature, bringing the total to 120 tasks. We will describe this evaluation setup explicitly in the main text and point readers to the reference IDs in the appendix.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. The revisions below are highlighted in blue in the updated manuscript.

Expanded Evaluation and Baselines

  • Expanded evaluation. We expanded the benchmark from 20 to 120 tasks across two Gemma-2 models and five SAE configurations and report how agent performance varies across configurations (Tables 8–9 and Figures 8–9 — nTwc W3/W4/Q3, uXBD W1/Q1).
  • Non-agent baselines. We added four baselines, including Activation Top-3, which tests steering on three candidates before selecting a feature. Four of nine agents outperformed it on both Activation and Steering in the expanded evaluation (Table 9 — nTwc W2/Q1, uXBD W3/Q3, 2tnT W2).

Steering Evaluation

  • Joint Steering. We added a metric that combines concept relevance and task preservation and assigns zero to degenerate responses. Gemini 3.8 Flash led the agents under both Steering and Joint Steering, while several other agents changed positions (Figure 10 — 2tnT W1).
  • Human verification and judge comparison. Both judges’ relevance ratings correlated strongly with blinded human ratings. GPT-4o and GPT-5.5 produced identical Steering rankings across all ten agents in the original evaluation (Appendix C.3 and Table 10 — nTwc W1/Q2, uXBD W2/Q2).

Task Instructions and Expert Reference

  • Task instructions and tool use. We provided the instructions for the expanded evaluation and reported how often agents used the activation and steering tools (Figures 11–12 — 2tnT Q2).
  • Role of the Expert feature. The human-curated Expert feature serves as a point of comparison. Agents are scored on the held-out performance of the feature they choose. On some tasks, agents found features that outperformed the Expert reference (nTwc W3/Q5).

We would be glad to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.