acceptodds
Under review as a conference paper at ICLR 2027

Representational Simplicity and Circuit Size Dissociate in a Threshold-Dependent Way: A Controlled Test via Adversarial Training

Abstract

Sparse-autoencoder decomposability and concentrated feature attribution are increasingly treated as evidence that a model's computation is easier to reverse-engineer. Whether this representational and attributional cleanliness actually predicts a smaller or more tractable causal circuit remains an open question. We test this directly using adversarial training as a controlled instrument: it reliably reshapes internal representations, but this alone does not constitute a test of circuit size. We investigate this question through reverse-engineering complexity: the causal structure required to recover a model's behavior at a fixed level of faithfulness. To our knowledge, this is the first controlled empirical test of whether representational or attributional simplicity translates into causal simplicity at the circuit level. Starting from the same pretrained GPT-2 Small checkpoint, we apply matched standard and adversarial continual training, requiring both conditions to retain competence on indirect object identification and pass independent robustness verification before comparing mechanisms. We then compare the models along three complementary axes: sparse-autoencoder decomposability, SAE feature engagement in task attribution, and the size of faithful circuits recovered from the raw computational graph. The robust model is more SAE-decomposable and engages fewer SAE features in task attribution. Circuit size is regime-dependent: on competence-matched IOI, standard leads or ties below 85% faithfulness, but robust needs substantially fewer edges at high faithfulness (90%, 95%), a pattern established on the primary pair while representational trends generalize across a seven-point sweep and a second corpus.

Primary area:
interpretability and explainable AI
Keywords:
Adversarial robustness, Superposition, Mechanistic interpretability, Adversarial training, Circuit discovery, Interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6crsOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether a cleaner internal representation implies a smaller causal circuit. From one GPT-2 Small checkpoint it continues training with a standard objective and with ten-step PGD on token embeddings, keeps both models competent at indirect object identification, and checks that the robust model is robust under the same attack used in training. It then compares SAE reconstruction error, how many SAE features the task attribution uses, and how many edges EAP-IG must keep to recover IOI behavior at a given faithfulness. The robust model is more SAE-decomposable and engages fewer features. Circuit size depends on the cutoff: the standard model leads or ties through 85% faithfulness, while on the primary pair the robust model reaches 90% with about 1,000 edges against 4,000, and 95% with about 2,000 against 16,000. The area under the same curve favors the standard model, 0.87 to 0.60.

Critical strengths

The comparison holds task competence fixed before it compares mechanisms, and it says in advance that representational cleanliness and circuit size are different quantities. Figure 2 gives both readings of one curve: the models meet at 85%, the standard model leads through the steep part of the curve, and the robust model pulls ahead only at high faithfulness. The paper states that no cutoff is a field standard, that the 90% and 95% counts were computed for the primary pair only, and that Greater-Than is not competence-matched because the robust model scores 0.442 there against 0.671. The representational trend, unlike the high-faithfulness gap, is repeated across a seven-point sweep and a second corpus. That is a usable result.

Critical weaknesses

The result that would support a stronger score is the edge gap at 90% and 95%. It comes from one pair, 200 prompts, and a coarse grid on which a reported count means the crossing fell in the preceding bin. Those crossings were not recomputed for the sweep or the second corpus. SAE reconstruction uses three seeds; the circuit comparison does not. Adversarial training changes more than feature entanglement, so the dissociation is shown for this training instrument on IOI, not for representational simplicity in general. The paper already states these limits. They are why the acceptance is weak.

Questions for authors

On the models already trained for the seven-point sweep, do the 90% and 95% crossings still fall in different grid bins, in the same direction, on the same 200-prompt protocol? A stable gap would move this toward a clear acceptance. A reversal would move it back to a weak rejection.

Reviewer_r9gHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether representational simplicity predicts causal simplicity. Starting from one GPT-2 Small checkpoint, the authors continue training a standard model and six adversarially trained models (embedding-space PGD) on the same 1B tokens. They keep one standard/robust pair that both retain IOI accuracy and pass a robustness check. They compare the pair on three axes: SAE reconstruction error at the block-8 residual stream, SAE feature engagement in task attribution, and the size of faithful circuits found by EAP-IG on the raw graph. The robust model is more SAE-decomposable (2.65% lower MSE) and has fewer active task features (1,196 vs 1,612). Circuit size depends on the faithfulness threshold: standard leads or ties up to 85%, while robust needs fewer edges at 90% (1,000 vs 4,000) and 95% (2,000 vs 16,000). The representational trends are shown over a seven-model sweep and a FineWeb pair, but the circuit result only for the primary pair.

Critical strengths
  1. The question is well posed and tested with a careful design: shared checkpoint, same data stream, an IOI competence floor, a masking check, and an SAE-free α-ReQ control. Separating “easier to decompose” from “smaller circuit” is useful for work that treats SAE metrics as proxies for tractability.

  2. The reporting is honest. The result is framed as threshold-dependent, GT is flagged as not competence-matched, and the failed EAP-GP pilot and the non-seed-robust ablation-KL are disclosed. This makes the evidence easy to assess.

Critical weaknesses
  1. The headline causal claim rests on very little. The 90/95% result is one model pair, one task, one training run (seed 42) and 200 prompts, with no uncertainty on a coarse grid. It sits on a flat part of the curve: robust has F = 0.91 at 1,000 edges, while standard has 0.87 at 1,000 and 0.89 at 2,000. A gap of a few faithfulness points becomes a 4× gap in “circuit size”, and the robust curve is itself non-monotone (0.97 at 2,000, 0.92 at 4,000). The 90/95% crossings were not computed for the six other models or the FineWeb pair, although the sweep shows the 85% crossing flat while representational metrics move. EAP-GP, the only second estimator, failed. The primary robust model is the sweep’s extreme and has the largest GT drop (0.671 to 0.442), so the effect may reflect general capability change rather than robustness.

  2. The motivation and scope are limited. I could not see how the results connect to interpretability for AI safety, even after the introduction, and the paper itself declines to draw conclusions about auditability. Everything is on GPT-2 Small and one competence-matched task (IOI). Scalability and generality are untested, although adversarially trained LLMs or efficient ways to build them exist (e.g., ReFAT).

Questions for authors
  1. Can you report the 90/95% crossings for the other six models and the FineWeb pair, with a finer grid and prompt-bootstrap intervals, plus Y(clean) and Y(corrupt) per model?

  2. What is the variation between two standard models trained with different seeds or data order? Please also report the original GPT-2 checkpoint, for SAE MSE, pool size and circuit crossings.

  3. The abstract says “independent robustness verification”, but Appendix B says training and qualification use the same attack implementation. Can you evaluate a different attack and report clean loss for each model?

  4. The SAE axes use block-8 features and the circuits use head-level edges. Can you test the link at one level, e.g., with sparse feature circuits?

Reviewer_GMhbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether adversarial training makes neural-network computations easier to reverse engineer, and more specifically whether simpler internal representations imply smaller causal circuits. Starting from the same GPT-2 Small checkpoint, the authors compare standard and adversarial continual training while attempting to control initialization, data, training budget, and task competence.

The analysis separates three notions that are often conflated: (1) SAE decomposability, (2) the number of SAE features engaged by a task, and (3) the number of computational-graph edges required to recover behavior at a specified level of faithfulness. The main experiments focus on indirect object identification (IOI), with Greater-Than as a secondary, non-competence-matched task.

The principal result is nuanced. Adversarial training consistently produces representations that are more easily reconstructed by matched SAEs and engages fewer SAE features. However, this does not translate monotonically into smaller causal circuits. For IOI, the standard and robust models require the same number of edges at 85% faithfulness, while at 90% and 95% the robust model requires substantially fewer edges. Across a seven-model robustness sweep and an additional FineWeb pair, the representational trend generalizes, whereas the high-faithfulness causal result is demonstrated only for the principal model pair.

Critical strengths
  1. The paper makes an important conceptual distinction between representational and causal simplicity. The strongest contribution is not merely finding that adversarial training changes representations, but explicitly testing whether this translates into simpler causal computation. Prior work has associated adversarial training with changes in learned representations and interpretability. Related vision work has also shown that training under strong image transformations can yield more robust latent representations and improve robustness to adversarial perturbations (Malik et al., 2023). The present paper goes beyond this representational question by asking whether such changes correspond to smaller faithful causal circuits. I find this distinction both important and consequential.
  2. The experimental controls are stronger than in many mechanistic-interpretability comparisons. The shared checkpoint, matched continual-training setup, explicit robustness qualification, multiple SAE seeds, SAE-free α-ReQ analysis, second-corpus experiment, and conditional co-ablation provide useful checks on several obvious alternative explanations. Most importantly, the paper does not overinterpret the Greater-Than result when the task competence differs between models, and it explicitly acknowledges that the circuit result has not been established across the full sweep.
Critical weaknesses
  1. The central high-faithfulness circuit result is supported by only one competence-matched model pair. The headline result is that the robust model needs 1,000 versus 4,000 edges at 90% faithfulness and 2,000 versus 16,000 at 95%. Yet these thresholds are evaluated only for the selected standard/robust IOI pair. Across the seven-model sweep and the FineWeb pair, the only circuit statistic reported is the 85% crossing, where the robust models do not demonstrate smaller circuits. This creates an asymmetry in the evidence: the representational claim is replicated broadly, while the most interesting causal claim is effectively a single-pair observation. This matters because the paper's main thesis concerns whether representational simplicity predicts causal simplicity. Without 90/95% circuit recovery across more of the robustness sweep, it is difficult to determine whether the striking high-faithfulness effect is systematic, specific to the selected robust checkpoint, or sensitive to the circuit-recovery procedure.
  2. The circuit-size conclusion is highly dependent on the chosen faithfulness regime and relatively coarse edge grid. At 70% faithfulness the standard model requires fewer edges, at 80-85% the models tie, and only at 90-95% does the robust model become substantially smaller. Moreover, the standard model has the higher overall partial-recovery AUC. Thus, there is no threshold-independent sense in which the robust computation is simply "smaller." I appreciate that the paper itself recognizes this, but I think the framing should remain very conservative: the evidence supports a high-faithfulness circuit-size advantage for one matched pair, rather than a general reduction in reverse-engineering complexity under adversarial training.
Questions for authors
  1. Clarification that would materially affect my assessment: Can the authors compute the 90% and 95% IOI circuit crossings for the other models in the adversarial-strength sweep, or at least for multiple additional qualified robust checkpoints? This seems like the most important missing experiment. The representational quantities vary systematically across the sweep, but the reported 85% circuit crossing largely does not. Establishing whether the high-faithfulness circuit advantage also varies with adversarial strength would considerably strengthen the central causal claim.
  2. Clarification that would materially affect my assessment: How stable are the 90% and 95% crossings to the prompt subsample used for circuit recovery? Circuit recovery uses 200 IOI prompts whereas other analyses use 1,000. Given the large reported difference at high faithfulness, replication across independently sampled prompt subsets would help establish that the crossing behavior is not particular to the selected subset.
  3. Suggestion for positioning rather than a requirement for acceptance: The relationship to previous robustness-and-representation work could be discussed somewhat more broadly. For example, Malik et al. (2023) show that training with extreme image transformations can produce more robust latent object representations and improve robustness to adversarial attacks. This does not address mechanistic circuit size and therefore does not compete with the paper's main contribution, but it reinforces the motivation for explicitly separating representational robustness/simplicity from causal simplicity.

References

Malik, G., Crowder, D., & Mingolla, E. (2023). Extreme image transformations facilitate robust latent object representations. arXiv preprint arXiv:2310.07725. https://doi.org/10.48550/arXiv.2310.07725

Malik, G., Crowder, D., & Mingolla, E. (2023). Extreme image transformations affect humans and machines differently. Biological Cybernetics, 117(4–5), 331–343. https://doi.org/10.1007/s00422-023-00968-7

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.