acceptodds
Under review as a conference paper at ICLR 2027

Automated Circuit Discovery for Equivariant Molecular Networks

42 views

Abstract

Graph neural network force fields predict molecular energies and forces with near-quantum accuracy and increasingly run as foundation models that users never retrain. Whether such a model can be trusted beyond its test set depends on what it computes: which geometric features it responds to, where in the network the response is carried, and whether the same computation recurs from molecule to molecule. Mechanistic interpretability answers questions of this kind for language models by finding circuits, small sets of internal components that together implement one behaviour, and validating them with interventions. Its tools assume discrete inputs, hidden states that can be copied between examples and evidence from single inputs, whereas molecular geometry is continuous, equivariant hidden states are tied to a coordinate frame, and a mechanism is established only when it recurs across molecules. We present an automatic circuit-discovery pipeline for GNN force fields built around these conditions. Typed chemical triangles define families of small, valid perturbations; an exact Möbius decomposition over all switch combinations of six internal units assigns every part of the response to a set of units; components whose effect recurs and survives dose and content controls are localised by a greedy search over units typed by their role relative to the triangle, so a circuit found on one set of molecules is defined on every molecule of its family; and each circuit is confirmed on molecules the search never saw by sufficiency, necessity and a random-subgraph contrast under one multiplicity correction. Run as a single command on MACE, Orb and NequIP, the pipeline returns 44 confirmed circuits whose interventions read as chemistry: angle circuits separate from centres and bond-stretch circuits behave as harmonic springs.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
mechanistic interpretability, equivariant neural networks, machine learning interatomic potentials, molecular simulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2N8uOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies automated circuit discovery for GNN force field models. They define a behaviour, a set of chemically meaningful pertubations on small local substructures (stretching a bond, or opening an angle) that change the model output by a certain margin. They then find parts of the computational graph of the model, the units, that explain a certain fraction of this model output change in an intervention on most of the molecules. They then try to find a concrete small circuit across different molecules that is both sufficient and necessary for this change. They find such small circuits across three different force field models and can relate selected examples to concrete pyhsical properties/laws.

Critical strengths

S1: They find small circuits in trained force field models and are able to interprete some of them in a case study. The relation to concrete physical laws/properties seems interesting. However, I am lacking expertise to really judge how meaningful these explanations are and how much insight they provide.

S2: The motivation to apply circuit discovery (which is mostly used for LLMs) to force field models make sense. And the authors convincingly argue that this application is not straightforward as molecules contain continuous position features, force field models use equivariant transformations, and it is not obvious how to define meaningful perturbations.

Critical weaknesses

W1: Lack of clarity: The paper is very hard to read and understand. While I acknoledge that part of this might be caused by my lack of expeertise in circuit discovery, I still belief that there is a significant lack of clarity:

  • The paper would benefit from a background section on circuit discovery that then forms the basis of the subsequent discussion and makes clear what parts of the pipeline are just taken over from already existing circuit discovery pipelines and what parts are new/adapted to the force field setting.
  • The definitions are not well-organized and not clear enough.
    • Many definitions contain terms that weren't defined previously or aren't sufficeintly explained: "An interface is a declared finite set of nodes of the computation graph of fθ at one boundary" (ll. 167). What does boundary mean here? "An intervention on a subset A ⊆ VI at strength α ∈ [0, 1] evaluates the model on G with aca_c replaced by ac(G)+α∆aca_c (G) + α ∆a_c for c ∈ A and recomputes every downstream node; when the written value comes from a molecule in a different orientation it is transported through the unit’s group action." What does "different orientation" mean here? What does "written value" refer to here?
  • The same symbol is defined in different ways. In Definition 2 v(A) denotes the model output after intervening on A (it explicitly says v(∅)=yBv(∅) = y_B), whereas in Definition 3 seems to denote the change in model output (it explicitly says now v(∅)=0v(∅) = 0).

This lack of clarity makes it very difficult to understand the precise method of the paper (which also seems very complicated and involves many different stages).

Questions for authors
  • Can you more clearly describe the different stages of the algorithm?
  • For how many of the discovered circuits can you find concrete interpretable connections to phyical properties/laws? What concrete insight or practical consequences follow from these connections?
6 comments
AuthorsOpenReview ↗

Response to reviewer 2N8u, part 1 of 2

We thank the reviewer for their critical review and for listing exactly which terms were unclear. We have fixed each one in the revised PDF (changes are highlighted in blue).

W1a. Background on circuit discovery.

"The paper would benefit from a background section on circuit discovery ... what parts of the pipeline are just taken over from already existing circuit discovery pipelines and what parts are new/adapted to the force field setting."

The introduction now clarifies that circuit discovery was developed on language models (as the reviewer notes), and Section 3 opens with a table detailing what we borrowed from existing literature and what are our contributions. Briefly, standard circuit discovery (e.g., ACDC; Conmy et al., 2023) isolates a behavior on discrete inputs, copies activations between clean and corrupted runs, and prunes the computation graph to the smallest part that reproduces the behavior.

  • While we build upon existing frameworks such as activation patching, ACDC-style pruning, faithfulness and completeness scores, and Möbius (Harsanyi) coefficients, none of these methods were directly applicable off-the-shelf and required specific adaptations.
  • Several modifications were made to adapt the methodology to our setting. Because the input space is continuous, target behaviors were defined as small, chemically typed geometric perturbations. Units are identified by chemical role, meaning a single circuit can be defined across different molecular graphs. Furthermore, we integrated a recurrence test across molecules using false-discovery control, incorporated controls that respect activation symmetries, and validated all findings on held-out molecules.

W1b. "Boundary".

"'An interface is a declared finite set of nodes of the computation graph ... at one boundary' (ll. 167). What does boundary mean here?"

A boundary is a cut through the network at one depth, a set of activations that every path from the input to the output crosses. If we set all of them to their perturbed values, we get the perturbed output. Definition 2 now clarifies this.

W1c. "Written value" and "different orientation".

"What does 'different orientation' mean here? What does 'written value' refer to here?"

The written value is the activation we put into a unit during an intervention, ac(G)+α Δaca_c(G)+\alpha\,\Delta a_c: its clean unperturbed value plus a fraction α\alpha of the change that the perturbation caused. Equivariant activations are vectors and tensors expressed in a coordinate frame. When the change comes from another site or another molecule, as in our transfer tests, we first have to rotate it into the receiving frame, and that rotation is what "transported through the group action" means. We have rewritten the sentence in order to clarify this to readers.

W1d. v(A)v(A).

"The same symbol is defined in different ways. In Definition 2 v(A) denotes the model output after intervening on A ... whereas in Definition 3 seems to denote the change in model output."

We thank the reviewer for the observation regarding the mismatch and clarify here: we inadvertently used the symbol in two different ways. We have corrected the text so v(A)v(A) consistently denotes the change in output under the intervention. This means v(∅)=0v(\emptyset)=0 across every definition, and admissibility now reads v(∅)=0v(\emptyset)=0 and v(VI)=ΔyBv(V_I)=\Delta y_B. Because the appendix and our codebase already used this exact meaning, no results change.

Q1. Describing the stages.

"Can you more clearly describe the different stages of the algorithm?"

We show here a summary of the table which is now detailed in Section 3, to describe the different stages of the algorithm:

stagewhat it doeswhat decides it
1 Behaviourssplits each molecule into typed bonds and angles; one small step per family (+0.02+0.02 Å or +2∘+2^\circ)validity and range checks
2 Decompose and screenswitches six internal units between clean and perturbed values in all 64 combinations, which gives every unit set's share of the energy change; keeps the sets that matter in most moleculesexact binomial test with false-discovery control
3 Causal checkwrites the component's change at four doses, and wrong content of the same sizedose response; must beat every control
4 Localiseexpands the component into role-typed units and finds the smallest set that reproduces its effectfaithfulness within [0.8,1.2][0.8,1.2], then pruning
5 Confirmtests the circuit on molecules no earlier stage sawsufficiency, necessity, better than random subgraphs

Appendix B.1 follows one perturbation, a C=C stretch in MACE, through all five stages, as an example.

AuthorsOpenReview ↗

Response to reviewer 2N8u, part 2 of 2

Q2. How many circuits connect to physical properties, and what follows?

"For how many of the discovered circuits can you find concrete interpretable connections ... What concrete insight or practical consequences follow from these connections?"

Beyond the case studies, we evaluated all 58 circuits using strict chemical criteria that we defined before we ran the sweep, informed by the seven case studies (Section 4.3, Appendix G.3).

  • 55 of 58 act as harmonic springs: When we displace the coordinate in either direction within a molecule, the energy carried by the circuit curves upward, mirroring the harmonic term of a classical force field (Hooke's law).
  • 36 of 58 exhibit a restoring trend across molecules: Over molecules where the bond or angle starts at different lengths, the circuit's force consistently points back toward a single, shared equilibrium.
  • 15 of 58 pass all four criteria: restoring across molecules, zero-force value within 5° or 0.05 Å of its own component's, textbook order between hybridisation classes, and zero-force value within 5° or 0.05 Å of the textbook value. For calibration, the full model passes the three criteria that apply to it (the first, third and fourth) for only 21 of the 58 circuits, and 5 circuits pass where the full model fails.
  • They recover chemistry from scratch: For example, MACE's H–C–C circuit places sp$^3$ and sp$^2$ carbon at 109° and 120° (textbook 109.5° and 120°), and Orb's two C=C circuits place the bond at 1.30 and 1.32 Å and its two C=O circuits at 1.19 and 1.22 Å (textbook 1.33 to 1.34 and 1.21 to 1.22 Å). No model was given hybridisation or bond order; they receive elements and coordinates only.

We draw four consequences from this:

  1. First, the models rebuild classical chemistry internally. Networks trained only on energies and forces organically organize their computation by standard chemical categories. A circuit is the exact same small set of units across every molecule in a family, and its component carries roughly a fifth of the total response (medians of 22%, 18%, and 24% in MACE, Orb, and NequIP).
  2. Second, the explanations concern correct behaviour. New DFT calculations at 693 perturbed geometries agree with the models' energy changes at r≥0.9988r\ge0.9988 (Appendix A.1). So, the circuits explain behaviours that the models get right.
  3. Third, a circuit shows where in the network a property is set. In MACE's C=C bond, one component carries 73% of the energy response and 37% of the stiffness, while the rest of the network shifts the bond length to its final value. If a developer spots a bond-length error, the circuit tells them exactly which part of the network to examine (something accuracy metrics cannot do!)
  4. Fourth, we can compare architectures by what they compute. Most MACE circuits sit in the receiver atom's own features and readout, while Orb and NequIP circuits reach back along the messages from the moving atom. The three reach similar accuracy by different internal routes.

We hope the rewritten definitions, the stage table and the worked example make the method clear, and that the sweep over all circuits answers your second question. We are glad to clarify anything further during the discussion period.

Thank you for your detailed answers and the new results.

The interpretations of the circuits look interesting and it is very good that you can find clear interpretations for most of them. (However, due to my lack of expertise it is hard for me to accurately judge their significance.)

Additionally, I have gone through the revisions in the definition in the updated paper. The specific examples that I orginally listed have largely been fixed. However I still have strong concerns about the clarity of the paper. To illustrate this, I have gone (only) over Definitions 1 & 2 and listed the points that remains unclear (to me). While the indended meaning can sometimes be inferred from the context, this makes reading just very hard. And in other cases one has to guess what the authors mean.

  • A perturbation specification is said to be parametrized by "target coordinate, a displaced atom, a step size η\eta, and fixed geometric constraints" (l.156). This is quite vague. What does target coordinate mean here? The new coordinate of the displaced atom? But what is then the influence of the step size? What are the fixed geometric constraints? Are we only considering bond stretching and angle bends or are ther other "local geometric transformations"?

  • "Finally, a pair B=(F,T)B = (\mathcal{F}, T) constitutes a behavior of fθf_\theta on D\mathcal{D} if valid, range-constrained occurrences of F\mathcal{F} appear in at least a minimum number of molecules DF⊆D\mathcal{D}_{\mathcal{F}} \subseteq \mathcal{D}". Valid-and range-constrained is never defined for occurences of F\mathcal{F}, just for applications of TT. Also what happens if there are multiple occurences of F\mathcal{F} in GG. Then ΔyB(G)=yB(TG)−yB(G)\Delta y_B(G) = y_B(TG) - y_B(G) is not uniquely defined.

  • A unit is defined as a node in the computational graph (l.175) but is later to be said to be e.g., a set of edge messages or an aggregated group (l.177). How does the computational graph look like? Wouldn't an aggregated group correspond to multiple nodes? Similary, we then define aca_c as the activation of unit cc. How is the activation of an aggregated group then defined here? Is it a vector of the individual activations?

  • One condition of vv to be called an admissible interface is that v(∅)=0v(\emptyset) = 0, isn't this always true for any interface? Similarly, isn't v(VI)=ΔyBv(V_I) = \Delta y_B also always true as an interface is defined to cut through every directed path? So does there exist any non-admissible interface?

  • The definition of a recorruing term is not clear to me:

    • What does "molecuels in family F\mathcal{F}" (l.192) mean? F\mathcal{F} is defined to be a typed (sub)graph). Is molecules containing F\mathcal{F} meant here?

    • Also it is not clear to me how SS is defined here? Is it allowed to depend on the input graph? E.g. can it be something like "all messages to the central crabon atom in the occurence of F\mathcal{F} in GG. So does the set SS consist of roles instead of concrete nodes in a computational graph (as l.175 would suggest).

While each point on its own might not be that significant or might possibly seem clear from the context, the amount of those imprecisions makes this paper very hard to follow. For this reason, I maintain my original score.

Reviewer_u8TbOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an automatic circuit-discovery pipeline for equivariant GNN force fields. It perturbs typed local motifs by a small step. It then applies a Möbius decomposition over all combinations of six internal units to find unit-set terms, single units and their interactions, that recur across molecules. These terms are filtered by a causal check with dose and wrong-content controls. A greedy search over units typed by their chemical role then localizes each surviving component into a small circuit, which is confirmed on held-out molecules. Applied to MACE, Orb, and NequIP, the pipeline returns 44 confirmed circuits, which the case studies interpret chemically.

Critical strengths
  1. The paper addresses a relevant and underexplored problem: bringing mechanistic interpretability to molecular force fields, where continuous inputs, equivariant hidden states, and molecule-dependent graphs break the assumptions of existing circuit-discovery tools.
Critical weaknesses
  1. The presentation is hard to follow. Many notations and concepts are introduced densely, and some are not defined at all. Although the problem and method can be understood intuitively, these presentation and notation problems make them very difficult to assess rigorously.

  2. Although the authors present many details to illustrate the designs, several key choices lack enough justification. Examples include the single six-unit interface, the restriction to bond and angle in motifs, the fixed step sizes, and the confirmation thresholds. These choices are mainly checked by sensitivity analysis rather than motivated, which makes it difficult to understand why the designs make sense and why the method works. Since the main parts are adapted from existing tools (Möbius interaction indices, ACDC), the novelty and contributions of the paper are also difficult to evaluate.

Questions for authors
  1. The motifs are defined only as bonded pairs and three-atom angles. What's the motivation to focus on these motifs and what methods are used to decompose molecules into motifs? Why are torsional angles not perturbed, given that many molecular properties and conformational energies are closely related to torsions? Also, how are the bonds, bond orders, and sp2/sp3 hybridization labels assigned, since the models themselves only take element types and coordinates as input?

  2. The perturbed structures are no longer at equilibrium. Could the authors show that the model’s energy predictions remain reliable on these perturbed geometries, e.g., by comparing with reference QM energies, so that the circuits are not explaining out-of-distribution behaviour?

  3. Why were these three models selected? Since Orb is not equivariant and the pipeline only requires swapping internal activations, the method seems applicable to invariant GNNs as well. What is specific to equivariant models here? And would the findings change for other equivariant architectures with different designs, such as Allegro, which is strictly local without message passing?

  4. Could the authors explain the findings in Appendix E.2 in more detail? Also, what is the relationship between the GNN formulation (e.g., different message passing schemes and tensor orders) and the discovered circuits?

  5. In the case studies, some circuits match chemical understanding well, such as the angles for sp3 and sp2 carbons. However, they cover only a few selected circuits, and some of them give values different from the full model (e.g., the C=C bond length). Do most of the 44 circuits have a clear chemical meaning, or only the selected ones?

9 comments
AuthorsOpenReview ↗

Response to reviewer u8Tb, part 1 of 5

We thank the reviewer for the detailed review, which led us to strengthen our claims in the paper. We detail answers to the concerns raised below.

W1. Presentation.

"The presentation is hard to follow. Many notations and concepts are introduced densely ... very difficult to assess rigorously."

We thank the reviewer for their analysis and have made changes in the revised edition, which are now written in blue. These are listed in item 7 of the general response, and our reply to 2N8u (W1b–W1d, Q1) explains each clarified definition (the stage overview is in Table 1).

  • Section 3 now includes a table of the five stages, including a description for each stage and which parts are new or borrowed from existing literature. In addition, the formal algorithm is now detailed in Appendix B.
  • Appendix B.1 adds a notation table and a worked example that follows one perturbation, a C=C stretch in MACE, through all five stages.
  • Definition 2 now defines "boundary" and the rotation of an activation into another frame. In addition, v(A)v(A) means the change in output everywhere, with v(∅)=0v(\emptyset)=0.

W2. Why each design choice, and what is new.

"... several key choices lack enough justification. Examples include the single six-unit interface, the restriction to bond and angle in motifs, the fixed step sizes, and the confirmation thresholds. ... the novelty and contributions of the paper are also difficult to evaluate."

We address all examples detailed in the question below:

Six-unit interface. To ensure the coefficients sum exactly to the response, the interface must be mathematically complete, i.e., its units have to intercept every possible computational route from the perturbation to the energy output.

To achieve this, we use a six-unit interface. The receiver atom's own state provides one unit per irrep block. Two additional units then close the cut: one captures everything arriving through other atoms, and the final one accounts for any geometry reused downstream. We chose this specific design because it is the coarsest complete split that successfully separates the receiver's irreps while keeping the exact decomposition computationally affordable (requiring only 64 forward passes per perturbation). Moving to a finer split just refines these exact same terms without fundamentally changing them. For example, if we split the "other atoms" unit in two, 395 of the 399 terms that do not involve it still recur. Similarly, splitting the channels doesn't generate any interactions above order two, as we detail in Appendix F.5.

Bonds and angles. See Q1a below.

Step sizes. A step has to stay inside the training distribution and still give a measurable response. Our steps of +0.02+0.02 Å and +2∘+2^\circ are 0.49 and 0.29 standard deviations of that coordinate's spread across the population, and the median response, 20 to 49 meV, is more than 2,000 times the detection threshold.

Thresholds.

  • δ=10 μ\delta=10\,\mueV sits far above numerical error in double precision (≤4×10−12\le4\times10^{-12} eV) and far below the response.
  • Prevalence above one half means the term occurs in most molecules of its family.
  • Sufficiency and necessity above 0.5 mean the circuit alone carries most of the effect, and removing it removes most of the effect.
  • The faithfulness band [0.8,1.2][0.8,1.2] lets the search stop only at a set that reproduces the effect to within 20%.

Appendix F reports how the counts change under each choice, and sections 2 and 3 now give each motivation next to its definition.

A clear detail of novelty: We acknowledge in Table 1 that we borrow Möbius coefficients and ACDC-style pruning, but neither can be applied to a force field as-is. ACDC relies on a single fixed graph with discrete inputs, while our molecules have unique graphs and continuous coordinates. Likewise, Möbius coefficients describe single inputs, whereas circuit claims require population-level proof. Our contributions are the exact technical steps required to bridge this gap:

  1. behaviors defined as chemically typed geometric perturbations, which give a finite vocabulary with replication across molecules;
  2. units typed by chemical role, so that one circuit is defined on every molecule of a family;
  3. a population recurrence test with false-discovery control;
  4. content controls that respect the symmetry of equivariant activations;
  5. confirmation on scaffold-disjoint molecules under a design frozen before we saw any outcome.

With this pipeline, we found 44 circuits across three distinct architectures. To our knowledge, these are the first circuits in a molecular force field confirmed on held-out molecules.

AuthorsOpenReview ↗

Response to reviewer u8Tb, part 2 of 5

Q1a. Why bonds and angles, and how motifs are found.

"What's the motivation to focus on these motifs and what methods are used to decompose molecules into motifs?"

Stretches and bends are the two lowest-order internal coordinates and the leading terms of every classical force field. Each moves one atom and is fixed by atoms within two bonds, so it gives a local behaviour with a chemical label and a strong response. We enumerate motifs exhaustively from the bond graph: every bonded pair and every bonded triple jj–ii–kk, typed by element, and by bond order for pairs. We keep a family when enough molecules contain it, which gives 38 families, and a fixed hash picks one occurrence per molecule, ensuring that motifs are not hand-picked.

Q1b. Torsions.

"Why are torsional angles not perturbed, given that many molecular properties and conformational energies are closely related to torsions?"

While we limited this paper to bonds and angles, the method itself also runs on torsions. A torsion that moves a single atom (such as rotating a hydroxyl H around its C–O axis) passes through our current interface without any modification.

To verify this, we ran a pilot study on that exact hydroxyl torsion across all three models, using our identical code, interface, and thresholds (a 10° rotation; 90 molecules per model: 30 for discovery and 60 held out; median response of 14 to 15 meV).

Hydroxyl torsion pilotMACEOrbNequIP
Terms that recur on held-out molecules24179
Components that pass the causal check031
Circuits localised031
Share of live units in the circuitn/a21 to 38%17%
Necessity on held-out moleculesn/a0.99 to 1.000.98
Sufficiency on held-out moleculesn/a0.57 to 0.630.78

The pipeline successfully identified recurring structure for this torsion across all three models, ultimately extracting three circuits in Orb and one in NequIP. On the held-out molecules, each of these circuits proved necessary for its component and outperformed random subgraphs of the same size. However, with only 60 held-out molecules, the sufficiency interval did not quite clear 0.5 under our strict full correction (m=813m=813). As a result, the pipeline conservatively classifies these four circuits as "necessary and better than random" rather than fully "confirmed." (In MACE, the two largest terms failed the content controls, so the pipeline properly returned no circuit).

Fundamentally, the exact same gates evaluate a torsion as a bend. If a torsion rotates an entire fragment and moves many atoms, the single receiver simply becomes a set of receivers. This extends the interface, but leaves all other stages of the pipeline completely unchanged. We chose to focus initially on stretches and bends because they are stiff, highly localized terms that allow us to test circuit claims very cleanly, and their 38 families already provided 6,439 perturbations. We have updated the Discussion to explicitly clarify this scope, and we now report the full details of this torsion pilot in Appendix G.4.

Q1c. Labels.

"... how are the bonds, bond orders, and sp2/sp3 hybridization labels assigned, since the models themselves only take element types and coordinates as input?"

We extract bonds and bond orders from the dataset's atom-mapped SMILES (SPICE) via RDKit, which we also use to determine hybridization. These labels are entirely hidden from the models; they exist in our pipeline solely to group perturbations and name unit roles. Because the models never see this metadata, the chemistry we recover is a genuine result. A circuit that distinguishes sp$^2$ from sp$^3$ carbon does so based entirely on elements and coordinates. Section 2 now clarifies this.

AuthorsOpenReview ↗

Response to reviewer u8Tb, part 3 of 5

Q2. Are the models reliable on the perturbed geometries?

"The perturbed structures are no longer at equilibrium. Could the authors show that the model's energy predictions remain reliable on these perturbed geometries ... so that the circuits are not explaining out-of-distribution behaviour?"

Yes. We verified this through three checks, including new DFT calculations (now detailed in Appendix A.1).

  • The perturbed geometries stay in distribution: 99.7% of perturbed coordinates lie inside the population's own 1st to 99th percentile for that motif type, and 0.14% lie beyond three standard deviations.
  • First-order forces agree: Across the panel, the energy change predicted by the dataset's reference forces matches each model at r=0.9995r=0.9995, 0.99940.9994, and 0.99670.9967 (excluding one molecule with internally inconsistent stored forces).
  • New DFT calculations confirm reliability: We recomputed DFT single points at the SPICE reference level (ωB97M-D3BJ/def2-TZVPPD) for 693 perturbed geometries and their clean counterparts. The clean energies reproduce the dataset to a median of 0.5 meV.
modelnnPearson rrslopeMAE (meV)same sign
MACE6930.99961.0021.299.3%
Orb6120.99961.0001.397.9%
NequIP6930.99880.9972.398.3%

With a median DFT energy change of 33 meV, the model error is quite small: just 2% to 4% of the total response. This near-perfect agreement holds up when evaluating stretches and bends separately (r≥0.997r\ge0.997), as well as within each of the 38 individual families for MACE (r≥0.996r\ge0.996). (Orb's panel covers 37 families, hence n=612n=612). Furthermore, because we calculated these single points at the exact geometries used by our pipeline, they naturally account for the small incidental distance changes that occur during an angular step.

Q3a. Why these three models, and what is specific to equivariance.

"Why were these three models selected? Since Orb is not equivariant ... What is specific to equivariant models here?"

We study the models as released. We chose three that cover the same organic chemistry using three distinct designs: many-body equivariant products (MACE), equivariant tensor-product message passing (NequIP, checkpoint nequip_organics_02, now named in Appendix A), and a non-equivariant network that learns approximate symmetry from data (Orb).

The reviewer is correct in their observation that the pipeline also runs on invariant networks; we included Orb specifically to show this. However, three parts of the method exist specifically to handle equivariance:

  • Units are whole irrep blocks, so each has a geometric type and no intervention breaks the symmetry.
  • We built the content controls to commute with rotation, ensuring the wrong content is still valid equivariant content of the same norm.
  • When we move an activation to another site or molecule, we rotate it using the model's own Wigner matrices.

Section 4 now explicitly clarifies both points: the pipeline runs without equivariance, and where a model does have it, these are the three specific places the pipeline takes advantage of it.

Q3b. Allegro.

"... would the findings change for other equivariant architectures with different designs, such as Allegro, which is strictly local without message passing?"

Allegro would enter the pipeline through an interface declared on its pair features, just as Orb entered through its node and edge states. We excluded it because, to our knowledge, no Allegro checkpoint for organic molecules has been publicly released. The available Allegro foundation models are trained on inorganic materials, and the SPICE-trained models we are aware of were published as configurations without weights.

Training our own model would violate our strict rule to study models exactly as released, because the results would then reflect our specific training run rather than a community baseline. Whether a strictly local model organizes these terms differently remains a question for future work. In the meantime, Appendix E.3 shows which of the current findings depend on the architecture (see Q4 below).

Reviewer_gBc4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

he paper proposes an automated circuit-discovery pipeline for molecular neural-network force fields. Chemically typed triangles and bond stretches define geometric perturbations. An exact Möbius decomposition over all 262^6 interventions at a six-unit internal interface identifies recurring interaction terms. Selected components undergo dose-response and content-control tests, followed by circuit localization on a graph whose units are defined by chemical roles. Circuits are subsequently evaluated on independent molecules using sufficiency, necessity, and comparisons with random subgraphs.

Across MACE, Orb, and NequIP, the paper reports 44 confirmed circuits under the strictly held-out evaluation protocol. Case studies relate selected circuits to bond stretching, angular responses, and atomic hybridization.

Critical strengths

The problem is important, and the adaptation to molecular models is substantive. Continuous geometric inputs, equivariant hidden representations, and computational graphs that vary across molecules create meaningful challenges for circuit discovery. Defining internal units by chemical roles provides a useful way to instantiate the same circuit across different molecules.

The experimental protocol is unusually explicit. The paper distinguishes recurring terms, components, and confirmed circuits, and evaluates dose responses, content specificity, sufficiency, and necessity. Appendix C specifies independent confirmation molecules and scaffold separation. Strict and topped-up results are reported separately.

The supplementary evaluation is substantial. Ablations examine interface granularity, perturbation magnitude, interface depth, random initialization, locality baselines, and threshold sensitivity. Reporting the fraction of the full energy response carried by each component helps readers understand the scope of an explanation.

Critical weaknesses
  1. The connection between discovering interaction terms and validating the final circuits needs stronger evidence. references : def 2-3, p.4, Appendix E and F. 3 candidate selection uses Mobius Coef. c(S), whereas circuit localization and confirmation target the joint intervention effect:
eC=v(S)−v(∅)=∑∅≠T⊆Sc(T).e_C=v(S)-v(\varnothing) =\sum_{\varnothing\neq T\subseteq S}c(T).

For a pair S={i,j}S=\{i,j\}, this becomes

eC=c({i})+c({j})+c({i,j}).e_C=c(\{i\})+c(\{j\})+c(\{i,j\}).

Consequently, a circuit that successfully reproduces eCe_C may primarily reproduce the singleton contributions rather than the interaction that motivated selecting the component. This is mathematically consistent with the definitions, but it leaves the contribution of the interaction-discovery stage insufficiently established.

  1. The baselines do not adequately separate mechanism discovery from computational-path connectivity. The nearest-unit and stratum-matched random baselines are useful starting points. However, the paper explicitly notes that random subgraphs in Orb and NequIP rarely contain a complete upstream–inside–downstream path, explaining their near-zero sufficiency.

The observed advantage may therefore partly reflect the search preserving a complete information-transmission path while the baselines disconnect it. Matching unit counts and regional composition does not fully control for this difference.

A stronger comparison would use either path-connected random subgraphs or an explicitly evaluated attribution top-kk baseline on the same role-typed graph. Using attribution internally for ranking is not equivalent to reporting its performance as a standalone baseline. One such comparison would be sufficient to address this concern without requiring extensive adaptations of prior methods.

Questions for authors

How does restricting candidate selection to singleton or pairwise terms affect the final confirmed circuits, their explained response fractions, and computational cost? Does the sufficiency advantage persist against a baseline that preserves complete computational paths? How many of the 11 confirmed NequIP circuits originate from interaction terms that fail the precision-margin screen? Does double-precision recomputation change their selection or interpretation?

6 comments
AuthorsOpenReview ↗

Response to reviewer gBc4, part 1 of 3

We thank the reviewer for their review and detail our responses to each point raised, below.

W1. Selection uses c(S)c(S), confirmation targets eCe_C.

"The connection between discovering interaction terms and validating the final circuits needs stronger evidence ... it leaves the contribution of the interaction-discovery stage insufficiently established."

We appreciate the reviewer's observation: for a pair, eCe_C contains both singleton terms. Empirically however, each confirmed circuit (44/44 strict, 58/58 topped-up) comes from a single-unit component S={i}S=\{i\}. For a single unit eC=v({i})=c({i})e_C=v(\{i\})=c(\{i\}). Pairs were eligible candidates throughout the process and initially performed well, taking 10 of the top-3 slots when the screen ranked all 63 unit sets. Ultimately, they were filtered out by the later stages:

  • Three had too few admitted molecules to grade.
  • Three failed the stage-3 controls.
  • Four reached confirmation but fell short on sufficiency.

Due to this, no reported circuit ends up reproducing singletons in place of an interaction that selected it. We have added a clarification to Section 4.1 to ensure this is clear.

While a one-unit screen does yield the exact same list of circuits (as we detail in Q1 below and now state plainly in the revision), our full decomposition measures two critical properties that a simple one-unit screen cannot.

First, it tells us whether single units are additive, i.e. whether each keeps its effect when the other units change. A one-unit patch only measures v({i})v(\{i\}) in a clean context, so it can't reveal if unit ii maintains its effect when other units change. To evaluate this, we split each circuit unit's effect into its main term cic_i and its interaction mass Ii=∑i∈S, ∣S∣≥2c(S)I_i=\sum_{i \in S,\,\vert{}S\vert{}\ge2}c(S) across all 9,222 stored decompositions:

MACEOrbNequIPall
median ∣Ii∣/∣ci∣\lvert I_i\rvert/\lvert c_i\rvert0.0210.0080.0130.012

These interactions are quite small. The median ratio is under 10% in 53/58 circuits and under 25% across all circuits. Even when every other unit is perturbed, a unit's effect remains 1.003 times its clean effect (at the median) and keeps the same sign in 99.2% of cases. The confirmed circuits represent the norm here: across all 617 recurring single units (102,369 unit-perturbation pairs), the median ratio is just 4.5%, with 564 staying under 25%. The only real exceptions are NequIP's directional and non-local units, though its scalar channel (which 9 of its 15 circuits use) remains additive across all 38 families.

Second, the decomposition provides a map of which units actually interact. While interactions make up 56% to 63% of the recurring terms, they carry only 2% to 8% of the coefficient mass. Importantly, the model learned these interactions: randomly initialized copies of MACE produce no recurring interactions, whereas our test successfully recovers the pairwise term of a classical force field in 23 out of 23 families (Appendix E.2; details in our reply to u8Tb, Q4).

Together, these findings support a clear conclusion, which we now explicitly state in Section 4.1 and the Discussion: single units handle the primary bond and angle responses, while interactions merely provide small, learned corrections. We have also updated Section 3 to clarify that out of our 64 evaluations, 7 act as one-unit patches, while the remaining 57 explicitly test whether a unit maintains its effect when the surrounding context changes.

Q1. Restricting selection to singletons or pairs.

"How does restricting candidate selection to singleton or pairwise terms affect the final confirmed circuits, their explained response fractions, and computational cost?"

To test this, we first verified that our code reproduced every original production q-value exactly. We then re-ran stages 1 and 2 under each restriction, applying the Benjamini–Yekutieli correction to the correspondingly smaller families.

selection fromevaluations/perturbationstrict circuits (MACE/Orb/NequIP)topped-upexplained fractionpipeline compute hours (MACE/Orb/NequIP)
all 63 sets6416 / 17 / 1158as reported36 / 50 / 671
singletons and pairs22same 44same 58unchanged36 / 49 / 583
singletons7same 44same 58unchanged35 / 49 / 522

Restricting the selection does not change anything about the confirmed circuits or their explained fractions. When we limit the screen to singletons, the 10 slots previously occupied by pairs simply roll over to singletons that the screen had already chosen anyway. As a result, no new candidates ever enter stages 3 through 5.

The only real impact is on compute. Dropping to a 7-evaluation screen cuts NequIP's cost by 22%. For MACE and Orb, the savings are minimal (3% or less) because the decomposition step only accounts for 1 to 3% of their total pipeline. We report this 7-evaluation screen in Appendix E.4 as a cheaper variant of the pipeline that gives the same circuits.

AuthorsOpenReview ↗

Response to reviewer gBc4, part 2 of 3

W2, Q2. Baselines that keep a complete computational path.

"The baselines do not adequately separate mechanism discovery from computational-path connectivity. ... A stronger comparison would use either path-connected random subgraphs or an explicitly evaluated attribution top-k baseline on the same role-typed graph."

"Does the sufficiency advantage persist against a baseline that preserves complete computational paths?"

We ran both baselines across all 58 circuits and every admitted held-out molecule (2,082 molecule-circuit pairs), scoring them against the paper's original criteria with the same Bonferroni correction (m=813m=813).

baselinemedian sufficiency, circuit vs baselinecircuit betterbaseline would be confirmed
path-connected random, size-matched0.86 vs 0.00558/580/58
path-connected random, size- and region-matched0.86 vs 0.00758/580/58
attribution top-k,k=∣S∗∣k,k=\lvert S^\ast\rvert0.86 vs 0.8524 better, 24 identical sets, 1 tied, 9 worse31/58

Path Connectivity (W2) The reviewer was correct about the original pipeline for Orb and NequIP: only 31% and 22% of those random draws actually contained a complete computational path (compared to 88% in MACE). However, when we explicitly force the random subgraphs to be path-connected, they still fail. Per model, our circuits reach median sufficiencies of 0.99, 0.84, and 0.82, while the path-connected random draws flatline at 0.10, 0.00, and 0.00.

Ultimately, the two random baselines test slightly different things, and the circuits beat both. For MACE, the original baseline was actually a stricter test because it matched the exact unit count per stratum (e.g., messages, channel bands, readouts). Because MACE's graphs are small, 64 of the 420 random draws were the circuit itself, which is why the baseline retained a median 0.34 of the effect (0.39 over the 16 strict circuits). The new path-connected baseline only matches size and region, which is why it drops to 0.10 in MACE. But for Orb and NequIP, forcing path connectivity proves that while a complete path is necessary (a subgraph without one scores exactly 0), it is far from sufficient. The specific units on that path dictate the outcome.

We have updated Sec 3, Algorithm 4, and App. F to define the strata using this terminology for clarification.

Attribution Top-kk (Q2)

For this baseline, we gave top-kk a head start by using the exact size (kk) that our search found. This isolates what our faithfulness band and pruning steps add over a naive ranking. The search matches or beats top-kk in 49 of the 58 cases. In the 9 cases where top-kk "wins" on the sufficiency score, it only does so by vastly overshooting the effect (by 1.35x to 2.99x; a set that triples the effect does not explain it) which is why our confirmation criteria enforce a two-sided faithfulness band. Eight of those nine top-kk circuits fail confirmation for this reason.

The value of the search scales with the complexity of the graph:

  • MACE (10–24 live units): In these small graphs, ranking alone is nearly enough. Top-kk finds the same circuit in 16 of 21 cases, and overshoots the effect in the other 5.
  • Orb (102–445 live units): Here, the band and pruning are essential. The searched circuit wins in 19 of 22 cases. Top-kk would only be confirmed for 5 of the 22 cases, logging a median sufficiency of 0.60 against our 0.84. Furthermore, in 5 of Orb's "stretch" circuits, the top-kk set fails to even form a complete path, scoring 0.
  • NequIP: 7 sets are identical, our search wins in 5, ties in 1, and top-kk overshoots in 2.

We report both baselines with this per-model breakdown in Sec 4.2. App F.2 contains the full table and the counts (top-kk is identical in 24 cases, scores lower in 24, ties in 1, and overshoots in 9).

AuthorsOpenReview ↗

Response to reviewer gBc4, part 3 of 3

Q3. NequIP precision.

"How many of the 11 confirmed NequIP circuits originate from interaction terms that fail the precision-margin screen? Does double-precision recomputation change their selection or interpretation?"

None of the 11: all 11 strict NequIP circuits (and all 15 topped-up) are single-unit components, and the precision screen keeps all 190 recurring NequIP singletons, as it only filters out pairs. Additionally, stages 3-5 were already run in float64 for every model. Double-precision recomputation does not change their selection or interpretation. We verified this by replaying all 6,439 NequIP stage-1 perturbations in float64 (190 CPU hours, zero failures), finding that:

  • Selections are completely stable: No stage-2 selection changes in any of the 123 NequIP components. All 190 recurring singletons remain, and recurring pairs shift only from 240 to 238 (6 lost, 4 gained).
  • Negligible error: The reconstruction error is at most 1.8×10−141.8\times10^{-14} eV.
  • Our margin was highly conservative: The float32 error on pair coefficients was at most 9.2×10−69.2\times10^{-6} eV. In fact, 119 of the 125 pairs our screen originally set aside actually recur in float64.

We have added the full details of the above experiments to Appendix E.1.

We thank the reviewer for their constructive feedback, which has helped us strengthen our arguments and empirical results. We hope these new ablations address the concerns and all three of your questions. We would be glad to run any further analyses during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision and general comments

We thank the reviewers for their careful and constructive feedback. All reviewers recognized the value of automating circuit discovery in this domain, while appropriately highlighting areas where the manuscript needed clearer exposition and stronger empirical backing. In the revised PDF (where we have marked all material changes in blue), we have restructured the narrative for readability (changes across the paper) and added six new ablations and analyses. Crucially, with no reported circuit changed, the results of our algorithm stay strong and consistent under these rigorous stress tests.

  1. In order to evaluate model accuracy on displaced structures (u8Tb Q2), we computed DFT reference energies (ωB97M-D3BJ/def2-TZVPPD) for 693 perturbed geometries. The predicted energy changes closely match the DFT values, yielding Pearson correlation coefficients of 0.9996 (MACE), 0.9996 (Orb), and 0.9988 (NequIP). Mean absolute errors range from 1.2 to 2.3 meV against a median energy change of 33 meV. These structural perturbations represent 0.3 to 0.5 standard deviations of each coordinate's natural variance (Section 4, Appendix A.1).
  2. Since a one-unit screen cannot measure two critical properties: whether a unit keeps its effect when the other units change, and the precise inter-unit interactions, the full decomposition is required to quantify these relationships. By mapping the full decomposition, we demonstrate that single units are enough to reconstruct our list of circuits (for a single unit, c({i})=eCc(\{i\}) = e_C), as higher-order interactions contribute a median of just 1.2% to the main effect and stay under 10% for 53 of 58 circuits. Restricting selection to single units reduces evaluations from 64 to 7, but the full decomposition provides the evidence for our physical interpretation: bond and angle responses localize to single units, with interactions acting only as minor learned corrections (Sections 3, 4.1, 5, Appendix E.4).
  3. In comparison against random and attribution baselines (gBc4 W2, Q2), we demonstrate that the extracted circuits exhibit much higher fidelity than random subgraphs containing a complete path. The random subgraphs retain only a 0.005 fractional effect (failing to be confirmed in any of the 58 cases) compared to 0.86 for the extracted circuits. We also evaluated an attribution-based baseline using the top-ranked units. Our extraction method matches or beats this attribution set in 49 of 58 cases, while in the remaining 9 cases, the attribution baseline overshoots the actual effect (Section 4.2, Appendix F.2).
  4. We show that double precision doesn't change any circuit (gBc4 Q3), by replaying all 6,439 NequIP perturbations in float64, and no selection or circuit changed (Appendix E.1).
  5. In order to interpret the discovered circuits physically (u8Tb Q5, 2N8u Q2), we analyzed the chemical behavior of all 58 confirmed circuits. In 55 cases, the circuit's energy contribution curves upward as the target bond or angle is displaced within a molecule, acting as a harmonic spring with positive stiffness. Furthermore, 36 circuits maintain this restoring behavior across different molecules. We evaluated the circuits against a strict four-part heuristic based on textbook chemical principles, with criteria pre-defined prior to the sweep using our seven case studies. Fifteen circuits satisfy all four criteria; for reference, the full models pass 21 of 58 on the three criteria that apply to them (C1, C3, C4), which shows how demanding the across-molecule fit is (Section 4.3, Appendix G.3).
  6. We show a generalization to torsional degrees of freedom (u8Tb Q1b). While the primary scope of this manuscript is restricted to bonds and angles, the extraction pipeline also runs on torsions without changes. Evaluating a hydroxyl torsion through the unmodified pipeline across all three models, we identified 50 recurring terms on held-out molecules. Additionally, four extracted circuits in Orb and NequIP are necessary for their respective components (necessity 0.98–1.00) and significantly outperform random subgraphs, though their sufficiency does not yet clear the confirmation bar with 60 held-out molecules (Appendix G.4).
  7. To improve the overall clarity of the manuscript, Section 3 now opens with a summary table outlining the five stages of the pipeline, clearly delineating our novel contributions from established methods. We have also added a comprehensive notation table and a fully worked example (Appendix B.1). Furthermore, Definition 2 has been updated to formally define both the "boundary" and the frame rotation. Finally, we standardized the notation v(A)v(A) to denote the change in output throughout the text and provided explicit justifications for all methodological design choices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.