acceptodds
Under review as a conference paper at ICLR 2027

What Drives the Prediction? How Neural Models Rely on Physical Information

Abstract

Physical quantities such as energy and momentum can be decoded accurately from a neural model even when its predictions respond differently to a change in available signals. We study this gap between physical representation and predictive reliance. Matched Invariant Interchange (MII) compares activation exchanges selected for their correlation with a physical quantity against random exchanges of the same size. A complementary regression task varies the availability of a target-correlated hint during training and removes its informativeness at evaluation. Our contributions are: i) Decodability–coupling dissociation: across five dynamics systems and five exchange sizes per system, high decodability coexists with a maximum absolute MII gap of 0.0068. ii) Distribution-dependent reliance: across five invariant-regression systems, informative-hint and decorrelated-hint training both yield decoding –, yet mean absolute output–target correlation after hint permutation is 0.09 versus approximately 1.00. Decorrelated-hint training also increases sensitivity to invariant information relative to hint information. iii) Output-local coupling: in pretrained Poseidon-B, the decoder MII gap is 0.893 for the spatial integral of one velocity component, , and 0.027 for kinetic energy. The comparisons distinguish readable physical information, training-dependent signal reliance, and sensitivity arising near the output. They support evaluating physical representations through matched interventions and behavior after a predictive training signal changes.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
mechanistic interpretability, scientific machine learning, physical representations, activation interventions, shortcut learning, neural dynamics models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MKH5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper demonstrates a dissociation between network representation and prediction (or behavior). The authors defined a new metric called MII to quantify the additional information about a physical variable gained from selective neuronal activations compared to random activations. Across dynamical systems, the authors demonstrate that the decodability of physical variable can remain high with almost zero MII. In a controlled shortcut learning experiment, the authors also showed that networks replying on shortcut learning will have a critical dissociation between decodability and network behavior.

Critical strengths
  1. The proposed metric (MII) is useful and intuitive. It is also well controlled, since it accounts for the effect of swapping the same number of random neurons.
  2. The experiments cover a broad range of settings, including both small models and a large pretrained model.
  3. The shortcut-learning section is a nice demonstration of the decoupling between decoding and network behavior.
Critical weaknesses
  1. This paper lacks critical details about the task, the model, and key experimental setups. In section 4.1 and section 5, the authors talked about changing objectives and 8 objective modifications, but those alternative objectives aren't introduced anywhere and it is entirely unclear to me what those are and how they're supposed to improve decoding of physical variables.
  2. Results in Section 5 is mostly expected from prior literature and work in shortcut learning. The central claim that "representation dissociates from prediction" is also well established in probing research.
  3. Without specifying the number of swapped units, the model size, and the layer in which the neurons are swapped, it is hard to judge whether near-zero MII means the variable isn’t used, or just that swapping so few units can’t change the prediction at all. The authors should show a sweep of prediction change and MII with respect to number of neurons swapped.
Questions for authors
  1. Can you fully specify the 8 objective modifications in section 4.1 and 5, and clarify why they should in principle improve decoding? And why some of them don't improve decoding at all?
  2. How many neurons were swapped in section 4? Can you add a sweep of prediction change and MII with respect to number of neurons swapped?
  3. At which layers are those neurons most correlated with physical quantities located?
  4. What is the model size and hidden dimension for those small models? At which layers are those swapped neurons generally located?
  5. For section 6, can you test more layers and more quantities in Poseidon to see whether the large effect really appears only near the output?
Reviewer_hKhWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the decodability of physical information in neural models relates to their reliance on that information for prediction. The authors compare activation exchanges on physical-quantity-correlated units against random units—along with experiments across layers and shortcut signals—to separate whether a physical quantity can be read from hidden representations from whether it actually drives predictions. They further test how the readout form affects intervention responses. One design choice I liked is computing post-intervention physical quantities directly from the model's predicted state rather than re-reading them with the same probe, and matching donors and unit counts gives a clean control for the comparisons.

My main reservation is how much we learn here beyond what is already well known. Separating decodability from causal use was already the central point of Amnesic Probing (Elazar et al., 2021), and prior work has shown that changing the training distribution shifts which cues a given architecture relies on (e.g., texture vs. shape in Geirhos et al., 2019). It is not yet clear what new insight this combined diagnostic offers over those findings. In particular, an near-zero MII gap only shows that picking correlated units does no better than random selection—it does not tell us whether the model uses physical information or not.

Incomplete reporting makes the main dynamics results even harder to interpret. The paper omits baseline predictive performance for the dynamics models as well as the two individual component responses behind the MII gap, leaving it unclear whether a small gap means both interventions had an effect or neither did. Key model configurations and data splits are missing too. While the hint experiment shows clearer behavioral differences, it only tests physical-quantity regression and does not resolve these issues in the dynamics prediction setup.

Critical strengths

Evaluating intervention effects directly from the predicted state is a sound design choice. While the probe measures how easily physical quantities can be read from hidden representations, post-intervention energy and momentum are computed from the model's predicted state via exact physical formulas. This checks whether modifying hidden units actually propagates to the predictor's output, rather than just shifting the probe's readout, and avoids circularity from using the same probe to both define and test the intervention. The random baseline in MII is also well-controlled. By matching both the donor examples and the number of exchanged units between the correlated and random conditions, the setup rules out unit count or donor differences as confounders and directly tests whether targeting physically correlated units tracks the donor's physical quantity better than random patching.

Critical weaknesses
  1. I think the paper needs to explain more clearly what new diagnostic insight MII provides. The distinction between information being readable and being used by a model was already explicitly discussed by Elazar et al. (2021), and the observation that training data changes which cues a model relies on has experimental precedents such as Geirhos et al. (2019). MII specifically measures the selection advantage of physical-quantity-correlated units over random units. A small advantage, however, does not distinguish between the model not relying on that information and the information contributing to prediction through a distributed or redundant representation. The authors interpret the metric cautiously, but this also means that the paper needs to explain what these results add to our understanding of physical prediction, beyond showing another instance where decodability and a particular intervention response do not agree.

  2. The main dynamics experiments lack information that directly affects the interpretation of the results. The paper emphasizes small MII gaps without showing the individual responses to correlated-unit and random-unit exchanges, leaving readers unable to tell whether both are equally effective or equally ineffective. Predictive errors under clean inputs are also not systematically reported for the main predictors, making it difficult to judge the quality of the dynamics models in which these intervention results occur. In addition, the appendices provide insufficient detail about the small models’ widths, intervention layers, and data splits between probe fitting, unit selection, and final evaluation. These details are necessary to assess the reliability and meaning of the central negative results and should not be treated as minor implementation details.

Questions for authors
  1. Could you report the individual responses to correlated-unit and random-unit exchanges for the main MII results, together with the corresponding predictors’ prediction errors on inputs without intervention? I would like to understand whether the small gaps reflect two interventions with substantial effects or two weak interventions, and how well the underlying models predict the dynamics.

  2. Could you specify the widths and intervention layers of the main small models, and explain which data are used for probe fitting, correlated-unit selection, and final intervention evaluation? How are the data split between these steps, and are they separated by independent trajectories?

  3. What diagnostic information do you think MII adds beyond existing probing and activation-intervention analyses? In particular, for the small MII gaps that form the main results, could you use a concrete example from the existing experiments to explain what new understanding they provide beyond the distinction between decodability and use?

Reviewer_FqELOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper tries to give an answer to whether linearly decodable physical quantities (energy etc...) from a dynamics' model activations are actually used by the neural network for predictions. To solve this they introduce the metric called MII: first they swap the most correlated activations with the decodable quantity of a recipient with the same units of a donor; then they measure the correlation between the energy of the predicted state with the donor's energy and finally they repeat the process choosing now random units. Finally they compute the difference between the computed correlations in the 2 cases. On five toy systems they show that although the R2R^2 is almost perfect ∼1\sim 1 the MII gap has very low values ∼0\sim 0 and adding physic inspired objectives improve the some probe score without substantial changes of the response. Moreover they show that in a hint-regression task two models with the same decodability can have opposite reliance under hint-permutation.Finally they show that in a pretrained Poseidon-B model a strong selective effect is present only at the decoder for a linear readout with a much smaller effect for the kinetic energy. To conclude, the paper suggests that a linear probe should be complemented with matched interventions and behavioral tests.

Critical strengths

The paper shows that current methods to evaluate physical representations are actually prone to errors. It shows that a conservation objective can have high probe scores while not changing the models's actual reliance on the physics, with the extreme case of two model having the same probe score but two very different behaviors. Thus the common practice of certifying physical models through decodability can be misleading. With their new methodology, the authors propose a more resilient protocol.

Critical weaknesses
  1. The MII has not positive control as the paper never shows that is able to spot realince in a dynamics model where the reliance is known. For instance, the Poseidon decoder effect is related to the output map and very low gaps could be linked to the selection rule rather than low physical reliance.
  2. Given the experimental nature of the paper, the generality is not proven enough. The paper focuses on small models tested on toy problems. Even if they may be sufficient challenge the linear probe, they are not enough to propose the new methodology as a standard protocol.
Questions for authors

I would appreciate comments and experiments related to the aforementioned weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.