- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper demonstrates a dissociation between network representation and prediction (or behavior). The authors defined a new metric called MII to quantify the additional information about a physical variable gained from selective neuronal activations compared to random activations. Across dynamical systems, the authors demonstrate that the decodability of physical variable can remain high with almost zero MII. In a controlled shortcut learning experiment, the authors also showed that networks replying on shortcut learning will have a critical dissociation between decodability and network behavior.
Critical strengths
- The proposed metric (MII) is useful and intuitive. It is also well controlled, since it accounts for the effect of swapping the same number of random neurons.
- The experiments cover a broad range of settings, including both small models and a large pretrained model.
- The shortcut-learning section is a nice demonstration of the decoupling between decoding and network behavior.
Critical weaknesses
- This paper lacks critical details about the task, the model, and key experimental setups. In section 4.1 and section 5, the authors talked about changing objectives and 8 objective modifications, but those alternative objectives aren't introduced anywhere and it is entirely unclear to me what those are and how they're supposed to improve decoding of physical variables.
- Results in Section 5 is mostly expected from prior literature and work in shortcut learning. The central claim that "representation dissociates from prediction" is also well established in probing research.
- Without specifying the number of swapped units, the model size, and the layer in which the neurons are swapped, it is hard to judge whether near-zero MII means the variable isn’t used, or just that swapping so few units can’t change the prediction at all. The authors should show a sweep of prediction change and MII with respect to number of neurons swapped.
Questions for authors
- Can you fully specify the 8 objective modifications in section 4.1 and 5, and clarify why they should in principle improve decoding? And why some of them don't improve decoding at all?
- How many neurons were swapped in section 4? Can you add a sweep of prediction change and MII with respect to number of neurons swapped?
- At which layers are those neurons most correlated with physical quantities located?
- What is the model size and hidden dimension for those small models? At which layers are those swapped neurons generally located?
- For section 6, can you test more layers and more quantities in Poseidon to see whether the large effect really appears only near the output?