- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Trajectory-Induced Modes (TIMs), a technique for decomposing activation trajectories of individual neural network inputs into components that are transient, persistent or amplified through layers in the model. The approach derives "persistence scores" from an eigendecomposition of an affine surrogate fitted on the activation trajectory of a single input - these scores along with their corresponding eigenvectors are used to construct representative activation-space vectors (TIM representatives) at varying levels of persistence. The authors argue that TIM representatives are both predictive of the final hidden state (hence they capture representational retention) and functionally relevant directions (which they demonstrate via ablation). They also explore prevalence of transient, persistent and amplified modes across the modalities of vision, language and audio in 108 pretrained models, and evaluate the alignment of TIM representatives to cortical responses.
Critical strengths
The analysis covers a real breadth of models and modalities - 108 models across vision, language, audio. They also include many of their results broken down per-model in the appendix which is good to see.
The evaluation is also relatively broad, including a comparison of the distribution of eigenvalues between the three modalities and evaluations of TIM representatives for predicting the final hidden state, functional contributions of these directions to task performance, cortical alignment and robustness to perturbation.
The authors are careful to note what each evaluation does and does not establish and they use this to inform the set of experiments they run, which helps form a coherent narrative, e.g:
Spectral persistence alone does not establish... we therefore...
and
Greater terminal-output alignment does not establish... we therefore...
Critical weaknesses
Soundness concerns that affect some central claims of the paper:
- It seems the transient representative (and perhaps , depending on its definition) may not always actually be "transient" - e.g. if a trajectory had for all , it is unclear what represents given that all the modes involved are actually "persistent". This seems especially plausible for certain inputs in the case of language or vision, based on Figure 1.
- The authors discuss the effect of residual connections on spectral distributions but they do not discuss the implications of residual identity on their results in Table 1 and Table 4, where this may have a significant effect (it seems easy to predict layer N+1 if it is equal to layer N + some small update, and this effect is not isolated in the results).
Additionally, the significance or applications of this work are not immediately clear to me; the paper would be significantly strengthened if the authors could describe a research question or use case for which TIMs are better-suited or more informative than existing techniques.
Questions for authors
In addition to the points raised in the "weaknesses" section, I would like to request a few clarifications:
- What is the exact construction of and and how do these differ from the and in the previous section? I think it is important to add some detail here as these vectors are a key part of the ablation experiment.
- How does this approach deal with models that have layers of varying width, which is probably the case for many/most models being used here? The paper appears to assume a fixed dimensionality across all layers of a given model.
- When computing the representatives, what is the motivation for deriving them from the depth-averaged representation ? Why not derive them from the first layer instead? Do the results in section 4.3 (on representational retention) depend on this choice?