- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper starts from a puzzling experimental observation concerning in-vivo neural control via presented (generated) visual stimuli: models that scored favourable on held-out neural predictivity were not automatically good model for neural control (i.e. via model-based control obtained by gradient ascent on the model's prediction), and viceversa; moreover even poor-controller models were generally good at estimating the efficacy of synthesised control-images (i.e. good "peer-review" predictivity). In this work the author propose a linear theory to mechanistically explain this puzzling behaviour: in a simplified teacher-student framework they show how this asymmetry fundamentally stems from the fact that control evaluates models on a different distribution than what predictivity measures do. Fundamentally, prediction is weighted by the data covariance matrix, while control puts equal weight on all weight-direction, so control fails when control optimisation steps into low-variability data directions. The author offer a precise geometrical picture for this fact and validate their predictions via Monte Carlo estimates. Finally, the authors apply their linear theory to non-linear network by considering the Jacobian-linearization and showing that their geometrically-derived factor yields a better predictor for the model performance as controller then the corresponding held-out neural predictivity.
Critical strengths
In this work the authors have tackled an initially puzzling experimental finding and proceeded to build a methodical mechanistic explanation for the phenomenon in a simplified linear theory which yielded an intuitive and satisfying explanation: prediction and control are evaluated on different distributions, the latter being generated by the controller itself. The geometrical intuition behind this asymmetry (conveyed in Fig.2) is particularly helpful. The author then support their claim via extensive Monte Carlo simulations and an explicit construction that recreates the same phenomenon in the studied teacher-student setting. The final extension to the non-linear network case is also appreciated, where the theory-inform geometrical factor provides a better (although not perfect) predictor of downstream controller performance than the standard held-out neural predictivity. This last piece might provide especially useful to the experimental practitioners for designing the next neural control-based experiments and informing the model-selection process.
Critical weaknesses
Minor points:
- Figure 1 is hard to read due to the small fonts and high visual complexity. Perhaps the author can consider simplifying the design to focus and convey the most important aspects (panel B,C) and moving the rest to the appendix.
Questions for authors
Some curiosity-based questions for the authors:
- What are the author recommendations for obtaining reliable neural controller? Can those theory-derived insights be applied to training a model specifically for control?
- Adversarially trained models tend to be the best neural controller. Do the authors think this is because adversarial training tends to probe-and-exploit low-variance directions, thus implicitly regularising the data covariance to be "more spherical", with in turns reduces the risk of mis-control? Or is there some other mechanism involved?
3 comments
Dear Reviewer fGdA,
Can you please take a look at the authors' response and provide your further input? Thank you.
AC
I thank the authors for their response, especially for the recommendations on how to improve future controller, both at the training and generation stage. I confirm my score and recommend acceptance for this work.
Response to `fGdA`
We appreciate reviewer
fGdAfor their detailed reading, insightful comment, and positive evaluation of our work!W1:
Thanks for the helpful comment! We are remaking Figure 1 to break it down and make it easier to read.
Q1: Recommendations for obtaining reliable neural controller
Training side: Our current insight is to directly regularize or supervise the model’s gradients so that they have a more naturalistic structure. Empirically, adversarial training is an effective way to encourage natural-image-like gradients. In practice, adversarial fine-tuning offers a good recipe: start from a strong foundation model (CNN or transformer) and then make its gradients more perceptually aligned. Among our ten models, CLIPAG is the result of this kind of adversarial fine-tuning, and it performed well on neural control, as did ResNet50-robust (which was adversarially trained from scratch).
One interesting future direction is to find alternative training method other than adversarial training: is directly regularizing frequency spectrum of input gradient a competitive alternative?
Generation side: According to the theory, the failure of control is a form of out-of-distribution (OOD) generalization failure. If a stronger prior is supplied during the image generation process (e.g., diffusion models, GANs, etc. Pierzchlewicz et al. 2023; Wang & Ponce 2026), we expect the predictive model’s judgements to be better calibrated and the control to be more successful. We think this will be especially helpful for models that were not robustly trained.
Q2: Cause of Adv. Robust models performing better
For adversarially trained models, they tend to have gradients more aligned with low freq / high variance dimension of data — as we can see the gradients from adv. trained models generally have better spatial frequency profiles (decay like natural images), and visually they looks more like shapes instead of meaningless noise.
So related to the feature regression theory, when we do accentuation and derive control stimuli from adv robust models, the image sweeps are more aligned with natural data manifold, so we think the models have less out-of-distribution prediction error for their images. Thus on these robust model generated images the robust models have better prediction of neural out come thus control score. Actually, as a matter of fact, all other models have higher neural prediction scores on these robust model generated images — suggesting they may be closer to training distribution.