- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
Authors investigate an important and underexplored problem in weight-space learning, which is about understanding how meta-weight transformers (MWTs) perform classification. In the general setting, meta-learning is applied to develop implicit neural representations (INRs) that both represent target images and have parameters that can be processed to classify the represented image. However, it is generally unclear how the weights are organized to jointly enable image classification and representation.
The mechanistic analysis of MWT is the main contribution of the paper. Here, authors conduct several experiments, including modifying and analysing INR biases, which they found to be important for the classification. Authors also modify the MWT method in several ways, slightly improving performance over the baseline.
Critical strengths
-
The investigated problem is important for the field. While several approaches to INR classification exist, it is not well understood how these methods work. In MWT, it is unclear how meta-learning organizes classification related information within the INR weights.
-
Authors perform broad experiments, including PCA based analysis and various interventions on INR weights. Obtained results illustrate that MWT organizes the weigths in a way that support classification (but this is not suprising in some sense).
-
Authors improve the original MWT method. On Imagenette (Table 3), the Composite method achieves accuracy of 62.33 compared to 60.65 for the baseline.
Critical weaknesses
-
The interpretability analysis could be more systematic. The applied interventions are mostly handcrafted (e.g., zeroing biases). The paper would benefit from using methods that estimate parameter importance automatically. The authors mention in the Related Work that gradient based attribution methods have already been used to determine the importance of INR parameters for classification [Janik et al., ECCV 2026]. Given that ECCV was in September, close to the ICLR submission deadline, I would not consider the lack of comparison with this paper an issue for this work. However, using some kind of automatic attribution method for INR parameters would help understand whether biases are important for classification without directly targeting them (do they emerge as important?). Another possibility would be to adapt randomized masking methods, such as RISE (Petsiuk et al., BMVC 2018), from image pixels to groups of INR parameters. In my understanding, many methods developed for black-box classifiers or transformers can be applied here.
-
The methodological novelty is limited. The authors apply established methods (probes, regression, PCA, etc.) to the investigated problem. There is no tool that is specific to INR classification. Also, the main results (bias dependence) are less suprising given the FUNCTA method/paper.
-
Authors do not investigate the WT method (no meta-learning).
-
Aside from the interpretability study, authors introduce an MWT modification named Composite. However, the Composite method includes various modifications (extra embeddings and schedulers for some training parameters), which do not feel connected to the interpretability results presented by authors. Also, the modifications feel more like engineering than a fundamental improvement of the baseline. For instance, the feedback coefficient is changed from 0.001 to 0.1 during training. In the MWT paper, the reported ablation results stated that setting this value to 0.001 (vs 0.01) actually decreases performance (Table 4). As of now, it is difficult to understand which modifications are actually responsible for the improvements without extra ablations and explanations.
-
The pixel sampling fractions reported in Table A2 raise questions about whether the baseline reproduces the original CIFAR-10 training procedure. In the original MWT paper, the sampling fraction for CIFAR-10 was set to 100% (no undersampling requirement for low-resolution images). It is a little bit unclear why the authors decided to apply sampling in this case.
-
To better illustrate the usefulness of various parameters, the authors could train an MWT that only tokenizes the biases in INRs. For original MWT, they could limit the gradient flow from the classification branch to the biases only. Also, it would be interesting to use a SIREN without the biases just to check if the MWT can place the classification related infromation in the weights. The interpretation of bias dependence is incomplete.
-
Authors perform various interventions, such as zeroing the biases. Could authors also include some images showing how the interventions change the images reconstructed by the INRs? This would be a nice extra qualitative result. Alternatively, does the modification of the bias changes the reconstruction PSNR? In general, do these modifications actually impact the reconstructed images, or do the biases perhaps serve as a placeholder for information required for classification, which happen to be related to image statistics?
Questions for authors
My initial recommendation is reject.
I read this study as addressing an important problem, but I consider the analysis to be somehow preliminary (in its current form). The main issue is the lack of attribution methods that would automatically highlight biases as important for the classification. Also, the Composite method includes several modifications, which are not clearly motivated/explained in the text.
Given the rebuttal window, to improve the odds of changing the rating, I would suggest:
- Use a gradient based method (or RISE like technique) to determine parameter importance scoresfor the baseline. This should be straitforward given a checkpoint on Imagenette/CIFAR-10. Report the importance distributions for weights vs biases.
- Train a baseline MWT without the biases on Imagenette. Would MWT just place the classification required information in the INR weights? Optionally, you can increase the width of the INR to show that extra parameters do not compensate the lack of the biases.
- Better explain the Composite method.
- Clarify the role of biases in image reconstruction.
Thank you for your review
As the discussion period comes to a close, we would like to thank you again for your thoughtful review, and we hope you might have a chance to look over our response and revision plan. Your point comparing color removal with the leading principal components, along with your suggestions on held-out regression, unified probe splits, and the Composite ablation, helped us clarify the specific role of color in the paper. If any question remains, we would be glad to address it, and we would greatly appreciate your feedback.