- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper considers the case of meta-networks that take as input model weights with the goal of predicting some property of the model. In this case its main argument and contribution is that weight symmetries (i.e. symmetries that preserve the model’s function) and task symmetries (i.e. the subset of weight symmetries that preserve the quantity to predict) should be treated separately. Specifically, the authors argue and show that a meta-network should be invariant exactly to task symmetries (and not necessarily to all weight symmetries). Otherwise, i.e., if it is invariant to a larger set of symmetries, there is an irreducible error.
Critical strengths
Positive points:
- The argument that weight symmetries and task symmetries should be distinguished, although kind of obvious in retrospect, is indeed a useful observation. The authors also quantify the irreducible error that occurs when a larger set of invariances is enforced.
- They prove the strict LoRA ladder and show the optimizer dependent symmetry in LoRA
Critical weaknesses
Negative points:
- Concerning the theorems in section 3, they do not bring any really new theoretical results. The new thing here is the framing of how they are used (i.e. in the context of meta-networks that take as input network weights and predict network properties). The same applies to the optimizer equivariance proposition 5.1. The theoretical novelty is thus mainly concentrated on the LoRA results.
- The synthetic zoo LoRA experiments use very small weight matrices. Moreover, each LoRA adapter is randomly reparameterized, whereas in practice the initialization of the second LoRA matrix is typically zero, which, to a large extent, constrains the parameterization of the LoRA adapters in conjunction with training dynamics. This raises the question of how realistically these experiments reflect the conditions encountered in the case of large scale models. The authors need to show results on a real zoo of LoRA adapters to convincingly demonstrate how important the issue actually is in practice (i.e. a real zoo + meta-model experiment). Note that this is not shown by the reparameterization tests reported in section 7.4.
Questions for authors
The authors should address the issues raised about experimental evaluation and practical importance.
2 comments
Response to Reviewer 54DG (2/2)
Weakness 1
We agree that the results in Section 3 use standard tools, namely projection onto invariant functions and orbit averaging. The revised paper now presents Section 3 as the framework. It credits these tools at the start of the section. What we claim there is a new use of these results in weight-space learning. They give a criterion that can be computed and that selects the architecture. The ceiling can be estimated on a zoo before any meta-model is trained. It also grows monotonically as the group gets larger. Reviewer 1kyk points to this as a strength, and we now test it on real checkpoints. There the ceilings are exact wherever predicted, and they rank the targets by how much they need non-invariant information (App. D).
The technical results beyond Section 3 are the following.
- The strict LoRA ladder . It finds the largest subgroup that survives for each class of targets (Thm 4.3).
- The curvature ladder (Prop A.3).
- The result that the optimizer selects the symmetry. It links the init law and the update rule to the invariance of the zoo (Sec 5, Remark 5.3).
- The counting result for permutations. Permutation-invariant linear order- targets are only a fraction of all such targets. For symmetric tensors the fraction is at most . This shows how much information permutation quotienting removes for reference targets (Thm A.5, Sec 6).
We have rewritten the contributions in the introduction in these terms. The paper also has two new practical parts. One is a procedure for choosing or adapting the symmetry of a metanetwork (Procedure 1). The other extends the criterion and Thm 3.3 to equivariant weight-to-weight outputs (App. C).
Response to Reviewer 54DG (1/2)
We thank the reviewer for the careful reading of our paper and their valuable comments. Here's our response:
Thanks for your suggestion. We ran exactly the experiment the reviewer asked for in the revised version. The new results are in Sec 7.4 (Table 1) and App. D of the paper.
Why B=0 does not make the problem go away. In fact, standard training is a direct case of the theory.
We agree that the reparameterization tests in the old Sec 7.4 (now Sec 7.5) do not involve a learned predictor. They now serve only as supporting evidence. The new real-zoo experiment is the main evidence.