- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
MapPerturb proposes a new method for finetuning neural networks. The paper has a nice figure 1 and provides a novel algorithm for selecting update directions. However, this paper should be rejected because a key claim is not backed up and the results substantially lack analysis.
Critical strengths
The paper has a nice figure 1 and provides a novel algorithm for selecting update directions. Many experiments are run, and extensive theory is provided.
Critical weaknesses
In general, the results list numbers but provide very little analysis. This makes it challenging to understand the relevance of the results. Here are two examples, though many are present in the paper:
- One paragraph in section 5.2 reads: “From d = 2 to 16, gradient-basis R2 0 increases by 0.208 ([0.137, 0.278]), random by 0.070 ([0.062, 0.078]), and cyclic decreases by 0.078 (change interval [−0.118, −0.038]). The gradient– cyclic gap therefore widens by 0.286 ([0.194, 0.377]). Gradient bases lead at every tested d (eight Holm-adjusted p ≤ 0.0014).” It is unclear why the gradient-cyclic gap widens, why gradient bases lead, and what the overall significance of these results are.
- In table 3, the authors compare 9 methods, but it is unclear why the table is there. What is the relevance of Grad-SVD? It seems to perform best but it’s only introduced in the appendix. The other proposed methods do not seem to perform substantially better than LoRA and VeRA.
Furthermore, one of the paper’s claimed contributions is that their method can “match or exceed LoRA, VeRA, and full tuning.” But it is not clear where this claim is shown. It seems like Grad-SVD beats LoRA in table 3, but it is not clear if this is statistically significant. When LoRA is compared again in appendix table 10, it seems like it outperforms all of the authors’ methods by a much larger margin. Finally, runtime results and some critical statistical significance results are missing, so it is challenging to compare the proposed methods to existing ones.
Questions for authors
What is the relevance of Grad-SVD? It seems to perform best in table 3 but it’s only introduced in the appendix.
How does LoRA compare to the proposed methods? Why does it outperform all of the authors’ methods in table 10?
How do the proposed methods (e.g. Grad-SVD) perform relative to LoRA and VeRA in terms of runtime?
What is the statistical significance of the results in tables 3 and 10?
Thank you for taking the time to read and assess our paper, and for your positive comments on Figure 1 and the direction construction. We appreciate the opportunity to clarify the paper's claims and analysis. We have updated the manuscript.
Summary/W1: method and dimension trend. MapPerturb is a parameter-efficient fine-tuning framework that adapts a pretrained GNN through low-dimensional coordinates, with multi-gradient factors as one construction. We removed the abstract's broad “match or exceed” claim. From d=2 to 16, gradient R02 rises by 0.208 while cyclic R02 falls by 0.078, widening their gap by 0.286 (95% CI [0.194, 0.377]). Gradient directions lead at every tested dimension (eight Holm-adjusted p≤.0014). In a descriptive analysis of 30 saved contexts, fitting-gradient capture (the fraction of an independent fitting gradient's squared norm projected into the span) averages 0.305/0.00463/0.000112 for gradient/cyclic/random. Their fitting cross-entropy decreases by 0.159/0.0296/0.00596, consistent with stronger alignment to task descent. In the Appendix C.6 estimator rerun, the cyclic d=2→16 change is −0.088 with ridge-and-rescaling and +0.080 with direct radius-constrained fitting. The gradient gain remains positive (+0.235). Thus direction and fitting rule must be evaluated together.
W2/Q1: Table 3 and Grad-SVD. Table 3 tests classification across three non-citation graphs and three backbones with a fitted frozen head and common radius. Grad-SVD is a simple supervised control: it applies layerwise SVD to one aggregate training gradient and trains eight shared coefficients over eight fixed directions. It has the highest pooled Table 3 accuracy, exceeding multi-gradient by 0.538 pp (95% CI [0.169, 0.908], Holm p=.0465, 15 contrasts). Section 3.1 now defines this control and §5.5 explains its role. Descriptive Grad-SVD−LoRA-r1 means are +1.228/+1.238/+0.061 pp on Chameleon/Roman-empire/Squirrel, with positive means in 8/9 graph–backbone cells. Controls matched on the discovery objective leave the DGI multi−single-SVD difference unresolved, while masked single-SVD exceeds multi by 2.161 pp in the saved cohort and 1.969 pp in a fresh 30-anchor cohort (Holm p=.000155 and 2.53×10−5). Both factors still improve on head-only and matched-random controls in the fresh pipeline (Appendix C.24). The contribution is controlled direction analysis and verified label-free gains, not multi-gradient being the best accuracy method.
Q2: Table 3 vs. Table 10. The protocols differ in graph/backbone, head training, radius, rank, discovery signal, and endpoint. In Table 10, LoRA-r4 exceeds DGI/masked factors by 2.370/2.098 pp, while full tuning exceeds them by 3.453/3.181 pp. A matched 30-anchor study (480 endpoints) varies head and radius from a common support-fitted head (Appendix C.26). Removing the radius improves LoRA-r4 by 7.613/6.140 pp with fixed/joint heads, while factor-map radius effects do not pass correction. With a joint head, LoRA-r4 minus masked changes from +5.620 pp without the radius to −0.433 pp with it (95% CI [−1.240, 0.373]). Radius materially changes relative accuracy, but other differences between the original tables remain.
Q3/Q4: runtime and statistical evidence. On the same 90 Table 3 contexts, including discovery, construction, both adaptation trials, and validation selection, Grad-SVD takes 2.708 s per context, LoRA-r1 2.811 s, VeRA-r16 2.941 s, and multi-gradient 3.528 s. We synchronize the GPU and rotate order across three repeats. Shared pretraining and head/radius setup are excluded for all methods (Appendix C.28.2). Grad-SVD is faster in this protocol. The paired geometric time ratio of multi-gradient to LoRA-r1 is 1.280 (95% CI [1.258, 1.302], Holm p=4.91×10−10). Peak memory is similar (380.55 MiB for multi-gradient, 379.90 for LoRA-r1).
For Table 3, paired inference averages the nine graph–backbone cells within each of ten split blocks and corrects 15 secondary contrasts. Grad-SVD exceeds LoRA-r1 by 0.842 pp (Holm p=.000546) and VeRA-r16 by 1.045 pp (p=.000232). Multi-gradient exceeds VeRA-r16 (p=.0100), while its LoRA-r1 and full-tuning contrasts do not pass correction (p=.131/.739). Table 10 uses separate external (15) and core (4) families, averaging shots within 30 graph–anchor units. LoRA-r4/r8 and full tuning exceed both factors after correction, while both factors improve over head-only and matched random. Appendix C.21 reports every contrast and interval. The revised claims follow these paired results and distinguish the two protocols.