acceptodds
Under review as a conference paper at ICLR 2027

MapPerturb: Dimension and Direction in Compact GNN Adaptation

Abstract

Low-dimensional weight maps reduce the trainable parameters of a GNN, but their coordinate count does not determine which prediction changes they can realize. We study how update directions affect the benefit of adding coordinates. MapPerturb parameterizes weight updates around a pretrained anchor through a fixed affine map. We analyze first-order changes in centered logits, separating the dimension of the reachable response space from its alignment with a specified target response. On three citation graphs, gradient directions yield larger reconstruction gains from additional coordinates than random directions at matched weight-update budgets. This gradient-direction advantage persists under direct radius-constrained fitting. Additionally, weight-space retention and held-out response reconstruction also rank the tested maps differently. Low-rank gradient factors retain a reconstruction advantage over spectrum-matched random factors, use – fewer stored basis values than dense maps, and match or exceed LoRA, VeRA, and full encoder tuning on further graphs. In transductive few-shot transfer, directions constructed from DGI or masked-feature reconstruction without task labels improve pooled accuracy over head-only adaptation by and percentage points, respectively, with eight trainable encoder coordinates and a full linear head.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph neural networks, Parameter-efficient fine-tuning, Hypernetworks, Intrinsic dimensionality, Gradient alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_63Q7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

MapPerturb proposes a new method for finetuning neural networks. The paper has a nice figure 1 and provides a novel algorithm for selecting update directions. However, this paper should be rejected because a key claim is not backed up and the results substantially lack analysis.

Critical strengths

The paper has a nice figure 1 and provides a novel algorithm for selecting update directions. Many experiments are run, and extensive theory is provided.

Critical weaknesses

In general, the results list numbers but provide very little analysis. This makes it challenging to understand the relevance of the results. Here are two examples, though many are present in the paper:

  1. One paragraph in section 5.2 reads: “From d = 2 to 16, gradient-basis R2 0 increases by 0.208 ([0.137, 0.278]), random by 0.070 ([0.062, 0.078]), and cyclic decreases by 0.078 (change interval [−0.118, −0.038]). The gradient– cyclic gap therefore widens by 0.286 ([0.194, 0.377]). Gradient bases lead at every tested d (eight Holm-adjusted p ≤ 0.0014).” It is unclear why the gradient-cyclic gap widens, why gradient bases lead, and what the overall significance of these results are.
  2. In table 3, the authors compare 9 methods, but it is unclear why the table is there. What is the relevance of Grad-SVD? It seems to perform best but it’s only introduced in the appendix. The other proposed methods do not seem to perform substantially better than LoRA and VeRA.

Furthermore, one of the paper’s claimed contributions is that their method can “match or exceed LoRA, VeRA, and full tuning.” But it is not clear where this claim is shown. It seems like Grad-SVD beats LoRA in table 3, but it is not clear if this is statistically significant. When LoRA is compared again in appendix table 10, it seems like it outperforms all of the authors’ methods by a much larger margin. Finally, runtime results and some critical statistical significance results are missing, so it is challenging to compare the proposed methods to existing ones.

Questions for authors

What is the relevance of Grad-SVD? It seems to perform best in table 3 but it’s only introduced in the appendix.

How does LoRA compare to the proposed methods? Why does it outperform all of the authors’ methods in table 10?

How do the proposed methods (e.g. Grad-SVD) perform relative to LoRA and VeRA in terms of runtime?

What is the statistical significance of the results in tables 3 and 10?

1 comment

Thank you for taking the time to read and assess our paper, and for your positive comments on Figure 1 and the direction construction. We appreciate the opportunity to clarify the paper's claims and analysis. We have updated the manuscript.

Summary/W1: method and dimension trend. MapPerturb is a parameter-efficient fine-tuning framework that adapts a pretrained GNN through low-dimensional coordinates, with multi-gradient factors as one construction. We removed the abstract's broad “match or exceed” claim. From d=2d=2 to 16, gradient R02R_0^2 rises by 0.208 while cyclic R02R_0^2 falls by 0.078, widening their gap by 0.286 (95% CI [0.194, 0.377]). Gradient directions lead at every tested dimension (eight Holm-adjusted p≤.0014p\le.0014). In a descriptive analysis of 30 saved contexts, fitting-gradient capture (the fraction of an independent fitting gradient's squared norm projected into the span) averages 0.305/0.00463/0.000112 for gradient/cyclic/random. Their fitting cross-entropy decreases by 0.159/0.0296/0.00596, consistent with stronger alignment to task descent. In the Appendix C.6 estimator rerun, the cyclic d=2→16d=2\to16 change is −0.088 with ridge-and-rescaling and +0.080 with direct radius-constrained fitting. The gradient gain remains positive (+0.235). Thus direction and fitting rule must be evaluated together.

W2/Q1: Table 3 and Grad-SVD. Table 3 tests classification across three non-citation graphs and three backbones with a fitted frozen head and common radius. Grad-SVD is a simple supervised control: it applies layerwise SVD to one aggregate training gradient and trains eight shared coefficients over eight fixed directions. It has the highest pooled Table 3 accuracy, exceeding multi-gradient by 0.538 pp (95% CI [0.169, 0.908], Holm p=.0465p=.0465, 15 contrasts). Section 3.1 now defines this control and §5.5 explains its role. Descriptive Grad-SVD−LoRA-r1 means are +1.228/+1.238/+0.061 pp on Chameleon/Roman-empire/Squirrel, with positive means in 8/9 graph–backbone cells. Controls matched on the discovery objective leave the DGI multi−single-SVD difference unresolved, while masked single-SVD exceeds multi by 2.161 pp in the saved cohort and 1.969 pp in a fresh 30-anchor cohort (Holm p=.000155p=.000155 and 2.53×10−52.53\times10^{-5}). Both factors still improve on head-only and matched-random controls in the fresh pipeline (Appendix C.24). The contribution is controlled direction analysis and verified label-free gains, not multi-gradient being the best accuracy method.

Q2: Table 3 vs. Table 10. The protocols differ in graph/backbone, head training, radius, rank, discovery signal, and endpoint. In Table 10, LoRA-r4 exceeds DGI/masked factors by 2.370/2.098 pp, while full tuning exceeds them by 3.453/3.181 pp. A matched 30-anchor study (480 endpoints) varies head and radius from a common support-fitted head (Appendix C.26). Removing the radius improves LoRA-r4 by 7.613/6.140 pp with fixed/joint heads, while factor-map radius effects do not pass correction. With a joint head, LoRA-r4 minus masked changes from +5.620 pp without the radius to −0.433 pp with it (95% CI [−1.240, 0.373]). Radius materially changes relative accuracy, but other differences between the original tables remain.

Q3/Q4: runtime and statistical evidence. On the same 90 Table 3 contexts, including discovery, construction, both adaptation trials, and validation selection, Grad-SVD takes 2.708 s per context, LoRA-r1 2.811 s, VeRA-r16 2.941 s, and multi-gradient 3.528 s. We synchronize the GPU and rotate order across three repeats. Shared pretraining and head/radius setup are excluded for all methods (Appendix C.28.2). Grad-SVD is faster in this protocol. The paired geometric time ratio of multi-gradient to LoRA-r1 is 1.280 (95% CI [1.258, 1.302], Holm p=4.91×10−10p=4.91\times10^{-10}). Peak memory is similar (380.55 MiB for multi-gradient, 379.90 for LoRA-r1).

For Table 3, paired inference averages the nine graph–backbone cells within each of ten split blocks and corrects 15 secondary contrasts. Grad-SVD exceeds LoRA-r1 by 0.842 pp (Holm p=.000546p=.000546) and VeRA-r16 by 1.045 pp (p=.000232p=.000232). Multi-gradient exceeds VeRA-r16 (p=.0100p=.0100), while its LoRA-r1 and full-tuning contrasts do not pass correction (p=.131/.739p=.131/.739). Table 10 uses separate external (15) and core (4) families, averaging shots within 30 graph–anchor units. LoRA-r4/r8 and full tuning exceed both factors after correction, while both factors improve over head-only and matched random. Appendix C.21 reports every contrast and interval. The revised claims follow these paired results and distinguish the two protocols.

Reviewer_TitnOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MapPerturb, a framework for compact GNN adaptation using fixed affine weight maps. It analyzes the first-order centered-logit image, separating the coordinate count, response sensitivity, and alignment with a specified target. Controlled reconstruction experiments compare gradient, random, and cyclic directions at matched weight-update radii. The authors compress gradient atoms into low-rank factors and construct label-free directions using DGI or masked-feature objectives. Few-shot experiments jointly train eight encoder coordinates and a full linear head, with separate studies examining other graphs, backbones, PEFT baselines, and map-selection policies. The strongest evidence concerns the effect of direction choice on target-response reconstruction and the gains over head-only or matched-random adaptation.

Critical strengths

1.A carefully controlled empirical analysis of direction versus dimension. The use of cross-fitting, common weight radii, and spectrum-matched random factors effectively isolates orientation from scale and storage effects. The constrained-fitting rerun is particularly valuable: it reverses the cyclic dimension trend while preserving a gradient-direction advantage, successfully distinguishing an estimator effect from a capacity claim. The finding that weight retention and response reconstruction can rank maps differently is also highly relevant for subspace adaptation.

2.Concrete compact constructions with appropriately reported limitations. The factor maps retain a substantial reconstruction advantage over matched random factors. Both label-free objectives improve pooled few-shot performance over head-only adaptation and their respective random controls. The paper appropriately reports paired uncertainty, clearly distinguishes trainable coordinates from frozen storage, and acknowledges unresolved functional-mixing and selection results. This supports a bounded, well-scoped contribution.

Critical weaknesses

1.The response analysis has not established a reliable bridge to adapter choice or task gains. Much of the theoretical apparatus relies on standard local linear-algebra results, making the empirical implications critical for the paper's significance. Yet, the correlation between reconstruction gains and accuracy gains is approximately -0.19 and -0.16 (with intervals containing zero). Furthermore, the prospective reconstruction-based selector does not resolve an advantage over the fixed development prior. The analysis helps understand a specified reference response, but its added prescriptive value over simpler gradient-informed constructions remains limited.

2.The practical advantage is narrower than the parameter count suggests. In the jointly trained-head comparison, LoRA-r=4/8 and full encoder tuning outperform the proposed factor variants. On Cora, the factor adapter stores 210,364 bytes versus 14,820 for LoRA-r=2, despite training fewer values. The external fixed-head, common-radius comparison addresses a different setting, and the simpler Grad-SVD construction performs better overall than the multi-gradient map in that context. Consequently, the evidence establishes a trainable-state tradeoff rather than a general accuracy or total-resource advantage. The unresolved cross-run anchor differences further limit exact reproducibility, although the within-run pairing checks are reassuring.

Questions for authors

1.Contribution clarification: Given the inconclusive prospective selector and the weak reconstruction–accuracy association, what actionable conclusion does the response analysis support beyond evaluating a specified local target? Please clarify whether any existing results establish an advantage of the multi-gradient construction over simpler Grad-SVD/GiVA-style alternatives under comparable adaptation conditions, or explicitly delimit the contribution to the controlled analysis.

2.Practical-value and reproducibility clarification: In which measured resource regime is the factor map preferable to LoRA when frozen payload, construction cost, and adaptation cost are all counted? Please also clarify which saved anchors and configuration/environment records define the canonical reproducible results, given the unresolved same-seed cross-run differences noted in Appendix C.11. This would materially affect my assessment of the construction’s practical value.

2 comments

Thank you for your careful reading and recognition of the controlled analysis. We appreciate the opportunity to discuss your concerns about task value, resources, and reproducibility. We have updated the manuscript.

W1/Q1: interpretation and simpler constructions. Reconstruction evaluates fidelity to a specified graph response. Section 4 and Appendix B.8 bound loss differences from a fixed reference. Preserving a task improvement requires a useful reference, so reconstruction alone is not an accuracy selector. The submitted reconstruction–accuracy correlations are −0.19/−0.16 with intervals including zero. The selector also has no statistically resolved advantage over the fixed development prior that always selects masked factors: the cross-entropy difference is −0.003751 (95% CI [−0.009880, 0.003399]).

The analysis supports three evaluation rules: match realized update radii, check the fitting rule before making a capacity claim, and rank maps by their graph responses rather than weight retention. At d=8d=8, gradient/cyclic weight retention is 0.0367/0.0792, but response R02R_0^2 is 0.621/0.221. Direct constrained fitting reverses the cyclic decline while preserving the gradient lead.

Task value is established separately. Across 30 saved anchors, DGI/masked factors improve over head-only by 3.270/3.464 pp and over matched random by 3.191/3.377 pp. A fresh 30-anchor, 630-endpoint pipeline reproduces gains of 3.219/3.409 pp over head-only and 3.172/3.369 pp over random (Holm p=5.67×10−6p=5.67\times10^{-6}). These results support the label-free factors, not a general accuracy advantage for multi-gradient. Grad-SVD, a single aggregate-gradient layerwise SVD control, exceeds multi-gradient by 0.538 pp on Table 3 (95% CI [0.169, 0.908], Holm p=.0465p=.0465). With the discovery objective and coordinate budget matched, DGI multi vs. single-SVD is unresolved, while masked single-SVD exceeds multi by 2.161 pp in the saved cohort and 1.969 pp in the fresh cohort (Holm p=.000155p=.000155 and 2.53×10−52.53\times10^{-5}). Thus the added value is the controlled direction–dimension analysis and measured task gains, not the necessity or superiority of combining gradients.

GiVA also exceeds DGI/masked multi by 1.724/2.948 pp under matched discovery signals, using 136 rather than eight encoder variables on Cora (Appendix C.23).

W2/Q2: parameters, resources, and accuracy. We do not establish a regime where the factor map is cheaper overall than LoRA. It trains 463 values on Cora (eight encoder coordinates plus a 455-value head), versus 3,705 for LoRA-r2, but its task-specific payload is 210,364 B versus 14,820 B. In the synchronized Table 3 protocol, multi-gradient takes 3.528 s per context versus 2.811 s for LoRA-r1, with similar peak memory (380.55 vs. 379.90 MiB). The compact-storage claim concerns compression relative to a dense aligned basis, not total-resource superiority.

The accuracy ranking also depends on protocol. Table 3 uses a fitted frozen head and common update radius. Table 10 jointly trains the head without that projection. There, LoRA-r4 exceeds DGI/masked factors by 2.370/2.098 pp, while full tuning exceeds them by 3.453/3.181 pp after correction. In a matched 30-anchor study, removing the radius raises LoRA-r4 by 7.613/6.140 pp with fixed/joint heads, while factor-map radius effects do not pass correction. With a joint head, LoRA-r4 minus masked changes from +5.620 pp without the radius to −0.433 pp with it (95% CI [−1.240, 0.373]). Radius therefore affects relative accuracy, but it does not explain every difference between the original protocols (Appendix C.26).

Q2: reproducibility and anchor differences. A seed alone does not identify an anchor. We define each reported result by its saved anchor, construction state, support split, selected update, configuration, hashes, and environment (Appendix C.30). The archive covers 90 Table 3 contexts, five ogbn-arxiv anchors, 30 saved citation anchors, and 30 fresh anchors. The recorded environment is Python 3.9.13, PyTorch 1.13.0+cu117, CUDA 11.7, PyG 2.5.3, and an RTX 3060 Laptop GPU (6 GB). All predicted classes in the fresh 630-endpoint cohort are reproduced on replay. A separate nine-anchor repeat has positive DGI/masked gains (+4.411/+3.596 pp), but its four primary contrasts do not pass Holm correction (p=.0663p=.0663) and are reported separately.

For two Cora seeds, separate GPU processes had identical initial tensors, RNG states, and DGI permutations, yet gradients differed from the first step. Repeated CPU runs were identical. The primary few-shot study and Table 10 also have different anchor hashes despite the same nominal seeds and supports. We cannot attribute that historical difference to a single cause because the original checkpoints are unavailable. The saved states and environment define the reproducible results, while the fresh pipeline tests whether gains persist under repeated pretraining and discovery.

The link between better reconstruction of reference responses and better task performance remains insufficiently established. Moreover, the additional theoretical validation still does not demonstrate a practically useful guarantee of task improvement.

Reviewer_xY8EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how the dimension and orientation of low-dimensional weight-update subspaces affect GNN adaptation. The authors propose MapPerturb, which represents encoder updates through a fixed affine mapping from a small number of trainable coordinates.

Critical strengths
  1. the paper shows an interesting perspective on parameter-efficient adaptation
  2. Gradient-based maps consistently outperform random and cyclic alternatives in the main reconstruction experiments
  3. Empical study is useful and easy to read
Critical weaknesses
  1. The main theoretical observations largely follow from standard linear algebra and local Taylor approximations. In particular, the rank bound for the graph-conditioned Jacobian, the projection-based reconstruction error, and the quadratic subspace approximation are relatively direct consequences of existing results. Could the author clarify the motivation of the theory

  2. why choose reconstruction improvement instead of downstream performance directly?

  3. The results on ogbn-arxiv are less convincing, with the proposed method performing below LoRA and full fine-tuning.

  4. I am sure this problem is promising for LLM infra or model, but for graph, i think still needs some time. But i will be glad if the author can explain why this work is important for graph community.

Questions for authors

Could you clarify the novelty and compare with recent work like "GiVA: Gradient-Informed Bases for Vector-Based Adaptation" and "Gradient Intrinsic Dimensionality Alignment"

1 comment

Thank you for the time you spent reading our paper and for recognizing its direction-based perspective and reconstruction evidence. We appreciate the opportunity to discuss your concerns and clarify our contribution. We have updated the manuscript.

W1/Q1: theory and contribution. Our theory asks how a trainable coordinate changes a GNN's predictions. For Δθ=βEz\Delta\theta=\beta Ez, the local centered-logit response with a fixed classifier is βC(JGEz)\beta C(J_GE z). Thus equal coordinate counts and update radii need not produce equal graph responses. At d=8d=8, gradient directions retain less reference displacement energy than cyclic directions (0.0367 vs. 0.0792) yet reconstruct the response better (R02=0.621R_0^2=0.621 vs. 0.221). Direct radius-constrained fitting reverses the cyclic dimension decline. Our contribution is a controlled analysis of direction, dimension, fitting rule, and graph-conditioned response, plus label-free factors for few-shot adaptation. Rank bounds and SVD are standard tools, not novelty claims.

W2: reconstruction and task value. Reconstruction measures fidelity to a fixed graph response under a matched head and radius. It is neither an accuracy proxy nor an adapter selector, so we test task accuracy separately. On 30 saved anchors, DGI/masked factors improve over head-only by 3.270/3.464 pp and over matched random by 3.191/3.377 pp. In a fresh 30-anchor pipeline with 630 endpoints fixed before scoring, the gains are 3.219/3.409 pp over head-only and 3.172/3.369 pp over random (Holm p=5.67×10−6p=5.67\times10^{-6}). Controls matched on the discovery objective leave the DGI multi vs. single-SVD difference unresolved. Masked single-SVD exceeds multi-gradient by 2.161 pp in the saved cohort and 1.969 pp in the fresh cohort (Holm p=1.55×10−4p=1.55\times10^{-4} and 2.53×10−52.53\times10^{-5}). Eq. (13) bounds loss change relative to a fixed reference, whose own quality matters. Reconstruction–accuracy correlations (−0.19/−0.16) and the selector's cross-entropy difference from the fixed development prior (−0.003751, 95% CI [−0.009880, 0.003399]) do not validate accuracy selection.

W3: ogbn-arxiv. We agree that the tested task result favors LoRA-r1 and full tuning: 27.89% vs. 30.84%/31.35% on five anchors, using 800 labels and a fitted frozen head, not the standard full-data OGB protocol. Removing the common radius changes multi-gradient by only +0.004 pp (95% CI [−0.003, 0.010], Holm p=.556p=.556). The step-50 direction refresh gives +1.987 pp, with an interval including zero (95% CI [−1.116, 5.090], Holm p=.300p=.300). Initially separate equal-seed prefixes differed by up to 5.16×10−55.16\times10^{-5} in step-50 weight displacement. Rerunning both arms from one exact saved prefix reproduces the accuracy difference and uncertainty. Since this pairing audit followed inspection of the initial test results, it is diagnostic rather than independent confirmation (Appendix C.27.1). Neither test establishes a reliable recovery of the observed gap.

W4: relevance to graph learning. We agree that the infrastructure payoff will take time. We see the path to LLM infrastructure as running through graph learning. Message passing filters centered-logit changes: ΔF=A^XΔW⊤Pc\Delta F=\widehat A X\Delta W^\top P_c, where PcP_c centers classes. On two nodes, mean aggregation annihilates (1,−1)(1,-1), while identity propagation preserves it. The same nonzero update can therefore have different prediction effects across graphs (Appendix B.8). This motivates matching radius, head, and fitting rule while measuring response alongside accuracy. The label-free gains above address graph adaptation when task labels are scarce. Looking ahead, we view graphs as a lightweight structural core for relational multimodal state in continuously adapting agent and LLM systems. Efficient, label-free adaptation could help graph foundation models serve that role, motivating graph-aware measurement before infrastructure demand arrives.

Q1: GiVA and GIDA/RaLoRA. GiVA scales frozen gradient-informed frames, while our factor map mixes 16 gradient atoms with eight shared coordinates. GIDA estimates gradient intrinsic dimension to configure RaLoRA blocks. We reimplemented RaLoRA-r1/r4 and RaLoRA-Pro for this GNN comparison (Appendix C.23). Under a common adaptation protocol on 90 support sets, GiVA with the same discovery objective exceeds DGI/masked multi by 1.724/2.948 pp. The 95% CIs are [0.897, 2.552]/[1.784, 4.111], with Holm p=.00234/.000215p=.00234/.000215. On Cora, GiVA uses 136 encoder variables and multi uses eight. RaLoRA-r1/r4 differences from LoRA at the same rank do not pass correction, while RaLoRA-r4/Pro exceed both factors. RaLoRA-Pro reaches 59.580% pooled accuracy, using 4,018 encoder variables on Cora. Our supported contribution is controlled response analysis and compact representation relative to a dense basis, together with measured label-free gains. We do not claim accuracy superiority over GiVA or RaLoRA.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.