- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes model embeddings (MODEMs) as a representation of fine-tuning task adaptations. Given a pretrained weight matrix , its singular directions are split into blocks, and the update for task is parameterized as , with fixed bases taken from the singular vectors of and small trainable coordinate matrices . The collection of these matrices across layers is the embedding of task , and all tasks share the same coordinate system, with the zero embedding corresponding to the pretrained model. The authors show that the map is linear and distance preserving, so linear operations on embeddings (addition, subtraction, interpolation) are well defined. They then relate differences between embeddings to differences in the model function through a path-averaged Jacobian, and show that under a shared local geometry these operations correspond approximately to combining the functional changes of the tasks. Experiments include a controlled MNIST generation study, where a transformer pretrained on EMNIST letters is adapted to all possible 1023 nonempty digit subsets. Embedding distances are correlated with transfer regret and other task relation metrics and compared with LoRA and full fine-tuning. Addition and subtraction of embeddings are tested, and the embeddings are decomposed into a shared component plus digit-associated directions. The authors also include experiments with CLIP ViT-B/32 on 20 image datasets, where the shared-geometry assumption of Corollary 2 does not generally hold, which motivates the introduction of a path-based task relation and a path-aware merging procedure. Additional experiments in the appendix use GPT-2 on 20 text datasets.
Critical strengths
-
Well written and easy to follow. The idea of a fixed coordinate system tied to the pretrained model, where adaptations for different tasks can be compared and combined directly, is interesting and useful for studying task adaptations. Since the decoder is built only from the pretrained weights, the formulation is agnostic to the specific layer and architecture. This work can open the path to more complex manipulations of the MODEM embedding space.
-
Exhaustive controlled experiment. The controlled MNIST study is well designed. Using all possible nonempty digit subsets gives task relations known by construction, and the authors measure them in three independent ways (Jaccard, MMD, transfer regret), over more than 500k task pairs. MODEM, LoRA and full fine-tuning are trained and evaluated on the same tasks and data. The generated images are evaluated with an independent 47-class classifier that also covers the letters of the pretraining distribution, so generations that remain letter-like are not counted as digits, and both target-digit and other-digit rates are reported (Table 1). The evolution of the embeddings during training is visualized in a single PCA basis fitted on all epochs jointly (Fig. 1, App. B.3). This avoids the artifacts of refitting the projection at each epoch, so movement in the plot reflects real changes of the adapters. In my opinion this is a good curated setting for testing whether the properties discussed in Section 2.2 show up empirically.
Critical weaknesses
-
Limited novelty of the parameterization. The update in Eq. 6 (fixed singular bases from , small trainable matrices between them) is essentially the mechanism of SVFT (Lingam et al., 2024), which is not discussed in the related work. The authors say they do not claim novelty for the arithmetic, so the contribution rests on the representational analysis. In my opinion that analysis is not strong enough on its own (see point 2,3).
-
Evidence that MODEMs are a better representation than plain fine-tuning updates is limited.
- In Figure 2 and Table 6 the advantage over full fine-tuning appears only with Euclidean distance. With cosine distance, full fine-tuning is equal or slightly better on all three relation metrics (e.g. transfer: 0.949/0.970 vs 0.945/0.962). The paper does not explain why the Euclidean behaviour should matter, and it does not check whether the gap is caused by different update norms across fine-tuning methods.
- The operations are not compared with any merging method (e.g. TIES, TSV-Merge, which is cited, or data-dependent methods such as AdaMerging and RegMean). This matters because the paper introduces a new merging procedure (Eq. 21). A comparison in Table 4 would have been interesting. Furthermore, in Table 4 plain MODEM addition is on par with full fine-tuning (77.42 vs 77.54). Plain subtraction is worse at removing task B than both LoRA and full fine-tuning (residual 65.64 vs 51.99 and 38.83).
- In Table 2, the constructed triplet generates the added digit only 15.4% of the time, far from the ~29% obtained by the adapter trained directly on the triplet (28.9%/29.0%/28.9%, same table) and from the ~33% of a uniform triplet. It is also unclear whether the table shows a single example or an average over tasks.
3 Clarity of the real-model experiments and applicability of the method. On real models and datasets the assumptions of the method basically do not hold. The authors themselves say that the shared-geometry condition of Corollary 2 does not generally hold for ViT, and results improve substantially only with the path-based relation and path merging (Tables 3 and 4), which need the stored training trajectories and Jacobians along them. The experimental setting is also hard to follow from the main text. To understand the vision setting I had to read the supplementary. Section 4 only says “20 image-classification tasks”. The datasets, the 10% training subsets, the checkpoint selection and the fact that parameter distances are computed only on the value projections are all in App. C.1, and GPT-2 is only in App. C.2.
Smaller inconsistencies
- The abstract on OpenReview is non-negligibly different from the abstract in the submitted PDF. For example, it promises an additional experiment on constructing held-out task adaptations that is not in the paper.
- Sec. 3.1 states that all methods use 500 MNIST training examples per target digit, while App. B.1 states that all available MNIST training samples of the selected digits are used.
- The header says ICLR 2026.
Questions for authors
-
Why adapt only the value projections? This choice is never motivated, and there is no ablation. The scope also changes between experiments (V only for MNIST and ViT, V plus MLP for GPT-2), and the baselines are not always compared on the same scope: in MNIST, LoRA updates the attention projections (Sec. 3.1, apparently not only V) while MODEM adapts only V; in ViT, Sec. 4 says full fine-tuning updates the whole vision encoder, while App. C.1 says full fine-tuning updates are restricted to the value projections for the parameter-space comparison. It is unclear whether full fine-tuning is trained only on V or the V weights are extracted afterwards. Do the correlations in Table 6 and Table 3 hold if MODEMs are applied to other layers (Q/K, MLP), and if all methods are trained on the same scope? Related to this, how are the block dimension and the set of selected blocks chosen? In the value-projection experiments the blocks coincide with attention heads, so the SVD-based block construction of Section 2.1 is only used for the GPT-2 MLP layers, with a single configuration (four 8×8 blocks in a rank-32 basis; which singular directions are used?). App. B.1 refers to a "primary dense experiment": were sparse block selections tried? An ablation on and on the selected directions would show how sensitive the representation properties are to these choices.
-
How does the method differ from SVFT, beyond the representational point of view?
-
Can you explain why Euclidean correlation is the relevant property, given that cosine distance on full fine-tuning updates already gives equal or higher correlation? Does the Euclidean gap remain if updates are normalized?
-
Can you compare addition, subtraction and path merging with standard merging methods, ideally also in the multi-task setting (more than two tasks)?
6 comments
Rebuttal (Part2)
(W2c) Clarification on Table 2.
A: In Table 2, our primary objective is to investigate whether model embeddings support basic addition and subtraction operations. Therefore, we use direct arithmetic, , without introducing additional weighting coefficients, to isolate the effects of these operations. We additionally evaluate weighted merging in MODEM space to investigate whether task-specific behaviors can be continuously controlled through embedding-space operations.
Specifically, we construct triplet adaptations from a digit-pair embedding and a singleton embedding using where The experimental setting is the same as in Table 2.
We report the average generated frequencies of the corresponding digits (a/b/c).
| Pair + Singleton → Triplet (a / b / c) % | KL ↓ | JSD ↓ | Entropy ↑ | |
|---|---|---|---|---|
| 0.0 | 41.73 / 42.93 / 1.60 | 0.423 | 0.129 | 1.107 |
| 0.1 | 40.53 / 37.53 / 3.93 | 0.303 | 0.090 | 1.217 |
| 0.2 | 38.80 / 30.93 / 8.00 | 0.195 | 0.056 | 1.345 |
| 0.3 | 34.40 / 24.40 / 14.07 | 0.127 | 0.033 | 1.467 |
| 1/3 | 32.60 / 22.20 / 16.93 | 0.111 | 0.028 | 1.504 |
| 0.4 | 27.87 / 18.27 / 23.07 | 0.125 | 0.031 | 1.516 |
| 0.5 | 21.07 / 12.27 / 34.40 | 0.212 | 0.053 | 1.419 |
| 0.6 | 15.33 / 7.73 / 47.73 | 0.367 | 0.097 | 1.164 |
| 0.7 | 10.07 / 4.20 / 61.53 | 0.610 | 0.164 | 0.828 |
| 0.8 | 5.60 / 2.00 / 73.53 | 0.890 | 0.240 | 0.513 |
| 0.9 | 3.13 / 0.80 / 82.20 | 1.137 | 0.309 | 0.284 |
| 1.0 | 1.33 / 0.53 / 89.00 | 1.350 | 0.366 | 0.148 |
| Trained triplet | 28.40 / 27.20 / 27.73 | 0.000 | 0.000 | 1.576 |
The reported metrics are defined as follows:
- Distribution: Average generated digit frequencies across the three examples in the triplet.
- KL / JSD: Mean divergence relative to each corresponding trained triplet. KL uses .
- Entropy: Mean conditional Shannon entropy across the three target digits, normalized within those digits.
- All information-theoretic metrics use base-2 logarithms (bits).
Key observation: The theoretically motivated coefficient achieves the lowest KL divergence (0.111) and JSD (0.028), while achieves the highest conditional entropy (1.516). These results suggest that appropriately weighted combinations of model embeddings can produce distributions that more closely align with the target distribution.
Rebuttal (Part3)
(W3, Q1) Clarify the experimental setting. In our ViT experiment, we follow the setting as stated in the main paper: Full fine-tuning updates the vision encoder and visual projection while keeping the text encoder fixed. LoRA learns low-rank updates to the attention block of the vision transformer. MODEM adapts the value projections of the attention block.
Why MODEM adapts only the value projections. We treat the multi-head attention (MHA) block as a whole, whose structure naturally aligns with our proposed SVD-based decomposition. Specifically, the SVD-based parameterization takes the form , while the MHA-based parameterization is expressed as . Both share the same structural form, with a trainable core matrix between two fixed bases.
In the SVD-based formulation, and are fixed singular-vector bases, whereas in the MHA-based formulation, and are fixed projections inherited from the pretrained model. Therefore, although we describe this implementation as adapting the value projection for simplicity, the trainable core effectively modifies the entire MHA block rather than independently fine-tuning the value-projection weights. We will clarify this distinction in the revised manuscript to avoid ambiguity regarding the adaptation scope.
LoRA and Full. Since MODEM adapts the MHA block as a whole, we align the adaptation scope of the baselines accordingly: Full fine-tuning updates the entire attention block, while LoRA is applied to all four attention projections (Q/K/V/O), ensuring a consistent comparison of adaptation scope.
Additional ablation. To evaluate whether MODEM generalizes to other model components, we independently apply it to the query (Q), key (K), and two MLP projections (FC1 and FC2) under the same experimental setting. For these projections, we treat each as an individual linear layer and construct the fixed adaptation bases from the SVD of its pretrained weight matrix. This differs from our value-projection (V) implementation, where the fixed bases are derived from the pretrained multi-head attention block rather than the value-projection weights alone.
Adaptation results
| MHA (v) | Q | K | FC1 | FC2 | |
|---|---|---|---|---|---|
| Acc. | 83.46 | 82.21 | 81.95 | 83.75 | 83.69 |
All settings use the same trainable-parameter budget of 0.59M. The results demonstrate that MODEM achieves competitive adaptation performance across different attention and MLP projections, highlighting its flexibility across model components.
Do the correlations hold if MODEMs are applied to other layers (Q/K, MLP)? We further evaluate the correlation between parameter-space distances of the adapters trained under the aforementioned settings and task-space relationships measured by transfer regret.
| Method | Pearson | Spearman |
|---|---|---|
| MODEM (Q) | 0.531 | 0.550 |
| MODEM (K) | 0.524 | 0.535 |
| MODEM (FC1) | 0.473 | 0.609 |
| MODEM (FC2) | 0.453 | 0.545 |
| MODEM (V) | 0.443 | 0.494 |
+ Path-aware
| Method | Pearson | Spearman |
|---|---|---|
| MODEM (Q) | 0.781 | 0.717 |
| MODEM (K) | 0.705 | 0.633 |
| MODEM (FC1) | 0.559 | 0.680 |
| MODEM (FC2) | 0.627 | 0.596 |
| MODEM (V) | 0.774 | 0.766 |
As shown in the table above, MODEM achieves comparable task–parameter correlation results across different model components under both evaluation settings. Furthermore, incorporating path-aware analysis consistently improves these correlations, demonstrating that the observed representational properties extend beyond the value projection.
Apply path-aware to LoRA and Full. To isolate the contribution of the representation from that of the path-aware operation, we additionally applied the same procedure to LoRA and Full under the same evaluation settings as Section 4.
Path-aware correlation
| Method | Pearson | Spearman |
|---|---|---|
| LoRA | 0.34 | 0.42 |
| LoRA + Path | 0.38 | 0.44 |
| Full | -0.43 | -0.46 |
| Full + Path | 0.41 | 0.39 |
| MODEM | 0.44 | 0.49 |
| MODEM + Path | 0.77 | 0.77 |
The results confirm that incorporating adaptation trajectories improves task-relationship estimation beyond MODEM. This supports the general applicability of the path-aware formulation. Importantly, MODEM achieves the strongest correlations both with and without path information,
Path-aware merging
| Method | Addition | Path-aware | Computational Cost | Path Overhead |
|---|---|---|---|---|
| LoRA | 75.06% | 78.68% | 0.603 PFLOP | 2.64 GB |
| Full | 77.54% | 80.09% | 0.647 PFLOP | 2.64 GB |
| MODEM | 77.42% | 80.19% | 0.603 PFLOP | 0.22 GB |
Path-aware merging achieves 78.68% accuracy for LoRA and 80.09% for Full fine-tuning, demonstrating that the mechanism is broadly applicable beyond MODEM. However, applying path-aware merging to these methods can incur substantially greater storage overhead, as their parameter-update checkpoints are considerably larger than MODEM embeddings.
Rebuttal (Part1)
We thank the reviewer for their constructive feedback. We address the comments below and have revised the manuscript accordingly.
(W1, Q2) Comparison with SVFT.
A: SVFT primarily investigates parameter-efficient fine-tuning by learning selected coefficients in the pretrained spectral basis, using sparsity patterns such as diagonal, banded, random, and Top-k. In contrast, MODEM is motivated by a different research question: Can task adaptations be represented and studied within a compact, structured model space, rather than the increasingly high-dimensional parameter space of large pretrained models?
Beyond the representational perspective, MODEM's blockwise structure also enables substantially more efficient computation. Unlike SVFT's general sparse coefficient matrix, MODEM exploits the correspondence between spectral blocks and the pretrained model's architectural components, enabling efficient reuse of intermediate projections and small blockwise matrix multiplications.
To quantify this advantage, we additionally compare MODEM with SVFT Top-k under an identical trainable-parameter budget:
Both methods achieve nearly identical adaptation accuracy. We compare the additional MACs required to evaluate the adaptation branches. While SVFT uses full-dimensional spectral projections, MODEM organizes its computation into headwise spectral blocks. This structure can reduce additional computation while maintaining comparable adaptation accuracy.
Most importantly, MODEM goes beyond parameter-efficient adaptation by investigating the representational properties of independently learned task adaptations within a shared, structured coordinate space. Unlike SVFT's primary focus on fine-tuning efficiency, MODEM examines embedding stability, task-related geometry, functional component interventions, and model arithmetic, including addition, subtraction, and pathwise operations, to understand and manipulate task-specific adaptations.
(W2a, Q3) Why is Euclidean correlation the relevant property?
A. We examine both cosine and Euclidean distances because they capture complementary aspects of the representation geometry.
Cosine similarity characterizes the angular relationships between task adaptations, independently of their magnitudes, whereas Euclidean distance captures both directional and magnitude differences. The observation that full fine-tuning performs comparably under cosine distance but differs under Euclidean distance suggests that the representations capture similar directional task relationships while exhibiting different magnitude-dependent geometry.
Our findings indicate that MODEM preserves meaningful task relationships within a compact, structured parameter space, despite operating with substantially fewer trainable parameters than full fine-tuning. This supports the representational utility of MODEM without requiring dominance across every geometric metric.
We also include the normalized Euclidean distance here for reference.
Notably, L2-normalized Euclidean distance is a monotonic transformation of cosine distance, dnorm=2dcos, so both capture the same angular geometry and produce near-identical pairwise rankings.
(W2b, Q4) Compare with standard merging methods.
A. To provide a more comprehensive evaluation, we additionally compare MODEM against established merging methods, including TIES, TSV-Merge, AdaMerging, and RegMean, under the same experimental setting as in Section 4.
MODEM achieves 77.42% mean accuracy, comparable to TIES and AdaMerging, although below RegMean. Incorporating path-aware merging improves accuracy to 80.19%, outperforming all evaluated baselines, including RegMean.