acceptodds
Under review as a conference paper at ICLR 2027

Structured Model Embeddings as Representations of Task Adaptations

Abstract

Fine-tuning a shared pretrained model produces a collection of task-specific parameter updates. Beyond serving as independent adapters, these updates offer a way to represent learned task behavior. A useful representation must do more than store parameters: it should retain adaptation capacity, expose meaningful task relationships, and support interpretable changes in behavior. We study this problem through a structured model embedding space whose coordinates are defined by the pretrained model. Each embedding specifies a parameter update through fixed bases, providing both a compact description of an adaptation and a direct means of executing it. We organize the study around three properties: adaptation fidelity, relational geometry, and operational usefulness. We establish exact properties of the coordinate system and conditional bounds connecting embedding changes to functional changes and task arithmetic. A controlled MNIST generation study is designed to examine these properties with explicitly specified task relations, followed by evaluations using ViT on image datasets and GPT-2 on NLP datasets. We further investigate the construction of held-out task adaptations from existing embeddings as a test of the representation's practical scope.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
model embedding, adaptation, model relations

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_8ntmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes model embeddings (MODEMs) as a representation of fine-tuning task adaptations. Given a pretrained weight matrix W0=UΣV⊤W_0 = U\Sigma V^\top, its singular directions are split into blocks, and the update for task tt is parameterized as ΔWt=∑hU(h)Mt(h)V(h)⊤\Delta W_t = \sum_h U^{(h)} M_t^{(h)} V^{(h)\top}, with fixed bases U(h),V(h)U^{(h)}, V^{(h)} taken from the singular vectors of W0W_0 and small trainable coordinate matrices Mt(h)∈Rrh×rhM_t^{(h)} \in \mathbb R^{r_h \times r_h}. The collection of these matrices across layers is the embedding of task tt, and all tasks share the same coordinate system, with the zero embedding corresponding to the pretrained model. The authors show that the map is linear and distance preserving, so linear operations on embeddings (addition, subtraction, interpolation) are well defined. They then relate differences between embeddings to differences in the model function through a path-averaged Jacobian, and show that under a shared local geometry these operations correspond approximately to combining the functional changes of the tasks. Experiments include a controlled MNIST generation study, where a transformer pretrained on EMNIST letters is adapted to all possible 1023 nonempty digit subsets. Embedding distances are correlated with transfer regret and other task relation metrics and compared with LoRA and full fine-tuning. Addition and subtraction of embeddings are tested, and the embeddings are decomposed into a shared component plus digit-associated directions. The authors also include experiments with CLIP ViT-B/32 on 20 image datasets, where the shared-geometry assumption of Corollary 2 does not generally hold, which motivates the introduction of a path-based task relation and a path-aware merging procedure. Additional experiments in the appendix use GPT-2 on 20 text datasets.

Critical strengths
  1. Well written and easy to follow. The idea of a fixed coordinate system tied to the pretrained model, where adaptations for different tasks can be compared and combined directly, is interesting and useful for studying task adaptations. Since the decoder is built only from the pretrained weights, the formulation is agnostic to the specific layer and architecture. This work can open the path to more complex manipulations of the MODEM embedding space.

  2. Exhaustive controlled experiment. The controlled MNIST study is well designed. Using all possible nonempty digit subsets gives task relations known by construction, and the authors measure them in three independent ways (Jaccard, MMD, transfer regret), over more than 500k task pairs. MODEM, LoRA and full fine-tuning are trained and evaluated on the same tasks and data. The generated images are evaluated with an independent 47-class classifier that also covers the letters of the pretraining distribution, so generations that remain letter-like are not counted as digits, and both target-digit and other-digit rates are reported (Table 1). The evolution of the embeddings during training is visualized in a single PCA basis fitted on all epochs jointly (Fig. 1, App. B.3). This avoids the artifacts of refitting the projection at each epoch, so movement in the plot reflects real changes of the adapters. In my opinion this is a good curated setting for testing whether the properties discussed in Section 2.2 show up empirically.

Critical weaknesses
  1. Limited novelty of the parameterization. The update in Eq. 6 (fixed singular bases from W0W_0, small trainable matrices between them) is essentially the mechanism of SVFT (Lingam et al., 2024), which is not discussed in the related work. The authors say they do not claim novelty for the arithmetic, so the contribution rests on the representational analysis. In my opinion that analysis is not strong enough on its own (see point 2,3).

  2. Evidence that MODEMs are a better representation than plain fine-tuning updates is limited.

    • In Figure 2 and Table 6 the advantage over full fine-tuning appears only with Euclidean distance. With cosine distance, full fine-tuning is equal or slightly better on all three relation metrics (e.g. transfer: 0.949/0.970 vs 0.945/0.962). The paper does not explain why the Euclidean behaviour should matter, and it does not check whether the gap is caused by different update norms across fine-tuning methods.
    • The operations are not compared with any merging method (e.g. TIES, TSV-Merge, which is cited, or data-dependent methods such as AdaMerging and RegMean). This matters because the paper introduces a new merging procedure (Eq. 21). A comparison in Table 4 would have been interesting. Furthermore, in Table 4 plain MODEM addition is on par with full fine-tuning (77.42 vs 77.54). Plain subtraction is worse at removing task B than both LoRA and full fine-tuning (residual 65.64 vs 51.99 and 38.83).
    • In Table 2, the constructed triplet generates the added digit only 15.4% of the time, far from the ~29% obtained by the adapter trained directly on the triplet (28.9%/29.0%/28.9%, same table) and from the ~33% of a uniform triplet. It is also unclear whether the table shows a single example or an average over tasks.

3 Clarity of the real-model experiments and applicability of the method. On real models and datasets the assumptions of the method basically do not hold. The authors themselves say that the shared-geometry condition of Corollary 2 does not generally hold for ViT, and results improve substantially only with the path-based relation and path merging (Tables 3 and 4), which need the stored training trajectories and Jacobians along them. The experimental setting is also hard to follow from the main text. To understand the vision setting I had to read the supplementary. Section 4 only says “20 image-classification tasks”. The datasets, the 10% training subsets, the checkpoint selection and the fact that parameter distances are computed only on the value projections are all in App. C.1, and GPT-2 is only in App. C.2.

Smaller inconsistencies

  • The abstract on OpenReview is non-negligibly different from the abstract in the submitted PDF. For example, it promises an additional experiment on constructing held-out task adaptations that is not in the paper.
  • Sec. 3.1 states that all methods use 500 MNIST training examples per target digit, while App. B.1 states that all available MNIST training samples of the selected digits are used.
  • The header says ICLR 2026.
Questions for authors
  1. Why adapt only the value projections? This choice is never motivated, and there is no ablation. The scope also changes between experiments (V only for MNIST and ViT, V plus MLP for GPT-2), and the baselines are not always compared on the same scope: in MNIST, LoRA updates the attention projections (Sec. 3.1, apparently not only V) while MODEM adapts only V; in ViT, Sec. 4 says full fine-tuning updates the whole vision encoder, while App. C.1 says full fine-tuning updates are restricted to the value projections for the parameter-space comparison. It is unclear whether full fine-tuning is trained only on V or the V weights are extracted afterwards. Do the correlations in Table 6 and Table 3 hold if MODEMs are applied to other layers (Q/K, MLP), and if all methods are trained on the same scope? Related to this, how are the block dimension rhr_h and the set of selected blocks Ht\mathcal H_t chosen? In the value-projection experiments the blocks coincide with attention heads, so the SVD-based block construction of Section 2.1 is only used for the GPT-2 MLP layers, with a single configuration (four 8×8 blocks in a rank-32 basis; which singular directions are used?). App. B.1 refers to a "primary dense experiment": were sparse block selections tried? An ablation on rhr_h and on the selected directions would show how sensitive the representation properties are to these choices.

  2. How does the method differ from SVFT, beyond the representational point of view?

  3. Can you explain why Euclidean correlation is the relevant property, given that cosine distance on full fine-tuning updates already gives equal or higher correlation? Does the Euclidean gap remain if updates are normalized?

  4. Can you compare addition, subtraction and path merging with standard merging methods, ideally also in the multi-task setting (more than two tasks)?

6 comments
AuthorsOpenReview ↗

Rebuttal (Part1)

We thank the reviewer for their constructive feedback. We address the comments below and have revised the manuscript accordingly.

(W1, Q2) Comparison with SVFT.

A: SVFT primarily investigates parameter-efficient fine-tuning by learning selected coefficients in the pretrained spectral basis, using sparsity patterns such as diagonal, banded, random, and Top-k. In contrast, MODEM is motivated by a different research question: Can task adaptations be represented and studied within a compact, structured model space, rather than the increasingly high-dimensional parameter space of large pretrained models?

Beyond the representational perspective, MODEM's blockwise structure also enables substantially more efficient computation. Unlike SVFT's general sparse coefficient matrix, MODEM exploits the correspondence between spectral blocks and the pretrained model's architectural components, enabling efficient reuse of intermediate projections and small blockwise matrix multiplications.

To quantify this advantage, we additionally compare MODEM with SVFT Top-k under an identical trainable-parameter budget:

MethodAcc.Params.extra MACs
SVFT83.41% ± 0.15589,82414.746M
MODEM83.46% ± 0.07589,8241.769M

Both methods achieve nearly identical adaptation accuracy. We compare the additional MACs required to evaluate the adaptation branches. While SVFT uses full-dimensional spectral projections, MODEM organizes its computation into headwise spectral blocks. This structure can reduce additional computation while maintaining comparable adaptation accuracy.

Most importantly, MODEM goes beyond parameter-efficient adaptation by investigating the representational properties of independently learned task adaptations within a shared, structured coordinate space. Unlike SVFT's primary focus on fine-tuning efficiency, MODEM examines embedding stability, task-related geometry, functional component interventions, and model arithmetic, including addition, subtraction, and pathwise operations, to understand and manipulate task-specific adaptations.

(W2a, Q3) Why is Euclidean correlation the relevant property?

A. We examine both cosine and Euclidean distances because they capture complementary aspects of the representation geometry.

Cosine similarity characterizes the angular relationships between task adaptations, independently of their magnitudes, whereas Euclidean distance captures both directional and magnitude differences. The observation that full fine-tuning performs comparably under cosine distance but differs under Euclidean distance suggests that the representations capture similar directional task relationships while exhibiting different magnitude-dependent geometry.

Our findings indicate that MODEM preserves meaningful task relationships within a compact, structured parameter space, despite operating with substantially fewer trainable parameters than full fine-tuning. This supports the representational utility of MODEM without requiring dominance across every geometric metric.

We also include the normalized Euclidean distance here for reference.

MethodDistanceJaccardMMDTransfer regret
MODEMFrobenius0.873 / 0.8650.668 / 0.7260.741 / 0.826
MODEML2-normalized Frobenius0.942 / 0.9400.902 / 0.9240.923 / 0.965
LoRAEuclidean−0.178 / −0.194−0.490 / −0.440−0.372 / −0.315
LoRAL2-normalized Euclidean0.871 / 0.8940.795 / 0.8670.825 / 0.921
FullEuclidean0.655 / 0.6420.359 / 0.4270.481 / 0.569
FullL2-normalized Euclidean0.943 / 0.9410.901 / 0.9230.923 / 0.964

Notably, L2-normalized Euclidean distance is a monotonic transformation of cosine distance, dnorm=2dcosd_{\mathrm{norm}}=\sqrt{2d_{\mathrm{cos}}}, so both capture the same angular geometry and produce near-identical pairwise rankings.

(W2b, Q4) Compare with standard merging methods.

A. To provide a more comprehensive evaluation, we additionally compare MODEM against established merging methods, including TIES, TSV-Merge, AdaMerging, and RegMean, under the same experimental setting as in Section 4.

MethodAcc.
TIES77.55%
TSV-Merge75.05%
AdaMerging77.69%
RegMean79.80%
MODEM77.42%
MODEM + Path80.19%

MODEM achieves 77.42% mean accuracy, comparable to TIES and AdaMerging, although below RegMean. Incorporating path-aware merging improves accuracy to 80.19%, outperforming all evaluated baselines, including RegMean.

AuthorsOpenReview ↗

Rebuttal (Part2)

(W2c) Clarification on Table 2.

A: In Table 2, our primary objective is to investigate whether model embeddings support basic addition and subtraction operations. Therefore, we use direct arithmetic, mab+mcm_{ab} + m_{c}, without introducing additional weighting coefficients, to isolate the effects of these operations. We additionally evaluate weighted merging in MODEM space to investigate whether task-specific behaviors can be continuously controlled through embedding-space operations.

Specifically, we construct triplet adaptations from a digit-pair embedding mab\mathbf m_{ab} and a singleton embedding mc\mathbf m_c using m(λ)=(1−λ)mab+λmc,\mathbf m(\lambda)=(1-\lambda)\mathbf m_{ab}+\lambda\mathbf m_c, where λ∈[0,1].\lambda\in[0,1]. The experimental setting is the same as in Table 2.

We report the average generated frequencies of the corresponding digits (a/b/c).

λ\lambdaPair + Singleton → Triplet (a / b / c) %KL ↓JSD ↓Entropy ↑
0.041.73 / 42.93 / 1.600.4230.1291.107
0.140.53 / 37.53 / 3.930.3030.0901.217
0.238.80 / 30.93 / 8.000.1950.0561.345
0.334.40 / 24.40 / 14.070.1270.0331.467
1/332.60 / 22.20 / 16.930.1110.0281.504
0.427.87 / 18.27 / 23.070.1250.0311.516
0.521.07 / 12.27 / 34.400.2120.0531.419
0.615.33 / 7.73 / 47.730.3670.0971.164
0.710.07 / 4.20 / 61.530.6100.1640.828
0.85.60 / 2.00 / 73.530.8900.2400.513
0.93.13 / 0.80 / 82.201.1370.3090.284
1.01.33 / 0.53 / 89.001.3500.3660.148
Trained triplet28.40 / 27.20 / 27.730.0000.0001.576

The reported metrics are defined as follows:

  • Distribution: Average generated digit frequencies across the three examples in the triplet.
  • KL / JSD: Mean divergence relative to each corresponding trained triplet. KL uses DKL(Pmerged∣Ptrained)D_{\mathrm{KL}}(P_{\mathrm{merged}}\\|P_{\mathrm{trained}}).
  • Entropy: Mean conditional Shannon entropy across the three target digits, normalized within those digits.
  • All information-theoretic metrics use base-2 logarithms (bits).

Key observation: The theoretically motivated coefficient λ=1/3\lambda=1/3 achieves the lowest KL divergence (0.111) and JSD (0.028), while λ=0.4\lambda=0.4 achieves the highest conditional entropy (1.516). These results suggest that appropriately weighted combinations of model embeddings can produce distributions that more closely align with the target distribution.

AuthorsOpenReview ↗

Rebuttal (Part3)

(W3, Q1) Clarify the experimental setting. In our ViT experiment, we follow the setting as stated in the main paper: Full fine-tuning updates the vision encoder and visual projection while keeping the text encoder fixed. LoRA learns low-rank updates to the attention block of the vision transformer. MODEM adapts the value projections of the attention block.

Why MODEM adapts only the value projections. We treat the multi-head attention (MHA) block as a whole, whose structure naturally aligns with our proposed SVD-based decomposition. Specifically, the SVD-based parameterization takes the form ∑hU(h)Mi(h)V(h)⊤\sum_h U^{(h)}M_i^{(h)}V^{(h)\top}, while the MHA-based parameterization is expressed as ∑hWv(h)Mi(h)Wo(h)⊤\sum_h W_v^{(h)}M_i^{(h)}W_o^{(h)\top}. Both share the same structural form, with a trainable core matrix Mi(h)M_i^{(h)} between two fixed bases.

In the SVD-based formulation, U(h)U^{(h)} and V(h)V^{(h)} are fixed singular-vector bases, whereas in the MHA-based formulation, Wv(h)W_v^{(h)} and Wo(h)W_o^{(h)} are fixed projections inherited from the pretrained model. Therefore, although we describe this implementation as adapting the value projection for simplicity, the trainable core effectively modifies the entire MHA block rather than independently fine-tuning the value-projection weights. We will clarify this distinction in the revised manuscript to avoid ambiguity regarding the adaptation scope.

LoRA and Full. Since MODEM adapts the MHA block as a whole, we align the adaptation scope of the baselines accordingly: Full fine-tuning updates the entire attention block, while LoRA is applied to all four attention projections (Q/K/V/O), ensuring a consistent comparison of adaptation scope.

Additional ablation. To evaluate whether MODEM generalizes to other model components, we independently apply it to the query (Q), key (K), and two MLP projections (FC1 and FC2) under the same experimental setting. For these projections, we treat each as an individual linear layer and construct the fixed adaptation bases from the SVD of its pretrained weight matrix. This differs from our value-projection (V) implementation, where the fixed bases are derived from the pretrained multi-head attention block rather than the value-projection weights alone.

Adaptation results

MHA (v)QKFC1FC2
Acc.83.4682.2181.9583.7583.69

All settings use the same trainable-parameter budget of 0.59M. The results demonstrate that MODEM achieves competitive adaptation performance across different attention and MLP projections, highlighting its flexibility across model components.

Do the correlations hold if MODEMs are applied to other layers (Q/K, MLP)? We further evaluate the correlation between parameter-space distances of the adapters trained under the aforementioned settings and task-space relationships measured by transfer regret.

MethodPearsonSpearman
MODEM (Q)0.5310.550
MODEM (K)0.5240.535
MODEM (FC1)0.4730.609
MODEM (FC2)0.4530.545
MODEM (V)0.4430.494

+ Path-aware

MethodPearsonSpearman
MODEM (Q)0.7810.717
MODEM (K)0.7050.633
MODEM (FC1)0.5590.680
MODEM (FC2)0.6270.596
MODEM (V)0.7740.766

As shown in the table above, MODEM achieves comparable task–parameter correlation results across different model components under both evaluation settings. Furthermore, incorporating path-aware analysis consistently improves these correlations, demonstrating that the observed representational properties extend beyond the value projection.

Apply path-aware to LoRA and Full. To isolate the contribution of the representation from that of the path-aware operation, we additionally applied the same procedure to LoRA and Full under the same evaluation settings as Section 4.

Path-aware correlation

MethodPearsonSpearman
LoRA0.340.42
LoRA + Path0.380.44
Full-0.43-0.46
Full + Path0.410.39
MODEM0.440.49
MODEM + Path0.770.77

The results confirm that incorporating adaptation trajectories improves task-relationship estimation beyond MODEM. This supports the general applicability of the path-aware formulation. Importantly, MODEM achieves the strongest correlations both with and without path information,

Path-aware merging

MethodAdditionPath-awareComputational CostPath Overhead
LoRA75.06%78.68%0.603 PFLOP2.64 GB
Full77.54%80.09%0.647 PFLOP2.64 GB
MODEM77.42%80.19%0.603 PFLOP0.22 GB

Path-aware merging achieves 78.68% accuracy for LoRA and 80.09% for Full fine-tuning, demonstrating that the mechanism is broadly applicable beyond MODEM. However, applying path-aware merging to these methods can incur substantially greater storage overhead, as their parameter-update checkpoints are considerably larger than MODEM embeddings.

Reviewer_ZV1YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • summary: The paper proposes model embeddings (MODEMs): adapters written as small coordinate matrices in fixed bases taken from the pretrained weights (SVD blocks, or per-head value projections).
  • contribution: It gives linearity and isometry results and a first-order link between differences in embeddings and differences in model outputs.
  • validation: Experiments cover 1,023 MNIST digit-subset tasks, CLIP ViT-B/32 on 20 tasks and GPT-2 on 20 tasks, looking at embedding distance vs transfer regret, a shared-plus-digit decomposition, and adding and subtracting adapters.
Critical strengths
  1. The controlled MNIST study is clean. With known task relations, a shared component plus one direction per digit explains 88% of the update energy, and moving along v0v_0 adds digit 0 (Fig. 3).
  2. Treating adapters as representations that can be compared and also decoded back into models is a useful framing.
Critical weaknesses
  1. The comparisons and the theory don't isolate what MODEM adds.
  • Prop. 1 (isometry) needs orthonormal blocks, but the attention form used in the experiments, ΔW(h)=M(h)⊤W0,v(h)\Delta W^{(h)} = M^{(h)\top} W_{0,v}^{(h)}, is not orthonormal.
  • MODEM distances are computed on coordinates, LoRA and full fine-tuning distances on ΔW\Delta W, and the trained scopes differ. With cosine distance, full fine-tuning correlates as well as MODEM (0.949 vs 0.945, Table 6).
  • Missing: a random-basis and matched-scope control, and a comparison with LoRA-XS and SVFT, which use the same structure. The first-order results hold for any linear parameterization.
  1. The evidence for operational usefulness is weak.
  • On MNIST, addition gives the added digit in only 15.4% of samples (Table 2). The paper's own decomposition (Eq. 19) suggests (2/3)m01+(1/3)m2(2/3)m_{01} + (1/3)m_2 rather than m01+m2m_{01} + m_2.
  • On ViT, MODEM addition equals full fine-tuning (77.42% vs 77.54%), subtraction trades retention for forgetting, and there are no merging baselines (TIES, RegMean, AdaMerging, TSV).
  • The held-out construction experiment mentioned in the OpenReview abstract is not in the PDF.
  1. Presentation
  • section 5 as “related work” appears a little bit too late in the main paper
Questions for authors

Clarifications:

  1. Should the numerator of Eq. 18 be L^j(mi)−L^j(mj)\hat{L}_j(m_i) - \hat{L}_j(m_j)? And how is the directed regret made symmetric?
  2. Are distances computed on mm or on D(m)D(m), and do the results change on D(m)D(m)?
  3. Does each task use 500 examples per digit (Sec. 3.1) or all samples (App. B.1)?
  4. Can you build embeddings for held-out digit subsets from the fitted directions and evaluate their generations?
  5. Add the joint-PCA and surface-construction details from Appendix B.3 to the relevant figure caption, making clear that the rendered surface is a visualization aid rather than evidence of an underlying continuous task manifold.
  6. Section 3.1 specifies the use of 500 training samples per target digit for each MNIST task, whereas Appendix B.1 specifies the use of all available training samples for the selected digit classes. Do these descriptions refer to different experiments? Please clarify which sampling scheme was employed for the results presented in Tables 1, 2, and 6.
  7. Regarding the ViT experiments, how do the path-aware relationships (as defined in Equation 20) and the path-merging mechanism (in Equation 21), perform when applied to LoRA and full fine-tuning trajectories, under consistent data access and evaluation conditions?

Suggestions (not affecting my score):

  • Relate the work to CAVIA, LEO and weight-space learning (hyper-representations). Also, the page header still says ICLR 2026.
  • Building on the mean values presented in Table 4, we provide additional statistical results broken down by task or task-pair. This will help readers assess whether the improvements, both in terms of "merging" operations and the trade-off between "retaining" and "removing" elements, are consistent across different datasets.
3 comments
AuthorsOpenReview ↗

Rebuttal (Part1)

We thank the reviewer for their constructive feedback. We address the comments below and have revised the manuscript accordingly.

Our primary motivation is to investigate the representation, relationships, and arithmetic of learned task adaptations within a structured subspace induced by a pretrained model. We first study addition and subtraction under approximately linear functional geometry, where embedding-space operations can induce corresponding changes in task behavior. To account for nonlinear functional effects and variations in local geometry, we further introduce a path-aware approach that incorporates trajectories to improve the operations.

(W1a) Apply Prop.1 to attention.

A: Given attention weight updates ΔWv(h)=M(h)Wv(h),\Delta W_v^{(h)}=M^{(h)}W_v^{(h)}, where Wv(h)∈Rdh×dW_v^{(h)}\in\mathbb R^{d_h\times d} is the fixed pretrained value projection for head hh, and M(h)∈Rdh×dhM^{(h)}\in\mathbb R^{d_h\times d_h} is the trainable embedding matrix.

For two MODEMs mim_i and mjm_j, define ΔMij(h)=Mi(h)−Mj(h).\Delta M_{ij}^{(h)}=M_i^{(h)}-M_j^{(h)}. The difference between their decoded updates is Dh(mi)−Dh(mj)=ΔMij(h)Wv(h).D_h(m_i)-D_h(m_j)=\Delta M_{ij}^{(h)}W_v^{(h)}.

By the singular-value bounds for matrix multiplication,

σmin⁡(Wv(h))∥∣ΔMij(h)∥∣F≤∥∣ΔMij(h)Wv(h)∥∣F≤∥∣Wv(h)∥2∥ΔMij(h)∥∣F.\sigma_{\min}(W_v^{(h)})\||\Delta M_{ij}^{(h)}\||_F \le \||\Delta M_{ij}^{(h)}W_v^{(h)}\||_F \le \||W_v^{(h)}\|_2\|\Delta M_{ij}^{(h)}\||_F.

Here, σmin⁡\sigma_{\min} denotes the smallest singular value, including zero if the matrix is rank-deficient.

Since the attention update concatenates the head-wise updates, we define

cmin⁡=min⁡hσmin⁡(Wv(h)),cmax⁡=max⁡h∥Wv(h)∥2.c_{\min}=\min_h\sigma_{\min}(W_v^{(h)}), \qquad c_{\max}=\max_h\|W_v^{(h)}\|_2.

Then the full decoder satisfies

cmin⁡∥∣mi−mj∥∣F≤∥∣D(mi)−D(mj)∥∣F≤cmax⁡∥∣mi−mj∥∣F.c_{\min}\||m_i-m_j\||_F \le \||D(m_i)-D(m_j)\||_F \le c_{\max}\||m_i-m_j\||_F.

We will provide the generalized distance relationship for Prop.1.

(W1b, Q1, Q2) Clarify the correlation between task space and param space.

A: We clarify the definition of transfer regret, the distinction between distances measured in the embedding and decoded parameter spaces, and the complementary roles of cosine and Euclidean distances.

Definition of transfer regret. We clarify that the directional transfer regret from Eq. (18) is defined as Ri→j=L^j(mi)−L^j(mj)L^j(0)−L^j(mj)R_{i\rightarrow j}=\frac{\hat{L}_j(m_i) - \hat{L}_j(m_j)}{\hat{L}_j(0) - \hat{L}_j(m_j)}. We could make it symmetric by defining R=12(Ri→j+Rj→i)R=\frac{1}{2} (R_{i\rightarrow j} + R_{j \rightarrow i}).

Notably, directional transfer regret can be used directly in correlation analysis by comparing each ordered task pair with its corresponding embedding distance.

Correlations in mm vs. D(m)D(m). The correlations reported in the original manuscript are computed using distances between the learned model embeddings mim_i and mjm_j. To examine whether these relationships are specific to the latent coordinate representation, we additionally evaluate distances between the corresponding decoded parameter updates D(mi)D(m_i) and D(mj)D(m_j).

DistanceRelationmm: Pearson / SpearmanD(m)D(m): Pearson / Spearman
EuclideanTask Jaccard0.873 / 0.8650.911 / 0.907
Functional MMD0.668 / 0.7260.733 / 0.793
Transfer regret0.741 / 0.8260.794 / 0.880
CosineTask Jaccard0.928 / 0.9370.930 / 0.940
Functional MMD0.912 / 0.9160.920 / 0.924
Transfer regret0.945 / 0.9620.946 / 0.965

Strong task–parameter correlations are consistently observed in both spaces, indicating that the relationships captured by MODEM are not an artifact of measuring distances exclusively in the latent coordinates. Secondly, decoding generally strengthens the observed correlations, particularly for Euclidean distance. For example, the Pearson correlation with functional MMD increases from 0.668 to 0.733, while the Spearman correlation with transfer regret increases from 0.826 to 0.880.

The compact embedding space already captures substantial task-related structure without requiring reconstruction of the full parameter updates for similarity evaluation. Thus, MODEM provides a compact representation for analyzing task relationships, while the decoded space offers an additional perspective on their geometry.

We acknowledge that full fine-tuning achieves a comparable cosine correlation to MODEM (0.949 vs. 0.945). However, our goal is not to outperform full fine-tuning across all metrics, but to preserve meaningful task relationships in a substantially more compact adaptation space. We evaluate both cosine and Euclidean distances because they capture complementary geometric properties: cosine measures directional relationships, while Euclidean also reflects magnitude differences. The comparable cosine correlations demonstrate that MODEM preserves task-related directional structure despite using substantially fewer trainable parameters.

AuthorsOpenReview ↗

Rebuttal (Part2)

(W1c) Comparison with LoRA-XS, SVFT.

SVFT primarily investigates parameter-efficient fine-tuning by learning selected coefficients in the pretrained spectral basis, using sparsity patterns such as diagonal, banded, random, and Top-k. In contrast, MODEM addresses a different research question: whether task adaptations can be represented and analyzed within a compact, structured model embedding space, rather than the high-dimensional parameter space of pretrained models.

LoRA-XS learns a small core matrix between fixed low-rank projection bases. Its parameterization can be viewed as a special case of MODEM with a single block, provided the projection bases and their dimensions are chosen equivalently.

We additionally compare MODEM against SVFT and LoRA-XS under matched trainable-parameter budgets.

MethodAcc.Paramsextra MACs
SVFT83.43% ± 0.15589,82414.746M
LoRA-XS83.48% ± 0.02591,4084.683M
MODEM83.46% ± 0.07589,8241.769M

We compare the additional MACs required to evaluate the adaptation. While SVFT uses full-dimensional spectral projections and LoRA-XS uses truncated low-rank projections, MODEM organizes its computation into headwise spectral blocks. This structure can reduce additional computation while maintaining comparable adaptation accuracy.

MODEM achieves comparable adaptation accuracy to both methods while requiring approximately 8.33× and 2.65× fewer ideal additional MACs than SVFT and LoRA-XS, respectively.

(Q7) Apply Path to Full and LoRA.

To isolate the contribution of the representation from that of the path-aware operation, we additionally applied the same procedure to LoRA and Full under the same evaluation settings as Section 4.

Path-aware correlation

MethodPearsonSpearman
LoRA0.340.42
LoRA + Path0.380.44
Full-0.43-0.46
Full + Path0.410.39
MODEM0.440.49
MODEM + Path0.770.77

The results confirm that incorporating adaptation trajectories improves task-relationship estimation beyond MODEM. This supports the general applicability of the path-aware formulation. Importantly, MODEM achieves the strongest correlations both with and without path information,

Path-aware merging

MethodAdditionPath-awareComputational CostPath Overhead
LoRA75.06%78.68%0.603 PFLOP2.64 GB
Full77.54%80.09%0.647 PFLOP2.64 GB
MODEM77.42%80.19%0.603 PFLOP0.22 GB

Path-aware merging achieves 78.68% accuracy for LoRA and 80.09% for Full fine-tuning, demonstrating that the mechanism is broadly applicable beyond MODEM. However, applying path-aware merging to these methods can incur substantially greater storage overhead, as their parameter-update checkpoints are considerably larger than MODEM embeddings.

(W2a) Why not (2/3)m01+(1/3)m2(2/3) m_{01} + (1/3) m_{2} rather than m01+m2m_{01} + m_{2}?

In Table 2, our primary objective is to investigate whether model embeddings support basic addition and subtraction operations. Therefore, we use direct arithmetic, m01+m2m_{01}+m_2, without introducing additional weighting coefficients, to isolate the effects of these operations.

We additionally evaluate weighted merging and interpolation in MODEM space. Specifically, we construct triplet adaptations from a digit-pair embedding mab\mathbf m_{ab} and a singleton embedding mc\mathbf m_c using m(λ)=(1−λ)mab+λmc,\mathbf m(\lambda)=(1-\lambda)\mathbf m_{ab}+\lambda\mathbf m_c, where λ∈[0,1].\lambda\in[0,1]. The experimental setting is the same as in Table 2.

We report the average generated frequencies of the corresponding digits (a/b/c).

λ\lambdaPair + Singleton → Triplet (a / b / c) %JSD↓
0.041.73 / 42.93 / 1.600.129
0.140.53 / 37.53 / 3.930.090
0.238.80 / 30.93 / 8.000.056
0.334.40 / 24.40 / 14.070.033
1/332.60 / 22.20 / 16.930.028
0.427.87 / 18.27 / 23.070.031
0.521.07 / 12.27 / 34.400.053
0.615.33 / 7.73 / 47.730.097
0.710.07 / 4.20 / 61.530.164
0.85.60 / 2.00 / 73.530.240
0.93.13 / 0.80 / 82.200.309
1.01.33 / 0.53 / 89.000.366

In the table, JSD means the divergence between the resulting distribution and that of the corresponding trained triplet. It shows that λ=1/3\lambda=1/3 achieves the lowest JSD, indicating the closest match to the target distribution.

AuthorsOpenReview ↗

Rebuttal (Part3)

(W2b) Additional merging baseline.

To provide a more comprehensive evaluation, we additionally compare MODEM against established merging methods, including TIES, TSV-Merge, AdaMerging, and RegMean, under the same experimental setting as in Section 4.

MethodAcc.
TIES77.55%
TSV-Merge75.05%
AdaMerging77.69%
RegMean79.80%
MODEM77.42%
MODEM + Path80.19%

MODEM achieves 77.42% mean accuracy, comparable to TIES and AdaMerging, although below RegMean. Incorporating path-aware merging improves accuracy to 80.19%, outperforming all evaluated baselines, including RegMean.

Subtraction and the retention–forgetting tradeoff. Plain MODEM preserves more performance on the removed task than LoRA and full fine-tuning, indicating weaker forgetting despite maintaining competitive retained-task accuracy.

However, the pretrained backbone already achieves 55.3% accuracy on the removed task, suggesting that part of the corresponding capability is encoded in the pretrained model rather than exclusively in the task-specific adaptation. Consequently, subtracting task-specific updates alone cannot be expected to eliminate all pretrained capabilities.

(W2c, Q4) The held-out construction experiment.

We clarify that our original paper already investigates held-out task construction, where the target triplet adapter is excluded from the available source adapters, and its functionality is approximated through arithmetic operations on independently trained singleton and pair adapters.

To further strengthen this evaluation, we conduct additional experiments on the held-out target 6,7,8\\{6,7,8\\}, systematically examining how source-target overlap influences reconstruction quality.

Motivated by the component-wise representation in Eq. (19), we construct the target embedding as a linear combination of available source adapters: $ \hat{\mathbf m}_{678} =\sum_{S\in\mathcal C}\alpha_S\mathbf m_S, $ where C\mathcal C denotes the source-adapter set.

We evaluate two levels of source-target overlap:

  • Partial overlap: Each source adapter contains 1-2 target digits, e.g., 6\\{6\\}, 6,8\\{6,8\\}, and 7,8\\{7,8\\}.
  • Full overlap with distractors: Each source adapter contains all three target digits plus an additional digit, e.g., 6,7,8,0\\{6,7,8,0\\}, 6,7,8,1\\{6,7,8,1\\}, and 6,7,8,2\\{6,7,8,2\\}.

We evaluate the resulting generative behavior using four complementary metrics: target-digit rate, Jensen–Shannon divergence (JSD), feature-space FID, and paired SSIM. The directly trained triplet adapter serves as the reference for evaluating reconstruction quality.

SettingTarget-digit rate ↑JSD ↓Feature FID ↓Paired SSIM ↑
Trained {6,7,8} reference72.07%001
Partial overlap (1 digit)24.02%0.2224741.420.7294
Partial overlap (2 digits)45.51%0.0984303.180.8217
Full overlap + distractor60.35%0.018061.130.9244

First, MODEM supports held-out task construction without directly training the target adapter. The constructed embeddings recover a substantial portion of the target behavior using only previously learned source adaptations, demonstrating the feasibility of compositional reuse in the model embedding space.

Second, source–target overlap substantially influences reconstruction quality. Compared with partial-overlap sources, full-overlap sources with distractors produce stronger semantic and distributional alignment. This suggests that task-related information encoded in existing adaptations can be reused to approximate a target distribution, even when individual source adapters contain additional unwanted behaviors.

(W3). Section 5 as “related work” appears a little bit too late in the main paper

A: We could move it earlier

(Q3, Q6) Clarify the experimental setting.

A: We use 500 per digit for Tables 1, 2, 6 and will correct Appendix B.1

(Q5) Provide surface-construction details to the figure caption.

A: We will explain the surface with more detail.

(Q8) Suggestions

A: We will include them in the related work. We will update our paper accordingly.

Reviewer_AeufOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MODEM, a structured model embedding space within a shared subspace induced by a pretrained model, to represent task-specific adaptations. It studies four properties: adaptation fidelity, relational geometry, functional component structure, and operational usefulness. These properties are examined through theoretical analysis and experiments on MNIST generation and 20 vision classification datasets using ViT. The paper further introduces a relation based on the MODEM path and MODEM path merging.

Critical strengths

The idea of representing task adaptations in a shared, structured subspace of a pretrained model is interesting. Using fixed pretrained bases to represent parameter updates provides a useful perspective for studying task relationships and functional operations.

Critical weaknesses

Insufficient methodological details: Several parts of the method are not clearly explained. For example, Eq. (19) does not specify how the digit-associated directions (vi) are obtained, while Eq. (21) does not explain how the path-aware merging optimization is performed. Section 4 also lacks detailed training details, how the number of path checkpoints (K) is selected, and the additional computational cost of path-aware optimization. These makes the results difficult to reproduce.

Insufficient evidence for MODEM's effectiveness: The results of MODEM alone show no clear advantage over Full Fine-tuning and LoRA. The largest improvements come from MODEM + Path, which introduces additional path optimization. However, training trajectories can also be recorded for Full Fine-tuning and LoRA, and similar path-aware approaches could potentially be applied to them. Without such comparisons, it is difficult to determine whether the improvements come from MODEM itself or the additional path optimization, raising concerns about the fairness of the comparison.

Limited novelty: The paper proposes a structured model embedding space based on fixed pretrained bases and investigates task relationships, functional components, and task arithmetic. However, similar aspects seem to have been explored in existing work, such as Task2Vec [1], Editing Models with Task Arithmetic [2], Task Arithmetic in the Tangent Space [3], and Spectral Adapter [4]. The paper does not clearly explain what new technical contributions MODEM offers compared with these existing approaches.

References: [1] Task2Vec: Task Embedding for Meta-Learning. ICCV 2019. [2] Editing Models with Task Arithmetic. ICLR 2023. [3] Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. NeurIPS 2023. [4] Spectral Adapter: Fine-Tuning in Spectral Space. NeurIPS 2024.

Questions for authors
  1. How are the digit-associated directions vi in Eq. (19) obtained?

  2. How is the path-aware merging in Eq. (21) actually optimized? In particular, how is the path of the merged model constructed, and what initialization, optimizer, and number of optimization steps are used?

  3. Could the authors provide more details on the training configurations in Section 4, including the learning rates, training epochs, path checkpoint selection, and additional computational costs? Could similar path-aware optimization be applied to Full Fine-tuning and LoRA for a fairer comparison?

  4. In Table 2, addition produces only 15.4% of the added digit, rather than approximately 33.3% for the intended uniform distribution. Similarly, subtraction produces 82.0% of the target digit instead of approaching 100%. Could the authors explain these deviations from the intended distributions and account for the remaining generated samples?

  5. In Table 3, incorporating the MODEM path substantially improves the correlation with task relationships. Since training trajectories can also be recorded for Full Fine-tuning and LoRA, could the authors evaluate whether similar path-aware measures improve these baselines?

  6. The experiments mainly evaluate addition and subtraction, while existing work has explored more general task composition and model merging methods. Could the authors evaluate other operations, such as interpolation and weighted merging, to better demonstrate the operational usefulness of MODEM?

5 comments
AuthorsOpenReview ↗

Rebuttal (Part1)

We thank the reviewer for their constructive feedback. We address the comments below and have revised the manuscript accordingly.

Our primary motivation is to investigate the representation, relationships, and arithmetic of learned task adaptations within a structured subspace induced by a pretrained model. We first study addition and subtraction under approximately linear functional geometry, where embedding-space operations can induce corresponding changes in task behavior. To account for nonlinear functional effects and variations in local geometry, we further introduce a path-aware approach that incorporates trajectories to improve the operations.

(W1, Q1) Provide details of how to get digit-associated directions vi\mathbf v_i in Eq. (19).

For every training task SS, define its semantic feature vector aS,i=1[i∈S]∣S∣.a_{S,i}=\frac{\mathbb 1[i\in S]}{|S|}. For example, a{2,5,7}=[0,0,13,0,0,13,0,13,0,0].a_{\{2,5,7\}}=[0,0,\tfrac13,0,0,\tfrac13,0,\tfrac13,0,0].

We estimate the parameters by minimizing

min⁡mshared,v0,…,v9∑S∈Ttrain∥mS−mshared−∑iaS,ivi∥22\min_{\mathbf m_{\mathrm{shared}},\mathbf v_0,\ldots,\mathbf v_9} \sum_{S\in\mathcal T_{\mathrm{train}}} \left\| \mathbf m_S- \mathbf m_{\mathrm{shared}}- \sum_i a_{S,i}\mathbf v_i \right\|_2^2; s.t. ∑ivi=0.\sum_i\mathbf v_i=0.

Each vi\mathbf v_i is therefore the digit-ii effect that jointly best explains all training adapters containing digit ii.

(W1, Q2) Provide details of path-aware merging.

We store the trajectories mt(k)k=1K\\{\mathbf m_t^{(k)}\\}_{k=1}^K of two specialists, where tt represents taskA or taskB. We then convert the two specialist trajectories into checkpoint increments, Δmt(k)=mt(k)−mt(k−1).\Delta \mathbf m_t^{(k)}=\mathbf m_t^{(k)}-\mathbf m_t^{(k-1)}. The merged trajectory is constrained to use linear combinations of the corresponding source increments: ΔmC(k)=αkΔmA(k)+βkΔmB(k).\Delta \mathbf m_C^{(k)}=\alpha_k\Delta\mathbf m_A^{(k)}+\beta_k\Delta\mathbf m_B^{(k)}. Thus, only two scalar coefficients, αk\alpha_k and βk\beta_k, are selected at each path segment.

At the current merged state mC(k−1)\mathbf m_C^{(k-1)}, we compute the two Jacobian-vector products Φ(x;mC(k−1))ΔmA(k)\Phi(x;\mathbf m_C^{(k-1)}) \Delta\mathbf m_A^{(k)}, Φ(x;mC(k−1))ΔmB(k).\Phi(x;\mathbf m_C^{(k-1)}) \Delta\mathbf m_B^{(k)}.

The target is formed from the specialists' actual changes: y(k)=∑t∈{A,B}λtΦ(x;mt(k−1))Δmt(k).\mathbf y^{(k)} = \sum_{t\in\{A,B\}}\lambda_t \Phi(x;\mathbf m_t^{(k-1)}) \Delta\mathbf m_t^{(k)}.

We then solve the following problem:

(αk,βk)=arg⁡min⁡α,β∥∣αΦ(x;mC(k−1))ΔmA(k)+βΦ(x;mC(k−1))ΔmB(k)−y(k)∥∣22(\alpha_k,\beta_k)=\arg\min_{\alpha,\beta} \left\||\alpha \Phi(x;\mathbf m_C^{(k-1)}) \Delta\mathbf m_A^{(k)}+ \beta \Phi(x;\mathbf m_C^{(k-1)}) \Delta\mathbf m_B^{(k)}-\mathbf y^{(k)}\right\||_2^2

i.e., (αk,βk)=arg⁡min⁡α,β∥∣Φ(x;mC(k−1))ΔmC(k)−∑t∈{A,B}λtΦ(x;mt(k−1))Δmt(k)∥∣22.(\alpha_k,\beta_k)=\arg\min_{\alpha,\beta} \left\|| \Phi(x;\mathbf m_C^{(k-1)}) \Delta \mathbf m_C^{(k)} -\sum_{t\in\{A,B\}}\lambda_t \Phi(x;\mathbf m_t^{(k-1)}) \Delta\mathbf m_t^{(k)}\right\||_2^2.

This linear system can be solved directly with torch.linalg.solve. Therefore, there is no gradient-based optimizer or learning rate. JVPs are computed with forward-mode automatic differentiation without constructing the full Jacobian. We have (λA,λB)=(1,1)(\lambda_A,\lambda_B)=(1,1). The merged path starts from the zero adapter. We use K=50K=50 checkpoints. Only a small set of training data is used to select these coefficients.

(W2, Q3, Q5) Apply path to Full and LoRA.

A: To isolate the contribution of the representation from that of the path-aware operation, we additionally applied the same procedure to LoRA and Full under the same evaluation settings as Section 4.

Path-aware for merging.

MethodAdditionPath-awareComputational CostOverhead
LoRA75.06%78.68%0.603 PFLOP2.66 GB
Full77.54%80.09%0.647 PFLOP2.64 GB
MODEM77.42%80.19%0.603 PFLOP0.22 GB

Path-aware merging achieves 78.68% accuracy for LoRA and 80.09% for Full fine-tuning, demonstrating that the mechanism is broadly applicable beyond MODEM. However, applying path-aware merging to these methods can incur substantially greater storage overhead, as their parameter-update checkpoints are considerably larger than MODEM embeddings.

Path-aware for correlation.

MethodPearsonSpearman
LoRA0.340.42
LoRA + Path0.380.44
Full-0.43-0.46
Full + Path0.410.39
MODEM0.440.49
MODEM + Path0.770.77

The results confirm that incorporating adaptation trajectories improves task-relationship estimation beyond MODEM. This supports the general applicability of the path-aware formulation rather than suggesting that it is unique to MODEM. Importantly, MODEM achieves the strongest correlations both with and without path information,

Experimental details. To fine-tune the adapters (MODEM, LoRA, Full), we use the Adam optimizer with a 3×10−43\times 10^{-4} learning rate and 50 training epochs for all tasks.

AuthorsOpenReview ↗

Rebuttal (Part2)

(W3) Identify the contribution compared with existing approaches.

A: As pretrained models become increasingly large, directly analyzing task adaptations and their interactions in the full parameter space becomes computationally expensive and difficult to interpret. We therefore construct a compact, shared coordinate space that enables systematic investigation of task relationships and functional operations while maintaining a direct connection to the underlying pretrained model.

Our work differs from the previous studies in the following aspects:

  • Task2Vec: Task2Vec constructs Fisher-information-based embeddings to characterize task similarity and transferability. In contrast, MODEM represents learned parameter adaptations themselves, where each embedding directly maps to a model update through a fixed decoder.

  • Task Arithmetic: Task Arithmetic demonstrates that fine-tuned parameter differences can be combined to edit model behavior. Our motivation is to investigate these operations within a compact, structured subspace of the pretrained model, rather than directly manipulating high-dimensional task vectors.

  • Task Arithmetic in the Tangent Space: While first-order functional approximations are relevant to both analyses, MODEM focuses on the representation, relationships, and operations of learned model adaptations within a structured parameter space.

  • Spectral Adapter: It exploits pretrained spectral information for parameter-efficient adaptation through additive updates or orthogonal rotations of selected singular-vector factors. In contrast, the SVD-based formulation of MODEM keeps the pretrained singular-vector bases fixed and learns task-specific coefficient matrices within selected spectral blocks.

(Q4) Clarify why addition and subtraction cannot recreate the intended distributions.

A: Combining independently learned adaptations can introduce interference between their functional effects. Such interference may weaken certain task-specific behaviors, causing generated samples to be classified as other digits or EMNIST letters rather than the intended target digits. Consequently, addition can introduce the desired new digit without achieving the ideal uniform triplet distribution, while subtraction can suppress unwanted digits without perfectly reconstructing the target singleton distribution.

Importantly, these deviations are not exclusive to embedding arithmetic. Even independently trained triplet and singleton specialists do not perfectly reproduce their target distributions: triplet specialists may generate non-uniform digit frequencies, while singleton specialists may produce samples outside the target class. Thus, the imperfect distributions resulting from arithmetic reflect both the inherent limitations of the learned generative models and additional interference introduced by combining task adaptations.

(Q6) Could the authors evaluate other operations, such as interpolation and weighted merging

A: We additionally evaluate weighted merging and interpolation in MODEM space to investigate whether task-specific behaviors can be continuously controlled through embedding-space operations. Specifically, we construct triplet adaptations from a digit-pair embedding mab\mathbf m_{ab} and a singleton embedding mc\mathbf m_c using m(λ)=(1−λ)mab+λmc,\mathbf m(\lambda)=(1-\lambda)\mathbf m_{ab}+\lambda\mathbf m_c,, where λ∈[0,1].\lambda\in[0,1]. The experimental setting is the same as Table 2.

We report the average generated frequencies of the corresponding digits (a/b/c).

λ\lambdaPair + Singleton → Triplet (a / b / c) %
0.041.73 / 42.93 / 1.60
0.140.53 / 37.53 / 3.93
0.238.80 / 30.93 / 8.00
0.334.40 / 24.40 / 14.07
1/332.60 / 22.20 / 16.93
0.427.87 / 18.27 / 23.07
0.521.07 / 12.27 / 34.40
0.615.33 / 7.73 / 47.73
0.710.07 / 4.20 / 61.53
0.85.60 / 2.00 / 73.53
0.93.13 / 0.80 / 82.20
1.01.33 / 0.53 / 89.00

These findings demonstrate that MODEM supports weighted composition beyond simple addition and subtraction, allowing task-specific behaviors to be progressively adjusted through embedding-space operations. The resulting distributions exhibit predictable changes in response to the composition coefficients.

Thanks for the detailled response and the many new experiments. Feedback:

  • The comparison with LoRA-XS and SVFT, the merging baselines, the weighted merging, and the distances on D(m) address most of my questions.
  • The weighted merging also confirms that plain addition is not the right operation, which is good to see.
  • However, my main concern is still what MODEM adds. The accuracy is the same as LoRA-XS and SVFT, and the path-aware gains also show up for LoRA and full fine-tuning, so the advantage is mainly in compute and storage. The random-basis and matched-scope control is also not done yet.
  • Without the path, MODEM is comparible to TIES and AdaMerging but below RegMean, and the path uses training trajectories that the other merging methods don’t have. For the held-out construction, the best result needs sources that already contain all target digits, while partial overlap stays quite far from the reference.
  • I would frame the contribution more around efficiency and the analysis of task relations.

I keep my score of 2.

Reviewer_m5BjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper propose a model embedding method, which embedding the parameter updates to a subspace via basis from UV decomposition. Based on the basis, the paper proposes a low-rank adaptation method, with path strengthening. The experimental results provide varied ablations for the proposed method.

Critical strengths
  1. The idea is super interesting. Having model embedding is an interesting topic and I'm super excited to see the paper designs something working.

  2. The visualization + the Path method is a plus for the base method MODEM.

Critical weaknesses
  1. How does "Operations in MODEM space affect task space" work with epoch? Q1: does the formulation (21) give different weights to epoch 0 and epoch 100? Assuming converging, I expect different weight for differenr epoch. Q2: how does #epoch affect the formulation (21)? Saying I train 10 epoch & 100000 epoch.

  2. I did not get Figure 1. Why is model embedding space becomes bigger and bigger?

  3. Though the idea is super interesting, the paper test is mostly on images, with limited language test, limiting the generalization.

Questions for authors
  1. Figure 2 show a very interesting behavior. Full finetuning is more aligned with MODEM than LoRA. MODEM might be a better method than LoRA. Is MODEM specifically better than LoRA on image datasets?

  2. what is the boundary of the proposed method? Consider dataset & model. Is the method limited to vision dataset and CNN & ViT?

1 comment

We thank the reviewer for their constructive feedback. We address the comments below and have revised the manuscript accordingly.

(W1) Clarification of how operations in MODEM space affect task space work.

A: We clarify that our primary objective is to study the representational and operational properties of converged task embeddings (i.e., endpoint), rather than intermediate training states that may not yet adequately represent the target tasks.

Equation (21) extends endpoint operations by incorporating optimization trajectories. Since all task embeddings originate from the same pretrained initialization, their trajectories provide corresponding sequences of updates that can be used to improve operations.

Q1: Does Equation (21) assign different weights to different epochs? Equation (21) does not assign predetermined weights to epochs. Instead, it adaptively combines checkpoint increments, Δmt(k)=mt(k)−mt(k−1)\Delta\mathbf m_t^{(k)}=\mathbf m_t^{(k)}-\mathbf m_t^{(k-1)}, based on their functional effects under the local Jacobian Φ(x;m)\Phi(x;\mathbf m). Thus, different training stages may contribute differently, depending on both the magnitude of their updates and the local functional geometry.

Q2: How does the number of training epochs affect Equation (21)? The formulation does not explicitly depend on the number of epochs, but on the recorded optimization trajectory and its checkpoint discretization. Training for 10 versus 100,000 epochs does not change the formulation. Once training converges, additional epochs with negligible updates are expected to contribute little.

(W2) Why does the model embedding space get bigger in Fig.1?

A: The figure visualizes how task-specific embeddings evolve during optimization. All embeddings are initialized from the same pretrained model, corresponding to the zero embedding. As training progresses, different tasks induce different parameter updates, causing their embeddings to move away from the common initialization and become increasingly separated in the shared coordinate space.

Importantly, all epochs are projected using a single PCA basis with identical axis limits. Therefore, the apparent expansion reflects the increasing dispersion of the learned task embeddings.

(W3) Generalization of this work to other non-vision tasks.

A: MODEM is not inherently restricted to vision architectures: its general formulation applies to linear transformations with fixed shared projection structures, including attention and feed-forward layers in transformer-based models.

Our GPT-2 experiments provide an initial evaluation beyond vision, using task-specific adaptations in both attention and MLP components. These experiments support the applicability of the framework to language models.

We will clarify this and distinguish the architectural generality of the parameterization from the empirical scope of our current evaluations.

(Q1) Is MODEM better at representing the task space than LoRA on image datasets?

A: Our experiments on MNIST (Table 6), vision datasets (Table 3), and language datasets (Appendix C.2) suggest that LoRA generally exhibits weaker correspondence between parameter-space and task-space relationships.

One possible explanation is: LoRA constrains each adaptation to be low-rank, but independently trained adapters may learn different low-rank subspaces. Consequently, their parameter updates are not explicitly constrained to a common low-dimensional subspace, potentially making their geometric relationships less consistent with task relationships.

In contrast, MODEM learns task-specific coordinates within a fixed, pretrained-model-induced subspace shared across tasks. This provides a consistent coordinate system for comparing independently learned adaptations and helps preserve meaningful task relationships.

(Q2) What datasets or models is this method suited for?

A: MODEM is not restricted to specific architectures or data modalities; its applicability primarily depends on whether task adaptations can be represented within a shared, fixed coordinate structure derived from a common pretrained model.

At the architectural level, the general SVD-based formulation applies to linear transformations, including attention projections and feed-forward layers. At the task level, MODEM assumes that independently learned adaptations share a compatible pretrained backbone and coordinate system. This is important because direct geometric comparisons and arithmetic operations require embeddings to have consistent meanings across tasks.

The framework has one potential limitation. A restricted adaptation space may not capture tasks requiring substantial changes outside the selected subspace. Overall, MODEM provides a general framework for representing and studying task adaptations across architectures and modalities, while broader evaluation on larger models and more diverse tasks remains an important direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.