acceptodds
Under review as a conference paper at ICLR 2027

What Can a Task Checkpoint Identify? Invariant Task-Core Fusion for Model Merging

Abstract

Model merging combines independently fine-tuned experts into one model, but interference can degrade performance. Without task data, data-free methods rely on checkpoints and often use task vectors, the parameter changes induced by fine-tuning, as proxies for inaccessible task data. In a controlled linear setting, we show that different training data can produce the same final weights. A task vector therefore does not uniquely characterize its originating data, and treating checkpoint geometry as if it faithfully reflected the original task data can misguide fusion. We propose Invariant Task-Core Fusion (ITCF), which builds basis-invariant task cores from checkpoint-identifiable directions and uses core overlap to capture shared support and reconcile expert updates. Across eight vision and language configurations and more than 500 task compositions, ITCF remains competitive across model families and merge scales, including full-model merges of up to 166 experts. Core fusion is roughly two orders of magnitude faster than iterative optimization, with no additional parameters or inference cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model merging, data-free model merging, task vectors, checkpoint-only model merging, task checkpoints, task cores, foundation models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uWGkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the identifiability of task-related information from fine-tuned checkpoints in data-free model merging and proposes Invariant Task-Core Fusion (ITCF). The authors first show in a linear regression setting that different input covariance structures can produce identical final checkpoints, indicating that checkpoints cannot uniquely determine the relative importance of input directions. Motivated by this observation, ITCF constructs isotropic task cores from the dominant right-singular subspaces of task vectors and leverages their overlap to coordinate expert updates. An adaptive spectral regularization scheme further balances joint response matching with summed task updates. The method requires neither task data nor validation-based hyperparameter tuning. Experiments across multiple vision and language models, covering over 500 task compositions and up to 166 experts, demonstrate competitive performance, scalability, and computational efficiency.

Critical strengths
  1. The paper investigates what task-related information can be reliably inferred from fine-tuned checkpoints. By characterizing the ambiguity of input covariance structures and deriving a minimax-optimal comparison geometry under the considered uncertainty, it provides an interesting theoretical perspective on data-free model merging and motivates the proposed isotropic task-core construction.

  2. The proposed method combines task-core overlap with an adaptive spectral recovery scheme, requiring neither task data nor validation-based tuning. Experiments across multiple vision and language model families and up to 166 experts demonstrate competitive performance against strong baselines. The closed-form construction also offers substantial computational advantages over iterative merging methods, making the approach practically appealing.

Critical weaknesses
  1. Proposition 1 proves that isotropic task cores minimize worst-case response-comparison distortion under a specific uncertainty set. However, this does not guarantee better multi-expert fusion or downstream performance, as also illustrated by the two-expert counterexample in Appendix B.3. The ablation results provide limited support for the practical benefit of isotropic weighting. In Table 4, ISO outperforms SV2 by only 0.052 percentage points (95% CI: [-0.155, +0.260]). ISO also performs worse on RoBERTa (-0.213 pp) and Qwen2.5-1.5B (81.82 vs. 81.91; Table 31). In contrast, the proposed recovery scheme improves over ordinary ridge by 0.277 pp when ISO is fixed. These results leave the contribution of the minimax-derived geometry to downstream performance unclear.

  2. Although the paper discusses Iso-C (Marczak et al., 2025), it does not compare ITCF with either Iso-C or Iso-CTS. Both methods use task-vector subspaces and isotropic spectral transformations to mitigate task interference. They are therefore particularly relevant baselines. Without comparisons under matched checkpoints, task compositions, and comparable tuning budgets, it is difficult to assess whether ITCF offers a meaningful improvement over existing isotropic merging methods.

Questions for authors

Could the authors analyze whether a lower task-core response-matching loss corresponds to better preservation of individual task performance? It would be helpful to examine this relationship under different levels of task-core overlap and disagreement between expert updates. Such evidence would clarify when the proposed minimax-derived geometry is a useful proxy for downstream task preservation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer uWGk

Dear Reviewer uWGk,

Thank you for your constructive review and for recognizing ITCF's theoretical foundation and practical efficiency. We have added full-response analysis, direct Iso-C/Iso-CTS comparisons, and network interventions across task-overlap and update-disagreement regimes.

1. From minimax comparison to a beneficial fusion decision.

Under its stated uncertainty model, Proposition 1 gives the minimax isotropic comparison metric Gi=Pi/riG_i=P_i/r_i on the retained task support. ITCF converts it into participation A=∑iGiA=\sum_i G_i and requested response B=∑iΔiGiB=\sum_i\Delta_i G_i, then computes the actual anchored update Mτ=M0+(B−M0A)QΛ(Λ2+τ2I)−1Q⊤M_\tau=M_0+(B-M_0A)Q\Lambda(\Lambda^2+\tau^2I)^{-1}Q^\top. Its modewise weight λj2/(λj2+τ2)\lambda_j^2/(\lambda_j^2+\tau^2) follows shared requests more strongly along well-supported directions and stays closer to the anchor where support is weak.

Theorem L.3 and Corollary L.4 establish a full-response success criterion for this actual output, allowing calibrated within-core inputs, off-core energy, and cross-covariance (sufficient condition ag−dS>∑iLiag-dS>\sum_i L_i). Importantly, the factorial study also identifies where the empirical gains arise: recovery is +0.277/+0.323 and geometry–recovery interaction +0.161/+0.155 under setting/task weighting, alongside isolated geometry +0.052/−0.039. With the original scalar selector fixed, isotropic geometry additionally improves accuracy in all five ViT task–composition comparisons across three distinct tasks. Retaining the core correction improves task-native scores in 24 of 28 matched full-network interventions (p. 9, line 485; p. 75, lines 4008–4009).

Thus, the minimax geometry supplies a principled comparison structure, while anchored recovery and its interaction with that geometry deliver the principal pooled benefits of the actual fusion rule.

(p. 4, lines 188–215; p. 5, lines 218–260; p. 60, lines 3204–3222; p. 66, lines 3530–3555; p. 67, lines 3602–3617; p. 79, lines 4238–4254.)

2. Direct Iso-C and Iso-CTS comparisons.

We evaluated 22 matched task compositions and five training seeds under three validation-selection tracks. In the particularly informative Track B, the original checkpoint-only ITCF uses zero validation candidates while Iso-C and Iso-CTS receive 30 each; ITCF still leads by +0.568 (paired 95% CI [+0.511, +0.624]) and +0.350 ([+0.302, +0.397]) scaled mixed-native macro points, with 5/5 positive paired seed aggregates against each baseline.

Under matched 30/30/30 budgets in Track C, validation-selected ITCF leads by +0.753/+0.536 points, again 5/5 positive seed aggregates and 18/22 favorable task-composition means against each comparator.

ITCF leads both tuned isotropic baselines without validation and remains ahead under equal tuning budgets.

(p. 72, lines 3882–3887; p. 73, lines 3889–3918.)

3. Core response matching and preservation of individual-task ability.

Separately from the original-core retention experiment in Point 1, we compared controlled withdrawals of the task-support direction in 13 task contexts under two fixed network backgrounds, while holding expert untargeted parameters, task heads, inputs, and complementary directions fixed within each pair.

The measured core response-matching loss decreases in 26/26 background-conditioned comparisons; gold-label CE decreases in 19/26, and the task-specific readout improves in 17/26 (four ties, five decreases). By background, the matching-loss/CE co-improvement occurs in 9/13 (B0) and 10/13 (B1).

For RoBERTa CoLA, CE falls by 0.575/0.713 nats per example and MCC improves by 0.421/0.289; for ViT Cars, CE falls by 0.226/0.244 with accuracy gains of 6.25/7.81 percentage points. Classification uses native task metrics; T5 uses fixed gold-prefix token readouts.

We further stratified these same comparisons by projector overlap tr⁡(GiGj)\operatorname{tr}(G_iG_j) and task-projected own/foreign update-direction disagreement Cdir=1−cos⁡(⋅,⋅)C_{\mathrm{dir}}=1-\cos(\cdot,\cdot), using within-family median strata. In the sum-anchor background, ViT's CE improvements span 2/2 low, 1/1 median, and 2/2 high-overlap cases; RoBERTa's high-disagreement group improves in 2/2 cases under each background.

These controlled withdrawals reveal a direct empirical relationship between task-core response matching and individual-task outcomes across distinct overlap and update-disagreement regimes. Together with the original-core retention interventions in Point 1, they provide complementary functional evidence for ITCF's task-core construction.

(p. 77, lines 4153–4157; p. 78, lines 4158–4209.)

We have uploaded the revised PDF, incorporating the additional theoretical results, experimental evidence, and clarifications discussed above. We sincerely hope these additions address your concerns and further demonstrate the scientific contribution of ITCF. Thank you again for your thoughtful feedback and consideration.

AuthorsOpenReview ↗

To Reviewer uWGk

Dear Reviewer uWGk,

Thank you again for your thoughtful review and valuable suggestions.

We have submitted our detailed responses and uploaded the revised manuscript, incorporating additional theoretical results and experimental evidence addressing your concerns, including the matched-budget Iso-C/Iso-CTS comparisons and complete-network functional interventions.

As the discussion period draws to a close, we would greatly appreciate it if you could review these additions and share any remaining questions or concerns.

We would be happy to provide further clarifications during the remaining discussion period.

Thank you very much for your time and consideration.

Best regards, The Authors

Reviewer_uWGkOpenReview ↗

Response to Authors

Thank you for the detailed rebuttal and supplementary experiments. The matched Iso-C/Iso-CTS comparisons and full-network interventions substantially address my earlier concerns and strengthen the empirical validation of ITCF. However, the isolated downstream benefit of the theoretically motivated isotropic geometry remains limited, while the recovery scheme and its interaction with geometry show clearer gains. Overall, I appreciate the additional evidence but maintain my rating of 3 (Weak Acceptance).

AuthorsOpenReview ↗

To Reviewer uWGk

Dear Reviewer uWGk,

Thank you very much for your thoughtful reassessment and constructive feedback.

We sincerely appreciate your recognition that the additional Iso-C/Iso-CTS comparisons and full-network interventions have substantially addressed your earlier concerns and strengthened the empirical validation of ITCF.

We also appreciate your thoughtful observations regarding the respective roles of isotropic geometry and recovery. Your feedback has been valuable in helping us further refine our understanding and presentation of the method.

Thank you again for your time, careful consideration, and positive assessment of our work.

Best regards, The Authors

Reviewer_XWjpOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates what task-specific information can be recovered from pretrained and fine-tuned checkpoints when the original training data are unavailable. Through a theoretical analysis of centered linear regression, it establishes that checkpoints identify the input directions affected by fine-tuning but cannot uniquely determine the relative importance of these directions. Based on this information boundary, the authors propose Invariant Task-Core Fusion (ITCF), which constructs isotropic task cores from retained right-singular subspaces and combines overlapping expert supports through a regularized fusion procedure. The resulting update interpolates between a summed-update reference and a joint response-matching solution according to the participation spectrum. Experiments cover vision, encoder and encoder-decoder language models, multilingual translation, and full-model merging of up to 166 experts, together with extensive ablation, stability, and efficiency analyses.

Critical strengths
  1. The paper formulates an important theoretical question—whether final model weights can uniquely reveal the input-related information needed for data-free model merging—and, rather than assuming that task-vector geometry directly represents the original task distribution, the authors explicitly characterize the uncertainty remaining after observing the final checkpoint. Theorem 1 (Sec. 3.2, p. 3) establishes that distinct input covariances can produce identical endpoint updates while inducing different normalized geometries within the observed update subspace, and Appendix A.1–A.3 (pp. 15–17) provides explicit constructions for fixed-label ambiguity and profiled checkpoint identification, strengthening the mathematical foundation of the information-boundary claim. Proposition 1 (Sec. 3.3, p. 4) derives the unique isotropic comparison geometry under the specified worst-case response-distortion objective, providing a principled justification for assigning equal strength to retained directions. This is a meaningful theoretical contribution because it distinguishes information that is identifiable from checkpoints from information that remains fundamentally ambiguous, rather than relying on potentially unjustified assumptions about the original training data.
  2. ITCF translates the proposed theoretical perspective into a practical checkpoint-only merging procedure, and its fusion mechanism separates shared support from requested expert updates and introduces regularized recovery to prevent excessive corrections along weakly supported directions. Equations (4)–(13) (pp. 4–5) formulate the participation operator, exact joint fit, and anchored regularized recovery, providing a mathematically interpretable fusion mechanism. Tables 1–3 (pp. 7–8) demonstrate competitive performance across vision and language configurations, with the large-scale T5-base experiment maintaining the best mean performance among the reported methods from 8 to 166 experts. Table 6 (p. 9) and Appendix H.15 (p. 46) provide construction-time comparisons: at 166 experts, ITCF requires 3.19 seconds, compared with 1,154.68 seconds for WUDI300 and 1,179.98 seconds for TSV-Merge under the reported timing protocol. The combination of a data-free construction, interpretable regularization, large-scale evaluation, and relatively low inference overhead makes the framework practically relevant to model merging.
Critical weaknesses
  1. The theoretical guarantees do not sufficiently establish the connection between the linear information boundary and practical neural-network merging: the central theoretical contribution is developed under a controlled linear-regression setting, while the practical method is evaluated on considerably more complex neural architectures, and although the manuscript acknowledges this distinction, the theoretical bridge remains conditional. Theorem 1 (Sec. 3.2, p. 3) establishes the checkpoint information boundary for centered linear regression under specific full-batch gradient-descent assumptions, but the result does not directly establish the same identifiability characterization for general nonlinear neural networks. Appendix G, Equation (106) (p. 33), derives a conditional network-level prediction bound containing separate contributions from task-core-controlled discrepancies, off-core responses, and internal hybrid-network differences, and the manuscript explicitly states that the uniform smallness of these remainder terms in arbitrary deep networks is neither assumed nor established (Appendix G, p. 33). Furthermore, Appendix H.3 (pp. 35–36) shows that some parameters, including embeddings, normalization parameters, and biases, remain untargeted by the construction. Consequently, although the proposed method achieves promising empirical results, the theoretical analysis does not yet establish that the checkpoint-identifiable task-core component adequately captures the dominant source of functional discrepancy in practical neural networks, which limits the explanatory and predictive strength of the theoretical contribution.
  2. The empirical benefits of ITCF are heterogeneous, and the contribution of its central isotropic geometry remains insufficiently isolated: the overall experimental evaluation is extensive, but several results indicate that improvements depend on task composition and model family, and more importantly, the ablation results suggest that the recovery mechanism contributes more consistently to the aggregate performance gains than the isotropic task-core geometry alone. Table 1 (p. 7) reports only a +0.17 percentage-point improvement over TSV-Merge on the Vision32 panel, with a paired 95% confidence interval of [-0.20, +0.55], which does not establish a statistically clear advantage in this comparison. Appendix Table 14 (p. 38) shows that ITCF performs 0.798 percentage points below WUDI on the two-task vision setting, and Appendix Table 15 (p. 38) also reports a 2.594 percentage-point deficit on the held-out RoBERTa-base combinations, indicating that improvements are not uniform across evaluation settings. Table 4 (p. 8) separates geometry and recovery effects: the isolated isotropic-versus-squared-singular-weighting effect is only +0.052 percentage points across the 48-setting panel and is family-dependent, whereas the recovery contribution under isotropic geometry is +0.277 percentage points, and Appendix J (p. 49) further confirms that the geometry–recovery interaction varies across model families. These observations do not invalidate the overall effectiveness of ITCF, but they leave some uncertainty about how much of its practical benefit comes from the theoretically motivated invariant task-core geometry versus the anchored regularized recovery mechanism, and they also suggest that the generality of the reported improvements should be characterized more carefully.
Questions for authors
  1. Can you provide empirical measurements of the different remainder terms in Equation (106), particularly the off-core response, untargeted-parameter contribution, and internal hybrid-network discrepancy, on representative ViT, RoBERTa, and T5 models? This matters because the central theoretical analysis is established in linear regression while the practical method operates on complex neural networks, and the relative magnitude of these remainder terms determines whether the task-core-controlled component provides a meaningful explanation of actual network-level behavior. Please report the relative contributions of the task-core term and the remainder terms across representative settings, and investigate whether the conditional prediction bound is sufficiently informative for actual neural-network evaluations.
  2. How do you explain the negative results on the two-task vision setting and the held-out RoBERTa-base combinations? Can you provide additional uncertainty estimates that account for the overlap among task compositions derived from the same underlying task pools and checkpoints? This matters because a substantial number of evaluated settings are constructed from a limited collection of underlying tasks, and consequently performance improvements across many compositions do not necessarily represent equally many independent task-level validations. Please provide family-level and task-clustered comparisons where feasible, and clarify whether the observed advantages remain robust when accounting for shared source tasks and checkpoint seeds.
  3. Table 4 indicates that the isolated geometry effect is relatively small, whereas the regularized recovery contributes more consistently to aggregate performance. Can you clarify whether the principal practical improvement of ITCF comes from the invariant isotropic task core, the anchored recovery mechanism, or their interaction? This matters because the distinction is important for understanding the scientific contribution of the proposed task-core formulation: the minimax result establishes a theoretically justified comparison rule, but it does not necessarily imply superior downstream performance under every task distribution. Please further discuss the relationship between the theoretical minimax guarantee and the observed family-dependent performance results, particularly whether the isotropic geometry offers measurable benefits beyond those obtained through the regularized recovery mechanism.
3 comments
AuthorsOpenReview ↗

Response to Reviewer XWjp (Part 1/2)

Dear Reviewer XWjp,

Thank you for your detailed and constructive review, and for recognizing the checkpoint-information boundary and the efficiency of ITCF. Following your suggestions, we have developed finite-change analyses for complete nonlinear networks, measured the functional roles of task-core and complementary responses, investigated the highlighted ViT/RoBERTa cases, and examined task dependence and geometry–recovery attribution. We respond to your two critical weaknesses and three questions in two consecutive comments.

1. Connection from checkpoint identifiability to nonlinear-network functionality.

Our new five-state analysis decomposes actual finite network-output changes into supported-core, off-core, untargeted, and signed interaction effects. It further relates finite logit changes to prediction margins and gold-label loss. In complete ViT, RoBERTa, and T5 networks, retaining the original ITCF core correction improves task-native scores in 24/28 matched comparisons (ViT 10/10, RoBERTa 10/14, T5 4/4). Relative to a common sum-anchor reference, the empirically calibrated core-plus-off-core reference bound decreases in 13/13 contexts, while actual endpoint quotient-seminorm discrepancy and gold-label CE each decrease in 11/13.

These interventions demonstrate that checkpoint-derived task cores have direct and measurable functional value in complete nonlinear networks, providing empirical support for the practical significance of ITCF's task-core construction.

(p. 58, lines 3128–3131; p. 59, lines 3132–3178; p. 75, lines 3998–4017 and 4037–4049; p. 76, lines 4073–4091.)

2. Real-network response components and Eq. (106).

We conducted two complementary analyses.

First, the exact finite-output squared-energy decomposition measures core, off-core, untargeted, and signed cross contributions to the observed output change. For RoBERTa QQP, the normalized decomposition yields core, off-core, and untargeted squared-response contributions of 0.188, 0.415, and 0.010, together with a substantial +0.386 signed cross contribution. This illustrates how task-core responses interact with complementary directions to shape complete-network behavior, highlighting the importance of their joint functional effects.

Second, Eq. (106) provides a distinct conditional norm upper bound comprising a core-controlled term, an off-core hybrid response, and an untargeted-parameter response. We evaluated finite-path behavior across 14 complete-network contexts and 1,128 layer-occurrence–task measurements. With common finite calibration, the core-plus-off-core reference bound improves in 13/13 contexts; actual endpoint discrepancy and gold CE improve in 11/13 each, and propagated-path increments improve in 12/13.

The refined groupwise margin analysis further identifies 96/960 strict-margin events, including 74 satisfying all measured finite-path caps. These findings demonstrate that component-wise finite-path analysis provides informative sample-level prediction diagnostics, complementing the general conditional response bound in Eq. (106).

Together, the exact output decomposition, measured finite-path responses, and groupwise prediction diagnostics establish a direct empirical connection between checkpoint-derived task cores and observable nonlinear-network behavior. These results strengthen the theoretical and experimental foundation of ITCF by demonstrating how its task-core construction contributes to actual network responses and prediction outcomes.

(p. 35, lines 1859–1878; p. 59, lines 3142–3161; p. 75, lines 4019–4049; p. 76, lines 4051–4103.)

In Part 2, we address the specific ViT/RoBERTa outcomes, dependence across reused tasks and checkpoints, and the contributions of geometry and recovery.

AuthorsOpenReview ↗

Response to Reviewer XWjp (Part 2/2)

Response to Reviewer XWjp — Part 2/2 (continued).

3. Mechanisms behind the highlighted ViT and RoBERTa outcomes. We conducted matched directional interventions with independently trained diagnostic checkpoints for the relevant task compositions, separating core and complementary effects under fixed inputs and task heads. In the two-task ViT setting, the core direction contributes +17 correct Cars predictions and +3 DTD predictions; the complementary direction contributes −37/−10, with signed interaction accounting for the complete-model behavior. In the RoBERTa ITCF–WUDI direction study (18 task contexts), the K=3/5 core direction increases gold loss in 16 background-conditioned comparisons, whereas the seven K=8 classification tasks show lower CE and higher native scores in 14/14 background-conditioned comparisons. These interventions identify the mechanism behind the review's contrasting outcomes: merged-task performance reflects the joint functional action of the core, complementary directions, and network background. (p. 77, lines 4105–4151.)

4. Statistical dependence from shared tasks and checkpoint banks. We analyzed the 48-setting, 204-task-occurrence factorial using both setting-weighted and source-task-balanced aggregation. Recovery remains positive at +0.277/+0.323 scaled mixed-native points, and the geometry–recovery interaction at +0.161/+0.155, respectively. All 22 whole-source-task deletions retain positive pooled recovery and interaction effects; eligible whole-bank deletions do as well. On a separate 36-setting, five-bank ViT-B/32 panel, the bank-balanced ITCF gain is +0.685 accuracy points, with positive leave-bank [+0.627,+0.769] and leave-source-task [+0.040,+0.763] ranges. The five-seed comparisons use paired seed-level statistics on their corresponding fixed panels. The recovery gain and its constructive interaction with geometry persist under task-balanced aggregation and source-level sensitivity, supporting the observed benefit beyond repeated-composition weighting. (p. 60, lines 3188–3229; p. 70, lines 3768–3777; p. 78, line 4211; p. 79, lines 4212–4219 and 4231–4236.)

5. Isolating invariant geometry, recovery, and interaction. The four-cell, 48-setting factorial separately measures their effects: geometry contributes +0.052/−0.039, recovery +0.277/+0.323, and geometry–recovery interaction +0.161/+0.155 under setting-weighted/task-balanced aggregation. We also held the original supports, sum anchor, and scalar selector (k,τ)(k,\tau) fixed while changing only retained-direction weighting. Isotropic geometry improves accuracy over native magnitude weighting in all five ViT task–composition comparisons (three distinct tasks, one seed), with lower CE in 4/5. Independently, under matched seven-candidate selection and five training seeds, squared recovery exceeds ordinary anchored ridge by +0.109 scaled mixed-native points (paired 95% CI [+0.100,+0.118], 5/5 positive seed-level contrasts). The experiments isolate a direct task-level benefit of invariant geometry and establish recovery plus its constructive geometry interaction as consistent sources of ITCF's aggregate gains. (p. 8, lines 425–430; p. 60, lines 3204–3222; p. 72, lines 3835–3843 and 3869–3871; p. 79, lines 4238–4254.)

We have uploaded the revised PDF, incorporating the additional theoretical results, experimental evidence, and clarifications discussed above.We hope these additions address your concerns and strengthen your assessment of ITCF. If you find the new evidence persuasive, we would sincerely appreciate your consideration of a more favorable rating.

AuthorsOpenReview ↗

To Reviewer XWjp

Dear Reviewer XWjp,

Thank you again for your careful review and constructive feedback.

We have provided detailed point-by-point responses to your concerns and uploaded a revised manuscript incorporating additional theoretical analyses, direct functional experiments, expanded baseline comparisons, and further empirical evidence.

As the discussion period is approaching its end, we would greatly appreciate it if you could kindly revisit our responses and the revised manuscript when your schedule permits. We would particularly welcome your assessment of whether the additional results have addressed your original concerns.

We remain very happy to clarify any outstanding questions or provide further explanations during the remaining discussion period.

Thank you very much for your time and thoughtful consideration.

Best regards, The Authors

Reviewer_i6k8OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies data-free model merging using only a pretrained checkpoint and independently fine-tuned expert checkpoints. It argues that a task update identifies the directions on which the update acts, but not the relative input-covariance weights within those directions. Based on a minimax comparison principle, ITCF represents each expert by an isotropic task core constructed from the retained right-singular subspace of its update. It then combines these cores through a participation operator and uses an anchored, participation-dependent regularizer to obtain a closed-form merged update. The broad evaluations and efficient closed-form construction are appealing.

Critical strengths

Strengths 1. Comprehensive and insightful theoretical analysis.

The paper provides a rigorous characterization of what task information can be recovered from fine-tuned checkpoints. In particular, it establishes an important identifiability boundary: checkpoints reveal the input directions involved in parameter updates but cannot uniquely determine their relative importance in the original task distribution. Building on this insight, the paper derives a minimax task-core representation and a regularized fusion framework, providing useful theoretical guidance for data-free model merging.

Strengths 2. Extensive experimental evaluation.

The paper conducts comprehensive experiments across diverse model architectures, task domains, and numbers of experts. The extensive comparisons with existing methods, ablation studies, and large-scale merging experiments provide strong empirical support for the proposed approach.

Critical weaknesses

Weakness 1: The proposed regularization is not sufficiently connected to downstream performance, and its advantage over ordinary L2 regularization is limited

A central component of ITCF is the participation-dependent regularizer

\qquad M_0=\sum_i\Delta_i,\quad A=\sum_i G_i.$$ The paper does provide a mathematical derivation based on minimum-energy synthesis, together with spectral-stability results and supplementary statistical interpretations. My concern is therefore **not** that the term is mathematically unmotivated, but that its relevance to the *actual task-merging objective* remains unclear. In particular, if $$J(M)=\sum_i\operatorname{tr} \left[(M-\Delta_i)G_i(M-\Delta_i)^\top\right]$$ is the proposed core-matching objective, its unregularized minimizer $M^\star$ necessarily satisfies $$J(M^\star)\leq J(M_\tau).$$ Thus, the regularizer trades off fidelity to the paper's own response-matching surrogate against a smaller, more stable correction from the summed-update anchor. The bounds on inverse amplification explain why this can improve numerical robustness, but they do not establish when that trade-off reduces *true downstream prediction error*. The conditional prediction analysis in Appendix G includes off-core and other residual terms and does not show that $M_\tau$ outperforms $M^\star$ in downstream risk. A clearer link between participation strength, correction uncertainty, and downstream loss would materially strengthen the theoretical justification for the regularizer and its specific spectral weighting. The empirical evidence for the more elaborate regularizer over an ordinary **anchored L2** penalty is also relatively modest. In the controlled comparison in Table 4 and Appendix J, the reported mean improvement over ordinary anchored ridge is **+0.277 percentage points** across 48 settings (95% paired confidence interval **[+0.113, +0.450]**). The effect varies by model family: **+0.221** on vision, **+0.675** on RoBERTa, and **−0.008** on T5. The aggregate difference is statistically supported, but its magnitude is small and it does not consistently favor ITCF across families. Moreover, this comparison uses the paper's checkpoint-only selection rules rather than a comprehensive, matched-budget comparison against tuned L2 regularization. **Requested clarification:** Can the authors characterize conditions under which participation-dependent shrinkage improves downstream risk over unregularized core matching or ordinary anchored L2 regularization? A controlled comparison across regularization strengths, with identical selection budgets and any downstream-tuned oracle clearly separated from the data-free protocol, would help establish whether the proposed geometry-dependent penalty provides a substantive benefit. ## Weakness 2: Insufficient justification of task-core subspace selection and limited applicability to low-rank merging A fundamental concern is whether the retained task-core subspaces adequately capture the input directions necessary for preserving downstream task performance. Proposition 1 establishes that the top-$r$ right singular vectors optimally approximate the task update in Frobenius norm. However, **optimal compression of parameter updates does not necessarily imply preservation of task-relevant input directions**, since the update geometry need not align with the underlying task input covariance. This raises a structural limitation. When all task updates have full column rank and their entire right singular spaces are retained, ITCF reduces to ordinary weight averaging. Conversely, truncating updates to lower-dimensional task cores implicitly assumes that the retained directions are sufficiently representative of the original task-response geometry. The paper does not establish the conditions under which this assumption is justified. More importantly, even an exact low-rank representation of each task update does not eliminate this issue. Consider the following simple linear regression counterexample. Let $W_0\in\mathbb{R}^{2\times 2}$ be an arbitrary pretrained initialization. Consider two tasks sharing the input distribution $x\sim\mathcal{N}(0,I_2)$, with noiseless labels generated by $$y_i=(W_0+\Delta_i)x,$$ where $$\Delta_1=\begin{pmatrix}1&0\\\0&0\end{pmatrix}, \\qquad \Delta_2=\begin{pmatrix}0&0\\\0&1\end{pmatrix}.$$ Assume independent fine-tuning exactly recovers $W_i=W_0+\Delta_i$. Each expert thus achieves zero prediction loss on its own task. Both updates are rank one, and the task cores can represent them exactly, without any truncation error. In particular, $$G_1=\begin{pmatrix}1&0\\\0&0\end{pmatrix}, \qquad G_2=\begin{pmatrix}0&0\\\0&1\end{pmatrix}.$$ Under ITCF, the summed-update anchor, participation operator, and requested-response operator are $$M_0=\Delta_1+\Delta_2=I_2,$$ $$A=G_1+G_2=I_2, \qquad B=\Delta_1G_1+\Delta_2G_2=I_2.$$ Because $B-M_0A=0$, the regularized recovery formula $$M_\tau=M_0+(B-M_0A)Q\Lambda(\Lambda^2+\tau^2 I)^{-1}Q^\top$$ returns $M_{\mathrm{ITCF}}=I_2$ for **any** regularization strength $\tau$. Its task-core matching objective is exactly zero: $$\begin{aligned} J(M_{\mathrm{ITCF}}) &=\sum_{i=1}^2 \operatorname{tr}\left[ (M_{\mathrm{ITCF}}-\Delta_i)G_i (M_{\mathrm{ITCF}}-\Delta_i)^\top \right]\\\\ &=0. \end{aligned}$$ However, both tasks have *full-rank* input covariance. Their true aggregate squared prediction loss is $$\mathcal{R}(M) = \sum_{i=1}^{2} \mathbb{E}_{x}[\Vert (W_0+M)x-y_i \Vert_2^2]$$ $$\mathcal{R}(M) = \sum_{i=1}^{2} \mathbb{E}_{x}[\Vert (M-\Delta_i)x \Vert_2^2]$$ $$\mathcal{R}(M) = \sum_{i=1}^{2} \Vert M-\Delta_i \Vert_F^2$$ For the ITCF solution, $$\mathcal{R}(M_{\mathrm{ITCF}})=\mathcal{R}(I_2)=2.$$ By contrast, minimizing the true aggregate prediction loss yields $$M^\star=\frac{\Delta_1+\Delta_2}{2}=\frac{I_2}{2}, \qquad \mathcal{R}(M^\star)=1.$$ Thus, **even with exact experts, exact low-rank task-core representations, and zero core-matching loss, the resulting merged model can be strictly suboptimal for the original tasks**. In this example, ordinary weight averaging halves the true aggregate squared prediction loss. The failure occurs because a task's update subspace describes where fine-tuning changed its parameters, not every input direction on which its predictions must be preserved: the second expert modifies a direction outside the first expert's core, and vice versa. Those cross-expert changes are invisible to the respective core-matching terms. The gap need not be confined to two tasks. For $n\ge 2$ tasks with $x\sim\mathcal{N}(0,I_n)$ and updates $\Delta_i=e_ie_i^\top$, $i=1,\ldots,n$, each update and its retained rank-one core are exact. ITCF gives $M_{\mathrm{ITCF}}=I_n$ with $J(M_{\mathrm{ITCF}})=0$, whereas the true aggregate response loss satisfies $$\mathcal{R}(I_n)=n(n-1), \qquad \min_M\mathcal{R}(M)=\mathcal{R}(I_n/n)=n-1.$$ Consequently, the ratio of these losses is $n$, showing that there is no uniform task-count-independent approximation factor for the **true aggregate response loss** over this class of linear examples. This does not by itself imply an analogous ratio for real-world classification accuracy; rather, it exhibits a concrete gap between the proposed surrogate and the original task-response objective. This limitation is especially relevant to **LoRA merging**, where the rank of each update is explicitly constrained, while the task input covariance may remain high-rank. Although Appendix G includes an off-core response term, the proposed matching objective does not directly control it, and the paper does not establish conditions ensuring that this remainder is negligible. The current experimental evidence also does not systematically assess LoRA adapter merging across ranks. A stronger theoretical characterization of the alignment needed between task-update subspaces and task-input geometry, together with LoRA-rank and off-core interference experiments, would clarify the conditions under which task-core matching is a reliable surrogate for preserving downstream performance. ## Suggestion: Improve readability with a simpler linear example and a notation table The theoretical presentation is technically detailed, but its central intuition is harder to follow than necessary. **For readability, I suggest introducing checkpoint non-identifiability with a short, self-contained linear interpolation example before presenting the more elaborate finite-step full-batch gradient-descent analysis.** This would let readers understand exactly *which information is lost in a checkpoint* before they have to follow the training-dynamics construction. It would also make the transition from the information-boundary argument to the isotropic task-core design more intuitive. The stronger finite-step GD result remains valuable, but could be summarized in the main text and proved in the appendix, rather than carrying most of the introductory explanation. For example, the same intuition can be shown without assuming zero initialization. Let $W_0$ be arbitrary, let $Y=W^\star X$ be noiseless training labels, and consider the interpolating solution closest to $W_0$: $$W_T=\arg\min_{W:\,WX=Y}\|W-W_0\|_F^2 =W_0+(W^\star-W_0)XX^\dagger.$$ Writing the compact SVD $X=U\Sigma V^\top$ gives $$\Delta=W_T-W_0=(W^\star-W_0)UU^\top,$$ which **does not depend on the positive singular values** in $\Sigma$. Yet the normalized empirical input covariance is $$C_X=\frac{XX^\top}{\operatorname{tr}(XX^\top)} =U\frac{\Sigma^2}{\operatorname{tr}(\Sigma^2)}U^\top.$$ Keeping $U$, $W_0$, and $W^\star$ fixed while changing the positive singular values therefore produces the **same final checkpoint but different within-subspace covariance geometries**. If $(W^\star-W_0)U$ has full column rank, the row space of $\Delta$ even equals $\operatorname{span}(U)$, so the ambiguity persists when the update identifies the entire sample-input subspace. This concise example makes the information boundary immediate and clarifies why the unobserved singular-value weights matter for response matching. The finite-step result can then be presented as a stronger extension of this basic observation. **I also recommend adding a consolidated notation table** near the beginning of the paper Appendix. The manuscript introduces many closely related matrices, operators, subspaces, ranks, and spectral quantities across Sections 3–4 and the appendices. A table defining, for example, $W_0$, $W_i$, $\Delta_i$, $M$ versus $M_0$, $Q_i$, $P_i$, $G_i$, $r_i$, $A$, $B$, $Q$, $\Lambda$, $s$, $k$, $\tau$, and $J(M)$ would reduce the need to repeatedly search for definitions. In particular, distinguishing the **pretrained weights, individual task updates, summed-update anchor, and final merged update** would substantially improve readability and make the theoretical development easier to follow without changing its technical content. ## Missing Related Work Several relevant studies are missing from the related work. The authors are encouraged to consider the following references, grouped by their relevance to ITCF. **1. Closely Related Model Merging Methods** - MaTS (Tam et al., 2024): Task parameter subspace matching and linear-system formulations for model merging. - DC-Merge (Zhang et al., 2026): Singular-value smoothing and subspace alignment for reducing merging interference. - FDA (Shi et al., 2025): Checkpoint-derived synthetic inputs for functional response matching. - TALL-Masks and Consensus Merging (Wang et al., 2024): Identifying task-specific parameter supports and mitigating interference. **2. Low-Rank and LoRA Model Merging** - KnOTS (Stoica et al., 2025): SVD-based subspace alignment for LoRA merging. - Core Space Merging (Panariello et al., 2025): Efficient low-rank merging in a shared subspace. - OSRM (Zhang and Zhou, 2025): Task-input distributions, orthogonal LoRA subspaces, and interference reduction. - DO-Merging (Zheng et al., 2026): Magnitude-direction decoupling and orthogonalization for data-free LoRA merging. **3. Foundational Methods, Evaluation, and Scalability** - Model Soups (Wortsman et al., 2022): Weight averaging as a foundational model merging approach. - Realistic Evaluation of Model Merging (Tam et al., 2024): Evaluation protocols, computational costs, and practical merging considerations. - Why Do More Experts Fail? (Wang et al., 2026): Theoretical limitations of large-scale expert merging. Discussing these works would help clarify the novelty and applicability of ITCF, particularly its task-core construction, spectral treatment, and behavior in low-rank merging settings. ## Reference 1. Tam et al. (2024). [Merging by Matching Models in Task Parameter Subspaces](https://arxiv.org/abs/2312.04339). TMLR. 2. Zhang et al. (2026). [DC-Merge: Improving Model Merging with Directional Consistency](https://arxiv.org/abs/2603.06242). CVPR. 3. Shi et al. (2025). [Model Merging with Functional Dual Anchors](https://arxiv.org/abs/2510.21223). arXiv. 4. Wang et al. (2024). [Localizing Task Information for Improved Model Merging and Compression](https://proceedings.mlr.press/v235/wang24k.html). ICML. 5. Stoica et al. (2025). [Model Merging with SVD to Tie the Knots](https://proceedings.iclr.cc/paper_files/paper/2025/hash/0d4f8a5109c5083b5307fcd0bddae7a7-Abstract-Conference.html). ICLR. 6. Panariello et al. (2025). [Accurate and Efficient Low-Rank Model Merging in Core Space](https://arxiv.org/abs/2509.17786). NeurIPS. 7. Zhang and Zhou (2025). [Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging](https://aclanthology.org/2025.acl-long.1284/). ACL. 8. Wortsman et al. (2022). [Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy without Increasing Inference Time](https://proceedings.mlr.press/v162/wortsman22a.html). ICML. 9. Tam et al. (2026). [Realistic Evaluation of Model Merging for Compositional Generalization](https://arxiv.org/abs/2409.18314). TMLR. 10. Wang et al. (2026). [Why Do More Experts Fail? A Theoretical Analysis of Model Merging](https://aclanthology.org/2026.acl-long.2108/). ACL. 11. Zheng et al. (2026). [Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging](https://aclanthology.org/2026.findings-emnlp.1516/). Findings of EMNLP.
Questions for authors

See my weakness.

6 comments
AuthorsOpenReview ↗

Response to Reviewer i6k8 (Part 1/2)

Dear Reviewer i6k8,

Thank you for your thoughtful and constructive review and for recognizing the theoretical insight of checkpoint identifiability and the breadth of our evaluation. Following your suggestions, we have developed new prediction-risk guarantees for participation-aware recovery, conducted five-seed ridge comparisons with matched validation budgets, derived full-response conditions for low-rank task cores, and measured core corrections in complete neural networks. We have also clarified the full-support limit, adapter-rank geometry, notation, and related work. We address your two critical weaknesses, readability suggestions, and related-work concerns in two consecutive comments.

1. Prediction benefit of participation-dependent regularization. The actual ITCF recovery filter is hj=λj2/(λj2+τ2)h_j=\lambda_j^2/(\lambda_j^2+\tau^2), placing strongly shared directions closer to the fitted task requests and weakly shared directions closer to the sum anchor. The review's inequality J(M∗)≤J(Mτ)J(M^*)\leq J(M_\tau) compares retained-core fitting, whereas downstream risk also depends on recovery uncertainty and task-input geometry. Our new Theorem K.4 analyzes this distinction under the conditional second-moment model (K.9), including a zero signal–innovation cross moment, positive-risk modes, 1≤k<s1\leq k<s, and the innovation-to-signal ratios ρj=νj/αj\rho_j=\nu_j/\alpha_j in [λk+12,λk2][\lambda_{k+1}^2,\lambda_k^2]. Under these conditions, ITCF's original checkpoint-selected τ2=λkλk+1\tau^2=\lambda_k\lambda_{k+1} minimizes the worst-case normalized modewise prediction-risk ratio, with sharp factor 1+[(λk−λk+1)/(λk+λk+1)]2<21+[(\lambda_k-\lambda_{k+1})/(\lambda_k+\lambda_{k+1})]^2<2. Corollary K.5 establishes a strictly larger worst-case factor for scalar ordinary-ridge filters when the boundary eigenvalues differ; Lemma L.5 separately gives an exact improvement criterion with correlated anchor-relative errors. Thus, ITCF's spectral shrinkage has a precise prediction-risk rationale and a provable advantage in the stated signal–innovation regime. (p. 5, lines 244–261; p. 56, lines 2983–3003 and 3017–3023; p. 57, lines 3024–3072; p. 68, lines 3644–3667.)

2. Fairness against ordinary L2/Ridge regularization. Across 21 task compositions and five training seeds, the original checkpoint-only ITCF, with zero validation candidates, obtains 63.078 scaled mixed-native macro points, compared with 63.076 for validation-tuned ordinary anchored ridge; it exceeds the fixed ordinary-ridge control by +0.876 points. We also isolate the filter family under seven effective-degrees-of-freedom-matched validation candidates per family. The selected squared-recovery variant exceeds selected ordinary ridge by +0.109 points (paired 95% CI [+0.100, +0.118], 5/5 positive seed-level contrasts). These comparisons distinguish the original checkpoint-only decision from the separately tuned filter variant. ITCF attains competitive performance without validation selection, while its squared filter achieves a consistent advantage under equal search budgets. (p. 71, lines 3806–3833; p. 72, lines 3835–3877.)

3. Functional meaning of the task core. We intervened on the original ITCF core correction in complete ViT, RoBERTa, and T5 networks, with matched inputs, task readouts, and untargeted-parameter backgrounds. Retaining this correction improves the native task score in 24/28 comparisons (ViT 10/10, RoBERTa 10/14, T5 4/4). In a separate common-sum-anchor comparison, the controlled-plus-off-core empirical reference bound decreases in 13/13 contexts; the actual endpoint quotient-seminorm discrepancy and gold-label CE each decrease in 11/13. Finite-output decomposition further identifies the supported, off-core, and signed cross-response effects. Theorems K.7 and L.3 provide complementary input-geometric conditions for full-task response improvement, incorporating off-core and cross-response effects (see Point 5). These experiments establish the retained core as a measurable and beneficial functional component of the actual merged network. (p. 53, lines 2845–2861; p. 54, lines 2862–2894; p. 75, lines 3998–4017 and 4037–4049; p. 76, lines 4051–4091.)

4. Full-support limit. When every task retains the entire input space, Pi=InP_i=I_n and A=(K/n)InA=(K/n)I_n. The participation spectrum is flat, the original selector gives τ=0\tau=0, and the actual merger becomes Mτ=K−1∑iΔiM_\tau=K^{-1}\sum_i\Delta_i, i.e., active-expert weight averaging. ITCF therefore recovers the symmetric averaging rule in the full-support limit and adapts it through task participation when supports differ. (p. 55, lines 2945–2947.)

We continue with the low-rank counterexample, LoRA, identifiability, notation, and related work in Part 2.

AuthorsOpenReview ↗

Response to Reviewer i6k8 — Part 2/2 (continued).

  1. Orthogonal low-rank counterexample and full-task response. For the reviewer's orthogonal rank-one checkpoint construction with isotropic full-rank inputs, the exact response calculation gives RITCF=n(n−1)\mathcal R_{\rm ITCF}=n(n-1) versus Ravg=n−1\mathcal R_{\rm avg}=n-1. Proposition K.1 identifies the mechanism through Ri(M)=Ti(M)+Oi(M)+2Ci(M)R_i(M)=T_i(M)+O_i(M)+2C_i(M): off-core input action and signed cross response determine the relation between supported fitting and full-task risk. Keeping the same orthogonal checkpoint family and considering an input-aligned regime, Theorem K.7 proves a strict per-task advantage over averaging when the normalized off-core input mass satisfies εi<(n−1)/(2n)\varepsilon_i<(n-1)/(2n); the original isotropic case has εi=(n−1)/n\varepsilon_i=(n-1)/n. Theorem L.3 and Corollary L.4 extend this insight through calibrated within-core response, off-core energy, and cross-covariance to an explicit full-response success condition for the actual merger. The new analysis characterizes the input geometry that governs ITCF's full-task response advantage, converting the low-rank example into a precise decision criterion. (p. 53, lines 2845–2861; p. 54, lines 2862–2895; p. 62, lines 3307–3327; p. 63, lines 3355–3368; p. 66, lines 3530–3555; p. 67, lines 3602–3617.)

  2. LoRA rank and input geometry. We now distinguish adapter factorization rank, observed update rank, retained task-core rank, and task-input covariance rank. Because ITCF forms its core from the observed checkpoint update, the same checkpoint-level construction applies to rank-constrained adapter updates. The supported/off-core/cross-response decomposition then explicitly characterizes the input action outside the retained update support. This provides a principled account of how LoRA-style rank constraints interact with the task-input geometry relevant to fusion. A systematic empirical study across adapter ranks, adapted modules, and off-core interference is a natural direction for future work. (p. 53, lines 2815–2832 and 2845–2861; p. 54, lines 2862–2893; p. 66, lines 3530–3553.)

  3. Intuitive illustration of the identifiability boundary. Section 3 now introduces the interpolation intuition, with its full worked construction in Appendix K.1. For arbitrary pretrained weights W0W_0 and labels Y=W⋆XY=W^{\star}X, minimum-distance interpolation gives Wint=W0+(W⋆−W0)XX†W_{\rm int}=W_0+(W^{\star}-W_0)XX^\dagger. Varying the relative positive singular values of XX while holding its singular vectors fixed preserves this checkpoint endpoint and can change XX⊤/tr⁡(XX⊤)XX^\top/\operatorname{tr}(XX^\top). The example demonstrates the central reason for ITCF's invariant geometry: a checkpoint reveals the acted-on support, while relative input importance within that support is not identifiable from the endpoint alone. (p. 3, lines 149–161; p. 4, lines 162–199; p. 53, lines 2834–2843.)

  4. Notation and actual fusion decision. We clarify W0W_0 (pretrained weights), Δi\Delta_i (expert update), M0=∑iΔiM_0=\sum_i\Delta_i (sum anchor), M∗M^{\ast} (exact supported fit), and MτM_\tau (actual regularized fusion update), together with GiG_i, the participation matrix AA, the requested response BB, and the retained ranks. Their definitions in the method and appendix separate checkpoint-computable decisions from theoretical comparison quantities. The construction now has an explicit path from identifiable supports to ITCF's implemented fusion rule and its response guarantees. (p. 5, lines 218–261; p. 53, lines 2812–2826; p. 54, lines 2899–2915.)

  5. Related work and closest isotropic baselines. We have incorporated the eleven suggested references and clarified the distinction between checkpoint-identifiable task comparison and existing update transformations. On 22 task compositions and five training seeds, the original ITCF uses zero validation candidates while Iso-C/Iso-CTS each use 30 in Track B, yet ITCF leads by +0.568/+0.350 scaled mixed-native points, with 5/5 positive seed-level differences against each baseline. Under equal 30/30/30 budgets in Track C, validation-selected ITCF retains +0.753/+0.536 advantages. The direct comparison supports both ITCF's distinctive checkpoint-only formulation and its practical competitiveness against closely related isotropic baselines. (p. 2, lines 102–107; p. 3, lines 108–144; p. 72, lines 3882–3887; p. 73, lines 3889–3918.)

We have uploaded the revised PDF, incorporating the additional theoretical results, experimental evidence, and clarifications discussed above. We sincerely hope these additions address your concerns and further demonstrate the scientific contribution of ITCF. Thank you again for your thoughtful feedback and consideration.

The revision addresses several of my original concerns. The interpolation example and identifiability analysis clarify what is recoverable from checkpoints (Sec. 3.2; Appendix K.1); the new prediction-risk analysis gives conditional minimax guarantees for participation-dependent regularization and a comparison with ordinary Ridge (Theorem K.4, Corollary K.5); the supported/off-core/cross-response decomposition and input-geometry conditions strengthen the full-response analysis (Appendix K.2; Theorem K.7; Theorem L.3, Corollary L.4); and the matched-budget Ridge comparisons (Appendix M.2) and complete-network interventions (Appendix M.4) add empirical support for the method and the functional role of task-core corrections.

Two concerns persist. (i) The conditional signal–innovation assumptions underlying the prediction-risk guarantees are not verified for the actual sum-anchor construction, and Eq. (K.17) shows the cross moment can be nonzero. (ii) The input-geometry conditions are assumed rather than identified from checkpoints, and their validity on the evaluated benchmarks is not established (Appendix K.8; Appendix L.3), leaving the link between checkpoint-based rank selection and task-relevant input geometry incomplete. A systematic evaluation across LoRA ranks is also still missing (Appendix K.1).

Overall, considering the substantial improvements in the revision, i believe the paper is good enough for acceptance at ICLR, and I support its acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.