acceptodds
Under review as a conference paper at ICLR 2027

Receiver State Shapes Functional Compatibility: Internal Message Adaptation for Model Stitching

Abstract

Model stitching connects a frozen source model and a frozen target receiver through a learned interface, with functional compatibility commonly assessed by downstream task accuracy. However, downstream accuracy alone does not fully characterize functional compatibility, as the effect of a stitching mismatch can vary sharply with receiver state. In particular, receiver states can have nearly identical unperturbed accuracy yet respond very differently to the same controlled perturbation. Controlled token and attention interventions identify messages from patch tokens to CLS as a key pathway through which stitching mismatch affects downstream function. Guided by this pathway, we introduce RIMA, Receiver Internal Message Adaptation, which learns state conditioned corrections to CLS attention outputs while keeping the pretrained models and stitch frozen. This receiver state dependence is further validated across model families, stitching directions, interface locations, and objectives. We then evaluate RIMA across heterogeneous model pairs and classification benchmarks against conventional stitch optimization and parameter matched receiver adaptation. On DeiT B to DeiT S, RIMA adds 0.02M trainable parameters and improves ImageNet 1K top 1 accuracy from 68.538% to 73.752%. These results indicate that receiver state shapes the functional impact of stitching mismatch, while internal message adaptation can recover degraded performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model Stitching, Functional Compatibility, Representation Alignment, Vision Transformers, Parameter Efficient Adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rFyxOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The paper investigates model stitching and proposes RIMA to improve the stitching performance.

Critical strengths

The experiments might suggest potential benefits of the proposed method for model stitching.

Critical weaknesses
  • This paper is difficult to understand. For example, in line 46, the term “receiver states” appears in the Introduction without definition or explanation. The arrows in RIMA in Figure 1 are unclear, and there is a loop in RIMA. In Section 3.1, it is unclear how the stitch layer is trained and what the loss is. These are examples of the presentation issues.

  • The experimental results are not sufficient to demonstrate the superiority of the proposed method.

    • No ablation study to evaluate the contribution of each component of RIMA, the loss, and the number of RIMA modules.
    • Insufficient comparison with prior work, such as [1].
    • The benefits of RIMA may come from an additional training stage and the additional trainable parameters. What if using the same training strategy to directly finetune the stitching layer? What if introducing LoRA or an adapter to the stitching layer? What if keeping the stitching layer but introducing LoRA or AdaptFormer to the target model?

[1] Revisiting Model Stitching in the Foundation Model Era, CVPR 2026

Questions for authors

Suggestions

  • The paper could refer to the experimental section of prior work on model stitching and better design its experiments.

  • The paper could discuss closely related work, such as [2-5], and also refer to them to better present Section 3.

[2] Locating and Editing Factual Associations in GPT, NeurIPS 2022.

[3] Register and [CLS] Tokens Induce a Decoupling of Local and Global Features in Large ViTs, NeurIPS 2025.

[4] Revisiting [CLS] and Patch Token Interaction in Vision Transformers, ICLR 2026.

[5] Causal Attribution via Activation Patching, arXiv preprint arXiv:2603.13652.

11 comments
AuthorsOpenReview ↗

Response to Reviewer rFyx

Response to Reviewer rFyx (Part 1/5)

Thank you for your detailed review and constructive suggestions. We have revised the presentation of our method, clarified the training objectives, and added extensive experimental comparisons and ablations. We address each of your concerns below.

Key substantive additions and clarifications are marked in dark blue in the revised manuscript. New tables are identified by their blue captions.

A.1.1 Definition of Receiver States

Your comment: "The term 'receiver states' appears in the Introduction without definition or explanation."

Thank you for pointing this out. We have revised the Introduction to clarify the meaning of receiver state at its first substantive use. Specifically, we refer to the intermediate activations entering a block of the frozen target network as the receiver state. This clarification makes the central research question more precise and provides a clearer basis for the controlled interventions in Section 3.

A.1.2 Clarity of Figure 1

Your comment: "The arrows in RIMA in Figure 1 are unclear, and there is a loop in RIMA."

Thank you for pointing out the ambiguity in Figure 1(b). We have redrawn the figure to clarify the direction of information flow and the role of RIMA within the receiver.

The enlarged diagram illustrates a single receiver block equipped with RIMA. The current CLS query and patch summaries are used to predict a correction, which is added to the CLS attention output before subsequent receiver processing. The apparent loop represents the standard Transformer residual connection rather than recurrent feedback through RIMA.

We have also revised the figure caption to clarify that separate RIMA modules are applied to the last four available receiver blocks.

A.1.3 Stitching Layer Training and Objective

Your comment: "In Section 3.1, it is unclear how the stitch layer is trained and what the loss is."

Thank you for pointing out this missing detail. We have revised Section 3.1 to explicitly describe the stitching objective and training procedure.

For the diagnostic experiments in Section 3.1, we train the affine stitching layer using relative mean squared error (MSE) between the stitched token representations and their corresponding native target representations. The all-token objective supervises both CLS and patch tokens, whereas the CLS-only objective supervises only the CLS token. In both cases, the full stitched token sequence enters the receiver; the objective names do not describe which tokens are passed forward. The source model, target receiver, and classifier remain frozen.

RIMA is trained in a separate stage with the learned stitching layer frozen. Its training objective combines cross entropy (CE) on the task labels with GlobalKD, which matches the centered output logits of the stitched model to those of the frozen native target model. In the Section 3 diagnostics, relative token MSE trains the affine interface. In the subsequent RIMA stage, that interface is frozen while CE and GlobalKD optimize the receiver corrections.

We have clarified the stitching objective in Section 3.1, while the RIMA objective and training protocol are described in Section 4.2 and Appendix A.

AuthorsOpenReview ↗

Response to Reviewer rFyx

Response to Reviewer rFyx (Part 2/5)

A.2. Insufficient Experimental Evidence

Your comment: "The experimental results are not sufficient to demonstrate the superiority of the proposed method."

Thank you for raising this concern. We have strengthened the experimental evaluation with additional comparisons against alternative adaptation methods. The evidence now covers three complementary aspects.

First, in the original 38-setting evaluation across multiple model families and classification datasets, RIMA improves traditional stitching by 40.01 percentage points on average and exceeds the interface baseline with a matched parameter budget by 8.32 points. The paired 95% confidence interval for the RIMA advantage has a positive lower bound in every evaluated condition (Section 5.4 and Appendix C.3).

Second, we have added comparisons with Receiver LoRA, Interface LoRA, and Receiver Bottleneck under approximately matched parameter budgets. Across four model pairs on Flowers102 and CIFAR-100, RIMA achieves the highest mean accuracy in every setting, exceeding the strongest alternative by 2.66 to 5.16 percentage points. Each configuration is evaluated over three independent seeds (Table 4 and Appendix C.8).

Third, the original full ImageNet-1K validation results provide evidence at larger evaluation scale. RIMA improves DeiT-B to DeiT-S stitching from 68.538% to 73.752% (+5.214 pp) and ViT-B to DeiT-S stitching from 57.000% to 69.658% (+12.658 pp).

Together, these results provide evidence that RIMA improves functional compatibility across different stitching settings and offers advantages over the evaluated interface and receiver adaptation baselines.

A.3. Missing Ablation Studies

Your comment: "No ablation study to evaluate the contribution of each component of RIMA, the loss, and the number of RIMA modules."

Thank you for this suggestion. We have added three sets of ablation experiments to Appendix E, examining the conditioning features, training objectives, and number of RIMA modules. All experiments use ViT-S/16 to DeiT-S/16 stitching on Flowers102, with 5,000 optimizer steps and three independent seeds. Results are reported as mean ± sample standard deviation.

(1) Contribution of conditioning features.

We zero-mask each conditioning feature individually while retaining the full 256-dimensional input and nominal 20,800-parameter RIMA architecture. The CLS-query-only variant uses the input [q_cls, 0, 0, 0].

ConfigurationTop-1 Accuracy (%)
Full RIMA79.93 ± 0.34
Without CLS query74.77 ± 0.79
Without patch context74.71 ± 0.88
Without mean patch key73.07 ± 0.32
Without mean patch value73.95 ± 0.74
CLS query only72.32 ± 0.82

Removing any individual feature reduces accuracy by 5.16 to 6.86 percentage points. Using only the CLS query reduces accuracy by 7.61 points. These results demonstrate the contribution of the four conditioning features to RIMA's performance.

(2) Contribution of training objectives.

We separately evaluate cross entropy (CE), GlobalKD, and their combination.

Training ObjectiveTop-1 Accuracy (%)
CE + GlobalKD79.93 ± 0.34
CE only78.01 ± 0.32
GlobalKD only77.29 ± 0.50

The combined objective outperforms CE alone by 1.93 points and GlobalKD alone by 2.65 points, calculated from unrounded accuracies. Both terms therefore contribute to the final performance in this setting.

(3) Number of RIMA modules.

We vary the number of adapted receiver blocks while keeping the bottleneck dimension fixed at r = 16.

ModulesAdapted BlocksParametersTop-1 Accuracy (%)
1125,20070.39 ± 1.58
211-1210,40075.62 ± 0.25
310-1215,60078.59 ± 0.64
49-1220,80079.93 ± 0.34

Accuracy increases from 70.39% with one module to 79.93% with four modules. The four-module configuration achieves the highest mean accuracy among the evaluated settings.

These three ablations directly address the requested component, loss, and module count analyses. We have included the complete results and experimental details in Appendix E (Tables 26, 27, and 28), "Ablations of RIMA Components, Training Losses, and Module Count," and added a reference to these experiments in Section 4.2.

AuthorsOpenReview ↗

Response to Reviewer rFyx

Response to Reviewer rFyx (Part 3/5)

A.4. Comparison with Prior Work

Your comment: "Insufficient comparison with prior work, such as [1]."

Thank you for highlighting the importance of this comparison. We have added experiments based on the two-stage interface optimization strategy of Mai et al. (CVPR 2026), which uses final feature matching (FFM) followed by task loss training (TLT).

We adapt this strategy to four model pairs across Flowers102 and CIFAR-100. During FFM, we match the final pooled CLS features after normalization and before the frozen classifier, using MSE for 5,000 steps. TLT then continues from the FFM checkpoint for 5,000 steps using CE only. The interface is a two-layer ReLU MLP applied to each token, with hidden width equal to the target feature dimension. The source, receiver, and classification head remain frozen.

We use the name Adapted Mai MLP deliberately: the hidden width is our implementation choice, and the training budget, model pairs, and evaluation protocol differ from the original work. We implemented the two-stage procedure independently from the method described in Mai et al.

We also evaluate a LayerNorm-plus-linear interface with TLT using either CE or CE + GlobalKD after FFM initialization. These modified affine variants are our extensions.

All results are validation top-1 accuracy (%), reported as mean ± sample standard deviation over three independent seeds.

The settings are C1: ViT-S → DeiT-S on Flowers102; C2: MAE-B → CLIP-B/16 on Flowers102; C3: CLIP-B/16 → DeiT-B on Flowers102; and C4: ViT-B → ViT-S on CIFAR-100.

SettingAdapted Mai MLPModified affine, CEModified affine, CE+KDRIMA
C178.20 ± 0.9776.96 ± 0.2669.08 ± 0.2879.93 ± 0.34
C278.99 ± 1.0382.39 ± 0.7180.75 ± 0.2588.46 ± 0.25
C387.58 ± 0.6489.12 ± 0.6986.83 ± 0.3290.16 ± 0.59
C471.63 ± 0.2872.75 ± 0.0068.30 ± 0.3072.15 ± 0.31

RIMA achieves higher mean accuracy than Adapted Mai MLP in all four settings, with improvements of 1.73, 9.48, 2.58, and 0.52 percentage points, respectively.

Among all four evaluated methods, RIMA achieves the highest mean accuracy in C1, C2, and C3. In C4, the modified affine interface trained with CE achieves 72.75%, compared with 72.15% for RIMA.

These experiments provide a direct comparison with an adapted implementation of Mai et al.'s training strategy under our model pairs and datasets. We have added the experimental protocol, parameter counts, and complete results to Appendix C.9, "Comparison with Adapted Mai Interface Optimization" (Table 24).

A.5.1 Additional Training Stage and Parameter Budget

Your comment: "The benefits of RIMA may come from an additional training stage and the additional trainable parameters."

Thank you for raising this concern. We address it through a controlled comparison with an interface adaptation baseline that receives the same additional training stage, supervision, and optimization schedule as RIMA.

Both methods start from the same trained stitching interface. RIMA adds 20,800 trainable parameters inside the receiver, while the interface baseline updates 20,763 or 20,790 parameters of the existing stitching layer, depending on the model configuration. Both use the same training data, task labels, native target distillation, and evaluation protocol.

Across 38 evaluated conditions, RIMA exceeds the parameter matched interface baseline by 8.32 percentage points on average, with a positive lower bound for the paired 95% confidence interval in every condition.

These results show that the additional training stage and trainable parameter budget alone do not account for RIMA's advantage over the evaluated interface adaptation baseline. The comparison is reported in Section 5.4, Table 3, and Appendix C.3, with implementation details in Appendix A.

Reviewer_xmkuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines the influence of receiver states on functional compatibility in model stitching. Through token swaps and attention interventions, the authors show that states with similar clean accuracy can respond very differently to the same perturbation and identify patch-to-CLS messages as an important pathway. These observations motivate Receiver Internal Message Adaptation (RIMA), which learns lightweight, state-conditioned residual corrections inside the receiver while freezing the pretrained models, stitch, and task head. Experiments across multiple model families and classification datasets, including full ImageNet-1K, show consistent improvements over the original stitches and the evaluated parameter-matched interface baseline, with the main configuration requiring only 20,800 trainable parameters.

Critical strengths

1, The motivation and method are well connected: receiver-side interventions directly inform the correction location and conditioning information.

2, Token swaps and selective key/value replacements provide stronger mechanistic evidence than representation similarity alone.

3, Figures 2–3 clearly communicate the intervention logic, and the statistical reporting distinguishes evaluation uncertainty from training-seed variation.

Critical weaknesses

1, The interface baseline updates only selected rows of an affine stitch. Low-rank and nonlinear interface adapters would better isolate the advantage of receiver-side adaptation.

2, Direct ablations of state conditioning, individual summary features, and adaptation location are missing, leaving the necessity of these design choices unclear.

3, Receiver states and perturbations are constructed from related reference trajectories. Independent perturbations and naturally occurring stitched states would strengthen the generality of the analysis.

4, RIMA receives additional labeled training and distillation, while the native target remains frozen. A matched native adapter and separate supervision ablations are needed to interpret the native-target comparisons.

Questions for authors

see weaknesses

6 comments
AuthorsOpenReview ↗

Response to Reviewer xmku

Response to Reviewer xmku (Part 1/4)

Thank you for your careful review and constructive suggestions. We appreciate your positive assessment of the connection between our diagnostic findings and the proposed adaptation mechanism, as well as your recognition of the mechanistic value of the token and attention interventions. Your comments have helped us strengthen the experimental controls and clarify the scope of our conclusions. We have added comparisons with low-rank and nonlinear interface adaptation, direct ablations of RIMA's design choices, supplementary perturbation diagnostics, and a parameter-matched native-target adaptation control. We address each weakness below.

Key substantive additions and clarifications are marked in dark blue in the revised manuscript. New tables are identified by their blue captions.

W1. Stronger Interface Adaptation Baselines

Your comment: "The interface baseline updates only selected rows of an affine stitch. Low-rank and nonlinear interface adapters would better isolate the advantage of receiver-side adaptation."

We added Interface LoRA and a Nonlinear Interface Adapter alongside the original Partial Stitch baseline to compare different interface-side adaptation mechanisms. Interface LoRA adds a low-rank linear term to the frozen parent projection. The Nonlinear Interface Adapter adds a 384 → 26 → 384 GELU branch driven by the parent's frozen LayerNorm output; it is a residual correction, not a replacement nonlinear stitch. Its output layer is zero-initialized, preserving the parent prediction at initialization.

We conduct a controlled comparison in C1 (ViT-S/16 to DeiT-S/16 on Flowers102), using the same trained parent stitch, task supervision, 5,000 adaptation steps, and evaluation protocol. All four methods are evaluated over three independent seeds, and we report mean accuracy and sample standard deviation.

The results are:

MethodTrainable parametersTop-1 accuracy (%)
Full RIMA20,80079.93 ± 0.34
Interface LoRA20,73674.77 ± 0.40
Nonlinear Interface Adapter20,37875.49 ± 0.26
Partial Stitch20,79072.88 ± 0.40

RIMA exceeds Interface LoRA, the Nonlinear Interface Adapter, and Partial Stitch by 5.16, 4.44, and 7.06 percentage points, respectively. The trainable parameter budgets are approximately matched, ranging from 20,378 to 20,800 parameters. Gains are computed before rounding the displayed accuracies.

These results provide a stronger comparison than the original partial-row affine baseline. In particular, the Nonlinear Interface Adapter improves upon Partial Stitch, but does not close the gap to RIMA.

We additionally evaluate Interface LoRA across four model-pair configurations in Flowers102 and CIFAR-100. RIMA achieves higher mean accuracy in all four settings, with gains ranging from 3.65 to 5.26 percentage points over Interface LoRA.

We have added the nonlinear and partial-interface results and their implementations to Appendix C.8. The broader LoRA comparison appears in Section 5.4 (Table 4), with individual-seed results in Appendix C.8 (Table 23). The new C1 controls are reported independently of the original 38-setting suite in Appendix C.3. These findings support receiver-side message correction relative to the evaluated interface adaptation methods under comparable parameter and training budgets.

AuthorsOpenReview ↗

Response to Reviewer xmku

Response to Reviewer xmku (Part 2/4)

W2. Necessity of State Conditioning, Summary Features, and Adaptation Location

Your comment: "Direct ablations of state conditioning, individual summary features, and adaptation location are missing."

Thank you for identifying these missing controls. We have added direct experiments addressing all three aspects in C1, using 5,000 adaptation steps and three independent seeds.

State conditioning. We investigate whether RIMA benefits from sample-specific receiver-state information rather than merely introducing additional correction parameters. Full RIMA achieves 79.93 ± 0.34% accuracy, whereas constant conditioning achieves 70.26 ± 0.63% and shuffled conditioning achieves 69.44 ± 0.45%. These reductions of 9.67 and 10.49 percentage points support the importance of both sample-dependent conditioning information and its correct association with the current input.

We also evaluate a sample-independent static correction, which achieves 68.17 ± 0.34%. This control has fewer trainable parameters (1,536 versus 20,800), so we do not interpret its performance difference as an isolated effect of state conditioning. The constant and shuffled controls provide the more direct evidence.

Individual summary features. We separately zero-mask the CLS query, patch attention context, mean patch key, and mean patch value. Every variant retains the 256-dimensional conditioning input and nominal 20,800-parameter architecture. The weight columns corresponding to zeroed inputs receive zero gradients. The results are:

Conditioning inputTop-1 accuracy (%)
Full RIMA79.93 ± 0.34
Without CLS query74.77 ± 0.79
Without patch context74.71 ± 0.88
Without mean patch key73.07 ± 0.32
Without mean patch value73.95 ± 0.74
CLS query only72.32 ± 0.82

Each individual removal lowers mean accuracy. Together, these results support the usefulness of both the CLS query and patch-side information for predicting the correction in this configuration.

Adaptation location. We compare the original CLS attention-output correction with alternative locations while matching the 20,800-parameter budget and training protocol. Full RIMA achieves 79.93 ± 0.34%, compared with 76.21 ± 0.15% for CLS MLP-output correction, 74.48 ± 0.70% for CLS hidden-state correction, and 74.80 ± 1.58% for patch-token output correction.

The original CLS attention-output location achieves the highest mean accuracy among these alternatives, providing direct experimental support for the design choice motivated by our patch-to-CLS pathway analysis.

The complete feature, conditioning, and correction-location ablations are reported in Appendix E (Tables 26 and 29). Together, they support the chosen conditioning information and correction location under the tested configurations.

AuthorsOpenReview ↗

Response to Reviewer xmku

Response to Reviewer xmku (Part 3/4)

W3. Independent Perturbations and Naturally Occurring Stitched States

Your comment: "Receiver states and perturbations are constructed from related reference trajectories. Independent perturbations and naturally occurring stitched states would strengthen the generality of the analysis."

We appreciate this observation. The original diagnostic experiments construct receiver states through controlled interventions and use stitched-minus-native K/V differences as perturbations. To investigate the scope of the resulting state dependence, we have added two supplementary diagnostics on ImageNet-100, each using 2,000 evaluation images and a fixed perturbation seed. No additional model training is performed.

Independent perturbation directions. We evaluate Gaussian and donor-derived perturbations on two reconstructed receiver states, P{9,10} and P{10,11}. To avoid confounding perturbation direction with magnitude, we calibrate the key and value perturbations separately to the corresponding per-image, per-head reference difference norms. The directions are independently generated, although their magnitudes remain reference-calibrated.

Under Gaussian perturbations, the accuracy drops are 0.10 and -0.05 percentage points, respectively; the latter is a 0.05-point increase. Donor perturbations produce accuracy drops of 0.50 and 1.25 percentage points. These are accuracy drops, defined as clean minus perturbed accuracy.

The responses are small in this reconstructed setting. The reconstructed diagnostic uses reference trajectories different from the historical Figure 3 checkpoints.

Naturally trained stitched states. We further examine two independently trained stitching checkpoints, E02 and E03, without the preceding-block interventions used to construct the receiver states in the original analysis.

We construct a shared block-12 patch K/V perturbation from the frozen E02 stitched trajectory and the native target trajectory, then apply the same sample-matched perturbation at full strength to both naturally trained states.

Before perturbation, E02 and E03 achieve top-1 accuracies of 87.70% and 87.95%, differing by only 0.25 percentage points. Under the shared perturbation, their accuracies decrease to 66.70% and 58.45%, corresponding to losses of 21.00 and 29.50 percentage points.

The difference in accuracy drops is 8.50 percentage points, with a paired bootstrap 95% confidence interval of [6.45, 10.55].

Thus, the two naturally trained states have closely matched unperturbed accuracy but exhibit substantially different functional responses to the same perturbation. This extends the qualitative receiver-state observation beyond the manually constructed states used in the main analysis.

We also clarify an important limitation: the shared perturbation is derived from E02, so it is not independent of both evaluated natural states. The independent-direction and natural-state experiments address complementary aspects of the receiver-state analysis.

We have added the experimental protocols, complete results, and interpretation to Appendix B.5, Independent Perturbations and Naturally Trained Stitched States (Table 12). These findings support receiver-state-dependent functional responses in the evaluated natural-state setting.

Reviewer_vqEKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the question of functional similarity between networks via stitching. Through controlled experiments, the authors find that the target receiver network's state plays a key role in compatibility. Namely, the same perturbation added to similarly performing states yields substantially different drops in performance, and performance can be recovered by manipulating the attention delivered to the CLS token. They then propose an adaptation mechanism to patch the receiver model's internal states, dubbed RIMA (Receiver Internal Message Adaptation). Their evaluation across five model families and six datasets yields improvements in stitching performance ranging from +1 to +40 percentage points.

Critical strengths
  • The paper is very well written, easy to read, straight to the point, and well organized.
  • The findings on the influence of the receiver state and the proposed adaptation method, RIMA, are interesting contributions.
  • The evaluation is well executed and sound.
Critical weaknesses

1. Insufficient comparison with alternative adaptation methods

  • The effectiveness of RIMA is not evaluated against equivalent LoRA adaptation or any other parameter-efficient fine-tuning method. Whether at an equivalent parameter count or not, knowing where these methods stand (lower, similar, or a clear upper bound) compared to RIMA is crucial to appreciating the value brought by RIMA.
  • L129–130: You state that RIMA is also related to adapters without clearly stating how they are related, e.g., whether it is also an adapter that could be used in general. In the same sense, your stated difference doesn't make it clear why they are not included in the evaluation.
  • L128–129: You claim that you keep the learned interface fixed, implicitly assuming that it is trained with task loss. Balogh & Jelasity (2025) propose a middle ground between structural and functional similarity with “direct matching.” It would be good to evaluate RIMA on it too.

2. Missing experimental controls and ablations

  • L341: An ablation on the hyperparameter r is missing.
  • CLS-only training: Is the low accuracy with CLS-only training not a capacity issue? Wouldn't it be the same if we were only training the patches or a subset of the patches? A controlled increase in capacity could answer this question.

3. Presentation and interpretation of results

  • L035–036: “Despite sharing similar Transformer architectures, these models can develop substantially different internal representations.” This statement is too general to be a falsifiable claim. It could be true or false depending on what is meant by “different internal representations.” Functional compatibility is a precise existing question, and stitching is one precise way to approach it. I suggest making the statement more precise and citing relevant references.
  • Figure 1: (a) is hard to understand, especially in conveying the fact that the same perturbation gives two different effects for previously similarly performing states. Maybe using the numbers directly instead of the robust vs. fragile symbols could help? (b) The zoomed version seems to be for only one block with the corresponding RIMA block, but the illustration arrows cover the whole model.
  • Figure 3b: The accuracy similarities are not visible, making it hard to read. As it is now, it just shows the variability in the responses but not the comparisons.
  • Table 4: Please consider adding the interface baseline to this table. If the interface baseline is already better than RIMA, what is the point of such a comparison? By transitivity, RIMA being better than the interface baseline would be enough.
Questions for authors

1. Perturbation and controlled interventions

  • L050: Please specify whether the perturbation is applied to a sample in the input space or the representation space.
  • L051–052: Can you add a qualifier or some precision to this point regarding what kind of perturbation is used, or at least its magnitude, to put this statement into perspective?
  • L053/L080/L098: How can the “replacing or repairing patch information” part be considered part of the controlled experiments when you are also proposing a patching method? This raises the question of whether these are observation tools or should be held out as methods for comparison. The same precision is lacking in L098 regarding “token and attention interventions.”

2. Receiver processing and interpretation of the experiments

  • L154: “and the CLS similarity falls to 0.0493 after receiver processing.” What processing are you talking about? Nothing was announced.
  • Figure 2: Are there only four blocks in the receiver? If not, could you please provide the plot over all blocks, even if there is saturation? (b) seems to contradict L151–152. Namely, those lines seem to say that 2.14% occurs when only a CLS objective is used for the stitch, with patches kept native, whereas Figure 2b says that 2.14% also occurs with both CLS and patches stitched. Can you clarify this?

3. RIMA and evaluation details

  • L105: What do you mean by “requires no native target trajectory at inference time”? It could be clearer.
  • L423–424: What is the “stronger cut-6 parent stitch”? It was not introduced before.
9 comments
AuthorsOpenReview ↗

Response to Reviewer vqEK

(Part 2/6)

A.1.3 Comparison with Direct Matching

Your comment: "Balogh & Jelasity (2025) propose a middle ground between structural and functional similarity with 'direct matching.' It would be good to evaluate RIMA on it too."

Thank you for this suggestion. We have conducted additional experiments using Direct Matching (DM), following Balogh and Jelasity (2025).

We construct the stitching interface by solving a least-squares feature-matching problem using the pseudoinverse and representations from 100 training images, including CLS and patch tokens. The interface is constructed without task labels. We subsequently freeze the DM interface and train RIMA for 5,000 adaptation steps using cross-entropy and GlobalKD supervision.

We evaluate four model pairs on Flowers102 and CIFAR-100, using the same evaluation protocol as the corresponding task-trained interface comparisons. RIMA results are averaged over three independent seeds.

The validation top-1 accuracies (%) are:

Source → ReceiverDatasetDMDM+RIMAGain (pp)Task+RIMA
ViT-S → DeiT-SFlowers10217.7578.33+60.5979.93
MAE-B → CLIP-B/16Flowers10224.5180.26+55.7588.46
CLIP-B/16 → DeiT-BFlowers10254.2286.44+32.2290.16
ViT-B → ViT-SCIFAR-10048.7266.43+17.7172.15

RIMA improves Direct Matching in all four settings, with gains ranging from 17.71 to 60.59 percentage points. After adaptation, the accuracy gap between DM+RIMA and RIMA applied to the task-trained interface ranges from 1.60 to 8.20 percentage points.

These findings demonstrate that RIMA remains effective when the stitching interface is constructed through Direct Matching rather than the original task-based training procedure. The improvements over DM alone include the effect of the additional task and distillation supervision used during RIMA training.

We have added the complete experimental details and comparisons to Appendix C.5, "RIMA with Direct Matching Interfaces" (Table 20).

A.2.1 Ablation on Bottleneck Dimension r

Your comment: "An ablation on the hyperparameter r is missing."

Thank you for this suggestion. We have conducted an additional bottleneck-width ablation using ViT-S to DeiT-S stitching on Flowers102, evaluating r ∈ {8, 16, 32, 64}. All configurations adapt the last four receiver blocks and use 5,000 optimization steps and three independent seeds.

The results are:

Bottleneck width rTrainable parametersTop-1 accuracy (%)
810,52878.14 ± 0.64
16 (default)20,80079.93 ± 0.34
3241,34479.71 ± 0.74
6482,43278.24 ± 0.43

Increasing r from 8 to 16 improves the mean accuracy by 1.80 percentage points, calculated from unrounded accuracies. Further increasing r to 32 or 64 does not improve mean accuracy. Among the evaluated settings, r = 16 achieves the highest mean accuracy while maintaining a substantially smaller parameter budget than the wider alternatives.

These results support our choice of r = 16 as the default bottleneck dimension.

We have added this experiment to Appendix C.7, "Sensitivity to Bottleneck Dimension" (Table 22).

AuthorsOpenReview ↗

Response to Reviewer vqEK

Response to Reviewer vqEK (Part 1/6)

Thank you for your careful and constructive review. We appreciate your assessment of the receiver-state findings, the proposed method, and the experimental evaluation. We have added the requested adaptation baselines, Direct Matching experiments, bottleneck-width ablation, and capacity and token-supervision controls. We have also clarified the diagnostic interventions and revised the figures. We address each comment and question below.

Key substantive additions and clarifications are marked in dark blue in the revised manuscript. New tables are identified by their blue captions.

A.1.1 Comparison with LoRA and Other Adaptation Methods

Your comment: "The effectiveness of RIMA is not evaluated against equivalent LoRA adaptation or any other parameter-efficient fine-tuning method."

Thank you for identifying this missing comparison. We have added three parameter-efficient adaptation baselines: Receiver LoRA, Interface LoRA, and Receiver Bottleneck. Receiver LoRA adapts the attention output projections in the last four receiver blocks. Interface LoRA adds a low-rank correction to the frozen stitching projection, and Receiver Bottleneck provides a conventional residual-adapter control. Interface LoRA is distinct from the original parameter-matched interface baseline in Table 3, which updates a fixed subset of rows in the affine stitching interface.

We evaluate four source-receiver pairs across Flowers102 and CIFAR-100. Within each setting, all methods use the same task-trained stitching configuration, supervision, and evaluation protocol, with 5,000 adaptation steps and three independent seeds (20260919, 20260920, and 20260921). RIMA has 20,800 trainable parameters, whereas the alternatives have between 18,432 and 21,504 parameters, depending on the configuration.

The validation top-1 accuracies (%), reported as mean ± sample standard deviation, are:

Source → ReceiverDatasetReceiver LoRAInterface LoRABottleneckRIMA
ViT-S → DeiT-SFlowers10271.80 ± 1.6374.77 ± 0.4067.68 ± 1.3379.93 ± 0.34
CLIP-B/16 → DeiT-BFlowers10284.97 ± 0.4186.50 ± 0.0683.73 ± 2.1490.16 ± 0.59
MAE-B → CLIP-B/16Flowers10284.28 ± 0.9383.20 ± 0.1178.10 ± 1.3188.46 ± 0.25
ViT-B → ViT-SCIFAR-10069.49 ± 0.0768.50 ± 0.3168.41 ± 0.3972.15 ± 0.31

RIMA achieves the highest mean accuracy in all four settings. Its improvements over the strongest competing method are +5.16, +3.66, +4.18, and +2.66 percentage points, respectively. The individual-seed results further show that RIMA outperforms all three alternatives in every evaluated seed and setting.

These experiments provide direct evidence that the evaluated LoRA and bottleneck mechanisms do not match RIMA under comparable parameter budgets and training protocols. The new three-seed PEFT evaluation is an independent comparison, reported alongside the original 38-setting results in Table 3 and Appendix C.3.

We have added the main comparison to Section 5.4 (Table 4) and report the exact parameter counts, experimental protocol, and individual-seed results in Appendix C.8 (Table 23).

A.1.2 Relationship Between RIMA and Existing Adapters

Your comment: "related to adapters without clearly stating how they are related"

We agree that this relationship was not sufficiently explained in the original manuscript. RIMA shares the bottleneck residual structure of conventional adapters and keeps the pretrained receiver frozen. Its distinction lies in the information used to condition the correction and the location at which the correction is applied.

RIMA uses the current CLS query, patch attention context, mean patch key, and mean patch value to predict a residual correction specifically for the CLS attention output. In the C1 Receiver Bottleneck control, a conventional bottleneck instead directly transforms that CLS attention output. The distinction therefore concerns the conditioning information, not simply the presence of a residual MLP. Our patch-to-CLS diagnostic findings motivate the RIMA inputs and correction location.

RIMA is currently designed for model stitching with Transformer receivers containing a CLS token.

We have revised the Related Work section to clarify the relationship between RIMA and conventional adapters.

AuthorsOpenReview ↗

Response to Reviewer vqEK

Response to Reviewer vqEK (Part 3/6)

A.2.2 Capacity Control for CLS-Only Training

Your comment: "Is the low accuracy with CLS-only training not a capacity issue? Wouldn't it be the same if we were only training the patches or a subset of the patches? A controlled increase in capacity could answer this question."

Thank you for suggesting these controls. We have conducted six additional experiments in the original Figure 2 setting: DeiT-B/16 to DeiT-S/16 stitching on ImageNet-100, with the interface placed after block 8 (L2).

All configurations use relative raw-token MSE for the supervised tokens, 20,000 optimization steps, and three independent seeds. The source, receiver, and classification head remain frozen. Importantly, CLS-only and patch-only refer to the alignment objectives, not to which tokens are passed to the receiver. The full stitched token sequence enters the receiver in every configuration.

The validation top-1 results (%), reported as mean ± sample standard deviation, are:

GroupSupervised tokensInterfaceParametersAccuracy (%)
ACLS onlyLinear295,2963.15 ± 0.93
BCLS onlyMLP885,88813.79 ± 2.47
CPatches onlyLinear295,29673.58 ± 0.14
D50% patchesLinear295,29673.63 ± 0.29
EAll tokensLinear295,29688.21 ± 0.50
FAll tokensMLP885,88891.65 ± 0.20

Increasing interface complexity from a linear mapping (A) to a larger nonlinear MLP (B) raises CLS-only accuracy by 10.64 percentage points. This contrast changes both parameter count and nonlinearity, so it is not a parameter-count-only intervention. Nevertheless, CLS-only accuracy remains substantially below the all-token condition even with the larger MLP.

Most importantly, under the same 885,888-parameter MLP architecture, all-token supervision (F) achieves 91.65%, compared with 13.79% for CLS-only supervision (B), a difference of 77.86 percentage points. Thus, the severe degradation associated with CLS-only alignment persists under the evaluated higher-capacity interface.

Under the same linear architecture, patch-only supervision (C) achieves 73.58%, outperforming CLS-only supervision (A) by 70.43 percentage points. Supervising a fixed 50% subset of patches (98 of 196 positions, shared across seeds) achieves a similar 73.63%. Joint CLS and patch supervision (E) further increases accuracy to 88.21%, exceeding patch-only supervision by 14.63 percentage points.

With identical 885,888-parameter MLP interfaces, all-token supervision reaches 91.65% and CLS-only supervision reaches 13.79%. The patch-only and joint-supervision results further identify the contribution of patch-side alignment to downstream accuracy.

We have added the full protocol and three-seed results to Appendix D, "Capacity and Token-Supervision Controls for Stitching" (Table 25).

The original Figure 2 results, including 2.14% for CLS-only training and 88.60% for all-token training, remain unchanged.

A.3.1 Precision of the Introduction (L035-036)

Your comment: "This statement is too general to be a falsifiable claim."

We agree that the original description of substantially different internal representations was insufficiently precise.

We have revised the Introduction to focus on the concrete question of functional compatibility: whether intermediate representations from different pretrained models can be connected through a learned interface and processed successfully by a frozen target receiver.

We also added relevant references to prior model stitching research, including Bansal et al. (2021) and Csiszárik et al. (2021).

A.3.2 Clarity of Figure 1

Your comments: "using the numbers directly" and "the illustration arrows cover the whole model."

We have revised Figure 1(a) to display the actual accuracies before and after perturbation. The two receiver states begin with accuracies of 88.96% and 89.10%, respectively, yet the same per-image perturbation produces accuracy drops of 19.22 and 68.86 percentage points.

For Figure 1(b), we have clarified that the enlarged diagram represents a single receiver block equipped with RIMA. Separate RIMA modules are applied to the last four available receiver blocks. We have adjusted the illustration and updated its caption to distinguish the individual-block mechanism from the complete receiver architecture.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.