- Rating
- 3(+2)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
The paper investigates model stitching and proposes RIMA to improve the stitching performance.
Critical strengths
The experiments might suggest potential benefits of the proposed method for model stitching.
Critical weaknesses
-
This paper is difficult to understand. For example, in line 46, the term “receiver states” appears in the Introduction without definition or explanation. The arrows in RIMA in Figure 1 are unclear, and there is a loop in RIMA. In Section 3.1, it is unclear how the stitch layer is trained and what the loss is. These are examples of the presentation issues.
-
The experimental results are not sufficient to demonstrate the superiority of the proposed method.
- No ablation study to evaluate the contribution of each component of RIMA, the loss, and the number of RIMA modules.
- Insufficient comparison with prior work, such as [1].
- The benefits of RIMA may come from an additional training stage and the additional trainable parameters. What if using the same training strategy to directly finetune the stitching layer? What if introducing LoRA or an adapter to the stitching layer? What if keeping the stitching layer but introducing LoRA or AdaptFormer to the target model?
[1] Revisiting Model Stitching in the Foundation Model Era, CVPR 2026
Questions for authors
Suggestions
-
The paper could refer to the experimental section of prior work on model stitching and better design its experiments.
-
The paper could discuss closely related work, such as [2-5], and also refer to them to better present Section 3.
[2] Locating and Editing Factual Associations in GPT, NeurIPS 2022.
[3] Register and [CLS] Tokens Induce a Decoupling of Local and Global Features in Large ViTs, NeurIPS 2025.
[4] Revisiting [CLS] and Patch Token Interaction in Vision Transformers, ICLR 2026.
[5] Causal Attribution via Activation Patching, arXiv preprint arXiv:2603.13652.
11 comments
Response to Reviewer rFyx
Response to Reviewer rFyx (Part 2/5)
A.2. Insufficient Experimental Evidence
Your comment: "The experimental results are not sufficient to demonstrate the superiority of the proposed method."
Thank you for raising this concern. We have strengthened the experimental evaluation with additional comparisons against alternative adaptation methods. The evidence now covers three complementary aspects.
First, in the original 38-setting evaluation across multiple model families and classification datasets, RIMA improves traditional stitching by 40.01 percentage points on average and exceeds the interface baseline with a matched parameter budget by 8.32 points. The paired 95% confidence interval for the RIMA advantage has a positive lower bound in every evaluated condition (Section 5.4 and Appendix C.3).
Second, we have added comparisons with Receiver LoRA, Interface LoRA, and Receiver Bottleneck under approximately matched parameter budgets. Across four model pairs on Flowers102 and CIFAR-100, RIMA achieves the highest mean accuracy in every setting, exceeding the strongest alternative by 2.66 to 5.16 percentage points. Each configuration is evaluated over three independent seeds (Table 4 and Appendix C.8).
Third, the original full ImageNet-1K validation results provide evidence at larger evaluation scale. RIMA improves DeiT-B to DeiT-S stitching from 68.538% to 73.752% (+5.214 pp) and ViT-B to DeiT-S stitching from 57.000% to 69.658% (+12.658 pp).
Together, these results provide evidence that RIMA improves functional compatibility across different stitching settings and offers advantages over the evaluated interface and receiver adaptation baselines.
A.3. Missing Ablation Studies
Your comment: "No ablation study to evaluate the contribution of each component of RIMA, the loss, and the number of RIMA modules."
Thank you for this suggestion. We have added three sets of ablation experiments to Appendix E, examining the conditioning features, training objectives, and number of RIMA modules. All experiments use ViT-S/16 to DeiT-S/16 stitching on Flowers102, with 5,000 optimizer steps and three independent seeds. Results are reported as mean ± sample standard deviation.
(1) Contribution of conditioning features.
We zero-mask each conditioning feature individually while retaining the full 256-dimensional input and nominal 20,800-parameter RIMA architecture. The CLS-query-only variant uses the input [q_cls, 0, 0, 0].
| Configuration | Top-1 Accuracy (%) |
|---|---|
| Full RIMA | 79.93 ± 0.34 |
| Without CLS query | 74.77 ± 0.79 |
| Without patch context | 74.71 ± 0.88 |
| Without mean patch key | 73.07 ± 0.32 |
| Without mean patch value | 73.95 ± 0.74 |
| CLS query only | 72.32 ± 0.82 |
Removing any individual feature reduces accuracy by 5.16 to 6.86 percentage points. Using only the CLS query reduces accuracy by 7.61 points. These results demonstrate the contribution of the four conditioning features to RIMA's performance.
(2) Contribution of training objectives.
We separately evaluate cross entropy (CE), GlobalKD, and their combination.
| Training Objective | Top-1 Accuracy (%) |
|---|---|
| CE + GlobalKD | 79.93 ± 0.34 |
| CE only | 78.01 ± 0.32 |
| GlobalKD only | 77.29 ± 0.50 |
The combined objective outperforms CE alone by 1.93 points and GlobalKD alone by 2.65 points, calculated from unrounded accuracies. Both terms therefore contribute to the final performance in this setting.
(3) Number of RIMA modules.
We vary the number of adapted receiver blocks while keeping the bottleneck dimension fixed at r = 16.
| Modules | Adapted Blocks | Parameters | Top-1 Accuracy (%) |
|---|---|---|---|
| 1 | 12 | 5,200 | 70.39 ± 1.58 |
| 2 | 11-12 | 10,400 | 75.62 ± 0.25 |
| 3 | 10-12 | 15,600 | 78.59 ± 0.64 |
| 4 | 9-12 | 20,800 | 79.93 ± 0.34 |
Accuracy increases from 70.39% with one module to 79.93% with four modules. The four-module configuration achieves the highest mean accuracy among the evaluated settings.
These three ablations directly address the requested component, loss, and module count analyses. We have included the complete results and experimental details in Appendix E (Tables 26, 27, and 28), "Ablations of RIMA Components, Training Losses, and Module Count," and added a reference to these experiments in Section 4.2.
Response to Reviewer rFyx
Response to Reviewer rFyx (Part 3/5)
A.4. Comparison with Prior Work
Your comment: "Insufficient comparison with prior work, such as [1]."
Thank you for highlighting the importance of this comparison. We have added experiments based on the two-stage interface optimization strategy of Mai et al. (CVPR 2026), which uses final feature matching (FFM) followed by task loss training (TLT).
We adapt this strategy to four model pairs across Flowers102 and CIFAR-100. During FFM, we match the final pooled CLS features after normalization and before the frozen classifier, using MSE for 5,000 steps. TLT then continues from the FFM checkpoint for 5,000 steps using CE only. The interface is a two-layer ReLU MLP applied to each token, with hidden width equal to the target feature dimension. The source, receiver, and classification head remain frozen.
We use the name Adapted Mai MLP deliberately: the hidden width is our implementation choice, and the training budget, model pairs, and evaluation protocol differ from the original work. We implemented the two-stage procedure independently from the method described in Mai et al.
We also evaluate a LayerNorm-plus-linear interface with TLT using either CE or CE + GlobalKD after FFM initialization. These modified affine variants are our extensions.
All results are validation top-1 accuracy (%), reported as mean ± sample standard deviation over three independent seeds.
The settings are C1: ViT-S → DeiT-S on Flowers102; C2: MAE-B → CLIP-B/16 on Flowers102; C3: CLIP-B/16 → DeiT-B on Flowers102; and C4: ViT-B → ViT-S on CIFAR-100.
| Setting | Adapted Mai MLP | Modified affine, CE | Modified affine, CE+KD | RIMA |
|---|---|---|---|---|
| C1 | 78.20 ± 0.97 | 76.96 ± 0.26 | 69.08 ± 0.28 | 79.93 ± 0.34 |
| C2 | 78.99 ± 1.03 | 82.39 ± 0.71 | 80.75 ± 0.25 | 88.46 ± 0.25 |
| C3 | 87.58 ± 0.64 | 89.12 ± 0.69 | 86.83 ± 0.32 | 90.16 ± 0.59 |
| C4 | 71.63 ± 0.28 | 72.75 ± 0.00 | 68.30 ± 0.30 | 72.15 ± 0.31 |
RIMA achieves higher mean accuracy than Adapted Mai MLP in all four settings, with improvements of 1.73, 9.48, 2.58, and 0.52 percentage points, respectively.
Among all four evaluated methods, RIMA achieves the highest mean accuracy in C1, C2, and C3. In C4, the modified affine interface trained with CE achieves 72.75%, compared with 72.15% for RIMA.
These experiments provide a direct comparison with an adapted implementation of Mai et al.'s training strategy under our model pairs and datasets. We have added the experimental protocol, parameter counts, and complete results to Appendix C.9, "Comparison with Adapted Mai Interface Optimization" (Table 24).
A.5.1 Additional Training Stage and Parameter Budget
Your comment: "The benefits of RIMA may come from an additional training stage and the additional trainable parameters."
Thank you for raising this concern. We address it through a controlled comparison with an interface adaptation baseline that receives the same additional training stage, supervision, and optimization schedule as RIMA.
Both methods start from the same trained stitching interface. RIMA adds 20,800 trainable parameters inside the receiver, while the interface baseline updates 20,763 or 20,790 parameters of the existing stitching layer, depending on the model configuration. Both use the same training data, task labels, native target distillation, and evaluation protocol.
Across 38 evaluated conditions, RIMA exceeds the parameter matched interface baseline by 8.32 percentage points on average, with a positive lower bound for the paired 95% confidence interval in every condition.
These results show that the additional training stage and trainable parameter budget alone do not account for RIMA's advantage over the evaluated interface adaptation baseline. The comparison is reported in Section 5.4, Table 3, and Appendix C.3, with implementation details in Appendix A.
Response to Reviewer rFyx
Response to Reviewer rFyx (Part 1/5)
Thank you for your detailed review and constructive suggestions. We have revised the presentation of our method, clarified the training objectives, and added extensive experimental comparisons and ablations. We address each of your concerns below.
Key substantive additions and clarifications are marked in dark blue in the revised manuscript. New tables are identified by their blue captions.
A.1.1 Definition of Receiver States
Your comment: "The term 'receiver states' appears in the Introduction without definition or explanation."
Thank you for pointing this out. We have revised the Introduction to clarify the meaning of receiver state at its first substantive use. Specifically, we refer to the intermediate activations entering a block of the frozen target network as the receiver state. This clarification makes the central research question more precise and provides a clearer basis for the controlled interventions in Section 3.
A.1.2 Clarity of Figure 1
Your comment: "The arrows in RIMA in Figure 1 are unclear, and there is a loop in RIMA."
Thank you for pointing out the ambiguity in Figure 1(b). We have redrawn the figure to clarify the direction of information flow and the role of RIMA within the receiver.
The enlarged diagram illustrates a single receiver block equipped with RIMA. The current CLS query and patch summaries are used to predict a correction, which is added to the CLS attention output before subsequent receiver processing. The apparent loop represents the standard Transformer residual connection rather than recurrent feedback through RIMA.
We have also revised the figure caption to clarify that separate RIMA modules are applied to the last four available receiver blocks.
A.1.3 Stitching Layer Training and Objective
Your comment: "In Section 3.1, it is unclear how the stitch layer is trained and what the loss is."
Thank you for pointing out this missing detail. We have revised Section 3.1 to explicitly describe the stitching objective and training procedure.
For the diagnostic experiments in Section 3.1, we train the affine stitching layer using relative mean squared error (MSE) between the stitched token representations and their corresponding native target representations. The all-token objective supervises both CLS and patch tokens, whereas the CLS-only objective supervises only the CLS token. In both cases, the full stitched token sequence enters the receiver; the objective names do not describe which tokens are passed forward. The source model, target receiver, and classifier remain frozen.
RIMA is trained in a separate stage with the learned stitching layer frozen. Its training objective combines cross entropy (CE) on the task labels with GlobalKD, which matches the centered output logits of the stitched model to those of the frozen native target model. In the Section 3 diagnostics, relative token MSE trains the affine interface. In the subsequent RIMA stage, that interface is frozen while CE and GlobalKD optimize the receiver corrections.
We have clarified the stitching objective in Section 3.1, while the RIMA objective and training protocol are described in Section 4.2 and Appendix A.