- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes MERIT, a framework for recovering temporal reasoning capabilities in video-language models by merging their self-attention parameters with those of their pre-adaptation language checkpoints. The method uses CMA-ES to search for binary layer-wise dominance assignments, together with a shared interpolation coefficient, under an objective that rewards temporal reasoning performance and penalizes temporal perception degradation.
Critical strengths
- The paper addresses temporal reasoning degradation following multimodal adaptation. Comparisons between the original and adapted language backbones on text-only temporal reasoning provide empirical support for investigating capability recovery.
- The experiments are extensive, including independently tuned interpolation coefficients, random layer selections with matched cardinality, and random search under the same evaluation budget.
- It requires no additional inference components or decoding steps, making the resulting model convenient to integrate into existing deployment pipelines.
Critical weaknesses
- The robustness of recipe selection to the calibration data is insufficiently established. MERIT evaluates up to 8,001 candidate configurations against the same 232 examples.Although model weights are not optimized through gradients, the layer assignments and interpolation coefficient are still selected using supervised feedback and can overfit these examples.
- The methodological contribution and the justification for the search space remain limited. The framework combines parameter interpolation, binary layer selection, and an existing evolutionary optimizer. Given prior work on layer-wise and evolutionary model merging, the contribution appears to lie primarily in the particular recovery setting and perception-aware objective. The current comparisons establish advantages over uniform merging and random selection, but do not sufficiently explain why binary dominance assignments with a shared interpolation coefficient are preferable to alternative structured approaches.
- The offline computational cost is substantial relative to the demonstrated gains. For InternVL3-8B, the reported full search takes approximately 8.5 hours on four A100 GPUs, corresponding to roughly 34 GPU-hours. The reported LoRA and full-parameter fine-tuning runs use approximately 6.4 and 4 GPU-hours, respectively. These methods optimize different parameter spaces, but the comparison shows that avoiding gradient updates does not itself imply lower adaptation cost.
- The primary preservation signal consists of only 55 examples that also participate in recipe selection. For LongVA and Qwen3-VL, the reported TP improvements each correspond to approximately one additional correct answer. These results are consistent with preservation on the diagnostic set, but provide limited support for a broader preservation claim.
- The intervention studies do not fully support the mechanistic interpretation. The attention-versus-MLP ablation reuses a layer pattern optimized specifically for attention and only retunes the interpolation coefficient for other modules. This design favors attention and does not establish that it would remain the best intervention target if each module received an independent search under the same budget. Similarly, suppressing the selected layers demonstrates sensitivity at those locations, but the absence of matched random-layer suppression controls makes it difficult to establish their particular importance for temporal reasoning.
Questions for authors
Please refer to the Weaknesses for details.
5 comments
Author Response [2/2]
W4: Evidence for temporal perception preservation.
A4: The original submission evaluates diagnostic TP preservation alongside independent checks of capability retention. Table 1 reports maintained or improved TP across all three models and absolute accuracy changes of +0.3, +0.5, and −0.7 percentage points on the 2,468 held-out non-target examples. Table 2 additionally reports overall performance on four unseen benchmarks. These measurements support preservation on the diagnostic TP subset and limited disruption on the evaluated held-out capabilities.
The added calibration-split experiment directly evaluates held-out TP, achieving 80.0% versus 74.5% for the adapted InternVL3-8B backbone (Appendix D.1). Each held-out prediction uses a recipe selected without that fold. This supplements the original capability-retention checks with perception-specific evidence beyond selection-set performance.
W5: Interpretation and controls for intervention studies.
A5: The original module ablation evaluates restoration targets at the same MERIT-selected locations, independently tuning α for each target (Appendix D.2; Tables 5–6 in the original submission, Tables 9–10 in the revised version). Attention-only merging achieves 57.6% Video-MME TR and 53.2% LVBench Reasoning, compared with 52.0–54.8% and 49.8–51.2% for alternative modules. This establishes attention-only merging as the strongest tested variant under the fixed layer configuration.
The original Fig. 3 separately examines functional sensitivity by suppressing selected attention outputs in the unmodified VLM. The revision adds 20 matched-cardinality random masking controls per model, including Qwen3-VL-4B (§4.4; new Table 3).
At κ = 0.4, reasoning specificity is the relative performance drop in Reasoning minus the relative drop in Others, expressed in percentage points:
| Model | Selected-layer specificity (pp) | Random-layer specificity, mean ± SD (pp) |
|---|---|---|
| LongVA-7B | 32.4 | 1.5 ± 6.9 |
| InternVL3-8B | 11.4 | 1.7 ± 3.2 |
| Qwen3-VL-4B | 10.1 | 1.4 ± 4.6 |
Selected layers exhibit greater reasoning specificity than the random-control mean across all three models. These controls strengthen the functional evidence for the recovery locations as reasoning-sensitive intervention points.
Gentle reminder of the author-reviewer discussion deadline
Dear Reviewer 7Zjg:
We are eager to engage in further discussions with you! In our earlier responses, we have actively addressed your concerns by providing clarifications of:
- The generalization evidence in the original submission, supplemented by calibration-split analyses and additional benchmark evaluations.
- The search-space design and existing controlled comparisons, supplemented by structured layer-selection baselines.
- The offline computational costs reported in the original submission and unchanged inference requirements.
- The scope of perception preservation and the original held-out capability checks, supplemented by out-of-fold TP evaluation.
- The interpretation of the original module ablation and masking analysis, supplemented by matched random-layer masking controls.
As the author-reviewer discussion deadline is approaching, please let us know if you have any additional questions or concerns about the paper. We would be delighted to continue the conversation with you.
We sincerely hope that our responses have effectively addressed your concerns and may encourage a more favorable reconsideration of our paper.
Best regards,
Submission 1729 Authors
Author Response [1/2]
Thank you for recognizing the empirical motivation, controlled comparisons, and deployment convenience of MERIT. We summarize the evidence in the original submission and the additional evaluations addressing each concern. Where figure or table numbering has changed, both versions are specified.
W1: Calibration robustness and potential overfitting.
A1: The original submission evaluates generalization separately from diagnostic optimization (§4.1). Table 1 reports performance on 2,468 Video-MME examples excluded from selection. Table 2 evaluates frozen recipes on four unseen benchmarks, with reasoning-aligned improvements across all 12 model–benchmark comparisons. These results establish transfer beyond the calibration examples.
The revision adds five 80/20 calibration-split searches on InternVL3-8B (Appendix D.1; new Table 8). Four selections reproduce the original nine-layer recipe exactly; the fifth retains all nine layers and adds four. Mean pairwise Jaccard similarity is 0.877. Pooled out-of-fold accuracy reaches 55.9% TR and 80.0% TP, compared with 52.0% and 74.5% for the adapted backbone. This directly tests calibration sensitivity and recovery on examples excluded during each selection.
We also added the full TOMATO[1] benchmark. The same frozen recipes improve overall accuracy by 6.94, 4.85, and 2.76 percentage points for LongVA-7B, InternVL3-8B, and Qwen3-VL-4B, respectively (Appendix F.2; new Table 17), extending transfer evidence to dedicated visual temporal reasoning.
[1] TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models. In ICLR 2025.
W2: Contribution and justification for the search space.
A2: MERIT’s contribution is selective recovery of adaptation-weakened reasoning under a perception-preservation objective. Binary assignments give each layer an LLM weight of α or 1−α, allowing heterogeneous restoration with a single strength parameter.
The original submission evaluates the contribution of this intervention structure through two controls:
The revision adds matched-cardinality Late-k and Best Contiguous baselines with independently tuned α (Appendix E.2; Fig. 10 and Table 16). MERIT exceeds both structured baselines on independent LVBench Reasoning across all three models, with gains over Best Contiguous of 2.0, 6.9, and 5.5 percentage points. This directly evaluates the benefit of flexible layer selection against contiguous and late-layer alternatives.
W3: Offline computational cost.
A3: MERIT’s demonstrated benefit is transferable capability recovery using existing checkpoint parameters, with unchanged inference requirements. The original submission reports the offline cost in Appendix D.3 and Table 7 (Table 11 in the revised version): approximately 34 GPU-hours for the full search, compared with 6.4 for LoRA and 4 for FFT.
These measurements document the higher offline search cost. “Gradient-free” describes the absence of gradient updates during recovery. After selection, the recipe is compiled into a single checkpoint that retains the original architecture and inference cost.