acceptodds
Under review as a conference paper at ICLR 2027

Lost in Adaptation: Layer-Selective Recovery of Temporal Reasoning in Video-Language Models

Abstract

Multimodal adaptation can weaken temporal reasoning (TR) in video-language models (VLMs) even when the visual evidence needed for an answer is correctly perceived. This weakened competence, however, remains partly recoverable from the model's paired pre-adaptation language checkpoint. We introduce MERIT, a gradient-free framework that selectively restores this competence across the language backbone while protecting temporal perception (TP). It assigns each self-attention layer a VLM-dominant or LLM-dominant interpolation and uses the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to search the recipe space under a perception-constrained objective. Across three VLM families and five video benchmarks, MERIT improves reasoning-aligned performance in all 15 model–benchmark evaluations while preserving TP across all three Video-MME settings. Recipes selected using 232 diagnostic examples transfer to four unseen benchmarks, with relative gains up to 27.8%, and outperform full-backbone, uniform-attention, and alpha-tuned random-layer merging. Interventional masking shows that MERIT-selected layers are disproportionately important for TR, while frame-level attribution reveals increased reliance on temporally distributed, causally relevant evidence. These results show that adaptation-induced reasoning degradation can be repaired post hoc through selective reuse of pre-adaptation competence, without learning new parameters or adding inference overhead.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
temporal reasoning, video-language models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_7ZjgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MERIT, a framework for recovering temporal reasoning capabilities in video-language models by merging their self-attention parameters with those of their pre-adaptation language checkpoints. The method uses CMA-ES to search for binary layer-wise dominance assignments, together with a shared interpolation coefficient, under an objective that rewards temporal reasoning performance and penalizes temporal perception degradation.

Critical strengths
  1. The paper addresses temporal reasoning degradation following multimodal adaptation. Comparisons between the original and adapted language backbones on text-only temporal reasoning provide empirical support for investigating capability recovery.
  2. The experiments are extensive, including independently tuned interpolation coefficients, random layer selections with matched cardinality, and random search under the same evaluation budget.
  3. It requires no additional inference components or decoding steps, making the resulting model convenient to integrate into existing deployment pipelines.
Critical weaknesses
  1. The robustness of recipe selection to the calibration data is insufficiently established. MERIT evaluates up to 8,001 candidate configurations against the same 232 examples.Although model weights are not optimized through gradients, the layer assignments and interpolation coefficient are still selected using supervised feedback and can overfit these examples.
  2. The methodological contribution and the justification for the search space remain limited. The framework combines parameter interpolation, binary layer selection, and an existing evolutionary optimizer. Given prior work on layer-wise and evolutionary model merging, the contribution appears to lie primarily in the particular recovery setting and perception-aware objective. The current comparisons establish advantages over uniform merging and random selection, but do not sufficiently explain why binary dominance assignments with a shared interpolation coefficient are preferable to alternative structured approaches.
  3. The offline computational cost is substantial relative to the demonstrated gains. For InternVL3-8B, the reported full search takes approximately 8.5 hours on four A100 GPUs, corresponding to roughly 34 GPU-hours. The reported LoRA and full-parameter fine-tuning runs use approximately 6.4 and 4 GPU-hours, respectively. These methods optimize different parameter spaces, but the comparison shows that avoiding gradient updates does not itself imply lower adaptation cost.
  4. The primary preservation signal consists of only 55 examples that also participate in recipe selection. For LongVA and Qwen3-VL, the reported TP improvements each correspond to approximately one additional correct answer. These results are consistent with preservation on the diagnostic set, but provide limited support for a broader preservation claim.
  5. The intervention studies do not fully support the mechanistic interpretation. The attention-versus-MLP ablation reuses a layer pattern optimized specifically for attention and only retunes the interpolation coefficient for other modules. This design favors attention and does not establish that it would remain the best intervention target if each module received an independent search under the same budget. Similarly, suppressing the selected layers demonstrates sensitivity at those locations, but the absence of matched random-layer suppression controls makes it difficult to establish their particular importance for temporal reasoning.
Questions for authors

Please refer to the Weaknesses for details.

5 comments
AuthorsOpenReview ↗

Author Response [1/2]

Thank you for recognizing the empirical motivation, controlled comparisons, and deployment convenience of MERIT. We summarize the evidence in the original submission and the additional evaluations addressing each concern. Where figure or table numbering has changed, both versions are specified.


W1: Calibration robustness and potential overfitting.

A1: The original submission evaluates generalization separately from diagnostic optimization (§4.1). Table 1 reports performance on 2,468 Video-MME examples excluded from selection. Table 2 evaluates frozen recipes on four unseen benchmarks, with reasoning-aligned improvements across all 12 model–benchmark comparisons. These results establish transfer beyond the calibration examples.

The revision adds five 80/20 calibration-split searches on InternVL3-8B (Appendix D.1; new Table 8). Four selections reproduce the original nine-layer recipe exactly; the fifth retains all nine layers and adds four. Mean pairwise Jaccard similarity is 0.877. Pooled out-of-fold accuracy reaches 55.9% TR and 80.0% TP, compared with 52.0% and 74.5% for the adapted backbone. This directly tests calibration sensitivity and recovery on examples excluded during each selection.

We also added the full TOMATO[1] benchmark. The same frozen recipes improve overall accuracy by 6.94, 4.85, and 2.76 percentage points for LongVA-7B, InternVL3-8B, and Qwen3-VL-4B, respectively (Appendix F.2; new Table 17), extending transfer evidence to dedicated visual temporal reasoning.

[1] TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models. In ICLR 2025.


W2: Contribution and justification for the search space.

A2: MERIT’s contribution is selective recovery of adaptation-weakened reasoning under a perception-preservation objective. Binary assignments give each layer an LLM weight of α or 1−α, allowing heterogeneous restoration with a single strength parameter.

The original submission evaluates the contribution of this intervention structure through two controls:

  • Interpolation strength: Full-Backbone, All-Attention, and matched-cardinality Random-k independently tune α on the same diagnostic set. Their comparison with MERIT supports the contribution of layer identity beyond coefficient selection (Tables 1–2).
  • Search strategy: CMA-ES achieves higher final diagnostic fitness across all three models under the same α, recipe space, objective, and evaluation budget (Fig. 6). The InternVL3-8B comparison also shows stronger downstream performance (Table 4 in the original submission, Table 7 in the revised version).

The revision adds matched-cardinality Late-k and Best Contiguous baselines with independently tuned α (Appendix E.2; Fig. 10 and Table 16). MERIT exceeds both structured baselines on independent LVBench Reasoning across all three models, with gains over Best Contiguous of 2.0, 6.9, and 5.5 percentage points. This directly evaluates the benefit of flexible layer selection against contiguous and late-layer alternatives.


W3: Offline computational cost.

A3: MERIT’s demonstrated benefit is transferable capability recovery using existing checkpoint parameters, with unchanged inference requirements. The original submission reports the offline cost in Appendix D.3 and Table 7 (Table 11 in the revised version): approximately 34 GPU-hours for the full search, compared with 6.4 for LoRA and 4 for FFT.

These measurements document the higher offline search cost. “Gradient-free” describes the absence of gradient updates during recovery. After selection, the recipe is compiled into a single checkpoint that retains the original architecture and inference cost.

AuthorsOpenReview ↗

Author Response [2/2]

W4: Evidence for temporal perception preservation.

A4: The original submission evaluates diagnostic TP preservation alongside independent checks of capability retention. Table 1 reports maintained or improved TP across all three models and absolute accuracy changes of +0.3, +0.5, and −0.7 percentage points on the 2,468 held-out non-target examples. Table 2 additionally reports overall performance on four unseen benchmarks. These measurements support preservation on the diagnostic TP subset and limited disruption on the evaluated held-out capabilities.

The added calibration-split experiment directly evaluates held-out TP, achieving 80.0% versus 74.5% for the adapted InternVL3-8B backbone (Appendix D.1). Each held-out prediction uses a recipe selected without that fold. This supplements the original capability-retention checks with perception-specific evidence beyond selection-set performance.


W5: Interpretation and controls for intervention studies.

A5: The original module ablation evaluates restoration targets at the same MERIT-selected locations, independently tuning α for each target (Appendix D.2; Tables 5–6 in the original submission, Tables 9–10 in the revised version). Attention-only merging achieves 57.6% Video-MME TR and 53.2% LVBench Reasoning, compared with 52.0–54.8% and 49.8–51.2% for alternative modules. This establishes attention-only merging as the strongest tested variant under the fixed layer configuration.

The original Fig. 3 separately examines functional sensitivity by suppressing selected attention outputs in the unmodified VLM. The revision adds 20 matched-cardinality random masking controls per model, including Qwen3-VL-4B (§4.4; new Table 3).

At κ = 0.4, reasoning specificity is the relative performance drop in Reasoning minus the relative drop in Others, expressed in percentage points:

ModelSelected-layer specificity (pp)Random-layer specificity, mean ± SD (pp)
LongVA-7B32.41.5 ± 6.9
InternVL3-8B11.41.7 ± 3.2
Qwen3-VL-4B10.11.4 ± 4.6

Selected layers exhibit greater reasoning specificity than the random-control mean across all three models. These controls strengthen the functional evidence for the recovery locations as reasoning-sensitive intervention points.

AuthorsOpenReview ↗

Gentle reminder of the author-reviewer discussion deadline

Dear Reviewer 7Zjg:

We are eager to engage in further discussions with you! In our earlier responses, we have actively addressed your concerns by providing clarifications of:

  1. The generalization evidence in the original submission, supplemented by calibration-split analyses and additional benchmark evaluations.
  2. The search-space design and existing controlled comparisons, supplemented by structured layer-selection baselines.
  3. The offline computational costs reported in the original submission and unchanged inference requirements.
  4. The scope of perception preservation and the original held-out capability checks, supplemented by out-of-fold TP evaluation.
  5. The interpretation of the original module ablation and masking analysis, supplemented by matched random-layer masking controls.

As the author-reviewer discussion deadline is approaching, please let us know if you have any additional questions or concerns about the paper. We would be delighted to continue the conversation with you.

We sincerely hope that our responses have effectively addressed your concerns and may encourage a more favorable reconsideration of our paper.

Best regards,

Submission 1729 Authors

Reviewer_YDD4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles the problem of losing the temporal reasoning capability of the language backbone during multimodal adaptation, and explores parameter-space merging of the self-attention layers of a VLM and its LLM backbone counterpart. The proposed method assigns each self-attention layer to a VLM- or LLM-dominant interpolation with a single global coefficient α\alpha for the merging intensity. The search uses the CMA-ES algorithm with an objective that rewards temporal reasoning gains while penalizing temporal perception degradation on Video-MME diagnostic samples. Evaluation is conducted on held-out Video-MME and four unseen benchmarks (LongVideoBench, LVBench, MMBench-Video, Video-Holmes) with three models (LongVA-7B, InternVL3-8B, Qwen3-VL-4B), showing improvements on average without learning new parameters.

Critical strengths

Interesting research question with clear motivation. The paper asks an important research question about multimodal adaptation: can we recover the reasoning capability of the LLM backbone that is lost during VLM training? The motivation is supported by evidence of degradation in a text-only comparison (Fig. 1), and the paper clearly positions itself within the related work on model merging and VLMs.

Critical weaknesses
  1. Limited methodological rigor in the layer-selection design.

    • The parameterization is motivated as separating "where" from "how strongly" to merge (Sec. 3.2), but the selected global α is 1.0 or 0.9 for all models (Table 3). Under Eq. 4, this is close to replacing the selected attention layers with those of the LLM, so the role of α seems limited, and alternatives such as layer-wise α are not explored.
    • The selected layers appear scattered, without a mechanistic insight into which layer ranges mainly contribute to reasoning (e.g., in relation to the finding of [1] that perception is mainly encoded in early layers and reasoning in middle-to-late layers) or a stability check (e.g., the variance of the top-N selected recipes from the proposed search). Given that they are searched on 232 Video-MME examples, I am not fully sure whether they reflect meaningful structure.
  2. Limited experimental evidence for the effectiveness of MERIT.

    • The large Video-MME TR gains are in-sample (Table 1), while held-out gains are mostly within 1–2 points and are reported as relative gains on low baselines.
    • The LongVA baseline on Video-Holmes is below chance (15.2; 6-way multiple choice), so part of the gain might come from better answer-format compliance.
    • The masking analysis (Sec. 4.4) would be more convincing with a random-layer control (i.e., masking the same number of randomly chosen layers).

[1] S. Chen et al. Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging. ICML, 2025.

Questions for authors
  1. In the masking analysis of Sec. 4.4 (Fig. 3), could the authors add a control that suppresses the same number of randomly chosen self-attention layers, and report the results for Qwen3-VL-4B as well (currently only LongVA-7B and InternVL3-8B are shown)?
  2. Are the selected layers stable across diagnostic splits, and could absolute gains with variance be reported?
  3. The current layer-selection baselines cover either no selection (FULL-BACKBONE, ALL-ATTENTION) or random selection (RANDOM-K, random search). Would a simple structured selection, e.g., merging a contiguous block of late layers with the same number of layers (similar in spirit to PlaM [2]), perform comparably to the CMA-ES-selected recipe? This would help clarify the benefit of the search.
  4. What output format was used for each benchmark, and what is the answer-parsing failure rate, especially for LongVA on Video-Holmes?

[2] Z. Wang et al. PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMs. Findings of ACL, 2026.

4 comments
AuthorsOpenReview ↗

Author Response [1/2]

Thank you for recognizing the research question, the text-only evidence supporting our motivation, and the positioning within model merging and VLM research. We have added structured merging baselines, calibration-split evaluations, matched masking controls, and an answer-extraction audit to address the design and evaluation concerns. All references refer to the revised manuscript.


1. Interpolation strength and layer selection (W1a).

A1: The selected α values indicate that strong restoration works well at the identified locations. Under Eq. 4, α = 1 restores the selected self-attention parameters exactly and retains VLM parameters elsewhere. At α = 0.9, selected and remaining layers receive 90% and 10% LLM weight, respectively.

To control for interpolation strength, we independently tune α for Full-Backbone, All-Attention, and matched-cardinality Random-k on the same diagnostic set (Appendix E.1, Tables 13–14). All-Attention selects α = 0.1 across all three models, while the preferred α varies across random layer patterns. MERIT matches or exceeds these tuned baselines across the reported reasoning-aligned evaluations (Tables 1–2). This shows that suitable restoration strength depends on the intervention configuration and supports the contribution of the selected layer identities.

We use a shared α to keep strength selection compact for the 232-example diagnostic set. Binary layer assignments determine whether each layer receives an LLM weight of α or 1−α, allowing heterogeneous restoration across layers with a single strength parameter.


2. Comparison with structured layer selections (W1b, Q3).

A2: We added two structured baselines with matched layer counts: Late-k, which selects the final k layers, and Best Contiguous, which searches every contiguous k-layer block. For each candidate block, we tune α over {0.6, 0.7, 0.8, 0.9, 1.0} using the same perception-constrained diagnostic objective. Selected recipes are evaluated on LVBench without further tuning (Appendix E.2, Figure 10, Tables 15–16).

Independent LVBench Reasoning accuracy is:

ModelBase (%)Late-k (%)Best Contiguous (%)MERIT (%)
LongVA-7B45.346.344.846.8
InternVL3-8B51.249.346.353.2
Qwen3-VL-4B39.341.840.846.3

MERIT exceeds both structured baselines across all three models. Its gains over Best Contiguous are 2.0, 6.9, and 5.5 percentage points, respectively. For Qwen3-VL-4B, Best Contiguous matches MERIT’s diagnostic fitness, while MERIT achieves higher diagnostic TP (70.9% versus 65.5%) and stronger independent reasoning performance. This directly tests the suggested layer-range prior and supports the transfer benefit of the discovered configuration.


3. Recipe stability and variation across calibration splits (W1b, Q2).

A3: We added recipe selection across five 80/20 calibration splits on InternVL3-8B (Appendix D.1, Table 8). All five selections retain the same nine core layers. Four reproduce the original recipe exactly; one adds layers 2, 5, 8, and 9. Mean pairwise Jaccard similarity is 0.877.

Pooled out-of-fold accuracy is 80.0% TP and 55.9% TR, compared with 74.5% and 52.0% for the adapted backbone. These correspond to absolute gains of 5.5 and 3.9 percentage points on examples excluded during the corresponding recipe selection.

Independent-benchmark performance across the five split-selected recipes is:

Independent evaluationBase (%)Mean ± SD (%)Gain over Base, mean ± SD (pp)
LVBench Reasoning51.252.90 ± 0.67+1.70 ± 0.67
LVBench Overall44.844.94 ± 0.13+0.14 ± 0.13
Video-Holmes36.537.12 ± 0.18+0.62 ± 0.18

These approximate summaries use the four identical nine-layer recipes and one expanded recipe in Appendix D.1 and Table 8. Means and sample SDs are calculated using the rounded accuracies reported there.

Separately, with the diagnostic set fixed, three independent CMA-ES runs recover the same final recipe and objective for each model (Figure 6). The experiments therefore assess both search randomness and calibration sensitivity, supporting a stable core selection with positive out-of-fold gains.

AuthorsOpenReview ↗

Author Response [2/2]

4. Reasoning specificity under matched masking controls (W2c, Q1).

A4: We added both requested analyses in Sec. 4.4. Figure 3 now includes Qwen3-VL-4B across the full suppression sweep. Table 3 compares MERIT-selected layers with 20 random layer sets per model, matching the number of suppressed layers.

On LVBench, we scale the selected self-attention outputs in the unmodified VLM by κ = 0.4. Reasoning specificity is the relative performance drop in Reasoning minus the relative drop in Others, expressed in percentage points:

ModelNumber of masked layersMERIT-selected specificity (pp)Random-layer specificity, mean ± SD (pp)
LongVA-7B1132.41.5 ± 6.9
InternVL3-8B911.41.7 ± 3.2
Qwen3-VL-4B1610.11.4 ± 4.6

For Qwen3-VL-4B, selected-layer suppression reduces Reasoning by 19.6% and Others by 9.5%. Across all three models, selected layers show greater reasoning specificity than the random-control mean. This provides functional evidence that the recovery locations are especially sensitive intervention points for reasoning in the unmodified VLM.


5. Held-out transfer and absolute improvements (W2a).

A5: Table 1 distinguishes the 55 TP and 177 TR selection examples as diagnostic, with in-sample entries shaded. The remaining 2,468 Video-MME examples assess non-target performance. Reasoning transfer is evaluated using frozen recipes on independent benchmarks.

We additionally evaluate the full TOMATO benchmark [1], which specifically tests visual temporal reasoning, using the same Video-MME-selected recipes without further tuning (Appendix F.2, Table 17). Absolute accuracy gains over Base are:

ModelLVBench Reasoning gain (pp)Video-Holmes gain (pp)TOMATO gain (pp)
LongVA-7B+1.5+3.7+6.94
InternVL3-8B+2.0+0.7+4.85
Qwen3-VL-4B+7.0+1.4+2.76

The gains vary across models and tasks, with improvements across all three models on the dedicated temporal benchmark. The calibration-split results in A3 additionally establish positive TP/TR gains under out-of-fold evaluation. Tables 1–2 also show that LoRA and full fine-tuning achieve higher diagnostic accuracy but transfer less reliably, supporting the distinction between calibration fit and generalization.

[1] TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models. In ICLR 2025.


6. Output formats and answer-parsing effects (W2b, Q4).

A6: We follow the default benchmark evaluation protocols without modifying answer extraction or scoring. Appendix F.1 specifies multiple-choice QA for Video-MME, LongVideoBench, LVBench, Video-Holmes, and TOMATO, and open-ended QA for MMBench-Video. Appendix C.4 documents the implementations: lmms-eval for Video-MME, LongVideoBench, and LVBench, and VLMEvalKit for MMBench-Video and Video-Holmes. Video-Holmes uses GPT-4-0125 as the answer judge under the default VLMEvalKit pipeline.

We added an answer-extraction audit covering all 1,837 Video-Holmes questions:

ModelVariantAccuracy (%)Valid parse (%)Parse failuresCommon nCommon accuracy (%)
LongVA-7BBase15.250.291559230.6
LongVA-7BMERIT18.951.189959241.2
InternVL3-8BBase36.599.921,83536.5
InternVL3-8BMERIT37.2100.001,83537.2
Qwen3-VL-4BBase31.599.841,83031.6
Qwen3-VL-4BMERIT32.999.831,83033.1

Common n denotes questions successfully parsed for both Base and MERIT; common accuracy is evaluated on that shared subset.

For LongVA-7B, approximately half of the outputs fail answer extraction under the default protocol, helping explain its low absolute score. On the 592 questions successfully parsed for both variants, MERIT achieves 41.2% versus 30.6% accuracy, an improvement of 10.6 percentage points and a net gain of 63 correct answers. Those 63 answers contribute 3.43 percentage points to the full-benchmark improvement of 3.7 points.

Improvements also persist on the common parseable subsets for InternVL3-8B (+0.7 pp) and Qwen3-VL-4B (+1.5 pp). This audit demonstrates improved answer accuracy on questions where both variants produce parseable outputs. All reported benchmark scores retain the original evaluation protocols.

Thank you very much for the detailed response. My assessment is a bit mixed: some methodological limitations remain (α being close to binary, and relatively small gains in several held-out transfer settings), but the other aspects that can be addressed empirically are resolved (layer sensitivity, comparison with structured selection, fair comparisons, and additional evaluations), and I think this paper covers an interesting topic and approach that merits a weak acceptance. I will therefore raise my score from 2 to 3.

AuthorsOpenReview ↗

Thanks for your support

Thanks for your support! We appreciate your openness in sharing your concerns and are glad that our additional experiments effectively addressed the empirical questions.

Your feedback helped us strengthen the empirical support for our central finding: temporal reasoning weakened during multimodal adaptation can be recovered through selective reuse of pre-adaptation competence. This reinforces the value of capability recovery as a practical direction for improving VLM reasoning without retraining. We are sincerely grateful for your constructive feedback.

Reviewer_y7pyOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper discovered a very interesting phenomenon, after doing multimodal adaptation to the backbone LLM, models' temporal (text) understanding capability will decrease. Stemming from this observation, the authors make an assumption that the temporal reasoning weakness might not come solely from the perception, maybe from multimodal adaptation. To do that, they introduce MERIT, a gradient free framework for layer selective capability recovery, where the key idea is to determine where the adapted VLM should retain VLM parameters and where it should contain the LLM one. The paper shows that MERIT recovers temporal reasoning while preserving perception capability.

Critical strengths
  1. Motivation and observation is very interesting. Actually, (only for sharing purpose), [1] proves that we can improve models' temporal reasoning capability from pure text temporal reasoning. But I've never thought about the other direction, I like the cutting angle of this paper.

[1] Temporal Reasoning Transfer from Text to Video, ICLR 2025

  1. Experiments are very solid. They are not only comparing with original VLMs but also compare with multiple layer selection baseline.

  2. Demonstrating good video reasoning capability transfer in other bnechmarks.

Critical weaknesses
  1. In the related work, you've listed many visual temporal reasoning benchamrks, why don't you consider evlauating MERIT on these benchmarks tailored for visual temporal reasoning rather than choosing general video understanding?

  2. Does the model improvement truly comes from recovering lost temporal reasoning? The logic is like: multimodal adaptation leads to temporal reasonig degradation --> recover LLM params to VLMs --> benchmark score up --> original temporal reasoning is being recovered. Im not sure if the last one can be demonstrated, as your merge recipe comes from video temporal reasoning accuracy. Even though original LLMs didn't lose such capability in multimodal adapation, this merging might found a better parameters for VLMs.

I'm happy to raise my score if you can answer my #2 well : ) I really like your perspective.

Questions for authors

see weakness

4 comments
AuthorsOpenReview ↗

Author Response

Thank you for recognizing the adaptation-loss perspective, the layer-selection comparisons, and cross-benchmark reasoning transfer. We address both concerns below. All references refer to the revised manuscript.


W1: Evaluation on benchmarks specifically designed for visual temporal reasoning.

A1: We have added TOMATO [1], a dedicated visual temporal reasoning benchmark designed to emphasize multi-frame integration and sensitivity to frame order. It contains 1,484 questions covering action count, direction, rotation, shape and trend, velocity and frequency, and visual cues.

We evaluate all three models on the full benchmark using the same recipes selected on Video-MME, with no additional tuning. The overall results in Table 17, Appendix F.2 are:

ModelBase accuracy (%)MERIT accuracy (%)Gain (percentage points)
LongVA-7B21.4328.37+6.94
InternVL3-8B30.6635.51+4.85
Qwen3-VL-4B32.9535.71+2.76

MERIT improves overall accuracy across all three models. Category-level gains include +22.86 percentage points on velocity and frequency for LongVA-7B and +12.67 points on action count for InternVL3-8B. These results extend the transfer evidence to dedicated visual temporal reasoning tasks. We describe TOMATO in Appendix F.1 and summarize the findings in Sec. 4.2.


W2: Whether the gains reflect recovery of temporal reasoning lost during multimodal adaptation.

A2: Our new question-level analysis shows that MERIT restores specific temporal reasoning answers lost during multimodal adaptation. We track the same questions across the original LLM, the adapted backbone, and MERIT, using text-only data excluded from recipe selection (Sec. 4.5, Tables 4–5).

1. Recovery is evaluated independently of recipe selection.

We compare the three checkpoints on 9,823 text-only TempReason questions covering time–event and event–event reasoning. The recipes remain frozen after selection on Video-MME; TempReason is used exclusively for evaluation. Within each model family, all three checkpoints use the same task prompt and deterministic decoding, with their native chat templates (Appendix B).

Figure 5 shows that adaptation reduces overall text-only accuracy in all three model families and that MERIT restores part or all of this aggregate loss. This evaluation directly tests temporal reasoning within the language backbone without visual input.

2. The improvements predominantly recover previously correct answers.

We define a lost example as one answered correctly by the original LLM and incorrectly by the adapted backbone. A recovered example is a lost example answered correctly by MERIT. Recovery rate measures the fraction of lost examples recovered; recovery share measures the fraction of all newly correct answers that belong to this lost set.

The question-level results in Table 4 are:

ModelLost examplesRecovered examplesRecovery rate (%)Recovery share (%)
LongVA-7B15510869.6890.76
InternVL3-8B25318673.5290.29
Qwen3-VL-4B16710965.2789.34

These recoveries account for 89.34–90.76% of all newly correct answers. After accounting for regressions, the net increases are 77, 133, and 63 correct answers, respectively. The aggregate improvements therefore predominantly reflect restoration of predictions that were correct before adaptation.

3. The selected layer configuration strengthens this recovery.

We further compare MERIT with Random-k merging on InternVL3-8B, matching the number of selected layers. Table 5 reports:

MethodEM (%)Recovery rate (%)Recovery share (%)EM gain (percentage points)
Adapted backbone14.020N/AN/A
Random-k14.26 ± 0.1014.23 ± 1.5055.18 ± 1.36+0.24 ± 0.10
MERIT15.3773.5290.29+1.35

Random-k values are mean ± standard deviation over five runs. MERIT achieves substantially higher recovery of lost examples, a higher recovery share, and a larger net accuracy gain. This supports the contribution of the discovered layer configuration to restoring pre-adaptation performance.

MERIT recovers 65.27–73.52% of adaptation-lost predictions, with these recoveries accounting for approximately 90% of its newly correct answers. Obtained on text-only questions excluded from recipe selection, these findings provide direct behavioral evidence that the video-selected intervention restores temporal reasoning behavior weakened during multimodal adaptation.


[1] TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models. In ICLR 2025.

AuthorsOpenReview ↗

Gentle reminder of the author-reviewer discussion deadline

Dear Reviewer y7py:

We are eager to engage in further discussions with you! In our earlier responses, we have actively addressed your concerns by providing:

  1. Evaluation on TOMATO, a dedicated visual temporal reasoning benchmark, with improvements across all three model families.
  2. Question-level evidence of adaptation-lost reasoning recovery, showing that MERIT recovers 65.27–73.52% of lost predictions on text-only questions excluded from recipe selection.
  3. A Random-k merging comparison supporting the contribution of the selected layer configuration to capability recovery.

As the author-reviewer discussion deadline is approaching, please let us know if you have any additional questions or concerns about the paper. We would be delighted to continue the conversation with you.

We sincerely hope that our responses have effectively addressed your concerns and may encourage a more favorable reconsideration of our paper.

Best regards,

Submission 1729 Authors

Thank you for your detailed response and my sincere apology for the late reply (unfortunately another conf's deadline is on Monday). I think my concerns on W2 is resolved. Your perspective is very interesting and I will increase my score. Apology to the late reply again and hope it won't be too late...

AuthorsOpenReview ↗

Thanks for your support

Thank you very much for your kind response and for raising your score! We truly appreciate you taking the time to revisit our work despite your busy schedule.

We are glad that our additional experiments and analyses addressed your central concern about W2. We also sincerely appreciate your encouragement of our research perspective. Thank you again for your thoughtful feedback and support.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear AC,

Thank you for overseeing the review of our submission. We summarize the shared strengths, updated assessments, and key rebuttal evidence below. All manuscript references refer to the revised version.


Shared Recognition of Strengths

  • Research question and empirical motivation: All three reviewers acknowledged the value of investigating temporal reasoning degradation during multimodal adaptation, supported by comparisons between the original and adapted language backbones.
  • Controlled evaluation: Reviewers y7py and 7Zjg recognized the extensive comparisons, including independently tuned interpolation coefficients, matched random-layer selections, and matched-budget random search.
  • Transfer and deployment: Reviewer y7py highlighted cross-benchmark reasoning transfer, while 7Zjg recognized that MERIT requires no additional inference components or decoding steps.

Positive Feedback During Rebuttal

We are encouraged that our clarifications and additional experiments addressed the reviewers’ key concerns and strengthened their support for the paper:

  • Reviewer y7py (score increased: 3 → 4) explicitly confirmed that their central concern about whether MERIT truly recovers temporal reasoning lost during multimodal adaptation (W2) was resolved, and raised their score to clear acceptance.
  • Reviewer YDD4 (score increased: 2 → 3) explicitly acknowledged that the empirical concerns regarding “layer sensitivity, comparison with structured selection, fair comparisons, and additional evaluations” are resolved, and concluded that the topic and approach merit weak acceptance.
  • Reviewer 7Zjg (score increased: 2 → 3) stated that the new experiments “address most of my original concerns” and raised their score.

All three reviewers increased their scores during rebuttal, with current recommendations of 4/3/3 (one clear acceptance and two weak acceptances).


How We Addressed Each Reviewer’s Concerns

We supplemented the original evidence with targeted experiments and clarified the scope of our claims:

  • y7py: Direct evidence of capability recovery and dedicated temporal evaluation. On text-only questions excluded from recipe selection, MERIT recovers 65.27–73.52% of predictions lost during adaptation. These recoveries account for approximately 90% of newly correct answers, providing direct behavioral evidence for the recovery interpretation (Sec. 4.5, Tables 4–5). We also added TOMATO, a dedicated visual temporal reasoning benchmark, where frozen recipes improve accuracy across all three models by 2.76–6.94 percentage points (Table 17).

  • YDD4: Stronger controls for layer selection and evaluation. We added structured merging baselines, calibration-split evaluations, matched random-layer masking, Qwen3-VL masking results, and an answer-extraction audit. MERIT exceeds both late-layer and best-contiguous baselines on independent LVBench Reasoning across all three models (Table 16). Selected layers exhibit greater reasoning specificity than random-control means (Table 3), and Video-Holmes improvements persist on questions successfully parsed for both model variants.

  • 7Zjg: Clarification of existing evidence and additional robustness tests. We clarified the original transfer and capability-retention results (Tables 1–2), reported offline costs (Table 11), and the scope of the module ablation. New calibration-split experiments retain the same nine core layers across all five selections and yield out-of-fold gains of 3.9 percentage points in reasoning and 5.5 in perception on InternVL3-8B (Appendix D.1).

With direct evidence of capability recovery, stronger controls for layer selection, and reasoning transfer across benchmarks, we are confident that MERIT advances our understanding of reasoning loss during multimodal adaptation. It demonstrates the value of selectively restoring pre-adaptation capabilities, opening a practical direction for improving VLM reasoning without additional training or inference overhead.

Sincerely,

Submission 1729 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.