acceptodds
Under review as a conference paper at ICLR 2027

Beyond Answering: Small Models Empower Large Models for Long-Video Understanding

Abstract

Recent advances in long-video reasoning enable models to actively search for visual evidence through tool use. However, these approaches commonly assign evidence acquisition and answer generation to the same model, coupling repeated visual exploration with that model’s computational cost and reasoning limitations. We find that small models can more readily locate and verify relevant evidence yet struggle to integrate it into a correct answer, revealing distinct capability requirements for evidence acquisition and reasoning. Building on this observation, we propose an evidence-driven collaboration framework in which a small model handles repeated, multi-step evidence search and verification, while a large model integrates the acquired evidence and produces the final answer in a single invocation. Crucially, we distinguish evidence search under sparse observations from direct evidence localization: when decisive details remain unobserved, the true evidence location may not be a search target supported by the available cues. We therefore supervise search priorities grounded in the current observations, teaching the small model where to inspect next, while dense local verification determines each candidate's actual evidential value. This decoupled framework supports flexible test-time scaling through configurable evidence acquisition and answer routing. Experiments demonstrate that small models can strengthen large-model reasoning through evidence acquisition, extending the collaborative value of small models beyond answering. Pairing a 4B evidence search and verification model with a frozen 27B model for final reasoning achieves 74.8% on VideoMME, 79.2% on MLVU, and 58.7% on LVBench. Code, data, and models will be released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic, Long video understanding, Small-large model collaboration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VscUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a small–large model collaboration framework for long-video understanding. A shared 4B model performs routing, temporal search, and local evidence verification, while a frozen larger model integrates the retained visual evidence in one final answering call. Search supervision is derived from teacher-assessed priorities under the same sparse observations available to the student, rather than directly targeting evidence boundaries that may be unobservable. The framework supports different inference budgets. With a frozen 27B reasoner, the highest-effort configuration reports 74.8% on VideoMME, 79.2% on MLVU, and 58.7% on LVBench, with ablations examining candidate priorities, verification, and search supervision.

Critical strengths
  1. The strongest evidence is the improvement over the corresponding frozen reasoner, rather than comparisons across unrelated systems. The 4B+27B configuration improves VideoMME, MLVU, and LVBench by 6.0, 13.6, and 15.3 points, respectively. Improvements with a frozen 4B reasoner further support the value of the acquired evidence. These results demonstrate that a small model can contribute meaningfully beyond its standalone answering capability.

  2. The method appropriately separates proposing promising intervals from establishing whether those intervals contain useful evidence. Top-ranked candidates outperform lower-ranked ones, and verification improves performance by 1.1–2.2 points on the evaluated deferred subsets. Preserving partial evidence without requiring the verifier to solve the entire question is also a sensible design for multi-evidence reasoning.

Critical weaknesses
  1. The efficiency analysis uses parameter-weighted video-input sessions, excluding actual input lengths, continuation tokens, output generation, caching, and preprocessing. The appendix appropriately acknowledges these limitations. Nevertheless, even this proxy reaches 2.28×–2.77× the standalone baseline for Extra-High inference, while Medium costs 1.10×, 1.33×, and 2.19× across the three benchmarks. The paper therefore demonstrates a useful accuracy–resource trade-off, but does not establish superiority over alternative ways of allocating the same total computation, such as additional large-model observations or multi-window inference.

  2. Appendix E.2 defines “evidence-missing” examples using the same teacher-derived priority profile that supplies Seeker supervision. A low teacher relevance score is not an independent measurement of whether decisive evidence is visually absent. Furthermore, Seeker retains multi-turn supervision while the localization model predicts a single interval, despite both being evaluated with one interval. Consequently, Figure 7 may partly reflect teacher-defined partitioning and differences in supervision, rather than solely the advantage of observation-aligned search targets.

Questions for authors
  1. How does the framework compare with dense or multi-window 27B inference under matched end-to-end latency or GPU-time budgets, including all search, verification, and generation costs?
  2. Can evidence visibility be independently annotated from the actual sparse frames, and can the search/localization comparison match supervised-token budgets and the number of target intervals?
  3. What happens when the Verifier rejects every candidate, and how frequently does it discard evidence that becomes useful only in combination with another interval?
4 comments
AuthorsOpenReview ↗

Response to W1

We appreciate the reviewer’s valuable suggestion.

W1. How does the framework compare with dense or multi-window 27B inference under matched end-to-end latency or GPU-time budgets, including all search, verification, and generation costs?

To test under equivalent compute budget, we doubled the standalone 27B baseline’s uniform frame budget from 64 to 128 and compared it with our framework using the same frozen 27B Reasoner. We used concurrent inference in vLLM to keep both H200 GPUs fully utilized. The GPU time accounts for routing, search, verification, and final reasoning where applicable. The revised appendix reports the stage-wise breakdown.

Each cell shows accuracy (%) / GPU time (s per question).

ConfigurationVideoMMEMLVULVBench
Standalone 27B, 64 frames68.8 / 1.3265.6 / 1.3643.4 / 1.34
Standalone 27B, 128 frames72.8 / 1.5973.3 / 1.6745.5 / 1.62
Ours (Low)67.7 / 0.6070.1 / 0.8354.2 / 1.09
Ours (Medium)69.1 / 0.6472.6 / 0.8756.1 / 1.21
Ours (Extra-High)74.8 / 1.5579.2 / 1.5458.7 / 1.63

Extra-High stays within 5% of the 128-frame baseline’s GPU time on every benchmark and improves accuracy by 2.0, 5.9, and 13.2 percentage points, respectively. Moreover, visual tokens are a major contributor to GPU memory use in video QA. Obviously, Increasing the baseline to 128 frames can raise peak GPU memory demand, whereas our framework scales test-time computation without requiring a larger peak GPU memory budget.

Relative to the 64-frame baseline, Extra-High uses 1.13–1.22× the measured GPU time, compared with 2.28–2.77× under the original parameter-weighted session proxy. One reason for the lower ratios is output generation. The 27B Reasoner generates a reasoning trace, and the Router, Seeker, and Verifier emit only short, structured outputs. The original proxy omitted this decoding cost, which increased the apparent cost of the small-model stages relative to standalone 27B inference.

AuthorsOpenReview ↗

Response to W2

W2. Evidence visibility and matched search/localization supervision. We reran the ablation with visibility labels derived from the actual sparse frames, independently of the teacher priorities used to train Seeker. We first sampled each video globally at 2 fps with a 64-frame cap, then asked Qwen3.8-27B to answer using only the sampled frames near the annotated evidence interval. We labeled an example as evidence-missing when the model answered incorrectly from these sparse frames and correctly from denser local frames around the same interval.

For each training mixture, EL and Seeker used the same source QA examples, initialization, and training settings. We matched the supervision budget at one target interval per example for both models. At evaluation, both predicted one interval, and we matched the interval durations before passing them to the same answering pipeline.

MethodEvidence-missing shareVideoMMEMLVU
EL0%54.152.8
EL50%52.7 (−1.4)50.3 (−2.5)
Seeker0%53.653.4
Seeker50%55.6 (+2.0)55.1 (+1.7)

EL and Seeker perform similarly with 0% evidence-missing examples. Raising the share to 50% lowers EL accuracy on both benchmarks and improves Seeker accuracy. The original trend therefore holds with independently derived visibility labels and the same number of supervised intervals. We have replaced the original ablation in the main text with this more closely matched comparison and updated the appendix with the experimental protocol.

AuthorsOpenReview ↗

Response to W3

W3. What happens when the Verifier rejects every candidate, and how frequently does it discard evidence that becomes useful only in combination with another interval?

If the Verifier rejects every candidate, the frozen Reasoner answers from frames sampled uniformly across the full video under the same 64-frame budget.

Besides, we agree that verifying intervals independently can discard evidence that is useful only in combination. In the revised appendix, we analyze 692 Router-deferred VideoMME questions with at least one retained and one rejected interval. For the 247 questions answered incorrectly using the retained intervals alone, we test each rejected interval in isolation and then add it back to the retained set.

In 79 of the 692 questions (11.42%), at least one rejected interval led to a correct answer when combined with the retained evidence, even though Reasoner answered incorrectly with either input alone. This identifies a limitation of interval-wise verification. Evaluating candidate intervals in the context of the evidence already retained is a valuable direction for future work.

Thank you again for your valuable suggestions. Your suggestions helped us make the comparison between evidence search and evidence localization more rigorous and better isolate the effect of the supervision target under sparse observations. We would be appreciate to clarify any remaining points during the discussion period.

AuthorsOpenReview ↗

Follow-up on Our Responses

Dear Reviewer VscU,

Thank you once again for your thoughtful comments on our submission. As the discussion phase is approaching its end, we would appreciate your feedback on whether our responses have sufficiently addressed your concerns.

We have responded to each point by adding a GPU-time comparison with a denser 27B baseline, rerunning the search–localization experiment with independently assessed evidence visibility and matched interval supervision, and clarifying the fallback strategy while quantifying complementary evidence rejected by the Verifier. These additions are included in the revised manuscript.

If any concerns remain, we would be grateful for the opportunity to clarify them.

Thank you for your time and consideration.

Reviewer_kDssOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a collaborative framework for long-video QA that decouples evidence acquisition from answer generation. A 4B vision-language model acts as the Router, Seeker, and Verifier, and a frozen model answers from the selected frames. The Router handles directly answerable questions, the Seeker proposes candidate intervals from sparse observations, and the Verifier selects evidence from dense local observations.

The paper distinguishes evidence search from evidence localization. Instead of supervising exact evidence boundaries, it uses teacher judgments under the same sparse observations to prioritize informative intervals.

Critical strengths

1.The paper tackles a well-motivated problem: combining the reasoning ability of large models with the efficiency of small ones. The hierarchical design assigns evidence acquisition and verification to a small model and final reasoning to a large model, effectively leveraging their complementary strengths.

2.The framework achieves substantial gains on MLVU and LVBench (Table 1), demonstrating its effectiveness.

  1. The methodology is coherent and supported by extensive experiments against multiple baselines on VideoMME, MLVU, and LVBench.
Critical weaknesses
  1. The framework is evaluated only with Qwen models. Experiments with other open-source backbones, such as InternVL and LLaVA, are needed to assess its generalizability.

2.The proposed framework offers only a marginal gain over the state-of-the-art method VideoSEAL in the 4B+4B configuration, whereas the 4B+27B configuration yields substantial improvements. This raises the concern that the gains stem from the stronger answering model rather than the framework itself. Comparisons under matched model sizes and computational budgets are needed to isolate the framework’s contribution.

3.The ablations over model configurations are limited. Systematic experiments with larger evidence-acquisition models or smaller answering models would clarify how model capacity, and its allocation between the two stages, affects performance.

Questions for authors

Please see the weakness

11 comments
AuthorsOpenReview ↗

Response to W1

We appreciate the reviewer’s valuable suggestion.

W1. The framework is evaluated only with Qwen models. Experiments with other open-source backbones, such as InternVL and LLaVA, are needed to assess its generalizability.

With the compute available during the rebuttal period, we trained a naive model of our evidence-acquisition module on InternVL3.5-4B for 100 steps with a batch size of 128. Our standard training recipe uses one full epoch with a batch size of 256. We then paired this same briefly trained module with two frozen Reasoners, InternVL3.5-4B and Qwen3-VL-4B, to test whether it could improve both within-family and cross-family configurations.

Our standalone InternVL scores were lower than those reported in the repository, possibly due to differences in visual input budgets, prompt templates, or other inference settings. We include the repository scores for reference. To ensure a fair comparison, we evaluated our standalone and evidence-guided InternVL runs under identical answering conditions, using the same frozen Reasoner, QA prompt, visual input budget, and decoding settings. The only difference was which video frames the Reasoner received.

Evidence acquisitionFrozen ReasonerVideoMMEMLVULVBench
Standalone (repository-reported)InternVL3.5-4B64.870.443.1
Standalone (our run)InternVL3.5-4B57.263.338.8
InternVL3.5-4B (ours)InternVL3.5-4B61.2 (+4.0)70.8 (+7.5)47.9 (+9.1)
Standalone (our run)Qwen3-VL-4B61.562.633.6
InternVL3.5-4B (ours)Qwen3-VL-4B63.9 (+2.4)72.8 (+10.2)48.8 (+15.2)

Parentheses show gains in percentage points over our standalone run with the same Reasoner. The InternVL evidence-acquisition module, trained for 100 steps, improves both frozen Reasoners across all three benchmarks.

AuthorsOpenReview ↗

Response to W2

W2. The proposed framework offers only a marginal gain over the state-of-the-art method VideoSEAL in the 4B+4B configuration, whereas the 4B+27B configuration yields substantial improvements. This raises the concern that the gains stem from the stronger answering model rather than the framework itself. Comparisons under matched model sizes and computational budgets are needed to isolate the framework’s contribution.

The comparison with full VideoSEAL reflects differences in model capacity, visual input, and preprocessing. VideoSEAL uses the much larger DeepSeek-V3.2 to filter retrieved evidence. Both systems cap video inputs at 64 frames, but VideoSEAL uses a higher frame resolution. It also captions and embeds every 16-second clip in each video, adding heavy offline computation.

Specifically, we increased the per-frame pixel cap of our 4B+4B configuration from 32 × 32 × 64 to 32 × 32 × 320.

ConfigurationPixels per frame (cap)VideoMME OverallVideoMME LongMLVULVBench
VideoSEAL (8B + 7B + DeepSeek-V3.2)32 × 32 × 32062.953.468.255.1
VideoSEAL (8B + 7B, without DeepSeek-V3.2)32 × 32 × 320———51.2
Ours (4B + 4B, original)32 × 32 × 6464.752.974.350.7
Ours (4B + 4B, higher resolution)32 × 32 × 32066.056.375.254.5

At the higher pixel budget, our 4B+4B configuration exceeds full VideoSEAL by 3.1, 2.9, and 7.0 percentage points on VideoMME Overall, VideoMME Long, and MLVU, respectively. On LVBench, removing the DeepSeek-V3.2 filter reduces VideoSEAL’s reported score from 55.1 to 51.2. Our 4B+4B reaches 54.5, exceeding that ablation by 3.3 points and coming within 0.6 points of the full system. In summary, With comparable per-frame pixel budgets, our framework shows a clear accuracy advantage over full VideoSEAL on VideoMME Overall, VideoMME Long, and MLVU, and outperforms the no-DeepSeek variant on LVBench.

AuthorsOpenReview ↗

Response to W3

W3. the ablations over model configurations are limited. Systematic experiments with larger evidence-acquisition models or smaller answering models would clarify how model capacity, and its allocation between the two stages, affects performance.

We added a 2×2 experiment to the main results, pairing a 4B or 8B Reasoner with a Seeker and Verifier that are scaled jointly from 4B to 8B.

Seeker & VerifierReasonerVideoMME OverallMLVU M-AvgLVBench OverallMean
4B4B64.774.350.763.2
8B4B65.674.050.863.5
4B8B67.775.353.065.3
8B8B68.975.454.166.1
Scale Seeker & Verifier+1.1−0.1+0.6+0.5
Scale Reasoner+3.2+1.2+2.8+2.4

The last two rows report gains in percentage points, averaged over both sizes of the other stage. Scaling the Reasoner yields larger gains on every benchmark, averaging 2.4 points versus 0.5 from scaling the Seeker and Verifier. The 4B Seeker and Verifier paired with an 8B Reasoner also outperform the reverse allocation on all three benchmarks. These results support our design choice to use smaller models for evidence acquisition and allocate more capacity to reasoning over the selected evidence. We have included this particularly valuable experiment in the main results of the revised manuscript 3.2 Section to clarify how model capacity contributes at each stage.

Thank you again for your suggestions. The experiments you prompted allowed us to verify that the framework’s gains extend to InternVL, and identify which stage benefits most from additional model capacity. We would welcome any follow-up questions and would be glad to address them during the discussion period.

Reviewer_9JKJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a small–large model collaboration framework for long-video understanding. Specifically, a small model searches for and verifies visual evidence, and a frozen large model produces the final answer. The small model is trained using teacher-generated supervision to predict which video intervals to inspect next based on sparse visual observations. Experiments on three benchmarks show improved accuracy over standalone inference and flexible accuracy–cost trade-offs.

Critical strengths
  1. The paper makes an interesting observation: a small model can find useful evidence even when it cannot answer the question correctly. This gives a clear motivation for separating evidence acquisition from final reasoning.
  2. The distinction between searching for evidence and locating known evidence is well explained. Supervising where to look next based on visible cues is a sensible choice when sparse frames may miss the decisive details.
Critical weaknesses
  1. The comparison with existing methods is not comprehensive, omitting closely related approaches such as VideoSeek[A]. Moreover, most baselines in Table 1 use models of around 8B parameters, while the proposed method uses a 27B reasoner, raising concerns about whether the comparison adequately controls for model capacity. Moreover, including more benchmarks, such as LongVideoBench, EgoSchema, would help establish the framework's generalizability.

[A] Jingyang Lin, Jialian Wu, Jiang Liu, Ximeng Sun, Ze Wang, Xiaodong Yu, Jiebo Luo, Zicheng Liu, and Emad Barsoum. Videoseek: Long-horizon video agent with tool-guided seeking. arXiv preprint arXiv:2603.20185, 2026.

  1. If the large model were replaced with a more advanced model, such as GPT-6, would the gains provided by the small model still be substantial? Since the paper’s core contribution is introducing a trained small model for evidence acquisition, demonstrating continued benefits with advanced models would strengthen the case for the framework’s value.

  2. The gains from evidence acquisition are not fully disentangled from the benefits of additional visual observations. A comparison under a matched total compute budget is needed to clarify the source of these gains.

  3. The verifier assesses each clip independently, and the paper does not quantify whether this leads to discarding evidence that is useful only in combination with other clips.

Questions for authors

See Weaknesses.

6 comments
AuthorsOpenReview ↗

Response to W1

We thank the reviewer for the careful reading and constructive feedback.

W1. The comparison with existing methods is not comprehensive, omitting closely related approaches such as VideoSeek[A]. Moreover, most baselines in Table 1 use models of around 8B parameters, while the proposed method uses a 27B reasoner, raising concerns about whether the comparison adequately controls for model capacity. Moreover, including more benchmarks, such as LongVideoBench, EgoSchema, would help establish the framework's generalizability.

(1) Comparison with VideoSeek. The revised manuscript cites VideoSeek[A] and compares its results with ours in Table 4 of Section 3.2.

Please see the response to W2.

[A] Jingyang Lin, Jialian Wu, Jiang Liu, Ximeng Sun, Ze Wang, Xiaodong Yu, Jiebo Luo, Zicheng Liu, and Emad Barsoum. Videoseek: Long-horizon video agent with tool-guided seeking. arXiv preprint arXiv:2603.20185, 2026.

(2) Additional benchmarks. We added LongVideoBench and EgoSchema subset. On both benchmarks, Extra-High improves over standalone inference with the same frozen Reasoner.

ConfigurationLongVideoBenchEgoSchema (subset)
4B standalone56.561.2
Ours 4B+4B (Extra-High)64.667.0
27B standalone59.469.8
Ours 4B+27B (Extra-High)73.476.4

(3) Comparison with 8B Reasoners. To address the capacity concern, we also evaluated a frozen 8B Reasoner with the same 4B Seeker and Verifier. The 4B+8B configuration exceeds ParaVT on all three benchmarks and VideoSEAL on VideoMME and MLVU, and outperforms the no-DeepSeek variant on LVBench.

MethodModel configurationVideoMME (Overall)MLVU (M-Avg)LVBench
ParaVT8B62.165.039.8
VideoSEAL8B + 7B + DeepSeek-V3.262.968.255.1
VideoSEAL (w/o DeepSeek filter)8B + 7B——51.2
Ours4B + 4B64.774.350.7
Ours4B + 8B67.775.353.0
AuthorsOpenReview ↗

Response to W2

W2. If the large model were replaced with a more advanced model, such as GPT-6, would the gains provided by the small model still be substantial? Since the paper’s core contribution is introducing a trained small model for evidence acquisition, demonstrating continued benefits with advanced models would strengthen the case for the framework’s value.

We evaluated DeepSeek Flash as the Reasoner, with each input frame capped at 32 × 32 × 64 pixels. Table 4 of the revised manuscript reports the following results.

MethodModels usedLVBenchVideoMME-LongLongVideoBench-Long (val.)
StandaloneGPT-560.167.964.5
StandaloneDeepSeek Flash53.667.758.6
MR. VideoGemini 2.0 Flash + GPT-4o60.863.461.6
VideoSeekGPT-568.470.173.5
OursQwen3-VL-4B + DeepSeek Flash63.4 (+9.8)70.7 (+3.0)67.0 (+8.4)

Notably, VideoSeek uses GPT-5 to plan its search, and MR. Video uses GPT-4o for planning and final synthesis, with Gemini 2.0 Flash for visual analysis. In our framework, a much smaller 4B model handles both evidence search and verification. We call DeepSeek Flash only to answer the question using the selected evidence.

Although differences in commercial model capabilities, visual input budgets, and inference settings limit direct comparisons across methods, our results show that evidence gathered by 4B model still delivers substantial gains when a much larger commercial Reasoner is used, yielding competitive accuracy across all three long-video benchmarks.

AuthorsOpenReview ↗

Response to W3

W3. The gains from evidence acquisition are not fully disentangled from the benefits of additional visual observations. A comparison under a matched total compute budget is needed to clarify the source of these gains.

To test under equivalent compute budget, we doubled the standalone 27B baseline’s uniform frame budget from 64 to 128 and compared it with our framework using the same frozen 27B Reasoner. We used concurrent inference in vLLM to keep both H200 GPUs fully utilized. The GPU time accounts for routing, search, verification, and final reasoning where applicable. The revised appendix reports the stage-wise breakdown.

Each cell shows accuracy (%) / GPU time (s per question).

ConfigurationVideoMMEMLVULVBench
Standalone 27B, 64 frames68.8 / 1.3265.6 / 1.3643.4 / 1.34
Standalone 27B, 128 frames72.8 / 1.5973.3 / 1.6745.5 / 1.62
Ours (Low)67.7 / 0.6070.1 / 0.8354.2 / 1.09
Ours (Medium)69.1 / 0.6472.6 / 0.8756.1 / 1.21
Ours (Extra-High)74.8 / 1.5579.2 / 1.5458.7 / 1.63

Extra-High stays within 5% of the 128-frame baseline’s GPU time on every benchmark and improves accuracy by 2.0, 5.9, and 13.2 percentage points, respectively. Moreover, visual tokens are a major contributor to GPU memory use in video QA. Obviously, Increasing the baseline to 128 frames can raise peak GPU memory demand, whereas our framework scales test-time computation without requiring a larger peak GPU memory budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.