acceptodds
Withdrawn from ICLR 2027

DynFrame: Adaptive Reasoning-Driven Multimodal Framework with Dynamic Frame Augmentation for Complex Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent video multimodal large language models (MLLMs) increasingly couple step-by-step reasoning with on-demand visual evidence retrieval, allowing models to revisit relevant video segments during inference. However, two structural gaps remain in existing thinking-with-video systems. (i) Sampling density is not a learnable decision: existing methods may let the model decide where to look, but the per-window frame rate is largely fixed. As a result, fine-grained evidence is often recovered through repeated retrieval calls or overly dense frame sampling, both of which increase inference context length and training difficulty. (ii) Retrieval and answer generation are usually optimized with a single trajectory-level advantage, so the "where to look" tokens and the "how to answer" tokens receive the same credit even when one is correct and the other is not. To address these gaps, we present DynFrame, a framework that emits the temporal window and the sampling density as native tokens within a single autoregressive pass; this learnable span–density retrieval acquires multi-granularity evidence with a single retrieval step. Based on the above tokenized retrieval interface, we further introduce Segment-Decoupled GRPO (SD-GRPO), which splits each rollout at the retrieval boundary and assigns role-specific token-level advantages, separately crediting the sampling decision and the answer. Trained on the curated DM-CoT-74k and DM-RL-45k, DynFrame-4B is competitive with strong 7B–8B baselines across six benchmarks (NExT-GQA, Charades-STA, ActivityNet-MR, Video-MME, MLVU, LVBench), and DynFrame-8B sets a new state of the art on most metrics. Code, data, and model weights will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video understanding, dynamic retrieval, multimodal large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PkzbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DynFrame makes both the temporal window and the sampling rate part of a video model’s generated retrieval command. Starting from uniformly sampled frames, the model requests one additional clip, receives its visual embeddings, and continues reasoning. Training combines supervised retrieval-and-reasoning trajectories with SD-GRPO, which adds a separately normalized retrieval advantage before frame injection while retaining answer-derived feedback throughout. Evaluation covers six benchmarks for temporal grounding, grounded VideoQA, and long-video understanding.

Critical strengths
  1. Learning the temporal window and sampling density together is a useful extension of video retrieval. The model can request different observation granularities through a compact interface within its reasoning trajectory. The dynamic-density ablation in Table 2d improves over both tested fixed rates, providing evidence that this design helps answer questions requiring different amounts of temporal detail.

  2. The temporal grounding results are convincing relative to the backbone. On ActivityNet-MR, DynFrame-8B obtains 51.5 mIoU compared with 36.2 for Qwen3-VL-8B (Table 1). The evaluation covers several task families, and the sparse-observation experiment shows that the framework can retain performance when the initial video pass contains less evidence. These results support the practical usefulness of retrieving additional frames during reasoning.

Critical weaknesses
  1. I am not yet convinced that the RL gains establish the benefit of segment-specific credit assignment, which is a central contribution. Table 2b shows that SD-GRPO improves over vanilla GRPO, but the baseline’s exact training reward is not stated clearly enough to establish that it receives the same retrieval supervision. Figure 4 evaluates both methods with the same diagnostic reward definitions; that does not establish that those rewards were used identically during optimization. The missing control is to retain the same separately normalized sampling and answer advantages, apply their sum throughout the generated response, and then change only the segment assignment. Until this is tested, the improvement could come from explicit retrieval feedback rather than the proposed credit-assignment mechanism.

  2. The benefit of learned sampling density is not established at comparable visual cost. Table 2d supports higher accuracy than the tested fixed rates, but changing FPS also changes the number of retrieved frames. Average frame counts, visual-token counts, and end-to-end latency are not reported, so I cannot assess whether the policy allocates evidence better for a given budget. This matters because reducing repeated retrieval and context growth is part of the motivation, while the implementation rebuilds the KV cache by prefilling the full extended context after retrieval. A comparison against fixed density with a comparable average visual budget would help establish the value of learning density rather than simply choosing a different amount of visual input.

Questions for authors
  1. What exact reward does vanilla GRPO optimize in Table 2b? Does its training reward include the same span and FPS feedback as SD-GRPO?

  2. Does SD-GRPO still improve when the sum of the same separately normalized sampling and answer advantages is applied throughout the generated response, with the other training settings held fixed?

  3. What are the average retrieved frame counts, visual-token counts, and inference latency for dynamic versus fixed density? Does the accuracy advantage persist when their average visual budgets are comparable, and how often does the retrieval frame cap activate?

I am open to increasing my score if the authors can provide convincing responses to the above questions and sufficiently address my concerns.

2 comments
AuthorsOpenReview ↗

## Response to Reviewer Pkzb

Thank you for specifying the controls needed to strengthen the attribution and for your willingness to consider raising your score.

W1/Q1/Q2 — Reward definition and credit assignment. Vanilla GRPO in Table 2(b) uses the answer reward only, without span/FPS feedback. We agree that this baseline alone does not isolate credit assignment. The normalization-matched control retains the same separately normalized sampling and answer-plus-format advantages and applies their sum to every generated token.

We conducted an additional GRPO experiment using Ranswer+RsampleR_{\mathrm{answer}} + R_{\mathrm{sample}} with a shared advantage. As shown in the updated Figure 4, its final answer reward is lower than that of GRPO trained with RanswerR_{\mathrm{answer}} alone. Its sampling reward improves overall and approaches that of SD-GRPO, whereas its answer reward initially rises and then declines. This suggests that applying sampling feedback to post-injection reasoning may hinder answer learning, even as sampling reward improves.

We further compare these variants on NExT-GQA, Charades-STA, MLVU and LVBench. SD-GRPO consistently outperforms both controls across all four benchmarks, whereas the shared-advantage variant does not improve upon answer-only GRPO. Together with the normalization-matched control, these results support the importance of segment-specific credit assignment beyond adding sampling feedback alone in our setting.

VariantNExT-GQA Acc.Charades-STA mIoUMLVU M-AvgLVBench Acc.
Answer-only GRPO78.361.076.256.5
Same rewards/normalization, shared advantage78.060.975.956.4
SD-GRPO79.361.777.156.9

W2/Q3 — Visual budget and inference cost. In our training implementation, the coarse observation and retrieved clip are each capped at 128 frames, giving at most 256 sampled frames per trajectory. The visual-token count is computed from the sampled frame count and resized resolution according to Qwen3-VL's spatial and temporal merging scheme. Our updated cap statistics show that only 0.1% / 0.9% / 1.6% of retrievals exceed the cap on NExT-GQA, Video-MME and MLVU, respectively. So cap-induced changes to sampling density affect only a small minority of retrievals.

BenchmarkRetrievalsRequested frames, meanRequests above 128 (rate-derived)Cap activation (%)
NExT-GQA5,5001550.1
Video-MME2,70017240.9
MLVU5091981.6

To assess inference latency under different FPS settings, we first analyze DynFrame's predicted retrieval rates across all evaluated benchmarks, obtaining a mean FPS of 2.6. Based on this distribution, we compare the Qwen3-VL backbone and DynFrame with fixed 2 and 3 FPS. As shown in the table, DynFrame has lower reported inference latency than the backbone on NExT-GQA and LVBench, indicating that DynFrame can retrieve clearer visual evidence, which enables more focused and efficient reasoning. Compared with fixed 2 and 3 FPS, dynamic FPS keeps inference latency on the same order of magnitude while improving performance on all four benchmarks (Table 2(d)).

Model / retrieval policyNExT-GQA (s/query)Charades-STA (s/query)Video-MME (s/query)LVBench (s/query)
Qwen3-VL-8B, no retrieval12.510.115.319.5
DynFrame-8B, fixed 2 FPS11.217.119.017.2
DynFrame-8B, fixed 3 FPS11.418.321.418.9
DynFrame-8B, predicted FPS12.015.916.615.5

Moreover, we compare DynFrame with other retrieval-based models on the long-video understanding benchmarks LVBench and Video-MME, running each baseline with the inference settings in its official scripts. As shown in the table below (Table 7 in Appendix B), DynFrame-8B has the lowest inference latency on both benchmarks and processes 5.3–6.3× fewer tokens than the other methods, including 6.3× fewer than VideoChat-R1.5. Visual revisits and token counts are averaged over the two benchmarks.

MethodSizeVisual revisitsAverage total processed tokens (k)LVBench latency (s/query)Video-MME latency (s/query)
VideoChat-R1.5 (3 perception rounds)7B290.9156.3160.6
LOVE-R1 (zoom-in)7B1.276.782.968.5
LongVT-RFT (tool calls)7B1.085.736.330.8
DynFrame8B114.515.516.6

I thank the authors for their response and have decided to maintain my original score.

Reviewer_G72MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DynFrame is a video reasoning framework that predicts both the temporal window and sampling rate for retrieving additional visual evidence during inference. Retrieved frames are injected into the reasoning context to support subsequent answer generation. To improve training, the authors introduce Segment-Decoupled GRPO (SD-GRPO), which adds retrieval-specific feedback to the sampling segment while retaining answer-based feedback throughout the generated trajectory. Two curated datasets support supervised fine-tuning and reinforcement learning, respectively. Across six video understanding benchmarks, DynFrame achieves generally competitive results against recent retrieval-augmented methods. Ablation studies examine adaptive sampling, frame injection, and the proposed reinforcement learning objective.

Critical strengths
  1. DynFrame jointly predicts temporal windows and sampling rates, allowing the model to adjust the density of retrieved visual evidence during reasoning. This design aims to better capture brief or fast-changing events that sparse sampling may miss. The comparisons with fixed FPS further support the benefit of adaptive sampling.

  2. Results across six video understanding benchmarks are competitive with recent retrieval-augmented approaches. DynFrame-8B achieves 61.7 mIoU on Charades-STA and 51.5 mIoU on ActivityNet-MR, outperforming several reported baselines. The comparison with SFT without executable retrieval in Table 2(a) also shows that access to additional visual evidence provides clear benefits beyond textual retrieval supervision alone.

Critical weaknesses
  1. Using a tokenized span–FPS interface is a reasonable design. However, FPS annotations already follow activity-dependent guidelines, while the reported comparisons of learned FPS mainly use fixed sampling rates. Although Appendix A.3 further supports the benefit of density prediction, it remains unclear how much additional value the learned policy provides over a simpler content-adaptive sampling strategy.

  2. Equation (7) does assign different advantages to the two segments, and Table 2(b) reports improvements over vanilla GRPO. However the comparison does not distinguish whether these gains come from segment-specific credit assignment or simply from introducing an additional retrieval reward.

Questions for authors
  1. Beyond the activity-dependent guidelines used for annotation, does the learned FPS policy make different decisions for questions requiring different visual details within similar video segments? Examples of such cases would help clarify what the model learns beyond the predefined sampling preferences.

  2. In Equation (7), the sampling segment retains the answer advantage even when retrieval is inaccurate. How does SD-GRPO handle cases where the model retrieves an incorrect segment but still produces the correct answer? Could conflicting sampling and answer feedback affect the learned retrieval policy?

1 comment
AuthorsOpenReview ↗

## Response to Reviewer G72M

Thank you for recognizing the benefits of executable retrieval and adaptive density.

W1/Q1 — Learned FPS versus content-adaptive sampling. To assess whether the learned FPS policy improves upon content-adaptive sampling, we evaluated a Gemini-based FPS selector using the same FPS-annotation prompt as our data-construction pipeline. We generated Gemini FPS labels for the evaluation examples and substituted them for DynFrame's predicted FPS, keeping the retrieval spans, downstream answering model and sampling configuration fixed, without retraining. Replacing the learned FPS with Gemini-selected FPS reduces downstream performance on the evaluated benchmarks, supporting the value of learning the policy beyond reapplying the annotation guidelines. As shown in the table below, DynFrame-8B performs better with its learned FPS policy than with Gemini-selected FPS across all four benchmarks. Through SFT and subsequent RL, DynFrame learns and further refines its FPS policy, which enables more fine-grained, question-dependent density selection.

FPS policyNExT-GQA Acc.Charades-STA mIoUVideo-MME Acc.LVBench Acc.
Replace predicted FPS with Gemini-selected FPS77.260.870.556.5
DynFrame-predicted FPS79.361.772.356.9

For a qualitative example, we use the same bracelet-display segment (227–229 s) and two counting questions. Following our rule-based FPS-selection prompt, Gemini selects 2 FPS for both, whereas DynFrame selects 1 FPS for bracelet counting and 4 FPS for diamond counting:

Question about the same segmentGemini FPSDynFrame FPS
How many bracelets does the woman display?21
How many diamonds are on the bracelet she displays?24

With the activity and temporal window unchanged, DynFrame varies density according to the visual detail requested by the question, rather than applying a fixed activity-dependent sampling preference.

W2 — Isolating SD-GRPO. We acknowledge that answer-only GRPO alone does not isolate credit assignment. To distinguish the benefit of adding sampling feedback from that of segment-specific credit assignment, we additionally trained a shared-advantage GRPO control using Ranswer+RsampleR_{\mathrm{answer}} + R_{\mathrm{sample}}, with the same separately normalized sampling and answer advantages as SD-GRPO but applying their sum to every generated token. As shown in the expanded Table 2(b) below, SD-GRPO consistently outperforms both GRPO baselines across all four benchmarks, whereas the shared-advantage variant does not improve upon answer-only GRPO.

VariantNExT-GQA Acc.Charades-STA mIoUMLVU M-AvgLVBench Acc.
Answer-only GRPO78.361.076.256.5
Shared-advantage GRPO78.060.975.956.4
SD-GRPO79.361.777.156.9

The updated Figure 4 further shows that the shared-advantage variant achieves a sampling reward approaching that of SD-GRPO, but its answer reward initially rises and then declines, ending below both answer-only GRPO and SD-GRPO. This suggests that improving sampling reward alone is insufficient: restricting sampling feedback to the pre-injection segment may help preserve answer learning.

Q2 — Correct answers with inaccurate retrieval. In SD-GRPO, retrieval tokens receive both sampling- and answer-derived advantages. Within the same group and for equal answer-derived advantage, a lower sampling reward yields a lower combined advantage for retrieval tokens, even when both answers are correct. Thus, inaccurate retrieval receives relatively less credit than accurate retrieval. When the combined advantage is negative, the update steers the policy away from incorrect sampling actions.

Reviewer_ejMQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DynFrame, which lets a video model predict both a temporal window and its sampling frame rate during reasoning. Retrieved frames are inserted before the model continues answering. The authors introduce two training datasets and SD-GRPO. This objective applies retrieval-specific feedback before frame injection and answer feedback throughout the trajectory. Experiments cover grounded question answering, temporal grounding, and long-video understanding with 4B and 8B models.

Critical strengths
  1. Jointly predicting the retrieval window and sampling density is intuitive. It gives the model explicit control over the visual evidence it receives.

  2. The evaluation covers six benchmarks and two model sizes. The gains on NExT-GQA and ActivityNet-MR are particularly encouraging.

  3. The appendix contains useful controls, including crossed span/density comparisons, alternative-teacher agreement, and density perturbations under human-annotated evidence windows.

Critical weaknesses
  1. My main concern is whether Table 2b isolates segment-specific credit assignment. SD-GRPO adds a span/FPS reward, but the exact reward used to train vanilla GRPO is unclear. Computing sampling rewards afterward for Figure 4 does not establish that both methods receive the same training signal. If vanilla GRPO uses only answer and format rewards, the comparison also changes supervision. A shared trajectory-level advantage with the same rewards is the relevant control.

  2. The treatment of injected metadata needs clarification. It says that the system inserts frame timestamps and vision delimiters. Equation (8) and Appendix A.2 retain these tokens in the policy loss and KL term. They appear to be observations rather than sampled policy actions, so their importance ratios need justification. Table 4 also shows a large performance collapse when metadata targets are masked. This makes the distinction consequential, although that ablation does not isolate the reinforcement-learning stage.

  3. The efficiency motivation lacks direct evidence. It rebuilds the cache over the complete extended context after retrieval. The paper reports no latency, realized frame counts, or visual-token budgets, and the maximum frame cap is unspecified. One retrieval call does not reduce the total cost. The cap can also cause the requested FPS to differ from the actual sampling density. The fixed-FPS results support accuracy improvements but leave the computational tradeoff unresolved.

Questions for authors

As shown in the weakness.

1 comment
AuthorsOpenReview ↗

## Response to Reviewer ejMQ

Thank you for the constructive review.

W1 — Reward supervision versus credit assignment. Vanilla GRPO in Table 2(b) uses the answer reward only. We chose this reward because we found that combining Ranswer+RsampleR_{\mathrm{answer}} + R_{\mathrm{sample}} and applying the sum of separately normalized sampling and answer advantages throughout the trajectory resulted in lower final answer reward and performance. The updated Figure 4 includes this additional variant. Although its sampling reward approaches that of SD-GRPO, its answer reward initially rises and then declines, ending below both answer-only GRPO and SD-GRPO. This suggests that broadcasting sampling feedback to post-injection reasoning tokens may adversely affect answer learning, despite improving the sampling reward.

We also compare these variants on four downstream benchmarks using DynFrame-8B, as shown in the expanded Table 2(b) below. SD-GRPO consistently outperforms both GRPO baselines across all four benchmarks, whereas the shared-advantage variant does not improve upon answer-only GRPO.

VariantNExT-GQA Acc.Charades-STA mIoUMLVU M-AvgLVBench Acc.
GRPO (RanswerR_{\mathrm{answer}} only)78.361.076.256.5
GRPO (Ranswer+RsampleR_{\mathrm{answer}} + R_{\mathrm{sample}}, shared advantage)78.060.975.956.4
SD-GRPO79.361.777.156.9

W2 — Injected metadata. Thank you for pointing this out. We agree that frame timestamps and vision delimiters are observations rather than sampled policy actions and should not contribute directly to the RL objective. In our training experiments, the performance collapse in Table 4 primarily results from removing metadata supervision during SFT. We suppose these SFT targets help align the injected frames with their timestamps and visual boundaries. By contrast, excluding injected metadata from the losses only during RL changes the final benchmark scores by approximately ±0.1 points, which we regard as a noise-level fluctuation. We retain metadata supervision during SFT and have revised the definition of Mi\mathcal{M}_i used in Eq. (8) to exclude system-injected observations from both the policy loss and KL penalty.

W3 — Computational cost. In our training implementation, the coarse observation and retrieved clip are each capped at 128 frames, giving at most 256 sampled frames per trajectory. The visual-token count is computed from the sampled frame count and resized resolution according to Qwen3-VL's spatial and temporal merging scheme. Our updated cap statistics show that only 0.1% / 0.9% / 1.6% of retrievals exceed the cap on NExT-GQA, Video-MME and MLVU, respectively. These requests are uniformly subsampled within the predicted window, so cap-induced changes to sampling density affect only a small minority of retrievals.

BenchmarkRetrievalsRequested frames, meanRequests above 128Capped ratio (%)
NExT-GQA5,5001550.1
Video-MME2,70017240.9
MLVU5091981.6

To assess inference latency under different FPS settings, we first analyze DynFrame's predicted retrieval rates across all evaluated benchmarks, obtaining a mean FPS of 2.6. So we compare the Qwen3-VL backbone and DynFrame with a fixed 2/3 FPS. We find DynFrame has lower inference latency than the backbone on NExT-GQA and LVBench, indicating that clearer visual evidence enables more focused and efficient reasoning, potentially offsetting part of the retrieval overhead.

When updating the KV cache after retrieval, we reuse the pre-computed prefix states, encode only the newly retrieved frames, and prefill only the uncached suffix. We also measured the full-context cache-rebuilding implementation, in which post-injection visual encoding and prefill account for 4–8% of inference time. This suggests that FPS-dependent visual processing may have a limited impact on overall inference latency. The performance ablation shows improved task performance with dynamic FPS, while the timing measurements identify encoding and prefill as a relatively small component of inference. The table below reports inference times for the backbone and fixed/dynamic FPS variants in seconds per query:

Model / retrieval policyNExT-GQA (s/query)Charades-STA (s/query)Video-MME (s/query)LVBench (s/query)
Qwen3-VL-8B, no retrieval12.510.115.319.5
DynFrame-8B, fixed 2 FPS11.217.119.017.2
DynFrame-8B, fixed 3 FPS11.418.321.418.9
DynFrame-8B, predicted FPS12.015.916.615.5

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.