acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Reasoning Routing for Efficient Online Video Object Segmentation

Abstract

Online reasoning video object segmentation aims to segment targets in an incoming video stream based on complex or implicit language queries. Maintaining accurate target grounding requires timely adaptation to evolving visual evidence. However, existing invocation schedules incur substantial reasoning costs without consistent accuracy gains. To address this, we propose Dynamic Reasoning Routing (DRR), a training-free framework that adaptively schedules semantic reassessment using motion-compensated residuals. By monitoring changes in the target and its context at low cost, DRR reduces redundant reasoning while remaining responsive to evolving visual evidence. To limit error accumulation during propagation, DRR further introduces confidence-weighted mask encoding, which softens masks from low-confidence reassessments. Experiments on five reasoning and referring VOS benchmarks show an improved accuracy–efficiency trade-off. On ReasonVOS, DRR improves accuracy at – the throughput of matched periodic baselines, and surpasses the prior art by & with a speedup. A paired-replay analysis further demonstrates that DRR provides greater average benefit for subsequent segmentation than periodic scheduling. Code will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Object Segmentation, Adaptive Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KkY1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dynamic Reasoning Routing (DRR), a training-free framework for online reasoning video object segmentation. It uses motion-compensated residuals to adaptively trigger semantic reassessment and confidence-weighted mask encoding to reduce error accumulation. Experiments on five benchmarks demonstrate an improved trade-off between segmentation accuracy and computational efficiency.

Critical strengths
  1. Experiments on five benchmarks demonstrate competitive performance against state-of-the-art methods, with substantial efficiency gains.

  2. The paper is well written and clearly organized.

Critical weaknesses
  1. Motion-compensated residuals may respond to illumination changes, deformation, or occlusion that do not alter the queried target, while missing subtle interactions or relevant events outside the monitored neighborhood. Since the monitored regions depend on the predicted mask, incorrect tracking may also compromise subsequent invocation decisions. The aggregate results are encouraging, but the reliability and failure conditions of this proxy need further analysis.

  2. The contribution of the scheduling policy is not fully separated from changes to memory management. Section 3.3 states that the baseline resets the propagation state when accepting new grounding, whereas DRR retains earlier references. However, Table 3 does not explicitly isolate this change from residual-guided scheduling and confidence-weighted encoding. Consequently, it remains unclear how much of the accuracy improvement comes from better invocation timing versus retaining historical information.

  3. The confidence-weighted memory mechanism relies on an insufficiently justified confidence proxy. The selector’s Yes/No probability measures its confidence that a frame is suitable for grounding, which does not necessarily indicate the accuracy of the mask produced by the grounding module. Its additional gain in Table 3 is modest, from 56.7 to 57.1, and fixed weights achieve similar results in Table 4. Moreover, the target-absent analysis still shows substantial degradation after reassessment, suggesting that the mechanism provides limited protection against unreliable updates.

Questions for authors
  1. How does residual-guided scheduling behave under irrelevant appearance changes, subtle semantic events, and incorrect target tracking? Could the authors report missed reassessments and the delay between relevant evidence appearing and successful target correction?

  2. Could the authors compare fixed, random, and adaptive scheduling under identical memory-management settings, and separately ablate resetting versus retaining historical memory?

  3. At comparable invocation budgets, how does the proposed residual signal compare with simpler alternatives, such as direct frame differences, residuals without motion compensation, and global residual aggregation?

  4. Given the substantial negative reassessment benefit when the target is absent, how does DRR handle target disappearance and reappearance? Could the authors report false-positive predictions during absence and recovery delays after reappearance?

  5. Please include the component-wise time cost and memory usage to support the contribution of each submodule.

4 comments

Thank you for your review. Our responses follow below.

Re: Residual reliability and recovery (W1, Q1). We agree that residual cues may respond to irrelevant appearance changes or miss relevant events. DRR includes safeguards for these failure modes through selector verification and fallback evaluation. First, appearance changes can trigger evaluation, but a No verdict leaves the propagated mask and memory unchanged, costing one 380 ms selector call. Normalization and target–context contrast suppress shared changes; the selector declines 13.8% of residual-triggered evaluations. Second, events outside the monitored neighborhood may not trigger evaluation, but the 32-frame maximum gap forces one and supplies the first evaluation in 19% of DRR’s failure episodes. Third, tracking errors can shift the monitored regions. An empty mask requests evaluation at the next frame permitted by the minimum gap; a wrong non-empty track remains subject to the maximum gap.

On ReasonVOS, DRR recovers 78.5% of tracking-failure episodes versus 71.6% for fixed-period scheduling. Mean recovery delay falls from 7.6 to 5.1 frames among recovered episodes, and the share receiving no evaluation falls from 45.9% to 31.2%.

ScheduleRecovered %Mean recovery framesNo evaluation %
Fixed period71.67.645.9
Random73.46.940.8
DRR78.55.131.2

An episode starts when target-present IoU falls below 0.5 after reaching that threshold and ends at recovery or disappearance. Each schedule generates its own episodes. These results measure tracking recovery, not delay from an independently annotated semantic event; the 19% figure measures fallback evaluation coverage, not successful correction. App. H.1 reports the analysis.

Re: Scheduling and memory (W2, Q2). With shared memory retention and comparable selector-call rates, DRR improves J&F by 5.5 over fixed-period scheduling and 2.1 over random scheduling, without confidence weighting. All results below use LLaVA-1.5-7B on ReasonVOS.

ScheduleMemorySelector calls %J&F
Fixed period, ξ=4\xi=4Reset25.452.0
Fixed period, ξ=4\xi=4Retain25.452.8
Fixed period, ξ=20\xi=20Reset5.550.1
Fixed period, ξ=20\xi=20Retain5.551.2
RandomRetain5.254.6
DRRRetain5.256.7

Keeping each fixed-period schedule unchanged, memory retention adds 0.8 J&F at ξ=4\xi=4 and 1.1 at ξ=20\xi=20. With retention shared and comparable call budgets, DRR still reaches 56.7 J&F versus 51.2 for fixed-period scheduling. Memory retention alone therefore does not explain DRR’s advantage. The original Table 3 retains memory for random and adaptive variants and preserves CoT-RVS’s published reset configuration. We have added fixed-period retention baselines to the revised Table 3 and clarified each variant’s memory policy in Sec. 4.4.

Re: Confidence weighting (W3). The reviewer highlights an important distinction between frame suitability and mask accuracy. Selector confidence measures frame suitability, and our results show that it is also informative about the accuracy of the resulting mask. Across confidence bins on ReasonVOS, mean grounding IoU rises from 0.41 to 0.78, while the share of target-absent updates falls from 24.4% to 3.3%. These trends support using selector confidence to control a grounding’s influence on memory.

Selector confidenceMean grounding IoUTarget-absent updates %
0.50≤pt<0.600.50\le p_t<0.600.4124.4
0.60≤pt<0.750.60\le p_t<0.750.4917.2
0.75≤pt<0.900.75\le p_t<0.900.629.1
0.90≤pt≤1.000.90\le p_t\le1.000.783.3

Further, confidence weighting improves segmentation by selectively attenuating low-confidence updates. It achieves 57.1 J&F, compared with 56.7 without weighting and 56.4 with a fixed weight of 0.5. The stratified replay shows why weighting by confidence helps. Low-confidence target-absent updates are more damaging before weighting, with a mean IoU loss of 15.8 versus 12.1 for high-confidence updates. Weighting attenuates the low-confidence group and reduces its loss to 13.3 IoU, a 16% relative reduction.

Importantly, the reported negative reassessment in Fig.7 reflects a local update effect measured by paired replay. It measures mean 16-frame IoU loss relative to skipping the update from the same saved state, with further reassessment blocked. We also report an online comparison with both branches continuing normal reassessment. The weighted loss is 4.6 IoU online versus 12.7 in replay; online loss without weighting is 7.3. Together, selective memory weighting and subsequent reassessment mitigate the downstream impact of incorrect groundings; see Re: Harmful updates on target-absent frames (W2, Q3), Reviewer oFGS. Sec. 3.3 in the revision clarifies confidence-weighted encoding, and App. E.1 reports these analyses.

Re: Simpler visual cues (Q3). We thank the reviewer for suggesting these comparisons. DRR achieves 56.7 J&F on ReasonVOS at the same 5.2% selector-call rate, outperforming uncompensated frame differences over the same target and context regions by 1.5 J&F, whole-frame motion-compensated residuals by 2.2 J&F, and direct whole-frame differences by 2.4 J&F. These comparisons show benefits from both motion compensation and regional aggregation; see Re: Frame differences (Q2), Reviewer KAx1.

SignalRegionJ&FJ&F, subset
Random—54.652.9
Frame differenceWhole frame54.352.4
Motion-compensated residualWhole frame54.553.1
Frame differenceTarget + context55.254.0
DRRTarget + context56.755.7

All variants share anchor, memory, bootstrap and fallback policies and omit confidence weighting. To examine clips most affected by alignment, we select the third with the largest reduction in pixel mismatch with the grounding anchor as the subset. DRR's gains are larger on these clips, at 3.3 J&F over direct whole-frame differences and 1.7 J&F over uncompensated target–context differences. We have expanded App. G.4 of the revised manuscript to include these comparisons.

Re: Disappearance and reappearance (Q4). DRR reduces false positives during absence and recovers faster after reappearance. The reasons are three fold. First, the target-absent loss in Fig. 7 measures local update damage with reassessment blocked for 16 frames. Allowing online reassessment reduces weighted loss from 12.7 to 4.6 IoU over the same horizon; see Re: Confidence weighting (W3) above. Second, EgoMask distinguishes target-present accuracy from performance including absence. On its medium and long subsets, DRR-LLaVA-1.5-7B has lower IoU_gold and T_recall but higher IoU_all than CoT-RVS, consistent with fewer spurious foreground predictions during absence. App. G.3 and Fig. 9 report these results. Third, the direct ReasonVOS comparison below uses retained references and comparable invocation budgets.

ScheduleAbsence false positives %Median recovery framesRecovered within 16 frames %
Fixed period74.61933.8
DRR51.8859.4

As shown, DRR lowers false positives from 74.6% to 51.8%, reduces median recovery delay from 19 to 8 frames, and raises recovery within 16 frames from 33.8% to 59.4%. Mechanistically, foreground or context changes can request reassessment, while an empty mask requests fallback evaluation subject to the minimum gap; these mechanisms provide further correction opportunities. App. H.1 reports this analysis.

Re: Time and memory per component (Q5). Residual computation and confidence-weighted encoding together require only 2.6 ms and approximately 0.43 GFLOPs per frame, with no additional model weights. The selector and reasoning segmenter run on 5.2% and 4.3% of frames, yet account for 47% of total component runtime and 63% of FLOPs. We measure the following costs on ReasonVOS using one H200, with GPU synchronization.

Componentms per frameGFLOPs per frameWeights MiBPeak activations MiB
Residual cues2.10.00300
Selector19.851013,4721,120
Reasoning segmenter49.592516,6722,350
Weighted encoding0.50.43020
Propagation76.08508561,640

Component costs total 147.9 ms, plus 1.1 ms framework overhead, giving 149 ms per frame and 6.7 FPS. Encoding includes the full mask-encoding stage for an accepted grounding. Peak GPU memory is 33,400 MiB, comprising 31,000 MiB of weights and approximately 2,400 MiB of activations and other runtime memory; component activation peaks are not additive. The two reasoning models dominate parameter memory, and reducing their invocation frequency provides the throughput gain. App. G.5 reports the full breakdown, with a summary in Sec. 4.4.

Reviewer_KkY1OpenReview ↗

reply to authors

I thank for the authors' efforts to provide the detailed explanation and additional experiments. Most of my concerns are well addressed. This paper provided a well-engineered pipeline with convincing results. However, this also raises another concerns about the technical contributions, since the proposed framework is composed with existing tools. I hope the authors can provide more explanations.

Thank you for your response. We respectfully note that building on established components does not preclude a methodological contribution, and prior works [R1, R2] likewise build on the same components. Deciding when to invoke costly reasoning models is an active research problem in streaming video understanding, with recent works [R3, R4] using learned decision mechanisms.

Our main contribution is to establish motion-compensated residuals, referenced to semantic grounding, as a training-free signal for allocating costly reasoning. The substantive finding is that these residuals identify opportunities for useful reassessment beyond what can be inferred from visual change alone. At the same invocation rate, motion compensation improves ReasonVOS J&F by 1.5 over uncompensated differences in the same target and context regions. On a common pool of saved states, DRR-selected reassessments yield a mean downstream IoU gain of 1.6, compared with 0.6 for frame-difference selection. Grounding refreshes the monitoring reference, closing this loop without a learned scheduling model. The policy’s transfer to language-specified tracking further supports its value as a reusable computation-allocation mechanism; see Re: Methodological contribution (W1), Reviewer KAx1. We welcome clarification on any remaining concerns.

[R1] “Decomposed Attention Fusion in MLLMs for Training-Free Video Reasoning Segmentation,” ICLR 2026.

[R2] “Towards Streaming Referring Video Segmentation via Large Language Model,” CVPR 2026.

[R3] “Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction,” CVPR 2025.

[R4] “StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant,” NeurIPS 2025.

Reviewer_oFGSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DRR is a training-free framework for online reasoning video object segmentation. It uses motion-compensated visual residuals to decide when to invoke a language-conditioned keyframe selector and grounding model, and propagates masks with SAM 2 between accepted groundings. The framework also retains earlier references and weights new mask logits by selector confidence. The experiments evaluate segmentation accuracy, inference cost, and the effect of different routing policies.

Critical strengths
  1. The training-free system improves the accuracy–speed trade-off, including in comparisons that use the same selector.

  2. The routing controls and paired replays examine when reassessment helps, rather than reporting only the final segmentation score.

Critical weaknesses
  1. The comparison with CoT-RVS changes two things at once: DRR changes the invocation schedule and retains earlier references rather than resetting propagation memory. Without a matched-memory comparison, the accuracy gain does not isolate the benefit of routing.

  2. On target-absent frames, confidence-weighted reassessment still causes substantial damage to later masks. The weighting only slightly reduces this loss, so it offers limited protection against storing an incorrect foreground mask in memory.

Questions for authors
  1. Can you compare fixed, random, and adaptive schedules with the same reference-retention rule and a matched invocation rate? This would separate the scheduling benefit from the change in propagation memory.

  2. Your replay pairs an update with a skip from the same state, but each scheduling policy supplies different states and frames. On a common set of saved states and frames, do the residual cues still select updates with greater downstream benefit?

  3. For accepted updates on target-absent frames, how does selector confidence relate to subsequent damage? Please separate high- and low-confidence cases to show whether harmful masks are actually downweighted.

  4. Please report latency separately for reassessment frames and propagation-only frames, including the slowest reassessment pauses. The sequence-average processing time does not show these interruptions.

2 comments

We thank the reviewer for the feedback and address each point below.

Re: Matched memory and call budgets (W1, Q1). We agree that matching the memory settings better isolates the contribution of scheduling policies. With shared memory retention and comparable selector-call rates, DRR improves J&F by 5.5 over fixed-period scheduling and 2.1 over random scheduling, without confidence weighting. All results below use LLaVA-1.5-7B on ReasonVOS.

ScheduleMemorySelector calls (%)J&F
Fixed period, ξ=4\xi=4Reset25.452.0
Fixed period, ξ=4\xi=4Retain25.452.8
Fixed period, ξ=20\xi=20Reset5.550.1
Fixed period, ξ=20\xi=20Retain5.551.2
RandomRetain5.254.6
DRRRetain5.256.7

Within fixed-period scheduling, retention adds 0.8 J&F at ξ=4\xi=4 and 1.1 at ξ=20\xi=20. With both methods retaining memory and using comparable call budgets, DRR still achieves 56.7 J&F versus 51.2 for fixed-period scheduling. Thus, memory retention alone does not explain DRR’s advantage. The original Table 3 retains memory for random and adaptive variants and preserves CoT-RVS’s published reset configuration. We have added fixed-period retention baselines to the revised Table 3 and clarified each variant’s memory policy in Sec. 4.4.

Re: Harmful updates on target-absent frames (W2, Q3). Confidence weighting reduces mean downstream IoU loss from 13.9 to 12.7 for target-absent updates, but does not prevent harmful updates. Before weighting, low-confidence updates with pt<0.75p_t<0.75 incur greater mean loss than high-confidence updates, 15.8 versus 12.1 IoU. Weighting attenuates the low-confidence group and reduces its loss to 13.3 IoU, a 16% relative reduction.

Selector confidenceIoU loss without weightingIoU loss with weighting
Low, pt<0.75p_t<0.7515.813.3
High, pt≥0.75p_t\ge0.7512.112.1
All target-absent updates13.912.7

Note that these losses compare updating with skipping from the same saved state over the next 16 frames, with further reassessment blocked. They measure local update damage, not a dataset-wide performance drop; target-absent updates account for 17.2% of replayed updates. The online comparison uses the same update events and starting states, but both branches continue normal reassessment. The online IoU loss is 7.3 without weighting and 4.6 with weighting, indicating that both weighting and subsequent reassessment help limit the damage caused by incorrect updates.

Target-absent updatesReplay IoU lossOnline IoU lossJ&F
No weighting13.97.356.7
Confidence weighting12.74.657.1
Oracle skip0057.9

At the dataset level, weighting improves J&F from 56.7 to 57.1 without ground truth. An oracle that uses ground-truth presence to skip target-absent updates reaches 57.9. Weighting therefore recovers 0.4 of the oracle’s 1.2 J&F gain, or one-third, while leaving room for further improvement. Sec. 3.3 clarifies this scope, and App. E.1 reports these analyses.

Re: Common-state paired replay (Q2). Yes they do! DRR selects updates with greater downstream benefit even when all rules use the same candidate pool. Each rule selects 5.2% of saved ReasonVOS states, with the same frames, memory banks and grounding anchors available to every rule. Selected states are those where a rule requests reassessment; unselected states are the remainder. For both groups, paired replay measures the mean 16-frame IoU gain from updating relative to skipping from the same state. An update is beneficial when this gain is positive.

Selection ruleIoU gain, selectedIoU gain, unselectedBeneficial selected %Beneficial unselected %
Random+0.4+0.49.89.7
Frame difference+0.6+0.410.99.6
DRR+1.6+0.317.99.3

DRR-selected states gain 1.6 IoU versus 0.3 for its unselected states, while random selection gains 0.4 in both groups. Beneficial updates account for 17.9% of DRR-selected states, compared with 9.3% of its unselected states and 9.8% of randomly selected states. These results show that DRR identifies more useful reassessments from the same available states. Full results are in App. E.2, referenced in Sec. 4.3.

Re: Latency of reassessment pauses (Q4). Propagation-only frames average 80 ms, whereas reassessment frames average 1.41 s, with a median of 1.48 s and a maximum observed latency of 1.92 s. We measure both frame types separately on ReasonVOS using one H200 and LLaVA-1.5-7B.

Frame typeFrames %Mean msMedian msMax ms
Propagation only94.88078121
Reassessment5.21,4101,4801,920
DRR overall100149781,920

Each reassessment adds a 380 ms selector call to propagation. A No verdict ends the reassessment, while a Yes verdict also invokes the reasoning segmenter for 1,150 ms. The 1.41 s mean includes both cases. DRR reduces the share of frames requiring reassessment from 25.4% to 5.2%, approximately 80% fewer pauses, while individual reassessments still take up to 1.92 s. Averaging 94.8% propagation-only frames and 5.2% reassessment frames gives 149 ms per frame. App. G.5 of the revised manuscript reports the separate latency and component measurements.

Reviewer_KAx1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Dynamic Reasoning Routing (DRR), a training-free framework for efficient online reasoning video object segmentation. DRR uses motion-compensated residuals within the tracked foreground and its surrounding context to determine when to invoke semantic reassessment, reducing redundant calls to a multimodal language model. It also retains previous grounding references and introduces confidence-weighted mask encoding to mitigate error accumulation from unreliable grounding updates. Experiments on five reasoning and referring segmentation benchmarks demonstrate improved accuracy–efficiency trade-offs. On ReasonVOS, the matched LLaVA-based configuration improves J&F from 52.0 to 57.1 while increasing throughput from 1.1 to 6.7 FPS.

Critical strengths
  1. Good accuracy and speed gains without additional training. With the same LLaVA selector, DRR improves ReasonVOS performance from 52.0 to 57.1 J&F while increasing throughput from 1.1 to 6.7 FPS. I find this a useful practical result, especially since it does not require retraining the existing models. The gains suggest that the invocation schedule deserves more attention in online reasoning segmentation.

  2. The routing ablation supports the main idea. At the same reported invocation rate of 5.2%, local-contrast routing achieves 55.2 J&F compared with 51.4 for random routing. Adding foreground residuals further improves performance to 56.7. This gives some evidence that the visual cues help identify useful reassessment times, rather than the improvement coming only from fewer reasoning calls.

Critical weaknesses
  1. Limited methodological novelty. I find the proposed routing approach reasonable, and the accuracy and speed gains are useful. However, the main contribution is a heuristic policy that uses motion-compensated residuals to decide when to invoke existing reasoning and segmentation models. While effective, I see this as an incremental extension of the existing pipeline, and I am not convinced that the methodological contribution is substantial enough for acceptance.
  2. The effect of routing is not sufficiently isolated. DRR changes both when reasoning is invoked and how propagation memory is maintained. CoT-RVS resets the propagation state after accepting a new grounding, whereas DRR retains earlier conditioning references. Table 3 does not include a fixed-period baseline with the same memory-retention policy. Without this comparison, I find it difficult to tell how much of the accuracy improvement comes from residual-based routing, which is the paper’s main contribution.
Questions for authors
  1. In Table 3, do the fixed-period, random, and adaptive variants use the same memory-retention policy? If not, could you report a fixed-period baseline that retains earlier conditioning references as DRR does? This would help clarify how much of the gain comes from routing rather than memory retention.
  2. How does DRR compare with a trigger based on frame differences relative to the same grounding anchor, but without motion compensation? Using the same memory policy, fallback rules, and a matched invocation rate would help show whether motion compensation improves over a simpler change-detection rule.
1 comment

Thank you for your constructive review. We address each point below.

Re: Methodological contribution (W1). DRR’s novelty lies in coupling semantic grounding with subsequent invocation decisions. Fixed-period invocation [R1] ignores intervening visual changes, while mask-consistency-based invocation [R2] relies on mask stability, which does not guarantee that the tracked target satisfies the query. In contrast, DRR monitors motion-compensated changes in the predicted target and its context relative to the last accepted grounding frame. Each accepted grounding updates both the segmentation and the visual reference used for subsequent invocation. This links fast perception and semantic reasoning [R3] [R4] through a feedback mechanism based on motion-compensated residuals, without an additional learned routing model [R5].

Beyond the original RVOS setting, the same invocation mechanism can guide when a tracking pipeline acquires new grounding. We demonstrate this by integrating DRR into the detector-based pipeline in [R6] for language-specified tracking on TNL2K [R7]. Replacing only the fixed detection interval with our residual trigger in Eq. (8) raises AUC from 49.6% to 52.9% at comparable detector-call rates of 5.0% and 4.9%, respectively. The revised manuscript discusses the design in Sec. 2 and reports the transfer results in App. G.4.

Detection scheduleCalls (%)AUC (%)P_norm (%)P (%)FPS
Every 4 frames25.051.257.650.99.8
Every 20 frames5.049.655.849.014.6
Random5.050.356.749.814.5
DRR4.952.959.652.514.2

Re: Memory retention (W2, Q1). With shared memory retention and comparable selector-call rates, DRR improves J&F by 5.5 over fixed-period scheduling and 2.1 over random scheduling, without confidence weighting. All results below use LLaVA-1.5-7B on ReasonVOS. Separately, keeping each fixed-period schedule unchanged, retaining memory raises J&F from 52.0 to 52.8 at ξ=4\xi=4 and from 50.1 to 51.2 at ξ=20\xi=20. Memory retention therefore provides gains of 0.8 and 1.1 J&F, while DRR’s scheduling advantage remains under shared retention and comparable call budgets.

ScheduleMemorySelector calls (%)J&F
Fixed period, ξ=4\xi=4Reset25.452.0
Fixed period, ξ=4\xi=4Retain25.452.8
Fixed period, ξ=20\xi=20Reset5.550.1
Fixed period, ξ=20\xi=20Retain5.551.2
RandomRetain5.254.6
DRRRetain5.256.7

Note that the original Table 3 retained memory for the random and adaptive variants while preserving CoT-RVS’s published memory-reset configuration. We have added the fixed-period memory-retention baselines to the revised Table 3 and clarified each variant’s memory policy in Sec. 4.4.

Re: Frame differences (Q2). We appreciate the question and compare DRR against a frame-difference trigger on ReasonVOS. For patch PP, direct differences are Etdiff(P)=1∣P∣∑x∈P∣Yt(x)−Yk(x)∣E_t^{\mathrm{diff}}(P)=\frac{1}{|P|}\sum_{x\in P}|Y_t(x)-Y_k(x)|, where YtY_t and YkY_k are the current and last accepted grounding frames’ downsampled luma images. All variants share anchor, memory, bootstrap and fallback policies, without confidence weighting, at a 5.2% call rate.

SignalRegionJ&FJ&F, subset
Random—54.652.9
Frame differenceWhole frame54.352.4
Motion-compensated residualWhole frame54.553.1
Frame differenceTarget + context55.254.0
DRRTarget + context56.755.7

DRR uses the same invocation budget more effectively than direct whole-frame differences, improving J&F by 2.4 overall and 3.3 on the subset that comprises the third of clips where alignment most reduces pixel mismatch with the grounding anchor. Motion compensation adds 1.5 J&F overall and 1.7 on the subset over target–context frame differences. We will expand App. G.4 with these comparisons.

[R1] “CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videos,” ICLR 2026.

[R2] “Towards Streaming Referring Video Segmentation via Large Language Model,” CVPR 2026.

[R3] “Neural Reasoning, Fast and Slow, for Video Question Answering,” IJCNN 2020.

[R4] “JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence,” 2026.

[R5] “RouteLLM: Learning to Route LLMs with Preference Data,” ICLR 2025.

[R6] “Grounded SAM 2: Ground and Track Anything in Videos,” official code repository, 2024.

[R7] “Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark,” CVPR 2021.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for recognizing DRR’s training-free accuracy–efficiency gains (Reviewers KAx1, oFGS, KkY1), analysis of reassessment benefits (Reviewers KAx1, oFGS), and clear presentation (Reviewer KkY1). The revisions clarify how grounding informs invocation, with controlled analyses showing scheduling gains and reduced damage from unreliable updates.

First, DRR’s novelty lies in using semantic grounding to guide invocation. Unlike scheduling from elapsed time or propagation-mask stability, DRR monitors motion-compensated changes in the predicted target and its context relative to the last accepted grounding frame. Each accepted grounding updates both the segmentation and the visual reference for subsequent invocation, linking perception and reasoning without an additional scheduling model. Beyond RVOS, replacing only the fixed detection interval in Grounded SAM 2 improves tracking accuracy on TNL2K at comparable detector-call rates, supporting transfer of the invocation mechanism. See Re: Methodological contribution (W1), Reviewer KAx1.

Second, the scheduling benefit persists under shared memory retention and comparable call budgets. Without confidence weighting, DRR outperforms fixed-period and random scheduling, while retention alone provides a smaller gain within fixed-period scheduling. Matched comparisons show benefits from both motion compensation and regional aggregation. On a shared pool of saved states, DRR selects updates with greater mean downstream benefit than random selection. These results support both the invocation design and its ability to select useful updates. See Re: Memory retention (W2, Q1) and Re: Frame differences (Q2), Reviewer KAx1, and Re: Common-state paired replay (Q2), Reviewer oFGS.

Third, confidence weighting and subsequent reassessment jointly reduce damage from unreliable updates. Weighting improves segmentation accuracy by reducing damage from low-confidence target-absent updates. Paired replay compares updating with skipping from the same saved state, with further reassessment blocked. Allowing reassessment reduces update damage further, with weighting providing an additional reduction. DRR also speeds recovery after target reappearance. See Re: Harmful updates on target-absent frames (W2, Q3), Reviewer oFGS, and Re: Disappearance and reappearance (Q4), Reviewer KkY1.

Summary of revisions to the manuscript

Revised text is highlighted in blue.

  • Methodological contribution: Accepted grounding supplies the visual reference for subsequent invocation, linking perception and reasoning without a learned scheduling model (Sec. 2).

  • Scheduling and memory: Comparisons with shared memory retention distinguish scheduling gains from retention gains (revised Table 3; Sec. 4.4).

  • Frame differences and transfer: Matched comparisons quantify gains from motion compensation and regional aggregation; TNL2K demonstrates transfer beyond RVOS (App. G.4; Sec. 4.4).

  • Confidence and selection quality: Confidence and IoU analyses, stratified replay, online and oracle comparisons, and selection from shared states characterize grounding quality and downstream update effects (Apps. E.1 and E.2; Secs. 3.3 and 4.3).

  • Failure and recovery: Failure episodes, absence errors and reappearance delays quantify system behavior; unrecovered events remain in recovery-rate denominators (App. H.1).

  • Computational cost: Separate frame latencies and component time, FLOPs and memory distinguish average throughput from reassessment pauses (App. G.5; Sec. 4.4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.