acceptodds
Under review as a conference paper at ICLR 2027

VideoSlate: Stateful Commit Control for Temporally Grounded Agentic Video Reasoning

Abstract

Agentic video reasoning systems gather and assess video evidence across stages, selecting relevant spans as visual evidence for answering questions. New candidates and observations prompt reassessment of the current span, but revisions can correct localization errors or discard useful evidence already present in the adopted span. Systems therefore need to connect evidence organization across stages with temporal state control, allowing accumulated information to jointly inform revision assessment while explicit adoption decisions govern how assessment outcomes affect downstream visual evidence. We introduce VideoSlate, a stateful commit-control framework built around shared evidence. A typed evidence blackboard links candidate spans, provenance, construction, and verifier observations, enabling an incumbent-relative certifier to assess revision support from the candidate, current span, and related proposals. Based on certification, the controller authorizes a revision or retains the current span, separating evidence accumulation from state updates and recording each decision and its evidence on the blackboard. Extensive evaluations across video grounding and grounded question-answering benchmarks show that VideoSlate can improve temporal localization through selective revision.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Temporal video grounding, Agentic video reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_JbMsOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VideoSlate, a framework for selectively revising temporal intervals in video grounding and grounded question answering. It maintains an incumbent interval, constructs alternative candidates from frozen VideoMind proposals, and records candidate provenance and verifier observations in an evidence blackboard. A logistic classifier uses seven source-derived features to score whether a challenger should replace the incumbent. A threshold-based controller then either commits the selected candidate or retains the incumbent. Training labels reflect candidate–incumbent differences in ground-truth IoU. The paper reports results on 5 benchmarks and controlled experiments addressing candidate selection and replacement decisions. The central empirical finding is that selective revision can improve localization while reducing harmful replacements relative to unconditional verifier selection. Overall, I believe this paper offers some novelty and promising empirical results.

Critical strengths
  1. The paper addresses a practical and well-motivated problem. A candidate can look relevant to the query while still being worse than the current interval. Comparing a proposed revision against what it would replace is a sensible idea, and the examples make clear why blindly choosing the verifier’s favorite candidate can hurt localization.

  2. The results on ReXTime are promising. Compared with unconditional verifier selection, VideoSlate improves mIoU from 33.55 to 41.18 and [email protected] from 28.88 to 38.98. It also reduces harmful replacements while recovering more missed events, suggesting that the policy does more than simply avoid changes.

  3. The paper includes useful controlled comparisons. The fixed-budget experiment holds the number of replacements constant, while the Charades-STA controls use a shared candidate pool. These comparisons help distinguish better replacement decisions from gains that might otherwise come from having more candidates or making more updates.

  4. The evaluation covers several datasets and both grounding and grounded QA. Localization improvements on Charades-STA and ActivityNet-Captions suggest that the approach has value beyond the main ReXTime setting. The gains are not equally large everywhere, but this broader evaluation is a useful strength.

Critical weaknesses
  1. Calibration and testing use different tie-breaking rules. During calibration, the method uses ground-truth IoU to resolve certifier-score ties, whereas testing uses an annotation-free rule. Since derived candidates can inherit identical features, these ties may matter in practice. I would like to see how often they occur and whether the results hold when calibration uses the exact test-time rule.

  2. The evidence for the broader stateful framework is limited. The main controlled experiment replays a single revision step over frozen candidates and observations. This supports the usefulness of the replacement policy, but does not show why persistent memory or a blackboard is needed. A comparison with a simpler implementation using the same features would help clarify the architectural contribution.

  3. The feature ablations do not fully establish which inputs are necessary. The paper removes features while keeping the remaining coefficients and thresholds fixed. The performance drops could therefore reflect a poorly calibrated model after feature removal, rather than information that a simpler model cannot recover. Retraining and recalibrating each variant would make the conclusions more convincing.

  4. The grounded-QA comparisons do not isolate the effect of the proposed method. VideoSlate uses Qwen3-VL-8B as its answerer, while the published baselines retain their own answering configurations. Better answer accuracy and joint QA scores may therefore come partly from the answerer. A comparison with the answer model, prompts, and visual inputs controlled would make the source of these gains clearer.

Questions for authors
  1. How frequently do candidates tie in certifier score? What thresholds and evaluation results are obtained when out-of-fold calibration uses exactly the annotation-free selection rule in Equation 2?

  2. Do the feature-ablation conclusions hold when each reduced-feature model is retrained and recalibrated? How does the full model compare with a similarly tuned classifier using only incumbent support and candidate support?

Reviewer_B96cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

VideoSlate treats temporal refinement as a selective replacement decision relative to an incumbent interval. A typed evidence blackboard stores candidate spans, provenance, and verifier observations. A logistic certifier uses seven features to estimate whether a challenger should replace the incumbent, and a thresholded controller either commits the replacement or retains the current span. The paper reports improvements on grounded video QA and temporal grounding benchmarks using frozen VideoMind components. Its main controlled evaluation replays one state transition per query. The results suggest value in incumbent-relative selection, but calibration inconsistencies and limited validation of the broader stateful framework remain important concerns.

Critical strengths

A candidate’s absolute verifier score need not measure its improvement over an already adequate incumbent. The paper addresses this distinction directly. In the REXTIME replay, the full policy improves [email protected] from 25.30 for KEEP-only and 28.88 for unconditional verifier selection to 38.98. These controlled comparisons provide substantive evidence that selective replacement can improve the use of an existing candidate pool.

The seven-feature logistic model operates on frozen grounding and verification outputs, avoiding additional large-model fine-tuning. Recording the selected candidate, feature values, thresholds, and decision provides a concrete basis for auditing revisions. This is an appealing practical property of the proposed selection layer.

Critical weaknesses

Section 3.5 states that primary out-of-fold score ties are resolved using training-candidate IoU, whereas evaluation uses the annotation-free rule in Equation 2. Thus, calibration evaluates a selection policy that has access to information unavailable at deployment. This can overestimate calibration quality when ties occur; inherited source features make ties between distinct derived candidates a relevant possibility. The manuscript also does not adequately specify the certifier-training datasets, sample counts, or the dataset-specific policies used outside REXTIME. These omissions prevent a confident assessment of the frozen thresholds and cross-benchmark generalization.

The primary evaluation performs one offline state transition over frozen candidates and observations. It therefore provides limited evidence about accumulated evidence, repeated updates, or long-horizon state management. Keep and Defer return the same interval and differ only in their recorded trace. Moreover, the feature ablations remove terms while retaining the original coefficients and thresholds, which measures intervention sensitivity rather than the performance of properly refitted alternatives. Published QA comparisons also use different answering configurations. The current evidence most clearly supports a selective reranking layer; the additional contribution of the blackboard and stateful control remains insufficiently established.

Questions for authors
  1. What datasets and video-disjoint splits are used for fitting and calibrating each benchmark’s policy? Please report sample counts, candidate-generation settings, and all dataset-specific parameters.

  2. How frequently do certifier ties occur, and what thresholds and test results are obtained when the exact deployment tie-breaking rule is used throughout out-of-fold calibration?

  3. Do the gains persist against refitted feature-reduced rankers and a simple accept/retain selector using the same candidates, data, and answerer? What evidence specifically demonstrates benefits from multiple state updates or the blackboard?

Reviewer_HwZ6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies temporal grounding for video question answering. It builds on VideoMind, where a grounder proposes candidate segments and a verifier selects one. The authors argue that this selection can replace a good initial segment with a worse one. The proposed method, VideoSlate, keeps the grounder’s top segment as the current answer and decides whether an alternative should replace it. A logistic certifier, trained on features from the verifier, scores each alternative relative to the current segment. A controller switches only when the score exceeds a threshold. Candidates and decisions are recorded in an “evidence blackboard.” The method is evaluated on five grounding and grounded QA benchmarks. It shows the largest improvement on ReXTime, supported by ablations and controlled comparisons.

Critical strengths
  • The problem is concrete and well motivated. A new interval can appear relevant while still being worse than the current selection, and the examples in figures 1 and 3 illustrate this failure clearly. This gives a good reason to assess replacement quality explicitly rather than relying on the highest-scoring candidate.
  • The improvement on ReXTime is convincing. The proposed mode is a logistic regression over the 7 features added on top of the frozen VideoMind models. From table 1 and table 5, localization improves a lot with the proposed model.
  • The controlled comparisons help explain why the method works. In table 6, With the number of updates held fixed, the learned score selects which queries to update more effectively than source-support or source-margin ranking . The bootstrap intervals in the appendix (table S10) indicate that these differences are not due to sampling noise. Reporting rescues and harmful replacements separately also makes the tradeoffs easier to assess.
  • The code is provided which supports reproducibility.
Critical weaknesses
  • While the improvement on ReXTime is clear, it is less clear that it comes from the selective revision idea in the paper. The relevant numbers appear only in the appendix. Table S8 (candidate-family interventions) shows that the controller accepts the challenger in 881 of 921 queries, defers in 40, and never keeps. In Table S9 (budget grid), committing the challenger for every query gives nearly the same result as the full policy. This suggests the gain comes mainly from selecting a better challenger, not from deciding when to commit.

  • VideoSlate uses Qwen3-VL-8B as the answerer, while the published baselines keep their own answerer configurations. Part of the QA gains may therefore come from the answerer rather than the proposed method.

  • The claims extend beyond what the experiments evaluate. The introduction motivates agentic video reasoning over multiple steps, but the evaluation replays a single state transition per query. This provides limited support for the broader claim about accumulating and reusing evidence across reasoning stages.

  • The discussion of related work is limited. The paper does not clearly explain how the approach differs from REVISOR or others mentioned.

Questions for authors
  1. As mentioned in the weaknesses, the controller accepts the challenger in 881 of 921 ReXTime queries (table S8), and committing every challenger gives nearly the same result (table S9). What evidence shows that the commit rule adds value beyond the certifier’s choice of challenger?
  2. Could the authors report QA results using the same answerer for both VideoMind and VideoSlate? This seems particularly important for CG-Bench.
  3. Line 274 states that θkθ_k “follows the grid tie convention”. Could the authors clarify what this means? In addition, both selected thresholds are the lowest values in the search grid. Were lower thresholds considered?
  4. Table 6 reports a single random run, whereas table S9 reports a three seed average that is slightly higher. Is there a reason the main table uses the single run?
  5. Section 3.2 states that merging “takes the maximum of each stored numeric feature”. Does this include the rank rir_i? Since equation 2 prefers a lower rank, taking the maximum would assign the merged candidate its worst rank.
  6. Figure 2 is quite dense. A simplified version would make the method easier to follow.
Reviewer_VGe4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VideoSlate studies whether a newly proposed video segment should replace the segment currently used for reasoning. It builds on a frozen VideoMind grounder and verifier. The method records candidate segments, their sources, and verifier scores in a shared evidence store. A logistic classifier uses seven features to score whether each candidate would improve temporal overlap over the current segment. The controller selects the highest-scoring candidate and accepts it only when its score reaches a threshold.

The paper reports results on five benchmarks. Its strongest controlled result is on ReXTime, where mIoU increases from 33.55 with unconditional verifier selection to 41.18 with VideoSlate.

Critical strengths

1. The paper addresses a practical failure in video evidence selection.

Replacing a reasonable segment with another plausible segment can reduce localization quality. Comparing a candidate against the current segment and allowing the system to retain its earlier choice is a useful design. The reported rescue and hurt counts help show both the benefits and risks of revision.

2. The controlled comparisons provide evidence beyond the main benchmark tables.

On ReXTime, VideoSlate improves mIoU from 33.55 to 41.18 over unconditional verifier selection. Table 6 also compares update allocation at the same commit budget, while Table 7 uses shared frozen candidates on Charades-STA. These controls help separate better selection from simply generating more candidates.

Critical weaknesses

1. Calibration does not use the same candidate-selection rule as evaluation.

Section 3.5 states that out-of-fold score ties are resolved using training-candidate IoU, while evaluation uses verifier support, rank, and candidate identifiers. Ground-truth IoU is therefore involved in choosing the candidate whose output is used to calibrate the threshold. This can make calibration results look better than the rule available at test time. The issue may matter because different derived intervals can inherit the same features and receive equal classifier scores. The paper also does not identify the datasets and splits used to fit and calibrate the classifier, and Section 4.1 mentions dataset-specific policies without describing them. These details are necessary to assess whether the reported results come from a clearly separated training and evaluation process.

2. The experiments support selective candidate replacement more clearly than the full stateful framework.

The primary evaluation replays one transition per query using frozen candidates and observations. It does not test how the shared evidence store helps across repeated revisions or changing observations. In this setting, the tested decision can be implemented as candidate scoring followed by a threshold. Moreover, Table 5 removes feature terms while keeping the remaining coefficients and threshold fixed. The resulting drops show that the fitted model depends on those terms, but do not show whether a smaller model could recover the gains after retraining and calibration. Stronger controls are needed to establish which parts of the proposed framework are necessary.

Questions for authors

1. How often do candidates receive equal classifier scores?

Could you repeat calibration using exactly the annotation-free selection rule in Eq. (2), then report the selected threshold, localization results, and hurt counts?

2. Could you retrain and recalibrate the reduced-feature models in Table 5?

Please also clarify which measured benefit requires the shared state across revision steps. An evaluation with repeated revisions would strengthen this part of the contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.