acceptodds
Withdrawn from ICLR 2027

GroundBoth: Support-Counter Grounding for Video Question Answering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Grounded video question answering (VideoQA) asks a model to support its answer with localized temporal evidence. However, a correct answer with accurately grounded evidence does not by itself show that the model understood the video. When a wrong option describes an event that also occurs in the video and resembles the correct one, a model can choose correctly without ever telling the two apart. We extend grounded video question answering to support–counter grounded VideoQA: a model grounds the selected answer and, for each rejected option, grounds the competing event it refers to and explains why that event does not answer the question, making each rejection inspectable rather than taken on trust. To study this task, we build SC-Ground by refining NExT-GQA, ReXTime, and CG-Bench so that every distractor is anchored to a real event elsewhere in the video, with support and counter annotations. Used as a diagnostic, it reveals that even strong proprietary models ground the selected answer well yet largely miss the events behind the wrong options; grounding the answer alone is far from enough. We therefore curate a support–counter training set and introduce GroundBoth, a 2B model that emits the answer, its support, and per-option counter evidence in a single pass, trained by role-wise on-policy distillation (Role-wise OPD) and support–counter GRPO (SC-GRPO) that rewards both evidence roles. Despite its size, GroundBoth rivals 7B models on the original benchmarks, raising grounded answering by 18.0, 15.5, and 2.7 points over its 2B backbone, and leads all evaluated open-source single-call models on SC-Ground, lifting joint grounding nearly eightfold on NExT-GQA-SC (1.48 to 11.67).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video understanding, Grounded Video Understanding, On-Policy Distillation, Vision-language Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zon6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces support–counter grounded VideoQA, where a model must answer a multiple-choice question, localize supporting evidence, and localize the events associated with rejected alternatives while explaining why they do not answer the question. The authors construct SC-GROUND by refining three existing benchmarks and propose GroundBoth, a 2B model trained with role-wise on-policy distillation followed by support–counter GRPO. The model substantially improves grounding over its backbone on both the refined and original benchmarks.

Critical strengths
  1. The task formulation is meaningful. Correctly answering a question and localizing its support do not establish that a model distinguished the relevant event from plausible competing events. Option-conditioned localization provides a concrete way to make these alternatives inspectable.

  2. The empirical results are promising for a small model. GroundBoth achieves competitive results on the original NExT-GQA and ReXTime benchmarks, suggesting that the benefits of the overall training procedure extend beyond the revised distractors in SC-GROUND.

Critical weaknesses
  1. The benchmark lacks independent validation against the underlying videos. As described in Appendix A, distractor verification and answer-uniqueness checks operate on machine-generated event records. Using a different verifier reduces direct self-verification, but cannot identify visual facts that the shared record omits or misrepresents. Only approximately 30–40% of the original questions are retained, and the paper acknowledges incomplete record coverage as one source of filtering failures. Without a human audit against the original videos, the reliability of event existence, temporal boundaries, answer uniqueness, and the resulting selection bias remains uncertain. This is my primary concern because the benchmark is a central contribution.

  2. The contribution of counter learning to the original-benchmark gains is not isolated. Table 3 supports counter rewards on SC-GROUND, but does not report the corresponding support-only versus support–counter comparison on the original benchmarks. Consequently, Table 2 establishes the effectiveness of the complete training recipe rather than the specific contribution of counter learning. Similarly, comparing role-wise OPD against pure SFT and pure OPD does not isolate role assignment from the benefit of combining annotation supervision and distillation; a conventional mixed SFT–OPD baseline would clarify this distinction.

Questions for authors
  1. Can the authors provide an independent human audit against the original videos, reporting event validity, answer uniqueness, and temporal annotation quality?

  2. On a modest evaluation subset, how often are the rejection rationales correct? Can a question-and-options-only baseline help distinguish language-based rejection from visually grounded discrimination?

  3. How do support-only and support–counter reward configurations compare on the original benchmarks under the same initialization? Does role-wise OPD outperform a conventional mixed SFT–OPD objective?

  4. Are the key joint-score improvements stable under paired confidence intervals, preferably computed by resampling videos to account for multiple questions sharing a video?

Reviewer_VKWkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes adding a second grounding target to grounded VideoQA. Besides the interval that supports the chosen answer, the model must give, for each rejected option, the interval of the event that option describes and a short reason why it does not answer the question. Three existing benchmarks are rewritten with an LLM pipeline so that every distractor points to a real event in the video, and a VLM is trained with a mixed gold/distillation cold start followed by GRPO with rewards on both evidence roles. The model clearly improves over its base model on the new sets and on the original benchmarks.

Critical strengths

The task is a natural extension. A correct answer with a correct support interval still leaves open whether the model noticed the competing event, and asking for that event is a direct way to test it. The reward design is simple, and the ablations in Table 3 are organized around clear questions (cold start, reward terms, aggregation). Table 2 shows that the extra counter objective does not hurt standard grounded QA for this backbone, which is a useful sanity check.

Critical weaknesses

The major weaknesses are as follows.

  • An LLM (GPT) writes the clip summaries, proposes the distractors and their intervals from those summaries, and another LLM (Gemini) checks the candidates against the same summaries. No human looks at any video, and the paper itself reports that the verifier often finds no option supported by the record. Counter intervals are also padded with neighboring events to reach a minimum length, and GPT-5.6 is the strongest system in Table 1. As it stands, mIoU− measures agreement with an LLM transcript rather than with the video. A human check on a sample is needed before the benchmark can carry the claims.

  • Every wrong option in SC-Ground has an interval, and a missing counter interval simply scores zero. A model that always emits some interval for every option is never penalized, and the rejection analysis text is never scored. The restraint reward is trained but never tested. So the benchmark cannot separate a model that knows why an option is wrong from one that retrieves the option text in the video.

  • Divid-2B, run once per option, scores 19.79 on ReXTime-SC against 18.46 for GroundBoth, and the gaps on the other two sets are small. Since each distractor was written from one chosen interval, the counter task as built is close to per-option temporal grounding of that sentence. GroundBoth is not run in the same multi-call protocol, and no per-option grounder trained on the same data is reported, so the benefit of learning both roles jointly is not shown.

  • The training and test annotations come from the same pipeline with the same interval rules, while VideoChat-R1 and TVG-R1 only receive an extra sentence appended to prompts they were not trained for. The same issue appears in Table 2, where GroundBoth is trained on NExT-QA and ActivityNet videos and the 7B baselines are not. The comparisons mix training data with method.

Questions for authors

Did you check a sample of counter intervals against the raw video by hand? What fraction were correct? What does Qwen3-VL-8B score on SC-Ground under the single-call prompt? What is the Acc@IoU± of the Role-wise OPD policy before SC-GRPO? How does GroundBoth do under the M+1 call protocol used for Divid and VideoMind? With 64 frames over a 25-minute video, one frame covers about 23 seconds while a median clue lasts 10 seconds. How much of CG-Bench-SC is answerable at all in this setting?

Reviewer_q9nVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces support–counter grounded VideoQA, requiring models to localize both the evidence supporting the correct answer and the events associated with rejected options. The authors construct SC-Ground by refining NExT-GQA, ReXTime, and CG-Bench, and train a Qwen3-VL-2B model using role-wise on-policy distillation and support–counter GRPO. The model improves grounding scores over its backbone on both the refined and original benchmarks.

Critical strengths

The distinction between evidence supporting the correct answer and evidence associated with rejected alternatives addresses a concrete limitation of conventional grounded VideoQA. Evaluating on both refined and original benchmarks also helps assess whether the new training objective preserves standard answer-support grounding.

Critical weaknesses
  1. The reliability and difficulty of SC-Ground are insufficiently established. Section 3.2 and Appendix A describe a benchmark construction pipeline in which GPT-5.6 generates distractors and temporal annotations, while Gemini-3.1-Pro verifies them against model-generated video records rather than the original videos. No human validation statistics are provided for temporal boundaries, distractor plausibility, or answer uniqueness. More importantly, the example in Figure 4 asks why chickpeas are cooked, while the rejected options describe hanging laundry, feeding a dog, and using a smartphone. These alternatives are semantically unrelated to the question and can be rejected without viewing the video, undermining the intended evaluation of plausible competing events. Furthermore, only 40%, 34%, and 30% of the original questions are retained across the three datasets, raising concerns about selection bias.
  2. The evaluation does not establish the central claim of grounded rejection. The Acc@IoU± metric defined in Equation (8) awards partial credit when only some rejected options are localized, without evaluating whether the generated rejection explanations are correct. Thus, successful temporal localization does not necessarily demonstrate that the model understands why an alternative is incorrect. Table 1 reports joint grounding scores of only 11.67, 18.46, and 0.93 across the three benchmarks, with particularly limited performance on CG-Bench-SC. Moreover, the uniform sampling of only 64 frames is poorly suited to videos lasting approximately 25 minutes with median evidence intervals of just 10 seconds, potentially leaving relevant events unobserved.
  3. The experimental comparisons do not sufficiently isolate the proposed methodological contributions. In Table 1, TVG-R1 and VideoChat-R1 are adapted to counter grounding through additional prompting, whereas GroundBoth receives dedicated counter annotations, 8B-teacher distillation, and reinforcement learning. The comparison therefore conflates task-specific supervision with methodological differences. GroundBoth only marginally exceeds VideoChat-R1 on NExT-GQA-SC (11.67 vs. 11.28), despite lower answer accuracy (55.3 vs. 61.0). Furthermore, the reward ablations in Table 3(b) retain counter-supervised role-wise initialization, making it difficult to isolate the contribution of counter annotations from that of the proposed counter reward. The reported improvements therefore do not adequately establish which components are responsible for the gains.
Questions for authors

1.Have the generated support/counter intervals and distractors been independently verified by human annotators viewing the original videos? Can the authors report annotation agreement, distractor plausibility, and text-only answering accuracy on SC-Ground? 2.What proportion of questions have the correct answer, correct support interval, and all counter intervals simultaneously localized? How is the correctness of the generated rejection explanations verified beyond temporal IoU? 3.What are the results of the cold-start variants in Table 3 before SC-GRPO? How much improvement remains when counter annotations are excluded from the cold-start stage?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.