acceptodds
Under review as a conference paper at ICLR 2027

Who Does What, and When? Synchronized Conjunctive Reasoning for Image-Disambiguated Video Temporal Grounding

Abstract

Image-Disambiguated Video Temporal Grounding (ID-VTG) aims to localize a temporal segment using a reference image that identifies the target entity and a text query that describes the target event. The task is inherently conjunctive: entity and event conditions must be supported at the same temporal location, while satisfying either condition alone is insufficient. Existing ID-VTG methods effectively integrate image and text cues, yet their entangled proposal relevance leaves temporal entity–event conjunction implicit, limiting their ability to reliably determine when both conditions hold simultaneously. To address this problem, we first introduce Synchronized Conjunctive Grounding (SCG), which factorizes frame-wise entity and event evidence and models their temporal conjunction through a non-compensatory product. The resulting conjunctive evidence is propagated across the temporal pyramid and injected as a signed residual to calibrate proposal relevance, favoring moments jointly supported by both conditions. With entity–event conjunction explicitly modeled, we further characterize its underlying validity structure through Factor-Selective Counterfactual Learning (FCL), making the validity of entity and event factors identifiable. Rather than collapsing intervened queries into generic negatives, FCL exposes the underlying factorial validity structure of entity and event conditions and learns a structured four-state energy with unary validity and pairwise interaction potentials. Together, SCG and FCL form a coherent factorize–conjoin–attribute framework for effective entity–event conjunctive grounding. Extensive experiments on existing ID-VTG benchmarks validate the effectiveness of the proposed framework, with consistent improvements across temporal granularities, ambiguity settings, and cross-domain generalization scenarios.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video temporal grounding, vision-language grounding, multimodal video understanding, compositional reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ipZROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies image-disambiguated video temporal grounding. The authors build on VGD-Agg and introduce two components. SCG separately estimates entity and event evidence and combines them through pointwise multiplication. The combined evidence receives supervision during training and is used to calibrate proposal scores at inference. FCL augments the training queries by replacing the reference image, the event description, or both, while keeping the original video and annotated interval fixed. The experimental results provide some support for the effectiveness of the proposed method.

Critical strengths

The paper tackles a useful problem: finding the right event when a text query alone cannot identify the intended person. The design is straightforward, with SCG combining entity and event scores and FCL adding supervision that distinguishes which condition is satisfied. The results show consistent gains over the reproduced VGD-Agg baseline on Gym, InternVid, and Web. The ablations also provide useful evidence for the role of each component, including comparisons with different supervision objectives on the same counterfactual samples. The additional inference cost is small, and FCL is used only during training.

Critical weaknesses
  1. SCG uses conventional frame-level auxiliary supervision, scores entity and event evidence separately, and combines them through multiplication to calibrate localization scores across temporal scales. FCL adds factor-level supervision by replacing the reference image, the text query, or both, following ideas related to selective counterfactual training. The four-state supervision within a fixed interval offers some novelty, and the performance gains are substantial. Still, the overall contribution appears incremental, mainly adapting and combining existing mechanisms for ID-VTG.

  2. High scores from both branches at the same time do not necessarily mean that the target person performs the queried action. For example, the target person may stand in the scene while someone else performs the action, allowing both branches to assign high scores. FCL may help address this issue, but the paper does not specifically evaluate such cases or report the accuracy of the individual branches or four validity states. The results support improved localization, but provide limited direct evidence for person–action binding and factor identifiability.

Questions for authors
  1. Do the test sets include cases where the target person is present but someone else performs the queried action? If so, how do SCG and FCL perform on these cases? Accuracy results for the entity and event branches, or the four validity states, would also help substantiate the claim of factor identifiability.
  2. Does the gain from FCL remain when SCG score calibration is disabled at inference? Please compare the SCG and SCG+FCL models with lambda=0 for both. This would help clarify whether FCL also improves the base grounding model or whether its gains mainly come through the SCG calibration signal.
5 comments
AuthorsOpenReview ↗

Response to Reviewer ipZR (Part 1)

We thank the reviewer for recognizing the useful task, straightforward design, substantial and consistent gains, informative ablations, low inference overhead, and the novelty of fixed-interval four-state supervision. We address your concerns below.

Q1. A task-driven factorization–conjunction–attribution framework for ID-VTG

A1. Our contribution lies in a task-driven formulation and a unified framework tailored to the structural ambiguity of ID-VTG.

1) The design follows the conjunctive structure of ID-VTG. A valid interval must simultaneously match the reference entity and the described event, whereas conventional joint scores can remain high for entity-only or event-only partial matches. SCG therefore factorizes the two conditions and evaluates their same-time conjunction to adaptively enhance or suppress proposal scores. The binary validity of the entity and event conditions naturally gives rise to four states under selective image and text interventions, motivating FCL to supervise which conditions remain valid.

2) The components form a unified factorization–conjunction–attribution framework. SCG converts factorized evidence into an explicit, scale-compatible conjunctive correction: it aligns entity and event responses on a shared fine temporal grid, uses a non-compensatory Product so that strong evidence for one condition cannot mask weak evidence for the other, and propagates the resulting signal across scales as a signed residual. FCL then makes the constituent factors attributable through controlled image/text interventions that preserve the video and temporal support. Rather than collapsing all counterfactuals into generic negatives, it retains the complete four-state validity structure and supervises which conditions remain valid.

3) Controlled and broad evaluations validate the design. The complete framework consistently improves performance across three benchmarks, different query granularities, and multiple ambiguity types. Under identical counterfactual samples, Table 5 shows that FCL outperforms conventional binary counterfactual supervision and independent factor-wise supervision. Table 4 further shows that SCG’s non-compensatory Product consistently outperforms Mean and Minimum. Our analysis explains how Product combines non-compensatory conjunction with responsiveness to both factors, properties that are particularly important for ID-VTG.


Q2. Do the test sets include cases where the target person is present but someone else performs the queried action? If so, how do SCG and FCL perform on these cases?

A2. We thank the reviewer for highlighting this challenging case. We analyze its frequency in the current benchmark, mitigation through our framework design, and the performance of SCG and SCG+FCL on the identified cases.

1) Only a few such cases are found in the current benchmark. We manually inspect 200 randomly sampled IDVTG-Gym training videos and 200 test videos, identifying 5 potential cases in training and 3 in test (2.5% and 1.5%, respectively). This indicates that such cases occur at a low frequency in IDVTG-Gym. All three manually identified test cases are included in the revised paper: one in Fig. 4 and the remaining two in Appendix Fig. 9(a–b).

2) The overall framework design helps mitigate such distractors. SCG combines entity and event evidence through same-time conjunction before producing a signed correction. Its synchronization loss assigns zero targets outside the annotated GT interval, penalizing high conjunctive responses at distractor locations during training. The base path jointly models the reference image, video context, and event query, while FCL provides factor-validity supervision under selective interventions. Together, the overall framework design helps suppress mismatched intervals.

3) Both SCG and SCG+FCL achieve good localization performance on all 3/3 identified test cases. These results show that both SCG and the complete model can successfully localize the correct intervals even when another entity simultaneously performs the queried action.

AuthorsOpenReview ↗

Response to Reviewer ipZR (Part 2)

Q3. Direct validation of factor validity and mismatch-source attribution

A3. We thank the reviewer for requesting direct validation of the evidence branches’ semantics and their ability to distinguish mismatch sources. Our controlled four-state evaluation directly supports factor identifiability and mismatch-source attribution.

We evaluate the frozen SCG+FCL model on 1,000 inputs derived from 250 randomly sampled IDVTG-Gym test queries. For each test query, we automatically construct three counterfactual states using our existing procedure and manually verify that each state corresponds to its intended factor-validity label. Together with the factual input, this yields 250 examples per state.

Four-state predictions use softmax–argmax over the aggregated state logits. Entity and event predictions are obtained by thresholding the corresponding aggregated branch probabilities at 0.5.

MethodAcc(++)Acc(+-)Acc(-+)Acc(--)Four-state Acc.Entity Acc.Event Acc.
SCG+FCL81.20%85.60%88.00%92.40%86.80%91.40%94.80%

Here, ++, +-, -+, and -- denote factual, text-replaced, image-replaced, and dual-replaced inputs, respectively. The 85.60% and 88.00% accuracies on text- and image-replaced inputs show that the model can distinguish event mismatches from entity mismatches. The 91.40% entity accuracy and 94.80% event accuracy further support the intended semantic roles of the two evidence branches, providing direct factor-level evidence beyond localization gains.


Q4. Does the gain from FCL remain when SCG score calibration is disabled at inference?

A4. Yes. FCL continues to provide clear performance gains when SCG calibration is disabled at inference for both models. We compare the separately trained SCG and SCG+FCL models and set λ=0\lambda=0 only at inference for both. This removes SCG’s signed residual correction from both models during inference. FCL provides additional training supervision and introduces no inference-time branch.

IDVTG-Gym Overall, inference λ=0\lambda=0R@1(.5)R@1(.7)mIoU
SCG67.0861.5958.76
SCG+FCL71.7666.2363.04
FCL gain+4.68+4.64+4.28

Under this controlled setting, FCL improves R@1(.5), R@1(.7), and mIoU by 4.68, 4.64, and 4.28 points, respectively. These gains show that FCL improves the learned grounding model during training and that its benefit does not depend solely on SCG’s inference-time calibration.

AuthorsOpenReview ↗

A Brief Follow-up

Dear Reviewer ipZR,

Thank you again for your time and constructive feedback.

As the author–reviewer discussion period draws to a close, we would appreciate your thoughts on our responses and additional results. If any concerns remain, we would be happy to provide further clarification.

Reviewer_cTttOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses image-disambiguated video temporal grounding, where a reference image identifies the target entity and a text query specifies the event to localize. It proposes Synchronized Conjunctive Grounding, which combines separate entity and event evidence at corresponding temporal locations to refine proposal scores. The framework also introduces Factor-Selective Counterfactual Learning, using selectively replaced image and text queries to supervise distinct validity configurations. Experiments report consistent improvements across benchmarks, temporal granularities, ambiguity settings, and an out-of-domain evaluation, with ablations supporting the contributions of both components.

Critical strengths
  1. The proposed framework consistently improves upon a reproduced baseline using the same pretrained features. Gains extend across datasets, temporal granularities, and ambiguity settings, while evaluation on an out-of-domain dataset provides evidence of transfer beyond the training distribution. These results support the practical value of explicitly modeling entity and event evidence.

  2. SCG and FCL offer complementary mechanisms for synchronizing evidence and supervising factor-specific validity. The ablations examine the coupling operator, synchronization supervision, proposal calibration, and counterfactual learning objectives, including comparisons using the same counterfactual samples. The framework also maintains modest inference overhead, with FCL operating exclusively during training.

Critical weaknesses
  1. The rationale for calibrating VGD-Agg outputs, rather than building a standalone grounding model around the proposed principles, is insufficiently established. Table 4 shows that residual calibration helps within the chosen architecture, but does not explain why this formulation is preferable to directly predicting temporal intervals from factorized entity–event evidence with counterfactual supervision. A controlled comparison with such a standalone alternative would clarify the benefits of retaining VGD-Agg. Moreover, the SnAG results in Table 6 provide encouraging transfer evidence, but one additional backbone is insufficient to establish consistent gains across diverse grounding architectures. Evaluation on further, architecturally distinct backbones is needed to substantiate the broader applicability of the method.

  2. Direct visual evidence for SCG’s claimed corrective behavior is limited. Figures 3 and 6 show improved final temporal intervals, but these comparisons do not reveal how SCG changes proposal scores and rankings within the same trained model. The paper lacks a visualization jointly showing pre- and post-calibration scores, ranking changes, the signed residual, and the underlying entity and event evidence. Such an analysis would help assess whether SCG promotes correct proposals and suppresses partial matches as intended. Although Table 4 supports the aggregate benefit of calibration, it does not directly illustrate how the factor-specific evidence produces appropriate corrections for individual proposals.

Despite these limitations, I consider the core idea a potentially valuable contribution and therefore give the paper a positive overall assessment. Stronger empirical evidence addressing the concerns above would make the paper more convincing.

Questions for authors
  1. How does SCG handle cases where the reference entity is visible while a different entity simultaneously performs the queried action? In other words, since the event scorer is not conditioned on the reference image, how does SCG avoid assigning a high conjunctive score when the reference entity is visible but another entity performs the queried action, while the reference entity itself does not? Are such cases represented in the evaluation benchmarks, and if so, can the authors provide a targeted analysis?

  2. Could the authors quantify the additional training cost of SCG+FCL compared with VGD-Agg, particularly in terms of training time and peak GPU memory usage under comparable experimental settings?

  3. How does the proposed framework compare in localization accuracy and inference cost with a representative recent MLLM-based grounding method, such as TimeLens or Enrich and Detect, when adapted to the same image-text query setting?

4 comments
AuthorsOpenReview ↗

Response to Reviewer cTtt (Part 1)

We thank you for your careful review and constructive feedback. We appreciate your overall positive assessment of our work, especially your recognition of the substantial gains, complementary roles of SCG and FCL, informative ablations, modest inference overhead, and potential value of the core idea. We address your concerns below.

Q1. Rationale for calibrating the base grounding scores rather than replacing them with factorized evidence

A1. The base path provides multimodal contextual grounding, while synchronized factor evidence adaptively enhances or suppresses proposal scores, motivating signed residual calibration. We support this design through its functional motivation and a direct comparison with a factorized-evidence alternative.

1) Design motivation. The base path jointly integrates the reference image, event text, and multiscale video context to estimate proposal relevance and temporal boundaries. However, one query condition may dominate the joint score, allowing entity-only or event-only partial matches to remain highly ranked. SCG explicitly evaluates synchronized entity–event support and converts it into a signed residual: proposals with stronger conjunctive support are enhanced, whereas partial matches are suppressed. This retains the base path’s multimodal contextual grounding and multiscale boundary-estimation capacity while selectively correcting its proposal scores.

2) Comparison with a factorized-evidence alternative. To assess whether synchronized factor evidence can replace the base classification path, we train an SCG-logits-only alternative that uses factorized synchronized evidence as its classification score while retaining the same boundary-regression architecture and other training losses.

IDVTG-InternVid OverallmIoUR@1(.5)R@1(.7)
VGD-Agg†49.4552.1541.89
SCG-logits-only47.1748.9537.22
Full SCG+FCL53.7957.5246.34

† denotes our reproduced baseline. The full model exceeds the SCG-logits-only alternative by 6.62 mIoU points. These results support retaining multimodal contextual grounding and using synchronized evidence as an adaptive signed correction rather than as a standalone replacement.


Q2. Generalization across structurally distinct grounding architectures

A2. We thank you for suggesting broader evaluation across different grounding architectures. Our method yields substantial gains on both dense-proposal and DETR-style architectures, demonstrating applicability beyond VGD-Agg. Beyond the SnAG evaluation already included in our submission, we further validate our framework on CG-DETR, a structurally distinct DETR-style architecture.

MethodIDVTG-GymIDVTG-InternVidIDVTG-Web
SnAG reported46.73/53.64/48.8241.70/44.26/35.4616.48/9.06
SnAG†47.21/54.26/49.8142.13/44.45/35.7016.08/9.10
SnAG† + Ours60.67/69.21/64.2448.87/52.23/42.3423.50/13.36
CG-DETR reported26.99/27.96/23.3643.80/44.81/31.6213.86/5.91
CG-DETR†27.40/28.15/23.4643.90/45.16/30.0513.22/5.77
CG-DETR† + Ours43.60/45.74/38.0849.12/50.58/36.2116.98/8.13

IDVTG-Gym and IDVTG-InternVid report mIoU/R@1(.5)/R@1(.7); IDVTG-Web reports R@1(.5)/R@1(.7). † denotes our reproduced baseline.

The substantial gains over the reproduced baselines across all three benchmarks demonstrate effective transfer across dense-proposal SnAG and DETR-style CG-DETR.


Q3. Direct visualization of factor evidence and inference-time signed score correction

A3. We thank you for suggesting this direct analysis. Fig. 5 in the revised paper visualizes SCG’s inference-time corrective behavior by jointly showing the underlying entity and event evidence, pre- and post-calibration proposal scores, signed residual, and ranking changes.

In this fine-grained sub-action example, the distractor contains the same entity but a mismatched sub-action. SCG’s inference-time calibration increases the target proposal logit from 1.88 to 1.98 while suppressing the partial-match proposal logit from 2.07 to 1.33. This moves the correct proposal to the top rank and enables accurate localization. The example directly demonstrates how synchronized factor evidence enhances the target and suppresses a partial match through signed score correction.

AuthorsOpenReview ↗

Response to Reviewer cTtt (Part 2)

Q4. Analysis of simultaneous distractors when the reference entity is present but does not perform the queried action

A4. We thank you for highlighting this challenging case. We analyze it from three perspectives: its frequency in the current benchmark, mitigation through our framework design, and targeted evaluation on the identified cases.

1) Only a few such cases are found in the current benchmark. We manually inspect 200 randomly sampled IDVTG-Gym training videos and 200 test videos, identifying 5 potential cases in training and 3 in test (2.5% and 1.5%, respectively). This indicates that such cases occur at a low frequency in the current benchmark.

2) The overall framework design helps mitigate such distractors. SCG combines entity and event evidence through same-time conjunction before producing a signed correction. Its synchronization loss assigns zero targets outside the annotated GT interval, penalizing high conjunctive responses at distractor locations during training. The base path jointly models the reference image, video context, and event query, while FCL provides factor-validity supervision under selective interventions. Together, the overall framework design helps suppress mismatched intervals.

3) Our method correctly localizes all three such test cases. Fig. 4 in the revised paper presents one representative case with an IoU of 0.987 , while Appendix Fig. 9 presents the remaining two cases. These results show that our method maintains good localization performance even when another entity simultaneously performs the queried action.


Q5. Additional training time and GPU-memory cost

A5. SCG+FCL achieves substantial performance gains with moderate training overhead. Both methods use the same training configuration and identical pre-extracted CLIP ViT-L/14 video and reference-image features. † denotes our reproduced baseline.

MethodmIoU (%)Training time (h)Peak memory (GiB)
VGD-Agg†55.075.4131.58
SCG+FCL64.337.1333.28

SCG+FCL improves IDVTG-Gym mIoU by 9.26 points, while peak reserved GPU memory increases by only 5.4%. Training takes 1.32× as long (5.41 → 7.13 h). During FCL training, the factorized entity and event evidence for the original and replaced conditions is reused and recombined to form the four states, avoiding four fully independent grounding passes. FCL introduces no additional inference-time branch.


Q6. Accuracy and inference-cost comparison with MLLM-based TimeLens-8B

A6. Our method achieves higher localization accuracy and greater inference efficiency than TimeLens-8B under matched multimodal inputs. We discuss both TimeLens and ENRICH in the revised Related Work. For quantitative evaluation, we compare our method with TimeLens-8B on the Web dataset under a shared transfer setting, where neither method uses Web-specific training or fine-tuning.

To adapt TimeLens to the same image–text query setting, we provide the complete video, a separate reference image, and the event text. The prompt requests one temporal interval in which the reference entity performs the described event. Both methods sample videos at 2 fps. Inference time is measured per query on a single A800 and includes video-frame processing and feature extraction.

MethodmIoUR@1(.5)R@1(.7)Time (s/query)
Ours (SCG+FCL)29.0429.6916.585.62
TimeLens-8B24.6221.8810.647.63

On the Web dataset, our method improves mIoU by 4.42 points and R@1(.5/.7) by 7.81/5.94 points over TimeLens-8B, while being 26.3% faster under the same end-to-end timing protocol.

I thank the authors for their careful rebuttal, which has addressed most of my concerns. I therefore maintain my positive rating.

AuthorsOpenReview ↗

Thank you for your positive feedback.

Thank you for your positive feedback. We are glad to hear that our rebuttal has addressed most of your concerns. We sincerely appreciate your time, constructive suggestions, and continued support.

Reviewer_kZF4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies image-disambiguated video temporal grounding, which localizes the interval in which an entity specified by a reference image performs an event described by a text query. Building on VGD-Agg, the authors introduce two components. Synchronized Conjunctive Grounding (SCG) computes separate temporal evidence for entity and event matching, combines them through a pointwise product, and uses the resulting signal to calibrate proposal scores. Factor-Selective Counterfactual Learning (FCL) replaces reference images and event descriptions to construct four validity states, providing structured supervision for the two factors and their interaction. Experiments show consistent improvements on IDVTG-Gym, IDVTG-InternVid, and the out-of-domain Web evaluation, with ablations examining coupling operators, supervision objectives, and counterfactual construction strategies.

Critical strengths
  1. The method explicitly addresses partial matches by combining entity and event evidence at fine-grained temporal locations, helping reduce the aggregation of evidence from different moments. SCG is straightforward and integrates directly into the existing proposal-scoring pipeline.

  2. The experiments provide targeted comparisons of the main design choices, including product versus mean and minimum coupling, as well as binary, independent factor-wise, and structured four-state supervision using the same counterfactual samples. These controls help distinguish the benefits of the proposed structure from those of additional training examples.

  3. The method improves performance across temporal granularities, ambiguity types, and an out-of-domain evaluation, with additional results on the SnAG backbone. FCL is used only during training, and the modest additional inference overhead reported in the appendix supports the method’s practical utility.

Critical weaknesses
  1. Strong entity and event evidence at the same temporal location does not necessarily establish that the target entity performs the event. Both branches could respond strongly when the target person is present but another person performs the queried action, and the paper lacks a dedicated analysis of such simultaneous distractors to establish the extent to which SCG resolves entity–event binding.

  2. FCL’s four-state labels rely on replacement queries being invalid within the original annotated interval, which similarity filtering, deduplication, and temporal non-overlap do not guarantee. The paper does not quantify counterfactual label error rates, leaving the effects of incorrect supervision on factor learning and grounding unclear.

  3. The evidence primarily demonstrates improved localization, rather than directly supporting the stronger claim that entity and event validity become identifiable and attributable. The appendix establishes structural properties of the scoring function, but direct validation of the two evidence branches’ semantics and their ability to distinguish sources of mismatch is missing.

Questions for authors
  1. How does SCG determine which entity performs the queried action when the target entity and an acting distractor appear simultaneously? Could the authors provide a dedicated evaluation or controlled experiment distinguishing temporal co-occurrence detection from entity–event binding?

  2. What are the label accuracies of entity-intervened, event-intervened, and dual-intervened counterfactuals? Could a manual audit identify the main error types, and do FCL’s gains persist with more strictly filtered or verified counterfactual samples?

5 comments
AuthorsOpenReview ↗

Response to Reviewer kZF4 (Part 1)

We thank you for your careful review and constructive feedback. We appreciate your overall positive assessment of our work, especially your recognition of its focus on partial-match ambiguity, controlled comparisons, substantial gains across settings, and practical inference efficiency. We address your concerns below.

Q1. Analysis and evaluation of simultaneous distractors performing the queried action while the reference entity remains visible but inactive in the same scene

A1. We thank you for highlighting this challenging case involving a simultaneous distractor. We analyze these cases from three perspectives: (i) their frequency in the current benchmark based on a manual audit of randomly sampled videos, (ii) mitigation through our framework design, and (iii) targeted empirical evaluation on the identified cases.

1) Only a few such cases are found in the manual audit. We manually inspect 200 randomly sampled training videos and 200 randomly sampled test videos from IDVTG-Gym. We identify 5 potential cases in training and 3 in test (2.5% and 1.5%, respectively). All three manually identified test cases are included in the revised paper: one in Fig. 4 and the remaining two in Appendix Fig. 9(a–b). This audit indicates that such cases occur at a low frequency in the current IDVTG-Gym benchmark.

2) The overall framework design helps mitigate such distractors. SCG’s synchronization loss assigns zero targets outside the annotated GT interval, penalizing high conjunctive responses at distractor locations during training. The base path jointly models the reference image, video context, and event query. The final grounding objective is applied to the calibrated logits, allowing the base and SCG paths to be learned jointly, while FCL provides factor-validity supervision under selective interventions. Together, the overall framework design helps suppress mismatched intervals.

3) Our method correctly localizes all 3/3 identified test cases. Fig. 4 in the revised paper visualizes one representative case with an IoU of 0.987, while Appendix Fig. 9 presents the remaining two cases. These results indicate that our method maintains good localization performance even in these challenging cases.


Q2. Counterfactual-label accuracy, main error types, and performance after stricter filtering

A2. We thank you for suggesting this analysis. We examine counterfactual reliability through manual label auditing, error-type analysis, and retraining with more strictly filtered samples.

1) A manual audit confirms high label accuracy in the sampled counterfactuals. We manually audit three counterfactual variants for each of 200 randomly sampled IDVTG-Gym query groups, yielding 600 counterfactual inputs. The audited samples achieve 96.2% overall label accuracy, with accuracies of 98.0%, 96.0%, and 94.5% for entity-, event-, and dual-intervened counterfactuals, respectively. These results support the effectiveness of our construction and filtering strategy in producing valid factor-specific counterfactuals. The factual ++ state inherits the benchmark annotation and is excluded from this audit.

InterventionEntity (-+)Event (+-)Dual (--)Overall
Audited inputs200200200600
Incorrect labels481123
Label accuracy98.0%96.0%94.5%96.2%

2) Event-side semantic compatibility is the main error type. Event-intervention errors are more frequent than entity-intervention errors (8 versus 4) and primarily involve coarse-grained event descriptions with overlapping semantics. Entity-side errors occur when the replacement image still matches the target entity. The two error groups overlap once, yielding 11 dual-intervention errors.

3) Stricter filtering of event-text counterfactuals yields slightly better overall performance. On top of our standard FCL filtering, we apply an additional automated filtering procedure assisted by Qwen3-8B to all event-text replacements in the training set according to predefined semantic rules, while entity-image replacements retain the standard filtering procedure. We retrain the model on the retained samples under the same training budget.

Counterfactual filteringDatasetmIoUR@1(.5)R@1(.7)
Standard FCL filteringIDVTG-Gym64.3373.5667.08
Standard + Qwen-assisted text filteringIDVTG-Gym65.0074.3667.55
Standard FCL filteringIDVTG-InternVid53.7957.5246.34
Standard + Qwen-assisted text filteringIDVTG-InternVid54.2657.7646.03

These results show that FCL remains effective under stricter counterfactual filtering, with slightly improved overall localization performance.

AuthorsOpenReview ↗

Response to Reviewer kZF4 (Part 2)

Q3. Direct validation of factor validity and mismatch-source attribution

A3. We thank you for requesting direct validation of the evidence branches’ semantics and their ability to distinguish mismatch sources. Our controlled four-state evaluation directly supports factor identifiability and mismatch-source attribution.

We evaluate the frozen SCG+FCL model on 1,000 inputs derived from 250 randomly sampled IDVTG-Gym test queries. For each query, we automatically construct three counterfactual variants using our existing procedure and manually verify that each variant corresponds to its intended factor-validity state. Together with the factual input, this yields 250 examples per state.

Four-state predictions are obtained using softmax–argmax over the temporally aggregated state logits. Entity and event predictions are obtained by thresholding the corresponding aggregated branch probabilities at 0.5.

MethodAcc(++)Acc(+-)Acc(-+)Acc(--)Four-state Acc.Entity Acc.Event Acc.
SCG+FCL81.20%85.60%88.00%92.40%86.80%91.40%94.80%

Here, the two signs indicate entity and event validity, respectively: ++, +-, -+, and -- denote factual, text-replaced, image-replaced, and dual-replaced inputs. The 85.60% and 88.00% accuracies on text- and image-replaced inputs show that the model can distinguish event mismatches from entity mismatches. The 91.40% entity accuracy and 94.80% event accuracy further support the intended semantic roles of the two evidence branches. Together, these results provide direct factor-level evidence beyond the reported localization gains.

AuthorsOpenReview ↗

A Brief Follow-up

Dear Reviewer kZF4,

Thank you again for your positive assessment and constructive feedback.

As the author–reviewer discussion period draws to a close, we would be grateful for your thoughts on our responses and additional results. Please let us know if any concerns remain; we would be happy to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.