- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies image-disambiguated video temporal grounding. The authors build on VGD-Agg and introduce two components. SCG separately estimates entity and event evidence and combines them through pointwise multiplication. The combined evidence receives supervision during training and is used to calibrate proposal scores at inference. FCL augments the training queries by replacing the reference image, the event description, or both, while keeping the original video and annotated interval fixed. The experimental results provide some support for the effectiveness of the proposed method.
Critical strengths
The paper tackles a useful problem: finding the right event when a text query alone cannot identify the intended person. The design is straightforward, with SCG combining entity and event scores and FCL adding supervision that distinguishes which condition is satisfied. The results show consistent gains over the reproduced VGD-Agg baseline on Gym, InternVid, and Web. The ablations also provide useful evidence for the role of each component, including comparisons with different supervision objectives on the same counterfactual samples. The additional inference cost is small, and FCL is used only during training.
Critical weaknesses
-
SCG uses conventional frame-level auxiliary supervision, scores entity and event evidence separately, and combines them through multiplication to calibrate localization scores across temporal scales. FCL adds factor-level supervision by replacing the reference image, the text query, or both, following ideas related to selective counterfactual training. The four-state supervision within a fixed interval offers some novelty, and the performance gains are substantial. Still, the overall contribution appears incremental, mainly adapting and combining existing mechanisms for ID-VTG.
-
High scores from both branches at the same time do not necessarily mean that the target person performs the queried action. For example, the target person may stand in the scene while someone else performs the action, allowing both branches to assign high scores. FCL may help address this issue, but the paper does not specifically evaluate such cases or report the accuracy of the individual branches or four validity states. The results support improved localization, but provide limited direct evidence for person–action binding and factor identifiability.
Questions for authors
- Do the test sets include cases where the target person is present but someone else performs the queried action? If so, how do SCG and FCL perform on these cases? Accuracy results for the entity and event branches, or the four validity states, would also help substantiate the claim of factor identifiability.
- Does the gain from FCL remain when SCG score calibration is disabled at inference? Please compare the SCG and SCG+FCL models with lambda=0 for both. This would help clarify whether FCL also improves the base grounding model or whether its gains mainly come through the SCG calibration signal.
5 comments
Response to Reviewer ipZR (Part 2)
Q3. Direct validation of factor validity and mismatch-source attribution
A3. We thank the reviewer for requesting direct validation of the evidence branches’ semantics and their ability to distinguish mismatch sources. Our controlled four-state evaluation directly supports factor identifiability and mismatch-source attribution.
We evaluate the frozen SCG+FCL model on 1,000 inputs derived from 250 randomly sampled IDVTG-Gym test queries. For each test query, we automatically construct three counterfactual states using our existing procedure and manually verify that each state corresponds to its intended factor-validity label. Together with the factual input, this yields 250 examples per state.
Four-state predictions use softmax–argmax over the aggregated state logits. Entity and event predictions are obtained by thresholding the corresponding aggregated branch probabilities at 0.5.
| Method | Acc(++) | Acc(+-) | Acc(-+) | Acc(--) | Four-state Acc. | Entity Acc. | Event Acc. |
|---|---|---|---|---|---|---|---|
| SCG+FCL | 81.20% | 85.60% | 88.00% | 92.40% | 86.80% | 91.40% | 94.80% |
Here, ++, +-, -+, and -- denote factual, text-replaced, image-replaced, and dual-replaced inputs, respectively. The 85.60% and 88.00% accuracies on text- and image-replaced inputs show that the model can distinguish event mismatches from entity mismatches. The 91.40% entity accuracy and 94.80% event accuracy further support the intended semantic roles of the two evidence branches, providing direct factor-level evidence beyond localization gains.
Q4. Does the gain from FCL remain when SCG score calibration is disabled at inference?
A4. Yes. FCL continues to provide clear performance gains when SCG calibration is disabled at inference for both models. We compare the separately trained SCG and SCG+FCL models and set only at inference for both. This removes SCG’s signed residual correction from both models during inference. FCL provides additional training supervision and introduces no inference-time branch.
| IDVTG-Gym Overall, inference | R@1(.5) | R@1(.7) | mIoU |
|---|---|---|---|
| SCG | 67.08 | 61.59 | 58.76 |
| SCG+FCL | 71.76 | 66.23 | 63.04 |
| FCL gain | +4.68 | +4.64 | +4.28 |
Under this controlled setting, FCL improves R@1(.5), R@1(.7), and mIoU by 4.68, 4.64, and 4.28 points, respectively. These gains show that FCL improves the learned grounding model during training and that its benefit does not depend solely on SCG’s inference-time calibration.
A Brief Follow-up
Dear Reviewer ipZR,
Thank you again for your time and constructive feedback.
As the author–reviewer discussion period draws to a close, we would appreciate your thoughts on our responses and additional results. If any concerns remain, we would be happy to provide further clarification.
Response to Reviewer ipZR (Part 1)
We thank the reviewer for recognizing the useful task, straightforward design, substantial and consistent gains, informative ablations, low inference overhead, and the novelty of fixed-interval four-state supervision. We address your concerns below.
Q1. A task-driven factorization–conjunction–attribution framework for ID-VTG
A1. Our contribution lies in a task-driven formulation and a unified framework tailored to the structural ambiguity of ID-VTG.
1) The design follows the conjunctive structure of ID-VTG. A valid interval must simultaneously match the reference entity and the described event, whereas conventional joint scores can remain high for entity-only or event-only partial matches. SCG therefore factorizes the two conditions and evaluates their same-time conjunction to adaptively enhance or suppress proposal scores. The binary validity of the entity and event conditions naturally gives rise to four states under selective image and text interventions, motivating FCL to supervise which conditions remain valid.
2) The components form a unified factorization–conjunction–attribution framework. SCG converts factorized evidence into an explicit, scale-compatible conjunctive correction: it aligns entity and event responses on a shared fine temporal grid, uses a non-compensatory Product so that strong evidence for one condition cannot mask weak evidence for the other, and propagates the resulting signal across scales as a signed residual. FCL then makes the constituent factors attributable through controlled image/text interventions that preserve the video and temporal support. Rather than collapsing all counterfactuals into generic negatives, it retains the complete four-state validity structure and supervises which conditions remain valid.
3) Controlled and broad evaluations validate the design. The complete framework consistently improves performance across three benchmarks, different query granularities, and multiple ambiguity types. Under identical counterfactual samples, Table 5 shows that FCL outperforms conventional binary counterfactual supervision and independent factor-wise supervision. Table 4 further shows that SCG’s non-compensatory Product consistently outperforms Mean and Minimum. Our analysis explains how Product combines non-compensatory conjunction with responsiveness to both factors, properties that are particularly important for ID-VTG.
Q2. Do the test sets include cases where the target person is present but someone else performs the queried action? If so, how do SCG and FCL perform on these cases?
A2. We thank the reviewer for highlighting this challenging case. We analyze its frequency in the current benchmark, mitigation through our framework design, and the performance of SCG and SCG+FCL on the identified cases.
1) Only a few such cases are found in the current benchmark. We manually inspect 200 randomly sampled IDVTG-Gym training videos and 200 test videos, identifying 5 potential cases in training and 3 in test (2.5% and 1.5%, respectively). This indicates that such cases occur at a low frequency in IDVTG-Gym. All three manually identified test cases are included in the revised paper: one in Fig. 4 and the remaining two in Appendix Fig. 9(a–b).
2) The overall framework design helps mitigate such distractors. SCG combines entity and event evidence through same-time conjunction before producing a signed correction. Its synchronization loss assigns zero targets outside the annotated GT interval, penalizing high conjunctive responses at distractor locations during training. The base path jointly models the reference image, video context, and event query, while FCL provides factor-validity supervision under selective interventions. Together, the overall framework design helps suppress mismatched intervals.
3) Both SCG and SCG+FCL achieve good localization performance on all 3/3 identified test cases. These results show that both SCG and the complete model can successfully localize the correct intervals even when another entity simultaneously performs the queried action.