- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes the Temporal Structure Learner (TSL) for video question answering and the localization of temporal evidence. The method divides the video into temporal units and predicts five states, organizing evidence intervals through valid transition constraints and refining boundaries using continuous offsets. TSL is integrated into the decoding layers of a frozen large-scale multimodal model and trained using a combination of state supervision, path and endpoint supervision, answer generation loss, and reinforcement learning. The model re-examines the video evidence based on the predicted intervals to generate answers.
Critical strengths
-
The problem is of practical significance, and the link between localization and answering is direct. The paper focuses on the consistency between the answer and its temporal evidence; by feeding the localized evidence back into the model for answer generation, the localization process actively contributes to the generation of the answer. This design facilitates verifiable evidence for the answer and offers a logical pathway for jointly optimizing localization quality and answer accuracy.
-
The combination of discrete state constraints and continuous boundary prediction is well-founded. State transition constraints ensure the structural validity of the predicted segment's start and end points, while continuous offsets allow for boundary refinement within temporal units, thereby balancing structural constraints with localization precision. Theoretical analysis establishes a reversible relationship between the standardized state-offset representation and the set of temporal evidence supporting the answer—given specific temporal segmentation—providing a formal justification for this representation design.
Critical weaknesses
1. Key implementation details are incomplete. There are multiple instances of "Appendix ??" and "equation ??" in the main text and appendices regarding key steps such as transition score calculation, feasibility mapping for boundary offsets, and RL sampling and gradient computation.
2. The sources of training supervision and the experimental comparison setup remain insufficiently clear. The state supervision, endpoint regression, and RL rewards described in the methodology all rely on annotated evidence; however, Appendix A.2 notes that the NExT-GQA training set lacks temporal evidence annotations and that the validation and test sets are merged for evaluation. The paper needs to further clarify the data and data splits used for TSL's training supervision, as well as the data used for model selection. Although Appendix A.3 defines the metrics and basic post-processing, it is necessary to specify whether the baselines were re-run on the same evaluation set and how task prompts, the joint QA pipeline, and video input budgets were aligned.
3. The ablation study has not yet disentangled the contributions of path supervision and boundary regression. Table 4 presents results where the combined term is either included or excluded in its entirety; consequently, the respective roles of path structure learning and continuous boundary regression cannot be determined. Although the answer generation loss is retained across configurations, it relies on annotated evidence (); furthermore, for the configuration where both flow and RL are removed, the training path and initialization of the boundary prediction head require clarification. Moreover, existing experiments primarily compare loss combinations and have not yet independently validated the roles of the four TSL components.
Questions for authors
see weakness