- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies whether the benefits of individual frames add up in video question answering. It also measures the loss from selecting frames based on their individual scores. The authors use controlled experiments with fixed frame budgets, surrounding frames, and candidate positions. They measure individual frame benefits and interactions through changes in the model’s score for the correct answer.
Critical strengths
-
The core experiments cover six models and three datasets with annotated evidence. The results show that evidence frames are more useful, but their combined benefits often differ from the sum of their individual benefits.
-
The paper tests selecting three frames from four candidates. Using individual frame scores alone finds the highest-scoring combination in 55.2% of cases, while adding measured pairwise interactions raises this rate to 68.6%.
-
The paper evaluates several frame selection methods. Section 6 tests the stability of the findings by changing frame order, candidate positions, surrounding frames, and frame budget.
Critical weaknesses
-
Limited new insights and no real progress toward the research goal: The paper uses many experiments to confirm intuitive findings that evidence frames are more useful, and ranking frames by individual scores can miss the best combination. However, it provides little further explanation of these findings. The analysis does not clearly identify the specific causes of selection failures. It also does not propose and test a practical strategy to address these causes. This is my main concern with the paper.
-
The experiments miss the core issue, and the analysis is unconvincing: A large, even excessive, number of experiments repeatedly confirm the advantage of evidence frames over outside frames. In contrast, the four-candidate experiment directly studies combination selection but receives little analysis. Figure 3 (right) compares the absolute interaction of evidence-frame pairs and outside-frame pairs. However, the frames in the two groups already differ. The individual and joint gains of outside frames are both close to zero, so their interaction is naturally close to zero as well. The paper does not clearly explain why comparing these values is meaningful. Section 6 and Table 2 then repeatedly verify this gap through stress tests under various settings. The extensive analysis with unclear value is another major reason for my recommendation to reject the paper.
Overall, the paper has a reasonable motivation, but it provides no substantial analysis of the causes and no solution. And the writing is also difficult to understand. I recommend clear rejection.
Questions for authors
-
The four-candidate experiment relies on scores for the correct answer. How can these scores be obtained when the answer is unknown?
-
Methods such as DPP and MDP3 in Section 5.3 select frames using embedding-based scores. How do their frame rankings relate to rankings based on correct-answer scores? Can the preceding analysis help these methods select frames?