The paper has multiple contradictions that fail to strengthen the argument. For example, the authors asserted that observations can be traced to evaluated media (page 5, paragraph 1), yet also assert that “category grouping branch in Figure 2 [does not] imply a prerequisite” (pg 4, paragraph 2). The verbiage in the first three sections on fixed graph of visual requirements is confusing at best.
The discussion on the test harness lacked details (and the appendix never elaborated on it either). Primarily, a description of ideal image conditions were never made, hence it was difficult to accept the assertions on performance. Similarly, Figures 1 and specifically Figure 3, show multiple “pipelines” where videos are not exactly the same for each “pipeline” test, biasing likely complex scenes unfairly to other methods, and likely introducing better scores for GIVER. If this was a deficiency on the LLM aspect of the method, then it was not discussed. A fair comparison with fair scoring would have attempted to ensure similarities in frames generated from prompts as possible. There were no assertions that the same prompt (prose, stance, etc) was used across all pipelines, which needs to be clarified.
The initial conditions should have been normalized across all tests. Therefore, the evidence is a bit lacking: For example, acceptance is underspecified. §3 defines a graph score but acceptance (as logic would dictate) depends on critical features, that then dictate the nature of failures, vetoes, applicability rules, and waivers. The discussion on waivers was short and could have used more elaboration. The repair methods were also mostly assertions lacking any implementation discussions. There were discussions on “requirements” which suggests that there may be bias on input prompts that can cascade through the workflow, never mind the fact that the initial images also likely can drive cascading effects into the workflow that would penalize one method over another.
There are a number of concerns regarding multiple aspects of GIVER. The graph score could have benefitted from more elaboration on the method and quantification. The repair mechanism is largely asserted, but bereft of information on how the repair is actually done, or what specific features trigger it. So repeatability may be difficult to perform. This is somewhat demonstrated by the discrepancies in the pipeline (results) illustrated in Figure 3, where a cascade of prompt inputs may have altered the complexity of initial scenes per pipeline, which in turn may have unfairly penalized certain pipelines when compared to GIVER. Finally, the verification and validation discussion (§ 4.2 and beyond) appear to have many nuances in scoring discrepancies. Discussion on the discrepancy in human evaluation versus automated methods is lacking. Some of the SF and Phys tests differed by more than 0.02, and others 0.15 respectively. The parameters for each test are also concerning, with ViMax-Bench having significantly more test samples than ST-Bench-10, likely biasing the results. The human test is also not in parity with the automated test, where human evaluations were consistent for both ViMax-Bench and ST-Bench-10 in terms of sample size compared to each other, but significantly less in sample for the automated test. Hence, it’s difficult to trust the results, as good as the assertions may be. It is not clear why GIVER had two tests (Figure 4) when the other pipelines/methods did not. There were no justifications given for this “retry” (§ 4.4, and Figure 4).
Therefore, this reviewer thinks there’s insufficient specifications or elaborations on assertions to strengthen claims made regarding performance. The evaluated outputs received favorable scores for the introduced method, but they fail to establish the reliability of the proposed mechanism or explain why the gains were significantly better than existing methods. Therefore, any claims to novelty disappear due to lack of evidence.