- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies fine-grained limb-motion fidelity in dense, multi-shot video captioning, emphasizing the distinction between omitted details and incorrect assertions. It introduces FlexBench, comprising 660 video sequences, 3,105 shots, and 18,161 evaluation queries, with human-verified person identities and shot-specific annotations of actions, limb states, and visibility. Its Graded Physical Alignment (GPA) metric combines positive credit with explicit penalties for incorrect motion claims. The proposed Graded Margin DPO (GM-DPO) uses synthetic caption perturbations to assign error-severity-dependent preference margins and loss weights. Experiments compare four preference-learning baselines across three VLM backbones and three training seeds. GM-DPO improves GPA over DPO by 2.02–3.40 points and reduces weighted hallucination rate by 21.3% on Qwen3-8B, while retaining long-form captions and competitive performance on three additional multimodal benchmarks.
Critical strengths
-
The benchmark addresses an important and insufficiently measured failure mode of video captioning. FlexBench evaluates freely generated captions at the person, limb, and shot levels, including stationary and hidden limbs rather than only salient actions. Distinguishing omissions from false assertions is particularly consequential when captions serve as downstream training supervision. The multi-round human verification and reported evaluator–human agreement provide substantial support for the benchmark’s usefulness. I regard this targeted evaluation resource as the paper’s strongest contribution.
-
The empirical improvements are consistent across architectures and training seeds. Under a shared preference dataset and auxiliary NLL objective, GM-DPO achieves the highest substantive-action and GPA scores among the evaluated preference methods on all three backbones. The positive paired GPA gains across seeds make the results more convincing than a single-model demonstration. Component ablations also support combining margins and weights, while caption-length statistics indicate that the gains are not simply obtained by producing shorter, less informative outputs. This establishes practical value for the proposed training recipe.
Critical weaknesses
- The experiments do not isolate the benefit of severity conditioning from stronger generic preference optimization. The weight-only and margin-only ablations establish that these components help, but both remain severity-dependent. The paper lacks controls using a constant margin and constant weight matched to their average magnitudes, shuffled severity assignments, or comparable tuning of DPO’s optimization strength.
Questions for authors
Have you compared GM-DPO against controls using (a) constant margins and weights matched to the averages in GM-DPO, and (b) shuffled severity labels? How does the evaluator handle an unsupported action claim that does not correspond to any reference-derived checklist item?
4 comments
Reviewer comment (verbatim)
"How does the evaluator handle an unsupported action claim that does not correspond to any reference-derived checklist item?"
Response (continued)
To check whether improving limb-motion accuracy affects broader video understanding, we evaluated all five Qwen3-8B policies on four complementary benchmarks. These more directly probe the factual and temporal understanding needed for accurate captions than our original Video-MME, POPE, and Video-ChatGPT evaluations.
- VideoHallucer [30] tests intrinsic and extrinsic hallucinations across five categories. Both answers in each yes/no pair must be correct; we report the macro-average across categories.
- VidHalluc [14] tests action (ACH: binary pairs and MCQ) and temporal sequence (TSH) hallucinations, with 13,426 responses per model.
- TempCompass [18] tests action, direction, speed, attribute change, and event order through MCQ and caption generation (CG). CG provides candidate information; Qwen3.8-Max answers a question from the generated caption to check its temporal content.
- VinoGround [34] tests temporal matching on 500 video–caption groups. Text (video-to-caption) and Video (caption-to-video) require both choices in each direction to be correct; Group requires all four. We use local inference and the official scorer.
All scores are percentages. Results marked with † are cited from the original benchmark papers for comparison; the ACH binary references use Binary QA Pair. Bold marks the best score among our five policies.
The table shows VideoHallucer paired accuracy (VH), VidHalluc action-hallucination binary pairs (ACH pair) and temporal sequence hallucinations (TSH), TempCompass caption generation (TC-CG), and VinoGround group accuracy (VG-Group). Table 4 reports all nine metrics.
| Model | VH | ACH pair | TSH | TC-CG | VG-Group |
|---|---|---|---|---|---|
| DPO | 58.56 | 78.69 | 90.33 | 66.62 | 17.00 |
| cDPO | 58.58 | 78.36 | 90.67 | 66.52 | 17.80 |
| IPO | 58.46 | 78.27 | 92.00 | 66.02 | 17.20 |
| SimPO | 58.69 | 78.11 | 91.33 | 65.82 | 17.20 |
| GM-DPO | 58.98 | 79.20 | 92.80 | 67.23 | 18.60 |
| GPT-4o † | 53.30 | 66.79 | 82.00 | — | 24.60 |
| GPT-4o (CoT) † | — | — | — | — | 35.00 |
| Gemini-1.5-Pro † | 37.80 | 59.10 | 83.83 | — | 10.20 |
| Qwen2-VL-72B † | — | — | — | — | 17.40 |
| Qwen2-VL-7B † | — | — | — | — | 15.20 |
| LLaVA-OneVision-Qwen2-7B † | — | — | — | — | 14.60 |
| PLLaVA-7B † | 38.10 | — | — | — | — |
| LLaVA-NeXT-Video-DPO-7B † | 32.00 | — | — | — | — |
| VideoLLaMA2-7B † | — | 29.09 | 26.17 | — | — |
| VideoChat2-7B † | 7.80 | — | — | 38.50 | — |
| Qwen-VL-Chat-7B † | — | — | — | 37.30 | — |
| Video-LLaVA-7B † | 17.80 | 9.87 | 27.17 | 34.80 | 6.60 |
In the full results (Table 4), GM-DPO leads on eight of the nine metrics. Over DPO, it gains 0.51 and 2.47 percentage points on VidHalluc ACH binary pairs and TSH, 0.61 points on TempCompass caption generation, and 1.60 points on VinoGround Group. These modest improvements support our aim of improving limb-motion factuality while retaining broader video understanding, including temporal information conveyed in generated captions.
We have replaced the original evaluations in Sec. 5.6 (lines 479–492; Table 4) with these results and discussion, and moved the Video-MME, POPE, and Video-ChatGPT results to Appendix C.4.
References
[30] Wang et al. VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models. 2024.
[14] Li, Im, and Fazli. VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding. CVPR 2025.
[18] Liu et al. TempCompass: Do Video LLMs Really Understand Videos?. Findings of ACL 2024.
[34] Zhang, Cai, and Lee. Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videos. 2024.
Dear Reviewer kCuW,
Thank you again for your careful review and thoughtful comments. I very much appreciate the time and effort you have devoted to reviewing this paper.
The authors have now replied to your initial comments. Could you please take a look at their responses and continue the discussion with them when you have a chance?
Thank you so much for your continued help and engagement!
Best wishes,
Your AC
Dear Reviewer kCuW,
Thank you once again for the time and effort you dedicated to reviewing our paper. Your insightful comments—especially regarding isolating the effect of severity conditioning—have helped us further clarify the contribution of GM-DPO and significantly improve our work.
We fully understand that you have a busy schedule, so we just wanted to gently follow up and see if our responses, the newly added control experiments (constant margins/weights and shuffled severity labels), and the results across four additional benchmarks have adequately addressed your concerns. If there are any remaining questions, or if we have misunderstood any of your points, please do let us know—we are more than happy to provide further clarification.
If you feel that your concerns have been satisfactorily resolved, we would be sincerely grateful if you would kindly consider raising your rating.
Best regards,
The Authors
Reviewer comments (verbatim)
Response
Thank you for the helpful suggestions. We have added both controls and evaluations on four external benchmarks, and clarified the scope of our evaluation. All changes in the revised manuscript are highlighted in blue.
1) Severity-conditioning controls
Thank you for suggesting these controls. We ran both with the same seeds and evaluation protocol as in the paper:
Both use α=0.1, γ=0.2, β=0.1, and 1.0LNLL, with the paper's multi-judge evaluation.
As the table shows, both controls improve GPA over DPO, so part of the gain can come from changing the margin and weight. GM-DPO nevertheless improves GPA over these controls by 2.60 and 2.94 points, while reducing WHR by 3.66 and 7.02 percentage points. Shuffling gives slightly higher GPA_clip (30.64 vs. 30.52), but also higher WHR (21.82% vs. 14.80%). This supports using each pair's actual error severity, beyond changing the overall optimization strength.
We have added these results and discussion to Sec. 5.5 (lines 474–477) and Appendix C.3 (lines 1004–1018; Table 9).
2) Unsupported actions and hallucination coverage
Thank you for raising this point. We should have explained the evaluator's inputs more clearly. Each question is scored using the full structured limb reference. The original Sec. 3.2 (lines 159–160) describes this setup, and the submitted
FlexBench/data/FlexBench_en.jsonland its Chinese counterpart already containgt_extracted_actions_by_shotandquestions. The annotations list each person's actions, laterality, order, and limb states by shot. For each Qi, the evaluator receives the full reference, generated caption, and question list.This allows questions about inactive or invisible limbs to catch invented actions as well. For example, if the reference states that both hands are invisible, a claim about table-tapping receives −1 under the non-substantive/no-hand rubric. We have clarified this in Sec. 3.2 (lines 180–183) and added a visual example in Appendix A.3, Fig. 8 (lines 731–745; caption, lines 803–808).
We agree, however, that an extra claim unrelated to all checklist items may still escape a penalty. Our detailed annotations reduce this risk. FlexBench focuses on limb-motion accuracy within dense captions; it does not evaluate all scene, camera, or audio descriptions. This limitation was already acknowledged in our original Conclusion. The revised Sec. 6 (lines 524–539) retains this scope and outlines future evaluation, with further discussion in Appendix D.