- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies clinical skill assessment from video and proposes ATV, an Anticipate-Then-Verify framework that treats assessment as verification against learned normative latent dynamics. ATV first represents videos as temporally aligned object-centric states using recurrent Slot Attention. It then learns forward and reverse latent dynamics from correctly performed procedures, and finally compares observed state transitions with these normative references to produce rubric-conditioned ordinal scores. The discrepancy representation decomposes transition differences into progress, orthogonal deviation, and endpoint mismatch rather than relying only on raw prediction error. The paper also introduces ClinSkill-V, a physician-annotated dataset of clinical skill examinations covering CPR, AED, and bag-mask ventilation. The dataset contains 15,775 step-aligned clips from 544 full-length videos. ATV is compared with general video MLLMs, medical video MLLMs, omni-modal models, and latent predictive models, and reports the strongest overall performance, particularly on fine-grained ordinal scoring.
Critical strengths
The strongest aspect of the paper is the formulation of procedural assessment as verification against normative dynamics rather than direct video-to-score prediction. In particular, the distinction between predicting what is likely to happen and modeling how a correctly executed procedure should evolve is well motivated. The bidirectional design and the structured discrepancy representation also go beyond simply treating prediction error as an anomaly score. I also found the ablation results informative. Removing normative dynamics or discrepancy conditioning clearly degrades the ordinal metrics, while raw prediction errors improve binary MCC but perform worse on fine-grained scoring. This supports the paper's argument that detecting whether an execution is abnormal and estimating the degree of procedural deviation are different problems. The dataset is another strength: it is collected from real clinical skill examinations, uses physician-defined rubrics, and includes dual-expert annotations with adjudication.
Critical weaknesses
My main concern is that the experimental task is narrower than some of the paper's procedural-state and world-model framing suggests. Each sample is already segmented into a rubric-defined procedural step, and the model performs assessment within that step. Therefore, the current experiments do not test whether ATV can maintain procedural state across a complete multi-step procedure or reason about dependencies between earlier and later steps. The proposed formulation is still meaningful for step-level assessment, but the distinction between within-step temporal modeling and broader procedural state tracking should be made clearer. A second concern is that the comparison with external baselines does not fully isolate the contribution of normative dynamics. ATV uses a dedicated ceiling-aware hierarchical ordinal objective that explicitly separates full-score detection, non-full severity estimation, and ordinal distance, whereas the MLLM baselines are trained with their own score-label objectives. The component ablations convincingly support the contribution of BNDM and discrepancy reasoning within ATV, but part of the gap in Table 2 could still come from the scoring objective rather than the proposed dynamics model itself. A matched scoring-head/objective comparison on one or two strong baselines would make the main comparison more conclusive.
Questions for authors
Q1: The current benchmark uses rubric-aligned step clips. Can ATV also model state across multiple consecutive procedural steps, or is the method intended only for within-step assessment? A full-procedure or multi-step experiment would help clarify how strongly the current results support the broader procedural-state tracking claim.
Q2: ATV uses a custom hierarchical ordinal objective, while the MLLM baselines use different scoring objectives. Please provide a comparison in which a strong baseline such as Molmo-2 or Qwen3-VL uses the same ordinal scoring head/objective as ATV.
Q3: How are “normative” training samples selected for BNDM? Please report the number of full-score/correct samples available for each rubric step and clarify whether all steps have sufficient normative examples.
Q4: Correct executions can follow different valid trajectories across operators, speeds, and styles. Please evaluate whether the learned normative dynamics remain tolerant to such variation, for example with operator-held-out evaluation or by training on one subset of correct performers and testing on another.
Q5: OPSR is central to the claim of object-centric state tracking, but its evaluation is mainly qualitative. Please provide a quantitative comparison against a non-object-centric representation, such as pooled/frame-level DINO features or temporally aligned patch features, to show whether the slot-based state representation is necessary.
7 comments
Continuation of Response to Reviewer NDDt, Comment 1
(2) Scoring Objective Control
Our internal ablations provide controlled evidence for the effectiveness of ATV's core methodological components. By retaining the same scoring backbone and objective across ablation variants, these experiments demonstrate that BNDM and discrepancy reasoning contribute to fine-grained procedural assessment independently of differences in scoring objectives. Therefore, the effectiveness of these components is supported not only by comparisons with external baselines but also by controlled experiments within ATV.
We appreciate your concern that differences in scoring objectives may affect cross-model comparisons. To further address this potential confounding factor, we select the best-performing model from each of the four baseline categories in Table 2—Molmo-2, Qwen2.5-Omni, SurgVidLM, and V-JEPA 2.1—and conduct additional controlled experiments under a matched scoring objective. We replace the original scoring objective of each selected baseline with the same ceiling-aware hierarchical ordinal objective used by ATV, while keeping the remaining experimental settings consistent with those reported in the main comparison (Table 2). This experiment directly examines whether ATV's performance advantage persists when differences in scoring objectives are controlled. The experimental results are as follows:
| Model | Scoring Objective | NMAE^M ↓ | BAcc ↑ | [email protected] ↑ | BAcc@1 ↑ | MCC ↑ |
|---|---|---|---|---|---|---|
| Molmo-2 | Original | 0.415 | 0.347 | 0.423 | 0.659 | 0.309 |
| Molmo-2 | Hierarchical Ordinal | 0.529 | 0.238 | 0.318 | 0.477 | -0.076 |
| Qwen2.5-Omni | Original | 0.422 | 0.326 | 0.400 | 0.653 | 0.237 |
| Qwen2.5-Omni | Hierarchical Ordinal | 0.422 | 0.345 | 0.414 | 0.638 | 0.077 |
| SurgVidLM | Original | 0.425 | 0.335 | 0.416 | 0.656 | 0.217 |
| SurgVidLM | Hierarchical Ordinal | 0.411 | 0.352 | 0.426 | 0.656 | 0.279 |
| V-JEPA 2.1 | Original | 0.429 | 0.335 | 0.404 | 0.647 | 0.183 |
| V-JEPA 2.1 | Hierarchical Ordinal | 0.429 | 0.331 | 0.405 | 0.645 | 0.208 |
| ATV (Ours) | Hierarchical Ordinal | 0.324 | 0.410 | 0.512 | 0.714 | 0.312 |
The results reveal that the impact of the ceiling-aware hierarchical ordinal loss varies across model architectures. While SurgVidLM benefits from this objective across the reported metrics, Qwen2.5-Omni exhibits modest improvements in ordinal accuracy, V-JEPA 2.1 shows limited changes, and Molmo-2 experiences substantial performance degradation. Importantly, even the strongest baseline under the matched scoring objective, SurgVidLM, remains substantially behind ATV, with BAcc of 0.352 versus 0.410 and [email protected] of 0.426 versus 0.512. These findings indicate that differences in scoring objectives alone cannot account for ATV's performance advantage, providing further evidence for the effectiveness of our proposed methodological design. We have made the corresponding updates in Appendix F.5 (Controlling Scoring Objectives and Prompting Strategies).
Response to Questions For Authors from Reviewer NDDt
Q1 The current benchmark uses rubric-aligned step clips. Can ATV also model state across multiple consecutive procedural steps, or is the method intended only for within-step assessment? A full-procedure or multi-step experiment would help clarify how strongly the current results support the broader procedural-state tracking claim.
Response:
Thank you for highlighting the distinction between within-step state modeling and procedural-state tracking across multiple steps. ATV is currently designed and evaluated for rubric-aligned within-step procedural assessment, rather than end-to-end tracking across an entire clinical procedure.
Importantly, a rubric-defined step is not necessarily a single atomic action. It often involves multiple temporally dependent interactions among the operator, patient, and instruments, requiring the assessment of how their states evolve throughout execution. Our objective is therefore to move beyond action recognition or likely-future prediction toward verifying whether observed state transitions conform to normative procedural dynamics. Rubric-aligned segmentation allows us to investigate this question while separating normative dynamics modeling from additional challenges such as procedural step localization and cross-step rubric alignment.
Within this scope, OPSR represents evolving object-centric states, BNDM models normative temporal dynamics, and DRPV evaluates discrepancies between observed and normative state transitions. The current results demonstrate the effectiveness of this framework for within-step procedural verification. However, they do not establish that ATV can maintain state continuity or reason about dependencies across multiple consecutive procedural steps. Extending ATV to multi-step or full-procedure assessment would require addressing additional challenges, including long-video temporal sampling, joint interpretation of multiple rubric criteria, and temporal localization of individual procedural steps. Our use of rubric-aligned clips allows us to focus on modeling and verifying interaction-driven state transitions without introducing these additional complexities. We have clarified this distinction in the revised manuscript to more precisely define the scope of our contributions and experimental evidence.
Q2 ATV uses a custom hierarchical ordinal objective, while the MLLM baselines use different scoring objectives. Please provide a comparison in which a strong baseline such as Molmo-2 or Qwen3-VL uses the same ordinal scoring head/objective as ATV.
Response:
We have conducted additional controlled experiments by replacing the original scoring objectives of four representative baselines—Molmo-2, Qwen2.5-Omni, SurgVidLM, and V-JEPA 2.1—with the same ceiling-aware hierarchical ordinal loss used by ATV, while keeping the remaining experimental settings consistent with Table 2.
Within-model Comparison. The results show that adopting the ceiling-aware hierarchical ordinal loss has model-dependent effects. Molmo-2 experiences substantial performance degradation, whereas Qwen2.5-Omni exhibits modest improvements in BAcc and [email protected] but declines in MCC. SurgVidLM benefits from the objective, improving BAcc from 0.335 to 0.352 and [email protected] from 0.416 to 0.426, while V-JEPA 2.1 shows only marginal changes in ordinal scoring performance. These results indicate that the hierarchical ordinal loss does not consistently improve assessment performance across different model architectures.
Cross-model Comparison. Importantly, even after adopting the same scoring objective, all four baselines remain substantially behind ATV in fine-grained ordinal assessment. In particular, SurgVidLM, the strongest baseline under the matched objective, achieves a BAcc of 0.352 and [email protected] of 0.426, compared with 0.410 and 0.512 for ATV. This demonstrates that, although the scoring objective can influence baseline performance, it alone cannot account for ATV's advantage. Together with our internal controlled ablations, these results provide further evidence supporting the contributions of normative dynamics modeling and discrepancy reasoning to fine-grained procedural assessment.
Response to Comment from Reviewer NDDt
Response:
Thank you for recognizing the value of our formulation for step-level procedural assessment and for acknowledging that our component ablations provide convincing evidence of the contributions of bidirectional normative dynamics modeling (BNDM) and discrepancy reasoning within ATV.
(1) Scope of Procedural State Tracking. We appreciate your concern regarding the scope of procedural state tracking. Clinical skill execution consists of multiple rubric-defined steps, each representing a clinically meaningful unit of interaction involving the operator, patient, instruments, and environment. In our work, procedural state tracking refers specifically to modeling how these interactions update object states within an individual step and verifying whether the resulting state transitions conform to normative procedural dynamics. Our contribution therefore lies in within-step state-transition modeling and verification, rather than long-horizon, continuous state tracking across an entire multi-step procedure. Our choice of step-level clips is motivated by three considerations. First, processing full-length videos requires substantial temporal sampling. For example, a two-minute video contains 120 frames at 1 FPS, whereas widely used Video-MLLMs often employ limited frame-sampling budgets (e.g., up to 32 frames in the original LLaVA-OneVision configuration), potentially omitting fine-grained state-transition evidence. Second, the complete clinical assessment rubric comprises 33 steps, of which 29 visually assessable steps are retained in ClinSkill-V. Jointly interpreting multiple scoring criteria introduces additional rubric-comprehension complexity. Third, full-procedure assessment requires models to temporally localize the video segments corresponding to individual rubric items, introducing temporal grounding as an additional confounding factor. Using step-aligned clips allows us to focus more directly on whether models can capture and verify interaction-driven state transitions.
We agree that maintaining procedural states across complete multi-step procedures and reasoning about dependencies between earlier and later steps represent important and broader research directions. We have clarified this distinction in the revised manuscript to more precisely define the scope of our contributions and experimental evidence. Specifically, in the Introduction, we explicitly clarify the scope of our study: "Our study focuses on modeling and verifying interaction-driven state transitions within rubric-defined procedural steps, rather than tracking procedural states over long horizons or reasoning about dependencies across a complete multi-step procedure" (Lines 106–107). In the Conclusion, we further clarify the scope of our experimental evaluation and identify long-horizon procedural state tracking and cross-step dependency reasoning as important directions for future research: "Our current evaluation focuses on modeling and verifying state transitions within rubric-defined procedural steps, rather than examining continuous state tracking or dependencies across complete multi-step procedures. Extending ATV to long-horizon procedural state tracking and cross-step dependency reasoning remains an important direction for future research" (Lines 484–489).