acceptodds
Under review as a conference paper at ICLR 2027

Learning Bidirectional Normative Latent Dynamics with Discrepancy Reasoning for Clinical Skill Assessment

Abstract

Vision-Language Models (VLMs) have become increasingly capable of recognizing events in videos. However, reliable procedural reasoning requires more than identifying what happened: a model must track how interactions update the underlying world state and verify whether the resulting state evolution is valid. In this work, we study this problem through clinical skill assessment, where the correctness of an action depends on the procedural state accumulated through prior interactions among the operator, patient, instruments, and environment. We propose ATV, an Anticipate-Then-Verify framework that formulates procedural assessment as verification against normative latent dynamics. ATV first organizes video observations into temporally aligned object-centric states. It then learns bidirectional dynamics exclusively from correctly performed procedures, where a forward predictor anticipates how object states should evolve and a reverse-time reflector captures whether the resulting states are compatible with their expected antecedents. At inference time, ATV converts the discrepancies between observed and normative state transitions into rubric-conditioned evidence for procedural scoring. We further construct ClinSkill-V, a large-scale dataset of 15,775 step-aligned clinical skill video clips collected from real-world clinical skill examinations, with expert rubric-based ordinal annotations. Extensive experiments against open-source VLMs and latent predictive models demonstrate that ATV consistently improves procedural assessment, with particularly pronounced gains in fine-grained ordinal scoring.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
State Transition Modeling, Video understanding, Vision-language Models, Clinical skill assessment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NDDtOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies clinical skill assessment from video and proposes ATV, an Anticipate-Then-Verify framework that treats assessment as verification against learned normative latent dynamics. ATV first represents videos as temporally aligned object-centric states using recurrent Slot Attention. It then learns forward and reverse latent dynamics from correctly performed procedures, and finally compares observed state transitions with these normative references to produce rubric-conditioned ordinal scores. The discrepancy representation decomposes transition differences into progress, orthogonal deviation, and endpoint mismatch rather than relying only on raw prediction error. The paper also introduces ClinSkill-V, a physician-annotated dataset of clinical skill examinations covering CPR, AED, and bag-mask ventilation. The dataset contains 15,775 step-aligned clips from 544 full-length videos. ATV is compared with general video MLLMs, medical video MLLMs, omni-modal models, and latent predictive models, and reports the strongest overall performance, particularly on fine-grained ordinal scoring.

Critical strengths

The strongest aspect of the paper is the formulation of procedural assessment as verification against normative dynamics rather than direct video-to-score prediction. In particular, the distinction between predicting what is likely to happen and modeling how a correctly executed procedure should evolve is well motivated. The bidirectional design and the structured discrepancy representation also go beyond simply treating prediction error as an anomaly score. I also found the ablation results informative. Removing normative dynamics or discrepancy conditioning clearly degrades the ordinal metrics, while raw prediction errors improve binary MCC but perform worse on fine-grained scoring. This supports the paper's argument that detecting whether an execution is abnormal and estimating the degree of procedural deviation are different problems. The dataset is another strength: it is collected from real clinical skill examinations, uses physician-defined rubrics, and includes dual-expert annotations with adjudication.

Critical weaknesses

My main concern is that the experimental task is narrower than some of the paper's procedural-state and world-model framing suggests. Each sample is already segmented into a rubric-defined procedural step, and the model performs assessment within that step. Therefore, the current experiments do not test whether ATV can maintain procedural state across a complete multi-step procedure or reason about dependencies between earlier and later steps. The proposed formulation is still meaningful for step-level assessment, but the distinction between within-step temporal modeling and broader procedural state tracking should be made clearer. A second concern is that the comparison with external baselines does not fully isolate the contribution of normative dynamics. ATV uses a dedicated ceiling-aware hierarchical ordinal objective that explicitly separates full-score detection, non-full severity estimation, and ordinal distance, whereas the MLLM baselines are trained with their own score-label objectives. The component ablations convincingly support the contribution of BNDM and discrepancy reasoning within ATV, but part of the gap in Table 2 could still come from the scoring objective rather than the proposed dynamics model itself. A matched scoring-head/objective comparison on one or two strong baselines would make the main comparison more conclusive.

Questions for authors

Q1: The current benchmark uses rubric-aligned step clips. Can ATV also model state across multiple consecutive procedural steps, or is the method intended only for within-step assessment? A full-procedure or multi-step experiment would help clarify how strongly the current results support the broader procedural-state tracking claim.

Q2: ATV uses a custom hierarchical ordinal objective, while the MLLM baselines use different scoring objectives. Please provide a comparison in which a strong baseline such as Molmo-2 or Qwen3-VL uses the same ordinal scoring head/objective as ATV.

Q3: How are “normative” training samples selected for BNDM? Please report the number of full-score/correct samples available for each rubric step and clarify whether all steps have sufficient normative examples.

Q4: Correct executions can follow different valid trajectories across operators, speeds, and styles. Please evaluate whether the learned normative dynamics remain tolerant to such variation, for example with operator-held-out evaluation or by training on one subset of correct performers and testing on another.

Q5: OPSR is central to the claim of object-centric state tracking, but its evaluation is mainly qualitative. Please provide a quantitative comparison against a non-object-centric representation, such as pooled/frame-level DINO features or temporally aligned patch features, to show whether the slot-based state representation is necessary.

7 comments
AuthorsOpenReview ↗

Response to Comment from Reviewer NDDt

Comment: My main concern is that the experimental task is narrower than some of the paper's procedural-state and world-model framing suggests. Each sample is already segmented into a rubric-defined procedural step, and the model performs assessment within that step. Therefore, the current experiments do not test whether ATV can maintain procedural state across a complete multi-step procedure or reason about dependencies between earlier and later steps. The proposed formulation is still meaningful for step-level assessment, but the distinction between within-step temporal modeling and broader procedural state tracking should be made clearer. A second concern is that the comparison with external baselines does not fully isolate the contribution of normative dynamics. ATV uses a dedicated ceiling-aware hierarchical ordinal objective that explicitly separates full-score detection, non-full severity estimation, and ordinal distance, whereas the MLLM baselines are trained with their own score-label objectives. The component ablations convincingly support the contribution of BNDM and discrepancy reasoning within ATV, but part of the gap in Table 2 could still come from the scoring objective rather than the proposed dynamics model itself. A matched scoring-head/objective comparison on one or two strong baselines would make the main comparison more conclusive.

Response:

Thank you for recognizing the value of our formulation for step-level procedural assessment and for acknowledging that our component ablations provide convincing evidence of the contributions of bidirectional normative dynamics modeling (BNDM) and discrepancy reasoning within ATV.

(1) Scope of Procedural State Tracking. We appreciate your concern regarding the scope of procedural state tracking. Clinical skill execution consists of multiple rubric-defined steps, each representing a clinically meaningful unit of interaction involving the operator, patient, instruments, and environment. In our work, procedural state tracking refers specifically to modeling how these interactions update object states within an individual step and verifying whether the resulting state transitions conform to normative procedural dynamics. Our contribution therefore lies in within-step state-transition modeling and verification, rather than long-horizon, continuous state tracking across an entire multi-step procedure. Our choice of step-level clips is motivated by three considerations. First, processing full-length videos requires substantial temporal sampling. For example, a two-minute video contains 120 frames at 1 FPS, whereas widely used Video-MLLMs often employ limited frame-sampling budgets (e.g., up to 32 frames in the original LLaVA-OneVision configuration), potentially omitting fine-grained state-transition evidence. Second, the complete clinical assessment rubric comprises 33 steps, of which 29 visually assessable steps are retained in ClinSkill-V. Jointly interpreting multiple scoring criteria introduces additional rubric-comprehension complexity. Third, full-procedure assessment requires models to temporally localize the video segments corresponding to individual rubric items, introducing temporal grounding as an additional confounding factor. Using step-aligned clips allows us to focus more directly on whether models can capture and verify interaction-driven state transitions.

We agree that maintaining procedural states across complete multi-step procedures and reasoning about dependencies between earlier and later steps represent important and broader research directions. We have clarified this distinction in the revised manuscript to more precisely define the scope of our contributions and experimental evidence. Specifically, in the Introduction, we explicitly clarify the scope of our study: "Our study focuses on modeling and verifying interaction-driven state transitions within rubric-defined procedural steps, rather than tracking procedural states over long horizons or reasoning about dependencies across a complete multi-step procedure" (Lines 106–107). In the Conclusion, we further clarify the scope of our experimental evaluation and identify long-horizon procedural state tracking and cross-step dependency reasoning as important directions for future research: "Our current evaluation focuses on modeling and verifying state transitions within rubric-defined procedural steps, rather than examining continuous state tracking or dependencies across complete multi-step procedures. Extending ATV to long-horizon procedural state tracking and cross-step dependency reasoning remains an important direction for future research" (Lines 484–489).

AuthorsOpenReview ↗

Continuation of Response to Reviewer NDDt, Comment 1

(2) Scoring Objective Control

Our internal ablations provide controlled evidence for the effectiveness of ATV's core methodological components. By retaining the same scoring backbone and objective across ablation variants, these experiments demonstrate that BNDM and discrepancy reasoning contribute to fine-grained procedural assessment independently of differences in scoring objectives. Therefore, the effectiveness of these components is supported not only by comparisons with external baselines but also by controlled experiments within ATV.

We appreciate your concern that differences in scoring objectives may affect cross-model comparisons. To further address this potential confounding factor, we select the best-performing model from each of the four baseline categories in Table 2—Molmo-2, Qwen2.5-Omni, SurgVidLM, and V-JEPA 2.1—and conduct additional controlled experiments under a matched scoring objective. We replace the original scoring objective of each selected baseline with the same ceiling-aware hierarchical ordinal objective used by ATV, while keeping the remaining experimental settings consistent with those reported in the main comparison (Table 2). This experiment directly examines whether ATV's performance advantage persists when differences in scoring objectives are controlled. The experimental results are as follows:

ModelScoring ObjectiveNMAE^M ↓BAcc ↑[email protected] ↑BAcc@1 ↑MCC ↑
Molmo-2Original0.4150.3470.4230.6590.309
Molmo-2Hierarchical Ordinal0.5290.2380.3180.477-0.076
Qwen2.5-OmniOriginal0.4220.3260.4000.6530.237
Qwen2.5-OmniHierarchical Ordinal0.4220.3450.4140.6380.077
SurgVidLMOriginal0.4250.3350.4160.6560.217
SurgVidLMHierarchical Ordinal0.4110.3520.4260.6560.279
V-JEPA 2.1Original0.4290.3350.4040.6470.183
V-JEPA 2.1Hierarchical Ordinal0.4290.3310.4050.6450.208
ATV (Ours)Hierarchical Ordinal0.3240.4100.5120.7140.312

The results reveal that the impact of the ceiling-aware hierarchical ordinal loss varies across model architectures. While SurgVidLM benefits from this objective across the reported metrics, Qwen2.5-Omni exhibits modest improvements in ordinal accuracy, V-JEPA 2.1 shows limited changes, and Molmo-2 experiences substantial performance degradation. Importantly, even the strongest baseline under the matched scoring objective, SurgVidLM, remains substantially behind ATV, with BAcc of 0.352 versus 0.410 and [email protected] of 0.426 versus 0.512. These findings indicate that differences in scoring objectives alone cannot account for ATV's performance advantage, providing further evidence for the effectiveness of our proposed methodological design. We have made the corresponding updates in Appendix F.5 (Controlling Scoring Objectives and Prompting Strategies).

AuthorsOpenReview ↗

Response to Questions For Authors from Reviewer NDDt

Q1 The current benchmark uses rubric-aligned step clips. Can ATV also model state across multiple consecutive procedural steps, or is the method intended only for within-step assessment? A full-procedure or multi-step experiment would help clarify how strongly the current results support the broader procedural-state tracking claim.

Response:

Thank you for highlighting the distinction between within-step state modeling and procedural-state tracking across multiple steps. ATV is currently designed and evaluated for rubric-aligned within-step procedural assessment, rather than end-to-end tracking across an entire clinical procedure.

Importantly, a rubric-defined step is not necessarily a single atomic action. It often involves multiple temporally dependent interactions among the operator, patient, and instruments, requiring the assessment of how their states evolve throughout execution. Our objective is therefore to move beyond action recognition or likely-future prediction toward verifying whether observed state transitions conform to normative procedural dynamics. Rubric-aligned segmentation allows us to investigate this question while separating normative dynamics modeling from additional challenges such as procedural step localization and cross-step rubric alignment.

Within this scope, OPSR represents evolving object-centric states, BNDM models normative temporal dynamics, and DRPV evaluates discrepancies between observed and normative state transitions. The current results demonstrate the effectiveness of this framework for within-step procedural verification. However, they do not establish that ATV can maintain state continuity or reason about dependencies across multiple consecutive procedural steps. Extending ATV to multi-step or full-procedure assessment would require addressing additional challenges, including long-video temporal sampling, joint interpretation of multiple rubric criteria, and temporal localization of individual procedural steps. Our use of rubric-aligned clips allows us to focus on modeling and verifying interaction-driven state transitions without introducing these additional complexities. We have clarified this distinction in the revised manuscript to more precisely define the scope of our contributions and experimental evidence.

Q2 ATV uses a custom hierarchical ordinal objective, while the MLLM baselines use different scoring objectives. Please provide a comparison in which a strong baseline such as Molmo-2 or Qwen3-VL uses the same ordinal scoring head/objective as ATV.

Response:

We have conducted additional controlled experiments by replacing the original scoring objectives of four representative baselines—Molmo-2, Qwen2.5-Omni, SurgVidLM, and V-JEPA 2.1—with the same ceiling-aware hierarchical ordinal loss used by ATV, while keeping the remaining experimental settings consistent with Table 2.

Within-model Comparison. The results show that adopting the ceiling-aware hierarchical ordinal loss has model-dependent effects. Molmo-2 experiences substantial performance degradation, whereas Qwen2.5-Omni exhibits modest improvements in BAcc and [email protected] but declines in MCC. SurgVidLM benefits from the objective, improving BAcc from 0.335 to 0.352 and [email protected] from 0.416 to 0.426, while V-JEPA 2.1 shows only marginal changes in ordinal scoring performance. These results indicate that the hierarchical ordinal loss does not consistently improve assessment performance across different model architectures.

Cross-model Comparison. Importantly, even after adopting the same scoring objective, all four baselines remain substantially behind ATV in fine-grained ordinal assessment. In particular, SurgVidLM, the strongest baseline under the matched objective, achieves a BAcc of 0.352 and [email protected] of 0.426, compared with 0.410 and 0.512 for ATV. This demonstrates that, although the scoring objective can influence baseline performance, it alone cannot account for ATV's advantage. Together with our internal controlled ablations, these results provide further evidence supporting the contributions of normative dynamics modeling and discrepancy reasoning to fine-grained procedural assessment.

Reviewer_1N2bOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ATV, which learns forward and reverse object-state dynamics from correctly performed procedures and uses deviations from these dynamics for rubric-based scoring. It also introduces ClinSkill-V, containing 15,775 step-aligned video clips with expert scores. Experiments show improved ordinal scoring across three skills

Critical strengths

A useful approach to procedural assessment. The method connects object-state prediction with rubric-based scoring. Comparing observed transitions with learned expectations provides an explicit reference for assessing execution quality.

Critical weaknesses

Limited evidence for state-transition reasoning. The paper presents state-transition tracking as its main distinction from direct video scoring. The experiments show improved scores and useful model components, but do not isolate the contribution of temporal order and state dependencies. An order-invariant baseline using the same frame encoder and scoring objective is missing.

Questions for authors

Does ATV retain its advantage on rubric items that specifically depend on action order? Maybe you can compare it with an order-invariant baseline using the same frame encoder, training data, and scoring objective. This would help establish whether the gains come from tracking state transitions.

4 comments
AuthorsOpenReview ↗

Response to Comment from Reviewer 1N2b

Comment: Limited evidence for state-transition reasoning. The paper presents state-transition tracking as its main distinction from direct video scoring. The experiments show improved scores and useful model components, but do not isolate the contribution of temporal order and state dependencies. An order-invariant baseline using the same frame encoder and scoring objective is missing.

Response:

Thank you for recognizing the significance of our research problem, which shifts clinical skill assessment from direct video scoring to tracking interaction-driven state transitions and verifying their procedural correctness. We also appreciate your acknowledgment of the effectiveness of our proposed Anticipate-Then-Verify (ATV) framework, which learns bidirectional normative latent dynamics and performs discrepancy-based verification for fine-grained procedural assessment.

Specifically, ATV explicitly models temporal state evolution through three complementary components. First, OPSR constructs temporally aligned object-centric representations to track how object states evolve across frames. Second, BNDM learns bidirectional normative dynamics by predicting future states from preceding states and reconstructing plausible antecedents from subsequent states, thereby modeling temporal dependencies in procedural execution. Third, DRPV compares observed state transitions with their normative references and characterizes directional deviations as evidence for rubric-conditioned assessment. Together, these components are designed to move beyond recognizing visual content toward modeling and verifying how procedural states evolve over time. Our existing ablation studies demonstrate the effectiveness of normative dynamics modeling and discrepancy reasoning. Specifically, removing the normative dynamics branches or discrepancy conditioning leads to substantial performance degradation, while removing the reverse-time branch also reduces fine-grained scoring accuracy.

We agree that a more direct mechanistic attribution would provide a clearer understanding of whether the observed performance gains arise specifically from modeling temporal order and state dependencies. While our existing ablation studies demonstrate the effectiveness of normative dynamics modeling and discrepancy reasoning, the suggested order-invariant baseline would provide a more controlled evaluation of the contribution of temporal ordering to procedural assessment, beyond the information available from visual representations alone. In particular, evaluating rubric items that explicitly depend on action order would help clarify the contribution of temporal state-transition modeling to procedural assessment.

To address this concern, we introduce an order-invariant baseline using the same frozen DINOv2 frame encoder, training data, language backbone, and hierarchical ordinal scoring objective as ATV. Unlike ATV, which explicitly models temporally ordered object-state transitions, this baseline independently extracts frame-level visual features and aggregates them through permutation-invariant pooling, preventing the scorer from accessing temporal ordering information. The results of the controlled comparison are presented below:

ModelNMAE^M ↓BAcc ↑[email protected] ↑BAcc@1 ↑MCC ↑
Order-invariant Baseline0.3420.3730.4580.6930.350
ATV (Ours)0.3240.4100.5120.7140.312

ATV outperforms the order-invariant baseline across all four ordinal assessment metrics, reducing NMAE^M by 5.3% and improving BAcc, [email protected], and BAcc@1 by 3.7, 5.4, and 2.1 percentage points, respectively. The largest improvement occurs in [email protected], indicating stronger fine-grained agreement with expert-assigned scores. In contrast, the order-invariant baseline achieves a higher MCC (0.350 vs. 0.312), indicating that unordered visual representations remain effective for distinguishing full-score from non-full-score executions, despite their weaker ordinal assessment performance.

This pattern is consistent with our existing ablations, where removing reverse-time modeling or replacing structured discrepancies with raw prediction errors similarly improves MCC while degrading ordinal assessment. Together, these findings suggest that visual evidence sufficient for identifying procedural errors does not necessarily capture the degree of deviation required for fine-grained scoring. More importantly, ATV's consistent advantage across ordinal metrics provides further evidence that explicitly modeling temporally ordered state evolution contributes to procedural assessment beyond aggregating unordered visual information. We have made the corresponding updates in Appendix F.8 (Temporal Order and State-Transition Reasoning).

AuthorsOpenReview ↗

Response to Questions For Authors from Reviewer 1N2b

Q1: Does ATV retain its advantage on rubric items that specifically depend on action order? Maybe you can compare it with an order-invariant baseline using the same frame encoder, training data, and scoring objective. This would help establish whether the gains come from tracking state transitions.

Response:

ATV retains an advantage in normalized scoring error on the identified action-order-related rubric items. We identify five rubric items involving ordered actions based on their procedural descriptions, before examining model predictions. These items cover the following action-order requirements:

Rubric ItemAction-Order Requirement
AED:step26       Resume CPR immediately after defibrillation.
AED:step28       Power on AED → attach and connect electrode pads → rhythm analysis → defibrillation → resume CPR.
BMV:step32       Squeeze the bag to deliver ventilation → observe chest rise → release the bag to allow exhalation.
CPR:step13       Complete 30 chest compressions before initiating rescue breaths.
CPR:step17       Inhale → deliver ventilation and observe chest rise → release the patient's nose after ventilation.

On these five items, ATV achieves lower average NMAEᴹ than the order-invariant baseline (0.294 vs. 0.307). In particular, on AED, which explicitly specifies the required sequence of AED operations, ATV reduces NMAEᴹ from 0.367 to 0.338.

We introduce an order-invariant baseline using the same frozen DINOv2 frame encoder, training data, language backbone, and hierarchical ordinal scoring objective as ATV. Unlike ATV, which explicitly models temporally ordered object-state transitions, this baseline independently extracts frame-level features and aggregates them through permutation-invariant pooling, preventing access to temporal ordering information.

ATV outperforms the order-invariant baseline across all four ordinal assessment metrics, reducing NMAEᴹ by 5.3% and improving BAcc, [email protected], and BAcc@1 by 3.7, 5.4, and 2.1 percentage points, respectively. The largest improvement in [email protected] highlights ATV's stronger fine-grained agreement with expert scores. Conversely, the order-invariant baseline achieves a higher MCC (0.350 vs. 0.312), suggesting that unordered visual information remains useful for distinguishing correct from incorrect executions but is less effective for assessing the severity of procedural deviations. These findings are consistent with our existing ablations and provide further evidence that modeling temporally structured state evolution benefits fine-grained procedural assessment beyond aggregating unordered visual features.

Reviewer_1N2bOpenReview ↗

Response to author

Thank you for the additional experiments. The authors have indeed addressed my main concern by introducing an order-invariant baseline and evaluating action-order-related rubric items. However, the observed improvements appear relatively modest, leaving some uncertainty about the extent to which state-transition reasoning contributes to the overall performance. Overall, I appreciate the authors' efforts and will maintain my original rating of 3 (Weak Accept).

AuthorsOpenReview ↗

Acknowledgment to Reviewer 1N2b

Thank you for acknowledging that our additional experiments have addressed your main concern. Temporal order is one aspect of state-transition modeling, and our additional experiments provide evidence for the value of temporally structured modeling within our normative verification framework. We greatly appreciate your time and dedication to providing us with your valuable feedback.

Reviewer_bMAkOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ATV, a framework for clinical skill assessment that learns object-centric video representations and bidirectional dynamics from correctly performed examples. It uses discrepancies between observed and predicted states to produce rubric-based scores. The paper also introduces ClinSkill-V, a dataset of step-level clinical examination videos with physician annotations. I have the following comments.

Critical strengths

1、ClinSkill-V is a useful contribution to procedural skill assessment. It contains 15,775 step-aligned clips from 544 examination recordings, with rubric-based scores independently assigned by two physicians and disagreements resolved through adjudication. The full-recording split and exclusion of speech-dependent steps are sensible design choices. These annotations support studying differences in execution quality beyond recognizing which action occurs.

Critical weaknesses

1、The baseline comparisons leave an important scoring confound unresolved. ATV uses a ceiling-aware hierarchical ordinal loss (Appendix E.3), whereas MLLM baselines use label-token prediction and latent predictive baselines use a different scoring loss (Appendix G). The baseline prompt also explicitly directs MLLMs away from endpoint scores when evidence is uncertain (Figure 9). These choices can affect score distributions and ordinal accuracy. Although the internal ablations retain the same objective, the paper does not establish how much of the advantage over external baselines remains under comparable scoring objectives and a neutral rubric prompt.

2、The central claim about learning normative dynamics is not directly tested. The “w/o Norm.” ablation removes both dynamics branches, changing the architecture and available evidence. It does not compare the same dynamics model trained on correct examples alone against one trained on both correct and incorrect examples. Likewise, the rubric-conditioning experiments in Appendix F.2 address a different question. The results support the usefulness of the proposed components, but do not establish that restricting dynamics learning to correct executions is what makes them effective. This distinction matters because it underpins the paper’s argument that generic predictive modeling is insufficient.

Questions for authors

1、How do the strongest baselines perform with a neutral rubric prompt? Can you also compare a baseline using the same hierarchical ordinal scoring objective as ATV?

2、Does training BNDM only on correct executions outperform training it on a size-matched mixture of correct and incorrect executions, with the architecture and downstream training held fixed?

3、Does the full-recording split also keep trainees disjoint across training and testing? If trainees overlap, is a result on a trainee-disjoint split available?

8 comments
AuthorsOpenReview ↗

Response to Comment 1 from Reviewer bMAk

Comment 1: The baseline comparisons leave an important scoring confound unresolved. ATV uses a ceiling-aware hierarchical ordinal loss (Appendix E.3), whereas MLLM baselines use label-token prediction and latent predictive baselines use a different scoring loss (Appendix G). The baseline prompt also explicitly directs MLLMs away from endpoint scores when evidence is uncertain (Figure 9). These choices can affect score distributions and ordinal accuracy. Although the internal ablations retain the same objective, the paper does not establish how much of the advantage over external baselines remains under comparable scoring objectives and a neutral rubric prompt.

Response:

Thank you for recognizing the significance of our motivation to move beyond conventional action recognition and future prediction toward verifying whether observed state transitions conform to normative procedural dynamics. We also appreciate your positive assessment of our model design and acknowledgment of the controlled experimental settings in our ablation studies. Our internal ablations retain the same scoring objective and demonstrate the effectiveness of ATV's core design: learning normative dynamics and explicitly reasoning about observed–normative discrepancies improve fine-grained procedural assessment. Importantly, this finding does not rely solely on comparisons with external baselines. By controlling for the scoring objective, our internal ablations isolate the contributions of these methodological components from loss-related confounding factors, providing independent empirical evidence for the effectiveness of our proposed approach.

The original scoring prompt was designed to encourage evidence-grounded clinical assessment and discourage unsupported endpoint predictions. Specifically, the Mandatory Endpoint Veto Rule requires direct visual evidence before assigning the highest or lowest score, while the Global Anti-Collapse Rule discourages using either endpoint as a default across cases and emphasizes independent assessment based on each video's evidence (Figure 9).

We appreciate your concern regarding the comparability of scoring strategies across different models. We agree that differences in scoring objectives and prompting strategies may influence predicted score distributions and therefore warrant further examination. To address these potential confounding factors and ensure fair cross-model comparisons, we select the best-performing model from each baseline category in the main experiments and conduct controlled experiments on scoring objectives and prompting strategies.

(1) Scoring Objective Control. We replace the original scoring objective of each selected baseline with ATV's ceiling-aware hierarchical ordinal loss, while keeping the remaining experimental settings consistent with those reported in the main comparison (Table 2). This experiment examines whether ATV's performance advantage can be attributed to differences in scoring objectives. The experimental results are as follows:

ModelScoring ObjectiveNMAE^M ↓BAcc ↑[email protected] ↑BAcc@1 ↑MCC ↑
Molmo-2Original0.4150.3470.4230.6590.309
Molmo-2Hierarchical Ordinal0.5290.2380.3180.477-0.076
Qwen2.5-OmniOriginal0.4220.3260.4000.6530.237
Qwen2.5-OmniHierarchical Ordinal0.4220.3450.4140.6380.077
SurgVidLMOriginal0.4250.3350.4160.6560.217
SurgVidLMHierarchical Ordinal0.4110.3520.4260.6560.279
V-JEPA 2.1Original0.4290.3350.4040.6470.183
V-JEPA 2.1Hierarchical Ordinal0.4290.3310.4050.6450.208
ATV (Ours)Hierarchical Ordinal0.3240.4100.5120.7140.312

The results reveal that the impact of the ceiling-aware hierarchical ordinal loss varies across model architectures. While SurgVidLM benefits from this objective across the reported metrics, Qwen2.5-Omni exhibits modest improvements in ordinal accuracy, V-JEPA 2.1 shows limited changes, and Molmo-2 experiences substantial performance degradation. Importantly, even the strongest baseline under the matched scoring objective, SurgVidLM, remains substantially behind ATV, with BAcc of 0.352 versus 0.410 and [email protected] of 0.426 versus 0.512. These findings indicate that differences in scoring objectives alone cannot account for ATV's performance advantage, providing further evidence for the effectiveness of our proposed methodological design.

AuthorsOpenReview ↗

Continuation of Response to Reviewer bMAk, Comment 1

(2) Prompt Control. For representative MLLM baselines, we replace the original scoring prompt with a neutral rubric prompt. Specifically, we remove (i) the Endpoint Veto Rule, which discourages assigning endpoint scores when visual evidence is uncertain, and (ii) the Global Anti-Collapse Rule, which introduces additional constraints on score predictions (Figure 9). We also remove the corresponding restrictions repeated in the Evaluation Procedure. The neutral prompt allows models to assign any valid score based solely on the rubric criteria and observed video evidence, without additional endpoint-related constraints. We retain the original rubric text, allowed score labels, scoring task, output format, and scoring objective, with all other experimental settings unchanged. The neutral prompt is used consistently during training and inference. This experiment evaluates whether the original prompting constraints influence baseline performance and the observed performance gap. The results are as follows:

ModelPromptNMAE^M ↓BAcc ↑[email protected] ↑BAcc@1 ↑MCC ↑
Molmo-2Original0.4150.3470.4230.6590.309
Molmo-2Neutral0.4290.3360.4070.6500.242
Qwen2.5-OmniOriginal0.4220.3260.4000.6530.237
Qwen2.5-OmniNeutral0.4370.3260.3950.6380.000
SurgVidLMOriginal0.4250.3350.4160.6560.217
SurgVidLMNeutral0.4270.3360.4140.6520.245
ATV (Ours)—0.3240.4100.5120.7140.312

The within-model comparisons show that removing the endpoint-related prompting constraints does not consistently improve baseline performance. Molmo-2 exhibits degradation across all five metrics, while Qwen2.5-Omni shows lower ordinal assessment performance and a substantial decrease in MCC (0.237 to 0.000). In contrast, SurgVidLM maintains comparable ordinal assessment performance with a modest improvement in MCC (0.217 to 0.245). Importantly, the cross-model comparisons show that ATV continues to outperform all three MLLM baselines across all five metrics under the neutral prompt setting. Even SurgVidLM, which benefits from the neutral prompt in MCC, remains behind ATV in BAcc (0.336 vs. 0.410) and [email protected] (0.414 vs. 0.512). These findings demonstrate that ATV's performance advantage persists even after removing the original prompting constraints.

Overall, these controlled experiments demonstrate that ATV consistently maintains its performance advantage over representative baselines under matched scoring objectives and neutral prompting conditions, respectively. Together with our internal ablations conducted under the same scoring objective, these findings demonstrate that ATV maintains a substantial performance advantage when differences in scoring objectives and prompting strategies are controlled separately, providing further evidence for the effectiveness of normative dynamics modeling and discrepancy-based verification in fine-grained clinical procedural assessment. We have made the corresponding updates in Appendix F.5 (Controlling Scoring Objectives and Prompting Strategies).

AuthorsOpenReview ↗

Response to Comment 2 from Reviewer bMAk

Comment 2: The central claim about learning normative dynamics is not directly tested. The “w/o Norm.” ablation removes both dynamics branches, changing the architecture and available evidence. It does not compare the same dynamics model trained on correct examples alone against one trained on both correct and incorrect examples. Likewise, the rubric-conditioning experiments in Appendix F.2 address a different question. The results support the usefulness of the proposed components, but do not establish that restricting dynamics learning to correct executions is what makes them effective. This distinction matters because it underpins the paper’s argument that generic predictive modeling is insufficient.

Response:

Thank you for recognizing the effectiveness of Bidirectional Normative Dynamics Modeling (BNDM) and Discrepancy Reasoning for Procedural Verification (DRPV), as they enable a shift from recognizing observed actions or predicting likely futures to verifying whether observed state transitions conform to normative procedural dynamics. BNDM learns bidirectional dynamics exclusively from correct executions, establishing normative references through forward anticipation and reverse-time reflection. DRPV then characterizes discrepancies between observed and normative state transitions and interprets them under clinical scoring rubrics to enable fine-grained procedural assessment.

Our design of BNDM is inspired by physicians' assessment process, in which clinicians can anticipate both correct and incorrect procedural outcomes based on their clinical knowledge and judge whether observed executions satisfy procedural requirements. However, BNDM does not inherently possess explicit knowledge of what distinguishes normative from non-normative state transitions. We therefore train BNDM exclusively on correctly performed procedures to establish normative references, which DRPV subsequently uses to characterize observed deviations for rubric-conditioned assessment.

Importantly, generic predictive modeling and normative dynamics modeling share a similar underlying principle: learning state evolution from examples that satisfy a particular notion of validity. Generic predictive models typically learn from real-world observations governed by physical laws, whereas our BNDM learns from executions that satisfy clinical procedural requirements. In this sense, both learn from valid state transitions, but the notion of validity differs: physical plausibility versus procedural correctness. Our correct-only training strategy explicitly selects procedurally correct executions to establish normative references. Beyond learning these dynamics, ATV further employs discrepancy reasoning to verify whether observed state transitions conform to the learned normative references, rather than merely predicting plausible future states.

We agree that a more direct mechanistic attribution would provide a clearer explanation of the observed performance gains, particularly by disentangling the contribution of learning dynamics exclusively from correct executions from that of the dynamics modeling architecture itself. To this end, we introduce a Mixed Training variant that trains BNDM on both correct and incorrect executions, while retaining the same architecture, training objectives, and optimization protocol as the original ATV. The downstream DRPV is independently retrained under the original scoring protocol, with all other settings unchanged. By comparing this variant against the original ATV, we directly isolate the contribution of correct-only dynamics learning to procedural assessment performance. Specifically, we evaluate three proportions of non-full-score executions in BNDM training: 10.0%, 18%, and 27%(including all available non-full-score executions), while keeping the total number of training samples identical across all settings. The results are as follows:

Non-full-score shareNMAE^M ↓BAcc ↑[email protected] ↑BAcc@1 ↑MCC ↑
0% (Main Experiment)0.3240.4100.5120.7140.312
10.00%0.3420.3960.4860.7130.293
18.00%0.3450.4210.5060.7000.298
27.00%0.3410.3830.4830.6980.328

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.