acceptodds
Under review as a conference paper at ICLR 2027

Events, Not Frames: Detecting Physical Violations from a Single Video

Abstract

Infants look longer when a box hangs in mid-air instead of falling; detecting such violations of physical expectation is a hallmark of intuitive physics and a long-standing goal for machines. Current methods often use a matched plausible clip to interpret a prediction error, or annotated violations to train an evaluator. Infants need neither to judge a single event from ordinary experience alone. Here we show that event structure may help close this gap by separating the physical expectations that frame-level prediction errors conflate. To this end, our model, Physics Using Latent State-Events (PULSE), parses tracked 3D object states into events and asks, stage by stage, whether an object should be visible (Visibility), whether the interaction begins as it should (Onset), and whether the state it leads to is consistent (Outcome). Each of these checks asks the same question of every video, and plausible videos alone define the normal range against which an unmatched clip is judged. On three benchmarks, including PhysEvent, our new benchmark with stage-specific violations and ground-truth 3D states, PULSE achieves the strongest overall single-video detection and paired ranking among the evaluated continuous scorers. Each decision also names the event, check, and objects behind it. More broadly, this work points toward machines that turn experience with ordinary events into an understanding of what can and cannot happen.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Intuitive Physics, Physical Reasoning, Video Understanding, Violation-of-Expectation Detection, Event-Centric Representations, Object-Centric Video Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_5d26OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper detects physical violations in a single, unpaired video, using only plausible videos for training, model selection, and calibration. Proposed model (PULSE) parses tracked 3D object boxes into events and applies three checks for each event (i.e., Visibility, Onset, Outcome) through four heads; each head's maximum error is normalized by its certain range on plausible validation videos and a video is flagged when any head exceeds its certain percentage, and the maximizing target names the event, check, and objects. The paper also introduces a new benchmark named as PhysEvent which consists of 10,800 Isaac Sim videos, six stage-targeted violation types, native 3D states). Across three datasets (including PhysEvent), PULSE reaches state of the art performance on all three benchmarks with three different metrics (i.e., Direct, AUC, Compare)

Critical strengths
  1. Novel objective with strong gain of performance Plausible-only standalone decisions seems to be a useful target. PULSE raises Direct from 0.519 to 0.665 on all datasets and from 0.516 to 0.580 on the public ones, with smal variance (Table A2), and the shared-state baselines, threshold sweeps (Table A10), and adapter peak pooling (Table A15) rule out several simple explanations.

  2. Diagnostic benchmark, inspectable decisions. PhysEvent's stage-targeted interventions, matched pairs, event-non-occurrence controls, and native 3D states enable controlled analysis. Without violation-type supervision, the dominant check matches the targeted stage in five of six categories (Figure 6b), and the readout identifies the responsible object pair (Figure 7).

Critical weaknesses
  1. Isolation of event alignment Beyond parsing, PULSE differs from the shared-state "-b" baselines in pair-relative states, a categorical objective, object-centered features, piecewise-linear fits, multiple heads, and per-head referencing. The check ablations (Figure 6a, Table A5) remove checks after training and Table A8 keeps parsing, so nothing shows that parsed boundaries are necessary; the evidence supports the full pipeline more than events, not frames

  2. Limited reliability on standalone The Public Direct/AUC are 0.580/0.624, versus 0.821/0.893 on PhysEvent, and the false-positive rate is 41.3% on average and 75.0% on IntPhys (Table A4). With four heads each thresholded at 90th percentiles, a union bound allows an averaged validation FPR of up to about 40%, and the realized test rate is at that level. The paper attributes the IntPhys rate to shared thresholds but does not separate this from a shift between the plausible references (held-out IntPhys training videos, PhysicalConcepts freeform videos) and the test videos.

  3. The JEPA baselines may be under-configured. V-JEPA and V-JEPA-2 average per-window surprise (Section F.1), whereas Garrido et al. (2025) favor maximum surprise for single videos and find it markedly better for V-JEPA-H on IntPhys (refer Table S5 in Garrido et al. (2025)). Table A15 covers only the PLATO/SlotFormer adapters. The released non-RoPE ViT-H used here also appears to differ from their RoPE-pretrained models (App. A.3) and reaches 0.778 IntPhys Compare (0.817 in Table A16), versus their ~98% on the same dev set. Both bear on the claim that strong paired-ranking baselines stay near chance standalone.

  4. Two supporting analyses could be overreaching. Table A9 compares aggregation rules at a fixed threshold of 1, although every alternative scores no higher than the maximum, so its Direct column mixes in operating-point effects. Table A7 uses one seed per condition and is strongly non-monotonic (Direct 0.538 at 7.5% box noise vs. 0.731 at 10%), too noisy to show that predicted boxes retain most native-state performance.

Questions for authors
  1. Fixed windows: If parsed boundaries are replaced by fixed windows of matched duration (or random boundaries matched in number), with everything else unchanged, how do all-dataset and public Direct/AUC change after retraining?
  2. Video-level calibration: If the final score is calibrated to 10% and 20% validation FPR, what are the per-dataset TPR, validation FPR, and test FPR?
  3. JEPA baselines: What Direct/AUC do V-JEPA and V-JEPA-2 reach with maximum-surprise pooling under the same calibration, and with the checkpoint evaluated by Garrido et al. if it is available?
  4. Supporting analyses: Does Table A9's ordering hold when each rule is calibrated to the same video-level FPR, and how much do Table A7's conditions vary across seeds?
5 comments
AuthorsOpenReview ↗

Summary and Event window and Standalone reliability (Part 1/2)

Summary

Dear Reviewer 5d26,

Thank you for the precise technical questions. The additional controls below preserve the main conclusions.

(W1; Q1) Adaptive boundaries improve detection within the event-based framework. With the same representations, network, objectives and calibration procedure, seed-42 fixed-window retraining reduces Direct/AUC from 0.680/0.696 to 0.608/0.630 on All and from 0.583/0.625 to 0.557/0.583 on Public.

(W2; Q2) PULSE's standalone advantage persists at matched validation video-FPR targets. At the 10% and 20% pooled validation targets, PULSE reaches All Direct of 0.612 and 0.669, respectively, versus 0.517 and 0.521 for the best baselines.

(W3; Q3) Recovering JEPA's paired performance does not yield strong standalone detection. Restoring original IntPhys frames raises V-JEPA Compare from 0.817 to 0.917, matching the reported level of the public checkpoint. Across both JEPA models and mean/max pooling, All Oracle Direct remains 0.514–0.536 even with test-label threshold selection.

(W4; Q4) Max retains its advantage at matched validation FPR, and predicted-box performance stays close to native states across both seeds. At the 10%/20% pooled validation targets, Max Direct is 0.612/0.669 versus 0.607/0.642 for the best alternative aggregation. Predicted boxes reduce Direct by only 3.8 and 2.9 percentage points at seeds 42 and 2026, respectively, compared with 19.6 and 22.2 points for predicted depth.

1. Adaptive boundary selection (W1; Q1)

The event perspective concerns how physical expectations are organized and checked; adaptive boundary selection is one implementation choice within it. We retrained with fixed windows to examine the benefit of alignment.

Window spans and Middle positions were fixed using plausible training/validation statistics, with spans chosen to approximate the adaptive parser's mean event count per video; no test-performance tuning was used. Fitted trajectories, state representation, network, objectives and calibration procedure were retained. Both variants use seed 42. Public recalibrates detection references using only its two datasets. The window settings are specified in revised Appendix D.2.

ProtocolModelDirectAUCCompare
AllAdaptive PULSE (Table A8 (original) / Table A11 (revised))0.6800.6960.735
AllFixed-window PULSE0.6080.6300.715
PublicAdaptive PULSE0.5830.6250.634
PublicFixed-window PULSE0.5570.5830.633

Both variants retain event-stage checks. Lower Direct and AUC under All and Public support the additional contribution of adaptive boundaries, while the fixed-window model's detection ability remains consistent with the broader event-based formulation.

2. Standalone reliability and IntPhys false positives (W2; Q2)

Thank you for the question about standalone reliability. Figure 4 and Table A2 (original and revised) show clear standalone detection gains over the evaluated continuous baselines. For the matched-FPR comparison, we adjust the four heads' common upper percentile to meet the pooled validation video-FPR targets (revised Appendix D.4):

Validation FPR targetPULSE All DirectBest baseline: All Direct
10%0.6120.517
20%0.6690.521

PULSE remains ahead at both targets, supporting the reliability of the standalone detection gains across operating points.

The requested per-dataset validation and test rates are:

Validation FPR targetDatasetActual validation FPRTest FPRTest TPRDirect
10%IntPhys17.77%63.33%79.44%0.581
10%PhysicalConcepts3.38%10.64%23.80%0.566
10%PhysEvent8.83%5.67%43.83%0.691
10%All9.993%26.55%49.03%0.612
20%IntPhys34.30%70.00%87.78%0.589
20%PhysicalConcepts10.34%19.92%39.20%0.596
20%PhysEvent15.33%12.83%77.00%0.821
20%All19.991%34.25%67.99%0.669

As you helpfully note, shared calibration alone does not explain IntPhys's high test FPR. At the 10% pooled validation target, IntPhys's own validation FPR is 17.77%; this dataset-level difference arises under shared calibration. Its further increase to 63.33% on test reflects a validation-to-test distribution difference. IntPhys test videos lack usable GT tracks and require SAM2 identity propagation (Table A11 (original) / Table A18 (revised); Appendix E.1). The resulting tracking and state-quality difference can raise prediction errors on plausible videos and may contribute to this additional gap. Revised Appendix D.4 distinguishes these effects and reports the per-dataset validation and test results. The matched-validation-FPR comparisons continue to support PULSE's standalone detection gains.

AuthorsOpenReview ↗

JEPA and Supporting analyses (Part 2/2)

3. JEPA configuration and reproduction (W3; Q3)

Our original experiments already used V-JEPA's official code and public checkpoint. The reproduction gap came from the common video compression applied across datasets. Compression-quality measurements show small average pixel differences (RGB MAE 3.4569 on the 0–255 scale; PSNR 35.77 dB) and high structural similarity (luminance SSIM 0.9655; revised Appendix F.2). Under the same evaluation protocol as the original appendix, restoring uncompressed frames raises IntPhys Compare from 0.817 to 0.917, consistent with the authors' approximately 0.920 for that checkpoint (official Issue #2). The approximately 0.980 result uses a different RoPE checkpoint (Garrido et al., Appendix A.3).

We further compared mean and maximum surprise for both JEPA models, using original uncompressed frames for IntPhys and PhysicalConcepts and retaining the compressed videos for PhysEvent.

We additionally report Oracle Direct, the highest balanced accuracy over all thresholds on the same fixed test scores, selected using test labels. It provides an optimistic thresholding bound on this test set and is reported separately from plausible-validation-calibrated Direct. For each model and pooling rule, All uses one common threshold across the three datasets, maximizing equal-dataset-weighted mean Direct; IntPhys independently optimizes one threshold for that dataset:

Model, 16 framesDatasetMean AUCMean CompareMean Oracle DirectMax AUCMax CompareMax Oracle Direct
V-JEPAAll0.5150.7200.5180.5230.6510.536
V-JEPAIntPhys0.5490.9170.5530.5740.7890.592
V-JEPA-2All0.5110.6190.5140.5070.5770.516
V-JEPA-2IntPhys0.5150.7060.5280.5330.6890.542

Our target is standalone detection across diverse scenes and violation types with a shared decision rule. The All protocol directly evaluates this setting: across both JEPA models and pooling rules, Oracle Direct remains 0.514–0.536 even when the common threshold is optimized using test labels. Thus, threshold selection alone does not resolve the limitations of these scores for the proposed task. Together with the AUC/Compare results, this supports our conclusion for the evaluated public checkpoints that strong paired ranking does not ensure reliable standalone detection across heterogeneous physical events.

4. Aggregation and state-quality repetitions (W4; Q4)

Max retains the strongest local violation evidence. For each aggregation rule, we adjust the four heads' common upper percentile separately to meet the same pooled validation video-FPR target (revised Appendix D.4):

AggregationDirect at 10% validation targetDirect at 20% validation target
Max0.6120.669
Top-2 mean0.6070.634
Softmax-weighted0.6070.642
Mean0.5700.593
Log-mean-exp0.5990.625

Max remains best at both targets, so its advantage is not just an effect of the original shared threshold.

We also completed the seed-2026 repeat of Table A7 (original) / Table A9 (revised):

PhysEvent conditionDirect, seed 42Direct, seed 2026AUC, seed 42AUC, seed 2026Compare, seed 42Compare, seed 2026
Native states0.7700.73500.8430.84860.8980.9067
1% box noise0.6700.67500.7840.80550.7720.8600
2.5% box noise0.7820.72000.8570.80440.9120.8600
5% box noise0.6550.69500.7520.80470.7700.8217
7.5% box noise0.5380.54750.6380.69040.6720.7617
10% box noise0.7310.60330.7860.70160.8220.7067
15% box noise0.6350.51830.7090.62760.7530.6367
Predicted boxes0.7320.70580.8450.80960.8700.8550
Predicted depth0.5740.51330.6820.63700.6920.6517

The overall conclusions are unchanged. Predicted-box Direct stays close to native-state performance in both seeds, with drops of approximately 3.8 and 2.9 percentage points; predicted depth produces larger drops in both. Stronger noise also generally reduces performance. Differences in the ordering of individual noise conditions do not alter these conclusions: Table A7 (original) / Table A9 (revised) characterizes full-pipeline sensitivity, not a strictly monotonic noise–performance law (Appendix D.1). Both seeds are reported in revised Appendix D.1.

AuthorsOpenReview ↗

W1 — Representation and prediction targets

To follow up on W1, we tested the relative-state representation and prediction targets within PLATO and SlotFormer (Table A26). For each backbone, the box control and the adapted model receive the same object-level DINO descriptors and visual context. The adaptation retains the temporal backbone and predicts object occlusion and discretized relative states with two separately calibrated heads.

Model and readoutAll DirectAll AUCPublic DirectPublic AUC
PLATO-b, box prediction0.4980.5330.4980.526
PLATO-relative, mean0.5480.5910.5130.540
PLATO-relative, max0.5730.5940.5160.540
SlotFormer-b, box prediction0.5140.5230.5080.540
SlotFormer-relative, mean0.5640.6370.5270.560
SlotFormer-relative, max0.5880.6440.5200.558
Full PULSE0.6800.6960.5830.625

The adapted models improve both Direct and AUC on each backbone, and full PULSE scores higher on both metrics. This comparison tests the change in representation, targets, and calibration together. The fixed-window retraining in our earlier reply tested boundary placement within PULSE.

Reviewer_56sbOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PULSE for detecting physical violations from a single video. Instead of using frame-level errors, it parses 3D object trajectories into events and checks Visibility, Onset, and Outcome separately. The model is trained on plausible videos only and evaluated on three benchmarks.

Critical strengths
  • PULSE achieves 0.665 standalone balanced accuracy compared with 0.519 for the next-best evaluated baseline. This is a meaningful improvement for the single-video setting.

  • Figure 6 shows that different checks respond to different violations. For example, Onset detects 87% of Collision violations, while Visibility mainly responds to Continuity and Height. This makes the model's decisions easier to interpret.

Critical weaknesses
  • The method depends on accurate 3D object states. Figure 8 includes predicted boxes, but their 2D localization still uses ground-truth-mask-derived prompts. So the experiment doesn't fully test an automatic video-to-decision pipeline.

  • Figure 6 shows that Containment violations trigger Visibility 98% of the time but Outcome only 8%, even though Containment is an Outcome-targeted category. The authors link this to independent trajectory fitting, but it highlights a limitation of the current event representation.

  • In Figure 8, the predicted-box experiment still uses ground-truth-mask-derived prompts. What happens if object detection, tracking, and 3D reconstruction are all automatic? Does the standalone detection performance drop significantly?

The response partially solved my concerns, I will remain the original score.

Questions for authors

NA

2 comments
AuthorsOpenReview ↗

Summary and Perception and Containment (Part 1/1)

Summary

Dear Reviewer 56sb,

Thank you for your positive assessment of the detection improvement and the diagnostic value of PULSE. We appreciate your suggestions on perception and event modeling.

(W1, W3) PULSE's physical-scoring gain is supported under controlled perception. In the shared-state comparison, PULSE achieves 0.665 All Direct versus 0.519 for the best baseline; the predicted-box control uses GT-mask-derived prompts to examine input sensitivity, with fully automatic perception discussed as a further extension.

(W2) Containment demonstrates the benefit of complementary checks. The ball exposed when the cup moves away triggers Visibility in 98.3% of Containment violations, compared with 8.0% for Outcome (Figure 6b), allowing detection through a different consequence of the same intervention.

1. Perception and broader applicability (W1, W3)

We share your interest in extending PULSE to a fully automatic pipeline. Our current evaluation focuses on physical scoring under controlled perception (Section 3.1), consistent with PLATO's use of supplied masks and identities and X-VoE's allowance for GT segmentation. Shared-state baselines compare the physical scorers directly (Figure 4), while the predicted-box experiment using GT-mask-derived prompts and state-quality controls examine sensitivity to estimated inputs (Figure 8; Table A7 (original) / Table A9 (revised)). These evaluations support the physical-scoring contribution. Fully automatic detection, tracking and 3D recovery would be a valuable extension toward broader applications, as discussed in Section 6.

2. Containment (W2)

Thank you for highlighting this case. The strong Visibility response arises when the cup moves away and exposes the ball, providing a useful violation cue (Figure 6b; Appendix C). The category label specifies the intervention stage, while the checks can capture complementary consequences of the same intervention (Section 3.4; Appendix A).

We agree that better modeling of hidden ball–cup motion would strengthen Outcome. Independent trajectory fits currently miss this relation; relation-aware parsing is a promising extension discussed in Section 6. The existing Visibility response illustrates how complementary checks can still provide useful detection evidence in this case.

AuthorsOpenReview ↗

[Additional experiment] Cross-dataset transfer to ADEPT with automatic object perception

Thank you for suggesting that we test PULSE without prompts derived from ground-truth masks. We have now completed this experiment on ADEPT (Smith et al., 2019), an external benchmark held out from PULSE training and calibration. The 378 videos (189 pairs) cover the three scenarios used in PLATO’s transfer study (Piloto et al., 2022).

W1. Object perception without ground-truth prompts

Mask R-CNN detected the objects, SAM2 tracked them, and Video Depth Anything provided depth for 3D reconstruction. Only the fixed camera parameters and ground-plane depth were supplied as ground truth.

W3. Effect on standalone detection accuracy

With PULSE and its thresholds frozen, automatic perception lowered single-video balanced accuracy by 1.3 percentage points.

Object statesDirectAUCCompare
Supplied55.56%68.20%77.78%
Automatically reconstructed54.23%56.18%60.32%
Reviewer_AoqtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents PULSE, a framework that leverages 3D object states and related features to detect physical violations in a single video. Specifically, it parses 3D object trajectories to segment videos into events and then applies multiple checks on each event to detect physical violations. The three checks are Visibility, Onset, and Outcome, all learned from plausible videos. The paper also introduces PhysEvent, a synthetic benchmark dataset with stage-specific violations and ground-truth 3D states. Experimental results show that PULSE outperforms existing methods on single-video physical violation detection across three benchmark datasets.

Critical strengths
  1. Important and well-motivated problem. The authors address the challenge of detecting physical violations from a single video without requiring matched plausible references or violation labels. The motivation is clear and aligns well with current research trends.
  2. Intresting event-based formulation. Separating physical expectations into Visibility, Onset, and Outcome is intresting. Calibrating each check using errors from plausible videos provides a useful way to combine different prediction signals without requiring violation labels.
  3. Carefully designed benchmark dataset. The proposed benchmark includes various types of physical violations, and the associated ground-truth 3D object states enable detailed analysis of the model's results.
  4. Comprehensive experiments. The paper presents experimental results on three benchmark datasets and compares the proposed method with various baselines in both their original and adapted forms. The evaluation also includes ablation studies, threshold sensitivity analysis, comparisons of pooling strategies, and state-quality controls.
Critical weaknesses
  1. Important ablations are missing. The proposed method uses 3D object states together with visual features to detect physical violations. However, these context features would include surrounding objects and may directly reveal violations without requiring physical dynamics reasoning. Therefore, an ablation using only 3D object states, without visual features, should be conducted to assess their contribution.
  2. Complex pipeline. Although the overall idea is straightforward, the proposed method involves multiple steps, including 3D object tracking, trajectory parsing, and multiple prediction heads. This introduces substantial complexity and potential error propagation, which may affect the robustness of the system. It is also unclear whether all these components are necessary to achieve the reported improvements.
  3. Dependence on accurate 3D object tracking. The proposed method requires 3D object trajectories, which are difficult to obtain from real-world videos, especially under occlusion and complex interactions. As shown in Appendix D.1 (Table A7), performance is sensitive to the quality of the estimated 3D boxes. This dependence may limit the method's applicability to real-world scenarios.
  4. Presentation needs improvement. The motivation is clear, but important details, including the input–target definitions (Table A12) and event-parsing rules, are only provided in the appendix. Figure 1 lacks a clear legend and leaves some notation unexplained. Figure 2 illustrates predicted states as scene images rather than actual model outputs. Some terminology, particularly regarding temporal intervals, is also not clearly defined, making the method difficult to follow.
Questions for authors
  1. Fairer baseline comparison. SlotFormer-b uses absolute coordinates and regression-based prediction, while PULSE uses pairwise relational states and categorical outputs. Could the authors provide a baseline with comparable representations and prediction to better isolate the benefit of event-based prediction?
  2. Interpretability. Since PhysEvent contains controlled physical interventions, could the authors quantify how often the predicted event (e^*) matches the actual violation event? This would provide stronger evidence for the claimed event-level localization.
  3. High false-positive rate on IntPhys. Table A4 shows 75% of plausible IntPhys videos are incorrectly classified as violations. While the authors attribute this to shared calibration thresholds, could they clarify why IntPhys is particularly affected and whether errors in 3D state estimation also contribute?
6 comments
AuthorsOpenReview ↗

Summary and Ablation and Pipeline (Part 1/2)

Summary

Dear Reviewer Aoqt,

Thank you for recognizing the event-based formulation and for the suggestions on analysis and presentation.

(W1) Additional visual features are not the main source of PULSE's detection gain. Removing both object descriptors and visual context and retraining changes All Direct/AUC only from 0.680/0.696 to 0.677/0.692 at seed 42.

(W2; Q1) Adaptive event alignment adds value under matched representations and prediction objectives. Keeping the relative states, network, categorical objectives and calibration procedure unchanged, fixed-window retraining lowers Direct from 0.680 to 0.608 on All and from 0.583 to 0.557 on Public.

(W3; Q3) Shared calibration alone does not explain IntPhys's high test FPR. At the 10% pooled validation target, IntPhys has 17.77% validation FPR and 63.33% test FPR; the test split's reliance on SAM2 identity propagation introduces a state-quality difference that may contribute to this additional gap.

(Q2) Manual assessment supports event-level localization conditional on successful detection. The observed violation falls within the flagged event interval in 82 of 120 assessed videos (68.3%), sampled as 20 randomly selected, correctly detected videos per category.

(W4) The main text now makes the event-parsing rules and prediction targets explicit. Sections 3.2–3.3 present the key parsing rules and a compact input–target summary, while the revised Figure 1–2 captions explain the notation and distinguish schematic scenes from actual model outputs.

1. Framework-level contribution and ablation evidence (W1)

Our contribution concerns the event-based framework, supported by the overall comparisons (Figure 4; Table A2 (original and revised)); further component-wise attribution extends this analysis. To test whether visual inputs supply the main violation cues, we removed object descriptors and visual context and retrained:

Model, All datasetsDirectAUCCompare
Full PULSE (Tables A8–A9 (original) / Tables A11–A12 (revised))0.6800.6960.735
State-only PULSE0.6770.6920.729

Performance is largely preserved, showing that the main detection gain does not depend on these additional visual inputs.

2. Pipeline design and framework-level contribution (W2; Q1)

PULSE follows a simple, task-driven logic: locate meaningful changes in motion, check the physical expectations at the corresponding event stages, and combine the evidence into a video-level decision. Each step uses familiar modeling tools. Piecewise-linear trajectory fitting identifies event boundaries; relative object states and relational graph networks represent interactions (Sanchez-Gonzalez et al., 2020); and categorical prediction accommodates multiple plausible state values (Trajectory Transformer). The separate prediction heads naturally follow the distinct physical questions—visibility, interaction onset, and outcome—and their corresponding targets. Percentile-based calibration places the prediction errors on a common plausible-video reference scale, while maximum aggregation implements the rule that any violated expectation can flag a video. The pipeline therefore directly reflects the task structure using standard building blocks. Its framework-level contribution lies in organizing these tools around event-level physical expectations for plausible-only, standalone VoE detection. The controls below further examine these design choices.

To test adaptive alignment with matched representations and prediction objectives, we replaced adaptive boundaries with fixed windows set from plausible training/validation statistics and retrained at seed 42, retaining the relative states, network, categorical objectives and calibration procedure:

EvaluationModelDirectAUCCompare
AllAdaptive PULSE (Table A8 (original) / Table A11 (revised))0.6800.6960.735
AllFixed-window PULSE0.6080.6300.715
PublicAdaptive PULSE0.5830.6250.634
PublicFixed-window PULSE0.5570.5830.633

The decline in Direct and AUC supports adaptive alignment within the event-based framework.

Replacing relative states with absolute states gives:

Representation, All datasetsDirectAUC
Relative states (Table A8 (original) / Table A11 (revised))0.6800.696
Absolute states0.6210.652

This supports the suitability of relative states; Table A8 (original) / Table A11 (revised) additionally reports discretization sensitivity.

In progress: Object-level DINO controls for PLATO-b/SlotFormer-b.

AuthorsOpenReview ↗

Applicability and Interpretability and Presentation (Part 2/2)

3. Applicability and IntPhys false positives (W3; Q3)

Dependence on 3D perception does not invalidate evaluation of the physical model under controlled inputs. This is an established VoE setting: PLATO uses supplied masks and identities, and X-VoE permits GT segmentation. Shared-state baselines test PULSE's physical scorer on the same 3D inputs (Figure 4). More general tracking and 3D recovery are system-level extensions discussed in Section 6.

For IntPhys, test videos lack usable GT tracks and require SAM2 identity propagation (Table A11 (original) / Table A18 (revised); Appendix E.1). The resulting tracking and state-quality difference from validation can affect reconstructed trajectories and event states, raising prediction errors even on plausible videos. Table A7 (original) / Table A9 (revised) supports sensitivity to input quality.

At a global 10% validation video-FPR target:

DatasetValidation FPRTest FPRTest TPR
IntPhys17.77%63.33%79.44%

The 10% target is pooled across datasets; IntPhys's 17.77% validation FPR reflects its operating point under shared calibration. The further increase to 63.33% on test is a separate validation-to-test gap, to which the tracking and state-quality difference may contribute. Revised Appendix D.4 discusses these effects separately.

4. Event localization and attribution (Q2)

The original paper already reports check and object attribution: Figure 7 illustrates the readout, and Figures A3–A4 connect object indices to scenes and summarize their frequencies (Appendix C).

In the Collision scenes, the barrier's collision property is disabled at scene initialization, but the violation becomes observable only when the object should collide with the barrier. This later violation-onset time is not recorded in the dataset.

We manually assessed 20 randomly sampled, correctly detected videos per category. A localization was judged correct if manual inspection confirmed that the physical violation occurred within the event interval flagged by the model:

PhysEvent categoryCorrect / assessedTop-1 localization correctness
Continuity20 / 201.000
Height18 / 200.900
Collision14 / 200.700
Gravity13 / 200.650
Attachment7 / 200.350
Containment10 / 200.500
Overall, equal category weights82 / 1200.683

The 82/120 agreement supports event localization conditional on successful detection. This assessment is included in revised Appendix C.1.

5. Presentation (W4)

Thank you for identifying where the presentation can be clearer. The revised Sections 3.2–3.3 present the key event-parsing rules and a compact input-target summary, with full input-target definitions in Table A12 (original) / Table A19 (revised):

HeadState inputsPrediction target
Stable VisibilityStart and MiddleVisibility probabilities over the stable interval
Boundary VisibilityMiddle and EndVisibility probabilities over the transition interval
OnsetStartCategorical distributions over Middle relative positions
OutcomeStart and MiddleCategorical distributions over End relative positions and velocities

The revised Section 3.2 defines Start as the preceding event's End (or the first frame), and Middle/End as the earliest/latest times in a merged cluster of motion changes. A terminal event covers the remaining interval (Appendix E.2). Figure 1's caption explains the check colors, event indices and plausible-only decision threshold.

Figure 2 retains the schematic scenes, and its caption explicitly distinguishes illustrations of state targets from actual model outputs:

Scene images illustrate state targets; the model outputs visibility probabilities and categorical state distributions, not future images.

Thanks the authors for the detailed rebuttal and additional experiments. Several of my concerns have been addressed.

In particular, the state-only ablation shows that additional visual features provide limited performance gains. This suggests the features may be unnecessary and could be removed to simplify the framework.

However, my main concern remains the overall presentation and clarity of the paper. The proposed framework involves multiple components, but their roles, and implementation details are not clearly explained. Although the authors have made revisions, I believe the paper still requires substantial improvements in its organization and presentation to meet the standards of a top-tier conference.

Therefore, I keep my original rating.

Reviewer_so65OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article studies the detection of single video physical expectation violation. The author points out that correct paired ranking does not necessarily mean reliable judgment of a single video. Therefore, PULSE is proposed: first segment the 3D object trajectory into events, and then construct three types of checks: Visibility, Onset, and Outcome at each stage, and calibrate them with the error distribution of normal videos. Independent judgment can be made without pairing videos. On the newly established PhysEvent, PhysicalConcepts, and IntPhys, PULSE’s single video detection overall outperforms existing continuous scoring methods and can provide event level and object level localization.

Critical strengths

The paper distinguishes between paired ranking and independent judgment, points out the limitations of existing evaluation protocols, and the problem is valuable. Dividing the inspection into three categories based on the stage of the event, with clear physical motivations and some interpretability of the results. The experiment covered two publicly available benchmarks and a new dataset, and conducted a comprehensive analysis of state quality, threshold, and error aggregation methods.

Critical weaknesses
  1. Lack of comparison to validate core claims, and the comparison conditions are inconsistent. Compared to baseline, PULSE introduces event segmentation, three types of prediction tasks, relational networks, maximum error aggregation, and head by head calibration, while existing ablation mainly removes certain types of checks and cannot determine whether event segmentation itself is necessary. Although PLATO-b and SlotFormer-b obtained the same 3D box states, they only used full frame features, and PULSE additionally used object level visual descriptions and event context; A pure RGB baseline has no structured 3D input at all. Therefore, it is impossible to determine the improvement comes from which design.
  2. The handling mechanism for violations such as "changes that should have occurred have not occurred" is unclear. The event segmentation depends on the observed motion changes, and the paper does not explain where the event boundary comes from at this time. The Onset detection rate of Gravity class is 46%, indicating that the method is not completely ineffective. However, the detection rates of the three types of checks for directional inertia were only 0.8%, 0.0%, and 7.6% (Table A6), and the author attributed the reason to the lack of visible interaction during movement changes.
  3. Insufficient generalization evidence. The training and testing sets of PhysEvent have the same scene types; All sets are used for joint training with three datasets, while Public sets only recalibrate the threshold. The Direct values of the three datasets are 0.821, 0.594, and 0.580, respectively, with significant differences. The paper also did not test rare but reasonable difficult normal samples, making it difficult to distinguish whether the model learned physical expectations or fitted the statistical patterns of the dataset.
  4. The false alarm rate is relatively high. If any of the four heads exceeds their respective P90, it is considered abnormal, and the video level false alarm rate will be significantly higher than 10% (Appendix D.3); The baseline only takes P90 for a single score, and the two are not compared at the same false positive rate. 75.0% of normal videos on IntPhys were falsely reported (Table A4). In addition, each Head takes the maximum error over the entire video, which may be affected by the length of the video and the number of targets: in Figure A2, the false positive rate of normal videos generally increases with the average duration of the category. 5、The LikePhys replication gap is not explained. On IntPhys, the reproduced PPE is 39.4%, while the original paper reports 30.5% (lower is better).
Questions for authors
  1. How much does the performance decrease if we keep the input, network, and calibration unchanged and only replace the event boundaries with uniform time partitioning?
  2. Can you provide a baseline with completely consistent input features? Does PULSE still have an advantage when not relying on ground-truth masks and box prompts?
  3. How to determine the event boundary for the violation of “changes that should have occurred have not occurred”?
  4. Can experiments be conducted under unseen physical conditions and on difficult normal samples?
  5. Does the conclusion change compared to a fixed video level false positive rate (such as 5% or 10%)?
  6. What is the reason for the discrepancy in LikePhys replication? Does it affect the comparative conclusion?
7 comments
AuthorsOpenReview ↗

Summary and Core claim and Not occurred (Part 1/3)

Summary

Dear Reviewer so65,

Thank you for recognizing the motivation and interpretability of PULSE.

(W1; Q1–Q2) Adaptive alignment improves standalone detection, and additional visual inputs are not the main source of the gains. In seed-42 controls, matched fixed-window retraining reduces All Direct/AUC from 0.680/0.696 to 0.608/0.630, whereas removing object descriptors and visual context leaves Direct/AUC at 0.677/0.692.

(W2; Q3) Missing interactions can be detected without a boundary at the expected interaction time. Onset compares the expected next-stage state with the observation at the next boundary or terminal frame; Figure 7 demonstrates this through a detected missed collision.

(W3; Q2, Q4) Frozen evaluation on ADEPT provides additional cross-dataset transfer evidence. Using supplied ground-truth states, the main seed-42 model achieves 68.20% AUC, 77.78% Compare and 55.56% Direct on 378 ADEPT videos, with no retraining or threshold recalibration.

(W4; Q5) PULSE's detection advantage persists when validation false-alarm rates are matched. At the 10% pooled validation video-FPR target, All Direct is 0.612 versus 0.517 for the best baseline, and PULSE also leads at the 5% and 20% targets.

(W5; Q6) Recovering LikePhys's published paired performance does not overturn the single-video detection conclusion. Replacing compressed IntPhys inputs with original PNGs improves PPE from 0.394 to 0.272, compared with the published 0.305, while original-frame single-video AUC remains near chance at 0.503 on IntPhys and 0.500 on PhysicalConcepts.

1. The core claim and supporting evidence (W1; Q1–Q2)

We appreciate the request to examine the sources of the gains. The existing comparisons already support our core claim: PULSE provides an effective event-based framework for plausible-only single-video VoE detection, improving overall detection relative to the evaluated continuous scorers, including baselines given the same 3D states (Figure 4; Table A2 (original and revised)). Further component-wise attribution deepens this evidence without being a prerequisite for the framework-level contribution. The controls below examine adaptive alignment and whether additional visual inputs explain the gains.

We replaced adaptive boundaries with fixed windows set from plausible training/validation statistics, retained the state representation, network, objectives and calibration procedure, and retrained the scorer (seed 42). The window settings are specified in revised Appendix D.2.

EvaluationAdaptive PULSE: Direct / AUCFixed-window PULSE: Direct / AUC
All0.680 / 0.696 (Table A8 (original) / Table A11 (revised))0.608 / 0.630
Public0.583 / 0.6250.557 / 0.583

Both variants retain event-stage checks; lower Direct and AUC support the additional benefit of adaptive alignment.

Removing object descriptors and visual context and retraining gives:

Model, All datasetsDirectAUCCompare
Full PULSE (Tables A8–A9 (original) / Tables A11–A12 (revised))0.6800.6960.735
State-only PULSE0.6770.6920.729

Performance is largely preserved, so additional visual inputs are not the main source of the gain.

In progress: Object-level DINO controls for PLATO-b/SlotFormer-b.

2. When an expected interaction does not occur (W2; Q3)

An absent expected interaction is already a violation that Onset tests. It compares the expected next-stage state with the observation at the next boundary or terminal frame, even when the expected interaction never occurs (Section 3.3; Appendix E.2). Figure 7 demonstrates this through a missed collision detected by Onset.

Directional inertia concerns motion after contact (official dataset). Here, noisy predicted depth and the 15-frame clips may destabilize boundary fitting during training, impairing learning of the expected interaction transitions (Table A11 (original) / Table A18 (revised); Appendix F.1). Revised Appendix C explains how depth noise can affect boundary recovery and the transition checks.

AuthorsOpenReview ↗

Generalization and Calibration (Part 2/3)

3. Generalization and perception (W3; Q2, Q4)

Our contribution is the event-based framework's effectiveness on single-video detection: 0.665 Direct versus 0.499–0.519 across three distinct benchmarks (Figure 4; Table A2 (original and revised)). PhysEvent varies object properties and placements, with disjoint initial configurations and event-non-occurrence controls. PhysicalConcepts separates freeform training/validation from concept probes; IntPhys likewise uses separate evaluation scenes (Appendices A, E.4).

This focused evaluation follows IntPhys's diagnostic testing of specific physical concepts. PLATO also treats arbitrary-shape dynamics as a separate generalization challenge. Its GT masks and consistent identities, together with X-VoE's allowance for GT segmentation, support studying physical reasoning under controlled perception. PULSE's shared-state comparisons and state-quality controls serve this purpose (Figure 4; Table A7 (original) / Table A9 (revised)).

Broader transfer calls for richer event data and reliable tracking and 3D recovery, through annotations or stronger perception (Section 6). These are natural extensions, rather than prerequisites for establishing the present framework-level contribution.

We also conducted a small transfer evaluation on PLATO's three ADEPT subsets (378 videos, 189 pairs), using supplied GT states with weights and thresholds frozen:

EvaluationCompareROC AUCDirect
ADEPT (three subsets, overall)77.78%68.20%55.56%

The above-chance results provide supplementary transfer evidence without retraining or recalibration.

4. Video-level calibration and duration (W4; Q5)

PULSE's four heads implement three complementary physical checks: any violated expectation can invalidate a video. Removing any check lowers Direct (Figure 6a), supporting this design. Each head deliberately receives its own plausible P90 reference (Section 3.4; Appendix D.3 (original) / Appendix D.4 (revised)).

We adjust the four heads' common upper percentile to meet each pooled validation video-FPR target (revised Appendix D.4):

Validation video-FPR targetPULSE All DirectBest of six baselines: All Direct
5%0.5870.511
10%0.6120.517
20%0.6690.521

PULSE remains ahead at every target, so its advantage persists when validation false-alarm rates are aligned.

The target applies to the pooled, equally weighted validation datasets, so it need not hold separately for each dataset. At the global 10% target:

DatasetValidation FPRTest FPR
IntPhys17.77%63.33%

IntPhys's 17.77% validation FPR is its operating point under shared calibration; the further increase to 63.33% on test reflects a validation-to-test distribution difference. Its test videos lack usable GT tracks and require estimated tracking, creating a state-quality difference from validation. The resulting tracking and state errors can raise plausible-video scores and contribute to this additional gap (Table A11 (original) / Table A18 (revised); Appendix E.1). Revised Appendix D.4 distinguishes these calibration and input-quality effects.

Duration and object-count sensitivity reflect the multiple-testing problem studied in anomaly detection and sequential monitoring (Rebjock et al., NeurIPS 2021, Section 2; Podkopaev & Ramdas, ICLR 2022, Appendix A.1). With fixed local thresholds, checking more events or objects creates more opportunities for at least one false alarm. This is a general calibration issue in repeated detection, not a property unique to event-based modeling. Grouping PhysEvent by duration gives:

Duration quartileDuration (s)FPR
Shortest20.633–56.4176.21%
Longest90.733-104.56729.58%

The longer-video group has higher FPR, consistent with this sensitivity.

AuthorsOpenReview ↗

LikePhys (Part 3/3)

5. LikePhys reproduction (W5; Q6)

We used the official code and traced the reproduction gap to our common compressed-input preprocessing. Even visually minor pixel changes can affect LikePhys's noise-prediction error (revised Appendix F.2). Restoring original IntPhys PNGs gives:

LikePhys evaluationPPE ↓
Published result0.305
Reproduction with compressed inputs0.394
Reproduction with original PNGs0.272

Original inputs recover and exceed the published PPE result.

The remaining PPE–Compare discrepancy is largely due to score ties: the released PPE implementation counts ties as correct, whereas Compare gives half credit:

DiagnosticResult
PPE-derived score with half credit for ties0.556
Compare0.553

Consistent tie handling nearly reconciles the paired scores.

The original-frame single-video comparison is:

DatasetLikePhys, original frames: AUCPULSE, compressed inputs (Table A3): AUC
IntPhys0.5030.629
PhysicalConcepts0.5000.633

LikePhys AUC remains near chance and below PULSE on both datasets. Recovering its published paired performance therefore does not overturn the single-video conclusion.

Comments on the paper

3 comments

Dear reviewers, the authors have posted their responses to your reviews. Pls have a look and discuss.

Thanks, AC for #7156

AuthorsOpenReview ↗

Summary of manuscript revisions

We thank all four reviewers for their detailed feedback. We have incorporated the additional experiments and clarified the method and its decision process. The five text colors identify the revision topics below. We have also updated a few table references in our earlier responses to match the revised manuscript numbering.

Teal — Method presentation and decision traces (Aoqt). We revised Figures 1 and 2 and their captions to make the event-level checks and model pipeline easier to follow. Figure 2 shows the GNN shared by Onset and Outcome, their separate decoders, and the prediction–target comparisons. Appendix G provides pseudocode, while Appendix H traces actual video decisions through their event-level predictions (Table A28, Figures A14 and A21). Appendix C.1 reports the localization and object–check attribution assessments.

Olive green — Design and state-quality controls (so65, 5d26, Aoqt). Table 1 brings together the fixed-window, state-only, and absolute-state controls under both All and Public calibration, with settings in Appendix D.2. Tables A9 and A10 report the state-quality controls and their repeat at a second training seed.

Purple — Baseline comparisons (Aoqt, so65, 5d26). Table A26 compares the original-target and relative-state variants of PLATO and SlotFormer with the same object-level visual features. Table A27 extends the comparison to matched validation false-positive-rate targets. Appendix F.2 also consolidates the LikePhys and JEPA reproduction checks, including evaluations on original public frames (Tables A23–A25).

Burnt orange — Calibration and tracking (so65, 5d26). Appendix D.4 distinguishes the shared validation FPR target from per-dataset rates. We re-tracked the same 3,000 plausible IntPhys validation videos with the test-time SAM2 procedure and recalibrated the shared thresholds while keeping PULSE and all test losses fixed. Table A16 reports both IntPhys and three-dataset results. This appendix also includes matched-FPR comparisons and duration analysis.

Burgundy — ADEPT transfer and automatic perception (56sb, so65). Appendix B.1 and Table A5 report transfer to 378 ADEPT videos (189 pairs), comparing supplied and automatically reconstructed object states with PULSE weights and its original thresholds frozen. Automatic reconstruction uses object detection, tracking, and estimated depth, with camera parameters and ground-plane depth supplied.

AuthorsOpenReview ↗

Summary of rebuttal discussion

1. Contributions recognized by the reviewers.

We thank all four reviewers for the constructive discussion. The reviews recognized the value of plausible-only single-video violation detection (so65, Aoqt, 5d26), the motivation for organizing physical expectations around event stages (so65, Aoqt), and the meaningful detection gains and diagnostic responses of the checks (56sb). Aoqt and 5d26 also highlighted PhysEvent’s controlled design and the ability to inspect model decisions. These observations identify the central contribution: learning from plausible experience to judge an individual video and identify the event and physical expectation underlying that judgment.

2. Progress acknowledged in the discussion.

so65 acknowledged that fixed-window retraining supports adaptive event alignment, that the visual-input controls substantially alleviate concerns about additional features, and that matching validation FPR improves comparison fairness. The reviewer also welcomed the relative-state baselines, accepted the explanation of missing interactions, and indicated that the original-frame LikePhys results largely resolve the reproduction concern. Aoqt likewise acknowledged the state-only finding that additional visual features provide limited performance gains. These responses provide concrete support for the event-based formulation and strengthen the interpretation of the comparisons.

3. Further evidence and presentation improvements.

The additional comparisons show that PULSE retains its advantage over the relative-state baselines at matched validation FPR targets (so65, 5d26). The IntPhys tracking and recalibration experiments examine the validation-to-test discrepancy (so65), while the ADEPT evaluation with automatically reconstructed states tests dependence on supplied object states (56sb). Our responses to 5d26 also provide the requested All/Public fixed-window results, JEPA reproduction checks, aggregation comparisons, and second-seed state-quality evaluation. In response to Aoqt’s emphasis on presentation, the revised Figures 1 and 2 and worked examples connect the model architecture, prediction targets, and event-level evidence to the final video decision.

4. Remaining limitations and the significance of the advance.

We acknowledge the remaining challenges in fixed-threshold reliability and transfer (so65, 5d26), sensitivity to object-state quality (Aoqt, 56sb), and hidden object relations in Containment (56sb). The original Limitations section discussed dependence on reliable object states, transfer beyond the evaluated domains, and the coverage of the current event vocabulary; the additional experiments make these boundaries more concrete. Nevertheless, PULSE offers a new perspective on physical-violation detection through two complementary ideas: using events as the units of analysis and treating different types of physical phenomena separately. Our experiments demonstrate the effectiveness of this framework. We believe this constitutes a substantive methodological and conceptual advance, even though broad applicability and robust generalization remain open challenges.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.