- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper detects physical violations in a single, unpaired video, using only plausible videos for training, model selection, and calibration. Proposed model (PULSE) parses tracked 3D object boxes into events and applies three checks for each event (i.e., Visibility, Onset, Outcome) through four heads; each head's maximum error is normalized by its certain range on plausible validation videos and a video is flagged when any head exceeds its certain percentage, and the maximizing target names the event, check, and objects. The paper also introduces a new benchmark named as PhysEvent which consists of 10,800 Isaac Sim videos, six stage-targeted violation types, native 3D states). Across three datasets (including PhysEvent), PULSE reaches state of the art performance on all three benchmarks with three different metrics (i.e., Direct, AUC, Compare)
Critical strengths
-
Novel objective with strong gain of performance Plausible-only standalone decisions seems to be a useful target. PULSE raises Direct from 0.519 to 0.665 on all datasets and from 0.516 to 0.580 on the public ones, with smal variance (Table A2), and the shared-state baselines, threshold sweeps (Table A10), and adapter peak pooling (Table A15) rule out several simple explanations.
-
Diagnostic benchmark, inspectable decisions. PhysEvent's stage-targeted interventions, matched pairs, event-non-occurrence controls, and native 3D states enable controlled analysis. Without violation-type supervision, the dominant check matches the targeted stage in five of six categories (Figure 6b), and the readout identifies the responsible object pair (Figure 7).
Critical weaknesses
-
Isolation of event alignment Beyond parsing, PULSE differs from the shared-state "-b" baselines in pair-relative states, a categorical objective, object-centered features, piecewise-linear fits, multiple heads, and per-head referencing. The check ablations (Figure 6a, Table A5) remove checks after training and Table A8 keeps parsing, so nothing shows that parsed boundaries are necessary; the evidence supports the full pipeline more than events, not frames
-
Limited reliability on standalone The Public Direct/AUC are 0.580/0.624, versus 0.821/0.893 on PhysEvent, and the false-positive rate is 41.3% on average and 75.0% on IntPhys (Table A4). With four heads each thresholded at 90th percentiles, a union bound allows an averaged validation FPR of up to about 40%, and the realized test rate is at that level. The paper attributes the IntPhys rate to shared thresholds but does not separate this from a shift between the plausible references (held-out IntPhys training videos, PhysicalConcepts freeform videos) and the test videos.
-
The JEPA baselines may be under-configured. V-JEPA and V-JEPA-2 average per-window surprise (Section F.1), whereas Garrido et al. (2025) favor maximum surprise for single videos and find it markedly better for V-JEPA-H on IntPhys (refer Table S5 in Garrido et al. (2025)). Table A15 covers only the PLATO/SlotFormer adapters. The released non-RoPE ViT-H used here also appears to differ from their RoPE-pretrained models (App. A.3) and reaches 0.778 IntPhys Compare (0.817 in Table A16), versus their ~98% on the same dev set. Both bear on the claim that strong paired-ranking baselines stay near chance standalone.
-
Two supporting analyses could be overreaching. Table A9 compares aggregation rules at a fixed threshold of 1, although every alternative scores no higher than the maximum, so its Direct column mixes in operating-point effects. Table A7 uses one seed per condition and is strongly non-monotonic (Direct 0.538 at 7.5% box noise vs. 0.731 at 10%), too noisy to show that predicted boxes retain most native-state performance.
Questions for authors
- Fixed windows: If parsed boundaries are replaced by fixed windows of matched duration (or random boundaries matched in number), with everything else unchanged, how do all-dataset and public Direct/AUC change after retraining?
- Video-level calibration: If the final score is calibrated to 10% and 20% validation FPR, what are the per-dataset TPR, validation FPR, and test FPR?
- JEPA baselines: What Direct/AUC do V-JEPA and V-JEPA-2 reach with maximum-surprise pooling under the same calibration, and with the checkpoint evaluated by Garrido et al. if it is available?
- Supporting analyses: Does Table A9's ordering hold when each rule is calibrated to the same video-level FPR, and how much do Table A7's conditions vary across seeds?
5 comments
JEPA and Supporting analyses (Part 2/2)
3. JEPA configuration and reproduction (W3; Q3)
Our original experiments already used V-JEPA's official code and public checkpoint. The reproduction gap came from the common video compression applied across datasets. Compression-quality measurements show small average pixel differences (RGB MAE 3.4569 on the 0–255 scale; PSNR 35.77 dB) and high structural similarity (luminance SSIM 0.9655; revised Appendix F.2). Under the same evaluation protocol as the original appendix, restoring uncompressed frames raises IntPhys Compare from 0.817 to 0.917, consistent with the authors' approximately 0.920 for that checkpoint (official Issue #2). The approximately 0.980 result uses a different RoPE checkpoint (Garrido et al., Appendix A.3).
We further compared mean and maximum surprise for both JEPA models, using original uncompressed frames for IntPhys and PhysicalConcepts and retaining the compressed videos for PhysEvent.
We additionally report Oracle Direct, the highest balanced accuracy over all thresholds on the same fixed test scores, selected using test labels. It provides an optimistic thresholding bound on this test set and is reported separately from plausible-validation-calibrated Direct. For each model and pooling rule, All uses one common threshold across the three datasets, maximizing equal-dataset-weighted mean Direct; IntPhys independently optimizes one threshold for that dataset:
| Model, 16 frames | Dataset | Mean AUC | Mean Compare | Mean Oracle Direct | Max AUC | Max Compare | Max Oracle Direct |
|---|---|---|---|---|---|---|---|
| V-JEPA | All | 0.515 | 0.720 | 0.518 | 0.523 | 0.651 | 0.536 |
| V-JEPA | IntPhys | 0.549 | 0.917 | 0.553 | 0.574 | 0.789 | 0.592 |
| V-JEPA-2 | All | 0.511 | 0.619 | 0.514 | 0.507 | 0.577 | 0.516 |
| V-JEPA-2 | IntPhys | 0.515 | 0.706 | 0.528 | 0.533 | 0.689 | 0.542 |
Our target is standalone detection across diverse scenes and violation types with a shared decision rule. The All protocol directly evaluates this setting: across both JEPA models and pooling rules, Oracle Direct remains 0.514–0.536 even when the common threshold is optimized using test labels. Thus, threshold selection alone does not resolve the limitations of these scores for the proposed task. Together with the AUC/Compare results, this supports our conclusion for the evaluated public checkpoints that strong paired ranking does not ensure reliable standalone detection across heterogeneous physical events.
4. Aggregation and state-quality repetitions (W4; Q4)
Max retains the strongest local violation evidence. For each aggregation rule, we adjust the four heads' common upper percentile separately to meet the same pooled validation video-FPR target (revised Appendix D.4):
| Aggregation | Direct at 10% validation target | Direct at 20% validation target |
|---|---|---|
| Max | 0.612 | 0.669 |
| Top-2 mean | 0.607 | 0.634 |
| Softmax-weighted | 0.607 | 0.642 |
| Mean | 0.570 | 0.593 |
| Log-mean-exp | 0.599 | 0.625 |
Max remains best at both targets, so its advantage is not just an effect of the original shared threshold.
We also completed the seed-2026 repeat of Table A7 (original) / Table A9 (revised):
| PhysEvent condition | Direct, seed 42 | Direct, seed 2026 | AUC, seed 42 | AUC, seed 2026 | Compare, seed 42 | Compare, seed 2026 |
|---|---|---|---|---|---|---|
| Native states | 0.770 | 0.7350 | 0.843 | 0.8486 | 0.898 | 0.9067 |
| 1% box noise | 0.670 | 0.6750 | 0.784 | 0.8055 | 0.772 | 0.8600 |
| 2.5% box noise | 0.782 | 0.7200 | 0.857 | 0.8044 | 0.912 | 0.8600 |
| 5% box noise | 0.655 | 0.6950 | 0.752 | 0.8047 | 0.770 | 0.8217 |
| 7.5% box noise | 0.538 | 0.5475 | 0.638 | 0.6904 | 0.672 | 0.7617 |
| 10% box noise | 0.731 | 0.6033 | 0.786 | 0.7016 | 0.822 | 0.7067 |
| 15% box noise | 0.635 | 0.5183 | 0.709 | 0.6276 | 0.753 | 0.6367 |
| Predicted boxes | 0.732 | 0.7058 | 0.845 | 0.8096 | 0.870 | 0.8550 |
| Predicted depth | 0.574 | 0.5133 | 0.682 | 0.6370 | 0.692 | 0.6517 |
The overall conclusions are unchanged. Predicted-box Direct stays close to native-state performance in both seeds, with drops of approximately 3.8 and 2.9 percentage points; predicted depth produces larger drops in both. Stronger noise also generally reduces performance. Differences in the ordering of individual noise conditions do not alter these conclusions: Table A7 (original) / Table A9 (revised) characterizes full-pipeline sensitivity, not a strictly monotonic noise–performance law (Appendix D.1). Both seeds are reported in revised Appendix D.1.
W1 — Representation and prediction targets
To follow up on W1, we tested the relative-state representation and prediction targets within PLATO and SlotFormer (Table A26). For each backbone, the box control and the adapted model receive the same object-level DINO descriptors and visual context. The adaptation retains the temporal backbone and predicts object occlusion and discretized relative states with two separately calibrated heads.
| Model and readout | All Direct | All AUC | Public Direct | Public AUC |
|---|---|---|---|---|
| PLATO-b, box prediction | 0.498 | 0.533 | 0.498 | 0.526 |
| PLATO-relative, mean | 0.548 | 0.591 | 0.513 | 0.540 |
| PLATO-relative, max | 0.573 | 0.594 | 0.516 | 0.540 |
| SlotFormer-b, box prediction | 0.514 | 0.523 | 0.508 | 0.540 |
| SlotFormer-relative, mean | 0.564 | 0.637 | 0.527 | 0.560 |
| SlotFormer-relative, max | 0.588 | 0.644 | 0.520 | 0.558 |
| Full PULSE | 0.680 | 0.696 | 0.583 | 0.625 |
The adapted models improve both Direct and AUC on each backbone, and full PULSE scores higher on both metrics. This comparison tests the change in representation, targets, and calibration together. The fixed-window retraining in our earlier reply tested boundary placement within PULSE.
Summary and Event window and Standalone reliability (Part 1/2)
Summary
Dear Reviewer 5d26,
Thank you for the precise technical questions. The additional controls below preserve the main conclusions.
(W1; Q1) Adaptive boundaries improve detection within the event-based framework. With the same representations, network, objectives and calibration procedure, seed-42 fixed-window retraining reduces Direct/AUC from 0.680/0.696 to 0.608/0.630 on All and from 0.583/0.625 to 0.557/0.583 on Public.
(W2; Q2) PULSE's standalone advantage persists at matched validation video-FPR targets. At the 10% and 20% pooled validation targets, PULSE reaches All Direct of 0.612 and 0.669, respectively, versus 0.517 and 0.521 for the best baselines.
(W3; Q3) Recovering JEPA's paired performance does not yield strong standalone detection. Restoring original IntPhys frames raises V-JEPA Compare from 0.817 to 0.917, matching the reported level of the public checkpoint. Across both JEPA models and mean/max pooling, All Oracle Direct remains 0.514–0.536 even with test-label threshold selection.
(W4; Q4) Max retains its advantage at matched validation FPR, and predicted-box performance stays close to native states across both seeds. At the 10%/20% pooled validation targets, Max Direct is 0.612/0.669 versus 0.607/0.642 for the best alternative aggregation. Predicted boxes reduce Direct by only 3.8 and 2.9 percentage points at seeds 42 and 2026, respectively, compared with 19.6 and 22.2 points for predicted depth.
1. Adaptive boundary selection (W1; Q1)
The event perspective concerns how physical expectations are organized and checked; adaptive boundary selection is one implementation choice within it. We retrained with fixed windows to examine the benefit of alignment.
Window spans and Middle positions were fixed using plausible training/validation statistics, with spans chosen to approximate the adaptive parser's mean event count per video; no test-performance tuning was used. Fitted trajectories, state representation, network, objectives and calibration procedure were retained. Both variants use seed 42. Public recalibrates detection references using only its two datasets. The window settings are specified in revised Appendix D.2.
Both variants retain event-stage checks. Lower Direct and AUC under All and Public support the additional contribution of adaptive boundaries, while the fixed-window model's detection ability remains consistent with the broader event-based formulation.
2. Standalone reliability and IntPhys false positives (W2; Q2)
Thank you for the question about standalone reliability. Figure 4 and Table A2 (original and revised) show clear standalone detection gains over the evaluated continuous baselines. For the matched-FPR comparison, we adjust the four heads' common upper percentile to meet the pooled validation video-FPR targets (revised Appendix D.4):
PULSE remains ahead at both targets, supporting the reliability of the standalone detection gains across operating points.
The requested per-dataset validation and test rates are:
As you helpfully note, shared calibration alone does not explain IntPhys's high test FPR. At the 10% pooled validation target, IntPhys's own validation FPR is 17.77%; this dataset-level difference arises under shared calibration. Its further increase to 63.33% on test reflects a validation-to-test distribution difference. IntPhys test videos lack usable GT tracks and require SAM2 identity propagation (Table A11 (original) / Table A18 (revised); Appendix E.1). The resulting tracking and state-quality difference can raise prediction errors on plausible videos and may contribute to this additional gap. Revised Appendix D.4 distinguishes these effects and reports the per-dataset validation and test results. The matched-validation-FPR comparisons continue to support PULSE's standalone detection gains.