acceptodds
Under review as a conference paper at ICLR 2027

Atomic Physical Transitions: A Structured Representation for Physical Dynamics in Video

Abstract

Current video understanding often compresses multiple physical processes into clip-level labels or answers, allowing correct event-level predictions without explicitly recovering the underlying state changes. These processes can instead be represented as compositions of reusable causal units—physical state transitions defined by the mechanisms that drive them rather than by scene appearance. We introduce Atomic Physical Transitions (APTs): minimal, temporally localized state changes that bind visual evidence to an active mechanism and a before/after physical state, forming an ordered transition chain for each video. For diagnosis and supervision, we construct APT-BENCH, covering 14 transition types across contact, gravity, friction, and rotation/stability, with approximately 55,000 timed instances over 2,500 trials. A human-anchored, rule-guided simulation pipeline scales annotation using realized state traces rather than intended scene templates. Evaluation of twelve vision-language models reveals low zero-shot transition recovery, and a denser-input check shows that the recovery gap persists. APT-only fine-tuning improves detection but can degrade event-level answering. We therefore introduce APT-Tune, a parameter-efficient recipe combining multimodal answer-only masking with format-conditional co-training across APT, multiple-choice, and description targets. Across six open-source backbones, it substantially improves transition recovery while retaining event-level performance on MVBench. APT-Tune consistently improves performance on PhysBench across the evaluated backbones. These results support APTs as an intermediate representation between coarse event recognition and full physical simulation, useful for both diagnosis and transferable supervision. The project page is at https://apt-physical-transitions-2027.netlify.app/

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Physical Video Understanding, Atomic Physical Transitions, Vision-Language Models, Physics-Aware Video Reasoning, Parameter-Efficient Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gsS1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on physical understanding of videos. Current VLMs may correctly understand a physical statement of an object, but cannot understand the full trajectory of object physical statement transition. To push current VLM physics understanding to fine-grained atomic physical statement, authors propose APT-Bench. Extensive zero shot evaluations including both commercial and open-source models show that current models suffer from this fine grained physics understanding. Authors further post-train open-source models and can make some improvement on this task.

Critical strengths
  1. The motivation of this paper is clear, It focuses on evaluating fine-grained physical state transitions within video events, which are often overlooked by existing physical video understanding benchmarks.

  2. The study is relatively comprehensive. This paper includes APT definition, Benchmark building, and zero shot evaluation on both commercial and public models, and post-training on open-source models.

Critical weaknesses
  1. I am concerned about the claimed contribution of evaluating causal physical understanding. By defining fine-grained atomic physical state transitions and checking the correctness of model predictions, the current evaluation essentially measures transition type classification and temporal localization. It can be viewed as another temporal event localization task, just replacing atomic actions with atomic physical state transitions. The current APT-Bench evaluation does not explicitly assess causality. Although the authors acknowledge this limitation, I think this conceptual concern needs to be addressed during rebuttal: either reframe APT-Bench as a fine-grained, mechanism-typed temporal localization benchmark for physical state transitions, or add an explicit causality evaluation that checks the causal relationships between physical state transitions, as well as the structured representation of physical mechanisms, before/after states, and visual evidence, instead of the current evaluation, which only focuses on transition types and timestamps.

  2. A natural follow-up question is how to ensure that the physical mechanisms covered by APT-Bench are visually identifiable. What if the same observed motion transition could be explained by multiple different physical mechanisms? Does APT-Bench suffer from such ambiguity? Does APT-Bench consistently measure visually identifiable physical transitions, or does it sometimes require information that is only accessible through simulator state traces?

Questions for authors

Please see the critical weaknesses section.

2 comments

Thank you for the two conceptual questions. Both are answered below, with new results for each. W1. What APT-Bench measures. We adopt your framing: APT-Bench is a fine-grained, mechanism-typed temporal localization benchmark for physical state transitions, the scope stated in Sec. 3.2. Wording changes:

  • Abstract: "reusable causal units" → "reusable mechanism-typed units".
  • Sec. 2: "transition-level causal supervision" → "transition-level supervision".
  • Sec. 3.1 and App. B.1: "causal process" → "sequence of physical state changes".
  • Table 10: "Causal interpretation" → "Physical interpretation".

Each type is one mechanism family plus one before/after state pair (Table 2), so a correct type and timestamp fix the mechanism, the state change and the time. A new entity-aware track on the simulator test split also requires the correct object or object pair: recall is 6.8% zero-shot, 23.1% for the submitted APT-Tune model and 31.0% with entity targets.

Difference from action localization.

  1. The unit is a physical regime change, defined by mechanism and state change independently of object, material and agent (Sec. 3.4). Several transitions occur inside one action label (App. A.3).
  2. Targets are dense and brief: 20.4 transitions per 8.46 s clip, each scored within ±200 ms (Table 3).
  3. Current VLMs lack this ability: fourteen models, including GPT-5, Gemini 3.1, Claude Sonnet 5 and open models up to 32B, recover 10.0–15.2% zero-shot. 6 of the 14 types have zero zero-shot recall on Qwen3-VL-2B (Table 14).
  4. The supervision transfers: PhysBench improves by 9.8–14.2 points on all eight backbones without PhysBench data. On the same 1,000 videos, APT chains reach 50.8, against 43.5 and 46.1 for object-description and event-level-QA targets.

W2. Visual identifiability.

  1. Observable states. Each type is a change between coarse visible states (Table 2). Hidden quantities (mass, force magnitude, friction, restitution) are excluded (App. B.5), and a transition requires a visible cue over at least three native frames (App. B.3).
  2. Existing human labels. All 14 types are annotated by humans who see only the video: The data includes 27,303 transitions annotated by two independent annotators and a third adjudicator (Sec. 4). They make up 5,551 of the 10,200 test APTs and cover all 14 types, so no type requires simulator state.
  3. Filtered trace labels. A trace event becomes a label only when the change is visible in the rendered video (App. C.10, Listing 14).
  4. Ambiguity. Table 8 resolves competing mechanisms by the visible before-state: an object that stops on impact is inelastic_capture; one that decelerates while sliding is sliding_arrest.

New blind audit (additional annotation; benchmark labels unchanged). Two annotators who see only the video independently label the same 40 simulator test clips (744 trace-derived APTs). Under the benchmark's matching rule, they recover 89.2% of the trace-derived labels (90.1% and 88.3%), and 93.4% of their labels have a trace-derived counterpart. Median timing difference: 33 and 67 ms. Labels admitting a second plausible type: 4.1%.

Recall by label source (existing predictions of the Table 5 models). Human-labeled APTs (5,551): 8.0% → 38.2%. Simulator APTs (4,649): 12.4% → 38.0%. The tuned model, which sees only the video, recovers trace-derived labels at the same rate as human labels.

In the revision, we report the new blind audit in App. C.10 and the per-source results in Sec. 6.

Thanks for the response. Authors addressed my concerns and I maintain my positive rating.

Reviewer_69nEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Atomic Physical Transitions (APTs), which represent a physical event in video as an ordered chain of timestamped state changes, each typed by its physical mechanism. The authors define 14 transition types across contact, gravity, friction and rotation. They build APT-Bench, which combines human annotations on CLEVRER and Physion++ with simulator-generated videos, for about 55,000 transitions in total. Twelve VLMs recover these transitions poorly in the zero-shot setting. The authors then propose APT-Tune, a LoRA fine-tuning recipe that mixes APT targets with SSv2 question answering and description data. It substantially improves transition recovery on six open-source backbones and also improves results on PhysBench while keeping MVBench performance.

Critical strengths

The paper proposes a well-motivated representation that sits between clip-level event labels and full physical simulation, and grounds it in work on event cognition and physics expertise. The taxonomy is carefully defined, with clear rules for separating similar types and detailed examples in the appendix. The data pipeline is thoughtfully designed: simulator labels are derived from the dynamics that actually occur rather than from the intended templates, and failed simulations are filtered out. The experiments are broad. They cover twelve zero-shot models, including several commercial ones, and six fine-tuned backbones across two model families and three sizes. The paper also includes useful controls and analyses, namely a prediction-count control, per-type results, a frame-budget study with a timestamp coverage analysis, and transfer tests on MVBench and PhysBench.

Critical weaknesses

W1. The test set lacks detailed distribution statistics. Table 3 and Table 11 report counts by source, and Fig. 3 shows overall type coverage. However, the number of instances per type in the test split is not reported, nor is its breakdown by source and domain. App. D.3 notes that per-type differences remain descriptive for this reason.

W2. The transfer gains cannot yet be attributed to the APT structure. The PhysBench gains of 11 to 14 points come from a recipe that adds 1,000 physical videos with APT targets. As the authors note in Sec. 6.3 and App. D.4, the SSv2-only baseline changes both the training data and the supervision targets. Matched-video supervision controls are therefore needed to separate the effect of the APT structure from the effect of simply seeing more physical videos. Without such a control, it is hard to tell how much of the transfer comes from the representation itself.

W3. The evaluation ignores which objects undergo each transition. Eq. 1 defines an APT with a mechanism, a before and after state, and visual evidence, and Sec. 3.4 states that each label is assigned to a specific entity or interacting pair. However, the evaluation prompt in Listing 16 asks only for type and time, and Sec. 3.2 confirms that entity identity is not checked. In scenes where several objects collide at similar times, a prediction can be matched even if the model misidentifies the objects involved. This weakens the claim that APT chains capture the structure of physical events, especially since the simulator traces already record the objects for each transition.

W4. The rules for co-occurring labels appear inconsistent. Sec. 3.4 states that each instance receives one label, but that different changes may share a timestamp, for example contact_init and landing at an impact. Table 8 and App. B.6, however, present landing and contact_init as alternatives, using landing when a falling object becomes supported. B.6 also says annotators “may” mark contact_init in this case. In contrast, the simulator pipeline labels both at the same moment, as in Listing 9, and Listing 14 treats a missing landing as an error to fix. Human-labeled and simulator-labeled data may therefore follow different conventions, which would affect recall. The paper does not report how often such co-occurring labels appear.

W5. The selection of APT training data is not described. The training split contains 2,000 trials, but APT-Tune uses only 1,000 of them, described as “APT-BENCH train + Isaac Lab generated” in App. C.13. The paper does not explain how these 1,000 trials were selected, how they are distributed across sources.

Questions for authors

Q1. On distribution statistics. Could the authors provide a table of per-type counts in the test split, broken down by source and domain?

Q2. On matched-video supervision. I would appreciate a matched-video control, even on a subset. For example, the authors could train Qwen3-VL-2B on the same 1,000 APT videos, but with non-APT targets such as the object descriptions or event-level QA already exported in App. C.11, while keeping the SSv2 data unchanged. These targets should not describe the transitions themselves. Comparing APT recall, PhysBench and MVBench under this setting would show how much of the gain comes from the APT structure.

Q3. On entity-aware evaluation. Could the authors clarify whether the human annotations record the objects involved in each transition, and whether the APT training targets include them? It would be helpful to report an entity-aware evaluation, in which a prediction must also name the correct object or object pair.

Q4. On co-occurring labels. Could the authors state the exact rule for when contact_init and landing, or other pairs, are labeled together, and confirm whether human annotators and the simulator parser follow the same rule? Reporting the fraction of co-occurring labels in each source would also help.

Q5. On the training subset. How were the 1,000 APT training trials selected.

3 comments

Thank you for the close reading of the appendix. Each of the five points is addressed below with new data.

W1/Q1. Per-type test statistics. Table R3 counts the existing test labels (human for CLEVRER and Physion++, trace-derived for the simulator). No label was added or changed.

Table R3. Test APTs by type, domain and source (500 trials).

TypeCLEVRERPhysion++SimulatorAll
Contact & Collision1,0192,6961,2154,930
contact_init4721,1927772,441
contact_break4721,1241601,756
elastic_rebound75361146582
inelastic_capture019132151
Gravity & Projectile02909091,199
free_fall_onset014208222
projectile_apex0194261
landing0257659916
Friction, Surface & Medium476811,1581,886
static_breakaway0136402538
sliding_arrest47332465844
surface_transition0213291504
Rotation & Stability1346841,3672,185
rotation_onset89275180544
toppling088687775
rotation_arrest45212493750
settling01097116
Total1,2004,3514,64910,200

Every type has at least 61 test instances. Table R3 has been added to App. C.12, and Table 14 now lists the count of each type.

W2/Q2. Matched-video control. We ran the control you proposed on Qwen3-VL-2B: same 1,000 APT videos, SSv2 data and budget; only changing target on the 1,000 videos. A free text field of transition prose, rather than object descriptions and event-level QA is used as an extra variant to describe the transitions. Table R2. Matched-video control on Qwen3-VL-2B (%). †: new. Other entries: Tables 5, 7, 12 and 15.

Target on the 1,000 physical videosAPT recallPhysBench AvgDynamicsMVBench Avg
Zero-shot, no training10.038.538.251.4
SSv2 only, no physical videos11.640.239.8†52.4†
Object description† (no motion, contact or state change)13.543.543.952.0
Event-level QA† (no transition types or timestamps)16.446.149.353.2
Transition prose† (the chain as free text, no types or timestamps)31.347.653.253.3
APT chain (APT-Tune)38.150.856.753.6
  • Seeing the videos: +3.3 (object description) and +5.9 (event-level QA) PhysBench points over SSv2-only training.
  • The APT structure: APT chains add 10.6. The APT target thus accounts for 4.7–7.3 of the 10.6 PhysBench points, 7.4–12.8 of the 16.9 Dynamics points and 21.7–24.6 of the 26.5 recall points.
  • Content vs. structure: transition prose reaches 47.6 on PhysBench and 31.3% recall; the typed, timestamped structure adds 3.2 and 6.8 points.

MVBench is preserved in every variant. The results from Table R2 help us strengthen the claim against the caveat from the existing results Sec. 6.3 and App. D.4.

W3/Q3. Entity identity.

  • Annotations. An `objects’ field records both sources, the object or object pair of each transition: clip-local identifiers for human labels, the simulator trace for simulator labels (App. C.8, Listing 9). The original metric does not use it.
  • Training targets. Only type and t_ms (App. C.12, C.13). The objects field of Listing 15 is not part of them.
  • New entity-aware track. On the simulator test split (250 trials, 4,649 APTs), a prediction counts only if type, time (±200 ms) and object or object pair all match.

Table R4. Entity-aware evaluation, simulator test split, Qwen3-VL-2B (%). Rows 1–2: new evaluation of existing models. Row 3: new training run.

ModelRecall (type + time)Recall (type + time + entity)Entity accuracy among type-and-time matches
Zero-shot12.46.854.8
APT-Tune, submitted targets (type + time)38.023.160.8
APT-Tune, targets with entities39.331.078.9

The gain holds when entities are required: 6.8% → 23.1% without entity supervision, and 31.0% with entity targets (entity accuracy 78.9%, type-and-time recall retained). The revision adds the track, releases the entity fields and reports both metrics.

W4/Q4. Co-occurring labels. The wording of Sec. 3.4, Table 8 and App. B.6 was not aligned; thank you for catching this. All three now state one rule:

A change in one state variable of an entity or pair (contact, support, sliding, rotation, stability) receives exactly one label, the most specific type. Two labels share a timestamp only when two different state variables change at that moment, or when they belong to different entities.

  • A falling object touches a surface and stays supported: contact_init (contact) and landing (support), same timestamp.
  • A lateral contact without a support change: contact_init only.

Human annotators and the simulator parser follow the same rule: in both sources, every landing has a co-timed contact_init.

Table R5. Co-occurring labels in the existing test labels (same entity or pair, timestamps within one native frame).

SourceTest APTsAPTs in co-timed groupslanding with a co-timed contact_init
CLEVRER (human)1,200120 (10.0%)no landing
Physion++ (human)4,351640 (14.7%)257 / 257 (100%)
Simulator (trace)4,6491,500 (32.3%)659 / 659 (100%)
All10,2002,260 (22.2%)916 / 916 (100%)

The simulator's higher share reflects its larger number of landings (659 against 257 in Physion++).

W5/Q5. Training subset. The 1,000 trials are a proportional stratified sample of the 2,000-trial training split (stratum: source; seed 42; without replacement): 200 CLEVRER, 298 Physion++ and 502 simulator trials, matching the pool (400, 596 and 1,004). Each trial contributes one APT-chain example, alongside 1,000 SSv2 multiple-choice and 1,000 SSv2 description records (1:1:1, App. C.13). No test trial is included. The revision states this in App. C.13 and releases the trial IDs.

Thanks for the response. Authors addressed my concerns and I raise my score to 3.

Reviewer_TLmjOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Atomic Physical Transitions (APTs), a structured representation that decomposes physical events in videos into temporally localized state transitions, such as contact, rebound, loss of support, and sliding arrest.

The main contributions include:

  1. Defining 14 types of APTs across four physical domains. Each transition is associated with a timestamp, physical mechanism, transition type, before/after state change, and local visual evidence.
  2. Constructing APT-Bench with approximately 55K annotated transitions across 2,500 video trials, and introducing APT-Tune to incorporate transition-level supervision into VLM training.
  3. Evaluating the approach across six VLM backbones, demonstrating substantial improvements in APT recovery and transfer to physical understanding benchmarks.

In particular, APT-Tune improves PhysBench by 11.1–14.2 percentage points without using PhysBench training data, while maintaining or slightly improving general video understanding performance on MVBench.

Critical strengths
  1. Interesting and meaningful representation for physical video understanding. The idea of decomposing physical events into reusable, temporally localized state transitions is interesting. Compared with event-level labels, APT provides a more structured way to represent the underlying physical processes and could be useful for physical reasoning and embodied intelligence.

  2. Promising experimental results and transferability. The paper provides relatively comprehensive evaluations across six VLM backbones. The improvements on PhysBench are particularly encouraging, with gains exceeding 11 percentage points across all evaluated models and 16.6–20.3 points on the Dynamics subtask. Since PhysBench is not used for training, these results provide useful evidence that the training benefits extend beyond APT-Bench. The results on MVBench also suggest that the proposed training recipe can preserve general video understanding capabilities.

Critical weaknesses
  1. Limited validation at larger model scales.

    • The experiments are conducted on models up to 8B parameters.
    • It remains unclear whether the proposed APT supervision provides similar improvements for substantially larger and stronger VLMs.
    • Given the importance of scaling in current multimodal foundation models, evaluating the approach on stronger backbones would make the conclusions more convincing.
  2. The contribution of APT representation is not sufficiently isolated from the training data and recipe.

    • The improvements may come from the structured APT representation, additional physical video training data, or the mixed-format SFT recipe.
    • Although the paper includes ablation studies, the current experiments do not fully distinguish these factors.
    • A controlled comparison using the same physical videos and training budget, but different supervision formats, would better establish the unique value of APT representation.
Questions for authors
  1. Scaling to stronger VLMs: Have the authors evaluated APT-Tune on larger models beyond 8B? Does the improvement in physical understanding remain significant as the baseline model becomes stronger?

  2. Isolating the contribution of APT representation: Can the authors compare APT supervision with conventional physical video descriptions or event-level QA supervision using exactly the same videos, data scale, and training settings? This would help clarify whether the improvements come specifically from the proposed atomic transition representation or from additional physical video training data.

  3. Understanding the transfer gains: The improvements on PhysBench are substantial, while the gains on MVBench are relatively modest. Could the authors provide further analysis of which types of physical reasoning benefit most from APT supervision and whether these improvements generalize beyond the current mechanics-focused taxonomy?

1 comment

Thank you for the positive assessment. Each of your three questions is answered below with new results. W1/Q1. Larger backbones. The improvements transfer to larger VLMs. Two new models are fine-tuned with the unchanged recipe and evaluated with the protocol of Tables 4, 5, 7 and 12.

Table R1. Zero-shot → +APT-Tune (gain). 8B rows: from the submission. 32B and 14B rows: new.

BackboneAPT recall (%)PhysBench Avg (%)MVBench Avg (%)
Qwen3-VL-8B12.0 → 48.6 (+36.6)45.0 → 56.6 (+11.6)56.8 → 57.6 (+0.8)
InternVL3.5-8B13.0 → 53.4 (+40.4)45.2 → 59.4 (+14.2)57.1 → 58.3 (+1.2)
Qwen3-VL-32B12.3 → 58.0 (+45.7)52.5 → 62.3 (+9.8)61.0 → 62.2 (+1.2)
InternVL3.5-14B13.8 → 57.3 (+43.5)50.4 → 61.1 (+10.7)59.3 → 60.4 (+1.1)
  • The recall gain grows with scale. Qwen3-VL (2B, 4B, 8B, 32B): +28.1, +31.5, +36.6, +45.7. InternVL3.5 (2B, 4B, 8B, 14B): +32.8, +35.0, +40.4, +43.5 (Table 5).
  • PhysBench transfer holds: +9.8 and +10.7 (Table 7). The two new models are the best of the eight on both APT recall (58.0, 57.3) and PhysBench (62.3, 61.1).
  • Scale alone does not help. Zero-shot recall is 12.3% at 32B and 13.8% at 14B, against 10.0% and 14.0% at 2B. The tuned 2B model (38.1%) exceeds every zero-shot system by at least 22.9 points.

W2/Q2. Representation, videos, or recipe. We ran the comparison you describe on Qwen3-VL-2B: same 1,000 physical videos, SSv2 records, budget and LoRA configuration; only the target on the physical videos changes.

Table R2. Matched-video control on Qwen3-VL-2B (%). †: new. Other entries: Tables 5, 7, 12 and 15.

Target on the 1,000 physical videosAPT recallPhysBench AvgDynamicsMVBench Avg
Zero-shot, no training10.038.538.251.4
SSv2 only, no physical videos11.640.239.8†52.4†
Object description† (no motion, contact or state change)13.543.543.952.0
Event-level QA† (no transition types or timestamps)16.446.149.353.2
Transition prose† (the chain as free text, no types or timestamps)31.347.653.253.3
APT chain (APT-Tune)38.150.856.753.6
  • Three factors. Of the 12.3-point PhysBench gain: recipe on SSv2 alone, 1.7; physical videos with conventional targets, a further 3.3–5.9; APT chains on the same videos, a further 4.7–7.3. Format mixing: APT chains alone reach 44.7 (Table 15).
  • Structure. The margin over the conventional targets is larger on Dynamics (7.4–12.8) and on APT recall (21.7–24.6). Typed, timestamped chains add 3.2 points over transition prose.
  • MVBench is preserved in every variant. Table R2 replaces the caveat in Sec. 6.3 and App. D.4.

Q3. Transfer analysis.

  • MVBench is a retention check. APT-only tuning lowers SSv2 accuracy (72.5 → 68.5); the full recipe keeps MVBench at or above zero-shot on all eight backbones (+0.5 to +2.2).
  • Dynamics benefits most: the largest gain on all eight backbones (+15.7 to +20.3), ahead of Property (+6.0 to +13.3) and Relationships (+4.9 to +11.9) (Table 7). On Qwen3-VL-8B, collision (+21.1), manipulation (+20.5), throwing (+17.7) and motion (+17.5) lead (Table 13). In Table R2, APT chains lead event-level QA by 7.4 points on Dynamics, against 4.7 on the average.
  • Beyond mechanics: fluid, light, chemistry, temperature and air change by −0.3 to +0.2 (Table 13). Transfer follows the supervised content. Extending the vocabulary adds trace rules to the same pipeline (App. B.11); the simulator already supports garment, fluid and granular assets (App. C.3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.