- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PAVE, a video encoder for learning physics-aware representations. It combines a 515K-clip corpus, causal future-feature prediction, a sparse MoE encoder with a shared expert, and physics VQA alignment with a baseline-adjusted cross-entropy loss (BACE). The authors report gains on MVBench, PhysBench, video generation, spatial reasoning, and robotic manipulation.
Critical strengths
- The problem is important for physical reasoning, video generation, and embodied AI.
- The paper evaluates the encoder across several downstream settings.
- The paper includes ablations for alignment, BACE, causal masking, initialization, and data scale.
Critical weaknesses
-
Incomplete attribution of gains. Table 3 isolates alignment and BACE gains within PAVE. However, baseline encoders receive no equivalent alignment stage. Their relative performance therefore does not isolate the contribution of PAVE’s physics-oriented pretraining.
-
Missing controls for two central design claims. Table 9 varies data quantity while preserving composition. It does not establish the value of physics-centric curation. The dense comparison also changes active parameters from 86M to 105M, leaving the benefit of expert routing unresolved.
-
The methodological advance remains unclear. The framework largely combines established components. BACE improves accuracy, but its distinctive benefit beyond PAVE is untested. Stronger controls would clarify the contribution of the proposed combination.
Questions for authors
-
Would a strong baseline encoder retain a performance gap after the same VQA alignment, with and without BACE? This would separate the benefit of the encoder from that of the alignment procedure.
-
Does PhysCentric outperform an equally sized generic-video corpus under matched training compute? The current scale ablation does not test the central data-curation claim.
-
Does MoE outperform a dense encoder with comparable active parameters and compute under identical training? This would clarify whether routing contributes beyond increased capacity.
3 comments
Response to Reviewer UN8f (Part 2/2)
Q2. Does PhysCentric outperform an equal-sized generic-video corpus under matched training compute?
We compare PhysCentric with a random OpenVid subset of the same size, using the same architecture, initialization rule, training budget, and downstream protocol.
| Pretraining corpus | Clips | PhysBench (%) ↑ | MVBench (%) ↑ |
|---|---|---|---|
| PhysCentric | 515,377 | 61.40 | 56.37 |
| Random OpenVid subset | 515,377 | 50.08 | 50.53 |
| PhysCentric − OpenVid (pp) | — | +11.32 | +5.84 |
Both conditions follow the same 30-epoch Stage I configuration. PhysCentric performs better on both benchmarks under this matched comparison.
Unlike the original scale ablation, this experiment changes corpus composition while holding clip count and training configuration fixed. It supports the corpus’s usefulness relative to the tested generic-video control, but does not separately identify the effects of source diversity, domain composition, or individual filtering operations.
Q3. Does MoE outperform a dense model with comparable active parameters and compute under identical training?
We add Dense105 to replace the original capacity-mismatched comparison as the main architectural control.
| Architecture | PhysBench (%) ↑ | MVBench (%) ↑ |
|---|---|---|
| Dense105 | 59.45 | 53.83 |
| MoE105 | 61.40 | 56.37 |
| MoE − Dense (pp) | +1.95 | +2.54 |
| Metric | Dense105 | MoE105 |
|---|---|---|
| Active parameters per token | 105.123M | 105.151M |
| Encoder matrix-operation FLOPs | 419.879 GFLOPs | 419.956 GFLOPs |
Computation is measured for 16 frames at 224 × 224 resolution, counting one MAC as two FLOPs. FLOPs cover the encoder’s major matrix operations.
Both models use the same training data, random seed, epochs, optimizer update count, effective batch size, optimization settings, and downstream protocol. Both undergo 30-epoch Stage I pretraining without BACE Stage II.
MoE performs better on both benchmarks with closely matched active capacity and matrix computation under identical training conditions. This supports an architectural benefit beyond the parameter-count difference in the original Dense86 comparison, without establishing that MoE is universally necessary.
I appreciate the authors for the additional experiments. They address most of my attribution concerns. However, my remaining concerns still lie:
- Limited novelty. The results are stronger, but the methodological contribution remains modest.
- Uncertain robustness. The gains over equally aligned baselines are small, with no uncertainty estimates.
- Incomplete connection to the final model. The data and architecture controls use 30-epoch pretraining. It remains unclear whether these gains persist after longer training and alignment.
The response has improved my assessment. I remain borderline, but am not yet convinced that the contribution is strong enough for acceptance.
Response to Reviewer UN8f (Part 1/2)
Thank you for identifying the controls needed to clarify attribution. We respond to each weakness (W) and question (Q) in the original order.
W1. Incomplete attribution because baseline encoders lack equivalent alignment
We agree that comparing aligned PAVE only with unaligned encoders does not isolate the encoder contribution. We therefore directly apply physics-QA Stage II to four pretrained baseline encoders, updating each encoder and its projector. They do not first undergo PAVE Stage I.
CE and BACE use equivalent alignment budgets and the same downstream adaptation protocol. The complete results are reported under Q1 below. They allow us to distinguish within-encoder alignment gains from comparisons between encoders.
W2. Missing controls for data composition and MoE capacity
We add both requested controls: an equal-sized OpenVid comparison and a Dense105 model with closely matched active parameters and encoder matrix-operation FLOPs. The results and controlled settings are reported under Q2 and Q3, respectively.
These comparisons address different factors: Q2 changes the pretraining corpus while keeping the model and training configuration fixed; Q3 changes the architecture under the same data and training protocol.
W3. Unclear methodological contribution and untested BACE benefit beyond PAVE
BACE trains visual features to provide additional evidence beyond a frozen VLM by correcting answer logits against the baseline. Its benefit extends beyond PAVE’s architecture. We demonstrate this by comparing BACE with CE on four pretrained encoders under matched training conditions:
The physical-QA improvement occurs across every tested encoder, supporting BACE as a transferable alignment objective. The DINOv3 decrease on MVBench also establishes a boundary: physics-supervised BACE does not uniformly improve all downstream tasks. Together with the corpus and architecture controls below, these results provide more specific attribution than the original comparisons.
Q1. Does PAVE retain an advantage after equivalent VQA alignment, with and without BACE?
We compare all encoders under unaligned, CE-aligned, and BACE-aligned conditions:
“Unaligned” means no Stage II, not zero-shot evaluation. PAVE and unaligned baseline results are from the paper; aligned baseline results are additional experiments following the same protocol.
The answer depends on the alignment objective. Under BACE, PAVE retains the highest accuracy on both benchmarks. Under CE, VideoMAE V2-L exceeds PAVE on PhysBench, while PAVE leads on MVBench. We will report this distinction rather than claiming that PAVE is strongest under every alignment condition.
Equivalent alignment reduces the supervision confound, but differences in encoder capacity and historical pretraining remain. The architecture and corpus controls below address complementary aspects of attribution.