acceptodds
Under review as a conference paper at ICLR 2027

PAVE: Learning Physics-Aware Visual Representations from Videos

Abstract

Physical knowledge is essential for physical reasoning, video generation, and robotic manipulation. However, general-purpose visual encoders primarily learn semantic and spatial features, with limited emphasis on physical properties and interactions. We introduce PAVE, a vision encoder that learns transferable physical representations from videos. For data, we curate PhysCentric, a corpus of 515,377 unlabeled video clips covering diverse physical interactions. For architecture, we combine sparsely routed experts with a shared expert to capture specialized and common physical cues. For training, we first learn physical dynamics through causal future-feature prediction, then apply Physics Knowledge Alignment using physical question-answer supervision and Baseline-Adjusted Cross-Entropy (BACE) to support physical reasoning. PAVE achieves 58.93% on MVBench and 70.04% on PhysBench, surpassing the strongest compared encoders by 3.08 and 2.01 percentage points, respectively. Downstream gains over the respective baselines include 9.7 percentage points in VideoPhy semantic adherence, 6.4 points in VSI-Bench average score, and 3.2 percentage points in average success rate across five RoboTwin 2.0 tasks, which shows the strong transferability. The data, code, and model will be publicly available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Physics-aware representation learning, Video representation learning, Self-supervised learning, Predictive representation learning, Mixture-of-experts, Physical reasoning, Multimodal learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UN8fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PAVE, a video encoder for learning physics-aware representations. It combines a 515K-clip corpus, causal future-feature prediction, a sparse MoE encoder with a shared expert, and physics VQA alignment with a baseline-adjusted cross-entropy loss (BACE). The authors report gains on MVBench, PhysBench, video generation, spatial reasoning, and robotic manipulation.

Critical strengths
  • The problem is important for physical reasoning, video generation, and embodied AI.
  • The paper evaluates the encoder across several downstream settings.
  • The paper includes ablations for alignment, BACE, causal masking, initialization, and data scale.
Critical weaknesses
  1. Incomplete attribution of gains. Table 3 isolates alignment and BACE gains within PAVE. However, baseline encoders receive no equivalent alignment stage. Their relative performance therefore does not isolate the contribution of PAVE’s physics-oriented pretraining.

  2. Missing controls for two central design claims. Table 9 varies data quantity while preserving composition. It does not establish the value of physics-centric curation. The dense comparison also changes active parameters from 86M to 105M, leaving the benefit of expert routing unresolved.

  3. The methodological advance remains unclear. The framework largely combines established components. BACE improves accuracy, but its distinctive benefit beyond PAVE is untested. Stronger controls would clarify the contribution of the proposed combination.

Questions for authors
  1. Would a strong baseline encoder retain a performance gap after the same VQA alignment, with and without BACE? This would separate the benefit of the encoder from that of the alignment procedure.

  2. Does PhysCentric outperform an equally sized generic-video corpus under matched training compute? The current scale ablation does not test the central data-curation claim.

  3. Does MoE outperform a dense encoder with comparable active parameters and compute under identical training? This would clarify whether routing contributes beyond increased capacity.

3 comments
AuthorsOpenReview ↗

Response to Reviewer UN8f (Part 1/2)

Thank you for identifying the controls needed to clarify attribution. We respond to each weakness (W) and question (Q) in the original order.

W1. Incomplete attribution because baseline encoders lack equivalent alignment

We agree that comparing aligned PAVE only with unaligned encoders does not isolate the encoder contribution. We therefore directly apply physics-QA Stage II to four pretrained baseline encoders, updating each encoder and its projector. They do not first undergo PAVE Stage I.

CE and BACE use equivalent alignment budgets and the same downstream adaptation protocol. The complete results are reported under Q1 below. They allow us to distinguish within-encoder alignment gains from comparisons between encoders.

W2. Missing controls for data composition and MoE capacity

We add both requested controls: an equal-sized OpenVid comparison and a Dense105 model with closely matched active parameters and encoder matrix-operation FLOPs. The results and controlled settings are reported under Q2 and Q3, respectively.

These comparisons address different factors: Q2 changes the pretraining corpus while keeping the model and training configuration fixed; Q3 changes the architecture under the same data and training protocol.

W3. Unclear methodological contribution and untested BACE benefit beyond PAVE

BACE trains visual features to provide additional evidence beyond a frozen VLM by correcting answer logits against the baseline. Its benefit extends beyond PAVE’s architecture. We demonstrate this by comparing BACE with CE on four pretrained encoders under matched training conditions:

EncoderBACE − CE on PhysBench (pp)BACE − CE on MVBench (pp)
PAVE+3.00+2.50
DINOv3-L+5.98−1.68
V-JEPA 2.1-L+1.82+0.61
SigLIP 2-L+1.52+2.60
VideoMAE V2-L+0.83+2.61

The physical-QA improvement occurs across every tested encoder, supporting BACE as a transferable alignment objective. The DINOv3 decrease on MVBench also establishes a boundary: physics-supervised BACE does not uniformly improve all downstream tasks. Together with the corpus and architecture controls below, these results provide more specific attribution than the original comparisons.

Q1. Does PAVE retain an advantage after equivalent VQA alignment, with and without BACE?

We compare all encoders under unaligned, CE-aligned, and BACE-aligned conditions:

EncoderAlignmentPhysBench (%) ↑MVBench (%) ↑
PAVEUnaligned64.8552.65
PAVECE67.0456.43
PAVEBACE70.0458.93
DINOv3-LUnaligned60.9055.85
DINOv3-LCE61.6052.23
DINOv3-LBACE67.5850.55
V-JEPA 2.1-LUnaligned59.0553.83
V-JEPA 2.1-LCE64.9454.72
V-JEPA 2.1-LBACE66.7655.33
SigLIP 2-LUnaligned55.5851.88
SigLIP 2-LCE58.2653.98
SigLIP 2-LBACE59.7856.58
VideoMAE V2-LUnaligned68.0352.92
VideoMAE V2-LCE68.3654.72
VideoMAE V2-LBACE69.1957.33

“Unaligned” means no Stage II, not zero-shot evaluation. PAVE and unaligned baseline results are from the paper; aligned baseline results are additional experiments following the same protocol.

The answer depends on the alignment objective. Under BACE, PAVE retains the highest accuracy on both benchmarks. Under CE, VideoMAE V2-L exceeds PAVE on PhysBench, while PAVE leads on MVBench. We will report this distinction rather than claiming that PAVE is strongest under every alignment condition.

Equivalent alignment reduces the supervision confound, but differences in encoder capacity and historical pretraining remain. The architecture and corpus controls below address complementary aspects of attribution.

AuthorsOpenReview ↗

Response to Reviewer UN8f (Part 2/2)

Q2. Does PhysCentric outperform an equal-sized generic-video corpus under matched training compute?

We compare PhysCentric with a random OpenVid subset of the same size, using the same architecture, initialization rule, training budget, and downstream protocol.

Pretraining corpusClipsPhysBench (%) ↑MVBench (%) ↑
PhysCentric515,37761.4056.37
Random OpenVid subset515,37750.0850.53
PhysCentric − OpenVid (pp)—+11.32+5.84

Both conditions follow the same 30-epoch Stage I configuration. PhysCentric performs better on both benchmarks under this matched comparison.

Unlike the original scale ablation, this experiment changes corpus composition while holding clip count and training configuration fixed. It supports the corpus’s usefulness relative to the tested generic-video control, but does not separately identify the effects of source diversity, domain composition, or individual filtering operations.

Q3. Does MoE outperform a dense model with comparable active parameters and compute under identical training?

We add Dense105 to replace the original capacity-mismatched comparison as the main architectural control.

ArchitecturePhysBench (%) ↑MVBench (%) ↑
Dense10559.4553.83
MoE10561.4056.37
MoE − Dense (pp)+1.95+2.54
MetricDense105MoE105
Active parameters per token105.123M105.151M
Encoder matrix-operation FLOPs419.879 GFLOPs419.956 GFLOPs

Computation is measured for 16 frames at 224 × 224 resolution, counting one MAC as two FLOPs. FLOPs cover the encoder’s major matrix operations.

Both models use the same training data, random seed, epochs, optimizer update count, effective batch size, optimization settings, and downstream protocol. Both undergo 30-epoch Stage I pretraining without BACE Stage II.

MoE performs better on both benchmarks with closely matched active capacity and matrix computation under identical training conditions. This supports an architectural benefit beyond the parameter-count difference in the original Dense86 comparison, without establishing that MoE is universally necessary.

I appreciate the authors for the additional experiments. They address most of my attribution concerns. However, my remaining concerns still lie:

  • Limited novelty. The results are stronger, but the methodological contribution remains modest.
  • Uncertain robustness. The gains over equally aligned baselines are small, with no uncertainty estimates.
  • Incomplete connection to the final model. The data and architecture controls use 30-epoch pretraining. It remains unclear whether these gains persist after longer training and alignment.

The response has improved my assessment. I remain borderline, but am not yet convinced that the contribution is strong enough for acceptance.

Reviewer_ADFMOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on learning physics-aware visual representations from videos. It argues that models such as DINOv3, SigLIP2 do not explicitly prioritize physical properties and interactions during pretraining, even though their features may contain physical information. To address this, the authors introduce PAVE and build PhysCentric, a dataset containing 515,377 video clips from nine sources. The dataset combines real-world videos, robot interactions, and simulations. The filtering process removes duplicate and invalid clips, excludes clips with too few frames, and reduces repeated content.

PAVE uses a mixture-of-experts architecture and two training stages. The first stage uses causal future-feature prediction to learn representations of physical dynamics. The second stage aligns these representations with physics question-answer supervision while keeping the VLM backbone frozen. The paper also introduces Baseline-Adjusted Cross-Entropy (BACE), which encourages the added PAVE features to improve answer predictions beyond a frozen baseline. Under the reported evaluation protocol, PAVE outperforms the compared encoders on MVBench and PhysBench and improves performance on several downstream tasks.

Critical strengths
  1. PAVE achieves better scores on MVBench and PhysBench than the compared encoders while using fewer active visual-encoder parameters.
  2. The authors demonstrate the effectiveness of PAVE on two downstream applications, video generation and spatial reasoning, and show consistent gains over backbone models.
  3. BACE is an interesting training objective. Its idea of encouraging additional features to improve predictions beyond a frozen baseline may also be useful for tasks outside physical understanding.
Critical weaknesses
  1. The dataset combines nine sources, but the paper does not study how much each source contributes to the results. The experiment in Table 9 compares the full dataset with a smaller subset while keeping the source proportions unchanged. This shows the effect of data scale, but it does not explain which sources are most useful or whether some sources contribute little.
  2. The filtering process mainly checks data validity, repeated content, and the number of available frames. It is unclear whether this is sufficient to ensure that the retained videos provide reliable observations of physical processes. Videos from sources such as WISA and OpenVid may contain edited or reversed sequences that introduce physically implausible motion. The described filtering does not clearly address these cases. Such clips could remain in PhysCentric, but the paper does not estimate their frequency or study their effect on learning physical representations.
  3. I find the novelty of the predictive pretraining stage relatively limited, since its main idea builds on existing feature-prediction methods such as V-JEPA. The combination of causal prediction, SIGReg, and sparse experts is reasonable, and the ablations support some of these choices. However, it remains unclear how much this specific pretraining stage and architecture add beyond applying the same Physics Knowledge Alignment stage to an existing pretrained encoder. A direct comparison would help clarify the need for training PAVE from scratch.
Questions for authors
  1. What happens if you apply Stage II directly to pretrained V-JEPA 2.1 or DINOv3 encoders, updating the encoder and projector with the same QA supervision and BACE objective? Do these variants perform worse than PAVE under the same evaluation protocol? Table 4 studies initialization followed by PAVE pretraining, which is different from this comparison. This experiment would help clarify the contribution of Stage I and the proposed architecture.
  2. Could PAVE be used in a VLM-as-a-judge setup to score the physical quality of generated videos, for example on a scale from 1 to 5? Have you tested whether these scores can improve best-of-N video selection on a benchmark such as Physics-IQ?
  3. Do you have an estimate of how many clips in PhysCentric contain physically implausible motion or editing that makes them unsuitable for learning physical dynamics? Was any manual or automatic check performed for these cases?
  4. Beyond the reported downstream evaluations, have you tested the two-stage training approach or BACE using supervision from a task that is not focused on physics, such as general video question answering? This would help show whether the benefits are specific to physics or apply more broadly.
4 comments
AuthorsOpenReview ↗

Response to Reviewer ADFM (Part 1/2)

Thank you for your constructive review and for recognizing the potential of BACE. We address each weakness (W) and question (Q) in the order presented.

W1. Contributions of the nine data sources

We agree that the original scale ablation preserved source proportions and therefore did not identify source contributions. We add source-group experiments comparing the full mixture with real-world, robotic, or simulated sources alone, under the same training and evaluation configuration.

Pretraining dataPhysBench (%) ↑MVBench (%) ↑
Full PhysCentric61.4056.37
Real-world sources only53.9352.58
Robotic sources only50.2748.33
Simulation sources only38.9036.38

The full-data reference is from the paper; source-group experiments follow the same configuration.

The complete mixture outperforms each individual group. These experiments add evidence about source composition beyond the original scale ablation. However, they are group-level comparisons, not leave-one-source-out estimates for all nine datasets. Where training-set sizes differ, the gaps reflect both composition and scale. We therefore do not claim to have established each source’s isolated marginal contribution or redundancy.

W2. Whether filtering ensures physically reliable observations

We agree that technical validity, deduplication, and minimum-frame checks do not guarantee physical plausibility. The filtering described in the paper should not be interpreted as a validated detector of reversed sequences or editing-induced discontinuities.

We will clarify the distinction between technical validity and physical reliability. The source-group and corpus comparisons measure downstream utility, but do not establish how frequently problematic clips occur or isolate their effect on learning. Consequently, we do not infer that such clips are absent, or that their effect is negligible, from the reported accuracy.

W3. Contribution beyond existing predictive methods and direct Stage II alignment

PAVE connects predictive learning of video dynamics with physical-QA alignment to learn representations useful for physical reasoning.

To test the alternative you identify, we directly apply Stage II to pretrained DINOv3-L, V-JEPA 2.1-L, SigLIP 2-L, and VideoMAE V2-L, without first running PAVE Stage I. Each encoder and its projector receive the same physics-QA supervision, with matched CE/BACE alignment budgets and downstream adaptation protocols.

EncoderPhysBench CE (%)PhysBench BACE (%)MVBench CE (%)MVBench BACE (%)
PAVE67.0470.0456.4358.93
DINOv3-L61.6067.5852.2350.55
V-JEPA 2.1-L64.9466.7654.7255.33
SigLIP 2-L58.2659.7853.9856.58
VideoMAE V2-L68.3669.1954.7257.33

PAVE results are from the paper; the added baseline alignment experiments follow the same protocol.

EncoderBACE − CE on PhysBench (pp)BACE − CE on MVBench (pp)
PAVE+3.00+2.50
DINOv3-L+5.98−1.68
V-JEPA 2.1-L+1.82+0.61
SigLIP 2-L+1.52+2.60
VideoMAE V2-L+0.83+2.61

PAVE retains the highest accuracy after equivalent alignment, while BACE improves physical QA for every tested encoder. This supports the objective’s usefulness beyond PAVE. It does not alone prove that training from scratch is necessary, because encoder capacity and pretraining history still differ. We also report the DINOv3 trade-off rather than claiming uniform improvements across tasks.

Q1. What happens when Stage II is applied directly to V-JEPA 2.1 or DINOv3?

The new experiments in W3 directly address this setting. Unlike the initialization experiment in the original Table 4, they do not perform PAVE pretraining after loading the existing encoder. The pretrained encoder and its projector are directly updated with CE or BACE supervision.

The table shows that these aligned encoders are competitive, while PAVE remains stronger on both reported benchmarks. We will explicitly distinguish this direct-alignment experiment from the original initialization ablation.

AuthorsOpenReview ↗

Response to Reviewer ADFM (Part 2/2)

Q2. PAVE as a physical-quality judge and for best-of-N selection

We evaluate best-of-4 video selection using 50 prompts with four candidate videos per prompt, yielding 200 candidates. Qwen3-VL-4B and its BACE-PAVE-augmented counterpart score the same candidate pool and select one video per prompt. Two human annotators rate each candidate for physical plausibility and text consistency on a 1–5 scale, with higher scores indicating better quality.

Selection methodPhysical plausibility ↑Text consistency ↑
Random selection3.163.54
Qwen3-VL-4B3.723.67
Qwen3-VL-4B + BACE-PAVE3.933.78
PAVE-augmented − baseline judge+0.21+0.11

The table reports human-rated quality of the selected videos.

Selection by the PAVE-augmented judge achieves higher human-rated physical plausibility and text consistency than selection by the baseline judge. These results provide evidence that PAVE can support a VLM-as-a-judge application and improve best-of-4 selection on the evaluated candidate set.

Q3. Estimated frequency of physically implausible or unsuitable clips

Our curation removes duplicate and invalid clips, excludes clips with insufficient decodable frames, and segments long videos to preserve temporal context for future-feature prediction.We do not report a reliable quantitative estimate in the current evidence. As clarified in W2, the documented checks concern validity, duplication, and available frames.

We will make this limitation explicit rather than equating successful preprocessing with physically reliable dynamics. A defensible prevalence estimate requires source-stratified inspection with defined criteria for reversed motion, editing discontinuities, and uncertain cases. The downstream results cannot substitute for that estimate.

Q4. BACE under non-physics supervision

We additionally compare CE and BACE using general video-QA supervision. Both conditions use PAVE with the same initialization and training configuration.

Supervision and objectivePhysBench (%) ↑MVBench (%) ↑
General QA + CE65.0156.67
General QA + BACE66.7059.97
BACE − CE (pp)+1.69+3.30

This experiment changes the supervision domain, rather than only changing the encoder under physics supervision. BACE improves both evaluations, supporting its applicability beyond physics-focused alignment. This specifically tests the alignment objective under non-physics supervision, not the entire two-stage pipeline across arbitrary tasks.

Thank you for the detailed response and additional experiments. Overall, the authors' responses have largely addressed my concerns, I am keeping my positive score.

Thank you for your constructive feedback and for taking the time to evaluate our additional experiments. We are glad that our responses have largely addressed your concerns.

Reviewer_e9qDOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes PAVE, a video encoder for physics-aware representations, trained on PhysCentric, a 515K-clip merge of nine public video datasets. It proposes a MoE-based vision encoder and adds a physics VQA supervised finetuning to update the weights of the vision encoder via gradient produced by the downstream Qwen model. A BACE loss is used to better supervise the visual encoder. In general, the contribution is limited and the comparisons are not sufficient to justify the overall design. The MoE part isn't well studied and reads rather ad hoc. In sum, the recommendation is reject.

Critical strengths
  1. Physics-aware visual representation is a relevant topic having many practical downstream applications.
  2. The encoder is small (105M active parameters) and gives competitive results against encoders of about 300M parameters under a single evaluation interface.
Critical weaknesses
  1. Limited dataset contribution. The proposed dataset reads like a combination of several existing publicly available datasets. Thus the contribution would be limited.
  2. Unfair comparison with the baselines. The baseline encoders are kept frozen, but the proposed vision encoder is supervised by gradient from the downstream model. "We update the complete PAVE encoder and its projector while keeping Qwen3-VL-2B's language model and native visual pathway frozen (Bai et al., 2025)." (Appendix B, Training Details) "Each external encoder is frozen and connected to Qwen3-VL-4B (Bai et al., 2025) through a newly initialized projector that maps its temporal features into the language-model embedding space." (Appendix B, Evaluation Protocol on PhysBench and MVBench)
  3. Unfair MoE vs. dense comparison. The MoE part reads rather ad hoc with insufficient evidence it is necessary. The authors only report a 105M active parameter MoE and an 86M dense version of it, which is unfair and not directly comparable. It is unclear why MoE is needed here. For an encoder of about 100M parameters, MoE seems rather unnecessary.
Questions for authors
  1. Please report results under a fair evaluation protocol: if the proposed encoder is supervised by the Qwen model's gradient, so should the baseline encoders.
  2. Does the method generalize to newer Qwen models and other LLMs with vision capability? The applied backbone is rather dated.
3 comments
AuthorsOpenReview ↗

Response to Reviewer e9qD (Part 1/2)

Thank you for your careful review. We respond to each weakness (W) and question (Q) below, following their original order.

W1. Limited dataset contribution

PhysCentric is curated to support learning physical dynamics from temporally ordered observations. We segment long videos to avoid excessive temporal gaps, remove duplicate and invalid clips, and retain clips with at least 16 decodable frames. Our four-segment temporal sampling preserves both clip-wide coverage and local temporal detail for future-feature prediction. To evaluate the resulting corpus, we compare it with an equally sized random OpenVid subset under the same model architecture, initialization rule, training budget, and downstream adaptation/evaluation protocol.

Pretraining corpusClipsPhysBench (%) ↑MVBench (%) ↑
PhysCentric515,37761.4056.37
Random OpenVid subset515,37750.0850.53
PhysCentric − OpenVid (pp)—+11.32+5.84

Both conditions follow the same 30-epoch Stage I configuration. This supports the usefulness of PhysCentric relative to the tested equal-scale generic-video corpus. We will describe this as evidence for the corpus composition as a whole, rather than attributing the entire improvement to a particular filtering operation.

W2. Unequal alignment supervision for baseline encoders

We agree that a common downstream evaluation interface alone does not eliminate differences in alignment supervision. We therefore directly apply Stage II to four pretrained baseline encoders, updating each encoder and its projector with the same physics-QA supervision.

The aligned comparisons use 28 alignment epochs, with AdamW reset after epoch 15, followed by three epochs of Qwen3-VL-4B adaptation. Qwen3-VL-2B is frozen during Stage II; the external encoder is frozen during downstream adaptation. “Unaligned” means no Stage II, not zero-shot evaluation.

EncoderAlignmentPhysBench (%) ↑MVBench (%) ↑
PAVEUnaligned64.8552.65
PAVECE67.0456.43
PAVEBACE70.0458.93
DINOv3-LUnaligned60.9055.85
DINOv3-LCE61.6052.23
DINOv3-LBACE67.5850.55
V-JEPA 2.1-LUnaligned59.0553.83
V-JEPA 2.1-LCE64.9454.72
V-JEPA 2.1-LBACE66.7655.33
SigLIP 2-LUnaligned55.5851.88
SigLIP 2-LCE58.2653.98
SigLIP 2-LBACE59.7856.58
VideoMAE V2-LUnaligned68.0352.92
VideoMAE V2-LCE68.3654.72
VideoMAE V2-LBACE69.1957.33

PAVE and unaligned baseline results are from the paper; aligned baseline results are additional experiments following the same protocol.

PAVE retains the highest accuracy after the baselines receive equivalent alignment. BACE also improves physical QA across all four baseline encoders. We will use this expanded comparison rather than attributing the entire advantage over unaligned baselines to the encoder. We also report the DINOv3 trade-off explicitly: physical-QA gains do not necessarily imply improved general video understanding.

W3. Unfair MoE–dense comparison and the need for MoE

We add a Dense105 control to address the capacity mismatch in the original Dense86 comparison.

ArchitecturePhysBench (%) ↑MVBench (%) ↑
Dense10559.4553.83
MoE10561.4056.37
MoE − Dense (pp)+1.95+2.54
MetricDense105MoE105
Active parameters per token105.123M105.151M
Encoder matrix-operation FLOPs419.879 GFLOPs419.956 GFLOPs

Computation is measured for 16 frames at 224 × 224 resolution, counting one MAC as two FLOPs. FLOPs cover the encoder’s major matrix operations.

Both models use the same training data, random seed, epochs, optimizer update count, effective batch size, optimization settings, and downstream protocol. Both undergo 30-epoch Stage I pretraining without BACE Stage II.

The advantage persists with closely matched active capacity and matrix computation under identical training conditions. This supports an architectural benefit beyond the original capacity difference, without implying that MoE is universally necessary.

AuthorsOpenReview ↗

Response to Reviewer e9qD (Part 2/2)

Q1. Give baseline encoders the same supervision through Qwen gradients

Yes. The experiments in W2 directly implement this request: gradients update the complete external baseline encoder and its projector during Stage II, while Qwen remains frozen. We test both CE and BACE, followed by the same downstream adaptation procedure. The comparison therefore no longer relies solely on frozen, unaligned baseline encoders.

Q2. Generalization to newer Qwen models and other vision-language models

To clarify the backbone identity, the paper uses Qwen3-VL-2B for alignment and Qwen3-VL-4B for downstream evaluation. We additionally test transfer to InternVL3.5-4B, including an adapted control without PAVE to separate representation gains from additional training.

VLMConditionPhysBench (%) ↑MVBench (%) ↑
Qwen3-VL-4BNative visual baseline55.8650.82
Qwen3-VL-4BAdapted without PAVE58.4352.43
Qwen3-VL-4BAdapted with BACE-PAVE70.0458.93
InternVL3.5-4BNative model, no additional adaptation53.2851.40
InternVL3.5-4BAdapted without PAVE57.5053.32
InternVL3.5-4BAdapted with BACE-PAVE69.2958.05
VLMGain over adapted control: PhysBench (pp)Gain over adapted control: MVBench (pp)
Qwen3-VL-4B+11.61+6.50
InternVL3.5-4B+11.79+4.73

For each backbone, adaptation with and without PAVE uses the same budget and evaluation protocol. The improvements support transfer beyond the Qwen family and are not explained solely by additional adaptation training. The new backbone experiment tests another model family, rather than an additional Qwen version; our conclusion is limited to the tested backbones.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer e9qD,

Thank you again for your review. We would like to follow up on the additional experiments presented in our responses above (Parts 1 and 2). These include equivalent CE/BACE alignment for all baseline encoders, a Dense105 control with matched active parameters and compute, an equal-sized OpenVid comparison, and transfer to InternVL3.5-4B with an adapted control.

We hope these results help address your concerns about comparison fairness and the contributions of the data and architecture. We would appreciate your updated assessment in light of the additional evidence, and would be happy to clarify any remaining concerns.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.