acceptodds
Under review as a conference paper at ICLR 2027

PhysMRV: Physical Memory Retrieval and Verification for Physical Plausibility Reasoning

Abstract

Video-language models (VLMs) have achieved remarkable performance on traditional visual question answering tasks, yet they remain unreliable in reasoning about physical plausibility, where understanding object interactions, causal dynamics, and fundamental physical principles is essential. This limitation is particularly evident on challenging physical reasoning benchmarks, revealing a persistent gap in physical commonsense reasoning. To address this challenge, we propose PhysMRV, a training-free physical memory and verification framework for physical plausibility reasoning. PhysMRV constructs a Hierarchical Memory Bank of structured physical knowledge comprising three complementary levels: scene descriptions capturing general visual information, event graphs modeling object interactions and state transitions, and physics-rule summaries distilling reusable physical principles. During inference, PhysMRV retrieves physically relevant memories and leverages their structured physical evidence to guide the VLM to verify physical plausibility without requiring parameter updates. We evaluate PhysMRV on three challenging physical plausibility reasoning benchmarks: ImplausiBench, IntPhys2, and GRASP Level 2, across multiple state-of-the-art VLMs. Experimental results demonstrate consistent improvements across diverse VLM backbones and strong baselines, validating the effectiveness of structured physical memory and evidence-guided verification for physical plausibility reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Physical Reasoing, Memory, Multimodality, Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YydGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates physical plausibility in VLMs’ visual question answering. To eliminate the limitations of physical reasoning, the authors propose PhysMRV, a training-free physical memory and verification framework for physical plausibility reasoning. Experimental results demonstrate consistent improvements across diverse VLM backbones and strong baselines.

Critical strengths

The paper mainly concentrated on evaluating VLMs on real-world dynamics and the underlying physical mechanisms. This is a challenging question for highly capable AI agents.

Critical weaknesses

It is recommended to report the input and output token costs, as well as question-answering latency. More importantly, the cost of improving or maintaining the memory bank for the same or different types of questions should be shown.

As can be seen in Table 1, simple CLIP retrieval baselines achieve performance close to that of the complex framework, and removing any module in the ablation study in Table 3 results in performance lower than that of CLIP. This means that the problem addressed by the proposed methods could possibly be solved by post-training a specific CLIP model.

Questions for authors

Polish the formatting of the writing, including incorrect citation formatting. For example, “InPhyRe (Sreekumar & Boddeti, 2026)reports” is missing a space. Additionally, in lines 159–161, “shown in 1 with” is missing “Figure”.

It is recommended to conduct evaluations on open-sourced MLLMs, such as ChatGPT and Gemini.

How is the quality of AI tools ensured during offline construction?

It is recommended to explore the reason behind the improvement, as well as key takeaways for future work.

6 comments
AuthorsOpenReview ↗

Response to Reviewer YydG (1/2)

We thank the reviewer for the helpful suggestions and answer each point below.

1. Token cost, latency and memory-bank cost. Measured with Qwen3-VL-8B on A100 GPUs. Direct is the same backbone answering without memory, values are Direct → PhysMRV.

ImplausiBenchIntPhys 2GRASP L2
Total input tokens1,909 → 2,4851,472 → 2,0711,601 → 2,114
Answer time per sample (s)1.05 → 1.230.44 → 0.530.38 → 0.49
  • Latency. The longer context does increase latency, but only modestly. The retrieved memories add about 510–600 text tokens, which raises the total input by 30–41% and the answer time by 17% on ImplausiBench, 19% on IntPhys 2 and 29% on GRASP L2. The increase is small because video tokens make up most of the input. Times are the average per sample at the batch size of each run (32 for IntPhys 2, 16 for the other two).
  • Output tokens. On all three benchmarks both methods answer with a single token.
  • Maintaining the bank for the same type of questions. Adding examples requires only a caption, an event graph and a rule label for the new items: about 2.2 s of Qwen3-VL-8B per new video and about 1 s of GPT-5.5 per rule label. Existing instances are untouched.
  • Different types of questions. The memory bank needs no update. The same bank, built once from source videos, serves all three benchmarks and all four backbones without adding or relabelling any instance.

We have added this cost table to the revised version.

2. CLIP retrieval baseline and "removing any module is below CLIP". We apologize for the confusion caused by its name. The “CLIP retrieval” row in Table 1 is not a standalone CLIP model. It is PhysMRV with the retriever changed: CLIP ViT-B/32 video similarity instead of our physics-aware retrieval, while the query event graph, memory content and verifier are all kept. It is therefore an ablation of our retrieval module rather than a simple baseline. This also explains why every ablation in Table 3 falls below it. Each ablation removes a module (the verifier, the event graph, or the memory) that the CLIP variant still retains, so falling below it confirms that each module contributes.

3. Formatting. Thank you for pointing this. We fixed the citation spacing ("InPhyRe (Sreekumar & Boddeti, 2026) reports"), the missing "Figure" in L159–161, and other typos.

AuthorsOpenReview ↗

Response to Reviewer YydG (2/2)

4. Evaluation on ChatGPT and Gemini. We evaluated OpenAI gpt-6-luna and Google gemini-3-flash-preview on all three benchmarks. Both receive exactly the inputs used for previous backbones: Direct is the video plus the task, and PhysMRV adds the same query event graph and the same three retrieved memories. Retrieval, prompts and the memory bank are unchanged. Both reasoning effort is medium, Values are Direct → PhysMRV.

ModelImplausiBenchIntPhys 2GRASP L2
gemini-3-flash-preview60.33 → 69.3356.52 → 59.5863.48 → 64.99
gpt-6-luna69.67 → 72.6753.95 → 55.1457.47 → 59.55

PhysMRV improves both models on every benchmark, even though their direct accuracy is well above that of the 8B backbones. This indicates that a memory bank and event graphs built once with an 8B open-source model remain useful for substantially stronger proprietary models.

5. Offline construction: quality. We use the AI tools to produce (i) one caption and one event graph per video (Qwen3-VL-8B), which serve as retrieval keys, and (ii) one rule-family label per instance (GPT-5.5).

  • Constrained generation. Captions and event graphs are generated with deterministic decoding. Every event graph follows a fixed schema (type, participants, phase, states, state change, evidence), with event types and phases taken from predefined vocabularies. The rule label must be one of 15 fixed families, and GPT-5.5 chooses it from the question, the original explanation and the video.
  • Human check. Two annotators manually evaluated 100 randomly sampled memory instances against their source videos, demonstrating substantial agreement. They found the generated outputs highly accurate, judging 91% and 93% of the captions correct (96% agreement, Cohen's κ=0.73\kappa=0.73), and 98% and 99% of the event graphs correct (99% agreement, κ=0.66\kappa=0.66). Both annotators also confirmed the assigned rule labels were correct in all 100 instances. Crucially, the event graph captures physical details much better than the caption. For example, of the 9 captions marked incorrect by the first annotator (6 of which were confirmed by the second), the corresponding event graphs remained completely accurate. Seven of these 9 videos feature a physical violation (e.g., a ball passing through a mirror, or a paper towel that does not absorb water); in these cases, the caption merely defaults to the expected outcome, whereas the event graph faithfully records what actually happens. Generation also holds up on low-quality videos: across the 24 videos at 176×144 resolution, the first annotator judged 23 captions and all 24 event graphs to be correct.

6. Why PhysMRV helps, and takeaways for future work.

  • Knowing a rule is not enough; a matched case makes it usable. Retrieved rules alone perform on par with no memory (Rule Only vs. w/o Memory). Gains appear only when each rule comes with a similar case showing how it applies. This echoes InPhyRe: VLMs often know physical laws but fail to apply them.
  • Relevance carries the gain, not extra context or labels. Shuffled Memory keeps the same content and verdict distribution but performs no better than no memory, and the majority verdict of the retrieved memories is at chance. Ranking by event-graph similarity also beats semantic and visual retrieval.
  • Perception is the remaining bottleneck. Frozen VLMs mainly miss violations: Direct accuracy on implausible videos is far below that on plausible ones. Memories enter the prompt only as text, so the backbone must still locate the violation in the video. Accordingly, the gain is smallest for LLaVA-OneVision-2, the backbone with the weakest Direct accuracy.

Future work. Improving event perception, for example by training on paired counterfactual videos that differ only in the violation, and storing such contrastive pairs in memory.

We hope these answers address the reviewer's concerns, and we are glad to clarify anything further.

AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal

Dear Reviewer YydG,

Thank you again for your time and valuable comments. We have carefully addressed each of your concerns in our response. As the discussion period ends soon, we would greatly appreciate it if you could take a look at our rebuttal when convenient and let us know if any concerns remain. We are happy to provide any further clarification.

Best regards,

The Authors

Reviewer_ndXjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes a training-free method which builds a memory structure bank from a physics video QA dataset. It retrieves relevant entries via semantic recall and event-graph reranking, it also uses them as evidence to help VLMs judge physical plausibility during the inference. And the reports showed consistent but modest gains across four backbones and three benchmarks.

Critical strengths
  1. It is training-free and serves as a plug-in framework for all backbones.
  2. Across a total of 12 experiments, this framework can be said to achieve SOTA performance.
  3. The shuffled Memory ablation is an important analysis.
Critical weaknesses
  1. Regarding Line 309 (“Complete prompt templates are provided in the appendix.”) and the Reproducibility Statement, these claims are not accurate, because the appendix does not provide the complete prompt templates actually used in the experiments. I believe this is the most important part of training-free research. Please provide them.
  2. Line 706 states: “It also presents a saved qualitative example with its complete retrieved context.” This also does not appear to be provided. Please supply the content claimed here during the rebuttal phase and upload a revised version of the manuscript.
  3. Lines 885 and 913: where is the “source bundle”? If the authors promise to provide it, they should actually provide it, rather than leaving reviewers unable to find it.
  4. I believe the principles are the physical rules, but the manuscript does not seem to provide the principles for all rule categories (only one example is shown in Figure 1). Please clarify.
  5. The ablation study should report results for all backbones, so that we can truly know whether the components generalize across models.
  6. The proportion of plausible and implausible samples among the retrieved memories may be a key analysis for this study. Clearly showing its relationship with the final predictions would make this work more insightful.
  7. Table 1 shows that InternVL3.5-8B performs similarly to Qwen3-VL-8B-Instruct. Why did the authors choose the latter to generate captions and event graphs? I think a more feasible approach would be to compare candidate models on an existing benchmark such as CLEVRER before deciding which one to use. A more rigorous approach would be human evaluation, reporting accuracy and inter-annotator agreement.
  8. The authors should also evaluate on PhysBench, the benchmark used in the PhysNote paper (which you cite in the manuscript), to enable a fairer comparison.
  9. Results on PhysGame would support the claim in Line 399: “Overall, the results show that a single shared physical memory bank can effectively transfer across both benchmarks and backbones.” Gameplay videos differ substantially from TRAVL, but the underlying physical phenomena should be the same.
  10. How does the framework perform on each sub-category of each benchmark? This would give us a better understanding of the framework.
  11. Since all results use greedy decoding, standard deviations cannot be computed. Could the authors provide confidence intervals for comparisons against Direct and the strongest baselines, for example, using confidence intervals (e.g., bootstrap) or significance tests (e.g., McNemar’s test) ?
  12. Which components are removed in the “w/o Verification” variant?
  13. In Table 5, PhysMRV with InternVL3.5-8B achieves only 36.52% accuracy on plausible (P/Y) samples of GRASP L2, which is below the 50% expected from random guessing and a drop of nearly 60 points from Direct (96.39%). This contradicts the statement in Line 470 that PhysMRV “largely preserves” performance on plausible events. What causes this drop?
  14. Which backbone does TRAVL use?
Questions for authors

As mentioned in the Weaknesses.

11 comments
AuthorsOpenReview ↗

Response to Reviewer ndXj (1/5)

We thank the reviewer for the insightful comments. We answer each point in the order of the review.

1–4. Prompt templates (L309), qualitative example (L706), source bundle (L885, L913), principles of all rule families. We appreciate the reviewer for pointing out these issues. The revised PDF has added:

  • the complete prompt templates: caption, event-graph, and the Direct and PhysMRV verifier prompts for each benchmark;
  • a complete qualitative example: video frames, question, retrievaled memory and final prompt;
  • a table of all 15 rule families with their principles.

We will uploaded the source bundle too.

5. Ablations on all backbones. We ran the same three ablations on Cosmos-Reason2, InternVL3.5 and LLaVA-OneVision-2, using the same prompts as for Qwen. Values are accuracy (%), parentheses give the change from the PhysMRV.

ModelConfigurationImplausiBenchIntPhys 2GRASP L2
Cosmosw/o Query Graph53.33 (−5.00)51.19 (−2.17)52.54 (−3.17)
w/o Memory55.33 (−3.00)51.28 (−2.08)51.25 (−4.46)
w/o Verification56.67 (−1.67)52.17 (−1.19)55.20 (−0.51)
InternVLw/o Query Graph57.33 (−2.34)49.70 (−2.38)50.10 (−4.66)
w/o Memory57.67 (−2.00)50.30 (−1.78)51.68 (−3.08)
w/o Verification58.33 (−1.34)51.58 (−0.50)53.78 (−0.98)
LLaVAw/o Query Graph34.33 (−1.34)49.70 (−0.60)50.54 (−0.71)
w/o Memory34.67 (−1.00)49.60 (−0.70)50.59 (−0.66)
w/o Verification33.67 (−2.00)48.32 (−1.98)50.07 (−1.18)

The results confirm that our framework's components are universally beneficial: removing any module degrades performance across all 27 new test cases, proving the findings in Table 3 generalize beyond Qwen.

  1. The query graph acts as the core driver of the framework. Its removal universally damages performance, with the most significant penalties observed on Cosmos (−5.00% on ImplausiBench) and InternVL (−4.66% on GRASP L2).
  2. The utility of retrieved memory is highly task-dependent, peaking on GRASP L2 where it contributes up to 4.46% accuracy.
  3. The verification framing exhibits strong complementary effects based on the backbone. It yields modest gains for Cosmos, InternVL and LLaVA.

We have added this table to the revised version.

AuthorsOpenReview ↗

Response to Reviewer ndXj (2/5)

6. Proportion of plausible and implausible retrieved memories and its relationship with predictions.

We thank the reviewer for this suggestion. We agree that the plausible/implausible composition of the retrieved memories is central to understanding how PhysMRV uses memory.

Composition of the memory bank. The memory bank is built from the TRAVL tuning split, where each video is associated with several QA instances. Plausible videos contribute more QA instances on average (7.1 vs. 3.9 per video). As a result, the bank is even more plausible-dominated at the instance level than at the video level.

ImplausiblePlausible
Videos1,521 (43.7%)1,961 (56.3%)
Memory instances (QA)5,864 (29.8%)13,844 (70.2%)

Composition of the retrieved memories. In contrast, the retrieved Top-3 sets are dominated by implausible instances: 76.1% on ImplausiBench, 86.6% on IntPhys 2 and 90.2% on GRASP L2. This reversal comes from scene type. Plausible TRAVL videos are mostly real-world human activities. Implausible ones are mostly object-centric physics scenes (balls, ramps, containers), which are much closer to the object-physics videos in all three benchmarks. The proportion is nearly the same for plausible and implausible queries. It is therefore determined by what the scene looks like rather than by the correct answer.

Do the retrieved verdicts predict the answer? Because plausible and implausible queries retrieve memories with nearly the same verdict proportions, the retrieved verdicts by themselves cannot distinguish between the two classes. The majority-verdict baseline confirms this. It answers with the majority verdict of the three retrieved memories:

  • GRASP L2: 49.9%, compared with 55.20 / 54.76 for PhysMRV (Qwen3-VL / InternVL3.5).
  • IntPhys 2: 50.4%, compared with 53.75 / 52.08.
  • ImplausiBench: 52.3% even when scored only on the plausibility verdict, as the baseline cannot choose among the multiple-choice options. PhysMRV reaches 60.67 / 59.67 under the stricter original-option scoring.

Relationship with the final predictions. PhysMRV agrees with the majority verdict of its retrieved memories on 33.7% (Qwen3-VL) and 34.0% (InternVL3.5) of ImplausiBench videos, and on 44.4% (Qwen3-VL) and 69.9% (InternVL3.5) of GRASP L2 videos. Because the retrieved majority is implausible for almost every query, agreement mainly reflects how often a backbone answers "implausible". Both backbones receive identical retrieved memories, yet their agreement on GRASP L2 differs by more than 25 pp. The composition alone therefore does not determine the prediction. Rather, the prediction depends largely on how each backbone evaluates the current video.

Separating composition from relevance. The shuffled-memory control (Table 4, Qwen3-VL) directly tests whether the composition itself drives the predictions. Each query receives the three memories retrieved for another randomly chosen query from the same benchmark. The benchmark-level composition is unchanged (76.1% / 86.6% / 90.2% implausible), but the memories no longer match the current video. If the gains came from the proportion of implausible verdicts, shuffling would preserve them. Instead, accuracy falls to or below the no-memory level on all three benchmarks:

Variant (Qwen3-VL)ImplausiBenchIntPhys 2GRASP L2
w/o Memory57.6752.0852.56
Shuffled Memory56.6750.6952.56
PhysMRV (Full)60.6753.7555.20

We have added add these analyses to the revised version.

AuthorsOpenReview ↗

Response to Reviewer ndXj (3/5)

7. Why Qwen3-VL generates the captions and event graphs and human evaluation.

Thank you for raising this important point. We selected Qwen3-VL because the Qwen family is among the most widely used open-source VLMs, and in our development experiments Qwen3-VL produced reliable captions and consistent, structurally valid event-graph JSON. To directly assess the quality of these generated outputs, two annotators independently evaluated 100 randomly sampled memory instances from 100 distinct videos, checking each caption and event graph against its source video.

Annotator 1Annotator 2Percent agreementCohen's κ
Caption91%93%96%0.73
Event graph98%99%99%0.66

Agreement. Both annotators judge most captions and event graphs correct. Cohen's κ is 0.73 for captions and 0.66 for event graphs, both indicating substantial agreement. On event graphs, the two annotators disagree on only one of the 100 instances.

8. PhysBench. We thank the reviewer for this suggestion. We did not include PhysBench because it targets a different problem from the one studied in this paper. Our task is physical-plausibility reasoning: given a video that may contain a physics violation, the model must decide whether the observed event is consistent with physical laws. PhysBench instead evaluates broad physical understanding across object properties, object relationships, scene understanding and dynamics. Most of its questions ask the model to recognize a property or a relation, for example which liquid in a video is the most viscous or which object is closest to the camera, rather than to judge whether an event violates physics.

This mismatch also concerns the method itself. PhysMRV’s memory is built from TRAVL annotations explaining why a video is plausible or implausible, and its rule catalog summarizes the constraints that such violations break. The verifier uses these rules to check whether the interactions in the query event graph satisfy them. This design has no natural role in estimating viscosity or camera distance, so a result on PhysBench would mainly reflect general physical perception rather than the ability PhysMRV is designed to improve. In addition, many PhysBench items are image-only or interleaved multi-image inputs, for which the event graph, built from the temporal dynamics of one video, is not defined. We therefore evaluate on ImplausiBench, IntPhys2 and GRASP, which are built specifically around plausible and implausible events.

We agree that the comparison with PhysNote should be fair. Within our task, both methods use the same four frozen backbones, 1-FPS video sampling and final-answer budgets. The remaining mismatch is memory size, so we rebuilt PhysMRV’s memory from the same 500 TRAVL examples used to build PhysNote’s notes and re-ran PhysMRV on all four backbones and three benchmarks with everything else unchanged.

BackboneMethodImplausiBenchIntPhys 2GRASP L2
Qwen3VLDirect56.0049.9048.56
PhysNote (500 videos)49.0052.1751.66
PhysMRV-50060.3350.9954.35
CosmosDirect53.0049.8049.98
PhysNote (500 videos)45.0049.9053.27
PhysMRV-50057.6752.2754.74
InternVLDirect56.3349.4149.07
PhysNote (500 videos)43.6750.6951.03
PhysMRV-50058.3351.6853.12
LLaVADirect34.0048.4248.90
PhysNote (500 videos)36.3351.8849.76
PhysMRV-50036.0050.2050.73
  • With the same 500 videos, PhysMRV beats PhysNote in 9 of 12 cells (by up to +14.66 on ImplausiBench), so its margin over PhysNote does not come from using 7× more videos. It trails in Qwen IntPhys 2 (−1.18), LLaVA ImplausiBench (−0.33) and LLaVA IntPhys 2 (−1.68). In the two LLaVA cells, full PhysMRV also trails PhysNote.
  • PhysMRV-500 still beats Direct in all 12 cells (+1.09 to +5.79).

We have added this table to the revised version.

9. PhysGame. We thank the reviewer for this valuable suggestion and agree that gameplay videos would be a meaningful test of transfer, since they differ substantially from TRAVL in visual appearance while sharing the same underlying physical phenomena. We tried to include PhysGame in our evaluation, but unfortunately we were unable to obtain the data. The download link provided in the official GitHub repository is currently no longer accessible. We also noticed that other users have reported the same issue in the repository, but it has not yet received a response. We would be glad to include PhysGame results if the dataset becomes available again.

Reviewer_6DMWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PhysMRV, a training-free framework that helps video-language models judge whether a video is physically plausible. It builds a memory bank offline from the TRAVL training data, where each entry links a question and its explanation to a scene caption, a physical event graph, and a physical rule from a fixed set of fifteen families. At inference time, PhysMRV retrieves candidate memories by text similarity, reranks them by event-graph similarity, and gives the top three, together with the event graph of the current video, to the model for verification. Experiments with four backbones on ImplausiBench, IntPhys2 and GRASP Level 2 show consistent gains over direct prompting.

Critical strengths

The paper addresses a relevant and practical problem, and its approach requires no parameter updates. A single memory bank is reused across benchmarks and backbones, which makes the method easy to apply. The design is clearly presented, with a well-motivated split between scene context, physical events and general rules, and a simple two-stage retrieval process. The evaluation covers four different backbones and compares against several types of baselines, including general reasoning methods, visual retrieval and physics-specific approaches. Beyond removing individual components, the shuffled-memory control checks whether gains come from relevant memories rather than from extra context. Finally, the appendix documents baseline adaptations, answer parsing and the handling of invalid responses in detail, which makes the experiments easier to understand and reproduce.

Critical weaknesses

W1. Absolute performance remains close to chance, and the reliability of the gains is unclear. IntPhys2 is a balanced binary task with a chance level of 50%, and Table 5 suggests that GRASP L2 is also roughly balanced. The best results of PhysMRV on these two benchmarks are only 53.75 and 55.71.

W2. The gains mainly come from a shift in answer tendency rather than better discrimination. Table 5 shows large changes in both directions on GRASP L2. Accuracy on plausible samples drops sharply, from 94.87 to 62.45 for Qwen3-VL and from 96.39 to 36.52 for InternVL3.5, while accuracy on implausible samples rises sharply. InternVL3.5 effectively moves from answering “yes” almost always to leaning toward “no”. The sum of the two class accuracies increases by only about 11 to 13 points, so most of the change comes from redistributing predictions between the two classes.

W3. The retrieved explanations may reveal the verdicts of similar videos. According to App. A.1, the explanations stored in memory are the original annotations from the TRAVL training set, and they often state directly whether a video is plausible. The model may therefore be copying the conclusions of similar examples, in effect performing nearest-neighbor label transfer, rather than checking the current video against physical rules. The paper does not report the balance of plausible and implausible examples in the memory or in the retrieved sets. If implausible examples dominate, adding memory alone would push predictions toward “implausible”.

W4. The reproduced TRAVL results appear unreliable. In Table 2, the reproduced TRAVL reaches only 33.50 on IntPhys2, well below the 50% chance level of this balanced binary task. Its ImplausiBench score of 31.33 is also lower than the direct results of every untuned backbone in Table 1. This usually points to parsing failures, which App. A.2 counts as incorrect, or to a different backbone. However, the paper does not state which backbone TRAVL uses or how many of its responses are invalid.

W5. The comparison with PhysNote uses very different memory sizes. According to App. D.2, the PhysNote adaptation builds its notes from 500 training examples, while PhysMRV retrieves from 19,708 entries.

Questions for authors

Q1. On chance levels and statistical reliability. Could the authors state the number of options and the chance level for ImplausiBench, and the class balance of GRASP L2? I would also appreciate paired significance tests, such as McNemar’s test, or bootstrap confidence intervals for the comparison between PhysMRV and Direct on each benchmark.

Q2. On discrimination versus answer tendency. For the open-source backbones, could the authors report AUROC computed from the yes and no token probabilities? This measures discrimination independently of the decision threshold. It would also help to include two simple calibration baselines:

A prompt that reminds the model that the video may violate physical laws. A yes/no decision threshold tuned on a held-out set.

Q3. On possible verdict copying. Could the authors report the proportion of plausible and implausible examples in the memory bank and in the retrieved sets? It would be very informative to add a simple baseline that predicts the majority verdict of the three retrieved memories without VLM verification. It would also help to report how often PhysMRV’s prediction agrees with the retrieved verdicts. These results would show whether the model checks the current video or mainly follows similar examples.

Q4. On the TRAVL comparison. Which backbone was used for the reproduced TRAVL, and what fraction of its responses were invalid? It would help to compare these results with those reported in the original TRAVL paper.

Q5. On memory size in the PhysNote comparison. Could the authors run PhysMRV with a memory built from the same 500 examples used for PhysNote, or run PhysNote with a larger set of examples? Either comparison would separate the contribution of the method from that of memory size.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 6DMW (1/4)

We thank the reviewer for the insightful comments. We address your concerns and questions point by point below:

W1 / Q1. Near-chance accuracy, chance levels and statistical reliability.

Thank you for pointing this out. It is notable that IntPhys 2 and GRASP L2 are hard for all current VLMs.

  • IntPhys 2: The benchmark paper reports a best multimodal LLM of 55.63% on Gemini 2.5 Flash and 53.75% for GPT-4o. Each is the best of many runs and prompt variants. The 2026 work ChronoVision[1] reports "a strong 55.0% accuracy on IntPhys2, a highly challenging cross-domain benchmark".
  • GRASP L2: The original paper finds all tested models at or below chance. Pixels to Principles[2] paper reports that 72–78B open models and Gemini 2.0 Flash Thinking stay below 54%.

Our PhysMRV is training-free and keeps VLMs frozen. It reaches 53.75 on IntPhys 2 and 54.76–55.71 on GRASP L2 with three backbones, which is a competitive and low cost method.

Chance levels.

  • ImplausiBench has 7 options per item (A–G; G is always "None of the given reasons is entirely correct."), so uniform guessing gives 14.3%. Its 300 videos are 150 plausible and 150 implausible.
  • GRASP L2 is exactly balanced: 2,048 plausible videos and 2,048 implausible videos.

Significance. We used an exact two-sided McNemar test on paired per-sample correctness and a 95% paired bootstrap CI (2,000 resamples).

PhysMRV vs Direct:

BackboneImplausiBench Δ (p) [95% CI]IntPhys 2 Δ (p) [95% CI]GRASP L2 Δ (p) [95% CI]
Qwen3-VL+4.67 (0.009) [+1.3, +8.0]+3.85 (5e-8) [+2.5, +5.3]+6.64 (4e-11) [+4.7, +8.6]
Cosmos-Reason2+5.33 (0.004) [+2.0, +8.7]+3.56 (9e-7) [+2.2, +5.0]+5.74 (2e-9) [+3.9, +7.6]
InternVL3.5+3.33 (0.16) [−0.7, +7.7]+2.67 (0.004) [+0.9, +4.5]+5.69 (9e-6) [+3.1, +8.2]
LLaVA-OV-2+1.67 (0.125) [+0.0, +3.4]+1.88 (3e-4) [+1.0, +2.9]+2.34 (0.10) [−0.3, +5.1]

Nine of twelve backbone–benchmark cells are significant at p < 0.05, and all twelve point estimates are positive. PhysMRV delivers consistent, statistically reliable improvements across backbones and benchmarks. We have added this result to the revised version.

References [1] ChronoVision: Temporal Reasoning via Latent State Reconstruction (Shen et al., EMNLP 2026). [2] Pixels to Principles: Probing Intuitive Physics Understanding in Multimodal Language Models (Ballout et al., arXiv 2025).

AuthorsOpenReview ↗

Response to Reviewer 6DMW (2/4)

W2 / Q2. Discrimination vs. answer tendency.

Thank you for pointing this out. We agree that class-wise accuracy alone cannot separate discrimination from a threshold shift. We therefore report threshold-free AUROC below.

AUROC. We recorded the first answer-token probabilities under the same frozen prompts, and score each video by log P(positive token) − log P(negative token). Since only IntPhys 2 and GRASP L2 are binary-class benchmarks, we report AUROC for all four backbones on these two benchmarks. AUROC, Direct → PhysMRV (ΔAUROC):

IntPhys 2GRASP L2
Qwen3-VL0.520 → 0.551 (+0.031)0.504 → 0.572 (+0.068)
InternVL3.50.519 → 0.533 (+0.014)0.527 → 0.571 (+0.044)
Cosmos-Reason20.547 → 0.556 (+0.009)0.550 → 0.572 (+0.022)
LLaVA-OV-20.502 → 0.519 (+0.017)0.494 → 0.518 (+0.024)

PhysMRV raises AUROC in all eight cells, significantly for Qwen and InternVL on GRASP L2 and for Qwen on IntPhys 2. This indicates that the gain is not only a shift of the decision threshold.

Reminder-prompt baseline. For Qwen3-VL and InternVL3.5 we prefixed the Direct prompt with "Note: the video may contain events that violate physical laws.", leaving everything else unchanged. Direct / Reminder / PhysMRV on the two binary benchmarks:

Acc. GRASP L2Acc. IntPhys 2AUROC GRASP L2AUROC IntPhys 2
Qwen3-VL48.56 / 51.73 / 55.2049.90 / 51.78 / 53.750.504 / 0.527 / 0.5720.520 / 0.517 / 0.551
InternVL3.549.07 / 50.49 / 54.7649.41 / 50.40 / 52.080.527 / 0.548 / 0.5710.519 / 0.521 / 0.533

Qwen3-VL: PhysMRV beats the reminder significantly in GRASP L2 accuracy (+3.47 pp), and in AUROC on both benchmarks (+0.045; +0.034). On IntPhys 2 the reminder leaves AUROC unchanged (0.520 → 0.517): it moves only the threshold, whereas PhysMRV improves the ranking. InternVL3.5: PhysMRV outperforms the reminder on all four metrics. On GRASP L2, PhysMRV (54.76) surpasses the reminder (50.49) and roughly doubles its AUROC gain over Direct (+0.044 vs +0.021), confirming the improvement stems from better ranking, not a shifted threshold. On IntPhys 2, PhysMRV improves over the reminder by +1.68 pp in accuracy and +0.012 in AUROC, while the reminder again leaves AUROC essentially unchanged (0.519 → 0.521).

Held-out threshold baseline. We tuned Direct's yes/no threshold on the PID test split of PhyDetEx[1] (500 videos, 250 plausible / 250 implausible), which is disjoint from our benchmarks and from the memory bank. For each backbone, we select the threshold on the first-token log-odds, log P(yes) − log P(no), that maximizes balanced accuracy on PID, and apply it to GRASP L2 and IntPhys 2. ImplausiBench is omitted since it is multiple-choice.

Accuracy (%): Direct / Direct + PID threshold / PhysMRV:

GRASP L2IntPhys 2
Qwen3-VL48.56 / 48.66 / 55.2049.90 / 51.88 / 53.75
InternVL3.549.07 / 50.05 / 54.7649.41 / 51.09 / 52.08
Cosmos-Reason249.98 / 49.80 / 55.7149.80 / 52.08 / 53.36
LLaVA-OV-248.90 / 48.54 / 51.2548.42 / 49.11 / 50.30

The threshold works on PID itself (Direct's PID AUROC 0.77–0.82, balanced accuracy 72–76%). On GRASP L2 it moves Direct by less than 1 pp, and PhysMRV is significantly better than calibrated Direct on all four backbones, so its GRASP L2 gain cannot be obtained by recalibrating Direct on held-out data. On IntPhys 2, calibrated Direct comes within 1.0–1.9 pp of PhysMRV. This is partly because Direct’s IntPhys 2 AUROC is only 0.50–0.55, leaving little room for a threshold to improve upon.Calibrated Direct ends up at about 50–52% on every backbone. PhysMRV consistently outperforms calibrated Direct on all four backbones, with the largest gain on Qwen3-VL (+1.87 pp), confirming that its improvement goes beyond threshold recalibration.

References [1] PhyDetEx: Detecting and Explaining the Physical Plausibility of T2V Models (Wang et al., ECCV 2026)

AuthorsOpenReview ↗

Response to Reviewer 6DMW (3/4)

W3 / Q3. Possible verdict copying from retrieved explanations.

We thank the reviewer for raising this important check.

Composition of the memory bank and retrieved sets. The memory bank is built from the TRAVL tuning split, each video contributes several QA instances. Plausible videos contribute more instances on average (7.1 vs. 3.9 per video), so the bank is even more plausible-heavy by instance than by video.

ImplausiblePlausible
Videos1,521 (43.7%)1,961 (56.3%)
Memory instances (QA)5,864 (29.8%)13,844 (70.2%)

Although the bank is plausible-majority, 76.1% (ImplausiBench), 86.6% (IntPhys 2) and 90.2% (GRASP L2) of retrieved Top-3 instances are implausible. This skew is because plausible TRAVL videos are mostly real-world human activities, whereas implausible ones are mostly object-centric physics scenes (balls, ramps, containers), which resemble the object-physics videos in all three benchmarks. The skew is nearly identical for plausible and implausible queries, so it reflects scene type rather than the answer.

Majority-verdict baseline. Answering with the majority verdict of the three retrieved memories gives 49.9% on GRASP L2 and 50.4% on IntPhys 2, versus 55.20 / 54.76 and 53.75 / 52.08 for PhysMRV (Qwen3-VL / InternVL3.5). On ImplausiBench it reaches only 52.3% even when scored leniently at the verdict level, versus 60.67 / 59.67 for PhysMRV under the stricter original-option scoring.

Agreement with the retrieved majority. PhysMRV agrees with the majority verdict of its three memories on 33.7% (Qwen3-VL) and 34.0% (InternVL3.5) of ImplausiBench videos, and on 44.4% (Qwen3-VL) and 69.9% (InternVL3.5) of GRASP L2 videos.

Shuffled-memory control. In the shuffled-memory control (Table 4, Qwen3-VL), each query gets the three memories that were retrieved for another, randomly chosen query from the same benchmark. Since we only swap memories between queries, the share of implausible memories stays the same (76.1% / 86.6% / 90.2%), but the memories no longer match the current video. If PhysMRV simply copied the retrieved verdicts or was pushed toward “implausible” by them, shuffling should not hurt performance. Instead, accuracy drops to or below the no-memory level on all three benchmarks:

Variant (Qwen3-VL)ImplausiBenchIntPhys 2GRASP L2
w/o Memory57.6752.0852.56
Shuffled Memory56.6750.6952.56
PhysMRV (Full)60.6753.7555.20

Summary. PhysMRV does not copy the verdicts of its memories. The majority-vote baseline over retrieved verdicts performs at chance on all three benchmarks. Moreover, despite the retrieved sets being heavily skewed toward implausible examples (76–90%), PhysMRV’s predictions agree with the retrieved majority verdict only 34–44% of the time (Qwen3-VL), confirming that the model reasons over the current video rather than transferring labels from similar examples.

Comments on the paper

1 comment

Dear Reviewers,

Please read the authors' response. Please evaluate whether your concerns are addressed or if anything needs further attention. Please engage with the authors while we have time.

Thanks, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.