- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates physical plausibility in VLMs’ visual question answering. To eliminate the limitations of physical reasoning, the authors propose PhysMRV, a training-free physical memory and verification framework for physical plausibility reasoning. Experimental results demonstrate consistent improvements across diverse VLM backbones and strong baselines.
Critical strengths
The paper mainly concentrated on evaluating VLMs on real-world dynamics and the underlying physical mechanisms. This is a challenging question for highly capable AI agents.
Critical weaknesses
It is recommended to report the input and output token costs, as well as question-answering latency. More importantly, the cost of improving or maintaining the memory bank for the same or different types of questions should be shown.
As can be seen in Table 1, simple CLIP retrieval baselines achieve performance close to that of the complex framework, and removing any module in the ablation study in Table 3 results in performance lower than that of CLIP. This means that the problem addressed by the proposed methods could possibly be solved by post-training a specific CLIP model.
Questions for authors
Polish the formatting of the writing, including incorrect citation formatting. For example, “InPhyRe (Sreekumar & Boddeti, 2026)reports” is missing a space. Additionally, in lines 159–161, “shown in 1 with” is missing “Figure”.
It is recommended to conduct evaluations on open-sourced MLLMs, such as ChatGPT and Gemini.
How is the quality of AI tools ensured during offline construction?
It is recommended to explore the reason behind the improvement, as well as key takeaways for future work.
6 comments
Response to Reviewer YydG (2/2)
4. Evaluation on ChatGPT and Gemini. We evaluated OpenAI gpt-6-luna and Google gemini-3-flash-preview on all three benchmarks. Both receive exactly the inputs used for previous backbones: Direct is the video plus the task, and PhysMRV adds the same query event graph and the same three retrieved memories. Retrieval, prompts and the memory bank are unchanged. Both reasoning effort is medium, Values are Direct → PhysMRV.
| Model | ImplausiBench | IntPhys 2 | GRASP L2 |
|---|---|---|---|
gemini-3-flash-preview | 60.33 → 69.33 | 56.52 → 59.58 | 63.48 → 64.99 |
gpt-6-luna | 69.67 → 72.67 | 53.95 → 55.14 | 57.47 → 59.55 |
PhysMRV improves both models on every benchmark, even though their direct accuracy is well above that of the 8B backbones. This indicates that a memory bank and event graphs built once with an 8B open-source model remain useful for substantially stronger proprietary models.
5. Offline construction: quality. We use the AI tools to produce (i) one caption and one event graph per video (Qwen3-VL-8B), which serve as retrieval keys, and (ii) one rule-family label per instance (GPT-5.5).
- Constrained generation. Captions and event graphs are generated with deterministic decoding. Every event graph follows a fixed schema (type, participants, phase, states, state change, evidence), with event types and phases taken from predefined vocabularies. The rule label must be one of 15 fixed families, and GPT-5.5 chooses it from the question, the original explanation and the video.
- Human check. Two annotators manually evaluated 100 randomly sampled memory instances against their source videos, demonstrating substantial agreement. They found the generated outputs highly accurate, judging 91% and 93% of the captions correct (96% agreement, Cohen's ), and 98% and 99% of the event graphs correct (99% agreement, ). Both annotators also confirmed the assigned rule labels were correct in all 100 instances. Crucially, the event graph captures physical details much better than the caption. For example, of the 9 captions marked incorrect by the first annotator (6 of which were confirmed by the second), the corresponding event graphs remained completely accurate. Seven of these 9 videos feature a physical violation (e.g., a ball passing through a mirror, or a paper towel that does not absorb water); in these cases, the caption merely defaults to the expected outcome, whereas the event graph faithfully records what actually happens. Generation also holds up on low-quality videos: across the 24 videos at 176×144 resolution, the first annotator judged 23 captions and all 24 event graphs to be correct.
6. Why PhysMRV helps, and takeaways for future work.
- Knowing a rule is not enough; a matched case makes it usable. Retrieved rules alone perform on par with no memory (Rule Only vs. w/o Memory). Gains appear only when each rule comes with a similar case showing how it applies. This echoes InPhyRe: VLMs often know physical laws but fail to apply them.
- Relevance carries the gain, not extra context or labels. Shuffled Memory keeps the same content and verdict distribution but performs no better than no memory, and the majority verdict of the retrieved memories is at chance. Ranking by event-graph similarity also beats semantic and visual retrieval.
- Perception is the remaining bottleneck. Frozen VLMs mainly miss violations: Direct accuracy on implausible videos is far below that on plausible ones. Memories enter the prompt only as text, so the backbone must still locate the violation in the video. Accordingly, the gain is smallest for LLaVA-OneVision-2, the backbone with the weakest Direct accuracy.
Future work. Improving event perception, for example by training on paired counterfactual videos that differ only in the violation, and storing such contrastive pairs in memory.
We hope these answers address the reviewer's concerns, and we are glad to clarify anything further.
Gentle Follow-up on Our Rebuttal
Dear Reviewer YydG,
Thank you again for your time and valuable comments. We have carefully addressed each of your concerns in our response. As the discussion period ends soon, we would greatly appreciate it if you could take a look at our rebuttal when convenient and let us know if any concerns remain. We are happy to provide any further clarification.
Best regards,
The Authors
Response to Reviewer YydG (1/2)
We thank the reviewer for the helpful suggestions and answer each point below.
1. Token cost, latency and memory-bank cost. Measured with Qwen3-VL-8B on A100 GPUs. Direct is the same backbone answering without memory, values are Direct → PhysMRV.
We have added this cost table to the revised version.
2. CLIP retrieval baseline and "removing any module is below CLIP". We apologize for the confusion caused by its name. The “CLIP retrieval” row in Table 1 is not a standalone CLIP model. It is PhysMRV with the retriever changed: CLIP ViT-B/32 video similarity instead of our physics-aware retrieval, while the query event graph, memory content and verifier are all kept. It is therefore an ablation of our retrieval module rather than a simple baseline. This also explains why every ablation in Table 3 falls below it. Each ablation removes a module (the verifier, the event graph, or the memory) that the CLIP variant still retains, so falling below it confirms that each module contributes.
3. Formatting. Thank you for pointing this. We fixed the citation spacing ("InPhyRe (Sreekumar & Boddeti, 2026) reports"), the missing "Figure" in L159–161, and other typos.