- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that embodied-planning benchmarks reward linguistically plausible next-step prediction rather than physical causal reasoning. Validation also includes no-vision controls, a cross-judge comparison, human calibration of the judge, and a data-scaling trend.
Critical strengths
- Unusually thorough evaluation-reliability analysis. No-vision controls drop MCQ accuracy to near chance. Rankings are stable across three judges (Spearman 0.90–1.00). Human calibration over 2,700 ratings quantifies both judge agreement (Spearman 0.835) and judge bias (+3.04). This makes the benchmark’s measurements more trustworthy than those of typical LLM-judged benchmarks.
- A reusable, well-documented resource. The structured annotations cover preconditions, effects, counterfactuals and failure recovery, with source-level train/test separation, full prompts, rubrics and provenance. They offer value to the community independent of the model results.
Critical weaknesses
- Circular, in-distribution evidence. GPT-5.6-sol annotates the data, generates the questions, writes the reasoning traces, judges the open-ended answers and supplies the RL reward. Training and test data come from the same pipeline. Beating the teacher in-domain therefore likely reflects learning the format and the judge’s preferences, not “physical causal reasoning.” External gains are modest (+3–5 points, still below several closed models), and RoboVQA is both a training source and an external test set.
- Claims outrun the evidence. The paper frames itself as moving beyond token prediction toward world modeling and physical agency. Yet the method is next-token SFT plus GRPO, the evaluation is entirely offline QA with no closed-loop execution, and ablations come from single training runs with differences of only 2–3 points.
Questions for authors
See the weakness.
3 comments
Thank you for the detailed rebuttal and the additional experiments. Given that the main contributions are the benchmark and the dataset, I am raising my score to 3.
Thank You for Your Thoughtful Reconsideration and Constructive Feedback
Dear Reviewer W4sB,
We sincerely thank you for taking the time to carefully read our rebuttal and consider the additional experiments and analyses. We are truly grateful for your encouraging response and thoughtful reassessment.
Your constructive feedback has been invaluable in helping us strengthen our experimental evidence and more clearly articulate the central contributions of Causal-Plan-Bench and Causal-Plan-1M. We deeply appreciate your careful consideration of our responses and your willingness to revisit your evaluation.
Thank you again for your valuable feedback and support. Your feedback has greatly contributed to improving the quality and clarity of our manuscript, and we are sincerely grateful for your support and encouragement.
Thank You for Your Insightful Assessment and Valuable Guidance on Evaluation Reliability, Training Stability, and the Scope of Our Claims
We sincerely thank you for your thoughtful review and recognition of our resources. Your feedback motivated further experiments and helped us clarify our findings.
W1. Shared pipeline and external overlap. GPT-5.6-sol generates annotations, QA pairs, traces, RL rewards, and open-ended scores. We examined this potential bias using matched training, expert review, exact-match scoring, visual controls, and alternative judges.
For benchmark curation, Qwen3.5-397B-A17B ranks candidates, Gemini 3.7 Flash audits grounding, and two experts independently inspect the evidence, answers, traces, distractors, and rubrics. Expert agreement on 2,400 admission decisions is 96.29% (α=0.940; B.5–B.6). Table 11 rejects items answerable from language priors alone; plausible MCQ distractors change decisive physical details, such as the faucet outlet versus its neck (Figure 5).
Six MCQ tasks use exact matching without an LLM judge: Causal Planner scores 46.67% versus 35.83% for its backbone and 37.83% for GPT-5.6-sol (Table 16). Open-ended tasks use expert-designed, task-specific rubrics that assess state changes, dependencies, and recovery, penalize unsupported claims, and accept equivalent wording (B.18). Three experts scored 900 answers (2,700 ratings), with ICC(A,3)=0.929; GPT-5.6-sol correlates with their mean (Spearman 0.835; MAE 5.58; offset +3.04; B.11/Table 18). GPT-5.6-sol, Qwen3.5-9B, and Seed2.1 Pro also rescored the same blinded five-model answers. Causal Planner ranks first under each (+13.26/+11.16/+13.09 over its backbone; rank ρ 0.90–1.00; Table 4). Two alternative GPT-5.6-sol prompts, using fixed answers and rubrics, preserve its lead (+14.58/+13.94; rank ρ 1.00/0.90). GRPO uses separate categorical reward rubrics, distinct from continuous benchmark scoring (B.19/B.18).
We trained a matched general-CoT control with the same filtered QA pairs, visuals, GPT-5.6-sol teacher, initialization, filtering, and SFT/RL settings. Both retain rationales; only guidance on preconditions, state changes, and dependencies differs. Despite 5.6% more reasoning tokens, general CoT scores 42.46 versus 45.28 for Causal Planner (−2.82) and trails on all three external benchmarks (−3.58 on Cosmos-Reason1). With the teacher fixed, this supports causal guidance beyond trace length.
Visual controls offer further evidence. Removing videos, keyframes, and visual-derived metadata while keeping questions and scoring fixed lowers Causal Planner Overall from 45.28 to 22.90 and MCQ accuracy from 46.67% to 26%, near four-choice chance (25%). GPT-5.6-sol and Gemini 3.1 Pro also lose 16.92 and 18.76 points (Table 3). Task 10 tests how lid removal enables washing; Tasks 18–20 examine flawed-plan repair (30.50→41.80), blocked-pot counterfactuals (33.40→46.50), and kettle–faucet recovery (35.50→47.20; Table 15).
We screened SFT/RL data against all four benchmark evaluation sets using QA hashes, MinHash, embedding similarity, and available source/video/trajectory IDs, removing detected overlaps. Causal-Plan-Bench source videos and derivatives are excluded from training, and we preserve RoboVQA's official episode-disjoint split (B.24). Without adaptation, Causal Planner improves on EgoPlan-Bench2 (+3.45), RoboVQA (+4.88), and Cosmos-Reason1 (+4.60). Its external mean is 57.35 (rank 4/15), close to Seed2.1 Pro's 57.83. The same backbone trained separately on external QA data yields means of 53.81/53.31/56.43, versus 57.35 for Causal Planner. These results support broader transfer.
W2. Training stability and claim scope. To assess ablation stability, we repeated each variant across three independent seeds. Paired gains (mean±sample SD):
All four gains remain positive across seeds (Table 6; B.22).
For the full Causal Planner, the three training scores are 45.28, 45.06, and 46.32 (45.55±0.67), distinct from fixed-checkpoint evaluation variation (45.28±0.82, B.13). A paired bootstrap over 1,200 instances gives a +2.24-point advantage over GPT-6-astra (95% CI [1.30, 3.10]). Figure 4(c) also shows gains as supervision grows.
Your point about the scope of our claims is well taken. Our primary contributions are Causal-Plan-Bench and Causal-Plan-1M. Causal Planner uses next-token SFT and GRPO to test whether structured supervision improves reasoning beyond plausible action sequences. Our results support improved offline reasoning about preconditions, effects, dependencies, and hypothetical recovery. Our revised discussion emphasizes this offline scope; we also plan VLA-based closed-loop tests of manipulation, long-horizon planning, and recovery.
We sincerely appreciate your careful and constructive feedback.