- Rating
- 3
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Capek0.5, a VLM focused on the execution-centric capabilities of embodied AI. It first introduces a taxonomy comprising four capability families: Spatial Reasoning, Temporal Understanding, Action Guidance, and State Verification. Guided by this taxonomy, the authors construct a training dataset by integrating multiple data sources, including newly generated data targeting state verification in the BEHAVIOR-1K simulator. Using this dataset, the authors first train 4 specialized expert models as teachers, then merge their checkpoints to initialize a student model, which is further optimized through multi-teacher on-policy distillation. The resulting model is evaluated on a range of general and embodied reasoning benchmarks, as well as in simulated environments such as EmbodiedBench to assess its closed-loop decision-making capabilities. Experimental results demonstrate that Capek0.5 outperforms other models of comparable size.
Critical strengths
This paper presents a comprehensive pipeline encompassing training data construction, benchmark development, and model training, reflecting a substantial amount of work. The evaluation extends beyond conventional QA tasks and offline benchmarks to assess the performance of embodied VLMs in closed-loop decision-making scenarios, providing a more practical and meaningful assessment.
Critical weaknesses
- The proposed capability taxonomy and the composition of the training dataset do not appear to offer substantial novelty over prior work. For example, RoboBrain2.5[1] has already considered spatial grounding and temporal understanding, while ERQA[2] and Cosmos-Reason[3] have introduced similar capability taxonomies. Although the authors construct new data for state verification, related efforts have also been explored in previous work, such as RoboBench[4], Progress-Bench[5].
- The proposed training pipeline builds upon existing techniques, including GRPO, TIES, and MOPD, but the paper does not sufficiently establish the necessity or effectiveness of each stage. The ablation results in Table 2 suggest that the individual contributions of TIES and MOPD are relatively modest on certain benchmarks. Moreover, these ablations are conducted on only a small selection of benchmarks, making it difficult to assess whether the observed benefits exist across different capabilities and tasks. In addition,l while the authors include a mix-RL baseline, it would also be valuable to consider simpler alternatives, such as mix-SFT, or mix-SFT followed by mix-RL. I think the supervised finetuning objective may lead to smaller gap among capabilities, potentially reducing the need for the proposed merging and distillation procedures.
[1] RoboBrain 2.5: Depth in Sight, Time in Mind
[2] Gemini Robotics: Bringing AI into the Physical World
[3] Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning
[4] Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain
[5] ProgressLM: Towards Progress Reasoning in Vision-Language Models
Questions for authors
Please see weaknesses.
2 comments
Q3. How does Capek compare with simpler supervised alternatives?
A. Thank you for the suggestion. We train a Mix-SFT baseline from the same Qwen3.6-35B-A3B initialization using the pooled training examples from the four specialist families. Capek outperforms Mix-SFT on 16 of the 20 embodied evaluation rows, with advantages spanning spatial reasoning, temporal understanding, and action guidance.
| Benchmark | Qwen | Mix-SFT | Capek 0.5 |
|---|---|---|---|
| Spatial Reasoning | |||
| CV-Bench | 88.56 | 88.02 | 89.99 |
| VSI-Bench | 60.83 | 62.85 | 70.69 |
| OmniSpatial | 58.84 | 48.79 | 56.36 |
| MindCube | 58.67 | 81.43 | 69.90 |
| ERQA | 60.00 | 48.00 | 58.25 |
| OpenEQA | 67.48 | 58.69 | 69.51 |
| All-Angles-Bench | 65.43 | 57.97 | 64.12 |
| RoboSpatial-Home | 63.46 | 64.27 | 72.80 |
| EmbSpatial-Bench | 82.94 | 81.59 | 84.59 |
| Temporal Understanding | |||
| EgoTempo | 38.20 | 31.20 | 44.60 |
| QVHighlights-TimeLens | 54.82 | 50.57 | 58.26 |
| LongVideoBench (64 frames) | 60.48 | 62.98 | 64.89 |
| MVBench (64 frames) | 68.55 | 72.18 | 71.32 |
| Video-MME (64 frames) | 71.33 | 70.04 | 74.19 |
| Action Guidance | |||
| PointBench | 76.66 | 68.94 | 81.22 |
| Where2Place | 57.61 | 68.10 | 73.52 |
| RoboRefIt (test B) | 84.45 | 84.15 | 85.96 |
| VABench-point | 54.71 | 51.28 | 59.33 |
| State Verification | |||
| StateBench-P | 65.80 | 93.20 | 76.80 |
| StateBench-T | 43.76 | 80.57 | 46.21 |
| General Capabilities | |||
| IFEval | 92.42 | 87.99 | 91.31 |
| LiveCodeBench v6 | 77.31 | 59.91 | 72.91 |
| MMLU-Pro | 84.39 | 83.12 | 84.77 |
| BFCL-v3 | 58.75 | – | 60.12 |
Mix-SFT performs substantially better on MindCube and StateBench-P/T, and slightly better on MVBench. This is particularly evident on StateBench, where the supervised targets closely match the evaluation tasks. For physical-state verification, SFT directly learns from state labels. For PVE, it also learns the required structured output through token-level supervision. In Capek, the subsequent GRPO stage instead optimizes format validity and task performance through rewards. This difference in supervision may contribute to the large gap on StateBench-T. We consider these results an important strength of the Mix-SFT baseline.
Despite these advantages, Capek performs better on 16 of the 20 embodied benchmarks, with improvements across spatial reasoning, temporal understanding, and action guidance. It also better preserves the general capabilities of the Qwen backbone. Mix-SFT drops by 4.43 points on IFEval and 17.40 points on LiveCodeBench v6, whereas Capek scores 91.31 and 72.91, respectively. On BFCL-v3, Mix-SFT produces outputs that cannot be parsed under the evaluation protocol, while Capek scores 60.12, compared with 58.75 for Qwen. Thus, Capek achieves broader embodied performance while retaining more of the backbone's instruction-following and coding capabilities.
Thank you for your valuable comments and for recognizing the breadth of our training pipeline and the practical value of our evaluation. Below we address the raised questions and concerns.
Q1. What does Capek add beyond prior work?
A. RoboBrain 2.5 develops precise spatial reasoning and temporal value estimation; Gemini Robotics demonstrates broad embodied reasoning capabilities and introduces ERQA; and Cosmos-Reason1 uses capability ontologies to guide data construction and post-training. Capek builds on these advances to address how complementary execution capabilities can be developed through focused specialization and retained together in a unified model.
Our execution-centric organization connects capability grouping directly to specialist training and practical embodied execution. This makes it possible to examine both what each specialist acquires and how those gains change when integrated into one checkpoint. Through comparisons with joint training and alternative consolidation routes, we provide empirical evidence on the benefits and trade-offs of this recipe, alongside its transfer to closed-loop execution.
A further distinction is the role of State Verification. Related work studies state estimation and progress reasoning; Capek brings local physical-state verification and task-level progress assessment together as a dedicated capability family, with aligned supervision and evaluation. This explicitly connects checking execution outcomes with assessing task completion and what should happen next, complementing the spatial, temporal, and guidance capabilities within the same model.
Q2. Effectiveness of each training stage and broader ablation coverage
A. To better demonstrate the effectiveness of each training stage, we expand the original eight-benchmark consolidation study to 21 evaluation rows across all four capability families. The comparisons examine the role of each stage:
Across the complete comparison, TIES+MOPD outperforms both Mix-RL and the original Qwen backbone on most of the benchmarks. Together, these results characterize the distinct roles of the three stages: specialist RL acquires execution-oriented capabilities, TIES provides a merged initialization, and MOPD further refines the consolidated policy. The resulting model combines complementary capabilities in a single checkpoint, although the gains are not uniform across benchmarks. These results support the effectiveness of the evaluated recipe without implying that each stage improves every capability.