- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper asks which embodied VQA supervision actually transfers to downstream VLA policies, a practical question that has received surprisingly little systematic attention despite being implicit in nearly every VLA training recipe. The authors build an automatic episode-to-VQA pipeline that separates what a question teaches (task), which episodes it comes from (source), and how the answer is presented (interface), and that runs on both simulated RoboCasa and real-world DROID episodes.
Critical strengths
- The studied problem is an important and well-posed question, and the paper addresses this with a simple and effective decomposition, separating supervision into what is taught, where episodes come from, and how the question is asked. This decomposition both simple and well motivated.
- The paper provided an automated, modular pipeline that is not tied to a particular model or task, which is reusable beyond the specific recipes studied here.
- Strong experimental control, which makes the comparisons credible. All recipes start from the same checkpoint and use the same GRPO configuration; every adapted VLM is converted into a policy with one shared training recipe (30,000 steps, two seeds); and every policy is evaluated on the full LIBERO-plus benchmark. Recipe pairs are matched in example count and in the distribution of difficulty, and comparisons are made at the level of individual variants, separating gained from lost successes.
Critical weaknesses
- The experimental scope is too narrow to support the general conclusions drawn. Although the findings are stated in general terms ("embodied supervision should be selected by the control conditions it improves and the successes it retains"), they rest on a single VLM backbone (Qwen3.5-4B) and a single policy framework (StarVLA/QwenGR00T). The paper would be considerably stronger if the headline conclusions were scoped to this setting, or if the key comparisons were replicated with a second backbone or policy framework.
- The interface study is not a clean manipulation of the answer interface. The abstract states that the paper changes only the answer interface from multiple choice to direct generation, but the body notes that this conversion "also changes the answer target and its verifier." Because the target and the reward function change together with the format, the observed dissociation that the VLM average rising from 54.85 to 55.53 while policy success falls from 83.16 to 80.87 cannot be attributed to the answer format alone. As written, this comparison does not isolate the effect of how the question is asked.
Questions for authors
- Can you replicate the central comparisons with a second VLM backbone and/or a different policy framework? Even a partial replication would substantially strengthen the claim.
- The interface comparison changes the answer target and the verifier in addition to the format. Can you isolate the effect of the interface itself?
3 comments
Thanks to the authors for their detailed response, which addressed my main concerns. I will keep my positive assessment.
Thank you for your thoughtful feedback! We are happy to see that our rebuttal helped address your concerns and we hope our clarifications will also be helpful to the broader discussion.
Response to Reviewer 8DKz
We sincerely thank you for your time and constructive feedback. We appreciate your recognition of the research question and modular pipeline. To address your concerns, we conducted additional experiments with another backbone and policy architecture, together with a controlled study of answer options. We summarize the results below. All revisions in the updated PDF are highlighted in blue.
1: Another backbone and policy architecture (W1 & Q1)
We appreciate this suggestion. To examine whether the findings extend beyond the original setting, we conducted two additional experiments:
For each configuration, we compared the Cross-view-adapted backbone with its unadapted baseline under the same downstream training and evaluation settings. The Qwen3.5-4B VLMs are the same models as in the paper, so their VLM scores are taken from the paper. The first two rows are the results reported in the paper.
With StarVLA-π, Cross-view improves policy success by 1.46% despite a lower Embodied VLM Avg., showing that the VLM–policy mismatch also holds for a different policy architecture. In all three settings, Cross-view shows gains under Camera, Language, and Light perturbations. On Gemma, most of these gains are lost to a drop under Noise (−6.93%), which also falls in the other two settings. Therefore, the Camera, Language, and Light gains hold in both new settings, while the overall and robot-state gains vary with the backbone and action head. We have added these results to the main text as Sec. 5.7 (Table 4).
2: Controlled answer-option comparison (W2 & Q2)
Thank you for raising this point. In the original comparison (Sec. 5.6, Table 3), removing the options also changes the answer target and verifier. To isolate the options, we fix the target and reward: both variants use the same 2,000 examples, images, prompt, and training settings, and generate the answer text inside
<answer>...</answer>. They differ only in whether the four candidates are shown.Original interface experiment
Controlled option experiment
The shared reward gives 1 to an exact correct answer, 0 to an exact incorrect candidate, and bounded partial credit to close paraphrases of the target.
With the answer target and reward being fixed, adding the candidate options increases policy success by 2.64% and by 8.39% under Robot perturbations while the Embodied VLM Avg. remains nearly unchanged. Combined with the original comparison, this result attributes the performance gap primarily to the presence of candidate options. When the options are shown, changing the answer target and reward only has a marginal effect (changing from 83.16% to 82.41%), but removing them reduces success to 79.77%.
We have added this controlled experiment to Sec. 5.6 and extended Table 3 with the two new variants. We have also revised the abstract to accurately reflect these findings.
Thank you again for your constructive suggestions. We hope these additional experiments and explanations address your concerns.