- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Embodied Choice Tuning (ECT), a data-centric framework for improving embodied vision-language models through multiple-choice supervision. The authors construct a dataset of 10,000 examples covering eight embodied capabilities and compare four supervision formats: Letter, Content, Joint, and Open. The framework combines semantically confusable distractors with compact letter-based supervision. Experiments across three model families and nine Embodied Arena benchmarks show that ECT achieves the highest aggregate performance among the tested formats. Additional experiments investigate the contributions of candidate availability, distractor difficulty, and output target formulation.
Critical strengths
-
Well-controlled comparison of supervision formats. The authors construct four supervision variants from identical visual observations, questions, and correct answers. In particular, comparing Content and Open while keeping target sequences identical helps isolate the contribution of candidate information. This experimental design provides useful evidence about how supervision formulation affects embodied VLM adaptation.
-
Broad empirical evaluation with consistent improvements. The study evaluates three model families across nine benchmarks covering spatial understanding, visual reasoning, and embodied perception. ECT achieves the highest aggregate score among the four formats in all five reported model and seed comparisons. The additional Near versus Far experiments provide evidence that distractor construction affects transfer performance.
Critical weaknesses
-
Limited methodological novelty and relatively modest improvements. The proposed framework primarily combines established multiple-choice supervision with semantically similar distractors and short answer targets. Both multiple-choice training and hard distractor construction have been extensively explored in prior work. The main contribution is their empirical comparison in embodied VLMs rather than a fundamentally new learning method. Moreover, the average improvement over Content supervision is only 1.16 points on Qwen3.5-4B, making the practical significance of the specific Letter formulation less clear.
-
Incomplete isolation of the supervision mechanism. Although the authors carefully control training examples, the compared target formats differ substantially in sequence length and effective example weighting. Table 7 reports an average of 2 supervised tokens for Letter versus 8.76 for Content and 10.93 for Joint. Consequently, the observed advantage may partly reflect differences in loss weighting and optimization rather than the semantic benefits of compact choice supervision. Additionally, the Near versus Far comparison is conducted with only one training seed, and the evaluation focuses entirely on offline perception and reasoning rather than downstream robotic action performance.
Questions for authors
-
Supervision length confounding. Have the authors considered normalizing the training loss independently for each example, so that longer answer targets do not receive greater weight? Would Letter supervision retain its advantage over Content and Joint under this control?
-
Distractor difficulty. The Near versus Far experiments use only one training seed, and Appendix A reports that approximately 39% of examples retain identical distractors in both conditions. Can the authors provide additional evidence that the improvement is robust across seeds and directly attributable to distractor difficulty?
-
Practical significance for embodied learning. The reported improvements are measured on offline VLM benchmarks. Is there evidence that multiple-choice supervision improves downstream robotic decision-making or manipulation performance? Without such evidence, how should the claimed benefits for embodied models be interpreted?
8 comments
Response to Reviewer gjsU (2/6) — Q2 (1/2): Cross-seed and cross-model robustness
2. Near–Far robustness, unchanged distractors, and distractor difficulty
Question: Near–Far uses only one seed, and approximately 39% of examples have unchanged distractors. Can robustness and the role of distractor difficulty be better supported?
Response: We added cross-seed, cross-model, and random-distractor comparisons, all supporting Near's training value, and audited candidate replacement across the 10K corpus.
2.1 Additional experiments reproduce Near's aggregate advantage
The original comparison uses Qwen3.5-4B seed 42 with Letter, Content, and Joint targets:
Table 3a. Near–Far results already reported in the submission, retaining its two-decimal precision.
| Target (original Qwen, seed42) | Near | Far | Near − Far |
|---|---|---|---|
| Letter | 57.23 | 56.15 | +1.08 |
| Content | 56.72 | 54.68 | +2.04 |
| Joint | 56.71 | 54.95 | +1.76 |
We added Letter comparisons for Qwen seeds 43 and 44 and InternVL3.5-4B-Instruct seeds 42, 43, and 44. The original seed-42 result is reported separately rather than pooled with the new batch into a three-seed mean.
Table 3b. Additional paired Near–Far results with Letter supervision.
| Model (additional Letter comparison) | seed | Near | Far | Near − Far |
|---|---|---|---|---|
| Qwen3.5-4B | 43 | 57.470 | 55.366 | +2.104 |
| Qwen3.5-4B | 44 | 57.243 | 55.894 | +1.349 |
| Qwen3.5-4B | Two-seed mean | 57.356 | 55.630 | +1.726 |
| InternVL3.5-4B-Instruct | 42 | 44.225 | 43.681 | +0.544 |
| InternVL3.5-4B-Instruct | 43 | 44.395 | 43.518 | +0.877 |
| InternVL3.5-4B-Instruct | 44 | 44.249 | 43.371 | +0.878 |
| InternVL3.5-4B-Instruct | Three-seed mean | 44.290 | 43.524 | +0.766 |
The new Qwen seeds improve by 2.104 and 1.349 points, and the InternVL seeds by 0.544, 0.877, and 0.878 points. Every additional paired comparison is positive across both families, providing repeated evidence for the finding.
The comparison fixes the question, visual input, correct answer, correct position, supervision target, and training budget. Qwen Near uses sample normalization and Far uses token normalization; every Letter target contains two supervised tokens, including the terminator, so the two normalizations are mathematically equivalent in this comparison.
Response to Reviewer gjsU (3/6) — Q2 (2/2): Unchanged and random distractors
2.2 Unchanged distractors reflect constraints on valid alternatives
Near and Far contain the same 10K questions. We replace distractors where valid different alternatives can be constructed and retain the remaining examples:
Table 3c. Candidate-replacement coverage in the complete training corpus.
| Candidate changes | Questions | Share of training set |
|---|---|---|
| At least one distractor replaced | 6,098 | 60.98% |
| Distractors unchanged | 3,902 | 39.02% |
| Total | 10,000 | 100% |
Of the 6,098 changed examples, 6,017 replace all three distractors, 71 replace two, and 10 replace one, totaling 18,203 replaced incorrect options. Among the 3,902 unchanged examples, 3,874 have finite answer domains; the remaining 28 involve symmetrically retained source-annotation disputes or candidate overlap after review. Construction records are described in Appendix A of the submission.
The unchanged records are identical on both sides, preserving question count, training content, and capability coverage, with differences confined to the incorrect candidates of the other 6,098 questions. This intervention holds the complete training corpus fixed: changing incorrect candidates while retaining the correct answers and training content produces the repeatable transfer gains in Table 3b.
2.3 Random distractors provide additional evidence for candidate construction
We additionally sample incorrect candidates from the full answer pool while retaining questions, visual inputs, correct answers, and their positions.
Table 4. Additional three-seed comparisons of Near and random distractors with Letter targets.
| Model (three seeds) | Near mean | Random mean | Mean difference | seed42 difference | seed43 difference | seed44 difference | Positive seeds |
|---|---|---|---|---|---|---|---|
| Qwen3.5-4B | 57.340 | 55.360 | +1.981 | +2.985 | +1.149 | +1.809 | 3/3 |
| InternVL3.5-4B-Instruct | 44.290 | 41.538 | +2.751 | +2.896 | +2.646 | +2.712 | 3/3 |
Near's mean advantages over Random are +1.981 points for Qwen and +2.751 for InternVL, with positive differences in all three seeds of each model. Together with Near–Far, these results show that candidate construction influences transfer.
Near–Far is constructed according to confusability relative to visual evidence. Both sides preserve the question's answer domain and comparable answer granularity; Far distractors must clearly conflict with visual evidence and undergo structural checks and independent model review. The paired design and additional experiments jointly support the training value of constructing distractors according to semantic confusability.
Paired intervention, repeated training, and the random-distractor control together strengthen the paper's conclusion that confusable distractors have training value.
Response to Reviewer gjsU (1/6) — Q1: Sample-normalized loss
Thank you for your review and suggestions. We address your questions below and believe that the additional evidence and clarifications resolve your concerns.
1. Does Letter remain ahead of Content and Joint after sample normalization?
Question: Different target lengths may give longer answers greater training weight. Can loss be normalized independently per example, and does Letter retain its advantage?
Response: We completed this control. Letter remains ahead of both Content and Joint in all three seeds, retaining the highest aggregate among the four supervision formats.
The submission reports average supervised lengths of 2, 8.76, and 10.93 tokens for Letter, Content, and Joint. To address length-dependent example weighting directly, we use:
Lsample=N1∑i=1N(Ti1∑t=1Tiℓi,t).
Here, Ti is the supervised-token count for example i, and N is the number of examples in the actual gradient-accumulation window. We average within each example before averaging across examples, preventing longer answers from receiving greater example weight solely because they contain more supervised tokens.
We trained each of Letter, Content, Joint, and Open with seeds 42, 43, and 44, for 12 runs. They share the base model, the same 10K questions, two training epochs, effective batch size 64, and evaluation of the final step-314 checkpoint. Letter, Content, and Joint share Near candidates; Open removes candidates and supervises the corresponding correct answer text.
Table 1. Qwen3.5-4B nine-benchmark aggregates with sample-normalized loss.
Table 2. Paired advantages under the normalization control.
After normalization, Letter's average advantages are +1.309 points over Content, +0.364 over Joint, and +2.102 over Open. Both Letter–Content and Letter–Joint are positive in all three seeds. This directly answers your question: Letter's aggregate advantage remains after sample normalization. The original ordering withstands this additional control for example weighting.