- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
FlexSense uses one VLA policy across optional sensor configurations through shared and sensor-specific residual encoding, set-conditioned FiLM, and modality dropout. UtilSelect learns from a frozen policy's action errors to choose sensor inputs for each action chunk. Experiments cover LIBERO, ManiSkill, real robots, and transfer between demonstrations collected with different sensors.
Critical strengths
- One checkpoint approaches specialist performance. Across four configurations, average success is 97.14% versus 96.80% for separate specialists on LIBERO, and 56.4% versus 57.2% on ManiSkill (Tables 3–4, p.7). This supports reducing the need to train and maintain a policy for each configuration.
- Controlled transfer from heterogeneous demonstrations. With F data and evaluation fixed, adding separately collected D demonstrations yields 93.6% for ShareRes versus 89.8% for Independent. Initialization, splits, and optimization budgets are matched. This directly supports the value of sharing representations across configurations.
Critical weaknesses
Small component gains need stronger uncertainty estimates. The full architecture gains 1.00 percentage point over dropout alone; UtilSelect gains 0.45 points over the best fixed subset (Tables 6–7). Only one training seed is reported, and the scorer's fitting/selection/evaluation splits and fixed-subset selection rule are unclear (Appendix D.1). These concerns mainly affect component claims. Also, the selector reads all available sensor representations, so using fewer policy inputs does not directly establish savings in physical sensing energy.
Questions for authors
- Evaluation protocol: How are episodes split between scorer fitting, checkpoint selection, and final evaluation? On which data, and by what rule, is the best fixed subset selected?
- Existing-result analysis: Please report paired uncertainty for the main differences in Tables 6–7 using task or episode records. If additional training seeds already exist, please include them.
3 comments
Thanks for clarifying the splits and adding paired intervals. These address most of my original concerns. The scorer seeds provide useful evidence, though the VLA checkpoint remains fixed.
The remaining point is the “best fixed subset” baseline, which is actually the best of three random task-level assignments. Please label it accordingly and clarify how UtilSelect compares with each feasible constant subset using the same checkpoint, drawing on existing evaluations where available. I maintain my rating of 3, based mainly on configuration sharing and transfer.
Response to Reviewer 3Duc
Thank you for the clarification. We will rename the original baseline “Best of three random task-level assignments” (RTA, best of 3).
The table below compares UtilSelect with every feasible constant subset. All comparisons use the same underlying model checkpoint and evaluation protocol, without subset-specific retraining. Each constant subset is fixed across all tasks and queries. UtilSelect results are averaged over three scorer training seeds (Table 7 and Appendix Table A8).
| Available auxiliary sensors | Fixed ∅ | Fixed D | Fixed F | Fixed D+F | UtilSelect |
|---|---|---|---|---|---|
| D | 96.10% | 96.20% | — | — | 97.10% |
| F | 96.10% | — | 96.40% | — | 96.97% |
| D+F | 96.10% | 96.20% | 96.40% | 96.60% | 97.33% |
A dash indicates that the fixed subset requires a sensor that is unavailable in that setting.
Across the three sensor availability settings, UtilSelect improved mean success over every constant subset. We will include this comparison in the appendix and refer to it in the discussion of Table 7.
Thank you again for your constructive feedback.
Point-to-Point Response to Reviewer 3Duc
W1 and Q2 — Uncertainty and reproducibility of component gains
We performed task-level paired analyses for the main differences in Tables 6–7. Both analyses use 40 LIBERO tasks, with the results summarized below.
(i) Component ablation (Table 6). We paired the full architecture and Dropout only by task, averaging each task’s success rate across the four sensor configurations. The full architecture improved average success from 95.60% to 96.60%. The positive paired interval supports an average benefit from combining ShareRes and SetFiLM beyond modality dropout alone.
(ii) UtilSelect (Table 7). With the FlexSense checkpoint fixed, UtilSelect achieved 97.13 ± 0.08% success (mean ± sample SD) across three scorer seeds, outperforming the fixed-assignment baseline (96.67%) by 0.47 PP on average. For the paired analysis, we averaged across three sensor configurations and three scorer seeds per task, then computed the interval over 40 task-level differences. Configuration-level results are provided in our Overall Response, “UtilSelect reproducibility.” (Table 7; Appendix G.2, Table A8)
W2 and Q1 — UtilSelect evaluation protocol and fixed-subset baseline
(i) UtilSelect evaluation protocol. We fit the UtilSelect scorer on 1,597 expert episodes and selected its checkpoint on 179 separate expert episodes. We trained for 40 epochs and retained the checkpoint with the lowest validation macro-regret, breaking ties in favor of the earliest epoch. Final evaluation uses 50 official LIBERO initial states per task across four suites of 10 tasks each, yielding 2,000 closed-loop rollouts per availability condition. (Appendix A.1, Table A2; Appendix G.2)
(ii) Fixed-subset baseline. We apologize for the ambiguity in the original manuscript. For each sensor availability condition in Table 7, we randomly generated three assignments of feasible sensor subsets to tasks. Under each assignment, every task used its assigned subset throughout all evaluation rollouts. We evaluated all three assignments using the same LIBERO evaluation initial states and reported the assignment with the highest mean success rate across the four suites. We label this baseline as “Best fixed subset.” The protocol is now described in Section 4.4 and Appendix G.2.
W3 — Physical sensing and energy costs
We apologize for the confusion. We did not claim that passing fewer sensor inputs to the action policy reduces physical sensing energy consumption. UtilSelect operates on representations from all available sensors and therefore does not reduce sensing or encoding costs. Instead, it improves task performance by adaptively selecting which sensor representations are passed to the action policy, with only +0.41% inference overhead. The timing protocol and results are provided in our Overall Response, “Inference cost.” (Table 7; Appendix G.2, Table A9)
Moreover, a single FlexSense policy supports all four sensor configurations. In a matched training benchmark, FlexSense took 2.01 s per optimizer step, while the four configuration-specific specialists took 1.76–1.85 s per step. With the same number of updates, training one shared policy reduces training time by 72% across the four configurations. The savings can grow as more optional modalities create more configurations.