acceptodds
Under review as a conference paper at ICLR 2027

FlexSense: Flexible Sensor Configurations with a Unified Vision-Language-Action Model

Abstract

Vision-Language-Action (VLA) models increasingly generalize across tasks, environments, and robot embodiments, yet most remain coupled to a fixed sensory interface. In practice, robots expose different combinations of optional sensors, and training a separate policy for every configuration scales poorly and prevents demonstrations collected under different sensing setups from being fully shared. We introduce FlexSense, a unified VLA framework for robotic manipulation under heterogeneous sensor configurations. It consists of three modules: First, Shared-Residual Sensory Encoding (ShareRes) captures transferable control structure through a shared sensory backbone while preserving sensor-specific cues through lightweight residual adapters gated by zero-initialized scalars. Second, Sensor-Set-Conditioned FiLM (SetFiLM) explicitly conditions each action block on the active sensor set, complementing token masking with configuration-aware action generation. Third, Utility-Guided Sensor Selection (UtilSelect) derives subset-dependent supervision from counterfactual action errors of the frozen policy, distills it into a lightweight scorer, and selects the sensor subset with the lowest predicted action error at each action-chunk query. We evaluate FlexSense on LIBERO, ManiSkill, and real-world robotic manipulation across multiple sensor configurations. Across LIBERO and ManiSkill, FlexSense remains within 1.6 percentage points of configuration-specific specialists in configuration-level average success rate across all tested sensor configurations. Real-robot experiments further show that the same policy benefits from complementary depth and tactile sensing, reaching 100% success on Pick-and-Place and 80% on Precision Plug Insertion. Our code will be released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Robot Learning, Multimodal Learning, Heterogeneous Sensor Configurations, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3DucOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FlexSense uses one VLA policy across optional sensor configurations through shared and sensor-specific residual encoding, set-conditioned FiLM, and modality dropout. UtilSelect learns from a frozen policy's action errors to choose sensor inputs for each action chunk. Experiments cover LIBERO, ManiSkill, real robots, and transfer between demonstrations collected with different sensors.

Critical strengths
  • One checkpoint approaches specialist performance. Across four configurations, average success is 97.14% versus 96.80% for separate specialists on LIBERO, and 56.4% versus 57.2% on ManiSkill (Tables 3–4, p.7). This supports reducing the need to train and maintain a policy for each configuration.
  • Controlled transfer from heterogeneous demonstrations. With F data and evaluation fixed, adding separately collected D demonstrations yields 93.6% for ShareRes versus 89.8% for Independent. Initialization, splits, and optimization budgets are matched. This directly supports the value of sharing representations across configurations.
Critical weaknesses

Small component gains need stronger uncertainty estimates. The full architecture gains 1.00 percentage point over dropout alone; UtilSelect gains 0.45 points over the best fixed subset (Tables 6–7). Only one training seed is reported, and the scorer's fitting/selection/evaluation splits and fixed-subset selection rule are unclear (Appendix D.1). These concerns mainly affect component claims. Also, the selector reads all available sensor representations, so using fewer policy inputs does not directly establish savings in physical sensing energy.

Questions for authors
  • Evaluation protocol: How are episodes split between scorer fitting, checkpoint selection, and final evaluation? On which data, and by what rule, is the best fixed subset selected?
  • Existing-result analysis: Please report paired uncertainty for the main differences in Tables 6–7 using task or episode records. If additional training seeds already exist, please include them.
3 comments
AuthorsOpenReview ↗

Point-to-Point Response to Reviewer 3Duc

W1 and Q2 — Uncertainty and reproducibility of component gains

We performed task-level paired analyses for the main differences in Tables 6–7. Both analyses use 40 LIBERO tasks, with the results summarized below.

ComparisonMean paired gain (pp)95% paired tt-interval (pp)
Full vs. Dropout only1[+0.75, +1.25]
UtilSelect vs. Best fixed subset0.47[+0.25, +0.69]

(i) Component ablation (Table 6). We paired the full architecture and Dropout only by task, averaging each task’s success rate across the four sensor configurations. The full architecture improved average success from 95.60% to 96.60%. The positive paired interval supports an average benefit from combining ShareRes and SetFiLM beyond modality dropout alone.

(ii) UtilSelect (Table 7). With the FlexSense checkpoint fixed, UtilSelect achieved 97.13 ± 0.08% success (mean ± sample SD) across three scorer seeds, outperforming the fixed-assignment baseline (96.67%) by 0.47 PP on average. For the paired analysis, we averaged across three sensor configurations and three scorer seeds per task, then computed the interval over 40 task-level differences. Configuration-level results are provided in our Overall Response, “UtilSelect reproducibility.” (Table 7; Appendix G.2, Table A8)

W2 and Q1 — UtilSelect evaluation protocol and fixed-subset baseline

(i) UtilSelect evaluation protocol. We fit the UtilSelect scorer on 1,597 expert episodes and selected its checkpoint on 179 separate expert episodes. We trained for 40 epochs and retained the checkpoint with the lowest validation macro-regret, breaking ties in favor of the earliest epoch. Final evaluation uses 50 official LIBERO initial states per task across four suites of 10 tasks each, yielding 2,000 closed-loop rollouts per availability condition. (Appendix A.1, Table A2; Appendix G.2)

(ii) Fixed-subset baseline. We apologize for the ambiguity in the original manuscript. For each sensor availability condition in Table 7, we randomly generated three assignments of feasible sensor subsets to tasks. Under each assignment, every task used its assigned subset throughout all evaluation rollouts. We evaluated all three assignments using the same LIBERO evaluation initial states and reported the assignment with the highest mean success rate across the four suites. We label this baseline as “Best fixed subset.” The protocol is now described in Section 4.4 and Appendix G.2.

W3 — Physical sensing and energy costs

We apologize for the confusion. We did not claim that passing fewer sensor inputs to the action policy reduces physical sensing energy consumption. UtilSelect operates on representations from all available sensors and therefore does not reduce sensing or encoding costs. Instead, it improves task performance by adaptively selecting which sensor representations are passed to the action policy, with only +0.41% inference overhead. The timing protocol and results are provided in our Overall Response, “Inference cost.” (Table 7; Appendix G.2, Table A9)

Moreover, a single FlexSense policy supports all four sensor configurations. In a matched training benchmark, FlexSense took 2.01 s per optimizer step, while the four configuration-specific specialists took 1.76–1.85 s per step. With the same number of updates, training one shared policy reduces training time by 72% across the four configurations. The savings can grow as more optional modalities create more configurations.

Thanks for clarifying the splits and adding paired intervals. These address most of my original concerns. The scorer seeds provide useful evidence, though the VLA checkpoint remains fixed.

The remaining point is the “best fixed subset” baseline, which is actually the best of three random task-level assignments. Please label it accordingly and clarify how UtilSelect compares with each feasible constant subset using the same checkpoint, drawing on existing evaluations where available. I maintain my rating of 3, based mainly on configuration sharing and transfer.

AuthorsOpenReview ↗

Response to Reviewer 3Duc

Thank you for the clarification. We will rename the original baseline “Best of three random task-level assignments” (RTA, best of 3).

The table below compares UtilSelect with every feasible constant subset. All comparisons use the same underlying model checkpoint and evaluation protocol, without subset-specific retraining. Each constant subset is fixed across all tasks and queries. UtilSelect results are averaged over three scorer training seeds (Table 7 and Appendix Table A8).

Available auxiliary sensorsFixed ∅Fixed DFixed FFixed D+FUtilSelect
D96.10%96.20%——97.10%
F96.10%—96.40%—96.97%
D+F96.10%96.20%96.40%96.60%97.33%

A dash indicates that the fixed subset requires a sensor that is unavailable in that setting.

Across the three sensor availability settings, UtilSelect improved mean success over every constant subset. We will include this comparison in the appendix and refer to it in the discussion of Table 7.

Thank you again for your constructive feedback.

Reviewer_aTsuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents FlexSense, a unified vision-language-action framework designed to support robotic manipulation under heterogeneous sensor configurations. Motivated by the difficulty of deploying conventional VLA policies across robots with different sensing capabilities, the proposed method aims to enable a single policy to operate with varying combinations of optional sensors without configuration-specific retraining.

The framework consists of three main components. First, Shared-Residual Sensory Encoding (ShareRes) combines a shared sensory backbone with lightweight modality-specific residual adapters to capture transferable information while preserving sensor-specific representations. Second, Sensor-Set-Conditioned FiLM (SetFiLM) explicitly conditions action generation on the active sensor subset through feature-wise scale and shift modulation. Third, Utility-Guided Sensor Selection (UtilSelect) estimates the expected action prediction error associated with different sensor subsets and dynamically selects the most promising subset at each action-chunk query.

The authors evaluate FlexSense on LIBERO, ManiSkill, and real-world robotic manipulation tasks. The results demonstrate that a single unified policy can achieve performance close to configuration-specific specialists across multiple sensor configurations. Real-world experiments further illustrate the complementary benefits of depth and tactile sensing for manipulation tasks involving geometric alignment and contact-sensitive control.

Critical strengths
  1. The paper addresses an important practical challenge in generalist robot learning: the dependency of current VLA architectures on fixed sensory interfaces. In realistic robotic deployments, different platforms may provide different combinations of RGB, depth, force, and tactile observations. Training and maintaining separate policies for every possible configuration can be expensive and prevents effective reuse of demonstrations collected under heterogeneous sensing setups. The proposed formulation of configuration-compatible VLA learning is therefore meaningful. I particularly appreciate the distinction between the physically available sensor set and the subset actively used for each action prediction, as it provides a clear framework for studying both sensor compatibility and adaptive sensor utilization.

  2. The proposed framework provides a reasonable and systematic approach to handling variable sensor configurations. ShareRes combines shared sensory representations with modality-specific residual corrections, aiming to balance knowledge transfer and preservation of modality-specific information. SetFiLM further conditions the action expert on the active sensor subset, extending beyond simply masking unavailable sensory tokens. The configuration-constrained modality dropout strategy is also appropriate for heterogeneous demonstrations, since it avoids synthesizing unavailable modalities and only samples subsets supported by the original observations. Together, these components form a coherent architecture that can be integrated into existing pretrained VLA policies without requiring separate action-generation models for different sensor configurations.

  3. The paper provides several useful experiments that directly examine its central motivation. In particular, the specialist-versus-unified comparisons demonstrate that a single FlexSense checkpoint achieves performance close to separately trained specialists on both LIBERO and ManiSkill. The cross-configuration data utilization experiment is also valuable, showing that shared sensory representations can benefit from demonstrations collected under another sensing configuration. Furthermore, the real-world evaluation includes both pick-and-place and precision plug insertion tasks. The latter is particularly relevant because it involves contact-sensitive manipulation where tactile feedback can provide information beyond conventional visual observations. The improvement from 20% success without optional sensors to 80% with combined depth and tactile sensing is encouraging and demonstrates the practical value of incorporating complementary physical sensing modalities.

Critical weaknesses
  1. My primary concern is the extent of the methodological novelty introduced by the proposed framework. The individual components are closely related to established techniques in multimodal learning. ShareRes follows the general idea of shared-specific representation modeling, SetFiLM applies feature-wise conditioning to encode sensor availability, and configuration-constrained modality dropout builds upon existing missing-modality training strategies. Although their integration into a unified VLA framework is reasonable and practically motivated, the paper could more clearly establish which aspects constitute new methodological contributions beyond adapting these techniques to robotic manipulation. Furthermore, the component ablations suggest that the improvements attributable to the proposed architectural components are relatively modest. On LIBERO, modality dropout alone achieves 95.60% average success across configurations, while combining ShareRes and SetFiLM increases this to 96.60%, corresponding to a 1.00-percentage-point improvement. While these results support the usefulness of the proposed modules, additional comparisons with simpler configuration-aware fusion baselines would help clarify whether the architectural complexity is necessary.

  2. UtilSelect is presented as one of the three main contributions, but the current experiments provide relatively limited evidence for its advantages over simpler sensor-selection strategies. As shown in Table 7, UtilSelect achieves 97.12% average success on LIBERO, compared with 96.67% for the best fixed sensor subset and 96.40% when using all available sensors. These correspond to improvements of only 0.45 and 0.72 percentage points, respectively. Although these gains are positive, their practical significance remains unclear, particularly because the paper does not report variability across independent training seeds. More importantly, the real-world experiments primarily compare the performance of the same policy under different sensor configurations. They do not directly demonstrate that dynamic sensor selection improves physical manipulation performance over a fixed configuration using all available sensors. There is also a distinction between selecting which sensory representations are passed to the action policy and reducing the actual cost of sensor acquisition or processing. Since the selection scorer uses representations from the available sensory inputs, the reported selection mechanism does not by itself establish reductions in end-to-end sensing or computation costs. Consequently, I believe stronger evidence is needed to establish the independent practical contribution of UtilSelect, ideally through real-world adaptive-selection comparisons, repeated evaluations, and more explicit efficiency measurements.

  3. Although FlexSense is motivated by the diversity and variability of sensing configurations in real-world robotic systems, the experimental evaluation considers at most two optional sensors at a time, resulting in only four possible configurations per setting. This provides useful evidence that the proposed framework can support a small number of predefined sensor combinations, but it does not fully establish its scalability to substantially more heterogeneous sensing systems. Moreover, the experiments do not explicitly evaluate unexpected sensor loss, latency, or corruption during ongoing manipulation. These scenarios are particularly relevant to the motivation of flexible sensing, where sensor availability and reliability may change during deployment. While the cross-configuration data transfer experiments are encouraging, a substantial portion of the evaluation is conducted in simulation or through controlled modality masking. It remains unclear how effectively the framework would handle differences in sensor calibration, observation quality, and temporal alignment across independently collected real-world datasets. Additional experiments involving mid-episode sensor failures, noisy sensory inputs, and larger sensor sets would strengthen the generality and practical robustness claims.

Questions for authors
  1. Could the authors further clarify the main methodological contributions of ShareRes and SetFiLM relative to existing shared-specific representation learning, modality dropout, and feature-wise conditioning methods? In particular, could the authors compare FlexSense with simpler alternatives, such as modality dropout with learned sensor embeddings, independent sensor encoders with configuration conditioning, or lightweight fusion modules under matched computational budgets? Such comparisons would help determine which architectural components are essential for configuration-compatible VLA learning.

  2. Could the authors provide additional experiments directly evaluating the benefits of UtilSelect in real-world manipulation? For example, on the precision plug insertion task, how does dynamically selecting depth and tactile sensing compare with always using both sensors or using the best fixed subset? It would also be useful to report the variance of the observed improvements across multiple training seeds and to clarify whether adaptive selection reduces end-to-end sensing or inference costs when accounting for the scorer and the processing of available sensor inputs. These results would help establish the practical significance of the adaptive selection mechanism.

  3. How does FlexSense perform when sensor availability or reliability changes unexpectedly during execution? Could the authors evaluate scenarios involving mid-episode sensor dropout, corrupted force or tactile observations, or delayed sensor measurements? Additionally, can the proposed framework scale to settings with more than two optional sensors, where the number of candidate subsets grows exponentially? Such evaluations would provide stronger evidence for the framework's ability to support flexible sensing in realistic robotic deployments.

3 comments
AuthorsOpenReview ↗

Point-to-Point Response to Reviewer aTsu (1/2)

W1 and Q1 — Methodological contributions and simpler baselines

(i) Methodological novelty. We propose a modality-flexible approach to robotic manipulation using a single VLA policy. To the best of our knowledge, FlexSense is the first to jointly enable learning from demonstrations collected under heterogeneous native sensor configurations and adaptive sensor-subset selection during execution, without paired multimodal observations, modality imputation, or configuration-specific retraining. Our methodological contributions are threefold:

  1. Cross-configuration knowledge transfer. ShareRes transfers knowledge across heterogeneous sensor configurations while preserving modality-specific information.

  2. Configuration-aware action generation. SetFiLM directly modulates action computation according to the active sensor subset, rather than relying solely on input masking or sensor embeddings.

  3. Adaptive sensor utilization. UtilSelect selects observation-dependent sensor subsets using action-error supervision, allowing the same policy to adjust its sensory inputs during execution without retraining.

(ii) Comparison with prior methods:

To better address your concerns, we investigated representative methods in each direction and conducted the following comparative analyses:

1. Shared–specific representation learning. MISA [ref1], a representative work with over 2,000 citations, feeds shared and private vectors separately into a fusion Transformer for sentiment prediction. Its prediction head uses a fixed concatenation of these vectors, so applying this design to changing sensor sets requires a rule for handling missing positions. ShareRes adds a gated bottleneck residual directly to each sensor’s shared tokens, and the action policy attends only to the optional sensor tokens that are active. This keeps the private branch lightweight, preserves the token count per sensor, and lets one policy operate across sensor configurations. The shared Transformer also lets depth-only demonstrations update the parameters used for force-only control. With the force training pool held fixed and full depth-only data coverage, ShareRes achieves 93.6% force-only success versus 89.8% with independent encoders (Table 5).

2. Modality dropout. HeMIS [ref2], an early missing-modality method with about 600 citations, starts training with complete MRI inputs and then randomly drops modalities. This full-modality warm-up is difficult to apply to robot demonstrations collected with different sensor suites, where some recordings are already incomplete. FlexSense starts from the sensors actually recorded in each demonstration and intersects that set with the candidate dropout mask. The same expert action can then supervise multiple feasible configurations. For example, a depth-only demonstration can train the policy with depth enabled or with all optional sensors disabled. This lets demonstrations collected with different hardware train one policy. The resulting subset also conditions SetFiLM through the same interface used by UtilSelect at deployment. In our LIBERO ablation, modality dropout improves average success from 94.75% to 95.60% (Table 6).

3. Feature-wise conditioning. AdaIN [ref3], a widely cited conditional normalization method with over 7,500 citations, derives scale and shift from the feature statistics of a reference image. For a policy whose sensor inputs can change between action queries, feature statistics provide only an indirect signal about which sensor streams are active. SetFiLM maps the active sensor set to a separate scale and shift for every Action DiT block and adds them to the existing flow-timestep modulation. Zero initialization preserves the pretrained modulation when SetFiLM is introduced. During execution, switching from depth and force to force alone changes both the tokens visible to attention and the modulation inside each action block. Adding SetFiLM to modality dropout improves average LIBERO success from 95.60% to 96.10% (Table 6).

Taken together, each component introduces meaningful methodological advances tailored to robotic manipulation. More importantly, their synergistic integration establishes, to the best of our knowledge, the first modality-flexible approach for VLA-based robotic manipulation that jointly enables learning from heterogeneous native sensor configurations and adaptive sensor-subset selection within a single policy. This enables cross-configuration knowledge transfer, configuration-aware action generation, and dynamic sensor utilization without configuration-specific retraining.

[ref1] MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis, 2020.

[ref2] HeMIS: Hetero-Modal Image Segmentation, 2016.

[ref3] Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization, 2017.

AuthorsOpenReview ↗

Point-to-Point Response to Reviewer aTsu (2/2)

(iii) Comparisons with simpler baselines. Under the same backbone, data, and optimization budget, we compare against Dropout only, which includes learned sensor embeddings, and Dropout + Set Token, which uses independent sensor encoders and appends a learned configuration embedding to the action model’s input tokens. Results are shown below.

Model variantMean success (%)
Dropout only95.6
Dropout + Set Token95.65
Dropout + ShareRes + SetFiLM96.6

The FlexSense architecture outperforms Dropout only and Dropout + Set Token by 1.00 and 0.95 PP, respectively, supporting the benefit of ShareRes and SetFiLM. See Table 6 and Appendix G.1.

W2 and Q2 — Practical benefits of UtilSelect

(i) Real-world manipulation. We compared UtilSelect against a fixed baseline that always consumes all available depth and tactile inputs. Each setting was evaluated over 20 trials per task. Results are shown below.

PolicyAvailable auxiliary sensorsPick-and-place SR (%)Plug-insertion SR (%)Plug-insertion score
π₀.₅∅80207.5
FlexSense w/o UtilSelectD+T1007552.6
FlexSense + UtilSelectD+T1008065.3

UtilSelect improved insertion success rates and completion scores, supporting its practical value in real-world manipulation. (Table 8)

(ii) Seed variation. Across three scorer seeds, UtilSelect achieved 97.13 ± 0.08% success, outperforming the fixed-assignment baseline by 0.47 PP on average. Configuration-level results are provided in our Overall Response, “UtilSelect reproducibility.” (Table 7; Appendix G.2, Table A8)

(iii) Inference cost. UtilSelect leaves sensing and encoding costs unchanged and adds 0.41% to measured inference latency. Its benefit is improved task performance with little cost. The timing protocol and results are provided in our Overall Response, “Inference cost.” (Appendix G.2, Table A9)

W3 and Q3 — Robustness and scalability

(i) Robustness. In real-world robot deployment, sensors may experience unexpected dropout, corrupted measurements, or delays. To investigate these challenges, we simulated three representative disturbances during LIBERO-10 execution: temporary depth loss, force noise, and a 200-ms force delay, applied from 4.0 to 6.0 s.

ConditionSuccess rate (%)Force-selection rate (%)
Clean D+F9523.8
Depth dropout93.8—
Force noise93.617.48
Force delay93.420.34

Compared with clean D+F, success decreased by 1.2, 1.4, and 1.6 PP under depth loss, force noise, and force delay, respectively. UtilSelect selected force less frequently under noise or delay, showing that FlexSense can tolerate sensor disturbances and adapt its sensor use for real-world deployment.

(ii) Scalability. We trained a policy with depth, force, and proprioception as three optional inputs, increasing the candidate set from 4 to 8 subsets. The policy achieved 96.65% success with all optional inputs unavailable, extending configuration compatibility to optional proprioception. Time per optimizer step increased by only 3.43%. (Appendix G.3)

We then evaluated inference-time scaling by adding point clouds and making proprioception optional:

Optional modalitiesCandidate subsetsSelection latency (ms)Observation-to-action latency (ms)
D + F41.443301.11
D + F + PC81.441302.81
D + F + PC + proprio161.532303.11

Increasing the set from 4 to 16 added only 0.09 ms to selection and 0.66% to observation-to-action latency. These results demonstrate that exhaustive subset selection remains highly efficient, with virtually no additional selection latency even as the number of candidates quadruples. (Appendix G.2, Table A9)

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer aTsu,

Thank you again for your thoughtful feedback. We have addressed your concerns about methodological novelty and simpler baselines, the practical benefits of UtilSelect in real-world, and robustness to sensor disturbances and scalability through additional experiments and clarifications.

We would appreciate your thoughts on our responses and would be happy to clarify any remaining concerns.

Best regards,

All Authors

Reviewer_v6z8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

FlexSense enables one VLA checkpoint to operate under multiple optional-sensor configurations. ShareRes combines modality-specific input stems, a shared sensory backbone, and lightweight residual adapters. SetFiLM conditions action blocks on the active sensor subset, while configuration-constrained modality dropout trains the policy using feasible subsets of each demonstration’s available sensors. After policy training, UtilSelect learns to predict subset-specific imitation errors from offline policy evaluations against expert actions. At deployment, it selects the subset with the lowest predicted error for each action chunk. The main contributions are configuration-compatible policy learning, transfer across differently sensed demonstrations, and observation-dependent sensor selection, evaluated on LIBERO, ManiSkill, and physical manipulation.

Critical strengths

The specialist-versus-unified comparisons directly evaluate configuration compatibility across two simulation benchmarks, showing small configuration-level performance gaps.

The controlled cross-configuration experiment holds force supervision fixed while adding depth-only demonstrations, providing useful evidence of representation transfer.

Critical weaknesses

The architectural contribution largely combines established shared-specific encoding, FiLM, and modality dropout. Controlled comparisons against existing heterogeneous-sensor and adaptive-sensing policies are missing.

UtilSelect improves over the best fixed subset by only approximately 0.45 percentage points. Without repeated training seeds or confidence intervals, its incremental benefit remains uncertain.

Experiments use at most two optional sensors. Exhaustive subset scoring scales exponentially, and the scorer consumes all available sensor representations, so reduced policy inputs do not establish reduced sensing or encoding cost.

Some citations are redundant or missing.

Questions for authors
  1. How does FlexSense compare with existing heterogeneous-sensor and adaptive-sensing policies under matched backbones, data, and optimization budgets? This comparison is needed to establish its methodological advantage.

  2. Is UtilSelect’s approximately 0.45-percentage-point improvement over the best fixed subset reproducible across training seeds and paired evaluation rollouts? Please report uncertainty for this difference.

  3. Since UtilSelect consumes all available sensor representations, what are its end-to-end computational savings after acquisition, encoding, and selection costs? How does the approach scale beyond two optional sensors?

Still I think this is an interesting research question and it does make some sense. If the authors could give much more clear presentation and solve the problems above, I will consider raising my score.

2 comments
AuthorsOpenReview ↗

Point-to-Point Response to Reviewer v6z8

W1 and Q1 — Comparisons with existing methods

(i) Existing heterogeneous-sensor policies. We have added Fig. 1 to compare FlexSense with three open-source depth-aware VLAs on LIBERO-10. Under depth removal, success rates change from 91.6% to 90.0% for GuidedVLA, 55.5% to 33.6% for SpatialVLA, and 92.0% to 0.0% for Evo-Depth. FlexSense goes from 94.0% to 94.2%, and from 95.0% to 93.8% when force remains available. The 0.2 PP difference is within the granularity of our 500-rollout evaluation and does not suggest a meaningful improvement from removing depth. Rather, the comparable success rates indicate that FlexSense maintains its performance when depth is unavailable. These results highlight the sensitivity of existing depth-aware VLAs to depth removal, while demonstrating FlexSense’s robustness to changing sensor configurations. (Fig. 1; Appendix D.2)

(ii) Matched architecture comparison. To our knowledge, no open-source VLA policy explicitly supports dynamic sensor-subset selection. We therefore compare UtilSelect with fixed sensor-use strategies in Table 7. For a controlled architectural comparison, we additionally introduce Dropout + Set Token, which appends a learned sensor-configuration embedding to the action model’s input tokens. Results using the same backbone, training data, and optimization budget are shown below.

Model variantMean success (%)
Dropout only95.6
Dropout + Set Token95.65
Dropout + ShareRes96.05
Dropout + SetFiLM96.1
Dropout + ShareRes + SetFiLM96.6

The full architecture outperforms Set Token by 0.95 percentage points, with higher success rates in all four configurations, supporting the benefit of the proposed architectural design beyond adding a configuration token. (Table 6; Appendix G.1, Table A7)

W2 and Q2 — Seed variation and paired uncertainty of UtilSelect

With the FlexSense checkpoint fixed, all three scorer seeds outperformed the Best fixed subset baseline, giving a mean improvement of +0.47 PP with a sample standard deviation of 0.08 PP across scorer seeds. After averaging across sensor configurations and scorer seeds, the paired analysis over 40 tasks gave a 95% tt-interval of [+0.25, +0.69] pp, supporting a positive average improvement across tasks. Configuration-level results and the analysis protocol are provided in our Overall Response, “UtilSelect reproducibility.” (Table 7; Appendix G.2, Table A8)

W3 and Q3 — UtilSelect inference cost and scalability

UtilSelect leaves sensing and encoding costs unchanged and improves task success by adapting the inputs passed to the action policy. Its measured inference overhead was +0.41%. The timing protocol and results are provided in our Overall Response, “Inference cost.” Meanwhile, the same shared policy supports different sensor configurations, avoiding separate specialist training costs for each configuration.

To measure how UtilSelect scales beyond depth and force, we first added point clouds and then made proprioception optional. The scorer evaluates all 2m2^m feasible subsets in parallel for mm optional modalities. The measured latencies are shown below.

Optional modalitiesCandidate subsetsSelection latency (ms)Observation-to-action latency (ms)
D+F41.443301.11
D+F+point clouds81.441302.81
D+F+point clouds+proprioception161.532303.11

In this experiment, increasing the candidate count fourfold added just 0.09 ms to selection and 0.66% to observation-to-action latency. These results demonstrate that exhaustive subset selection remains highly efficient, with virtually no additional selection latency even as the number of candidates quadruples. (Appendix G.2, Table A9)

W4 — Citations

Thank you for pointing this out. We have added the missing references and removed redundant citations throughout the manuscript.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer v6z8,

Thank you again for your thoughtful feedback. We have addressed your concerns about comparisons with existing methods, the reproducibility of UtilSelect’s gains, and its inference cost and scalability through additional experiments and clarifications.

We would appreciate your thoughts on our responses and would be happy to clarify any remaining concerns.

Best regards,

All Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response by Authors

We sincerely thank all three reviewers for their careful evaluations and constructive feedback. To address the shared concerns regarding UtilSelect reproducibility, architectural comparisons, inference cost, robustness, scalability, etc., we have conducted additional experiments and revised the manuscript. Major additions and modifications are highlighted in blue.

UtilSelect reproducibility (v6z8, aTsu, 3Duc)

We trained UtilSelect scorers with three seeds while keeping the FlexSense checkpoint fixed and selected each scorer’s checkpoint on its validation set. Results are shown below. Each entry is the average success rate (%) over the four LIBERO suites. D, F, and D+F denote the available auxiliary sensor configurations. SD denotes the sample standard deviation across scorer seeds.

MethodScorer seedDFD+FAverage
Best fixed subset—96.596.696.996.67
UtilSelect4296.959797.497.12
UtilSelect1797.0596.997.2597.07
UtilSelect4797.39797.3597.22
UtilSelect (mean ± SD)—97.10 ± 0.1896.97 ± 0.0697.33 ± 0.0897.13 ± 0.08

The gains over the fixed-assignment baseline were +0.45, +0.40, and +0.55 PP, yielding +0.47 ± 0.08 PP across scorer seeds. The mean improvement is approximately 5.9 times the across-seed standard deviation, and all three seeds show positive gains, suggesting that the improvement is not driven by a particular scorer initialization.

For paired uncertainty, we first averaged across the three sensor configurations and three scorer seeds for each task, then matched UtilSelect and the fixed baseline across 40 tasks. The mean paired gain was +0.47 pp, with a 95% paired tt-interval of [+0.25, +0.69] pp. This interval reflects variation across tasks after averaging over scorer seeds. (Table 7; Appendix G.2, Table A8)

Architectural comparisons and methodological contributions (v6z8, aTsu)

We added Fig. 1 to examine depth removal in existing depth-aware VLAs and FlexSense, and introduced Dropout + Set Token as a simpler configuration-aware baseline under matched training conditions. The architecture comparisons are summarized below.

Model variantMean LIBERO success (%)
Dropout only95.6
Dropout + Set Token95.65
Dropout + ShareRes96.05
Dropout + SetFiLM96.1
Dropout + ShareRes + SetFiLM96.6

The full architecture outperforms Set Token by 0.95 PP and dropout alone by 1.00 PP, supporting the benefit of combining representation sharing with sensor-set conditioning. (Fig. 1; Table 6; Appendices D.2 and G.1)

Inference cost (v6z8, aTsu, 3Duc)

UtilSelect scores subsets after all available sensor streams have been acquired and encoded, so physical sensing and encoding costs remain unchanged. Its purpose is to improve task success by adapting the inputs passed to the action policy.

We measured 100 paired queries over 25 recorded LIBERO observations, with depth and force available. Results are shown below. Latency includes encoding, selection, and action generation.

Selection strategySelection latency (ms)Observation-to-action latency (ms)
All available sensors0299.87
UtilSelect1.44301.11

UtilSelect added 1.24 ms (+0.41%) to observation-to-action latency, improving sensor utilization with little added inference overhead. (Appendix G.2, Table A9)

Scalability and robustness (v6z8, aTsu)

We extended evaluation from 4 to 16 candidate subsets, with selection latency increasing from 1.44 to 1.53 ms. We also evaluated mid-episode sensor disturbances. FlexSense maintained 93.8% success under temporary depth dropout, while force noise reduced the selection rate of force-containing subsets from 23.80% to 17.48%, indicating adaptive use of sensory inputs. Detailed results are provided in Appendices G.2–G.4.

We have also added real-world comparisons of UtilSelect, clarified the fixed-subset baseline protocol, updated citations, and expanded implementation and evaluation details. These additions and the remaining reviewer-specific questions are addressed in our individual responses.

We thank the reviewers again for helping us strengthen the experimental evidence and presentation of our work.

AuthorsOpenReview ↗

General Response

We sincerely thank the Program Chairs, Area Chairs, and reviewers for their time and constructive feedback. We appreciate the reviewers for recognizing the strengths of our work:

  • Practical motivation. Addressing heterogeneous sensor configurations is an important problem for real-world robotic manipulation.
  • Strong experimental performance. FlexSense achieves competitive results on LIBERO and ManiSkill while maintaining performance across different sensor configurations.
  • Cross-configuration knowledge transfer. The experiments demonstrate the benefit of sharing information across demonstrations collected with different sensors.

In response to the reviewers' concerns, we conducted additional experiments and revised the manuscript, with major changes highlighted in blue. The main updates are:

  • Architectural comparisons. We added a simpler Set Token baseline and further clarified the contributions of ShareRes and SetFiLM. The full architecture outperforms Set Token by 0.95 percentage points.
  • UtilSelect reproducibility. Across three scorer seeds, UtilSelect improves over the best fixed-subset baseline by 0.47 ± 0.08 percentage points, with a 95% paired confidence interval of [+0.25, +0.69].
  • Efficiency and scalability. We measured only 0.41% additional inference latency and extended evaluation to 16 candidate sensor subsets.
  • Robustness and real-world evaluation. We added sensor-failure experiments and real-robot comparisons, where UtilSelect improves plug-insertion success from 75% to 80%. We hope these additional results and clarifications address the reviewers' concerns and would welcome any further feedback.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.