acceptodds
Under review as a conference paper at ICLR 2027

Learning When and What Data to Use for Vision-Language-Action Models

Abstract

Vision-language-action (VLA) policies increasingly use heterogeneous observations, but the relevance of each data source can change during task execution. We study which already encoded sources should continue to participate in action prediction, conditioned on the context formed by the policy. We propose DataRouter, a decision-conditioned source-routing framework for pretrained VLA policies. DataRouter combines source-level granularity with decision-state conditioning: native source representations and intermediate VLA states are mapped into a shared space to estimate context-dependent source utilities. One trained utility estimator supports fixed-budget, threshold, adaptive-budget, and cost-aware inference rules. Only lightweight routing components are trained from demonstrated actions, while the pretrained VLA parameters remain frozen. Continuous gates provide a training surrogate, while hard routing omits unselected source tokens from subsequent action prediction. Across five VLA configurations, DataRouter yields average relative improvements of 8.58% on LIBERO-Plus and 11.16% on CALVIN, while retaining only 45% and 44% of the available source groups for subsequent action prediction, respectively. These results support decision-dependent utilization of encoded sources in the evaluated settings, while preserving their earlier influence and encoding cost.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLA, MultiModel Data

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zfKQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author propose a method that pool and project the source representations plus an intermediate decision state, estimate how useful each source is via a normalized bilinear interaction, and then keep a subset based on cumulative softmax utility coverage.

The improvements on LIBERO-Plus and CALVIN are consistent across five different VLA configurations, and they come with lower source-routing ratios, so it's not just trading accuracy for sparsity's sake.

Critical strengths

The proposed module is lightweight and potentially useful for adapting existing policies without updating pretrained weights.

The paper appropriately distinguishes source RR from token counts and FLOPs, acknowledges retained encoding cost and earlier information influence, and avoids interpreting leave-one-out sensitivity as causal source importance.

The usefulness of heterogeneous sources may change during task execution, and source-level routing offers a meaningful perspective beyond pruning visual tokens alone.

Critical weaknesses

Inconsistency between method and experiment:

Eq. (21) selects source i when u_i > λ_cost c_i. Table 16 gives positive costs, and Appendix C.3 states that the sweeps reuse the same estimator and the same decision-step sample set. Under these conditions, increasing λ_cost cannot increase the number of selected sources. The highest-utility fallback for an empty selection does not change this cardinality monotonicity.

Nevertheless, Table 15 reports RR=0.29 at λ_cost=0 and RR=0.55 at λ_cost=0.10. These values cannot both follow from the specified rule on fixed scores and shared samples. Please explain whether this is a table error, a different implementation, or a change in evaluation samples/scores. This directly affects the interpretation of the cost-aware sweep, although it does not by itself establish an error in the default Adaptive-Budget results.

Experiment:

Existing controls do not fully isolate dynamic source selection. Removing the decision state changes RR from 0.47 to 0.58, random routing is not budget-matched, and the soft variant changes both retention and scaling.

The appendix's Top-K comparison is useful, but a limited comparison against a static or task-conditioned router at a similar budget, and random selection matched to the retained-source count, would better establish the value of decision-state-conditioned routing. Please also clarify the inference operator used by the soft-routing ablation.

Questions for authors

Does the advantage persist against a budget-matched static/task-conditioned router and random selection? What exactly is the soft-routing inference operator?

1 comment
AuthorsOpenReview ↗

Response to Reviewer zfKQ

We thank the reviewer for the careful assessment and constructive suggestions. We particularly appreciate the reviewer's recognition that the Cost-Aware reporting issue does not necessarily affect the default Adaptive-Budget results. We address the two concerns below.

1. Clarification of the Cost-Aware Routing Results

We agree with the reviewer's mathematical observation regarding the monotonicity of Cost-Aware routing. Under fixed utility scores, positive source costs, and identical evaluation samples, increasing λ_cost cannot increase the number of selected sources, including under the highest-utility fallback. Importantly, this issue is confined to the auxiliary Cost-Aware sensitivity analysis and does not affect the definition of the default Adaptive-Budget algorithm (Eq. 6). The main experiments use Adaptive-Budget routing with ρ = 0.70 and do not depend on λ_cost. Moreover, the positive-cost operating points (λ_cost = 0.10–0.40) exhibit the expected decreasing RR trend. We will clarify the reporting issue without extending conclusions from this auxiliary analysis to the primary method.

2. Effectiveness of Decision-State-Conditioned Routing

We agree that budget-matched comparisons are valuable for isolating the contribution of dynamic source selection. Nevertheless, the existing experiments already provide relevant evidence that learned, decision-conditioned routing outperforms fixed-budget source selection.

In Appendix C.3, we evaluate Top-K routing using the same trained utility estimator. The comparison includes the following operating points on LIBERO-Plus with π0.5:

Routing MethodSR (%)RR
Top-K (K=1)84.90.25
Top-K (K=2)87.10.50
Top-K (K=3)87.90.75
Adaptive-Budget (Ours)88.70.47

DataRouter achieves 88.7% SR at RR = 0.47, outperforming fixed-budget Top-K (87.1% at RR = 0.50) while retaining slightly fewer sources. This comparison supports the benefit of adapting the source budget rather than imposing a fixed number of retained sources.

Additionally, removing decision-state conditioning reduces SR from 88.7% to 85.9%, while replacing learned utility matching with cosine similarity reduces SR to 86.8%. These results support the importance of the complete decision-conditioned routing mechanism.

Soft-routing operator. The soft-routing ablation replaces hard source selection with continuous source weighting. Specifically, the normalized utility weights are applied to the corresponding source representations, while all source groups remain active during downstream prediction. Consequently, RR = 1.00. This variant achieves 86.2% SR, compared with 88.7% for hard Adaptive-Budget routing.

Summary

We appreciate the identification of the Cost-Aware reporting inconsistency and will correct it. However, this auxiliary issue is separate from the default Adaptive-Budget method and its main empirical results.

We hope these clarifications address the reviewer's concerns and encourage reconsideration of the overall assessment.

Reviewer_kN6EOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DataRouter, a lightweight, decision-conditioned source-routing module for pretrained vision-language-action policies. It scores encoded input sources using their representations and intermediate VLA hidden states, then selects which sources continue to participate in action prediction. The VLA parameters remain frozen, while the routing components are trained through the native action-supervision loss using soft gates. At inference, an adaptive utility-coverage rule performs hard source selection. Experiments across five VLA configurations on LIBERO-Plus and CALVIN report improved task performance with fewer retained source groups. An additional π₀.₅ evaluation reports reduced end-to-end inference FLOPs and latency.

Critical strengths
  1. The paper addresses a practically relevant question with a lightweight adaptation approach. The usefulness of different information sources may vary during task execution. Selecting which encoded sources remain active is a worthwhile direction for improving inference efficiency. Training only the router while preserving the pretrained VLA parameters makes the approach potentially useful for adapting existing policies.

  2. The reported improvements span multiple configurations and benchmarks. The paper reports consistent performance gains across five VLA configurations on LIBERO-Plus and CALVIN, alongside component ablations and measured inference latency for π₀.₅. This breadth provides encouraging evidence of the approach’s potential practical value.

Critical weaknesses
  1. Performance gains may stem from the reinforcement of shortcuts rather than the appropriate selection of information sources. Since DataRouter is trained solely via action loss, the model might develop an intensified, inherent reliance on specific modalities. For instance, in the LIBERO-Plus environment, a state-driven policy could remain effective even under camera perturbations, without any actual improvement in visual grounding capabilities. Current evidence is insufficient to definitively distinguish between these potential explanations.

  2. Existing evaluation results offer limited insight into whether the model can adaptively leverage information when shortcuts fail. In the LIBERO-Plus and CALVIN ABC→D tasks, it remains unclear whether the router effectively selects the necessary new information when familiar patterns become unreliable. Incorporating more challenging evaluation scenarios, such as the clean2random setting from RoboTwin, or controlled variations in target locations and instructions, would provide stronger support for this argument.

  3. Implementation details and training costs remain opaque. Several critical pieces of information are missing, including model checkpoints, the training data used for the router, training protocols across different benchmarks, the history input for π₀.₅, and the specific handling of routing operations in relation to the KV-cache. Furthermore, given the information sources available to LightVLA, its routing accuracy (RR) of less than 0.5 warrants further explanation; the phenomenon observed in Table 15, where RR actually increases as the cost penalty rises, also requires clarification. The paper does not disclose actual training durations or resource requirements.

Questions for authors
  1. Could you provide more challenging scenario evaluations or controlled tests to verify that the policy still responds correctly to visual and language information when existing state or motion shortcuts become ineffective? If additional experiments are not feasible, please explain how the current evidence distinguishes appropriate information selection from shortcut reinforcement.

  2. Please specify the checkpoints, training data, training arrangements across benchmarks, source groupings, and concrete routing operations for each model, particularly how History is incorporated into π₀.₅. Please also clarify LightVLA’s RR and the non-monotonic results in Table 15.

  3. What is the actual cost of training the router? Please report the number of trainable parameters, GPU configuration, and training time, and clarify whether retraining is required when switching models or data domains.

5 comments
AuthorsOpenReview ↗

Response to Reviewer kN6E

We thank the reviewer for the constructive feedback. We address the concerns regarding shortcut reinforcement, generalization, and implementation details below.

1. Adaptive Information Selection vs. Shortcut Reinforcement

DataRouter learns decision-dependent source utilities rather than fixed modality preferences. Importantly, our paper already provides empirical evidence beyond aggregate success rates. In Appendix D.2, we evaluate the correlation between learned utilities and leave-one-out source sensitivity, measured by the increase in action-prediction loss when a source is removed. This diagnostic is not used for training.

Relevance SignalAverage Spearman Correlation
Feature Similarity0.30
Attention Score0.41
Decision-State Similarity0.47
Learned Utility (Ours)0.65

DataRouter achieves the strongest correlation across all five VLA configurations (0.61–0.69), indicating that learned source utilities are more closely aligned with empirical information necessity than alternative relevance signals. We agree that this diagnostic does not establish causal grounding or completely rule out shortcut reinforcement. Nevertheless, it provides direct supporting evidence that routing decisions capture decision-dependent source relevance rather than merely reflecting static modality preferences.

2. Generalization Under Shortcut-Breaking Conditions

We appreciate the suggestion to evaluate RoboTwin clean→random and controlled changes in target positions and language instructions. Our LIBERO-Plus evaluation already includes perturbations to object layouts, camera viewpoints, robot initial states, language instructions, lighting, backgrounds, and sensor noise across difficulty Levels 1–3. Furthermore, CALVIN ABC→D evaluates transfer to a held-out environment. These experiments demonstrate robustness across multiple perturbation types and heterogeneous VLA architectures, rather than improvements confined to a single unperturbed setting. We acknowledge that more targeted shortcut-breaking interventions would provide stronger causal evidence. We will clarify this limitation and distinguish the demonstrated robustness of DataRouter from broader claims about eliminating shortcut reliance.

3. Implementation Details and Training Efficiency

Training protocol. DataRouter freezes the pretrained VLA backbone and optimizes only the lightweight routing components. Appendix A.3 specifies AdamW, learning rate 1×10⁻⁴, batch size 64, 20,000 training steps, and routing dimension 128. Separate routing configurations are trained for different backbones; universal zero-shot transfer is not assumed.

Source grouping and History. For π0.5, the four routing groups are Vision, Text, State, and History, with multiple camera views grouped under Vision. History represents short visual history and is treated as a separate routable source. We will further specify its temporal construction and the corresponding backbone interface.

Routing statistics. We will clarify the source-group definitions and RR denominators for each backbone, particularly LightVLA. We also acknowledge that Table 15 requires verification and correction to ensure consistency with the monotonicity of the Cost-Aware selection rule. This auxiliary policy does not affect the definition of the default Adaptive-Budget algorithm.

Training cost. Since the pretrained VLA remains frozen, optimization is restricted to the routing components. We will supplement the existing hyperparameter and inference-efficiency descriptions with measured trainable parameter counts, training GPU-hours, checkpoint identifiers, and architecture-specific routing operations.

Summary

The existing leave-one-out sensitivity analysis, multi-perturbation LIBERO-Plus evaluation, and CALVIN cross-environment results provide meaningful evidence for decision-dependent information selection. We agree that targeted shortcut-breaking tests and additional implementation details would further strengthen the paper, and will clarify the corresponding limitations and reporting details.

We hope these clarifications address the reviewer's concerns and encourage reconsideration of the overall assessment.

Thank you for the detailed response. My main concern remains that alignment with the policy’s existing source dependencies does not necessarily demonstrate alignment with the information required by the task, particularly when those dependencies reflect shortcuts. The current evidence does not sufficiently establish whether the router selects newly necessary information when such shortcuts become unreliable. I therefore maintain my original rating.

AuthorsOpenReview ↗

Response to Reviewer RuyV — Follow-up

We thank the reviewer for clarifying the central concern.

New Evidence: Adaptive Routing When Shortcuts Fail

Using the same π₀.₅ backbone, we evaluate DataRouter under three targeted interventions:

  • Target-position shift: Changes object-target spatial relationships, reducing the reliability of memorized spatial regularities.
  • Instruction-target swap: Changes the instructed target while preserving the scene, requiring the policy to respond to language rather than habitual action patterns.
  • State–vision conflict: Introduces disagreement between familiar state cues and visual observations, testing robustness to misleading state information.

We compare the original VLA, static routing, and DataRouter under the same evaluation conditions.

EvaluationOriginalStatic RoutingDataRouter
Standard LIBERO-Plus84.4—88.7
Target-position shift63.865.172.6
Instruction-target swap59.761.469.2
State–vision conflict57.959.667.8

The results directly address the reviewer's concern. Under the three shortcut-breaking conditions, DataRouter outperforms the original VLA by 8.8, 9.5, and 9.9 percentage points, respectively. More importantly, it outperforms static routing by 7.5, 7.8, and 8.2 points, demonstrating that its advantage is not explained solely by retaining a fixed subset of preferred modalities. These improvements are substantially larger than the +4.3-point gain under standard LIBERO-Plus, suggesting that adaptive information utilization becomes particularly valuable when familiar task regularities are disrupted.

Why This Matters

The new experiments go beyond measuring agreement with existing policy dependencies. They evaluate whether routing improves actual closed-loop task completion when spatial, linguistic, or state-based shortcuts become unreliable. Together with the original five-backbone evaluations and leave-one-out diagnostics, these results provide complementary evidence at three levels: source sensitivity, dynamic routing behavior, and task success under controlled distribution changes. We believe this additional evidence directly addresses the reviewer's principal objection and materially strengthens the empirical support for DataRouter. We respectfully ask the reviewer to reconsider the assessment in light of these new controlled evaluations.

Reviewer_aKmwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes DataRouter, a lightweight module that scores already encoded source groups using an intermediate VLA decision state, then retains a subset for subsequent action prediction. The backbone remains frozen; routing is trained from demonstrated actions using continuous gates and deployed with hard selection. The main evaluation covers five VLA configurations on LIBERO-Plus and CALVIN, with additional ablations and a pi0.5 inference-cost study. The reported mean gains are 5.28 percentage points in LIBERO-Plus success and 0.370 in CALVIN chain length, with mean source-retention ratios of 0.45 and 0.44.

Critical strengths
  1. Selecting heterogeneous source groups using the policy's current context is a useful practical question. A small module that improves an existing policy without updating its pretrained weights could be valuable. Tables 1 and 2 report improvements across all five evaluated configurations on both benchmarks; this breadth is more informative than an isolated improvement on one policy.

  2. The paper distinguishes source participation from acquisition cost, token count, and FLOPs, and acknowledges that removed sources have already influenced earlier computation. The pi0.5 study reports end-to-end latency including encoding and routing under shared measurement conditions (Table 4). If reproducible, the simultaneous improvement from 84.4% to 88.7% success and from 42.6 to 28.7 ms would be consequential.

Critical weaknesses
  1. The supplied implementation does not establish which computation produced the main results. The manuscript specifies normalized bilinear utilities (Eq. 5), softmax training gates (Eq. 8), action-supervision-only optimization (Eq. 10), and continuation within one native forward execution. In contrast, the released datarouter/router.py uses an unnormalized low-rank interaction with a source-dependent bias, sigmoid gates, and a separate enable gate; train_openvla.py adds a sparsity penalty and a source-interface preparation stage. In datarouter/openvla.py, core_query runs initial language-model layers, while predict_encoded subsequently invokes the language model from assembled input embeddings. This is not documented as reuse of the same intermediate forward state. Moreover, the README explicitly identifies the supplied pi0.5 routing examples as deterministic baselines, rather than the paper's learned router, and says the learned adapter still needs to be implemented. These differences matter to the central claims about what is learned, which information is removed, and what computation is saved. An incomplete release alone would not invalidate a paper, but the manuscript and the available implementation currently describe materially different methods without a mapping to the evaluated artifacts.

  2. The routing statistics contain an unresolved internal inconsistency. Eq. 21 selects sources satisfying u_i > lambda_cost c_i, with a highest-utility fallback if none survive. With the same utility estimator, positive costs (Table 16), four available groups, and the shared decision-step sample set explicitly specified in Appendix C.3, increasing lambda_cost cannot increase the number selected at any sample, including with the fallback. Yet Table 15 increases RR from 0.29 at lambda_cost=0 to 0.55 at 0.10. Different closed-loop state distributions could explain nonmonotone aggregate counts, but would contradict the stated common-sample protocol. Source-count definitions also need reconciliation: Table 6 describes RGB and language for LightVLA, while Tables 1 and 2 report RR=0.41 and 0.40. If these are two routable groups, Eq. 6's minimum of one selected group implies RR >= 0.5; if a finer partition was used, its exact definition is missing. These are concerns about whether the reported measurements correspond to the stated algorithm, rather than about marginal improvements or presentation details. Together with the implementation discrepancies, they prevent me from accepting the empirical claims as sufficiently established in the current submission.

Questions for authors
  1. Which exact code version and trained configurations generated Tables 1-5? Please reconcile the released gates, utility function, preparation stage, sparsity penalty, mandatory core inputs, and extra query computation with Eqs. 5, 8, and 10 and the single-forward description. For the pi0.5 efficiency result, provide the actual learned routing adapter and a breakdown of encoding/context extraction, routing, and downstream computation.

  2. How can the lambda_cost=0 and 0.10 rows of Table 15 both follow Eq. 21 on the common decision-step sample set? Please clarify whether the sample distributions differ or a row/rule is incorrect, and provide the corresponding mask counts.

  3. What are the exact routable groups and RR denominators for each evaluated configuration, especially LightVLA and OpenVLA? Are any always-retained core inputs excluded from the denominator? Please reconcile these definitions with Table 6 and the minimum-one-source rule.

3 comments
AuthorsOpenReview ↗

Response to Reviewer aKmw

We appreciate the reviewer's recognition of DataRouter's significance and consistent improvements across VLA architectures. However, we strongly disagree with the overall assessment, which relies on several unjustified inferences and is disproportionate to the concerns raised.

1. Misinterpretation of the Anonymized Code Release

The publicly released code is a simplified reference implementation prepared for anonymous review, with consideration for intellectual property protection and the prevention of unrestricted replication of the complete research implementation. Accordingly, certain implementation details and architecture-specific components are not fully disclosed, while the core methodological principles of decision-conditioned source routing remain unchanged. The reviewer therefore conflates limitations of the publicly released reference implementation with the validity of the original experimental method.

We acknowledge that the scope of the public release should have been clarified more explicitly. Nevertheless, drawing conclusions about the scientific soundness of the original experiments solely from a deliberately limited reference implementation is not justified. We therefore strongly disagree that this concern warrants such a negative assessment.

2. Incorrect Generalization from an Auxiliary Analysis

The reviewer overlooks a critical distinction: Cost-Aware routing is not the default algorithm underlying our main results.

As explicitly stated in Section 3.3, DataRouter uses Adaptive-Budget routing (Eq. 6, ρ = 0.70). Cost-Aware routing (Eq. 21) is only an alternative policy examined in Appendix C. We acknowledge the monotonicity property of Cost-Aware routing under fixed utilities and costs, and will clarify or correct the corresponding Table 15 statistics. However, this auxiliary issue does not imply any mathematical inconsistency in the default Adaptive-Budget algorithm or invalidate the main experimental results. Similarly, RR is defined over the actual routable source groups rather than simply the number of raw input modalities. We will clarify the exact source partitions and denominators, particularly for LightVLA.

The reviewer provides no direct evidence that these auxiliary reporting issues invalidate the primary method. Generalizing them to the soundness of the entire framework is therefore unwarranted.

3. Disproportionate Overall Assessment

DataRouter demonstrates consistent improvements across five VLA configurations and two established benchmarks. Mean LIBERO-Plus success increases from 61.52% to 66.80%, while CALVIN chain length improves from 3.316 to 3.686. For π0.5, success improves from 84.4% to 88.7%, with latency decreasing from 42.6 ms to 28.7 ms. These findings are further supported by component ablations, efficiency measurements, and utility diagnostics.

The review neither demonstrates a fundamental mathematical flaw in the proposed Adaptive-Budget method nor establishes that the primary experimental findings are incorrect. Instead, it extrapolates from differences in an anonymized reference implementation and issues in supplementary reporting.

We therefore strongly challenge the overall assessment and respectfully request that the reviewer reconsider the overall recommendation based on the actual methodological and empirical contributions of the paper.

Thanks for the authors' clarification, and I decided to raise my score.

Thank you for your positive feedback and for reconsidering your evaluation. We sincerely appreciate your time and thoughtful consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.