acceptodds
Withdrawn from ICLR 2027

ForgeVLA: Federated Vision-Language-Action Learning without Client-side Language Annotations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language-Action (VLA) models hold great promise for general-purpose robotic intelligence, yet scaling up such models is severely bottlenecked by the high cost of acquiring annotated training data. Fortunately, vision-equipped robots deployed across various domains already produce abundant vision-action pairs that can be leveraged to scale up VLA training more efficiently. However, these raw data cannot be centrally aggregated due to various constraints and also exhibit severe heterogeneity. To address these challenges, in this paper, we propose ForgeVLA, a federated VLA training framework that learns VLA models from distributed vision-action pairs without centralizing client-side raw data or requiring client-side manual annotations. Specifically, each client in ForgeVLA is equipped with an embodied instruction classifier that maps vision-action trajectories to a predefined instruction set, recovering the missing language modality and forming complete vision-language-action triplets. Beyond triplet construction, we also identify vision-language feature collapse as a critical challenge that has been largely overlooked in prior federated VLA research. To mitigate this issue, ForgeVLA combines a client-side contrastive planning loss with a server-side adaptive aggregation strategy to learn task-discriminative representations efficiently. Extensive experiments across multiple benchmarks show that ForgeVLA significantly outperforms other baselines, and ablation studies further validate the contribution of each component.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Federated Learning, Federated Vision-Language-Action Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2ghoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ForgeVLA, a federated framework for training VLA models from client-side vision–action trajectories without language annotations, while keeping raw data local. An instruction classifier trained on a small labeled public dataset assigns predefined task instructions to these trajectories. The framework combines a task-level contrastive loss with adaptive server aggregation to reduce training degradation caused by differences in client task distributions. Experiments on four LIBERO suites and two real-world tasks show improvements over federated baselines.

Critical strengths

The paper studies a meaningful research problem. Existing robot logs may lack language annotations and cannot always be shared across organizations. Studying how to use these distributed data for VLA training could help expand the available training data while keeping raw trajectories local.

Critical weaknesses

1.The experimental evaluation is limited in scope. Although the paper evaluates four LIBERO suites, all simulation experiments use the same benchmark, and the real-world evaluation includes only two tasks. This limits the evidence for the method’s effectiveness across a broader range of tasks and environments.

2.Important experimental details are missing or unclear. The paper does not clearly explain how the public labeled subset and client datasets are separated or how demonstrations are distributed across clients. For the real-world experiments, the number of training demonstrations and the instruction classifier’s training data and evaluation setup are not sufficiently described. These details are needed to assess and reproduce the results.

3.The methodological novelty is limited. The paper studies a meaningful problem, but its main contribution lies in combining instruction classification, task-level contrastive learning, and adaptive aggregation for federated VLA training. Overall, the method focuses on system integration, with relatively limited novelty in the individual components.

Questions for authors

Could the authors respond to the concerns raised in the weaknesses section?

Reviewer_RbuROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ForgeVLA trains VLA models across clients with vision-action trajectories but no language annotations. It uses a classifier to recover instructions from a predefined task set, then combines task-prototype contrastive learning with adaptive aggregation. Experiments are conducted on task-partitioned LIBERO data and two real-robot tasks.

Critical strengths

ForgeVLA improves over FedAvg with the same backbone on LIBERO and the real robot. The ablations show gains from both contrastive learning and adaptive aggregation, and the instruction classifier remains useful with few labeled demonstrations.

Critical weaknesses

The trajectories are already segmented into individual tasks, and the instruction set is known. In this setting, why is obtaining a task label expensive? The task ID could potentially be recorded during collection or recovered from the execution program, and manually assigning one label per trajectory also seems feasible. The experiments hide language annotations in existing datasets, but do not show why these simpler options are impractical for the deployment data the paper aims to reuse. Table 3 shows that the classifier works with few labels, but does not compare its cost with simply labeling the trajectories. I am not convinced that missing language is a substantial bottleneck under these assumptions.

The paper motivates reusing existing deployment logs, but the experiments mainly remove language from benchmark training data and redistribute it across clients. This does not test whether the method can reuse logs from deployed controllers with different behavior quality. Predicting an instruction only assigns a task to a trajectory; the subsequent imitation objective still fits the recorded actions, including mistakes if they are present. The real-robot section reports two task-skewed partitions without explaining who generated the trajectories or how they were filtered. The paper should specify the behavior policies and data-selection assumptions, and evaluate whether the gains hold when logs contain failed or suboptimal executions.

Questions for authors

See weaknesses.

Reviewer_FPdEOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies federated training of VLA policies when the robot trajectories have no language annotations. ForgeVLA first runs a classifier that assigns each trajectory an instruction from a fixed set. It then trains the policy with a task-level contrastive loss and an adaptive aggregation rule on the server. Against FedAvg with the same backbone, it does better on all four LIBERO suites. The biggest gap is on LIBERO-Goal (55.2% vs. 28.8%). On two real-robot tasks, overall success goes from 35% to 56%.

Critical strengths

The motivation is clear that learning from distributed robot data makes sense. The gains over same-backbone FedAvg are large. The ablations are useful: both the contrastive loss and the adaptive aggregation help, and aggregation accounts for more of the improvement.

Critical weaknesses
  1. The classifier can only choose from a predefined list of instructions, limiting the generalization and flexibility of the method. VLMs are becoming powerful on annotation videos. A direct comparison with VLM annotation data would be necessary to show whether classification actually wins on annotation quality, cost, or downstream success.
  2. I’m not convinced the feature analysis shows that vision-language feature collapse is what leads poor federated performance. Task representations moving closer together doesn’t necessarily mean task information is lost. It could just be a change in feature scale. Normalized distances, plus a linear probe for task identity, would make a much stronger case.
  3. Every main result uses InternVLA-M1. Keeping the backbone fixed is fine to isolate what the training method contributes. But it's unclear whether the gains carry over to other policies, such as π0.5 or world-action models.
Questions for authors
  1. Why classification instead of generative annotation? How do the two compare on annotation accuracy and downstream success?
  2. Does the feature-collapse result survive normalization, and does it actually go along with worse task discriminability?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.