acceptodds
Under review as a conference paper at ICLR 2027

FedHVLN: Heterogeneous Federated Vision-and-Language Navigation

Abstract

Federated vision-language navigation (VLN) trains agents from private environments, but averaging native models requires matching parameter shapes. We introduce FedHVLN, which shares compatible language-side modules while retaining each client’s navigator. Four stages train a human-instruction encoder and transformation, a federated Speaker that turns local route context into pseudo-instructions, a feature generator aligned with human features, and pseudo-instruction navigation. Clients upload only the shared modules designated at each stage; inference uses the human-instruction path. On the Room-to-Room (R2R) validation split (1,021 held-out instructions from 56 previously seen environments), the mixed federation reaches 64.45% success rate (SR) and 61.69% success weighted by path length (SPL). At fixed round 991, aggregating the shared interface across both model sizes improves pooled SR and SPL by 3.14 and 3.23 percentage points over matched single-size controls. Both evaluated subsets have positive point differences on all five navigation metrics, although exploratory scene-resampling intervals for the compact-model subset include zero. In homogeneous federations, FedHVLN improves all five metrics over FedVLN, with SR/SPL gains of 1.37/1.49 and 2.15/2.74 points for the wide and compact models, respectively. These results support sharing a compatible language interface across the two tested navigator structures on R2R validation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Federated Learning, Vision-and-Language Navigation, Model Heterogeneity, Embodied AI, Cross-Capacity Collaboration, Generative Replay

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_St7kOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FedHVLN lets navigation clients with incompatible model dimensions collaborate through a shared language interface and private projections. Its four stages combine human-instruction training, route-conditioned instruction generation, feature alignment, and pseudo-instruction navigation. Inference uses the human-instruction pathway. The paper's main contribution is applying this interface to federated VLN, where the meaning of an action depends on the client's current scene and viewpoint.

Critical strengths
  1. Language is a sensible sharing interface (Sections 2 and 3.2). Navigation clients do not have a common set of action meanings, whereas they all need to interpret instructions. Sharing instruction features through private projections addresses this distinction directly and accommodates different native model widths.

  2. The collaboration results give the interface a concrete purpose (Table 2). Cross-group aggregation improves SR by 3.07 points for wide clients and 3.25 points for compact clients while preserving each evaluated client's local training budget. Both groups benefit from experience contributed by models whose native parameters they cannot directly average.

  3. The training machinery stays out of the inference path (Figure 2; Section 3.7). The Speaker and generator supply additional supervision during training, but deployment uses the human encoder and shared transformation. This is a useful property for integrating the method with an existing navigator. Table 1 also shows gains over FedVLN in homogeneous settings.

Critical weaknesses
  1. The advance over shared auxiliary models needs a clearer statement (Section 2; Section 3.2). The paper identifies FedMRL as a close precedent, and the shared-module/private-projection design follows that general pattern. Conditioning a history-dependent navigator on language is a worthwhile application, but I am less clear about the new learning principle. The contribution would be easier to assess if the paper distinguished the VLN-specific insight from the underlying solution to incompatible parameter shapes.

  2. The reason for the full training sequence is not yet persuasive (Sections 3.3–3.6). Stage 1 already learns a common instruction interface through local navigation. The Speaker provides a natural way to use routes without human descriptions, but it is less clear why those descriptions require a separate generator aligned to the human encoder. The paper describes the stages carefully; the missing connection is why this separation is especially useful for sharing navigation knowledge across clients.

  3. The two language pathways depend on an important representation assumption (Section 3.5, Eqs. (3)–(4); Sections 3.6–3.7). Alignment initially brings pooled generator features toward the fixed human-encoder features. Stage 3 then adapts the generator locally and trains the shared transformation on its outputs, while inference applies that transformation to the human encoder. The method relies on compatibility surviving this adaptation. I would like a clearer explanation of how the local navigation objective is expected to preserve the relationship established by alignment.

Questions for authors
  1. In Sections 3.4–3.5, what is the intended advantage of a separate generator over encoding the Speaker's pseudo-instructions with the human-instruction encoder?
  2. In Section 3.6, how should the reader understand the balance between local generator adaptation and maintaining a useful shared transformation for the human-instruction path?
Reviewer_FPxROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FedHVLN, a federated learning framework for Vision-Language Navigation designed to bridge clients with structurally heterogeneous local models (i.e., different embedding and hidden dimensions). Because action spaces in VLN vary per viewpoint and parameter tensors cannot be directly averaged across heterogeneous architectures, FedHVLN introduces a shared, compatible language interface consisting of an auxiliary instruction encoder (E), a language transformation module (D), a Speaker (S), and a feature generator (G), while keeping native navigators and visual encoders local. Training follows a four-stage schedule:

  1. Human-instruction navigation to optimize and aggregate E and D;

  2. Federated Speaker training to generate route pseudo-instructions;

  3. Generator alignment matching generated text features to frozen target representations from E;

  4. Pseudo-instruction navigation optimizing local navigators and refining D.

At inference, the agent discards the Speaker and generator, utilizing only the human-instruction path (E→D→P_k). Evaluated on the Room-to-Room (R2R) benchmark across two different model width configurations, FedHVLN demonstrates positive point differences over single-size controls and outperforms FedVLN in homogeneous settings.

Critical strengths
  1. Well-Motivated Domain Problem: Tackles an important and realistic bottleneck in embodied AI and federated learning—enabling collaborative navigation learning across devices with varying memory and computation budgets without requiring matching model shapes or sharing private indoor environments.

  2. Task-Tailored Architecture Design: Smartly navigates the limitations of existing heterogeneous FL methods by decoupling navigation policy heads from language understanding and synchronizing only the shared language interface modules.

  3. Inference Efficiency: The auxiliary components required during the generative alignment and data-augmentation stages (the Speaker S and Generator G) are decoupled and omitted at inference, meaning deployment retains a streamlined forward pass (E→D→P_k) without runtime communication or generative overhead.

  4. Consistent Empirical Gains Across Setups: Outperforms FedVLN on all five reported navigation metrics across both wide and compact models in homogeneous settings, while also demonstrating positive cross-group collaboration improvements over matched within-group baselines.

Critical weaknesses
  1. Constrained Definition of Heterogeneity: The evaluation only tests two configurations of the identical Seq2Seq Listener topology differing strictly in hidden and word-embedding widths, while visual features (CLIP 512-d) and state decoders remain identical. It leaves unverified whether the approach generalizes across genuinely distinct architectures.

  2. Missing Empirical Resource Benchmarks: Despite hardware resource constraints being the core motivation for heterogeneous models, the paper does not measure actual communication volumes, memory footprints, training times, or FLOPs incurred by the multi-stage updates and auxiliary modules.

  3. Multi-Stage Training Complexity and Overhead: The protocol requires four sequential, interdependent training stages with multiple alternating broadcast-and-upload cycles. This design may be brittle to stragglers or asynchronous client dropouts common in real-world federated deployments.

  4. Marginal Statistical Significance on Compact Models: In the cross-group collaboration experiments, all five exploratory 95% scene-resampling intervals for the compact model subset cross zero, indicating that the performance gains for smaller devices exhibit high scene variance and weaker statistical confidence.

Questions for authors
  1. Can FedHVLN support structural differences beyond recurrent/embedding widths, such as integrating transformer-based VLN navigators alongside recurrent listeners, or varying the visual backbones?

  2. What is the total communication volume (in megabytes transferred per client) and computational overhead across all four stages compared to standard FedVLN?

  3. How robust is the four-stage framework if clients disconnect or drop out mid-stage—particularly during the federated Speaker training (Stage 2.1) or generator alignment (Stage 2.2)?

  4. How does the mixed federation perform on the R2R val_unseen split? Since the paper only reports results on the val_seen split (1,021 instructions across 56 seen scenes), demonstrating generalization to novel environments is critical for validating whether shared language representations overfit local scene layouts.

Reviewer_MKtbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedHVLN, a heterogeneous federated vision-language navigation framework that enables collaboration among navigation models with incompatible parameter dimensions. The method introduces a shared language interface and adopts a four-stage training procedure involving human-instruction learning, federated Speaker training, feature alignment, and pseudo-instruction navigation. By aggregating compatible language-side modules while preserving model-specific components, FedHVLN facilitates knowledge sharing across heterogeneous clients. Experiments on the R2R benchmark demonstrate improvements over FedVLN and positive gains from cross-structure collaboration.

Critical strengths
  1. The paper addresses an important and practical challenge in federated VLN, where clients may employ navigation models with different parameter dimensions. The proposed shared language interface provides a reasonable solution for transferring knowledge without directly averaging incompatible model parameters. The overall framework is well motivated and has potential applications in heterogeneous embodied systems.

  2. The experiments demonstrate that FedHVLN can support collaborative training across two model configurations. The matched cross-group and within-group comparisons show positive improvements in navigation performance, while additional scene-level analyses provide useful insights into the variability of these improvements.

Critical weaknesses
  1. The experiments only consider two configurations of the same Seq2Seq Listener architecture, differing mainly in hidden and embedding dimensions. Therefore, the results do not sufficiently demonstrate whether the proposed framework can accommodate more substantial architectural differences. Additionally, evaluation is restricted to R2R val_seen, leaving its generalization to unseen environments unclear. Broader evaluation would strengthen the central claim of heterogeneous federated navigation.

  2. Although the proposed framework integrates multiple training stages, the individual contributions of Speaker training, feature alignment, and pseudo-instruction navigation are not sufficiently investigated through ablation studies. Furthermore, the relationship between the pseudo-instruction feature path used during training and the human-instruction path used during inference deserves further clarification. These details would help establish the necessity of the proposed training procedure.

Questions for authors
  1. Could the authors clarify whether FedHVLN can support substantially different navigation architectures, beyond configurations with different hidden and embedding dimensions?

  2. Are there any available results on R2R val_unseen or other navigation benchmarks that could further demonstrate the generalization ability of the proposed framework?

  3. Could the authors provide additional clarification or existing ablation results regarding the individual contributions of Speaker training, feature alignment, and pseudo-instruction navigation?

  4. Section 3.7 states that native navigator parameters remain local, whereas Section 4.1 indicates that native parameters are aggregated within the same model-size group. Could the authors clarify the actual aggregation protocol?

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their careful reading of our paper. We summarize and clarify the following two themes: the gain of navigators of different widths from training together and the advantage FedHVLN retains in environments unseen during training.

Compact and wide navigators both benefit from training together. FedHVLN shares what clients learn about following instructions while keeping each client’s navigation policy and projection local. On R2R val_seen, we compared sharing across widths with sharing only among clients of the same width, holding each client’s training data and update budget fixed. Cross-width sharing increased success rate by 3.07 percentage points for wide navigators and 3.25 points for compact navigators. SPL also improved for both groups (Table 2 in the revised manuscript). Our replies to MKtb and FPxR explain how this result relates to broader architectural differences and when a navigator would need a different interface to use the shared instruction features.

In unseen environments, FedHVLN retains its advantage over both FedVLN baselines. FedHVLN led both baselines by at least 1.28 percentage points in SR and 0.81 points in SPL before pre-exploration, and by at least 0.60 and 0.65 points, respectively, afterward (Appendix A.6, Table 5). In response to your comments on generalization, we designed a new experiment to compare FedHVLN with both FedVLN baselines before and after pre-exploration. We used the same compact-to-wide navigator ratio in the 11 R2R val_unseen environments as in the seen environments and reserved all 2,349 human-written instructions for evaluation. Before pre-exploration, FedHVLN achieved 53.77% SR and 49.52% SPL, compared with 51.13% and 46.50% for wide FedVLN and 52.49% and 48.71% for compact FedVLN. After 30 rounds of pre-exploration on local routes with Speaker-generated instructions, FedHVLN reached 67.18% SR and 64.47% SPL, compared with 66.58% and 63.82% for wide FedVLN and 66.20% and 63.27% for compact FedVLN (Table 3 in the revised manuscript). Mixed-width FedHVLN outperformed both single-width baselines on both metrics before and after pre-exploration.

In our individual replies, we also report what training on additional routes contributed under a matched update budget, how the adapted model uses human instructions at inference, and the communication and computation costs of the four training stages.

Area_Chair_wQtYOpenReview ↗

Reminder: Please Review Authors’ Rebuttal and Provide Feedback

Dear Reviewers,

Thank you for your reviews. If you have not already done so, please read the authors' rebuttal and share your feedback in the discussion thread at your earliest convenience. Your timely responses would be greatly appreciated.

Best,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.