acceptodds
Under review as a conference paper at ICLR 2027

Shared yet Distinct: Rethinking Label-Scarce Vertical Federated Learning with Heterogeneity-Aware Meta-Prototype

Abstract

Label scarcity and aligned-sample sparsity have emerged as major barriers to the practical deployment of supervised Vertical Federated Learning (VFL). Existing self-supervised VFL frameworks designed for label-scarce settings primarily rely on instance-discriminative representation learning. However, this learning paradigm tends to enforce rigid alignment of representations in the latent space, thereby inducing severe representation collapse in VFL. Moreover, these self-supervised VFL frameworks remain heavily dependent on frequent inter-participant representation exchange and knowledge-transfer schemes, further limiting scalability in heterogeneous settings while introducing computational and security overhead. To overcome these fundamental limitations, we propose the Heterogeneity-Aware Meta-Prototype Learning (HAMPL) framework, which establishes a new paradigm for self-supervised VFL by introducing meta-prototypes as safer intermediaries. By accounting for inter-participant heterogeneity, HAMPL enables each participant model to learn shared and generalizable knowledge while preserving its distinctive participant-specific information, thereby exploiting the complementarity of heterogeneous feature subsets across participants. Furthermore, meta-prototypes act as a knowledge bridge across the two pre-training stages, naturally mitigating catastrophic forgetting that can arise from the two-stage learning paradigm. We further provide a theoretical characterization and a security analysis of HAMPL. Extensive experiments indicate that HAMPL achieves superior performance in label-scarce VFL and remains effective under heterogeneous settings.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vertical Federated Learning, Privacy preservation, Self-supervised learning, Representation learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iTyxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The manuscript proposes HAMPL, a self-supervised vertical federated learning framework that uses shared trainable prototypes to transfer knowledge across participants while preserving participant-specific information under scarce labels and limited sample overlap. It combines unbalanced optimal transport, entropy-guided consensus fusion, and consensus-guided assignment refinement within alternating cross-participant and local training stages, reporting improved downstream performance over the evaluated baselines.

Critical strengths

This work proposes a new self-supervised VFL under scarce labels and limited sample overlap.

Critical weaknesses

An intensive comparison study with the recent SOTA model is not provided, limiting the evidence for the advantages of HAMPL over existing models. Moreover, it does not systematically evaluate sensitivity to prototype count, initialization, or geometric diversity, leaving the robustness of this design unclear.

Questions for authors
  1. One recent VFL model, FALSE-VFL[1], is not considered as a baseline in the comparison analysis of this work. FALSE-VFL is relevant because it also combines participant-specific information through a shared latent structure while addressing scarce labels and limited alignment. Although the two methods communicate different quantities—latent-distribution parameters versus prototype assignments and updates—a direct comparison would clarify HAMPL’s advantages. Moreover, HAMPL’s stated training procedure does not exploit labels attached to unaligned samples, whereas FALSE-VFL can incorporate such supervision. [1] K. Hong et. al, Deep Latent Variable Model Based Vertical Federated Learning with Flexible Alignment and Labeling Scenarios, In International Conference on Learning Representation 2026.

  2. The OT objectives in equations (2) and (5) include positive entropy regularization terms, ϵaH(Q)\epsilon_a H(Q) and ϵgH(Q)\epsilon_g H(Q), which encourage softer assignments. In contrast, equation (3) assigns greater influence to participants with lower-entropy assignments through the confidence weight wapw_{a}^p. Could the authors explain how these two effects interact and how the entropy regularization coefficients affect consensus weighting?

  3. Please clarify the inference procedure, including whether the projector, prototype bank, or OT solver is used after fine-tuning. Does inference require features from every participant for each test sample? If some participants do not have that sample, how does HAMPL perform prediction?

  4. Since the prototype bank is central to HAMPL, please evaluate sensitivity to its size and initialization, and characterize its geometry before and after training. Useful diagnostics include pairwise distances and effective rank. A comparison between fixed and trainable banks would also help isolate the benefit of learning the prototypes.

16 comments

We sincerely thank the reviewer for their careful evaluation and constructive feedback. We have thoroughly revised the manuscript and submitted the updated version. Below, we address the reviewer's concerns and questions point by point.


Weakness 1. We thank the reviewer for this valuable comment. We have added an comparison with recent SOTA methods, as detailed in our response to Question 1. Furthermore, we have systematically evaluated the sensitivity of HAMPL to prototype count, initialization, and geometric diversity to further validate its robustness, as detailed in our response to Question 4.


Question 1. Thank you for highlighting this relevant work. We have carefully revisited FALSE-VFL and studied its mechanisms, and have added FALSE-VFL-I and FALSE-VFL-II to the comparison. Firstly, when feature-space expansion is accompanied by a change in the prediction target or label semantics, labels from the original task may become inapplicable. Therefore, in our experimental setting, we use labeled samples with complete feature blocks across all participants to supervise the joint prediction task. In Question 3 below, we discuss missing feature blocks in labeled samples during downstream supervised fine-tuning and acknowledge the corresponding limitations of our framework. Here, we focus on adapting FALSE-VFL to the datasets investigated in this work, using the encoder architectures and data partitions adopted in our homogeneous setting, while tuning its hyperparameters as extensively as possible. Revised Table 1 reports the resulting comparisons, showing that both FALSE-VFL variants achieve lower mean performance than HAMPL. For convenience, we reproduce the relevant part of Table 1 below.

Table 1 (partial): Performance evaluation of various methods.

Number of labeled samples2004006008001000
DatasetMethodPerformance
NUS-WIDEFALSE-VFL-I0.614±0.0020.638±0.0020.639±0.0010.645±0.0010.648±0.002
FALSE-VFL-II0.611±0.0050.637±0.0040.637±0.0020.645±0.0030.647±0.003
FALSE-VFL-II$^{*}$0.607±0.0010.628±0.0010.629±0.0020.636±0.0010.640±0.002
HAMPL0.672±0.0090.682±0.0140.699±0.0040.704±0.0050.714±0.007
AvazuFALSE-VFL-I0.565±0.0100.624±0.0060.646±0.0020.647±0.0020.652±0.001
FALSE-VFL-II0.514±0.0020.551±0.0030.581±0.0030.589±0.0020.599±0.002
FALSE-VFL-II$^{*}$0.546±0.0140.591±0.0010.617±0.0020.618±0.0020.624±0.002
HAMPL0.658±0.0080.662±0.0020.665±0.0130.675±0.0070.683±0.012
BHIFALSE-VFL-I0.753±0.0020.757±0.0010.757±0.0020.759±0.0020.760±0.002
FALSE-VFL-II0.765±0.0020.772±0.0010.779±0.0010.778±0.0010.777±0.002
FALSE-VFL-II$^{*}$0.771±0.0010.775±0.0010.782±0.0020.781±0.0010.781±0.001
HAMPL0.821±0.0040.822±0.0010.831±0.0010.838±0.0010.839±0.001
ModelNetFALSE-VFL-I0.523±0.0020.563±0.0030.604±0.0020.610±0.0040.616±0.003
FALSE-VFL-II0.524±0.0020.573±0.0040.606±0.0040.617±0.0020.623±0.002
HAMPL0.735±0.0100.753±0.0080.801±0.0020.811±0.0010.818±0.002

Note: For each dataset, the best-performing value in each column is marked in bold. ∗^{*} indicates that pre-training was extended to 100 epochs due to the larger dataset size.

We discussed possible explanations for this gap in Appendix D.8, which we summarize below together with the training-budget comparison.

First, the original missing-mask protocol in FALSE-VFL retains many examples observed by several participants. For its independent MCAR configurations, our calculation gives a single-party proportion of approximately 0.008%–9.5% within the randomly masked subset, conditional on at least one observed party. In our construction, 60% of all training examples are distributed into disjoint single-party subsets, which is common in two-participant setting. This provides fewer co-observations for estimating cross-participant dependencies and may make joint latent-distribution learning more challenging.

Second, FALSE-VFL aggregates posterior uncertainty through the inverse of summed local precisions. Holding local uncertainty estimates fixed, fewer observed parties provide less uncertainty reduction, and a single-party observation receives no benefit from combining multiple precision estimates. This may be particularly relevant to our two-party NUS-WIDE, Avazu, and BHI settings, where most training examples are observed by only one participant.

Question 1 (Continued). Third, our benchmarks combine larger training populations and scarce labels with heterogeneous feature blocks, long-tailed distributions, and limited cross-party co-observations. These conditions can complicate latent-distribution modeling, missing-feature generation, and the association between learned latent variables and task labels. We interpret FALSE-VFL as emphasizing inference of a common latent description, while HAMPL aims to acquire shared knowledge and preserve distinct participant-specific information. This difference may contribute to HAMPL’s stronger performance in the evaluated settings.

In addition, FALSE-VFL’s original pre-training budgets are at least 100 epochs, whereas our standard comparison uses 40 epochs for all methods. The shorter budget may therefore contribute to its weaker results. We additionally extend FALSE-VFL-II to 100 epochs on NUS-WIDE, Avazu, and BHI, reported as FALSE-VFL-II* in Table 1. Although this improves its results on Avazu and BHI, the gap to HAMPL remains on all three datasets. Its sampling-based training also requires repeated exchanges of posterior parameters, latent samples, likelihoods, and gradients. Appendix B.2.2 and Table 9 show that even FALSE-VFL-I’s aligned-sample communication subtotal exceeds HAMPL’s complete pre-training communication payload For convenience, we reproduce the relevant part of Table 9 below.

**Table 9 (partial): Communication counts and costs per participant per pre-training epoch. **

NUS-WIDEAvazuBHIModelNet
MethodTimesCost (MB)TimesCost (MB)TimesCost (MB)TimesCost (MB)
FALSE-VFL-I$^{\dagger}$1080r′+90ℓ1080\mathrm r^{\prime}+90\ell1,134.3115000r′+1250ℓ15000\mathrm r^{\prime}+1250\ell15,754.242064r′+172ℓ2064\mathrm r^{\prime}+172\ell2,167.78264r′+22ℓ264\mathrm r^{\prime}+22\ell277.27
HAMPL90q+93p90\mathrm q+93\mathrm p237.501250q+1253p1250\mathrm q+1253\mathrm p819.59172q+175p172\mathrm q+175\mathrm p113.1222q+25p22\mathrm q+25\mathrm p59.24

Question 2. Entropy regularization and confidence weighting are compatible: the former regulates assignment construction, and the latter evaluates the resulting uncertainty when forming consensus. For cross UOT in Eq. (2), εa\varepsilon_a scales representation–prototype similarities through Ka=exp⁡(Sap/εa)K_a=\exp(S_a^p/\varepsilon_a). A smaller εa\varepsilon_a amplifies similarity differences and typically produces sharper assignments, whereas a larger value promotes smoother assignments. Together, εa\varepsilon_a, the marginal regularization coefficients, and the current representation–prototype similarities affect the sharpness of local assignment matrices and thus their assignment confidence, which we measure using assignment entropy. Thus, although all participants use the same εa\varepsilon_a, their assignment entropies can differ because their actual matching results differ for each sample. We therefore use confidence weighting to evaluate assignment uncertainty when forming consensus, giving higher weights to more certain assignments to prevent the consensus from becoming unstable or having low confidence.

For consensus-guided OT in Eq. (5), expanding the KL term gives an effective entropy coefficient of εg+λ\varepsilon_g+\lambda. At fixed λ\lambda, increasing εg\varepsilon_g generally smooths the refined assignments. Since this step follows consensus fusion, it affects future weights indirectly through model and prototype updates, while the current weights are determined by the first-pass assignments.


Question 3. First, HAMPL follows the same forward procedure during inference as during downstream supervised fine-tuning. Each participant encodes its local features, and the active participant combines these representations and feeds them into the joint predictor. Following the representation-learning paradigm of methods such as SwAV [1], the projector, prototype bank, and OT solver are used only during pre-training to improve encoder representations and generalization. Downstream fine-tuning and inference use the local encoders and the task-specific joint predictor. We have clarified the relationship between fine-tuning and inference in the downstream fine-tuning description.

Second, conventional VFL methods, including supervised VFL and SSVFL, generally give limited consideration to joint inference with missing feature blocks [2]. Accordingly, HAMPL primarily targets collaborative inference in which all participants provide their respective features for each test sample. Its strong empirical performance under complete feature availability supports the effectiveness of its pre-training in improving the joint model’s generalization. During pre-training, HAMPL can also accommodate missing feature subsets by restricting collaboration to the subset of participants holding aligned observations. At inference time, if a participant lacks the features for a test sample, a simple fallback would be to use a zero vector in place of its representation before applying the joint predictor. This permits a prediction, but accuracy may deteriorate substantially. Possible adaptations include masking participant representations during supervised fine-tuning to improve robustness to missing feature blocks, or estimating missing representations using an approach inspired by FedCVT [3]. Exploring how the missing-data modeling in FALSE-VFL can complement HAMPL to improve prediction under incomplete observations is an interesting direction for future research, but it is beyond the scope of this paper.

Last but not least, our privacy analysis in Appendix B.1 indicates that label and attribute inference attacks based on a participant’s local model depend strongly on the predictive information acquired by that model during collaboration. When an adversary’s local model absorbs semantic information from other participants, its representations may become more similar to theirs, increasing its ability to infer their private information. HAMPL mediates knowledge exchange through multiple meta-prototypes while preserving participant-specific information. By avoiding rigid alignment to averaged representations, it can reduce encoder coupling and limit the acquisition of other participants’ private predictive information. The label-inference experiments in Appendix D.2 and Table 10 show that, under the evaluated protocol, collaboration adds less label-predictive information to a single pretrained HAMPL encoder than to its DACLSC counterpart. These results are consistent with HAMPL’s cooperative inference design, in which each participant contributes complementary information that may be difficult to recover from the other participants alone. Accordingly, HAMPL’s joint inference relies on the distinct information contributed by each participant.

[1] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, volume 33, pp. 9912–9924, 2020.

[2] Yang Liu, Yan Kang, Tianyuan Zou, Yanhong Pu, Yuanqin He, Xiaozhou Ye, Ye Ouyang, Ya-Qin Zhang, and Qiang Yang. Vertical federated learning: Concepts, advances, and challenges. IEEE Transactions on Knowledge and Data Engineering, 36(7):3615–3634, 2024. doi: 10.1109/TKDE.2024.3352628.

[3] Yan Kang, Yang Liu, and Xinle Liang. FedCVT: Semi-supervised vertical federated learning with cross-view training. ACM Transactions on Intelligent Systems and Technology, 13(4):1–16, 2022.

Reviewer_zFw4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to improve the performance of Vertical Federated Learning in label-scarce and aligned-sample sparsity scenarios. The author points out that existing self-supervised VFL frameworks tend to enforce alignment of representations, which may cause representation collapse. Moreover, existing methods have limited scalability and also introduce extra computational and security overhead. To solve these limitations, this paper proposes HAMPL, a Heterogeneity-Aware Meta-Prototype Learning framework. HAMPL introduces learnable meta-prototypes and a dual optimal transport mechanism enable knowledge sharing through participant-specific assignment targets. The learnable meta-prototypes connect cross-participant and local training, incorporating locally learned information into subsequent collaboration. Experiments show improved performance and continued effectiveness under heterogeneous settings.

Critical strengths

This paper aims to improve the performance of Vertical Federated Learning in label-scarce and aligned-sample sparsity scenarios. This paper shows technical advantages in the following aspects.

(1) The proposed design of HAMPL is well motivated. Different participants hold feature subsets, so enforcing rigid alignment of representations may discard useful information. Using shared prototypes while allowing participant-specific training targets is a reasonable approach to this problem.

(2) The experimental section demonstrates the performance of the method under different combinations of benchmarks and numbers of labeled samples. The ablation study section also shows the benefits obtained by each component in the method design.

Critical weaknesses

Overall, this paper has some merits, but there are a few weaknesses that stop me from giving a higher rating. My major concerns are as follows.

(1) The Introduction mentions that enforcing rigid alignment of representations may lead to severe representation collapse. However, there is a lack of quantitative metrics to support the claim that existing methods may cause representation collapse. The authors could report the data efficiency of the training phase and the communication overhead of the method. These metrics also impact the practical deployment of the method, so the authors could give a further comparison of the training and communication-related costs between HAMPL and baseline methods.

(2) The performance evaluation under heterogeneous settings mainly compares the performance of HAMPL itself under homogeneous and heterogeneous settings, demonstrating that it can run on the tested heterogeneous architectures. The coverage of heterogeneous settings in the experiments is relatively limited, which is insufficient to demonstrate the advantages of this method over other methods in terms of compatibility.

Questions for authors

Here are some questions for the authors.

(1) Could you provide further explanation regarding the possible representation collapse of existing methods and the mitigation effects of HAMPL on this issue?

(2) The paper provides relatively detailed hyperparameter settings. Could you additionally include a sensitivity analysis?

(3) The paper proposes that this method can alleviate catastrophic forgetting. Besides existing ablation experiments, is there more direct evidence to illustrate this claim?

4 comments

We sincerely thank the reviewer for their careful evaluation and constructive feedback. We have thoroughly revised the manuscript and submitted the updated version. Below, we address the reviewer's concerns and questions point by point.


Weakness 1. First, To quantitatively assess potential representation collapse, beyond the visualization in Figure. 6, we have added effective-rank trajectories measured using RankMe (Garrido et al., 2023) and spectral analyses of the meta-prototype bank in Figure 4, Section 3.6, and Appendix D.7. DACLSC and FedHSSL-M exhibit substantial effective-rank reductions on NUS-WIDE, consistent with dimensional collapse. In comparison, HAMPL’s representation effective rank shows a larger increase during training than those of DACLSC and FedHSSL-M and gradually stabilizes. Meanwhile its meta-prototype bank maintains a near-maximal effective rank of approximately 126 out of 128. These results provide quantitative evidence supporting our collapse-mitigation claim.

Second, we provide a detailed per-epoch computational complexity comparison in Appendix B.2.1 and Table 8. For convenience, we reproduce Table 8 below. On Avazu and BHI, HAMPL’s leading cross-training and local-training computational terms are respectively 0.4 and 0.0625 times those of DACLSC at matched batch size and representation width. Only NUS-WIDE incurs substantial computational overhead, as it uses longer queues to improve performance. When training costs become excessive, shortening or disabling the queues provides a lighter configuration that can still achieve competitive performance. Regarding data efficiency, Appendix D.1 and Figure 5 show that HAMPL using only 10% aligned samples consistently outperforms competing collaborative methods using 20% on BHI and ModelNet across the evaluated label budgets.

Table 8: Per-epoch computational overhead of HAMPL and DACLSC.

SideHAMPLDACLSC
Participant sideO(Ba(ma+Lap)NCR+Blp(mlp+Llp)NCR)O\bigl(B_{\mathrm a}(m_{\mathrm a}+L_{\mathrm a}^p)N_C R+B_l^p(m_l^p+L_l^p)N_C R\bigr)O(BamaR(ma+R)+BlpmlpLQR)O\bigl(B_{\mathrm a}m_{\mathrm a}R(m_{\mathrm a}+R)+B_l^p m_l^p L_Q R\bigr)
Server sideO(PBaNC(ma+R))O\bigl(PB_{\mathrm a}N_C(m_{\mathrm a}+R)\bigr)O(PBamaR+PAθ)O\bigl(PB_{\mathrm a}m_{\mathrm a}R+PA_\theta\bigr)

Third, we report communication counts and payload costs per participant per pre-training epoch in Appendix B.2.2 and Table 9. For convenience, we reproduce Table 9 below. HAMPL requires substantially less communication than even the aligned-sample subtotals reported for FALSE-VFL-I. Compared with DACLSC, HAMPL reduces communication volume by approximately 38%, 59%, and 50% on Avazu, BHI, and ModelNet, respectively, while incurring higher costs on NUS-WIDE. On Avazu and BHI, smaller prototype banks reduce HAMPL’s communication costs. On BHI and ModelNet, HAMPL also eliminates the substantial communication overhead associated with aggregating image-encoder parameters. On NUS-WIDE, however, the baseline’s linear-layer aggregation incurs a relatively small communication payload, substantially smaller than the payload associated with exchanging HAMPL’s assignment matrices and prototype parameters. Crucially, HAMPL introduces a different communication paradigm: participants exchange assignment matrices and meta-prototype updates while keeping encoder representations and model parameters local, thereby strengthening protection against their direct disclosure. Therefore, we consider additional communication overhead acceptable when changing the exchanged data types and interaction protocol improves predictive performance while providing stronger protection against direct disclosure of participant representations and model parameters.

Table 9: Communication counts and costs per participant per pre-training epoch.

NUS-WIDEAvazuBHIModelNet
MethodCountsCost (MB)CountsCost (MB)CountsCost (MB)CountsCost (MB)
FedHSSL-M45r+2m45\mathrm r+2\mathrm m55.62625r+2m625\mathrm r+2\mathrm m663.7986r+2m86\mathrm r+2\mathrm m184.6511r+2m11\mathrm r+2\mathrm m106.01
ISSVFL-M90r+90s+2m90\mathrm r+90\mathrm s+2\mathrm m196.991250r+1250s+2m1250\mathrm r+1250\mathrm s+2\mathrm m2627.31172r+172s+2m172\mathrm r+172\mathrm s+2\mathrm m454.8322r+22s+2m22\mathrm r+22\mathrm s+2\mathrm m140.56
DACLSC90r+2m90\mathrm r+2\mathrm m102.801250r+2m1250\mathrm r+2\mathrm m1319.15172r+2m172\mathrm r+2\mathrm m274.8322r+2m22\mathrm r+2\mathrm m117.54
FALSE-VFL-I$^{\dagger}$1080r′+90ℓ1080\mathrm r^{\prime}+90\ell1,134.3115000r′+1250ℓ15000\mathrm r^{\prime}+1250\ell15,754.242064r′+172ℓ2064\mathrm r^{\prime}+172\ell2,167.78264r′+22ℓ264\mathrm r^{\prime}+22\ell277.27
HAMPL90q+93p90\mathrm q+93\mathrm p237.501250q+1253p1250\mathrm q+1253\mathrm p819.59172q+175p172\mathrm q+175\mathrm p113.1222q+25p22\mathrm q+25\mathrm p59.24

Weakness 2. Representative SSVFL methods, including FedHSSL, ISSVFL, and DACLSC, rely on partial model aggregation, which requires compatible parameter structures in the aggregated components. Consequently, their original protocols cannot directly accommodate fully heterogeneous architectures. HAMPL removes this restriction by enabling collaboration through a shared meta-prototype interface. We have also added FALSE-VFL, which supports heterogeneous architectures in principle but exhibits substantially lower performance than HAMPL under homogeneous settings (Table 1). Different model architectures may introduce additional geometric discrepancies, further complicating its latent-distribution modeling. Heterogeneous settings therefore lead to a wider performance gap between FALSE-VFL and HAMPL.

In addition, our ablation study in Section 3.4 and Table 3 evaluates multiple variants under heterogeneous encoders, demonstrating the contributions of HAMPL’s components. We have further introduced two variants: DACLSC w/o KT, which removes knowledge transfer and uses one local model for both pre-training stages, and HAMPL w/o LP, which replaces prototype-based local training with MoCo. These results support the importance of meta-prototypes as information carriers between cross-participant and local training and provide additional evidence consistent with their role in mitigating catastrophic forgetting under heterogeneous setting.

Finally, we conduct sensitivity experiments under heterogeneous setting, covering meta-prototype bank size and initialization methods, consensus regularization strength, and marginal regularization coefficients, as detailed in Section 3.6, Appendix D.6, and Table 11. These results further characterize HAMPL’s effectiveness and robustness within the tested heterogeneous settings.


Question 1. To quantitatively assess potential representation collapse, beyond the visualization in Figure 6, we have added effective-rank trajectories measured using RankMe [1] and spectral analyses of the meta-prototype bank in Figure 4, Section 3.6, and Appendix D.7. RankMe is used to assess representation collapse in self-supervised learning by measuring representation diversity through the entropy of normalized singular values; lower values indicate greater concentration in fewer directions. DACLSC and FedHSSL-M exhibit substantial effective-rank reductions on NUS-WIDE, consistent with dimensional collapse. In comparison, HAMPL’s representation effective rank shows a larger increase during training than those of DACLSC and FedHSSL-M and gradually stabilizes. Meanwhile its meta-prototype bank maintains a near-maximal effective rank of approximately 126 out of 128. These results complement the qualitative visualization and provide quantitative evidence supporting HAMPL’s mitigation of representation collapse.

[1] Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann LeCun. RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 10929–10974. PMLR, 2023.


Question 2. We have added a comprehensive hyperparameter sensitivity analysis across all four datasets under heterogeneous settings, including the meta-prototype bank size, initialization methods, consensus regularization strength, and marginal regularization coefficients, as detailed in Appendix D.6 and Table 11. For convenience, we reproduce Table 11 below.

**Table 11: Sensitivity analysis.

HyperparameterNUS-WIDEAvazuBHIModelNet
Top-1 AccAUCF1-ScoreTop-1 Acc
λ\lambda0.50.651±0.0080.646±0.0090.819±0.0010.715±0.005
1.00.656±0.0070.652±0.006$^{*}$0.818±0.003$^{*}$0.720±0.004
1.50.664±0.004$^{*}$0.649±0.0100.813±0.0010.727±0.002$^{*}$
2.00.660±0.0060.641±0.0070.810±0.0020.718±0.005
NCN_C1280.638±0.0080.630±0.0080.795±0.0020.665±0.007
2560.643±0.0070.652±0.006$^{*}$0.818±0.003$^{*}$0.686±0.010
10240.664±0.004$^{*}$0.628±0.0090.801±0.0020.727±0.002$^{*}$
20480.650±0.0050.625±0.0080.783±0.0020.711±0.005
ρa\rho_{\mathrm a} & ρl\rho_l0.50.664±0.004$^{*}$0.652±0.006$^{*}$0.818±0.003$^{*}$0.727±0.002$^{*}$
1.00.654±0.0050.643±0.0090.811±0.0020.717±0.006
C0G\mathcal{C}_0^G initializationKaiming uniform0.664±0.004$^{*}$0.652±0.006$^{*}$0.818±0.003$^{*}$0.727±0.002$^{*}$
Gaussian0.659±0.0080.639±0.0160.795±0.0030.705±0.003
SobolSphere0.664±0.0090.653±0.0060.787±0.0010.704±0.002
Meta-prototype formfix0.649±0.0060.637±0.0090.808±0.0020.715±0.004
learnable0.664±0.004$^{*}$0.652±0.006$^{*}$0.818±0.003$^{*}$0.727±0.002$^{*}$

Note: The best-performing value within each hyperparameter is marked in bold. ∗^{*} indicates the configuration adopted in the main experiments under heterogeneous settings.

These results further demonstrate HAMPL’s effectiveness and robustness under the tested heterogeneous settings, with a detailed discussion provided in Appendix D.6.


Question 3. To further investigate this issue, we have added two targeted variants in Section 3.4 and Table 3:

  • HAMPL w/o LP: Removes meta-prototypes from local training and replaces this stage with MoCo, interrupting prototype-mediated knowledge transfer between the two stages.

  • DACLSC w/o KT: Removes DACLSC’s knowledge transfer mechanism and uses a single local model for both pre-training stages.

For convenience, we reproduce the relevant part of Table 3 below.

Table 3 (partial): Performance comparison across the variants.

MethodNUS-WIDEAvazuBHIModelNet
Top-1 AccAUCF1-ScoreTop-1 Acc
HAMPL w/o LP0.616±0.0090.627±0.0250.774±0.0030.652±0.010
DACLSC w/o KT0.630±0.0150.633±0.0110.787±0.0050.676±0.009
HAMPL0.664±0.0040.652±0.0060.818±0.0030.727±0.002

The substantial performance degradation of the two variants highlights the importance of mechanisms that preserve knowledge across the two pre-training stages. These results support the role of meta-prototypes as knowledge carriers: prototypes learned through cross-participant collaboration guide subsequent local training, while locally updated prototypes are aggregated to inform the next collaborative stage. This maintains continuity between the two learning objectives and provides further evidence consistent with the role of meta-prototypes in mitigating catastrophic forgetting during alternating training.

We sincerely thank the reviewer for the positive assessment and for taking the time to review the additional results and clarifications. We are glad that the concerns have been addressed.

Reviewer_mUuXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper proposes HAMPL, a meta-prototype-based self-supervised pre-training method for label-scarce VFL. Unlike existing alignment-based methods, it matches participant representations to shared learnable meta-prototypes instead of rigidly aligning them with fused anchors.

  • The method integrates unbalanced OT, entropy-weighted consensus fusion, and consensus-guided OT reassignment into a prototype-based framework. Pre-training alternates between a cross-training stage on aligned samples and a local-training stage on all private samples, with the meta-prototypes bridging the two stages.

  • The method outperforms the baselines on four benchmarks across various label budgets and remains effective under heterogeneous encoders and low aligned-sample ratios. It also shows improved privacy against label inference attacks compared with a representation-sharing baseline.

Critical strengths
  • The problem setting is well motivated, and the adaptation of several techniques to VFL is well structured, although most of the techniques are existing ones.

  • The experiments are comprehensive, and the method is analyzed from various perspectives, including ablation, aligned-sample ratio, privacy, and efficiency.

Critical weaknesses

Some of the claims may be overstated. Since the proposed method is largely based on existing techniques adapted to VFL, expressions such as “fundamentally rethink” may not be fully justified.

Questions for authors
  • Q1. HAMPL’s largest gain is on ModelNet, whose views are different viewpoints of the same object and thus highly redundant. This gain could also stem from other factors, such as long-tail handling or the four-party setting. How can the authors attribute it to accommodating cross-participant semantic discrepancies?

  • Q2. Could the authors clarify which components are novel beyond adapting existing techniques to VFL, and why matching to shared meta-prototypes fundamentally differs from aligning representations with a shared anchor?

4 comments

We sincerely thank the reviewer for their careful evaluation and constructive feedback. We have thoroughly revised the manuscript and submitted the updated version. Below, we address the reviewer's concerns and questions point by point.


Weakness 1. We thank the reviewer for this valuable comment. We have carefully revised the manuscript to moderate potentially overstated claims to ensure that our statements accurately reflect the scope and contributions of our work.


Question 1. We render each ModelNet object from 12 viewpoints using Blender and distribute these views equally among four participants. Compared with splitting a single image into left and right halves, this construction more naturally reflects a visual VFL scenario where different parties hold distinct observations of the same entity.

Although these views share substantial information about object identity, they differ in fine-grained structure and visible cues. Independently parameterized encoders further introduce differences in representation geometry, even with identical architectures. Thus, redundancy in object identity does not imply agreement among participant representations. This helps explain why existing SSVFL methods, despite receiving the same potentially redundant views, can still exhibit representation concentration, as observed in Figure 6. With raw data remaining local, collaboration must enable participants to learn shared semantics while retaining their distinctive knowledge. Reconciling these requirements can become more demanding as the number of participants increases. HAMPL therefore focuses on accommodating cross-participant semantic discrepancies through participant-specific prototype matching, avoiding rigid instance-level alignment that can suppress distinctive information. We regard long-tail handling as a complementary distributional aspect of this objective: unequal semantic frequencies, combined with view-dependent representations, can produce nonuniform sample-to-prototype matching preferences. Relaxing the marginals in cross and local UOT accommodates these preferences, while consensus-guided rematching incorporates shared information without imposing identical targets on all participants. Accordingly, long-tail handling and discrepancy-aware rematching are complementary components of the same heterogeneity-aware learning framework.


Question 2. First, to the best of our knowledge, HAMPL is the first framework to explicitly address representation collapse arising from cross-participant discrepancies in label-scarce VFL. We acknowledge that prototype matching is an established technique. However, centralized prototype-based SSL operates on augmented views of the same sample, whereas VFL participants hold heterogeneous feature subsets with distinct semantics. HAMPL therefore introduces a distributed interaction protocol that constructs a cross-participant consensus and uses it as a soft prior for participant-specific rematching, enabling personalized representation learning. Its learnable meta-prototypes further bridge cross-participant and local training, enabling one local model per participant to acquire both inter-domain and intra-domain knowledge, eliminating the need for separate cross/local model and the additional model-to-model knowledge transfer employed by representative SSVFL frameworks.

Second, HAMPL redesigns the pre-training communication paradigm. Participants exchange assignment matrices and meta-prototype updates while keeping local model parameters private and avoiding direct representation transmission. This changes the information available to an adversary. Appendix B.1 discusses a potential attack chain exploiting jointly exposed representations and model components, while Appendix D.3 analyzes how HAMPL changes these attack prerequisites and characterizes the remaining information exposure. Furthermore, representative SSVFL frameworks employ partial model aggregation, which requires compatible parameter structures in the aggregated components. HAMPL eliminates this step and enables collaboration through a shared meta-prototype interface, extending its applicability to settings with fully heterogeneous local model architectures.

Third, participant-specific semantic matching is another core advantage of HAMPL and also addresses your second question about how HAMPL differs from shared-anchor alignment. As illustrated in Figure 1, independently parameterized encoders can produce geometrically discrepant representations even when they share the same architecture, compounding differences in local feature semantics. Directly fusing these representations can produce a biased shared anchor, and enforcing alignment with it can suppress distinct participant-specific information and induce representation collapse. MoCo-style instance discrimination can additionally repel semantically related samples by treating them as negatives. HAMPL’s shared meta-prototypes instead provide a learnable semantic dictionary. Each participant uses the consensus as a soft prior and incorporates its own representation–prototype similarities when constructing matching targets. Consequently, multiple meta-prototypes can jointly capture semantic information, while different participants can obtain different soft assignments for the same entity. This flexibility enables each local model to acquire shared, generalizable knowledge while preserving participant-specific information. Beyond the visualization in Figure 6, we have added effective-rank trajectories measured using RankMe [1] and spectral analyses in Section 3.6 (Figure 4) and Appendix D.7. These analyses show that the effective rank of HAMPL’s representations increases during pre-training and gradually stabilizes. Meanwhile its meta-prototype bank maintains a near-maximal effective rank, providing further evidence of collapse mitigation.

[1] Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann LeCun. RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 10929–10974. PMLR, 2023.

I thank the authors for responding to my questions. I am maintaining my original score.

We sincerely thank the reviewer for the positive assessment and for taking the time to review the additional results and clarifications. We are glad that the concerns have been addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.