acceptodds
Under review as a conference paper at ICLR 2027

Transfer Learning for Multi-Channel HDP-LDA Topic Models Under Strict Data Separation Constraints

Abstract

Latent Dirichlet Allocation (LDA) models are increasingly used beyond text, including for discrete tabular data in which samples are treated as documents and multinomial variables as observation channels. Such models have found renewed success in the biomedical domain, where datasets are usually small and targeted to a specific health condition. Often, such models can be improved by including data from external sources and transfer learning is the canonical approach in such situations. However, patient data is extremely sensitive and hosted on isolated sandbox environments whose strict governance rules forbid exporting individual-level data to other systems and require manual approval of every data export request. These restrictions make traditional transfer and federated learning methods for LDA models inapplicable. In this paper, we therefore propose new methods for transfer learning for multi-channel LDA models in the presence of strict data-separation constraints and where no iterative communication between data hosting systems is allowed. We tackle this challenge by extracting sufficient statistics from a model fit on the source dataset and using them as frozen, weighted priors when modeling the target dataset, thus requiring a single data transfer between systems. To approximate the topics that would be obtained by combining the datasets, we tune the weight of the frozen source summaries to match their empirical covariance with the nominal covariance they induce in the target model. For target-topic estimation, we introduce exact auxiliary-variable Gibbs updates that infer the transfer strength and the global concentration of a Hierarchical Dirichlet Process (HDP) prior under the asymmetric concentrations induced by the frozen source summaries. Controlled synthetic experiments across a range of sample sizes and domain shifts characterize the methods' behavior, while a case study on the Cleveland heart disease dataset provides a real-world demonstration.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Transfer Learning, Privacy-Preserving Learning, Latent Dirichlet Allocation (LDA), Hierarchical Dirichlet Processes (HDP), Markov Chain Monte Carlo, Biomedical Data Mining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iCASOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks at transfer in multi-channel LDA with HDP prior when the data from both source and target cannot be pooled together, and where there can only be one source export. Summaries of posterior distributions of source counts of emissions and tables freeze to form priors in the target which are then used to obtain auxiliary variable Gibbs updates for the global concentration and transfer strength. The evaluation includes experiments with synthetic source/target pairs and an IID/OOD example involving heart disease data.

Critical strengths
  1. A practical contribution to making inferences under a well-specified restriction. The single-export setup is well motivated, and it is clear that the paper acknowledges that offset transfers make the standard global concentration update inapplicable.

  2. Evaluation considers two distinct estimands. Separating target-specific topics from the estimation based on pooled population resolves an essential issue in defining the notion of successful transfer. The synthetic experiment manipulates data size, domain shift, and topic overlap in the feature space and compares adaptive sampling and covariance matching with varying strengths, grid search oracle, Dirichlet MLE, and target mass matching.

Critical weaknesses
  1. Novel topic adaptation is not shown. The synthetic experiments employ the exact number of topics, K = 4, in each domain, with appropriate topics being created from bases in common between the two. It is therefore not an adequate way of evaluating topic adaptation or the uncertainty in truncation. Thus, the results show transfer between similar predetermined topic sets rather than adaptation of new topics in the target.

  2. The real data experiment fails to validate strict separation. The single real data experiment employs artificial separations within one aggregate set of 1,190 patients. Boundaries for quantiles are calculated based on the full aggregate data in Appendix C.4 before the split, introducing additional information bias apart from the exported source summary. Common categories are required, but the protocol needs to be repeated with boundaries determined by fitting the source data or being fixed externally.

  3. Heuristic validation is lacking in relation to a more general utility criterion. Source-only covariances alone do not show an approximation of a joint fit under the target shift. The document-count-weighted average of corresponding emissions in Appendix C.3 is an instance of synthetic estimand and it is not required that such instance will approximate the wrong pooled model. Cosine distance, therefore, does not show whether the heuristic improves held-out prediction or uncertainty calibration.

  4. Different concentration samplers bias the transfer comparison. An approximate Escobar-West update for the non-transfer finite-K fit is applied in Appendix C.1, while an exact finite-K update is used for transfer. Measured differences could thus depend on inference algorithm in addition to transferred information. Source-only, target-only, joint, and transfer models must use the same exact update, including a zero-transfer control.

Questions for authors
  1. Are you able to redo the Cleveland assessment using source-tuned or externally defined discretization limits that are sent along with the source summaries?

  2. Are the benefits maintained for the source-only, target-only, and combined fits with the same finite-K concentration updates, including the zero transfer version?

  3. How does transfer work with target-only topics and a wrong truncation model? Does matching of covariances help a hold-out prediction or calibration score?

Reviewer_2of6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes transfer learning methods for multi-channel HDP-LDA topic models under strict data separation constraints, where individual-level data cannot be shared between source and target datasets and iterative communication between data-hosting systems is prohibited. The methods extract emission counts and HDP table counts from the source model as frozen aggregate statistics and incorporate them into asymmetric Dirichlet priors for the target model.

The authors propose two approaches for distinct estimation objectives. First, Adaptive Gibbs automatically adjusts the transfer strength based on the target data to estimate the topics of the target population. To enable inference under the asymmetric priors induced by the transferred source statistics, the authors derive exact auxiliary-variable Gibbs updates for the transfer strength and the global HDP concentration parameter. Second, Dirichlet Covariance Matching calibrates the transfer strength by matching the empirical source posterior covariance to the nominal covariance of the induced Dirichlet prior, aiming to estimate the topics of the joint source-target population.

The proposed methods are evaluated on 5,832 synthetic dataset pairs spanning various sample sizes and domain shifts, as well as on the Cleveland heart disease dataset. The results show that Adaptive Gibbs performs well for target-topic estimation, while Covariance Matching is effective for joint-topic estimation. Overall, the experiments demonstrate that source information can improve topic estimation without requiring the original datasets to be combined.

Critical strengths
  1. Transfer learning and inference for multi-channel HDP-LDA under strict data separation constraints This paper addresses a realistic setting in which source and target datasets cannot be combined and iterative communication between data-hosting systems is prohibited. Under these constraints, the authors propose methods that incorporate aggregate statistics from the source dataset into asymmetric Dirichlet priors for the target model, enabling the transfer of topic distributions and global topic weights in multi-channel HDP-LDA. In particular, the derivation of exact auxiliary-variable Gibbs updates for the global HDP concentration parameter and transfer strength under the resulting asymmetric priors is a meaningful technical contribution. A key strength of this work is that it addresses the inference challenges arising from extending existing LDA-based prior-transfer approaches to multi-channel HDP-LDA.

  2. Clear distinction between estimation objectives and extensive empirical validation The paper distinguishes between estimating topics of the target population and estimating topics of the joint source-target population, proposing Adaptive Gibbs and Dirichlet Covariance Matching for these objectives, respectively. This distinction is particularly important when distribution shifts exist between the two datasets. Experiments on 5,832 synthetic dataset pairs demonstrate that Adaptive Gibbs performs well for target-topic estimation, while Covariance Matching is effective for joint-topic estimation. The Cleveland heart disease case study further supports the applicability of the proposed methods. The clear alignment between the estimation objectives, the proposed methods, and the empirical results strengthens the paper's methodological contribution and provides evidence of its practical relevance.

Critical weaknesses
  1. Insufficient ablation of the core HDP-specific components The paper compares a range of transfer-strength settings and discusses its relationship to existing prior-based transfer methods. However, the contribution of transferring global HDP table statistics to the observed performance improvements remains unclear, despite being one of the paper's main methodological contributions. In particular, the paper does not provide an ablation comparing the transfer of emission statistics alone with the transfer of both emission and global HDP table statistics. This makes it difficult to assess the incremental benefit of the HDP-specific components beyond conventional Dirichlet-prior transfer. Consequently, this limits the empirical evidence for the practical benefit of the proposed HDP-specific extension, although it does not undermine the technical validity of the derived inference updates.
Questions for authors
  1. Contribution of global HDP table statistics Could the authors clarify how transferring global HDP table statistics contributes to the observed performance improvements? Although the paper compares a range of transfer-strength settings, it does not provide a direct comparison between transferring emission statistics alone and transferring both emission and global HDP table statistics. Such an ablation would help assess the incremental contribution of the HDP-specific components beyond conventional Dirichlet-prior transfer.

  2. Statistical intuition behind covariance matching under distribution shift Appendix Table C.2 and Figure C.1 show that Covariance Matching achieves performance close to the oracle across various domain-shift settings. Could the authors elaborate on the statistical intuition or conditions under which a transfer strength determined solely from the source posterior covariance is appropriate for joint-topic estimation? In particular, since posterior uncertainty calibration and source-target distribution mismatch are distinct issues, it would be helpful to clarify how these two factors relate to the effectiveness of Covariance Matching.

1 comment

We sincerely thank you again for your thoughtful comments and suggestions. We believe that we have addressed all your concerns in our rebuttal and revised manuscript, and we would greatly appreciate hearing your thoughts. Please let us know if you have any further questions or comments, and we would be happy to provide additional clarification.

Reviewer_sqSoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes to move "sufficient statistics" from one private cache of documents to another in order to do privacy preserving, federated topic learning in the style of latent Dirichlet allocation (LDA).

The contribution is an an algorithmic approach to transfer learning in LDA.

There are synthetic and real data experiments to validate the approach.

Critical strengths

None.

Critical weaknesses

The paper is entitled ... HDP-LDA ... and then has nothing to do with HDP's ("finite approximation to the HDP structure with K topics").

Topic signatures; fine. This "per-sweep signatures are aligned to the running average to the previously aligned signatures via the Hungarian algorithm?"; what? No details are provided. And while technically, in theory, not a problem as exploring all permutations is not necessary, it is non-trivial and certainly non-standard; sufficiently so to require some explanation.

Fig. 1 "cosine distance to true topics" -- I don't see why there aren't error bars -- you write 5832 dataset pairs. I also don't understand whether this is an expected quantity (it should be) or something else. Details for "joint dataset", etc. are lacking. Guessing would suggest a magic result whereby the proposed approximate Gibbs algorithm produces joint topic inferences that better than running LDA on the joint dataset.

I look with great suspicion on results reported with 3 significant digits and p values with the same.

Fig. 2 reports "distance to joint fit" which, in a Bayesian setting, has no meaning as the "fit" is a distribution and distance between a distribution and a sample is ill-defined.

Questions for authors

Please address the points in the critical weaknesses section.

Comments on the paper

2 comments

Thanks for your updated experiments. In the second added table, Dirichlet Cov. yields statistically significantly worse held-out NLL than Target dataset alone and Fixed λ\lambda and it performs negatively when transferred to novel-topic regime (the third added table). This might show that matching source-only covariance statistics optimizes proximity at the expense of predictive generalization. Hence, I keep the rating unchanged.

Thank you for highlighting this trade-off. The distinction between held-out NLL and model interpretability is well established in the literature: Chang et al. (2009) found that “topic models which perform better on held-out likelihood may infer less semantically meaningful topics” and that human interpretability captures aspects “undetected by previous measures of model quality based on held-out likelihood.” This observation motivated subsequent work defining evaluation metrics that better correlate with human expert judgment, and evaluating topic models explicitly in terms of semantic coherence, including Newman et al. (2010), Mimno et al. (2011), and Röder et al. (2015).

Thus, while we agree that the results demonstrate a trade-off between predictive likelihood and preservation of the targeted topic structure, we do not believe that the decrease in held-out likelihood alone is sufficient evidence against the effectiveness of the proposed approach. In fact, the primary goal of our paper is to improve the quality of discovered topics, thus motivating our primary evaluation in terms of topic signatures.

We will explicitly mention this trade-off and better motivate our experimental evaluation arrangement in terms of topic signatures.

Chang, J. et al, Reading Tea Leaves: How Humans Interpret Topic Models, NIPS 2009 Newman, D. et al, Automatic Evaluation of Topic Coherence, NAACL 2010 Mimno, D. et al, Optimizing Semantic Coherence in Topic Models, EMNLP 2011 Röder, M. et al, Exploring the Space of Topic Coherence Measures. WSDM 2015

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.