- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper looks at transfer in multi-channel LDA with HDP prior when the data from both source and target cannot be pooled together, and where there can only be one source export. Summaries of posterior distributions of source counts of emissions and tables freeze to form priors in the target which are then used to obtain auxiliary variable Gibbs updates for the global concentration and transfer strength. The evaluation includes experiments with synthetic source/target pairs and an IID/OOD example involving heart disease data.
Critical strengths
-
A practical contribution to making inferences under a well-specified restriction. The single-export setup is well motivated, and it is clear that the paper acknowledges that offset transfers make the standard global concentration update inapplicable.
-
Evaluation considers two distinct estimands. Separating target-specific topics from the estimation based on pooled population resolves an essential issue in defining the notion of successful transfer. The synthetic experiment manipulates data size, domain shift, and topic overlap in the feature space and compares adaptive sampling and covariance matching with varying strengths, grid search oracle, Dirichlet MLE, and target mass matching.
Critical weaknesses
-
Novel topic adaptation is not shown. The synthetic experiments employ the exact number of topics, K = 4, in each domain, with appropriate topics being created from bases in common between the two. It is therefore not an adequate way of evaluating topic adaptation or the uncertainty in truncation. Thus, the results show transfer between similar predetermined topic sets rather than adaptation of new topics in the target.
-
The real data experiment fails to validate strict separation. The single real data experiment employs artificial separations within one aggregate set of 1,190 patients. Boundaries for quantiles are calculated based on the full aggregate data in Appendix C.4 before the split, introducing additional information bias apart from the exported source summary. Common categories are required, but the protocol needs to be repeated with boundaries determined by fitting the source data or being fixed externally.
-
Heuristic validation is lacking in relation to a more general utility criterion. Source-only covariances alone do not show an approximation of a joint fit under the target shift. The document-count-weighted average of corresponding emissions in Appendix C.3 is an instance of synthetic estimand and it is not required that such instance will approximate the wrong pooled model. Cosine distance, therefore, does not show whether the heuristic improves held-out prediction or uncertainty calibration.
-
Different concentration samplers bias the transfer comparison. An approximate Escobar-West update for the non-transfer finite-K fit is applied in Appendix C.1, while an exact finite-K update is used for transfer. Measured differences could thus depend on inference algorithm in addition to transferred information. Source-only, target-only, joint, and transfer models must use the same exact update, including a zero-transfer control.
Questions for authors
-
Are you able to redo the Cleveland assessment using source-tuned or externally defined discretization limits that are sent along with the source summaries?
-
Are the benefits maintained for the source-only, target-only, and combined fits with the same finite-K concentration updates, including the zero transfer version?
-
How does transfer work with target-only topics and a wrong truncation model? Does matching of covariances help a hold-out prediction or calibration score?
We sincerely thank you again for your thoughtful comments and suggestions. We believe that we have addressed all your concerns in our rebuttal and revised manuscript, and we would greatly appreciate hearing your thoughts. Please let us know if you have any further questions or comments, and we would be happy to provide additional clarification.