- Rating
- 1
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
Triangular transport maps are a useful approach for sampling models. However, finding a good structure for the map has been challenging. This paper provides a way to learn those structures jointly with the objective function. In some experiments, they show better performance empirically.
Critical strengths
- I think the problem they are trying to solve is important and needs to be explored further.
Critical weaknesses
- The writing (or presentation in general) of the paper should be improved a lot. For example, when the paper starts the introduction (Section 1), it does not explain what triangular transport is and just says, "Triangular transport maps represent a target distribution by transforming it into a simple reference distribution". This definition can be applied to many other transport maps, so it is not quite a definition of a triangular transport. The structure of the whole introduction makes it much more difficult to follow as well. The background section is hard to follow. Why BatchEnsemble after 2.2? It is also very hard to follow the main Section 3. It is more like listing the components of the methods but lacks a natural-language explanation of each step.
- Missing a lot of details in the experiments
Questions for authors
- What is the shape of in line 165? When you say "its" diagonal value in line 166, are you referring to 's diagonal value? Also, what do you mean by "index component by rank in this ordering"?
- What are $ p$ and $ q$ in line 202 and where did you define them? Also, what do you mean by "monotone inputs/outputs"?
- Not a question, but it would have been better if the paper had a one-to-one comparison between naive BatchEnsemble and monotone BatchEnsemble in the form of an algorithm or something. In its current form, it is hard to follow.
- Don't you need some citations in line 243, like [1]? I think missing this kind of citation is crucial.
- I think the paper does not contain any details of synthetic experiments. For example, what is the formulation of Neal's funnel that you use and what does mean in the funnel experiment? The original Funnel does not have observations but only latent variables ([2]).
- Did you run full-rank Gaussian or diagonal Gaussian VI with non-centering parameterization? That might perform a lot better in some of your benchmark experiments
- Did the experiment include some learning rate sweeps or multiple random seeds? How fair the experiment is for each of the baseline method?
[1] Bengio, Yoshua, Nicholas Léonard, and Aaron Courville. "Estimating or propagating gradients through stochastic neurons for conditional computation." arXiv preprint arXiv:1308.3432 (2013). (I could not find the exact publication venue of this preprint so just referring to the arxiv version of this paper) [2] Neal, Radford M. "Slice sampling." The annals of statistics 31.3 (2003): 705-767.
Thank you for the helpful feedback on the presentation. Based on your review, we have clarified the introduction, background and notation, and added the details of the synthetic experiments. We answer each question below.
Q1: o(k) is a scalar, the index of the variable at position k in the ordering. "Its" refers to Tk, whose diagonal variable is xo(k). "Rank k" means position k in the ordering. We have rewritten this paragraph at the start of Section 3 to make this clear.
Q2: p and q are the input and output widths of a layer. We added this definition to Section 2.2. Monotone inputs and outputs are those that depend on the diagonal variable. We added this to the definition of V and U in Section 3.1.
Q3: We added an algorithm to Appendix A that compares a standard and a monotone BatchEnsemble layer, and refer to it in Section 3.1.
Q4: Thank you. We added the citation [1] where the straight-through estimator is introduced in Section~3.2.
Q5: Thanks for pointing this out. We added the formulations of the funnel and the hierarchical model, and their true graphs, to Appendix B. N is the number of training samples. We draw N samples from the funnel and estimate its density from them, so the funnel is the target distribution and not a posterior with observations.
Q6: No, we did not. We are not sure if it would fit what we are doing. As we understand, non-centered Gaussian VI needs the true target density, for example that the root variable sets the scale of the others. We estimate a density from samples where the structure is unknown to the model, and SSTM learns that structure from data. Our Oracle variant, which is given the true structure, shows what known structure achieves.
Q7: All our runs use five seeds. On the synthetic data, each seed draws a new graph, dataset and initialization. We tuned MAF, NSF and GNF on validation data over the settings in Appendix B. We did not sweep learning rates, except for MAF, which was unstable at 10−3. We compared 10−4 and 3×10−4 on validation. NSF and GNF use the learning rates of their authors. Two-stage uses the NOTEARS paper settings. On UCI, MAF and NSF use the per-dataset settings of Durkan et al. (2019), and we report GNF's results from Wehenkel & Louppe (2021). On Sachs, GNF uses its authors' published settings. MAF, NSF and GNF report their best validation epoch, while all triangular maps report the final epoch, which favors the flows.