acceptodds
Under review as a conference paper at ICLR 2027

Learning the Structure of Triangular Transport Maps

Abstract

Triangular transport maps provide a flexible approach to sampling-based probabilistic modeling, including density estimation, generative modeling, and Bayesian inference. They transform an unknown target distribution into a simpler reference through a monotone triangular map. The map structure is defined by a variable ordering and sparsity pattern, which together encode a directed acyclic graph. Map quality can depend strongly on this structure, yet finding a good structure is computationally expensive because each candidate generally requires fitting a different map. A central challenge is therefore to learn density and structure jointly, while keeping computation manageable as dimension grows. We introduce Self-Structuring Transport Maps (SSTM), which learn the map, ordering, and sparsity jointly. We use SoftSort to learn the variable ordering and gates to learn the sparsity, while preserving a triangular structure. To keep the map scalable, we use a monotone BatchEnsemble that shares one weight matrix across all map components through rank-one adapters. Across synthetic and real data, jointly learning the structure and map gives better density estimates than estimating the structure first. When the structure is identifiable from the density, SSTM matches the density performance of a map fitted with the true structure and outperforms autoregressive flows. On large datasets, SSTM is competitive with autoregressive flows.

Primary area:
generative models
Keywords:
triangular measure transport, density estimation, structure learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jFJBOpenReview ↗
Rating
1
Confidence
4
Informativeness
2(+1)
Presentation
1
Soundness
2(+1)
Read review
Paper summary

Triangular transport maps are a useful approach for sampling models. However, finding a good structure for the map has been challenging. This paper provides a way to learn those structures jointly with the objective function. In some experiments, they show better performance empirically.

Critical strengths
  1. I think the problem they are trying to solve is important and needs to be explored further.
Critical weaknesses
  1. The writing (or presentation in general) of the paper should be improved a lot. For example, when the paper starts the introduction (Section 1), it does not explain what triangular transport is and just says, "Triangular transport maps represent a target distribution by transforming it into a simple reference distribution". This definition can be applied to many other transport maps, so it is not quite a definition of a triangular transport. The structure of the whole introduction makes it much more difficult to follow as well. The background section is hard to follow. Why BatchEnsemble after 2.2? It is also very hard to follow the main Section 3. It is more like listing the components of the methods but lacks a natural-language explanation of each step.
  2. Missing a lot of details in the experiments
Questions for authors
  • What is the shape of o(k)o(k) in line 165? When you say "its" diagonal value in line 166, are you referring to oo's diagonal value? Also, what do you mean by "index component TkT_k by rank kk in this ordering"?
  • What are $ p$ and $ q$ in line 202 and where did you define them? Also, what do you mean by "monotone inputs/outputs"?
  • Not a question, but it would have been better if the paper had a one-to-one comparison between naive BatchEnsemble and monotone BatchEnsemble in the form of an algorithm or something. In its current form, it is hard to follow.
  • Don't you need some citations in line 243, like [1]? I think missing this kind of citation is crucial.
  • I think the paper does not contain any details of synthetic experiments. For example, what is the formulation of Neal's funnel that you use and what does NN mean in the funnel experiment? The original Funnel does not have observations but only latent variables ([2]).
  • Did you run full-rank Gaussian or diagonal Gaussian VI with non-centering parameterization? That might perform a lot better in some of your benchmark experiments
  • Did the experiment include some learning rate sweeps or multiple random seeds? How fair the experiment is for each of the baseline method?

[1] Bengio, Yoshua, Nicholas Léonard, and Aaron Courville. "Estimating or propagating gradients through stochastic neurons for conditional computation." arXiv preprint arXiv:1308.3432 (2013). (I could not find the exact publication venue of this preprint so just referring to the arxiv version of this paper) [2] Neal, Radford M. "Slice sampling." The annals of statistics 31.3 (2003): 705-767.

1 comment

Thank you for the helpful feedback on the presentation. Based on your review, we have clarified the introduction, background and notation, and added the details of the synthetic experiments. We answer each question below.

  • Q1: o(k)o(k) is a scalar, the index of the variable at position kk in the ordering. "Its" refers to TkT_k, whose diagonal variable is xo(k)x_{o(k)}. "Rank kk" means position kk in the ordering. We have rewritten this paragraph at the start of Section 3 to make this clear.

  • Q2: pp and qq are the input and output widths of a layer. We added this definition to Section 2.2. Monotone inputs and outputs are those that depend on the diagonal variable. We added this to the definition of VV and UU in Section 3.1.

  • Q3: We added an algorithm to Appendix A that compares a standard and a monotone BatchEnsemble layer, and refer to it in Section 3.1.

  • Q4: Thank you. We added the citation [1] where the straight-through estimator is introduced in Section~3.2.

  • Q5: Thanks for pointing this out. We added the formulations of the funnel and the hierarchical model, and their true graphs, to Appendix B. NN is the number of training samples. We draw NN samples from the funnel and estimate its density from them, so the funnel is the target distribution and not a posterior with observations.

  • Q6: No, we did not. We are not sure if it would fit what we are doing. As we understand, non-centered Gaussian VI needs the true target density, for example that the root variable sets the scale of the others. We estimate a density from samples where the structure is unknown to the model, and SSTM learns that structure from data. Our Oracle variant, which is given the true structure, shows what known structure achieves.

  • Q7: All our runs use five seeds. On the synthetic data, each seed draws a new graph, dataset and initialization. We tuned MAF, NSF and GNF on validation data over the settings in Appendix B. We did not sweep learning rates, except for MAF, which was unstable at 10−310^{-3}. We compared 10−410^{-4} and 3×10−43\times10^{-4} on validation. NSF and GNF use the learning rates of their authors. Two-stage uses the NOTEARS paper settings. On UCI, MAF and NSF use the per-dataset settings of Durkan et al. (2019), and we report GNF's results from Wehenkel & Louppe (2021). On Sachs, GNF uses its authors' published settings. MAF, NSF and GNF report their best validation epoch, while all triangular maps report the final epoch, which favors the flows.

Reviewer_GVkCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Triangular maps are an effective way to transport one density into another. The authors introduce self-structuring transport maps that learn the map, ordering (of variables) and the sparsity pattern (of the map, corresponding to some kind of conditional independence) jointly. The framework is based on the BatchEnsemble architecture of prior work, where different transport maps share a single weight matrix and individualize using a rank-one adapter. Monotonicity and normalization are enforced while variable ordering is learned through a straight-through estimator and sparsity is encouraged through a regularizer. The authors conduct experiments on synthetic and UCI datasets (with dimension ranging from 6 to 43) and showed improved negative log-likelihood compared to baselines.

Critical strengths
  • A nice joint framework to learn the triangular map, the ordering of variables and the sparsity pattern of the map altogether. This is a very difficult problem and the authors managed to build several ideas from prior works into a coherent pipeline.
Critical weaknesses
  • the current writing is lacking in details: each of the components in Section 3 could be better described and justified.

  • the final evaluation is a bit underwhelming, as the main paper only reported negative log-likelihood while structural recovery was only mentioned in passing. This section would be much stronger if the authors could carefully demonstrate the improved recovery of the variable ordering and the sparsity pattern (not to mention the small scale of the experiments and the somewhat weak result on the Sachs data).

Questions for authors

The paper reads well up to Section 3.1 and then takes a downturn from there:

  • in Section 3.1, how is sigma_+(W) or sigma_+(R) defined exactly? Should the selectors U and V depend on the layer index l? It'd be much clearer if the authors could use Figure 1 to explain every step in Eq (6) and (7) and help readers see how monotonicity is maintained. While this is certainly more flexible, is there any practical advantage compared to just requiring the weights of the diagonal variable to be nonnegative (along with a monotonic activation function)?
  • efficient normalization: I am completely lost here. In Eqs (8) and (9), the left-hand side is a scalar but the right-hand side is a matrix (my guess, otherwise why would you need to transpose)? Is this detail really important to include here or it suffices to mention we do some kind of normalization?
  • Section 3.2, again, very lost: how do you get the STE estimator in Eq (12)? Why is the mask M even triangular? What is the purpose of doing the cumulative sum in Eq (13) and then immediately reverse it using Δ\Delta in Eq (14)? What is the justification of applying STE?
  • Section 3.3, same issue as in Section 3.2: can't seem to understand why one sets up things in this way, based on what principle and aiming to achieve what. I don't even see how L_0 in Eq (18) would encourage sparsity.
  • there are a lot of hyperparameters that are left undiscussed: β\beta in Eq (16), λsp\lambda_{sp} in Eq (18) and λ1\lambda_1 in Eq (21), possibly some more. When we sweep these hyper-parameters, should we still be surprised that we achieve better results compare to the baselines?

While I appreciate the authors' efforts in trying to learn three things (map, ordering and sparsity) jointly, I don't think the current draft explains the designs well, or provides sufficient justifications of each choice, or convincingly demonstrates the desired effects in the experiments.

1 comment

Thank you for your time and effort in providing us with your detailed and constructive feedback. Based on your review, we have made changes to Section 3 to make it clearer and easier to follow. This includes an example that uses Figure 1 to explain the monotone BatchEnsemble layer, moving the normalization details to Appendix A, and clarifying Sections 3.2 and 3.3. We answer each question below.

  • Q1: σ+\sigma_+ is the softplus. We added this definition to Section 3.1. Yes, UU and VV depend on ℓ\ell. We added a sentence stating this. Monotonicity holds because the diagonal variable enters only the monotone path, where all weights are positive, and the free path never depends on it. We added both points to Section 3.1.

    Requiring only the diagonal weights to be nonnegative forces all later layers to have nonnegative weights, so conditioning variables are only unrestricted in the first layer. Our free path keeps them unrestricted at every depth. Our ablation with all units monotone (ρ=1\rho=1 in Figure 8) is this alternative and performs worse on both density estimation and structure recovery (Appendix C.1).

    We added an example after Equation (7) that uses Figure 1 to show which inputs and outputs VV and UU mark, and how the orange and blue edges form the monotone and free terms.

  • Q2: You are right, the left-hand side is the matrix ∥w∥∈RK×q\lVert\mathbf{w}\rVert\in\mathbb{R}^{K\times q} of row norms, one for each output of each component. We have moved the formulas to Appendix A and now state the shape in Section 3.1.

  • Q3: We have revised Section 3.2. In the forward pass of the STE, P−sg(P)=0P-sg(P)=0, so it takes the value of the hard permutation P^\widehat{P}. In the backward pass, sg(P)sg(P) has no gradient, so the gradient is that of the SoftSort relaxation PP. We now name both terms explicitly in the text.

    The mask MM is triangular because row kk selects the first kk variables in the ordering. We added this sentence.

    You are right that the former Equation (14) undid the cumulative sum. Row kk of PSTEP^{\mathrm{STE}} already marks the diagonal variable of TkT_k, so we removed it and write Delta=PSTEDelta=P^{\mathrm{STE}}.

    We use the STE because the ordering is discrete. The STE keeps it discrete in the forward pass, so the map is exactly triangular at every step and the log-determinant in Equation (2) stays exact, while the scores receive gradients through SoftSort.

  • Q4: We have revised Section~3.3. The aim is to remove conditioning variables that a map component does not need. This gives a sparse map, and each removed variable is a missing edge in the learned graph. We follow L0L_0 regularization (Louizos et al., 2018). Each possible edge has a gate that is open with a learned probability. The penalty is the expected number of open gates, so every open gate has a cost. A gate stays open only if its variable improves the fit by more than this cost, and otherwise it closes. We keep the gates binary during training with the STE. Our ablation (Appendix C.1) compares this with hard-concrete gates (Louizos et al., 2018), which can be partly open. They give the same density but worse structure recovery, because they keep more false-positive edges.

  • Q5: No, we did not sweep these hyperparameters per setting. On the synthetic data, SSTM uses one setting for all 2424 settings. We take β\beta from Louizos et al. (2018), set λsp\lambda_{\mathrm{sp}} early in development, and fit the constant in λ1\lambda_1 once on validation data. MAF and NSF share one setting for all 2424 settings, chosen by a search on validation data. GNF was tuned separately for each sample size. On Sachs, SSTM, MAF and NSF use their synthetic settings, and GNF uses its published Sachs settings with more epochs so that it converges to a DAG. SSTM reports its final epoch, while MAF, NSF and GNF report their best validation epoch, which favors the baselines. We now state β=2/3\beta=2/3 in Section 3.3, and Appendix B reports the hyperparameters.

Reviewer_yvZmOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Self-Structuring Transport Maps (SSTM), a technique to jointly learn a triangular transport map and its structure. Triangular maps are of interest in probabilistic modeling because their structure allows for computationally efficient density evaluation through the pullback of a simple reference distribution. This makes them particularly relevant to applications such as density estimation and Bayesian inference.

Two key components determine the structure of a triangular map: the variable ordering and the sparsity pattern. The variable ordering determines the conditional density factorization, while the sparsity pattern determines the conditional dependencies represented by each map component. The guiding question of the paper is whether the map, its variable ordering, and its sparsity pattern can be learned jointly.

The proposed method combines three established techniques: BatchEnsemble, which is modified to preserve monotonicity and reduce the number of parameters, SoftSort, which provides a differentiable relaxation for learning variable orderings, and stochastic L0L_0 gates, which induce sparsity. These techniques are incorporated into a single optimization framework.

The experiments consider three settings: synthetic data with known density and graphical structure, real data without known structure, and real data with a reference graph. The authors compare SSTM against relevant SOTA methods, including GNF, MAF, NSF, and a two-stage approach. The results on synthetic datasets are generally favorable compared with GNF and the two-stage approach, although the advantages over random orderings are less consistent. On real-world datasets, SSTM demonstrates competitive density-estimation performance, while improvements in graphical structure recovery are less consistent.

Overall the paper is very well written

Critical strengths
  • The method builds on well-established techniques and introduces appropriate modifications when needed. In particular, the extension of BatchEnsemble to monotone BatchEnsemble allows the authors to share parameters across transport components while preserving the monotonicity required for triangular transport maps. Combined with SoftSort and stochastic binary gates, this provides a practical framework for jointly learning the map, ordering, and sparsit
  • The paper provides extensive comparisons against relevant methods in the literature. In particular, the comparisons with random ordering, two-stage structure estimation, and oracle structure are well chosen, as they help isolate the advantages of jointly learning the structure and transport map. The additional comparisons with GNF, MAF, and NSF provide useful benchmarks for evaluating density-estimation performanc
  • The ablation studies in Figure 8 provide evidence supporting the architectural choices of SSTM. In particular, the experiments demonstrate that parameter sharing and weight normalization improve both density estimation and structure recovery
  • The experimental results provide strong evidence for the effectiveness of SSTM on synthetic datasets. SSTM outperforms GNF and the two-stage approach in density estimation on 23 of 24 configurations. Furthermore, the experiments on funnel and hierarchical distributions demonstrate the advantages of learning the variable ordering compared with fixing a random ordering. The method also achieves favorable structure-recovery results on synthetic datasets, outperforming GNF in directed F1 on 22 of 24 configurations.
Critical weaknesses
  • Although SSTM shows strong performance compared with GNF and the two-stage approach, its advantages over a random-ordering baseline are less consistent. In Figure 3, the improvement is particularly noticeable on funnel and hierarchical distributions, while on Erd"os-Renyi and scale-free models, the results are often comparable. Similarly, on the Sachs dataset, random ordering achieves comparable density-estimation performance and better structure recovery. This raises questions about the general benefit of jointly learning the variable ordering, particularly for distributions without a strong hierarchical structure. Although the authors do not make this claim, this seems to be a limitation of the method's demonstrated benefits
  • The paper reports that SSTM requires substantially fewer optimization steps than GNF, a more detailed computational comparison would strengthen this claim. In particular, reporting wall-clock time and GPU memory requirements.
  • The real-world density-estimation experiments do not fully evaluate the proposed joint structure-learning procedure. In Appendix A.2, the authors mention that keeping all sparsity gates open provided the best validation performance for the four UCI datasets. In addition, the experiments do not include a random-ordering baseline. Could the authors provide comparisons with a fixed random ordering and report the results obtained when sparsity learning is enabled?
Questions for authors
  • I would appreciate the definition of monotonic activation being included in the paper (either in the main or supplemental text).

  • In Section 3.2, paragraph 1. The authors mention adding Gumbel nosie during training to sample multiple variable orderings Could the authors provide additional details on the noise distribution and its parameteres?I would appreciate the definition of the monotonic activation to be included in the paper (either main or supplemental text).

In Section 3.2, paragraph 1, it is mentioned that during training the authors add Gumbel noise. Can you please provide more details on the parameters of the Gumbel noise and whether any ablation studies were carried out on the effects of the Gumbel noise? Are there any ablation studies on the sensitivity of the method to the Gumbel noise?

  • The paper would benefit from having pseudocode for the complete SSTM

  • The authors report a parameter count of pq+K(p+q)pq + K(p+q) per layer. Does this expression includes the ordering scores and gate affinities?

  • Could the authors provide additional insights into the performance of SSTM on real-world datasets without reference graphs? In particular, Table 1 shows that SSTM outperforms NSF on only one of the four datasets and GNF on two. Is there an explanation for these differences, particularly given the favorable results on synthetic datasets?

  • The experiments consider dimensions up to K=43K=43. Could the authors comment on the expected scalability of SSTM to higher-dimensional problems? In particular, how do the computational cost and memory requirements scale with the dimension, considering the ordering and sparsity-learning components?.

2 comments

Thank you for your time and effort in providing us with your feedback. Based on your helpful review, we have added a random ordering to the UCI results, a cost comparison with GNF, the definition of the monotonic activation, and pseudocode for SSTM. We answer each question below:

  • Q1: We have added the definition of the monotonic activation to a new appendix and refer to it in Section 3.1.
  • Q2: We add independent standard Gumbel noise, Gj∼Gumbel(0,1)G_j \sim \mathrm{Gumbel}(0,1), to the ordering scores. The noise has no parameters to tune. The SoftSort relaxation has a temperature, and its schedule is given in Appendix B.1. We sample m=10m=10 orderings per input. We added these details to Section 3.2 and Appendix B.1. We did not ablate the noise.
  • Q3: We added Algorithm 2 to Appendix A, which gives one training step of SSTM, and refer to it in Section 3.4.
  • Q4: No, pq+K(p+q)pq+K(p+q) counts the weights of one layer. The ordering scores and gate affinities add KK and K2K^2 parameters for the whole map. We added this to Section 3.1.
  • Q5: We have added a fixed random ordering to Table 1, with open gates as for SSTM. SSTM gives a lower NLL than random on MINIBOONE, a similar NLL on GAS and HEPMASS, and a higher NLL on POWER. To separate the ordering from training, we also trained the same model with the ordering learned by SSTM fixed from the start, which we added to Appendix C. On MINIBOONE, the learned ordering gives a lower NLL than a random one. On POWER, it matches Random, so the gap there comes from training and not from the ordering. We have updated the text in the results and discussion accordingly. We believe SSTM helps most when structure matters, such as on small datasets with sparse graphs. The UCI datasets are large, and open gates performed best on validation for all four, suggesting little benefit from sparsity there. With this much data, the flexible spline layers of NSF can likely model dense dependencies well.
  • Q6: We added a cost comparison with GNF on the funnel at K=10K=10 and K=20K=20 (Appendix C). SSTM trains about 2020 times faster and samples 33 to 55 times faster, with a smaller inversion error. It uses more memory during training, mostly because each step evaluates the map under m=10m=10 sampled orderings per input. This memory doubled from K=10K=10 to K=20K=20, while training time stayed the same. The gate affinities add K2K^2 parameters and the ordering scores add KK. We do not evaluate beyond K=43K=43 and list this as a limitation.

Thank you for your reply. My main questions have been addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.