acceptodds
Under review as a conference paper at ICLR 2027

LapKDE Flow: Exact Invertible Transports Between Kernel Density Estimates

Abstract

Kernel density estimation and normalizing flows address opposite sides of the same problem: KDE is directly data-driven and effective in low-data regimes, but static; flows compose, sample, and train, but their scalar maps must be fitted. We close this gap with *LapKDE Flow*, a normalizing flow whose scalar map is , the monotone transport between two Laplace-kernel density estimates. The map is exactly invertible in closed form and admits an exact log-determinant. Anchoring the knots at data recovers KDE as a flow layer; adding rotations and depth yields a multivariate estimator; and adding gradient steps turns the same construction into a trained flow - three operating points of one construction rather than three separate models. Because its bandwidths act as statistical temperatures rather than free slopes, LapKDE Flow provides two independent routes to quantization and propagates gradient to every mixture component from every data point, unlike fixed-knot splines. With zero gradient steps, the anchored estimator outperforms classical small-sample estimators by up to bits per dimension on MNIST. When trained, the same layer improves on published Gaussianization-flow results on all seven standard benchmarks, with the advantage persisting under a matched parameter budget on three of the four comparable benchmarks. It also remains competitive on the circle and in linear ICA. The contribution is not another scalar transform, but a model family in which estimating a density and fitting a density become the same operation.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Kernel density estimation, normalizing flows, fixed-knot splines, Gaussianization flows

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_QU2iOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors build a normalizing flow from Laplace KDEs. The main benefit here is the closed-form inverse obtained by solving a quadratic within each interval between 'knots' (as they call them). They evaluate both data-only and trained versions.

Critical strengths
  • the closed-form inverse is useful: it avoids the numerical inversion required by the Gaussianization-flow. (though i'm not sure how novel this is?)
Critical weaknesses
  • the paper is difficult to read, which makes it difficult to assess
  • some central theoretical claims are dubious as stated. One proposition claims that taking both bandwidths to zero causes a diverging derivative ratio. But if the two mixtures are identical, then the map T is the identity, which implies that ratio is always 1.
  • fastICA is still stronger
Questions for authors
  • what assumptions are missing from Propositions 2 and 4?
  • What supports the claim that “two quantization paths beat one”? In Table 2, freezing h_2 gives better results than learning both bandwidths at r=10, r=100, and HEPMASS k=5
2 comments
AuthorsOpenReview ↗

Responses to the Reviewer - part 1

Q1. Missing assumptions in Props. 2 and 4.

You are right, and thank you for the counterexample: as stated, Prop. 4(iii) is false when the two mixtures coincide, and the "not C2C^2" part of Prop. 2 fails in the same degenerate cases. The missing hypotheses are:

  • (global) strictly positive weights wk>0w_k>0, coincident knots merged (Def. 1 allowed wk≥0w_k\ge 0; the implementation uses a softmax, so this always holds in practice);
  • Prop. 2: TT is C2C^2 iff (μ2,w2,h2)=(aμ1+b,w1,ah1)(\mu^2,w^2,h_2)=(a\mu^1+b, w^1, ah_1) for some a>0a>0, and then T(x)=ax+bT(x)=ax+b — your example is a=1a=1, b=0b=0; otherwise T′′T'' jumps at every breakpoint that is not both an inner knot and the preimage of an outer knot. The proof uses an exact identity in p=F(x)p=F(x): G(p):=(hf(F−1(p)))2=p2−2∑kwk(p−F(μk))+G(p):=\big(hf(F^{-1}(p))\big)^2=p^2-2\sum_k w_k (p-F(\mu_k))^{+}, so T′=h2h1G1/G2T'=\tfrac{h_2}{h_1}\sqrt{G_1/G_2} at p=F1(x)p=F_1(x) and T′′T'' is continuous exactly where G1/G2G_1/G_2 is C1C^1; matching kinks forces w1=w2w^1=w^2 by induction from G1=G2=p2G_1=G_2=p^2 below the first knot. We also correct "the K1+K2K_1+K_2 cells" to "at most K1+K2+1K_1+K_2+1 open intervals delimited by the inner knots and the preimages T−1(μ(m)2)T^{-1}(\mu^2_{(m)})" (same fix in Prop. 3);
  • Prop. 4(ii): the O(h2)O(h_2) collapse onto the outer knots holds pointwise for xx off the K2−1K_2-1 breakpoints F1−1(qk)F_1^{-1}(q_k), uniformly on compact sets avoiding them. The proof already assumes pp bounded away from the qkq_k (l. 676 of the submission); the statement did not. At a breakpoint T(x)T(x) tends to the interior of an outer gap, not to a knot;
  • Prop. 4(iii): with h1=h2=h→0h_1=h_2=h\to0, sup⁡T′/inf⁡T′\sup T'/\inf T' grows at least like ec/he^{c/h} unless F2F_2 is a translate of F1F_1, F2=F1(⋅−b)F_2=F_1(\cdot-b), in which case T(x)=x+bT(x)=x+b for every hh and the ratio is identically 1. We also drop the clause that the joint limit is "not a staircase", which is false in general; what sets (iii) apart is that the ratio diverges although both tails keep slope 1. Sketch: with h1=h2h_1=h_2, T′=G1/G2T'=\sqrt{G_1/G_2}, and on the jj-th cell G=(p−cj)2+ΛjG=(p-c_j)^2+\Lambda_j with Λj=Θ(e−gj/h)\Lambda_j=\Theta(e^{-g_j/h}), gjg_j the jj-th gap. If an inner level cjc_j is not an outer level, T′=O(e−gj1/2h)T'=O(e^{-g^1_j/2h}) at xj=F1−1(cj)x_j=F_1^{-1}(c_j); if an outer level is not an inner level, symmetrically T′=Ω(egk2/2h)T'=\Omega(e^{g^2_k/2h}); if all levels coincide (equal ordered weights), T′(xj)=Λj1/Λj2=e−(gj1−gj2)/2h(1+o(1))T'(x_j)=\sqrt{\Lambda^1_j/\Lambda^2_j}=e^{-(g^1_j-g^2_j)/2h}(1+o(1)), which tends to 0 or ∞\infty unless all gaps agree, i.e. unless F2F_2 is a translate. Both tails have slope 1 (Prop. 1), so inf⁡T′≤1≤sup⁡T′\inf T'\le1\le\sup T' and the ratio diverges. The proof in App. B.1 implicitly assumed the first alternative; the revised proof covers all three.

We verified the corrected statements numerically in 400-digit arithmetic (for small hh the effect is below float64 resolution), with a script added to the supplementary code: identical and translated pairs give log⁡10(sup⁡T′/inf⁡T′)=0.00\log_{10}(\sup T'/\inf T')=0.00 at all seven tested hh from 1 to 0.01; four non-translate pairs reach 13.0–51.8 at h=0.01h=0.01, with fitted rates c=0.300c=0.300, 1.2001.200, 0.6770.677 and 0.9930.993 (the first two where the case analysis predicts 0.3 and 1.2); T′′T'' jumps vanish exactly for identical and affine pairs and are nonzero at every inner knot of a generic pair; off the breakpoints dist(T(x),{μ2})/h2≤7.51\mathrm{dist}(T(x),\lbrace\mu^2\rbrace)/h_2\le 7.51 for h2h_2 from 0.10.1 to 0.0030.003, whereas at the breakpoints it grows from 3.9 to 133. Auditing the remaining statements, we also make explicit that Prop. 6 assumes uniform weights 1/Ki1/K_i; Def. 1 now requires wk>0w_k>0, which Prop. 5 also needs.

AuthorsOpenReview ↗

Responses to the Reviewer - part 2

Q2. What supports "two quantization paths beat one"?

Nothing in Table 2 supports this sentence; it also contradicts our own App. F.1 ("the second mixture buys nothing here"), and we withdraw it. What Table 2 does support is that a learnable bandwidth beats none: pinning every bandwidth is far worse on the sharp targets (excess 0.585 vs 0.017 at r=100r=100; test NLL −7.41-7.41 vs −23.72-23.72 at k=5k=5). Two aspects of this comparison require clarification: (i) the "h2h_2 frozen" arm is the K2=1K_2=1 link, which at matched parameter count gets twice the inner knots (K=32K=32 vs 16; 128 vs 64 on HEPMASS) and a fixed Φ−1∘σ\Phi^{-1}\circ\sigma outer map, so Table 2 does not isolate the number of quantization paths; (ii) on average the K2=1K_2=1 arm is ahead on the 1-D atoms at every r≥10r\ge 10 (in all three seeds at r=10r=10 and 30, in two of three at r=100r=100), but by only 0.002–0.006 nats against a 0.09–0.57 gap to "both frozen"; on HEPMASS the sign alternates with kk (two paths better at k=0,2k=0,2, worse at k=1,5k=1,5); and on 4-bit MNIST patches, a naturally discrete target, the two-path model is better by 1.82±0.171.82\pm0.17 nats in 3/3 seeds (94.09 vs 95.92). The revised Sec. 6.3 says that a learnable bandwidth is what wins and that, between the two arms, the sign depends on the target. Prop. 4's "two independent directions" is a structural statement (each path alone reaches a quantizer). We also corrected Remark 1: a rational-quadratic spline does reach a staircase, as its knot derivatives diverge, so the difference is one of parameterization (a single temperature per path, levels set by the estimated laws), not of function class. To test the two paths empirically we ran a 2×22\times 2 factorial at identical architecture (h1h_1 and h2h_2 each learned or frozen at e−1e^{-1}, same KK and parameter count, 5 seeds, all Table 2 targets and 4-bit patches; new table in App. F.1). The answer depends on the target. On the sharp 1-D atoms (r=10,100r=10, 100) the domain path alone matches both learned (on average within 0.005 nats), while the range path alone trails by 0.04 and 0.18 nats. On HEPMASS with five restored columns and on 4-bit patches both paths are needed at the same KK: two learned paths beat each single one in all 5 seeds (by 1.56 and 8.93 nats at k=5k=5; by 1.15 and 0.33 nats on 4-bit patches, where the range path alone recovers 85% of the gain); with twice the inner knots the K2=1K_2=1 link is again ahead at k=5k=5 (0.57 nats, 5/5 seeds). At r=1r=1 and on standard HEPMASS, with little to quantize, the four factorial arms are within 0.11 nats. The revision reports this per target, states the second path as a property of the class rather than a general gain, and softens the abstract accordingly.

Readability.

We agree. The revision compresses Secs. 3–6 and cuts part of the process narration from the main text; Sec. 6 keeps its opening roadmap from claims to experiments. For the final version we will add a notation table (knots, weights, bandwidths, levels cjc_j, qkq_k, cells) and one running example carried through Secs. 3–4.

Some theoretical claims are dubious as stated. See Q1 above: the statements are corrected, the missing hypotheses are stated, and the corrected statements are checked numerically.

FastICA is stronger.

Agreed, and the paper says so (Sec. 6.4: "FastICA is strongest overall", Tab. 3 caption). Our claim is the narrower within-search comparison: inside RADICAL's exhaustive Jacobi search, kernel marginals beat the m-spacing entropy estimate in 13 of 20 cells and on skewed sources at every dimension (App. G). The contributions paragraph and Sec. 6.4 of the revision state this scope explicitly.

Novelty of the closed-form inverse. Rational-quadratic splines also invert analytically; among mixture-CDF layers (GF, Flow++), which invert by bisection, the Laplace kernel is what makes the inverse closed-form and equal-cost (Sec. 3; Lem. 2, Thm. 1).

Reviewer_E46UOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes LapKDE Flow, a Gaussianization flow whose element-wise transform composes the CDF of one Laplace-kernel mixture with the inverse CDF of another. Because the Laplace CDF is piecewise exponential, the transform has a closed-form inverse. With the mixture components placed at data points, the model can be used without training. Otherwise it is trained by maximum likelihood like a standard Gaussianization flow.

Critical strengths
  • The closed-form inverse makes sampling faster than in bisection-based Gaussianization flows.
  • The authors report negative results and ablations openly.
Critical weaknesses
  • Limited novelty: The method is a Gaussianization flow (Meng et al., 2020) with Laplace instead of logistic mixture CDFs. The authors state that swapping the kernel gives "exactly the layer of Meng et al.".
  • The second mixture F2F_2, which motivates the title, does not improve likelihood, as the authors acknowledge (l. 372–377, Fig. 6).
  • Weak results: The model is worse than standard flows on most benchmarks:
    • MNIST: 1.25 bits per dimension, against 0.99–1.06 for Real NVP, Glow and FFJORD.
    • Tabular: MAF-DDSF is better on four of five datasets (Table 17).
    • CIFAR-10: 4.32 vs 3.35 for Glow (l. 482–483).
  • Missing baselines. Table 1 leaves out most modern flows, e.g., neural spline flows (Durkan et al., 2019), Flow++ (Ho et al., 2019), Block Neural Autoregressive Flows (De Cao et al., 2019) and Residual Flows (Chen et al., 2019). Flow++ is especially relevant because it already uses mixture CDFs as element-wise transforms. Given the model sizes (Table 6), it is unclear whether LapKDE Flow is competitive with any of them at matched parameter count. The exact inverse is not unique either, since coupling flows also have one.
  • The models are significantly larger than the Gaussianization flows they are compared against (Table 6). At matched parameter count, the gain on GAS disappears (Table 16).
  • Misleading presentation. In Tables 1 and 17, the whole LapKDE row is bold, including columns where other flows are clearly better.
Questions for authors
  • How does LapKDE Flow compare with neural spline flows and Flow++ at matched parameter count?
  • Why is the whole LapKDE row bold in Table 17, where MAF-DDSF is better on four of five datasets?
4 comments
AuthorsOpenReview ↗

Responses to the Reviewer - part 1

Q1. Comparison with neural spline flows and Flow++ at matched parameter count.

Published RQ-NSF (C) (Durkan et al., 2019, Tab. 1) reaches −0.64 / −13.09 / 14.75 / 9.67 / −157.54 nats on POWER / GAS / HEPMASS / MINIBOONE / BSDS300, against our −0.59 / −10.19 / 16.81 / 9.78 / −153.58 (on MINIBOONE within their error bar of ±0.47). Size alone does not explain the gap: NSF reports no tabular parameter counts, but instantiating its published coupling configurations in our re-implementation gives ≈2.69\approx2.69M parameters on BSDS300 (ours: 2.62M; NSF better by ≈4\approx4 nats) and ≈0.13\approx0.13M on MINIBOONE (ours: 0.55M). Flow++ reports image results only (3.08 bpd on CIFAR-10, with variational dequantization and attention conditioners), so there is no published tabular number to match. LapKDE Flow as evaluated in Tab. 1 is conditioner-free — element-wise maps and rotations, the class of GF and RBIG; conditioner networks are what NSF and Flow++ add on top of their scalar transforms.

To answer at matched size directly, we trained, in our own re-implementation, NSF's rational-quadratic spline and Flow++'s logistic-mixture transform (with residual-MLP conditioners, i.e. without Flow++'s attention and variational dequantization) in one coupling architecture (NSF's depth, blocks, bins and dropout per dataset), with the hidden width set to the largest whose exact instantiated parameter count fits the size of our Tab. 6 models (0.14M / 1.16M / 0.86M / 0.55M / 2.62M), and our transport T=F2−1∘F1T=F_2^{-1}\circ F_1 as a drop-in transform in the same architecture. All three transforms pass the same exactness tests (log-determinant vs. autograd and full Jacobian ≤10−14\le 10^{-14}, round trip ≤4⋅10−13\le 4\cdot10^{-13} in float64); ours (plus one Newton step) and the spline invert in closed form, the Flow++ transform by bisection. After training, the float32 round trip on 2,000 test points is off by at most 1.1⋅10−31.1\cdot10^{-3} for ours (7.6⋅10−37.6\cdot10^{-3} in one HEPMASS run), 3.4⋅10−43.4\cdot10^{-4} for the spline and 1.4⋅10−31.4\cdot10^{-3} for the logistic mixture.

Results (App. E.8, new Tabs. 22–23; 5⋅1045\cdot10^4 steps for every run, 3 seeds; test NLL in nats; failed seeds excluded; HEPMASS with 22 columns, App. C.3):

coupling flow at our budgetPOWERGASHEPMASSMINIBOONEBSDS300
NSF spline−0.544−12.0614.0711.93−156.67
Flow++ logistic mixture−0.549−12.13 (2 of 3)13.64 (2 of 3)11.29−157.37 (1 of 3)
LapKDE (ours)−0.573failed (0 of 3)13.8811.47−156.99

At 5⋅1045\cdot10^4 steps (every completed run outside MINIBOONE is still improving at its last step), our transform matches or beats the spline wherever it trained (paired, all seeds: −0.03, −0.46 and −0.32 nats on POWER, MINIBOONE and BSDS300, intervals excluding zero; −0.19 on HEPMASS, interval including zero); against the logistic mixture it is behind on HEPMASS (0.23 nats; two seeds, interval excluding zero) and on MINIBOONE (all three seeds, interval including zero) and ahead on POWER (all three seeds, interval including zero). It failed on GAS in all three seeds (in the seed we replayed, single points 22–30 standard deviations out gave non-finite log-densities, and a non-finite gradient then spread to every parameter). The logistic mixture failed in 4 of 15 runs (in those replayed, after its loss diverged), the spline in none; losses above 10410^4 also hit 4 of 12 completed LapKDE and 7 of 11 completed logistic-mixture runs (never the spline). Post hoc, skipping steps with a non-finite gradient and raising the stop limit from 100 to 2,500 skipped steps lets all seven failed runs train (4–199 of 5⋅1045\cdot10^4 steps skipped, three over 100; App. E.8): ours then reaches −12.15 on GAS (spline −12.06; interval including zero), and the logistic mixture leads ours on BSDS300 by 0.38 nats in every seed. With Flow++'s inverse at 40 bisection iterations (enough for float32), sampling through our closed-form inverse is faster only on POWER and GAS (2.4× and 1.7×) and 1.4–2.7× slower on the other three (at 30 iterations: 1.8× and 1.3× faster, 1.9–3.5× slower); it is 4–6× slower than the spline's, and a training step costs 3–5× more. At this size and step count the coupling flows beat our conditioner-free model (HEPMASS: its 22-column version) on GAS, HEPMASS and BSDS300, but not on MINIBOONE (where every coupling run peaks at the first validation, 5,000 steps) or POWER; the protocols differ, so this comparison is descriptive.

The revision adds this protocol and these results (App. E.8), the published RQ-NSF (C) row to Tab. 1, and RQ-NSF (AR), B-NAF and Residual Flows to Tab. 17; Flow++ reports image results only and is discussed in Sec. 2.

AuthorsOpenReview ↗

Responses to the Reviewer - part 2

Q2. The bold LapKDE row in Table 17.

The bolding rule of Tab. 1 ("best among conditioner-free models") was carried over to Tab. 17, which contains no conditioner-free competitor, so there it simply marks our own row. MAF-DDSF is better on all five tabular datasets. (In the submission our HEPMASS entry, 15.07, was ahead of MAF-DDSF's 15.09, but our loader had kept a 22nd column; re-run on the 21 published columns, our entry is 16.81 ± 0.06, App. C.3.) In the revision both tables bold the best entry per column over all rows, and Tab. 1 also underlines the best conditioner-free entry; the rule is stated in each caption. In Tab. 1 this moves the bold to FFJORD on HEPMASS, BSDS300 and MNIST and to Real NVP on Fashion-MNIST; with the published RQ-NSF (C) row added (Q1), NSF holds the bold in all five tabular columns of Tab. 1, and our row keeps only the conditioner-free underline.

Limited novelty ("GF with a Laplace kernel").

With K2=1K_2=1 and a logistic kernel our layer is GF's construction up to parametrization (learnable weights, one bandwidth per dimension rather than per anchor; App. E.7), and we say so. We went further and decomposed our margin over the published GF numbers (runs not in the submission; 3 seeds each): GF's logistic kernel in our single-mixture layer, at GF's published depth and number of anchors and trained with our recipe (anchored initialization, a single cosine schedule, Cayley rotations), reaches −0.571±0.002-0.571\pm0.002 / 9.91±0.109.91\pm0.10 on POWER / MINIBOONE, within 0.15 nats of our Tab. 1 entries (on POWER our entry also uses EMA), and 14.98±0.0914.98\pm0.09 on HEPMASS against 15.07 for our model when both see the 22 columns of our first loader (App. C.3). On MINIBOONE most of the margin in Tab. 1 therefore does not need the Laplace kernel: the recipe, our layer parametrization and Cayley rotations recover all but 0.13 nats of it. On POWER the margin needs EMA (not tested with the logistic kernel); on HEPMASS, where only our own 22-column runs can be compared, the logistic kernel matches ours. The revision states this (Sec. 6.2, new App. E.7), reframes Tab. 1 as "the closed-form inverse costs no measurable likelihood", and drops "it is the kernel, not the scale" from App. E.1. What the Laplace kernel buys is structural: a mixture-CDF transform with a closed-form, equal-cost inverse (Lem. 2, Thm. 1), exactly affine tails and computable Lipschitz constants (Props. 1, 3). Separately from the kernel question, the anchored zero-gradient operating point outperforms classical small-sample estimators by up to 0.49 bits/dim on MNIST (Sec. 6.1, Fig. 3).

Misleading presentation of Tables 1 and 17. See Q2 above: the bolding was our error, and both tables now bold the best entry per column over all rows.

AuthorsOpenReview ↗

Responses to the Reviewer - part 3

The second mixture F2F_2 does not improve likelihood.

Agreed; we report this ourselves. F2F_2 buys structure rather than likelihood: the family is closed under inversion at equal cost (Thm. 1), tails are exactly affine with slope h2/h1h_2/h_1 (Prop. 1), a frozen F2F_2 removes the sampling divergence of the deep gradient-free chain (100% vs 0% finite draws at T=1T=1 at unchanged BPD, App. D.4, Tab. 11), though it does not improve sample quality (FID 458 ± 22 vs 419 ± 3 at the FID-optimal T≈0.7T\approx0.7), and it gives exact C1C^1 flows on the circle (Cor. 2, Tab. 4). On near-discrete targets the sign depends on the target. In Tab. 2 and its full version in App. F.1, where the K2=1K_2=1 link also has twice the inner knots, the two-mixture model is ahead by 1.82±0.171.82\pm0.17 nats on 4-bit MNIST patches and by 0.34 nats with two restored HEPMASS columns, and behind on the 1-D atoms at r≥10r\ge10 (by 0.002–0.006 nats) and with one and five restored columns (by 0.25 and 0.50 nats; 3/3 seeds each). A 2×22\times2 factorial at identical architecture (App. F.1) shows that both bandwidth paths are needed on 4-bit patches and with five restored columns.

Weak results.

These comparisons are correct (on HEPMASS MAF-DDSF is now ahead too, see Q2; the CIFAR-10 number is a single exploratory run, now documented in App. E.9), and the revision makes them easier to read. They set a conditioner-free model against conditioner-based and autoregressive ones; within the conditioner-free class (RBIG, GF) the model is competitive, and the main empirical claim of the paper is the small-sample regime (Sec. 6.1), not the benchmark table.

Missing baselines (NSF, Flow++, B-NAF, Residual Flows); the exact inverse is not unique.

We add RQ-NSF (C) to Tab. 1 and RQ-NSF (AR), B-NAF and Residual Flows (MNIST 0.97, which takes the MNIST bold of Tab. 17) to Tab. 17; Flow++ reports only CIFAR-10/ImageNet and is discussed in Sec. 2 with the others; plus the matched-parameter coupling comparison of Q1.

We also agree that an exact inverse is not unique to our layer: RQ splines and coupling layers have one. The precise claim, which Sec. 3 of the revision now states, is that ours is a closed-form inverse of a mixture-CDF transform, which GF and Flow++ invert by bisection (5–251× slower sampling for a matched conditioner-free bisection stack, App. F.4; inside coupling layers our per-row inverse is slower than 40-iteration bisection on three of five datasets, see Q1).

Model size and Table 16.

Agreed, and the gap is larger than we stated: GF's image architecture is published (their App. D, Tab. 3: 10 layers, 50 anchors), so by the estimate of our Tab. 6 our models are larger on every benchmark except MINIBOONE, at least 3.9× on MNIST/Fashion-MNIST; the abstract and Sec. 6.2 now say so. On GAS (K=87K=87) the margin does not survive the matched budget (−10.06-10.06 vs −10.13-10.13, two of four seeds diverge). We re-ran the POWER, BSDS300 and HEPMASS rows of Tab. 16 with four seeds each (BSDS300 at K=38K=38, because our K=39K=39 model exceeds the unrounded GF estimate by 0.6%; HEPMASS on the 21 published columns, App. C.3): at GF's budget the model reaches −0.583 ± 0.003 / −153.50 ± 0.015 / 16.89 ± 0.11 against GF's −0.57 / −152.82 / 17.59, every seed ahead of GF, so on these three datasets the margin is not bought with parameters. The re-run agrees with the submitted rows (−0.584 ± 0.001; −153.49 ± 0.013 at K=39K=39; 16.90 ± 0.05) within two standard deviations. Two stale sentences in App. E.1 that contradicted Tab. 16 are corrected.

Note also the decomposition in our answer on novelty: at GF's own depth and number of anchors, GF's kernel in our layer, trained with our recipe, comes within 0.15 nats of our entries on POWER and MINIBOONE and of our 22-column model on HEPMASS, so on these datasets our larger size buys little (on MINIBOONE the sizes are equal; on POWER our entry's edge comes from EMA).

Thank you for addressing some of my concerns regarding the presentation of the paper.

However, my main criticisms remain:

  • Limited novelty. As far as I can tell, by the authors' own experiment in App. E.7, the layer is essentially the same as the one in Gaussianization Flows, and most of the improvement comes from the training recipe, not from the Laplace kernel.
  • Weak results. Coupling flows and published RQ-NSF are ahead on most tabular datasets.
  • Presentation. The paper is still very hard to follow, and so are the new appendices.

I will keep my score. In my opinion the paper is not publication-ready at a top-tier conference such as ICLR. I suggest a thorough rewrite, careful proofreading by colleagues, and resubmission.

Reviewer_cKJ1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces LapKDE Flow, a normalizing flow based on Laplace-kernel density estimates. It constructs exact, invertible transformations with closed-form Jacobians, enabling efficient likelihood evaluation and sampling. The method supports both gradient-free density estimation and can be trained end-to-end to further improve the precision of the estimated density. Their Experiments show that LapKDE Flow can achieve competitive results on several benchmarks compared to the baselines.

Critical strengths
  1. The proposed idea is decent; the transport between two Laplace-kernel CDFs has a closed-form inverse and a tractable log-det Jacobian. The symmetry between forward and inverse operations is elegant and useful in practice.

  2. The paper presents a useful unification; the parameters of the layer can be estimated from data to participate in a deep untrained model, and can be optimized as a normalizing flow.

Critical weaknesses

The main limitation of the proposed framework is its performance on high-dimensional data. As acknowledged by the authors, the method struggles to model high-dimensional distributions. For example, the untrained estimator fails to achieve improved likelihoods as the sample size increases on MINIBOONE, despite the dataset having only 43 dimensions. Furthermore, the authors mention that coupling-based flows can achieve better likelihoods, but provide no experimental results to substantiate this claim. It would be helpful to include a direct empirical comparison.

Additionally, the AI usage statement seems to be missing from the paper.

Questions for authors

My main question concerns the scalability of LapKDE Flow to high-dimensional data, particularly images at typical resolutions such as 128×128. How well does the method perform in this setting, and what challenges might arise when scaling to such dimensions?

3 comments
AuthorsOpenReview ↗

Responses to the Reviewer - part 1

Q1. Scalability to high-dimensional images (128×128).

In its current, conditioner-free form we do not expect LapKDE Flow to be competitive in likelihood at 128×128: already at 32×32 the trained model reaches 4.32 bits/dim on CIFAR-10 against Glow's 3.35 (Sec. 7; a single exploratory run, App. E.9). The highest resolution in the submission was the gradient-free chain on CelebA 64×64 (12,288 dimensions; App. D.3, Fig. 7c); for this response we also ran it at 128×128 (point (ii) below). The obstacles are concrete, and they concern the mixing rather than the element-wise layer:

  1. The element-wise layer's compute scales linearly. Its time is linear in the dimension DD (prefix sums once per update, a binary search per point; Lem. 1, Thm. 1), and a sandwich block has 2(2DK+D)2(2DK+D) parameters — 19.8M at D=49,152D=49{,}152 (128×128×3) and K=100K=100.
  2. Dense rotations do not. The trained model's D×DD\times D orthogonal layer has 2.4⋅1092.4\cdot10^{9} parameters at 128×128×3: 9.7 GB for the fp32 weight alone and ≈39 GB with its gradient and Adam state, and the Cayley map costs O(D3)O(D^3). Structured mixing (patch or block rotations, 1×1 convolutions, multi-scale squeezing) is required.
  3. The gradient-free chain avoids (2) with patch rotations (Dp2Dp^2 = 0.79M parameters for the 4×4 patches of our chain, 3.1M for 8×8), but it stores k×Dk\times D knots per layer: 59 GB in fp32, which sampling needs, for k=1000k=1000, m=300m=300 at 128×128 (14.7 GB at 64×64), and pushing the pool through a layer at once takes memory that grows with pool size and DD (8.6 GB for 2,000 points at 128×128), so the pool has to be pushed in chunks.
  4. Statistically, in the gradient-free chain, unlearned rotations plus marginal maps Gaussianize only marginally (App. D.4); long-range pixel dependence needs multi-scale mixing, which helps but does not close the gap (App. D.5, Tab. 14).

The natural route is to use T=F2−1∘F1T=F_2^{-1}\circ F_1 as the element-wise transform inside a conditioner-based, multi-scale architecture, where it keeps its closed-form inverse. The revision adds (i) the measured cost of each ingredient on one 10 GB RTX 3080 (App. E.9, new table): at 128×128×3 the element-wise block takes 21.6 ms forward with its log-determinant and 15.4 ms inverse (batch 64, K=100K=100), four times its time at 64×64; the dense rotation, recomputed from its Cayley parameters as in training, already takes 0.59 s and 6.7 GB per call at 64×64 (at 128×128×3 its fp32 weight alone would take 9.7 GB); the anchored mixture of one gradient-free layer (without its rotation and outer map) takes 145 ms and 8.6 GB at 128×128 when a pool of 2,000 points is pushed through it at once; (ii) the gradient-free chain on CelebA 128×128 (App. E.9 and a new figure; one run, k=500k=500 anchors, m=200m=200 layers, 4×4 patch rotations, pool 2,500): 7.04 bits/dim on 2,000 held-out images (7.32 at 64×64 with the same settings, 7.21 with the kk, mm and pool of the paper's 64×64 chain but our single-scale 4×4 patches and Laplace-mixture outer map; bits per dimension do not compare across resolutions). With the pool pushed in chunks it builds in 12.5 min with at most 4.1 GB of GPU memory, on a card shared with other jobs, and its knots take 19.8 GB of host memory. All 64 samples drawn at each temperature are finite, but at T≤0.7T\le0.7 they show nearly the same average face under strong high-frequency noise, and at T=0.8T=0.8 only noise: the construction runs at this size, but its latent is Gaussian only marginally, so isotropic draws do not reproduce the joint pixel dependence; and (iii) the drop-in comparison inside a coupling architecture at matched parameter count requested by Reviewer E46U (our re-implementation; App. E.8): at 5⋅1045\cdot10^4 steps our transform matches or beats NSF's spline wherever it trains and trails Flow++'s on HEPMASS, but it failed numerically on GAS, its loss exceeded 10410^4 at some point in 4 of its 12 completed runs, and it samples and trains more slowly than the spline. The Limitations paragraph (Sec. 7) now states the high-resolution limitation explicitly, and the abstract says that the dense model trails convolutional flows on natural images.

AuthorsOpenReview ↗

Responses to the Reviewer - part 2

MINIBOONE: the gradient-free estimator does not improve with nn.

Correct, and we report it as the boundary of the zero-gradient claim (Sec. 6.1, App. D.1, Tab. 8): the anchored column goes from 51.3 to 39.3 nats between n=50n=50 and 1000 while the trained arm reaches 26.9. The estimators that beat it there are exactly the two that exploit a global covariance (shrinkage Gaussian, full GMM), which suggests (we have not tested it) that the unlearned rotations, not the marginal estimator, are the bottleneck; on MNIST (D=784) the same anchored model wins at every nn. We state this per dataset rather than as a property of the method.

Coupling flows: claimed better without experimental results.

Agreed. We now report the published coupling and autoregressive results (e.g. RQ-NSF (C), Durkan et al., 2019: better than ours on all five tabular datasets, on MINIBOONE within their error bar) and add a matched-parameter comparison of the NSF, Flow++ and LapKDE transforms inside one coupling architecture (our re-implementation; App. E.8; see our answer to Reviewer E46U): at our model size and 5⋅1045\cdot10^4 steps these coupling flows beat our conditioner-free model (HEPMASS: its 22-column version) on GAS, HEPMASS and BSDS300, though not on POWER and MINIBOONE (a descriptive comparison: the protocols differ), and inside them our transform matches or beats the spline wherever it trains and trails Flow++'s on HEPMASS, but it fails numerically on GAS and is slower than the spline to sample and train (see Q1 (iii)).

AI usage statement.

It was in Appendix H ("Use of Large Language Models", the last page of the supplement): LLMs were used for copy-editing, for cross-checking that numbers quoted in the text match the tables, and for locating claims left unsupported after an experiment was rerun; all technical content, design decisions and interpretations are the authors'. In the revision it sits directly after the Ethics and Reproducibility statements, following the main text, so that it is not missed.

I thank the authors for their detailed response and for conducting additional experiments, including the evaluation of the gradient-free chain at a resolution of 128×128. I also appreciate their acknowledgment of the high-resolution limitation in the revised manuscript. Nevertheless, my main concern regarding the method's performance on realistic image resolutions and high-dimensional data remains unaddressed. I therefore keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.