acceptodds
Under review as a conference paper at ICLR 2027

The Wristband Gaussian Loss: Deterministic, Composable Latents via a Sphere–Interval Decomposition

Abstract

We introduce the Wristband Gaussian Loss, a deterministic batch regularizer for learning standard-Gaussian representations without per-step Gaussian reference sampling, KL penalties or an inner transport solve. Gaussianity is equivalent to joint uniformity of direction and CDF-transformed radius on a sphere–interval product. The full reflected-Neumann population energy has a unique Gaussian target, and its excess equals a squared maximum mean discrepancy (MMD), with analytic proofs and an accompanying Lean 4 formalization. Independent Gaussian blocks compose, and joint Gaussian codes support block resampling. GPU-friendly three-image pairwise and truncated spectral repulsion cost and , respectively, for radial modes and fixed angular truncation. Optional radial and moment penalties and Gaussian-null calibration balance the finite-batch training terms. At , , , three B300 input seeds give a spectral-over-pairwise speedup for full-loss forward/backward evaluation. Five-seed direct tests compare Gaussianization with analytic MMD, exact empirical assignment, refined Sinkhorn, and SIGReg. On learning representations: CIFAR-100, Shapes3D, and conditional-resampling experiments evaluate the trade-off between Gaussian representation structure and task performance. In the primary CIFAR-100 cohort, Wristband has lower held-out projector MMD than analytic MMD, with mean linear-probe accuracies of 29.56% and 30.88%, respectively.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, Gaussianization, deterministic distribution matching, latent-space regularization, spectral methods, Gaussian autoencoders, radial–angular dependence

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3UmCOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
3(+2)
Read review
Paper summary

This paper proposes the Wristband Gaussian Loss. It equivalently turns "the latent variable follows a standard Gaussian" into "the direction and the radius (after the χ² CDF transform) are jointly uniform on the sphere × [0,1]", and then uses a kernel with reflecting (Neumann) boundaries to build a deterministic batch level regularizer. It needs no Gaussian reference sampling at every step, no KL, and no inner transport solve, and it provides a spectral approximation that is linear in batch size.

Critical strengths
  1. The manuscript provides an O(N2d)O(N^2d) three-image pairwise repulsion term and an O(NdK)O(NdK) Neumann harmonic spectral approximation. It also has optional radial W2W_2 and moment terms, plus a self-calibration based on Gaussian null z-scores to balance the scales of the different terms.
  2. The manuscript's efficiency claim is backed by measurements. At N=8192N=8192 the spectral version is about 1.86× faster than the pairwise version, and much faster than Sinkhorn and CPU assignment.
Critical weaknesses
  1. The biggest problem with the manuscript is that its core idea overlaps heavily with existing work, and the experimental results are not enough to show that this new construction is more valuable than existing Gaussian regularization. The paper Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere proposed the Uniformity Loss:

Luniform=log⁡Eu,u′exp⁡(−t∥u−u′∥2)\mathcal L_{\mathrm{uniform}} = \log\mathbb E_{u,u'} \exp(-t\|u-u'\|^2)

while Wristband's Angular Kernel is:

kang(u,u′)=exp⁡(−βα2∥u−u′∥2)k_{\mathrm{ang}}(u,u') = \exp(-\beta\alpha^2\|u-u'\|^2)

So the two are the same in mathematical structure, just with different parameter notation. Wristband then multiplies by a radial kernel:

KW((u,r),(u′,r′))=kang(u,u′) krad(r,r′)K_W((u,r),(u',r')) = k_{\mathrm{ang}}(u,u')\,k_{\mathrm{rad}}(r,r')

which is essentially just extending the existing spherical uniformity kernel to a product kernel of the sphere and a radial interval. Of course, the radial product kernel and the Neumann boundary reflection are concrete technical contributions the authors can claim. But describing it as a brand new Gaussianization principle needs a much more convincing argument. Earlier work has already analyzed the representation learning effects of directly optimizing the spherical uniformity objective. I think this is a really fatal novelty problem.

  1. Also, the manuscript's theoretical guarantee is disconnected from the actual accelerated algorithm. The authors prove in Theorem 2 that:

E(P)−E(μ0)=MMD⁡KW2(P,μ0)\mathcal E(P)-\mathcal E(\mu_0) =\operatorname{MMD}_{K_W}^{2}(P,\mu_0)

and that the minimum is attained only when P=μ0P=\mu_0. So the full Wristband Kernel can uniquely characterize the standard Gaussian. But this conclusion only applies to the full infinite-image kernel, not to the spectral loss the paper uses for the speedup. In Section 5 the authors keep only ℓ∈{0,1}\ell\in\{0,1\}, i.e. the zeroth and first order spherical harmonics, and drop all higher order angular components. What is actually computed is:

Esp=λ0∑kakc0,k2+λ1∑kak∥c1,k∥2\mathcal E_{\mathrm{sp}} = \lambda_0\sum_k a_k c_{0,k}^{2} + \lambda_1\sum_k a_k\|c_{1,k}\|^2

This only checks radial statistics and information related to first order directional statistics. As a result, the experiments and the theory don't actually match. The results of the manuscript are not rigorous.

  1. The manuscript is also not that accurate on some other details. For example, the abstract says "In the primary CIFAR-100 cohort, Wristband has lower held-out projector MMD² than analytic MMD, with mean linear-probe accuracies of 29.56% and 30.88%, respectively." Here the authors only compare with analytic MMD, and leave out SIGReg (31.01%) and exact assignment (34.26%). The latter have clearly higher probe accuracy, and exact assignment is the highest in Table 3. Also, the evaluation metric itself is the MMD² with "analytic Gaussian target + five RBF bandwidths", which is of the same kind as the objective that the analytic MMD method optimizes. Wristband is actually lower on this metric, which shows that each method's λ was chosen by validation probe accuracy (main text: selection uses validation probe accuracy, not minimum MMD), so this is not a fair comparison under a distribution matching setting. The λ values differ a lot (0.1 vs 300), and Wristband's loss goes through z-score calibration, so the scales are not comparable to begin with. There are several problems like this throughout the paper, which confuses me too.

  2. The motivation of the manuscript is very weak. The motivation is stated like this: "A deterministic encoder with a standard-Gaussian output offers a useful interface…" There is no discussion of what is needed compared with existing routes such as VAEs, diffusion models, and flow models. Right after that it jumps into the technical construction.

  3. Currently the DGAE and MNIST experiments have no baselines at all: no unregularized, analytic MMD, SIGReg, VAE/CVAE, and also no control that does deterministic regression from the bottom half only. Without these, we can't tell whether Wristband contributes anything to completion quality. Gaussianity is measured only by the training loss's own calibrated score. There are no independent metrics (MMD, sliced W2W_2, KS). MNIST only has qualitative grids and its own reconstruction loss (0.0369 and 0.0533). It doesn't measure completion diversity, whether the residual is actually used by the decoder, or whether the context code and the residual code are independent. The experiments can't lead to any valid conclusion.

Questions for authors
  1. What is the fundamental methodological novelty of Wristband beyond extending the existing hyperspherical uniformity kernel with radial regularization and Neumann boundary reflections?
  2. How can the theoretical guarantees of the full Wristband kernel justify the spectral approximation when retaining only angular degrees ℓ∈{0,1}\ell\in\{0,1\} no longer guarantees Gaussianity?
  3. Why does the abstract selectively compare Wristband with analytic MMD while omitting stronger linear probe results from SIGReg and exact assignment, and how can Gaussianity comparisons be considered fair when regularization strengths are selected based on probe accuracy?
  4. What concrete limitations of existing approaches, such as VAEs, normalizing flows, and Gaussian regularization methods, motivate the proposed framework, and what practical advantages does Wristband offer over them?
  5. Without comparisons against existing Gaussian regularizers and conditional generation baselines, or independent evaluations of Gaussianity, conditional diversity, and latent independence, how can the DGAE and MNIST experiments substantiate the claimed benefits of Wristband?
7 comments

Thank you for the direct review. Several of your points identified real problems in how the submission presented itself, and we have changed the paper accordingly; on two points we believe the submission was misread, and we explain why while accepting that the writing invited it. We also ran the experiments you asked for, including baselines for both the mixture autoencoder and MNIST, and report their outcome: neither favours our method. We answer your five questions in order. Section, table and appendix numbers refer to the revised PDF; new text is in blue.


Q1. What is the methodological novelty beyond extending the hyperspherical uniformity kernel with a radial term and Neumann reflection?

You are correct that our angular factor is the uniformity kernel of Wang & Isola. This is by design and was stated in the submission ("Wristband extends the hyperspherical uniformity objective of Wang and Isola (2020)"; "Wristband retains this angular construction"). The submission did not, however, say precisely what is and is not new. Sec. 2 now does:

Shared. kang(u,u′)=exp⁡(−βα2∥u−u′∥2)k_{\rm ang}(u,u')=\exp(-\beta\alpha^2\|u-u'\|^2) is their kernel with t=βα2t=\beta\alpha^2. If all embeddings had equal radius, our repulsion would reduce to their loss up to constants.

New. Three things, none of which follows from "multiply by a radial kernel":

  1. The target space (Theorem 1). The uniformity loss cannot express a Gaussian target because it discards the radius. Theorem 1 shows that the radius, after the χd2\chi^2_d CDF transform, enters as one extra coordinate that must be uniform on [0,1][0,1] and independent of direction: Q=N(0,Id)Q=\mathcal N(0,I_d) if and only if Φ#Q=σd−1⊗Unif[0,1]\Phi_\#Q=\sigma_{d-1}\otimes\mathrm{Unif}[0,1]. The independence requirement is the substantive part. "Uniform directions plus a correct radial marginal" is strictly weaker; our RAC construction satisfies both marginals and is not Gaussian, and Radial-VCReg, which is the natural "uniformity plus radius" method, leaves the structure in App. G unrepaired.
  2. The reflection is what makes the minimizer correct (Theorem 2). A Gaussian kernel on an interval does not have the uniform law as its energy minimizer: under the uniform law the potential near an endpoint is about half its interior value, so repulsion pushes mass to the boundaries and biases the radius law. A naive product kernel would therefore be minimized by a non-Gaussian law. The Neumann reflection restores a constant potential, and this is exactly the property that yields the unique minimizer and the identity E(P)−C=MMDKW2(P,μ0)\mathcal E(P)-C=\mathrm{MMD}^2_{K_W}(P,\mu_0) on the product space. Reflection is not boundary cosmetics; without it the theorem is false.
  3. Structure for computation. The product of spherical harmonics and Neumann cosines gives the separable expansion behind the O(NdK)O(NdK) evaluation, with a radial truncation bound and an explicit characterization of what the truncation can and cannot see.

Why it matters in practice. Independent Gaussian blocks concatenate to a Gaussian; separately normalized spherical blocks do not concatenate to a uniform sphere (the energy split would have to be Beta-distributed). So the Gaussian target, not the spherical one, supports composition and block resampling.

We do not claim a new principle of Gaussianization in general; MMD, sliced and characteristic-function methods already target the Gaussian, and Sec. 2 says that linear-cost Gaussian regularization is not our novelty claim. We claim a specific kernel construction with a population guarantee, which the revised contributions paragraph states in one sentence.


Q2. The guarantee holds for the full infinite-image kernel, but the accelerated algorithm keeps only angular degrees ℓ∈{0,1}\ell\in\{0,1\}. How can the theory justify it?

It does not, and the paper does not use it that way. Two clarifications, now in a dedicated paragraph of Sec. 5 ("Which guarantee applies to which path"):

  1. The experiments on learned representations do not use the spectral loss. CIFAR-100, Shapes3D, MNIST and conditional resampling all use the pairwise path, Eq. (7). That path has no angular truncation: it evaluates the exact angular kernel. It differs from the theorem's kernel only in truncating the radial image sum to three images, with uniform error at most 2e−β/(1−e−3β)=6.7×10−42e^{-\beta}/(1-e^{-3\beta})=6.7\times10^{-4} at β=8\beta=8, and App. L converts this into an explicit two-sided bound on the empirical MMD. So for the pairwise path, theory and experiment match up to a stated, small, bounded term.
  2. The spectral path is an optional accelerator and is labelled as such. The submission already said that a finite angular expansion is "noncharacteristic", gave the hypercube-vertex counterexample, and stated that "the pairwise path has no angular truncation". Wherever spectral results appear (Table 1, Table 2, the angular-stress test) they are reported in a separate row named "spectral", and they are worse where theory predicts (X-10D: 7.32 vs. 2.13; angular stress: z=14.8z=14.8 vs. 2.67).

We accept that the submission's organization made it easy to read the spectral loss as "the method". The revision (i) states the scope of each path in one place, (ii) characterizes exactly what the spectral loss does guarantee (minimized iff T∼Unif[0,1]T\sim\mathrm{Unif}[0,1] and E[U∣T]=0\mathbb E[U\mid T]=0; App. L), and (iii) adds a pairwise-audit diagnostic (Sec. 5, App. Z.2) for detecting when the spectral features are insufficient. On the vertex counterexample at d=10d=10 the audit gives zpair=15.7±1.3z_{\rm pair}=15.7\pm1.3 while zsp=0.2±1.0z_{\rm sp}=0.2\pm1.0; on the spectrally trained X-10D clouds, 38.2±1.538.2\pm1.5.


Q3. Why does the abstract compare only with analytic MMD, omitting SIGReg and exact assignment? How is the Gaussianity comparison fair when λ\lambda is selected by probe accuracy?

The abstract. You are right, and we have rewritten it. The body of the submission did report all seven methods and said "assignment has the highest mean probe accuracy", but the abstract's single comparison was selective. The revised abstract states that Wristband's probe accuracy is about 1.5 points below SIGReg and analytic MMD (29.7% against 31.3% and 31.0% over two sets of runs) and well below exact assignment (34.3%). It also states that Wristband shows no advantage in two low-dimensional autoencoders (Q5), and that the experiments support one use: a more Gaussian high-dimensional projector, at some cost in probe accuracy. Reruns at the published settings on a second GPU reproduced the gap (SIGReg and MMD 1.6±1.31.6\pm1.3 and 1.3±1.21.3\pm1.2 points above Wristband over six paired differences), so we state it as real rather than as seed variation.

The metric. The external metric is the analytic-target RBF MMD², which is the statistic that the analytic-MMD baseline directly optimizes. A metric of the same kind as a baseline's objective favours that baseline, not us. To avoid relying on one metric, Table 4 now has four more diagnostics and a Gaussian-null row. Sliced Wasserstein and radial KS required projector features that the original runs had not saved, so we reran four methods with identical configurations and initial weights (App. Z.6):

Projector diagnostic (d=256d=256)WristbandSIGRegAnalytic MMDiid Gaussian samples
MMD² U (10−310^{-3})0.090.370.560.00±0.010.00\pm0.01
Gaussian-fit W22/dW_2^2/d0.0710.3250.4750.066
Effective rank (max 256)223.5148.6121.7225.8
Sliced W22/2W_2^2/2 (10−310^{-3}), full test split0.561.672.030.34
Radial KS0.080.240.210.03

(The sliced row is the full-test-split value from App. Z.6. Table 4 lists the n=1024n=1024 value, where the statistic is dominated by its sampling floor: 3.32 / 4.48 / 4.81 against 3.07 for Gaussian samples.)

Wristband is closest to the Gaussian reference on each. It is not at the reference on radial KS, and the paper says so.

The operating points. This is a fair criticism of the CIFAR-100 table as submitted (Table 3 there, Table 4 now): each λ\lambda was chosen by validation probe accuracy, so the table compares utility-selected points (the submission said "selection uses validation probe accuracy, not minimum MMD" but did not address the consequence). The revision separates method from operating point with Gaussianity–utility curves over every screened coefficient, on three seeds (Fig. 1; App. Z.3). Over a 20×20\times range of its coefficient Wristband stays at MMD² 1.11.1–2.0×10−42.0\times10^{-4} with 29.5–32.2% validation accuracy. No screened coefficient of analytic MMD or SIGReg reaches that Gaussianity at any accuracy: their most Gaussian settings are 5.4×10−45.4\times10^{-4} at 27.9% and 6.7×10−46.7\times10^{-4} at 26.6%. The Gaussianity gap is therefore not produced by the selection rule.

The same curves show that accuracy rankings depend on the training budget, which we report rather than select from: at the 50-epoch screening budget Wristband's selected point (32.2 ± 0.5%) is above the best MMD and SIGReg settings (29.2%, 29.5%); at the 200-epoch confirmation budget it is about 1.5 points below them. Coefficients were not re-tuned at the long budget for any method.

On the different magnitudes of λ\lambda (0.1 vs. 300). These are not comparable and we now say so. Wristband's loss is a zz-score against Gaussian batches; MMD's is a raw statistic of order 10−310^{-3}. Each method's coefficient was searched over its own six-value grid spanning about two orders of magnitude.


Q4. What limitations of VAEs, normalizing flows and Gaussian regularization motivate the framework?

The introduction now has a paragraph on this ("Why another route to Gaussian latents?"). In brief:

  • VAEs obtain the prior through a stochastic encoder and a per-sample KL. The KL acts on each conditional posterior, not on the aggregate code distribution, so the aggregate need not match the prior, and encoder noise trades against reconstruction. In our conditional-resampling study (Table 3) a VAE-style encoder has higher task error (0.056 vs. 0.035) and higher conditional error (0.075 vs. 0.062) than Wristband; in the new MNIST comparison a conditional-VAE arm failed on two of five seeds (App. Z.8), and in the new mixture comparison a VAE has the most Gaussian sampled codes but by far the largest reconstruction error (App. Z.9).
  • Normalizing flows give exact Gaussian latents but require invertible, dimension-preserving architectures and a likelihood; they cannot be attached to an arbitrary encoder or a projector head.
  • Diffusion / flow matching learn a transport to noise that is integrated at sampling time; they do not provide a compact deterministic code.
  • Batch regularizers (WAE-MMD, sliced Wasserstein, SIGReg) share our setting: arbitrary deterministic encoder, constraint on the aggregate law. They differ in which statistic is matched. Reference-sampled MMD and transport add sampling noise or an inner solve; projection-based statistics see the joint law through finitely many slices. Wristband matches a single kernel statistic whose population minimizer is exactly the Gaussian, with no reference samples, organized around direction and radius.

The measured practical differences are: on CIFAR-100, projector codes closer to Gaussian than analytic MMD or SIGReg reach at any screened coefficient (Fig. 1, Table 4); insensitivity of that Gaussianity to the coefficient over a 20×20\times range; and, for large batches, a spectral option whose memory is flat in batch size (84 MiB vs. 2.6 GiB for the pairwise path at N=8192N=8192; Table 2). The measured disadvantages are stated with equal weight: analytic MMD has lower sliced error on free point clouds; Wristband's CIFAR-100 probe accuracy is about 1.5 points lower at the full budget; the spectral path is as cheap as linear-time MMD but not cheaper (App. Z.7); on MNIST completion Wristband shows no advantage over MMD or SIGReg; and in the mixture autoencoder analytic MMD and SIGReg reconstruct and generate better than Wristband (Q5).

Reviewer_vW3pOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a new regularizer that directly targets Gaussianity rather than moment matching. The authors introduce the wristband map, which consists of a direction and the cumulative distribution function (CDF) value of the squared norm. They show that the push-forward distribution under this map is uniform on the product space of a sphere and the interval ([0,1]) if and only if the original distribution is Gaussian. They further construct a reflected kernel whose population energy is uniquely minimized by the standard Gaussian, with the excess energy characterized by squared maximum mean discrepancy (MMD).

For implementation, the authors propose a pairwise kernel-based loss, where the reflected kernel is decomposed into three radial terms accounting for the original point and its reflections across the boundaries at 0 and 1. They also consider optional radial and moment penalties and introduce an efficient implementation based on spectral approximation.

The experiments demonstrate improved Gaussianity of learned representations on CIFAR-100 and increased computational efficiency of the spectral loss at larger batch sizes. However, a trade-off between task performance and distributional fidelity is observed.

Critical strengths
  1. The proposed regularizer is theoretically well-motivated and directly targets Gaussianity, addressing limitations of existing regularizers based on moment or radial matching. The reflected kernel accounts for boundary conditions through Neumann reflection, and the authors provide a population-level characterization along with approximation methods for efficient computation.
  2. The experiments cover a range of settings. The point-cloud experiments help isolate the behavior of the proposed regularizer, while the representation learning and resampling experiments investigate its practical applicability.
Critical weaknesses
  1. The presentation is somewhat condensed, making parts of the methodology and experimental setup difficult to follow. A more detailed explanation of the pairwise repulsion loss in Eq. (7) would be helpful. Although the authors mention excluding self-interactions, it is unclear whether the diagonal contribution of the reflected image kernel is also excluded, particularly since kimg(t,t)≠1k_{\mathrm{img}}(t,t) \neq 1. It would also be helpful to clarify whether the logarithm is applied after averaging over the batch.

    In addition, the transition to the spectral approximation in Eqs. (13)–(15) is rather abrupt. Providing a brief derivation would help readers understand the approximation and its computational benefits.

    Similarly, although experimental details are available in the appendix, a concise description of the experimental protocols—including the model architectures, training procedures, and evaluation metrics—would improve the clarity of the main text.

  2. The practical advantages of the wristband loss over existing alternatives are not yet fully established. In particular, the spectral approximation may not preserve all the theoretical properties of the full kernel, and its practical implications could be discussed more explicitly.

    Empirically, the wristband loss often performs comparably to or slightly worse than analytic squared MMD. The computational cost and performance on CIFAR-100 also appear similar. Moreover, in the RAC experiments, the wristband loss does not fully eliminate radial–angular dependence, with a reported correlation coefficient more than three times that of analytic squared MMD.

    Given that analytic squared MMD provides a relatively straightforward distribution-matching objective, a more systematic comparison would help establish the practical benefits of the proposed method.

Questions for authors
  1. The squared MMD values reported for Wristband, analytic MMD, and SIGReg's projector in Table 3 are all relatively small. Could the authors clarify how these differences translate into actual Gaussianity? Reporting the sliced Wasserstein distance and radial one-sample KS discrepancy, as in other experiments, would help contextualize these results.
  2. Could the authors provide Gaussianity–utility trade-off curves across different regularization strengths for Wristband, analytic MMD, and SIGReg? Such a comparison would help assess the trade-off between distributional fidelity and downstream task performance.
  3. Could the authors clarify what the MNIST conditional completion experiment is intended to demonstrate about Wristband? Gaussian latent representations do not appear to be essential for this task, and the conditional reconstruction objective explicitly trains the decoder to use the observed bottom-half code together with Gaussian noise. It is therefore difficult to determine how the results isolate the benefits of the proposed regularizer. Additionally, what role does the invertible affine-coupling postprocessor ff play in this experiment?
  4. The population analysis relates the repulsion objective to the logarithm of a constant plus squared MMD. Could the authors elaborate on the practical motivation for applying the logarithm in Eq. (7)? Have they considered comparing this formulation with the kernel energy without the logarithmic transformation? Such an ablation would help clarify whether the logarithm provides additional optimization or performance benefits beyond the choice of kernel.
6 comments

Thank you for the detailed review. Your presentation concerns were justified and have led to the largest changes in the revision, and we ran every experiment you asked for: sliced-Wasserstein and radial KS on CIFAR-100, trade-off curves over regularization strength, MNIST baselines, and a logarithm ablation. Section, table and appendix numbers refer to the revised PDF; new text is in blue.


W1a. Eq. (7): is the diagonal of the reflected-image kernel also excluded? Is the logarithm applied after averaging over the batch?

Both are now stated explicitly under Eq. (7) ("Reading Eq. (7)").

  • The double sum has 3N23N^2 terms: one real and two reflected-image terms per ordered pair. On the diagonal the real term is exactly 1; subtracting NN removes only these NN real self-interactions, which is why the denominator is 3N2−N3N^2-N.
  • The 2N2N reflected diagonal terms, e−4βti2e^{-4\beta t_i^2} and e−4β(1−ti)2e^{-4\beta(1-t_i)^2}, are retained. A point does interact with its own mirror images; this is what stops points from accumulating at t=0t=0 or t=1t=1. As you noticed, these terms are not constant (they depend on tit_i), so they contribute gradients; App. L gives the resulting expectation, EAN=[(N−1)Eimg(P)+E r(T)]/(3N−1)\mathbb E A_N = [(N-1)\mathcal E_{\rm img}(P) + \mathbb E\, r(T)]/(3N-1).
  • The logarithm is applied once, after averaging over the batch ("global" reduction). This is the form used in the direct, CIFAR-100 and timing studies. The earlier autoencoder examples use a "per-point" variant (logarithm of each row mean, then averaged), which we now name as a different finite-batch objective.

W1b. The transition to Eqs. (13)–(15) is abrupt.

Sec. 5 now opens with the derivation. The kernel is a sum of separable terms, krad=∑kakcos⁡(kπt)cos⁡(kπt′)k_{\rm rad}=\sum_k a_k\cos(k\pi t)\cos(k\pi t') and kang=∑ℓ,mλℓYℓm(u)Yℓm(u′)k_{\rm ang}=\sum_{\ell,m}\lambda_\ell Y_{\ell m}(u)Y_{\ell m}(u'), so each mode's double sum over pairs factorizes into the square of a single sum over points:

1N2∑i,jKW(wi,wj)=∑ℓ,m,kλℓak(1N∑iYℓm(ui)cos⁡(kπti))2.\frac{1}{N^2}\sum_{i,j}K_W(w_i,w_j)=\sum_{\ell,m,k}\lambda_\ell a_k\Big(\frac1N\sum_i Y_{\ell m}(u_i)\cos(k\pi t_i)\Big)^2 .

Each retained mode costs O(N)O(N) instead of O(N2)O(N^2). There are O(dℓ)O(d^\ell) harmonics of degree ℓ\ell, so ℓ≤1\ell\le1 (the constant and d u\sqrt d\,u) is what can be computed in O(Nd)O(Nd) per radial mode; that gives Eqs. (13)–(14). App. Z.1 has the full derivation and the two differences from Eq. (7) that follow from it (the spectral statistic includes self-pairs, and it truncates the full Neumann kernel in frequency rather than in images).

W1c. A concise description of protocols in the main text.

Sec. 6 now opens with "Protocols in brief" (data, architecture, objective, selection rule, metrics for the direct and CIFAR-100 studies), and the full protocol appendices are in the same PDF rather than only in the supplement.


W2. Practical advantages over analytic MMD are not established; the spectral approximation loses properties; Wristband's RAC correlation is three times MMD's.

We ran the comparisons you asked for and have tried to make the revised paper say only what they support.

Spectral approximation. A new paragraph in Sec. 5 states which guarantee covers which path. Every learned-representation experiment (CIFAR-100, Shapes3D, MNIST, conditional resampling) uses the pairwise path, which keeps all angular degrees and differs from the theorem's kernel only by a radial image tail of at most 6.7×10−46.7\times10^{-4}. The spectral path is an optional accelerator for large batches; it is characteristic only for "TT uniform and E[U∣T]=0\mathbb E[U\mid T]=0", and we add a diagnostic for when that is not enough (App. Z.2).

RAC. You are right, and the revision says so more directly: RAC is the least favourable case for Wristband at this budget, and analytic MMD removes more of the coupling. New analysis (App. Z.2) explains why: the coupling lives in high-order, sign-symmetric angular structure, to which the finite-batch kernel statistic has little power at N=1024N=1024. The two methods are complementary rather than ordered. In X-10D and RAC-10D analytic MMD leaves radial KS of 0.107 and 0.137, against 0.008 and 0.007 for full Wristband, and the Wristband statistic flags the MMD-trained clouds at z>12z>12.

Where Wristband is and is not better than analytic MMD. Our summary after the new experiments:

  • Better: Gaussianity of a learned projector on CIFAR-100, on all five diagnostics of Table 4 and against every screened coefficient of analytic MMD (Q1, Q2 below); insensitivity of that Gaussianity to its coefficient.
  • Worse: sliced error on free point clouds (all four problems, stated in the abstract); CIFAR-100 probe accuracy at the full budget, by about 1.3 points over two sets of runs.
  • Worse: the 8-d mixture autoencoder, to which we added baselines at another reviewer's request (App. Z.9). With validation-selected weights analytic MMD reconstructs better (MSE 0.049 vs. 0.089) and its samples decoded from the Gaussian prior are closer to held-out data (energy distance 0.041±0.0090.041\pm0.009 vs. 0.113±0.0030.113\pm0.003, on all five seeds); SIGReg matches analytic MMD. Wristband keeps the better radius law and covariance there, as in the direct tests.
  • No difference: MNIST completion at matched regularization strength (Q3 below).

Q1. The MMD² values in Table 3 are all small. How do the differences translate into actual Gaussianity? Reporting sliced Wasserstein and radial KS would help.

Done. The CIFAR table (now Table 4) has five diagnostics and a Gaussian-null row.

Sliced Wasserstein and radial KS. The published runs had saved only scalar diagnostics, so we reran Wristband, SIGReg, analytic MMD and VICReg at the published settings (seeds 100–102, 200/100 epochs) with the projector features saved (App. Z.6). Configurations and initial weights are identical to the published runs; the GPU model differs. The reruns reproduce the published diagnostics (e.g. effective rank 223.4 / 148.1 / 122.2 / 224.2 against the published 223.5 / 148.6 / 121.7 / 224.1).

Projector diagnostic (d=256d=256)WristbandSIGRegAnalytic MMDVICRegiid Gaussian samples
MMD² U (10−310^{-3}), n=1024n=10240.09±0.030.09\pm0.030.37±0.020.37\pm0.020.56±0.030.56\pm0.0330.4±2.330.4\pm2.30.00±0.010.00\pm0.01
Gaussian-fit W22/dW_2^2/d, n=1024n=10240.0710.3250.4750.3120.066
Effective rank (max 256), n=1024n=1024223.5148.6121.7224.1225.8
Sliced W22/2W_2^2/2 (10−310^{-3}), n=1024n=10243.32±0.053.32\pm0.054.48±0.174.48\pm0.174.81±0.244.81\pm0.24130±11130\pm113.07±0.103.07\pm0.10
Sliced W22/2W_2^2/2 (10−310^{-3}), n=10,000n=10{,}0000.56±0.030.56\pm0.031.67±0.081.67\pm0.082.03±0.142.03\pm0.14126±11126\pm110.34±0.010.34\pm0.01
Radial KS, n=1024n=10240.080±0.0300.080\pm0.0300.241±0.0110.241\pm0.0110.209±0.0060.209\pm0.0060.864±0.0130.864\pm0.0130.027±0.0070.027\pm0.007
Radial KS, n=10,000n=10{,}0000.077±0.0130.077\pm0.0130.237±0.0200.237\pm0.0200.195±0.0060.195\pm0.0060.860±0.0150.860\pm0.0150.008±0.0030.008\pm0.003

(First three rows: original runs; sliced and radial rows: reruns. Mean ± SD over three seeds.)

  • Scale of MMD². For iid Gaussian samples of the same size the estimator has standard deviation 0.013×10−30.013\times10^{-3}. Wristband is 6–7 null SDs from zero, SIGReg about 28, analytic MMD about 42.
  • What the gaps mean. At their selected coefficients the SIGReg and MMD projectors have effective rank 149 and 122 out of 256: they use roughly half of the available directions. Wristband's is 224, against 226 for Gaussian samples.
  • Sliced error. At n=1024n=1024 the statistic is dominated by its sampling floor, so the methods look closer than they are; on the full test split Wristband is 1.6× the floor, SIGReg 4.9× and analytic MMD 5.9×.
  • Radial KS. Wristband's radius law is about three times closer than SIGReg's or analytic MMD's. It is not at the sampling floor (0.08 against 0.03 at n=1024n=1024), and the paper says so.

Q2. Gaussianity–utility trade-off curves across regularization strengths for Wristband, analytic MMD and SIGReg.

Added as Fig. 1, with every value in App. Z.3. Every screened coefficient of Wristband, analytic MMD, SIGReg and VICReg was run on three seeds (the original seed 17 plus two new ones; 50 SSL / 20 probe epochs). Ranges of the three-seed means:

MethodCoefficientsVal. probe (%)Projector MMD² U
Wristband0.01 – 0.2 (5 values)29.5 – 32.21.11.1 – 2.0×10−42.0\times10^{-4}
Analytic MMD3 – 1000 (6 values)21.5 – 29.21.4×10−11.4\times10^{-1} – 5.4×10−45.4\times10^{-4}
SIGReg0.3 – 10 (5 values)23.6 – 29.56.7×10−46.7\times10^{-4} – 1.7×10−31.7\times10^{-3}
VICReg0.1 – 30 (6 values)23.2 – 32.21.7×10−21.7\times10^{-2} – 2.0×10−12.0\times10^{-1}

(Wristband at its smallest screened coefficient, 0.003, is under-regularized: 30.1%, 6.9×10−26.9\times10^{-2}. SIGReg at 30 diverged in the original screen and was not repeated.)

Three conclusions.

  1. Wristband's Gaussianity is insensitive to its coefficient over a 20×20\times range.
  2. No screened coefficient of analytic MMD or SIGReg reaches Wristband's Gaussianity at any accuracy: their most Gaussian settings give 5.4×10−45.4\times10^{-4} at 27.9% and 6.7×10−46.7\times10^{-4} at 26.6%. The gap in the confirmation table is therefore not an artifact of each method being stopped at a different point on its own curve.
  3. Accuracy rankings depend on the training budget, and we report this rather than choose the favourable one. At this 50-epoch budget Wristband's selected point (32.2±0.5%32.2\pm0.5\%) is level with VICReg and above the best analytic-MMD and SIGReg settings (29.2%, 29.5%). At the 200-epoch confirmation budget the order reverses: over the original runs and the reruns, SIGReg and analytic MMD are 1.6±1.31.6\pm1.3 and 1.3±1.21.3\pm1.2 points above Wristband. Coefficients were selected at the short budget and not re-tuned at the long one, for every method. The Gaussianity ordering is the same at both budgets.

Q3. What is the MNIST experiment intended to demonstrate? Gaussian latents do not appear essential, and the results do not isolate a benefit of the regularizer. What is the role of ff?

You were right, and we ran the experiment that tests it (App. Z.8). Its answer is that the MNIST example does not show a benefit of Wristband over other Gaussian regularizers.

What we ran. Six arms with the same architecture and schedule, five seeds each: Wristband; analytic MMD and SIGReg on the same joint code; a conditional VAE (stochastic residual with a KL term); no regularizer; and a deterministic regression control whose decoder receives no residual. Metrics on 6,000 held-out images: reconstruction, conditional L1L_1, diversity of 32 completions per image, accuracy of a separately trained digit classifier (99.5%) on completed images, HSIC between context and residual codes with a permutation null, and Gaussianity of the joint code against a Gaussian null. Only an example script survives from the original experiment, so the protocol is partly reconstructed; every such choice is recorded.

ArmRec. MSE (10−310^{-3})Cond. L1L_1 (10−310^{-3})DiversityCompletion acc. (%)HSIC (10−410^{-4})Code MMD² U (10−410^{-4})Code radial KS
Wristband, λ=0.1\lambda=0.115.8±0.215.8\pm0.252.4±0.352.4\pm0.32.85±0.072.85\pm0.0783.8±0.283.8\pm0.29.4±0.79.4\pm0.77.9±0.17.9\pm0.10.014±0.0060.014\pm0.006
Analytic MMD, w=1w=1 (selected)8.0±0.18.0\pm0.140.6±0.240.6\pm0.20.54±0.160.54\pm0.1689.6±0.289.6\pm0.23.1±1.03.1\pm1.03.3±0.83.3\pm0.80.044±0.0070.044\pm0.007
Analytic MMD, w=1000w=1000 (one run)15.654.22.8783.31.71.60.027
SIGReg, w=10−3w=10^{-3} (selected)8.4±0.18.4\pm0.141.6±0.241.6\pm0.20.86±0.090.86\pm0.0989.0±0.289.0\pm0.22.4±0.42.4\pm0.42.7±0.32.7\pm0.30.057±0.0180.057\pm0.018
SIGReg, w=0.1w=0.1 (one run)14.352.42.8183.32.52.60.026
Regression control (no residual)17.1±0.217.1\pm0.242.4±0.842.4\pm0.8090.7±0.490.7\pm0.4–––
Null (permutation / iid Gaussian)––––0.60.0±0.10.0\pm0.10.011±0.0040.011\pm0.004

What it shows.

  1. Completion accuracy mostly rewards not sampling. The regression control has the highest accuracy with zero diversity, and for every regularizer accuracy falls as diversity rises with the weight. This supports your remark that Gaussian latents are not essential when the task is scored against the single true image.
  2. At matched use of the residual, the regularizers are interchangeable. Wristband's row is close to analytic MMD at w=1000w=1000 and SIGReg at w=0.1w=0.1 on every completion metric.
  3. No arm's joint code is Gaussian, and every arm fails the independence test. Wristband has the best radius law (at its null) and covariance, but a larger MMD² and a larger HSIC than the MMD and SIGReg arms.
  4. A regularizer on the joint code is needed. Without one the decoder ignores the sampled residual (variance of completed pixels across draws below 10−410^{-4}, against 0.012 for Wristband and for MMD and SIGReg at their strongest weights), and the unregularized and CVAE arms ended with exploded codes and poor reconstructions on two of five seeds each.
Reviewer_z1LuOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a way to organize a neural network’s learned representations into a predictable, "Gaussian" like distribution. The method starts with how far the samples are from the center. It encourages points to spread evenly in direction, follow the right pattern of distances, and avoid unwanted connections between direction and distance. The authors introduces a faster approximation for training on large batches. The main trade-off is that a better-organized representation is not always more useful for the task. Their experiments show improved matching to the desired distribution, but sometimes slightly worse classification accuracy.

Critical strengths

The main idea is to decompose embeddings into direction and radius, then transform the radius using the Gaussian radial CDF. Gaussianity becomes joint uniformity on a sphere × interval, requiring the correct directional distribution, radial distribution, and independence between them. The idea is very nice.

Strengths:

  1. The theory is clean, and the spectral approximation’s angular blind spot is proven with an explicit counterexample - this is very nice and my main motivation towards acceptance of this paper.
  2. The paper turns Gaussian representation learning into controlling the directions and distances of points, providing an interpretable geometric foundation
  3. The used regularizer avoids generating Gaussian reference samples at every training step or solving an inner transport problem, simplifying its computation.
  4. The spectral approximation reduces the cost of processing large batches, and the authors measure the resulting speed gains while acknowledging approximation limitations. This is super useful.
  5. Balanced experimental evaluation. Experiments examine distribution matching, classification, and latent resampling. The paper explicitly reports trade-offs between achieving Gaussian representations and preserving useful task information.
Critical weaknesses
  1. Better Gaussian matching does not clearly improve task performance. On CIFAR-100, Wristband produces representations closer to Gaussian but achieves lower linear-probe accuracy than analytic MMD. This leaves its practical advantage for representation learning uncertain - this is something I want authors to comment on during rebuttal, this is my main reason of weak rejection. 2.The full objective uniquely targets a Gaussian distribution, but the truncated spectral version can miss higher-order angular structure. Consequently, a low approximate loss does not necessarily indicate successful Gaussian matching. This is not necessarily a weakness but something to mention and discuss.
  2. The speed advantage is demonstrated in a limited setting. It does not establish a general efficiency advantage over other inexpensive Gaussian regularizers, such as SIGReg or linear-cost MMD alternatives.
Questions for authors
  1. The proposed model improves Gaussian matching but slightly reduces CIFAR-100 accuracy relative to analytic MMD. In which applications does its stronger Gaussian structure translate into a clear downstream advantage?
  2. Can the authors provide a diagnostic for when the approximation is insufficient?
  3. How does Wristband compare with SIGReg and linear-cost MMD in runtime, memory etc.?
  4. How well does the core Wristband loss perform without radial and moment penalties? Can one default configuration transfer across datasets, embedding dimensions, and batch sizes? This will be useful for someone trying in practice.
7 comments

Thank you for the careful and constructive review, and in particular for treating the proven angular blind spot as a strength rather than a defect. We answer your four questions in order. On two of them (downstream advantage and efficiency) the new experiments support your reading rather than our original framing, and the paper now says so. Section, table and appendix numbers refer to the revised PDF; new text is in blue.

Q1 / W1. Better Gaussian matching does not improve CIFAR-100 probe accuracy. Where does stronger Gaussian structure give a clear downstream advantage?

The probe gap is real. To add the diagnostics requested by another reviewer we reran four methods at the published settings with identical configurations and initial weights on a second GPU model (App. Z.6). The reruns reproduce the published runs, and across both sets SIGReg and analytic MMD are 1.6±1.31.6\pm1.3 and 1.3±1.21.3\pm1.2 points above Wristband (six paired differences each; pooled means 31.3%, 31.0% and 29.7%). We no longer describe this as within seed variation: the abstract, Sec. 6.4 and the conclusion say Wristband's probe accuracy is about 1.5 points lower at the full training budget. Exact assignment is higher still (34.3%) and clearly non-Gaussian.

It depends on the training budget. In the 50-epoch screens, now on three seeds, Wristband's selected coefficient reaches 32.2±0.5%32.2\pm0.5\% validation accuracy, level with VICReg (32.2±0.1%32.2\pm0.1\%) and above the best analytic-MMD and SIGReg settings (29.2%, 29.5%). At 200 epochs Wristband and VICReg are about two points lower than in the screens while MMD and SIGReg are one to three points higher. Coefficients were selected at the short budget and not re-tuned at the long one, for every method, so we report this as an observation, not as an explanation.

Why Gaussianity need not help a probe. The probe reads the 512-d backbone; the regularizer acts on the 256-d projector. A linear probe never consumes the projector's distribution.

Where the Gaussian law is itself used:

  • Block resampling (Table 3). In a synthetic two-factor model with a known conditional, one latent block is replaced by Gaussian noise and the predicted conditional law is compared with the truth. Without a Gaussian regularizer the conditional is wrong (W1=0.251W_1=0.251). Every Gaussian regularizer repairs it; Wristband has the lowest conditional error (0.062) and the lowest task error (0.035) among them, ahead of WAE-MMD (0.067 / 0.048), MMD (0.072 / 0.056), a VAE-style encoder (0.075 / 0.056) and sliced W2W_2 (0.097 / 0.063). These are retained aggregates without per-run records, so the ordering is descriptive.
  • Latent edits (Shapes3D). Wristband and sliced Wasserstein preserve edit accuracy (91.5% / 91.7%) where MMD at its selected weight does not (61.2%).
  • A counter-example we now report: MNIST completion (App. Z.8). We added MMD, SIGReg, conditional-VAE, unregularized and regression arms to the MNIST example. Wristband shows no advantage there: at matched use of the sampled residual, MMD and SIGReg give the same reconstruction, conditional error and completion accuracy (the baseline rows at those weights are single runs). What the example does show is that some Gaussian regularizer on the joint code is needed (the unregularized and CVAE arms failed on two of five seeds).
  • A second counter-example: the mixture autoencoder (App. Z.9). At another reviewer's request we added analytic-MMD, SIGReg, VAE and unregularized arms to the 8-d mixture autoencoder, each over a weight grid on five seeds, and scored samples decoded from the Gaussian prior against held-out data. Analytic MMD and SIGReg do better than Wristband: energy distance 0.041 for each against 0.113 (on all five seeds), with lower reconstruction error (0.049 and 0.062 against 0.089). Wristband's codes have the better radius law and covariance; the baselines' have lower MMD², sliced and coordinate errors.

So where is the clear advantage? In the Gaussianity of the 256-d CIFAR-100 projector (next paragraph), and, on retained aggregates only, in the synthetic resampling table. In the two low-dimensional autoencoders that we have now tested against full baselines there is none, and on the mixture the baselines are ahead. The abstract and the practical guidance now say this.

How much more Gaussian is the CIFAR projector? Table 4 now has five diagnostics with a Gaussian-null row:

Projector diagnostic (1,024 test images, d=256d=256)WristbandSIGRegAnalytic MMDiid Gaussian samples
MMD² U (10−310^{-3})0.090.370.560.00±0.010.00\pm0.01
Effective rank (max 256)223.5148.6121.7225.8
Gaussian-fit W22/dW_2^2/d0.0710.3250.4750.066
Sliced W22/2W_2^2/2 (10−310^{-3}), full test split0.561.672.030.34
Radial KS0.080.240.210.03

(The sliced row is the full-test-split value from App. Z.6. Table 4 lists the n=1024n=1024 value, where the statistic is dominated by its sampling floor: 3.32 / 4.48 / 4.81 against 3.07 for Gaussian samples.)

At their selected coefficients the SIGReg and MMD projectors use roughly half of the available directions. Wristband's radius law is about three times closer than theirs but not at the sampling floor.

Our answer, now in the Discussion: the evidence favours Wristband only where a high-dimensional projector should itself be Gaussian. For low-dimensional autoencoders and for probe accuracy, analytic MMD and SIGReg did as well or better and should be tried first.


W2 / Q2. A low approximate loss does not indicate Gaussian matching. Can you provide a diagnostic for when the approximation is insufficient?

Yes; this prompted a new paragraph in Sec. 5 and App. Z.2.

A priori. The spectral path is characteristic exactly for "TT uniform and E[U∣T]=0\mathbb E[U\mid T]=0" (App. L). Anything beyond that (higher-order angular structure, sign-symmetric radius–direction coupling) is invisible to it by construction.

In practice: a pairwise audit. Periodically evaluate the pairwise repulsion on a subsample of n≤1024n\le 1024 points with a sharp angular kernel, as a zz-score against Gaussian batches of the same shape, next to the spectral statistic's own zz-score. Cost is O(n2d)O(n^2 d) independent of the batch size. A law the truncated features cannot see shows zsp≈0z_{\rm sp}\approx 0 with zpair≫0z_{\rm pair}\gg 0. Results (N=1024N=1024, five seeds):

Cloudzpairz_{\rm pair}zspz_{\rm sp}
iid Gaussian control, d=10d=10−0.2±0.7-0.2\pm0.71.1±1.51.1\pm1.5
Hypercube-vertex counterexample (Sec. 5), d=10d=1015.7±1.315.7\pm1.30.2±1.00.2\pm1.0
X-10D after spectral training (Table 1)38.2±1.538.2\pm1.5−6.4±0.0-6.4\pm0.0
X-10D after pairwise training2.8±0.72.8\pm0.7−5.5±0.3-5.5\pm0.3
RAC-10D after spectral training0.2±0.10.2\pm0.1−6.3±0.1-6.3\pm0.1

The audit flags the case in Table 1 where spectral training is much worse than pairwise (X-10D: sliced error 7.32 vs. 2.13) and does not flag RAC-10D, where the two agree (2.36 vs. 2.03).

Its limits, reported in the same appendix. The audit is a finite-sample statistic. At N=1024N=1024 it does not detect the vertex law at d=128d=128 (whose fourth angular moment differs from the uniform sphere's by about 1%), nor the untrained RAC coupling.


W3 / Q3. Runtime and memory versus SIGReg and linear-cost MMD.

You were right that the submission did not establish a general efficiency advantage, and the new measurements confirm it for linear-cost MMD.

Memory (Table 2, B300, d=128d=128). Peak GPU memory at N=8192N=8192: spectral Wristband 84–88 MiB, pairwise Wristband 2.6 GiB, analytic MMD 2.1 GiB, SIGReg 3.3 GiB (1,024 directions; reducible by chunking).

Linear-cost MMD (new, App. Z.7). We reran the whole panel in one session on a second GPU (RTX 5090), adding the linear-time MMD estimator of Gretton et al. (2012) and a random-Fourier-feature MMD with the same kernels. Forward+backward, mean over three input seeds:

Objectivems, N=1024N{=}1024ms, N=8192N{=}8192MiB, N=1024N{=}1024MiB, N=8192N{=}8192
Wristband pairwise, full3.7820.011062636
Wristband pairwise, repulsion only1.6518.381052632
Wristband spectral, full3.383.766788
Wristband spectral, repulsion only1.341.346784
Analytic MMD1.9121.13982180
SIGReg (1,024 fresh directions)1.3410.764733333
Linear-pair MMD (fresh reference)1.671.75216
Random-feature MMD (5 × 1,024 features)0.673.171871031

What this supports:

  • Spectral Wristband removes the quadratic cost of its own pairwise path. The speedup at N=8192N=8192 is hardware-dependent: 2.97×2.97\times (repulsion) and 1.86×1.86\times (full loss) on the B300, 13.8×13.8\times and 5.3×5.3\times on the RTX 5090. Memory is identical on both GPUs.
  • Linear-pair MMD is as cheap: 0.3–0.5 ms slower than spectral repulsion and with less memory at every NN. The full spectral loss is about twice as slow as linear-pair MMD because of its auxiliaries.
  • At N=1024N=1024 everything takes 1–4 ms and there is no speed reason to prefer the spectral path. Inside a ResNet-18 training step all of this is a 1–3% effect (App. V).

The revised text therefore claims a reduction of Wristband's own quadratic cost and states that spectral Wristband is in the same cost class as linear-cost MMD, not cheaper. We did not compare the statistical efficiency of these estimators; the linear-pair statistic uses N/2N/2 kernel evaluations and has a larger variance than the quadratic ones. The second-GPU timings are reported in their own table and are not merged with the B300 numbers.


Q4a. How well does the core loss perform without radial and moment penalties?

It depends on whether points are free or produced by an encoder (ablation paragraph in Sec. 6.2):

  • Free points (Table 1, five new seeds). Repulsion-only is close to the full loss in sliced error on three of four problems (X-2D 2.04 vs. 1.84; RAC-10D 2.59 vs. 2.03; RAC-128D 2.36 vs. 2.37) and twice as large on X-10D (4.14 vs. 2.13). Its radial KS is larger on three of four.
  • Learned embeddings (CIFAR-100, earlier five-seed protocol, App. N). Repulsion-only at its selected coefficient gives a higher probe accuracy (33.0% vs. 30.8%) but leaves the projector essentially un-Gaussianized (MMD² 0.376, about the unregularized value, vs. 9×10−59\times10^{-5}).

So the auxiliaries are not needed for the population characterization, are optional for free points, and should be kept on for learned codes when Gaussianity is the goal.

Q4b. Can one default configuration transfer across datasets, dimensions and batch sizes?

For the kernel, largely yes; for the outer weight λ\lambda, less than we first claimed. The same kernel and weights were used for the mixture autoencoder (d=8d=8, N=1024N=1024), MNIST (d=21d=21, N=1024N=1024) and CIFAR-100 (d=256d=256, N=512N=512): pairwise path, β=8\beta=8, α2=1/12\alpha^2=1/12, wrad=0.1w_{\rm rad}=0.1, wmom=1w_{\rm mom}=1, outer weight λ=0.1\lambda=0.1. Each component is zz-scored against Gaussian batches of the training shape, which puts λ\lambda on a common scale across NN and dd. On CIFAR-100, over three seeds and a 20×20\times range λ∈[0.01,0.2]\lambda\in[0.01,0.2], projector MMD² stays within 1.11.1–2.0×10−42.0\times10^{-4} and validation accuracy within 29.5–32.2% (Fig. 1, App. Z.3).

Three caveats. In the mixture autoencoder, where we have now run a weight grid, validation selects λ=0.01\lambda=0.01 and not 0.10.1: larger weights raise reconstruction error (0.047 to 0.199 over four decades) without improving held-out Gaussianity, and generation at 0.1 is twice as far from the data as at 0.01 (App. Z.9). The guidance therefore says to tune λ\lambda per task. The exceptions are listed in a table (App. Z.5): the direct point-cloud tests select between this kernel and a sharper one per problem, and two earlier studies (Shapes3D, conditional resampling) used other settings. And, as noted under Q1, λ\lambda was selected for probe accuracy at 50 epochs and was not re-tuned for 200. A practical-guidance paragraph in the Discussion summarizes: pairwise path by default; spectral path only for large batches, with auxiliaries on and the pairwise audit running.

Comments on the paper

3 comments

General response to all reviewers

Summary and request. We ask each reviewer to reconsider their score and raise it for the revised paper. Three points:

  1. What is new. The hyperspherical uniformity loss of Wang & Isola (2020) discards the radius, so it cannot target a Gaussian. We prove that a law is standard Gaussian exactly when its direction is uniform, its CDF-transformed radius is uniform, and the two are independent (Theorem 1), and that a reflected radial kernel makes the Gaussian the unique minimizer of a repulsive energy equal to an MMD² (Theorem 2). Without the reflection the minimizer is not Gaussian. The idea is simple to state, but the closest prior method, Radial-VCReg (Kuang et al., 2025), constrains the radius marginal and second moments, not the joint law of radius and direction, and leaves non-Gaussian structure that Wristband removes (Tables 7–8).
  2. What we did. We ran every experiment the reviews asked for (285 new training runs) and report every result, including those that go against us.
  3. What the evidence supports. On CIFAR-100, Wristband's projector codes are closer to Gaussian than analytic MMD or SIGReg reach at any coefficient we screened. It does not improve probe accuracy and shows no advantage in two low-dimensional autoencoders, and the abstract now says so.

We think a proved construction with an evaluation that states its own limits is worth publishing, and we ask you to judge the revision on that basis, and consider the method is for "learning representations" which is directly aligned with name of the conference.


We thank the three reviewers. The reviews agree on what is solid (the direction–radius reduction, the reflected kernel, the explicit spectral blind spot) and on what was missing: a clear statement of novelty relative to hyperspherical uniformity, a clear statement of which guarantee covers which code path, evidence of practical value beyond "lower MMD", and a readable account of Eq. (7), Eqs. (13)–(15) and the protocols. The revised PDF addresses each point; new or rewritten text is in blue. The main text is 10 pages and now carries the full appendices in the same PDF.

We ran every experiment the reviews asked for. Some results strengthen the paper and some limit its claims, one of them substantially; we report all of them, and the abstract, the practical guidance and the conclusion have been rewritten to match.

We thank the three reviewers. The reviews agree on what is solid (the direction–radius reduction, the reflected kernel, the explicit spectral blind spot) and on what was missing: a clear statement of novelty relative to hyperspherical uniformity, a clear statement of which guarantee covers which code path, evidence of practical value beyond "lower MMD", and a readable account of Eq. (7), Eqs. (13)–(15) and the protocols. The revised PDF addresses each point; new or rewritten text is in blue. The main text is 10 pages and now carries the full appendices in the same PDF.

We ran every experiment the reviews asked for. Some results strengthen the paper and some limit its claims, one of them substantially; we report all of them, and the abstract, the practical guidance and the conclusion have been rewritten to match.

New experiments (App. Z)

RequestWhat we ranOutcome
Sliced-Wasserstein and radial KS for CIFAR-100 (vW3p)Reran Wristband, SIGReg, analytic MMD and VICReg at the published settings (3 seeds) with projector features saved (App. Z.6)Wristband is closest to Gaussian on both: sliced error 1.6× the sampling floor on the full test split vs. 4.9× (SIGReg) and 5.9× (MMD); radial KS 0.08 vs. 0.24 and 0.21. Its radius law is closer, not exact (floor 0.03).
Gaussianity–utility curves over regularization strength (vW3p, 3UmC)Every screened coefficient of the four methods, now on three seeds (Fig. 1, App. Z.3)Over a 20× range of its coefficient Wristband stays at MMD² 1–2×10⁻⁴. No screened coefficient of analytic MMD or SIGReg reaches that at any accuracy.
Runtime and memory vs. SIGReg and linear-cost MMD (z1Lu)Peak memory added to Table 2; whole timing panel rerun on a second GPU with linear-pair and random-feature MMD (App. Z.7)Spectral Wristband reduces its own quadratic cost (84 MiB vs. 2.6 GiB at N=8192; 3–14× faster depending on GPU). It is not cheaper than linear-pair MMD.
MNIST baselines and independent metrics (3UmC, vW3p)Analytic MMD, SIGReg, conditional VAE, no regularizer and a regression control, 5 seeds each, with diversity, independence and completion-accuracy metrics (App. Z.8)No advantage for Wristband. At matched use of the residual, MMD and SIGReg give the same completion quality. A regularizer on the joint code is needed (unregularized arms fail on 2 of 5 seeds), but the example does not favour ours.
Mixture-autoencoder baselines and independent metrics (3UmC)Analytic MMD, SIGReg, VAE and no regularizer added to the 8-d mixture autoencoder; five weights per method, with and without the coupling postprocessor, 5 seeds (185 runs); weights chosen on validation data by a rule fixed in advance (App. Z.9)Analytic MMD and SIGReg do better than Wristband. Samples decoded from the Gaussian prior are closer to held-out data (energy distance 0.041 for each vs. 0.113, on all five seeds) at lower reconstruction error (0.049 and 0.062 vs. 0.089). Code diagnostics are split: Wristband has the best radius law and covariance, the baselines the lower MMD², sliced and coordinate errors. The coupling postprocessor hurts every regularizer.
Diagnostic for an insufficient spectral approximation (z1Lu)"Pairwise audit" tested on the counterexample and on saved clouds (Sec. 5, App. Z.2)Flags the cases where spectral training is worst; has stated finite-sample limits.
Logarithm ablation (vW3p)Logged vs. unlogged repulsion on the direct benchmark, 80 runs (App. Z.4)The logarithm is a scale convention; results differ little and without a consistent sign.

What the new evidence changed in our claims

  1. Probe accuracy. The reruns reproduce the published runs, and across both sets SIGReg and analytic MMD are 1.6 ± 1.3 and 1.3 ± 1.2 points above Wristband. We no longer describe this gap as within seed variation; the abstract says Wristband's probe accuracy is about 1.5 points lower. (At the 50-epoch screening budget the order is reversed: Wristband 32.2 ± 0.5%, best MMD 29.2%, best SIGReg 29.5%; coefficients were not re-tuned for 200 epochs.)
  2. Efficiency. The claim is now restricted to a reduction of Wristband's own pairwise cost.
  3. MNIST. Reported as a comparison in which Wristband shows no advantage.
  4. Mixture autoencoder. Reported as a comparison in which analytic MMD and SIGReg reconstruct and generate better. We no longer recommend the coupling postprocessor. The guidance now says to tune λ\lambda per task (validation selects 0.01 here, not the 0.1 used elsewhere) and to try analytic MMD and SIGReg first for low-dimensional autoencoders.
  5. Scope. The abstract now says the experiments support one use: making a high-dimensional self-supervised projector more Gaussian, at some cost in probe accuracy. It no longer presents Wristband as the regularizer to prefer wherever the latent law is consumed.

Other changes

  • Novelty (Sec. 2). What is shared with Wang & Isola (the angular factor, by design) and what is new (the equivalence theorem, the reflected radial kernel without which the uniform law is not the energy minimizer, and the product-space MMD identity).
  • Theory vs. implementation (Sec. 5). A paragraph states which guarantee applies to which path. All learned-representation experiments use the pairwise path, which keeps every angular degree.
  • Clarity. Eq. (7) explained term by term; the logarithm motivated; Eqs. (13)–(15) derived in the main text and in App. Z.1; a "Protocols in brief" paragraph opens Sec. 6; a motivation paragraph relative to VAEs, flows and diffusion opens Sec. 1.
  • Conditional resampling with baselines (Table 3) moved from the supplement into the main text.
  • Formalization claim. The Lean development is now described as a partial formalization (it imports several facts as axioms). The theorems rest on the analytic proofs.

Provenance. The Gaussian-fit, effective-rank and memory numbers come from per-run records already in the supplement. The audit and the logarithm ablation are CPU computations on the frozen direct-benchmark code. The new GPU runs used rented RTX 5090 machines with the software versions of the published runs but a different GPU model from the B300 used originally; configurations and initial weights of the CIFAR reruns are identical to the published ones, and an unmodified rerun had to reproduce a published result within a tolerance fixed in advance before any other job started. All prescribed runs completed and none was excluded. For the mixture autoencoder the corresponding reproduction check was missed on one of its two criteria: the rerun of the submitted configuration gave reconstruction MSE 0.204 against a window of 0.21–0.27 set around the submitted 0.241±0.0060.241\pm0.006 (five seeds: 0.213±0.0080.213\pm0.008). The submitted value was transcribed from an experiment summary whose runs are not retained; the paper now quotes the reruns and says so. The generation statistics for that experiment were fixed in advance and proved coarse (a Gaussian with the data covariance scores better than every trained arm), so we added a precision/recall/coverage check on the two seeds whose samples are archived; it is labelled post hoc and gives the same ordering. Timings on the second GPU are reported separately and are not merged with the B300 table. The MNIST protocol had to be partly reconstructed from a surviving example script; every such choice is recorded and will be added to the supplement with the run records.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.