acceptodds
Under review as a conference paper at ICLR 2027

The Scale–Shape Effect on Sliced-Gaussian Regularizer-based Latent Prediction Methods

Abstract

Self-supervised regularizers like SIGReg match embedding distributions to isotropic Gaussians by independently penalizing deviations in scale and geometric configuration. However, the substitution rate between these regularization penalties, defined as the scale–shape curvature (), is fixed exclusively by the dimensionality of the latent space (called width here). Neither of the theoretical regularizer-objective's kernel (over infinite data and random slice directions), the bandwidth, the slice count, nor the slice dimension can alter this property, and no stationary kernel improves upon the conditioning achieved by the standard baseline. In this work, we propose , a single weight that rescales the shape component of the objective’s kernel, and with it , without adding a new penalty term. Consequently, the minimizer remains invariant, the kernel preserves positive definiteness for all , and the configuration exactly recovers the standard objective (for e.g. SIGReg). Every result reported here is obtained on the analytic infinite-slice objective, the closed-form limit of SIGReg; has not been applied to the stochastic -slice estimator itself. Modulating adjusts the rank of the regularized layer and we show empirical results across two modalities, time-series and image data. This rank modulation is subject to two explicit operational assumptions: sufficient headroom below the rank ceiling, and a non-monotonic layer response, such as an un-regularized backbone preceding a projection head where the regularizer is applied. In downstream evaluations, shifts performance bi-directionally depending on the specific task and readout mechanism. We therefore present this parameterized axis of variation rather than prescribing an optimal hyperparameter configuration.

Primary area:
learning theory
Keywords:
self-supervised learning, representation learning, covariance regularization, effective rank, kernel methods, dimensional collapse

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PYsMOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the relationship between scale and shape regularization in sliced-Gaussian latent prediction methods.

Critical strengths

This work explores an interesting theoretical property of distribution-matching regularizers and proposes a mathematically motivated mechanism for controlling representation geometry. The experiments cover both image and time-series modalities and provide useful observations about the dependence of rank and downstream performance on the regularization configuration.

Critical weaknesses

My main concern is that the proposed method is insufficiently specified and remains unconvincing. The central construction relies on quantities whose explicit definitions are not provided in the main text. And the relationship between the exact infinite-slice objective and the approximated kernel used in implementation is unclear.

Second, the theoretical curvature analysis does not establish a convincing connection to the learned representations. The observed accuracy changes are inconsistent.

Questions for authors
  1. The authors need to provide a complete and reproducible formulation of the proposed regularizer and clarify exactly which objective is implemented?

2, What meaningful understanding or improvement does the proposed scale–shape control enable beyond adjusting representation rank?

  1. How does the local curvature analysis explain the observed behavior of trained networks?
4 comments

Thank you for pressing on the specification; the appendix that should have carried the implemented objective was an unresolved reference in the submitted version, which we think explains much of the difficulty you report. It is now updated, and we have added experiments on what η does beyond rank and on how far the curvature analysis reaches on trained networks. We look forward to your reply!

  • The revised manuscript:
    • App. E states the implemented objective as an algorithm box (kernel, quadrature, shape-sector projection, strength calibration) with the closed forms of h and c; Sec. 2.1 now says what every η run minimises.
    • Secs. 2.1–2.2 rewritten to introduce ρ, η, K, τ and the three objects (sliced estimator, exact limit, implemented objective).
    • Sec. 4.3 now shows the added width experiment (new Fig. N), a label-budget ladder and a VISReg comparison; Sec. 5.1 lists what η gives beyond rank; Sec. 5.2 shows a limitation on the operating point; App. D (strength control) and App. F (covariance defects on trained networks) are new.

W1 / Q1. "The central construction relies on quantities whose explicit definitions are not provided … the relationship between the exact infinite-slice objective and the approximated kernel used in implementation is unclear. Provide a complete and reproducible formulation and clarify exactly which objective is implemented."

  • Every η run minimises L(Z) = ρ · λ · D_η(Z) with D_η = D − (1 − η) ‖Π_sh Δµ‖²_H (Eq. 5),described further in App. E.
  • Eq. 7 (τ = (K + 2)/(2η)) is a property of D_η at Σ = I_K; the empirical claims rest only on the rank and readout responses to η, as Sec. 3.2 now states.

Q2. "What meaningful understanding or improvement does the proposed scale–shape control enable beyond adjusting representation rank?"

  • Width-invariance: with η fixed, the response (few-shot accuracy and backbone rank) is the same at K = 256 / 1024 / 4096; with τ fixed (η ∝ K + 2) it is not. The quantity the baseline silently changes the shape weight (and η undoes it) when a projector is widened.
  • This is specifically used for low-dimensional narrow projectors for cases like autoencoder-based data compression.
  • The gain is a few-label gain, not a rank gain: doubling the strength at η = 1 raises the projector rank slightly more than η = 2 does (+29 vs +25, 5 seeds) and shrinks the shape defect as much, but gives none of the few-label gain (+0.06 [−0.04, +0.16] vs +0.80 [+0.67, +0.93] at 10 labels per class). Rank can be moved without the benefit; η moves both the rank and the gain on the image datasets. This combined with the flat time-series scores further motivates that the η knob always needs to be empirically verified and tuned per dataset, and is also an added interpretability tool to understand the optimal effective rank for the downstream task.
  • A label-free dial: the η ordering of backbone rank at epoch 40 of 80 equals the ordering at epoch 80 and equals the ordering of the final 10-shot readout.
  • We also now show that our results hold for VISReg as well apart from SIGReg, matching the scores on ResNET18 across 3 seeds at a similar shape tuning.

W2 / Q3. "The theoretical curvature analysis does not establish a convincing connection to the learned representations … How does the local curvature analysis explain the observed behavior of trained networks?"

  • The analysis predicts that raising η penalises the traceless (shape) part of the covariance of the regularized layer relative to its trace (scale) part.
Kshape defect relT, η = 1 → η = 20
2561.97 → 0.30
10244.31 → 1.68
40969.61 → 4.31
  • The shape defect decreases at every width (new App. F), and so the covariance of the regularized layer is closer to isotropic, which is the original objective of methods like SIGReg.
  • Sec. 3.2 now presents the curvature result as the mechanism for the direction of the effect and the rank/readout responses as the empirical content.

W3. "Difficult to follow despite careful reading."

  • We agree the initial version was unclear in places and we have made a significant effort to add relevant information where missing (the objective definitions etc.). New experiments were added (expanded the analysis to VISReg, added an experiment looking at the effect at different latent widths with narrow or broader projectors), and minor inconsistencies resolved now.
  • Secs. 2.1–2.2 rewritten to define ρ, η, K, τ; the two unresolved appendix references are fixed; the implementation is further described in one appendix (App. E); the image results in one place (Sec. 4.3) with the width figure first.
  • We thank you for your comment and hope the readability is now improved. We would love to hear your thoughts on the updated manuscript!

Dear reviewer,

We have updated the manuscript for added clarity and addressed your initial concerns in the response above, and the VISReg and the K sweep results we added supports our hypothesis even better, thanks to your comments.

We have also made a top-level general comment addressing all the improvements made in an effort to make our responses easier to parse.

We would be grateful to know your thoughts on the updates and responses before the close of the review period soon.

We thank you once again for your time and contributions through this review process!

Best, Authors

Dear Reviewer,

We have made a significant effort to improve the clarity of the manuscript and also added experiments and ablations that strengthen the utility of our method for anyone using sliced regularizers.

We think our method is a useful tool for other researchers, not just in terms of potentially improving their performance on a downstream task but also as an interpretability tool that allows a rank-sweep, making it clear if their algorithm has an optimal effective rank region that existing methods like SIGReg (and now VISReg, recently added and supportive of our hypothesis) are by default blind to.

We have received one score increase so far and we would be excited to hear your thoughts and your final score on the updated manuscript before the rebuttal closes.

Thank you once again for your time and contributions to our work!

Thanks for your work on rebuttal and revision. My concern has been mostly addressed. I would keep my current rating.

Reviewer_8mCXOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors show that the SIGReg family has an implicit relationship between scale and shape curvature determined by the local dimensionality. They introduce a parameter, eta, that adjusts the target ratio of scale to shape curvature: increasing eta typically increases the rank of the regularized representation and seems to shift the ballance. While adjusting eta does not reliably “improve” models in obvious ways (noted as future work), the experiments support the central claim that the geometry can be systematically manipulated by messing around with eta, giving an extra knob to play with. It is also a nice example of how different objectives can impose geometric preferences on representations in ways that are not obvious but are are likely important to understand.

Critical strengths

The authors further expose and discuss this interesting scale–shape curvature property -- which is interesting.

They introduce eta, which is mathematically interesting and seems to do the job. The experiments seem well done with honest results (e.g. no exaggerated claims, maybe even undersold).

Critical weaknesses

Downstream benefits are modest and inconsistent. Basically I think it is cool but I am not sure what to do with it.

Questions for authors

I dont have any good questions but I am curious if the authors could come up with a contrived setting where eta is then the solution?

2 comments

Thank you for the fair reading and for the one targeted question, which we found the most useful of the round: it made us find a setting in our new data where η does something the baseline's own knobs cannot. We look forward to your reply!

  • The revised manuscript:
    • Sec. 4.3: added a width experiment (three projector widths, new Fig. N), a label-budget ladder table and a VISReg comparison table.
    • Sec. 5.1 now gives the operational setting below, with its caveats, in place of the previous two-purpose paragraph.
    • App. D adds the strength-matched control (ρ = 15 at η = 1) that the setting relies on.

W1. "Downstream benefits are modest and inconsistent. Basically I think it is cool but I am not sure what to do with it."

  • Agreed they are modest, and we also feel that this is a cool finding as well as a useful tuning knob that other researchers using such slicing based methods should try tuning for their specific task to get rank-based interpretability results and in some cases better downstream performance.
  • Images: η = 2 beats η = 1 per class and at full data, 5/5 seeds, for ResNet-18 and ResNet-34; largest at 5 labels per class (+0.9 to +1.0 points), smallest at full data (+0.5 to +0.75). At the best η (10) the 10-shot gain is +2.2 to +2.6 at every projector width, 3/3 seeds.
  • We also added VISReg results to the paper and they match the SIGReg results at similar shape tuning.

Q1. "Could the authors come up with a contrived setting where eta is then the solution?"

  • Standard objective (η = 1), a narrow projector, few labels, and say you want more usable rank in the backbone for a richer representation. The baseline offers two knobs, width and strength but still they leave the effective rank fairly low, and this is not good given the width is already low in this case with the narrow projector. Neither of the standard knobs can change the effective rank; η does. CIFAR-100 / ResNet-18 / 80 epochs, 10-shot ridge on frozen backbone features:
intervention from the η = 1, K = 256 baseline10-shot Δbackbone rank Δ
widen the projector 16× (K = 4096, η = 1), 3 seeds+0.31 [+0.14, +0.47]+1.5
double the strength (ρ = 15, η = 1; paper's 5 seeds, K = 1024)+0.06 [−0.04, +0.16]+7 (projector +29)
η = 2 on the same 5 seeds, for reference+0.80 [+0.67, +0.93]+6 (projector +25)
η = 10 at the same narrow width (K = 256), 3 seeds+2.20 [+2.05, +2.36]+25.5
  • The narrow projector at η = 10 beats the 16× wider projector at η = 1 by +1.89 [+1.80, +1.99], 3/3 seeds.
  • While the gain is still small, the effect being more profound at smaller widths adds an interpretability result that could be used down the line to achieve better representations with larger dimensionality reduction - which is specially useful in cases like autoencoder-based lossy data compression.
  • We would love to hear if you have more use cases in mind that we can add to improve the manuscript!

I thank the authors for their reply. I raised my score to a 4. I think a 3 was probably fine but I thought I would throw this project a bone. Good luck.

Reviewer_Zt3nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new hyperparameter η\eta to rescale the shape component of the objective’s kernel, without adding a new penalty term. Consequently, the regularizer’s minimizer remains unchanged, the kernel stays positive definite for all η>0\eta>0, and setting η=1\eta=1 exactly recovers the baseline objective (e.g., SIGReg). The paper also did downstream evaluations where η shifts performance bi-directionally depending on the specific task and readout mechanism

Critical strengths
  • The paper provides an interpretable way (by proposing a new paramter η\eta to adjust representation geomery while reserving the regularization target under Gaussian distribution.

  • The theoretical analysis for \taoK=(K+2)/2\tao_{K}=(K+2)/2 provides interesting insight of how latent width changes the relative sensitivity to scale and shape, motivating the intervention.

*The experiments scope are broad. For example, the authors provide experiment results across images and time series to support substantial rank modulation, although saturation and nonmonotonic cases limit predictability.

Critical weaknesses
  • The writing needs to be polished. For example I see many missing citations. (line 80,.103).

  • Limited improvment shown in experiments. More specifically, Table 2 reports approximately +0.80, +0.41, and +0.85 percentage points in the three image settings, with positive paired changes across their five seeds. These results deserve credit.

However, the paper itself reports no successful primary time-series outcomes under its formal assessment. The ETTh1 forecasting endpoint also fails its mean-predictor check. Some secondary readouts improve, but their direction depends on the task and evaluation procedure. Overall, the experiment part does not establish a generally useful replacement for the baseline.

Meanwhile, the direct comparison between alternatives like VISReg is missing. The discussion of VISReg is extensive, but there is no corresponding empirical comparison. The same applies to other relevant regularizers.

  • There is a discrepancy between theory and corresponding empirical results. For example, If the motivating issue is τK∝K\tau_K\propto K, a particularly informative experiment would compare:
  • fixed η\eta across widths; and
  • η\eta scaled with K+2K+2, keeping the theoretical curvature ratio constant.
Questions for authors

See weakness.

3 comments

Thank you for the targeted weaknesses and in particular for the width experiment you proposed; we ran that design and it is a clear result strengthening the narrative (we found that at lower widths the η effect is more pronounced - see results below). The two "missing citations" were unresolved appendix references and have been fixed. We look forward to your reply!

  • The revised manuscript:
    • The two App. references (lines 80, 104) now resolved: App. D (strength-matched control, ρ = 15) and App. E (closed form of h and c; the implemented objective as an algorithm box). "Figures 3 gives" (line 385) fixed.
    • Sec. 4.3 gains the width experiment (new Fig. N: 10-shot and backbone-rank response against η and against τ at three projector widths), a label-budget ladder table, and the VISReg comparison table.
    • The VISReg paragraph in Sec. 1 now points to the empirical comparison.
    • Sec. 5.1 (operational utility) rewritten around the width result; Sec. 5.2 has one added limitation (the curvature analysis predicts direction, not the operating point).

W1. "The writing needs to be polished. For example I see many missing citations (line 80, 103)."

  • Lines 80 and 104 are two unresolved appendix cross-references for the App.; both now fixed.

W2. "Limited improvement … no successful primary time-series outcomes … ETTh1 fails its mean-predictor check … does not establish a generally useful replacement for the baseline."

  • Agreed on time-series: the primary task scores do not move under the scale-shape tuning correction, and the ETTh1 forecast is below a mean predictor for every arm including the unregularised one (App. B), so we report it as uninterpretable to further emphasize that this parameter needs to be validated empirically for each use case.
  • We do not claim a replacement for the baseline; the abstract and Sec. 5 present η as an axis of variation and we view this more as an interpretability tool to understand if at all you can extract more performance or if the objective defaults are already at optimal effective rank. This parameter we present can also be helpful in other studies that require explicit modifications of the effective rank for downstream model analyses. These have to be repeated per dataset to get a better understanding of the task.
  • On images the gain is small but consistent specifically at few labels and we posit that other researchers might benefit by tuning this parameter on their tasks.

W3. "The direct comparison between alternatives like VISReg is missing … The same applies to other relevant regularizers."

  • We agree that this was an important missing experiment and we have added it to the paper now and we see the positioning has gotten stronger with it, thank you for pointing it out.
  • VISReg (Wu et al. 2026, Alg. 1, 4096 slices per step) run in our pipeline at K = 1024 with everything else identical (encoder, predictor loss, schedule; ρ = 7.5 calibrated once on VISReg's default):
arm (scale, shape, centre)10-shot Δ vs SIGReg η = 1
VISReg default (1, 1, 1)−0.32 [−0.95, +0.30]
VISReg shape-leaning (0.5, 2, 0.5)+0.95 [+0.56, +1.34]
SIGReg η = 2 (reference)+0.97 [+0.15, +1.79]
  • VISReg default coincides with SIGReg η = 1; VISReg shape-leaning (shape η = 2) matches that of SIGReg η = 2 across 3 seeds.

W4. "If the motivating issue is τ ∝ K, a particularly informative experiment would compare fixed η across widths; and η scaled with (K + 2)."

  • Run as proposed: K = 256 / 1024 / 4096, η ∈ {2, 5, 10, 20} at every width over 3 seeds. CIFAR-100, ResNet-18, 80 epochs, ρ = 7.5 (the paper's phase-Q protocol). New Fig. N added.
  • Fixed η across widths (10-shot Δ vs η = 1, all 3/3 seeds):
η251020
K = 256+1.18+2.18+2.20+1.90
K = 1024+0.97+2.15+2.37+2.24
K = 4096+0.71+1.98+2.58+2.57
  • At a given η, smaller K widths show larger improvement deltas.
  • Fixed τ across widths (η scaled with K + 2):
τ held fixedK = 256K = 1024K = 4096
τ ≈ 102: 10-shot Δ+0.26+2.15+2.57
τ ≈ 102: backbone rank Δ+2.6+18.4+47.0
τ ≈ 256: 10-shot Δ−1.02+0.97+2.47
  • So the same η produces the same response at every width, the same τ does not. η is the coordinate in which the response is width-invariant; τ is what the baseline silently changes when a projector is widened. This is now added as the first result of Sec. 4.3.

Dear reviewer,

We have updated the manuscript for added clarity and addressed your initial concerns in the response above, and the VISReg and the K sweep in fact added very strong results supporting our hypothesis much better, thanks to your comments.

We have also made a top-level general comment addressing all the improvements made in an effort to make our responses easier to parse.

We would be grateful to know your thoughts on the updates and responses before the close of the review period soon.

We thank you once again for your time and contributions through this review process!

Best, Authors

Dear Reviewer,

We have made a significant effort to strengthen our ablations and experiments and we are grateful your comments added a very impressive result.

We think our method is a useful tool for other researchers using sliced regularization methods (which are becoming more popular in recent times), not just in terms of potentially improving their performance on a downstream task but also as an interpretability tool that allows a rank-sweep, making it clear if their algorithm has an optimal effective rank region that existing methods like SIGReg (and now VISReg, recently added and supportive of our hypothesis) are by default blind to.

We have received one score increase so far and we would be excited to hear your thoughts and your final score on the updated manuscript before the rebuttal closes.

Thank you once again for your time and contributions to our work!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of all changes made based on the reviews

We thank all three reviewers for their questions; the manuscript is clearer and better supported because of them. Our detailed replies are posted under each review, and we briefly summarise the main changes here.

  • Specification (PYsM). The two unresolved appendix references are fixed. The new App. E states the implemented objective as an algorithm box, with the closed forms of h and c, and Secs. 2.1–2.2 now define ρ, η, K and τ and say which objective every η run minimises.
  • Width experiment (Zt3n). We ran the experiment Zt3n proposed: K = 256 / 1024 / 4096 at fixed η and at fixed τ, 3 seeds each. At fixed η the response is the same at every width; at fixed τ it is not. This is now the first result of Sec. 4.3, with a new figure.
  • VISReg comparison (Zt3n, PYsM). VISReg is now run in our pipeline. At its default it coincides with SIGReg η = 1, and its shape-leaning setting matches SIGReg η = 2.
  • What η gives beyond rank (8mCX, PYsM). A strength-matched control (new App. D) raises rank about as much as η = 2 but gives none of the few-label gain. Sec. 5.1 now gives a setting where η does what the baseline's own knobs do not.
  • Curvature on trained networks (PYsM). The shape defect of the regularized layer decreases with η at every width (new App. F). Sec. 5.2 adds the limitation that the curvature analysis predicts the direction of the effect, not the operating point.

Two results from the rebuttal:

10-shot accuracy change vs η = 1 (CIFAR-100, ResNet-18, 80 epochs, 3/3 seeds in every cell):

η251020
K = 256+1.18+2.18+2.20+1.90
K = 1024+0.97+2.15+2.37+2.24
K = 4096+0.71+1.98+2.58+2.57

The narrow-projector setting (8mCX Q1), 10-shot ridge on frozen backbone features:

intervention from the η = 1, K = 256 baseline10-shot Δ
widen the projector 16× (K = 4096, η = 1), 3 seeds+0.31 [+0.14, +0.47]
double the strength (ρ = 15, η = 1; 5 seeds, K = 1024)+0.06 [−0.04, +0.16]
η = 10 at the same narrow width (K = 256), 3 seeds+2.20 [+2.05, +2.36]

We would be grateful if the reviewers could look at the replies and the revised manuscript, and we are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.