acceptodds
Under review as a conference paper at ICLR 2027

Toward Spectral Kinematics:Transformation Laws and Their Limits

Abstract

Pretrained encoders map samples to feature vectors that can be stored and reused across tasks. Pooling, compression, and reweighting change their spectral diversity, the effective number of directions carrying feature energy. We study what information determines pooled diversity and what partial observations can guarantee. Building on classical transformation identities, we construct sources with identical spectra, energies, and principal angles but different pooled effective ranks. These summaries miss which energetic directions overlap. For second-order effective rank, which emphasizes high-energy directions, source spectra and energies plus one energy-weighted overlap total determine the answer exactly. Bounding unmeasured contributions to this total yields computable relative-error guarantees. Compatible completions, which preserve all observations while varying unmeasured overlaps, distinguish conservative bounds from information that permits opposite diversity-gain decisions. In a four-encoder vision cohort, raw features require a median of 1% of direction-pair measurements to guarantee at most 1% relative error; centering raises the median to full coupling on the tested query schedule. Fewer measurements do not always reduce runtime: gains depend on cached source factors and bound-tightening costs. These results connect the information needed for pooled spectral diversity to reliable prediction and spectral-diversity batch admission, while delimiting their computational benefits.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
spectral kinematics, representation learning, spectral analysis, Rényi effective rank, feature pooling, subspace alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ecwdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies which information is sufficient to predict the spectral diversity of pooled feature matrices, measured by Rényi effective ranks of normalized squared singular values. It shows constructively that source spectra, energies, and principal angles do not determine pooled effective rank because they omit the association between energy and overlapping directions. For second-order effective rank, this dependence reduces to one energy-weighted overlap scalar: r2=E2/(Q+2L)r_2=E^2/(Q+2L). The paper bounds unmeasured contributions to LL and uses compatible coupling completions to distinguish conservative certificates from genuine ambiguity under partial observations. For validation, it includes synthetic constructions, numerical identity checks, frozen vision features, external supervised/DINO experiments, and fixed-base spectral admission decisions. Partial queries substantially reduce measurement counts for raw features, but centered features require much larger budgets, and runtime benefits depend strongly on cached source factors.

Critical strengths
  1. The paper identifies a precise information gap and demonstrates its consequences constructively. Proposition 3 preserves source spectra, energies, and principal angles while changing pooled Shannon effective rank. Corollary 4 converts this example into a quantitative prediction lower bound, and the direction-permutation experiments show that the phenomenon also arises from measured source factors. This is consequential because it establishes a failure of the retained information itself, beyond a failure of a particular estimator. The distinction between sufficient information for r2r_2 and for the broader effective-rank family is also clear and useful.

  2. The partial-observation analysis carefully separates accuracy, certification, and impossibility. Proposition 6 gives an observable deterministic error bound without requiring a positive merged-eigenvalue floor. The completion analysis then distinguishes a loose bound from observations compatible with opposite decisions.

Critical weaknesses
  1. The significance of the theoretical advance beyond established identities remains insufficiently demonstrated. The transformation laws are explicitly classical; full-Gram reconstruction and the upper envelope follow from standard Gram and majorization arguments; and the order-two correction is a direct trace expansion. Closely related prior work already expresses order-two Vendi novelty through an exact second-moment update and energy-weighted eigenvector projections. The distinctive contribution therefore rests primarily on partial-coupling certificates and completion diagnostics. These are useful, but the general completion optimum remains bracketed by a transport relaxation and a finite witness family; exact characterization is provided only for a two-mode family. The fixed-observation lower bounds also do not establish the efficiency of the selected query policy. I am consequently uncertain whether the paper provides a sufficiently substantial theoretical advance to support acceptance, despite its sound and careful presentation.

  2. The experiments establish a narrow computational benefit, leaving the practical importance of the framework uncertain. The strongest measurement savings occur on raw features, where the median mean-related overlap share is 99.31%. On centered features, the refined median certification budget is 64.645% in the cohort and 98.998% for external DINO features; every tested real-feature Shannon certificate requires full coupling at 1% tolerance. Table 10 further shows that direct products win all preparation-charged aggregate comparisons, while centered transport tightening costs about 4004.55 ms versus 2.30 ms for direct products even with cached inputs. Moreover, all raw and unit-row admission examples fall below the primary 5% gain threshold, and the downstream negative controls establish no selection or pruning advantage. Downstream accuracy is not necessary for a theoretical contribution, but these results leave little demonstrated application benefit to compensate for the uncertainty about theoretical significance.

Questions for authors
  1. Which specific theoretical statement constitutes the main advance beyond classical trace/majorization identities and the order-two eigenbasis update in Vendi Novelty Scores? Please explain whether the contribution is a new guarantee, a sharper information characterization, or their integration into a diagnostic framework.

  2. Does the existing evidence identify a realistic reuse regime in which certified partial coupling beats direct products after all necessary preparation is charged? In particular, how much reuse is needed, and are candidate-source factorizations assumed to be independently available? This would materially affect my assessment of the runtime contribution.

  3. Appendix F.1 states that the stochastic baselines are not calibrated for a per-pair guarantee. Can the existing results support a comparison at matched error tolerance and an explicit failure probability, including preparation costs? Seven disagreements at a fixed probe budget do not establish whether the proposed certificates offer a favorable cost–reliability trade-off relative to calibrated trace estimation.

1 comment

We have narrowed the theoretical focus from spectral transformation laws to the overlap information needed for pooling, with the exact deterministic worst-case query budget as the main new result.

What the pooling problem requires (Abstract and §1). Source spectra and principal angles can hide which energetic directions overlap (§3). The order-two trace identity, which specializes to the Vendi Novelty update for unit-row features and one added sample, identifies the energy-weighted overlap total that suffices given the source spectra. Bounds on its unread part can certify gain decisions. Two compatible completions on opposite sides of the threshold prove that the current readings cannot settle a decision (§§4.1 and 4.3).

Exact query complexity (§4.2, p. 7, Appendix I). Given positive spectra, exact queries in fixed source coordinates, and d≥kA+kBd\ge k_A+k_B, a maximum-weight matching on unread entries gives a fixed mask's minimax relative error over admissible overlaps. A maximum-matching partial permutation and zero overlap give the same all-zero readings. This yields a lower bound even for deterministic adaptive policies, attained by a minimum-size feasible fixed mask. Corollary 8 gives a closed form for equal flat spectra. For nonzero readings on a queried rectangle, §4.3 brackets the conditional completion optimum using feasible witnesses and a transport relaxation.

Distance to optimality (§5.4, Figure 5, Appendix N). At the 1% worst-case relative-error target, exact search closes 40 preselected low-budget cases among 240 full-spectrum pairs. For the 80 centered pairs, the median lower bound rises from 5.40% to 38.89% of entries. The shortest energy prefix gives a median feasible upper bound of 57.91%, with those intervals still open. In controlled partial-query truncations, energy order reaches the solver-certified optimum in 761/800 cases and the best leading rectangle in 0/800. The omitted tails preclude a uniform 1% guarantee on the full problem in all 1,200 truncations.

Cost and near-threshold decisions (§§5.4–5.5, Appendices J–M). Direct products have the lowest preparation-inclusive median runtime. Neither reuse pattern breaks even through 256 evaluations cycling the same four candidates per bank. Base-only reuse prepares the candidate each time, and two-sided reuse prepares each candidate once. In the matched 1% final-error comparison, Gaussian methods use pointwise 95% intervals with exact fallback. The 48-probe Hutch++ variant falls back in all 4,800 centered trials. The 96-probe version requires no fallback in any condition. At the separate 5% admission threshold and about 16% entry budget, the shared certificate settles 3/14, 1/14, and 0/14 near-threshold cases with rectangle, energy, and random queries, respectively, with zero automatic errors. Exact fallback handles the rest.

Reviewer_nkM8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how spectral diversity, measured by Rényi effective rank, changes when feature representations are pooled, reweighted, or restricted. The paper shows that source spectra, energies, and principal angles are generally insufficient to determine the effective rank after pooling because they do not preserve the association between mode energy and directional overlap. For second-order effective rank, the authors derive an exact characterization based on an energy-weighted overlap term and further develop error certificates when only partial coupling information is observed. The theoretical analysis is evaluated on frozen features from multiple vision encoders, with additional experiments on measurement budgets, spectral-diversity-based admission decisions, and computational cost.

Critical strengths
  1. The paper provides a careful characterization of the information required to determine pooled spectral diversity. The counterexamples clearly show that source spectra and principal angles alone can be insufficient because they discard the association between energy and direction. The exact order-two characterization is a clean and technically useful result.

  2. The treatment of partial observations is rigorous. The paper does more than propose an approximation: it derives computable error certificates and uses compatible completions to distinguish conservative bounds from genuine information insufficiency. This is a technically strong aspect of the work.

Critical weaknesses
  1. The connection to practical representation learning is not sufficiently established. The strongest theoretical results concern second-order effective rank, but effective rank itself is only a geometric property of the representation and does not directly measure downstream usefulness. The batch-admission experiments certify decisions in terms of spectral diversity, rather than showing that these decisions improve training or downstream accuracy. Moreover, the paper’s own negative controls show that the proposed spectral screening does not consistently outperform a budget-matched random strategy, while the pruning experiment does not demonstrate an improved accuracy–dimension trade-off. This makes the current contribution appear more like a specialized analysis of spectral statistics than a broadly impactful representation-learning result.

  2. The practical benefit of partial coupling is substantially weaker than the motivation suggests. The empirical advantage depends strongly on preprocessing. Raw features can often be certified using relatively few coupling entries, whereas centered features frequently require nearly full coupling. More importantly, the runtime study shows that once factor preparation or bound-tightening costs are included, direct computation can be faster; for centered features, the transport-based tightening is particularly expensive. Therefore, it is unclear how broad the regime is in which the proposed partial-observation machinery provides an actual computational advantage.

Questions for authors
  1. Can the authors provide stronger evidence that the proposed spectral quantities lead to useful downstream decisions?

  2. How should the computational contribution be interpreted given the centered-feature results?

1 comment

We have narrowed the practical claim to certifying spectral-gain decisions for frozen features and reducing file reads in the tested fixed-threshold monitoring setting.

Fixed label budget (§5.6, Appendix L). At 384 labels and five paired seeds per dataset, exact pooled-rank acquisition trails nearest-neighbor novelty in balanced accuracy on Pets (83.15% vs. 84.71%) and Flowers-102 (52.62% vs. 53.92%).

PACS monitoring (§5.6, p. 10, Appendix O). With eight reviewed windows per method, pooled rank, global DriftLens, and nearest neighbors each surface 94 of 597 classifier errors. In a separate file-backed evaluation, partial reads reproduce all 118 exact pooled-rank threshold decisions without fallback. They read 82.8–87.0% fewer file bytes on light-shift and original windows, with same-host steady-state p95 latency 1.42–1.80 times that of file-backed exact scoring. The frozen cutoff flags 9/15 clean windows in the separate holdout.

Threshold transfer and cost (Appendix O.1, Table 8, p. 24). At nominal 5% false-alarm cutoffs, the regrouped Photo arm has 0/12 false alarms, 7/12 detections, and 12/16 classifier errors surfaced at eight reviews. The fresh-image Art Painting arm has 3/22 false alarms, 11/22 detections, and 27/118 errors surfaced, with 29/118 for nearest neighbors. Photo trails nearest neighbors by 8.3 detection points, outside the prespecified five-point tolerance. Art Painting misses the false-alarm and review-coverage criteria. Both arms fall short of the prespecified criteria for another partial-access comparison. Centering raises query requirements in our tests, and direct products have the lowest preparation-inclusive median runtime (§5.5).

Reviewer_1misOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is so poorly written to the point where it is very difficult to write a summary. To my understanding, the authors study encoder embeddings using spectral methods, and certain transforms of the singular values and singular vectors. While I do not directly do research in this domain, I am familiar with much of the math involved. Despite this it is not clear what the problem set up is, what their theoretical results prove, or what their numerical experiments show.

Critical strengths

None

Critical weaknesses

W1: The introduction does a poor job setting up the problem. Consider the first paragraph:

W1.1: "A pretrained encoder maps samples to reusable feature vectors." An encoder is an extremely general object. What is meant by reusable feature vectors?

W1.2: "Pooling expands collections, compression reduces storage, and reweighting adjusts the relative energy carried by feature directions." What do you authors mean by pooling? And collections? These terms are so general, and you haven't set any context about what setting we are in. When I hear pooling a machine learning context, I think of max pooling used in CNNs. Is that what you are referring to?

W1.2: Now consider the paragraph beginning on line 43. What are you referring to as a source? What are cross source overlaps? What does it mean for a decision to be certified? What is this sentence saying, "This use requires a reliable gain interval, not just a point estimate"?

W2: Section 2 is equally unclear. Line 90, what is the input domain X? Something very general is okay, but you should state that or give an example. What are "checkpoint" and "layer" referring to in the first sentence? In the next setence, what are H_M, H_A, and H_B? Why do you specify without joint centering? The next sentence, "Raw features are the baseline; source centering and row normalization test sensitivity". What do you mean by "Raw features are the baseline", the baseline of what? Are you doing row normalization? What do you mean by test sensitivity? Sensitivity to what? Again, what is the H matrix, and what is the N dimension? I guessed that H is the output of the encoder, with N input samples and latent dimension d, but this is not explained any where.

W3: It would be nice somewhere in Section 2 to describe mathematically what the goal of this paper is, or identify the key quantities (such as spectral diversity) that you are trying to analyze.

W4: To summarize the weaknesses, I found the writing extremely unclear with many details missing, and many unnecessary, extraneous details included. I suspect these issues arise from the significant use of Generative AI in writing this paper because AI often doesn't know which details are important. AI writing is allowed by the conference and you are fully transparent about this in your AI statement, which I appreciate. However, the overuse of AI suggests an obvious solution: Rewrite every sentence in your own words.

Questions for authors

Q1: Line 107 "The order changes the observable, not the matrix." Is this sentence saying that the order alpha changes the quantities S_alpha and r_alpha, but not the matrix H? The seems pretty clear to me, I don't think you need this sentence (and in fact it made me think I missed something).

Q2: For the classical results (Theorem 1), it would be good to cite a paper or textbook with these results.

Q3: Is Corollary 4 a bound on a particular counterexample from Proposition 3? This feels like a strange result to have, generally bounds are more helpful when they cover many situations of interest.

1 comment

We have narrowed the paper from several spectral transformations to estimating pooled effective rank and deciding whether a new batch adds enough spectral diversity, using known source spectra and partial overlap measurements.

Why pool two batches (Abstract and §§1–2). One frozen image encoder at a fixed layer and checkpoint gives an N×dN\times d matrix for each batch, with NN images as rows and dd feature coordinates. Pooling stacks the bank and candidate matrices. Extra images may add little diversity when they reinforce directions already in the bank. Effective rank measures how broadly feature energy is spread. A batch with exactly two equally strong modes has effective rank two. The Objective (p. 3) specifies what is known (the two source spectra), what can be queried (inner products between their directions), and what we seek (pooled order-two rank or a spectral-gain decision).

What the three results establish (p. 2). Spectra, energies, and principal angles can hide which energetic directions overlap. The two inputs in Proposition 3 have identical summaries, so a deterministic summary-based predictor must give both the same prediction. On this construction, Corollary 4 (p. 4) shows that at least one of the two relative Shannon-rank errors is 29.66% or greater. For order two, the source spectra and one energy-weighted overlap total suffice. Proposition 6 (p. 5) bounds rank error from unread overlap. Related gain intervals can certify a decision, and compatible completions can prove the observed entries insufficient (§§4.1 and 4.3). Theorem 7 (p. 7) gives the exact deterministic worst-case query budget under the stated entry-access model.

How a query settles the running example. In Figure 1 (p. 2), the sole reading C11=0C_{11}=0 fits pooled ranks 2 and 4, with gains of 0% and 100%. Figure 3 (p. 6) shows how measured overlaps and a bound on unread ones give a gain interval. In Figure 4 (p. 6), the additional reading C12=0C_{12}=0 raises the minimum compatible gain to 33.3%, certifying admission at 5% while leaving the exact rank unknown.

Preprocessing and attribution (§2). Raw, separately source-centered, and unit-row-normalized features are three distinct inputs. We stack the chosen inputs directly. For raw features, joint centering would subtract the pooled mean and generally change the rank target. Mode restriction keeps selected singular-vector directions. Power reweighting changes their relative energies. We removed the sentence about changing the Rényi order. Section 2 cites sources for Theorem 1, and Appendix B derives the three classical identities.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.