acceptodds
Withdrawn from ICLR 2027

Compositional Objectives: Learning Structure in Structure

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

14 views

Abstract

Intelligence is defined in many ways. One of these definitions defines intelligence as the pursuit of learnable novelty. However, learnable novelty can be meaningless without the ability to compose the learned structures to take action and achieve goals. Learnable novelty builds on epiplexity, which is a way to measure learnable structure in data through a bounded observer. In this paper, we investigate a closed-form spectral approximation to compute epiplexity. We use a fixed-trace constraint and find that the epiplexity objective prefers a more uniform distribution of spectral mass rather than concentrating it in a small number of directions. However, a representation may spread information across many directions without organizing that information into features useful for a particular task. To address this gap, we propose a compositional objective whose observer measures the relationships between the parts and interactions of an image. We compare it with the original spectral objective given only the masked parts. In our ImageNet training runs, the spectral objective with masked parts produces an almost maximally spread representation while achieving the strongest frozen-feature classification performance on most evaluations, more than doubling the linear-probe accuracy of the whole-image baseline. Across multiple image benchmarks, changing what the observer sees matters more than adding relation and interaction tokens. At the same time, our prediction-oriented compositional objective produces substantially better held-out observer prediction but relatively weaker classification, revealing that spectral diversity, predictability, and downstream utility are distinct properties. These results suggest that the usefulness of spectral spreading depends not only on how much structure is preserved, but on which relationships the observer makes available to the objective.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Compositional Epiplexity, Bounded Observer, Minimum Description Length, Compositionality, Reservoir Computing, Computer Vision, Learnable Novelty

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1QouOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the idea of approximating epiplexity, building on ideas introduced in prior work on learnable novelty. The idea is to approximate the complexity of a computationally bounded learner (observer) using a fixed randomly initialized model with a linear readout. The work shows that the spectral objective prefers spreading the singular values of the readout across many directions. The paper argues that this does not necessarily lead to representations that are useful for downstream tasks. To address this, they (1) construct eight masked views of an image together with relation and interaction tokens, which are intended to capture relationships between the different parts; and (2) they introduce an objective that measures how much of the representation covariance can be explained by the observer, using the log ratio of covariance determinants before and after prediction -- intuitively measuring explained covariance of the representations as opposed to spectral complexity of the weights as done originally.

In the experiments, they show that using the original spectral objective introduced in prior work on learnable novelty, but with an observer receiving masked views, leads to better downstream performance on ImageNet compared to the whole-image baseline. However, their own proposed compositional objective does not improve upon this. They conclude that spectral spreading, observer predictability, and downstream usefulness are distinct properties.

Critical strengths
  1. The direction of extracting structure from data is interesting, applying notions of epiplexity and forms of its approximation is underexplored.
  2. The scope of the work is good.
Critical weaknesses

The biggest weakness is that the study provides neither convincing new insights into approximating epiplexity nor practically useful insights into how it could lead to practical gains.

The paper's conclusion reads, "when the spectral approximation to epiplexity is used to train a representation, what is it actually rewarding?. In theory, the answer is spread." But I don't see how this is established beyond the fixed-trace setting. The authors then write, "At fixed readout energy, the score prefers to distribute spectral mass evenly across as many directions as it can." However, this constraint is not enforced during training. While the authors empirically observe spectral spreading, their theoretical result concerns the readout weights, whereas the experiments measure the covariance of the learned representations. It's not clear how the former explains the latter.

Additionally, the authors write that "switching the spectral observer from the whole image to masked views more than doubled linear-probe top-1 accuracy." But (1) you can switch many more things to get better accuracy (just adopt inductive biases studied in self-supervised works), and (2) increasing accuracy x2 (from 3% to 7%) doesn't establish where the gains came from. The comparison also changes the preprocessing and compute.

Lastly, the authors write, "recognizing the leaves is not the same as knowing how they connect" and propose "building objectives that reward the connections themselves." But I don't see how this follows from their results. The paper shows that using masked views (with more compute and other changes) leads to better downstream performance. That's all. It does not show that the proposed objective learns relationships between parts.

More concretely:

  1. Their proposed method does not improve downstream performance. The biggest gains come from using eight masked views with the original objective. Their proposed compositional objective performs worse on almost all tasks.
  2. The motivation for multiple masked views comes out of nowhere. It's a particular choice of the frozen observer. The masked views go through the observer, and the expectation is that its outputs can predict the representations of the trainable encoder. This introduces a specific inductive bias whose justification does not follow from epiplexity. Moreover, this setup uses more compute, and the comparison also changes preprocessing and score calibration.
  3. The motivation for the new objective is unclear. Why should maximizing the amount of representation covariance explained by the observer lead to compositionality? It's not clear how this captures relationships between image parts. The experiments also do not demonstrate that it does.
Questions for authors

See weaknesses.

Reviewer_DAYjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies representation learning through the lens of epiplexity and asks what the recently proposed spectral approximation actually encourages an encoder to learn. The authors show that, under a fixed-trace constraint, the spectral objective prefers spreading mass across many directions, and provide a counterexample showing that identical representation spectra need not imply identical downstream utility. Motivated by this limitation, they introduce a compositional observer built from masked views, relation tokens, and interaction tokens, together with a predictive covariance objective. Experiments on ImageNet-1K and several transfer and relational benchmarks examine the relationship between spectral spread, observer predictability, and downstream utility.

Critical strengths
  1. The paper asks a well-defined and interesting question about what spectral epiplexity objectives actually optimize, rather than treating the spectral score as an unexplained proxy for representation quality.

  2. The theoretical observations are clean and useful: the fixed-trace analysis characterizes the preference for spectral spreading, while the counterexample clearly shows that spectrum alone cannot determine downstream utility.

  3. I appreciate that the experiments report negative results rather than hiding them. In particular, the paper clearly shows that masked observer inputs have a much larger effect than the proposed relation/interaction machinery, and that spread, predictability, and downstream utility can behave quite differently.

Critical weaknesses
  1. The proposed ScompS_{\mathrm{comp}} does not improve the main downstream results. It reaches only 3.30% ImageNet top-1 accuracy, compared with 7.02% for the simpler masked-view spectral objective, and shows the same pattern on most transfer benchmarks.

  2. The relational evaluations provide little evidence that the proposed objective actually improves compositional reasoning. On SpatialSense and the CLEVR relational subset, all three variants are within roughly 0.6 percentage points of each other, with the ordering changing across seeds.

  3. Most of the empirical gain appears to come from replacing the whole-image observer with masked views. This change alone more than doubles ImageNet linear-probe accuracy, whereas adding relation/interaction tokens and replacing the objective does not provide further improvement.

  4. The current ablations do not cleanly identify the source of the gains. Moving from SϕS^{\phi}to SϕcompS^{\phi_{\mathrm{comp}}} also changes preprocessing and score calibration, which makes the effect of the observer input difficult to isolate.

  5. The comparison between SϕcompS^{\phi_{\mathrm{comp}}} and ScompS_{\mathrm{comp}}changes both the observer construction and the optimization criterion. It is therefore unclear whether the resulting behavior is caused by the relation/interaction tokens, the predictive covariance objective, or their combination.

  6. The theoretical analysis mainly establishes what the original spectral objective prefers, not why the proposed objective should recover compositional structure. The counterexample shows that identical spectra can have very different task utility, but it does not specifically justify the proposed observer design.

  7. The term "compositional" remains only weakly operationalized. The paper shows that ScompS_{\mathrm{comp}} greatly improves held-out observer predictability (83.51% error reduction versus 18.56%), yet this does not translate into better relational or classification performance, making the connection between predictability and composition unclear.

  8. The experimental conclusions are also limited by the small number of runs. Only two seeds are used, while several relational differences are very small and even change ordering across seeds.

Questions for authors
  1. The experiments use only two seeds. Given that the relational evaluations differ by less than one percentage point and even change ordering across seeds, additional runs would be useful before drawing conclusions from these comparisons.

  2. The reported representation quality is quite low in absolute terms (e.g., 7.02% ImageNet linear-probe top-1 for the strongest variant after pretraining). While state-of-the-art SSL performance is clearly not the main goal, comparison with at least a few simple masked-prediction or redundancy-reduction baselines under a matched compute budget would help contextualize the practical strength of the learned representations.

  3. The relational evaluations combine encoder features with additional metadata, as the authors acknowledge. A cleaner probe using only the learned representation would make it easier to determine whether relational information is actually encoded by the representation itself.

Reviewer_EqncOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks a very specific question: when a model is trained with a certain "spectral score," what is that score actually rewarding? The authors report three main findings: The spectral score prefers to "spread information evenly" — the flatter the spectrum, the higher the score. It is like pouring a cup of water evenly into many cups. But "spreading evenly" does not mean "useful." The authors give a counterexample: two representations can have exactly the same spectrum, yet one supports perfect linear classification while the other is only at chance level. The authors propose an improved method that aims to learn not only "parts" but also "how parts combine into a whole." However, experiments show that this improved method does not outperform the simpler version. The paper's biggest finding is therefore not "our new method is better," but rather: changing what the observer sees (from the whole image to multiple masked views) helps more than changing the objective itself.

Critical strengths
  1. Instead of proposing yet another method, the paper first asks what existing methods actually reward. This "understand the principle first" approach is valuable.
  2. The authors use a simple mathematical result to show that, under a fixed total energy, the spectral score favors a flat distribution. This analysis is easy to follow and consistent with the experimental results.
  3. The authors do not hide the fact that their new method is worse than the simpler version on most tasks. This is commendable.
  4. The largest gains come from letting the observer see multiple masked views, not from a more complex objective. This is useful for future research.
Critical weaknesses
  1. The paper spends considerable space proposing a "compositional objective," but experiments show that it is worse than the simpler version on most classification tasks. This makes the reader wonder: why should I use the new method? The paper needs to clarify whether the value of the new method is analytical or practical.
  2. The authors themselves admit that the performance gain mainly comes from changing the input to the observer, not from changing the objective. This means the paper's core contribution (the compositional objective) is not the cause of the improvement. The paper's positioning should be adjusted more honestly.
  3. The theoretical analysis assumes "fixed total energy," but this condition does not hold during actual training. The analysis is more of an "ideal-case illustration" and cannot fully explain what happens in real training. The authors should state the scope of the analysis more clearly.
  4. SpatialSense and CLEVR are used to evaluate "relational reasoning," but the readouts also use metadata. This does not fully isolate the model's own ability. The conclusions based on these evaluations should be qualified.
Questions for authors
  1. If the new method does not improve performance, position the paper as an analytical study: focus on "what the spectral score rewards," "why spread, predictability, and utility diverge," and "why observer input matters," rather than "we propose a better compositional objective." This is more honest and more valuable.
  2. Compare the three methods under the same compute budget, or at least report "performance per unit of compute" to avoid the concern that the new method simply uses more computation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.