acceptodds
Under review as a conference paper at ICLR 2027

The Learning Geometry of Deterministic Views

Abstract

Adding deterministic views of an observation can change a model's predictions without adding information. This makes a standard view ablation difficult to interpret: the input expansion may also change the learning rule. We characterize this intervention for full-rank fixed linear views, each embedded by a trainable linear branch and summed before a shared receiver. Their frame operator preconditions the effective SGD update and determines the norm induced by branch regularization. Two view systems produce the same update for every loss exactly when their frame operators agree. With matched initialization, equal- systems follow identical trajectories under SGD and scalar-linear optimizer state, including momentum. The classical Parseval transform therefore supplies a neutral control relative to the parent-only model. Nonidentical equal- frames match exactly in our experiments, whereas equal-trace frames can separate. One multiscale frame changes subject-mean NLL by on Sleep-EDF and on ISRUC; Parsevalization removes both differences. Independent schedule selection retains a Sleep-EDF gap of (95% CI ). AdamW and branchwise nonlinearities break the tested equivalences. The frame operator thus specifies what to control when attributing a deterministic view's effect to the representation.

Primary area:
optimization
Keywords:
deterministic views, optimization geometry, representation learning, frame theory, reparameterization, stochastic gradient descent, Parseval normalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9w98OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how deterministic views can affect learning despite adding no new statistical information beyond the parent input. For fixed linear views with an additive interface, the authors show that the frame operator S=A⊤AS=A^\top A preconditions the effective SGD update and characterizes when different view systems follow identical training trajectories. Based on this result, they use the classical Parseval transform to construct a neutral control for deterministic-view ablations. Experiments on EEG and image classification tasks support the theoretical findings and show that changing the frame operator can either help or hurt performance.

Critical strengths
  1. The theoretical results are insightful and offer a promising perspective for understanding, designing, and refining multi-view learning frameworks.
  2. The experiments are carefully designed to test the theoretical predictions, with controlled comparisons involving equal frame operators, different spectral structures, and Parsevalization. The results consistently support the theoretical findings across EEG and image classification tasks.
  3. This paper also examines the conditions under which the proposed equivalence holds or breaks down, including different optimizers, branch nonlinearities, and numerical approximation errors. These analyses help clarify the scope and limitations of the theoretical results.
Critical weaknesses

The paper provides precise and insightful theoretical results within a controlled interface. However, their applicability is subject to restrictive architectural assumptions, which exclude many practical multi-view learning architectures. While these assumptions enable precise theoretical analysis, it remains unclear whether the theoretical results provide useful insights beyond the linear interface, particularly for architectures involving nonlinear view-specific encoders or cross-view interactions.

Questions for authors
  1. Theoretical novelty. The connection between linear reparameterization and gradient preconditioning is well established, and Parseval normalization is a classical technique. Could the authors clarify what the main theoretical advances are beyond these existing results, particularly regarding the characterization of frame-operator equivalence and the proposed neutral control?

  2. Section 4.4 shows that exact equivalence can break down under AdamW and branchwise nonlinearities. Beyond identifying these boundaries, do the current theoretical analysis or experiments support any diagnostic use of the frame operator or Parseval control in such settings?

Reviewer_cJuNOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper analyzes the effect of adding additional, non-informative "views" of data on changing the optimization landscape of linear models. The main mechanism is that it adds a preconditioning matrix to the SGD update, whose effect can also be cancelled by adding a classic Parseval transform.

Critical strengths

None

Critical weaknesses
  • The theoretical increments are quite limited: the main argument could be simply derived by using the chain rule on the gradient update and is also a rather straightforward corollary of Hennigen et al.
  • The considered "deterministic view" model only captures a very small portion of multi-view learning practice in deep learning:
    • The analyses and results only apply to a linear model, where the effect of adding a view is precisely characterized by preconditioning. Non-linear models such as neural networks could exhibit a much richer (and conceptually different) landscape that is not captured by the linear model.
    • The paper only considers adding views that do not bring additional information about XX. Practical learning with multi-view data could be more nuanced, with different views containing different informative features [1], or the model only receiving different views transformed from XX rather than the raw XX (i.e., I(Y;T1(X)∣X)=0,I(Y;T2(X)∣X)=0I(Y; T_1(X) | X) =0, I(Y;T_2(X) | X) =0 yet I(Y;T2(X)∣T1(X))≠0I(Y;T_2(X) | T_1(X))\ne 0), e.g., in contrastive learning with random crop data augmentation.

[1] Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. ICLR 2023.

Questions for authors

None

Reviewer_oVeUOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies why adding deterministic views can change predictions despite providing no new information. For fixed, full-column-rank linear views with trainable linear branches and additive fusion, it derives that the frame operator S=ATA governs effective SGD updates and induced branch regularization. It proves that equal-S systems have identical trajectories under matched initialization and specified scalar-linear optimizers, and proposes Parsevalization (AS-1/2) as a parent-neutral control.Experiments on Sleep-EDF, ISRUC, and CIFAR-100 support the equivalence while showing task-dependent effects and limitations under AdamW or branch nonlinearities.

Critical strengths

1.The paper clearly distinguishes information changes from optimization changes, addressing an important issue in interpreting deterministic-view ablation studies. 2.The frame-operator equivalence theorem provides a rigorous characterization of when different linear view systems induce identical effective training trajectories. 3.Equal-S comparisons, Parsevalization, and boundary tests provide a structured experimental framework for validating the theory and examining its limitations.

Critical weaknesses

1.The paper could more clearly distinguish its novel contributions from prior adapter-gradient duality results, particularly the added value of the frame-operator characterization and Parsevalized control. 2.The empirical validation is relatively concentrated, relying primarily on two EEG datasets with a limited number of seeds,broader datasets and more diverse task settings would better establish the external validity of the findings. 3.Although the frame operator jointly determines gradient preconditioning and branch-space regularization, the experiments do not isolate their respective contributions to task-dependent performance, which could be addressed through additional controls or discussion.

Questions for authors

see the weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.