acceptodds
Under review as a conference paper at ICLR 2027

Ghost Directions and Intersection Lattices in Shallow ReLU Networks

Abstract

The parameters that fit a given dataset with a shallow rectified linear unit (ReLU) network are far from unique. We study two global questions on finite data: when can two parameter settings with the same outputs be joined by a path that never changes those outputs, and where do several activation labels give algebraic explanations of the same output? We take an output-space view in which each activation cell is represented by its output-image closure, a linear branch. We compare the intersection of adjacent branches with the closure of their shared activation wall's image. Directions in the intersection but outside this wall-image closure are ghost directions. For a single wall, the dimension gap is at most one and is nonzero only when the branches coincide; a target outside the wall-image closure prevents a direct crossing at fixed output. Assembling these local incidences gives a global necessary condition for exact-fit paths, and refining activation cells by output-weight signs yields a finite graph that computes the exact fiber components by linear feasibility. On the multiplicity side, we study the maximal-branch intersection lattice: its dimension-labeled structure is generically constant at fixed uniform affine order type and width. Retaining all branches with activation-label weights stratifies outputs by their algebraic label multiplicity. For six planar inputs in general position and one hidden neuron, we classify the generic lattices across all 16 order types and determine their exceptional loci. Together the results separate shared outputs from feasible parameter transitions, bear on identifiability and mode connectivity, and supply finite, checkable examples.

Primary area:
learning theory
Keywords:
ReLU Geometry, Ghost Directions, Intersection Lattices, Algebraic Geometry

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_eEZXOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

The submission studies a specific problem in the theory of neural networks (single-hidden-layer, ReLU activation, scalar outputs): if two parameter configurations produce identical outputs on a datasets is there a path connecting them in parameter space that preserves all outputs? Adopting an algebraic approach, it establishes that even if two linear regions produce the same outputs, the intersection set between them might be lower-dimensional and therefore contain an obstruction to moving between the regions. Furthermore, the submission also studies how many ReLU activation patterns can result in identical predictions.

Critical strengths
  • The findings, in particular the observation of rank-1 barriers between identically-valued output cells of ReLU networks, are interesting and original.

  • The analysis described in the submission provides a new approach for analyzing the path connectedness of ReLU networks in parameter space on finite datasets. I expect several of the arguments and techniques to carry over to related problems. The tractable (6,2,1)-example for multiplicity counting might serve as a benchmark for future works, too.

  • The submission does a decent job presented the (algebraic and therefore rather abstract) results in a measured way and putting them into the context of prior work. While the work is nevertheless quite dense, and I cannot claim that I fully understood all aspects, I applaud the authors for trying to give concrete examples, and making explicit the relation between otherwise overly abstract quantities (e.g. Section 3.2).

Critical weaknesses
  • The scope is quite limited, even for a theoretical study: ReLU networks with a single hidden layer scalar output, connecting path are exactly output preserving. The provided insights from concrete examples are limited to a handful of points and typically 1 or 2 hidden neurons.

  • The work makes no connection to real training. For ICLR, it would have been nice to see some quantities characterized for real training runs, e.g. on Gaussian or even real data, such the emergence of ghost walls and the dist(y,Ga,b)2/(2n)dist(y, G_{a,b})^2/(2n) barrier height from the appendix.

  • The fully characterized six-point planar case appears disconnected from the earlier ghost-wall analysis, which do not emerge for m=1m=1.

Questions for authors
  • Theorem 3 makes statements about true walls, where all preactivations except the changing one are strictly signed. Is it obvious that these (always) exist?
Reviewer_oBngOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors consider shallow ReLU networks functional space with restricted domain. Given a finite sample they determine a structure, first via closure and then by fiber components, where they can identify the properties of the transition between two neighbouring activation cells in the parameter space. The paper organized into two distinct parts. First they explore pairwise intersections then int he second part they consider multiple intersections. The main purpose of the first part to specify continuous parameter paths with the same output image by constructing a graph out of neighbouring pairwise parameter cells while in the second part they consider multiple intersections and determine multiplicity of cells for a specific target variable.

In the first part they form the border or wall image highlights an interesting phenomena, the ghost directions. These ghost directions occur when the true walls, aka the walls separating cells with a single activation change, have different characteristic to the intersection of the separated cells. In these cases the dimension of the Zariski closure of the wall output image will be lower dimensional than the dimension of the intersection of the bordering cells closures. Next the authors analyze algebraic graph between the cells, and after inspecting the closure and the output image identify a fiber structure, a graph between cells. The final connected components, the exact fiber components, have nice properties, e.g. both the borders and the cells realize the target in the output image (linear feasibility). The novelty lies in the phenomena of ghost directions. In my opinion this construction may be a step into understading more about these networks generalization capability or robustness, e.g. identification of the reduced set of possible labeling based on the fiber structure may be connected to shattering (Vapnik-Chervonenkis theorem). I found the first part in that regard particularly valuable.

In the second part the authors consider intersections of multiple cells in a different view than parameter continuity. They are interested in counting the number of activations whose outputs contain a fixed output target in an intersection. Even though the authors do not discuss it in detail I found this argument regarding adversarial robustness and sensitivity interesting.

Imo next steps would be to extend the construction to deep networks and analyze sensitivity to sampling. Our understanding of generalization of mlps with relu activations would be an additional step.

I have found the paper consistent and mathematically sound and I did not find any major concern about the presented theorems. Imo the main contributions are the ghost directions.

My main concerns are the limited scope of the analysis (shallow relu) even if the detailed work suggests a more demanding work to make it more general and the missing argument about sampling. Another major concern is the limited scope and the missing literature work regarding irreducible networks.

Critical strengths
  1. Thoughtful work in the sense of exploiting the structure given by the Zariski closure of the output image.
  2. Ghost direction is a valuable contribution.
  3. Logical presentation and useful figures.
Critical weaknesses
  1. Limited scope: finite sample sensitivity and shallow ReLU networks. I acknowledge of the limitation of the construction in comparison to considering fully the fucntional space.

  2. Lack of greater focus: especially in the second part (the multiple intersection argument) I found the paper lacking a clear focus regarding our goal to understand these networks and their parameter space.

  3. Especially in the abstract there are several key concepts which are either introduced in the paper or not common enough (e.g. wall, cell, fiber). A more common introduction, note this is how the lack of focus connects imo, would benefit a lot and would greatly increase the possible audience.

  4. Imo the connection to results regarding identification of relu networks cannot be discarded. E.g. in [1] the authors identify equivalence classes of parameters through symmetries in overparametrized networks via critical subspaces. I think a discussion about the connection of the fiber and ghost directions and these results considering irreducible networks could increase the value of the paper. For more about functional equivalence in mlps see e.g. [2,3,4,5,6]

  5. Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In International Conference on Machine Learning, pages 9722–9732. PMLR, 2021.

  6. Héctor J. Sussmann. Uniqueness of the weights for minimal feedforward nets with a given input-outputmap. Neural Networks, 5(4):589–593, 1992.

  7. Francesca Albertini, Eduardo Sontag, and Vincent Maillot. Uniqueness of weights for neural networks. In Richard J. Mammone, editor, Artificial Neural Networks with Applications in Speech and Vision, pages 113–125, 04 1993.

  8. Paul C Kainen, Vera Kurková, Vladik Kreinovich, and Ongard Sirisaengtaksin. Uniqueness of network parametrization and faster learning. Neural Parallel Sci. Comput., 2(4):459–466, 1994

  9. Vera Kurková and Paul C. Kainen. Functionally equivalent feedforward neural networks. Neural Computation, 6(3):543–558, 1994.

  10. Verner Vlacic and Helmut Bölcskei. Affine symmetries and neural network identifiability. Advances in Mathematics, 376:107485, 2021

Questions for authors
  1. In the label mulitplicity part (starting from line 407) the authros consider the target defined as y in R^n. How this is handled in comparison to finite target?
  2. What could be a useful application of the exact-fiber paths regarding the network structure and parametrization?
Reviewer_NPSJOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a mathematical framework to study the invariances in ReLU networks. They study the geometry of shared activation patterns over a fixed set of samples, building a finite graph of activation cells. This allows them to identify cells that are adjacent but have so-called "ghost directions" that block individual crossings. They also form lattices from these structures to study the degree of multiple realizability in activation patterns.

Critical strengths
  • Rigorous mathematical treatment: Algebraic geometry is not my specialty, so I struggle to evaluate the correctness and significance of the mathematical results. However, the theoretical work in this paper is extensive. The appendix provides extensive proofs as well as examples computed on small networks. As far as I could tell, the results are correct, and the figures are helpful for understanding the concepts.

  • Topic of interest: The question of multiple realizability, and of how activation patterns relate to the evaluation of the network, is one of interest to the machine learning community (see the two references below for examples).

References: Braun, Lukas, Erin Grant, and Andrew M. Saxe. “Not All Solutions Are Created Equal: An Analytical Dissociation of Functional and Representational Similarity in Deep Linear Neural Networks.” Paper presented at Forty-second International Conference on Machine Learning. June 18, 2025. https://openreview.net/forum?id=YucuAuXMpT.

Simsek, Berfin, François Ged, Arthur Jacot, et al. “Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances.” Proceedings of the 38th International Conference on Machine Learning, July 1, 2021, 9722–32. https://proceedings.mlr.press/v139/simsek21a.html.

Critical weaknesses
  • Limited applications to real-world settings: Although the authors provide examples, they do not relate the paper to trained networks or real-world settings. It would be interesting to relate this work more closely to real-world networks, for instance by training two networks on the same tasks from two different initializations and predicting whether it is possible to move the network between the two solutions based on that. However, the authors do provide a comparison with the empirical NTK, showing that the NTK "jumps" when the network crosses a ghost wall, which is helpful.
Questions for authors
  • What is the significance of these results for neural network training or interpretation? Is it possible to compute these quantities on larger networks and predict whether a transfer is possible? Can they be related to multiple global minima in loss landscapes, for instance?
Reviewer_xrqrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies shallow (1 hidden layer) ReLU networks. The central object of interest is paths in parameter space that preserve the network outputs on a fixed finite batch X of input points. With regards to that fixed batch X, parameter space decomposes into regions on which the activation patterns of the neurons is constant. The central question this paper consider is: given two adjacent activation pattern regions (adjacent, in the sense of sharing a codimension one wall, over which a single neuron changes its sign for a single input point), and outputs that are potentially achievable for those patterns, could there be an output-preserving path across the wall? And how does that impact the connectivity of the set of parameters that realize this output vector?

The set of all vectors of outputs on the batch X realized on a given cell is difficult to exactly characterize or work with, so the paper’s strategy is to instead study a related but algebraically simpler object: the Zariski closure of this set.

Section 3 is about ghost directions and fiber connectivity. The idea of the ghost directions is that the intersections of the Zariski closures for adjacent cells can contain points not actually in the Zariski closure of the shared wall. If a particular output vector is not in the Zariski closure of the shared wall, then a path with this constant output cannot cross the wall. Theorem 3 describes the difference between the intersections of the Zariski closures and the Zariski closure associated to the wall, in particular showing that this set has dimension at most 1.

The idea of Theorem 4 seems straightforward. Namely, that if a constant output path cannot cross a wall due to this obstruction, then in order to connect points in the two regions the path must go around this call, passing through a series of cells and walls. This can be formulated as a graph of non-obstructed cell transitions, and Theorem 4 says that any such constant-output path must be compatible with this graph. The mathematical strength/key result of Theorem 4 is that no single-wall-crossing or transversality assumption is required.

Theorem 6 is the strongest result in this section. It proves a characterization of the connected components of the batch fiber. It yields an algorithm that could be used to actually calculate the number of connected components in a batch fiber. The graph is computable using linear feasibility. The resulting algorithm is finite and exact, but potentially computationally expensive, as the graph can have exponentially many vertices in the network width.

Section 4 is about the intersection lattice. The Zariski closures associated to the various cells are a collection of linear spaces. Such a collection naturally gives rise to an intersection lattice – the object that records how these linear spaces sit relative to each other.

Theorem 8 says that, assuming the order type (i.e. basically the combinatorial geometry of the inputs) is constant, the intersection lattice is generically constant. This seems not too surprising.

Theorem 9 describes a stratification of output space according to the number of algebraic activation pattern explanations of an output vector. This is obtained by enriching the intersection lattice with nonmaximal branches and multiplicities. However, these counts concern membership in Zariski closures, rather than actual realizability of the outputs under the specified activation patterns. I have not fully parsed the exact meaning or significance of Theorem 9; the manuscript would be improved by explaining this more.

The authors prove that the 16 uniform order types of six planar points produce 16 pairwise nonisomorphic generic intersection lattices, and identify the exceptional algebraic loci where the lattice changes within a fixed order type.

Critical strengths

The paper is mathematically rigorous and carefully written.

Being able to compute the number of connected components of batch fibers is a natural question.

Corollary 7 is striking.

The paper distinguishes between algebraic obstructions and actual realizability.

Critical weaknesses

The two main sections (section 3 and section 4) feel to some extent like different projects. There is not a lot of interaction between the two sections.

Passing to Zariski closures can introduce output vectors that are not realizable by parameters in the corresponding activation cells. Consequently, the significance of the intersection-lattice results for actual network realizability and parameter-fiber geometry is not clear. In particular, it would be helpful to understand what information these lattices provide about actual neural network behavior beyond the algebraic relationships among the branch closures.

Theorem 6 does not seem to need the ghost directions formalism.

It is not clear which, if any, insights or techniques might extend to deeper networks (with more than one hidden layer).

Overall, my biggest concern is that the paper does not sufficiently establish the significance of the algebraic and combinatorial structures it studies for understanding actual neural networks. The paper presents an interesting collection of results, but they feel somewhat incremental/disjointed, in that the paper does not adequately convey an overarching vision or new conceptual geometric understanding of actual network properties.

Questions for authors

As far as I can tell, the proof of Theorem 6 does not actually use the formalism of ghost directions. What additional understanding of the geometry does the ghost direction formalism provide?

The classification of intersection lattices for six points is an interesting computational result, but I’m not sure what broader insight into shallow ReLU networks it provides; can you explain?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.