Ghost Directions and Intersection Lattices in Shallow ReLU Networks

The parameters that fit a given dataset with a shallow rectified linear unit (ReLU) network are far from unique. We study two global questions on finite data: when can two parameter settings with the same outputs be joined by a path that never changes those outputs, and where do several activation labels give algebraic explanations of the same output? We take an output-space view in which each activation cell is represented by its output-image closure, a linear branch. We compare the intersection of adjacent branches with the closure of their shared activation wall's image. Directions in the intersection but outside this wall-image closure are ghost directions. For a single wall, the dimension gap is at most one and is nonzero only when the branches coincide; a target outside the wall-image closure prevents a direct crossing at fixed output. Assembling these local incidences gives a global necessary condition for exact-fit paths, and refining activation cells by output-weight signs yields a finite graph that computes the exact fiber components by linear feasibility. On the multiplicity side, we study the maximal-branch intersection lattice: its dimension-labeled structure is generically constant at fixed uniform affine order type and width. Retaining all branches with activation-label weights stratifies outputs by their algebraic label multiplicity. For six planar inputs in general position and one hidden neuron, we classify the generic lattices across all 16 order types and determine their exceptional loci. Together the results separate shared outputs from feasible parameter transitions, bear on identifiability and mode connectivity, and supply finite, checkable examples.