Representational Frustration: Local Incompatibility in Neural Networks

Examples that are nearby in a ReLU network's hidden-state space can nevertheless activate different sets of units. We call this local mismatch representational frustration and ask how it is organized across geometric scales and shaped by learning. We measure activation-support dissimilarity among pairs selected at increasing distance thresholds in hidden-state space. Since frustration has no intrinsic quality ordering, we investigate it through a temporary learning-rate increase, motivated by evidence that large steps can reorganize learning dynamics. We then restore the baseline and compare the resulting trajectory with a matched control. In CIFAR-10 MLPs, support incompatibility among control-nearby pairs remains displaced through the observed training horizon, after the learning rate returns to baseline and the large accuracy disruption subsides. Comparing local changes with the all-pair response reveals a reorganization across geometric scales. Under cross-entropy, the closest pairs initially change less than the global pair population but retain excess incompatibility afterward; under mean-squared error, positive local excess is already present at the measured pulse checkpoints. The retained local excess also depends on training conditions: it is negative in the full-batch runs, and different hidden layers can retain opposite signs. These observations show a persistent, scale-dependent organization of representational frustration that predictive accuracy alone does not describe. What these persistent differences retain from learning, and whether they play a role in what the network remembers, remain open questions.