Do Neural Networks Preserve Case Structure? Case-Based Decomposition, Interpretation, and Decision Consistency

Neural networks increasingly influence consequential decisions, yet the case-based foundations behind these decisions remain hidden. This makes it difficult to understand which learned experiences support a decision and when such decisions can be trusted. Motivated by Case-Based Decision Theory (CBDT), which views decisions as being formed from relevant past cases, we ask whether neural networks preserve a similar case structure during learning. We first show that trained neural networks preserve a recoverable case structure within their learned representations. This structure enables a CBDT-inspired decomposition that traces fitted decisions back to contributions from individual training cases. We then reveal a fundamental gap between the recovered case structure and classical CBDT assumptions. Although neural networks preserve case-based decision structure, the recovered case coefficients do not necessarily satisfy CBDT relevance requirements. Enforcing these stricter constraints changes selected decisions and reduces task-optimal accuracy by 4.2%–9.1% relative to signed reconstruction. Finally, we establish a verifiable decision-consistency certificate that identifies when the recovered case structure agrees with the original network decision. Across three real-world decision settings, certified decisions achieve substantially higher task accuracy (83.7% vs. 35.0% on the Adult Income dataset). Case-level analysis further identifies compact sets that preserve some certified fitted decisions. Together, our results reveal that neural networks can preserve case-based decision structures without fully satisfying CBDT assumptions. More importantly, they show that such imperfect but verifiable structures can still provide a principled foundation for reliable case-level understanding of neural network decisions.