From Information to Conditional Contribution: Residual-Informed Sparse Decoding for Traceable Neural Representation Analysis

Neural representations are organized patterns of population activity that carry information about external stimuli. Large-scale recordings make such information decodable, but it remains unclear which neural responses, at which times and temporal scales, support a given readout, or whether their conditional discriminative value changes with population context, stimulus, and decoding objective. Latent-component methods often form mixed coordinates that are difficult to trace to the original responses. Information-theoretic methods preserve traceability, but static rankings do not capture a response's value for the current readout. We introduce Residual-Informed Sparse Decoding (RISD), a residual-conditioned framework that constructs task-specific neural representations from traceable multiscale response coordinates. RISD first identifies an information core using conditional mutual information, then uses grouped out-of-fold residuals to add responses not captured by the current readout, yielding sparse, testable representations selected using training data only. We evaluate RISD on image-identity and semantic decoding using recordings from marmoset retina, the mouse visual pathway, and macaque visual cortex. Under matched support budgets, RISD outperforms baselines with compact response-coordinate sets and traces selected coordinates to specific neural regions, neurons, time windows, and scales. It reveals stimulus-specific response combinations and partially recoverable readout support in retinal activity, while showing across datasets that a response's discriminative value depends on selected support, stimulus, region, and decoding objective rather than its overall statistical information.