Temporal Hide&Seek: Instance-wise Time-Feature Selection for Time Series

Instance-wise feature selection seeks to identify inputs that are informative for individual predictions. For time series, this requires identifying not only which variables matter, but also when they matter. This creates distinct challenges: the relevance of an observation can depend on its temporal context, the relevant cells can change from one window to the next, and masking individual observations can disrupt the temporal structure the predictor relies on. We introduce Temporal Hide&Seek (T-HiSe), a sequence-aware method for instance-wise selection of time–feature pairs that learns a selector and a predictor jointly and end to end. A Transformer-based Hide network reads the full input window to produce a continuous mask over every time–feature pair, and an LSTM-based Seek network predicts from the masked sequence. Masked observations are replaced with complete feature trajectories drawn from strictly earlier training windows, preserving each feature’s temporal structure while avoiding future information, and an annealed sparsity penalty sharpens the masks as training progresses. On eight synthetic benchmarks with known ground-truth supports, spanning fixed and context-dependent supports, T-HiSe attains the best or tied-best support recovery on seven and averages 97.2%, with the largest gains where the relevant support switches between windows. In two real-world case studies, hourly bike-sharing demand and EEG seizure detection, T-HiSe achieves predictive performance comparable to the reported references while assigning high mask values to a relatively small part of each window. Deleting these highly ranked cells reduces predictive performance substantially more than deleting the same number of random cells.