Rethinking Encoder Learning in Time-Series Anomaly Detection

Time-series anomaly detection (TSAD) commonly learns representations from historical observations to distinguish normal from anomalous behavior. However, in the absence of anomaly supervision, learning alone faces a fundamental limitation: its objective does not directly specify which aspects of the data are actually relevant for anomaly detection. Historical observations, meanwhile, provide a concrete basis for evaluating new observations. This raises a fundamental question: if historical observations can serve directly as empirical references, is encoder learning really necessary for strong anomaly detection? As a direct test of this question, we introduce Fixed Anomaly Representation with Observed References (FARO), a deliberately simple architecture with no encoder training. FARO fixes a randomly initialized encoder, stores historical features in channel-specific reference banks, and scores new observations by their deviation from nearest historical references. The resulting channel-wise deviations support both anomaly detection and localization. Despite its simple design, FARO achieves state-of-the-art anomaly detection performance on TSB-AD-U/M and competitive channel localization on SMD, HAI, and SWaT. More importantly, controlled experiments show that, with historical references available, encoder training provides no consistent additional gain and can even degrade detection. This pattern also holds across multiple existing TSAD backbones. Together, these findings challenge the default reliance on encoder learning in TSAD and highlight empirical references as a strong alternative for exploiting historical observations. The source code is available at AnonymizedURL.