acceptodds
Under review as a conference paper at ICLR 2027

Beyond Covariance: How Shallow Autoencoders Learn Normal Relations for Multivariate Time Series Anomaly Detection

Abstract

A simple shallow autoencoder (ShallowAE) is highly competitive on the TSB-AD-M benchmark for multivariate time series anomaly detection, yet what it learns from normal data and how this information shapes anomaly scores remain unclear. We characterize the statistical sufficiency boundary of squared-error reconstruction objectives, proving that the mean and covariance determine the entire risk function for affine reconstruction but are not universally sufficient for nonlinear reconstruction objectives. Moment-matched interventions on real time series show that training information beyond second-order statistics can change ShallowAE's learning and detection behavior. Controls within a fixed PCA subspace further show that the network can assign different scores to different combinations of variables even when all inputs lie in the same subspace and have zero PCA residual. An analytically tractable relation-switching experiment reveals that, between normal environments with identical coordinate-wise marginals and covariance, reversing the conditional relationship changes both the network's low-residual regions and its scoring preferences on fixed inputs. These findings identify the learning of normal relationships beyond covariance as a mechanism through which ShallowAE can reshape reconstruction geometry and score inputs differently despite identical PCA reconstruction scores.

Primary area:
learning on time series and dynamical systems
Keywords:
multivariate time-series anomaly detection, shallow autoencoders, reconstruction-based anomaly detection, conditional relations, representation learning, principal component analysis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6Zc7OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates why a simple shallow autoencoder can perform competitively for multivariate time-series anomaly detection. The authors study whether reconstruction learning depends only on global first- and second-order statistics or also on higher-order joint information. They theoretically characterize when mean and covariance are sufficient for squared reconstruction objectives, and show that nonlinear ReLU autoencoders can depend on activation-conditioned statistics beyond global covariance. The paper further uses moment-matched interventions, fixed-PCA-subspace controls, and a synthetic relation-switching experiment to examine how changes in normal-variable relationships affect reconstruction geometry and anomaly scores. The main conclusion is that ShallowAE can exhibit relation-sensitive reconstruction behavior beyond global linear-subspace distance, although the detection benefit of such behavior is not consistent across real datasets.

Critical strengths

1.The paper addresses an important and relatively underexplored question: what information a nonlinear reconstruction model actually learns from normal multivariate time series. Compared with studies that mainly focus on benchmark performance, this work provides a more mechanistic analysis through theoretical characterization and controlled interventions.

  1. The controlled experiments are well aligned with the theoretical motivation. In particular, the moment matched interventions, fixed PCA subspace queries, and relation switching experiments directly test whether information beyond covariance changes reconstruction and anomaly scores. The large TSB-AD-M benchmark also shows that the simple baselines considered in the paper are practically competitive with recent methods.
Critical weaknesses

1.The connection between the controlled relation mechanism and the behavior observed on real data remains incomplete. The moment matched intervention shows that ShallowAE responds to information beyond the global mean and covariance, but this intervention can change several forms of higher order structure at the same time. The analytical relation switching experiment isolates a specific conditional relation, but it is conducted in a highly controlled synthetic setting. Therefore, the paper does not yet establish how much of the real data behavior is specifically explained by conditional normal relations rather than by other nonlinear joint structures.

2.The paper shows that relation-sensitive reconstruction can change anomaly scores, but it does not provide a clear criterion for when this behavior is beneficial or harmful. On real datasets, the effect can be either positive or negative. Although the paper provides qualitative explanations, it still lacks a quantitative condition that predicts whether the learned relations will remain valid at test time and improve anomaly separation.

Questions for authors

1.The moment-matched intervention preserves the global mean and covariance, but it may simultaneously alter many forms of higher-order structure. Can the authors further clarify to what extent the observed changes can specifically be attributed to normal-variable relations rather than other factors such as higher-order moments, local density, multimodality, or support geometry?

2.Have the authors analyzed the specific composition of the beyond-covariance information? For example, among higher-order statistics, local conditional statistics, and nonlinear cross-variable dependencies, which factors have the largest influence on ShallowAE’s reconstruction behavior?

3.The effect of relation-sensitive reconstruction is positive on some real datasets and negative on others. Can the authors provide a quantitative indicator or empirical criterion for predicting when this mechanism is likely to improve anomaly detection and when it may instead reduce performance?

4.The fixed-PCA-subspace experiments show that ShallowAE assigns different scores to different within-subspace combinations, but such selectivity does not necessarily imply that the model has learned stable normal relations. Can the authors further distinguish generalizable normal-relation learning from fitting to local support structure in the training distribution?

6 comments
AuthorsOpenReview ↗

Response to 6Zc7 (1/4): Real-data evidence and conditions for detection gains

Thank you for asking how the controlled mechanism relates to real data and when the additional selectivity helps detection.

W1. Connecting controlled conditional relations to real-data behavior

Training interventions, analytical relation switching, and real-event matching connect statistical responses, a conditional-relation mechanism, and natural-anomaly scoring.

  • Real-data training interventions: On 11 sequences from different dataset families, each sequence has the same effect sign under three moment-matching transforms. Signs vary across sequences, supporting a repeatable effect of information beyond second-order statistics on detection behavior.
  • Analytical relation switching: Reversing the conditional mean while preserving coordinate-wise marginals and full covariance changes both reconstruction relations and scoring preferences. This shows that a specified conditional relation suffices to produce additional selectivity.
  • Real-event matching: Matching normal and anomalous windows at similar PCA scores covers 88/223 real events. Additional selectivity improves anomaly rankings in some families and lowers them in SWaT, connecting the behavior to actual natural-anomaly scores.

The training interventions establish repeatable sensitivity to information beyond second-order statistics; the analytical construction identifies a conditional-relation mechanism; event matching links additional selectivity to changes in natural-anomaly ranking. The next comment (Q1-Q2) explains the statistical connection between conditional relations and other joint structures.

Manuscript changes: Section 3.2 and Section 4.7 explain the evidence roles. Appendices C, E, and D.2 provide training-intervention, relation-switching, and event-matching results, respectively.

W2. Quantitative conditions for beneficial relation-sensitive reconstruction

We address this through a score-separation condition and a model-selection rule, with theoretical and empirical support.

Revised Corollary 5 gives a geometric condition: if the normal-residual bound τ\tau, anomaly-to-normal-relation distance lower bound aa, and anomalous-output relation-deviation bound β\beta satisfy a>τ+βa>\tau+\beta, scores strictly separate. On sufficiently separated test queries from the original synthetic switching experiment, all 18 models across three seeds and six histories satisfy this condition and actually separate the classes.

The added selection experiment constructs within-subspace recombinations and off-subspace departures from normal data and compares ShallowAE and GWPCA on their discrimination. On 157 eligible sequences without labeled prefix anomalies and with computable indicators, mean VUS-PR is 0.44437, versus 0.43743 for fixed ShallowAE and 0.43177 for fixed GWPCA. The Q3 comment gives the rule and direction-prediction evaluation.

The geometric criterion connects score separation to normal reconstruction accuracy, anomaly-to-relation distance, and preservation of the relation by reconstructed outputs. The selection rule uses relative discrimination, assuming representative normal prefixes and controlled departures relevant to target anomalies. Held-out normal and Gaussian reference inputs further check normal compatibility and combination selectivity before internal constraints are enabled.

Manuscript changes: Section 3.4 and Appendix A.6 state and prove the geometric condition; Appendix E.4 and Table E.2 report finite-query checks. Section 4.6 and Table 4 report selection and fusion development comparisons. Appendices I.2-I.3 give selection and direction-evaluation methods; I.4 gives activation/fallback conditions. Section 4.7, paragraphs 2-3, links these criteria to detector design.

AuthorsOpenReview ↗

Response to 6Zc7 (2/4): Joint structure and quantitative risk contributions

Thank you for asking which information beyond covariance enters reconstruction and how its influence can be quantified.

Q1. Conditional relations versus other joint structures

Conditional relations, higher-order cross-moments, local density, multimodality, and support geometry are related descriptions of a joint distribution and can affect reconstruction through common regional statistics. Activation-conditioned moments describe this pathway, while the analytical construction identifies one conditional-relation mechanism.

Theorem 2 expresses a fixed ReLU reconstructor's squared risk through activation-region probability masses, conditional means, and conditional covariances. This specifies the statistics through which changes in joint structure enter the risk. Theorem 4 constructs two environments with identical coordinate-wise marginals and full covariance but opposite conditional means, with an exact excess risk of 1/31/3 for the ideal mismatched relation reconstructor at the specified representation.

The construction also makes the higher-order connection explicit. For U∼Unif⁡[−1,1]U\sim\operatorname{Unif}[-1,1], Vκ=κ(∣U∣−12)+ηV_\kappa=\kappa(|U|-\tfrac12)+\eta, κ∈{−1,+1}\kappa\in\{-1,+1\}, and independent zero-mean noise η\eta,

=\mathbb E[U^2\mathbb E(V_\kappa\mid U)] =\frac{\kappa}{12}.$$ The univariate marginals, including their higher moments, are identical, while the third-order cross-moment reverses with the conditional mean. Thus, conditional-relation change and joint higher-order statistical change describe the same mechanism in this construction. Real-data moment matching and fixed-subspace queries identify aggregate effects beyond second-order statistics and combination selectivity. The analytical construction specifies the relation change and its risk cost; bidirectional switching verifies that finite networks change reconstruction and scoring preferences with the training relation. **Manuscript changes:** Section 3.2 defines the information components and regional-statistics pathway; Section 3.4 connects conditional-mean reversal to relations and score separation. Section 4.7, paragraph 1, distinguishes aggregate real-data effects from analytical identification. Appendices D.1 and E.2 report the controls; A.5 gives the construction's statistics. The third-order cross-moment identity above is expanded in this response. **Q2. Information components and their relative influence** **For a fixed ReLU reconstructor, influence depends on activation-region probabilities, within-region means and covariances, and the residual map.** An activation region contains inputs with the same activation branches. In region $s$, write $r_\theta(x)=A_sx+b_s$ and $C_s=I-A_s$. With region probability, conditional mean, and conditional covariance $\pi_s,\mu_s,\Sigma_s$, Theorem 2 becomes $$\mathbb E_P S_\theta(X) =\sum_{s:\pi_s>0}\pi_s\left[ \operatorname{tr}(C_s\Sigma_sC_s^\top) +\|C_s\mu_s-b_s\|_2^2 \right].$$ Conditional covariance is weighted by $C_s$, the conditional mean contributes through $C_s\mu_s-b_s$, and regions are weighted by probability mass. Statistical changes of the same magnitude can therefore have different effects depending on their region and direction. This quantifies influence for a specified network, distribution, and intervention; the dominant component depends on that setting. The analytical construction above gives an exact risk cost for a specified relation change, and bidirectional switching tests the corresponding finite-network response. The temporal ablation additionally shows higher ShallowAE VUS-PR with original temporal order on 9/11 sequences, with a mean improvement of 0.04178, providing evidence for the detection value of temporal structure. **Manuscript changes:** Section 3.2 and Appendix A.3 specify the statistical components and weighting. Appendix H reports the temporal/channel ablations.
AuthorsOpenReview ↗

Response to 6Zc7 (3/4): Normal-data indicators and detection-gain direction

Thank you for asking for an indicator of when additional selectivity improves detection.

Q3. Predicting the direction of detection gains from normal data

We introduce a normal-data indicator of relative discrimination and test it through model selection and constraint activation.

Selection uses g=gR+gOg=g_R+g_O: choose ShallowAE when g>0g>0, otherwise GWPCA. The two controlled-query types change within-subspace combinations and introduce off-subspace departures. Here gRg_R and gOg_O are ShallowAE's mean AUC gains over GWPCA in distinguishing original normal samples from each controlled-query type. In the development comparison on 157 eligible sequences, this rule achieves mean VUS-PR 0.44437, versus 0.43743 for fixed ShallowAE and 0.43177 for fixed GWPCA. This supports average selection value on the evaluated sequences.

The sequence-level direction evaluation provides a separate test. On the 155 sequences with unequal model performance, fixed-threshold direction accuracy is 54.2% (84/155). The continuous indicator's direction-prediction AUROC is 0.57846, with a 95% interval of [0.41547, 0.75185]. Average selection gain and sequence-level direction prediction measure the policy's aggregate outcome and the indicator's discrimination of gain direction, respectively. Application assumes that normal prefixes represent the target regime and constructed departures are relevant to target anomalies.

The added fusion experiment also specifies when to enable an internal-combination constraint. It must accept at least 95% of held-out normal inputs and at most 95% of prespecified Gaussian reference inputs. This checks normal compatibility while requiring exclusion of some reference combinations. Both tests must pass; otherwise, scoring falls back to GWPCA. On 132 sequences meeting the normal-prefix block requirements, the complete fusion method achieves mean VUS-PR 0.45019, versus 0.44258 for GWPCA and 0.44409 for equal-weight raw-score fusion. This supports conditional use of combination constraints based on normal-data validation.

Manuscript changes: Section 4.6 outlines relative-discrimination selection and validated constraint fusion; Table 4 gives the complete comparisons. Appendix I.2 specifies query construction, indicator computation, and the zero-threshold rule. Appendix I.3 gives gain-direction evaluation methods; I.4 gives both 95% activation conditions, fallback, fusion scoring, and calibration.

Reviewer_w2zzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how shallow autoencoders use information beyond global means and covariances for multivariate time-series anomaly detection. Theoretical analysis and controlled experiments connect this information to reconstruction geometry and anomaly scores.

Critical strengths

The controlled experiments systematically examine which training information affects reconstruction learning and how these effects translate into changes in anomaly scores. The moment-matched interventions, fixed-subspace controls, and relation-switching experiments provide complementary evidence.

Critical weaknesses

The practical implications remain underdeveloped. While the paper demonstrates that information beyond covariance can affect detection, it offers limited guidance on when this information improves performance and how the findings can inform better anomaly detectors, particularly under changing operating conditions.

Questions for authors
  • Do the findings suggest concrete criteria for identifying normal relations that remain valid across operating states, or for deciding which relations to preserve versus adapt?
  • Much of the analysis also applies to tabular anomaly detection. Could ablations separating temporal dependencies from cross-channel dependencies clarify the insights specific to time series?
  • To what extent do the findings generalize to other nonlinear reconstruction models? Comparisons with additional architectures would help clarify the scope of the proposed mechanism.
  • The VUS-PR gap between ShallowAE and GWPCA is relatively small (0.456 vs. 0.448). How much of the strong benchmark performance is attributable to window representation and preprocessing rather than nonlinear reconstruction? Could you also clarify how ShallowAE and GWPCA differ from the AutoEncoder and PCA baselines in TSB-AD, and which implementation or configuration differences explain the much larger performance gaps between these corresponding methods?

I would be happy to raise my score if the authors provide more actionable insights and address the questions above.

5 comments
AuthorsOpenReview ↗

Response to w2zz (1/3): Practical design and constraint activation

Thank you for asking how the mechanism can guide detector design and constraint use under changing conditions.

W1. Practical guidance from the mechanism

We translate the mechanism into model selection and constraint fusion, both of which improve mean VUS-PR on the evaluated sequences. Selection constructs two types of queries from normal data: within-subspace recombinations and off-subspace departures. Their relative discrimination guides the choice between ShallowAE and GWPCA. Fusion retains off-subspace detection and adds internal-combination constraints after normal-data validation, combining the two capabilities.

EvaluationSequencesSame-protocol reference VUS-PRMechanism-guided VUS-PR
Selection157Fixed ShallowAE: 0.437430.44437
Fusion132Equal-weight raw-score fusion: 0.444090.45019

These are development comparisons on the inspected benchmark, with sequence-level decisions based on normal-prefix features. Selection requires prefixes without labeled anomalies and computable indicators; fusion requires prefixes without labeled anomalies and enough data for the development blocks. Selection assumes that the controlled departures are relevant to target anomalies. Fusion validates internal constraints on held-out normal data.

The resulting design rule is to select models by relative discrimination, check constraints for normal compatibility and combination selectivity, and combine validated constraints with off-subspace detection. The next response gives the retention and updating criteria.

Manuscript changes: Section 4.6 and Table 4 report the design and both development comparisons. Appendices I.1-I.4 give eligibility, query construction, selection rules, gain-direction evaluation, and fusion implementation. The final paragraph of Section 4.7 connects normal compatibility and controlled discrimination to model selection and complementary scoring.

Q1. Which relations should be retained or adjusted?

We assess the internal constraint as a whole by whether it accepts held-out normal data while excluding some other combinations. The fusion experiment requires acceptance of at least 95% of held-out normal inputs and at most 95% of prespecified Gaussian reference inputs. The latter checks that the constraint excludes some reference combinations. It is enabled only when both tests pass; otherwise, scoring falls back to GWPCA.

This criterion governs retention, fallback, and adjustment at the constraint-set level. A validated constraint is retained; a failed constraint triggers fallback, and confirmed normal data from a new regime can be used to rebuild the accepted combination set. The fusion experiment demonstrates executable constraint activation through normal-data validation. The original synthetic relation-switching experiment compares continued learning of the old relation with learning the new one and shows that reconstruction and scoring selectivity update with the training relation. These provide evidence for constraint-use decisions and for relation-dependent updating, respectively.

Manuscript changes: Section 4.6 outlines validation and fallback. Appendix I.4 specifies both 95% activation conditions and calibration, and describes rechecking coverage on confirmed normal segments, rebuilding sets, and recalibrating radii when necessary. Section 4.7, paragraphs 2-3, links constraint use to normal compatibility, anomaly discrimination, and complementary scoring.

AuthorsOpenReview ↗

Response to w2zz (2/3): Temporal/channel ablations and architecture tests

Thank you for suggesting separate tests of temporal/channel structure and additional reconstruction architectures.

Q2. Temporal versus cross-channel dependencies

We added ablations that separately permute temporal order and cross-channel pairing. Temporal ablation preserves each window's set of simultaneous multichannel observations. Retaining the original temporal order improves mean VUS-PR by 0.04178 for ShallowAE and 0.02403 for GWPCA, showing the detection value of ordered temporal organization.

Channel ablation preserves each channel's complete lag vectors while breaking their pairing across channels. For original pairing minus permuted pairing, ShallowAE's mean VUS-PR difference is -0.00792: original pairing scores higher on 9/11 sequences, while larger reverse effects occur on the other two. Thus, the detection effect of cross-channel pairing varies across sequences. Appendix H also provides training-order diagnostics for these cases.

The temporal ablation tests ordered placement in time; the channel ablation tests cross-variable pairing. These experiments examine information organization in time-series windows and complement the moment-matched experiments: temporal/channel ablations measure the overall effects of changing those structures and may change some covariances, whereas moment matching isolates information beyond global second-order statistics. This connects the general joint-distribution theory to specific dependencies in time series.

Manuscript changes: Appendix H reports the ablations, their time-series interpretation, and diagnostic controls.

Q3. Scope across nonlinear reconstruction architectures

We added DeepAE, GELU-AE, and a ReLU ConvAE, varying depth, activation, and temporal local connectivity. Each is tested for learning responses beyond second-order statistics, discrimination between combinations at equal PCA residuals, and updating of reconstruction and scoring under a changed normal relation.

All three added models exhibit learning and scoring responses to information beyond global second-order statistics. In the added switching experiment, all architectures learn the same two normal relations, allowing us to test whether changing the training relation redirects reconstruction and scoring toward the new relation. All three change in that direction for both switches, with switched AUROC of 0.907-0.964; the linear-AE control remains near 0.500. This supports the mechanism in the tested deep fully connected, smooth-activation, and convolutional reconstructors.

Theorem 2 covers DeepAE and ReLU ConvAE through their piecewise-affine structure. The statistical criterion for GELU-AE follows from Theorem 1.

Manuscript changes: Section 4.5, Table 3, and Appendix G report the architecture evidence. Section 3.2, Appendix A.3, and the Conclusion clarify the theoretical and empirical scope.

AuthorsOpenReview ↗

Response to w2zz (3/3): Performance sources and baseline implementations

Thank you for connecting performance attribution with differences between the benchmark implementations.

Q4. Window representation, preprocessing, nonlinearity, and TSB-AD baselines

We examine window length, activation, preprocessing, and scoring through single-factor comparisons, a scaling comparison, and implementation checks. Across the 180 benchmark sequences, ShallowAE wins on 96, GWPCA on 81, and 3 are numerical ties. Both use window representations. GWPCA captures global second-order structure to detect off-subspace departures, while ShallowAE adds selectivity over within-subspace combinations. This can help when anomalies violate valid normal relations; it can also raise normal scores or preserve low anomaly scores when test inputs do not align with that assumption. Its benefit therefore varies across settings.

Both longer windows and nonlinear activation improve mean VUS-PR within the parameter-sensitivity protocol. Appendix B.2 reports:

Controlled changeMean VUS-PRApproximate gain
Linear activation to ReLU, otherwise fixed0.450 to 0.470+0.020
Window length 25 to 100, otherwise fixed0.387 to 0.470+0.083

These comparisons support contributions from temporal information supplied by longer windows and from nonlinear reconstruction at a fixed window length.

We also compared channel-wise Z-score and Min-Max scaling, fitting both on the training prefixes. On the same 180 sequences, mean VUS-PR is 0.4561 and 0.4509, respectively, a difference of about 0.0052. Their aggregate performance is similar.

The key implementation differences are:

  • ShallowAE versus TSB-AD AutoEncoder: ShallowAE reconstructs multichannel temporal windows; the latter reconstructs a channel vector from one timestamp.
  • GWPCA versus TSB-AD PCA: Both use windows. GWPCA scores principal-subspace projection residuals, whereas TSB-AD PCA uses variance-weighted distances to principal-component vectors.

Together, these comparisons explain performance at two levels: single-factor experiments show average gains from window length and nonlinear activation, and the scaling comparison shows similar aggregate performance for the two scalers. Moment-matched training and fixed-subspace queries identify the additional learning responses and within-subspace combination selectivity available to nonlinear reconstruction.

Evidence locations: Appendix B.1 reports the complete benchmark ranking; Appendix B.2 gives the parameter comparisons. Sections 4.1-4.4 provide the training-intervention, subspace-query, and relation-switching mechanism evidence. The head-to-head counts, scaling comparison, and implementation distinctions are additionally explained in this response.

Reviewer_173gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper characterizes multivariate time-series anomaly detection by comparing ShallowAE and GWPCA. Using moment-matched training interventions and analytical constructions, the authors separate sensitivity to training information from global second-order statistics. The theoretical and experimental evidence identifies that ShallowAE relies on activation-conditioned moments to learn normal conditional relations. This produces within-subspace score selectivity, allowing the model to assign different scores to inputs at equal PCA residuals. Overall, the results connect training statistics to reconstruction geometry and anomaly separation, while also noting that relations specific to the training period can sometimes reduce separation.

Critical strengths
  1. Separating Sensitivity via Moment-Matched Interventions: The paper elegantly uses orthogonal moment-matched training interventions to separate sensitivity to training information from global second-order statistics. Combined with Theorem 2, it rigorously proves that reconstruction learning depends on ``activation-conditioned moments'', providing a clear statistical characterization of why nonlinear ShallowAE differs from linear GWPCA.

  2. Validation of Within-Subspace Score Selectivity and Residual Geometry: The study reveals ShallowAE's ``within-subspace score selectivity'' on real data. Through the analytical relation-switching experiment (P+P_+ vs. P−P_-), it successfully demonstrates how learned normal relations redirect "residual geometry" and reverse score preferences, directly linking training statistics to anomaly separation.

Critical weaknesses
  1. Dependence on Piecewise-Affine Structure (Theorem 2):The statistical characterization relies strictly on the piecewise-affine structure of ReLU networks. This is consequential to my assessment because it remains unclear whether the ``activation-conditioned moments'' framework generalizes to deep state-of-the-art architectures (e.g., Transformers) or non-linear activations, which slightly limits the broader applicability of the theory.

  2. Strong Assumption on Anomaly Approximation (Corollary 5):The condition for strict score separation assumes a uniform reconstruction approximation bound (ϵ\epsilon) over a query domain containing unseen anomalies. This is consequential because anomalies are inherently out-of-distribution, making this bound difficult to guarantee in practice and slightly weakening the realism of this specific theoretical claim?

  3. Unmitigated Transient Normal Relations (Section 4.5): The paper notes that selectivity over joint combinations can penalize new normal inputs when training relations are transient (e.g., GHL052). This is consequential because the lack of discussion on algorithmic mitigations (e.g., adaptation strategies) leaves a minor gap for the practical deployment of this mechanism in changing environments?

Questions for authors

For details, please refer to ``Critical Weaknesses''.

3 comments
AuthorsOpenReview ↗

Response to 173g (1/2): Theoretical scope and added architectures

Thank you for asking us to clarify the theoretical scope and test additional architectures.

W1. Network depth, activation functions, and Transformers

Theorem 2 requires piecewise-affine structure and is not restricted to shallow networks. We have also added mechanism experiments varying depth, activation, and convolutional structure. The revision distinguishes two cases:

  1. ReLU/Leaky-ReLU networks of any finite depth. Once activation branches are fixed, a finite composition of affine maps remains affine, giving rθ(x)=Asx+bsr_\theta(x)=A_sx+b_s within each region. Depth changes the partition and its coefficients, while preserving the risk decomposition. Convolution is affine with shared weights. Our added ReLU ConvAE uses fixed-size nearest-neighbor upsampling, which is linear, and therefore satisfies this condition. Appendix A.3 now explains the argument; this structural property is consistent with Balestriero and Baraniuk [1].

  2. Smooth activations and standard Transformers. For GELU, tanh, or standard Transformers with Softmax attention and input-dependent normalization, Theorem 1 still provides a statistical criterion: under its domain and moment conditions, a squared reconstruction score that is not globally quadratic admits distributions with identical means and covariances but different expected scores. This criterion does not require piecewise-affine structure; Theorem 2's finite regional-moment formula does.

We added a deeper ReLU autoencoder (DeepAE), a GELU autoencoder (GELU-AE), and a one-dimensional convolutional autoencoder (ConvAE). For each, we test whether information beyond second-order statistics changes learning, whether scores distinguish combinations at equal PCA residuals, and whether changing the normal relation redirects reconstruction and scoring preferences. All three added models exhibit these responses, supporting the mechanism in the tested deep, smooth-activation, and convolutional reconstructors.

DeepAE and ReLU ConvAE fall within Theorem 2's structural scope. GELU-AE provides empirical evidence with a smooth activation, with its statistical criterion supplied by Theorem 1.

Manuscript changes: Section 3.2 and Appendix A.3 clarify the scope. Section 4.5 and Table 3 add the architecture comparison; Appendices G.1-G.4 provide architectures, protocols, learning-signal comparisons under matched means and covariances, subspace scoring, and bidirectional relation switching.

[1] Balestriero, R., and Baraniuk, R. G. A Spline Theory of Deep Learning. ICML 2018, pp. 374-383. https://proceedings.mlr.press/v80/balestriero18b.html

AuthorsOpenReview ↗

Response to 173g (2/2): Geometric separation and changing normal relations

Thank you for raising the approximation assumption and deployment under changing normal relations.

W2. Uniform approximation over unseen anomalous inputs

We replaced the vector-approximation assumption in Corollary 5 with a relation-set condition that permits reconstructed outputs to move along the normal relation. For a predetermined normal relation set, let τ\tau bound normal reconstruction residuals, aa lower-bound anomaly-to-relation distance, and β\beta bound the distance of anomalous reconstructions to the relation. Then a>τ+βa>\tau+\beta guarantees strict score separation. The condition characterizes reconstruction behavior through output-to-set distance.

We checked the new condition using test queries from the sufficiently separated region of the original synthetic relation-switching experiment. All 18 models across three seeds and six training histories satisfy the condition and actually separate the classes, with a minimum margin of 0.0183. This establishes the correspondence between the geometric condition and score separation on the checked finite test queries.

Manuscript changes: Section 3.4 replaces Corollary 5; Appendix A.6 proves it, and Appendix E.4 reports the finite-query checks.

W3. Transient normal relations and changing operating conditions

We make the mitigation concrete through validation, fallback, and updating of internal-combination constraints. The added ShallowAE-GWPCA fusion retains off-subspace detection. It enables an internal constraint when that constraint accommodates held-out normal data while excluding some other combinations; otherwise, it falls back to GWPCA. On 132 eligible sequences, mean VUS-PR is 0.45019 versus 0.44409 for equal-weight raw-score fusion. This supports conditional use of internal constraints.

The original synthetic experiment also tests whether selectivity can update with the relation: from a shared model, one branch continues learning the old relation and the other learns the new relation. Evaluated with the new relation as normal, the updated reconstructions move toward it. In the sufficiently separated region, AUROC changes from 0.000 for the old-relation continuation to 1.000 for the updated branch in both directions.

For confirmed normal data from a new regime, constraint acceptance can therefore be rechecked, with the internal set rebuilt and thresholds recalibrated when needed. Normal acceptance and anomaly discrimination should both be evaluated.

Manuscript changes: Section 4.6 outlines validation and fallback; Table 4 reports the development comparison. Appendix I.4 gives activation thresholds, fusion scoring, calibration, and rechecking/rebuilding under a new regime. Appendix E.2 compares retained and switched branches under the same switch-target environment.

Reviewer_173gOpenReview ↗

Thanks for the response

Thanks for the response. I will maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Author Response: Summary of Revisions

We thank the reviewers and the Area Chair for their careful reading and constructive suggestions. The revision strengthens the connection between reconstruction theory, controlled mechanism evidence, and detector design in three ways.

  1. Theoretical scope and score separation. We clarify the scope of the activation-conditioned moment decomposition across depth, convolution, and activation functions. Revised Corollary 5 replaces the uniform vector-approximation assumption with a relation-set condition that allows reconstructed outputs to move along the normal relation. We also check this condition on finite test queries for all 18 relation-switching endpoints.

  2. Additional empirical evidence. We add DeepAE, GELU-AE, and ConvAE comparisons, separate temporal-order and cross-channel-pairing ablations, and real-data normal-coverage interventions with an independently validated relation reference and nearest-neighbor controls. These experiments examine architecture scope, time-series information, and relation preservation beyond training coverage.

  3. Practical detector design. We introduce mechanism-guided model selection and fusion with internal-constraint validation and fallback to GWPCA. We report their detection gains as development comparisons and specify the corresponding eligibility and calibration protocols.

The revised discussion connects normal compatibility, anomaly discrimination, and complementary scoring. Substantive additions and changes are marked in blue; wording-only edits are left unmarked. Our accompanying reviewer-specific responses follow the W/Q numbering and identify the corresponding manuscript locations.

We have updated our anonymous repository with the code and results for the additional experiments reported in this rebuttal. These materials are available at our anonymous repository.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.