acceptodds
Under review as a conference paper at ICLR 2027

FIRE: Learning Identifiable Operators for Multi-Environment Generalization

Abstract

Multi-environment data mix shared subject structure with environment-specific distortion. Domain generalization targets robust prediction and harmonization removes environment effects; neither is designed to identify the shared latent directions. Factored Identifiable Representation via Environments (FIRE) models each environment's observations as unknown full-rank linear transforms of a common subject representation. Stage 1 decomposes a reference environment's third cumulant; Stage 2 freezes the corresponding recovered rank-one operators during supervised prediction. Independent loadings with nonzero third cumulants and noise with zero third cumulant identify directions up to sign, permutation, and reference coordinates from one environment, without pairing. Under sub-Gaussian tails, a whitening-free estimator attains error up to a conditioning factor once ; without structural loading constraints, identification is impossible. Paired cross-environment second moments cancel independent noise. Synthetic recovery follows scaling, tolerates subject-dependent maps and poor conditioning, and degrades under nonlinear or spatially heterogeneous maps. Updating operators raises recovery error from to ; freezing preserves geometry without increasing prediction loss. Against ERM and six domain-generalization methods using identical losses, features, and splits, FIRE has the highest mean out-of-distribution accuracy on four of five benchmarks, with gains modest relative to seed variability. Random frozen directions predict equally well on Camelyon17. Spectral recovery supplies identifiable structure: its TCGA-BRCA directions separate PAM50 subtypes more strongly than random or PCA directions.

Primary area:
interpretability and explainable AI
Keywords:
identifiability, tensor decomposition, multi-environment learning, operator learning, out-of-distribution generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vFS9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to learn identifiable latent directions from multi-environment data and use them for supervised prediction. The authors propose a two-stage framework called FIRE. In Stage 1, latent directions are recovered by decomposing the third cumulant of a reference environment under a linear mixing model with independent, nonzero-skew loadings. In Stage 2, rank-one operators constructed from these directions are frozen while the encoder and prediction head are trained. The paper analyzes reference-coordinate identifiability and provides conditional finite-sample guarantees for a whitening-free estimator. Synthetic and biomedical experiments examine direction recovery, the effect of freezing, and out-of-distribution prediction, reporting modest predictive gains and stronger subtype separation in TCGA-BRCA.

Critical strengths
  1. The paper gives a clear theoretical formulation of the recovery problem. It establishes reference-coordinate identifiability under explicit loading assumptions and provides a conditional finite-sample analysis for a whitening-free estimator. These results give a concrete target and mathematical basis for evaluating structural recovery, which supports the technical value of the work.

  2. The TCGA-BRCA experiments provide some evidence for the usefulness of the recovered directions. Their stronger separation of PAM50 subtypes compared with random and raw PCA directions suggests that spectral recovery can capture task-relevant structure. This gives empirical support to the potential value of the recovery stage.

Critical weaknesses
  1. The connection between the recovery guarantee and the evaluated procedure is not sufficiently established. Proposition 2 analyzes a whitening-free estimator, whereas the experiments use covariance whitening and tensor power iteration. Whitening with observation noise can leave asymptotic bias, so the guarantee does not generally transfer to the implemented pipeline. This leaves uncertainty about whether the frozen operators reliably represent the identifiable directions, weakening the theoretical support for the practical method.
  2. The contribution of the recovered structure to prediction is not sufficiently isolated. The operators act after a trainable encoder, so freezing their stored geometry does not by itself preserve their alignment with the input features. Moreover, FIRE uses a deeper encoder than the main baselines, and spectral and random frozen directions have the same reported mean accuracy on Camelyon17. These results make the practical benefit of the identifiability-based predictor unclear.
Questions for authors
  1. In one existing noisy synthetic setting, how does the current whitening procedure compare with noise-corrected whitening or the analyzed whitening-free estimator in recovery error?
  2. What is the intended role of the recovered directions in the benchmark predictor, given the trainable encoder and random subject loadings? What evidence supports this role?
  3. Can you compare FIRE with ERM and random frozen operators on one existing benchmark, matching the encoder architecture and, where applicable, the operator initialization scale?
7 comments
AuthorsOpenReview ↗

Response to W1 (1/2)

Proposition 2 analyzes whitening-free recovery. Theorem 4 covers signal-covariance whitening under its conditions (§3.4, p. 5). Benchmark whitening can retain noise as sample size grows (Remark 3, p. 19). We therefore establish a conditional link between the implemented solver's recovered directions and the frozen operators used in prediction. This link is based on a bound for the directions returned by the whitening/tensor-power procedure.

Let π\pi match a returned unit direction b^j\hat b_j to a true unit direction vπ(j)v_{\pi(j)}. The projector error in spectral norm equals the sine of the angle between their lines.

= \sin \angle(\hat b_j, v_{\pi(j)}).$$ **Geometry and identification.** For $K$ latent directions $b_k$ in $d$ dimensions and reference map $A_{\mathrm{ref}}$, write $c_k=A_{\mathrm{ref}}b_k$ and $v_k=c_k/\|c_k\|$. Proposition 1 assumes a full-rank map, linearly independent directions, centered independent loadings with finite nonzero third cumulants $\mu_{3,k}$, and independent zero-third-cumulant noise. With weights $\rho_k$, the population third cumulant $\mathcal T$ satisfies $$\mathcal T=\sum_{k=1}^{K}\rho_k v_k^{\otimes3},\qquad \rho_k=\mu_{3,k}\|c_k\|^3.$$ This identifies reference lines up to sign and permutation. Latent lines also require the map. Independent Gaussian noise leaves the population third cumulant unchanged but changes the whitening covariance. Let $C=[c_1,\ldots,c_K]$ and write $I_m$ for the identity of size $m$. With unit-variance loadings and isotropic noise variance $\nu>0$, let $W\in\mathbb R^{d\times K}$ whiten $CC^\top+\nu I_d$ on the full signal subspace. The reference and whitened Gram matrices $G$ and $J$ satisfy $$J=(W^\top C)^\top(W^\top C) =G(G+\nu I_K)^{-1},\qquad G=C^\top C.$$ Since $G=\nu J(I_K-J)^{-1}$, whitened factors are orthogonal exactly when reference factors are orthogonal. For orthogonal reference factors, exact decomposition and unwhitening recover their lines when all $K$ factors are retained. **Guarantee scopes.** Proposition 2 assumes known $K$, i.i.d. jointly sub-Gaussian observations, two independent random unit contractions independent of the data, nonzero second-contraction coefficients, separated ratios and small perturbations. The pencil requires no signal-covariance estimate. In Theorem 4 (App. C, Eq. 8), signal whitening makes the variance-rescaled factors orthonormal. Its bound requires covariance and tensor errors below threshold and prescribed robust tensor-power iterations and restarts. Benchmark whitening uses reference auto-covariance. Paired whitening uses auto-covariance in a cross-moment subspace. Both can retain noise. Centered noise independent across environments and from loadings cancels in population cross-moments. Recovering reference signal covariance also requires alignment, as with identity maps. We therefore check the finite-budget solver’s outputs.
AuthorsOpenReview ↗

Response to W1 (2/2)

Conditional error bound. Write T,T^\mathcal T,\widehat{\mathcal T} for population and empirical tensors. For fitted weights τi\tau_i and returned directions b^i\hat b_i, fix T~=∑i=1Kτib^i⊗3\widetilde{\mathcal T}=\sum_{i=1}^K\tau_i\hat b_i^{\otimes3}. Assume population and candidate have rank K≥2K\geq2, linearly independent real unit directions in the same full reference coordinates and nonzero real weights of either sign.

Let F=[v1,…,vK]F=[v_1,\ldots,v_K] and B=[b^1,…,b^K]B=[\hat b_1,\ldots,\hat b_K] collect true and returned directions. Let UU be an orthonormal basis of the returned span, P=UU⊤P=UU^\top and H=U⊤FH=U^\top F. For ℓ=1,2\ell=1,2 and unit wℓw_\ell, set Mℓ=T(I,I,wℓ)M_\ell=\mathcal T(I,I,w_\ell), Nℓ=T^(I,I,wℓ)N_\ell=\widehat{\mathcal T}(I,I,w_\ell) and Rℓ=T~(I,I,wℓ)R_\ell=\widetilde{\mathcal T}(I,I,w_\ell). Require s=σK(M2)>0s=\sigma_K(M_2)>0 and invertible D=U⊤R2UD=U^\top R_2U. In matrix spectral norm, define

η=∥(Id−P)(R2−N2)∥2+∥(Id−P)(N2−M2)∥2s.\eta=\frac{∥(I_d-P)(R_2-N_2)∥_2+∥(I_d-P)(N_2-M_2)∥_2}{s}.

dℓ=∥U⊤(Rℓ−Nℓ)U∥2+∥U⊤(Nℓ−Mℓ)U∥2,ℓ=1,2.d_\ell=∥U^\top(R_\ell-N_\ell)U∥_2+∥U^\top(N_\ell-M_\ell)U∥_2,\quad\ell=1,2.

Thus η\eta bounds span mismatch and dℓd_\ell within-span error, combining fit and sampling errors. For η<1\eta<1, HH and U⊤M2UU^\top M_2U are invertible. Set Z=(U⊤M1U)(U⊤M2U)−1Z=(U^\top M_1U)(U^\top M_2U)^{-1} and let g>0g>0 be the minimum gap between ratios ⟨vi,w1⟩/⟨vi,w2⟩\langle v_i,w_1\rangle/\langle v_i,w_2\rangle. Rank ensures nonzero denominators. Numerators may vanish.

Let QQ denote the conditioning-adjusted perturbation bound for the recovery pencil, defined by

\qquad \operatorname{cond}_2(H)=\|H\|_2\|H^{-1}\|_2.$$ Here $d_1+\|Z\|_2d_2$ combines contraction errors, $\|D^{-1}\|_2$ measures amplification by inversion, and $\operatorname{cond}_2(H)$ accounts for projected factor geometry. Given the rank and invertibility conditions, $\eta<1$ and $Q<g/2$ yield a unique eigenvalue-based matching $\pi$ satisfying $$\max_j\sin\angle(\hat b_j,v_{\pi(j)}) \leq\beta:=\sqrt{\eta^2+(1-\eta^2)\left(\frac{Q}{g-Q}\right)^2}<1.$$ The projector identity then gives $\|\hat b_j\hat b_j^\top-v_{\pi(j)}v_{\pi(j)}^\top\|_2\leq\beta$. **Proof.** The second-contraction residual bounds the true span’s component outside the returned span by $\eta$. The columns of $H$ are the population pencil’s eigenvectors, and the contraction ratios are its eigenvalues. Because $Q$ bounds the conditioning-adjusted perturbation of the candidate pencil, each disjoint radius-$Q$ disk retains one eigenvalue along the straight perturbation path. In the $H$ basis, the residual in the other components is divided by a gap of at least $g-Q$. Transforming back gives within-span sine error at most $Q/(g-Q)$, since $Q$ already includes the conditioning of $H$. Combining the orthogonal errors gives $\beta$. Optimal permutation matching can only improve this bound. **Evaluation.** We fit $\tau_i$ to the full empirical reference tensor with directions fixed. Unit normalization transfers each saved column’s cubed norm to its weight, preserving the tensor. At $K=3$, we normalized the unique returned-span vectors with inner products $(-1,0,1)$ and $(1,1,1)$ against the saved columns of $B$. Column signs and order stayed fixed, without pair search or direction optimization. All 25 fits (five seeds per geometry) used unchanged settings, $N=60{,}000$ observations, $d=20$, $K=3$, exponential loadings and independent Gaussian noise of SD 1. Bounds use eigenvalue matching for passing outputs. Observed maximum sine errors use optimal permutation matching for all seeds. | Common factor cosine | Passing outputs | Sine-error bound | Observed maximum sine error | | --- | --- | --- | --- | | 0 | 5/5 | 0.172–0.380 | 0.0289–0.0389 | | 0.1 | 2/5 | 0.7124–0.7217 | 0.0544–0.0923 | | 0.3 | 0/5 | No bound | 0.2133–0.2651 | | 0.5 | 0/5 | No bound | 0.4755–0.5929 | | 0.7 | 0/5 | No bound | 0.8839–0.9190 | This pointwise synthetic assessment uses known population moments and realized errors at candidate-dependent contractions. It permits exponential loadings, outside Proposition 2's sub-Gaussian assumption. The bounds were loose where they applied. No run passed at higher cosines. At cosine 0.7, every observed worst-direction sine error exceeded 0.88. For each matched component, the conditional bound on direction error also bounds the error of the frozen rank-one operator used in prediction. With observation noise retained in whitening, this bound applied to all five orthogonal runs and two of five runs at factor cosine 0.1. **The bound applies under identifiable conditions, but its applicability becomes limited as factor correlation increases.**
AuthorsOpenReview ↗

Response to Q1: Current, Noise-Corrected and Whitening-Free Recovery

We compared current covariance whitening, oracle noise-corrected whitening, and the whitening-free pencil on the same 6,000 centered environment-0 observations. With K=3K=3 orthonormal factors and Gaussian noise of standard deviation (SD) σ=1\sigma=1, current whitening closely matched the oracle and had lower mean recovery error than the pencil. Mean errors were 0.108 ± 0.037 (current), 0.106 ± 0.036 (oracle), and 0.714 ± 0.408 (pencil). The pencil required no complex-eigenvector fallback in this setting.

The simulations used 20 features, independent exponential loadings, and independent Gaussian noise. The current solver whitens using the reference auto-covariance, applies tensor power iteration, and unwhitens. The oracle first subtracts the known synthetic noise covariance σ2I\sigma^2 I (II is the identity), which is unavailable on benchmarks. The whitening-free estimator uses a rank-KK restricted third-tensor contraction pencil and lifts its eigenvectors to feature coordinates. Each seed used one prespecified pair of normalized independent Gaussian contraction vectors drawn independently of the data.

Table 9 (p. 23) compared paired recovery, single-environment whitening, and FastICA using greedy matching (Appendix I.5). Here, each seed's error is the largest Euclidean distance between estimated and true unit directions after optimal sign and permutation matching, ranging from 0 to 2\sqrt{2}. W1's bound instead uses the sine of the angle between directions. All outputs were included, and results are reported as mean ± sample SD over five seeds.

We further varied the common pairwise factor cosine using 60,000 observations, holding noise SD at 1 and total signal power fixed.

Common pairwise cosineCurrent whiteningOracle correctionWhitening-free pencil
00.031 ± 0.0040.032 ± 0.0040.485 ± 0.510
0.10.073 ± 0.0140.032 ± 0.0040.486 ± 0.470
0.30.236 ± 0.0200.033 ± 0.0030.680 ± 0.551
0.50.536 ± 0.0590.035 ± 0.0030.529 ± 0.407
0.71.071 ± 0.0290.042 ± 0.0050.429 ± 0.282

Both whitening methods met the stopping criterion for every component in both experiments. In the correlation sweep, the pencil used normalized real parts of complex eigenvectors in one of five runs at each cosine of 0.3, 0.5, and 0.7. Runs at cosines 0 and 0.1 used real eigenvectors.

These finite-sample comparisons use exponential loadings, outside Proposition 2's sub-Gaussian assumption. With unit-variance loadings, nonzero isotropic noise, and the full signal subspace, population auto-covariance whitening preserves orthogonality exactly when the reference factors are orthogonal. In that case, exact decomposition and unwhitening recover their lines.

The results show that the effect of noise retained in whitening depends on factor geometry. Current whitening closely matched oracle correction for orthogonal factors, but their recovery errors diverged as the factors became more aligned.

Reviewer_mCpfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses domain generalization by identifying shared latent directions across environments while accounting for environment-specific distortions. The authors propose Factored Identifiable Representation via Environments (FIRE), a two-stage framework that recovers identifiable latent directions through third-order cumulant decomposition and incorporates them as frozen rank-one operators into a supervised predictor. Theoretical analysis establishes identifiability under specific assumptions on latent loadings, while freezing the recovered operators preserves their geometry during training. Experiments on synthetic and real-world benchmarks demonstrate competitive OOD generalization performance and improved latent direction recovery.

Critical strengths
  1. The proposed method achieves competitive performance across multiple DG benchmarks, including medical datasets. The ablation study further demonstrates the effectiveness of freezing the recovered directions.

  2. Identifying latent directions without ground-truth factor annotations is an interesting idea with potential real-world applications.

Critical weaknesses
  1. Limited validation of the biological interpretability of recovered directions. It remains unclear whether the recovered latent directions provide meaningful biological interpretations. Additional analysis showing how these directions relate to disease-associated factors would help establish their practical value, as Figure 3 does not reflect the final recovery method.

  2. Unclear connection between Stage 1 recovery and Stage 2 prediction. While Stage 1 recovers latent directions based on the theoretical formulation, the benchmark implementation randomly generates subject-specific loadings in Stage 2 and uses ridge regression to predict them for unseen subjects. It remains unclear how these random loadings relate to the recovered latent directions and whether the recovered structure meaningfully contributes to prediction.

Questions for authors

Please address the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to W1/ Q1: Limited validation of the biological interpretability of recovered directions

Figure 3 used cross-environment whitening on unpaired patients and therefore does not validate the final recovery method (p. 8). Table 3 presents a separate single-environment Fisher analysis of subtype separation and site alignment (§5.3, p. 9). The final method estimates the whitening covariance and third moment at a single reference site (§4, p. 6). Table 9 (p. 23) evaluates recovery against known synthetic directions. Across five seeds, the mean worst-direction error after sign alignment and greedy matching increased from 0.029 ± 0.003 at noise σ = 0.1 to 0.104 ± 0.014 at σ = 1 and 1.124 ± 0.157 at σ = 4.

To directly assess the biological relevance of recovered directions, we added two analyses using held-out patients from The Cancer Genome Atlas breast cancer cohort (TCGA-BRCA). First, we reconstructed each saved projection score as a weighted sum of measured log-expression values plus a centering constant, without using latent loadings or Gaussian predictor coefficients. Second, we evaluated the same directions for PAM50 subtype separation and associations with independently measured receptor status.

We used a subtype-stratified split of 632 training and 158 evaluation patients (790 total; one primary-tumour profile per patient). Training-only preprocessing defined a shared principal component analysis (PCA) space. Each method supplied three normalized axes, matching the predictor's direction count. Spectral recovery and independent component analysis (ICA) used the same reference site.

PAM50 subtype separation. Because PAM50 labels are expression-derived, this analysis provides an internal check. F averages the sample-weighted ratios of between-subtype to within-subtype variance across axes, with higher values indicating stronger separation. Bootstrap standard deviations (SDs) were estimated by resampling evaluation patients within subtype-by-site groups, holding the split and fitted models fixed.

MethodSubtype FBootstrap SD
Spectral0.07120.0215
ICA0.07190.0216
PCA0.96980.0989
Random0.14180.0284

Spectral separation was similar to ICA but weaker than PCA and random directions.

Receptor-status associations. We tested immunohistochemistry-measured estrogen receptor (ER; primary endpoint), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER2; secondary endpoints). For each endpoint, all methods used the same linked patients after excluding missing, indeterminate, and equivocal calls. T averages site-adjusted receptor-status partial R² across frozen axes. Within-site label permutations assume exchangeability under the null, and Holm adjustment covers all method-by-endpoint tests.

EndpointEvaluation patientsSpectral THolm-adjusted p
ER (primary)1470.013670.9306
PR1470.009331.0000
HER21110.009791.0000

No spectral association reached Holm-adjusted p < 0.05. PCA reached this threshold for all three receptors, while fixed random directions did so for ER and PR. Differences in significance alone do not establish method superiority.

These results distinguish mathematical identifiability and gene-level traceability from demonstrated biological relevance. Each recovered direction allows an individual patient's projection score to be traced to the weighted expression of measured genes. However, the current analyses do not establish associations with the examined disease-associated phenotypes. Such correspondence is not required by the identifiability assumptions. We will report these analyses and clarify this distinction in the revision.

AuthorsOpenReview ↗

Response to W2/Q2: Unclear connection between Stage 1 recovery and Stage 2 prediction

Stage 1 recovers latent directions; Stage 2 incorporates their frozen rank-one operators into a supervised predictor. For each subject, a coefficient scales the projection of the encoder output along a recovered unit direction (§3.3, Eq. 5, pp. 4–5). In the submitted benchmarks, each training subject receives a Gaussian coefficient vector sampled once and held fixed. Ridge regression maps source-training features to these draws and predicts coefficients for unseen subjects. These coefficients are auxiliary inputs, not estimates of individual latent loadings. The encoder and head remain trainable. Under the model assumptions, recovered reference directions are identifiable up to sign and permutation. We examined direction preservation under freezing, spectral versus random frozen operators, and held-out accuracy changes when the operator branch was bypassed.

Direction preservation. In synthetic regression with known generating loadings (Table 2, p. 8), frozen and trainable models achieved the same mean final-epoch training MSE (0.0102 across five seeds). Freezing better preserved the stored directions (mean errors 0.036 versus 1.075). Error was the maximum sign-adjusted Euclidean distance between normalized stored right factors and normalized generating directions after greedy absolute-cosine matching. Thus, in this synthetic setting, freezing preserved recovered geometry without improving training fit.

Prediction under Gaussian/ridge coefficients. On Camelyon17, controls matched encoder/head architecture, initialization, splits, feature dimension, tuning, and, where applicable, operator count and scale. All preprocessing, training, and validation were source-only. Results are mean held-out patch accuracy ± sample SD across five paired seeds, weighting the five held-out hospitals equally.

Camelyon17 modelHeld-out accuracy
No operator0.8616 ± 0.0063
Spectral frozen0.8634 ± 0.0032
Random frozen0.8625 ± 0.0055

Under the submitted Gaussian/ridge policy, spectral and random frozen operators achieved similar mean accuracy: the paired spectral-minus-random difference was +0.083 ± 0.286 percentage points (mean ± sample SD). Holding the encoder, head, coefficients, and examples fixed, the intact-minus-bypass accuracy difference for the recovered branch was −0.0040 percentage points (95% paired-t interval [−0.0119, +0.0039]). Thus, bypassing the recovered branch had negligible impact on held-out accuracy under this coefficient policy.

Exploratory fitted-LS comparison. We also fitted regularized least-squares (LS) coefficients to measured reference features in each consistently signed dictionary, matching Gaussian channel means and SDs on source-training data. At d = 50 and K = 3, Rotated-LS used one prespecified, data-independent orthogonal rotation per fold, preserving norms, the Gram matrix, and conditioning. The LS models were trained separately with matched encoder/head architecture, paired initializations and splits, training budgets, and source-validation learning-rate grids and selection rules. Gaussian/ridge retraining shared the Spectral-LS dictionary, paired initialization and splits, and validation budget.

The intact-minus-bypass accuracy difference was +0.8512 percentage points for Spectral-LS versus −0.0096 for Rotated-LS. Bypassing Rotated-LS changed the predicted class for 1.03% of held-out examples. The exploratory, post hoc spectral-minus-rotated contrast in bypass effects was +0.8608 percentage points (nominal, unadjusted 95% paired-t interval [0.6676, 1.0540]). With both models intact, the spectral-minus-rotated accuracy difference was +0.0408 percentage points (95% paired-t interval [−0.5854, +0.6670]). Intervals reflect five paired seeds with equal hospital weights, conditional on the observed patients, hospitals, splits, and rotations. Source validation was patch-level, and these hospitals informed earlier analyses.

Together, these analyses clarify how Stage 1 recovery enters Stage 2 prediction. Stage 1 defines the geometry of the frozen operators, while subject-specific coefficients govern their contribution. The synthetic experiments show that freezing preserves this geometry. The Camelyon17 ablations show a negligible intact-minus-bypass accuracy difference under Gaussian/ridge, but a larger difference for Spectral-LS than Rotated-LS under fitted coefficients. Because the LS models were trained separately, this exploratory contrast indicates different branch dependence, not an isolated effect of recovered geometry. We will clarify the role of subject-specific coefficients and the interpretation of these ablations in the revision.

Thanks for the detailed response. I will keep my original positive rating.

Reviewer_jXgJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper develops FIRE, a two-stage method to learn latent environment directions for multi-environment generalization. FIRE estimates latent directions using third-order cumulants, converts them into rank-one operators, and freezes these operators during downstream training. Synthetic experiments support latent direction recovery ability of FIRE. FIRE also achieves the best average OOD accuracy on 4/5 tested datasets, but the improvements are generally modest.

Critical strengths
  1. The idea of identifying latent directions provides an interesting perspective beyond conventional invariant representation learning.
  2. The experiments are generally broad, covering synthetic data, vision, and biomedical datasets, with several established domain-generalization baselines, as well as including controls and perturbation studies.
Critical weaknesses
  1. Theory-Implementation Mismatch: Proposition 2 requires consistent signal-covariance estimation and is developed for whitening-free estimators, whereas the implementation uses whitening that can retain observation-noise effects.
  2. Performance gains of FIRE over baseline methods are modest on 4/5 datasets, and FIRE underperfom than IRM on OASIS-1.
Questions for authors
  1. Can the authors compare the implemented estimator with the theoretically analyzed whitening-free under varying noise levels?
  2. Can the authors provide architecture-matched, consistently tuned comparisons among spectral frozen, random frozen, random trainable, recovered trainable, and no-operator models across all benchmarks? This is to assess whether gains come from operators rather than model capacity.
  3. Can the authors provide recovered vs. random frozen operators on every benchmark to see if identification really helps prediction?
  4. What are the runtime and memory costs, how do they scale with sample size, feature dimension, number of operators, and number of environments?
7 comments
AuthorsOpenReview ↗

Response to W1 (1/2). Recovery guarantees and solver outputs

Proposition 2 analyzes a whitening-free estimator and does not require signal-covariance estimation. That requirement belongs to Theorem 4’s whitening-based analysis (§3.4, p. 5). Both concern the same reference-coordinate directions used to construct FIRE’s frozen operators.

Let KK count the factors. Proposition 2’s matrix pencil requires known KK, i.i.d. jointly sub-Gaussian observations, two independent random unit contractions independent of the data, nonzero second-contraction coefficients, separated ratios and small perturbations. In Theorem 4, signal whitening makes the variance-rescaled factors orthonormal. Its bound requires covariance and tensor errors below threshold and prescribed tensor-power iterations and restarts (Appendix C, pp. 16–17).

What the implementation whitens. Benchmark whitening uses reference auto-covariance. Paired whitening uses auto-covariance within a cross-moment subspace. Both can retain noise, so applying Theorem 4 requires a bound on this covariance error (Remark 3, p. 19 and Appendix B, p. 16). Centered noise independent across environments and from loadings cancels in the population cross-moment. Recovering reference signal covariance also requires coordinate alignment, as with identity maps (Appendix C, p. 17).

To see how retained noise affects whitening, let dd be the feature dimension, bkb_k a latent direction and ArefA_{\mathrm{ref}} the reference map. The reference factor is ck=Arefbkc_k=A_{\mathrm{ref}}b_k, with unit direction vk=ck/∥ck∥v_k=c_k/\|c_k\|. Proposition 1 assumes a full-rank map, linearly independent latent directions, centered independent loadings with finite nonzero third cumulants μ3,k\mu_{3,k}, and independent noise with zero third cumulant. With weights ρk\rho_k and triple outer products vk⊗3v_k^{\otimes3}, the population third cumulant T\mathcal T is

T=∑k=1Kρkvk⊗3,ρk=μ3,k∥ck∥3.\mathcal T=\sum_{k=1}^{K}\rho_k v_k^{\otimes 3},\qquad \rho_k=\mu_{3,k}\|c_k\|^3.

This tensor identifies the reference lines up to sign and permutation. Recovering latent lines requires the reference map. Independent Gaussian noise preserves this tensor but affects whitening. Let C=[c1,…,cK]C=[c_1,\ldots,c_K] collect the reference factors and ImI_m denote the mm-dimensional identity. With unit-variance loadings and isotropic Gaussian noise of variance ν>0\nu>0, let W∈Rd×KW\in\mathbb R^{d\times K} whiten CC⊤+νIdCC^\top+\nu I_d on the full signal subspace. The reference Gram matrix G=C⊤CG=C^\top C and whitened Gram matrix JJ satisfy

=G(G+\nu I_K)^{-1},\qquad G=C^\top C.$$ Since $G=\nu J(I_K-J)^{-1}$, whitened factors are orthogonal exactly when reference factors are orthogonal. In that case exact decomposition and unwhitening recover their lines if all $K$ factors are retained. **From geometry to solver outputs.** Part 2 assesses the whitening solver’s returned directions using a separate conditional pencil. Let $\eta$ bound span error, $Q$ bound conditioning-adjusted pencil perturbation and $g$ be the minimum population eigenvalue gap. Given rank and invertibility, an output passes if $\eta<1$ and $Q<g/2$. With returned directions $B$ fixed, we fitted weights to the full empirical reference tensor. For $K=3$, choose unique $u_1,u_2$ in the span of $B$ with $B^\top u_1=(-1,0,1)$ and $B^\top u_2=(1,1,1)$, then set $w_\ell=u_\ell/\|u_\ell\|$ for $\ell=1,2$. Column order and signs stayed fixed without pair search or direction optimization. 25 outputs used five seeds per cosine, 60,000 observations, $d=20$, exponential loadings and independent Gaussian noise (SD 1), with unchanged settings. Bounds use eigenvalue matching for passing outputs. All observed errors use optimal permutation matching. | Common pairwise factor cosine | Conditional passes | Maximum sine-error bound | Observed maximum sine error | | --- | --- | --- | --- | | 0 | 5/5 | 0.172009–0.380335 | 0.028884–0.038933 | | 0.1 | 2/5 | 0.7124–0.7217 | 0.0544–0.0923 | | 0.3 | 0/5 | No bound | 0.2133–0.2651 | | 0.5 | 0/5 | No bound | 0.4755–0.5929 | | 0.7 | 0/5 | No bound | 0.8839–0.9190 | This pointwise check uses known population moments and realized errors at candidate-dependent contractions. Exponential loadings fall outside Proposition 2’s sub-Gaussian assumption. Worst-direction sine errors were below 0.04 at cosine 0 and above 0.88 at cosine 0.7 in all runs. The bound applied to all five orthogonal runs and two of five at cosine 0.1. **Under its stated conditions, the conditional bound connects the directions returned by the implemented whitening solver to the error of FIRE’s frozen operators.**
AuthorsOpenReview ↗

Response to W1 (2/2). Conditional error bound and proof

Claim. In dd reference dimensions, let K≥2K\geq2 count the components, vkv_k denote true unit directions and ρk\rho_k their weights in the population tensor T=∑k=1Kρkvk⊗3\mathcal T=\sum_{k=1}^K\rho_k v_k^{\otimes3} (triple outer products). For returned unit directions b^j\hat b_j, let π\pi match each to a true line and β\beta denote the sine-error bound. Here ∠\angle is the angle between lines and ∥⋅∥2\|\cdot\|_2 the Euclidean or spectral norm. Under the conditions below,

∥b^jb^j⊤−vπ(j)vπ(j)⊤∥2=sin⁡∠(b^j,vπ(j))≤β.∥\hat b_j\hat b_j^\top-v_{\pi(j)}v_{\pi(j)}^\top∥_2=\sin\angle(\hat b_j,v_{\pi(j)})\leq\beta.

Write F=[v1,…,vK]F=[v_1,\ldots,v_K] and B=[b^1,…,b^K]B=[\hat b_1,\ldots,\hat b_K]. With BB fixed, fit weights τi\tau_i to the full empirical tensor T^\widehat{\mathcal T} to form the candidate T~=∑i=1Kτib^i⊗3\widetilde{\mathcal T}=\sum_{i=1}^K\tau_i\hat b_i^{\otimes 3}. Both population and candidate tensors must have rank KK, linearly independent real unit directions in the same coordinates and nonzero real weights of either sign. For ℓ=1,2\ell=1,2, contract the third mode with unit vector wℓw_\ell to form the population, empirical and candidate matrices Mℓ,Nℓ,RℓM_\ell,N_\ell,R_\ell. Here II denotes the identity.

Mℓ=T(I,I,wℓ),Nℓ=T^(I,I,wℓ),Rℓ=T~(I,I,wℓ),ℓ=1,2.M_\ell=\mathcal T(I,I,w_\ell),\qquad N_\ell=\widehat{\mathcal T}(I,I,w_\ell),\qquad R_\ell=\widetilde{\mathcal T}(I,I,w_\ell),\quad \ell=1,2.

Let UU be an orthonormal basis of the returned span, P=UU⊤P=UU^\top its projector and H=U⊤FH=U^\top F the projected true directions. Require positive KK-th singular value s=σK(M2)s=\sigma_K(M_2) and invertible second candidate contraction D=U⊤R2UD=U^\top R_2U. The span-error bound η\eta and within-span errors dℓd_\ell combine fitting and sampling errors.

η=∥(Id−P)(R2−N2)∥2+∥(Id−P)(N2−M2)∥2s.\eta=\frac{∥(I_d-P)(R_2-N_2)∥_2+∥(I_d-P)(N_2-M_2)∥_2}{s}.

dℓ=∥U⊤(Rℓ−Nℓ)U∥2+∥U⊤(Nℓ−Mℓ)U∥2,ℓ=1,2.d_\ell=∥U^\top(R_\ell-N_\ell)U∥_2+∥U^\top(N_\ell-M_\ell)U∥_2,\quad\ell=1,2.

If η<1\eta<1, both HH and U⊤M2UU^\top M_2U are invertible. Form the projected population pencil Z=(U⊤M1U)(U⊤M2U)−1Z=(U^\top M_1U)(U^\top M_2U)^{-1}. Using the condition number cond⁡2(H)=∥H∥2∥H−1∥2\operatorname{cond}_2(H)=\|H\|_2\|H^{-1}\|_2 of the projected factors, define the conditioning-adjusted perturbation bound QQ by

Q=cond⁡2(H)(d1+∥Z∥2d2)∥D−1∥2.Q=\operatorname{cond}_2(H)(d_1+\|Z\|_2d_2)\|D^{-1}\|_2.

Here d1+∥Z∥2d2d_1+\|Z\|_2d_2 combines contraction errors and ∥D−1∥2\|D^{-1}\|_2 accounts for inversion. Let ri=⟨vi,w1⟩/⟨vi,w2⟩r_i=\langle v_i,w_1\rangle/\langle v_i,w_2\rangle be the population eigenvalues, r=(r1,…,rK)r=(r_1,\ldots,r_K) their vector, and gg their minimum pairwise separation. The rank of M2M_2 ensures nonzero denominators. Numerators may vanish. If η<1\eta<1, g>0g>0 and Q<g/2Q<g/2, a unique eigenvalue matching π\pi gives

\leq \beta:= \sqrt{\eta^2+(1-\eta^2) \left(\frac{Q}{g-Q}\right)^2}<1.$$ **Proof.** Since $(I_d-P)R_2=0$, we have $\|(I_d-P)M_2\|_2\leq s\eta$. The Moore–Penrose pseudoinverse $M_2^\dagger$ has norm $1/s$. Each unit $v_i$ lies in the range of $M_2$, so $v_i=M_2 M_2^\dagger v_i$ gives $\|(I_d-P)v_i\|_2\leq\eta$. The same holds on every unit vector in the true span, so $\eta<1$ makes $H$ invertible. Next, using $\mathrm{diag}$ for a diagonal matrix, the projected contractions factor as $H\,\mathrm{diag}(\rho_i\langle v_i,w_\ell\rangle)H^\top$ and $(U^\top B)\,\mathrm{diag}(\tau_i\langle\hat b_i,w_\ell\rangle)(U^\top B)^\top$. Since $s>0$, the second population diagonal factor and $U^\top M_2U$ are invertible. Thus $Z=H\,\mathrm{diag}(r)\,H^{-1}$, while the candidate pencil $\widetilde Z=(U^\top R_1U)D^{-1}$ has eigenvectors given by the columns of $U^\top B$. For contraction differences $\Delta_\ell=U^\top(R_\ell-M_\ell)U$ and pencil error $E=\widetilde Z-Z$, the identity $E=(\Delta_1-Z\Delta_2)D^{-1}$ gives $\|E\|_2\leq(d_1+\|Z\|_2d_2)\|D^{-1}\|_2$, hence $\operatorname{cond}_2(H)\|E\|_2\leq Q$. Let $t$ range from zero to one. By Bauer–Fike, each eigenvalue of $Z + tE$ lies within $tQ$ of some $r_i$. Since $Q < g/2$, the radius-$Q$ disks are disjoint. By continuity, each retains one eigenvalue, giving $\pi$. For a matched unit eigenvector $x$ with eigenvalue $\lambda$ near $r_i$, put $y = H^{-1}x$ and let $y_{-i}$ omit coordinate $i$. Then $(\operatorname{diag}(r) - \lambda I)y = -H^{-1}Ex$. All other population eigenvalues are at least $g-Q$ from $\lambda$, so $$\lVert y_{-i} \rVert_2 \leq \frac{\lVert H^{-1} \rVert_2 \lVert E \rVert_2}{g-Q}.$$ The distance from $x$ to the line of $U^\top v_i$ is at most $$\lVert H \rVert_2 \lVert y_{-i} \rVert_2 \leq \frac{Q}{g-Q}.$$ Finally, $Ux$ is the matched direction up to sign. Let $a=\|(I_d-P)v_i\|_2$ be the out-of-span length and $\phi$ the angle between $x$ and $U^\top v_i$. Orthogonal decomposition gives $\sin^2\angle(Ux,v_i)=a^2+(1-a^2)\sin^2\phi$. Using $a\leq\eta$ and $\sin\phi\leq Q/(g-Q)<1$ gives $\beta$. Signed weights and indefinite contractions are allowed, and optimal permutation matching can only improve the maximum error. **Thus the direction bound controls the error of each matched frozen Stage 2 projector.**
AuthorsOpenReview ↗

Response to W2

Identifiability determines whether the population distribution uniquely specifies the reference-coordinate directions under the model assumptions. Prediction accuracy measures how well a model predicts particular labels. Whether recovered structure improves accuracy depends on its relevance to those labels and how the predictor uses it; similar accuracy can therefore arise from representations that recover different structure.

For example, an identifiable direction might distinguish two disease subtypes, while the prediction task is healthy control versus disease. If the predictor already separates both subtypes from healthy controls, recovering their distinction need not improve binary accuracy, although it could help subtype classification.

FIRE has the highest mean accuracy on four of five benchmarks over ten matched seeds. On OASIS-1, FIRE achieves 0.509 ± 0.070 versus IRM's 0.531 ± 0.074 (mean ± seed SD), a 2.2-percentage-point gap. All five margins against the strongest baseline are within one seed SD. OASIS introduces an additional complexity: its environments are age quartiles, which may differ in the underlying patient factors relevant to dementia. Because FIRE recovers directions from a reference environment, structure captured there may not be equally informative for dementia classification in another age group. This could contribute to the observed difference on OASIS-1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.