acceptodds
Under review as a conference paper at ICLR 2027

What Intervention Loss Certifies: Sharp Bounds for Factor Separation

20 views

Abstract

An intervention loss can be small even when a frozen representation exhibits substantial factor mixing if the available actions fail to probe the relevant dependencies. We characterize precisely what separate upper bounds on such losses can certify about the representation. We show that the largest compatible off factor variation is given by a fractional cover linear program. An orthogonal variance decomposition reduces the exponential interaction structure to one variable per forbidden factor while an additive construction attains the resulting bound. To make the certificate data dependent we develop simultaneous confidence bounds that permit principled selection of certificate weights. We further reduce statistical conservatism through predictable variance adaptation and deterministic fiber envelopes without changing the certified quantity. On nonlinear neural encoders predictable concentration tightens a pair mask certificate from to compared with oracle leakage of ; restricting to overlapping tests yields a nearly two fold reduction. On the public Pendulum generator three image only autoencoder families achieve raw output certificates within factors of to of exact finite reference errors using pairs per action with exhaustive range audit costs reported separately. These results establish dependence certification as a distinct objective from normalized disentanglement scores full identifiability and information retention. They provide a quantitative framework for determining what intervention based evidence can and cannot certify about a representation.

Primary area:
causal reasoning
Keywords:
Disentangled Representation Learning, Identifiability, Causal Representation Learning, Interventional Learning, Representation Learning, Factor Mixing, Statistical Certification, Neural Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VnH9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In my reading, the paper asks a precise and, I think, worthwhile question: once an encoder F=f∘hF=f\circ h has been trained and frozen, what does a small intervention loss actually certify about factor mixing? Given latent factors C∈Ω=∏i=1KΩiC\in\Omega=\prod_{i=1}^{K}\Omega_i, action masks Ta⊆[K]T_a\subseteq[K] with probabilities pap_a, and the intervention loss

Lint(F;B)=EB∑j∉Ta(Fj(C′)−Fj(C))2,\mathcal L_{\mathrm{int}}(F;B)=\mathbb E_B\sum_{j\notin T_a}\big(F_j(C')-F_j(C)\big)^2,

the protected error is the off-factor variation Rj=EQ(Fj(C)−EQ[Fj(C)∣Cj])2\mathcal R_j=\mathbb E_Q\big(F_j(C)-\mathbb E_Q[F_j(C)\mid C_j]\big)^2 under a declared product reference law QQ (with a partial version Rjallow\mathcal R_j^{\mathrm{allow}} when masks leave some dependencies admissible).

The contributions, as I understand them, are four.

(1) Theorem 1 characterises zero loss through action signatures Si={a:i∈Ta}S_i=\lbrace a: i\in T_a\rbrace: Lint=0\mathcal L_{\mathrm{int}}=0 if and only if ∂iFj=0\partial_i F_j=0 whenever Si⊈SjS_i\not\subseteq S_j.

(2) Theorem 2 gives an aggregate bound, ∑jqjRj≤κ2Lint\sum_j q_j\mathcal R_j\le\frac{\kappa}{2}\mathcal L_{\mathrm{int}}, where qij=∑apa1{i∈Ta,j∉Ta}q_{ij}=\sum_a p_a\mathbf 1\lbrace i\in T_a, j\notin T_a\rbrace and qj=min⁡i≠jqijq_j=\min_{i\ne j}q_{ij}. The factor 1/(2q∗)1/(2q_{\ast}) is sharp.

(3) Theorem 3, the central result: given per-action half-energy upper bounds uaj≥eaju_{aj}\ge e_{aj}, the largest compatible residual is exactly a fractional-cover linear program,

Cj(u)=min⁡y≥0{∑a∈Ajyauaj : ∑a∈Aj: i∈Taya≥1  ∀i∈Ij}.\mathcal C_j(u)=\min_{y\ge0}\Big\lbrace\sum_{a\in\mathcal A_j}y_a u_{aj}\ :\ \sum_{a\in\mathcal A_j:\, i\in T_a}y_a\ge1\ \ \forall i\in I_j\Big\rbrace.

The exponentially many ANOVA interaction terms reduce to one variable per forbidden factor, and an additive construction attains the value.

(4) A finite-sample layer: empirical Bernstein and predictable plug-in (PrPl) bounds of Waudby-Smith and Ramdas with a union bound over action–output cells, deterministic fibre envelopes MajfibM_{aj}^{\mathrm{fib}}, and a zero-observation lower bound (Proposition 10) showing that a range term of order Mlog⁡(1/η)/nM\log(1/\eta)/n cannot be avoided.

The experiments use 25 small MLP encoders (8→64→64→48\to64\to64\to4) on four synthetic factors; controlled triangular-shear benchmarks with K∈{4,10,25}K\in\lbrace4,10,25\rbrace; and AE, VAE and β\beta-VAE encoders on a two-root Pendulum grid of 84×8784\times87 configurations. The authors report certificate-to-oracle ratios of 1.02–1.04 on Pendulum at 32,768 pairs per action, a reduction from 1.69×10−31.69\times10^{-3} to 1.30×10−31.30\times10^{-3} against an oracle leakage of 0.85×10−30.85\times10^{-3} on the neural pair-mask encoders, and a set of negative controls (rare separation, support holes, untargeted drift).

Critical strengths

Strength 1. In my opinion, Theorem 3 is a genuinely clean and useful piece of mathematics. I find the exact upper-information envelope to be the paper's real contribution. Through the orthogonal product-space (Hoeffding/ANOVA) decomposition Fj=∑U⊆[K]Fj,UF_j=\sum_{U\subseteq[K]}F_{j,U}, each action's half-energy becomes eaj=∑U∩Ta≠∅∥Fj,U∥2e_{aj}=\sum_{U\cap T_a\ne\varnothing}\lVert F_{j,U}\rVert^2. The authors then observe that every forbidden interaction U⊈DjU\not\subseteq D_j can have its variance moved to a single forbidden coordinate i(U)∈U∩Iji(U)\in U\cap I_j without increasing any action energy. This collapses an optimisation over 2K2^K interaction weights to a packing LP with ∣Ij∣|I_j| variables, and the additive construction G=∑ixi gi(Ci)G=\sum_i\sqrt{x_i}\,g_i(C_i) shows the value is attained. As far as I can verify, the argument in Appendix A.4 is correct.

I particularly appreciated the overlapping-mask example: with tests {1,2}\lbrace1,2\rbrace, {1,3}\lbrace1,3\rbrace and {2,3}\lbrace2,3\rbrace, the certificate is C4(a,b,c)=min⁡{a+b, a+c, b+c, a+b+c2}\mathcal C_4(a,b,c)=\min\lbrace a+b,\ a+c,\ b+c,\ \tfrac{a+b+c}{2}\rbrace. This makes concrete why fractional weighting beats both a fixed-weight and the best integral cover (1.063 versus 1.412 at K=25K=25 in Table 2). The analytic example where the aggregate bound is 49ε249\varepsilon^2 while the true error and the LP certificate are both ε2\varepsilon^2 is equally convincing. To my knowledge, this exact characterisation for masked intervention losses is new.

Strength 2. I would like to commend the statistical care and the intellectual honesty of the evaluation. From my perspective, the authors get several subtle points right that are often glossed over in the disentanglement literature:

  • the encoder, its output assignment and its normalisation are frozen before validation;
  • simultaneous validity over all action–output cells means the dual weights yy may be chosen using the same upper bounds without an additional union bound (Corollary 4);
  • a fitted validation maximum is never silently substituted for a support bound;
  • Proposition 10 proves that a range term Mmin⁡{12,1−η1/n}M\min\lbrace\tfrac12,1-\eta^{1/n}\rbrace cannot be removed by any energy-only procedure.

The negative controls are equally valuable. Proposition 6 and Appendices C.7–C.8 show explicitly how the certificate fails under support holes (κ=1/s→∞\kappa=1/s\to\infty) and untargeted drift (where the naive expression is (1−η)2/2(1-\eta)^2/2 but the true error stays 1/21/2). The nested and identical-signature designs correctly report "full separation unavailable" rather than a misleadingly small number. I believe this level of candour is a model for evaluation papers.

Critical weaknesses

Weakness1. In my view, the regime in which the certificate is both needed and valid is very narrow, and the experiments mostly sit in settings where it is not needed.

The certificate is valid only under strong premises: a declared product reference law QQ; exact (or κ\kappa-dominated) independent refreshes of the targeted factors, paΞa≤κBap_a\Xi_a\le\kappa B_a; and, for the coupling-corrected version, a verified discrepancy bound δF\delta_F. The authors themselves state that "neither action counts nor low loss verify these sampling premises", and Proposition 6, Table 10 and Table 11 show the certificate silently breaking when they fail. In practice, therefore, the guarantee holds only when one has a simulator that can refresh latent factors exactly from QQ.

But if one has such a simulator, I would ask: why not estimate the protected error directly? The paper's own oracle does exactly that. It holds CjC_j fixed, refreshes the remaining factors, and averages half the squared increment, so it estimates Rj\mathcal R_j itself. An empirical Bernstein interval on that one quantity would give a direct, essentially tight certificate with no LP and no cover. The LP adds value only when the action that refreshes [K]∖{j}[K]\setminus\lbrace j\rbrace is unavailable while exact product refreshes of other masks are available. I could not find a concrete real-world system with this combination motivated anywhere in the paper.

The Pendulum study makes the tension explicit. The authors enumerate all 84×87=730884\times87=7308 configurations to obtain exact envelopes and variance caps, and concede that "exhaustive enumeration already permits exact error computation, so this experiment is a finite-support validity and transfer audit". In other words, the headline 1.02–1.04 ratios are obtained in a setting where the certificate is, by the authors' own account, redundant.

Three further points narrow the scope, in my assessment:

  • The neural certificates do not bound the raw encoder. They concern Gj=tanh⁡(Fj)G_j=\tanh(F_j) rather than FjF_j; the paper states that this "is not automatically a bound for the raw output", and a saturating readout can make large raw mixing look small.
  • The scale is very small. There are four latent factors, an 8-dimensional observation and a 8→64→64→48\to64\to64\to4 MLP; Pendulum has two roots at 32×3232\times32 pixels.
  • The one visual pipeline that resembles realistic training fails. In Appendix F (Table 21), the conditioned RGB model recovers worse than the observational baseline in every setting (MCC 0.323 vs. 0.443, 0.333 vs. 0.457, 0.304 vs. 0.483).

Taken together, I am not yet convinced that a practitioner training a causal representation learner (CITRIS, iCITRIS, or a mechanism-sparsity model) would be able to use this certificate on their model.

Weakness 2. I think the theoretical novelty is narrower than the framing suggests, and the "sharpness" is a worst case over a weak information class that does not translate into practically tight certificates.

Theorem 1 is, as the authors acknowledge, the action-only form of Lachapelle et al. (Proposition 3 and Lemma 1). Theorem 2 is the classical weighted block-factorisation / Efron–Stein inequality. The finite-sample machinery is empirical Bernstein (Maurer and Pontil, 2009) and predictable plug-in bounds (Waudby-Smith and Ramdas, 2024) applied cell-wise with a union bound. That leaves Theorem 3 as the main new result. It is clean, but its proof is essentially one page of ANOVA bookkeeping plus LP duality.

More importantly, Theorem 3 is sharp only in the sense that some function consistent with the upper bounds uaju_{aj} attains Cj(u)\mathcal C_j(u). The authors themselves note that "the extremizer in the proof need not resemble the fitted encoder", and that coordinatewise optima are not simultaneously attainable. On the actual neural encoders, the certificates remain loose:

  • per-seed PrPl certificate/oracle ratios of 151.75±16.72151.75\pm16.72 (singleton) and 125.70±9.51125.70\pm9.51 (identical) at 2,048 pairs per action;
  • still 2.54±0.162.54\pm0.16 (singleton) at 262,144 pairs per action, which is over one million validation pairs to certify four latent factors (Tables 13 and 15);
  • a ratio of 7.14±1.257.14\pm1.25 in the mask-only neural check (Appendix D.5).

Proposition 10 shows this is not an artefact. Any energy-only certificate pays roughly Mlog⁡(m/δ)/nM\log(m/\delta)/n. Certifying a leakage level ε\varepsilon therefore needs na≳Mlog⁡(m/δ)/εn_a\gtrsim M\log(m/\delta)/\varepsilon pairs per action. The number of cells mm grows with both the number of actions and KK, and with M=2M=2 and ε≈10−3\varepsilon\approx10^{-3} this is already thousands of pairs per cell before any variance term. For the high-dimensional latent spaces that ICLR readers care about, I worry this becomes prohibitive.

Finally, the paper never shows that the certificate is useful for a decision. There are no experiments on model selection, hyperparameter or early-stopping choice, or ranking encoders, compared with IRS, DCI and MIG. Indeed, the authors explicitly state that "certificate tightness is not a ranking of representation quality" (Section 6.3) and that small leakage "can accompany a weakly informative coordinate" (Appendix E.4). If the certificate neither ranks representations nor guarantees information retention, I find it hard to see what downstream action it supports.

Questions for authors
  1. Concrete use case. Could you describe a realistic system in which (i) exact product-law refreshes of latent factors are available, so that the certificate is valid, but (ii) the refresh of [K]∖{j}[K]\setminus\lbrace j\rbrace needed to estimate Rj\mathcal R_j directly is unavailable, so that the certificate is needed? Without such an example, I find it difficult to judge the practical significance of Theorem 3.

  2. Direct estimation as a baseline. In the controlled and Pendulum experiments, could you add a baseline that applies the same empirical Bernstein / PrPl interval directly to the oracle half-energy that estimates Rj\mathcal R_j, at the same total pair budget? I believe this is the natural competitor to the LP certificate, and it is currently absent.

  3. Raw versus saturated output. The neural certificates bound the residual of Gj=tanh⁡(Fj)G_j=\tanh(F_j). Can you relate this to the residual of the raw FjF_j, for example via a lower bound on tanh⁡′\tanh' over the observed output range, or report raw-scale certificates using verified fibre envelopes as in the Pendulum study? Otherwise, how should a reader interpret a 10−310^{-3} certificate on a saturating readout?

  4. Verifying the overlap premise. For replay buffers collected by a behaviour policy (the setting that motivates CITRIS-style temporal learners), how would a practitioner obtain a justified κa\kappa_a or a coupling bound δF\delta_F? Is there any testable sufficient condition, or must these always be assumed?

  5. Scaling. Proposition 10 implies na≳Mlog⁡(m/δ)/εn_a\gtrsim M\log(m/\delta)/\varepsilon per cell. Could you report the total number of validation pairs needed to reach a certificate/oracle ratio below 2 as a function of KK for K∈{4,8,16,32}K\in\lbrace4,8,16,32\rbrace on a learned (not analytic) encoder? This would make the practical cost explicit.

  6. Decision usefulness. Can you show a single experiment in which the certificate leads to a better decision than IRS, DCI or MIG, for example selecting among checkpoints or intervention designs, or detecting a failure that these scores miss?

  7. Mechanistic interpretability and LLMs. Interchange-intervention objectives play a closely analogous role in the interpretability of large language models: interchange intervention training, distributed alignment search, and activation patching on Transformer residual streams, where a subspace is "intervened on" and off-target behaviour should remain unchanged. Does your LP certificate apply to such interchange interventions, where the "factors" are causal variables of a high-level model and FF is a learned subspace of the residual stream? If so, what plays the role of the product reference law QQ there? I think a short discussion would considerably broaden the paper's relevance to the ICLR audience.

  8. Missing related work. In my view, several closely related lines should be discussed and contrasted:

    • weakly supervised disentanglement from paired observations that differ in a subset of factors (Locatello et al., ICML 2020; Brehmer et al., NeurIPS 2022), which is very close to your paired-refresh setting;
    • sparse-perturbation and interventional causal representation learning (Ahuja et al., NeurIPS 2022 and ICML 2023);
    • interchange intervention training and causal abstraction (Geiger et al., ICML 2022). How does your certificate relate to the identifiability guarantees in these works?
  9. Simultaneous attainability. Theorem 3 is sharp coordinatewise, but you note that the optima need not be attained by a single map. Can you characterise, or at least bound, the gap between ∑jCj(u)\sum_j\mathcal C_j(u) and the joint supremum sup⁡G∑jRjallow(G)\sup_G\sum_j\mathcal R_j^{\mathrm{allow}}(G) under the same energy constraints?

  10. Reproducibility of Appendix F. The RGB results are described as "author-provided records" whose original checkpoints are not included, whereas the AI-use statement notes that generative AI assisted with the "implementation and execution" of the controlled and neural experiments. Could you clarify which experiments were run by the authors directly, and release the RGB checkpoints so that the negative result in Table 21 can be independently verified?

5 comments
AuthorsOpenReview ↗

Response to Reviewer VnH9: Decision Utility, CITRIS-NF Validation, and Theoretical Clarifications

We thank the reviewer. Revisions add Propositions 12–13 (Appendix H), Tables 22–23 (Appendix I), and the CITRIS-NF audit (Appendix J/Table 24).

Decision. This is a leakage gate, not a quality ranking. For prespecified frozen encoders, assignments, and scales, simultaneous bounds give Pr⁡(∃m:Rm>τ, Um≤τ)≤δ\Pr(\exists m:R_m>\tau,\ U_m\le\tau)\le\delta (familywise, not conditional, control). Um>τU_m>\tau means uncertified, not unsafe. Full leakage certification requires separation; otherwise, only the allowed-dependency residual is bounded. A utility floor is required. Among 40 held-out encoders (20 reference-safe/16 unsafe/4 unresolved), admissions [safe, unsafe, unresolved] are: fractional 14 [14,0,0]; direct PrPl 17 [17,0,0]; IRS 17 [13,3,1]; DCI 17 [14,2,1]; MIG 14 [12,2,0]. Thresholds use development data; independent reference intervals retain unresolved cases.

Independent visual audit. We test released CITRIS-NF (phlippe/saved_models: DL2/CRL/ae.ckpt, citris.ckpt), trained on seven-shape Temporal Causal3DIdent without our penalty. The frozen scalar is tanh⁡(zk)\tanh(z_k) for the lowest-index shape-block coordinate (half-increment envelope M=2M=2, variance cap vj=1v_j=1). A seven-class probe achieves 4,368/7,000=62.4% (chance 14.29%): evidence of utility, not disentanglement. Nuisances are position, object/spotlight rotation, and spotlight/background/object hue. Exact product-reference pairs give κ=1\kappa=1. Masks (12),(23),(13),(45),(56),(46)(12),(23),(13),(45),(56),(46) use 32,768 pairs each; direct six-factor refresh uses 196,608. Simultaneous PrPl bounds share δ=0.05\delta=0.05. Upper energies are u=(.012,.013,.011,.008,.010,.029)u=(.012,.013,.011,.008,.010,.029). Fractional/integral/fixed-uniform bounds are .0360/.0410/.0415.0360/.0410/.0415; direct is .0286.0286. At prespecified τ=.0400\tau=.0400, only LP and direct certify. Fractional weights y=(1/2,1/2,1/2,1,1,0)y=(1/2,1/2,1/2,1,1,0) match packing x=(.005,.007,.006,.008,0,.010)x=(.005,.007,.006,.008,0,.010), proving optimality. LP gains 12.2% versus integral and 13.3% versus uniform: triangle 1 favours halves (.018 vs .023 integral); triangle 2 favours action selection (.018 vs .0235 uniform). One triangle cannot strictly beat both. Uniform weights are prespecified; post-hoc tuning can recover LP. Uncovered factors lack finite bounds; empty forbidden sets yield zero. Table 16 also reports fractional .000661 versus integral .000854 (22.6% lower); fixed aggregate .000661 for balanced counts.

Direct refresh and access. At matched 98,304 pairs/encoder, mean PrPl bounds are direct .00042, fractional .00066, and integral .00085 (Table 22). Rendering and inference costs must be counted; encoders are replication units. Prefer direct refresh when feasible and cheaper. Composable exact masked refreshes implement their union; refreshing additional allowed factors gives a conservative bound. Our restricted menu models archived pairs, not renderer limitations. CITRIS's published singleton/path menu has a totally unimodular incidence matrix: Cfrac=CintC_{\mathrm{frac}}=C_{\mathrm{int}}, even after row/action restrictions or common variance caps. Temporal transitions are not valid reference pairs. Replay validity requires proved state–action–outcome pair-law domination or a valid coupling bound; action counts and overlap diagnostics cannot establish κ\kappa.

Raw output and scale. If ∣Fj∣≤B|F_j|\le B globally QQ-almost surely, then Rjallow(F)≤cosh⁡4(B)Rjallow(tanh⁡F)R_j^{\mathrm{allow}}(F)\le\cosh^4(B)R_j^{\mathrm{allow}}(\tanh F), also for the full residual (Prop. 13). Observed maxima do not suffice. The four-factor singleton ratio is 2.54 after 1,048,576 pairs. Learned 4/8/16/32-factor ratios at two million pairs are 1.7/1.9/2.8/4.6; pairs needed for mean ratio below 2 are 0.5m/1.5m/>2m/>2m (Table 23). This limits practical tightness; time costs also matter. LLM interchange interventions and activation patching require a declared product law and independent masked refreshes. We distinguish the identifiability work of Locatello/Brehmer and Ahuja, and Geiger's causal abstraction work, from our information bound.

Joint attainability and evidence. For separable output constraints, sup⁡G∑jRjallow(G)=∑jCj(u)\sup_G\sum_jR_j^{\mathrm{allow}}(G)=\sum_jC_j(u), or ∑jmin⁡{vj,Cj(u)}\sum_j\min\{v_j,C_j(u)\} with separate caps, via additive packing extremisers and row scaling (Prop. 12). For scalar factors, sufficiently large affine own-factor diagonal terms yield an invertible map without changing off-target energies. Pilot allocation is not checkpoint selection. Appendix F RGB relies on author-provided records without the original architecture, optimiser, checkpoints, or per-run logs; independent reproducibility is not claimed.

Thank you for the thorough response and the new material (Propositions 12–13, Tables 22–24 and the CITRIS-NF audit). The decision study, the raw-output bound and the K-scaling table answer my Q3, Q5, Q6 and Q9 well, and I appreciate the candid note that independent reproducibility of Appendix F "is not claimed".

Three short follow-ups:

  1. Your own numbers favour direct refresh: at matched 98,304 pairs it gives .00042 against .00066 for the fractional LP, and in the decision study it admits 17 versus 14 reference-safe encoders. Could you name one concrete setting where direct refresh is infeasible or costlier while product-law masked refreshes are available, ideally with a cost-matched comparison? This remains the crux of my Q1 and Q2.

  2. You note that CITRIS's published menu has a totally unimodular incidence matrix, so Cfrac=CintC_{\mathrm{frac}}=C_{\mathrm{int}} there. Since the 12.2% gain comes from designed masks, which intervention designs used in practice are non-TU?

  3. For Proposition 13, what is BB for your encoders? With B=2B=2, cosh⁡4(B)≈200\cosh^4(B)\approx200, which would make raw-scale certificates largely vacuous.

Thanks again. These points will decide whether I revise my score.

AuthorsOpenReview ↗

Response to Reviewer VnH9 (1/2): where restricted-mask certification is useful

1. Where restricted-mask certification is useful

Consider a retrospective audit of a frozen encoder from an archived intervention dataset. The auditor receives validated input pairs and encoder outputs for a prescribed collection of pairwise interventions, but can neither generate new counterfactual examples nor compose interventions on the same initial state. Direct nuisance-refresh pairs are then unavailable, while the archived pairwise measurements still suffice to certify residual dependence. Weakly supervised pair data, in which each pair changes a random subset of factors (Locatello et al., 2020), has this form when the pairs are released together with the factors each pair changes.

Our CITRIS-NF audit shows this measurement problem on an independently pretrained representation. Six pair-refresh masks each receive 32,768 independent validation pairs, 196,608 in total. All restricted-mask certificates use the same observations, confidence allocation and bounded-readout scale.

CertificatePairsUpper boundDecision at τ=0.0400\tau=0.0400
Fixed-weight aggregate196,6080.0415Abstain
Best integral cover196,6080.0410Abstain
Fractional-cover LP196,6080.0360Certify
Direct refresh, unrestricted renderer196,6080.0286Certify
Direct refresh, matched primitive-action cost49,1520.0310Certify

Among the procedures that use only the six pairwise measurements, only the LP certifies. It improves the integral certificate by 12.2% and the fixed-weight certificate by 13.3%.

The cost-matched comparison. With access restricted to the six pairwise actions, a complete refresh of the six nuisance factors must be composed from at least four primitive interventions, two per three-factor group. A budget of 196,608 primitive interventions therefore supports 49,152 composed direct pairs. Under the exact product-refresh law, a composed refresh has the same pair law as a direct one. We therefore drew a random subsample of 49,152 of our direct pairs, with the seed fixed in advance, and recomputed the bound with the same error allocation. It is 0.0310, so at matched primitive-action cost direct refresh still certifies, with a tighter bound than the LP's 0.0360. We add this row to Table 24.

Direct refresh is therefore preferable whenever it is available, even by composition at four times the cost per pair. The held-out decision study points the same way: 17 against 14 reference-safe encoders admitted, with no reference-unsafe admissions under either gate. The LP's advantage is confined to records where composition is impossible, such as archived pairs. There it extracts the strongest leakage certificate the measurements imply, and in this audit it is the only restricted-mask certificate that meets the budget.

Reviewer_ojepOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary
  • The paper considers an encoder that extracts a disentangled representation from a high-dimensional input generated by independent factors, and asks how much each output leaks information from factors it should not depend on.
  • Given a set of interventions, each perturbing one or several factors at once, together with the resulting output changes, the goal is a valid upper bound on the true leakage.
  • The paper first shows that the bound obtained from the aggregate intervention loss scales with the frequency of the rarest intervention and is therefore loose, since the rarest intervention may be irrelevant to the leakage actually present.
  • The proposed method instead keeps each intervention’s measured energy separately and combines them through a small linear program, which is proven to give a valid and tight bound. The interventions only need to jointly cover every factor; testing every factor combination is not required.
Critical strengths
  • Very cool theory work. An elegant formulation of the encoder deconfounding problem with some tight bounds. The problem formulation and assumptions seem reasonable.
  • Exact, tight answer to a well-posed question. Theorem 3 shows that the largest leakage consistent with per-intervention energy bounds is exactly the value of a small LP, attained by an explicit encoder. The reduction from exponentially many interaction terms to one variable per factor is what makes this both computable and tight. The motivating gap (a 49× loose aggregate bound in a three-factor example) is convincing.
Critical weaknesses
  • I have to first confess that I am not very familiar with this field and had a hard time reading this paper, so I'm probably not the best person to review it, and my understandings may be incorrect.
  • I would love to see a more realistic experiment that shows the method’s advantage. The LP only beats simpler bounds in Section 6.2, on constructed encoders or with one intervention’s sample count artificially reduced. In the experiments with trained models (Table 1) and real images (Table 3), the LP reduces to the aggregate or singleton bound. The paper never combines a realistic setting with a demonstration of its main contribution.
  • Readability. The abstract, introduction, and conclusion are dense enough that the problem and contribution are hard to extract without working through the examples in Section 4.2. Those examples should come much earlier.
Questions for authors
  • Why does Section 6.1 use balanced intervention frequencies, given that this is precisely the regime where the LP coincides with the aggregate bound?
  • Could you please show the LP’s gain on a standard disentanglement dataset with trained models and naturally unbalanced interventions?
  • The aggregate bound of Theorem 2 is derived in this paper. Is there any prior method that produces a comparable quantity to serve as an external baseline?
Reviewer_JXmKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper highlights an important issue: even a very small intervention loss does not guarantee that the learned representation is well disentangled. It addresses this issue by characterizing what intervention losses can certify about factor separation under explicitly stated assumptions. However, I am concerned that transferring the guarantees to replay data requires a strong domination assumption: the reference pair distribution must be dominated by the replay distribution with a finite density-ratio bound. This condition may fail for many practical data distributions, particularly when the collected data have gaps in state or intervention-outcome coverage, and it is generally difficult to verify from finite data. Although direct sampling from the reference intervention experiment avoids this transfer problem, the assumption limits the applicability of the replay-based guarantees.

Critical strengths

The paper addresses an important and potentially underappreciated issue: a small intervention loss can coexist with substantial factor mixing in a learned representation. Such mixing may undermine downstream experiments, making it valuable to understand what a low loss actually guarantees. The paper provides a principled way to assess this issue through upper bounds on off-factor variation. In particular, it proves that the fractional-cover certificate is the tightest uniform bound implied by the available action-specific energy upper bounds. This precise optimality result is a compelling theoretical contribution and provides a strong reason to view the paper positively.

Critical weaknesses

The guarantees for replay data rely on a strong distributional domination assumption: the reference pair distribution must be dominated by the replay distribution with a finite density-ratio bound. This condition may fail when the collected data do not cover relevant states or intervention outcomes, and it is difficult to verify from finite samples. Although direct sampling from the reference intervention experiment avoids this issue, the paper does not sufficiently establish how the assumption can be justified in practical offline-data settings. This limits the practical applicability of the replay-based guarantees.

Questions for authors

The Pendulum experiment uses exhaustive enumeration of all 7,308 configurations to obtain the range bounds and variance caps, but the same enumeration also permits direct computation of the exact leakage. The paper acknowledges this cost. Have you considered how to obtain valid, sufficiently tight range bounds without exhaustive enumeration? Can you clarify whether the resulting certificates would remain informative and provide a computational advantage over direct leakage evaluation in larger settings? This clarification would help assess the method's practical utility beyond the finite-grid audit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.