acceptodds
Under review as a conference paper at ICLR 2027

Adversarial Information Separation for Data-Scarce Generalization under Noisy Observations

Abstract

Learning from scarce data is difficult, e.g., in scientific domains where task-relevant information is often confounded with measurement noise and experimental artifacts. Compression-based regularization like the variational information bottleneck (VIB) improves generalization by compressing representations uniformly, but may suppress weak predictive information along with noise. In contrast, adversarial separation methods preserve such information, but require explicit separation labels. To address this challenge, we propose the Adversarial Information Separation Framework (AdverISF), which learns primary and residual representations from task supervision alone. It uses a task-guided adversarial separation mechanism to reduce the statistical dependence between the representations and a multi-layer separation architecture in which subsequent layers process residual representations to extract predictive information missed by earlier layers. Experiments show that AdverISF outperforms state-of-the-art methods under severe data scarcity. On a real materials design task, it generalizes to formulations independently prepared in a subsequent batch.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Adversarial Training, Variational Information Bottleneck, Supervised Learning, AI for Science

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BUSBOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AdverISF to improve generalization under data scarcity and noisy observations. It is motivated by the observation that conventional information bottleneck methods may suppress weak yet predictive information along with noise. AdverISF employs dual-branch encoders to learn primary and residual representations and uses adversarial training to reduce their statistical dependence. Its multi-layer separation architecture further recycles residual representations to progressively recover complementary predictive information missed by earlier blocks. Experiments demonstrate promising results under extreme data scarcity and on a real-world composite cement design task.

Critical strengths
  1. The paper addresses a practically important problem: in domains such as materials science, data are often scarce and affected by measurement noise and batch variation. AdverISF requires only task supervision and no explicit domain or separation labels, making it well suited to such settings.

  2. Recycling residual representations through subsequent separation layers to recover information missed by earlier blocks is an interesting design. AdverISF shows consistent gains in extremely data-scarce settings and generalizes well to independently prepared cement formulations from a subsequent experimental batch, providing evidence of its practical value.

Critical weaknesses
  1. The core mechanism is not directly validated. The experiments mainly report final predictive performance, without showing whether the statistical dependence between zpz_p and zrz_r is actually reduced or whether subsequent separation blocks recover subtle predictive information missed by earlier blocks. The current evidence therefore seems insufficient to identify which components drive the observed gains.

  2. The real-world composite cement experiment contains only nine test formulations, making the reported R2R^2 potentially sensitive to individual samples. Moreover, the full model and the variant without adversarial separation perform similarly in some settings, so the consistency of the gains from the adversarial separation mechanism remains unclear.

Questions for authors
  1. Could the authors provide more direct evidence that adversarial training reduces the statistical dependence between zpz_p and zrz_r, and that subsequent separation blocks recover subtle predictive information missed by earlier blocks? This would help determine whether the proposed mechanism works as intended.

  2. The real-world composite cement experiment includes only nine test formulations. Could the authors clarify how sensitive the reported R2R^2 is to individual samples and provide an uncertainty analysis? This would help assess the stability of the result.

4 comments

We thank Reviewer BUSB for the encouraging assessment and thoughtful questions on our mechanism validation and real-world evaluation, and below we address both Critical Weaknesses (W1–W2) and Questions (Q1–Q2) in detail.


W1, Q1: We thank the reviewer for raising this important point. We would like to clarify that we target functional predictive information separation rather than strict semantic disentanglement, since the latter is impossible without structural priors (Locatello et al., ICML 2019). We validate the core mechanism with two analyses on the Synthetic dataset. Table A shows that increasing λadv\lambda_{\text{adv}} from 0 to 0.5 reduces the statistical dependence between zp1z_p^1 and zr1z_r^1, lowering distance correlation (dCor) from 0.763 to 0.405 and the Hilbert-Schmidt Independence Criterion (HSIC) from 0.501 to 0.143 toward the empirical independence baselines.

Table B further probes what each layer captures on the Synthetic dataset. While zp1z_p^1 learns the primary predictive signal and zr1z_r^1 preserves complementary information that lifts R2R^2 from 0.268 (y1y_1) to 0.301 (y2y_2), the subtle component fsubf_{\text{sub}} is already near the noise floor in zr1z_r^1 (0.050±0.0670.050 \pm 0.067) because no factor-level supervision is available. Consequently, zp2z_p^2 recovers remaining dominant signal fdomf_{\text{dom}} left in zr1z_r^1 (R2=0.092R^2 = 0.092) rather than isolating fsubf_{\text{sub}} (R2=−0.017R^2 = -0.017), bringing the final R2R^2 to 0.304 (y3y_3) while all representations yield negative R2R^2 on measurement noise. Overall, these findings confirm functional recovery of missed predictive information rather than strict semantic separation between fdomf_{\text{dom}} and fsubf_{\text{sub}}.

Table A. Effect of the adversarial weight λadv\lambda_{\text{adv}} on the statistical dependence between representations on the Synthetic dataset.

λadv\lambda_{\text{adv}}dCor(zp1,zr1z_p^1, z_r^1)HSIC(zp1,zr1z_p^1, z_r^1)
00.7626 ± 0.01750.5009 ± 0.0279
0.10.4888 ± 0.04210.1997 ± 0.0354
0.30.4539 ± 0.04740.1671 ± 0.0365
0.50.4051 ± 0.09290.1430 ± 0.0661

Note. dCor and HSIC measure nonlinear statistical dependence between zp1z_p^1 and zr1z_r^1 (lower values indicate weaker dependence). Across settings, the empirical independence baselines obtained by randomly permuting one representation across samples are around 0.18 for dCor and 0.03 for HSIC.

Table B. Information about the true generative factors and measurement noise captured by each representation on the Synthetic dataset. The overall predictor R2R^2 is 0.268 for y1y_1, 0.301 for y2y_2, and 0.304 for y3y_3.

Representationfdomf_{\text{dom}} (R2R^2)fsubf_{\text{sub}} (R2R^2)Measurement Noise (R2R^2)
zp1z_p^10.353 ± 0.0860.079 ± 0.061−0.038 ± 0.010
zr1z_r^10.209 ± 0.1140.050 ± 0.067−0.074 ± 0.041
zp2z_p^20.092 ± 0.128−0.017 ± 0.051−0.042 ± 0.021

Note. For the overall prediction task, y1y_1, y2y_2, and y3y_3 denote the primary prediction from zp1z_p^1, the joint prediction from [zp1,zr1][z_p^1, z_r^1], and the final prediction from [zp1,zp2][z_p^1, z_p^2], respectively.

W2, Q2: We thank the reviewer for raising both concerns. To assess whether our results depend on individual samples, we ran Jackknife (leave-one-out) and Bootstrap (2,000 resamples) sensitivity analyses. Table C reports the results. Even in the Jackknife worst case, where the single most favorable sample is removed, the R2R^2 of AdverISF (Joint) remains at 0.871. This remains higher than the full-test R2R^2 of infoR-LSF (0.795). Among the methods included in this sensitivity analysis, AdverISF (Joint) also achieves the highest lower bound of the Bootstrap 95% CI (0.65). These results indicate that the improvement is consistent and is not driven by a few outlier samples.

Regarding the consistency of the adversarial separation mechanism, our analysis on Figure 4 (which directly isolates the effect of λadv\lambda_{\text{adv}}) indicates that its benefit scales with task difficulty. Under the harsh L2-15D setting with extreme noise and nonlinearity, introducing the adversarial penalty produces a statistically significant improvement (ΔR2=+0.015\Delta R^2 = +0.015, Wilcoxon signed-rank test p=0.049p = 0.049, t-test p=0.037p = 0.037). On simpler regimes where the primary encoder already captures most of the predictive structure, the gap remains small without harming performance, which explains why the full model and the variant without adversarial separation perform similarly in those settings.

Table C. Sensitivity analysis on the 9 blind-test samples.

MethodTest R² (Mean ± Std)Jackknife Worst CaseJackknife Best CaseMax Drop (ΔR²)Bootstrap 95% CI
MLP−1.559 ± 0.611−2.055−0.9670.496[−5.77, −0.28]
VIB0.617 ± 0.0790.5570.7210.060[−0.04, 0.78]
infoR-LSF0.795 ± 0.0770.7520.8760.043[0.36, 0.91]
AdverISF (Two-Stage)0.882 ± 0.0510.8540.9300.029[0.61, 0.97]
AdverISF (Joint)0.897 ± 0.0390.8710.9350.027[0.65, 0.97]

References

  • Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., & Bachem, O. (2019). Challenging common assumptions in the unsupervised learning of disentangled representations. Proceedings of the 36th International Conference on Machine Learning, PMLR 97:4114–4124.

Thank you for the author’s response. I will maintain my original score.

As the discussion period nears its end, thank you for taking the time to review our work and to engage with our response.

Reviewer_kzDNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the challenge of learning under the presence of confounding noise with limited data. The general idea of their approach is to learn disentangled representations to overcome these nuisance factors. Their method applies the variational information bottleneck (VIB) principle and also uses adversarial separation to encourage independence between different representations. Their proposed Adversarial Information Separation Framework (AdverSIF) has a primary representation branch that learns the representation zpz_p while applying the VIB principle, involving an encoder and predictor that are trained on a VIB-style loss (given in Equation 3) that effectively aims at maximizing I(zp;y)−βpI(zp;x)I(z_p; y) - \beta_p I(z_p; x), where yy is the task label and xx are the input features. AdverSIF also has a residual branch with an encoder/predictor that is also conditioned on zpz_p to learn the residual representation zrz_r via a conditional VIB-style loss (given in Equation 4) that effectively aims at maximizing I(zr;y∣zp)−βrI(zr;x∣zp)I(z_r; y | z_p) - \beta_r I(z_r; x | z_p). Another ingredient to their method is estimating a proxy for the dependence (via a Wasserstein GAN style loss) between zpz_p and zrz_r, which is added as an additional loss term weighted by λadv\lambda_\text{adv} to encourage these representations to be independent. Their methodology description also suggests how these AdverSIF blocks can be recursively layered (generally with two more representations per layer, except the last layer), but it seems that in their implemented evaluations, they only consider adding one additional and partial layer that only introduces one more "primary" branch representation zp2z_p^2. They also conduct a reasonable amount of experiments and ablation study with real and synthetic datasets, which seems to demonstrate the advantages of AdverSIF over baselines, particularly in the regime of limited training data.

Critical strengths

The paper seems to empirically demonstrate an advantage of their method in the low training data regime, across several different evaluation sets. However, this advantage seems to typically disappear (falling behind the baselines) as training data increases. The general concept of applying VIB and adversarial separation seems appealing.

Critical weaknesses

I think the general philosophical explanation of Sections 3.1-3.3 and Appendix A.3, which aim to motivate the design of the method, is a bit hand-wavy and not firmly justified. For example, in lines 220-221, the authors express that the aim is to separate and allocate task relevant features vs nuisance factors into zpz_p and zrz_r. However, it's not entirely clear that this is really what's going on with their method. Ultimately, all of the representations are trained to serve task prediction, and lines 357-358 seem to state that all of the learned representations are aggregated to make the final task prediction (note: the multi-layer aspect needs to be better clarified overall). If zrz_r only contained nuisance factors, it wouldn't make much sense to utilize that for task prediction. It seems that the adversarial separation could be interpreted as aiming to separate out independent nuisance factors, however the adversarial weights that are mainly used (and found to be most effective) are relatively small, with the main dominant training term for the residual representation appearing to be training it for task prediction, via the first term of the loss given in equation 4. Even the VIB-style regulation term, which is weighted by βr\beta_r, may just be playing a minor regularizing role, as βr\beta_r weights used were relatively small.

Note that another way to look at it is to consider the almost symmetry of how zpz_p and zrz_r are trained and how their losses are setup. Although they are not practically symmetric as the encoder/predictor for zrz_r is conditioned on zpz_p, the effective target for training zpz_p and zrz_r can be combined via the chain rule of mutual information, i.e., the maximization objectives could be combined as: $ I(z_p; y) - \beta_p I(z_p; x) + I(z_r; y | z_p) - \beta_r I(z_r; x | z_p) = I(z_p, z_r ; y) - \beta I(z_p, z_r; x), $ if we set βr=βp=β\beta_r = \beta_p = \beta. Thus, although the asymmetry of conditioning and differing β\beta weights may affect things in practice, we have that, in principle, they could almost be viewed somewhat symmetrically, and thus it's not entirely natural to say that one or the other should contain more or less task features vs nuisance factors.

The presentation of the paper could be significantly improved, with some key parts of the architecture not being clearly explained. Specifically, layering multiple Adversarial Information Separation Blocks is only vaguely and briefly described in Section 3.4, and the Figure 2 does not help shed much light either. It seems that if there were more than two layers, then all of the intermediate layers (not including the last one) should produce another primary and residual representation. However, the last layer seems to only produce another primary representation, but no residual? In their implementation and experiments, it seems that they always only consider two total layers, so there are three representations in total, zp1z_p^1, zr1z_r^1, and zp2z_p^2. In such a case, it's not entirely clear exactly what training loss applies to zp2z_p^2 or if zp2z_p^2 is involved in any adversarial separation terms.

I would guess that zp2z_p^2 is trained with a loss that is similar to equation 4, but with the following symbolic replacements: zr→zp2z_r \rightarrow z_p^2, zp→zp1z_p \rightarrow z_p^1, and x→zr1x \rightarrow z_r^1, and that the corresponding objective would be maximizing I(zp2;y∣zp1)−βrI(zp2;zr1∣zp1)I(z_p^2; y | z_p^1) - \beta_r I(z_p^2; z_r^1 | z_p^1). I recommend that the authors confirm and clarify this. Perhaps, instead of presenting it as a general LL architecture, they could just simplify the exposition as a two-layer architecture, which is anyways what the experiments appear to be limited to.

Section 3.3.3 presents the adversarial separation mechanism in terms of a vanilla GAN-like loss given in equation 5. This loss-term is also state as part of the overall "final training objective" given in equation 6. However, then the authors seem to actually use a different Wasserstein GAN style loss given in equation 7, i.e., it appears that LDL_D should replace LadvL_\text{adv} for the actual experiments. If only the Wasserstein GAN style loss is used, perhaps the exposition of the vanilla GAN alternative (in both section 3.3.3 and Appendix A.1) should just be dropped to save space and avoid confusion.

While the experiments so seem to demonstrate that the method works better in the regime of limited training data, it is also worth noting that performance tends to become comparatively worse when there is more training data available. It would be worthwhile to try to determine and understand the reasons for this. It would be disappointing if the method is only better for the limited data regime, or if hyper-parameters had to be tuned for effectiveness over different regimes.

I'm also curious if the experiments really establish a significant advantage for the adversarial separation. In Figures 5-8 in the Appendix, it appears that best performance is realized with the adversarial weight λadv\lambda_\text{adv} at the lowest end, and in the ablation study Figure 4 (from the main paper), which compares to the (A3) λadv=0\lambda_\text{adv} = 0 case, it seems that the difference is sometimes negative and sometimes positive, depending on the experimental conditions, and the significance of this difference is unclear. I wonder if it is not just maybe the VIB training loss doing the heavy lifting here, and the effect of adversarial separation may only be negligible or detrimental (for too large λadv\lambda_\text{adv}).

Questions for authors

It would appear the implementation for the experiments uses only two layers of the Adversarial Information Separation Blocks (i.e., L=2L = 2 for the architecture depicted in Figure 2), and that this second layer only has primary branch, so there are a total of three latent representations, zp1z_p^1, zr1z_r^1, and zp2z_p^2. Is this correct? Also, does the Adversarial Separation Mechanism only apply to zp1z_p^1 and zr1z_r^1, or is there something else to also adversarially separate zp2z_p^2 from the rest?

For this second layer, could you please clarify (or confirm) what the loss for training zp2z_p^2 would be? Note: please see a guess that I made about this in the Weaknesses box above.

Across the evaluations, it appears that the AdverISF method offers an advantage against the baselines in the regimes of limited training data, however the performance appears to typically be worse in regimes with more training data. Why does the advantage not persist (and instead the method falls behind) when more training data is available?

Has the advantage of using non-zero λadv\lambda_\text{adv} been established with statistical significance?

6 comments

We thank Reviewer kzDN for the careful and detailed review of our methodology, mathematical formulations, and experimental analysis, and below we address all Critical Weaknesses (W1-W7) and Questions (Q1-Q4) grouped by topic.


W1, W2: Thank you for this thoughtful critique. Your comments made us realize that our wording in Sections 3.1 to 3.3 and Appendix A.3 could read as claiming an unrealistic separation between pure task features and pure noise. In the revision, we will state our position explicitly.

Ideally, a single compressed representation zpz_p would capture all predictive signals while discarding all nuisance factors. Under data scarcity and bottleneck compression, however, zpz_p inevitably misses part of the useful signal. We therefore design zrz_r as a residual channel that retains this missed predictive information alongside residual nuisance variations, and include the joint task predictor on [zp,zr][z_p, z_r] so that zrz_r does not collapse into uninformative noise before reaching subsequent layers. Regarding your chain-rule observation, the equivalence holds only when both branches share the same prediction target and βp=βr\beta_p = \beta_r, and in our method neither condition holds. First, zpz_p is trained with its own standalone predictor qψ1(y∣zp)q_{\psi_1}(y \mid z_p) in addition to the joint predictor, whereas zrz_r has no marginal predictor and is conditioned on zpz_p, rewarding only complementary signal. Second, setting βp≠βr\beta_p \neq \beta_r imposes asymmetric bottleneck pressure, pushing zpz_p to prioritize dominant signals while leaving unabsorbed predictive content for zrz_r. Tables A and B on the Synthetic dataset support this functional asymmetry. Increasing λadv\lambda_{\text{adv}} from 0 to 0.5 lowers statistical dependence between zp1z_p^1 and zr1z_r^1 in Table A, while in Table B zr1z_r^1 lifts R2R^2 from 0.268 (y1y_1) to 0.301 (y2y_2) and zp2z_p^2 recovers remaining fdomf_{\text{dom}} signal (R2=0.092R^2 = 0.092, reaching 0.304 for y3y_3) without fitting measurement noise, even though weak fsubf_{\text{sub}} in zr1z_r^1 (0.050±0.0670.050 \pm 0.067) remains near the noise floor without factor supervision.

Table A. Effect of the adversarial weight λadv\lambda_{\text{adv}} on the statistical dependence between representations on the Synthetic dataset.

λadv\lambda_{\text{adv}}dCor(zp1,zr1z_p^1, z_r^1)HSIC(zp1,zr1z_p^1, z_r^1)
00.7626 ± 0.01750.5009 ± 0.0279
0.10.4888 ± 0.04210.1997 ± 0.0354
0.30.4539 ± 0.04740.1671 ± 0.0365
0.50.4051 ± 0.09290.1430 ± 0.0661

Note. dCor and HSIC measure nonlinear statistical dependence between zp1z_p^1 and zr1z_r^1 (lower values indicate weaker dependence). Across settings, the empirical independence baselines obtained by randomly permuting one representation across samples are around 0.18 for dCor and 0.03 for HSIC.

Table B. Information about the true generative factors and measurement noise captured by each representation on the Synthetic dataset. The overall predictor R2R^2 is 0.268 for y1y_1, 0.301 for y2y_2, and 0.304 for y3y_3.

Representationfdomf_{\text{dom}} (R2R^2)fsubf_{\text{sub}} (R2R^2)Measurement Noise (R2R^2)
zp1z_p^10.353 ± 0.0860.079 ± 0.061−0.038 ± 0.010
zr1z_r^10.209 ± 0.1140.050 ± 0.067−0.074 ± 0.041
zp2z_p^20.092 ± 0.128−0.017 ± 0.051−0.042 ± 0.021

Note. For the overall prediction task, y1y_1, y2y_2, and y3y_3 denote the primary prediction from zp1z_p^1, the joint prediction from [zp1,zr1][z_p^1, z_r^1], and the final prediction from [zp1,zp2][z_p^1, z_p^2], respectively.


W3, Q1: Thank you for the careful reading of our architecture. Because of space limits in the original submission, the description of the multi-layer design in Section 3.4 was too condensed. We will expand this section with explicit formulations in the revision.

Our architecture supports a general LL-layer recursion. For each intermediate layer n<Ln < L, the residual zrn−1z_r^{n-1} (with zr0=xz_r^0 = x) serves as the input for extracting both zpnz_p^n and zrnz_r^n, and a layer-specific discriminator performs adversarial separation between them. In each intermediate layer, the preceding primary representations zp1,…,zpn−1z_p^1, \dots, z_p^{n-1} are concatenated with zpnz_p^n and zrnz_r^n respectively and fed into two corresponding predictors to estimate yy. For the final layer n=Ln = L, no further residual needs to be passed down, so we only extract the primary representation zpLz_p^L, which is concatenated with zp1,…,zpL−1z_p^1, \dots, z_p^{L-1} and fed into a single predictor.

In this paper, we instantiate the basic L=2L = 2 configuration to validate the core framework while keeping the model compact under data scarcity. As you correctly inferred, this gives three representations (zp1z_p^1, zr1z_r^1, zp2z_p^2), and adversarial separation applies only to (zp1,zr1)(z_p^1, z_r^1).

W4, Q2: Your reconstruction of the training objective for zp2z_p^2 captures the essence of our framework. To be as precise as possible regarding our implementation, we would like to clarify the conditioning flow in the encoder.

In the second layer, the predictor qψ3(y∣zp1,zp2)q_{\psi_3}(y \mid z_p^1, z_p^2) explicitly conditions on zp1z_p^1, while on the encoder side the dependence on zp1z_p^1 is already carried implicitly through zr1∼pϕ(zr1∣x,zp1)z_r^1 \sim p_\phi(z_r^1 \mid x, z_p^1). Thus, the encoder takes only zr1z_r^1 as input and is parameterized as pθ(zp2∣zr1,zp1)=pθ(zp2∣zr1)p_\theta(z_p^2 \mid z _ r ^ 1, z _ p ^ 1) = p_\theta(z_p^2 \mid z_r^1). Specifically, the training objective for zp2z_p^2 is

Lp2=Ex,y,zp1,zr1[Ezp2∼pθ(zp2∣zr1)[−log⁡qψ3(y∣zp1,zp2)]+βp2⋅DKL[pθ(zp2∣zr1)∥r(zp2)]],\mathcal{L} _ {p^2} = \mathbb{E} _ {x, y, z_p^1, z_r^1}\left[\mathbb{E} _ {z_p^2 \sim p_{\theta}(z_p^2 \vert{} z_r^1)} [-\log q _ {\psi_3}(y\vert{}z_p^1,z_p^2)] + \beta _ {p2} \cdot D _ {KL}[p _ {\theta}(z_p^2\vert{}z_r^1) \parallel r(z_p^2)]\right],

where βp2\beta_{p2} (reported in Table 6 in the manuscript) is the KL weight. We will add these formulations to Section 3.4 to make the multi-layer mechanism fully clear. Thank you for helping us improve the technical precision of the paper.


W5: Thank you for pointing out the inconsistent loss notation across sections. In the original draft, we first introduced the standard adversarial formulation (Eqs. 5 and 6) to present the separation intuition in a familiar form, while adopting the WGAN-GP formulation (LDL_D in Eq. 7) in the experiments for more stable optimization, which is a common practice in adversarial representation learning. We recognize that keeping both forms in the exposition caused unnecessary confusion, and we appreciate you bringing this to our attention.

In the revised manuscript, we will follow your suggestion and unify the exposition around the WGAN-GP objective used in our implementation. Specifically, we will replace the vanilla GAN formulation in Section 3.3.3 and the overall training objective (Eqs. 5 and 6) with the WGAN-GP formulation (LDL_D in Eq. 7), and adjust Appendix A.1 accordingly so that the notation and exposition remain consistent throughout the paper. We thank you for helping us improve the clarity and consistency of the manuscript.


W6, Q3: Thank you for this sharp observation. To ensure a fair comparison, within each dataset we used a single set of hyperparameters across all sample sizes rather than tuning them separately for any specific data regime. Compared with standard baselines such as MLP or single-branch VIB, AdverISF combines multi-layer bottleneck compression (β\beta) with adversarial separation (λadv\lambda_{\text{adv}}), imposing stronger overall regularization. While this regularization strength helps prevent overfitting when samples are scarce, keeping the same constraints fixed in larger-data regimes restricts effective model capacity and can cause underfitting. Because AdverISF targets scarce and noisy settings, we will state this intended scope and regularization trade-off clearly in the revision.


W7, Q4: We thank the reviewer for this question. Using the data from Figure 4, we ran Wilcoxon signed-rank test and t-test across random seeds comparing non-zero λadv\lambda_{\text{adv}} against λadv=0\lambda_{\text{adv}} = 0. In simpler or low-noise settings, the differences fluctuate around zero and are not significant, indicating that the non-adversarial architecture (λadv=0\lambda_{\text{adv}} = 0) already handles mild regimes and that adding a small adversarial penalty does not degrade performance in the tested runs. In the high-noise, nonlinear L2-15D setting, however, non-zero λadv\lambda_{\text{adv}} yields a statistically significant gain (ΔR2=+0.015\Delta R^2 = +0.015, Wilcoxon signed-rank test p=0.049p = 0.049, t-test p=0.037p = 0.037), providing robustness beyond the λadv=0\lambda_{\text{adv}} = 0 variant. Similarly, in Appendix Figures 5-8, large λadv\lambda_{\text{adv}} over-penalizes useful signal, so effective values naturally lie at the lower non-zero end.

Reviewer_kzDNOpenReview ↗

Response to author's response

Thank you for the detailed response that carefully addresses various issues that I raised.

At this moment, I feel that I must maintain my score. While the author's response have indicated ways that they plan to address the presentation issues in the manuscript, it seems that the PDF has not been revised, so I cannot verify exactly how they have implemented those plans.

Further, I think some important issues remain open:

  1. It seems that the authors plan to keep an L-layer architecture presentation, while only validating the two-layer version in their experiments. I think that this may continue to undermine the presentation clarity and potentially cause confusion. Further, it remains unclear whether adding more layers would really offer any substantial gain, or if there are additional design issues that would need to be ironed out to make it worth while.
  2. The issue of demonstrating which regimes have statistically significant advantages is critical. The authors should spend more effort to carefully determine if whether anything they are doing is worth while. In the response headed with "W7, Q4", the authors have briefly sketched out some statistical significance tests that seem to suggest that dropping the adversarial loss (with λadv=0\lambda_\text{adv} = 0) may quite often work best? Further, regarding the response headed by "W6, Q3", it seems that it is fairly severe limitation that the method may have such hyper-parameter sensitivity the limits its performances in larger-data regimes. How would one practically delineate such regime in practice to know if AdverSIF should be employed, or how to tune the hyper-params to flexibly deal with this?
Reviewer_fPpQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdverISF to improve generalization under limited and noisy training data. It combines primary and residual representations, adversarial independence regularization, and hierarchical residual recycling to preserve complementary predictive information. Experiments on standard benchmarks, synthetic data, and a real-world cement design task show promising results, particularly in data-scarce settings. I find the motivation and overall design interesting, and the empirical results are promising. However, I would like to see stronger evidence that the proposed adversarial separation mechanism is responsible for the improvements, along with more comprehensive baseline comparisons.

Critical strengths

The paper addresses a meaningful problem, especially for scientific applications where labeled data are difficult to collect. The motivation that aggressive compression may discard weak but useful predictive signals is reasonable.

I find the idea of progressively extracting information from residual representations interesting. Combining this with task-guided adversarial separation provides a potentially useful approach without requiring additional separation labels.

The improvements in several extremely data-scarce settings are encouraging. I also appreciate the effort to evaluate the method on a newly prepared experimental batch rather than relying solely on conventional random splits.

Critical weaknesses

I am not fully convinced that encouraging independence between the primary and residual representations necessarily separates task-relevant information from noise. The theoretical analysis also seems to assume that complementary information is successfully preserved and recovered, rather than establishing when this happens.

Looking at Figure 3, the full model (A0) and the version without adversarial separation (A3) perform similarly in several settings. This makes it difficult to assess how much the adversarial mechanism actually contributes, especially without direct evidence of information separation.

Given the focus on small-data regression, I would have liked to see comparisons with classical methods such as Gaussian processes, SVR, or tree-based ensembles. Differences in training budgets and additional regularization also make the current comparisons somewhat difficult to interpret.

The real-world cement experiment is interesting, but the independent test set contains only nine selected formulations. This makes it difficult to judge how robust the reported improvements would be across different experimental batches.

Questions for authors

Since the synthetic dataset contains ground-truth dominant and subtle factors, could the authors examine which factors are captured by each representation and whether later layers actually recover additional predictive information?

Could the authors elaborate on why the adversarial component provides relatively limited gains in some ablation settings? It would be helpful to understand whether stronger representation independence consistently translates into better generalization.

How does AdverISF compare with classical small-data regression methods under comparable tuning budgets? Could the authors also clarify the contribution of stochastic KL-weight perturbation?

How were the nine formulations in the real-world experiment selected, and were they fixed before comparing different methods? Could the authors provide further evidence of robustness across experimental batches?

4 comments

We thank Reviewer fPpQ for the constructive feedback and encouraging comments on our motivation, progressive residual extraction, and real-world evaluation, and below we address all Critical Weaknesses (W1–W4) and Questions (Q1–Q4) grouped by topic.


W1, Q1: Thank you for this helpful suggestion. We would like to clarify that we do not aim for strict semantic disentanglement. As Locatello et al. (ICML 2019) showed, unsupervised disentanglement is impossible without structural priors. We instead target functional predictive disentanglement, with the goal of improving generalization. As shown in Table A on the Synthetic dataset, introducing λadv\lambda_{\text{adv}} (increasing it from 0 to 0.5) reduces the distance correlation (dCor) between zp1z_p^1 and zr1z_r^1 from 0.763 to 0.405 and reduces the Hilbert-Schmidt Independence Criterion (HSIC) from 0.501 to 0.143, moving toward the empirical independence baselines under random permutation.

Regarding factor attribution on the Synthetic dataset in Table B, the distinction between dominant (fdomf_{\text{dom}}) and subtle (fsubf_{\text{sub}}) components is itself a semantic factor partition that is not explicitly supervised during training. Under strong noise and limited samples, zp1z_p^1 captures the primary predictive signal, while zr1z_r^1 retains residual useful information that raises the overall R2R^2 from 0.268 (y1y_1) to 0.301 (y2y_2). Because the fsubf_{\text{sub}} signal is weak and already near the noise floor in zr1z_r^1 (0.050±0.0670.050 \pm 0.067), the second-layer representation zp2z_p^2 primarily recovers unabsorbed fdomf_{\text{dom}} information left in zr1z_r^1 (R2=0.092R^2 = 0.092) rather than isolating fsubf_{\text{sub}} (R2=−0.017R^2 = -0.017), reaching a final R2R^2 of 0.304 (y3y_3). In addition, no representation fits the measurement noise, as all corresponding R2R^2 values are negative. Overall, these results support functional residual recovery of missed predictive signal rather than semantic separation of zdominantz_{\text{dominant}} and zsubtlez_{\text{subtle}}.

Table A. Effect of the adversarial weight λadv\lambda_{\text{adv}} on the statistical dependence between representations on the Synthetic dataset.

λadv\lambda_{\text{adv}}dCor(zp1,zr1z_p^1, z_r^1)HSIC(zp1,zr1z_p^1, z_r^1)
00.7626 ± 0.01750.5009 ± 0.0279
0.10.4888 ± 0.04210.1997 ± 0.0354
0.30.4539 ± 0.04740.1671 ± 0.0365
0.50.4051 ± 0.09290.1430 ± 0.0661

Note. dCor and HSIC measure nonlinear statistical dependence between zp1z_p^1 and zr1z_r^1 (lower values indicate weaker dependence). Across settings, the empirical independence baselines obtained by randomly permuting one representation across samples are around 0.18 for dCor and 0.03 for HSIC.

Table B. Information about the true generative factors and measurement noise captured by each representation on the Synthetic dataset. The overall predictor R2R^2 is 0.268 for y1y_1, 0.301 for y2y_2, and 0.304 for y3y_3.

Representationfdomf_{\text{dom}} (R2R^2)fsubf_{\text{sub}} (R2R^2)Measurement Noise (R2R^2)
zp1z_p^10.353 ± 0.0860.079 ± 0.061−0.038 ± 0.010
zr1z_r^10.209 ± 0.1140.050 ± 0.067−0.074 ± 0.041
zp2z_p^20.092 ± 0.128−0.017 ± 0.051−0.042 ± 0.021

Note. For the overall prediction task, y1y_1, y2y_2, and y3y_3 denote the primary prediction from zp1z_p^1, the joint prediction from [zp1,zr1][z_p^1, z_r^1], and the final prediction from [zp1,zp2][z_p^1, z_p^2], respectively.


W2, Q2: Thank you for the careful observation. In practice, stronger independence between representations does not necessarily lead to better generalization. As discussed in Section 4.4, an overly strong adversarial constraint can discard useful information. Regarding the similar performance of A0 and A3 in Figure 3, the benefit of the adversarial term depends strongly on task complexity and noise level. While Figure 3 presents overall architectural ablations, Figure 4 is specifically dedicated to analyzing the effect of the adversarial weight λadv\lambda_{\text{adv}}. We therefore performed our statistical significance analysis using the data from Figure 4 rather than Figure 3. This analysis shows that the adversarial term yields a statistically significant gain (ΔR2=+0.015\Delta R^2 = +0.015, Wilcoxon signed-rank test p=0.049p = 0.049, t-test p=0.037p = 0.037 ) in complex tasks with strong noise and nonlinearity, where theory predicts a benefit (the L2-15D setting with extreme noise). In simpler or noise-free settings, the difference is not significant. We thus read the small gap in simpler settings as evidence that the adversarial term is non-intrusive and does not degrade the performance of the model without adversarial regularization, while adding meaningful robustness under harsh conditions.

W3, Q3: Thank you for suggesting classical small-sample baselines. We agree that this comparison is important. We have added comparisons with GPR, RF, and SVR, and Table C reports the results. On the standard tabular benchmark (AEP), classical methods show a clear advantage, with RF performing best at both ratios (0.677 and 0.744) followed by GPR (0.513 and 0.646). As task complexity and interference increase, however, AdverISF becomes more competitive and takes the lead. On the Synthetic dataset, AdverISF (Two-Stage) matches RF at 10% (0.251 vs. 0.250) and outperforms all baselines at 70% (0.565 vs. 0.521 for GPR). On the Real-world Cement task with real batch effects, AdverISF (Joint) reaches 0.897, ahead of GPR (0.846) and RF (0.802). These results support the claim in our introduction that deep disentanglement architectures are especially valuable for small-sample problems with high noise and strongly entangled features, while classical methods remain competitive on standard tabular data.

Table C. Test R2R^2 of deep and classical methods. The best result in each row is in bold.

SettingMLPMLP+WDVIBinfoR-LSFAdverISF (Joint)AdverISF (Two-Stage)GPRRFSVR
AEP 10%0.3220.3980.4260.3460.4300.4310.5130.6770.243
AEP 70%0.5580.5560.6040.5710.5970.4610.6460.7440.557
Synthetic 10%0.2010.2050.1590.1650.1790.2510.0820.2500.204
Synthetic 70%0.4680.4700.4780.4350.3550.5650.5210.3030.475
Real-world Task−1.5590.0170.6170.7950.8970.8820.8460.8020.718

Regarding the stochastic KL-weight perturbation, it acts as a dynamic regularizer that stabilizes bottleneck training and mitigates collapse on small datasets. In our paired check on VIB, adding this perturbation stabilizes scarce-data performance, for example sharply reducing test variance on AEP 10% (standard deviation dropping from 0.321 to 0.020) and lifting Real-world Task R2R^2 to 0.790 ± 0.097. Crucially, even with stochastic KL perturbation enabled, standard VIB still falls clearly short of AdverISF on the Real-world Task (0.790 vs. 0.897), confirming that our primary gains come from the adversarial separation architecture rather than KL perturbation alone.


W4, Q4: We appreciate your practical perspective on the real-world evaluation. To clarify the selection process, the 9 formulations were sampled in earlier experiments using a combination of Bayesian optimization and MLP-based surrogate optimization. They were fixed before we evaluated our method, and because of the project timeline AdverISF was not involved in their design. These formulations therefore serve as an independent blind test set. Each formulation was physically prepared and cured for the standard 28 days before its compressive strength was measured.

Regarding cross-batch robustness, preparing new batches would require another 28-day curing cycle. We therefore ran Jackknife (leave-one-out) and Bootstrap (2,000 resamples) sensitivity analyses on the test-set predictions to simulate sample perturbation. Table D reports the results. Even in the Jackknife worst case, where the single sample most favorable to our model is removed, the R2R^2 of AdverISF drops by only 0.027 to 0.871. This remains higher than the full-test R2R^2 of infoR-LSF (0.795). Among the methods included in this sensitivity analysis, AdverISF (Joint) also achieves the highest lower bound of the Bootstrap 95% CI (0.65). These results indicate that the advantage of our model is consistent and is not driven by a few favorable samples.

Table D. Sensitivity analysis on the 9 blind-test formulations.

MethodTest R² (Mean ± Std)Jackknife Worst CaseJackknife Best CaseMax Drop (ΔR²)Bootstrap 95% CI
MLP−1.559 ± 0.611−2.055−0.9670.496[−5.77, −0.28]
VIB0.617 ± 0.0790.5570.7210.060[−0.04, 0.78]
infoR-LSF0.795 ± 0.0770.7520.8760.043[0.36, 0.91]
AdverISF (Two-Stage)0.882 ± 0.0510.8540.9300.029[0.61, 0.97]
AdverISF (Joint)0.897 ± 0.0390.8710.9350.027[0.65, 0.97]

References

  • Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., & Bachem, O. (2019). Challenging common assumptions in the unsupervised learning of disentangled representations. Proceedings of the 36th International Conference on Machine Learning, PMLR 97:4114–4124.

Thanks for adding the classical baselines, factor-attribution check, and blind-test clarification. Still, the new results reinforce my main concerns: the second layer fails to isolate subtle factors and adds little to prediction, and the adversarial term only helps marginally in one extreme-noise setting. Meanwhile, RF and GPR outperform AdverISF on standard tabular data, Joint vs. Two-Stage performance is inconsistent, and KL perturbation explains much of the gain over VIB. I will maintain my current rating.

As the discussion period draws to a close, we are grateful for your thoughtful review and for taking the time to consider our responses and additional experiments.

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your thoughtful reviews. The authors have provided detailed responses to the concerns raised.

I would appreciate it if you could revisit the author responses and clarify whether your main concerns have been adequately addressed, particularly which issues remain critical to your overall assessment.

Given the differences in the initial recommendations, I also encourage you to consider and discuss one another's key arguments to help clarify the remaining disagreements and develop a more consistent understanding of the paper's merits and limitations.

Please feel free to update your ratings if your assessments have changed following the discussion.

Thank you for your continued contributions.

We thank you for overseeing the review process and for encouraging the reviewers to engage with our responses. Thank you for your time and effort in facilitating this discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.