- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AdverISF to improve generalization under data scarcity and noisy observations. It is motivated by the observation that conventional information bottleneck methods may suppress weak yet predictive information along with noise. AdverISF employs dual-branch encoders to learn primary and residual representations and uses adversarial training to reduce their statistical dependence. Its multi-layer separation architecture further recycles residual representations to progressively recover complementary predictive information missed by earlier blocks. Experiments demonstrate promising results under extreme data scarcity and on a real-world composite cement design task.
Critical strengths
-
The paper addresses a practically important problem: in domains such as materials science, data are often scarce and affected by measurement noise and batch variation. AdverISF requires only task supervision and no explicit domain or separation labels, making it well suited to such settings.
-
Recycling residual representations through subsequent separation layers to recover information missed by earlier blocks is an interesting design. AdverISF shows consistent gains in extremely data-scarce settings and generalizes well to independently prepared cement formulations from a subsequent experimental batch, providing evidence of its practical value.
Critical weaknesses
-
The core mechanism is not directly validated. The experiments mainly report final predictive performance, without showing whether the statistical dependence between and is actually reduced or whether subsequent separation blocks recover subtle predictive information missed by earlier blocks. The current evidence therefore seems insufficient to identify which components drive the observed gains.
-
The real-world composite cement experiment contains only nine test formulations, making the reported potentially sensitive to individual samples. Moreover, the full model and the variant without adversarial separation perform similarly in some settings, so the consistency of the gains from the adversarial separation mechanism remains unclear.
Questions for authors
-
Could the authors provide more direct evidence that adversarial training reduces the statistical dependence between and , and that subsequent separation blocks recover subtle predictive information missed by earlier blocks? This would help determine whether the proposed mechanism works as intended.
-
The real-world composite cement experiment includes only nine test formulations. Could the authors clarify how sensitive the reported is to individual samples and provide an uncertainty analysis? This would help assess the stability of the result.
4 comments
W2, Q2: We thank the reviewer for raising both concerns. To assess whether our results depend on individual samples, we ran Jackknife (leave-one-out) and Bootstrap (2,000 resamples) sensitivity analyses. Table C reports the results. Even in the Jackknife worst case, where the single most favorable sample is removed, the of AdverISF (Joint) remains at 0.871. This remains higher than the full-test of infoR-LSF (0.795). Among the methods included in this sensitivity analysis, AdverISF (Joint) also achieves the highest lower bound of the Bootstrap 95% CI (0.65). These results indicate that the improvement is consistent and is not driven by a few outlier samples.
Regarding the consistency of the adversarial separation mechanism, our analysis on Figure 4 (which directly isolates the effect of ) indicates that its benefit scales with task difficulty. Under the harsh L2-15D setting with extreme noise and nonlinearity, introducing the adversarial penalty produces a statistically significant improvement (, Wilcoxon signed-rank test , t-test ). On simpler regimes where the primary encoder already captures most of the predictive structure, the gap remains small without harming performance, which explains why the full model and the variant without adversarial separation perform similarly in those settings.
Table C. Sensitivity analysis on the 9 blind-test samples.
| Method | Test R² (Mean ± Std) | Jackknife Worst Case | Jackknife Best Case | Max Drop (ΔR²) | Bootstrap 95% CI |
|---|---|---|---|---|---|
| MLP | −1.559 ± 0.611 | −2.055 | −0.967 | 0.496 | [−5.77, −0.28] |
| VIB | 0.617 ± 0.079 | 0.557 | 0.721 | 0.060 | [−0.04, 0.78] |
| infoR-LSF | 0.795 ± 0.077 | 0.752 | 0.876 | 0.043 | [0.36, 0.91] |
| AdverISF (Two-Stage) | 0.882 ± 0.051 | 0.854 | 0.930 | 0.029 | [0.61, 0.97] |
| AdverISF (Joint) | 0.897 ± 0.039 | 0.871 | 0.935 | 0.027 | [0.65, 0.97] |
References
- Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., & Bachem, O. (2019). Challenging common assumptions in the unsupervised learning of disentangled representations. Proceedings of the 36th International Conference on Machine Learning, PMLR 97:4114–4124.
Thank you for the author’s response. I will maintain my original score.
As the discussion period nears its end, thank you for taking the time to review our work and to engage with our response.
We thank Reviewer BUSB for the encouraging assessment and thoughtful questions on our mechanism validation and real-world evaluation, and below we address both Critical Weaknesses (W1–W2) and Questions (Q1–Q2) in detail.
W1, Q1: We thank the reviewer for raising this important point. We would like to clarify that we target functional predictive information separation rather than strict semantic disentanglement, since the latter is impossible without structural priors (Locatello et al., ICML 2019). We validate the core mechanism with two analyses on the Synthetic dataset. Table A shows that increasing λadv from 0 to 0.5 reduces the statistical dependence between zp1 and zr1, lowering distance correlation (dCor) from 0.763 to 0.405 and the Hilbert-Schmidt Independence Criterion (HSIC) from 0.501 to 0.143 toward the empirical independence baselines.
Table B further probes what each layer captures on the Synthetic dataset. While zp1 learns the primary predictive signal and zr1 preserves complementary information that lifts R2 from 0.268 (y1) to 0.301 (y2), the subtle component fsub is already near the noise floor in zr1 (0.050±0.067) because no factor-level supervision is available. Consequently, zp2 recovers remaining dominant signal fdom left in zr1 (R2=0.092) rather than isolating fsub (R2=−0.017), bringing the final R2 to 0.304 (y3) while all representations yield negative R2 on measurement noise. Overall, these findings confirm functional recovery of missed predictive information rather than strict semantic separation between fdom and fsub.
Table A. Effect of the adversarial weight λadv on the statistical dependence between representations on the Synthetic dataset.
Note. dCor and HSIC measure nonlinear statistical dependence between zp1 and zr1 (lower values indicate weaker dependence). Across settings, the empirical independence baselines obtained by randomly permuting one representation across samples are around 0.18 for dCor and 0.03 for HSIC.
Table B. Information about the true generative factors and measurement noise captured by each representation on the Synthetic dataset. The overall predictor R2 is 0.268 for y1, 0.301 for y2, and 0.304 for y3.
Note. For the overall prediction task, y1, y2, and y3 denote the primary prediction from zp1, the joint prediction from [zp1,zr1], and the final prediction from [zp1,zp2], respectively.