- Rating
- 2
- Confidence
- 2
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ECTR for settings where spurious correlations change across environments and difficult examples receive too little attention within each environment. Its main idea is to learn sample weights and use the same weighted risk for prediction and the TV stationarity penalty. Weights are normalized within each environment, and a KL penalty limits their concentration. The paper analyzes the resulting weighting signal and evaluates the method through mixed-shift ablations, benchmark comparisons, and studies of corruption and environment information.
Critical strengths
-
The shared-risk design is well motivated. I find it sensible to evaluate stationarity on the same weighted risk used for prediction. Appendix A.4 makes the effect on the adversary update explicit. The Main-only ablation directly tests this choice. I consider this the paper's strongest result.
-
The additional controls help explain the results. I appreciate the comparison using identical frozen environment memberships for ECTR and TV-IRM , which helps separate the objective's benefit from environment inference. The corruption study also measures where the sample weight goes. This supports the claim that ECTR places less excess weight on corrupted examples than these fixed-tail rules.
Critical weaknesses
-
A shared weighted risk need not preserve the invariant prediction target. I understand the motivation for aligning prediction and stationarity. My main concern is that weights depend on the full input, including spurious features, so they can change the conditional label distribution given the invariant features. Preserving the outer environment coefficients does not prevent this change. For a simple squared-loss example, let the input be , with independent, uniform and . Set in one environment and in the other. Both environments have , and the invariant predictor has zero unweighted probe in both. A single input-only scorer can give three times the weight of . After normalization, these groups receive mass and in each environment. The weighted conditional means become and , and the probes at the same predictor become and . The KL cost relative to uniform weighting is finite, about . All samples initially have squared loss 1, so this violation is introduced by the stationarity-aware reweighting itself. This does not establish that ECTR fails in the reported tasks, but it shows why sharing a risk is not sufficient to justify the new stationarity target. KL controls the amount of reweighting; it does not require preservation of the invariant relationship. I would appreciate conditions or a controlled analysis explaining when this mechanism exposes useful tail variation without penalizing a valid invariant predictor. The current bounds, distributional formulas, and optimization results do not resolve this issue for me.
-
The benefit of using stationarity to learn the weights is not isolated. Full and Main-only differ in two ways: which risk is tested for stationarity, and whether TV contributes to the adversary's update. Their comparison supports the combined change, but does not distinguish the value of sharing the risk from the value of feeding stationarity back into weight learning. Appendix A.4 shows that the update directions differ; it does not establish that the additional direction favors examples useful for invariant prediction. I would find an aligned loss-only adversary particularly informative: learn weights from , then use those weights in both prediction and TV, without a TV gradient in the weight update. This retains the shared risk and removes only the stationarity feedback to the adversary. The Top- and CVaR controls use different selection rules and do not provide this comparison. Without it, I am not yet convinced that stationarity-dependent weight learning is responsible for the gains over simpler aligned reweighting. This distinction matters because that feedback can also alter the invariant target, as illustrated above.
Questions for authors
-
Under what conditions does the input-dependent reweighting preserve the invariant conditional prediction rule? How would ECTR behave in a controlled example such as W1, where a valid invariant predictor is stationary before reweighting but not afterwards?
-
Could you compare Full with a loss-only KL adversary whose weights are used in both prediction and TV, with TV excluded only from the adversary update? Using the same architecture and validation budget would help determine the contribution of stationarity-dependent weight learning.
4 comments
Q2 W2. Contributions of risk alignment and TV feedback
We thank the reviewer for their careful reading and thoughtful comments. Aligned loss-only isolates the feedback contribution. Both Aligned and Full use the conditional weighted risk in prediction and TV. Aligned's weight direction is . At the same parameter point , with fixed environment memberships and training time , the selected sample-adversary directions satisfy
=\frac{\lambda_{\mathrm{eff},t}}{E}\sum_e\operatorname{sign}_0(a_e) \sum_{i\in\mathcal I_e}b_i\nabla_\theta\pi_\theta(i\mid e),$$ where $b_i=\left.\nabla_w\ell_i(w)\right|_{w=1}$ and $\operatorname{sign}_0(0)=0$. The predictor uses the same form of shared weighted risk in both methods. The added direction passes stationarity contributions to the sample adversary; Section 3.2 and Appendix A.4 connect this term directly to the mechanism comparison. The matched four-method comparison uses common hyperparameters, backbone, budget, source-validation selection and five seeds. Full has the highest mean overall, worst-environment and worst-group accuracy among the four methods on all three evaluated datasets; Full raises worst-group accuracy from **57.28% to 58.48%**, **36.62% to 38.28%**, and **71.39% to 72.46%**, respectively (Table 2 (p. 7)). Three complementary measurements support this mechanism. The ESS study retains gains at comparable weight concentration (Table 3 (p. 8)); the frozen-predictor study measures lower spurious-feature sensitivity with higher original OOD accuracy (Table 6 (p. 9)); and the label-noise study measures higher noisy accuracy with NMR near one (Table 16 (p. 26), Table 18 (p. 27)). Together they connect the update-level feedback term to concentration-controlled prediction, intervention sensitivity and robustness to corrupted training labels.Thank you for the additional analyses. My first question was answered: the authors identified when reweighting preserves the prediction target and showed that Full can shift it in the controlled experiment. The Aligned ablation resolved my second concern by isolating TV feedback and showing its benefit on the evaluated tasks. My overall rating remains unchanged while rise the Informativeness.
Thank you for your follow-up and for confirming that our prediction-target analysis answers your first question, and that the Aligned ablation resolves your second concern by isolating TV feedback and demonstrating its benefit on the evaluated tasks.
We have revised the manuscript in response to feedback from all three reviewers. The revision incorporates the prediction-target analysis, matched mechanism ablations, controls at comparable effective sample sizes, and broader evaluation covering independently varied correlation and variance shifts and four DomainBed benchmarks. These additions clarify the theoretical scope of ECTR and provide further evidence for the contribution of shared-risk alignment and stationarity feedback.
We would appreciate your reassessment of the paper as a whole in light of the revised manuscript and the additional evidence. We note that your overall recommendation remains Weak Reject. Could you clarify which substantive concerns remain decisive for this recommendation? We would welcome the opportunity to address those specific points directly during the discussion period.
Q1 W1. Conditional target preservation and finite-support optimization
We thank the reviewer for their careful reading and thoughtful comments. The revision gives an exact criterion for preservation of a shared squared-loss conditional-mean target. For a nonnegative weight r(X) with finite positive normalization and the stated conditional moments,
=\frac{\mathbb{E}_e[r(X)(Y-\mu(Z))\mid Z]} {\mathbb{E}_e[r(X)\mid Z]}.$$ The denominator is positive, so weighted residual orthogonality is necessary and sufficient. We give sufficient conditions $r(X)=h(Z)$ and $r(X)\perp Y\mid Z,e$, and the equivalent zero-conditional-covariance characterization when the relevant moments exist. The proof is in Appendix C (p. 20), with the statement in Proposition 3 (p. 6). Fix $Z=h(X)$ as the feature variable defining the common target; weights may use the full input. For observed-environment empirical distributions, $r_\theta(X)=\exp s_\theta(X)$ gives ECTR's conditional weights after within-environment normalization. At $f_\mu(X)=\mu(Z)$, with finite reweighted second moments, the squared-loss probe is $$a_e^\mu(r)=-2\mathbb{E}_{e,r}[\mu(Z)\delta_e(Z)],\qquad \delta_e(Z)=\mathbb{E}_{e,r}[Y\mid Z]-\mu(Z).$$ This probe measures a weighted moment of conditional-target displacement. The finite-support study uses $X=(C,S)$, $Z=C$ and $\mu(Z)=C$, and reports $D_\mu$ alongside TV evaluated at each current predictor. Section 4.4 and Appendix C.1 connect these measurements to the preservation criterion. The finite-support model enumerates $C\in\{1,2\}$, $S\in\{-1,+1\}$, $Y=C+eS$ in two environments. Prescribed $3:1$ weights shift the conditional means to $C\pm0.5$ and produce probes $(-1.5,+1.5)$, $\mathrm{TV}=1.5$ and $\mathrm{KL}\approx0.1308$. The squared risk at fixed $f=C$ is one. These analytic quantities are given in Table 7 (p. 20). With a fixed predictor, Full's mean target displacement moves from **0.0380** to **0.8959**, while Aligned ends at **0.0372**. Joint optimization gives mean displacement **0.7004** for Full and **0.0372** for Aligned, with original-measure risks **1.2873** and **1.0039**. The complete TV, KL and uncertainty summaries appear in Table 9 (p. 21) and Table 8 (p. 21). ECTR's shared risk unifies the conditional measure used by the predictor and the stationarity constraint. Preservation of the original conditional prediction target is governed by Proposition 3's residual-orthogonality condition. Full-input weights can violate this condition, and the finite-support study quantifies the resulting target displacement and reference-risk changes. The paper specifies these target-preservation conditions and uses matched ablations and prediction interventions to evaluate the feedback mechanism on the studied tasks.