- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes CF-DGSA, an offline evaluation framework for deciding whether a generalised model should specialise to a target domain. The core issue is that fixed adaptation gain and episode-specific policy-selection gain should be assessed separately; otherwise, selection-induced optimism conflates “adaptation helps” with “selection helps.” The method uses symmetric cross-fitting to separate selection from evaluation, hierarchical bootstrap for uncertainty, and component projection for selection-premium intervals; it defines fixed gain, selection premium, and total gain. Experiments on PACS, Office-Home, and VLCS use ResNet-18/50 and reproduced ERM++ checkpoints. The main finding is that a positive selection premium can coexist with harmful total adaptation. The paper concludes that zero-shot inference, best fixed adaptation, and episode-specific selection should be compared separately.
Critical strengths
Separating fixed adaptation gain, selection premium, and total gain is a useful decomposition of selection-induced optimism in DG adaptation evaluation; the VLCS ResNet-50 case clearly shows that better selection need not imply positive total gain.
The paper is unusually careful about assumptions and limitations; the reproduced ERM++ results closely match published values, and the simulation operating characteristics add credibility.
Critical weaknesses
The paper compares one-way holdout and a same-test oracle, but does not systematically compare repeated holdout, nested evaluation, or a target-specific selector under matched label budgets and controlled Type-I error. Table 2 also shows small cell-level opportunities (e.g., 0.00 pp for Office-Home ResNet-18/50), so the additional complexity of episode-specific selection needs more direct held-out evidence.
The projection argument does not by itself guarantee component-interval calibration; the simulations are mainly Gaussian and paired-image settings and do not stress skewed effects, heteroscedasticity, or influential cells. The leave-one-cell-out analysis reports point-estimate sign stability but not CI-based decision stability.
Questions for authors
Under leave-one-cell-out recomputation, do the CI-based decisions (not only point-estimate signs) remain stable? Please report whether interval decisions flip.
Given the small cell-level opportunities in Table 2, how does episode-specific selection compare with a simpler target-specific selector under held-out evaluation?
Can the authors add a systematic comparison with repeated holdout / nested evaluation under matched label budgets and Type-I error to directly support the benefit of CF-DGSA over simpler protocols?
Thank you for the constructive review. The revision adds direct target-fixed comparisons, interval-based cell deletion, matched-budget alternatives, and dependence-aware calibration. Differences are in percentage points (pp); intervals are nominal 95% confidence intervals unless otherwise specified.
CI-based LOCO. The additional panel covers all nine Table 1 settings and 124 cell deletions. For fixed adaptation gain, H/A selection premium, and H/A total gain, 372/372 interval decisions remain unchanged with both 5,000 and 20,000 replicates (Supplement N). Secondary decisions change in both streams: one Office-Home ResNet-50 deletion changes the three-way-minus-H/A effect from unresolved to negative; five of twelve VLCS ERM++ deletions change the three-way premium from positive to unresolved. Supplement N reports these separately. The separate nine-condition cohort retains 370/372 decisions in both streams, with both changing configurations and full/deletion intervals in Supplement K. The larger ResNet-50 confirmation preserves all four prespecified decisions across 144 deletions (Supplement J.1).
Episode versus target-fixed selection. An independent ResNet-50 confirmation gives +0.46 [0.28,0.63], +0.37 [0.18,0.56], and +0.13 [0.05,0.18] on PACS/VLCS/Office-Home, with target-policy reselection in every replicate. Matched pooling also yields positive intervals on PACS/VLCS. Broader shared-image sensitivity retains positive pointwise evidence on PACS; all primary adjusted intervals include zero (Table S12). On the complete Office-Home audit sets under the Table 1 settings, H/A episode-minus-target-fixed accuracy is −0.191 [−0.264,−0.027] for ResNet-18 and +0.098 [−0.111,0.389] for ResNet-50. ERM++ H/A is unresolved. All three adjusted H/A intervals include zero; ResNet-18 Z/H/A remains negative after adjustment. The ERM++ Z/H/A contrast is −0.098 [−0.128,−0.006] nominally and unresolved after adjustment (Table S14(b)). Fresh validation using the separate Supplement K source cohort finds a negative PACS H/A contrast and no positive adjusted comparison (Supplement M.1). Repeated and nested evaluation. Four protocols share synthetic panels and 200 acquired audit labels per target. Independently calibrated detectors target positive-error probability at most .025 with 95% simultaneous calibration confidence over the prespecified null family. Worst observed null rejection rates are 1.98%/1.66%/1.82%/2.00% for single one-way/repeated one-way/symmetric/nested-margin assessment. At raw-margin scale 0.5 in the normal-cell, three-cell example, powers are 57.65%/93.00%/93.65%/83.90%. Repetition explains most of the improvement. Repeated bidirectional holdout has the same point estimator under matched candidates, partitions, directions, and weights. Table S3 gives all three datasets’ real-data comparisons; Table S7 reports the matched synthetic benchmark.
Component calibration. Projection requires valid component tails. Expanded studies examine skewed effects, heteroscedasticity, influential cells, and shared-image dependence. Independent calibration and validation cover 52 fixed-audit and 18 empirical reference laws, with explicit truth/calibration failure budgets. The interval containing both branches retains per-comparison marginal coverage over those 70 specified laws. Minimum observed coverage is 96.97%/99.24% at nominal 95%/98.333%; joint validation lower bounds, including truth uncertainty, are 95.92%/98.65%. Tables S11–S12 report coverage, widths, and all operational real-panel comparisons, distinguishing the projection argument from component-interval calibration.