acceptodds
Under review as a conference paper at ICLR 2027

When Should a Domain-Generalized Model Specialize?

Abstract

Limited-label adaptation raises two questions: whether a fixed policy improves a domain-generalized model, and whether episode-specific policy selection provides additional value. The Cross-Fitted Domain-Generalization Specialization Assessment (CF-DGSA) separates fixed adaptation gain, held-out selection premium, and total gain over zero-shot inference. Its comparator is the accuracy-maximizing fixed policy on a labeled audit panel, while a routing margin affects only selection. Symmetric cross-fitting and a matched-half diagnostic distinguish held-out value from same-fold reuse. Selection-premium intervals project paired component intervals, with operating characteristics evaluated under Gaussian and paired-image models. Across nine configurations on PACS, Office-Home, and VLCS, four selection-premium intervals are positive, but only three corresponding total-gain intervals are positive. VLCS ResNet-50 has a selection premium and a total gain: improved selection among adaptation policies can still underperform zero-shot inference. Reproduced ERM++ models further distinguish beneficial fixed adaptation without resolved selection value, observed policy degeneracy, and harmful adaptation. The assessment separates evidence for adaptation from evidence for policy selection under explicit audit-label access.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Domain Generalization, Limited-Label Adaptation, Policy Selection, Cross-Fitting, Uncertainty Quantification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Y5XAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CF-DGSA, an offline evaluation framework for deciding whether a generalised model should specialise to a target domain. The core issue is that fixed adaptation gain and episode-specific policy-selection gain should be assessed separately; otherwise, selection-induced optimism conflates “adaptation helps” with “selection helps.” The method uses symmetric cross-fitting to separate selection from evaluation, hierarchical bootstrap for uncertainty, and component projection for selection-premium intervals; it defines fixed gain, selection premium, and total gain. Experiments on PACS, Office-Home, and VLCS use ResNet-18/50 and reproduced ERM++ checkpoints. The main finding is that a positive selection premium can coexist with harmful total adaptation. The paper concludes that zero-shot inference, best fixed adaptation, and episode-specific selection should be compared separately.

Critical strengths

Separating fixed adaptation gain, selection premium, and total gain is a useful decomposition of selection-induced optimism in DG adaptation evaluation; the VLCS ResNet-50 case clearly shows that better selection need not imply positive total gain.

The paper is unusually careful about assumptions and limitations; the reproduced ERM++ results closely match published values, and the simulation operating characteristics add credibility.

Critical weaknesses

The paper compares one-way holdout and a same-test oracle, but does not systematically compare repeated holdout, nested evaluation, or a target-specific selector under matched label budgets and controlled Type-I error. Table 2 also shows small cell-level opportunities (e.g., 0.00 pp for Office-Home ResNet-18/50), so the additional complexity of episode-specific selection needs more direct held-out evidence.

The projection argument does not by itself guarantee component-interval calibration; the simulations are mainly Gaussian and paired-image settings and do not stress skewed effects, heteroscedasticity, or influential cells. The leave-one-cell-out analysis reports point-estimate sign stability but not CI-based decision stability.

Questions for authors

Under leave-one-cell-out recomputation, do the CI-based decisions (not only point-estimate signs) remain stable? Please report whether interval decisions flip.

Given the small cell-level opportunities in Table 2, how does episode-specific selection compare with a simpler target-specific selector under held-out evaluation?

Can the authors add a systematic comparison with repeated holdout / nested evaluation under matched label budgets and Type-I error to directly support the benefit of CF-DGSA over simpler protocols?

1 comment

Thank you for the constructive review. The revision adds direct target-fixed comparisons, interval-based cell deletion, matched-budget alternatives, and dependence-aware calibration. Differences are in percentage points (pp); intervals are nominal 95% confidence intervals unless otherwise specified.

CI-based LOCO. The additional panel covers all nine Table 1 settings and 124 cell deletions. For fixed adaptation gain, H/A selection premium, and H/A total gain, 372/372 interval decisions remain unchanged with both 5,000 and 20,000 replicates (Supplement N). Secondary decisions change in both streams: one Office-Home ResNet-50 deletion changes the three-way-minus-H/A effect from unresolved to negative; five of twelve VLCS ERM++ deletions change the three-way premium from positive to unresolved. Supplement N reports these separately. The separate nine-condition cohort retains 370/372 decisions in both streams, with both changing configurations and full/deletion intervals in Supplement K. The larger ResNet-50 confirmation preserves all four prespecified decisions across 144 deletions (Supplement J.1).

Episode versus target-fixed selection. An independent ResNet-50 confirmation gives +0.46 [0.28,0.63], +0.37 [0.18,0.56], and +0.13 [0.05,0.18] on PACS/VLCS/Office-Home, with target-policy reselection in every replicate. Matched pooling also yields positive intervals on PACS/VLCS. Broader shared-image sensitivity retains positive pointwise evidence on PACS; all primary adjusted intervals include zero (Table S12). On the complete Office-Home audit sets under the Table 1 settings, H/A episode-minus-target-fixed accuracy is −0.191 [−0.264,−0.027] for ResNet-18 and +0.098 [−0.111,0.389] for ResNet-50. ERM++ H/A is unresolved. All three adjusted H/A intervals include zero; ResNet-18 Z/H/A remains negative after adjustment. The ERM++ Z/H/A contrast is −0.098 [−0.128,−0.006] nominally and unresolved after adjustment (Table S14(b)). Fresh validation using the separate Supplement K source cohort finds a negative PACS H/A contrast and no positive adjusted comparison (Supplement M.1). Repeated and nested evaluation. Four protocols share synthetic panels and 200 acquired audit labels per target. Independently calibrated detectors target positive-error probability at most .025 with 95% simultaneous calibration confidence over the prespecified null family. Worst observed null rejection rates are 1.98%/1.66%/1.82%/2.00% for single one-way/repeated one-way/symmetric/nested-margin assessment. At raw-margin scale 0.5 in the normal-cell, three-cell example, powers are 57.65%/93.00%/93.65%/83.90%. Repetition explains most of the improvement. Repeated bidirectional holdout has the same point estimator under matched candidates, partitions, directions, and weights. Table S3 gives all three datasets’ real-data comparisons; Table S7 reports the matched synthetic benchmark.

Component calibration. Projection requires valid component tails. Expanded studies examine skewed effects, heteroscedasticity, influential cells, and shared-image dependence. Independent calibration and validation cover 52 fixed-audit and 18 empirical reference laws, with explicit truth/calibration failure budgets. The interval containing both branches retains per-comparison marginal coverage over those 70 specified laws. Minimum observed coverage is 96.97%/99.24% at nominal 95%/98.333%; joint validation lower bounds, including truth uncertainty, are 95.92%/98.65%. Tables S11–S12 report coverage, widths, and all operational real-panel comparisons, distinguishing the projection argument from component-interval calibration.

Reviewer_QzmeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CF-DGSA, an offline protocol for assessing limited-label adaptation of domain-generalized models. It uses a labeled target audit set and cross-fitting to separate fixed adaptation gain, episode-specific policy-selection gain, and total gain over zero-shot inference. Experiments compare head-only and final-block adaptation on PACS, Office-Home, and VLCS.

Critical strengths

Separating fixed adaptation gain, selection gain, and total gain avoids treating better selection among adaptation policies as evidence that adaptation outperforms zero-shot inference.

The cross-fitted evaluation avoids selecting and evaluating a policy on the same audit fold. The paper also reports cases where policy selection helps but adaptation still underperforms zero-shot inference.

Critical weaknesses

The technical and practical contribution is limited. CF-DGSA mainly combines a standard best-fixed comparison with sample splitting, cross-fitting, and hierarchical bootstrap. It also requires 284–889 additional labeled target examples per cell and is explicitly an offline assessment rather than a support-only selection rule. Its episode-level selector cannot choose zero-shot.

The experiments do not control the factors that differ between H and A. The two policies differ in both trainable depth and BatchNorm behavior: H freezes all BatchNorm modules, whereas A updates the final-block BatchNorm statistics. All real-data experiments also use only this policy pair and a single support setting k=10k=10, without controlled comparisons or experiments using different support and audit budgets. It is therefore unclear whether the results come from adaptation depth, BatchNorm updates, or the particular label setting.

Questions for authors

None

1 comment

Thank you for recognizing the distinction between adaptation benefit and selection benefit. The revision adds normalization controls, label-budget comparisons, and measured costs. Differences are in percentage points (pp); intervals are nominal 95% confidence intervals unless otherwise specified.

Assessment, labels, and cost. Support labels fit candidates; additional audit labels enable offline assessment. The aligned panels use 1,988/2,137/2,922 unique audit examples on PACS/VLCS/Office-Home. Matched episode-specific and target-fixed assessments reuse fitted candidates and labels; a prespecified single policy can avoid the other fit. Supplement L.2 reports the label requirements and measured H/A fitting costs, with the benchmark workload and hardware specified.

Depth and BatchNorm. BN-frozen A versus H holds BN mode fixed while expanding the trainable set; A versus BN-frozen A changes BN behavior with trainable parameters fixed. The assessment includes 440 all-BN-frozen A controls under the projected ResNet-50 settings of Table 1, preserving source models, supports, trainable parameters, policy-specific random streams, and updates for the paired A comparisons. On Office-Home, BN-frozen A minus H is −8.88 [−9.64,−8.14], and original A minus BN-frozen A is +7.30 [6.52,8.04]. Table S13 reports all three datasets' direct and assessment-level contrasts.

Candidate and label settings. Including Z yields a VLCS three-way total gain of +0.48 [0.30,0.70] (Table S8(b)). Support varies over k=5,10,20 on PACS and k=5,10 on VLCS/Office-Home, whose fixed pools preclude k=20. Directional budgets vary over 25/50/100/full-fold labels with predictors and evaluation examples fixed. At 25 labels on Office-Home, episode-minus-target-fixed value is −0.47 [−0.64,−0.22] (Tables S4–S5). Label accounting includes the complete evaluation union.

Reviewer_H8iDOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes CF-DGSA, an offline protocol for assessing whether adapting a domain-generalized (DG) model to a target domain with a few labels is worthwhile. It is an assessment protocol, not a new adaptation method. The protocol reports three quantities: the gain of the best single adaptation procedure over the unadapted model, the additional gain from choosing the procedure separately for each episode (one trained model with one draw of support labels), and their sum (Sec. 3.2). Adaptation uses a small labelled support set. A separate labelled audit set is used to choose on one half and evaluate on the other, with the halves then swapped.

The authors use the protocol to show three things. First, its interval for the selection gain is valid: they argue at least 95% coverage with a union bound (Sec. 3.3) and confirm it by running the procedure on simulated data (Sec. 4.2). Second, applied to two adaptation procedures (retraining the classifier only, or also the final residual block) for ResNet and ERM++ models, adaptation improves on the unadapted model on PACS and Office-Home but not on VLCS (Table 1). Third, selection can add value while adaptation as a whole is harmful: on VLCS with ResNet-50, selection beats the best fixed procedure by 0.80 pp, while the full pipeline is 3.00 pp below the unadapted model.

Critical strengths

The interval procedure is validated in simulation. The paper does not only derive its intervals but checks them in Gaussian and paired-image simulations, including settings with no real effect and near-ties between the two policies (Sec. 4.2). Under these conditions the procedure makes no false positive claims in 10,000 null trials and its intervals cover the true value at or above the nominal rate. This makes the reported intervals more trustworthy than a derivation alone would.

Critical weaknesses

1. Incremental contribution assembled from existing components. Each part of CF-DGSA is established practice: held-out selection and cross-fitting, comparison against the best single option from algorithm selection, and an unadapted baseline. The paper presents the assembly as its contribution (Sec. 2.3), but takes one option per component without discussing the alternatives. Several relevant ones are not cited: nested cross-validation, winner's-curse-corrected intervals, and data carving for the statistics; surgical fine-tuning, LP-FT and WiSE-FT for the adaptation choice. The interval procedure is compared only with one-way holdout and a same-data oracle. As a result, the paper does not show that its particular combination improves on existing ways of assessing adaptation, or that it changes any conclusion. On the competitive ERM++ models, which the paper uses to address the concern that its results reflect weak source models (Sec. 2.1), no selection premium is resolved on any dataset (Table 1: -0.02, 0.00 and +0.17 pp, all intervals containing zero).

2. Unclear contribution and hard-to-follow presentation. It is difficult to tell which parts of CF-DGSA are taken from the literature and which are new: the introduction lists contributions in the paper's own notation, and Sec. 2.3 states only that the contribution is not cross-fitting or resampling in isolation. It is also not stated plainly that the two adaptation procedures are the objects being assessed rather than the method. The method section introduces over a dozen named quantities to express three comparisons, which makes a modest idea difficult to follow despite careful reading. Figures and text also disagree in places: the third curve in Fig. 2(a) is labelled "Matched same-fold", while the text describes it as the same-test oracle, which is explicitly not the matched-half diagnostic; and Fig. 3(b) assigns four configurations to a "Same-fold-only" category that Table 1 does not use for the same results.

Suggested references

  • Bias in Error Estimation When Using Cross-Validation for Model Selection (Varma and Simon, BMC Bioinformatics 2006)
  • Inference on Winners (Andrews, Kitagawa and McCloskey, QJE 2024)
  • Optimal Inference After Model Selection (Fithian, Sun and Taylor), arXiv:1410.2597
  • Surgical Fine-Tuning Improves Adaptation to Distribution Shifts (Lee et al., ICLR 2023)
  • Fine-Tuning Can Distort Pretrained Features and Underperform Out-of-Distribution (Kumar et al., ICLR 2022)
  • Robust Fine-Tuning of Zero-Shot Models (Wortsman et al., CVPR 2022)
Questions for authors
  1. Three-way selection (additional analysis). What does a three-way selector over {zero-shot, H, A} give under the same audit protocol, and is the VLCS result (positive selection premium with negative total gain) expected by construction when zero-shot is not a candidate?
  2. Premium against a per-target policy (clarification). What is the selection premium against a per-target fixed policy? Our reading of Table 2 is that most of the resolved premium comes from D_T, the gain from choosing the better policy for each target domain, and that what remains for episode-specific selection appears much smaller, between -0.23 and +0.35 pp. It would be good to have this confirmed under held-out evaluation.
  3. Reproducibility (clarification). Will the authors release code, together with the support pools, audit manifests and fold partitions, so that the results can be reproduced? The paper has no reproducibility statement, and some setup details are not given, for example how the ResNet baselines were trained and how the classifier head in policy H is initialised.
3 comments

Thank you for the detailed review. The revision clarifies the assessment question and adds the requested comparisons. Differences are in percentage points (pp); intervals are nominal 95% confidence intervals unless otherwise specified. Contribution and presentation. CF-DGSA assesses adaptation benefit and the additional held-out value of finer selection. Table 1 makes the distinction concrete: VLCS ResNet-50 has a +0.80 selection premium but −3.00 total gain; all three original ERM++ premium intervals contain zero. Sections 1–3 introduce these decisions before notation, identify H/A as established candidates, and attribute cross-fitting and resampling to prior work. The assessment links explicit comparators, selection units, and target-label access to the corresponding adaptation decisions. All six suggested references are included; Supplement D distinguishes the inferential targets of nested evaluation, winner inference, and data carving. Figures 2–3 now use consistent protocol names and Table 1 outcome categories.

Three-way selection (Q1). The {Z,H,A} comparison uses the same within-panel predictors and audit folds. In the additional VLCS panel using the ResNet-50 settings of Table 1, including Z improves selected accuracy by +3.43 [2.54,4.33] and yields a +0.43 [0.09,0.86] total gain (Table S14(a)). The corresponding VLCS ERM++ panel gives +0.36 [0.06,0.69] total gain and a +3.16 [2.25,4.05] improvement over H/A (Table S14(a)). The larger controlled confirmation yields a +4.12 [3.81,4.44] improvement and a +0.48 [0.30,0.70] total gain on VLCS; including Z changes selected accuracy by −0.12 [−0.15,−0.08] on PACS and −0.19 [−0.25,−0.12] on Office-Home (Table S8(b)). Excluding Z permits negative total gain but does not force positive premium. If H is the best fixed adaptation policy and is always selected, the selection premium is zero even when H underperforms Z. Target-fixed comparison (Q2). The baseline pools selection-fold scores within each target and evaluates the chosen policy on the opposite globally aligned fold. Every paired bootstrap replicate reselects the target policy. An independent ResNet-50 confirmation gives episode-minus-target-fixed values of +0.46 [0.28,0.63], +0.37 [0.18,0.56], and +0.13 [0.05,0.18] on PACS/VLCS/Office-Home. Matched pooling also yields positive intervals on PACS/VLCS. Broader dependence sensitivity retains positive pointwise evidence on PACS; its primary adjusted intervals include zero on all three datasets (Tables S8(a),S12). Under the Table 1 settings, the Office-Home H/A comparison favors target-fixed selection nominally for ResNet-18 and is unresolved for ResNet-50 and ERM++; all three adjusted intervals include zero (Table S14(b)). This separates value over a target-fixed policy from premium over one policy shared across targets.

Adaptation alternatives. The additional experiments evaluate surgical and full fine-tuning, LP–FT, and parameter interpolation. The 13-candidate ResNet-50 union improves selected accuracy over {Z,H,A}; episode-minus-target-fixed intervals remain unresolved. The expanded-family estimates and intervals are reported in Table S10. Source training, retained source-classifier initialization, trainable modules, BN settings, and support/audit construction are specified in Supplements A and N. The additional nine-configuration assessment retains the original model settings and is reported separately from the historical Table 1 observations. Code and supporting research materials are available upon request.

Thank you for the detailed response and for the revised version. Here is where I stand on each point.

CW1. Contribution. Thank you for adding the suggested references and the comparison with repeated and nested holdout. Your reply to Reviewer Y5XA says that repetition explains most of the improvement, and that repeated bidirectional holdout has the same point estimator as CF-DGSA. I read this as confirming that the protocol is close to an established one. The ERM++ selection premium intervals still contain zero on all three datasets. My concern therefore remains: the paper does not show a case where CF-DGSA changes a conclusion compared with existing ways of assessing adaptation.

CW2. Presentation. Thank you for restructuring Sections 1-3 so that the decisions come before the notation, for crediting cross-fitting and resampling to prior work, and for making Figs. 2-3 consistent with Table 1.

Q1. Three-way selection. Thank you, this answers my question. With zero-shot included, VLCS ResNet-50 has a positive total gain (+0.43 pp). The VLCS case with a positive premium and a negative total gain therefore comes from leaving zero-shot out of the candidates. Since this is the paper's main example, I would say so next to the VLCS result in the main text.

Q2. Premium against a per-target policy. Thank you for running this comparison. Under the Table 1 settings all adjusted intervals include zero. In the new ResNet-50 run, the episode-level values of +0.13 to +0.46 pp also include zero after adjustment. This matches my reading that most of the resolved premium comes from choosing a policy for each target domain.

Q3. Reproducibility. Thank you for adding the setup details in Supplements A and N. Code available on request is not the same as a release, though. I would encourage releasing the code together with the support pools, audit manifests and fold partitions.

I appreciate the amount of work in the rebuttal and the revision. The new results answer my questions, but they mostly confirm the concern in CW1, so my overall assessment stays the same.

Thank you for the careful follow-up and for acknowledging the presentation improvements. Effects below are in percentage points (pp).

CW1. Contribution. The paper addresses when an observed accuracy gain supports adaptation and when policy selection provides additional value. CF-DGSA separates fixed adaptation gain, selection premium over the best policy shared across targets, and total gain over zero-shot. The added target-fixed comparison separately tests whether episode-specific choice provides further value within a target.

The original results show why these distinctions affect the interpretation of adaptation performance. Four selection-premium intervals are positive, but only three corresponding total-gain intervals are positive. In the VLCS ResNet-50 example, selection between {H,A} has a positive premium while the selected predictor remains below zero-shot. Conversely, Office-Home ERM++ selects H in every observed fold: fixed and selected adaptation both achieve +3.63 [3.27,4.00] total gain, with a premium of 0.00 [0.00,0.00] (Table 1). The observed improvement is therefore obtained by fixed adaptation, without an additional contribution from selection. This is a positive adaptation result with a different attribution from a positive selection-premium result.

Opposite-fold evaluation separates selection from evaluation, the matched-half diagnostic measures same-fold reuse, and paired intervals quantify the contrasts under the stated sampling and component-tail conditions. With candidates, partitions, directions, and evaluation weights matched, repeated bidirectional holdout gives the same selected-value point estimator; the protocol comparisons also attribute most of the power improvement to repetition. The contribution is the adaptation-specific assessment and the findings it supports: the reported gains provide different evidence for adaptation, selection relative to a shared policy, and additional selection within each target.

Q1. Three-way selection. Thank you for emphasizing this point. The negative-total-gain VLCS result in Table 1 uses the {H,A} selector, with zero-shot as an external reference. This candidate-set restriction is now stated directly beside the VLCS result in the main text. In the additional VLCS panel under the original ResNet-50 settings, including zero-shot improves selected accuracy over H/A by +3.43 [2.54,4.33] and yields +0.43 [0.09,0.86] total gain (Table S14(a)).

On that additional panel, retaining the unchanged model as an option produces a selector with positive total gain. The result makes the candidate-set choice consequential: choosing among adaptations and deciding whether to retain the unchanged model are different adaptation decisions.

Q2. Premium against a per-target policy. Thank you for motivating this direct comparison. The eligible original-setting Office-Home comparisons and the broader adjusted ResNet-50 confirmation do not establish additional episode-specific value over target-fixed selection (Tables S12 and S14). The held-out comparison reselects the target policy in every bootstrap replicate.

This clarifies what the original shared-policy premium supports. A positive premium over one policy used across targets does not by itself establish a benefit from selecting separately within each target. The added target-fixed comparison makes that extra flexibility a direct empirical question. Together, the comparisons tie each specialization claim to the simpler policy it would replace.

Q3. Reproducibility. Thank you for the recommendation. Code and supporting research materials remain available upon request, as stated in the manuscript.

The empirical message is that an accuracy gain can support adaptation without supporting added selection, and a selection premium can support one comparison without supporting finer specialization. The paper makes these claims separately testable and shows their different outcomes in the reported adaptation settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.