acceptodds
Under review as a conference paper at ICLR 2027

The Hiding Problem: When Unread Features Do and Do Not Drive Harmful Adaptation

Abstract

When a classifier adapts to new data without labels and its loss rises, the loss alone does not say why: the model may have acquired a harmful dependence on a component of its representation that it did not use before, or it may have become more confident in the mistakes it already made. We ask which of these happens when a later update reaches components that the current predictor does not read (the Hiding Problem). An exact binary construction (Theorem 1) shows that the first is possible: two predictors with identical calibrated predictions and zero population IRMv1 penalty take the same Euclidean entropy step, and the one whose unread coordinate is stretched climbs from 75% to 90% accuracy on the adaptation law and falls to 10% under a reversed nuisance association, while the other stays at 75% under both. Deliberate amplification produces the same behavior in synthetic and frozen-feature Camelyon17 representations. On six native ResNet-50 checkpoints for a naturally shifted hospital, three controls point to the second explanation instead: a label-free affine map of the initial score closely reproduces over 99% of the head-only loss increase, a selected unread direction keeps no mean advantage over random deletions once endpoint movement is matched, and the same parameters trained with 320 labels raise the ranking in every checkpoint, whereas label-free updates that reach the representation lower it in the three checkpoints that missed the most positives (a post hoc split). A preregistered replication on Waterbirds-95 with six fresh models reproduces the head-only account and the sign pattern of the representation arms; the widespread one-class collapse at a fixed learning rate did not replicate. Identical predictions do not imply identical responses to learning. An update can recruit an unused feature, increase confidence in existing mistakes, or appear less harmful simply because it changes the predictions less. Our results make these distinctions consequential: evaluating continued learning requires testing what an update changes and establishing which changes account for its harm; the checks here begin that test and lay the groundwork for safeguards that can be tested the same way.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, invariant risk minimization, test-time adaptation, spurious correlations, out-of-distribution generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KDeDOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The work appears to deal with emergence of spurious correlations in unsupervised adaptation.

Critical strengths
  1. The work is very honest about its use of AI.
  2. The figures have appropriately sized text.
Critical weaknesses
  1. Despite my best efforts, I was unable to understand in detail the problem this work is trying to solve, and how it aims to do so. The writing is unclear to a level that it seems almost impossible to understand. While some ideas can be made out occasionally, most text does not make sense to me. I could also not extract anything from Figures 1 and 2. At its current state, this issue makes it impossible to evaluate regarding its actual contribution.
Questions for authors
  1. Could you explain to me what the specific problem the work deals with, how it is trying to solve it, and point me to the evidence that it does indeed solve the problem?
1 comment
AuthorsOpenReview ↗

To Reviewer KDeD

Thank you for the candid review. The paper is rewritten for a first-time reader, and the answers to your three questions follow.

The problem. A classifier deployed on new data may keep learning from that data without labels. Sometimes its loss then rises. Your summary names the usual suspicion: the update makes the model rely on a feature it previously ignored, such as a spurious correlation. The other possibility is that the update only makes the model more confident in the mistakes it already made. The two call for different fixes, and the loss alone cannot tell them apart. The paper is about telling them apart.

How we address it. Three checks tell the two apart:

  1. Could a rescaling of the model's original scores, which keeps their order, produce the same loss? If so, confidence alone can explain the harm.
  2. When blocking part of an update seems to protect the model, is that only because the blocked update is smaller? We compare updates of the same size.
  3. Did the data hold something the label-free update missed? We train the same parameters with a few labels to see.

The evidence.

  • Ordinary trained models: when only the last layer adapts, on six tumor-detection models moved to a new hospital (Camelyon17), rescaling reproduces over 99% of the loss increase, and a replication on bird photos (Waterbirds), with its expected outcomes written down in advance, agrees. The harm is confidence. Check 2 shows that blocking a direction the model ignored helped only by making the update smaller, and when deeper layers adapt, check 3 shows that 320 labels on the same parameters improve the model's ranking.
  • Models whose last layer was retrained on a few labels, a common way to make a model more robust: there the same update changes the model beyond rescaling, on both datasets, and the model does begin to use what it had ignored. On Waterbirds the published recipe leaves such a layer nearly blind to the background of the photo, a spurious cue, and label-free adaptation brings the background back.
  • Where to look: pages 1 to 3 of the revision. Figure 1 (page 2) is redrawn as bars, Table 1 (page 3) lists the main studies with what each supports, and the caption of Figure 2 now says in plain words what each panel shows. Figure 3 (page 7) separates trained and refit layers at a glance.

We would be glad to clarify anything that is still unclear.

Reviewer_uB6eOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates two related problems that may occur during fixed-feature entropy adaptation (adapting to a new domain by training final prediction head on unlabelled data) and full-network entropy adaptation (full network adaptation on unlabelled data). These two problems are the "Hiding Problem" (retention of feature vectors that are orthogonal to prediction head, which may later become important once the prediction head updates) and overconfidence in pre-existing mistakes (the model doesn't learn anything new and instead continues to predict using the same incorrect rules it already had). Both can result in increasing loss. The paper then proves mathematically that hidden features can lead to poor performance if they are artificially magnified, but that in real-world tests, that the poor performance actually comes from simple score stretching (in head-only updates) and a drift toward the majority class (when deeper layers update). The other finding is that blocking those hidden features does not actually help and that blocking only shrinks the overall size of the update.

Critical strengths
  • The three tests to prove whether a model actually learned a new bad pattern or just became overconfident are broadly applicable and useful. Additionally, the finding that many "fixes" only look good because they shrink the size of the update, not because they fix the underlying problem.
  • This paper does not overstate the value of their findings and includes examples where these issues (hidden features degrading model performance) do not negatively impact real world models trained on real world data.
Critical weaknesses
  • The presentation / writing of the paper was overly dense and difficult to read. E.g., the second paragraph of the Introduction is talking about experiments and results using terms that have not yet been explained. Additionally, the unusual sentence structure (e.g., line 174, saying to write something, and an usual number of sentences with ; and :, e.g., line 208 which uses both colon and semicolon in one sentence!) made comprehension even more difficult. The paper appears to do many tests and be rigorous, but it is presented in a highly dense manner and is difficult to read. Please revise substantially to improve the accessibility to this work. Clearly explain the main problem, your approach, and your findings in a few simple sentences. These should be placed both in the abstract and introduction.
  • It appears that the hiding problem does not present a substantial issue in real-world test cases (unless features are amplified).
Questions for authors

Clarification:

  • What is the main contribution of your paper? It initially appeared as though unread features would cause harm based on Theorem 1 and the synthetic experiments, but then on real-world data, the harm could not be attributed to unread features. Is the Hiding problem only something that exists in theory, or are the diagnostic controls you presented the main contribution?
5 comments
AuthorsOpenReview ↗

To Reviewer uB6e

Thank you for the detailed comments on the writing. Having worked on this paper for quite a long time, we had gotten carried away with technicalities when trimming it to meet the page limit. The paper has been rewritten for a first-time reader, and we address each point you raised below.

  • Plain statements: As you asked, the abstract now states the problem in its first four sentences, the approach in the fifth, and the findings in the sixth, in plain words and before any measurement, and the introduction follows the same order.
  • Order: You noted that the introduction's second paragraph reported results in terms not yet explained. The introduction now goes in order: the problem, the Hiding Problem, what one update does to the final layer, the three checks, and only then the findings. Sections 2 and 4 define the terms the results use, and Table 1 (page 3) lists the main studies with what each supports.
  • The sentences you cited: Line 174 ("Write h = ...") is rewritten (page 4). Line 208 (a colon and a semicolon in one sentence) is now three plain sentences in Appendix A.2.

The main contribution is the three checks and what they find, in four sentences. A label-free update can harm a model in two ways: it can make the model rely on features it had ignored, or it can make the model more confident in mistakes it already made. Three checks test which happened: a rescaling of the original scores, updates compared at equal size, and the same parameters trained with a few labels. For a final layer trained together with its features, it is confidence: on six Camelyon17 models a rescaling of the model's original scores reproduces over 99% of the loss increase. For a final layer retrained on a few labels without retraining the features (a refit layer), part of the harm is no longer a rescaling and the model's ranking of the inputs falls, and one line of algebra says why (Section 2).

Is the Hiding Problem only theoretical? No: it appears in ordinary models whose last layer is retrained on a few labels, and no feature is amplified. In the natural models tested in the submitted version, the harm was confidence, as your review says, and that remains true for trained layers. The revision adds the case where confidence no longer explains all of the harm (Section 6). On Waterbirds a final layer retrained by a published method is nearly blind to the background, a spurious cue: its score tells the two backgrounds apart at .54, where .5 is chance, against .79 for the trained layer. Label-free adaptation of that layer brings the background back (median .73 after 1,000 steps). On Camelyon17 a refit layer loses a median 41% of its ranking advantage within 30 label-free steps, while the trained layer's ranking does not move, and the rescaling check separates the two (.098 against .0003 nats of loss that rescaling cannot reproduce). The predictions for that test were fixed before it was run.

Where to look (as we respect your time): Pages 1 to 3 (the abstract, the introduction, Figure 1 and Table 1), and Figure 3 on page 7, which separates the two kinds of final layer.

Thank you for your revisions; however, I am still finding this paper difficult to read and follow.

Plain statements: I do not find your revised abstract to be clear or plain. From your abstract:

The two call for different remedies, and the first is possible because a trained model carries more than it shows. What its final layer does not read leaves no trace in any prediction, yet stays within reach of the next update.

The "two" what? What "first" do you mean --- reason for loss rising, or remedy? I think "What its final layer does not read leaves no trace in any prediction, yet stays within reach of the next update." is cryptic at best, unfortunately. Please use plain language and terms common in the field of machine learning. I think I understand, with effort, what you're trying to say, but this type of wording throughout the paper makes it very difficult to read and follow.

Ordering in Introduction: Thank you, this is an improvement.

Additionally, would it be possible for you to please show edits in your paper using colour, so it is clear where changes have been made?

Main contribution: Thank you for your attempted clarification, though I still find it somewhat difficult to fully grasp. You gave me a four-sentence explanation, but those sentences still have a very unusual structure that I find difficult to understand. Could you please try again? Instead of using words such as "it", simply write out in full what "it" is. This will improve the clarity of your writing.

Table 1 does help make the experiments and results more clear, so thank you for that.

Thank you for the follow-up, and for the time a close reading takes in a heavy review cycle.

Fundamentally, the final layer of a classifier is a linear readout, so the final layer reads the features only along the layer's weight vectors, and whatever is orthogonal to all of the weight vectors changes no prediction, yet remains in the model, available to the next update. A trained model therefore carries more than the predictions show, which is what we call the Hiding Problem. The paper works out the binary case, in which the final layer reads one direction, and the identity of Section 2 extends to K classes (Section 9), though the experiments are binary.

An update of the final layer on unlabeled data can then raise the loss in two ways.

  1. The update can turn the final layer toward features that are orthogonal to that direction, and the classifier then depends on features that changed no prediction before, which is the first explanation in the abstract.
  2. The update can keep the direction of the final layer and only stretch the layer, and the classifier then makes the same predictions with more confidence, including the wrong predictions, which is the second explanation.

The main contribution is three checks that test whether the update turned the final layer or only stretched the final layer, and what the three checks find.

  • For final layers trained together with the features (Section 5), the loss increase is the stretch, and in each of six Camelyon17 models a rescaling of the original scores reproduces over 99% of the loss increase, and a preregistered Waterbirds replication agrees.
  • For final layers refit on 320 labels (Section 6), the checks find the turn as well, and in a Camelyon17 test registered in advance, the refit layer loses a median 41% of the AUROC advantage over the trained layer within 30 steps, which no rescaling of the scores can do.

Section 6 is also the answer to your question whether the Hiding Problem exists only in theory.

Why this matters: The two explanations call for different remedies, limiting what an update can reach when the update turns the final layer and controlling confidence when the update only stretches the layer, and the loss alone cannot say which explanation applies. Identical predictions do not imply identical responses to learning (Theorem 1 and Figure 1), so before harm, or protection, is attributed to the features a model uses, the update itself should be tested.

In the PDF: The abstract states the binary case of the mechanism in one sentence (page 1), and the abstract's sentences on the entropy step give the same picture.

Regarding the color request: In place of color, here is where to find the changes. Since the PDF you commented on, this version rewords those sentences of the abstract and two sentences of the introduction (page 1), changes one clause in Section 9 (page 10), adds one paragraph to Appendix V.2 (page 52), and changes the spelling of one word in Figures 8 and 9. For the revision of November 12, our comment of that day to all reviewers gives the main additions, each with the request it answers. The title, Theorem 1, and the three checks are those of the submission, and the submission's main result, that the harm is confidence for trained final layers, stands in Section 5.

So is the main contribution and significance clear?

Making this work accessible genuinely matters to us, since we have been the readers of difficult papers ourselves, so if any sentence is still unclear, please point us to the sentence, and we will gladly look into reworking the wording.

Reviewer_E3ACOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When a deployed classifier keeps learning from new unlabeled data, its error can go up, and it is hard to tell why. This paper investigates whether such harm comes from the model starting to rely on features it previously ignored (which the authors call the Hiding Problem) or from the model growing more confident in mistakes it already makes, and proposes three checks to tell the two causes apart. On tumor pathology images from different hospitals (Camelyon17), it shows that the harm comes almost entirely from overconfidence: a simple rescaling of the model's original scores accounts for over 99% of the increase in error when only the final layer is updated. It further shows with a small worked example that two models with identical predictions can react very differently to the same update, but finds reliance on previously ignored features only where the authors build it in, and a replication on Waterbirds reproduces the overconfidence finding.

Critical strengths

S1. The paper gives three checks that any test-time adaptation study can run before blaming harm on a change in which features the model uses: compare the update with a label-free rescaling of the original scores, compare restrictions only after rescaling all updates to the same size, and run the same update with labels (Section 7). Each check changed a conclusion in this paper. For example, removing the selected feature direction first appears to lower NLL by 0.0075 nats relative to random deletions, but the advantage disappears (+0.0007) once updates are matched in size (Figure 2d). These checks are the paper's most reusable contribution.

S2. The paper measures how much of the harm comes from overconfidence. A label-free rescaling of each model's original scores reproduces the increase in NLL from final-layer adaptation to within 6 × 10⁻⁴ nats in all six Camelyon17 models, over 99% of the harm, and a replication on Waterbirds with six new models finds the same. That replication fixed its predictions and pass criteria before any model was trained, so its analysis could not be adjusted after seeing the results (Appendix H). This is why I trust the head-only finding, even though the theory behind it is known (W2).

Critical weaknesses

W1. The representation-level result does not replicate in size. In the Waterbirds replication (Appendix H), BatchNorm-affine entropy changes AUROC by at most 9.4×10−49.4 \times 10^{-4} in every seed, within the paper's own 10−310^{-3} bound for an unchanged ranking (Table 15, P4a), compared with changes of up to 0.028 on Camelyon17. The collapse results also fail to replicate. At the common learning rate, the replication predicted in advance that BatchNorm-affine entropy would lower AUROC and collapse to one predicted class in at least 5 of 6 seeds, but it lowers AUROC in 1 seed and collapses in none (P9c). All-parameter entropy was predicted to collapse in all 6 seeds and collapses in 1 (P9d). The head-only result replicates in full, so the evidence supports the overconfidence account (S2) but not the claims about what happens when the representation adapts.

W2. The theorem and the final-layer result follow from known facts. Theorem 1 is the one-coordinate case of an established result that gradient descent is not invariant to rescaling the features (Amari 1998; Martens 2020, cited). In the final-layer result, each entropy step scales the head up by 1.06 to 1.14 (Section 6.1), which raises NLL for a model that is already overconfident, and this failure of entropy-based test-time adaptation has been documented in the literature (Niu et al. 2023; Zhao et al. 2023, which are already cited). Section 3.1 calls Theorem 1 elementary, so the contribution rests on the experiments, the three checks and the overconfidence measurement (S1, S2), and W1 limits the experimental evidence to the final layer.

W3. The main text has no tables, and results are given in prose. Sections 5 and 6 report results in running text, often to six decimal places (for example, mean target risk rising from .798979 to 1.745193, and loss rising from .232994 to 5.163395), while all ten tables they refer to are in the appendices. A main-text table with one row per study, giving the update, the intervention and the result, would make clear which experiment supports which claim.

Questions for authors

Q1. Is there a natural mechanism, such as a training choice or a normalization layer, that produces unread directions at scales above the threshold in Theorem 1? Evidence of one would show that the Hiding Problem can occur without deliberate rescaling.

Q2. Standard test-time adaptation baselines (EATA, Niu et al. 2022; SAR, Niu et al. 2023; DeYO, Lee et al. 2024) are not run. When they are run on the hospital-2 data, do the three proposed checks change the conclusion about whether each method helps? Separately, the hospital-1 study follows Tent (Wang et al. 2021, cited), which is known to fail when the class balance shifts (Niu et al. 2023, cited), so its harm does not test unused features.

Q3. With a bias term in the final layer, does a rescaling of the original scores still reproduce the harm?

Q4. How was the threshold of −0.01 nats for the deletion test chosen?

A positive answer to Q1, or evidence from Q2 that the checks change conclusions about standard test-time adaptation methods, would raise my score.

2 comments
AuthorsOpenReview ↗

To Reviewer E3AC

Thank you for stating what would raise your score. The revision has both.

Q1. A natural mechanism: yes, last-layer retraining, with no deliberate rescaling (Section 6, Figure 3).

  • Mechanism. Last-layer retraining on a small group-balanced sample is a standard way to improve robustness (Kirichenko et al., 2023). In our networks one direction holds .96 and .94 of the feature energy (Waterbirds, Camelyon17). A final layer trained with its features lies along it, and a layer refit on 320 labels does not, leaving it largely unread. An entropy step multiplies the layer by I+ηMI+\eta M (MM: the features' uncertainty-weighted second moment), which mostly stretches the trained layer and turns the refit layer toward that direction.
Medians, trained vs our refitCosine with dominant directionRecruit ratio, 30 stepsLoss rescaling cannot reproduce (nats)
Waterbirds (30 models).87 vs .18.24 vs 36.0001 vs .028
Camelyon17 (6 models).93 vs .25.12 vs 44.0003 vs .098
  • Theorem 1's threshold. In its construction the recruit ratio is a2∣ρ∣a^2|\rho|, and above 1 the update goes mostly into unread directions. The takeover of part (iii), where the unread part of the score exceeds the read part, needs a ratio above 1+1/(ηh)1+1/(\eta h) for one step and a smaller one for repeated steps. Post hoc, it comes within 30 steps in 26 of 30 Waterbirds models and after 30, 100 and 1,000 steps in 2, 5 and 6 of 6 Camelyon17 models.
  • Consequences. The refit layer's ranking falls in every model on both datasets, and on Camelyon17 it loses a median 41% of its advantage within 30 steps. On Waterbirds a head retrained as in Kirichenko et al. is nearly blind to the background (AUROC .54 against .79), which label-free steps at a matched rate bring back in all 30 models (median .73 after 1,000 steps). Two of the five predictions for that head, on the 30-step loss and ranking, failed (15 of 30 each).
  • Registration. For Camelyon17, 11 predictions were fixed before any refit layer was computed there, and all hold. The Waterbirds studies are post hoc, though each was registered before its labels were read, and on 24 new models all 13 predictions fixed before training came out as predicted.
  • Scope. At equal movement the trained layer's unreproduced loss is .0014 and .0050 nats against .028 and .098. The result is for updates of the final layer: with it frozen, as in Tent, rescaling reproduces the loss change (−.0001-.0001 nats) in all 30 refit-layer networks, contrary to two of our four registered predictions.

Q2. The checks change the conclusion for EATA and SAR and support DeYO's gain on Camelyon17 (Section 7, Table 2; current-batch statistics). Scoring rules and the EATA and SAR predictions were fixed in advance, and a blind second scorer agrees on all 5,366 values per dataset.

  • EATA and SAR look safer than Tent in the 3 Camelyon17 models where Tent's harm is material. At the published ϵ\epsilon and e0e_0, EATA takes one step of 30, while SAR, run on BatchNorm, resets at every step. Check 2 shows why they look safer: Tent moved equally far is equally harmless (within .0001 nats). Without EATA's diversity filter, or with SAR's e0e_0 scaled to two classes, they move .39 and .75 as far as Tent on Camelyon17 and match it at equal movement within .01 nats on average.
  • DeYO lowers the loss in 6 of 6 models on Camelyon17 and Waterbirds (−.055-.055 and −.019-.019 nats). On Camelyon17 rescaling reproduces about a third of the gain, and Tent extended to the same movement is .107 nats worse. On Waterbirds the matched advantage is .009, below our .01 convention. DeYO ran at its paper's Waterbirds rate, five times Tent's. Post hoc (Appendix W.1), at a shared rate the Camelyon17 matched advantage falls from .107 to .011 (above .01 in 2 of 6), yet at nearly equal rates DeYO still beats Tent in 6 of 6 models (−.055-.055 against +.050+.050 nats).
  • Hospital 1. It does not test unread features (Section 9, Appendix D).

W1. The final-layer result replicates, also in 22 of 24 further models. The sizes and the collapse do not, so we state them for Camelyon17 only (Section 5).

W2. Theorem 1 is elementary, and its job is to show that no statistic of the current predictions can rule recruitment out. What the paper adds is the measured attribution of the harm to confidence, the checks that produce it, and the account of when it stops holding (Section 6). Added in the revision: in the Fisher geometry the entropy step is exactly a rescaling (Section 2).

W3. Done: Table 1 has one row per main study.

Q3. Yes. The reference is a scale and a shift, so it covers a bias, and adapting thirteen heads with a trainable bias changes the harm by at most 4.5×10−44.5\times10^{-4} nats (post hoc; Section 4, Table 19).

Q4. The −.01-.01 nats is a round threshold fixed before the run. The conclusion does not depend on it: at matched movement the sign flips (+.0007+.0007).

Update on Q1: two further runs on Tent's update, each registered before it was made, and all 14 of their predictions held. Both runs are now in the PDF (Appendix V.2 on page 52, and one clause of Section 9 on page 10), and a blind second scorer agrees on every value it compares.

Why these runs. Our reply of November 12 reported that two of our predictions did not hold for Tent as published, which adapts only the BatchNorm affine parameters. The two runs test where the result of Section 6 applies. Both use the same 30 refit-layer Waterbirds networks, keep Tent's objective, optimizer and batch size, and change which parameters train.

What the runs show

  • Only the final layer trains (300 steps): the loss change goes beyond rescaling, with a check-1 difference above .01 nats (median .047).
  • Only the BatchNorm affine parameters train, as in Tent (300 steps): a rescaling still accounts for the loss change, with a check-1 difference below .01 nats (median .002).
  • The final layer is frozen and every other parameter trains (300 steps): the background AUROC still rises (median +.030, against +.003 with the trained layer, read without a prediction).
  • Every parameter trains (30 steps at Tent's rate): the background AUROC rises in 30 of 30 networks.

In the 300-step run, the eight predictions hold in 30 of 30 networks each. So the result of Section 6 appears when nothing but the final layer moves, the background still rises when the final layer is frozen and the layers below it adapt, and a rescaling still accounts for the loss change of Tent as published at ten times the earlier horizon.

Does any part of Q1 to Q4 or W1 to W3 remain open?

Comments on the paper

1 comment
AuthorsOpenReview ↗

To All Reviewers and the Area Chair

Thank you so much for taking the time and attention to read our paper and to give such in-depth feedback. Time and attention are the most valuable things anyone has to give, and it means a great deal to us that you gave them to our work.

The revision adds what the reviews asked for, and the submission's main results stand.

  • The paper in two sentences. When a label-free update harms a model, the loss does not say whether the model began to use features it had ignored or only became more confident. Three checks tell the two apart: for a final layer trained together with its features the harm is confidence, and for a final layer retrained on a few labels, the natural case the revision adds, part of the harm is not.
  • Added at Reviewer E3AC's request. Last-layer retraining on a few labels is a natural mechanism, with no deliberate rescaling, by which updates of the final layer go mostly into directions it did not read (Q1; Sections 2, 6 and 9). The three checks change the conclusion for EATA and SAR, which look safer than Tent only because they barely update, and support DeYO's gain on Camelyon17 (Q2; Section 7).
  • Narrowed (W1). The size of representation-level change is now stated for Camelyon17 only (Section 5). A post hoc pattern in the submission (Figure 2(b)), that the ranking falls in the models that missed the most positives, was tested on 24 further Waterbirds models and did not hold (Appendix V.4).
  • Rewritten for a first-time reader (Reviewers uB6e and KDeD). The abstract and introduction state the problem, the approach and the findings in plain sentences, Sections 2 and 4 define the terms the results use, and Table 1 lists the main studies with the claim each supports.
  • Unchanged. The title, Theorem 1, the three checks, the Camelyon17 and Waterbirds results, and Figure 2 (its caption is plainer and reports that test). The submitted Sections 4 to 6 are kept in full in the appendix. Figure 1 shows the same construction with the same numbers, redrawn as bars; the submitted drawing is in Appendix A.

The appendix says what was fixed in advance for each added study (Table 19 lists the main ones), and the added numbers were recounted by a second implementation written blind to the first, except those the appendix states were scored once.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.