acceptodds
Withdrawn from ICLR 2027

Test-Time Feature Alignment Under Uncertain Correspondence

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Feature alignment can improve a frozen classifier's accuracy under test-time distribution shift without establishing whether shifted examples recover the predictions made on their clean counterparts. This work asks whether prediction recovery can be assessed without knowing which shifted example corresponds to which clean example. It introduces an assignment framework that considers possible matchings subject to a cost constraint. With unpaired features, the framework bounds prediction disagreement relative to a baseline matching. Independent paired calibration provides simultaneous recovery-error bounds for a new batch drawn under the same conditions as the calibration batches. These bounds can certify that one method better recovers clean predictions than another without identifying the true matching. Across natural ImageNet shifts and ImageNet-C, substantial ambiguity remains despite accuracy gains. In paired ImageNet-C experiments, every certified ordering agreed with the hidden pairs. A randomized-cost control preserved observed coverage but could no longer distinguish methods, showing that coverage alone does not guarantee informative bounds.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Feature alignment, distribution shift, sample correspondence, prediction recovery, identifiability, conformal prediction, robust optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_S8MCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

In this paper, the authors study test-time feature alignment under distribution shift, focusing on assessing prediction recovery when the correspondence between shifted examples and their clean counterparts is unknown. The core idea is that improved prediction accuracy under distribution shift does not necessarily imply that the predictions of shifted examples are restored to those of their clean counterparts. The authors present an assignment framework in which potential correspondences are constrained by a feature-based matching cost. Specifically, they formulate a constrained assignment problem to quantify the maximum prediction disagreement over feasible correspondences and derive computable lower and upper bounds on this maximum. The authors further propose comparing alignment methods under the same unknown correspondence, which can establish their relative performance even when separately computed error bounds overlap.

Critical strengths
  • Evaluating test-time feature alignment is an interesting, practically relevant, and challenging problem. The core idea of distinguishing improved prediction accuracy from the recovery of clean-reference predictions is also interesting.

  • The proposed assignment-based formulation appears mathematically well motivated. The computable bounds on the maximum prediction disagreement under correspondence uncertainty are potentially valuable.

  • The experiments appear extensive, when including the additional analyses presented in the appendix.

Critical weaknesses
  • I found the paper, including its technical exposition, extremely difficult to follow. Even after reading the paper multiple times, I found it challenging to understand how the individual components fit together and contribute to the overall objective. The main challenge seems to lie not in the overall structure or the complexity of the mathematical subject, but in how the individual ideas and mathematical constructions are introduced and connected. Even the introduction moves abruptly into the technical setting without sufficiently developing the underlying motivation. The technical section presents a sequence of mathematical formulations and results without providing sufficient guidance on their roles and significance in the overall framework. For example, the authors introduce the anchor correspondence without clearly motivating why it is needed or explaining its role in the resulting ambiguity measure. Similarly, the transition from the unpaired ambiguity formulation to the paired-calibration analysis is rather abrupt. I believe that clearly explaining the motivation behind each component and design choice, as well as the logical connections between them, is as important as presenting the mathematical derivations. The presentation of the experiments could also be improved. For example, the last paragraph of Section 3 (post-hoc paired audits) describes three experimental procedures without clearly explaining the questions they are intended to address. Their significance becomes clearer only after reading the corresponding results.

  • The presented paired-calibration analysis of method-specific recovery errors appears to require clean and shifted example pairs with known correspondences in a calibration set. I do not think this is a limitation in itself, but it defines the operational setting of the proposed analysis. On the other hand, while the unpaired analysis does not require such correspondences, its bounds do not directly characterize the actual prediction-recovery error. This is acknowledged in the paper, but it could be explained more clearly.

  • The experiments focus on analyzing prediction recovery and correspondence uncertainty under feature alignment. The proposed analysis does not directly demonstrate how its findings can lead to improvements in downstream task performance. This is not necessarily a limitation of a scientific contribution, but it would be good to discuss how the findings of this work could be applied in practice.

Minor Comments

  • The appendix is quite long, which also contributes to the difficulty of reading the paper. Clearly referencing the relevant appendix sections and providing a brief description of their contents in the main paper could help readers.
Questions for authors

I was wondering whether the proposed lower and upper bounds are theoretically tight. Could the authors establish their sharpness by identifying nontrivial settings in which the bounds are attained?

Reviewer_KaDVOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies test-time feature alignment, where features of shifted test images (e.g., corrupted or stylized images) are adjusted to match clean feature statistics before being passed to a frozen classifier. This paper argues that accuracy gains from alignment do not show whether each shifted image receives the same prediction as its clean counterpart, and that this is hard to check without knowing which shifted image corresponds to which clean image. The paper proposes a framework that considers plausible matchings between clean and shifted images to quantify this uncertainty, and uses paired calibration data to compare alignment methods. Experiments on ImageNet shifts (ImageNet-C) report high uncertainty despite accuracy gains.

Critical strengths
  1. The question of whether accuracy gains reflect per-example recovery seems interesting and relevant to evaluating test-time alignment.

  2. The paper states the scope and limitations of its claims, and code is provided.

Critical weaknesses

I should note that this paper is outside my area of expertise, so my review focuses mainly on readability. However, I believe the presentation issues below would make the paper difficult to follow for a large part of the ICLR audience.

  1. The abstract and introduction are very hard to follow. Terms such as “prediction recovery,” “assignment framework,” “cost constraint,” and “baseline matching” are used without being explained. In the first paragraph, the statement that “marginal recovery, examplewise feature recovery, and prediction recovery are therefore distinct” is given without any intuitive explanation or example.

  2. The paper introduces many terms with little intuition, e.g., “anchored task-conditioned correspondence-ambiguity profile,” “endpoint multisets,” “indexed atoms,” and “signed worst case.” Section 2 moves directly into dense notation without first explaining the problem in readable words.

  3. A simple example of the problem could potentially demonstrate the method of the paper and help with reading. For instance, the paper could show a clean batch and a shifted batch where the overall predictions look correct but individual predictions are mismatched. Figure 1 illustrates the method comparison, but not the problem itself.

  4. Similarly, it is hard for me to understand the experiments. The results are reported as many counts and brackets (e.g., 263 vs. 220 orderings, 33 orderings on 32 batches, 10 numerically unresolved cases) with little explanation of what each number means or its significance. The practical takeaway for someone using test-time alignment is unclear.

  5. Overall, it is very difficult for me to understand why the problem is interesting and what the motivation is. After reading the paper, I still fail to understand why agreement with the clean prediction is more important than accuracy.

  6. The paper contains a very long appendix, yet the main paper does not provide many pointers. By skimming through the appendix, I also find that important definitions and explanations are buried, making the paper even harder to read.

Questions for authors

I would appreciate it if the authors could provide (1) a plain-language explanation and a simple example of the core problem early in the paper, (2) intuitive definitions of the key terms, and (3) a clear statement of how a practitioner would use the proposed measures? I expect the authors to substantially improve the presentation so that the paper is accessible to a broader ICLR audience.

Reviewer_LuGBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper explores the question of whether, under distribution shift, one can align individual examples between clean and shifted dataset (under distribution shift). The authors propose a method to determine which alignment is "better". Experiments on image datasets measure alignment, accuracy and their correspondence.

Critical strengths

None

Critical weaknesses

This papers has several flaws. I will only highlight the main ones.

  1. The setup is not realistic/does not make sense: the setup assumes a clean set of N points and shifted set of N points with an unknown correspondence between the two. For example, the shifted set is a blurred version of the clean set. However, in no realistic scenario would we have a shifted set with its corresponding clean set available to us! We may have a dataset from the clean distribution but that is different from what this paper assumes. This difference is crucial.

  2. The writing is extremely poor: The introduction is not clear. It does not clearly explain the problem, its motivation or the setting. Experimental setup is also not clearly explained. Lot of unnecessary jargon and complicated wording.

  3. The method is not well-motivated and does not make any significant contribuion: the method seems to be a combination of a bunch of seemingly unrelated stuff like conformal prediction and cost optimization put together. For conformal prediction to work, certain assumptions like train and test distributions match (of the predictor for conformal prediction) need to hold. I dont see that here.

Questions for authors

See weaknesses.

Comments on the paper

1 comment

Thank you for taking the time to review our submission. We wanted to kindly follow up on our rebuttal and ask whether our clarifications addressed your concerns. We'd be happy to provide any further explanations if needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.