- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
In this paper, the authors study test-time feature alignment under distribution shift, focusing on assessing prediction recovery when the correspondence between shifted examples and their clean counterparts is unknown. The core idea is that improved prediction accuracy under distribution shift does not necessarily imply that the predictions of shifted examples are restored to those of their clean counterparts. The authors present an assignment framework in which potential correspondences are constrained by a feature-based matching cost. Specifically, they formulate a constrained assignment problem to quantify the maximum prediction disagreement over feasible correspondences and derive computable lower and upper bounds on this maximum. The authors further propose comparing alignment methods under the same unknown correspondence, which can establish their relative performance even when separately computed error bounds overlap.
Critical strengths
-
Evaluating test-time feature alignment is an interesting, practically relevant, and challenging problem. The core idea of distinguishing improved prediction accuracy from the recovery of clean-reference predictions is also interesting.
-
The proposed assignment-based formulation appears mathematically well motivated. The computable bounds on the maximum prediction disagreement under correspondence uncertainty are potentially valuable.
-
The experiments appear extensive, when including the additional analyses presented in the appendix.
Critical weaknesses
-
I found the paper, including its technical exposition, extremely difficult to follow. Even after reading the paper multiple times, I found it challenging to understand how the individual components fit together and contribute to the overall objective. The main challenge seems to lie not in the overall structure or the complexity of the mathematical subject, but in how the individual ideas and mathematical constructions are introduced and connected. Even the introduction moves abruptly into the technical setting without sufficiently developing the underlying motivation. The technical section presents a sequence of mathematical formulations and results without providing sufficient guidance on their roles and significance in the overall framework. For example, the authors introduce the anchor correspondence without clearly motivating why it is needed or explaining its role in the resulting ambiguity measure. Similarly, the transition from the unpaired ambiguity formulation to the paired-calibration analysis is rather abrupt. I believe that clearly explaining the motivation behind each component and design choice, as well as the logical connections between them, is as important as presenting the mathematical derivations. The presentation of the experiments could also be improved. For example, the last paragraph of Section 3 (post-hoc paired audits) describes three experimental procedures without clearly explaining the questions they are intended to address. Their significance becomes clearer only after reading the corresponding results.
-
The presented paired-calibration analysis of method-specific recovery errors appears to require clean and shifted example pairs with known correspondences in a calibration set. I do not think this is a limitation in itself, but it defines the operational setting of the proposed analysis. On the other hand, while the unpaired analysis does not require such correspondences, its bounds do not directly characterize the actual prediction-recovery error. This is acknowledged in the paper, but it could be explained more clearly.
-
The experiments focus on analyzing prediction recovery and correspondence uncertainty under feature alignment. The proposed analysis does not directly demonstrate how its findings can lead to improvements in downstream task performance. This is not necessarily a limitation of a scientific contribution, but it would be good to discuss how the findings of this work could be applied in practice.
Minor Comments
- The appendix is quite long, which also contributes to the difficulty of reading the paper. Clearly referencing the relevant appendix sections and providing a brief description of their contents in the main paper could help readers.
Questions for authors
I was wondering whether the proposed lower and upper bounds are theoretically tight. Could the authors establish their sharpness by identifying nontrivial settings in which the bounds are attained?
Thank you for taking the time to review our submission. We wanted to kindly follow up on our rebuttal and ask whether our clarifications addressed your concerns. We'd be happy to provide any further explanations if needed.