- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper substantiates several narrowly scoped conclusions quite thoroughly, but does not establish equally convincingly what important misconceptions these conclusions correct, what new mechanisms they reveal, or what practically meaningful research decisions they would change. I view it as a carefully executed diagnostic study with some methodological value, but limited substantive insight and demonstrated broader significance. My concern is primarily the importance of the contribution rather than the internal consistency of the experiments. Stronger evidence about meaningful semantic errors, objective-versus-optimization limitations, or the causes of the dataset-dependent behavior would be more consequential than further confirmations that the diagnostics are not interchangeable.
Critical strengths
The paper distinguishes marginal representation structure, cross-modal relational agreement, and retrieval performance. It introduces hard negative re-pairing to examine the relationship between structural agreement and original-pair fidelity, and compares checkpoint selection by neighborhood agreement against selection by validation recall. The framing is clear, and the interventions are reasonably controlled: one-to-one re-pairing preserves the sample collections, representations and retrieval scores remain fixed, and the external selector does not consult the evaluated models’ outcomes. These properties make the construction more informative than random shuffling alone.
Critical weaknesses
- The re-pairing experiment offers limited insight into semantic alignment. Equation 2 explicitly favors semantically similar replacements by minimizing the total cosine dissimilarity between original and replacement text partners in an external encoder’s embedding space.Consequently, lower original-pair fidelity need not imply weaker semantic alignment, and retained CKA may reflect a reasonable response to preserved semantics rather than a failure to detect genuine semantic errors. The authors acknowledge that original-pair fidelity does not assess replacement validity and that close substitutes are expected to preserve some relational agreement. However, the experiments do not distinguish valid alternative descriptions from semantically similar captions containing incorrect attributes or relations; the object-overlap analysis does not resolve this, and no human validity assessment was conducted. A targeted assessment separating these cases would clarify whether the diagnostic overlooks meaningful semantic errors. Without that distinction, the result remains primarily a caution about exact pair identity.
- Non-monotonicity does not establish a substantive failure of CKA-based pair recovery. The ordering reversal demonstrates that CKA does not globally order pairings by original-pair fidelity. However, an optimization objective need not rank every suboptimal candidate monotonically by recovery accuracy to be useful. The original pairing still achieves the highest CKA in the main comparison, approximately 0.719 versus 0.659 for complete semantic re-pairing.The FAQ experiment moves closer to evaluating practical usefulness, but selecting the highest-CKA candidate changes recovery accuracy by only −0.25, −0.73, and +0.41 percentage points relative to randomly selecting an already optimized run.These results do not establish whether imperfect recovery mainly reflects the objective, the solver, or limited variation within the candidate pool. Comparing the ground-truth pairing’s objective with the selected candidates under identical preprocessing and anchors would help clarify this issue. Figure 2 cannot substitute for that comparison because Appendix C.6 uses additional feature standardization.
- The checkpoint-selection result supports a narrower recommendation than the discussion suggests. The 5.75-percentage-point held-out Recall@1 gap provides concrete evidence that agreement-based selection can be inferior within the evaluated panel.The paired bootstrap and model-pool sensitivity analyses are useful, but they do not establish the reliability of the entire selection procedure across new validation samples, model pools, and distributions. Moreover, the experiment concerns trained retrievers for which the target validation metric is already available. Demonstrating that an indirect criterion performs worse in this setting offers limited guidance about situations where a proxy is needed. In particular, Section 6’s claim that relational agreement is worth using “only before a retriever exists” is broader than the evidence supports.The practical contribution would be stronger with a clearer account of the conditions under which agreement-based selection is informative or misleading.
- A potentially more informative finding remains insufficiently explained. I find the dataset-dependent sign reversal in Appendix E.5 more promising than the repeated demonstrations of non-equivalence: the agreement–recall correlation is negative on COCO but positive on Flickr30k for the same checkpoints, and this difference persists after matching sample sizes at .The follow-up analysis locates most of the reversal in cross-family ordering changes, but does not identify the underlying data or model factors responsible.Explaining this discrepancy could provide more substantive insight into when structural agreement is useful and why it sometimes diverges from retrieval performance.
Questions for authors
- How can you distinguish retained agreement due to valid alternative descriptions from insensitivity to genuine semantic errors?
- What explains the opposite agreement–recall correlations on COCO and Flickr30k at , even after matching sample sizes?