- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that aggregate retrieval scores can hide regressions inside semantic groups when frozen CLIP/SigLIP embeddings are adapted post hoc. It proposes a "ledger–adapt–guard–re-ledger" loop on cached features. (i) A per-group ledger records the image–text centroid gap , within-modality compactness, top-1 group-hit consistency, and bidirectional recall. (ii) Two identity-initialized linear maps are trained with one of five objective branches, chosen per dataset–backbone slice by a fixed routing table. The gap-closing and preservation terms are weighted by and , assuming unit manifest weights. (iii) A held-out guard selects the largest for such that and . If no scale passes, it falls back to the frozen embeddings (). (iv) The evaluation set is re-ledgered to report the fraction of degraded groups.
Experiments cover 12 slices from 10 datasets with two backbones. They use two protocols: a strict protocol in which the fit, guard, and evaluation sets (, , ) are disjoint, and a transductive audit with . Under the strict protocol, the method reaches 38.27 R@1 with 20.01% degraded groups, compared with 38.53 R@1 and 21.65% for CLIP-Refine under the same guard.
Critical strengths
-
A relevant question, evaluated with a fair and transparent protocol. The paper asks whether aggregate gains from post-hoc adaptation hide group-level regressions, which matters for anyone updating deployed embeddings. In the disjoint fit/guard/eval experiment, the baseline receives exactly the same guard as the proposed method. The authors report the results transparently, including a comparison they do not win (Table 10), and support them with bootstrap intervals and leave-one-dataset-out checks. This is consequential because it makes the evidence trustworthy and makes the paper a useful reference point for evaluating cached-feature adaptation.
-
A useful, adapter-agnostic finding about guarded updates. Under held-out evaluation, unguarded adaptation falls below the frozen model (34.71 and 35.37 vs. 36.03 R@1). Guarded interpolation recovers about +3.55 R@1 and roughly halves the share of degraded groups (from 43.40% to 20.01%). Because the guard is simple and works with any adapter, this is the paper's most transferable contribution and the main reason I see a path to acceptance.
Critical weaknesses
-
The held-out results do not yet show that the ledger-specific components add value beyond the guard. Under the strict protocol, the fixed REFINE branch with the guard produces the same numbers as guarded CLIP-Refine (Tables 9–12). The routed method scores 0.26 R@1 lower than both (CI ). The AWARE and BALANCED branches trade lower R@1 for fewer degraded groups. The gap-derived weighting () is not ablated, since Table 3 uses uniform weights. The 1.64-point advantage in degraded groups has no confidence interval. This is consequential because it decides what the contribution is. If the contribution is the guard plus the audit protocol, the paper should be framed that way and compared with related interpolation-selection methods (e.g., WiSE-FT, PAINT). If the ledger-weighted objectives and the router are central, a held-out ablation is needed. Since all experiments run on cached features, I expect this to be addressable during the rebuttal.
-
Much of the evidence comes from the transductive audit, and the design of the routing table is not documented. Tables 2–7, the component ablation, and the qualitative figures come from fitting on the evaluation records (). Under held-out evaluation the gains over frozen shrink substantially: Winoground goes from to , and COCO-Attr from to . The routing table also includes backbone-specific branch choices and 14 per-slice overrides, and the paper does not explain how these were chosen. This is consequential because the strict experiment is the only evidence of generalization. Its credibility depends on the routing and overrides having been fixed without reference to the evaluation records. A clear description of the procedure, or a strict-protocol result with a single global configuration, would resolve most of this concern.
Questions for authors
Clarifications (answers could change my rating)
- Is the REFINE branch the same objective and implementation as CR-off? Since guarded REFINE matches guarded CR-off exactly, which part do you consider the main contribution: the guard and audit protocol, or the ledger-weighted objectives and the router?
- Under the strict protocol, could you report (a) ledger-derived weights vs. uniform weights (), and (b) the routed method vs. a single fixed configuration? A confidence interval for the degraded-group difference (20.01% vs. 21.65%) would also help. These should be inexpensive to run on cached features.
- How were the routing map (Table 13) and the overrides (Table 14) chosen? In particular, were any paper-scale results consulted before the strict split was made? Is the split into , , and done by image ID or by pair, given that some datasets have several captions or questions per image?
If the rebuttal shows that the ledger-aware components improve the held-out retrieval–degradation trade-off, or if it repositions the paper around the guard and audit protocol and resolves the clarifications above, I would be inclined to raise my rating.
Suggestions (not expected for the rebuttal)
- Discuss related work on interpolation and model patching (WiSE-FT, PAINT), on regression-free model updates (negative flip rate), on group-robust CLIP adapters (Zhang & Ré, 2022), and on the modality gap (Liang et al., 2022).
- Report the standard Winoground and SugarCrepe metrics alongside the retrieval-pool variants.
- Fix the typos and placeholder content in Figures 1–2, and align the supplementary router code with Table 13.
2 comments
Thank you for the detailed rebuttal and the substantial new experiments, I have raised the score to 3 (weak acceptance).
CR-off, a single global configuration, weighting controls and routing provenance
We thank the reviewer for recognising the fair protocol and the transferable guard. We ran the requested controls under a stricter protocol, full-v0, where fit, guard and evaluation sets are disjoint by global image ID, decoded pixels and source ID. It has 528 core cells, 11 fit-eligible slices × 2 backbones × 8 variants × 3 optimizer seeds, with ObjectNet used only for evaluation. Inference resamples seven shared-image source blocks, so the smallest Holm-adjusted p over our 14 predeclared tests is 0.21875; we report effects, 95% CIs and exact p-values.
Q1. REFINE against CR-off, and the main contribution. Yes. REFINE is deliberately the CLIP-Refine objective, included as a branch the ledger can route to, and fit-only checks at a common budget give maps identical bit for bit. The core contribution is the loop between ledger and guard, a group-wise audit coupled to adaptation and to update selection. Guarding alone, relative to the same maps at α = 1, removes 3.708 damage points for the router at R@1 +0.006 and 1.826 for REFINE at R@1 −0.045.
Q2(b). Routing against a single fixed configuration, default guard with tolerance 0.05 and cap 0.20:
The routed method exceeds REFINE by +0.472 R@1 points, CI [+0.077, +1.251], exact p = 0.09375, at level damage of −0.078 [−0.982, +1.473], and every leave-one-source-out difference stays positive, from +0.205 to +0.532. The single global configuration makes no per-dataset choice, matches REFINE retrieval to within 0.003 points and lowers degraded groups by 3.701 points [−11.087, +0.040].
Q2(a). Ledger-derived against uniform weights. With branch, coefficients, budget and guard fixed, gap minus uniform gives +0.000 R@1 [−0.011, +0.019] and −0.119 damage [−0.256, +0.026] for AWARE, and −0.010 R@1 [−0.024, −0.001] and +0.207 damage [+0.012, +0.392] for BALANCED. The gap weights are a tuning-free default that leaves retrieval in place; the ledger's damage reduction comes from its group-defined objectives, the 3.701 points above.
Q2. CI for the degraded-group difference. The submitted 20.01% against 21.65% is −1.648 points, source-cluster bootstrap CI [−5.302, +0.070], exact sign-flip p = 0.1875. On full-v0 at matched guard recall, the zero-gain target with all 66 cells matched, router minus REFINE is −0.002 R@1 and −0.617 damage.
W2. Held-out gains from the ledger-aware components. SHARPEN's transductive advantage holds out of sample. On ImageNet-A, -R and -1K, where the router selects SHARPEN, routed R@1 exceeds REFINE by +2.608, +0.547 and +0.932 points and frozen by +5.058, +1.560 and +1.722, and it is higher than REFINE in all six source and backbone cells. The damage advantage is robust to the guard thresholds: re-applying the guard on seed 42 at all 20 combinations of recall tolerance {0, .01, .025, .05} and damage cap {0, .05, .1, .2, .3}, fixed AWARE has fewer degraded groups than REFINE in every one, by 1.922 to 4.765 points, and the router's R@1 exceeds REFINE in 16.
Q3. Routing provenance and split unit. The routing map was assembled during development on these suites, which is why the rule for new data is the single global configuration above. On Flickr30K, never used in development, with 95,045 image-disjoint pairs after removing 223 overlapping images, none of the 18 new-source cells exceeds the damage cap or recall tolerance, and pooled over ten sources the router exceeds REFINE by +0.416 R@1 [+0.053, +1.249] while fixed AWARE lowers degraded groups by 2.998 points [−10.881, +0.273]. The submitted strict split was per pair within groups, Sec. 4.2; full-v0 keeps all captions of an image in one role.
Guard plus audit, compared with WiSE-FT and PAINT. Used to select the interpolation scale of native WiSE-FT and PAINT, the ledger guard raises evaluation-only ObjectNet accuracy by +0.528 points, paired 95% CI [+0.407, +0.645], and lowers degraded groups by 5.531 points over 18,574 images, compared with their own accuracy-based selection. In all six CLIP cells, where the selectors differ, accuracy rises by +0.716 to +1.228 and degraded groups fall by 6.195 to 17.699 points.
Suggestions. The guard keeps native compositional behaviour intact: Winoground group accuracy on 245 examples is 7.755% frozen and 7.347% routed for CLIP, one example apart, and 9.388% for both on SigLIP; SugarCrepe accuracy on 4,588 pairs is 78.357% and 78.269%, and 83.936% for both. On related work, the degraded-group rate is the group-level analogue of the negative flip rate, Yan et al. 2021; the ledger gap a_g is a per-group centroid modality gap, Liang et al. 2022; and contrastive adapters, Zhang & Ré 2022, target worst-group accuracy, not per-group change against the frozen model. The router code follows Table 13, and we will fix the typos in Figures 1 and 2.