acceptodds
Desk rejected at ICLR 2027

AlignLedger-CLIP: Group-Wise Auditing and Guarded Adaptation for Frozen Vision-Language Embeddings

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Aggregate retrieval scores can conceal regressions within semantic groups when adapting frozen vision–language embeddings. We introduce ALIGNLEDGER-CLIP, a framework that connects group-wise auditing to adaptation and update selection. A ledger records group geometry, directional retrieval, and representation changes; its statistics weight gap-closing and preservation objectives for lightweight cached- feature adapters. A held-out guard selects an interpolation scale subject to recall and group-degradation constraints, with the frozen representation as a fallback. The same ledger then measures the deployed update on evaluation records. We study 12 evaluation slices from 10 source datasets with CLIP and SigLIP, separating transductive audits from disjoint fit/guard/evaluation experiments. In the latter, guarded ALIGNLEDGER-CLIP achieves 38.27 average Recall@1 with 20.01% degraded groups, while identically guarded CLIP-Refine achieves 38.53 with 21.65%. These retrieval–degradation profiles expose the local cost of adaptation alongside its aggregate benefit.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision–language representations, Cross-modal retrieval, Embedding adaptation, Local geometry repair, Reliability calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wf9eOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that aggregate retrieval scores can hide regressions inside semantic groups when frozen CLIP/SigLIP embeddings are adapted post hoc. It proposes a "ledger–adapt–guard–re-ledger" loop on cached features. (i) A per-group ledger records the image–text centroid gap ag=∥μgI−μgT∥2a_g = \lVert \mu_g^{I} - \mu_g^{T} \rVert_2, within-modality compactness, top-1 group-hit consistency, and bidirectional recall. (ii) Two identity-initialized linear maps WI,WT∈Rd×dW_I, W_T \in \mathbb{R}^{d \times d} are trained with one of five objective branches, chosen per dataset–backbone slice by a fixed routing table. The gap-closing and preservation terms are weighted by wgclose∝ag/aˉw_g^{\mathrm{close}} \propto a_g/\bar{a} and wgkeep=max⁡(0.05,1−ag/aˉ)w_g^{\mathrm{keep}} = \max(0.05, 1 - a_g/\bar{a}), assuming unit manifest weights. (iii) A held-out guard selects the largest α∈{1,0.5,0.25,0.1,0.05,0.01}\alpha \in \lbrace 1, 0.5, 0.25, 0.1, 0.05, 0.01 \rbrace for z(α)=norm((1−α)z0+αzadapt)z(\alpha) = \mathrm{norm}\big((1-\alpha) z^{0} + \alpha z^{\mathrm{adapt}}\big) such that ΔRI→T,ΔRT→I≥−0.05\Delta R^{I\to T}, \Delta R^{T\to I} \ge -0.05 and Dguard(α)≤0.20D_{\mathrm{guard}}(\alpha) \le 0.20. If no scale passes, it falls back to the frozen embeddings (α⋆=0\alpha^\star = 0). (iv) The evaluation set is re-ledgered to report the fraction of degraded groups.

Experiments cover 12 slices from 10 datasets with two backbones. They use two protocols: a strict protocol in which the fit, guard, and evaluation sets (Dfit\mathcal D_{\mathrm{fit}}, Dguard\mathcal D_{\mathrm{guard}}, Deval\mathcal D_{\mathrm{eval}}) are disjoint, and a transductive audit with Dfit=Deval\mathcal D_{\mathrm{fit}} = \mathcal D_{\mathrm{eval}}. Under the strict protocol, the method reaches 38.27 R@1 with 20.01% degraded groups, compared with 38.53 R@1 and 21.65% for CLIP-Refine under the same guard.

Critical strengths
  1. A relevant question, evaluated with a fair and transparent protocol. The paper asks whether aggregate gains from post-hoc adaptation hide group-level regressions, which matters for anyone updating deployed embeddings. In the disjoint fit/guard/eval experiment, the baseline receives exactly the same guard as the proposed method. The authors report the results transparently, including a comparison they do not win (Table 10), and support them with bootstrap intervals and leave-one-dataset-out checks. This is consequential because it makes the evidence trustworthy and makes the paper a useful reference point for evaluating cached-feature adaptation.

  2. A useful, adapter-agnostic finding about guarded updates. Under held-out evaluation, unguarded adaptation falls below the frozen model (34.71 and 35.37 vs. 36.03 R@1). Guarded interpolation recovers about +3.55 R@1 and roughly halves the share of degraded groups (from 43.40% to 20.01%). Because the guard is simple and works with any adapter, this is the paper's most transferable contribution and the main reason I see a path to acceptance.

Critical weaknesses
  1. The held-out results do not yet show that the ledger-specific components add value beyond the guard. Under the strict protocol, the fixed REFINE branch with the guard produces the same numbers as guarded CLIP-Refine (Tables 9–12). The routed method scores 0.26 R@1 lower than both (CI [−0.51,−0.07][-0.51, -0.07]). The AWARE and BALANCED branches trade lower R@1 for fewer degraded groups. The gap-derived weighting (wgclose,wgkeepw_g^{\mathrm{close}}, w_g^{\mathrm{keep}}) is not ablated, since Table 3 uses uniform weights. The 1.64-point advantage in degraded groups has no confidence interval. This is consequential because it decides what the contribution is. If the contribution is the guard plus the audit protocol, the paper should be framed that way and compared with related interpolation-selection methods (e.g., WiSE-FT, PAINT). If the ledger-weighted objectives and the router are central, a held-out ablation is needed. Since all experiments run on cached features, I expect this to be addressable during the rebuttal.

  2. Much of the evidence comes from the transductive audit, and the design of the routing table is not documented. Tables 2–7, the component ablation, and the qualitative figures come from fitting on the evaluation records (Dfit=Deval\mathcal D_{\mathrm{fit}} = \mathcal D_{\mathrm{eval}}). Under held-out evaluation the gains over frozen shrink substantially: Winoground goes from +43.1+43.1 to −0.6-0.6, and COCO-Attr from +10.4+10.4 to +0.5+0.5. The routing table also includes backbone-specific branch choices and 14 per-slice overrides, and the paper does not explain how these were chosen. This is consequential because the strict experiment is the only evidence of generalization. Its credibility depends on the routing and overrides having been fixed without reference to the evaluation records. A clear description of the procedure, or a strict-protocol result with a single global configuration, would resolve most of this concern.

Questions for authors

Clarifications (answers could change my rating)

  1. Is the REFINE branch the same objective and implementation as CR-off? Since guarded REFINE matches guarded CR-off exactly, which part do you consider the main contribution: the guard and audit protocol, or the ledger-weighted objectives and the router?
  2. Under the strict protocol, could you report (a) ledger-derived weights vs. uniform weights (wgclose=wgkeep=1w_g^{\mathrm{close}} = w_g^{\mathrm{keep}} = 1), and (b) the routed method vs. a single fixed configuration? A confidence interval for the degraded-group difference (20.01% vs. 21.65%) would also help. These should be inexpensive to run on cached features.
  3. How were the routing map (Table 13) and the overrides (Table 14) chosen? In particular, were any paper-scale results consulted before the strict split was made? Is the split into Dfit\mathcal D_{\mathrm{fit}}, Dguard\mathcal D_{\mathrm{guard}}, and Deval\mathcal D_{\mathrm{eval}} done by image ID or by pair, given that some datasets have several captions or questions per image?

If the rebuttal shows that the ledger-aware components improve the held-out retrieval–degradation trade-off, or if it repositions the paper around the guard and audit protocol and resolves the clarifications above, I would be inclined to raise my rating.

Suggestions (not expected for the rebuttal)

  • Discuss related work on interpolation and model patching (WiSE-FT, PAINT), on regression-free model updates (negative flip rate), on group-robust CLIP adapters (Zhang & Ré, 2022), and on the modality gap (Liang et al., 2022).
  • Report the standard Winoground and SugarCrepe metrics alongside the retrieval-pool variants.
  • Fix the typos and placeholder content in Figures 1–2, and align the supplementary router code with Table 13.
2 comments
AuthorsOpenReview ↗

CR-off, a single global configuration, weighting controls and routing provenance

We thank the reviewer for recognising the fair protocol and the transferable guard. We ran the requested controls under a stricter protocol, full-v0, where fit, guard and evaluation sets are disjoint by global image ID, decoded pixels and source ID. It has 528 core cells, 11 fit-eligible slices × 2 backbones × 8 variants × 3 optimizer seeds, with ObjectNet used only for evaluation. Inference resamples seven shared-image source blocks, so the smallest Holm-adjusted p over our 14 predeclared tests is 0.21875; we report effects, 95% CIs and exact p-values.

Q1. REFINE against CR-off, and the main contribution. Yes. REFINE is deliberately the CLIP-Refine objective, included as a branch the ledger can route to, and fit-only checks at a common budget give maps identical bit for bit. The core contribution is the loop between ledger and guard, a group-wise audit coupled to adaptation and to update selection. Guarding alone, relative to the same maps at α = 1, removes 3.708 damage points for the router at R@1 +0.006 and 1.826 for REFINE at R@1 −0.045.

Q2(b). Routing against a single fixed configuration, default guard with tolerance 0.05 and cap 0.20:

ConfigurationR@1 %Degraded groups %
Fixed REFINE, equal to guarded CR-off31.05819.824
Routed AlignLedger-CLIP31.53019.746
Single global default, fixed AWARE31.05516.123

The routed method exceeds REFINE by +0.472 R@1 points, CI [+0.077, +1.251], exact p = 0.09375, at level damage of −0.078 [−0.982, +1.473], and every leave-one-source-out difference stays positive, from +0.205 to +0.532. The single global configuration makes no per-dataset choice, matches REFINE retrieval to within 0.003 points and lowers degraded groups by 3.701 points [−11.087, +0.040].

Q2(a). Ledger-derived against uniform weights. With branch, coefficients, budget and guard fixed, gap minus uniform gives +0.000 R@1 [−0.011, +0.019] and −0.119 damage [−0.256, +0.026] for AWARE, and −0.010 R@1 [−0.024, −0.001] and +0.207 damage [+0.012, +0.392] for BALANCED. The gap weights are a tuning-free default that leaves retrieval in place; the ledger's damage reduction comes from its group-defined objectives, the 3.701 points above.

Q2. CI for the degraded-group difference. The submitted 20.01% against 21.65% is −1.648 points, source-cluster bootstrap CI [−5.302, +0.070], exact sign-flip p = 0.1875. On full-v0 at matched guard recall, the zero-gain target with all 66 cells matched, router minus REFINE is −0.002 R@1 and −0.617 damage.

W2. Held-out gains from the ledger-aware components. SHARPEN's transductive advantage holds out of sample. On ImageNet-A, -R and -1K, where the router selects SHARPEN, routed R@1 exceeds REFINE by +2.608, +0.547 and +0.932 points and frozen by +5.058, +1.560 and +1.722, and it is higher than REFINE in all six source and backbone cells. The damage advantage is robust to the guard thresholds: re-applying the guard on seed 42 at all 20 combinations of recall tolerance {0, .01, .025, .05} and damage cap {0, .05, .1, .2, .3}, fixed AWARE has fewer degraded groups than REFINE in every one, by 1.922 to 4.765 points, and the router's R@1 exceeds REFINE in 16.

Q3. Routing provenance and split unit. The routing map was assembled during development on these suites, which is why the rule for new data is the single global configuration above. On Flickr30K, never used in development, with 95,045 image-disjoint pairs after removing 223 overlapping images, none of the 18 new-source cells exceeds the damage cap or recall tolerance, and pooled over ten sources the router exceeds REFINE by +0.416 R@1 [+0.053, +1.249] while fixed AWARE lowers degraded groups by 2.998 points [−10.881, +0.273]. The submitted strict split was per pair within groups, Sec. 4.2; full-v0 keeps all captions of an image in one role.

Guard plus audit, compared with WiSE-FT and PAINT. Used to select the interpolation scale of native WiSE-FT and PAINT, the ledger guard raises evaluation-only ObjectNet accuracy by +0.528 points, paired 95% CI [+0.407, +0.645], and lowers degraded groups by 5.531 points over 18,574 images, compared with their own accuracy-based selection. In all six CLIP cells, where the selectors differ, accuracy rises by +0.716 to +1.228 and degraded groups fall by 6.195 to 17.699 points.

Suggestions. The guard keeps native compositional behaviour intact: Winoground group accuracy on 245 examples is 7.755% frozen and 7.347% routed for CLIP, one example apart, and 9.388% for both on SigLIP; SugarCrepe accuracy on 4,588 pairs is 78.357% and 78.269%, and 83.936% for both. On related work, the degraded-group rate is the group-level analogue of the negative flip rate, Yan et al. 2021; the ledger gap a_g is a per-group centroid modality gap, Liang et al. 2022; and contrastive adapters, Zhang & Ré 2022, target worst-group accuracy, not per-group change against the frozen model. The router code follows Table 13, and we will fix the typos in Figures 1 and 2.

Thank you for the detailed rebuttal and the substantial new experiments, I have raised the score to 3 (weak acceptance).

Reviewer_4TwSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ALIGNLEDGER-CLIP, a framework for group-wise auditing and adaptation of frozen vision–language embeddings. A ledger guides lightweight adapters, while a held-out guard selects the update scale to limit recall loss and group degradation. Across 12 benchmark slices and two backbones, the method improves retrieval over frozen embeddings and yields fewer degraded groups than equally guarded CLIP-Refine, at slightly lower aggregate recall.

Critical strengths
  • Presents a unified framework that uses group-level alignment and retrieval statistics to guide adaptation, select updates, and audit their effects.
  • Introduces a simple calibration procedure that adjusts update strength to satisfy recall and group-degradation constraints on held-out data, reverting to frozen embeddings when necessary.
  • Validates the framework across 12 benchmark slices from 10 datasets and two backbones, measuring both overall retrieval and group-level regressions under disjoint and transductive settings.
Critical weaknesses
  • The routing strategy is heavily hand-designed, and its benefit remains unclear. The branches use the same pair of linear adapters and differ mainly in their losses and coefficients. The paper specifies dataset-, backbone-, and scale-dependent routing rules, but does not adequately explain how these choices were developed or how to apply them to a new dataset. Fixed-branch comparisons are provided, yet fixed REFINE achieves higher held-out retrieval than the routed method, at a slightly higher degraded-group rate. This leaves the need for the more elaborate routing scheme insufficiently established.

  • The evidence for the ledger’s role in adaptation is limited. The full-scale classification results show that ranking and preservation matter, but this profile uses uniform group weights and therefore does not test the proposed gap-derived weighting. Removing the gap term changes R@1 by only 0.18 points. Moreover, SHARPEN’s advantages on ImageNet are demonstrated in the transductive setting. A held-out comparison with the branch and training settings fixed would help establish whether ledger-based weighting adds value beyond the other loss terms.

  • The calibration rule and damage measure need stronger validation. The guard selects the largest passing interpolation scale from a fixed grid, using a five-point recall tolerance and a 20% damaged-group cap. The paper does not establish how sensitive its conclusions are to these choices. Damage is also defined by any decrease in group-hit consistency, without accounting for its magnitude or uncertainty. Since the main advantage over guarded CLIP-Refine is a reduction from 21.65% to 20.01% degraded groups, uncertainty estimates and regression magnitudes are needed to judge whether this difference is meaningful.

Questions for authors
  1. How were the routing table and dataset-specific coefficient overrides chosen? Were they informed by results on the evaluated benchmarks, and what rule would determine the branch and coefficients for a new dataset?

  2. Can you isolate the benefit of gap-derived group weighting under the disjoint protocol, comparing it with uniform weights while keeping the branch, training settings, and guard fixed? The main classification ablation uses uniform weights and therefore does not answer this question.

1 comment
AuthorsOpenReview ↗

Routing rule, gap-derived weighting and guard calibration

We thank the reviewer for recognising the unified framework. To answer all three concerns we used a stricter full-scale protocol, full-v0: fit, guard and evaluation sets are disjoint by global image ID, decoded pixels and source ID, whereas the submitted split was made per pair. It has 528 core cells, 11 fit-eligible slices × 2 backbones × 8 variants × 3 optimizer seeds, with ObjectNet used only for evaluation. Inference resamples seven shared-image source blocks jointly. The smallest attainable exact two-sided p is thus 0.015625 and the smallest Holm-adjusted p over our 14 predeclared tests 0.21875, so we report effect sizes, marginal 95% CIs and exact p-values.

Q1. Routing, and the rule for a new dataset. At execution time the router is a fixed lookup over dataset metadata, assembled during development on these suites. For that reason the rule for a new dataset is the single global default, fixed AWARE, which needs no per-dataset branch or coefficient. Under the default guard, recall tolerance 0.05 and damage cap 0.20:

ConfigurationR@1 %Degraded groups %
Fixed REFINE, the CLIP-Refine objective31.05819.824
Routed AlignLedger-CLIP31.53019.746
Global default, fixed AWARE31.05516.123
Fixed BALANCED31.03016.271

The routed method exceeds REFINE by +0.472 R@1 points, CI [+0.077, +1.251], exact p = 0.09375, at level damage of −0.078 points [−0.982, +1.473]. The gain is not carried by the REFINE entries of the table: excluding them it rises to +0.706 [+0.114, +1.513], and every leave-one-source-out estimate stays positive, from +0.205 to +0.532. The global default matches REFINE retrieval to within 0.003 points and lowers degraded groups by 3.701 points, CI [−11.087, +0.040], so a user who never consults the table already gets the main safety benefit.

We then applied the unchanged global rule to Flickr30K, never used in development, with 95,045 image-disjoint pairs after removing 223 overlapping images. None of the 18 new-source cells exceeds the damage cap or the recall tolerance, and pooled over all ten sources the router still exceeds REFINE by +0.416 R@1 [+0.053, +1.249] while fixed AWARE lowers degraded groups by 2.998 points [−10.881, +0.273].

Q2. Gap-derived weighting. With branch, data, coefficients, budget, batch order and guard held fixed, gap minus uniform weights gives +0.000 R@1 [−0.011, +0.019] and −0.119 damage [−0.256, +0.026] for AWARE, and −0.010 R@1 [−0.024, −0.001] and +0.207 damage [+0.012, +0.392] for BALANCED. The gap weights are thus a tuning-free default that moves retrieval by at most 0.010 points. The damage reduction comes from the group-defined objectives the ledger supplies: at the same guard and equal aggregate retrieval, AWARE has 3.701 fewer degraded-group points than REFINE. SHARPEN's advantage also holds under held-out evaluation. The router selects SHARPEN for the ImageNet-A, -R and -1K sources, and there routed R@1, averaged over backbones and three seeds, exceeds REFINE by +2.608, +0.547 and +0.932 points and frozen by +5.058, +1.560 and +1.722, higher than REFINE in all six source and backbone cells.

Calibration rule and damage measure. The submitted 1.64-point difference is −1.648 points, source-cluster bootstrap CI [−5.302, +0.070], exact sign-flip p = 0.1875. On full-v0, guarding the same maps relative to α = 1 removes 3.708 damage points for the router at R@1 +0.006, and 1.826 for REFINE at R@1 −0.045. Beyond the sign-only rate, the router has a lower group-size-weighted damaged fraction, 22.874% against 25.034% for REFINE, and fewer evaluation cells over the damage cap, 46.296% against 53.704%; mean negative-change magnitude is 0.659 against 0.588 points, and 0.592 for fixed AWARE. The conclusions are stable across the full threshold grid. Re-applying the guard on seed 42 for all 20 combinations of recall tolerance {0, .01, .025, .05} and damage cap {0, .05, .1, .2, .3}, fixed AWARE has fewer degraded groups than REFINE in all 20, by 1.922 to 4.765 points. The router's R@1 exceeds REFINE in 16 and is never more than 0.015 points lower, and its damage is lower in 15, by 1.207 to 1.634 points at every cap up to 0.1. At seven predeclared matched guard-recall targets, where evaluation never selects scales, router minus REFINE is −0.002 R@1 and −0.617 damage at the zero-gain target with all 66 cells matched, and +0.069 R@1 and −4.463 damage at the +1 point target.

The calibration rule also transfers. Used to choose the interpolation scale of native WiSE-FT and PAINT, ledger selection raises ObjectNet accuracy by +0.528 points, paired 95% CI [+0.407, +0.645], and lowers degraded groups by 5.531 points over 18,574 images. In all six CLIP cells, where the two selectors differ, accuracy rises by +0.716 to +1.228 and degraded groups fall by 6.195 to 17.699 points. The same thresholds therefore pick safer operating points for other methods without retuning, and fitting all 528 cells took 0.0869 CPU-hours plus 1.4717 CUDA-hours.

Reviewer_9KPUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets a real gap: aggregate retrieval metrics can conceal regression inside semantic groups after post-hoc adaptation of frozen CLIP/SigLIP embeddings. AlignLedger-CLIP closes a ledger → adapt → guard → re-ledger loop: group statistics (centroid gap agag, intra-modal compactness, bidirectional retrieval, group-hit consistency KgKg) are recorded, used to weight gap-closing and preservation objectives over identity-initialized linear maps on cached features, a metadata-only router selects one of five fixed branches, and a held-out guard picks the largest interpolation scale satisfying recall and degraded-group constraints, otherwise reverting to the frozen model. Evaluation spans 10 source datasets / 12 slices, two backbones, and three suites (multi-seed 720 runs, paper-scale 42,678 pairs, full-scale 2.51M pairs), under both transductive and disjoint fit/guard/eval protocols.

Critical strengths
  1. Well-motivated framing. Reporting retrieval jointly with group-level degradation addresses a genuine blind spot in cached-feature adaptation.
  2. Protocol discipline is the paper's strongest asset. Disjoint fit/guard/eval splits, the same guard, thresholds and candidate grid applied to every compared method, paired bootstrap, and leave-one-out worst case. This is above the norm for the area.
  3. The calibration guard is a real, reusable contribution. Identically applied, it lifts AlignLedger 34.71 → 38.27 and CLIP-Refine 35.37 → 38.53, cutting degraded groups from ~43% to ~20%, with frozen fallback as a safe default.
  4. Honest reporting. Null and negative results are stated explicitly (all transductive CIs contain zero; the strict-protocol deficit is disclosed), and the tables are arithmetically self-consistent.
Critical weaknesses

W1. The method loses to an identically guarded baseline on the primary metric under the only leak-free protocol. Guarded CLIP-Refine: 38.53 R@1 / 21.65% degraded groups; guarded AlignLedger-CLIP: 38.27 / 20.01%. The −0.26 gap is significantly negative (dataset-level CI [−0.51, −0.07]; slice-level [−0.52, −0.04]; W/T/L 4/13/7), while the sole claimed advantage — 1.64 points of degraded-group rate — has no confidence interval, no paired test, and is measured at a different operating point. The header claim of an improved retrieval–degradation profile is therefore unsupported. W2. Attribution fails: the gains come from the generic guard, not the ledger. Guard yields ~+3 points for both methods. In ablation, the name-bearing ledger-gap term is worth −0.18 R@1 and teacher stability −0.21, whereas the actual drivers are rank sharpening (−6.89) and preservation (−5.01). The authors further acknowledge that full-scale SHARPEN uses uniform group weights, so gap-derived weighting was never isolated in a branch where it applies. W3. The strongest branch replicates the baseline, and routing adds no demonstrated value. Per the routing table, 6 of 12 full-scale slices route to REFINE, whose objective matches CLIP-Refine's; those columns are numerically identical. Router vs. fixed REFINE is −0.26 (significantly negative), while fixed AWARE/BALANCED attain lower degraded-group rates (15.47/16.57) than the routed method. The router is a per-dataset lookup table with 14+ per-slice coefficient overrides (learning rate spanning 10−610−6–10−210−2, 1–10 epochs, one slice's candidate set pinned to {0.05}{0.05}) whose selection procedure is undisclosed, leaving benchmark-overfitting as an open explanation. W4. The degraded-group metric is too weak to carry the second claim. hg=1[ΔKˉg<0]hg=1[ΔKˉg<0] is sign-only and magnitude-blind, equally weights groups ranging from 2 to 1,469, has no noise floor, and is never tested for significance — yet it serves both as a hard guard constraint (0.20) and as the headline number. The thresholds (−0.05, 0.20) are unswept and unconnected to data properties, and the guard's ability to predict evaluation-time damage is never quantified. W5. No retrieval–degradation Pareto frontier. Single-point comparison cannot establish a better trade-off — notably, plain reference reversion already reaches 4.75% degraded groups at negligible retrieval cost. The paper needs a sweep over recall tolerance, degradation cap and scale, with comparison at matched operating points. W6. External validity and comparison gaps. The aggregate advantage rests on transductive audits whose CIs all contain zero; the strict protocol covers only paper-scale. DocVQA/ChartQA measure paired-text retrieval rather than answer accuracy, and Winoground/SugarCrepe use candidate-pool retrieval rather than their original coupled-matching or distractor tests, so numbers cannot be read against the literature. Near-floor slices (VG-R 0.48%, ChartQA 3.5–4.7%) receive equal weight in a 12-column mean, and the paper-scale gain (+0.08) is driven almost entirely by one ChartQA column (+9.71) while ImageNet-A/R regress (−3.47/−1.82). Group-robust and adapter baselines (You et al. 2024, Project-Probe-Aggregate, Tip-/CLIP-Adapter, CoOp family) are cited but never compared, and the guard's close lineage to WiSE-FT and L2-SP-style preservation is neither cited nor compared. Reproducibility: no code, cached features or checkpoints. Minor: repeated superscript and an undefined wgclosewgclose in the gap term, a missing scale-selection equation; an unclear self-similarity term in the within-modality loss; a routing/table inconsistency for ObjectNet; figure typos; cherry-picked qualitative cases; no compute-cost accounting.

Questions for authors
  1. Please provide the full (R@1, degraded%) Pareto frontier for both methods and the CI/test on the 1.64-point degraded-group difference.
  2. How many slices route to REFINE, and what is the aggregate excluding them? Why retain routing if always-REFINE scores higher?
  3. Isolate gap-derived weighting on AWARE/BALANCED against equal- and random-weight controls with identical losses, budget and guard.
  4. How were the routing and coefficient tables derived — was any independent development set used? What is the all-default-profile result?
  5. What is the noise floor of DgroupDgroup (frozen vs. frozen on independent splits)? Please add worst-group, magnitude, and group-size-weighted variants.
  6. Why do file-level partitions for ChartQA/DocVQA/SugarCrepe constitute semantic groups?
2 comments
AuthorsOpenReview ↗

Part 1 of 2: image-disjoint retrieval, attribution and routing

We thank the reviewer for the detailed review. To address W1 to W5 we ran a stricter full-scale protocol, full-v0, in which fit, guard and evaluation sets are disjoint by global image ID, decoded pixels and source ID. The submitted strict split was made per pair within groups, Sec. 4.2; full-v0 additionally keeps every caption of an image in a single role. It has 528 core cells, 11 slices × 2 backbones × 8 variants × 3 optimizer seeds, with ObjectNet used only for evaluation. Inference resamples seven shared-image source blocks jointly, so the smallest attainable exact p is 0.015625 and the smallest Holm-adjusted p over our 14 predeclared tests is 0.21875. We therefore report effects, marginal 95% CIs and exact p-values.

W1. Retrieval against identically guarded CLIP-Refine. The −0.26 gap the reviewer cites was measured on the pair-level split. Under image-level disjointness the routed method exceeds fixed REFINE, the CLIP-Refine objective with the same guard, by +0.472 R@1 points: 31.530% against 31.058%, marginal CI [+0.077, +1.251], exact block sign-flip p = 0.09375. Damage is level at 19.746% against 19.824%, a difference of −0.078 points [−0.982, +1.473]. For the 1.64-point difference in the submission, the requested CI is −1.648 points, with source-cluster bootstrap [−5.302, +0.070] and exact paired sign-flip p = 0.1875.

W2 and Q3. Attribution to the ledger. The guard is part of AlignLedger-CLIP. It selects scales under the ledger's group-degradation constraint, and we apply it to the baseline as well so the comparison is fair. The ledger-aware objectives add to what the guard does. With the same guard and equal retrieval, fixed AWARE at 31.055% R@1 lowers degraded groups by 3.701 points relative to REFINE, marginal CI [−11.087, +0.040], and BALANCED at 31.030% lowers them by 3.554 points, [−11.346, +0.005].

For the requested weight controls we held branch, losses, budget, batch order and guard fixed. Gap against uniform weights changes AWARE by +0.000 R@1, [−0.011, +0.019], and −0.119 damage, [−0.256, +0.026]; it changes BALANCED by −0.010 R@1, [−0.024, −0.001], and +0.207 damage, [+0.012, +0.392]. Against seeded permuted weights, AWARE moves by −0.004 R@1 and +0.308 damage, and both intervals contain zero. The gap weights are a tuning-free default that leaves retrieval where it is; the damage reduction comes from the group-defined objectives, as the 3.701 and 3.554 points above show.

W3, Q2 and Q4. Routing. The lookup has 14 REFINE entries out of 24. Two of them are evaluation-only ObjectNet entries, so 12 of the 22 fit-eligible cells use REFINE. If we exclude those cells, the router exceeds REFINE by +0.706 R@1, [+0.114, +1.513], and every leave-one-source-out difference stays positive, from +0.205 to +0.532. Where the lookup selects SHARPEN, on ImageNet-A, -R and -1K, routed R@1 averaged over backbones and seeds exceeds REFINE by +2.608, +0.547 and +0.932 points, and it is higher in all six source and backbone cells. Routing therefore beats always choosing REFINE, and the gain does not hinge on any single source. At run time the lookup is fixed and reads only metadata; it was assembled during development on these suites. For new data the rule is the all-default profile, which equals fixed AWARE: 31.055% R@1 and 16.123% damage, against 31.058% and 19.824% for REFINE.

We tested this rule on Flickr30K, which our development records show was never used. After removing 223 overlapping images it contributes 95,045 image-disjoint pairs. None of the 18 new-source cells exceeds the evaluation cap or the recall tolerance. Pooled over all ten sources, the router exceeds REFINE by +0.416 R@1, [+0.053, +1.249], and fixed AWARE lowers damage by 2.998 points, [−10.881, +0.273].

AuthorsOpenReview ↗

Part 2 of 2: damage measure, external validity, groups and reproducibility

W4, W5, Q1 and Q5. Damage measure, frontier and noise. Relative to the same maps at α = 1, guarding removes 3.708 damage points for the router at R@1 +0.006, and 1.826 for REFINE at R@1 −0.045. On the requested variants of the sign-only rate, router against REFINE gives a group-size-weighted damaged fraction of 22.874% against 25.034%, a mean worst-group consistency drop of 21.846 against 32.460 points, and 46.296% against 53.704% of evaluation cells over the cap with source-equal weighting. Mean negative-change magnitude is 0.659 against 0.588 points. For fixed AWARE it is 0.592, within 0.004 points of REFINE, so its 3.701-point reduction in degraded groups comes at essentially the same average drop size. Guard damage predicts evaluation damage: across the 66 default-guard cells, the Spearman correlation between guard and evaluation damage at the selected scale is 0.549 for the router, 0.531 for REFINE and 0.807 for AWARE, and along the seed-42 scale curves the two rank scales alike, with mean Kendall τ-b of 0.687, 0.638 and 0.756.

For the frontier we re-applied the guard on seed 42 at all 20 combinations of recall tolerance {0, .01, .025, .05} and damage cap {0, .05, .1, .2, .3}. Fixed AWARE has fewer degraded groups than REFINE in all 20, by 1.922 to 4.765 points. The router's R@1 exceeds REFINE in 16 and is never more than 0.015 points lower, and its damage is lower in 15, by 1.207 to 1.634 points at every cap up to 0.1. We also fixed seven matched guard-recall targets in advance; evaluation never selects scales. At the zero-gain target, where all 66 cells match, router minus REFINE is −0.002 R@1 and −0.617 damage points. At +1 point, with 8 of 66 cells matched, it is +0.069 R@1 and −4.463 damage. Reference reversion reaches 4.75% only by rejecting 77.8% of updates, and the routed method exceeds it by +2.17 R@1 points, CI [+0.80, +4.03], Table 10. Interpolation keeps the adaptation gain.

On the noise floor: frozen fit-versus-guard comparisons use disjoint image partitions with different candidate pools. In CLIP COCO-object, 49.37% of 79 groups score lower on guard, with a mean absolute difference of 8.70 points; in ImageNet-A the figures are 40.21% of 189 groups and 26.09 points. This partition-level variability is why every method contrast above is paired on identical evaluation images and seeds.

W6. External validity and baselines. All new evidence uses the strict image-disjoint protocol at full scale with source-equal averaging, so slices near the floor cannot dominate, and the leave-one-source-out range above shows that no single slice drives the gain. The guard keeps native compositional behaviour intact. Winoground group accuracy on 245 coupled examples is 7.755% frozen against 7.347% routed for CLIP, one example apart, and 9.388% for both on SigLIP. SugarCrepe micro accuracy on 4,588 pairs is 78.357% against 78.269% for CLIP and 83.936% for both on SigLIP. DocVQA and ChartQA are evaluated as paired-text retrieval.

We ran 318 external evaluations, 222 cached and 96 native, plus 72 native training runs, covering CFR, PPA, Tip-Adapter, CLIP-Adapter, L2-SP, CoOp, WiSE-FT and PAINT. Paper-equation implementations and SigLIP ports are labelled as such. WiSE-FT and PAINT fine-tune the full encoder, a heavier regime than our two cached-feature linear maps, and in the six transfer settings this buys them higher classification accuracy. The two approaches are complementary and combine well. Used as the interpolation selector inside WiSE-FT and PAINT, our ledger guard raises accuracy on evaluation-only ObjectNet by +0.528 points, joint paired-image 95% CI [+0.407, +0.645], and lowers degraded groups by 5.531 points compared with their accuracy-based selection, over 18,574 images, 2 backbones × 2 methods × 3 seeds. In each of the six CLIP cells where the two selectors differ, accuracy rises and degraded groups fall.

Q6. Groups. Groups come from dataset annotation fields, not from files. ChartQA uses the human_or_machine provenance field with two strata, DocVQA the first non-empty question_types annotation with nine types, and SugarCrepe its seven perturbation categories. Both ChartQA groups lie in one parquet file and DocVQA types span several files, so groups do not follow file boundaries.

Presentation and reproducibility. The close and keep weights are defined in Eq. (5) and scale selection in Eq. (14). Table 13, Table 14 and the router code agree on ObjectNet: Table 13 and the code route it to REFINE at full scale, checked before the SHARPEN rule for other classification sources, and the Table 14 override is stated for classification except ObjectNet. For unit-normalised vectors the self-similarity numerator within a modality is constant, so the loss acts through the competing similarities in the denominator. Fitting all 528 core cells took 0.0869 CPU-hours plus 1.4717 CUDA-hours. We will release the code, cached features, adapter checkpoints, configurations and per-cell provenance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Sincere Appreciation and Overall Response

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for their thoughtful evaluations and constructive feedback. Your comments have helped us clarify the contributions of AlignLedger-CLIP and substantially strengthen its experimental validation. In response to the main concerns, we conducted extensive additional experiments focusing on evaluation reliability, methodological contributions, and robustness and generalizability.

First, to strengthen evaluation reliability, we introduced a stricter full-scale protocol (full-v0), ensuring that fitting, calibration, and evaluation sets are disjoint by image identity, decoded pixels, and source identity. Across 528 experimental cells, AlignLedger-CLIP achieves 31.530% Recall@1 compared with 31.058% for identically guarded CLIP-Refine, with comparable degraded-group rates (19.746% versus 19.824%). Although the retrieval difference is positive, we acknowledge its statistical uncertainty and report the corresponding confidence intervals and exact tests.

Building on this, we further clarified the contributions of individual components. Controlled experiments demonstrate that the reduction in group degradation primarily comes from the group-defined adaptation objectives and calibration guard, rather than gap-derived weighting alone. Notably, a single global AWARE configuration achieves nearly identical retrieval to guarded CLIP-Refine (31.055% versus 31.058%) while reducing degraded groups from 19.824% to 16.123%. Additional routing analyses and evaluation on the previously unused Flickr30K dataset further examine the benefits and transferability of our approach.

Furthermore, to assess robustness and broader applicability, we evaluated 20 calibration-threshold combinations, matched-recall operating points, and alternative group-degradation measures. Fixed AWARE achieves fewer degraded groups than REFINE across all 20 threshold combinations. We also expanded comparisons to additional adaptation methods, including WiSE-FT and PAINT. Applying our ledger-based guard to these methods improves ObjectNet accuracy by 0.528 percentage points while reducing degraded groups by 5.531 points, demonstrating its potential as a reusable calibration mechanism.

Overall, these additional experiments reinforce our central contribution: a unified framework connecting group-wise auditing, lightweight adaptation, and guarded update selection to jointly evaluate retrieval performance and semantic-group degradation. We have also clarified the limitations and statistical uncertainties of individual components and will incorporate these findings, together with additional methodological and reproducibility details, into the revised manuscript.

We sincerely appreciate the Area Chair and reviewers for their valuable time and constructive suggestions, which have significantly improved the rigor and clarity of our work.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.