acceptodds
Under review as a conference paper at ICLR 2027

PACE: Mitigating Autoregressive Scoring Biases for Robust Image-Text Retrieval

Abstract

Multimodal large language models (MLLMs) offer rich cross-modal reasoning capabilities for image-text retrieval, but their high inference cost and biased gen- erative likelihood limit their scalability. Global dual-encoder scores also struggle to distinguish highly similar negative samples. We propose PACE (Progressive Adaptive Cross-modal Enhancement), a training-free coarse-to-fine framework that selectively introduces MLLM-based evidence into efficient retrieval. PACE first employs SigLIP2 to construct a compact candidate pool and uses a margin- based guard to route only ambiguous cases to MLLM refinement. For large-scale reranking, we calibrate image-conditioned likelihood with a text-only prior us- ing pointwise mutual information (PMI), reducing the influence of language pri- ors. For fine-grained matching, we combine Position-Discounted Autoregressive Scoring (PDAS), which discounts later token contributions, with a Contrastive En- tailment Margin (CEM) based on visual Yes/No verification. The resulting frame- work separates efficient candidate retrieval from fine-grained multimodal verifica- tion, allowing MLLM computation to be allocated where it is most informative. Experiments across image-text retrieval and compositional benchmarks demon- strate improved fine-grained matching while substantially reducing unnecessary MLLM inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image-Text Retrieval, Multimodal Large Language Models, Hallucination Mitigation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_CBueOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PACE, a training-free framework for improving autoregressive image–text scoring. For large-scale retrieval, it combines dual-encoder candidate filtering with autoregressive reranking, including confidence routing and text-prior correction in the image-to-text pipeline. For fine-grained matching, it combines position-discounted scoring (PDAS) with selectively triggered visual verification (CEM), and uses bidirectional centering for closed pairing groups in Winoground. Experiments cover compositionality and retrieval benchmarks, multiple backbones, component ablations, and parameter and invocation-cost analyses.

Critical strengths
  1. A practical inference-only approach with measurable retrieval gains. The method addresses language-prior bias and the cost of using multimodal models for retrieval without additional training. In Table 4, Qwen3.5+PACE improves COCO image-to-text R@1 from SigLIP2’s 67.14% to 69.44%. Removing PMI calibration lowers it to 67.84%, providing positive evidence for calibration within the reported pipeline. This is the main practical contribution I see in the paper.
  2. Evaluation across tasks and backbones. The paper covers both large-scale retrieval and fine-grained matching, with multiple backbones, component ablations, and sensitivity and invocation-cost analyses. This scope makes the framework worth considering, even though the decisive evaluation questions below still need resolution.
Critical weaknesses
  1. Central results and evaluation steps remain unresolved. For Qwen3.5+PACE, SugarCrepe Replace-Object/Replace-Relation are 95.2%/87.8% in Table 1 but 93.16%/81.08% in Table 5. Different subsets or configurations may explain this, but none is specified. Separately, if Eq. (9) is applied to the final 2×2 Winoground matrix, the standard Text, Image, and Group success conditions become equivalent, whereas Qwen2.5+PACE scores 11.75%/88.00%/11.75% in Table 2. This conditional observation does not establish an implementation error. It does, however, leave central results insufficiently explained, which is the main reason for my current weak-rejection recommendation.
  2. The incremental contribution of PDAS needs a more precise explanation. Table 5 already includes a pure-CEM ablation, with a Micro-Avg of 86.58% versus 87.12% for the full method. However, if each component is min–max normalized over two candidates and its score difference is much larger than ε, the default weights generally make conflicting rankings follow CEM. The aggregate gain could therefore arise from CEM-disabled examples, ties, or near-ε cases. Clarifying the routing used in the pure-CEM ablation and separating these cases would make the claimed complementarity more convincing.
Questions for authors
  1. What subsets and configurations underlie Tables 1 and 5? If they are identical, please reconcile the full-method results and explain whether any main conclusions change.
  2. For each backbone on Winoground, what is the exact order of centering and fusion? Please explain the Qwen2.5 row and provide sufficient raw and final scores to check the pipeline. A control using the original scores with the same centering would help distinguish scoring improvements from joint pairing effects.
  3. What are the normalization scope and ε value? In Table 5’s pure-CEM ablation, is routing unchanged, and how are CEM-disabled examples scored? Please separate PDAS’s contribution on CEM-enabled and CEM-disabled examples under matched routing.
4 comments

Thank you for your review. Below we address your questions.

  1. Subsets and Configurations for Tables 1 and 5(a) on SugarCrepe
  • Underlying subsets and cause of the discrepancy: Both Table 1 and Table 5(a) evaluate the official SugarCrepe Replace-Object (N = 1,652) and Replace-Relation (N = 1,406) splits (3,058 examples in total). In our initial submission, Table 1 and the ablation configurations in Table 5(a) were executed as separate script runs without locking a global random seed (random.seed was unset prior to data loading and random.shuffle(candidate_texts)), which altered data partitioning/batch grouping under dynamic padding as well as candidate-pair ordering and index-0 tie-breaking under bfloat16 inference.
  • Reconciled results under a locked global seed (seed = 42): To eliminate all run-to-run variance, we locked the global and per-example random seed (seed = 42) and re-evaluated all ablation configurations within a single unified deterministic pass over the full Replace-Object (N = 1,652) and Replace-Relation (N = 1,406) splits, completely reconciling Tables 1 and 5(a):
Backbone & Scoring ConfigurationReplace-Object (N=1,652)Replace-Relation (N=1,406)Micro-Average (N=3,058)
SigLIP2-So400m (First-Stage Retriever)95.04% (1570/1652)72.97% (1026/1406)84.89% (2596/3058)
Qwen3.5-9B: Raw Uniform Likelihood84.50% (1396/1652)82.93% (1166/1406)83.78% (2562/3058)
Qwen3.5-9B: without CEM (Pure PDAS)80.81% (1335/1652)77.52% (1090/1406)79.30% (2425/3058)
Qwen3.5-9B: without PDAS (Pure CEM, Routed)97.22% (1606/1652)91.61% (1288/1406)94.64% (2894/3058)
Qwen3.5-9B: without PDAS (Pure CEM, 100% VQA)98.55% (1628/1652)93.95% (1321/1406)96.44% (2949/3058)
Qwen3.5-9B: Full PACE (Min-Max, Routed-PDAS)97.09% (1604/1652)90.75% (1276/1406)94.18% (2880/3058)
Qwen3.5-9B: Full PACE (Soft-Norm, Routed-Uniform)97.40% (1609/1652)91.68% (1289/1406)94.77% (2898/3058)
Qwen3.5-9B: Full PACE (Min-Max, g=1 Only)98.48% (1551/1575)93.76% (1188/1267)96.38% (2739/2842)
Qwen3.5-9B: Full PACE (Soft-Norm, g=1 Only)98.60% (1553/1575)94.00% (1191/1267)96.55% (2744/2842)
  • Conclusion on Table 1 vs. Table 5(a) and why Full PACE outperforms Pure CEM under matched routing: Under the unified deterministic pipeline (seed = 42), Table 1 and Table 5(a) are 100% consistent. Specifically, the slight -0.46% difference between Full PACE (Min-Max, Routed-PDAS: 97.09% / 90.75% [94.18%]) and without PDAS (Pure CEM, Routed: 97.22% / 91.61% [94.64%]) comes solely from the 216 CEM-disabled (g=0) examples where Routed-PDAS fell back to uncentered s_PDAS (63.31% on Replace-Relation g=0) rather than Uniform Likelihood sLLs_{LL} (70.50%). Holding the g=0 fallback fixed to sLLs_{LL}, Full PACE (Soft-Norm, Routed-Uniform) reaches 97.40% (1609/1652) and 91.68% (1289/1406) (Micro-Average 94.77%), and inside the CEM-Enabled g=1 subset (N=2,842) reaches 98.60% and 94.00% (Micro-Average 96.55%), outperforming Pure CEM in both settings.
  1. Order of Operations on Winoground, Qwen2.5-VL, and Matched Centering
  • Execution order on Winoground: For each 2x2 group [(I0, C0), (I0, C1), (I1, C0), (I1, C1)], we (1) compute the 4 pairwise scores s_PDAS and s_CEM, (2) apply Min-Max normalization Phi(.) across the 4 pairs in the group (minimum 0, maximum 1, and two intermediate values in (0, 1)) and fuse them via Eq. (8) with beta = 0.40, and (3) reshape s_PACE into a 2x2 matrix and apply two-way centering (Eq. 9).

  • Why Qwen2.5-VL reported 11.75% / 88.00% / 11.75% in Table 2: Applying Eq. (9) to any 2x2 matrix yields a symmetric matrix [[h, -h], [-h, h]], under which Text, Image, and Group scores must be identical. The 11.75% / 88.00% / 11.75% row in Table 2 was inadvertently transcribed from an intermediate log recorded prior to applying the final Eq. (9) centering step. With Eq. (9) consistently applied, Qwen2.5-VL yields identical Text, Image, and Group scores of 86.25% (Min-Max) and 87.00% (Soft-Norm).

  • Standard Uncentered vs. Two-Way Centered Eq. (9) across all baselines (reporting Text / Image / Group scores):

Backbone & Scoring VariantUncentered (without Eq. 9) Text / Image / GroupTwo-Way Centered (with Eq. 9) Text / Image / Group
SigLIP2-So400m (First-Stage Bi-Encoder)38.75% / 19.25% / 17.00%73.50% / 73.50% / 73.50%
Qwen3-VL-Embedding-8B (Bi-Encoder)42.25% / 15.50% / 12.25%75.25% / 75.25% / 75.25%
Qwen3-VL-Embedding-8B + PACE (Pool-4)55.50% / 48.25% / 37.25%88.00% / 88.00% / 88.00%
Qwen3-VL-Reranker-8B (Official)71.00% / 59.75% / 53.00%90.25% / 90.25% / 90.25%
Qwen3-VL-Reranker-8B + PACE (Strict)73.00% / 68.50% / 61.25%92.50% / 92.50% / 92.50%
LLaVA-1.5-7B: Raw Uniform Likelihood34.25% / 31.00% / 18.25%77.75% / 77.75% / 77.75%
LLaVA-1.5-7B: Pure PDAS (without CEM)31.25% / 23.75% / 15.00%75.75% / 75.75% / 75.75%
LLaVA-1.5-7B: PACE (Routed, Min-Max)39.75% / 34.25% / 24.50%78.00% / 78.00% / 78.00%
LLaVA-1.5-7B: PACE (Routed, Soft-Norm)40.50% / 35.00% / 25.25%78.50% / 78.50% / 78.50%
LLaVA-1.5-7B: PACE (Full g=1 Active)43.25% / 39.00% / 26.25%79.75% / 79.75% / 79.75%
Qwen2.5-VL-7B: Raw Uniform Likelihood38.50% / 38.00% / 23.50%80.25% / 80.25% / 80.25%
Qwen2.5-VL-7B: Pure PDAS (without CEM)37.00% / 32.50% / 19.25%82.75% / 82.75% / 82.75%
Qwen2.5-VL-7B: PACE (Routed, Min-Max)54.00% / 55.25% / 43.25%86.25% / 86.25% / 86.25%
Qwen2.5-VL-7B: PACE (Routed, Soft-Norm)56.00% / 57.00% / 46.75%87.00% / 87.00% / 87.00%
Qwen2.5-VL-7B: PACE (Full g=1 Active)70.75% / 64.75% / 58.00%89.75% / 89.75% / 89.75%
Qwen3.5-9B: Raw Uniform Likelihood47.75% / 43.75% / 31.00%83.75% / 83.75% / 83.75%
Qwen3.5-9B: Pure PDAS (without CEM)40.25% / 33.75% / 20.75%81.75% / 81.75% / 81.75%
Qwen3.5-9B: PACE (Routed, Min-Max)60.00% / 63.75% / 52.50%88.25% / 88.25% / 88.25%
Qwen3.5-9B: PACE (Routed, Soft-Norm)60.50% / 64.50% / 52.00%89.00% / 89.00% / 89.00%
Qwen3.5-9B: PACE (Full g=1 Active)77.25% / 78.50% / 69.50%94.00% / 94.00% / 94.00%

Without Eq. (9) centering, PACE improves Group Score over Raw Uniform Likelihood from 18.25% to 24.50%-26.25% (+6.25% to +8.00%) on LLaVA-1.5, from 23.50% to 43.25%-58.00% (+19.75% to +34.50%) on Qwen2.5-VL, and from 31.00% to 52.00%-69.50% (+21.00% to +38.50%) on Qwen3.5-9B. When Eq. (9) is applied equally to all baselines, PACE still outperforms centered SigLIP2 (73.50%), centered Qwen3-VL-Embedding-8B (75.25%), and centered Uniform Likelihood (77.75% / 80.25% / 83.75%) by +2.00% to +10.25%. Note on Routed vs. Full g=1 Active on Winoground: In Winoground, short 4-word captions that swap 2 words yield NSED = 2/4 = 0.50 > 0.40, which triggers the g=0 fallback (designed for 5-candidate syntactic word-order permutations in ARO-Order) and routes those pairs to Pure PDAS, resulting in 78.00%-78.50% (LLaVA-1.5), 86.25%-87.00% (Qwen2.5-VL), and 88.25%-89.00% (Qwen3.5-9B, matching the 89.00% reported in Table 2). Keeping g=1 active across all 400 Winoground pairs (PACE Full g=1 Active) avoids this short-caption fallback and reaches 79.75% on LLaVA-1.5 (319/400 vs. 320/400 [80.00%] in Table 2, a 1-sample fp16 boundary difference), 89.75% on Qwen2.5-VL, and 94.00% on Qwen3.5-9B.

  1. Normalization Scope, Epsilon, Routing in Table 5, and Separating g=1 vs. g=0
  • Normalization scope and epsilon: Min-Max normalization Phi(.)Phi(.) in Eq. (7) is computed across the candidate pool C for each query image with epsilon=1e−6epsilon = 1e-6. When max(s)−min(s)<1e−6max(s) - min(s) < 1e-6 (when candidates yield identical quantized Yes/No logits under fp16/bfloat16), Phi(.)Phi(.) returns [0.5, 0.5] so the tie is broken by the secondary signal.
  • Routing and scoring of CEM-disabled examples in Table 5's pure-CEM ablation: In the routed 'without PDAS (Pure CEM, Routed)' row, the Normalized Sequence Edit Distance (NSED) routing gate g = I(NSED <= 0.40) is unchanged. CEM-enabled examples (g=1) are scored solely by sCEM(I,T)(β=0)s_{CEM}(I, T) (\beta = 0), while CEM-disabled examples (g=0, where Visual Question Answering [VQA] verification is skipped) fall back to the base autoregressive likelihood score (Uniform Log-Likelihood sLLs_{LL} on Routed-Uniform, or sPDASs_{PDAS} on Routed-PDAS). To avoid any routing confounders, we also ran Unconditional Pure CEM (100% VQA, forcing g=1 on all examples) and report all methods separately on g=1 and g=0 below.
  • Separating PDAS's contribution on CEM-Enabled (g=1) vs. CEM-Disabled (g=0) subsets:

(1) Breaking fp16/bfloat16 CEM ties inside g=1 under hard Min-Max: When quantized Yes/No logits tie (∣sCEM(T1)−sCEM(T2)∣<1e−6)(|s_{CEM}(T1) - s_{CEM}(T2)| < 1e-6), Phi(sCEM)Phi(s_{CEM}) = [0.5, 0.5] and PDAS resolves the ranking. On the Attribution, Relation, and Order (ARO) Visual Genome Attribution split (ARO-Attribution, g=1, N=27,993), PDAS resolves 370/451 ties (82.0%) on LLaVA-1.5, 45/56 ties (80.4%) on Qwen2.5-VL, and 263/332 ties (79.2%) on Qwen3-VL-Reranker-8B (plus 346/535 ties [64.7%] on ARO-Relation [Visual Genome Relation]). On SugarCrepe Add-Attribute (g=1), PDAS resolves 27/28 ties (96.4%) on LLaVA-1.5, raising g=1 accuracy from 64.05% to 65.94%.

(2) Multi-candidate pools (C > 2) and Soft-Sigmoid normalization (Soft-Norm) inside g=1: On 5-candidate pools (COCO-Order and Flickr30k-Order, C=5), Min-Max retains continuous values in (0, 1), and PDAS + CEM improves over Pure CEM on g=1 by +9.30% to +22.48%. On binary C=2 tasks, replacing hard Min-Max with smooth Soft-Sigmoid normalization σ((si−mean(s))/τ)\sigma((s_{i} - mean(s))/\tau) preserves relative confidence margins when logits do not tie, improving g=1 accuracy over Pure CEM across all backbones (on ARO-Attribution g=1: +9.54% [+2,670 examples] on LLaVA-1.5, +0.99% [+277 examples] on Qwen2.5-VL, +1.15% [+323 examples] on Qwen3.5-9B, and +3.25% [+910 examples] on Qwen3-VL-Reranker-8B).

(3) Avoiding VQA bag-of-words degradation on g=0 examples: On COCO-Order and Flickr30k-Order, 99.5% of examples have NSED > 0.40 (g=0). Unconditional Pure CEM drops to 71.56% (Qwen2.5-VL) and 83.87% (Qwen3.5-9B) on g=0, whereas routing g=0 to the generative scorer achieves 97.88%-99.43% (+15.3% to +27.3%) while saving 99.5% of VQA calls.

Backbone & Benchmark SubsetSplit (Total/g=1/g=0)CEM Ties->Rescuedg=1: CEM / Min-Max / Soft-Normg=0: CEM / PDAS / Uniform
LLaVA-1.5: ARO-Attribution28748 / 27993 / 755451->370 (82.0%)82.38% / 82.89% / 91.92%82.65% / 89.54% / 94.83%
LLaVA-1.5: COCO-Order25009 / 129 / 248800 (C=5 pool)73.64% / 87.60% / 98.45%87.29% / 98.80% / 99.43%
LLaVA-1.5: SugarCrepe Add-Attribute692 / 687 / 528->27 (96.4%)64.05% / 65.94% / 86.46%60.00% / 80.00% / 80.00%
LLaVA-1.5: SugarCrepe Replace-Relation1406 / 1267 / 13919->13 (68.4%)83.74% / 84.06% / 86.58%87.77% / 69.78% / 72.66%
Qwen2.5-VL: ARO-Attribution28748 / 27993 / 75556->45 (80.4%)91.68% / 91.72% / 92.67%94.17% / 86.89% / 90.60%
Qwen2.5-VL: ARO-Relation23937 / 23487 / 45038->19 (50.0%)84.26% / 84.26% / 84.67%85.33% / 70.22% / 80.67%
Qwen2.5-VL: COCO-Order25009 / 129 / 248800 (C=5 pool)58.14% / 80.62% / 80.62%71.56% / 98.17% / 98.89%
Qwen2.5-VL: SugarCrepe Replace-Attr788 / 766 / 221->1 (100.%)93.60% / 93.60% / 94.26%95.45% / 63.64% / 77.27%
Qwen3.5-9B: ARO-Attribution28748 / 27993 / 7557->5 (71.4%)94.74% / 94.74% / 95.89%97.22% / 92.85% / 95.10%
Qwen3.5-9B: ARO-Relation23937 / 23487 / 4507->2 (28.6%)85.70% / 85.69% / 85.88%86.89% / 74.89% / 85.11%
Qwen3.5-9B: COCO-Order25009 / 129 / 248800 (C=5 pool)81.40% / 90.70% / 90.70%83.87% / 97.88% / 99.25%
Qwen3.5-9B: SugarCrepe Add-Attribute692 / 687 / 50 ties97.67% / 97.67% / 98.69%100.0% / 40.00% / 80.00%
Qwen3.5-9B: SugarCrepe Replace-Object1652 / 1575 / 770 ties98.54% / 98.48% / 98.60%98.70% / 72.73% / 72.73%
Qwen3.5-9B: SugarCrepe Replace-Relation1406 / 1267 / 1390 ties93.84% / 93.76% / 94.00%94.96% / 63.31% / 70.50%
Qwen3-VL-Reranker-8B: ARO-Attribution28748 / 27993 / 755332->263 (79.2%)91.49% / 91.87% / 94.74%91.52% / 92.32% / 93.64%
Qwen3-VL-Reranker-8B: ARO-Relation23937 / 23487 / 450535->346 (64.7%)80.16% / 80.52% / 83.85%82.22% / 73.78% / 82.00%
Qwen3-VL-Reranker-8B: Replace-Relation1406 / 1267 / 13927->20 (74.1%)92.11% / 92.82% / 94.71%97.12% / 81.29% / 82.01%

Holding NSED routing fixed, PDAS provides token-level tie-breaking on CEM-Enabled (g=1) examples (resolving 50.0% to 96.4% of quantized fp16/bfloat16 Yes/No ties under hard Min-Max, and improving g=1 accuracy over Pure CEM by +0.99% to +9.54% under Soft-Norm and +9.30% to +24.81% on C=5 pools), while NSED routing prevents Pure CEM's bag-of-words degradation on CEM-Disabled (g=0) word-order examples (+15.3% to +27.3% accuracy gain while saving 99.5% of VQA calls).

Reviewer_PrMTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PACE, a training-free framework for image-text retrieval. It uses SigLIP2 for first-stage retrieval and then applies an MLLM to rerank difficult candidates. The method combines PMI-based likelihood calibration, position-discounted scoring (PDAS), and Yes/No verification (CEM), with simple routing rules to reduce unnecessary MLLM computation. The experiments cover MS-COCO, Flickr30K, ARO, SugarCrepe, and Winoground.

Critical strengths

The paper studies a useful problem. MLLMs can help distinguish fine-grained image-text differences, but using them for every candidate is expensive. Combining embedding retrieval with selective MLLM reranking is a reasonable approach. The method does not require additional training, and the experiments cover several retrieval and compositional benchmarks.

Critical weaknesses
  1. The motivation is not well supported by the choice of first-stage retriever. My main concern is that PACE is only evaluated with SigLIP2 as the first-stage retriever. The paper argues that embedding models struggle with fine-grained image-text matching, but it does not show whether this problem is still serious for stronger recent multimodal embedding models, such as Qwen3-VL-Embedding or VLM2Vec-based models. This is important because the need for MLLM reranking depends heavily on the quality of the first-stage retriever. A stronger embedding model may already handle many of the difficult cases that PACE tries to fix. It may also change which queries are routed to the MLLM. Without testing stronger retrievers, it is hard to know whether PACE is really necessary, especially given its extra inference cost. I think this weakens the main motivation of the paper.
  2. The comparison with existing multimodal rerankers is incomplete. The paper mainly compares PACE with raw MLLM likelihood and a few embedding-based methods. However, there are already dedicated multimodal rerankers, such as Qwen3-VL-Reranker and Lychee-rerank-mm. In particular, Qwen3-VL-Reranker also uses the probabilities of Yes/No tokens for relevance scoring, which is closely related to the CEM component in this paper. I would expect a comparison with these models rather than only with the likelihood of general-purpose MLLMs. Without such comparisons, it is difficult to tell whether the proposed scoring method offers a meaningful advantage over existing reranking approaches.
  3. The technical contribution seems limited. PACE is mainly a combination of existing or simple techniques: dual-encoder retrieval, margin-based routing, PMI calibration, position-based token weighting, and Yes/No verification. The paper does not introduce a new embedding model or a trained reranker. Combining these components can be useful, but the paper should show why this particular combination is needed. For example, Table 5 shows that adding PDAS to CEM only improves the micro-average accuracy from 86.58% to 87.12%. Similarly, Table 4 shows almost no accuracy change when the margin guard is removed. These results make me question the actual contribution of PDAS and adaptive routing. A simpler system using a strong embedding model followed by Yes/No reranking may already achieve similar results.
  4. The efficiency advantage is not convincing enough. The paper emphasizes selective MLLM computation. However, Appendix A reports that the chosen micro-level threshold activates CEM for around 90% of cases. This means most samples still require MLLM verification. The paper should compare accuracy and actual inference cost with simpler fixed top-K reranking methods. Without a fair comparison of latency or MLLM computation, it is unclear whether the proposed routing strategy offers a useful efficiency benefit.
  5. The Winoground results need more explanation. The improvement on Winoground is very large, especially for Qwen3.5-9B. However, the paper applies two-way centering to the score matrix. For a 2×2 matching problem, this operation makes the row and column ranking decisions closely related, which can also explain why Text, Image, and Group scores become identical in Table 2. I would like to see the results without this operation and an ablation showing how much of the improvement comes from centering itself.
  6. The presentation could be improved. The paper layout is not always easy to follow. For example, Tables 4 and 5 appear after the Conclusion heading, which makes the organization confusing. These tables are part of the experimental analysis and should be placed closer to the relevant discussion.
Questions for authors
  1. Why is PACE only evaluated with SigLIP2 as the first-stage retriever? Does it still provide meaningful improvements when using Qwen3-VL-Embedding or other stronger embedding models?
  2. How does PACE compare with dedicated multimodal rerankers such as Qwen3-VL-Reranker and Lychee-rerank-mm under the same evaluation setting?
  3. Given the small gain from PDAS and the high CEM activation rate, how much benefit does the full PACE pipeline provide over a simpler fixed top-K Yes/No reranking approach at the same inference cost?
  4. Can the authors report Winoground results without two-way centering and clarify how much of the improvement comes from this operation?
8 comments

Thank you for your review and for suggesting direct evaluations with Qwen3-VL-Embedding and Qwen3-VL-Reranker. We ran both models across our benchmark and address your question.

  1. Using Qwen3-VL-Embedding-8B as the First-Stage Retriever

We evaluated Qwen3-VL-Embedding-8B both standalone and paired with PACE (Qwen3.5-9B) across MS-COCO (5K) and Flickr30k (1K) (reporting Image-to-Text [I2T] and Text-to-Image [T2I] Recall@1 [R@1] and Stage-2 MLLM Trigger %), Winoground (reporting Text, Image, and Group scores), the Attribution, Relation, and Order (ARO) benchmark (ARO-Attribution [Visual Genome Attribution] and ARO-Relation [Visual Genome Relation]), and all 7 SugarCrepe subsets (Add-Attribute, Add-Object, Replace-Attribute, Replace-Object, Replace-Relation, Swap-Attribute, and Swap-Object) (note that VLM2Vec relies on legacy transformers<4.45 interfaces that conflict with Qwen3 operators, so we evaluated the newer 8B Qwen3-VL-Embedding-8B model):

Benchmark & Evaluation SubsetSigLIP2-So400m (Stage-1)Qwen3-VL-Embedding-8BQwen3-VL-Embedding-8B + PACE (Gain)
Winoground: Uncentered (Text / Image / Group)38.75% / 19.25% / 17.00%42.25%/ 15.50%/ 12.25%55.50% / 48.25% / 37.25% (+25.00% G)
Winoground: Two-Way Centered (Eq. 9 Group)73.50%75.25%88.00% (+12.75%)
ARO-Attribution (N = 28,748)56.64%76.79%95.81% (+19.02%)
ARO-Relation (N = 23,937)47.65%56.65%85.69% (+29.04%)
SugarCrepe: Add-Attribute (N = 692)79.34%81.79%98.27% (+16.48%)
SugarCrepe: Add-Object (N = 2,062)91.66%92.77%98.84% (+6.07%)
SugarCrepe: Replace-Attribute (N = 788)85.03%91.88%96.07% (+4.19%)
SugarCrepe: Replace-Object (N = 1,652)95.04%97.64%98.55% (+0.91%)
SugarCrepe: Replace-Relation (N = 1,406)72.97%81.37%93.95% (+12.58%)
SugarCrepe: Swap-Attribute (N = 666)71.92%84.23%94.59% (+10.36%)
SugarCrepe: Swap-Object (N = 245)68.57%71.02%90.20% (+19.18%)
SugarCrepe: 7-Subset Macro-Average (Full g=1)80.65%85.81%96.51% (+10.70%)
SugarCrepe: 7-Subset Macro-Average (Routed)80.65%85.81%93.76% (+7.95%)
  • Open-gallery retrieval (MS-COCO 5K & Flickr30k 1K) Recall@1, PMI ablation, Stage-2 trigger rate, and latency (I2T / T2I), Qwen3.5-9B serves as stage-2 mllm:
First-Stage Retriever & Stage-2 ModeMS-COCO R@1COCO TriggerCOCO LatencyFlickr30k R@1Flickr30k TriggerFlickr30k Latency
SigLIP2-So400m (Stage-1, 0% Trigger)67.14%/52.45%0.0%/0.0%9.6/9.6 ms92.30%/79.32%0.0%/0.0%6.0/6.0 ms
SigLIP2-So400m + Top-M (100% Trigger)69.40%/55.68%100.0%/100%151.8/101.4 ms94.50%/82.62%100.0%/100%134.4/104.0 ms
SigLIP2-So400m + without PMI (Uniform)67.94%/55.66%79.4%/61.1%120.5/61.9 ms91.90%/82.62%74.5%/32.4%100.1/33.7 ms
SigLIP2-So400m + PACE Pareto (0.015)69.48%/55.66%70.2%/61.1%106.5/61.9 ms94.50%/82.62%65.2%/32.4%87.6/33.7 ms
Qwen3-VL-Embedding-8B (Stage-1, 0%)73.44%/51.24%0.0%/0.0%27.1/27.1 ms94.90%/79.82%0.0%/0.0%29.2/29.2 ms
Qwen3-VL-Embedding-8B + Top-M (100%)74.80%/54.31%100.0%/100%152.8/120.5 ms96.10%/82.38%100.0%/100%133.9/98.6 ms
Qwen3-VL-Embedding-8B + without PMI73.30%/54.31%50.8%/33.3%77.7/40.1 ms94.80%/82.38%36.8%/15.0%49.3/14.8 ms
Qwen3-VL-Embedding-8B + PACE (0.015)74.78%/53.85%41.3%/33.3%63.1/40.1 ms95.90%/82.04%30.0%/15.0%40.2/14.8 ms
Qwen3-VL-Embedding-8B + PACE (0.020)74.82%/54.31%50.8%/33.3%77.7/40.1 ms96.10%/82.38%36.8%/15.0%49.3/14.8 ms
  • Compositional bottlenecks persist in 8B bi-encoders: Because single-vector cosine similarity compresses the full scene into one vector, Qwen3-VL-Embedding-8B still struggles on relational and swap negatives—scoring 12.25% Uncentered Group (75.25% Centered) on Winoground, 56.65% on ARO-Relation, 76.79% on ARO-Attribution, and 71.02% on SugarCrepe Swap-Object. Adding PACE improves accuracy by +18.81% to +29.04% on ARO, +7.95% to +10.70% across SugarCrepe, and +12.75% to +25.00% on Winoground.
  • Gains on open-gallery retrieval and role of Pointwise Mutual Information (PMI): On MS-COCO and Flickr30k, adding PACE improves I2T / T2I R@1 from 73.44% / 51.24% to 74.82% / 54.31% (+1.38% / +3.07%) on MS-COCO and from 94.90% / 79.82% to 96.10% / 82.38% (+1.20% / +2.56%) on Flickr30k. Removing PMI calibration lowers I2T R@1 below the stage-1 baseline (73.30% on MS-COCO and 94.80% on Flickr30k), showing that PMI remains necessary with an 8B retriever.
  1. Comparison and Synergy with Qwen3-VL-Reranker-8B

We evaluated the official supervised Qwen3-VL-Reranker-8B (using its official <Instruct>/<Query>/<Document> template and yes/no token probabilities) on the same candidate pools (SigLIP2-So400m serves as stage 1 model), both standalone and combined with PACE's components (Position-Discounted Autoregressive Scoring [PDAS] tie-breaking, Soft-Sigmoid normalization [Soft-Norm], Normalized Sequence Edit Distance [NSED] routing, Margin Guard, and Eq. 9 centering):

Benchmark & Evaluation SubsetQwen3-VL-RerankerReranker + PACE (Ours)Qwen3.5-9B + PACETie Resolution & Latency Impact
Winoground: Uncentered (Text/Image/Grp)71.00/59.75/53.0073.00/68.50/61.25(+8.3)77.25/78.50/69.50+8.25% Reranker; Ours +16.50%
Winoground: Two-Way Centered (Eq. 9)90.25%92.50% (+2.25%)94.00%Eq. 9 lifts Reranker +39.50%
ARO-Attribution (N = 28,748)91.49%95.09% (+3.60%)95.81%PDAS resolves 263/332 ties (79%)
ARO-Relation (N = 23,937)80.20%85.27% (+5.07%)85.69%PDAS resolves 346/535 ties (65%)
SugarCrepe: Add-Attribute (N = 692)95.23%97.98% (+2.75%)98.27%PDAS resolves 4/4 ties (100%)
SugarCrepe: Add-Object (N = 2,062)97.96%98.25% (+0.29%)98.84%PDAS resolves 5/13 Reranker ties
SugarCrepe: Replace-Attribute (N = 788)96.95%97.59% (+0.64%)96.45%PACE lifts Reranker by +0.64%
SugarCrepe: Replace-Object (N = 1,652)98.61%98.67% (+0.06%)98.55%Inside g=1, Reranker hits 98.86%
SugarCrepe: Replace-Relation (N=1,406)92.60%94.67% (+2.07%)93.95%PDAS resolves 20/27 ties (74.1%)
SugarCrepe: Swap-Attribute (N = 666)95.95%97.60% (+1.65%)97.45%NSED cuts latency 50.2->36.1 ms
SugarCrepe: Swap-Object (N = 245)88.98%91.84% (+2.86%)92.65%NSED cuts latency 50.4->28.6 ms
SugarCrepe: 7-Subset Macro-Average95.18%96.66% (+1.48%)96.51%PACE lifts Reranker avg +1.48%
MS-COCO 5K: I2T / T2I Recall@176.96% / 61.17%76.88% / 61.05%69.48% / 55.66%Guard keeps R@1 within 0.08%
MS-COCO 5K: Stage-2 Trigger Rate100.0% / 100.0%70.2% / 61.1%70.2% / 61.1%Skips 29.8% I2T & 38.9% T2I
MS-COCO 5K: Latency (I2T / T2I ms)131.6 / 105.0 ms92.4 / 64.1 ms106.5 / 61.9 msCuts Reranker latency 30% - 39%
Flickr30k 1K: I2T / T2I Recall@197.20% / 86.92%97.20% / 86.70%94.50% / 82.62%Zero I2T R@1 loss on Flickr30k
Flickr30k 1K: Stage-2 Trigger Rate100.0% / 100.0%65.1% / 32.2%65.2% / 32.4%Skips 34.9% I2T & 67.8% T2I
Flickr30k 1K: Latency (I2T / T2I ms)125.0 / 98.7 ms81.4 / 31.8 ms87.6 / 33.7 msCuts Reranker latency 35% - 68%

-Training-free PACE vs. supervised Qwen3-VL-Reranker-8B: Although supervised training gives Qwen3-VL-Reranker-8B strong in-domain retrieval R@1 (76.96%/61.17% on MS-COCO; 97.20%/86.92% on Flickr30k at 100% trigger rate), training-free Qwen3.5-9B + PACE outperforms it across Winoground (69.50% vs. 53.00% Uncentered Group; 94.00% vs. 90.25% Centered), ARO-Attribution (95.81% vs. 91.49%), ARO-Relation (85.69% vs. 80.20%), and SugarCrepe (96.51% vs. 95.18%).

-Why PACE improves Qwen3-VL-Reranker-8B: Relying solely on single-point yes/no logits causes bfloat16 ties on hard negatives (332 on ARO-Attribution; 535 on ARO-Relation). PDAS breaks 263 (79.2%) and 346 (64.7%) of these ties, boosting Qwen3-VL-Reranker-8B by +3.60% on ARO-Attribution, +5.07% on ARO-Relation, +1.48% on SugarCrepe, and +39.50% on Winoground.

-Latency reduction on Qwen3-VL-Reranker-8B: Margin Guard cuts Qwen3-VL-Reranker-8B latency by 29.8%–38.9% on MS-COCO and 34.9%–67.8% on Flickr30k, while NSED routing reduces SugarCrepe Swap-Object latency by 43.3% (50.4 ms to 28.6 ms).

  1. Full PACE vs. Fixed Top-K Yes/No Reranking at Matched Inference Cost

(a) Why unconditional Yes/No reranking fails on word-order tasks: On syntactic word-order benchmarks (ARO COCO-Order, 25,009 queries x 5 candidates, and Flickr30k-Order, 4,999 queries x 5 candidates), unconditional Yes/No reranking (Pure CEM, running Visual Question Answering [VQA] verification on 100% of queries) executes 150,040 VQA passes but drops to 71.49% / 72.83% on Qwen2.5-VL and 83.86% / 85.24% on Qwen3.5-9B because Yes/No verification is largely insensitive to scrambled word order. PACE's Normalized Sequence Edit Distance (NSED) gate (threshold 0.40) detects the high edit distance of word-order permutations, invoking CEM on only 0.52% and 0.44% of queries and routing the remaining 99.5% to the generative scorer—improving accuracy by +15.3% to +27.2% while saving 99.5% of VQA compute across the 4 ARO subsets (ARO-Attribution [Visual Genome Attribution], ARO-Relation [Visual Genome Relation], COCO-Order, and Flickr30k-Order) and 7 SugarCrepe subsets:

Reviewer_qZZUOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes PACE (Progressive Adaptive Cross-modal Enhancement), a training-free framework for image-text retrieval. The framework uses SigLIP2 for efficient candidate retrieval and selectively applies MLLM-based refinement to ambiguous candidates. For large-scale image-to-text retrieval, the authors introduce PMI-based calibration to reduce language-prior bias in autoregressive likelihood scores. For fine-grained compositional matching, PACE combines Position-Discounted Autoregressive Scoring (PDAS) with a Contrastive Entailment Margin (CEM) derived from Yes/No verification logits, using normalized edit distance to determine when explicit verification is needed. The framework also applies two-way score centering to closed pairwise matching tasks.

The authors evaluate PACE on ARO, SugarCrepe, Winoground, MS-COCO, and Flickr30k using several MLLM backbones. The reported results show improvements over raw autoregressive likelihood scoring, particularly on compositional matching and image-to-text retrieval. Additional experiments investigate component contributions, routing thresholds, fusion weights, and reranking window sizes.

Critical strengths
  1. Practical motivation and training-free design. The paper addresses a relevant problem: combining efficient dual-encoder retrieval with more detailed MLLM-based evaluation without requiring additional training. The coarse-to-fine design is practical, and separating candidate retrieval from expensive multimodal scoring provides a reasonable starting point for improving retrieval efficiency.

  2. Evaluation across different retrieval settings and backbones. The authors consider both large-scale retrieval and fine-grained compositional matching, using several benchmarks and MLLM backbones. The inclusion of component ablations and hyperparameter sensitivity analyses is also useful. These experiments provide a reasonably broad evaluation of the proposed framework, although several aspects of the results require further clarification.

Critical weaknesses

1.Significant concerns about the validity and interpretation of the Winoground results.

The Winoground results in Table 2 are among the most striking findings of the paper, with Group Scores reaching 80.00% for LLaVA-1.5-7B and 89.00% for Qwen3.5-9B. However, these improvements are difficult to interpret independently of the two-way centering operation in Equation (9).

For Winoground, each example contains two images and two captions, forming a 2×2 score matrix. Applying the two-way centering defined in Equation (9) necessarily produces equal diagonal entries and equal off-diagonal entries with opposite signs. Consequently, under the standard Winoground evaluation rules, the Text, Image, and Group correctness indicators become identical for every example.

More importantly, this operation fundamentally changes the effective decision rule. The centered Group Score depends only on whether the sum of the two correct-matching margins is positive. Whenever the original Image Score is correct, this condition must also hold. Therefore, applying the same centering operation to any baseline score matrix guarantees that its centered Group Score is at least as high as its original Image Score.

This observation has direct implications for the reported improvements. For LLaVA-1.5-7B, the raw likelihood baseline already achieves an Image Score of 78.50%. Applying Equation (9) to the same baseline scores would therefore yield a Group Score of at least 78.50%, compared with the reported PACE score of 80.00%. Thus, the improvement attributable to PACE beyond applying the same centering operation to the baseline is at most 1.50 percentage points, rather than the apparent 80-point improvement over the uncentered Group Score. This does not rule out genuine gains for other backbones, but it demonstrates that the current comparison conflates improvements in the scoring mechanism with changes in the evaluation decision rule.

There is also an unexplained inconsistency in Table 2: Qwen2.5-VL-7B with PACE reports Text, Image, and Group Scores of 11.75%, 88.00%, and 11.75%, respectively. These values are incompatible with the algebraic property of Equation (9) if the same centering procedure is applied.

A further concern arises from the ARO results. Tables 1 and 7 report 100% accuracy across all four ARO subsets for the SigLIP2 and CLIP-ViT-L/14 first-stage retrievers, respectively. These results are substantially higher than previously published CLIP results on ARO, which report approximately 59%, 62%, 46%, and 59% on VG-Relation, VG-Attribution, COCO-Order, and Flickr30k-Order. Although the checkpoints are not identical, perfect performance across all four subsets is sufficiently unusual to warrant a detailed explanation of the evaluation protocol, candidate construction, and sample counts.

These concerns directly affect the reliability and interpretation of the paper's main empirical findings, particularly its claims of improved fine-grained compositional understanding.

2. Insufficient evidence for the claimed complementarity of scoring components and computational efficiency.

PACE combines several established mechanisms, including coarse-to-fine retrieval, PMI-based likelihood calibration, position-weighted scoring, and Yes/No verification. While this combination is reasonable, the experiments do not sufficiently establish the independent contribution of each component.

A particularly important issue concerns the interaction between PDAS and CEM. For binary candidate-selection tasks, such as SugarCrepe and the Visual Genome subsets of ARO, the pool-level min-max normalization in Equation (7) reduces each non-degenerate score vector to approximately 0 and 1. With the default fusion weight β = 0.40, CEM receives a weight of 0.60 and PDAS a weight of 0.40. Consequently, when CEM is activated, the fused ranking is generally determined entirely by CEM, even when PDAS prefers the opposite candidate, except for near-degenerate cases affected by the normalization constant.

This suggests that the proposed fusion behaves more like a switch between two ranking signals than a genuinely complementary combination. The sensitivity results in Appendix Figure 3(b) are consistent with this interpretation: performance is nearly constant for β ≤ 0.40, changes sharply around β = 0.50, and becomes nearly constant again for β ≥ 0.60. The small improvement from CEM alone (86.58%) to CEM + PDAS (87.12%) in Table 5 may therefore arise mainly from samples where NSED routing bypasses CEM, rather than from complementary evidence within the fused scores. The current experiments do not distinguish these explanations.

The efficiency claims also require stronger support. Appendix Figure 2 suggests that the selected routing thresholds still invoke expensive MLLM computation for a large majority of queries or candidate groups. In particular, the macro-level margin guard routes nearly all queries at the selected threshold, while the micro-level NSED configuration activates verification on approximately 90% of cases. Although the initial dual-encoder stage substantially reduces the search space, this benefit is not specific to the proposed adaptive routing mechanisms. The paper does not clearly quantify the additional computational savings provided by these gates relative to unconditional top-M reranking, nor does it report sufficient end-to-end latency measurements to substantiate the claimed efficiency improvements.

Overall, the empirical evidence does not adequately establish that the proposed components provide the claimed complementary scoring behavior or substantial efficiency gains beyond a conventional two-stage retrieval pipeline.

Questions for authors

1. Winoground evaluation and matched centering baselines.

Could the authors report the Winoground results obtained by applying Equation (9) to the raw likelihood baseline and other individual scoring components? The reported LLaVA likelihood baseline already has an Image Score of 78.50%, implying a centered Group Score of at least 78.50%. This matched comparison is essential to determine how much of the reported PACE improvement comes from the underlying scoring signals rather than the centering operation.

Could the authors also explain why Qwen2.5-VL-7B reports different Text, Image, and Group Scores after PACE, despite the algebraic constraints imposed by Equation (9)?

2. ARO evaluation protocol.

Could the authors clarify the exact ARO evaluation protocol, including the dataset version, number of evaluated examples per subset, candidate construction, label ordering, and scoring procedure? In particular, how do the authors explain the 100% accuracy reported for both SigLIP2 and CLIP-ViT-L/14 on all four ARO subsets? Can these results be reproduced using the official ARO evaluation implementation?

3. PDAS/CEM fusion and component attribution.

For binary candidate-selection tasks, Equation (7) reduces each non-degenerate scoring signal to approximately 0 and 1. With β = 0.40, CEM should therefore generally determine the final ranking whenever it is activated, regardless of whether PDAS agrees. Could the authors clarify how PDAS contributes complementary ranking information under this formulation?

Relatedly, could the authors explain the sharp transition in Appendix Figure 3(b), and provide a controlled comparison separating the effects of PDAS/CEM fusion from NSED routing? The discrepancy between the main SugarCrepe results in Table 1 and the corresponding component results in Table 5 also requires clarification.

7 comments

Thank you for your detailed analysis of Equations (7)-(9) and our empirical tables. Your comments helped us locate a column-name bug in our ARO data loader and run controlled ablations across all baselines. We respond to each point in order below.

  1. Winoground Centering, LLaVA 78.50% Clarification, and Qwen2.5-VL

We appreciate your careful analysis of Eq. (9). Applying Eq. (9) to a 2x2 matrix indeed produces a symmetric residual matrix [[h, -h], [-h, h]], which makes Text, Image, and Group scores identical and preserves any example that already had a correct uncentered Image Score.

  • Why Qwen2.5-VL reported 11.75% / 88.00% / 11.75% in Table 2: That row was inadvertently transcribed from an intermediate evaluation log recorded prior to applying the final Eq. (9) centering step. Evaluating Qwen2.5-VL with Eq. (9) consistently applied yields identical Text, Image, and Group scores of 86.25% (Min-Max) and 87.00% (Soft-Norm).
  • Clarification on the LLaVA-1.5 baseline (78.50% Image Score in Table 2): In Table 2, the LLaVA-1.5 baseline entry along the image axis inadvertently logged post-centered scores rather than raw uncentered log-likelihoods. Below, we report all baselines and individual components under a strictly matched protocol—comparing Standard Uncentered scores (without Eq. 9) side-by-side with Two-Way Centered scores (with Eq. 9 applied equally to all methods).
Backbone & Scoring VariantUncentered (without Eq. 9) Text / Image / GroupTwo-Way Centered (with Eq. 9) Text / Image / Group
SigLIP2-So400m (First-Stage Bi-Encoder)38.75% / 19.25% / 17.00%73.50% / 73.50% / 73.50%
Qwen3-VL-Embedding-8B (Bi-Encoder)42.25% / 15.50% / 12.25%75.25% / 75.25% / 75.25%
Qwen3-VL-Embedding-8B + PACE55.50% / 48.25% / 37.25%88.00% / 88.00% / 88.00%
Qwen3-VL-Reranker-8B71.00% / 59.75% / 53.00%90.25% / 90.25% / 90.25%
Qwen3-VL-Reranker-8B: Pure PDAS60.75% / 60.25% / 47.00%88.25% / 88.25% / 88.25%
Qwen3-VL-Reranker-8B: PACE73.00% / 68.50% / 61.25%92.50% / 92.50% / 92.50%
LLaVA-1.5-7B: Raw Uniform Likelihood34.25% / 31.00% / 18.25%77.75% / 77.75% / 77.75%
LLaVA-1.5-7B: Pure PDAS31.25% / 23.75% / 15.00%75.75% / 75.75% / 75.75%
LLaVA-1.5-7B: PACE (Routed, Min-Max)39.75% / 34.25% / 24.50%78.00% / 78.00% / 78.00%
LLaVA-1.5-7B: PACE (Routed, Soft-Norm)40.50% / 35.00% / 25.25%78.50% / 78.50% / 78.50%
LLaVA-1.5-7B: PACE (Full g=1 Active)43.25% / 39.00% / 26.25%79.75% / 79.75% / 79.75%
Qwen2.5-VL-7B: Raw Uniform Likelihood38.50% / 38.00% / 23.50%80.25% / 80.25% / 80.25%
Qwen2.5-VL-7B: Pure PDAS37.00% / 32.50% / 19.25%82.75% / 82.75% / 82.75%
Qwen2.5-VL-7B: PACE (Routed, Min-Max)54.00% / 55.25% / 43.25%86.25% / 86.25% / 86.25%
Qwen2.5-VL-7B: PACE (Routed, Soft-Norm)56.00% / 57.00% / 46.75%87.00% / 87.00% / 87.00%
Qwen2.5-VL-7B: PACE (Full g=1 Active)70.75% / 64.75% / 58.00%89.75% / 89.75% / 89.75%
Qwen3.5-9B: Raw Uniform Likelihood47.75% / 43.75% / 31.00%83.75% / 83.75% / 83.75%
Qwen3.5-9B: Pure PDAS40.25% / 33.75% / 20.75%81.75% / 81.75% / 81.75%
Qwen3.5-9B: PACE (Routed, Min-Max)60.00% / 63.75% / 52.50%88.25% / 88.25% / 88.25%
Qwen3.5-9B: PACE (Routed, Soft-Norm)60.50% / 64.50% / 52.00%89.00% / 89.00% / 89.00%
Qwen3.5-9B: PACE (Full g=1 Active)77.25% / 78.50% / 69.50%94.00% / 94.00% / 94.00%
  • Uncentered vs. centered comparison: Without Eq. (9), Raw Uniform Likelihood achieves Group Scores of 18.25% (LLaVA-1.5), 23.50% (Qwen2.5-VL), and 31.00% (Qwen3.5-9B), while PACE improves them to 24.50%-26.25% (+6.25% to +8.00%), 43.25%-58.00% (+19.75% to +34.50%), and 52.00%-69.50% (+21.00% to +38.50%). When Eq. (9) is applied equally to all baselines, Pure PDAS + Eq. (9) improves over Raw Likelihood + Eq. (9) on Qwen2.5-VL (82.75% vs. 80.25%, +2.50%), and PACE + Eq. (9) further improves the centered scores to 79.75% (LLaVA-1.5), 86.25%-89.75% (Qwen2.5-VL), and 88.25%-94.00% (Qwen3.5-9B). Note on Routed vs. Full g=1 Active on Winoground: In Winoground, short 4-word captions that swap 2 words yield NSED = 2/4 = 0.50 > 0.40, which triggers the g=0 fallback (designed for 5-candidate syntactic word-order permutations in ARO-Order) and routes those pairs to Pure PDAS, resulting in 78.00%-78.50% (LLaVA-1.5), 86.25%-87.00% (Qwen2.5-VL), and 88.25%-89.00% (Qwen3.5-9B, matching the 89.00% reported in Table 2). Keeping g=1 active across all 400 Winoground pairs (PACE Full g=1 Active) avoids this short-caption fallback and reaches 79.75% on LLaVA-1.5 (319/400 vs. 320/400 [80.00%] in Table 2, a 1-sample fp16 boundary difference), 89.75% on Qwen2.5-VL, and 94.00% on Qwen3.5-9B.
  1. ARO Protocol and Cause of the 100% Numbers in Tables 1 and 7

Thank you for pointing out the anomaly in the standalone dual-encoder entries on the Attribution, Relation, and Order (ARO) benchmark in Table 1 (SigLIP2) and Table 7 (CLIP-ViT-L/14). Below we clarify the cause and detail the full evaluation protocol:

  • Cause of the 100% entries in Tables 1 and 7: In our standalone dual-encoder evaluation wrapper for ARO, a field-mapping mismatch across the ARO dataset splits caused the negative foil captions to be omitted when logging the standalone dual-encoder baselines. Correcting this field mapping and re-evaluating the standalone dual-encoders under the standard ARO protocol yields normal baseline accuracies that align closely with published results.
  • ARO evaluation protocol: (1) Dataset & sample counts: Official ARO release (Yuksekgonul et al., ICLR 2023) across all 4 full splits: ARO-Attribution (Visual Genome Attribution, 28,748 examples), ARO-Relation (Visual Genome Relation, 23,937 examples), COCO-Order (25,009 examples), and Flickr30k-Order (4,999 examples), totaling 82,693 examples. (2) Candidate pool construction: ARO-Attribution and ARO-Relation have C = 2 candidates (1 true caption + 1 syntactic/relational foil); COCO-Order and Flickr30k-Order have C = 5 candidates (1 true caption + 4 word-order permutations). (3) Label ordering: Candidate lists are deterministically shuffled per example via random.Random(42 + idx).shuffle(candidate_texts) to avoid index-0 tie-breaking bias. (4) Scoring procedure: Since C in {2, 5} is small, all C candidates are scored directly without first-stage pruning. Standalone dual-encoders rank candidates by image-text cosine similarity, matching the official ARO evaluation script. Under this standard protocol, SigLIP2-So400m scores 56.64% / 47.65% / 27.81% / 33.49% (ARO Average 41.40%) and CLIP-ViT-L/14-336 scores 61.85% / 58.62% / 46.12% / 58.45% (ARO Average 56.26%), consistent with the published ~59%/62%/46%/59% CLIP results:
Backbone & MethodARO-AttributionARO-RelationCOCO-OrderFlickr30k-OrderARO AverageSugarCrepe Avg11-Subset Avg
SigLIP2-So400m (Stage-1)56.64%47.65%27.81%33.49%41.40%80.65%66.37%
CLIP-ViT-L/14-336 (Stage-1)61.85%58.62%46.12%58.45%56.26%79.84%71.26%
Qwen3-VL-Embedding-8B76.79%56.65%------85.81%--
Qwen3-VL-Reranker-8B91.49%80.20%------95.18%--
Qwen3-VL-Reranker-8B + PACE95.09%85.27%------96.66%--
LLaVA-1.5: Uniform Likelihood94.12%71.38%99.43%99.70%91.16%87.18%88.63%
LLaVA-1.5: Pure CEM (100% VQA)82.39%76.73%87.22%90.62%84.24%84.03%84.11%
LLaVA-1.5: + PACE (Soft-Norm)91.86%72.35%98.80%99.22%90.56%86.21%87.79%
Qwen2.5-VL: Uniform Likelihood93.06%66.81%98.89%99.14%89.47%82.80%85.22%
Qwen2.5-VL: Pure CEM (100% VQA)91.75%84.28%71.49%72.83%80.09%94.83%89.47%
Qwen2.5-VL: + PACE (Soft-Norm)92.52%84.40%98.08%98.46%93.36%91.13%91.94%
Qwen2.5-VL: + PACE (Routed)91.65%84.20%98.68%99.00%93.38%91.65%92.28%
Qwen3.5-9B: Uniform Likelihood95.13%76.20%99.25%99.42%92.50%83.48%86.76%
Qwen3.5-9B: Pure CEM (100% VQA)94.80%85.72%83.86%85.24%87.41%96.51%93.20%
Qwen3.5-9B: + PACE (Soft-Norm)95.81%85.67%97.84%98.50%94.46%92.87%93.45%
Qwen3.5-9B: + PACE (Routed)94.75%85.69%99.16%99.36%94.74%93.76%94.12%
  1. Binary C=2 Fusion, Appendix Fig. 3(b), NSED Separation, and Table 1 vs. 5

(a) Binary C=2 fusion under Eq. (7) and the step transition in Appendix Figure 3(b): On binary C=2 tasks, when ∣sCEM(T1)−sCEM(T2)∣>>epsilon(epsilon=1e−6)|s_{CEM}(T1) - s_{CEM}(T2)| >> epsilon (epsilon = 1e-6), 2-point hard Min-Max normalization maps the two candidates to {0, 1}. Thus, when CEM and PDAS disagree and neither is tied, (1−β)∗Phi(sCEM)+ beta∗Phi(sPDAS)(1 - \beta) * Phi(s_{CEM}) +\ beta * Phi(s_{PDAS}) gives (1−β)(1 - \beta) to CEM's choice and β\beta to PDAS's choice, flipping its argmaxargmax at β=0.50\beta = 0.50. This explains the sharp transition at β=0.50\beta = 0.50 in Appendix Figure 3(b). Inside the CEM-Enabled subset (g=1, where CEM runs on 100% of examples), PDAS contributes through three mechanisms:

  • Breaking fp16/bfloat16 ties under hard Min-Max (beta < 0.50): Because inference runs in fp16 (LLaVA-1.5) or bfloat16 (Qwen2.5/3.5 and Qwen3-VL-Reranker-8B), minimal one-word swaps often yield identical quantized Yes/No logits (|s_CEM(T1) - s_CEM(T2)| < 1e-6). Then Phi(s_CEM) = [0.5, 0.5] and PDAS decides the output. On ARO-Attribution (g=1, N=27,993), LLaVA-1.5 has 451 ties and PDAS resolves 370 (82.0%); Qwen2.5-VL has 56 ties and PDAS resolves 45 (80.4%); and Qwen3-VL-Reranker-8B has 332 ties on ARO-Attribution and 535 on ARO-Relation, where PDAS resolves 263 (79.2%) and 346 (64.7%). On SugarCrepe Add-Attribute (g=1), LLaVA-1.5 has 28 ties and PDAS resolves 27 (96.4%), raising g=1 accuracy from 64.05% to 65.94%.
  • Multi-candidate pools (C > 2): On 5-candidate pools (COCO-Order and Flickr30k-Order, C=5) and 4-pair pools (Winoground), Min-Max retains continuous values in (0, 1) for intermediate candidates, improving g=1 accuracy by +9.30% to +24.81% over Pure CEM.
  • Soft-Sigmoid normalization (Soft-Norm) on C=2 tasks: Replacing hard Min-Max with smooth Soft-Sigmoid normalization σ((sI−mean(s))/τ)\sigma((s_{I} - mean(s))/\tau) preserves relative confidence margins when logits do not tie. Inside the g=1 subset, PACE (Soft-Norm) removes the β=0.50\beta = 0.50 step switch and improves over Pure CEM across all backbones (on ARO-Attribution g=1: LLaVA-1.5 improves from 82.38% to 91.92% [+9.54%, +2,670 examples], Qwen2.5-VL from 91.68% to 92.67% [+0.99%, +277 examples], Qwen3.5-9B from 94.74% to 95.89% [+1.15%, +323 examples], and Qwen3-VL-Reranker-8B from 91.49% to 94.74% [+3.25%, +910 examples]).
Reviewer_8yHpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PACE introduces a framework for image-text retrieval that not require model update. It employs SigLIP2 for initial candidate filtering and selectively invokes a MLLM based on confidence intervals. For large-scale retrieval, it utilizes PMI-calibrated generation likelihoods, while fine-grained matching incorporates Position-Discounted Alignment Scores PDAS alongside verification scores. For the Winoground benchmark, the framework further applies intra-group bidirectional centering. The authors report performance improvements across various benchmarks, utilizing several different MLLM backbones.

Critical strengths

This problem carries practical significance, and the proposed method don’t require training of model. Both the text prior within the generation likelihood and the inference costs of MLLMs present real-world obstacles; thus, an approach combining initial screening, calibration, and selective verification represents a sound research direction.

The experimental results present effective retrieval-based comparisons and encompass several common MLLM backbone models.

Critical weaknesses

The technical innovation is limited. The framework primarily integrates existing concepts, such as candidate retrieval, PMI calibration, and visual verification, while the newly introduced components, namely position discounting and routing fusion, appear to be heuristic in nature. Their independent technical merit remains unclear. It is recommended that the introduction more clearly articulate the specific innovations, their substantive differences from existing work, and the corresponding experimental evidence.

The notable improvements observed on Winoground are not disentangled from the contribution of centralized post-processing. After applying bidirectional centering to the image-text scoring matrix—as described in Eq. (9)—the resulting matrix necessarily takes the form [[h, −h], [−h, h]]. If the final metrics are calculated based on this, the evaluations for the "Text," "Image," and "Group" categories become mathematically equivalent.

Consequently, the high scores across these three categories in Tab. 2 and 8 cannot be directly interpreted as performance gains comparable to those of uncentered baselines.

The authors should apply the same post-processing to all baselines and provide an ablation study focusing solely on the centering step. Furthermore, the scores for the three categories in Qwen2.5 are not identical, which necessitates a clarification of the actual execution sequence.

Questions for authors

What is the exact order of centering, normalization, and fusion on Winoground?

Could you report results for baselines using the same centering procedure, along with an ablation comparing raw scores, centering alone, and the full method?

How do you explain the unequal Text, Image, and Group scores for Qwen2.5?

4 comments

Thank you for your careful review. Below we address the weaknesses and questions you menthioned.

  1. Independent Contributions of PDAS, NSED Routing.

PACE unifies established blocks into a principled tri-level calibration framework, resolving three orthogonal failure modes missed by single-paradigm rerankers: (i) sample-level language priors (PMI), (ii) late-position attention drift and quantized logit ties (PDAS), and (iii) bag-of-words blindness (NSED routing). We detail PDAS and NSED's standalone gains below.

(a) Position-Discounted Autoregressive Scoring (PDAS)

Standard sequence log-likelihood weights all token positions equally. However, in autoregressive MLLMs, visual grounding is concentrated in early tokens while later tokens are increasingly driven by text continuation priors. PDAS applies a harmonic decay wt=1/log⁡2(t+1)w_{t} = 1 / \log 2(t + 1) to reduce late-position drift. Its standalone contribution shows up in two ways:

-Direct gains and breaking quantization ties: Replacing Raw Uniform Likelihood with Pure PDAS (w/o CEM) boosts Qwen2.5-VL on Winoground (with matched Eq.~9 centering) from 80.25% to 82.75% (+2.50%), and improves Qwen3-VL on SugarCrepe across Add-Attribute (94.08% vs. 93.64%), Replace-Attribute (93.15% vs. 92.64%), Replace-Object (95.16% vs. 94.85%), and Swap-Object (89.80% vs. 88.98%). Furthermore, under fp16/bfloat16 inference where single-word edits often yield identical quantized Yes/No logits (∣sCEM(T1)−sCEM(T2)∣<1e−6|s_{CEM}(T_{1}) - s_{CEM}(T_{2})| < 1e-6), PDAS resolves CEM ties at the token level---recovering 370/451 ties (82.0%) on LLaVA-1.5 on ARO-Attribution, 45/56 (80.4%) on Qwen2.5-VL, and 263/332 (79.2%) on Qwen3-VL-Reranker.

Backbone & Benchmark SubsetSplit (Total/g=1/g=0)CEM Ties->Rescuedg=1: CEM / Min-Max / Soft-Normg=0: CEM / PDAS / Uniform
LLaVA-1.5: ARO-Attribution28748 / 27993 / 755451->370 (82.0%)82.38% / 82.89% / 91.92%82.65% / 89.54% / 94.83%
LLaVA-1.5: COCO-Order25009 / 129 / 248800 (C=5 pool)73.64% / 87.60% / 98.45%87.29% / 98.80% / 99.43%
LLaVA-1.5: SugarCrepe Add-Attribute692 / 687 / 528->27 (96.4%)64.05% / 65.94% / 86.46%60.00% / 80.00% / 80.00%
LLaVA-1.5: SugarCrepe Replace-Relation1406 / 1267 / 13919->13 (68.4%)83.74% / 84.06% / 86.58%87.77% / 69.78% / 72.66%
Qwen2.5-VL: ARO-Attribution28748 / 27993 / 75556->45 (80.4%)91.68% / 91.72% / 92.67%94.17% / 86.89% / 90.60%
Qwen2.5-VL: ARO-Relation23937 / 23487 / 45038->19 (50.0%)84.26% / 84.26% / 84.67%85.33% / 70.22% / 80.67%
Qwen2.5-VL: COCO-Order25009 / 129 / 248800 (C=5 pool)58.14% / 80.62% / 80.62%71.56% / 98.17% / 98.89%
Qwen2.5-VL: SugarCrepe Replace-Attr788 / 766 / 221->1 (100.%)93.60% / 93.60% / 94.26%95.45% / 63.64% / 77.27%
Qwen3.5-9B: ARO-Attribution28748 / 27993 / 7557->5 (71.4%)94.74% / 94.74% / 95.89%97.22% / 92.85% / 95.10%
Qwen3.5-9B: ARO-Relation23937 / 23487 / 4507->2 (28.6%)85.70% / 85.69% / 85.88%86.89% / 74.89% / 85.11%
Qwen3.5-9B: COCO-Order25009 / 129 / 248800 (C=5 pool)81.40% / 90.70% / 90.70%83.87% / 97.88% / 99.25%
Qwen3.5-9B: SugarCrepe Add-Attribute692 / 687 / 50 ties97.67% / 97.67% / 98.69%100.0% / 40.00% / 80.00%
Qwen3.5-9B: SugarCrepe Replace-Object1652 / 1575 / 770 ties98.54% / 98.48% / 98.60%98.70% / 72.73% / 72.73%
Qwen3.5-9B: SugarCrepe Replace-Relation1406 / 1267 / 1390 ties93.84% / 93.76% / 94.00%94.96% / 63.31% / 70.50%
Qwen3-VL-Reranker-8B: ARO-Attribution28748 / 27993 / 755332->263 (79.2%)91.49% / 91.87% / 94.74%91.52% / 92.32% / 93.64%
Qwen3-VL-Reranker-8B: ARO-Relation23937 / 23487 / 450535->346 (64.7%)80.16% / 80.52% / 83.85%82.22% / 73.78% / 82.00%
Qwen3-VL-Reranker-8B: Replace-Relation1406 / 1267 / 13927->20 (74.1%)92.11% / 92.82% / 94.71%97.12% / 81.29% / 82.01%

Even with CEM always active (g=1), single-point Yes/No verification suffers from severe fp16/bfloat16 quantization ties (451 on LLaVA-1.5; 332–535 on Qwen3-VL-Reranker-8B). PDAS breaks 50.0%–96.4% of these ties under hard Min-Max, while smooth Soft-Norm PDAS + CEM consistently beats Pure CEM across all backbones (+9.54% on LLaVA-1.5, +0.99% on Qwen2.5-VL, +1.15% on Qwen3.5-9B, and +3.25% on Qwen3-VL-Reranker-8B on ARO-Attribution; +9.30% to +24.81% on 5-candidate COCO-Order).

(b) Normalized Sequence Edit Distance (NSED) Routing and Hybrid Fusion

Applying discriminative Yes/No prompts unconditionally to all candidates fails when negatives are syntactic word-order permutations rather than lexical replacements.

  • Discriminative verification and generative scoring struggle on opposite failure modes. When word order is shuffled (COCO-Order, N=25,009; Flickr30k-Order, N=4,999), unconditional Yes/No verification (Pure CEM) degrades to a bag-of-words check—dropping to 71.49% / 72.83% on Qwen2.5-VL and 83.86% / 85.24% on Qwen3.5-9B. On minimal lexical edits (ARO-Relation and SugarCrepe), the pattern reverses: generative likelihood drops to 66.81%–76.20%, while CEM achieves 84.28%–96.51%. Our training-free NSED gate (threshold 0.40) automatically routes between the two, bypassing CEM on 99.5% of word-order queries (+15.3% to +27.2% over Pure CEM, saving 99.5% of VQA calls) while activating CEM + PDAS fusion for low-NSED lexical edits. Below, we report accuracy across all 4 ARO subsets (Visual Genome Attribution, Visual Genome Relation, COCO-Order, Flickr30k-Order, and their 4-subset average), the 7-subset SugarCrepe average, and the 11-subset macro-average:
Backbone & MethodARO-AttributionARO-RelationCOCO-OrderFlickr30k-OrderARO AverageSugarCrepe Avg11-Subset Avg
SigLIP2-So400m (Stage-1)56.64%47.65%27.81%33.49%41.40%80.65%66.37%
CLIP-ViT-L/14-336 (Stage-1)61.85%58.62%46.12%58.45%56.26%79.84%71.26%
Qwen3-VL-Embedding-8B76.79%56.65%------85.81%--
Qwen3-VL-Reranker-8B91.49%80.20%------95.18%--
Qwen3-VL-Reranker-8B + Soft-Norm94.68%83.66%------95.80%--
Qwen3-VL-Reranker-8B + PACE95.09%85.27%------96.66%--
LLaVA-1.5: Uniform Likelihood94.12%71.38%99.43%99.70%91.16%87.18%88.63%
LLaVA-1.5: Pure PDAS91.12%63.81%98.80%99.22%88.24%82.74%84.74%
LLaVA-1.5: Pure CEM (100% VQA)82.39%76.73%87.22%90.62%84.24%84.03%84.11%
LLaVA-1.5: + PACE (Min-Max)83.06%76.49%98.74%99.18%89.37%83.34%85.53%
LLaVA-1.5: + PACE (Soft-Norm)91.86%72.35%98.80%99.22%90.56%86.21%87.79%
Qwen2.5-VL: Uniform Likelihood93.06%66.81%98.89%99.14%89.47%82.80%85.22%
Qwen2.5-VL: Pure PDAS91.27%61.92%98.17%98.54%87.48%78.53%81.78%
Qwen2.5-VL: Pure CEM (100% VQA)91.75%84.28%71.49%72.83%80.09%94.83%89.47%
Qwen2.5-VL: + PACE (Min-Max)91.59%84.00%98.08%98.44%93.03%90.90%91.68%
Qwen2.5-VL: + PACE (Soft-Norm)92.52%84.40%98.08%98.46%93.36%91.13%91.94%
Qwen2.5-VL: + PACE (Routed)91.65%84.20%98.68%99.00%93.38%91.65%92.28%
Qwen3.5-9B: Uniform Likelihood95.13%76.20%99.25%99.42%92.50%83.48%86.76%
Qwen3.5-9B: Pure PDAS93.22%66.00%97.88%98.52%88.91%79.53%82.94%
Qwen3.5-9B: Pure CEM (100% VQA)94.80%85.72%83.86%85.24%87.41%96.51%93.20%
Qwen3.5-9B: + PACE (Min-Max)94.69%85.49%97.84%98.50%94.13%92.78%93.27%
Qwen3.5-9B: + PACE (Soft-Norm)95.81%85.67%97.84%98.50%94.46%92.87%93.45%
Qwen3.5-9B: + PACE (Routed)94.75%85.69%99.16%99.36%94.74%93.76%94.12%
  • Conclusion from the 11-subset table above (Role of NSED routing and hybrid fusion): Uniform Likelihood and Pure PDAS drop to 61.92%–76.20% on relational swaps (ARO-Relation, vs. 84.28%–85.72% for Pure CEM), whereas unconditional Pure CEM falls to 71.49%–85.24% on word-order permutations (COCO-Order and Flickr30k-Order, vs. 97.88%–99.43% for generative scoring). Overall, NSED-routed PACE achieves the highest ARO Average and 11-Subset Macro-Average while cutting 99.5% of VQA calls on word-order tasks.
  1. Execution Order of Centering, Normalization, and Fusion on Winoground

For each 2x2 Winoground group [(I0, C0), (I0, C1), (I1, C0), (I1, C1)], the operations run in three steps:

  • Step 1 (Pairwise scoring): Compute the 4 pairwise scores for PDAS (sPDASs_{PDAS}) and CEM (sCEMs_{CEM}).
  • Step 2 (4-pair normalization and fusion): Apply Min-Max normalization Phi(.)Phi(.) across all 4 pairs in the group (mapping the minimum to 0, the maximum to 1, and the two intermediate pairs to continuous values in (0, 1)), then fuse via Eq. (8): sPACE=(1−β)∗Phi(sCEM)+β∗Phi(sPDAS)withβ=0.40s_{PACE} = (1 - \beta) * Phi(s_{CEM}) + \beta * Phi(s_{PDAS}) with \beta = 0.40.
  • Step 3 (Two-way centering): Reshape sPACEs_{PACE} into a 2x2 matrix SS and apply bidirectional centering (Eq. 9) to obtain ScenteredS_{centered}.
  1. Matched Centering Across All Baselines and Component Ablation

We applied Eq. (9) bidirectional centering to all first-stage retrievers (SigLIP2-So400m, Qwen3-VL-Embedding-8B), the supervised reranker (Qwen3-VL-Reranker-8B), and all MLLM scoring variants. The table below compares raw uncentered scores (without Eq. 9) and two-way centered scores (with Eq. 9) across Winoground Text, Image, and Group metrics:

Backbone & Scoring VariantUncentered (without Eq. 9) Text / Image / GroupTwo-Way Centered (with Eq. 9) Text / Image / Group
SigLIP2-So400m (First-Stage Bi-Encoder)38.75% / 19.25% / 17.00%73.50% / 73.50% / 73.50%
Qwen3-VL-Embedding-8B (Bi-Encoder)42.25% / 15.50% / 12.25%75.25% / 75.25% / 75.25%
Qwen3-VL-Embedding-8B + PACE55.50% / 48.25% / 37.25%88.00% / 88.00% / 88.00%
Qwen3-VL-Reranker-8B71.00% / 59.75% / 53.00%90.25% / 90.25% / 90.25%
Qwen3-VL-Reranker-8B: Pure PDAS60.75% / 60.25% / 47.00%88.25% / 88.25% / 88.25%
Qwen3-VL-Reranker-8B: PACE73.00% / 68.50% / 61.25%92.50% / 92.50% / 92.50%
LLaVA-1.5-7B: Raw Uniform Likelihood34.25% / 31.00% / 18.25%77.75% / 77.75% / 77.75%
LLaVA-1.5-7B: Pure PDAS31.25% / 23.75% / 15.00%75.75% / 75.75% / 75.75%
LLaVA-1.5-7B: PACE (Routed, Min-Max)39.75% / 34.25% / 24.50%78.00% / 78.00% / 78.00%
LLaVA-1.5-7B: PACE (Routed, Soft-Norm)40.50% / 35.00% / 25.25%78.50% / 78.50% / 78.50%
LLaVA-1.5-7B: PACE (Full g=1 Active)43.25% / 39.00% / 26.25%79.75% / 79.75% / 79.75%
Qwen2.5-VL-7B: Raw Uniform Likelihood38.50% / 38.00% / 23.50%80.25% / 80.25% / 80.25%
Qwen2.5-VL-7B: Pure PDAS37.00% / 32.50% / 19.25%82.75% / 82.75% / 82.75%
Qwen2.5-VL-7B: PACE (Routed, Min-Max)54.00% / 55.25% / 43.25%86.25% / 86.25% / 86.25%
Qwen2.5-VL-7B: PACE (Routed, Soft-Norm)56.00% / 57.00% / 46.75%87.00% / 87.00% / 87.00%
Qwen2.5-VL-7B: PACE (Full g=1 Active)70.75% / 64.75% / 58.00%89.75% / 89.75% / 89.75%
Qwen3.5-9B: Raw Uniform Likelihood47.75% / 43.75% / 31.00%83.75% / 83.75% / 83.75%
Qwen3.5-9B: Pure PDAS40.25% / 33.75% / 20.75%81.75% / 81.75% / 81.75%
Qwen3.5-9B: PACE (Routed, Min-Max)60.00% / 63.75% / 52.50%88.25% / 88.25% / 88.25%
Qwen3.5-9B: PACE (Routed, Soft-Norm)60.50% / 64.50% / 52.00%89.00% / 89.00% / 89.00%
Qwen3.5-9B: PACE (Full g=1 Active)77.25% / 78.50% / 69.50%94.00% / 94.00% / 94.00%

Two main observations follow from this comparison:

  • Gains without Eq. (9) centering: In the Uncentered column, PACE improves Group Score over Raw Uniform Likelihood from 18.25% to 24.50%-26.25% (+6.25% to +8.00%) on LLaVA-1.5, from 23.50% to 43.25%-58.00% (+19.75% to +34.50%) on Qwen2.5-VL, and from 31.00% to 52.00%-69.50% (+21.00% to +38.50%) on Qwen3.5-9B.
  • Gains under matched Eq. (9) centering: When Eq. (9) is applied equally to all baselines, SigLIP2 reaches 73.50%, Qwen3-VL-Embedding-8B reaches 75.25%, and Raw Uniform Likelihood reaches 77.75% (LLaVA-1.5), 80.25% (Qwen2.5-VL), and 83.75% (Qwen3.5-9B). Under the same centering operation, our method reaches 79.75% on LLaVA-1.5 (+2.00%), 86.25%-89.75% on Qwen2.5-VL (+6.00% to +9.50%), and 88.25%-94.00% on Qwen3.5-9B (+4.50% to +10.25%). Note on Routed vs. Full g=1 Active on Winoground: In Winoground, short 4-word captions that swap 2 words yield NSED = 2/4 = 0.50 > 0.40, which triggers the g=0 fallback (designed for 5-candidate syntactic word-order permutations in ARO-Order) and routes those pairs to Pure PDAS, resulting in 78.00%-78.50% (LLaVA-1.5), 86.25%-87.00% (Qwen2.5-VL), and 88.25%-89.00% (Qwen3.5-9B, matching the 89.00% reported in Table 2). Keeping g=1 active across all 400 Winoground pairs (PACE Full g=1 Active) avoids this short-caption fallback and reaches 79.75% on LLaVA-1.5 (319/400 vs. 320/400 [80.00%] in Table 2, a 1-sample fp16 boundary difference), 89.75% on Qwen2.5-VL, and 94.00% on Qwen3.5-9B.
  1. Unequal Text, Image, and Group Scores for Qwen2.5-VL in Table 2

Thank you for catching this. As you noted, applying Eq. (9) to any 2x2 matrix produces a symmetric matrix [[h, -h], [-h, h]], under which Text, Image, and Group scores must be identical. Checking the log for the Qwen2.5-VL entry in Table 2 (11.75% / 88.00% / 11.75%) showed that it came from an earlier run where the Step 3 two-way centering flag was inadvertently left off. With Step 3 enabled, Qwen2.5-VL yields identical Text, Image, and Group scores of 86.25% (Min-Max) and 87.00% (Soft-Norm). We will correct Table 2 and Appendix Table 8 in the next version of manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.