- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PACE, a training-free framework for improving autoregressive image–text scoring. For large-scale retrieval, it combines dual-encoder candidate filtering with autoregressive reranking, including confidence routing and text-prior correction in the image-to-text pipeline. For fine-grained matching, it combines position-discounted scoring (PDAS) with selectively triggered visual verification (CEM), and uses bidirectional centering for closed pairing groups in Winoground. Experiments cover compositionality and retrieval benchmarks, multiple backbones, component ablations, and parameter and invocation-cost analyses.
Critical strengths
- A practical inference-only approach with measurable retrieval gains. The method addresses language-prior bias and the cost of using multimodal models for retrieval without additional training. In Table 4, Qwen3.5+PACE improves COCO image-to-text R@1 from SigLIP2’s 67.14% to 69.44%. Removing PMI calibration lowers it to 67.84%, providing positive evidence for calibration within the reported pipeline. This is the main practical contribution I see in the paper.
- Evaluation across tasks and backbones. The paper covers both large-scale retrieval and fine-grained matching, with multiple backbones, component ablations, and sensitivity and invocation-cost analyses. This scope makes the framework worth considering, even though the decisive evaluation questions below still need resolution.
Critical weaknesses
- Central results and evaluation steps remain unresolved. For Qwen3.5+PACE, SugarCrepe Replace-Object/Replace-Relation are 95.2%/87.8% in Table 1 but 93.16%/81.08% in Table 5. Different subsets or configurations may explain this, but none is specified. Separately, if Eq. (9) is applied to the final 2×2 Winoground matrix, the standard Text, Image, and Group success conditions become equivalent, whereas Qwen2.5+PACE scores 11.75%/88.00%/11.75% in Table 2. This conditional observation does not establish an implementation error. It does, however, leave central results insufficiently explained, which is the main reason for my current weak-rejection recommendation.
- The incremental contribution of PDAS needs a more precise explanation. Table 5 already includes a pure-CEM ablation, with a Micro-Avg of 86.58% versus 87.12% for the full method. However, if each component is min–max normalized over two candidates and its score difference is much larger than ε, the default weights generally make conflicting rankings follow CEM. The aggregate gain could therefore arise from CEM-disabled examples, ties, or near-ε cases. Clarifying the routing used in the pure-CEM ablation and separating these cases would make the claimed complementarity more convincing.
Questions for authors
- What subsets and configurations underlie Tables 1 and 5? If they are identical, please reconcile the full-method results and explain whether any main conclusions change.
- For each backbone on Winoground, what is the exact order of centering and fusion? Please explain the Qwen2.5 row and provide sufficient raw and final scores to check the pipeline. A control using the original scores with the same centering would help distinguish scoring improvements from joint pairing effects.
- What are the normalization scope and ε value? In Table 5’s pure-CEM ablation, is routing unchanged, and how are CEM-disabled examples scored? Please separate PDAS’s contribution on CEM-enabled and CEM-disabled examples under matched routing.
4 comments
| Backbone & Scoring Variant | Uncentered (without Eq. 9) Text / Image / Group | Two-Way Centered (with Eq. 9) Text / Image / Group |
|---|---|---|
| SigLIP2-So400m (First-Stage Bi-Encoder) | 38.75% / 19.25% / 17.00% | 73.50% / 73.50% / 73.50% |
| Qwen3-VL-Embedding-8B (Bi-Encoder) | 42.25% / 15.50% / 12.25% | 75.25% / 75.25% / 75.25% |
| Qwen3-VL-Embedding-8B + PACE (Pool-4) | 55.50% / 48.25% / 37.25% | 88.00% / 88.00% / 88.00% |
| Qwen3-VL-Reranker-8B (Official) | 71.00% / 59.75% / 53.00% | 90.25% / 90.25% / 90.25% |
| Qwen3-VL-Reranker-8B + PACE (Strict) | 73.00% / 68.50% / 61.25% | 92.50% / 92.50% / 92.50% |
| LLaVA-1.5-7B: Raw Uniform Likelihood | 34.25% / 31.00% / 18.25% | 77.75% / 77.75% / 77.75% |
| LLaVA-1.5-7B: Pure PDAS (without CEM) | 31.25% / 23.75% / 15.00% | 75.75% / 75.75% / 75.75% |
| LLaVA-1.5-7B: PACE (Routed, Min-Max) | 39.75% / 34.25% / 24.50% | 78.00% / 78.00% / 78.00% |
| LLaVA-1.5-7B: PACE (Routed, Soft-Norm) | 40.50% / 35.00% / 25.25% | 78.50% / 78.50% / 78.50% |
| LLaVA-1.5-7B: PACE (Full g=1 Active) | 43.25% / 39.00% / 26.25% | 79.75% / 79.75% / 79.75% |
| Qwen2.5-VL-7B: Raw Uniform Likelihood | 38.50% / 38.00% / 23.50% | 80.25% / 80.25% / 80.25% |
| Qwen2.5-VL-7B: Pure PDAS (without CEM) | 37.00% / 32.50% / 19.25% | 82.75% / 82.75% / 82.75% |
| Qwen2.5-VL-7B: PACE (Routed, Min-Max) | 54.00% / 55.25% / 43.25% | 86.25% / 86.25% / 86.25% |
| Qwen2.5-VL-7B: PACE (Routed, Soft-Norm) | 56.00% / 57.00% / 46.75% | 87.00% / 87.00% / 87.00% |
| Qwen2.5-VL-7B: PACE (Full g=1 Active) | 70.75% / 64.75% / 58.00% | 89.75% / 89.75% / 89.75% |
| Qwen3.5-9B: Raw Uniform Likelihood | 47.75% / 43.75% / 31.00% | 83.75% / 83.75% / 83.75% |
| Qwen3.5-9B: Pure PDAS (without CEM) | 40.25% / 33.75% / 20.75% | 81.75% / 81.75% / 81.75% |
| Qwen3.5-9B: PACE (Routed, Min-Max) | 60.00% / 63.75% / 52.50% | 88.25% / 88.25% / 88.25% |
| Qwen3.5-9B: PACE (Routed, Soft-Norm) | 60.50% / 64.50% / 52.00% | 89.00% / 89.00% / 89.00% |
| Qwen3.5-9B: PACE (Full g=1 Active) | 77.25% / 78.50% / 69.50% | 94.00% / 94.00% / 94.00% |
Without Eq. (9) centering, PACE improves Group Score over Raw Uniform Likelihood from 18.25% to 24.50%-26.25% (+6.25% to +8.00%) on LLaVA-1.5, from 23.50% to 43.25%-58.00% (+19.75% to +34.50%) on Qwen2.5-VL, and from 31.00% to 52.00%-69.50% (+21.00% to +38.50%) on Qwen3.5-9B. When Eq. (9) is applied equally to all baselines, PACE still outperforms centered SigLIP2 (73.50%), centered Qwen3-VL-Embedding-8B (75.25%), and centered Uniform Likelihood (77.75% / 80.25% / 83.75%) by +2.00% to +10.25%. Note on Routed vs. Full g=1 Active on Winoground: In Winoground, short 4-word captions that swap 2 words yield NSED = 2/4 = 0.50 > 0.40, which triggers the g=0 fallback (designed for 5-candidate syntactic word-order permutations in ARO-Order) and routes those pairs to Pure PDAS, resulting in 78.00%-78.50% (LLaVA-1.5), 86.25%-87.00% (Qwen2.5-VL), and 88.25%-89.00% (Qwen3.5-9B, matching the 89.00% reported in Table 2). Keeping g=1 active across all 400 Winoground pairs (PACE Full g=1 Active) avoids this short-caption fallback and reaches 79.75% on LLaVA-1.5 (319/400 vs. 320/400 [80.00%] in Table 2, a 1-sample fp16 boundary difference), 89.75% on Qwen2.5-VL, and 94.00% on Qwen3.5-9B.
- Normalization Scope, Epsilon, Routing in Table 5, and Separating g=1 vs. g=0
- Normalization scope and epsilon: Min-Max normalization in Eq. (7) is computed across the candidate pool C for each query image with . When (when candidates yield identical quantized Yes/No logits under fp16/bfloat16), returns [0.5, 0.5] so the tie is broken by the secondary signal.
- Routing and scoring of CEM-disabled examples in Table 5's pure-CEM ablation: In the routed 'without PDAS (Pure CEM, Routed)' row, the Normalized Sequence Edit Distance (NSED) routing gate g = I(NSED <= 0.40) is unchanged. CEM-enabled examples (g=1) are scored solely by , while CEM-disabled examples (g=0, where Visual Question Answering [VQA] verification is skipped) fall back to the base autoregressive likelihood score (Uniform Log-Likelihood on Routed-Uniform, or on Routed-PDAS). To avoid any routing confounders, we also ran Unconditional Pure CEM (100% VQA, forcing g=1 on all examples) and report all methods separately on g=1 and g=0 below.
- Separating PDAS's contribution on CEM-Enabled (g=1) vs. CEM-Disabled (g=0) subsets:
(1) Breaking fp16/bfloat16 CEM ties inside g=1 under hard Min-Max: When quantized Yes/No logits tie , = [0.5, 0.5] and PDAS resolves the ranking. On the Attribution, Relation, and Order (ARO) Visual Genome Attribution split (ARO-Attribution, g=1, N=27,993), PDAS resolves 370/451 ties (82.0%) on LLaVA-1.5, 45/56 ties (80.4%) on Qwen2.5-VL, and 263/332 ties (79.2%) on Qwen3-VL-Reranker-8B (plus 346/535 ties [64.7%] on ARO-Relation [Visual Genome Relation]). On SugarCrepe Add-Attribute (g=1), PDAS resolves 27/28 ties (96.4%) on LLaVA-1.5, raising g=1 accuracy from 64.05% to 65.94%.
(2) Multi-candidate pools (C > 2) and Soft-Sigmoid normalization (Soft-Norm) inside g=1: On 5-candidate pools (COCO-Order and Flickr30k-Order, C=5), Min-Max retains continuous values in (0, 1), and PDAS + CEM improves over Pure CEM on g=1 by +9.30% to +22.48%. On binary C=2 tasks, replacing hard Min-Max with smooth Soft-Sigmoid normalization preserves relative confidence margins when logits do not tie, improving g=1 accuracy over Pure CEM across all backbones (on ARO-Attribution g=1: +9.54% [+2,670 examples] on LLaVA-1.5, +0.99% [+277 examples] on Qwen2.5-VL, +1.15% [+323 examples] on Qwen3.5-9B, and +3.25% [+910 examples] on Qwen3-VL-Reranker-8B).
(3) Avoiding VQA bag-of-words degradation on g=0 examples: On COCO-Order and Flickr30k-Order, 99.5% of examples have NSED > 0.40 (g=0). Unconditional Pure CEM drops to 71.56% (Qwen2.5-VL) and 83.87% (Qwen3.5-9B) on g=0, whereas routing g=0 to the generative scorer achieves 97.88%-99.43% (+15.3% to +27.3%) while saving 99.5% of VQA calls.
| Backbone & Benchmark Subset | Split (Total/g=1/g=0) | CEM Ties->Rescued | g=1: CEM / Min-Max / Soft-Norm | g=0: CEM / PDAS / Uniform |
|---|---|---|---|---|
| LLaVA-1.5: ARO-Attribution | 28748 / 27993 / 755 | 451->370 (82.0%) | 82.38% / 82.89% / 91.92% | 82.65% / 89.54% / 94.83% |
| LLaVA-1.5: COCO-Order | 25009 / 129 / 24880 | 0 (C=5 pool) | 73.64% / 87.60% / 98.45% | 87.29% / 98.80% / 99.43% |
| LLaVA-1.5: SugarCrepe Add-Attribute | 692 / 687 / 5 | 28->27 (96.4%) | 64.05% / 65.94% / 86.46% | 60.00% / 80.00% / 80.00% |
| LLaVA-1.5: SugarCrepe Replace-Relation | 1406 / 1267 / 139 | 19->13 (68.4%) | 83.74% / 84.06% / 86.58% | 87.77% / 69.78% / 72.66% |
| Qwen2.5-VL: ARO-Attribution | 28748 / 27993 / 755 | 56->45 (80.4%) | 91.68% / 91.72% / 92.67% | 94.17% / 86.89% / 90.60% |
| Qwen2.5-VL: ARO-Relation | 23937 / 23487 / 450 | 38->19 (50.0%) | 84.26% / 84.26% / 84.67% | 85.33% / 70.22% / 80.67% |
| Qwen2.5-VL: COCO-Order | 25009 / 129 / 24880 | 0 (C=5 pool) | 58.14% / 80.62% / 80.62% | 71.56% / 98.17% / 98.89% |
| Qwen2.5-VL: SugarCrepe Replace-Attr | 788 / 766 / 22 | 1->1 (100.%) | 93.60% / 93.60% / 94.26% | 95.45% / 63.64% / 77.27% |
| Qwen3.5-9B: ARO-Attribution | 28748 / 27993 / 755 | 7->5 (71.4%) | 94.74% / 94.74% / 95.89% | 97.22% / 92.85% / 95.10% |
| Qwen3.5-9B: ARO-Relation | 23937 / 23487 / 450 | 7->2 (28.6%) | 85.70% / 85.69% / 85.88% | 86.89% / 74.89% / 85.11% |
| Qwen3.5-9B: COCO-Order | 25009 / 129 / 24880 | 0 (C=5 pool) | 81.40% / 90.70% / 90.70% | 83.87% / 97.88% / 99.25% |
| Qwen3.5-9B: SugarCrepe Add-Attribute | 692 / 687 / 5 | 0 ties | 97.67% / 97.67% / 98.69% | 100.0% / 40.00% / 80.00% |
| Qwen3.5-9B: SugarCrepe Replace-Object | 1652 / 1575 / 77 | 0 ties | 98.54% / 98.48% / 98.60% | 98.70% / 72.73% / 72.73% |
| Qwen3.5-9B: SugarCrepe Replace-Relation | 1406 / 1267 / 139 | 0 ties | 93.84% / 93.76% / 94.00% | 94.96% / 63.31% / 70.50% |
| Qwen3-VL-Reranker-8B: ARO-Attribution | 28748 / 27993 / 755 | 332->263 (79.2%) | 91.49% / 91.87% / 94.74% | 91.52% / 92.32% / 93.64% |
| Qwen3-VL-Reranker-8B: ARO-Relation | 23937 / 23487 / 450 | 535->346 (64.7%) | 80.16% / 80.52% / 83.85% | 82.22% / 73.78% / 82.00% |
| Qwen3-VL-Reranker-8B: Replace-Relation | 1406 / 1267 / 139 | 27->20 (74.1%) | 92.11% / 92.82% / 94.71% | 97.12% / 81.29% / 82.01% |
Holding NSED routing fixed, PDAS provides token-level tie-breaking on CEM-Enabled (g=1) examples (resolving 50.0% to 96.4% of quantized fp16/bfloat16 Yes/No ties under hard Min-Max, and improving g=1 accuracy over Pure CEM by +0.99% to +9.54% under Soft-Norm and +9.30% to +24.81% on C=5 pools), while NSED routing prevents Pure CEM's bag-of-words degradation on CEM-Disabled (g=0) word-order examples (+15.3% to +27.3% accuracy gain while saving 99.5% of VQA calls).
Thank you for your review. Below we address your questions.
Execution order on Winoground: For each 2x2 group [(I0, C0), (I0, C1), (I1, C0), (I1, C1)], we (1) compute the 4 pairwise scores s_PDAS and s_CEM, (2) apply Min-Max normalization Phi(.) across the 4 pairs in the group (minimum 0, maximum 1, and two intermediate values in (0, 1)) and fuse them via Eq. (8) with beta = 0.40, and (3) reshape s_PACE into a 2x2 matrix and apply two-way centering (Eq. 9).
Why Qwen2.5-VL reported 11.75% / 88.00% / 11.75% in Table 2: Applying Eq. (9) to any 2x2 matrix yields a symmetric matrix [[h, -h], [-h, h]], under which Text, Image, and Group scores must be identical. The 11.75% / 88.00% / 11.75% row in Table 2 was inadvertently transcribed from an intermediate log recorded prior to applying the final Eq. (9) centering step. With Eq. (9) consistently applied, Qwen2.5-VL yields identical Text, Image, and Group scores of 86.25% (Min-Max) and 87.00% (Soft-Norm).
Standard Uncentered vs. Two-Way Centered Eq. (9) across all baselines (reporting Text / Image / Group scores):