acceptodds
Under review as a conference paper at ICLR 2027

PAIQ: Patch-Aligned Semantic Injection with an Orthogonal Bridge

Abstract

Language-aligned and self-supervised visual encoders offer complementary strengths in semantic abstraction and spatial detail. Making use of this complementarity requires incorporating language-aligned semantics into local visual representations. We introduce PAIQ, a visual fusion method that enriches DINOv3 patch features with SigLIP features through patch-wise residual updates. PAIQ uses DINOv3 features as the base representation and projects both visual streams into the language-model dimension. Learned content similarities and a finite-step Sinkhorn procedure determine how SigLIP features are aggregated for each patch. A shared orthogonal transformation maps the difference between the aggregated and base features, and the transformed residual is added back with a fixed scale. This design learns update directions while preserving residual norms and pairwise angles, providing an explicit geometric constraint on semantic enrichment. Both visual encoders and the language model remain frozen; only projection and fusion parameters are trained, while the fused sequence retains 196 visual tokens. Across five language backbones and four evaluated datasets, PAIQ achieves higher correctness and lower hallucination severity than single-encoder baselines in most settings. PAIQ also compares favorably with mainstream multi-encoder and token-compression baselines, including an adapted CoME-VL baseline in the 2B setting. These results demonstrate that geometrically constrained residual updates provide an effective way to integrate complementary visual representations while maintaining a compact visual interface.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MLLM, Visual Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_w8vFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper addresses the problem of visual representation learning and proposes a method that integrates two distinct representations, DINOv3 and SgiLIP. More specifically, the proposed method first computes the similarity between every possible pair of DINOv3 patch features and SigLIP tokens, and then integrates the semantically rich SigLIP tokens into the spatially rich DINOv3 patch features based on this similarity. Experimental evaluations on image captioning and visual question answering demonstrate that the proposed method outperformed the baselines (DINOv3 and SigLIP) and closely related methods (GRANULON, CoME-VL, and MERV) in most cases.

Critical strengths

S1. The problem dealt with in this paper is significant. Visual representation learning is one of the most significant and fundamental research problems in computer vision. Among various visual representations, DINO is the most popular and relies solely on still-image inputs. However, as a result, visual representations produced by the DINO framework lack detailed semantic information. Meanwhile, the recent emergence of large-scale vision-language models (LVLMs) would generate another line of semantically rich visual representations under their guidance. In this sense, the problem of enhancing DINO to obtain semantically rich information is becoming one of the research problems attracting attention from computer vision researchers.

S2. The structure of the proposed method, as shown in Figure 3, is reasonable and technically sound. Although the way of aligning SigLIP tokens onto DINOv3 patch features seems brute-force, narrowing the selection to a small number and aligning the features of both is technically sound. Also, the technique of integrating aligned features with optimized rotations is reasonable.

S3. The experimental evaluations in image captioning and visual question answering demonstrate the unreasonable effectiveness of the proposed method. The proposed method greatly outperforms CoME-VL, which also integrates DINOv3 and SigLIP with cross-attention architectures, in almost all the settings.

Critical weaknesses

W1. The current manuscript is extremely difficult to read, which is a great loss given the excellent technical content. Until I looked at Figure 3, I could not even grasp the basic outline of how the proposed method was structured or what its key features were. What the authors should do first is to ensure that readers understand what can be achieved using the proposed method. However, the current manuscript skips it and focuses on explaining individual technical components. Also, it deliberately tries to explain something that could easily be conveyed concisely in a complicated way. I would recommend that the authors majorly revise Sections 1 and 2 in the camera-ready version. Also, Lemma 1 and related descriptions can be removed from the main content, as they are not the focus of this paper.

W2. I could not understand why the current methodology, called orthogonal bridge injection, is the best for integrating two aligned features. I understand that it might be better to contain trainable parameters in the methodology. However, I could not understand why the current shape is the best in terms of performance rather than computational cost. As far as I checked, it has not been well justified, neither literally, theoretically, nor empirically. For example, we can easily find simpler ways, such as concatenating D and ¥overline{S}.

Questions for authors

Q1. I am not familiar with the automatic judge used in this paper and Granulon. I would know its reproducibility against LLM deprecation.

2 comments
AuthorsOpenReview ↗

Manuscript Clarity, Fusion Design, and Evaluation Reproducibility

Thank you for recognizing PAIQ's excellent technical content. Your constructive comments and suggestions are very helpful, and we will carefully incorporate them into the revised manuscript.

W1: Manuscript readability

Response. We acknowledge that the writing needs substantial improvement; English not being our first language may have contributed. In the camera-ready version, we will:

  • Rewrite Sections 1 and 2 to present PAIQ's goal and overall structure before its components, using clearer, more concise language.
  • Move Lemma 1 and related descriptions out of the main text.
  • Reposition Figure 3 and refine other passages.

W2: Orthogonal Bridge Injection Design Explanation

1. Theoretical rationale

ComponentRationale
Sinkhorn matchingFive alternating row/column normalizations couple source allocation across patches, unlike independent row-wise Softmax.
Residual injectionDINOv3 remains the spatial base, while SigLIP adds semantics through an incremental residual update.
Orthogonal transformA shared Cayley-parameterized matrix learns the update direction while preserving residual norms and pairwise inner products.

2. Empirical evidence

EvidenceResults
Fixed-QQ ablation (Fig. 6)Fixing Q=IQ=I reduces Qwen3.5-2B task-macro Accuracy 68.53→36.72 and increases Hallucination 40.13→71.24.
Component ablationsRemoving orthogonality or replacing Sinkhorn with row-wise Softmax degrades performance (see Reviewer tjSd, W1+Q1).
Simple fusion baselinesPAIQ outperforms Resize+Addition, Learnable Residual Fusion, and simple channel-concatenation-based fusion methods. (see Reviewer tjSd, W1+Q1).
Existing methodsTable 1 shows PAIQ's gains over GRANULON, CoME-VL, MERV, and LLaVA-Mini across four datasets and multiple backbones.

3. Related-work support

We draw inspiration from these established works, which provide relevant theoretical and implementation support for our design choices.

PrincipleRepresentative work
Residual learningHe et al. (CVPR 2016); Rebuffi et al. (NeurIPS 2017).
Orthogonal mappings / updatesXing et al. (NAACL 2015); Artetxe et al. (EMNLP 2016); Bansal et al. (NeurIPS 2018); Oh et al. (NeurIPS 2025).
Cayley transformHelfrich et al. (ICML 2018).
Sinkhorn / optimal transportCuturi (2013).
Orthogonal residual fusionOpenSGG-VL (ACM ICMR 2026), for cross-modal 2D/3D integration with a different operator.

Q1: Automatic Judge and Reproducibility

1. Automatic judge

Following GRANULON [1], each response is independently scored on two axes:

MetricRangeBetter
Accuracy0–100Higher
Hallucination0–100Lower
TaskJudge input
SEED-Bench / A-OKVQAImage, question, all options, reference answer, unedited response, and fixed scoring prompt.
FLUX / CaptionImage, unedited response, and fixed caption-scoring template (no question or reference answer).

The two metrics are requested separately, with no shared context. We use GPT-4o with fixed settings and record each model identifier and response ID. Prompts, scoring rules, and request settings are in Appendix H, C.3 (Table 8), and C.4 (Table 9).

2. Reliability

Multimodal LLM-based evaluation has been widely adopted in visual benchmarks and evaluation frameworks, including Prometheus-Vision (ACL 2024), WildVision-Bench (NeurIPS 2024), CapArena (ACL 2025), DCScore (ICLR 2025), and other related works. Manual checks also show high correlation between LLM scores and output quality (Appendix G).

3. Reproducibility without an LLM judge

  • Setting: We evaluated the same Qwen3.5-2B setup and samples as Table 1 on the multiple-choice portions of SEED-Bench and A-OKVQA.
  • Scoring: Given each image and question, we compute the teacher-forced sum of log probabilities for every candidate option and select the highest-scoring one.
  • Evaluation: We use the official benchmark evaluation scripts.
MethodSEED-Bench (Acc. %)A-OKVQA (Acc. %)
SigLIP34.437.1
DINOv340.638.9
GRANULON39.337.7
CoME-VL38.942.3
LLaVA-Mini35.829.0
PAIQ42.143.6

PAIQ achieves the best accuracy on both benchmarks without an LLM judge.

References

[1] Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM

[2] Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation

[3] WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences

[4] CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era

[5] Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning

Thank you very much for your time and for the constructive discussion throughout the review process. We truly appreciate your thoughtful feedback, which has been very helpful in improving the clarity and completeness of our work. We have carefully considered your comments and made every effort to address the concerns you raised. We hope that our responses have provided sufficient clarification.

We would also be very happy to address any remaining questions or concerns you may have. If there are any aspects of our methodology, experiments, or explanations that you feel still need further clarification, please do not hesitate to let us know. We would be more than willing to provide additional explanations or evidence. Thank you again for your patience and consideration. We sincerely appreciate the opportunity to discuss our work with you.

Reviewer_tjSdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PAIQ, which fuses the local features of DINOv3 with the semantic features of SigLIP. Built upon DINOv3 patches, the method retrieves SigLIP features via content matching and Sinkhorn aggregation, then updates residuals through orthogonal transformation, ultimately retaining 196 visual tokens. During training, the two visual encoders and the language model are frozen, and only the projection and fusion modules are updated. The authors conduct experiments across five base language configurations and four sets of captioning and question-answering tasks, reporting correctness and hallucination severity scores generated by an automated evaluation model. The results demonstrate that this method outperforms single encoders under most settings and surpasses other fusion methods listed in the table with the Qwen configuration.

Critical strengths
  1. An approach that augments SigLIP semantics based on the local features of DINOv3. Orthogonal residuals impose explicit constraints for feature updates, and the authors also analyze the conditions under which distinct patches remain discriminable with shared semantics.
  2. The authors conduct validations across multiple language backbones and model sizes. The full method outperforms single encoders under most settings, and surpasses other fusion methods in the table under the Qwen configuration. 196 visual tokens are retained after fusion, allowing the two streams of information to be fed into the language model via a fixed-length interface.
Critical weaknesses
  1. Simple fusion baselines are missing, and the existing ablation studies fail to disentangle the contributions of each module. Prismatic VLMs [1] and Eagle [2] have demonstrated the performance of simple channel concatenation, yet this paper does not compare addition after resizing, learnable residuals, or concatenation followed by MLP. I wonder whether there exist simpler and more effective ways to fuse siglip and dino. Moreover, separate validation of the contributions of Sinkhorn and orthogonal constraints is required to further justify the necessity of the method.
  2. The performance of the single SigLIP baseline differs drastically between Qwen3.5-2B and Qwen3.5-9B. For instance, the FLUX score rises from 31.72 to 80.51. The paper does not explain this phenomenon or provide convergence information, which somewhat undermines the evaluation of the improvement brought by the proposed method.

[1] Karamcheti, S., Nair, S., Balakrishna, A., Liang, P., Kollar, T. & Sadigh, D.. (2024). Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models. <i>Proceedings of the 41st International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 235:23123-23144 Available from https://proceedings.mlr.press/v235/karamcheti24a.html. [2] Shi, Min, et al. "Eagle: Exploring the design space for multimodal llms with mixture of encoders." International Conference on Learning Representations. Vol. 2025. 2025.

Questions for authors
  1. Please provide comparisons with simple yet effective fusion methods such as those in Prismatic VLMs and Eagle under the authors' setting, to better demonstrate the necessity and importance of the proposed method.
  2. Have you attempted to train Qwen3.5 4B? I believe the performance of the 2B variant should not fall so far below that of the 9B model.
4 comments
AuthorsOpenReview ↗

W2 + Q2: SigLIP Scaling Analysis and Qwen3.5-4B Evaluation

W2 + Q2: SigLIP scaling behavior and Qwen3.5-4B evaluation

W2: The reviewer notes the substantial performance gap between the single-SigLIP baselines on Qwen3.5-2B and Qwen3.5-9B, and requests an explanation and training-convergence information.

Q2: Have you attempted to train Qwen3.5-4B? The reviewer expects its performance to fall between the 2B and 9B models.

Authors' Response:

1. Within-backbone comparison clarify

Table 1 is designed to evaluate PAIQ by comparing different methods under the same backbone, keeping backbone capacity fixed.

  • On Qwen3.5-2B, PAIQ achieves higher Accuracy and lower Hallucination than the single-encoder and fusion baselines across all four datasets.
  • The same consistent improvements are observed on Qwen3.5-9B.

These results demonstrate PAIQ's effectiveness under different backbone scales.

2. Additional Qwen3.5-4B evaluation

To address the scaling gap you highlighted, we trained the single-SigLIP baseline on Qwen3.5-4B using the same data and training recipe as in the paper. We evaluated it on the same samples using the same GPT-4o scoring protocol.

Single-SigLIP baselineFLUX (Acc. ↑ / Hall. ↓)Caption (Acc. ↑ / Hall. ↓)SEED (Acc. ↑ / Hall. ↓)A-OKVQA (Acc. ↑ / Hall. ↓)
Qwen3.5-2B31.72 / 75.4219.40 / 85.5929.77 / 74.7127.68 / 78.86
Qwen3.5-4B44.04 / 64.1024.99 / 79.6635.33 / 70.4038.00 / 70.93
Qwen3.5-9B80.51 / 31.0258.62 / 50.9753.44 / 50.9850.11 / 53.77

Key findings:

  • Accuracy consistently increases from 2B to 4B and then to 9B across all four datasets.
  • Hallucination consistently decreases with increasing backbone scale.
  • The 4B results fall between the 2B and 9B results on all four datasets for both metrics, supporting the consistency of the observed scaling trend.

3. Training convergence analysis

We further provide the training-loss progression at different steps:

Training stepSigLIP (2B)SigLIP (9B)DINOv3 (2B)DINOv3 (9B)
1001.6291.4211.6331.544
5001.4151.1741.4051.241
1,0001.3621.1021.3441.178
2,0001.3360.9661.3021.126
3,0001.3350.9611.2981.123
Average train loss1.3741.0561.3471.183

The average training-loss reductions from Qwen3.5-2B to Qwen3.5-9B are:

  • SigLIP: 1.374 → 1.056 (−0.318).
  • DINOv3: 1.347 → 1.183 (−0.164).

The loss reduction of the Qwen+SigLIP combination is substantially larger than that of Qwen+DINOv3, indicating a more pronounced scaling effect. This helps explain the larger performance improvement observed in the single-SigLIP baseline.

4. Supporting evidence from prior studies

Similar scaling effects have also been reported in prior studies using the Qwen+SigLIP combination. These works report approximately 45–50-point gains on visual tasks when scaling Qwen from 0.5B to 7B:

  • [1] LLaVA-OneVision: Easy Visual Task Transfer, TMLR 2025.
  • [2] Face-Human-Bench: A Comprehensive Benchmark of Face and Human Understanding for Multi-modal Assistants, NeurIPS 2025.
  • [3] More Images, More Problems? A Controlled Analysis of VLM Failure Modes, Findings of ACL 2026.

Thank you again for taking the time to review our work and provide valuable feedback. We sincerely appreciate your thoughtful comments, which have helped us identify several aspects of the paper that needed further clarification and improvement. We have made every effort to carefully address your concerns through detailed explanations and additional experiments. We genuinely hope that our responses have resolved the issues you initially raised and have demonstrated the technical soundness and merits of our approach more clearly.

We fully respect your judgment and understand that evaluating a paper requires careful consideration. However, if you feel that our clarifications and additional evidence have adequately addressed your concerns, we would be truly grateful if you could reconsider your overall assessment and score.

Thank you once again for your time, patience, and constructive engagement with our work. We sincerely appreciate your consideration.

Reviewer_tjSdOpenReview ↗

additional experiments and clarifications.

Thank you for the additional experiments and clarifications. They have addressed my main concerns, and I’m happy to raise my score. I also read your responses to the other reviewers, which helped clear up some of my remaining questions. I suggest adding the key experiments addressing these concerns to the revised paper.

Thank you very much for your positive feedback and for raising your score. We sincerely appreciate your recognition and careful consideration of our responses.

We are glad that our clarifications have addressed your concerns. We fully agree with your suggestion and will incorporate the key experiments and clarifications addressing all the concerns into the revised manuscript.

Thank you again for your time, constructive feedback, and encouragement!

Reviewer_KYRbOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

PAIQ is a visual feature-fusion method that incorporates semantic information from SigLIP into DINOv3 patch features. It aggregates SigLIP features using learned content similarities and finite-step Sinkhorn-style normalization, applies an orthogonal transformation to the difference between the aggregate and the DINOv3 feature, and adds the transformed residual back to the base feature. Both visual encoders and the language model remain frozen; only projection and fusion parameters are trained, and the output retains 196 visual tokens.

The paper compares automated-judge correctness and hallucination-severity scores across five language-backbone configurations and four datasets. It analyzes residual geometry and patch separability and explains in Appendix A.4 that, under specified assumptions, the method and direct interpolation represent the same function class.

Critical strengths

1. The geometric constraint introduced into feature fusion is clearly specified.

Section 3.3 shows that the orthogonal transformation preserves residual norms and pairwise angles, while Lemma 1 gives conditions under which patch differences are preserved for fixed intermediate features. The design objective of combining two encoders while retaining 196 visual input tokens for the language model is also clear.

2. The experiments include multiple backbones and strong fusion baselines.

Table 1 compares against CoME-VL, MERV, and LLaVA-Mini in addition to single-encoder baselines. Figure 6 includes a Q = I control that retains the matching and aggregation mechanisms. The authors also disclose variation in performance across backbones and tasks, as well as anomalies in the evaluation records.

Critical weaknesses

Update (17 November 2026): The authors’ reconciliation addresses my concerns about the evaluation accounting and arithmetic, and the clarification of matched training conditions addresses my concern about the fixed-Q comparison. I therefore update Soundness from 2 to 3 and no longer count these concerns as reasons for rejection. My overall rating currently remains 2, while I reassess the strength and scope of the contribution in light of the corrected results and added controls.

Original review:

1. It remains unclear whether the small gains over strong comparators persist after correcting evaluation errors.

Appendix C.3 and Table 7 report that, among 330 empty outputs, 17 received Accuracy scores of 100 and four received scores of 95. Of 71 empty VQA outputs, 52 violated the stated scoring rules, and the original scores were included in the reported means.

Appendix C.2 reports 1,000 fixed, aligned records per evaluation cell, and Appendix D.2, Eq. (48), specifies an arithmetic mean that retains empty outputs in the denominator. Setting only the nonzero Accuracy scores in Table 7 to zero would produce a total correction of 2,080 score points. Under this reported aggregation rule, the correction would be at most 2.08 points even if all these errors occurred in a single cell. These listed errors alone would not account for the large gaps against single-encoder baselines, but could affect some close comparisons with stronger fusion baselines in Table 1. The distribution of these errors by method and evaluation cell, together with consistently corrected results, is needed to assess those close comparisons.

The reported metric is also a judge score. Where the existing output format permits a faithful answer-correctness evaluation, applying the corresponding benchmark protocol would provide additional evidence for the performance claims.

2. The empirical benefit of the orthogonal parameterization needs a clearer training comparison.

Lemma 2 in Appendix A.4 establishes that PAIQ and the Q = I model represent the same function class when the projections and matching maps are unrestricted. Function-class equivalence does not rule out differences in optimization or generalization under finite training. The empirical question is what benefit the orthogonal parameterization provides under a comparable training budget.

Figure 6 compares a learned orthogonal Q with a fixed Q = I, which is itself orthogonal; the projection and matching maps remain trainable in the latter control. This comparison supports a benefit of learning Q under the reported training recipe, but does not isolate the benefit of the orthogonality constraint relative to a learned unconstrained Q. Section 4.1 and Appendix C.1 report a shared training recipe; confirming that it also applies to the fixed-Q control and clarifying how its settings were selected would help interpret this comparison. A learned unconstrained-Q comparison, if available under comparable training conditions, would provide supplementary evidence about the role of the constraint.

My weak rejection is driven primarily by uncertainty about the gains over strong fusion baselines. Corrected, method-level evaluation results that preserve those gains, together with a fair Q = I training comparison, would be the main reasons to reconsider my recommendation.

Questions for authors

Q1. Using the same evaluation records and denominator, please report consistently corrected method-level scores and the distribution of empty-output scoring errors across evaluation cells. Do the close comparisons with strong fusion baselines remain favorable after correction? Where the existing SEED-Bench and A-OKVQA output formats permit a faithful answer-correctness evaluation, do those gains also persist under the corresponding benchmark protocol?

Q2. Section 4.1 and Appendix C.1 report a shared training recipe. Please confirm that this recipe was also used for the fixed-Q control in Figure 6 and clarify how its settings were selected, including whether any control-specific tuning was performed. If available, a learned unconstrained-Q control under comparable training conditions would provide supplementary evidence about the orthogonality constraint, beyond the current learned-Q versus fixed-Q comparison.

12 comments
AuthorsOpenReview ↗

W1+Q1: Empty-Output Scoring and Benchmark-Protocol Evaluation

Thank you for recognizing PAIQ's clear geometric design and its strong results across multiple backbones and fusion baselines.

We sincerely appreciate your openness to reconsider the score upon addressing the concerns. We’ll carefully revise to fully respond to your insightful feedback.

Comment 1: Empty-output scoring

W1: The reviewer identifies two empty-output issues: among 330 empty outputs, 17 received Accuracy scores of 100 and four received 95; among 71 empty VQA outputs, 52 violated the stated scoring rules. Since each evaluation cell contains 1,000 records and empty outputs remain in the denominator, setting the 21 nonzero scores to zero would change a cell mean by at most 2.08 points. These errors may affect close comparisons with strong fusion baselines, so the reviewer requests method- and cell-level error distributions and consistently corrected scores.

Q1: Using the same evaluation records and denominator, please report consistently corrected method-level scores and the distribution of empty-output scoring errors across evaluation cells. Do the close comparisons with strong fusion baselines remain favorable after correction?

Authors' Response:

We clarify the handling of both groups of cases:

  • 1. The 52 VQA cases violating the scoring rules. As disclosed in Appendix C.3, sample-specific content triggered model refusals in these cases. We regenerated their answers using Qwen3-VL-235B-A22B-Instruct and rescored them with the same prompt and criteria used for all other cases.

  • 2. The 21 empty outputs with nonzero Accuracy scores. Of the 330 empty outputs, 17 had received 100 and four had received 95. We applied the same regeneration and rescoring procedure to these 21 cases.

Thus, the 2,080 score points identified by the reviewer come from regenerated answers evaluated under the same protocol, rather than from scoring empty outputs. We will clarify this procedure in Appendix C.3.


Comment 2: Benchmark-protocol answer-correctness evaluation

W1: The reviewer notes that our reported metric relies on judge scores and asks whether SEED-Bench and A-OKVQA can additionally be evaluated using their benchmark-specific answer-correctness protocols, where the output formats permit.

Q1: Where the existing SEED-Bench and A-OKVQA output formats permit a faithful answer-correctness evaluation, do those gains also persist under the corresponding benchmark protocol?

Authors' Response:

We evaluated both datasets using benchmark-aligned closed-ended answer-correctness protocols, without an LLM judge.

1. SEED-Bench [1]. Following the official answer-ranking protocol [1], we select the candidate with the highest teacher-forced log-likelihood conditioned on the image and question. Let the image be II, the question be qq, and the four candidate options be o1,…,o4o_1,\ldots,o_4. After tokenization, option oko_k is represented as (tk,1,…,tk,Lk)(t_{k,1},\ldots,t_{k,L_k}). Its score is:

\log p_\theta(t_{k,j}\mid I,q,t_{k,<j}).$$ The predicted option is: $$\hat{k} = \arg\max_{k\in\{1,\ldots,4\}} s_k.$$ Accuracy is calculated as: $$\mathrm{Acc}_{\mathrm{SEED}} = 100 \times \frac{N{\mathrm{correct}}}{N}, \qquad N=1000.$$ Here, $N_{\mathrm{correct}}$ is the number of questions for which the predicted option matches the ground truth. **2. A-OKVQA [2].** We evaluate the selected option under the official A-OKVQA multiple-choice protocol: $$\mathrm{Acc}_{\mathrm{AOKVQA}} = 100 \times \frac{N{\mathrm{correct}}}{N}, \qquad N=1000.$$ Here, $N_{\mathrm{correct}}$ is the number of predictions matching the official correct-choice index. The random-choice accuracy is 25%. **Quantitative results.** Results are: | Method | SEED-Bench (Acc. ↑) | A-OKVQA (Acc. ↑) | |:--|--:|--:| | SigLIP | 34.4 | 37.1 | | DINOv3 | 40.6 | 38.9 | | GRANULON | 39.3 | 37.7 | | CoME-VL | 38.9 | 42.3 | | LLaVA-Mini | 35.8 | 29.0 | | **PAIQ** | **42.1** | **43.6** | **PAIQ achieves the highest accuracy on both datasets**, consistent with our open-ended evaluation. For open-ended generation, PAIQ also produces more accurate, natural, and nuanced answers (see **Appendix G**). **References** [1] B. Li et al., “SEED-Bench: Benchmarking Multimodal Large Language Models,” *CVPR*, 2024. [2] D. Schwenk et al., “A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge,” *ECCV*, 2022.
AuthorsOpenReview ↗

W2+Q2: Controlled Comparisons for the Orthogonal Parameterization

Comment 3: Orthogonal parameterization and training controls

W2: Although PAIQ and fixed Q=IQ=I have the same function class under unrestricted projections and matching maps, their optimization and generalization may differ under finite training. Figure 6 compares learned orthogonal QQ with fixed Q=IQ=I, but does not isolate the orthogonality constraint from the effect of learning QQ. The reviewer requests confirmation that the fixed-QQ control uses the same training recipe without separate tuning, and suggests adding a learned unconstrained-QQ control. Preserving the gains over strong fusion baselines and providing a fair Q=IQ=I comparison would be the main reasons to reconsider the recommendation.

Q2: Section 4.1 and Appendix C.1 report a shared training recipe. Please confirm that this recipe was also used for the fixed-QQ control in Figure 6 and clarify how its settings were selected, including whether any control-specific tuning was performed. If available, a learned unconstrained-QQ control under comparable training conditions would provide supplementary evidence about the orthogonality constraint, beyond the current learned-QQ versus fixed-QQ comparison.

Authors' Response:

We clarify the fixed-QQ training setup and provide the additional learned unconstrained-QQ comparison below.

  • 1. Fixed-Q=IQ=I control in Figure 6.

    • The control uses exactly the same training recipe as Section 4.1 and Appendix C.1: the data, training steps, batch size, learning rate, optimizer, and frozen components are identical. No control-specific hyperparameter tuning was performed.
    • In PAIQ, QQ is initialized as the identity matrix. The fixed-Q=IQ=I control simply keeps QQ at this initialization while all other parameters remain trainable. Thus, whether QQ is learned is the only difference between the two settings, making this a fair comparison.
    • Training logs further show that the losses in both settings stabilize in the later training stage, indicating that neither setting is undertrained. Under these identical training conditions, learning QQ performs better than fixing Q=IQ=I.
  • 2. Learned unconstrained-QQ control.

    • We additionally trained a learned unconstrained-QQ control using the same training configuration. We only replace PAIQ's orthogonal matrix QQ with a standard trainable unconstrained matrix; the fusion form remains unchanged:

      z=d+0.6(sˉ−d)Q⊤.z=d+0.6(\bar{s}-d)Q^\top.

    • Quantitative results. Each entry reports Accuracy ↑ / Hallucination ↓:

      MethodFLUXCaptionSEEDA-OKVQA
      Fixed Q=IQ=I47.32 / 62.6134.52 / 74.0032.78 / 72.4732.27 / 75.86
      Learned unconstrained QQ87.19 / 21.6876.77 / 34.3352.09 / 54.8547.74 / 57.29
      PAIQ89.97 / 20.1681.27 / 32.2454.02 / 52.6348.85 / 55.48

PAIQ outperforms both controls across all four datasets in terms of Accuracy and Hallucination. Compared with learned unconstrained QQ, the*orthogonally constrained QQ achieves higher Accuracy and lower Hallucination on every dataset. These results are consistent with our theoretical analysis: the orthogonal constraint allows QQ to adjust the update direction while preserving the residual's geometric structure, improving the separability of different patches and leading to better performance.

Reviewer_KYRbOpenReview ↗

Follow-up on the evaluation procedure and additional comparisons

Thank you for confirming the training configuration and adding the learned unconstrained-Q comparison. These additions make the training controls clearer and provide a useful comparison with another learned residual map. The judge-free SEED-Bench and A-OKVQA results are also welcome additional evidence.

My main evaluation concern, however, does not yet appear to be resolved. The original request was to correct the scoring of the same model outputs, whereas the response describes regenerating answers and then rescoring them. I am not yet able to tell what was regenerated or which outputs contributed to the original and updated results. Clarifying this distinction would help assess how the reported gains should be attributed to the evaluated models.

1. What was regenerated, and when?

Could you clarify whether Qwen3-VL-235B-A22B-Instruct regenerated the evaluated models' candidate answers, regenerated only the evaluator's scoring responses, or was used in both roles?

In the manuscript I reviewed, Appendix C.3 describes Table 7 as scores returned for empty candidate outputs and states that the original evaluator scores were retained in the reported means. Appendix C.4 separately describes Qwen as a recovery route for missing evaluator responses, with the candidate answer unchanged. I may be reading “regenerated answers” differently from what was intended, so it would be helpful to explain how these descriptions relate to the response's statement that the 2,080 score points came from regenerated answers.

Was this procedure applied before the originally reported results or during the rebuttal, and which reported tables include its results? The distinction matters because replacing candidate answers would introduce an external generation component, whereas recovering evaluator responses would leave the evaluated models' outputs unchanged.

2. What are the corrected results with the original candidate outputs retained?

Could you provide the affected-record counts and before/after Accuracy and Hallucination scores by method and evaluation cell, keeping the original candidate outputs, evaluation records, and full denominator? A compact table showing the correction rules and their application across methods within each task would make the effect on the comparisons much easier to assess.

If candidate answers were regenerated, could those results be presented separately from corrections that retain the original outputs? The latter comparison would directly address whether the close margins over strong fusion baselines remain after consistent scoring.

3. Which setting does the additional judge-free evaluation cover?

Could you specify the backbone, checkpoints, and evaluation records used in the new SEED-Bench and A-OKVQA table? The SEED-Bench likelihood-ranking evaluation provides useful evidence independent of the judge scores, although it is a different evaluation from rescoring the existing open-ended answers.

Were the scores or predictions produced by each evaluated system's own checkpoint without external candidate-answer replacement, and how were the A-OKVQA choices selected? It would also help to identify which backbone and baseline comparisons the new table directly corroborates.

The training-control additions address an important part of my review. The remaining questions concern the interpretation and comparability of the evaluation results. A clear account of candidate generation, evaluator recovery, and scoring correction would help determine how far the combined evidence supports the paper's performance claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.