acceptodds
Under review as a conference paper at ICLR 2027

PolarVLM: When Polarization Meets Vision-Language Models

10 views

Abstract

Mainstream vision-language models (VLMs) fundamentally struggle with severe optical ambiguities, such as reflections and transparent objects, due to the inherent limitations of standard RGB inputs. While polarization imaging captures polarimetric physical parameters that resolve these ambiguities, existing methods are constrained by fixed-format outputs and remain isolated from open-ended reasoning. To bridge this semantic-physical gap, we introduce **PolarVLM**, the first multimodal framework integrating polarimetric physical parameters into VLMs. By employing a dual-stream architecture and a progressive two-stage training strategy, PolarVLM effectively prevents physical misinterpretations while largely retaining general visual abilities. Complementing our architecture, we construct **PolarVQA**, the first benchmark for polarization-aware VQA, featuring 75K physics-grounded instruction-tuning pairs targeting reflective and transparent scenes. Experiments show that PolarVLM surpasses the RGB baseline by 25.4% overall across five evaluation tasks, with remarkable gains of 26.6% in reflection recognition and 34.0% in glass counting, successfully unlocking physics-aware semantic understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, polarization imaging, visual question answering, multimodal fusion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_21o9OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a model and a benchmark. PolarVLM integrates RGB and polarization inputs into a vision-language model using two visual branches and two-stage training. To support this, paper also introduces PolarVQA for understanding transparent objects and reflections. The reported results show consistent improvements over matched RGB-only models across several backbones and evaluation methods.

Critical strengths
  1. The paper is clearly structured and well motivated. Polarization offers a promising approach to scene understanding when reflections and transparency make RGB-based understanding difficult.

  2. The evaluation covers multiple backbones, different LLM judges, and human evaluation.

  3. Extensive experiments, including performance comparisons, attention analysis, computational cost, and ablation studies, help explain the contributions of the dual-encoder model and training strategy.

Critical weaknesses

The scene-level split procedure and the extent and results of manual dataset checks need clarification.

Questions for authors
  1. Is scene-level separation maintained between the training data for both stages and the test set? How many dataset annotations were manually checked, and what error rate was found?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 21o9: Scene-Level Splits and Human Verification (W1, Q1, Q2)

We thank the reviewer for the positive assessment. Our responses to W1, Q1, and Q2 are as follows.

Response to W1. We thank the reviewer for the suggestion and provide the scene-level split statistics and the human verification results, which confirm scene-level separation and a factual accuracy of 96.4% on the verified samples.

PolarVQA is split at the scene level, so all Stage 1 captions and Stage 2 QA pairs derived from the same scene belong to a single split. The training scenes come from the official training splits of PolarFree and RGBP-Glass, and the test scenes come from the official test splits, with no overlap between the two. The number of scenes in each split is given in Table R1.

Table R1. Scene-level split statistics. Each PolarFree sequence or RGBP-Glass capture is treated as a scene.

SplitUnique ScenesReflection ScenesGlass Scenes
Stage 1 / Stage 2 train2,979932,886
Stage 1 / Stage 2 val33010320
Test3847377

For the human verification, which was not presented in the submission, we randomly sample 300 of the 45.8K training and validation instruction-tuning pairs (limited to the five task types in Table 1) and 200 of the 1,000 test samples, stratified by task. Two annotators independently verify each sample against the original RGB image, the reflection-removed reference image, or the official segmentation mask, and label it as Correct (factually correct), Minor (factually correct, but the wording could be more precise), or Error (factually incorrect). The inter-annotator agreement on the independent labels before adjudication is 98.8% with a Cohen's κ of 0.90, and factually correct references account for 96.4%. Results are given in Table R2.

Table R2. Human verification of PolarVQA annotations. The task and split rows are two breakdowns of the same 500 samples.

BreakdownVerifiedCorrectMinorErrorFactual Acc. (C+M)
By task
Glass Count585800100.0%
Glass Loc.50444296.0%
Glass Desc.51463296.1%
Scene Desc.28627151096.5%
Refl. Recog.55474492.7%
By split
Train + Val (from 45.8K)300280101096.7%
Test (from 1,000)2001866896.0%
All500466161896.4%

Minor cases are mainly wording issues, such as a phrase referring to the reference image (which is not present in the clean version) while the reflected content is correct. A typical Error case is a Glass Desc. reference that describes a vertical display window as a horizontal surface. This error stems from the Florence-2-large description of the RGB image, where transparent glass overlapping its background misleads the orientation estimate, which is exactly the ambiguity that polarization addresses. On the 200 verified test samples, PolarVLM achieves a relative gain of 24.0% over RGB-only, in line with the 25.4% on the full test set. Both tables will be included in the appendix of the revision. Before the dataset release, we will further expand the scope of human verification.

Response to Q1. Yes, scene-level separation is maintained between the training data of both stages and the test set.

Both training stages use the same set of training scenes, while all 384 test scenes come from the official test splits and appear in neither the training nor the validation data of either stage. The statistics are given in Table R1 of our response to W1.

Response to Q2. The human verification covers 500 samples with an error rate of 3.6%.

We verified a total of 500 samples, 300 from the 45.8K training and validation instruction-tuning pairs (limited to the five task types in Table 1) and 200 from the 1,000 test samples, with a factual accuracy of 96.4% and an error rate of 3.6%. Per-task results are given in Table R2 of our response to W1. Before the dataset release, we will further expand the scope of human verification.

Reviewer_21o9OpenReview ↗

Response to authors

Thanks for the authors' reply, which addresses my concerns. I thus would like to maintain my rating.

We sincerely thank the reviewer for the positive assessment and for confirming that our responses have addressed the concerns. The scene-level split statistics and the human verification results will be incorporated into the revised paper. We are grateful for the reviewer’s support of this work and hope these additions will also be helpful in the subsequent discussion.

Reviewer_bjdpOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper address a real problem in current VLMs. RGB sensors only capture intensity and color, so models struggle with reflections and transparent surfaces. Polarization carries exactly the physical information that would resolve this, but it has never been integrated into a VLM.

PolarVLM does that. Raw polarization measurements are converted into a three-channel input of DoLP plus sin(2Φ) and cos(2Φ). The three-channel format means a second CLIP ViT-L/14 branch can be initialized from RGB weights rather than trained from scratch. The two streams are concatenated along the sequence dimension and the LLM's own self-attention handles the fusion. Training is two-stage: stage one bypasses RGB entirely so the polarization branch has to learn semantics on its own, and stage two activates both streams with LoRA.

They also build PolarVQA, a 75K benchmark of captions and instruction-tuning pairs over reflective and transparent scenes, generated from PolarFree and RGBP-Glass through an automated pipeline. Reported results: 6.08 overall against 4.85 for a matched RGB-only baseline, beating GPT-4.1 and cascaded polarimetric pipelines.

Critical strengths
  • S1) Architectural design decisios are explained, and the explanations hold up: The sin/cos fix for AoLP periodicity, the three-channel constraint, the frozen RGB anchor, the RGB bypass in stage one, are all motivated rather than asserted. I came away understanding why the model is built this way, which is rarer than it should be.

  • S2) The range of comparisons is top-notch: A controlled RGB-only model on identical data, cascaded pipelines on two backbones, open-weight and proprietary zero-shot models, then three judges, a judge-free protocol, human eval, four backbones, significance testing, and real ablations. That makes the core claim hard to dispute.

Critical weaknesses

My main concern is with the dataset, which is presented as one of the headline contributions.

  • W1) The data generation pipeline is too thin to evaluate: It's compressed into about three paragraphs in Appendix C, and the parameters that determine data quality aren't given. For example (not limited to these though), reflection regions are localized using "high DoLP and a substantial RGB domain difference" with no values and no sensitivity analysis, and Figure 5 mentions an unspecified area threshold. It may work fine in practice and authors may have domain knowledge that makes these judgements; I just can't judge the quality/soundness of the pipeline from what's written.

  • W2) Nothing in the pipeline validates annotation quality: The rule-based check filters formatting. The Verification Prompt in Figures 6 and 7 rewrites captions to strip colors and person details, so it checks for leaked RGB semantics rather than correctness, and it never rejects a sample. There's no human in the loop anywhere, which is fine for an automated pipeline, but then validation results have to carry the weight and there aren't any. The only B.3 human study evaluates model outputs, not the data. So no number anywhere tells me how accurate these 75K annotations are, which matters both for the results and for the dataset's value to the community.

  • W3) The counting metric rewards phrasing over accuracy: The judge rubric sets the band by whether the count is right, then splits 6-7 from 8-10 on "clear understanding." PolarVLM is tuned on PolarVQA's own caption style, so it writes answers that match that description, while a zero-shot model giving the right number concisely caps at 6-7. The rubric also rewards "a clear and definitive count rather than a vague or hedging answer," which penalizes GPT-4.1 for its alignment training. I was expecting a clear exact match or just a simple accuracy here.

  • W4) The localization metric also has a strange design: The rubric matches only broad regions and gives partial credit for "any reasonable attempt," even though the ground truth comes from segmentation masks with exact coordinates. Every model lands between 4.88 and 5.51 and PolarVLM's margin over Qwen2.5-VL is 0.13, so the column isn't separating models while still feeding the weighted Overall. Its first scoring criterion is also "glass count consistency," so counting gets measured twice across two columns that both roll up.

  • W5) Two benchmark tasks are missing from the main results: Figure 3 defines seven tasks; Table 1 reports five. Glass Detection and Counterfactual Reasoning are both absent without explanation. Counterfactual reasoning is highlighted in the introduction as a core contribution and has test data allocated to it, so this is surprising.

  • Minor: Figure 2 is just absolutely hard to read. There's a lot jammed in it and I had very hard time tracing the flow. I suggest simplifying the figure or just breaking it up into multiple figures for easier understanding.

Questions for authors
  • Q1) Is there any human-verified subset of PolarVQA with a reported accuracy rate? Even a few hundred samples checked by annotators, with an error rate and ideally inter-annotator agreement, would substantially raise confidence in both the results and the dataset release.

  • Q2) Why aren't Glass Detection and Counterfactual Reasoning in Table 1?

15 comments
AuthorsOpenReview ↗

Response to Reviewer bjdp (Part 1/4): Pipeline Parameters and Sensitivity Analysis (W1)

(Part 1/4) We thank the reviewer for the careful review. Our response consists of four parts, with Part 1 covering W1, Part 2 covering W2 and Q1, Part 3 covering W3 and W4, and Part 4 covering W5, Q2, and the minor comment on Figure 2.

Response to W1. We provide the complete pipeline parameters together with a sensitivity analysis, which shows that the reflection regions are insensitive to the DoLP threshold.

Fig. 5 and Appendix C of the paper describe the structure of the pipeline, and we list the parameters following its modules below. All thresholds are applied uniformly to the training, validation, and test data and were not tuned on the test set. The complete parameters will be included in Appendix C of the revision.

(a) Visual structure extraction. The area threshold mentioned in Fig. 5 refers to retaining only Florence-2-large bounding boxes covering 1% to 85% of the image area, where smaller boxes are treated as fragments and larger ones are absorbed into the scene-level description. Duplicate boxes are merged at an IoU of 0.5. Reflection-only objects are obtained from the detection difference between the observed image and the reflection-removed reference image, with the matching criterion being an IoU above 0.3, or a label overlap above 0.3 together with an IoU above 0.15. A candidate is accepted only when more than 20% of the pixels within its bounding box belong to the reflection regions defined in (b). Glass instances directly use the official annotations of RGBP-Glass.

(b) Physical quantity analysis for reflection regions. A pixel is labeled as reflection only when it satisfies two conditions. First, its DoLP exceeds 0.12, meaning that at least 12% of the incident light is linearly polarized. Second, its per-channel mean RGB difference from the reflection-removed reference image exceeds 20, which is about 8% of the dynamic range given the 0 to 255 range of 8-bit images. The DoLP threshold lies above the typical noise level of diffuse surfaces, and the difference threshold of 20 retains weak reflections on dark surfaces while excluding sensor noise. Connected regions smaller than 0.3% of the image are discarded as noise, and those reaching 2% are marked as salient reflections.

(c) Physical quantity analysis for glass polarization levels. Glass polarization is graded by the P90 of DoLP within each glass region, with values above 0.25 labeled strong, above 0.10 moderate, above 0.04 weak, and the remainder very weak. When the glass P90 exceeds twice the background P90, the glass is labeled as noticeably higher than background.

Sensitivity analysis. On 250 images from the PolarFree training set, we separately vary the DoLP threshold and the RGB difference threshold and compare against the default setting in terms of the IoU of the reflection regions and the proportion of images whose reflection-presence judgment flips.

Table R1. Sensitivity of reflection regions (IoU / flip rate).

DoLP \ RGB diff102030
0.080.42 / 13.2%0.85 / 2.4%0.53 / 14.4%
0.120.52 / 11.2%1.00 / 0%0.54 / 14.8%
0.160.57 / 7.6%0.81 / 4.4%0.47 / 19.2%

With the RGB difference threshold fixed at 20, varying the DoLP threshold between 0.08 and 0.16 keeps the IoU at or above 0.81 and the flip rate within 4.4%, showing that the results are insensitive to this parameter. The RGB difference threshold mainly controls the spatial extent of the reflection regions, and the default value of 20 balances the retention of weak reflections on dark surfaces against sensor noise, as described in (b). This table measures the stability of the reflection regions under threshold changes, while annotation accuracy is assessed by the human verification in our response to W2.

(Continued in Part 2/4.)

AuthorsOpenReview ↗

Response to Reviewer bjdp (Part 2/4): Human Verification of Annotations (W2, Q1)

(Part 2/4) This part addresses W2 and Q1.

Response to W2. We thank the reviewer for raising this point and add a human verification of the PolarVQA QA annotations, which shows a factual accuracy of 96.4% on the verified samples.

The Verification Prompts in Fig. 6 and Fig. 7 are designed to remove RGB semantics, and we complement them with a human verification, not presented in the submission, that directly assesses annotation correctness. We randomly sample 300 of the 45.8K training and validation instruction-tuning pairs (limited to the five task types in Table 1) and 200 of the 1,000 test samples, stratified by task. Two annotators independently verify each sample against the original RGB image, the reflection-removed reference image, or the official segmentation mask, and label it as Correct (factually correct), Minor (factually correct, but the wording could be more precise), or Error (factually incorrect).

Table R2. Human verification of PolarVQA annotations. The task and split rows are two breakdowns of the same 500 samples.

BreakdownVerifiedCorrectMinorErrorFactual Acc. (C+M)
By task
Glass Count585800100.0%
Glass Loc.50444296.0%
Glass Desc.51463296.1%
Scene Desc.28627151096.5%
Refl. Recog.55474492.7%
By split
Train + Val (from 45.8K)300280101096.7%
Test (from 1,000)2001866896.0%
All500466161896.4%

Factually correct references account for 96.4% of the verified samples, with 93.2% fully correct and another 3.2% factually correct but with wording that could be more precise. Since 300 of the verified samples are drawn from the training and validation splits, this accuracy covers the QA annotations used for training as well as the test references, with 96.7% on the training and validation portion and 96.0% on the test portion. The Glass Count answers are taken from the official instance counts, and all 58 are correct. The inter-annotator agreement on the independent labels before adjudication is 98.8% with a Cohen's κ of 0.90. On the 200 verified test samples, PolarVLM achieves a relative gain of 24.0% over RGB-only, in line with the 25.4% on the full test set. Minor cases are mainly wording issues, such as a phrase referring to the reference image (which is not present in the clean version) while the reflected content is correct. A typical Error case is a Glass Desc. reference that describes a vertical display window as a horizontal surface. This error stems from the Florence-2-large description of the RGB image, where transparent glass overlapping its background misleads the orientation estimate, which is exactly the ambiguity that polarization addresses. Table R2 will be included in the appendix of the revision. Before the dataset release, we will further expand the scope of human verification.

Response to Q1. Yes, a human-verified subset of 500 samples has a factual accuracy of 96.4% and a Cohen's κ of 0.90.

We verified 500 samples stratified by task, 300 drawn from the 45.8K training and validation instruction-tuning pairs (limited to the five task types in Table 1) and 200 from the test set, with two annotators independently verifying each sample. The factual accuracy is 96.4% and the error rate is 3.6%, with an inter-annotator agreement of 98.8% and a Cohen's κ of 0.90. Per-task results are given in Table R2 of our response to W2. Before the dataset release, we will further expand the scope of human verification.

(Continued in Part 3/4.)

AuthorsOpenReview ↗

Response to Reviewer bjdp (Part 3/4): Judge-Free Evaluation of Counting and Localization (W3, W4)

(Part 3/4) This part addresses W3 and W4.

Response to W3. We thank the reviewer for the attention to the evaluation metrics, and we show with two judge-free evaluations that the counting gains do not depend on phrasing.

We have provided a judge-free evaluation in Appendix B.2 of the paper, PolarVQA-MCQ, which may have been easy to overlook in the appendix. It consists of 1,000 four-option questions. Correct options are derived from the ground-truth annotations, distractors are constructed through perturbations such as altering the glass count and swapping spatial locations, and scoring uses exact letter-match accuracy without invoking any language model at evaluation time. The Glass Count column of Appendix Table 6 is reproduced below.

Table R3. Glass Count accuracy on PolarVQA-MCQ (from Appendix Table 6 of the paper). Δ% = (Polar − RGB) / RGB × 100.

ModelRGBPolarΔ%
LLaVA-1.5-7B38.9%58.2%+49.6%
LLaVA-1.5-13B46.2%69.3%+50.0%
Qwen3-VL-8B69.7%82.4%+18.2%
InternVL3.5-14B66.4%84.3%+27.0%

Without any influence of phrasing, adding polarization clearly improves counting accuracy on all four backbones. To further address the reviewer's concern regarding zero-shot models, we additionally apply exact matching to the open-ended answers of the models in Table 1, extracting the first number from each Glass Count answer and comparing it with the ground truth, again without any judge.

Table R4. Exact-match accuracy of open-ended Glass Count answers (162 samples, additional experiment).

ModelExact-match Acc.
BLIP2-Flan-T5-XL13.6%
InternVL3-14B40.7%
Qwen2.5-VL-7B42.6%
GPT-4.145.1%
LLaVA-1.5-13B (RGB-only, trained on PolarVQA)46.3%
Glass Seg. + LLaVA-1.5-13B14.2%
Glass Seg. + GPT-4.162.3%
PolarVLM (Ours)69.8%

PolarVLM reaches an exact-match accuracy of 69.8%, which is 24.7 percentage points above GPT-4.1 and 23.5 points above the RGB-only model trained on the same data. This is consistent with PolarVQA-MCQ, where the same pair of models attains 46.2% and 69.3%, a gap of 23.1 points. MCQ selects among four options while the open-ended setting requires the number to be stated directly, yet the two evaluations, though different in form, yield nearly identical gains. Since GPT-4.1 also trails PolarVLM by a large margin under exact matching, its lower rubric score is not caused by the preference for definitive phrasing. In the revision, we will cite the MCQ results in the main text and add the exact-match accuracy.

Response to W4. We thank the reviewer for the comment, and we show that localization is also evaluated by exact matching and that the conclusion does not depend on this column.

Localization questions are also scored by exact matching in PolarVQA-MCQ (Appendix B.2 of the paper), where distractors are constructed by swapping the spatial locations of glass and scoring uses exact letter-match accuracy without relying on the rubric. The Glass Loc. column of Appendix Table 6 is reproduced below.

Table R5. Glass Loc. accuracy on PolarVQA-MCQ (from Appendix Table 6 of the paper). Δ% = (Polar − RGB) / RGB × 100.

ModelRGBPolarΔ%
LLaVA-1.5-7B70.6%78.7%+11.5%
LLaVA-1.5-13B80.6%87.4%+8.4%
Qwen3-VL-8B90.5%96.1%+6.2%
InternVL3.5-14B90.3%95.8%+6.1%

On localization questions scored by exact matching, adding polarization improves accuracy on all four backbones. Since open-ended localization outputs natural language rather than segmentation results, and the reference answers are likewise textual descriptions derived from the masks, the rubric relies on broad spatial region matching and allows different anchor objects to describe the same location. The narrow spread across models in Table 1 reflects this tolerance for anchor choice, and PolarVLM still obtains the highest Glass Loc. score among all models in Table 1. Even so, in the controlled comparison trained on the same data, this column still separates PolarVLM from RGB-only, with the score rising from 4.88 to 5.51. The rubric includes glass count consistency because, in scenes with multiple pieces of glass, the location of each piece can be matched one to one only when the count is correct. This is a prerequisite for correct localization and does not double-count the counting score. Regarding the composition of the Overall score, removing Glass Loc. yields a relative gain of 26.9% for PolarVLM over RGB-only (6.15 versus 4.85), so the conclusion does not depend on this column.

(Continued in Part 4/4.)

Reviewer_csS2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PolarVLM, which adds a second vision encoder for polarization measurements to LLaVA-1.5, concatenates its tokens with the RGB tokens, and proposes a two stage training method for PolarVLM. It also builds PolarVQA, an automatically generated polarization-language dataset for reflective and transparent scenes. PolarVLM outperforms an RGB-only model trained on the same data and general-purpose VLMs including closed source models.

Critical strengths
  1. Polarization is a well-motivated modality for reflections and transparent objects, where RGB alone is ambiguous, and the physically motivated input encoding is simple and helps in the ablations.
  2. The evaluation is thorough. A controlled RGB-only baseline trained on the same data, cascaded polarization pipelines, several LLM judges, a judge-free multiple-choice test, human evaluation, and multiple backbones all point the same way.
  3. PolarVQA is a sizable polarization-language resource that should be useful to others, and the paper is open about the costs of the approach, including some loss on general benchmarks and higher latency.
Critical weaknesses
  1. The architectural contribution is limited. The dual-stream design and two-stage training are listed as a main contribution, but adding a second vision encoder to a multimodal LLM is a well-explored design, as in multi-encoder models such as EAGLE, that the paper itself cites, and the alignment-then-tuning schedule follows LLaVA. The novelty lies mainly in bringing polarization into this setting, through the input encoding and the data, and the paper would be more accurate framing its contribution that way.
  2. It is unclear whether the reflection-related questions actually require polarization. Scene-description answers come from the reflection-free RGB reference and can depend on color, and reflection labels are derived from RGB detections. On scene description, the polarization-only model, which has no color information, scores only slightly below the RGB-only model, which suggests that part of the score may come from language priors. A text-only baseline, and tagging each question by the modality needed to answer it, would show where the gains come from.
  3. The independence of the splits and the generality of the results need clarification. The paper states that the splits do not overlap but not whether they are separated by question, image, or scene, including across the two training stages, which matters because each scene yields several captions and questions. Reporting the number of unique scenes per split and confirming that test scenes appear in neither training stage would resolve this.
  4. The qualitative evaluation would benefit from clearer visualizations. Several figures are hard to read, with presentation issues in others for example Figure 12. larger examples that show where polarization resolves the ambiguity would make the benefit of the extra modality easier to see.
Questions for authors

See Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer csS2 (Part 1/3): Revised Contributions (W1)

(Part 1/3) We thank the reviewer for the constructive comments. Our response consists of three parts, with Part 1 covering W1, Part 2 covering W2, and Part 3 covering W3 and W4.

Response to W1. We thank the reviewer for the suggestion and agree that framing the contributions from this perspective is more accurate, so we revise them to focus on the polarimetric encoding, the training recipe, and the dataset.

The revision will present dual-stream concatenation as an implementation choice, and the revised contributions are as follows.

  • PolarVLM, the first multimodal framework integrating polarimetric physical parameters into VLMs for open-ended VQA under severe optical ambiguities.
  • A continuous polarimetric encoding and polarization-tailored alignment recipe to bridge the semantic-physical gap, preventing physical misinterpretations while largely retaining general visual abilities.
  • PolarVQA, the first benchmark for polarization-aware VQA, providing 75K physics-grounded captions and instruction-tuning pairs targeting reflective and transparent scenes.

The second contribution corresponds to two design choices. The first is the continuous encoding [DoLP, sin(2Φ), cos(2Φ)], which removes the discontinuity of AoLP at π and allows the polarization branch to be initialized directly from RGB CLIP weights. It achieves the highest Overall score among the four encodings compared in Table 11 of the submission. The second is bypassing the RGB branch in Stage 1, where only polarization tokens are fed into the LLM and the patch embedding of the polarization branch is trained to adapt to the distribution of polarimetric inputs. This differs in purpose from the first stage of LLaVA, which trains only the projector while the input remains RGB. Removing Stage 1 reduces the Overall score from 6.08 to 5.46 (Table 2).

(Continued in Part 2/3.)

AuthorsOpenReview ↗

Response to Reviewer csS2 (Part 2/3): Necessity of Polarization and Language Priors (W2)

(Part 2/3) This part addresses W2.

Response to W2. We thank the reviewer for this suggestion and add two analyses, grouping questions by the information required to answer them and a text-only baseline, together with a clarification of how reflection labels are generated, which show that the gains come from the polarimetric input rather than from language priors.

Grouping by required information. Polar-only below refers to the w/o RGB Branch variant in Table 2. Using a color vocabulary, we divide the 512 Scene Desc. questions into three groups, namely questions that require color to answer, questions that do not ask for color but whose reference answers mention it, and questions for which neither the question nor the reference answer involves color.

Table R1. Scene Desc. grouped by color dependence (GPT-4o-mini judge).

SubsetnPolar-onlyRGB-onlyPolarVLM
Color required1413.484.915.82
Color mentioned1273.874.134.75
No color2445.364.706.43
All5124.474.625.85

On questions that require color, RGB-only leads, which is consistent with the fact that polarization carries no color information. On the 244 questions that do not involve color, however, Polar-only outperforms RGB-only, 5.36 versus 4.70. The differences in the two groups go in opposite directions and cancel out in the overall average, which is why the two models obtain similar total scores. If the scores were driven mainly by language priors, such a reversal across question types would not occur. Moreover, even on questions that require color, PolarVLM improves from 4.91 to 5.82, because polarization suppresses color misjudgments caused by reflections, as in the guitar example in Fig. 4.

Text-only baseline. We feed only the question text without any image and score the same 512 questions. Results are shown in Table R2.

Table R2. Text-only baseline (512 Scene Desc. questions, GPT-4o-mini judge).

ModelWith imageText-onlyNo color, with imageNo color, text-only
RGB-only4.623.194.703.14
Polar-only4.473.445.363.52

Without images, both models score only 3.2 to 3.4. On questions that do not involve color, adding the polarization image raises Polar-only from 3.52 to 5.36, and this portion of the score comes from the polarization image itself rather than from language priors.

On reflection labels generated from RGB. The generation of reflection labels does use RGB, but the decision does not rest on RGB detections alone. The Florence-2-large detection difference only proposes candidate objects, while the reflection regions are determined jointly by the RGB difference and the DoLP. A candidate is retained only when more than 20% of the pixels within its bounding box satisfy both a per-channel mean RGB difference above 20 (8-bit, about 8% of the dynamic range) and a DoLP above 0.12, so the selection of reflected targets is constrained by polarimetric physical quantities. On the other hand, the reflection-removed reference image used to generate the labels is not available to any model at test time. An RGB-only model has to distinguish real objects from reflections in a single image containing reflections, where a single RGB image is ambiguous, and polarization supplies exactly this missing information. PolarVLM and RGB-only use the same labels, and the only difference in their inputs is the presence of polarization. Under this condition, Refl. Recog. improves from 3.23 to 4.09 (+26.6%). In the judge-free MCQ evaluation in Appendix B.2 of the paper, the accuracy of LLaVA-1.5-13B on this task also increases from 83.7% to 92.8% (Appendix Table 6). We will include this analysis in the revision.

(Continued in Part 3/3.)

AuthorsOpenReview ↗

Response to Reviewer csS2 (Part 3/3): Scene-Level Splits and Visualizations (W3, W4)

(Part 3/3) This part addresses W3 and W4.

Response to W3. PolarVQA is split at the scene level with no test scene in either training stage, and PolarVLM outperforms RGB-only on every reflection test sequence.

Section 5 of the paper states that the splits are non-overlapping, and here we further specify the split unit. All Stage 1 captions and Stage 2 QA pairs derived from the same scene belong to a single split. Both training stages use the same set of training scenes, all drawn from the official training splits of PolarFree and RGBP-Glass, while the 384 test scenes come from the official test splits and appear in neither the training nor the validation data of either stage. The number of scenes in each split is given in Table R3, which will be added to the revision.

Table R3. Scene-level split statistics. Each PolarFree sequence or RGBP-Glass capture is treated as a scene.

SplitUnique ScenesReflection ScenesGlass Scenes
Stage 1 / Stage 2 train2,979932,886
Stage 1 / Stage 2 val33010320
Test3847377

Regarding the generality of the results, the reflection test set comes from 7 sequences, comprising 139 distinct images and 592 QA pairs. In a per-sequence analysis, PolarVLM outperforms RGB-only on all 7 sequences, and on the two largest sequences, which together contain 434 QA pairs, the gains are 32.9% and 31.8%, respectively. In addition, Appendix Table 5 of the paper shows consistent gains of 17.1% to 20.0% on three additional backbones.

Response to W4. The qualitative evidence on how polarization resolves ambiguities is presented in the paper, and we will further improve the clarity of these figures.

Qualitative comparisons showing how polarization resolves the ambiguity are presented in Fig. 1, Fig. 4, and Appendix Fig. 11 of the paper, together with the corresponding DoLP and AoLP maps. Fig. 12 shows the Polarization Attention Ratio across the LLM decoder layers. On a complex facade with 32 pieces of glass, this ratio averages 68.7%, and it drops to 47.7% in a scene with only 3 pieces of glass and distinct boundaries, showing that the model relies more on polarization in more ambiguous scenes. The Δ in the figure marks the largest per-layer gap between the two curves, rather than the difference between their averages.

Reviewer_ZWJPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PolarVLM, a vision-language model that combines RGB images with polarization measurements to improve understanding of reflective and transparent scenes. The model uses separate visual branches, token concatenation, and a two-stage training procedure that first aligns polarization features and then performs joint instruction tuning. The authors also construct PolarVQA, containing approximately 75K captions and QA pairs. Experiments show improvements over RGB-only baselines across five tasks, with additional validation using multiple backbones, LLM judges, multiple-choice evaluation, and human assessment.

Critical strengths

1.The problem is well motivated. Reflective and transparent scenes expose genuine limitations of RGB-based perception. Introducing polarization measurements into VLMs is a meaningful direction, and the paper connects these physical cues to concrete language-based tasks.

2.PolarVQA is a useful resource contribution. The dataset provides both polarization-alignment captions and task-specific instruction pairs, covering complementary aspects of glass and reflection understanding. This could lower the barrier to studying polarization-aware VLMs, where suitable language supervision has been scarce.

3.The design choices have empirical support. The dedicated polarization branch and progressive training form a straightforward adaptation recipe. Removing Stage 1 reduces the overall score from 6.08 to 5.46 (Table 2), while Appendix G provides additional comparisons of fusion strategies and polarization encodings.

4.The evaluation goes beyond a single model or scoring method. Improvements persist across four backbones, three LLM judges, multiple-choice evaluation, and a 200-sample human study (Appendix B). This consistency makes the reported advantage more convincing than the main benchmark comparison alone.

Critical weaknesses

1.Reflection annotation quality remains insufficiently quantified. Appendix C and Figure 8 identify reflected content through differences between automatic detections and captions, which can also arise from recognition errors. Although manual spot-checks are mentioned, their size and error rates are missing, leaving uncertainty about the benchmark’s reference answers.

2.The source of the gains is not fully isolated. Removing the polarization branch also removes an encoder and additional visual tokens. Without a comparably trained RGB+RGB control, the experiments do not fully separate the contribution of polarization measurements from that of additional capacity and adaptation.

3.The split protocol needs clarification. The paper states that the splits are non-overlapping, but does not specify whether all captions and QA pairs from the same source image or scene stay together across both training stages. This makes test-set independence difficult to assess.

4.The conclusion about cascaded methods is broader than the comparison supports. Table 1 treats reflection recognition as unavailable after reflection removal. A cascade retaining both the original and processed images could preserve this information, so the reported limitation is not necessarily inherent to cascaded designs.

Questions for authors

1.Annotation validation: How many test-set reflection references were independently checked against the original and reflection-free images, what error rate was found, and do the gains persist on the verified subset?

2.Attribution control: How does PolarVLM compare with an RGB+RGB model using the same two-branch architecture, visual-token budget, and comparable training schedule?

3.Split clarification: Are the splits disjoint at the original-image or scene level across Stage 1 and Stage 2, including every caption and QA pair derived from each source?

4.Cascade comparison: How does a cascade receiving both the original RGB image and the predicted reflection-removed image perform on scene description and reflection recognition?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ZWJP (Part 1/3): Human Verification of Annotations (W1, Q1)

(Part 1/3) We thank the reviewer for the positive assessment and constructive comments. Our response consists of three parts, with Part 1 covering W1 and Q1, Part 2 covering W2, Q2, W3, and Q3, and Part 3 covering W4 and Q4.

Response to W1. We would like to first clarify that the detection difference only proposes candidates that must further satisfy physical constraints, and we then provide a human verification, which shows a factual accuracy of 96.4% on the verified samples.

Role of the detection difference. In the PolarVQA generation pipeline, the detection difference produced by Florence-2-large on the observed image and the reflection-removed reference image only proposes candidate objects, and whether a candidate is accepted as reflected content is decided jointly by two physical quantities. The first is the per-channel mean RGB difference between the observed image and the reflection-removed reference image, which must exceed 20 on the 8-bit scale (about 8% of the dynamic range). The second is the DoLP, which must exceed 0.12, meaning that at least 12% of the incident light is linearly polarized. A candidate is accepted only when more than 20% of the pixels in its bounding box satisfy both conditions, so detection differences without a supporting physical reflection signal are filtered out. When generating QA pairs, GPT-4o-mini compares the two captions and also receives this filtered list with the location and intensity of each reflection region, and is instructed to select the reflected target preferentially from it.

Quantitative assessment. We report a human verification not presented in the submission. We randomly sample 300 of the 45.8K training and validation instruction-tuning pairs (limited to the five task types in Table 1) and 200 of the 1,000 test samples, stratified by task. Two annotators independently verify each sample against the original RGB image, the reflection-removed reference image, or the official segmentation mask, and label it as Correct (factually correct), Minor (factually correct, but the wording could be more precise), or Error (factually incorrect). The inter-annotator agreement on the independent labels before adjudication is 98.8% with a Cohen's κ of 0.90, and factually correct references, namely Correct and Minor combined, account for 96.4%. Per-task results are reported in Table R1, which will be added to the appendix.

Table R1. Human verification of PolarVQA annotations. The task and split rows are two breakdowns of the same 500 samples.

BreakdownVerifiedCorrectMinorErrorFactual Acc. (C+M)
By task
Glass Count585800100.0%
Glass Loc.50444296.0%
Glass Desc.51463296.1%
Scene Desc.28627151096.5%
Refl. Recog.55474492.7%
By split
Train + Val (from 45.8K)300280101096.7%
Test (from 1,000)2001866896.0%
All500466161896.4%

Minor cases are mainly wording issues, such as a phrase referring to the reference image (which is not present in the clean version) while the reflected content is correct. A typical Error case is a Glass Desc. reference that describes a vertical display window as a horizontal surface. This error stems from the Florence-2-large description of the RGB image, where transparent glass overlapping its background misleads the orientation estimate, which is exactly the ambiguity that polarization addresses.

Response to Q1. The human verification shows an error rate of 3.6%, and the gains persist on the verified test samples (+24.0%).

We verified 500 samples stratified by task, 300 from the training and validation splits and 200 from the test set, with a factual accuracy of 96.4% and an error rate of 3.6% (Table R1). On the test portion alone, 192 of the 200 references are factually correct (96.0%), including 15 of the 16 Refl. Recog. references (93.8%) and 99 of the 103 Scene Desc. references (96.1%). On the 200 verified test samples, PolarVLM achieves a relative gain of 24.0% over RGB-only, in line with the 25.4% on the full test set. Before the dataset release, we will further expand the scope of human verification.

Table R2. Results on the 200 verified test samples and the full test set (GPT-4o-mini judge).

ModelnGlass CountGlass Loc.Glass Desc.Scene Desc.Refl. Recog.OverallRel. Gain
RGB-only2005.574.835.684.583.194.79--
PolarVLM2007.165.606.895.704.125.94+24.0%
RGB-only1,0005.654.885.774.623.234.85--
PolarVLM1,0007.575.516.875.854.096.08+25.4%

(Continued in Part 2/3.)

AuthorsOpenReview ↗

Response to Reviewer ZWJP (Part 2/3): RGB+RGB Capacity Control and Scene-Level Splits (W2, Q2, W3, Q3)

(Part 2/3) This part addresses W2, Q2, W3, and Q3.

Response to W2. We thank the reviewer for this suggestion and add a capacity-matched RGB+RGB control, which shows that the gains stem from polarization rather than from additional capacity.

The RGB+RGB model shares the same architecture as PolarVLM, with 576+576 visual tokens and identical trainable modules and hyperparameters in Stage 2. The only difference is that the second stream receives the same RGB image. Since the second stream already takes RGB input, it can directly inherit the pretrained RGB weights, and the polarimetric alignment of Stage 1 is therefore unnecessary.

Table R3. RGB+RGB capacity control (LLaVA-1.5-13B, 1,000 test samples). Polar+RGB denotes PolarVLM.

JudgeInputGlass CountGlass Loc.Glass Desc.Scene Desc.Refl. Recog.Overallvs. RGB
GPT-4o-miniRGB5.654.885.774.623.234.85--
GPT-4o-miniRGB+RGB5.895.055.864.652.944.92+1.4%
GPT-4o-miniPolar+RGB7.575.516.875.854.096.08+25.4%
GPT-5.5RGB5.704.935.824.703.304.92--
GPT-5.5RGB+RGB5.865.135.914.723.034.97+1.0%
GPT-5.5Polar+RGB7.655.566.935.904.166.13+24.6%
Claude Sonnet 4.5RGB5.735.015.784.663.264.91--
Claude Sonnet 4.5RGB+RGB5.915.125.844.672.994.94+0.6%
Claude Sonnet 4.5Polar+RGB7.625.586.925.864.156.11+24.4%

Across all three judges, RGB+RGB improves over the single-stream RGB baseline by only 0.6% to 1.4%, and its Refl. Recog. score even decreases slightly, whereas PolarVLM outperforms RGB+RGB by 23.3% to 23.7%. Even PolarVLM without Stage 1 (Overall 5.46, Table 2) remains clearly ahead of RGB+RGB (4.92). We will add this control to Table 4 in the revision.

Response to Q2. As reported in our response to W2, the gains are attributable to polarization rather than to additional capacity.

The setup and results are given in Table R3 of our response to W2. With the same architecture and visual-token budget, RGB+RGB improves over the single-stream RGB baseline by at most 1.4%, whereas PolarVLM outperforms RGB+RGB by 23.3% to 23.7%.

Response to W3. PolarVQA is split at the scene level, so no test scene appears in either training stage.

Section 5 of the paper states that the splits are non-overlapping, and here we further specify the split unit. All Stage 1 captions and Stage 2 QA pairs derived from the same scene belong to a single split, and both training stages use the same set of training scenes. The training scenes of the reflection and glass subsets come from the official training splits of PolarFree and RGBP-Glass, respectively, and the test scenes come from the official test splits, with no overlap between the two. The number of scenes in each split is given in Table R4, which will be added to the revision.

Table R4. Scene-level split statistics. Each PolarFree sequence or RGBP-Glass capture is treated as a scene.

SplitUnique ScenesReflection ScenesGlass Scenes
Stage 1 / Stage 2 train2,979932,886
Stage 1 / Stage 2 val33010320
Test3847377

Response to Q3. Yes, the splits are disjoint at the scene level between training and test.

All captions and QA pairs derived from the same scene belong to a single split, and the scene-level statistics are given in Table R4 of our response to W3. Stage 1 and Stage 2 intentionally share the same training scenes, first aligning polarimetric features and then performing instruction tuning, so the separation that matters is between training and test. All 384 test scenes come from the official test splits and appear in neither the training nor the validation data of either stage.

(Continued in Part 3/3.)

AuthorsOpenReview ↗

Response to Reviewer ZWJP (Part 3/3): Dual-Input Cascade Comparison (W4, Q4)

(Part 3/3) This part addresses W4 and Q4.

Response to W4. We thank the reviewer for the suggestion, and we clarify the cascade setting in Table 1 before reporting an additional dual-input cascade, which PolarVLM still outperforms.

The cascades in Table 1 were designed to evaluate reflection removal as a preprocessing step, so the VLM receives only the reflection-removed image, and reflection recognition is by construction not applicable in that setting. Following the reviewer's suggestion, we further evaluate a dual-input cascade that receives both the original RGB image and the PolarFree reflection-removed output. GPT-4.1 takes the two images through its native multi-image input. Since LLaVA-1.5 does not support multiple images, we concatenate the two images side by side. Both models are used zero-shot, with LLaVA using the official llava-v1.5-13b weights, and the evaluation setting is identical to that of Table 1.

Table R5. Comparison with dual-input cascades (GPT-4o-mini judge). Weighted denotes the sample-weighted average over the two tasks.

MethodScene Desc.Refl. Recog.Weighted
PolarFree + LLaVA-1.5-13B (Table 1)4.34N/A--
PolarFree + GPT-4.1 (Table 1)5.71N/A--
Original RGB & PolarFree + LLaVA-1.5-13B4.903.404.70
Original RGB & PolarFree + GPT-4.15.653.845.41
PolarVLM (Ours)5.854.095.61

With the original image retained, both cascades can perform reflection recognition, yet PolarVLM still outperforms the strongest dual-input cascade, GPT-4.1, on both scene description and reflection recognition. In the revision, Table 1 and Section 6.2 report the dual-input cascades.

Response to Q4. PolarVLM outperforms the dual-input cascade on both tasks.

The results are given in Table R5 of our response to W4. The dual-input GPT-4.1 cascade scores 5.65 on Scene Desc. and 3.84 on Refl. Recog., compared with 5.85 and 4.09 for PolarVLM.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.