- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper provides a model and a benchmark. PolarVLM integrates RGB and polarization inputs into a vision-language model using two visual branches and two-stage training. To support this, paper also introduces PolarVQA for understanding transparent objects and reflections. The reported results show consistent improvements over matched RGB-only models across several backbones and evaluation methods.
Critical strengths
-
The paper is clearly structured and well motivated. Polarization offers a promising approach to scene understanding when reflections and transparency make RGB-based understanding difficult.
-
The evaluation covers multiple backbones, different LLM judges, and human evaluation.
-
Extensive experiments, including performance comparisons, attention analysis, computational cost, and ablation studies, help explain the contributions of the dual-encoder model and training strategy.
Critical weaknesses
The scene-level split procedure and the extent and results of manual dataset checks need clarification.
Questions for authors
- Is scene-level separation maintained between the training data for both stages and the test set? How many dataset annotations were manually checked, and what error rate was found?
3 comments
Response to authors
Thanks for the authors' reply, which addresses my concerns. I thus would like to maintain my rating.
We sincerely thank the reviewer for the positive assessment and for confirming that our responses have addressed the concerns. The scene-level split statistics and the human verification results will be incorporated into the revised paper. We are grateful for the reviewer’s support of this work and hope these additions will also be helpful in the subsequent discussion.
Response to Reviewer 21o9: Scene-Level Splits and Human Verification (W1, Q1, Q2)
We thank the reviewer for the positive assessment. Our responses to W1, Q1, and Q2 are as follows.
Response to W1. We thank the reviewer for the suggestion and provide the scene-level split statistics and the human verification results, which confirm scene-level separation and a factual accuracy of 96.4% on the verified samples.
PolarVQA is split at the scene level, so all Stage 1 captions and Stage 2 QA pairs derived from the same scene belong to a single split. The training scenes come from the official training splits of PolarFree and RGBP-Glass, and the test scenes come from the official test splits, with no overlap between the two. The number of scenes in each split is given in Table R1.
Table R1. Scene-level split statistics. Each PolarFree sequence or RGBP-Glass capture is treated as a scene.
For the human verification, which was not presented in the submission, we randomly sample 300 of the 45.8K training and validation instruction-tuning pairs (limited to the five task types in Table 1) and 200 of the 1,000 test samples, stratified by task. Two annotators independently verify each sample against the original RGB image, the reflection-removed reference image, or the official segmentation mask, and label it as Correct (factually correct), Minor (factually correct, but the wording could be more precise), or Error (factually incorrect). The inter-annotator agreement on the independent labels before adjudication is 98.8% with a Cohen's κ of 0.90, and factually correct references account for 96.4%. Results are given in Table R2.
Table R2. Human verification of PolarVQA annotations. The task and split rows are two breakdowns of the same 500 samples.
Minor cases are mainly wording issues, such as a phrase referring to the reference image (which is not present in the clean version) while the reflected content is correct. A typical Error case is a Glass Desc. reference that describes a vertical display window as a horizontal surface. This error stems from the Florence-2-large description of the RGB image, where transparent glass overlapping its background misleads the orientation estimate, which is exactly the ambiguity that polarization addresses. On the 200 verified test samples, PolarVLM achieves a relative gain of 24.0% over RGB-only, in line with the 25.4% on the full test set. Both tables will be included in the appendix of the revision. Before the dataset release, we will further expand the scope of human verification.
Response to Q1. Yes, scene-level separation is maintained between the training data of both stages and the test set.
Both training stages use the same set of training scenes, while all 384 test scenes come from the official test splits and appear in neither the training nor the validation data of either stage. The statistics are given in Table R1 of our response to W1.
Response to Q2. The human verification covers 500 samples with an error rate of 3.6%.
We verified a total of 500 samples, 300 from the 45.8K training and validation instruction-tuning pairs (limited to the five task types in Table 1) and 200 from the 1,000 test samples, with a factual accuracy of 96.4% and an error rate of 3.6%. Per-task results are given in Table R2 of our response to W1. Before the dataset release, we will further expand the scope of human verification.