- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ICON is a learned text rewriter that conditions on the original question, answer options when present, and a frozen VLM's initial response, then issues one revised question to the same VLM. Gold labels are used offline to construct and filter rewrite targets; deployment uses a single rewrite without labels or candidate search. Matched question-and-options-only versus response-conditioned experiments test the value of the initial response. Further evaluations train separate rewriters on eight benchmarks and transfer a fixed Robo2VLM rewriter across frozen answerers on the same task.
Critical strengths
-
The matched QO/QOA comparison provides clear evidence for the main contribution. Adding the initial response improves Robo2VLM accuracy by 4.84 and 4.30 percentage points for the two rewriter sizes, beyond the question-and-options-only controls. This supports a specific empirical result about response conditioning rather than only the observation that question wording matters, and is the main reason I lean toward acceptance.
-
The method addresses the practical risk that rewriting can break initially correct answers through identity targets and fix-versus-break analysis. The distinction between offline label use and label-free deployment is explicit, and standard inference requires only one learned rewrite. These choices make the limited contribution practically interpretable.
Critical weaknesses
- Successful rewrites have not been independently shown to preserve the original requested judgment. Training proposals see the gold answer, while disclosure rules and black-image checks do not establish semantic equivalence. The paper acknowledges this limitation, and the current evidence does not establish leakage. A modest blinded audit would nevertheless strengthen the interpretation that the gains reflect better elicitation of the same visual judgment.
- The broader results support benchmark-specific usefulness rather than a universally transferable rewriter: Table 3 explicitly trains a separate rewriter for each benchmark, and several gains are small. Means over three seeds do not show uncertainty, which matters particularly for VSR's 0.20-point gain with the 8B rewriter. Appendix A establishes disjoint question IDs but leaves image/video grouping worth clarifying; actual contamination is not established. These qualifications limit the breadth of the claim without negating the main Robo2VLM improvement.
Questions for authors
- Focused validation of semantic fidelity: What fraction of a blinded sample of successful and unsuccessful deployed rewrites preserves the original requested judgment under independent assessment?
- Clarification and focused uncertainty analysis: Can the authors provide seed variation or paired uncertainty for the reported gains, especially the small gains, and clarify whether custom partitions separate shared images or videos as well as question IDs?