- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a training-free method for understanding image metaphors through perception, search, and reasoning. It tries to frame failures in understanding image implications as potentially arising from missing contextual knowledge. Let Androids Dream (LAD) extracts structured descriptions, acquires relevant context through parametric answering or web search, and uses this information to infer the image's implication. Experiments report improvements across four backbones and multiple benchmarks.
Critical strengths
1- The paper motivates and addresses a relevant problem: understanding image implications requires connecting visual content with cultural and contextual knowledge. The explicit separation of perception, context acquisition, and reasoning provides a useful structure for organizing and inspecting this process.
2- The pipeline improves the reported open-ended answer scores across the tested backbones and improves GPT-4o-mini on the general-VQA benchmarks in Table 4(b). These results narrow the performance gap with GPT-4o on MMMU and MMStar, while slightly exceeding GPT-4o on SeedBench.
Critical weaknesses
1- My main concern is that the largest English MCQ improvements occur on weaker baselines. In Table 4(a), LAD improves GPT-4o-mini by 30 percentage points and Qwen2.5-VL-7B by 18 points, compared with gains of 4 points for Qwen2.5-VL-72B and 6 points for GPT-4o. The paper acknowledges this pattern. However, although newer models are included as baselines, LAD itself is not applied to them. It remains unclear whether the pipeline provides substantial additional value when the underlying model already has stronger perception, reasoning, and contextual knowledge. Evaluating LAD on a stronger recent backbone under a consistent, verified setup would help clarify this.
2- I also have concerns about the methodological novelty. LAD combines prompting, question generation, retrieval, ranking, and final reasoning, with its web-search component building on MindSearch. I find this structure useful, but I am still not fully convinced what the distinct methodological contribution is. In particular, the individual benefits of composition-guided queries and implication-aware selection are not isolated in the experiments. Could the authors clarify the technical contribution and provide targeted comparisons that establish the benefits of these design choices?
3- Table 3(b) provides useful search-enabled comparisons involving Grok-3, GPT-4o, and GPT-4o-mini, which help assess the contribution of search. However, in the full-benchmark results in Table 6, LAD uses web search while the other systems remain closed-book. This makes it difficult to separate the benefits of the proposed pipeline from those of additional information access. Could the authors extend the Stages I + III ablation to the full benchmarks in Table 6? This would help establish how much of the improvement comes from structured perception and reasoning, and how much is added by the search stage.
Questions for authors
-
Does LAD improve a stronger, more recent backbone (Like Gemini) when evaluated with and without LAD under the same model version and evaluation protocol?
-
Does LAD retain its advantage when competing pipelines have comparable web access and inference budgets?
-
The formulation would benefit from clearer positioning relative to prior work on polysemous vision-language understanding, such as PVSE [1]. How does the proposed contextual-alignment formulation relate to these approaches?
[1] Yale Song and Mohammad Soleymani. “Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
2 comments
Reminder: revised PDF uploaded for the discussion period
We thank Reviewer aD4U for the detailed review. The revised PDF is now uploaded, with all additions marked in blue.
W1 and Q1, a stronger backbone. Table 5(a) now reports Gemini-3.0-Pro with and without LAD, under the same prompts and the same OSQ judge as every other row: MCQ 76%/76% to 82%/78%, OSQ 3.82/3.96 to 4.30/4.46. The English MCQ gain does narrow as closed-book accuracy rises, +4 to +6 for the three backbones at 72-76%, and Chinese gains follow the same ceiling pattern, reaching +2 at 76%. OSQ improves on all five backbones, by +0.30 to +2.06.
Q3, positioning against PVSE. Section 2.2 now contrasts PVSE, which learns K embeddings per instance at training time and uses no external knowledge, with LAD's construction of context at test time from the perceived composition.
W3, closed-book comparison. Table 5(b) adds a closed-book Stages I+III row on the general benchmarks, 62.4/74.5/57.8 against the 59.4/72.8/54.8 base, with Stage II adding a further +5.5/+2.7/+2.5. Section 4.4.4 now states which of the two is a controlled comparison and which is not.
We will add a search-and-answer baseline using the same backbone, the same search engine and a comparable budget, together with separate ablations of composition-guided against entity-only queries and of implication-aware ranking against unranked selection, to the final version.
We would welcome your view on whether the new results change your assessment.
If you have any further questions, please feel free to let us know.
Response to Reviewer aD4U
We thank Reviewer aD4U for the constructive review. We appreciate that you found the problem relevant, the split into perception, context acquisition, and reasoning a useful structure for organizing and inspecting the process, and the gains on open-ended answers across backbones and on general VQA, where LAD narrows the gap to GPT-4o and slightly exceeds it on SeedBench.
New results for your points are in the General Response above: Table B (Gemini-3.0-Pro, W1/Q1), Table A (Stages I+III on general VQA, W3), and Table C (LAD without web search, W2). We answer each point below.
W1 & Q1: Evaluating LAD on a stronger recent backbone (Gemini-3.0-Pro). We ran full LAD on Gemini-3.0-Pro (a primary Table 2 baseline) on S100 with the same Appendix L prompts and
gpt-4o-2024-11-20OSQ judge (General Response, Table B). MCQ rises from 76% / 76% to 82% (+6) / 78% (+2) and OSQ rises from 3.82 / 3.96 to 4.30 (+0.48) / 4.46 (+0.50), with the closed-book row matching Table 2. Across all five backbones in Table B, English MCQ gains narrow at 72%–76% closed-book accuracy (+4 to +6), likely because stronger models already rule out more distractors. OSQ improves on every backbone (+0.30 to +2.06). We will add these rows to Table 4(a), Figure 4(c), and Section 4.4.3 in the revised version.W2: Methodological contribution vs. MindSearch and query/selection ablations. LAD credits MindSearch for WebSearch (Table 1, Section 3.3). Our contribution is test-time latent-context construction (c^=S(R(Q(z)))) for compositional visual metaphors. We will add targeted ablations comparing composition-guided against entity-only queries and implication-aware ranking against unranked selection to the revised version. Unlike entity-lookup RAG, LAD combines three designs:
Adding web access through routing lifts Qwen2.5-VL-7B from 56.7% / 44.7% to 64% / 46% (Table C). In Table 3(b), with GPT-4o-mini and Stages I+III fixed, OpenAI native search raises MCQ over LAD-CoT (68 / 44 → 72 / 48) but lowers OSQ (3.84 / 3.58 → 3.62 / 3.34). LAD-Search reaches 74 / 52 and 4.02 / 3.66. We will clarify this boundary in Sections 1 and 3.3 of the revised version.
W3: Separating Stages I+III from Stage II on larger benchmarks. We will add the closed-book Stages I+III ablation on the full II-Bench and CII-Bench benchmarks in Table 6 to the revised version. On MMMU, SeedBench, and MMStar (General Response, Table A), closed-book Stages I+III improves GPT-4o-mini on all three datasets (+3.0 / +1.7 / +3.0). Stage II adds +5.5 / +2.7 / +2.5 on top, where retrieved domain facts may contribute on knowledge-intensive items (Appendix I). On full II-Bench (n=1,399) and CII-Bench (n=800), full LAD improves GPT-4o-mini by +17.7 / +18.2 MCQ and +1.29 / +1.02 OSQ (Table 6). We will also add Table A to Table 4(b) in the revised version.
Q2: Comparable web access and inference budgets. Under live web search on the same backbones (Table 3(b)), generic search raises MCQ in five of six cases. LAD achieves higher OSQ in both same-backbone comparisons: GPT-4o + LAD gets 4.14 / 4.26 vs. 2.88 / 3.28 for Perplexity (80% / 66% MCQ tie), and GPT-4o-mini + LAD-Search gets 74% / 52% MCQ and 4.02 / 3.66 OSQ vs. 72% / 48% and 3.62 / 3.34 for OpenAI native search with Stages I+III fixed. Grok-3's built-in search leaves OSQ flat (3.24 → 3.25, 2.96 → 2.92). Stage II takes 15–45 s (~860–1,070 search-stage tokens/image) plus 3 Stage I+III calls (Appendix C). Commercial APIs use different indices and hide retrieval tokens, so we will add a matched DuckDuckGo baseline in Section 4.4.2 of the revised version.
Q3: Positioning relative to PVSE. PVSE (Song & Soleymani, CVPR 2019) learns K embeddings per instance at training time to handle polysemy in cross-modal retrieval. It does not use external knowledge. Visual metaphors instead require unseen cultural conventions c to infer one intended implication y from element compositions (Section 3.1). LAD constructs an explicit context c^=S(R(Q(z))) at test time on a frozen MLLM from parametric and web sources before reasoning pθ(y∣x,z,c^). We will add PVSE to Section 2.2 in the revised version.
If you have any further questions, please feel free to let us know.