acceptodds
Under review as a conference paper at ICLR 2027

Let Androids Dream: From Visual Elements to Emergent Implications via Contextual Alignment

Abstract

Image metaphor understanding differs from general VQA: the implied meaning is rarely explicit in pixels and instead emerges from the compositional interaction of visual elements together with social intent and culture-specific background. We identify missing context – the gap between surface perception and the external knowledge required to infer emergent implications – as a failure mode of current multimodal large language models (MLLMs). We formalize Contextual Alignment as inference over a latent context: selecting background knowledge that is relevant to the image's composition and useful for inferring its implication. We then propose Let Androids Dream (LAD), a training-free framework that operationalizes contextual alignment through test-time computation: (1) Perception extracts compositional visual structure, (2) Search acquires missing context via perception-guided retrieval, and (3) Reasoning aligns compositions with context to derive emergent implications. On S100, a benchmark of 100 high-level implication questions in English and Chinese, LAD raises the lightweight GPT-4o-mini from 44% to 74% (English) and from 42% to 52% (Chinese) Multiple-Choice Question (MCQ) accuracy, on par with the closed-book GPT-5.4 in English, and from 2.98 to 4.02 and from 3.36 to 3.66 on Open-Style Questions (OSQ); with GPT-4o, LAD obtains the highest OSQ scores among directly comparable systems (4.14 and 4.26). LAD improves all four backbones we test, including the open-source Qwen2.5-VL-7B (+18 MCQ and +1.30 OSQ points in English), and its gains hold on the full II-Bench and CII-Bench. Adding other search tools raises MCQ accuracy in five of six cases but changes OSQ scores by to , whereas LAD's search stage improves both, suggesting that how context is acquired matters beyond access to search. Our OSQ judge is within one point of trimmed human means for 95.7% of answers, and LAD with web search also improves MMMU, SeedBench, and MMStar.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Multimodal Reasoning, Image Metaphor Understanding, Visual Commonsense Reasoning, Tool-augmented Reasoning, Theory of Mind, Social Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aD4UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a training-free method for understanding image metaphors through perception, search, and reasoning. It tries to frame failures in understanding image implications as potentially arising from missing contextual knowledge. Let Androids Dream (LAD) extracts structured descriptions, acquires relevant context through parametric answering or web search, and uses this information to infer the image's implication. Experiments report improvements across four backbones and multiple benchmarks.

Critical strengths

1- The paper motivates and addresses a relevant problem: understanding image implications requires connecting visual content with cultural and contextual knowledge. The explicit separation of perception, context acquisition, and reasoning provides a useful structure for organizing and inspecting this process.

2- The pipeline improves the reported open-ended answer scores across the tested backbones and improves GPT-4o-mini on the general-VQA benchmarks in Table 4(b). These results narrow the performance gap with GPT-4o on MMMU and MMStar, while slightly exceeding GPT-4o on SeedBench.

Critical weaknesses

1- My main concern is that the largest English MCQ improvements occur on weaker baselines. In Table 4(a), LAD improves GPT-4o-mini by 30 percentage points and Qwen2.5-VL-7B by 18 points, compared with gains of 4 points for Qwen2.5-VL-72B and 6 points for GPT-4o. The paper acknowledges this pattern. However, although newer models are included as baselines, LAD itself is not applied to them. It remains unclear whether the pipeline provides substantial additional value when the underlying model already has stronger perception, reasoning, and contextual knowledge. Evaluating LAD on a stronger recent backbone under a consistent, verified setup would help clarify this.

2- I also have concerns about the methodological novelty. LAD combines prompting, question generation, retrieval, ranking, and final reasoning, with its web-search component building on MindSearch. I find this structure useful, but I am still not fully convinced what the distinct methodological contribution is. In particular, the individual benefits of composition-guided queries and implication-aware selection are not isolated in the experiments. Could the authors clarify the technical contribution and provide targeted comparisons that establish the benefits of these design choices?

3- Table 3(b) provides useful search-enabled comparisons involving Grok-3, GPT-4o, and GPT-4o-mini, which help assess the contribution of search. However, in the full-benchmark results in Table 6, LAD uses web search while the other systems remain closed-book. This makes it difficult to separate the benefits of the proposed pipeline from those of additional information access. Could the authors extend the Stages I + III ablation to the full benchmarks in Table 6? This would help establish how much of the improvement comes from structured perception and reasoning, and how much is added by the search stage.

Questions for authors
  1. Does LAD improve a stronger, more recent backbone (Like Gemini) when evaluated with and without LAD under the same model version and evaluation protocol?

  2. Does LAD retain its advantage when competing pipelines have comparable web access and inference budgets?

  3. The formulation would benefit from clearer positioning relative to prior work on polysemous vision-language understanding, such as PVSE [1]. How does the proposed contextual-alignment formulation relate to these approaches?

[1] Yale Song and Mohammad Soleymani. “Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

2 comments
AuthorsOpenReview ↗

Response to Reviewer aD4U

We thank Reviewer aD4U for the constructive review. We appreciate that you found the problem relevant, the split into perception, context acquisition, and reasoning a useful structure for organizing and inspecting the process, and the gains on open-ended answers across backbones and on general VQA, where LAD narrows the gap to GPT-4o and slightly exceeds it on SeedBench.

New results for your points are in the General Response above: Table B (Gemini-3.0-Pro, W1/Q1), Table A (Stages I+III on general VQA, W3), and Table C (LAD without web search, W2). We answer each point below.

W1 & Q1: Evaluating LAD on a stronger recent backbone (Gemini-3.0-Pro). We ran full LAD on Gemini-3.0-Pro (a primary Table 2 baseline) on S100 with the same Appendix L prompts and gpt-4o-2024-11-20 OSQ judge (General Response, Table B). MCQ rises from 76% / 76% to 82% (+6) / 78% (+2) and OSQ rises from 3.82 / 3.96 to 4.30 (+0.48) / 4.46 (+0.50), with the closed-book row matching Table 2. Across all five backbones in Table B, English MCQ gains narrow at 72%–76% closed-book accuracy (+4 to +6), likely because stronger models already rule out more distractors. OSQ improves on every backbone (+0.30 to +2.06). We will add these rows to Table 4(a), Figure 4(c), and Section 4.4.3 in the revised version.

W2: Methodological contribution vs. MindSearch and query/selection ablations. LAD credits MindSearch for WebSearch (Table 1, Section 3.3). Our contribution is test-time latent-context construction (c^=S(R(Q(z)))\hat{c} = S(R(Q(z)))) for compositional visual metaphors. We will add targeted ablations comparing composition-guided against entity-only queries and implication-aware ranking against unranked selection to the revised version. Unlike entity-lookup RAG, LAD combines three designs:

  1. Composition-guided queries Q(z)Q(z) from scene elements, emotion, domain, rhetoric, and puns.
  2. An explicit context c^\hat{c} that can be inspected, removed, or replaced (Table 1).
  3. Implication-aware ranking and refinement SS by relevance to the implied metaphor.

Adding web access through routing lifts Qwen2.5-VL-7B from 56.7% / 44.7% to 64% / 46% (Table C). In Table 3(b), with GPT-4o-mini and Stages I+III fixed, OpenAI native search raises MCQ over LAD-CoT (68 / 44 → 72 / 48) but lowers OSQ (3.84 / 3.58 → 3.62 / 3.34). LAD-Search reaches 74 / 52 and 4.02 / 3.66. We will clarify this boundary in Sections 1 and 3.3 of the revised version.

W3: Separating Stages I+III from Stage II on larger benchmarks. We will add the closed-book Stages I+III ablation on the full II-Bench and CII-Bench benchmarks in Table 6 to the revised version. On MMMU, SeedBench, and MMStar (General Response, Table A), closed-book Stages I+III improves GPT-4o-mini on all three datasets (+3.0 / +1.7 / +3.0). Stage II adds +5.5 / +2.7 / +2.5 on top, where retrieved domain facts may contribute on knowledge-intensive items (Appendix I). On full II-Bench (n=1,399n = 1{,}399) and CII-Bench (n=800n = 800), full LAD improves GPT-4o-mini by +17.7 / +18.2 MCQ and +1.29 / +1.02 OSQ (Table 6). We will also add Table A to Table 4(b) in the revised version.

Q2: Comparable web access and inference budgets. Under live web search on the same backbones (Table 3(b)), generic search raises MCQ in five of six cases. LAD achieves higher OSQ in both same-backbone comparisons: GPT-4o + LAD gets 4.14 / 4.26 vs. 2.88 / 3.28 for Perplexity (80% / 66% MCQ tie), and GPT-4o-mini + LAD-Search gets 74% / 52% MCQ and 4.02 / 3.66 OSQ vs. 72% / 48% and 3.62 / 3.34 for OpenAI native search with Stages I+III fixed. Grok-3's built-in search leaves OSQ flat (3.24 → 3.25, 2.96 → 2.92). Stage II takes 15–45 s (~860–1,070 search-stage tokens/image) plus 3 Stage I+III calls (Appendix C). Commercial APIs use different indices and hide retrieval tokens, so we will add a matched DuckDuckGo baseline in Section 4.4.2 of the revised version.

Q3: Positioning relative to PVSE. PVSE (Song & Soleymani, CVPR 2019) learns KK embeddings per instance at training time to handle polysemy in cross-modal retrieval. It does not use external knowledge. Visual metaphors instead require unseen cultural conventions cc to infer one intended implication yy from element compositions (Section 3.1). LAD constructs an explicit context c^=S(R(Q(z)))\hat{c} = S(R(Q(z))) at test time on a frozen MLLM from parametric and web sources before reasoning pθ(y∣x,z,c^)p_\theta(y \mid x, z, \hat{c}). We will add PVSE to Section 2.2 in the revised version.

If you have any further questions, please feel free to let us know.

AuthorsOpenReview ↗

Reminder: revised PDF uploaded for the discussion period

We thank Reviewer aD4U for the detailed review. The revised PDF is now uploaded, with all additions marked in blue.

W1 and Q1, a stronger backbone. Table 5(a) now reports Gemini-3.0-Pro with and without LAD, under the same prompts and the same OSQ judge as every other row: MCQ 76%/76% to 82%/78%, OSQ 3.82/3.96 to 4.30/4.46. The English MCQ gain does narrow as closed-book accuracy rises, +4 to +6 for the three backbones at 72-76%, and Chinese gains follow the same ceiling pattern, reaching +2 at 76%. OSQ improves on all five backbones, by +0.30 to +2.06.

Q3, positioning against PVSE. Section 2.2 now contrasts PVSE, which learns K embeddings per instance at training time and uses no external knowledge, with LAD's construction of context at test time from the perceived composition.

W3, closed-book comparison. Table 5(b) adds a closed-book Stages I+III row on the general benchmarks, 62.4/74.5/57.8 against the 59.4/72.8/54.8 base, with Stage II adding a further +5.5/+2.7/+2.5. Section 4.4.4 now states which of the two is a controlled comparison and which is not.

We will add a search-and-answer baseline using the same backbone, the same search engine and a comparable budget, together with separate ablations of composition-guided against entity-only queries and of implication-aware ranking against unranked selection, to the final version.

We would welcome your view on whether the new results change your assessment.

If you have any further questions, please feel free to let us know.

Reviewer_2a3cOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

It is challenging for multimodal models to understand image metaphor due to the lack of social intent and culture-specific background knowledge needed given the visual elements. To mitigate the issue, the paper propose Let Androids Dream (LAD), a training-free framework that involves three steps: (1) prompt the model to generate one-paragraph coarse-grained visual description and condense it into seven keywords, (2) use the keywords to generate search questions (for which the model can either answer with its own parametric knowledge or use web search tool and choose to adopt the answer with self-reported confidence score, rank questions by their relevance to the image description, and generate a summary of the search result), and (3) perform reasoning and generate the output given the summary. The method outperforms multiple baselines on 100 high0level implication questions in english and Chinese.

Critical strengths
  • The paper proposes a cheap, training-free framework to prompt model to self-judge and leverage web search tool for better answering high-level implication VQA and show promising improvement across multiple baselines and model families.
  • The paper has thorough ablation studies and analyses to show significance in the improvement.
Critical weaknesses
  • The core of the method seems to not be always as effective as the introduction claimed: the core is claimed to be the planning and searching part that enable filling in missing context. However, the ablation in table 2 (removing II) can increase the metric score from 44 to 68, showing that standard CoT or maybe just even better prompting could result in better results. Similar in table 3. Unclear of whether the paper tries to optimize the prompt
  • Related to the previous bullet point: the paper should have more finegrained ablation on different choices made in Stage II to support its main claim. Stage II includes both self-judging, planning, answering, and external search. Given the introduction claims that the paper focuses on mitigating lack of external knowledge that cannot be accessed by the model itself / via CoT, there should be experiment that ablates the external search only for obtaining external knowledge to support the claim, otherwise it’s possible that the model itself can still recall sufficient knowledge via better prompting.
Questions for authors

Same as the two weaknesses stated above: would want to see more results to show that external search is indeed necessary over simple prompt optimization / CoT reasoning without external search to support the main goal and claim made in the introduction of the paper.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2a3c

We thank Reviewer 2a3c for the constructive review. We appreciate that you found LAD a cheap, training-free framework that improves multiple baselines and model families, and our ablations and analyses thorough in showing the significance of the gains.

The ablations you asked for, a single-call metaphor prompt and LAD without web search, are in the General Response above (Table C), and the stage-wise general-VQA ablation is in Table A. We answer each point below.

W1 & Q1: Missing-context framing, CoT baselines, and prompt optimization. You are right that Stages I+III gives a large closed-book gain on S100 English MCQ (44% → 68% before Stage II reaches 74%). Our Abstract, Introduction, and Section 3.3 phrased missing context too narrowly as missing external knowledge. In the revised version, we will align them with Section 3.1: missing context covers both unused parametric conventions and absent external facts, with web search as an adaptive component.

  1. CoT and reasoning baselines vs. Stages I+III: On S100 (Table 3(a)), generic Std CoT moves GPT-4o-mini from 44% / 42% MCQ and 2.98 / 3.36 OSQ to 50% / 42% and 3.10 / 3.28. LAD-CoT (Stages I+III) reaches 68% / 44% MCQ and 3.84 / 3.58 OSQ. Better prompting alone does not close this gap: on Qwen2.5-VL-7B, putting the same metaphor cues into one call gives 43.3% / 42.0%, no better than Std CoT (46.7% / 42.0%), while LAD-CoT reaches 54.7% / 44.0% (Table C). In Table 5, the reasoning variant of Grok-3 raises English MCQ (66% → 74%) but not OSQ (3.24 / 2.96 → 3.06 / 2.92).
  2. Incremental gains from Stage II: Stage II improves all four metrics over Stages I+III on GPT-4o-mini (Table 3(b)). MCQ rises 68% → 74% and 44% → 52%, and OSQ 3.84 / 3.58 → 4.02 / 3.66. Stage II also adds +5.5 / +2.7 / +2.5 over closed-book Stages I+III on MMMU, SeedBench, and MMStar (General Response, Table A).
  3. Prompt optimization: All prompt templates (Appendix L) were written once without tuning on S100 items and reused unchanged across all backbones and benchmarks (Appendix A), including Gemini-3.0-Pro (Table B). We will state this explicitly in Section 4.1 of the revised version.

W2 & Q1: Ablating external search vs. parametric recall, and Stage-II subcomponents. To isolate external web search from parametric recall, we ran LAD with Stage II restricted to ModelSearch-only on Qwen2.5-VL-7B (General Response, Table C). We will also add separate ablations of question planning and implication-aware ranking within Stage II to the revised version.

  1. LAD without external search vs. full LAD (General Response, Table C): Without web access, ModelSearch-only gives 56.7% / 44.7% on S100 MCQ, 2.0 / 0.7 points above closed-book Stages I+III (54.7% / 44.0%). Full LAD with web search reaches 64% / 46% (+9.3 / +2.0 over Stages I+III). On Qwen2.5-VL-7B, the gain over the base model (46% / 40%) without web access comes from Stages I+III, and external search adds context that parametric recall does not provide.
  2. Adaptive Self-Judge routing (Section 3.3, Appendix L): Self-Judge scores each planned question from 1 to 5 and sends it to WebSearch only when score ≤3\le 3 and decision = WebSearch (otherwise ModelSearch). The rubric assigns scores below 3 for internet/meme culture, highly popular knowledge, a large share of real-time content, niche entities, or more than two entities. This directs meme-specific, time-sensitive, or niche references to web search.
  3. Structured Stage II vs. generic web search (Table 3(b)): Fixing GPT-4o-mini and Stages I+III, OpenAI native search gets 72% / 48% MCQ and lowers OSQ to 3.62 / 3.34. LAD-Search reaches 74% / 52% MCQ and 4.02 / 3.66 OSQ (+0.40 / +0.32).

In the revised version, we will add Table C and the planning/ranking ablations to Section 4.4.1 and Section 4.4.2.

If you have any further questions, please feel free to let us know.

AuthorsOpenReview ↗

Update: prompting alone vs. LAD stages and web search (W1, W2 & Q1)

We have added Stages I+III and a single-call metaphor prompt to Table C in the General Response (2/2), all on Qwen2.5-VL-7B. Putting the metaphor cues into one call gives 43.3% / 42.0%, no better than Std CoT (46.7% / 42.0%), while LAD-CoT reaches 54.7% / 44.0%. Without web access, Stage II reaches 56.7% / 44.7%, 2.0 / 0.7 points above Stages I+III, and full LAD with web search reaches 64% / 46%. So better prompting alone does not recover LAD's gain, and external search supplies most of the Stage-II gain. Our replies to W1 and W2 are updated accordingly.

If you have any further questions, please feel free to let us know.

AuthorsOpenReview ↗

Reminder: revised PDF uploaded for the discussion period

We thank Reviewer 2a3c for the careful reading. The revised PDF is now uploaded, with all additions marked in blue. Both of your points are addressed in the main paper.

W1, prompting versus the staged design. New Table 4 (Section 4.4.1) separates the two on one backbone. Putting LAD's emotion, domain and rhetorical-device instructions into a single call, with the output cap matched to LAD-CoT, reaches 43.3%/42.0%, which is not above Std CoT (46.7%/42.0%), while running the same instructions as separate stages reaches 54.7%/44.0%. Section 4.1 now also states that every prompt template was written once, without tuning on the S100 items, and reused unchanged across all backbones and benchmarks.

W2, ablating external search on its own. The same table isolates it. With every stage, prompt and decoding setting unchanged, answering each planned Stage-II question from the model's own parameters reaches 56.7%/44.7%, and routing to web search reaches 64%/46%. Web access therefore supplies 7.3 of the 9.3 English points that Stage II contributes, and parametric recall supplies 2.0.

Following your first point, we have also aligned the Abstract, Introduction and Section 3.3 with the Section 3.1 definition of missing context, which covers both conventions already in the weights that single-pass inference leaves unused and facts absent from them.

We will add separate ablations of question planning and of implication-aware ranking within Stage II to the final version.

We would be glad to know whether this resolves your concerns.

If you have any further questions, please feel free to let us know.

Reviewer_HR7yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Let Androids Dream (LAD), a training-free framework for visual metaphor understanding. The framework extracts image descriptions and metaphor-related keywords, acquires contextual knowledge through model-based or web search, and combines this information with the original image to infer its meaning. Experiments on English and Chinese benchmarks show improvements across multiple MLLM backbones, with additional results on general visual question answering tasks. Strengths

Critical strengths
  • The paper addresses an interesting problem, as understanding visual metaphors often requires cultural and contextual knowledge beyond recognizing visible objects.
  • The framework is conceptually intuitive, and the intermediate descriptions, queries, and retrieved information make the process easy to inspect.
  • Experiments on four backbones and substantial improvements on the full II-Bench and CII-Bench provide evidence for the framework’s effectiveness.
  • The evaluation includes multiple-choice and open-ended questions, stage-wise ablations, search baselines, and human validation of the automatic evaluator.
Critical weaknesses
  • The contribution of metaphor-specific prompting is not sufficiently isolated. LAD-CoT introduces image descriptions, keyword extraction, targeted instructions on emotion, domain, and rhetorical devices, and additional model calls. The comparison with standard CoT does not separate these factors. A stronger baseline that directly incorporates the same metaphor-analysis instructions, together with comparisons under matched inference budgets, would help clarify the contribution of the proposed decomposition.
  • The advantage of the full pipeline over simpler search augmentation is not fully established. On S100, GPT-4o with Perplexity and GPT-4o with full LAD achieve identical English and Chinese MCQ accuracy, at 80% and 66%, respectively, although LAD obtains higher OSQ scores. A comparison using the same backbone, search engine, and comparable search budget would better establish the additional value of LAD’s perception and reasoning stages beyond access to external information.
  • The component analysis is limited to a small evaluation subset. S100 contains only 50 examples per language, and adding search improves GPT-4o-mini’s MCQ accuracy from 68% to 74% in English and from 44% to 52% in Chinese, corresponding to three and four additional correct answers. The full-benchmark results support the overall improvement, but larger ablations and paired statistical analysis would provide stronger evidence for individual components.
  • The open-ended evaluation relies on a single GPT-4o judge. The human assessment is useful, but agreement within one point on a five-point scale does not establish reliable discrimination between closely performing methods, particularly the 0.08–0.18 OSQ gains from adding search. Independent judge models or blinded human pairwise comparisons would strengthen these conclusions.
Questions for authors
  • How much improvement comes from metaphor-specific prompting alone? Could the authors directly add the same emotion, domain, and rhetorical-analysis instructions to the baseline and compare under matched inference budgets?

  • Does full LAD outperform a simple search-and-answer baseline using the same backbone, search engine, and comparable search budget? This would help isolate the additional contribution of the perception and reasoning stages.

  • Could the authors extend the stage-wise ablations to the full benchmarks and use paired statistical tests to assess the incremental gains from adding search?

  • Do the OSQ improvements, particularly the small gains from adding search, remain consistent when evaluated by independent judge models or blinded human pairwise comparisons?

4 comments
AuthorsOpenReview ↗

Response to Reviewer HR7y

We thank Reviewer HR7y for the positive assessment. We appreciate that you found the problem interesting, since metaphors need cultural and contextual knowledge beyond visible objects; the framework intuitive, with descriptions, queries, and retrieved context that are easy to inspect; the gains on four backbones and the full II-Bench and CII-Bench convincing; and the evaluation broad, with MCQ, OSQ, stage-wise ablations, search baselines, and human validation.

New results for your points are in the General Response above: Table A (stage-wise general-VQA ablation, W1/W3/W4) and Table C (single-call baseline and LAD without web search, W1/Q1/W4). We answer each point below.

W1 & Q1: Metaphor-specific prompting and matched inference budget. We ran the single-call baseline you asked for on Qwen2.5-VL-7B (General Response, Table C). It puts the same emotion, domain, and rhetorical-device instructions into one prompt, with the output cap matched to LAD-CoT. It scores 43.3% / 42.0% MCQ, close to Std CoT (46.7% / 42.0%), while LAD-CoT with separate stages reaches 54.7% / 44.0%. So the gain comes from the staged design, not the metaphor instructions alone. We will add this baseline to Section 4.4.1 of the revised version.

  1. CoT and reasoning baselines: On S100 (Table 3(a)), Std CoT moves GPT-4o-mini from 44% / 42% MCQ and 2.98 / 3.36 OSQ to 50% / 42% and 3.10 / 3.28, whereas LAD-CoT (Stages I+III) reaches 68% / 44% MCQ and 3.84 / 3.58 OSQ. In Table 5, the reasoning variant of Grok-3 raises English MCQ (66% → 74%) but not OSQ (3.24 / 2.96 → 3.06 / 2.92).
  2. Stage II with Stage I+III prompts fixed: Holding Stage I+III prompts fixed, Stage II raises GPT-4o-mini on S100 to 74% / 52% MCQ (+6 / +8) and 4.02 / 3.66 OSQ (Table 3(b)). It also improves all three general-VQA benchmarks over Stages I+III (Table A).

W2 & Q2: Comparison with simpler search augmentation. We will add a simple search-and-answer baseline using the same backbone, DuckDuckGo search engine, and comparable search budget to the revised version.

  1. Search baselines without Stages I+III (Table 3(b)): On GPT-4o, Perplexity ties LAD on S100 MCQ (80% / 66%, up from 74% / 58%). It lowers OSQ from 2.94 / 3.76 to 2.88 / 3.28, while GPT-4o + LAD reaches 4.14 / 4.26. Across all six non-LAD search pairs in Figure 4(b), OSQ changes span -0.48 to +0.01.
  2. Fixed-backbone Stage-II swap (Table 3(b)): Fixing GPT-4o-mini and Stages I+III, OpenAI native search gets 72% / 48% MCQ and lowers OSQ to 3.62 / 3.34 (vs. 3.84 / 3.58 without search). LAD-Search reaches 74% / 52% MCQ and 4.02 / 3.66 OSQ (+2 / +4 MCQ and +0.40 / +0.32 OSQ over native search). Search indices and API budgets differ in Table 3(b), which our DuckDuckGo baseline will align in Section 4.4.2 of the revised version.

W3 & Q3: Full-benchmark stage-wise ablations and paired statistical tests. We will add stage-wise ablations on the full II-Bench and CII-Bench benchmarks with paired statistical tests to the revised version.

  1. Stage-wise general VQA (Table A): Closed-book Stages I+III improves GPT-4o-mini on MMMU, SeedBench, and MMStar (+3.0 / +1.7 / +3.0), and Stage II adds +5.5 / +2.7 / +2.5.
  2. Full metaphor benchmarks (Table 6 & Table 7): S100 has 50 items per language, so +3 and +4 items (+6 / +8 points) have wide intervals on S100 alone. On all 1,399 II-Bench and 800 CII-Bench items, full LAD lifts GPT-4o-mini from 63.5% → 81.2% and 35.6% → 53.8% MCQ (two-proportion z-test, p=1.1×10−25p = 1.1 \times 10^{-25} and 3.1×10−133.1 \times 10^{-13}; Table 7). OSQ rises from 2.93 / 3.29 to 4.22 / 4.31.

W4 & Q4: Single GPT-4o OSQ judge and small search gains. We will add evaluations with independent OSQ judge models and a blinded human pairwise comparison to the revised version. In Appendix G (16 bilingual researchers, 50 S100 images), 95.7% agreement within one point of the trimmed human mean (∣h~−g∣≤1\lvert \tilde{h} - g \rvert \le 1) validates coarse alignment rather than 0.08–0.18 differences. We will describe these small gains as directional in Section 4.3 of the revised version. Beyond those margins:

  1. Judge-free MCQ improves with Stage II on S100 (+6 / +8 on GPT-4o-mini, Table 3(b), plus +7.3 / +1.3 over no-web on Qwen2.5-VL-7B, Table C) and on general VQA (Table A).
  2. OSQ gaps over generic search are larger: +0.40 / +0.32 over OpenAI native search and +1.26 / +0.98 over Perplexity (Tables 3(b) and 4(a)).
  3. On Qwen backbones, full LAD also raises English OSQ by +1.30 / +2.06 (Table 4(a)), suggesting the gains are not limited to the judge's model family.

If you have any further questions, please feel free to let us know.

AuthorsOpenReview ↗

Update: single-call baseline (W1 & Q1)

We have added the single-call baseline you asked for to Table C in the General Response (2/2). On Qwen2.5-VL-7B, putting LAD's metaphor instructions into one call gives 43.3% / 42.0% MCQ, close to Std CoT (46.7% / 42.0%), while LAD-CoT with separate stages reaches 54.7% / 44.0%. Our reply to W1 & Q1 is updated accordingly.

If you have any further questions, please feel free to let us know.

Reviewer_HR7yOpenReview ↗

keep score

Thank you for adding this baseline. It addresses my concern about metaphor-specific prompting. I maintain my rating of 3 (Weak acceptance).

The controlled search comparison, full-benchmark stage-wise ablations with paired tests, and independent OSQ evaluations remain pending. However, the consistent gains across backbones and full benchmarks continue to support my positive assessment.

AuthorsOpenReview ↗

Revised PDF uploaded: where your points landed in the main paper

We thank Reviewer HR7y for the follow-up and for maintaining the positive assessment. The revised PDF is now uploaded, with all additions marked in blue, and the points you raised are in the main paper.

W1 and Q1, metaphor-specific prompting under a matched budget. New Table 4 (Section 4.4.1) separates the instructions from the staged design on one backbone. A single call carrying LAD's emotion, domain and rhetorical-device instructions, with the output cap matched to LAD-CoT, reaches 43.3%/42.0%, which is not above Std CoT (46.7%/42.0%), while the same instructions run as separate stages reach 54.7%/44.0%. The same table also separates parametric recall from web access: 56.7%/44.7% without web search against 64%/46% with it.

W3, stage-wise evidence beyond S100. Table 5(b) adds a closed-book Stages I+III row on MMMU, SeedBench and MMStar, 62.4/74.5/57.8 against the 59.4/72.8/54.8 base, with Stage II adding a further +5.5/+2.7/+2.5. Section 4.4.4 states which of the two is a controlled comparison.

W4 and Q4, the OSQ judge. Section 4.3 and Appendix G now state that the 95.7% human check uses a one-point tolerance on a five-point scale and therefore validates coarse alignment, and the +0.08 to +0.18 OSQ gains from adding search are described as directional.

Table 5(a) also adds Gemini-3.0-Pro with and without LAD, so the pipeline is now shown on five backbones.

We will add the controlled search comparison, the full-benchmark stage-wise ablations with paired tests, and the independent OSQ evaluations to the final version.

If you have any further questions, please feel free to let us know.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/2)

We thank all reviewers for their constructive feedback. Reviewers noted that:

  • the problem is relevant and interesting, since image metaphors need cultural and contextual knowledge beyond recognizing visible objects (HR7y, aD4U);
  • the perception, context-acquisition, and reasoning stages give an intuitive structure whose descriptions, queries, and retrieved context are easy to inspect (HR7y, aD4U);
  • LAD is a cheap, training-free framework that improves multiple backbones and model families, including the full II-Bench and CII-Bench and general VQA (2a3c, HR7y, aD4U);
  • the evaluation is broad and the ablations are thorough, covering MCQ, OSQ, stage-wise ablations, search baselines, and human validation of the judge (HR7y, 2a3c).

Contribution and Gain Decomposition. Image metaphor understanding fails when frozen MLLMs miss the cultural, social, or rhetorical background cc needed to interpret how visual elements combine. We formalize this test-time failure mode as missing context (Section 3.1): it covers both conventions stored in model weights that single-pass inference leaves unused and external facts absent from weights. In the revised version, we will align the Abstract, Introduction, and Section 3.3 with this Section 3.1 definition. LAD constructs context in three stages. First, Perception extracts a structured description and 7 keywords for scene elements, emotion, domain, rhetoric, and puns. Second, Search plans keyword-combination questions Q(z)Q(z), routes each via Self-Judge between parametric ModelSearch and MindSearch-based WebSearch, and ranks and refines answers by implication relevance SS. Third, Reasoning infers the implication from x,z,c^x, z, \hat{c}. External web search acts as an adaptive component rather than a source required for every image.

Additional Results Referenced Across Reviews.

Table A: Stage-wise general-VQA ablation on GPT-4o-mini (same Stage I+III prompts with description and 7 keywords, and same splits as Table 4(b)):

Setting (GPT-4o-mini)MMMUSeedBenchMMStar
Base (closed-book, Table 4(b))59.472.854.8
+ LAD Stages I+III (no search, closed-book)62.4 (+3.0)74.5 (+1.7)57.8 (+3.0)
+ LAD Stages I+II+III (full, Table 4(b))67.9 (+8.5)77.2 (+4.4)60.3 (+5.5)
Reference: GPT-4o (closed-book, Table 4(b))70.776.765.1

Stages I+III alone improve all three (+3.0 / +1.7 / +3.0). Stage II adds a further +5.5 / +2.7 / +2.5. As noted in Appendix I, full LAD uses web search while reference rows are closed-book, so retrieved domain facts may contribute on knowledge-intensive items such as MMMU.

Table B: Cross-backbone S100 results with Gemini-3.0-Pro (same Appendix L prompts and gpt-4o-2024-11-20 OSQ judge as Table 2):

Backbone (S100)w/o LAD (MCQ en/zh, OSQ en/zh)w/ LAD (MCQ en/zh, OSQ en/zh)Δ\Delta (MCQ en/zh, OSQ en/zh)
Qwen2.5-VL-7B (Table 4(a))46 / 40 / 2.34 / 2.5864 / 46 / 3.64 / 3.36+18 / +6 / +1.30 / +0.78
GPT-4o-mini (Table 4(a))44 / 42 / 2.98 / 3.3674 / 52 / 4.02 / 3.66+30 / +10 / +1.04 / +0.30
Qwen2.5-VL-72B (Table 4(a))72 / 56 / 1.56 / 3.1276 / 62 / 3.62 / 3.68+4 / +6 / +2.06 / +0.56
GPT-4o (Table 4(a))74 / 58 / 2.94 / 3.7680 / 66 / 4.14 / 4.26+6 / +8 / +1.20 / +0.50
Gemini-3.0-Pro (additional)76 / 76 / 3.82 / 3.9682 / 78 / 4.30 / 4.46+6 / +2 / +0.48 / +0.50

English MCQ gains narrow when closed-book accuracy is already high (+4 to +6 at 72%–76%). OSQ improves across all five backbones (+0.30 to +2.06), reaching 4.30 / 4.46 on Gemini-3.0-Pro.

(Continued in Comment 2/2.)

AuthorsOpenReview ↗

General Response to All Reviewers (2/2)

Single-Call Prompting, Staged LAD, and Search Without Web Access (Addresses HR7y Q1, 2a3c W1/W2/Q1). To separate the metaphor instructions, the staged design, and external web search, we ran the variants below on S100 MCQ (judge-free, same 100 questions, mean over runs) with Qwen2.5-VL-7B (Table 4(a) backbone). The single-call row puts LAD's emotion, domain, and rhetorical-device instructions into one prompt, with the output cap matched to LAD-CoT. ModelSearch-only keeps all stages, prompts (Appendix L), and decoding unchanged, but answers every planned Stage-II question via ModelSearch without web access.

Table C: Stage-wise ablation on S100 MCQ (Qwen2.5-VL-7B):

Qwen2.5-VL-7B, S100 MCQEN (%)ZH (%)
No LAD (base, Table 4(a))4640
Std CoT (Figure 8 prompt, new)46.742.0
Single call with LAD's metaphor instructions (new)43.342.0
LAD-CoT, Stages I+III (new)54.744.0
+ Stage II, ModelSearch-only (new)56.744.7
+ Stage II, Self-Judge: ModelSearch + WebSearch (full LAD, Table 4(a))6446
  1. Instructions vs. stages: Putting the metaphor instructions into one call does not help (43.3 / 42.0 vs. 46.7 / 42.0 for Std CoT). Running the same instructions as separate stages reaches 54.7 / 44.0 (+11.4 / +2.0 over the single call). The gain comes from the staged design, not the instruction wording alone.
  2. Web search vs. parametric recall: Answering the planned Stage-II questions from the model’s own parameters alone adds 2.0 / 0.7 points over Stages I+III (56.7 / 44.7 vs. 54.7 / 44.0). With web search, full LAD reaches 64 / 46, +9.3 / +2.0 over Stages I+III. On this backbone, external retrieval supplies 7.3 of those 9.3 English points.

This matches the two parts of missing context in Section 3.1: Stages I+III draw out conventions already in the model (46 → 54.7 EN), and web search adds facts beyond them (54.7 → 64 EN).

Summary of Revisions for the Revised Version.

  • Reframed contribution (Abstract, Section 1, Section 3.3, Section 4.1): Define missing context as covering both unelicited parametric conventions and absent external knowledge, with web search as an adaptive component. State in Section 4.1 that prompts were written once without tuning on S100.
  • Added results reported above (Table 4, Section 4.4): Include the GPT-4o-mini Stages I+III general-VQA row (Table A), the Gemini-3.0-Pro ± LAD results (Table B), and the Qwen2.5-VL-7B single-call and ModelSearch-only ablations (Table C, Section 4.4.1).
  • Additional baselines and ablations committed for the revised version:
    1. Simple search-and-answer baseline using the same backbone, DuckDuckGo search engine, and comparable search budget (Section 4.4.2, HR7y Q2, aD4U Q2).
    2. Stage-wise ablations including closed-book Stages I+III on the full II-Bench and CII-Bench benchmarks in Table 6, with paired statistical tests (Section 4.4.4, Appendix E, HR7y Q3, aD4U W3).
    3. OSQ evaluations with independent judge models and a blinded human pairwise comparison, plus clarification of the one-point tolerance in the 95.7% human-agreement check (Section 4.3, Appendix G, HR7y Q4).
    4. Sub-component ablations within Stage II: separate ablations of question planning and implication-aware ranking, including composition-guided vs. entity-only queries and implication-aware ranking vs. unranked selection (Section 4.4.2, 2a3c W2, aD4U W2).
  • Related work (Section 2.2): Add PVSE (Song & Soleymani, CVPR 2019) and contrast polysemous retrieval embeddings with test-time context construction (aD4U Q3).

If you have any further questions, please feel free to let us know.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.