acceptodds
Under review as a conference paper at ICLR 2027

Synthetic Multimodal In-Context Learning

Abstract

One of the central questions in multimodal in-context learning concerns how to construct effective multimodal contexts that provide appropriate task and reasoning guidance for multimodal large language models (MLLMs). Existing methods largely build such contexts by selecting demonstrations from a predefined corpus, limiting their ability to provide task- and reasoning-aligned context when suitable examples are unavailable. To overcome this limitation, we introduce synthetic multimodal in-context learning, a paradigm that shifts multimodal context construction from fixed-corpus demonstration selection toward query-adaptive demonstration synthesis at inference time. To instantiate it, we propose SynMIC, a self-evolving agentic framework that synthesizes demonstrations aligned with the task and reasoning schema required for in-context inference on each target query. Specifically, it abstracts the target query into a synthesis specification capturing its content, task, and reasoning schemas, and uses the specification to plan diverse demonstrations by specifying new visual scenarios, queries, pre-committed answers, and answer-supporting visual constraints. These plans are realized through tool-mediated synthesis by adaptively assembling and executing toolchains over code-based rendering, model-based image generation/editing, and auxiliary perception. Meanwhile, a verification-guided self-evolution loop verifies synthesized demonstrations, resynthesizes invalid cases, and distills failure experience into an evolving skill that assists subsequent synthesis. Experiments across diverse benchmarks and MLLMs demonstrate consistent improvements over existing baselines, highlighting the promise of shifting multimodal context construction from corpus-bounded retrieval toward adaptive synthesis. Code will be publicly available.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
in-context learning, agent system

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_wp1NOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the concern that prediction may require deep representations to retain excessive low-level details. It introduces mirrored hierarchical cross-attention between the encoder and decoder, together with timestamp-driven ROPE to capture temporal dependencies. The method is evaluated across multiple tasks, and controlled topology ablations support the effectiveness of the mirrored connectivity on video classification task.

Critical strengths
  1. The motivation is clear and convincing. It provides a useful perspective on representation learning through hierarchical information flow.

  2. The mirrored hierarchical cross-attention directly addresses the stated motivation, and the topology ablations provide evidence of its effectiveness.

  3. The explicit treatment of timestamps is well motivated for video and continuous time multimodal streams.

  4. The evaluation covers video classification, text-to-motion, and text-to-speech, demonstrating applicability beyond a single task.

Critical weaknesses
  1. Encoder–decoder skip connections are well established. But the comparison with closely related methods are insufficient to clearly establish the novelty of the proposed method.

  2. There is no direct evidence that shallow layers handle low-level details while deep layers learn more abstract representations. The single visualization does not provide a sound evidence.

  3. The method does not demonstrate consistent performance improvements across benchmarks. The video classification and text-to-speech results show competitive performance in some settings, but not a clear overall advantage.

  4. The topology ablations primarily focus on video classification with no corresponding ablations for text-to-motion or text-to-speech, leaving the effectiveness to these generation results unclear.

  5. The paper lacks ablations of the proposed ROTE, making it difficult to determine their individual contributions.

  6. The main text states that “full-causal attention” allows access to the “full caption and motion history,” whereas the appendix specifies that motion positions “attend causally within a 2 s window.” Several essential details like the data composition, model configurations, and training schedules, are not clearly summarized in the main text. The confusing distinction and poor organization make the paper hard to follow.

Questions for authors
  1. What distinguishes the proposed mirrored hierarchical cross-attention from closely related designs?

  2. Could the authors provide solid evidence supporting the claim that skip connections allow shallow layers to retain low-level details while deep layers learn more abstract features?

  3. Given that the method does not show consistent performance advantages across benchmarks, how do the authors interpret these results as evidence of its effectiveness?

  4. Does mirrored connectivity also improve on text-to-motion and text-to-speech?

  5. How does timestamp-driven RoTE compare with rotary embeddings based on frame or token indices under otherwise identical settings?

1 comment

Dear Reviewer wp1N,

Thank you very much for your time and effort devoted to reviewing our submission!

We would like to respectfully bring to your attention a possible discrepancy between the review comments and the content of our manuscript. Our paper introduces SynMIC, an agentic framework for synthetic multimodal in-context learning. Upon carefully reading your comments, we noticed that some of the technical details discussed in the review seem to refer to a different work, possibly titled RoTE, which introduces mirrored hierarchical cross-attention and timestamp-driven RoPE for video classification. To the best of our understanding, these technical components and research topics differ substantially from those presented in our submission.

Given these differences, we wonder whether there might have been an inadvertent mix-up with the review of another submission. We fully understand that such situations can occasionally occur, and we would be sincerely grateful if you could kindly double-check whether the review was intended for our manuscript.

We would also be more than willing to address any questions or concerns you may have regarding our work during the rebuttal and author–reviewer discussion period. At the same time, given the limited time remaining in the discussion period, we are worried about that there may not be sufficient time to thoroughly address too many new questions or concerns that might arise. Should any of your subsequent concerns overlap with those raised by other reviewers, we sincerely hope that our responses to their comments may also help provide useful clarifications.

Thank you again for your time, understanding, and consideration!

Best regards,

Authors

Reviewer_XipdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SynMIC, an agentic framework that constructs query-adaptive synthetic demonstrations for multimodal in-context learning. Given a target image-query pair, it abstracts the content, task, and reasoning structure, plans demonstrations with pre-committed answers and supporting visual constraints, and synthesizes images through code-based rendering and image-generation/editing tools. A verification-and-refinement loop corrects invalid demonstrations, while an evolving natural-language skill reuses failure-derived guidance across cases. Experiments across six benchmarks and five downstream models report consistent improvements over fixed-corpus baselines. Additional experiments examine demonstration content, alternative inference strategies, synthesis components, and same-backbone self-improvement.

Critical strengths
  1. The paper identifies a meaningful limitation of fixed-corpus multimodal ICL: retrieval cannot supply suitable demonstrations when the available corpus lacks examples matching the target task and reasoning requirements. SynMIC provides a coherent approach to constructing such demonstrations, connecting task/reasoning abstraction, answer-supporting visual constraints, synthesis, and verification. This is an interesting research direction, and the framework is clearly presented.

  2. The reported improvements extend across multiple models and task types. In Table 1, SynMIC improves average accuracy over ContextNav by approximately 2.3–2.7 percentage points across all five downstream models. The supplementary experiments provide useful controls: Table 5 examines visual demonstration content and alternative inference strategies, Table 6 compares simpler synthesis pipelines, and Table 8 demonstrates improvements when the same backbone performs synthesis and downstream inference. These results provide encouraging evidence that the constructed multimodal demonstrations can be useful, although the concerns below limit the strength of the overall conclusion.

Critical weaknesses
  1. The comparative advantage of allocating substantial inference resources to demonstration synthesis remains insufficiently established. Appendix B.1 matches the upstream policy reasoning budget, but SynMIC also uses code-rendering and image-generation models whose costs are not incorporated into a unified end-to-end comparison. Table 7 reports substantial synthesis costs per valid image without a complete per-target-query comparison against alternatives. Given the approximately 2.3–2.7 percentage-point average improvements over ContextNav, this leaves an important question unresolved: whether synthesis offers a compelling advantage over other ways of allocating comparable total inference resources. The reuse experiment in Table 9 reports the fraction of context slots filled by existing synthetic demonstrations, but not the resulting accuracy or realized cost savings on unseen queries. Consequently, it does not yet demonstrate that reuse preserves the benefits while meaningfully amortizing synthesis costs. This limits the significance I can assign to the reported gains.

  2. The evidence does not sufficiently establish the proposed coverage bottleneck or the robustness of the incremental gains. The results show that query-adaptive synthetic demonstrations can improve accuracy, but do not quantify how often fixed-corpus failures arise specifically from inadequate task/reasoning coverage. The augmentation experiment in Table 3 is supportive, yet does not separate improved coverage from other changes in demonstration quality and query alignment. In addition, several benchmarks contain only 100–150 evaluation instances, and medians over five trials are reported without dispersion estimates or paired confidence intervals. For GPT-5 on CharXiv, the improvement over ContextNav corresponds to approximately five additional correct answers among 150 queries. The consistent direction across settings is encouraging, but the uncertainty leaves the magnitude and stability of the contribution insufficiently established. Together with the incomplete resource comparison, this leads me to a borderline, weak-reject assessment.

Questions for authors
  1. In Table 5, which model produces the direct-inference solutions, and precisely which SynMIC computation is included in the matched budget? Please clarify whether verification and skill-update calls are included, and report the additional code-rendering and image-generation costs per target query. What evidence supports an advantage under a comparable total inference cost?

  2. What quantitative evidence connects the observed gains to insufficient task/reasoning coverage in the original demonstration corpus? In Table 3, are the added synthetic demonstrations generated specifically for the query being evaluated, or are they drawn from a previously generated corpus? This distinction would help interpret the coverage argument.

  3. For the unseen-query reuse setting in Table 9, what downstream accuracy and per-query cost are achieved? How do these compare with retrieval from the original corpus and with synthesizing new demonstrations for each query?

  4. Can you provide the five trial-level results and paired confidence intervals for the main SynMIC–ContextNav comparisons? Please also clarify whether the same sampled evaluation instances are shared across trials.

11 comments

Response to Reviewer Xipd (Part 1)

We sincerely thank you for your valuable time and thoughtful, constructive feedback on our manuscript, which has greatly helped us improve its quality. We try to make every effort to address each of your comments in detail below.


Weakness 1.1 - Performance advantage over baselines under matched end-to-end inference budgets

Thank you very much for this insightful comment. To assess whether SynMIC's performance advantages persist under matched inference budgets, we conduct additional experiments on CharXiv and MathVision, controlling the end-to-end inference budget per target query. Since exact matching is difficult, we conservatively allocate equal or slightly larger budgets to the baselines to avoid favoring SynMIC. As FLOP-level accounting is unavailable for proprietary API-based models, we use token costs as a common proxy for the entire inference pipeline, including synthesis-tool overhead. For model-based image generation and editing, we assign each generated image a cost equivalent to 17.6K tokens under our default policy, based on GPT-Image-2's medium pricing tier. We also disable SynMIC's auxiliary perception tools (e.g., OCR and SAM), whose costs cannot be reliably converted into token equivalents. For ContextNav, we allocate its budget to multiple independent rollouts with voting, following the strategy in Table 5 of the manuscript. The results are shown below.

MethodCharXiv Cost per Query (Token)↓CharXiv Gain (%)↑MathVision Costs per Query (Token)↓MathVision Gain (%)↑
ManyICL0.357M8.70.345M9.8
ContextNav0.360M10.20.348M12.5
SynMIC (w/o Perception Tools)0.356M15.90.345M15.2

Under matched end-to-end inference budgets, SynMIC consistently outperforms baselines, achieving better ICL gains than ContextNav even without auxiliary perception tools and under a conservative cost estimate for image generation. These results further support the effectiveness of SynMIC's synthetic demonstration construction in driving its performance advantages. The corresponding results and discussion have been added in Appendix C.2 of the revised manuscript.


Weakness 1.2 & Question 3 - Effectiveness and Cost Savings of Demonstration Reuse on Unseen Queries

Thank you very much for this constructive comment. To assess whether previously synthesized demonstrations remain effective for unseen queries, we follow the reuse setting in Table 9 (now incorporated into Table 3 of the revised manuscript) and the corpus-augmentation protocol in Table 3 of the manuscript. We compare retrieval from the original demonstration corpus with retrieval from an augmented corpus containing demonstrations synthesized for the original evaluation queries, without additional query-adaptive synthesis for unseen queries. The results show that reusing previously synthesized demonstrations also improves downstream performance on unseen queries.

MethodCorpusCharXiv Reuse RateCharXiv Gain (%)MathVision Reuse RateMathVision Gain (%)↑ERQA Reuse RateERQA Gain (%) ↑
ManyICLUnseen Original-3.0-1.2--4.4
+Augmentation57.44.627.14.174.92.2
ContextNavUnseen Original-6.1-5.5-2.2
+Augmentation53.29.140.89.636.96.5

Building on these results, we further examine the cost savings from demonstration reuse. Since previously synthesized demonstrations can serve unseen queries without additional synthesis, the one-time corpus construction cost remains fixed, while the amortized synthesis cost per query decreases as more queries are processed. As shown below, using ContextNav as an example, demonstration reuse consistently improves performance on unseen queries while reducing amortized synthesis costs. We further introduce a Full Synthesis setting, which independently synthesizes demonstrations for each unseen query without using the original corpus or previously synthesized demonstrations, and compare its performance and synthesis costs with those of reuse. The results show that demonstration reuse amortizes synthesis costs while retaining much of the performance benefit of full query-adaptive synthesis, highlighting its potential to balance inference effectiveness and computational efficiency.

CorpusQuery NumberCost per Unseen QueryCharXiv Reuse RateCharXiv Gain (%)MathVision Reuse RateMathVision Gain (%)
Unseen Original150--6.1-5.5
200--6.5-6.0
250--5.6-5.1
Unseen+Augmentation1500.351M53.29.140.89.6
2000.263M52.28.641.710.0
2500.211M55.19.343.610.3
Unseen+Full Synthesis1500.351M-10.6-12.3
2000.363M-10.8-11.0
2500.358M-11.2-12.0

The corresponding results and discussion have been added in Appendix C.3 of the revised manuscript.

Response to Reviewer Xipd (Part 2)


Weakness 2.1 & Question 2.1 - Quantitative Evidence of Performance Gains from Alleviating the Coverage Bottleneck

Thank you very much for this insightful comment. To quantitatively assess whether SynMIC's performance gains are associated with alleviating the coverage bottleneck, we conduct additional evaluations on ERQA and CharXiv. Specifically, we use GPT-6 Astra as a judge to evaluate the in-context demonstrations along three dimensions: coverage, demonstration quality, and query alignment, using a five-point rating scale. Each demonstration set is evaluated jointly with its target query, consistent with the context used for downstream inference. We then average the scores across target queries and compare them with downstream performance. All other settings follow our standard ablation evaluation protocol in the manuscript. Additionally, for cases where parsing occasionally failed due to formatting issues in the MLLM outputs, we reran the corresponding experiments to resolve these failures, ensuring that the evaluation was conducted on the complete test set.

The results on ERQA are shown below:

MethodCoverageDemonstration QualityQuery AlignmentOverall
ManyICL2.7803.9203.4600.410
ContextNav3.4603.9704.7900.430
SynMIC4.8203.9104.1000.470

The results on CharXiv are shown below:

MethodCoverageDemonstration QualityQuery AlignmentOverall
ManyICL3.2474.1073.3930.487
ContextNav3.9604.2004.6470.507
SynMIC4.7604.1934.0330.540

These results provide quantitative evidence for the importance of coverage in downstream performance. Despite lower query alignment scores than ContextNav, which employs a dedicated query-rewriting mechanism, and comparable demonstration quality to other baselines, SynMIC achieves superior downstream performance. This suggests that improving coverage offers benefits beyond demonstration quality and query-level alignment alone, further supporting limited coverage in fixed demonstration corpora as an important bottleneck for MM-ICL. The corresponding results and discussion have been added in Sec. 4.4 of the manuscript.

The detailed evaluation prompts are provided below and added in Appendix F of the revised manuscript:

(Coverage Rating Prompt)
Please rate the coverage of the provided in-context demonstration set relative to the target image-query pair. The target image-query pair and the set of in-context demonstrations are provided below.

Target image: {target_image}

Target query: {target_query}

Demonstration set: {demonstrations}

Each demonstration in {demonstrations} consists of an image-query pair. Evaluate the entire demonstration set relative to the target image-query pair and assign a single overall score. Focus on transferable task and reasoning structures rather than surface-level similarities. Demonstrations with different visual scenarios may still provide strong coverage if they instantiate the required task/reasoning patterns. 

Five-point anchors:

[1] = The demonstration set provides essentially no coverage of the target task or required reasoning patterns.

[2] = The set covers only broad task characteristics or limited reasoning elements, leaving major requirements unaddressed.

[3] = The set covers the main task or reasoning pattern partially, but important operations or evidence dependencies remain missing.

[4] = The set covers most essential task and reasoning requirements, with only minor gaps or omissions.

[5] = The set comprehensively covers the essential task structure, reasoning operations, and evidence-to-answer dependencies required by the target query.

Return only JSON with an integer score and a brief reason:

{"score": ..., "reason": "..."}

(Continued in the next part)

Response to Reviewer Xipd (Part 3)


Weakness 2.1 & Question 2.1 (Continued) - Quantitative Evidence of Performance Gains from Alleviating the Coverage Bottleneck

(Demonstration Quality Rating Prompt)
Please rate the demonstration quality of the provided in-context demonstration set. The target image-query pair and the set of in-context demonstrations are provided below.

Target image: {target_image}

Target query: {target_query}

Demonstration set: {demonstrations}

Each demonstration in {demonstrations} consists of an image-query pair. Evaluate the entire demonstration set and assign a single overall score. Focus on the intrinsic quality of the demonstrations, including visual clarity, image-query consistency, and whether the visual evidence is sufficient to support a clear and unambiguous answer. Assess demonstration quality independently of its relevance or alignment to the target query.

Five-point anchors:

[1] = The demonstration set is largely invalid or uninterpretable, with severe visual ambiguity or inconsistencies between images and queries.

[2] = The set contains substantial quality issues, including unclear visual evidence, poorly grounded queries, or significant ambiguity.

[3] = The set is generally interpretable and answerable, but noticeable issues in visual clarity, image-query consistency, or evidence sufficiency remain.

[4] = Most demonstrations are visually clear, well-grounded, and answerable, with only minor quality issues.

[5] = The demonstration set consistently provides clear, coherent, and unambiguous image-query pairs with sufficient visual evidence to support reliable answers.

Return only JSON with an integer score and a brief reason:

{"score": ..., "reason": "..."}
(Query Alignment Rating Prompt)
Please rate the query alignment of the provided in-context demonstration queries relative to the target query. The target query and the set of in-context demonstration queries are provided below.

Target query: {target_query}

Demonstration queries: {demonstration_queries}

Each query in {demonstration_queries} corresponds to one in-context demonstration. Evaluate the entire set of demonstration queries relative to the target query and assign a single overall score. Focus on semantics, question intent, referenced concepts, queried attributes, and question formulations. 

Five-point anchors:

1 = The demonstration queries are unrelated to the target query, with essentially no overlap in question intent or queried content.

2 = The set shares only broad semantic themes or general question characteristics, with substantial differences in specific query intent or content.

3 = The set shows moderate intent alignment with the target query, but noticeable differences in queried attributes, or content remain.

4 = Most demonstration queries closely align with the target query in intent, with only minor differences.

5 = The demonstration queries exhibit strong intent alignment with the target query, preserving its specific question intent despite possible differences in wording or referenced entities.

Return only JSON with an integer score and a brief reason:

{"score": ..., "reason": "..."}

Comments on the paper

7 comments

Response to Reviewer xsZY (Part 1)

We sincerely thank you for your valuable time and thoughtful, constructive feedback on our manuscript, which has greatly helped us improve its quality. We try to make every effort to address each of your comments in detail below.


Weakness 1.1 - Robustness to Invalidity in Initial Schema and Demonstration Planning

Thank you very much for this insightful suggestion. To assess the reliability of initial synthesis specifications and the robustness of subsequent demonstration planning, we conduct additional analyses under our standard ablation setting. First, we evaluate the validity of initial synthesis schemas using Qwen3.5-9B, the default policy (gemini-3-flash), and GPT-6 Astra, supplemented by human review of GPT-6 Astra's assessments. Each evaluator independently assesses whether the initial content, task, and reasoning schemas correctly capture the target query's requirements. The evaluation prompt is provided below, with human reviewers following the same criteria:

Verify whether the synthesis schema correctly captures the content, task, and reasoning requirements of the target multimodal query.

The original target image is attached.

original query: {original_query}

content abstraction: {content_abstraction}

task schema: {task_schema}

reasoning schema: {reasoning_schema}

Check whether the content abstraction reflects the query-relevant visual evidence, the task schema correctly identifies the underlying task objective and requirements, and the reasoning schema provides a plausible and consistent evidence-to-answer reasoning pattern.

Judge the schemas against the original image and query rather than assuming their correctness. Consider a schema invalid if it contains substantive errors, unsupported assumptions, or inconsistencies that could misguide subsequent demonstration planning. Allow reasonable variations in abstraction and reasoning formulations.

Return a JSON object with:

"validity decision": whether the initial synthesis specification is valid.

"verification feedback": a brief explanation of the assessment, including any identified schema errors or inconsistencies.

Please note that the evaluation should remain objective and evidence-based. Do not deliberately seek faults or mark a schema as invalid merely to identify potential issues; base all judgments on the actual evidence and substantive correctness.

As shown by the Schema Invalidity Rate below, fundamental misalignment with the target task or reasoning requirements is rare, potentially because schema construction mainly involves structured abstraction of the target image-query pair with relatively explicit requirements. Nevertheless, even when initial schemas accurately capture the target query's content, deviations in the policy's reasoning process may lead to demonstration plans that fail to produce valid demonstrations, requiring additional replanning. To assess SynMIC's robustness in such cases, we conduct a controlled comparison by replacing only the validation judge with each of the aforementioned evaluators, while retaining the default policy for all other components. We report the Replan Rate (fraction of cases requiring replanning during validation), Correction Rate (proportion of these cases passing the corresponding judge's validation within three replanning rounds), and resulting downstream performance gains. The results are shown below.

JudgeSchema Invalidity RateReplan RateCorrection RateGain
Qwen-3.5-9B0.020.0930.9215.5
Default Policy0.000.0471.0019.0
GPT-6 Astra0.000.0731.0020.7
GPT-6 Astra + Human Review0.000.0671.0020.7

The results show that the default policy successfully resolves most cases flagged for replanning under its own validation criteria. Using "GPT-6 Astra + Human Review" as the human-reviewed reference, we find that validation reliability is related to the judge capability. For example, the weaker Qwen3.5-9B judge triggers more replanning and yields lower downstream gains, suggesting that less reliable validation may affect accepted demonstration quality, while also introduce unnecessary revisions. Overall, fundamental errors in initial synthesis schemas are uncommon, while validation-guided replanning recovers most identified planning or synthesis failures. Despite judge-dependent differences, the default policy achieves a high correction rate and competitive downstream performance relative to stronger or human-reviewed validation, further supporting the reliability of SynMIC's initial schema construction and the robustness of its verification-guided refinement process. The corresponding results and discussion have been added in Appendix B.3 of the revised manuscript.

Response to Reviewer xsZY (Part 2)


Weakness 1.2 - Reliability of the Validation Process and Independent Evaluation of Accepted Demonstrations

Thank you very much for this valuable comment. Building on our analysis in Weakness 1.1, which found very few fundamentally invalid initial schemas, we further investigate the reliability of SynMIC's validation process and potential discrepancies between self-verification and independent evaluation. Specifically, under our standard ablation setting, we first collect the same pool of initially synthesized demonstrations before validation and independently assess them using three evaluators: the default policy, GPT-6 Astra, and GPT-6 Astra with human evaluation. We then further examine the subset of demonstrations accepted by the default policy, independently re-evaluating their validity using GPT-6 Astra and GPT-6 Astra with human evaluation. All evaluators follow the same verification criteria as specified in the manuscript, with all other experimental settings kept unchanged. The resulting validation pass rates are summarized below.

DemonstrationsJudged by PolicyJudged by GPT-6 AstraJudged by GPT-6 Astra with Human Evaluation
Before Policy Validation0.9060.8550.882
Accepted by Default Policy-0.9430.973

The results indicate that the vast majority of policy-accepted demonstrations remain valid under independent assessment, despite the potential for occasional self-verification errors. However, such discrepancies are relatively limited, supporting the effectiveness of the verification mechanism in maintaining the quality of accepted synthetic demonstrations. The corresponding results and discussion have also been added in Appendix B.3 of the revised manuscript.


Weakness 2.1 - Budget Consistency Across Reasoning Variants

Thank you very much for this insightful comment. In Table 5 of Appendix B.1 in the original manuscript, we described budget matching at the levels of "demonstration planning" and "tool-invocation reasoning." However, our explanation may not have been sufficiently detailed, potentially leading to ambiguity or misunderstanding. Specifically, "demonstration planning" covers all upstream reasoning and planning costs for demonstration synthesis, while "tool-invocation reasoning" includes synthesis-tool calls and internal execution costs, verification loops, skill updates, and downstream inference with tool-synthesized demonstrations. We disable auxiliary perception tools (e.g., OCR and SAM), whose costs cannot be reliably converted into token equivalents, to avoid unaccounted overhead. The comparison therefore matches end-to-end inference budgets. We have clarified this accounting in the revised manuscript.

To further quantify tool-related overhead, we decompose the average per-query inference cost into three components: (1) Other Costs, covering demonstration planning, policy-side synthesis-tool invocation, verification and resynthesis loops, skill updates, and final downstream inference; (2) Code-based Rendering, covering internal model inference in code-rendering tools; and (3) Model-based Image Generation, covering image generation and editing, whose cost is equivalent to 17.6K tokens per image under our default policy, based on GPT-Image-2's medium pricing tier. Results averaged across CharXiv, MathVision, MMStar, and ERQA are shown below.

Other CostsCode-based RenderingModel-based Image GenerationTotal Cost per Query
0.147M0.118M0.073M0.338M

The corresponding results, discussion and more clarification have been added in Appendix B.1 and C.1 of the revised manuscript.

Response to Reviewer xsZY (Part 3)


Weakness 2.2 - Performance Advantage over Baselines under Comparable End-to-End Inference Budgets

Thank you very much for this insightful comment. To assess whether SynMIC's performance advantages persist under matched inference budgets, we conduct additional experiments on CharXiv and MathVision, controlling the end-to-end inference budget per target query. Since exact matching is difficult, we conservatively allocate equal or slightly larger budgets to the baselines to avoid favoring SynMIC. As FLOP-level accounting is unavailable for proprietary API-based models, we use token costs as a common proxy for the entire inference pipeline, including synthesis-tool overhead. For model-based image generation and editing, we assign each generated image a cost equivalent to 17.6K tokens under our default policy, based on GPT-Image-2's medium pricing tier. We also disable SynMIC's auxiliary perception tools (e.g., OCR and SAM), whose costs cannot be reliably converted into token equivalents. For ContextNav, we allocate its budget to multiple independent rollouts with voting, following the strategy in Table 5 of the manuscript. The results are shown below.

MethodCharXiv Cost per Query (Token)↓CharXiv Gain (%)↑MathVision Costs per Query (Token)↓MathVision Gain (%)↑
ManyICL0.357M8.70.345M9.8
ContextNav0.360M10.20.348M12.5
SynMIC (w/o Perception Tools)0.356M15.90.345M15.2

Under matched end-to-end inference budgets, SynMIC consistently outperforms baselines, achieving better ICL gains than ContextNav even without auxiliary perception tools and under a conservative cost estimate for image generation. These results further support the effectiveness of SynMIC's synthetic demonstration construction in driving its performance advantages. The corresponding results and discussion have been added in Appendix C.2 of the revised manuscript.


Weakness 2.3 - Effectiveness and Cost Savings of Demonstration Reuse on Unseen Queries

Thank you very much for this constructive comment. To assess whether previously synthesized demonstrations remain effective for unseen queries, we follow the reuse setting in Table 9 (now incorporated into Table 3 of the revised manuscript) and the corpus-augmentation protocol in Table 3 of the manuscript. We compare retrieval from the original demonstration corpus with retrieval from an augmented corpus containing demonstrations synthesized for the original evaluation queries, without additional query-adaptive synthesis for unseen queries. The results show that reusing previously synthesized demonstrations also improves downstream performance on unseen queries.

MethodCorpusCharXiv Reuse RateCharXiv Gain (%)MathVision Reuse RateMathVision Gain (%)↑ERQA Reuse RateERQA Gain (%) ↑
ManyICLUnseen Original-3.0-1.2--4.4
+Augmentation57.44.627.14.174.92.2
ContextNavUnseen Original-6.1-5.5-2.2
+Augmentation53.29.140.89.636.96.5

Building on these results, we further examine the cost savings from demonstration reuse. Since previously synthesized demonstrations can serve unseen queries without additional synthesis, the one-time corpus construction cost remains fixed, while the amortized synthesis cost per query decreases as more queries are processed. As shown below, using ContextNav as an example, demonstration reuse consistently improves performance on unseen queries while reducing amortized synthesis costs. We further introduce a Full Synthesis setting, which independently synthesizes demonstrations for each unseen query without using the original corpus or previously synthesized demonstrations, and compare its performance and synthesis costs with those of reuse. The results show that demonstration reuse amortizes synthesis costs while retaining much of the performance benefit of full query-adaptive synthesis, highlighting its potential to balance inference effectiveness and computational efficiency.

CorpusQuery NumberCost per Unseen QueryCharXiv Reuse RateCharXiv Gain (%)MathVision Reuse RateMathVision Gain (%)
Unseen Original150--6.1-5.5
200--6.5-6.0
250--5.6-5.1
Unseen+Augmentation1500.351M53.29.140.89.6
2000.263M52.28.641.710.0
2500.211M55.19.343.610.3
Unseen+Full Synthesis1500.351M-10.6-12.3
2000.363M-10.8-11.0
2500.358M-11.2-12.0

The corresponding results and discussion have been added in Appendix C.3 of the revised manuscript.


Finally, we sincerely thank you again for these valuable insights, which have significantly improved the quality of our work. We have incorporated all the additional results and discussion into revised manuscript. If you have any further suggestions, please let us know, we will revise or supplement the paper accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.