- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Distill-ICL, a training-free framework for few-shot multimodal in-context learning. The method first employs an ICD-set-oriented MCTS to identify a shared task-level reasoning chain and then distills chain-local experiences from successful and failed rollouts to provide step-specific visual guidance. Experiments across multiple benchmarks and LVLM backbones demonstrate consistent improvements over existing methods.
Critical strengths
- The paper identifies two meaningful limitations of existing multimodal ICL methods: the lack of shared task-level reasoning guidance and insufficient support for reliable visual perception during reasoning. The proposed combination of a normative reasoning chain and chain-local experiences provide a reasonable approach to addressing these limitations.
- By distilling reasoning guidance during context construction, the proposed method achieves relatively low online inference costs compared with other reasoning-augmented ICL methods, making it potentially useful when the distilled context can be reused across multiple queries.
Critical weaknesses
- The method’s novelty is limited. The proposed framework primarily combines and extends existing techniques, including MCTS-based reasoning search, contrastive analysis of reasoning trajectories, and experience summarization. In particular, the ICD-set-oriented MCTS mainly differs from standard MCTS by aggregating candidate actions across demonstrations and introducing a variance-penalized reward. The improvement over standard MCTS is relatively modest (62.5% vs. 61.6% in Table 2). Similarly, identifying critical reasoning steps by contrasting successful and failed rollouts appears to be a relatively straightforward extension of existing experience-based reasoning methods. Therefore, the methodological novelty of the proposed components remains somewhat limited.
- Insufficient comparisons with simpler baselines. Although the paper compares the proposed method with several reasoning-based baselines, it remains unclear whether the relatively complex MCTS search and experience distillation procedures are necessary. For example, a simpler approach could directly prompt the LVLM to summarize a shared reasoning procedure and identify discriminative or error-prone visual attributes from the entire ICD set, without performing MCTS or multiple controlled rollouts. Furthermore, the effectiveness of the JSD-based critical-node identification could be better justified by comparing it with random-node selection or experience extraction from all reasoning steps. Compute-matched comparisons with these simpler alternatives would help establish whether the improvements arise from the proposed mechanisms rather than additional computation and LLM-based summarization.
Questions for authors
See weakness
2 comments
Author Response to Reviewer ZTtw (Part 2/2)
(W2) Comparison with Simpler Baselines
Thanks for this valuable suggestion. We conduct an additional ablation study to evaluate the necessity of our proposed two components.
Experimental Setup. We use three benchmarks following Tab. 2. We evaluate two aspects: (I) the necessity of structured reasoning distillation, and (II) the necessity of critical-node identification. For the former, we introduce the Direct Prompting baseline, which directly prompts the LVLM to summarize a shared reasoning chain, with or without discriminative and error-prone visual attributes. For the latter, we compare our JSD-based critical-node identification with two alternatives: Random-Node Selection, which randomly selects the same number of nodes, and All-Node Extraction, which extracts experiences from every reasoning step. To ensure fairness, all critical-node selection variants are based on the same Normative Reasoning Chain .
Table R2: Comparison with simpler alternatives using Qwen3-VL-32B.
| Methods | SMMILE | EndoBench | FinMME | Avg. |
|---|---|---|---|---|
| Baseline | ||||
| Few-shot CoT | 63.9 | 50.0 | 66.3 | 60.1 |
| Block I: Structured Distillation vs. Direct Prompting | ||||
| Direct Prompting (Chain Only) | 63.2 | 50.0 | 65.7 | 59.6 |
| Ours (Chain Only) | 66.3 | 53.2 | 68.0 | 62.5 |
| Direct Prompting (Chain + Experiences) | 63.9 | 50.4 | 66.5 | 60.3 |
| Ours | 69.5 | 56.3 | 70.5 | 65.4 |
| Block II: Node Selection | ||||
| Random-Node Selection | 67.1 | 53.7 | 67.7 | 62.8 |
| All-Node Extraction | 67.6 | 54.4 | 68.0 | 63.3 |
| Ours (JSD-based) | 69.5 | 56.3 | 70.5 | 65.4 |
Key Observations. (1) Both Direct Prompting variants achieve performance comparable to Few-shot CoT (59.6%/60.3% vs. 60.1%), while substantially underperforming our corresponding methods (62.5%/65.4%). This suggests that effectively distilling shared reasoning chains and chain-local experiences from limited demonstrations is non-trivial and cannot be readily achieved through direct prompting alone, showing the importance of our structured search and experience distillation mechanisms.
(2) Based on the same chain, our JSD-based approach outperforms both Random-Node Selection (65.4% vs. 62.8%) and All-Node Extraction (65.4% vs. 63.3%). Extracting experiences from all steps may introduce redundant guidance that obscures truly pivotal or error-prone steps, underscoring the importance of targeted experience distillation.
We recognize that these additional ablations are valuable for further validating the necessity of the proposed two components. We have therefore incorporated them into the revised manuscript as the third part of the ablation studies in Sec. 4.3.
Author Response to Reviewer ZTtw (Part 1/2)
We sincerely thank you for your effort in reviewing our paper, and for your valuable comments and insightful feedback. We also appreciate your recognition of “reasonable approach” and “low inference cost”. Below, we address each question individually.
(W1) Clarification of Methodological Novelty
We sincerely thank the reviewer for this thoughtful comment. We appreciate this concern and realize that our original presentation may not have clearly conveyed the key idea behind our design.
Our key insight is to first establish a shared task-level reasoning procedure and then identify critical and difficult-to-interpret visual cues along this shared structure. To our knowledge, this is a new perspective in multimodal reasoning-based ICL. A shared reasoning procedure alone cannot guarantee reliable visual interpretation, while experiences derived from independent trajectories lack a common structure for effective comparison and reuse. Our contribution lies in connecting these two aspects to make fuller use of limited demonstrations.
Our two components serve distinct but closely connected roles in realizing this idea:
(1) ICDs-MCTS: Beyond reasoning accuracy to a shared foundation for experience distillation. Unlike instance-specific search, our ICDs-MCTS searches for a shared Normative Reasoning Chain across the entire ICD set through the proposed collective expansion and universality-aware rewarding. More importantly, this shared chain provides a common reference for aligning and comparing step-wise reasoning behaviors across ICDs and rollouts. Thus, its value goes beyond the standalone accuracy improvement (62.5% vs. 61.6%), as it provides the foundation for subsequent experience distillation. Without this shared structure, experiences from different reasoning trajectories would be more difficult to compare and organize into reusable guidance.
(2) Chain-Local Experience Distillation: Beyond shared reasoning to reliable visual execution. While the Normative Reasoning Chain specifies what reasoning steps to follow, it does not guarantee their correct execution. We therefore identify two complementary types of critical nodes along this shared chain: Inter-ICD Pivotal Nodes, which capture answer-discriminative visual cues, and Intra-ICD Fragile Nodes, which reveal error-prone visual interpretations. These two types of experiences provide targeted guidance for both visual discrimination and reliable step-wise execution. Their effectiveness is supported by the improvement from 62.5% to 65.4% (copy from Tab. 2) when incorporating Chain-Local Experiences.
Our design emphasizes simplicity and effectiveness rather than algorithmic complexity. Rather than making each component unnecessarily complex, we use established techniques to realize a unified idea: using a shared reasoning chain as the foundation for distilling complementary, execution-level visual guidance. We believe the methodological contribution lies in this coherent integration of two interdependent components, enabling more effective use of limited demonstrations in multimodal ICL.
We sincerely appreciate the reviewer for helping us better clarify the motivation and contribution of our work.