GuidedEvolve: Feedback-Guided Prompt Evolution for Discovering Implicit Decision Criteria

Reliable decision-making with Vision-Language Models (VLMs) often requires adherence to implicit, domain-specific decision policies that go beyond generic reasoning instructions. Prompt optimization provides a practical way to elicit such behavior, yet existing methods largely rely on stochastic search or scalar performance feedback, offering limited insight into why a prompt fails or how it should be improved. To address this limitation, we propose GuidedEvolve, a feedback-guided prompt evolution framework for decision-making that transforms case-level feedback analysis into actionable decision policies. GuidedEvolve identifies fine-grained failure patterns, distills them into transferable decision strategies, and uses these strategies to guide subsequent prompt evolution, turning prompt optimization from metric-driven exploration into interpretable and directed refinement. We evaluate GuidedEvolve on multimodal misinformation detection across public benchmark datasets. Experimental results show that GuidedEvolve consistently outperforms existing prompting and prompt-optimization methods, demonstrating the effectiveness of explicit feedback-guided evolution for complex multimodal decision-making.