Faithful Reasoning under Intent Ambiguity via Toulmin-Augmented Chain-of-Thought

Vision-Language Models (VLMs) tend to give confident answers on driving scenes that even humans find ambiguous. We argue that free-form chain-of-thought (CoT) inherits a habit of human reasoning: the reasons are generated after the conclusion, and the premise that actually produced the conclusion is left out. Following Toulmin's model of argumentation, we build a multi-agent pipeline in which each agent has its own responsibility. Several agents describe what is visible in the video, others propose a warrant for each possible conclusion (in Toulmin's terms, the general rule that leads from the observations to the conclusion), and a judge weighs the two warrants against the video and reaches a conclusion. Because the observations are written before any conclusion exists, the trace cannot be a rationalization of an answer reached in advance. We evaluate on 200 ambiguous pedestrian intent samples with two expert human studies and standard classification metrics. First, expert annotators compared each method's reasoning output against the video. The Toulmin pipeline produces 39% fewer traces with a major hallucination than self-consistency CoT (31.4% against 51.5%). Second, in a study without the video, experts recover self-consistency CoT's conclusion from its observations alone in 94.6% of cases, and its reason changes their conclusion in only 5.0%, which is the signature of post-hoc rationalization. By contrast, the Toulmin warrant changes the reader's conclusion in 37.0% of cases. On the most ambiguous cases, self-consistency CoT is rated clearer, yet it is correct on only 8.9%, against 35.6% for Toulmin. On standard classification metrics, Toulmin also improves accuracy, by 6.5 points over the warrant-ablated pipeline (48.0% against 41.5%) and 21 points over self-consistency CoT (27.0%).