acceptodds
Under review as a conference paper at ICLR 2027

Faithful Reasoning under Intent Ambiguity via Toulmin-Augmented Chain-of-Thought

Abstract

Vision-Language Models (VLMs) tend to give confident answers on driving scenes that even humans find ambiguous. We argue that free-form chain-of-thought (CoT) inherits a habit of human reasoning: the reasons are generated after the conclusion, and the premise that actually produced the conclusion is left out. Following Toulmin's model of argumentation, we build a multi-agent pipeline in which each agent has its own responsibility. Several agents describe what is visible in the video, others propose a warrant for each possible conclusion (in Toulmin's terms, the general rule that leads from the observations to the conclusion), and a judge weighs the two warrants against the video and reaches a conclusion. Because the observations are written before any conclusion exists, the trace cannot be a rationalization of an answer reached in advance. We evaluate on 200 ambiguous pedestrian intent samples with two expert human studies and standard classification metrics. First, expert annotators compared each method's reasoning output against the video. The Toulmin pipeline produces 39% fewer traces with a major hallucination than self-consistency CoT (31.4% against 51.5%). Second, in a study without the video, experts recover self-consistency CoT's conclusion from its observations alone in 94.6% of cases, and its reason changes their conclusion in only 5.0%, which is the signature of post-hoc rationalization. By contrast, the Toulmin warrant changes the reader's conclusion in 37.0% of cases. On the most ambiguous cases, self-consistency CoT is rated clearer, yet it is correct on only 8.9%, against 35.6% for Toulmin. On standard classification metrics, Toulmin also improves accuracy, by 6.5 points over the warrant-ablated pipeline (48.0% against 41.5%) and 21 points over self-consistency CoT (27.0%).

Primary area:
foundation or frontier models, including LLMs
Keywords:
chain-of-thought faithfulness, post-hoc rationalization, Toulmin argumentation, vision-language models, multi-agent reasoning, human evaluation, pedestrian intent prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_re4pOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a multi-agent pipeline that separates visual observations, opposing Toulmin warrants, and final judgment for ambiguous pedestrian-intent prediction. It evaluates classification on 200 PSI windows and uses two human studies to assess visual hallucinations and the informational contribution of explanations.

Critical strengths
  1. The distinction between visually grounded observations and informative explanations addresses a relevant problem.

  2. The paper includes human evaluation and reports per-class results and a warrant-ablated baseline.

Critical weaknesses
  1. The methodological novelty is limited. Structured arguments as explanations and multimodal debate already appear in Cakar and Kristensson and Adhikari and Lapata. The proposed advance mainly consists of assigning these operations to prompted agents. The paper does not establish why this orchestration offers a distinct reasoning mechanism beyond structured prompting and debate.

  2. Generation order does not establish faithful reasoning. Writing observations before the final answer does not prevent an implicit prior conclusion, nor prevent the judge from independently deciding from the video and subsequently selecting a matching warrant. The abstract’s claim that the trace “cannot be a rationalization” is therefore unsupported. The authors explicitly acknowledge that they did not intervene on warrants or test post-hoc selection (section6, Limitations).

  3. The primary human-study outcome is confounded by changing the question. Part 1 asks what conclusion the reader would reach. Part 2 asks what conclusion the reasoning reaches (Appendix E, p. 16). These measure personal inference and textual attribution, respectively. Their difference cannot cleanly establish the warrant’s contribution, much less diagnose model rationalization. The automatic sentence partition underlying this comparison also lacks independent human validation.

  4. The warrant ablation does not isolate the claimed contribution. The full method supplies both a general-rule warrant and a visible applicability caveat, which the judge checks. The ablation replaces these with observations and changes judgment to evidence counting (section 3.1 and 3.2). Thus, the 6.5-point improvement cannot uniquely be attributed to the warrant. Moreover, the human studies omit this ablation, leaving the reported faithfulness improvements confounded with the broader agentic pipeline.

  5. Quantitative reporting contains material inconsistencies. The reported 85 not_sure cases out of 200 imply a 42.5% prior, not the repeatedly stated 38.5% (section 4; Tables 1 and 4). Figure 5 also appears to conflate different category sets: the three trajectory categories dominating SC-CoT account for 25% of Toulmin outputs, whereas its caption states 58%. These errors undermine interpretations based on class priors and reason distributions.

  6. The technical presentation does not adequately specify or illustrate the mechanism. The purported deterministic judgment uses undefined conditions such as “clearly stronger” and “close or mixed,” without executable thresholds or complete agent prompts (section 3.1). Figure 2 omits key caveat-checking and uncertainty branches. Figure 1(b) is itself unconvincing: parallel sidewalk walking and a rule about usually crossing at crosswalks do not establish that the pedestrian will not cross.

  7. The clarity of the images in the entire article is insufficient, and all the images have a strong AI drawing flavor, and they are not vector graphics.

Questions for authors

What intervention demonstrates that the warrants causally influence the model’s answer rather than explain a separately reached decision?

Does the human-study conclusion survive identical question wording and independently validated sentence partitioning?

Does the benefit remain when the applicability caveat and judging procedure are held fixed and only the general-rule warrant is removed?

Reviewer_U22BOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Toulmin-structured multi-agent pipeline for pedestrian crossing-intent prediction under ambiguity. Separate agents extract visual grounds, generate opposing warrants with observable applicability conditions, and re-examine the video to select a conclusion or abstain. Evaluation uses 200 selected PSI 2.0 clips, self-consistency CoT, and a warrant-ablated agentic baseline. The proposed method achieves 48.0% accuracy, compared with 41.5% and 27.0%, respectively. Expert assessments report fewer major perceptual hallucinations and greater changes in readers’ conclusions after warrants are revealed. The paper also finds that clearer, more preferred explanations can accompany less accurate predictions. Its main contribution is an inspectable inference procedure and an evaluation of perceptual grounding and explanatory information.

Critical strengths

The explicit separation of visual grounds, warrants, and applicability checks provides a concrete interface for inspecting reasoning. The warrant-ablated comparison controls agent count and computational budget, and the 6.5-percentage-point accuracy improvement provides useful evidence that the structured argument component contributes beyond agentic debate alone.

The evaluation examines evidence accuracy, the incremental contribution of reasons, and reader perceptions together. The observation that SC-CoT can receive better clarity and trust ratings despite worse predictions is consequential: subjective explanation quality can diverge from task correctness. This makes the paper informative beyond its classification results.

Critical weaknesses

The central human-study evidence has substantial validity concerns. Perceptual judgments were provided by three research-team members with method names visible. In the reasoning study, Part 1 asks what conclusion the reader would reach, whereas Part 2 asks what conclusion the reasoning reaches (Appendix E). The reported shift rate can therefore reflect a change in task as well as added information. Observation/reason segmentation was also produced by an LLM without independent human validation. These issues weaken the interpretation of the main explanatory findings.

The attribution of faithfulness gains to explicit warrants remains insufficiently isolated. Human evaluations compare the full pipeline only with SC-CoT, although these systems also differ in staged perception, agent organization, and video inspection. The warrant-ablated baseline is evaluated for classification but not perceptual faithfulness. Consequently, the accuracy ablation does not establish which component reduces hallucinations. Evidence is additionally limited to one base VLM and 200 selected clips, with only 76 clips entering the reasoning study. A targeted human evaluation of the existing ablation would materially strengthen the central contribution.

Questions for authors

Clarifications:

  1. Can you explain the different Part 1 and Part 2 question wording, report an audit of the automatic sentence segmentation, and clarify how these choices affect interpretation of conclusion shifts?
  2. Please reconcile the reported not_sure prior of 38.5% with 85/200 = 42.5%, and the Figure 5 statement that three subtypes contain 58% of Toulmin reasons with the displayed counts 33 + 3 + 14 = 50 out of 200. Additional evidence that could change my assessment:
  3. What are the warrant-ablated baseline’s perceptual-faithfulness results under the same evaluation? Please also report paired uncertainty for its 6.5-percentage-point accuracy gap from the full method.
  4. Do the main human-study differences persist with identical questions at both stages and independent, blinded assessment? These are targeted checks of the current claims, rather than requests for broad future-work extensions.
Reviewer_ypp6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a Toulmin-structured multi-agent pipeline for ambiguous pedestrian crossing intent prediction. It separates visual observations, opposing warrants, and video-grounded judgment. Evaluation on 200 PSI clips and two expert studies reports improved classification performance, fewer major perceptual hallucinations, and greater changes in readers’ conclusions after warrants are revealed.

Critical strengths
  • Explicitly separating observations from inference rules makes the resulting explanations easier to inspect and potentially correct.

  • Evaluating both prediction accuracy and human responses to explanations is valuable. The finding that clearer explanations can accompany poorer predictions highlights an important limitation of perceived explanation quality.

Critical weaknesses
  • The evidence does not establish the central faithfulness claim. Fewer perceptual errors and changes in readers’ conclusions do not demonstrate that the model’s answer causally depends on its stated warrant. Generating observations before an explicit answer also does not exclude implicit answer formation or post-hoc warrant selection by the judge. Although the limitations acknowledge this distinction, stronger claims elsewhere remain unsupported.

  • The evaluation does not adequately isolate the proposed contribution. The warrant ablation also changes how evidence is checked, and the human studies omit this ablated system. Consequently, improvements cannot be clearly attributed to explicit warrants rather than perception separation, applicability checks, or presentation differences. Moreover, the ambiguity definition places all ground-truth not_sure cases in HIGH, confounding ambiguity with class composition. The novelty also needs clearer differentiation from existing Toulmin-based multi-agent systems, particularly MADRA and SMADE-IE.

Questions for authors
  • Which components differ between the full pipeline and the warrant ablation, particularly applicability conditions and the judge’s decision procedure? Is there evidence that warrants influence model decisions beyond influencing readers?

  • Please report class counts within HIGH and LOW and paired uncertainty estimates for the 48.0% versus 41.5% accuracy comparison. How should the ambiguity-specific results be interpreted given the class-composition difference?

  • Please reconcile the reported not_sure prior of 38.5% with 85/200 = 42.5%. In Figure 5, the three trajectory-related Toulmin counts sum to 50/200 = 25%, whereas the caption reports 58%. Which figures are correct?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.