acceptodds
Withdrawn from ICLR 2027

The Role of Society of Thought in Vision-Language Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reasoning models sometimes produce chain-of-thought traces that look less like a single linear solution and more like a dialogue between internal perspectives. Prior work by Kim et a. (2026) has studied this “Society of Thought” (SoT) behavior in text only models, but it remains unclear whether the same phenomenon appears in vision-language models (VLMs), where reasoning must stay grounded in visual evidence. We analyze how this behavior varies across model families, post-training variants, tasks, difficulty, and answer correctness. Society-like reasoning appears across models, but is most visible in models trained to deliberate and in traces for more difficult benchmark items. In contrast to earlier findings in text only models, higher society scores are associated with lower accuracy, including when questions are separated by difficulty. We further distinguish the presence of multiple voices from whether different perspectives communicate and reach a coherent conclusion. Finally, we use society-based rewards to test whether the observed behavior can be changed through reinforcement learning. The rewards successfully alter the reasoning traces and produce modest performance gains without using answer correctness as a training signal. These results show that SoT is present and steerable in VLM reasoning, while its benefit depends on how the different perspectives are coordinated.

Primary area:
interpretability and explainable AI
Keywords:
VLMs, Society of Thought, Multimodal Reasoning, Chain-of-Thought, Multi-Persona Reasoning, Internal Dialogue, Reward Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HitROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The work extends the study of the "Society-of-Thought" phenomenon (Kim et al 2026) in the chain-of-thought of reasoning models to VLMs. This pertains to the model exhibiting many different perspectives and personalities in its chain-of-thought before outputting the answer. The authors however do not find this phenomenon in VLMs, but instead say that society score is negatively correlated with accuracy within a model. They claim to show that when trained with an RL reward to penalise society score, the accuracy improves.

Critical strengths

The concept of "society-of-thought" and its role in the structure of the CoT seems important and worth of investigation

Critical weaknesses

For a paper showing a negative/conflicting result, this needs to be much more rigorous and exhaustive to be convincing. At the very least, since the framework is largely borrowed from Kim et al (2026), the paper should match or improve upon their analysis. Here is a few of the shortcomings:

  1. Models analysed are much smaller (4B - 30B with 3.3B active) compared to Kim et al (8B-671B). Pixtral and GLM are also not matched with their non-reasoning counterparts.
  2. Judge model used is much weaker (Qwen3-30B vs Gemini-2.5-Pro), also no validation on real ground truth. Kim et al also measure judge–human agreement and judge-judge agreement (with GPT 5.2) more thoroughly.
  3. No supporting analysis such as SAE feature steering, spontaneous emergence of society of thought under RL with accuracy reward, etc.
  4. Importantly, Kim et al seem to claim that society score is correlated with performance across models, but this paper measures correlation within the same model but on different questions. This is fundamentally a different question.

In short, the delta over Kim et al seems limited especially since the paper makes some conflicting claims.

Questions for authors

In future versions, please make sure to at least match/exceed Kim et al on all dimensions.

Reviewer_KVWGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the Society of Thought (SoT) phenomenon previously characterized in text-only reasoning models extends to vision-language models. The authors evaluate five VLMs across MMMU, MMBench, and MMLU, introducing two judge-based measurements: v1, which measures visible multiplicity of perspectives, conflict, and persona separation; and v2, which emphasizes productive communication and reconciliation. The paper reports three main findings. First, society-like structures are more prevalent in models trained for extended deliberation and on difficult questions. Second, higher v1 scores are negatively associated with answer accuracy, while v2 exhibits a more complex relationship with correctness. Third, reinforcement learning with a reward that penalizes v1 reduces the measured society structure and produces modest accuracy improvements without using correctness labels.

Critical strengths

The paper examines whether the SoT phenomenon described by Kim et al. (2026) generalizes to multimodal reasoning. The reported negative association between society scores and correctness contrasts with the findings motivating the study.

Critical weaknesses

The operational definition of Society of Thought does not sufficiently distinguish multiple perspectives from ordinary reasoning operations. The relationship between society scores and accuracy remains confounded by reasoning length, difficulty, and judge information. The RL intervention demonstrates steerability of the score, but does not isolate the mechanism responsible for accuracy improvements. The evidence that coordinated society-like reasoning is beneficial is weaker than the discussion suggests. The central construct is not yet separated convincingly from ordinary self-correction, verbosity, and answer uncertainty. The observational evidence is therefore difficult to interpret mechanistically, while the RL intervention lacks the principal control needed to distinguish suppression of society-like behavior from generic trace compression.

Questions for authors

Does the negative v1–accuracy association remain after controlling for trace length and independently estimated item difficulty? Does negative-v1 RL outperform a length-only reward under comparable training conditions?

Reviewer_chWqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the "Society of Thought" (SoT) phenomenon in VLMs. Using an LLM judge across five VLMs and three benchmarks, the authors measure visible multi-persona structure (v1v_1) and propose an 8-axis communication/reconciliation rubric (v2v_2). Unlike prior findings in text-only models, higher v1v_1 correlates with lower accuracy in VLMs, while accuracy under v2v_2 recovers only when perspectives are reconciled. Using RL (GDPO), the authors show that penalizing v1v_1 (v1−v_1^-) compresses reasoning traces and yields modest accuracy gains.

Critical strengths
  1. Insightful counter-finding: Demonstrates that visible multi-voice structure in VLMs primarily reflects uncertainty and reasoning drift rather than productive deliberation, offering a grounded counterpoint to prior text-only SoT claims.

  2. Clear conceptual distinction: Separating surface-level persona multiplicity (v1v_1) from perspective reconciliation and closure (v2v_2) is well-motivated.

  3. Transparent Appendix Controls: Commendable reporting of random-reward controls, ground-truth sensitivity checks, and negative v2+v_2^+ RL outcomes in the appendices.

Critical weaknesses

W1. Confounded cross-model comparisons: In Table 1, higher v1v_1 scores in Qwen3-VL-4B variants are attributed to post-training rather than scale. However, Appendix D.1 reveals that max_new_tokens was increased 4×4\times exclusively for Qwen3-VL-4B models, alongside mismatched temperatures (e.g., 1.0 for Thinking vs. 0.7/0.8 for others). Longer budgets and higher sampling entropy naturally induce more self-corrections and stance shifts, confounding RQ1/RQ2.

W2. Missing length-penalty RL baseline: Penalizing v1v_1 cuts trace length by 40%–50%. Since prior VLM work shows that simply shortening CoT traces mitigates visual forgetting and improves accuracy, the lack of a rule-based Length-Penalty RL control makes it unclear whether the modest accuracy gains stem from suppressing SoT or merely from trace compression.

W3. Unlike v1v_1, the 8-axis v2v_2 rubric has zero human calibration. Moreover, Table 9 shows extreme bucket sparsity, making intermediate points in Figure 3 (right) noisy. Additionally, online RL swaps the 30B judge for Qwen3.5-4B without reporting inter-judge agreement.

W4. Although Section 3.2 argues that v2v_2 captures productive reasoning, the main RL experiments only use v1−v_1^-; v2+v_2^+ RL is relegated to Appendix Table 11, where raw v2+v_2^+ degrades Instruct accuracy (0.564 to 0.529). Furthermore, SoT is evaluated purely via textual discourse markers (Figure 2 even uses a text-only MMLU prompt) without analyzing visual grounding or attention.

Questions for authors

please see the weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.