- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The work extends the study of the "Society-of-Thought" phenomenon (Kim et al 2026) in the chain-of-thought of reasoning models to VLMs. This pertains to the model exhibiting many different perspectives and personalities in its chain-of-thought before outputting the answer. The authors however do not find this phenomenon in VLMs, but instead say that society score is negatively correlated with accuracy within a model. They claim to show that when trained with an RL reward to penalise society score, the accuracy improves.
Critical strengths
The concept of "society-of-thought" and its role in the structure of the CoT seems important and worth of investigation
Critical weaknesses
For a paper showing a negative/conflicting result, this needs to be much more rigorous and exhaustive to be convincing. At the very least, since the framework is largely borrowed from Kim et al (2026), the paper should match or improve upon their analysis. Here is a few of the shortcomings:
- Models analysed are much smaller (4B - 30B with 3.3B active) compared to Kim et al (8B-671B). Pixtral and GLM are also not matched with their non-reasoning counterparts.
- Judge model used is much weaker (Qwen3-30B vs Gemini-2.5-Pro), also no validation on real ground truth. Kim et al also measure judge–human agreement and judge-judge agreement (with GPT 5.2) more thoroughly.
- No supporting analysis such as SAE feature steering, spontaneous emergence of society of thought under RL with accuracy reward, etc.
- Importantly, Kim et al seem to claim that society score is correlated with performance across models, but this paper measures correlation within the same model but on different questions. This is fundamentally a different question.
In short, the delta over Kim et al seems limited especially since the paper makes some conflicting claims.
Questions for authors
In future versions, please make sure to at least match/exceed Kim et al on all dimensions.