- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SocialAgent connects collaborative social reasoning with single-model improvement. SocialAgent-Scaffold uses role analysis, parallel role-specific reasoning, and arbitration to answer video questions without parameter updates. A curator abstracts correctly answered training examples into an Evolving Experience Pool (EEP) of reusable strategies. SocialAgent-Evolve then applies Strategy-guided Learning (SGL) to successful scaffold outputs paired with strategies, followed by Multi-strategy Policy Optimization (MSPO) on student failures. MSPO samples under alternative strategies, normalizes rewards across each question's rollouts, and updates a shared policy. At inference, the student retrieves one strategy and generates one response. The authors report improvements on Social-IQ 2.0 and scaffold-only MuMA-ToM evaluation, together with reduced inference latency after internalization.
Critical strengths
-
S1: A coherent connection between collaboration, memory, and learning. Sections 3.2–3.3 (pp. 4–7) explain how the same experience pool guides both collaborative reasoning and subsequent student refinement. Using the same backbone for scaffold agents and the student makes the approach relevant to settings without a stronger external teacher.
-
S2: Encouraging reported performance and inference savings. Table 1 (p. 8) reports Social-IQ 2.0 multiple-choice accuracy increasing from 63.49% for the backbone to 71.04% for Scaffold and 83.49% for Evolve. Evolve also exceeds the same-backbone vanilla SFT+GRPO baseline by 3.13 points. The reported latency reduction from 72.29 s to 15.35 s illustrates a practical benefit of replacing repeated collaboration with single-agent inference.
-
S3: A useful controlled exploration comparison. Table 4 (pp. 8–9) holds the total at eight rollouts per question while varying strategy allocation; two strategies with four rollouts improve accuracy by 2.02 points over one strategy with eight. Appendix C (pp. 14–15) also specifies a frozen training-derived EEP and excludes same-video/question entries during MSPO retrieval. These are meaningful controls, although they do not resolve the separate issues below.
Critical weaknesses
-
W1: MuMA-ToM overall scores cannot be reconciled with the stated evaluation. Table 1 (p. 8) reports Scaffold category accuracies of 81.68%, 54.95%, and 46.57%, whose mean is 61.07%, versus an Overall of 56.33%. The backbone similarly gives 54.27% by averaging its categories, versus 49.78% reported. The original MuMA-ToM benchmark contains 300 questions per category, so full-set overall accuracy should equal that mean. Section 4.1 and Appendix C do not specify alternative subsets or denominators. This is a substantial, potentially repairable reporting/protocol issue; it does not erase the reported improvements within every category. Provide evaluated counts, aggregation rules, and corrected values if needed. The broad state-of-the-art wording (pp. 1, 3) also needs qualification: the benchmark paper reports LIMP at 76.6% overall. Its different backbone and inference procedure preclude a simple controlled comparison, but Table 1 establishes superiority only over the selected baselines.
-
W2: The contribution of pairing strategies with demonstrations is not isolated. Section 3.3.1 (p. 6, Eqs. 6–7) attributes value to combining scaffold trajectories with their strategies. However, every row in Table 3 (p. 8) already starts with SGL. Table 1 includes vanilla SFT, but neither Section 4.1 nor Appendix C defines whether that baseline uses identical successful examples, scaffold reasoning targets, optimization settings, and test-time retrieval. Consequently, the SGL improvement cannot be assigned specifically to strategy pairing rather than demonstration quality, filtering, or retrieval differences. This is a moderate, repairable attribution limitation, not evidence that the complete system is ineffective. Clarify the baseline recipe; if it is not matched, compare trajectory-only learning with strategy-paired learning under otherwise equivalent conditions, or narrow the mechanism claim.
-
W3: The open-ended evaluation protocol is insufficiently specified for the headline comparison. Section 4.1 (p. 7) describes GPT-3.5-turbo judging, whereas the supplied Role-Play/Arbiter prompts (Appendix D, Fig. 7–8, p. 16) include answer choices and request an answer index. No separate open-ended generation prompt or exact scoring prompt is provided. It therefore remains unclear whether open-ended responses are generated without choices, conditioned on choices, or obtained by converting selected options into text. These settings test different capabilities and affect interpretation of the 56.10% versus 54.90% VEGAS comparison. Giving options to the judge is consistent with the cited evaluation tradition and is not itself an error. This is a substantial but potentially clarification-resolvable reproducibility issue. Provide the generation/scoring data flow, evaluation split and sample counts, and identify which comparator scores were recomputed on identical examples versus taken from prior reports.
-
W4: Closely related experience-learning and multi-agent distillation work is missing. Section 2 (pp. 3–4) emphasizes social benchmarks and multimodal models but does not discuss Zhao et al.'s ExpeL: LLM Agents Are Experiential Learners (AAAI 2024), which extracts reusable insights from training experiences without parameter updates, or Luo et al.'s AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agent (first posted February 2026), which filters multi-agent reasoning traces and studies supervised and reinforcement-based distillation. These overlap with EEP and the collaboration-to-student motivation, respectively, and are absent from the bibliography (pp. 10–12). Differences remain: ExpeL targets interactive agents, while AgentArk uses different teacher/distillation machinery; SocialAgent adds social role decomposition, explicit strategy conditioning, and failure-focused cross-strategy exploration. This is a moderate positioning gap, not proof of redundancy. Subject to the applicable cutoff, explain those distinctions and identify the methodological advance beyond combining existing components; missing citations alone do not mandate new experiments.
Questions for authors
- W1: What category sample counts and aggregation rule produce the MuMA-ToM Overall column? Can the discrepancy with the category means be reconciled from existing evaluation records?
- W2: Does vanilla SFT use the same filtered scaffold examples and reasoning targets as SGL? What differs in training configuration and test-time strategy retrieval?
- W3: Please specify whether answer choices are visible during open-ended generation, what text is scored, and the exact judge prompt. Which split/examples are evaluated, and which baseline results were reproduced under that setup?
- W4: What distinguishes EEP and SMSR from ExpeL-style experience abstraction and AgentArk-style multi-agent distillation beyond the social-video application?