acceptodds
Under review as a conference paper at ICLR 2027

SocialLab: An Data-Grounded Framework for Automated Empirical Research in the Social Sciences

Abstract

Large language models (LLMs) are enabling new approaches to autonomous scientific research. However, empirical social science remains limited by the difficulty of transforming fragmented real-world evidence into reliable research variables and integrating evidence construction with research design, empirical reasoning, and scientific interpretation. To address this challenge, we introduce SocialLab, a data-grounded multi-agent framework for autonomous empirical research in the social sciences. SocialLab coordinates specialized agents across the entire research pipeline, including research planning, evidence construction, empirical analysis, and scientific review. Specifically, the Plan Agent identifies evidence requirements and analysis strategies, the Data Agent constructs and analyzes research datasets from heterogeneous sources, and the Advisor Agent independently reviews research plans, data decisions, and empirical conclusions. A Coordinator orchestrates these agents through a closed-loop workflow, while maintaining alignment with research objectives. We further introduce CSD-Bench, a benchmark for evaluating open-ended empirical research capabilities in the social sciences under heterogeneous, multi-source data settings. Experiments on ReplicatorBench show average gains over Replicator of 18.21, 43.99, 11.14, and 33.14 points in information extraction, research design, execution, and interpretation, respectively. On CSD-Bench, SocialLab achieves a mean quality–coverage harmonic score of 89.75 and a mean research evidence utility score of 88.89. Finally, a case study reveals how individual agents contribute to research design, data validation, and evidence grounding.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Autonomous Scientific Research, Empirical Social Science, Multi-Agent Systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QyoMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces SocialLab, a multi-agent framework for autonomous empirical research in the social sciences using heterogeneous data. It coordinates research planning, data construction and analysis, and independent review in a structured framework. For evaluation, the authors introduce CSD-Bench, a six-task benchmark including cross-source evidence construction and research evidence utility. Comprehensive evaluations show improvements over Replicator in research reconstruction and over tool-augmented LLM and OpenCode baselines on CSD-Bench. Component ablations and a public-attitudes case study further examine the contributions of planning, specialized data execution, and independent review.

Critical strengths
  1. The paper addresses evidence validity in empirical social science research through a workflow that connects research requirements, data construction, and the review of conclusions.
Critical weaknesses
  1. The evaluation lacks comparisons with relevant frameworks such as data-to-paper and Data Interpreter [2], leaving SocialLab’s added value over existing research and data analysis workflows insufficiently established.
  2. The CSD-Bench evaluation relies on rubric-based LLM judgments without reported validation against expert assessments or tests of their ability to detect known methodological errors. It therefore remains unclear whether the reported score improvements translate into fewer substantive errors and more reliable empirical conclusions, weakening support for the paper’s central claim.

[1] Ifargan, Tal, et al. "Autonomous LLM-driven research—from data to human-verifiable research papers." NEJM AI 2.1 (2025): AIoa2400555.

[2] Hong, Sirui, et al. "Data interpreter: An llm agent for data science." Findings of the association for computational linguistics: ACL 2025. 2025.

Questions for authors
  1. Could you explain how targeted revision works in practice? How is feedback linked to specific processing steps, which analyses are rerun, and how do you check whether the revisions meet the acceptance criteria? How does this differ from a standard review-and-revise loop?
  2. When were the benchmarks and their source materials created or published relative to the models’ training cutoffs? Did you check whether the models might have seen the source papers, reference analyses, or replication code during training?
  3. Why did you use GPT-5.6-luna to evaluate utility on CSD-Bench, but GPT-5.6-sol and Gemini-3.8-flash on ReplicatorBench? Have you checked whether using different evaluators changes the conclusions?
10 comments

Thank you for suggesting data-to-paper and Data Interpreter and for asking what our reviews catch in practice.

Response to Weakness 1:

The evaluation lacks comparisons with relevant frameworks such as data-to-paper and Data Interpreter [2], leaving SocialLab’s added value over existing research and data analysis workflows insufficiently established.

We ran data-to-paper and Data Interpreter on the six CSD-Bench briefs with DeepSeek-V4-Flash. All five judges gave SocialLab higher mean H and U scores than either baseline under the same rubric. For data-to-paper we used the fixed-goal hypothesis-testing workflow with Crossref retrieval and adaptations for execution and reporting. We gave Data Interpreter staged requests for data construction, modeling, and reporting. The additional runs used archived inputs, including GDELT data from sampled dates for T2. In T3, Data Interpreter received four separately normalized source tables and performed the merge and analysis.

data-to-paper provides a workflow from annotated input data to a traceable manuscript, including hypothesis testing and statistical checks. Data Interpreter decomposes data-science work into a graph with stepwise execution and verification. SocialLab adds a research-design and review layer around cross-source evidence construction. Its approved plan specifies the construct, sample and estimand. The data package records how the sources were aligned and measures constructed. The Advisor checks whether the executed results support the claim and can send a specific step back for revision.

Each cell is H / U (0–100). Luna is the judge used for the submitted CSD-Bench results. Sol, Grok 4.7, Gemini 3.8 Flash, and Kimi K3.1 Pro independently scored the same six cases per baseline.

MethodTaskLunaSolGrokGeminiKimi
SocialLabT195.65 / 100.0088.11 / 83.3397.14 / 100.00100.00 / 100.0095.65 / 100.00
SocialLabT285.71 / 75.0088.11 / 83.3386.42 / 83.33100.00 / 100.0091.58 / 91.67
SocialLabT395.65 / 91.6794.12 / 100.0089.23 / 75.00100.00 / 100.0090.91 / 83.33
SocialLabT490.91 / 75.0087.50 / 75.0097.22 / 91.67100.00 / 100.00100.00 / 100.00
SocialLabT585.30 / 83.3375.76 / 66.6791.33 / 75.00100.00 / 100.0091.58 / 83.33
SocialLabT697.14 / 100.0087.50 / 83.3398.59 / 100.00100.00 / 100.0097.14 / 100.00
SocialLabMean91.73 / 87.5086.85 / 81.9493.32 / 87.50100.00 / 100.0094.48 / 93.06
DIT160.28 / 41.6736.36 / 25.0070.51 / 66.6789.74 / 91.6775.05 / 58.33
DIT255.81 / 58.3348.15 / 50.0068.44 / 58.3387.50 / 91.6777.11 / 66.67
DIT364.81 / 66.6760.00 / 66.6770.51 / 58.3395.65 / 100.0070.15 / 66.67
DIT437.04 / 58.3338.72 / 33.3351.46 / 50.0055.00 / 66.6739.68 / 33.33
DIT546.85 / 41.6747.53 / 41.6757.11 / 66.6743.77 / 33.3342.44 / 41.67
DIT645.40 / 16.6760.00 / 41.6755.28 / 33.3371.26 / 58.3356.57 / 0.00
DIMean51.70 / 47.2248.46 / 43.0662.22 / 55.5673.82 / 73.6160.17 / 44.44
DTPT148.75 / 50.0035.48 / 25.0060.00 / 33.3365.76 / 33.3350.56 / 16.67
DTPT247.51 / 41.6747.51 / 41.6760.99 / 58.3380.79 / 75.0063.77 / 50.00
DTPT328.42 / 16.6730.77 / 16.6710.00 / 0.0031.28 / 0.0023.15 / 0.00
DTPT449.86 / 25.0046.14 / 8.3352.42 / 33.3352.85 / 25.0061.70 / 16.67
DTPT563.83 / 58.3360.00 / 50.0067.80 / 58.3363.89 / 58.3362.22 / 66.67
DTPT656.22 / 58.3358.14 / 50.0072.12 / 75.0086.95 / 91.6768.44 / 66.67
DTPMean49.10 / 41.6746.34 / 31.9453.89 / 43.0663.58 / 47.2254.97 / 36.11

The primary judge, Luna, gave SocialLab 91.73/87.50, DI 51.70/47.22, and DTP 49.10/41.67. Across judges, SocialLab's mean lead over DI was 26.18–40.03 points in H and 26.39–48.62 in U. Its lead over DTP was 36.42–42.63 points in H and 44.44–56.95 in U. SocialLab had a higher H score in all 30 task-by-judge comparisons with each baseline. For U it led in all 30 comparisons with DTP and in 29 with DI, with one tie.

The largest differences in T3 concern the evidence available for analysis. DI linked the normalized source tables into 714 state-year observations across 51 jurisdictions. Gemini scored its U at 100, tying SocialLab. DTP's run from the archived source files produced zero usable state-year observations after integration. Its U ranged from 0 to 16.67. DTP did substantially better on T6, where Gemini gave it 86.95/91.67. For T3, the DI and DTP scores reflect different starting inputs as well as the integration each workflow achieved.

Response to Weakness 2:

The CSD-Bench evaluation relies on rubric-based LLM judgments without reported validation against expert assessments or tests of their ability to detect known methodological errors.

The original GPT-5.6-luna grader found a measurement error in the DeepSeek–OpenCode run for T4. The report used vote-weighted dispersion of party positions as a sensitivity check, but the code centered the squared deviations on the unweighted mean. This inflated the dispersion. Luna cited the code and downstream results and scored C3 (measurement construction and weighting) 1/4. Sol, Grok, and Kimi flagged the same error when given the same scoring packet, so four of the five judges picked it up.

Using the saved party-level data, we reproduced all 112 original country-by-wave-by-policy-dimension dispersion values. Holding the sample and vote weights fixed, we recomputed the deviations around the weighted mean. The correction reduced dispersion in all 112 cells. The table shows one country-wave example and the effect on the reported correlations with income inequality in both policy dimensions.

T4 quantityOriginal implementationAfter correcting the weighted center
Hungary 2019 sociocultural dispersion3.85952.6225
Spearman ρ, economic dispersion and Gini−0.1582−0.1792
Spearman ρ, sociocultural dispersion and Gini−0.3116−0.2895

Both correlations remained negative after correction. What changed was the magnitude of the weighted sensitivity estimates that the report relied on.

We also checked one inference requirement across all six archived T4 main runs. For each principal inequality regression, we checked whether the analysis used a separate cross-section for each wave or accounted for repeated country observations when pooling waves. Three runs met this check and averaged 81.69 H / 80.56 U. The three runs reporting pooled HC1 inference averaged 68.02 H / 72.22 U. Luna gave the three runs in the first group 4, 3, and 2 out of 4 on AN2 (inference), and 1 out of 4 to each run in the second group. For DeepSeek–OpenCode, we reproduced the reported estimate for sociocultural dispersion and then clustered by country on the same sample. The Gini coefficient remained −0.06075, and the p-value moved from 0.0126 to 0.0300 across 54 observations in 27 countries. The group with the stronger inference practice averaged 13.67 H and 8.34 U points higher. The AN2 scores distinguish the groups more clearly than the total scores. In the re-estimation, clustering changed the uncertainty without changing the coefficient, which is precisely what the inference item is meant to check.

We have also completed a human evaluation of the generated research artifacts, in addition to the expert participation in prompt design described in k5Gh Q1. Five evaluators with relevant backgrounds, including four at the doctoral level and one at the master's level, each scored all 36 anonymized main-experiment artifacts using the same rubric as the model judges. The table reports their mean scores across all six tasks and both generation backbones.

WorkflowHuman HHuman U
SocialLab92.1890.28
OpenCode86.7985.56
Tool-augmented LLM91.1085.42

The human panel ranked SocialLab first on both pooled measures. Across the 36 artifacts, the five-person human means and the five-model judge means had Spearman correlations of 0.621 for H and 0.412 for U. These results provide direct human support for the aggregate workflow comparison, complementing the code-level checks of methodological errors reported above.

Response to Question 1:

Could you explain how targeted revision works in practice?

The Advisor tells the Coordinator which processing step needs to change, which results to preserve, and what to check after the rerun. If the data change, the responsible agent rebuilds the affected sample and reruns the estimates that depend on it. If the issue is in analysis, the agent starts from the approved data package. The Advisor applies the recorded tests to the new code and results and checks the wording against them.

Check failed → identify step + required change + acceptance test
              → rerun affected sample / estimates / tables
              → verify test in outputs → update corresponding claim

In T5, the first Advisor analysis review found that the estimator compared integer Eurostat break flags with a string. The purported exclusion removed zero rows and returned the primary fit. The Advisor issued a revise decision naming the flag-reading step, requiring a real exclusion and refit, and holding the figure and writing stages until review. The Analysis Agent corrected the flag parsing and reran the break sensitivities from the frozen panel using current, lag-1, and lag-2 flags. The affected vote-weighted economic-polarization estimate for lagged unemployment changed as follows.

T5 checkObservationsRows excludedβCountry-clustered p
Original exclusion check6600.0373090.043585
Revised exclusion check5790.0386000.062669

Before approving the rerun, the Advisor independently verified the nine removals, reproduced the displayed refit, and checked the 96 reissued break-sensitivity results and exact agreement on all 720 protected result rows. The final report disclosed that the finding was sensitive to the exclusion. The revision was scoped to the affected artifact and its downstream outputs. The workflow reused unaffected results and approved the rerun against the acceptance tests recorded at the first review.

Response to Question 2:

When were the benchmarks and their source materials created or published relative to the models’ training cutoffs?

The ReplicatorBench preprint appeared on 11 February 2026, after the 39 source studies had been published. We verified that the reconstruction runs received the original paper, the focal claim, and the benchmark data package. The evaluator read the expert preregistration and replication report only after the run. Public papers, analyses, and replication code could still have appeared in training. (See ReplicatorBench data and evaluation protocol.)

We used the six CSD-Bench briefs for our September 2026 submission experiments. Each workflow had to build its own joins, measures, and estimates, which we scored using the saved artifacts. For T1 the run used GSS Release 3a, published in July 2026, and matched 37,583 respondent records to macroeconomic data for 13 survey years. The evaluation checked that join and the estimates produced from it. (See GSS release record.)

These dates establish when the materials were public. DeepSeek reports that it collected some training data by about May 2025 and gives no cutoff for the whole corpus. The Qwen3.8-Flash model card gives no training cutoff. We therefore cannot determine from the published information which source papers, reference analyses, or replication code the models encountered in training. (See DeepSeek training disclosure and Qwen model card.)

Reviewer_zabXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new multi-agent framework for data-grounded multi-agent framework for autonomous empirical research in the social sciences, called SocialLab. It further introduces CSD-bench, a benchmark to evaluate such research under heterogeneous data settings. Experiments on both ReplicatorBench and CSDBench showed that SocialLab is outperforming existing approaches.

Critical strengths

The design of the multi-agent system is sound, and the experiments are comprehensive and covers all required aspects such as ablation and comparison to coding agents.

The construction of a new dataset CSD-Bench is also useful for the community.

The paper is very easy to follow and read!

Critical weaknesses

My only concern is the paper did not evaluate against SOTA coding agents such as Codex/Claude Code. Some of the core features, such as planner-data-agent decomposition, is available in Codex, and I'd be very curious to know how SocialLab compares with it. Alternatively, the author could include a comparison between OpenCode against Codex to help users understand if they have similar functionalities, or if any significant part of Codex's capability is missing from OpenCode. The paper also did not evaluate SOTA model's performance on this benchmark.

Questions for authors

What backbone model are the approaches in Table 1 based on? This does not seem immediately clear from Section 4.

Have you tried SOTA/frontier models for experiments in Table 1 or Table 2?

4 comments

Thank you for the careful reading and comments.

Response to Weakness 1:

My only concern is the paper did not evaluate against SOTA coding agents such as Codex/Claude Code.

Codex and OpenCode both provide general-purpose agent harnesses for executing tasks, managing context, and using tools. Codex can also be optimized for specific model pairings. For example, OpenAI reports adapting the Codex harness for GPT-5.3-Codex. OpenCode supports multiple model providers, which lets us run both of our generation backbones through the same general-purpose harness.

We chose OpenCode 1.18.4 to measure what DeepSeek-V4-Flash and Qwen3.8-Flash can accomplish on empirical research tasks in a general-purpose harness. In Table 2, each backbone runs in both OpenCode and SocialLab with the same briefs and access to the same search and code-execution backends. This comparison measures how the resulting research artifacts change when the workflow adds domain-specific design, data provenance, and scientific review. Appendix F traces those steps in T1 from the survey-year join through few-cluster inference to the final interpretation.

With DeepSeek-V4-Flash, SocialLab scored 91.73/87.50 and OpenCode 72.20/80.56 (H/U), a gain of 19.53 H and 6.94 U points. With Qwen3.8-Flash, the scores were 87.77/90.28 and 75.38/75.00, a gain of 12.39 H and 15.28 U points. The end-to-end workflow improved both means with each backbone. The larger gap is in evidence construction with DeepSeek and in evidence utility with Qwen. Our reply to QyoM W1 reports the comparison with two research-oriented workflows.

Response to Question 1:

What backbone model are the approaches in Table 1 based on?

Each backbone row in Table 1 uses the same generator for Replicator and SocialLab. GPT-5.6-sol and Gemini-3.8-flash supplied the reconstruction scores.

Table 1 rowReplicator generatorSocialLab generator
DeepSeekDeepSeek-V4-FlashDeepSeek-V4-Flash
QwenQwen3.8-FlashQwen3.8-Flash
PooledMean over both backbonesMean over both backbones

We will put the full generator names and the evaluator names directly in the Table 1 caption or notes.

Response to Question 2:

Have you tried SOTA/frontier models for experiments in Table 1 or Table 2?

The generation runs in both tables used DeepSeek-V4-Flash and Qwen3.8-Flash. Table 1 uses the evaluators named above, and Table 2 uses GPT-5.6-luna. The additional judges rescore existing CSD-Bench artifacts. The new data-to-paper and Data Interpreter artifacts were generated with DeepSeek-V4-Flash. We have no generation runs with Codex, Claude Code, or a third backbone in these tables. The cost of this work has already exceeded US$1,000; if the paper is accepted, we will consider adding experiments with SOTA/frontier models.

Dear Reviewer zabX,

Thank you again for your thoughtful feedback. In our responses, we clarify which models generated the research outputs and which models scored them in Tables 1 and 2. We also explain why we used OpenCode to compare SocialLab with a general-purpose agent harness under the same generation models.

As the discussion period draws to a close, we would appreciate any further feedback you may have. Please let us know if any points remain unclear or require further discussion. We would be happy to clarify them.

Best regards, The Authors

I have read through the author's response. I am keeping my positive score and will not further increase it, since no further experiments against codex were conducted.

Thank you for taking the time to review our responses and for maintaining your positive assessment. We appreciate your constructive feedback on comparisons with frontier coding agents and will clarify the scope of our evaluation in the revised manuscript.

Reviewer_k5GhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SocialLab, a data-grounded multi-agent framework with Plan/Data/Advisor agents plus a Coordinator, which uses inspectable data packages and feedback routing to keep research requirements, constructed evidence, and empirical claims consistent. It also introduces CSD-Bench, a six-task benchmark for cross-source evidence construction. On ReplicatorBench, SocialLab improves all four stages (design +43.99, interpretation +33.14); on CSD-Bench it achieves H=89.75H=89.75 and U=88.89U=88.89, outperforming baselines.

Critical strengths
  1. The problem is real: under-specified research requirements, evidence drift, and non-traceable corrections.
  2. The framework is cleanly specified; the data package B=(XK,ρP,L)B=(X_K,\rho_P,\mathcal{L}) and the Advisor review formulation are technically coherent.
Critical weaknesses
  • W1: The central claim is not directly tested — design soundness, revision closure, and social-science validity all lack evidence. The paper claims to support "reliable open-ended social-science research," but no experiment directly tests the transformation from an open topic to a sound research design.

  • W2: Baseline comparison does not control for compute, and CSD-Bench's scale and "open-ended" framing are mismatched. SocialLab uses multiple agents and multiple review rounds, while the baselines are a minimal tool-use loop and a general-purpose harness. The paper reports no token, tool-call, or LLM-call budgets and provides no compute-matched baseline, so it is unclear how much of the gain comes from coordination/review mechanisms versus additional inference compute. CSD-Bench has only six tasks with given sources, time ranges, and target relationships, making it closer to semi-structured cross-source tasks than open-ended research; no variance or confidence intervals are reported, so task-level conclusions have limited statistical robustness.

  • W3: LLM-judge and ablation designs raise validity and attribution concerns. HH and UU are graded by LLMs against a rubric that is structurally isomorphic to the Plan Agent's specification, creating a circularity risk; there is no human–LLM agreement study, no inter-judge agreement, and no judge-swap robustness check. Ablations use only four DeepSeek tasks and a single backbone, and remove components in bundles (e.g., removing the Data Agent also removes its execution, guidance, interfaces, and review gates), so contributions cannot be cleanly attributed; Qwen and T2/T6 are not covered.

Questions for authors
  1. Can the authors provide human-expert evaluation of Plan Agent designs?
  2. Can the authors show multiple competing designs for the same topic and how the system recognizes or selects among them?
12 comments

Thank you for asking what the experiments establish about the reliability of the research, the cost of the workflow, and the validity of the evaluation.

Response to Weakness 1:

For design soundness, our framework audits every stage of social science research to ensure its reliability, including the link between the question, the data, and the proposed test. The Plan Agent specifies the unit of observation, measures, target relationship, and analysis. The Advisor checks that the available data can support the proposed measures and sample and that the test addresses the stated relationship. The approved plan then gives the Data Agent explicit requirements to follow and the Advisor a basis for checking whether the completed analysis implemented them (Sections 3.2 and 3.4).

For revision closure, SocialLab verifies whether the revisions have resolved the original problem after identifying and fixing the issues. The request also identifies the affected artifact and the constraints that must be preserved. The Coordinator sends it to the responsible step, and the Advisor checks the corrected output against the recorded criterion. It also reassesses any estimates or conclusions that depend on the change before approving the revision (Section 3.4). We can therefore verify that the original issue was resolved and that its consequences for the analysis were addressed.

For social-science validity, the report generated by SocialLab have a checkable empirical basis because the Data Agent records how the sample was built and links the estimates, uncertainty, and sensitivity results to the actual data and code. This record lets the Advisor verify that the measures represent the intended constructs, the inference fits the way the data are structured, and the results support the strength and scope of each reported claim (Sections 3.3 and 3.4). Thus, the validity and scope of each claim can be checked against its supporting evidence.

The paper claims to support "reliable open-ended social-science research," but no experiment directly tests the transformation from an open topic to a sound research design.

SocialLab supports open-ended social-science research because the input defines a research question and scope while leaving the empirical design to the system. The workflow must determine how to operationalize the concepts, which observations and sources can be combined, which estimand addresses the question, and which analyses and sensitivity checks are appropriate. These choices can lead to several defensible designs for the same question. Evaluating the resulting research therefore requires checking how those decisions produce evidence that answers the question.

To resolve these research decisions left open by the task prompt, SocialLab’s Plan Agent reviews the literature and explores available data to determine feasible observation units, measurements, target relationships, and analysis methods. The system therefore develops the empirical design itself. After Advisor Agent review, the Data Agent links sources and constructs measurements according to actual data coverage, then conducts the analysis and sensitivity checks. If the data or analysis reveal problems in the design, the Advisor Agent specifies the required changes, and the Coordinator routes them to the responsible step so that the plan and downstream results are updated in light of the evidence. Following approval, the Figure, Writing, and Paper Agents prepare the visualizations and report. Thus, the workflow resolves the design and analysis choices left open by the task and turns them into inspectable research evidence through review and revision, supporting “reliable open-ended social-science research.”

The reliability of social science research is supported by concrete verification at each stage. During planning, the Advisor Agent checks whether available data support the proposed measurements and whether the analysis addresses the question, establishing empirical feasibility. During data preparation, the Data Agent records each operation’s inputs, outputs, and validation results, while the Advisor Agent checks sample fidelity and construction consistency, making linkage and measurement decisions traceable. During analysis, each estimate is linked to its code, data, fitted sample, uncertainty diagnostics, and sensitivity results; the Advisor Agent uses these to assess the inference and support for the claims. During revision, feedback specifies acceptance criteria, and revised artifacts and affected downstream results must pass reassessment before advancement. Thus, design, data construction, inference, and revision all have explicit review objects and acceptance conditions, and conclusions can be traced to reviewed execution evidence. These mechanisms support the “reliable” part of our claim.

Response to Weakness 2:

Baseline comparison does not control for compute

SocialLab uses multiple agents and multiple review rounds, while the baselines are a minimal tool-use loop and a general-purpose harness.

The paper reports no token, tool-call, or LLM-call budgets and provides no compute-matched baseline, so it is unclear how much of the gain comes from coordination/review mechanisms versus additional inference compute.

SocialLab uses substantially more inference because its workflow constructs, checks, and revises empirical evidence before delivering a report. Our comparison evaluates the quality delivered by each complete workflow under the same task prompts, generation backbones, and search and code-execution backends. We imposed no task-level token cap on the baselines: OpenCode and the tool-augmented LLM could continue calling their models and tools until they submitted a final response. None of the 24 baseline runs reached the runner’s applicable operational limits (six hours per run and, for direct tool use, 500 requests); the maximum runtime was approximately 3 hours and 37 minutes, and the largest direct-tool run recorded 193 model responses. Their lower usage therefore reflects their completed workflows’ stopping decisions.

The table reports per-task averages from all 36 run logs, excluding judge calls. Totals include cached input and logged tool errors and cover the full run, including any work after the scored artifact was frozen. M denotes million tokens.

WorkflowDeepSeek total (M)Qwen total (M)Pooled total (M)Cached input (M)Model response recordsTool returns
SocialLab112.80432.51272.66260.692377.332828.75
OpenCode4.5412.298.418.01113.83119.25
Tool-Augmented LLM6.0726.2516.1615.50123.83132.50

SocialLab records 272.66M tokens per task on average, compared with 8.41M for OpenCode and 16.16M for the tool-augmented LLM, or 32.4 and 16.9 times their usage. Cached input accounts for 95.6% of SocialLab’s total. The role breakdown shows which research activities account for this expenditure.

SocialLab runtime roleDeepSeek (M)Qwen (M)Pooled (M)Share
Data21.31121.1071.2126.12%
Analysis10.6477.7144.1716.20%
Writing15.5968.2141.9015.37%
Coordinator19.5848.9934.2912.58%
Advisor11.4743.1227.3010.01%
Research (including planning)8.3425.6316.986.23%
Figure13.7119.9116.816.17%
Environment8.988.938.953.28%
Paper3.186.945.061.86%
Maintenance0.005.162.580.95%
Build0.005.022.510.92%
Task specialist0.001.790.890.33%

Data construction, analysis, and Advisor review account for 52.33% of recorded tokens. These functions are necessary because cross-source estimates are interpretable only when the sample, observation units, and measurements are constructed correctly; the analysis implements the intended tests; and the conclusions are checked against the resulting evidence. Coordinator and revision work carry required corrections through to dependent results, so that the final report reflects the corrected evidence. The additional usage therefore supports the evidence construction and verification required to complete the research.

The results also show why allocating inference to these functions is useful. SocialLab exceeds OpenCode by 15.96 H and 11.11 U points, and the tool-augmented LLM by 11.52 H and 9.72 U points. Across all six DeepSeek tasks, removing the Data Agent lowers mean H and U by 9.00 and 11.11 points; removing the Advisor lowers them by 3.37 and 2.78 points. These losses show that evidence construction and review contribute to the quality of the completed research. Thus, the additional expenditure has a concrete methodological purpose and is supported by the observed quality gains.

CSD-Bench's scale and "open-ended" framing are mismatched.

CSD-Bench has only six tasks with given sources, time ranges, and target relationships, making it closer to semi-structured cross-source tasks than open-ended research

CSD-Bench tasks are open-ended because the system must develop and execute the empirical design needed to answer the research question. Specifying sources, time ranges, and target relationships defines a common research problem; it does not specify how the concepts should be measured, which observations are comparable, how sources should be linked, what estimand and model should be used, or what evidence would justify the conclusion. The tasks supply neither a complete analysis protocol nor a predetermined empirical answer. The system must resolve these decisions using the literature and actual data, and revise its choices when feasibility or sensitivity checks require it.

These decisions determine the substance of the research: alternative measurements, samples, linkage rules, and specifications can yield different estimates and interpretations. CSD-Bench therefore evaluates whether each workflow constructs defensible evidence that answers the question, while allowing methodologically justified alternative designs. The specified sources and scope make the workflows comparable on the same problem; the six tasks determine the size of this evaluation. The research remains open-ended because its empirical design and evidence-based answer must be developed by the system.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.