acceptodds
Under review as a conference paper at ICLR 2027

MetaRigor: From Agentic Automation to Verifiable End-to-End Meta-Analysis Through Scientific Decision Boundaries

Abstract

Automated meta-analysis requires not only accurate numerical extraction but also the consistent preservation of result identity across stages, including the associated study, comparison, population, outcome, time point, and analysis conditions. We present MetaRigor, an LLM workflow framework for verifiable end-to-end evidence synthesis. MetaRigor organizes study results as Study Target units and operationalizes scientific decision boundaries at the task level through Task Contracts, programmatic rules, and source binding. The model performs semantic interpretation that depends on study context, whereas programs execute formalizable rules, computations, validation, and source verification. We conducted controlled evaluations across five stages: evidence acquisition, data extraction, risk of bias assessment, certainty of evidence assessment, and manuscript generation. MetaRigor achieved an score of 86.29% for complete result delivery and 80.00% agreement for domain-level certainty of evidence judgments. Generated manuscripts received scores of 94.63 and 88.25 under DeepSeek and GPT-5.6 Luna evaluation, respectively, both exceeding the generic CLI baseline. Cross-system tracing of archived outputs from three systems further showed that local mismatches or omissions in result identity could alter statistical inputs and the objects subjected to risk of bias assessment, with these errors subsequently propagating into the final report. Ablation experiments showed that task constraints and programmatic rules played distinct roles across different stages. These findings indicate that, in LLM-driven evidence synthesis workflows, consistently maintaining explicit result identity, decision boundaries, and provenance relationships is critical to improving end-to-end verifiability. Source code is available at https://anonymous.4open.science/status/Meta-Rigor-8C62.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI scientist, large language models, scientific agents, agentic workflow, systematic review, meta-analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5R4fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work aims to solve automated meta-analysis and propose metarigor, a llm workflow for verifiable evidence synthesis. It organises the results as study x target structure and use several strategies (task contracts, programmatic rules, etc) to standardise scientific decision boundaries. It frameworks separate models for semantic understanding and programs for executing validation.

the experiment results also imply that llm-centric evidence workflows, maintaining intermediate provenance relationship is important. while an error might further propagate and pollute the full report.

Critical strengths

the motivation is sound (especially fig1 show how human-ai collaboration way evovle). the study x target and focus on the result identity rather extraction accuracy. also, the paper propose trace errors end-to-end and see how it propagate and affect the final scores, instead of reporting individual stage scores, which makes the verifiability claim more trustable.

Critical weaknesses

My main concern is that despite this work compare existing paper in related works, these evidence-grounded auditable work. but these works are never be compared conceptually in term of methods or experiments.

the proposed framework sound like a prompt-level engineering strategy. notably, in experiments, it only compare general agents such as codex cli and openhands agents without the domain sota baseline (such as related works), which makes the setting unfair and gains unconvincing. such as reading the numerical results in abstract (86.29%) it still difficulty to judge the actual performance.

Questions for authors
  1. bad readability, lots of new defined concepts such as Study Packet, Scientific Decision Boundary, FAST, DEEP, also EA, DE, RoB, EC, MS.. it is hard to follow easily for humans.. such as Study Packet, it firstly appear at line 98, but do not have any explanation. also what's the formal definition of scientific decision boundary?

  2. what the both stage and cross-stage propagation means in line 106?

  3. what the SELECT2 means in line 463? (also first appear)

  4. the reference looks like ai-style, such as line 129-132. several works ASReview, PICO, TrialMind, LEADS, while reference only appear at the end of the sentences..

10 comments
AuthorsOpenReview ↗

Initial Response and Plans for Additional Experiments

We thank the reviewer for the constructive feedback. We clarify MetaRigor's methodological design and address the concerns regarding presentation and evaluation below.

R1. Related Work and Baseline Selection. We appreciate the concern and will expand Related Work to clarify the contributions of existing systems and their differences from MetaRigor. We initially selected Codex CLI and OpenHands CLI because few existing automated systems cover the full meta-analysis workflow, and we aimed to evaluate the autonomous planning and tool-use capabilities of general-purpose agents. We agree that these comparisons alone are insufficient. Therefore, we are adding Manalyzer[1] and meta-pipe[2] as supplementary baselines to evaluate MetaRigor against specialized meta-analysis systems.

R2. Paper Readability. We apologize for the unclear definitions and terminology. We will revise how key concepts are introduced and defined throughout the manuscript to improve readability.

R3. Scientific Decision Boundary. We appreciate the request for clarification. MetaRigor does not merely constrain model behavior through prompts; it implements decision boundaries through explicit task constraints, programmatic execution, and validation. We will clarify the formal definition of the Scientific Decision Boundary, which comprises three elements: the Object of Judgment, Admissible Facts, and Decision Authority assigned to models and programs. MetaRigor operationalizes these elements through Task Contracts and corresponding program logic.

R4. Stage-Level Evaluation and Cross-Stage Propagation. We agree that Line 106 requires clarification. The two terms refer to distinct evaluation settings: Stage-level evaluation measures each stage independently with fixed upstream inputs, whereas cross-stage propagation analysis traces how result-identity mismatches or omissions affect downstream statistical inputs, risk-of-bias assessments, and final reports. We will clarify this distinction in the manuscript.

R5. Clarification of SELECT2. SELECT2 is a clinical trial of endovascular thrombectomy for large ischemic strokes reported by Sarraj et al[3]. It is included in our end-to-end analysis to illustrate a result-identity mismatch. We will clarify this at its first mention.

R6. Citation Practices. We will revise the Related Work section to associate each study's description directly with its citation, rather than grouping multiple studies under a collective citation. We will also clarify their individual contributions and relationships to MetaRigor. We apologize for the lack of clarity.

We expect to share additional results by November 16, followed by a detailed response. We appreciate the reviewer's time and continued engagement.

References

[1] Xu, W., Zhang, W., Ling, F., et al. (2025). Manalyzer: End-to-end Automated Meta-analysis with Multi-agent System. arXiv:2505.20310.

[2] Lin, H.-T., & Yeh, J.-T. (2026). meta-pipe: An LLM-agent pipeline for end-to-end automated systematic review and meta-analysis. arXiv preprint arXiv:2606.28363.

[3] Sarraj, A., Hassan, A. E., Abraham, M. G., et al. (2023). Trial of endovascular thrombectomy for large ischemic strokes. New England Journal of Medicine, 388(14), 1259–1271. https://doi.org/10.1056/NEJMoa2214403.

AuthorsOpenReview ↗

Reply to Reviewer 5R4f

We have added specialized baselines and are revising the manuscript to improve its clarity and presentation.

W1. Additional Specialized Baselines. We added Manalyzer and meta-pipe to extend the comparison beyond general-purpose agents. Manalyzer was evaluated on EA and DE, while meta-pipe was evaluated across all five stages. The results for EA are shown below.

MethodTitle/Abstract F1 ↑Full-text F1 ↑Family Recall ↑Companion Recall ↑
Manalyzer59.74%60.53%36.00%27.59%
meta-pipe31.65%34.25%48.00%82.76%
MetaRigor67.61%76.19%88.00%93.10%

The corresponding DE and RoB results are shown below.

MethodDE F1 ↑DE Recall ↑DE Precision ↑RoB Domain ↑RoB Judgment ↑
Manalyzer48.65%32.31%98.44%——
meta-pipe53.02%40.51%76.70%64.38%50.00%
MetaRigor86.29%87.18%85.43%71.25%62.50%

MetaRigor achieves the best results across the reported EA metrics and outperforms both specialized baselines in DE F1 and the two reported RoB metrics. Although Manalyzer achieves higher DE Precision, its Recall is substantially lower. These additional comparisons provide evidence beyond the original general-purpose agent baselines.

W2. Manuscript Clarity and Presentation. We are revising the manuscript to address the readability concerns. Specifically, we are introducing key concepts and stage abbreviations more clearly at their first occurrence, clarifying the distinction between independent stage-level evaluation and error propagation across stages, and explaining SELECT2 when it first appears. We are also revising Related Work to place citations directly alongside the corresponding descriptions of individual studies and to clarify their respective contributions and differences from MetaRigor.

We are incorporating these changes into the revised manuscript, and the updated PDF will be uploaded shortly.

We appreciate the reviewer’s suggestions, which have helped us improve both the experimental comparisons and the clarity of the manuscript.

thanks for the responses. these replies address some of my concerns, but my main concerns remain: the significance of, and the difference from, existing work on evidence-based methods. The author has also acknowledged and incorporated them. I therefore maintain my original vote 2.

Reviewer_CgNeOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce MetaRigor, an LLM-based framework for automated meta-analysis. It uses a Study × Target representation to keep track of study results across different stages. The framework also introduces Task Contracts to define task requirements and separates LLM judgments from programmatic rules. The outputs are linked to their original evidence sources for verification. The paper evaluates MetaRigor across 5 stages, including evidence acquisition, data extraction, risk of bias assessment, evidence certainty assessment, and manuscript generation. The authors compare it with Codex CLI and OpenHands CLI and conduct several ablation studies. MetaRigor achieves 86.29% F1 in data extraction, 80.00% domain agreement in evidence certainty assessment, and a composite score of 89.63 for manuscript generation.

Critical strengths
  1. I think the paper addresses an important problem in automated meta-analysis. MetaRigor uses Study × Target to keep track of result identity across different stages. This matters because a model may extract the correct numerical value but associate it with the wrong outcome or time point. The SELECT2 example makes this problem clear, where OpenHands uses in-hospital mortality for a 90-day mortality target.

  2. The paper also evaluates MetaRigor across 5 different stages and compares it with Codex CLI and OpenHands CLI. I found the ablation studies useful, especially the results showing that removing Code Logic reduces high-risk recall from 69.23% to 15.38%. The authors also analyze how errors in earlier stages can affect statistical inputs and final reports.

Critical weaknesses

Limited evaluation scale and end-to-end validation: The evaluation is relatively small, with only 5–20 cases per stage (Table 5). The paper does not report repeated runs, confidence intervals, or significance tests. Some large performance improvements are based on very few samples. For example, RoB high-risk recall of 69.23% vs. 7.69% corresponds to 9/13 vs. 1/13, and EC downgrade agreement of 50% vs. 15% corresponds to 10/20 vs. 3/20. More importantly, the end-to-end analysis includes only 2 complete MetaRigor runs and one archived case (Lin-2025). The paper does not systematically evaluate final-result accuracy across complete runs. This makes it difficult to assess the reliability of the full workflow.

Baseline comparison and evaluation limitations: The comparisons do not fully isolate the contributions of MetaRigor. The baselines are general-purpose CLI agents rather than systems designed for automated evidence synthesis. In the EA stage, MetaRigor uses different models from the baselines, which may partly explain the large performance gap. The Dual DeepSeek ablation also shows a large drop in title/abstract F1, from 67.61% to 22.94%. In addition, the manuscript evaluation relies heavily on LLM judges. Under human evaluation, MetaRigor scores 86.00 compared with 77.50 for OpenHands, a smaller difference than the overall composite scores suggest. These issues make it difficult to determine how much of the improvement comes from the framework itself.

Questions for authors
  1. Could you report the run-to-run variability or case-level confidence intervals for the main metrics in Tables 1 and 2? Also, how many distinct systematic reviews are represented by the cases in Table 5?

  2. Were all systems required to provide the same source anchors? What are the pre-filter claim-closure rates for Codex CLI and OpenHands CLI. Also, how many items were excluded by the filter in Appendix D.2? Could you also report recall, precision, and field accuracy without this filter for all three systems?

8 comments
AuthorsOpenReview ↗

Initial Response and Plans for Additional Experiments

We thank the reviewer for the constructive feedback. We address the evaluation concerns below and outline our planned experiments.

R1. Case-level Confidence Intervals and Evaluation Scale. We will report case-level bootstrap confidence intervals for the main metrics in Tables 1 and 2, supplement them with cluster-aware item-level bootstrap and leave-one-case-out sensitivity analyses, expand the EC evaluation to 100 cases, and extend Table 5 to distinguish stage-specific case counts from the number of distinct systematic reviews after deduplication. We will further expand the end-to-end case analyses and introduce quantitative evaluation metrics.

R2. Baseline Selection. We initially selected Codex CLI and OpenHands CLI because few existing automated systems cover the full meta-analysis workflow, and we aimed to evaluate the autonomous planning and tool-use capabilities of general-purpose agents. We agree that these comparisons alone are insufficient to establish MetaRigor's advantages. We will add Manalyzer[1] and meta-pipe[2] as specialized meta-analysis baselines and expand Related Work to clarify their differences from MetaRigor.

R3. Impact of the Underlying Model. We acknowledge that the EA experiments do not fully isolate the effects of underlying model capabilities. To address this, we will add a Dual-Luna (GPT-5.6 Luna) configuration and run Codex CLI and OpenHands CLI with GPT-5.6 Luna, allowing us to better distinguish model capability from MetaRigor's EA mechanisms.

R4. LLM-as-a-Judge Evaluation. We appreciate this concern and will place greater emphasis on human evaluation. We recognize that the performance gaps under human evaluation are smaller than those reflected by the composite scores. We will therefore interpret the results more cautiously and avoid overstating the overall performance advantage.

R5. Source Anchors and Claim Closure Filtering. We did not require Codex CLI or OpenHands CLI to provide structured Source Anchors in MetaRigor's format. Claim Closure is therefore not applicable to these baselines, and neither was subject to this filtering. For DE, MetaRigor requires every accepted numerical field to be linked to evidence in the original text through a Source Anchor. This is a core design requirement of the DE stage, rather than an additional criterion imposed for evaluation. In the evaluation, this requirement is operationalized through item-level closure filtering, where an Item is retained only if all delivered numerical fields have valid source attachments. To improve transparency, we will report the number of filtered items and pre-filter Recall, Precision, and Field Accuracy. The pre-filter field-level Claim Closure of 95.70% is already reported in Appendix D.6.1.

We expect to share additional results by November 16, followed by a detailed response. We appreciate the reviewer's time and continued engagement.

References

[1] Xu, W., Zhang, W., Ling, F., et al. (2025). Manalyzer: End-to-end Automated Meta-analysis with Multi-agent System. arXiv:2505.20310.

[2] Lin, H.-T., & Yeh, J.-T. (2026). meta-pipe: An LLM-agent pipeline for end-to-end automated systematic review and meta-analysis. arXiv preprint arXiv:2606.28363.

AuthorsOpenReview ↗

Reply to Reviewer CgNe (Part I)

We have completed additional baseline comparisons, an expanded EC evaluation, and model-controlled experiments in EA.

W1. Additional Baseline Comparisons. We added Manalyzer and meta-pipe as specialized evidence synthesis baselines. Manalyzer was evaluated on EA and DE, while meta-pipe was evaluated across all five stages. The EA results are shown below.

MethodT/A Precision ↑T/A F1 ↑T/A F2 ↑T/A NNR ↓Full Precision ↑Full F1 ↑Full F2 ↑Full NNR ↓Family Recall ↑Companion Recall ↑
Manalyzer44.23%59.74%75.66%2.2645.10%60.53%76.16%2.2236.00%27.59%
meta-pipe18.80%31.65%53.65%5.3220.66%34.25%56.56%4.8448.00%82.76%
MetaRigor52.17%67.61%82.19%1.9263.16%76.19%86.96%1.5888.00%93.10%

MetaRigor achieves the best results across all ten EA metrics, with F1 scores of 67.61% and 76.19% for title/abstract and full-text screening, respectively. Family Recall and Companion Recall reach 88.00% and 93.10%, respectively.

The corresponding DE and RoB results are shown below.

MethodDE F1 ↑DE Recall ↑DE Field Acc. ↑DE Precision ↑RoB Domain ↑RoB Judgment ↑RoB High-risk Recall ↑
Manalyzer48.65%32.31%35.75%98.44%———
meta-pipe53.02%40.51%44.93%76.70%64.38%50.00%7.69%
MetaRigor86.29%87.18%88.89%85.43%71.25%62.50%69.23%

In DE, MetaRigor achieves the best F1, Recall, and Field Accuracy. Manalyzer obtains higher Precision (98.44%) but substantially lower Recall (32.31%). In RoB, MetaRigor achieves the best results across all three metrics.

W2. Updated EC Evaluation. We expanded the EC evaluation to 100 cases. The updated point estimates are shown below.

MethodEC Domain Agreement ↑EC Downgrade Agreement ↑EC MAE ↓
Codex CLI62.20%45.00%0.5490
OpenHands CLI60.6%35.00%0.727
meta-pipe57.80%31.00%0.990
MetaRigor80.20%55.00%0.413

MetaRigor exceeds the strongest baselines by 18.00 and 10.00 percentage points in domain and downgrade agreement, respectively, while achieving the lowest MAE (0.413).

W3. Model-Controlled Experiments in EA. To distinguish underlying model capability from workflow contributions, we introduced Dual Luna, using GPT-5.6 Luna for both screening paths. We also evaluated Codex CLI and OpenHands CLI with GPT-5.6 Luna as their underlying model.

MethodT/A Precision ↑T/A F1 ↑T/A F2 ↑T/A NNR ↓Full Precision ↑Full F1 ↑Full F2 ↑Full NNR ↓Family Recall ↑Companion Recall ↑
Dual Luna46.15%62.34%78.95%2.1759.46%70.97%80.29%1.6884.00%79.31%
Dual DeepSeek12.95%22.94%42.66%7.7244.00%58.67%73.33%2.2784.00%86.21%
Codex CLI-Luna14.62%25.51%46.13%6.8453.49%67.65%80.42%1.8728.00%44.83%
OpenHands CLI-Luna9.64%17.52%34.38%10.3857.50%70.77%82.14%1.7432.00%17.24%
MetaRigor52.17%67.61%82.19%1.9263.16%76.19%86.96%1.5888.00%93.10%

Using GPT-5.6 Luna for both screening paths improves Dual Luna's title/abstract F1 from 22.94% to 62.34% and full-text F1 from 58.67% to 70.97%. However, Codex CLI-Luna and OpenHands CLI-Luna achieve full-text F1 scores of only 67.65% and 70.77%, respectively, both below MetaRigor's 76.19%. MetaRigor also achieves the best results across all ten metrics in this comparison.

AuthorsOpenReview ↗

Reply to Reviewer CgNe (Part II)

W4. Source Anchors and Claim Closure Filtering. As explained in our initial response, Codex CLI and OpenHands CLI were not required to provide structured Source Anchors in MetaRigor's format. Claim Closure is therefore not applicable to these baselines, and neither was subject to this filtering. For MetaRigor, the filtering removed 10 of 209 complete extraction items, of which 4 exactly matched the reference answers. Before filtering, MetaRigor achieved an exact-match Recall of 89.23%, Precision of 83.25%, Field Accuracy of 91.79%, and F1 of 86.14%. These results illustrate the trade-off between source verifiability and extraction coverage.

W5. Confidence Intervals and Sensitivity Analyses. We have completed case-level bootstrap analyses to quantify uncertainty in the main evaluation metrics. We further conducted cluster-aware item-level bootstrap analyses to account for the nested structure of the evaluation data, together with leave-one-case-out sensitivity analyses to examine the robustness of the results to individual cases. We also clarified the distinction between stage-specific case counts and the number of distinct systematic reviews after deduplication in Table 5. These analyses involve extensive results across multiple stages and metrics. To keep this response concise, we will include the detailed 95% confidence intervals, cluster-aware bootstrap results, and leave-one-case-out sensitivity analyses in the revised manuscript. We are preparing the updated PDF and will upload it shortly.

W6. Seven-Case End-to-End Evaluation. We added a seven-case end-to-end evaluation comparing MetaRigor, meta-pipe, Codex CLI, and OpenHands CLI across four stages: data extraction (DE), risk of bias assessment (RoB), evidence certainty assessment (EC), and manuscript generation (MS). All systems were evaluated against the same set of 2,133 adjudicable reference items, comprising 869 DE, 906 RoB, 114 EC, and 244 MS items. We first aggregated the counts across cases and stages, then calculated Precision (correct items / expressed items), Recall (correct items / reference items), and F1. Incorrectly expressed items are included in the Precision denominator, while omitted items remain in the Recall denominator.

SystemCorrect ItemsExpressed ItemsReference ItemsF1 ↑Precision ↑Recall ↑
MetaRigor1,5391,8642,13377.01%82.56%72.15%
meta-pipe8141,2352,13348.34%65.91%38.16%
Codex CLI1,3101,7642,13367.23%74.26%61.42%
OpenHands CLI9701,6102,13351.83%60.25%45.48%

MetaRigor achieves the highest F1 (77.01%), Precision (82.56%), and Recall (72.15%) among all four systems. Compared with the strongest baseline, Codex CLI, it improves F1, Precision, and Recall by 9.78, 8.30, and 10.73 percentage points, respectively. These results provide a quantitative end-to-end comparison of factual correctness and coverage across all four stages.

Reviewer_aFdPOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduce an LLM workflow framework for meta-analysis, which performs semantic interpretation that depends on study context, and use programs to execute formalizable rules, computations, validation, and source verification. The paper evaluate the proposed method for each stages, and the result is outperforming Codex and Openhand.

Critical strengths
  • The research question is well-motivated. Automating the meta-analysis process is a important challenge to tackle.
  • The design of Scientific Decision Boundary is interesting and intuitive.
Critical weaknesses
  • The main experiment compare MetaRigor with general-purpose baselines, which is an unfair comparison to me. MetaRigor is optimized for the task. Also from the appendix, the baseline prompt is clearly simper comparing to MetaRigor. The baseline might be too weak.
  • The evaluated model is generally too weak and limited ( Only DeepSeek V4 Flash and GPT-5.6 Luna). I understand the author might has some budget limitation, but there any many free stronger model in OpenRouter. We are not clear whether the low performance of baseline is due to the weak base model or the framework design.
  • I didn't find any end-to-end evaluation on the meta-analysis. The abstract and title claim this is end-to-end meta-analysis, but we don't know the performance of end-to-end task.
  • (minor weakness) The overall paper writing is a bit hard for me to understand.
Questions for authors
  • The code link in the abstract seems points to an unavailable codebase. It's not able to evaluate the code.
  • In the AI use statement, you mention AI tools only used for manuscript refinement, but you use codex and openhand as baselines. How AI tools is used in this submission?
7 comments
AuthorsOpenReview ↗

Initial Response and Plans for Additional Experiments

We thank the reviewer for the constructive feedback. We address the concerns below and outline our ongoing experiments.

R1. Baseline Selection and Evaluation Fairness. We initially selected Codex CLI and OpenHands CLI because few existing automated systems cover the full meta-analysis workflow, and we aimed to evaluate the autonomous planning and tool-use capabilities of general-purpose agents. We agree that comparisons with general-purpose agents alone are insufficient to establish MetaRigor's advantages. Therefore, we are adding Manalyzer[1] and meta-pipe[2] as supplementary baselines, with experiments currently underway. Appendix C.2 presents only a simplified baseline prompt; the actual experiments used complete stage-specific prompts. All systems received the same research questions, input materials, and task requirements. We will also expand Related Work to clarify how MetaRigor differs from existing systems.

R2. Underlying Model Capability and Computational Budget. Our experiments were primarily constrained by budget. The complete meta-analysis workflow requires processing extensive full-text literature and supplementary materials, making model inference and long-context processing costly. To address this limitation, we are conducting additional comparisons using stronger models at key stages.

R3. End-to-End Evaluation. The original manuscript includes end-to-end analyses of two cases, Lin and Turalde, but lacks a unified quantitative measure of end-to-end correctness. We will therefore expand the case analyses and introduce quantitative end-to-end metrics.

R4. Code Accessibility and AI Use Disclosure. We appreciate the reviewer for identifying issues with code accessibility and execution. We will correct the installation and execution entry points. Regarding AI use, we used generative AI tools solely for manuscript refinement, including translation, grammar correction, and improving clarity. Codex CLI and OpenHands CLI were used exclusively as experimental baselines.

We expect to share additional results by November 16, followed by a detailed response. We appreciate the reviewer's time and continued engagement.

References

[1] Xu, W., Zhang, W., Ling, F., et al. (2025). Manalyzer: End-to-end Automated Meta-analysis with Multi-agent System. arXiv:2505.20310.

[2] Lin, H.-T., & Yeh, J.-T. (2026). meta-pipe: An LLM-agent pipeline for end-to-end automated systematic review and meta-analysis. arXiv preprint arXiv:2606.28363.

AuthorsOpenReview ↗

Reply to Reviewer aFdP

We have completed additional comparisons with specialized baselines, stronger-model experiments using GPT-6.1 Terra, and a quantitative end-to-end evaluation.

W1. Additional Specialized Baselines. We added Manalyzer and meta-pipe as supplementary baselines. Manalyzer was evaluated on EA and DE, while meta-pipe was evaluated across all five stages (Secs. 2 and 4.1–4.2). The EA results are shown below.

MethodT/A Precision ↑T/A F1 ↑T/A F2 ↑T/A NNR ↓Full Precision ↑Full F1 ↑Full F2 ↑Full NNR ↓Family Recall ↑Companion Recall ↑
Manalyzer44.23%59.74%75.66%2.2645.10%60.53%76.16%2.2236.00%27.59%
meta-pipe18.80%31.65%53.65%5.3220.66%34.25%56.56%4.8448.00%82.76%
MetaRigor52.17%67.61%82.19%1.9263.16%76.19%86.96%1.5888.00%93.10%

MetaRigor achieves the best results across all ten EA metrics, with title/abstract and full-text F1 scores of 67.61% and 76.19%, respectively. Family Recall and Companion Recall reach 88.00% and 93.10%, respectively.

The corresponding DE and RoB results are shown below.

MethodDE F1 ↑DE Recall ↑DE Field Acc. ↑DE Precision ↑RoB Domain ↑RoB Judgment ↑RoB High-risk Recall ↑
Manalyzer48.65%32.31%35.75%98.44%———
meta-pipe53.02%40.51%44.93%76.70%64.38%50.00%7.69%
MetaRigor86.29%87.18%88.89%85.43%71.25%62.50%69.23%

In DE, MetaRigor achieves the best F1, Recall, and Field Accuracy. Although Manalyzer obtains higher Precision (98.44%), its Recall is only 32.31%. In RoB, MetaRigor achieves the best results across all three metrics.

W2. Stronger-Model Comparison with GPT-6.1 Terra. To address concerns about the strength of the underlying models, we conducted an additional comparison using GPT-6.1 Terra (Appendix I). The results are shown below.

MethodDE Recall ↑DE F1 ↑DE Field Acc. ↑DE Precision ↑RoB Domain ↑RoB Judgment ↑RoB High-risk Recall ↑EC Domain ↑EC Downgrade ↑EC MAE ↓
Codex CLI61.54%74.77%63.65%95.24%72.50%68.75%53.85%68.40%50.00%0.540
OpenHands CLI58.97%72.33%59.78%93.50%68.75%56.25%30.77%68.80%51.00%0.551
meta-pipe29.23%45.06%31.52%98.28%65.00%62.50%38.46%57.80%31.00%0.990
MetaRigor68.72%79.06%71.74%93.06%82.50%84.38%69.23%81.60%70.00%0.306

Under the GPT-6.1 Terra configuration, MetaRigor achieves the best results on nine of the ten metrics. Codex CLI obtains higher DE Precision (95.24% vs. 93.06%), while MetaRigor leads in the remaining metrics, including all three EC metrics.

W3. Seven-Case End-to-End Evaluation. We added a seven-case end-to-end evaluation comparing MetaRigor, meta-pipe, Codex CLI, and OpenHands CLI across four stages: data extraction (DE), risk of bias assessment (RoB), evidence certainty assessment (EC), and manuscript generation (MS) (Secs. 4.1 and 4.4; Table 4; Appendix G). All systems were evaluated against the same 2,133 adjudicable reference items, comprising 869 DE, 906 RoB, 114 EC, and 244 MS items. We aggregated counts across cases and stages before calculating Precision (correct items / expressed items), Recall (correct items / reference items), and F1. Incorrectly expressed items are included in the Precision denominator, while omitted items remain in the Recall denominator.

SystemCorrect ItemsExpressed ItemsReference ItemsF1 ↑Precision ↑Recall ↑
Full MetaRigor1,5391,8642,13377.01%82.56%72.15%
meta-pipe8141,2352,13348.34%65.91%38.16%
Codex CLI1,3101,7642,13367.23%74.26%61.42%
OpenHands CLI9701,6102,13351.83%60.25%45.48%

Full MetaRigor achieves the highest F1 (77.01%), Precision (82.56%), and Recall (72.15%). Compared with the strongest baseline, Codex CLI, it improves F1, Precision, and Recall by 9.78, 8.30, and 10.73 percentage points, respectively. These results provide a quantitative evaluation of factual correctness and coverage across the four stages of the end-to-end workflow.

We appreciate the reviewer's constructive suggestions, which have helped us strengthen the baseline comparisons, evaluate the impact of stronger models, and provide quantitative evidence for end-to-end performance. We are incorporating these additional experiments and analyses into the revised manuscript, and the updated PDF will be uploaded shortly.

Thank you again for your constructive feedback. We have now uploaded the latest revised PDF to the submission. It incorporates all the revisions and additional experiments described in our previous responses. All changes in the revised manuscript are highlighted in yellow.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Reply and Summary of Revisions

We thank all reviewers for their careful evaluation and constructive suggestions. We thank Reviewer 1 (aFdP) for comments on baseline fairness and model capability, Reviewer 2 (CgNe) for concerns about execution stability, evaluation scale, and statistical reliability, and Reviewer 3 (5R4f) for feedback on methodological contributions, related work, and readability. Their comments helped strengthen our experiments and improve the manuscript. All changes in the revised manuscript are highlighted in yellow.

  1. Additional Baselines (aFdP, CgNe, 5R4f). We added Manalyzer and meta-pipe as domain-specific baselines. Manalyzer covers EA and DE, while meta-pipe covers all five stages (Secs. 2 and 4.1–4.2; Tables 1–2; Appendix E).

  2. Clarified Contributions (aFdP, 5R4f). We clarified MetaRigor's contributions and differences from existing systems, emphasizing Study × Target, Task Contracts, programmatic rules, and source binding (Secs. 1–3; Table 1).

  3. End-to-End Evaluation (aFdP, CgNe). We added a seven-case evaluation across DE, RoB, EC, and MS, using actual upstream outputs and 2,133 reference facts to assess factual correctness and coverage (Secs. 4.1 and 4.4; Table 4; Appendix G).

  4. Stronger-Model Comparison (aFdP). We added comparisons under the GPT-6.1 Terra configuration for DE, RoB, and EC (Appendix I).

  5. Expanded EC Evaluation (CgNe). We expanded EC from 20 to 100 cases, covering 500 domain judgments and 100 overall downgrade judgments (Table 2; Appendices A.3 and D.4).

  6. Dual-Luna Experiments (CgNe). We added Dual-Luna, using GPT-5.6 Luna for both EA screening paths, and same-model comparisons with Codex CLI and OpenHands CLI (Secs. 4.2–4.3; Tables 2–3; Appendix E.2).

  7. Statistical Analyses (CgNe). We added case-level 95% bootstrap confidence intervals, cluster-aware item-level bootstrap, and leave-one-case-out sensitivity analyses (Appendices H).

  8. Code Fixes (aFdP). We fixed code accessibility, installation, and execution entry points (Code repository).

  9. Improved Readability (aFdP, 5R4f). We revised the Introduction and Method, clarified key concepts and abbreviations, explained SELECT2, distinguished stage-level evaluation from cross-stage error propagation, and improved citation placement (Secs. 1–3, 4.1, and 4.4; Appendices G.1 and G.3).

  10. Human Expert Evaluation (CgNe). We clarified the human reviewers' backgrounds, independent scoring procedures, and disagreement resolution, and reported human and model evaluation scores separately (Sec. 4.2; Figure 4; Appendices D.5.1–D.5.2).

The point-by-point responses below provide further experimental results and specific details of these revisions.

AuthorsOpenReview ↗

Author Summary

We sincerely thank the Area Chair and all reviewers for their careful evaluation and constructive feedback. Their comments have helped us further improve the research design, experimental validation, and presentation of our work.

In this work, we propose MetaRigor, a verifiable end-to-end evidence synthesis framework built around explicit Scientific Decision Boundaries. Its core contributions include maintaining Study × Target result identities, using Task Contracts, separating model interpretation from programmatic rule execution, and binding outputs to source evidence to ensure consistency and traceability across different stages. Importantly, although existing research has made progress on individual tasks such as literature screening, data extraction, and risk of bias assessment, the development of a unified framework that covers the complete evidence synthesis workflow and consistently preserves evidence relationships across stages remains a clear and largely unaddressed research gap. No systematic solution has yet been established for this direction. MetaRigor is proposed to address precisely this gap.

Experimentally, we conducted evaluations across five stages, covering 102 independent systematic reviews, and introduced additional domain-specific baselines, including Manalyzer and meta-pipe. Furthermore, our newly added seven-case downstream end-to-end evaluation achieved an aggregate F₁ score of 77.01%, exceeding the strongest baseline by 9.78 percentage points. We also conducted ablation studies to investigate the distinct contributions of task constraints and programmatic verification across different stages.

In response to the reviewers' major concerns, we further added paired confidence intervals and cluster-aware bootstrap analyses, expanded the EC evaluation, provided additional analyses of EA model configurations and baseline fairness, and substantially reorganized the related work and clarified our methodological contributions. Where the results remained subject to uncertainty, we also appropriately moderated the corresponding claims in the revised manuscript. All revisions are highlighted in yellow in the latest manuscript.

We believe these additional analyses and revisions have further strengthened the reliability of our experimental findings, the clarity of our methodological contributions, and the reproducibility of our work. We sincerely appreciate the time and effort invested by the Area Chair and all reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.