- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work aims to solve automated meta-analysis and propose metarigor, a llm workflow for verifiable evidence synthesis. It organises the results as study x target structure and use several strategies (task contracts, programmatic rules, etc) to standardise scientific decision boundaries. It frameworks separate models for semantic understanding and programs for executing validation.
the experiment results also imply that llm-centric evidence workflows, maintaining intermediate provenance relationship is important. while an error might further propagate and pollute the full report.
Critical strengths
the motivation is sound (especially fig1 show how human-ai collaboration way evovle). the study x target and focus on the result identity rather extraction accuracy. also, the paper propose trace errors end-to-end and see how it propagate and affect the final scores, instead of reporting individual stage scores, which makes the verifiability claim more trustable.
Critical weaknesses
My main concern is that despite this work compare existing paper in related works, these evidence-grounded auditable work. but these works are never be compared conceptually in term of methods or experiments.
the proposed framework sound like a prompt-level engineering strategy. notably, in experiments, it only compare general agents such as codex cli and openhands agents without the domain sota baseline (such as related works), which makes the setting unfair and gains unconvincing. such as reading the numerical results in abstract (86.29%) it still difficulty to judge the actual performance.
Questions for authors
-
bad readability, lots of new defined concepts such as Study Packet, Scientific Decision Boundary, FAST, DEEP, also EA, DE, RoB, EC, MS.. it is hard to follow easily for humans.. such as Study Packet, it firstly appear at line 98, but do not have any explanation. also what's the formal definition of scientific decision boundary?
-
what the both stage and cross-stage propagation means in line 106?
-
what the SELECT2 means in line 463? (also first appear)
-
the reference looks like ai-style, such as line 129-132. several works ASReview, PICO, TrialMind, LEADS, while reference only appear at the end of the sentences..
10 comments
Reply to Reviewer 5R4f
We have added specialized baselines and are revising the manuscript to improve its clarity and presentation.
W1. Additional Specialized Baselines. We added Manalyzer and meta-pipe to extend the comparison beyond general-purpose agents. Manalyzer was evaluated on EA and DE, while meta-pipe was evaluated across all five stages. The results for EA are shown below.
| Method | Title/Abstract F1 ↑ | Full-text F1 ↑ | Family Recall ↑ | Companion Recall ↑ |
|---|---|---|---|---|
| Manalyzer | 59.74% | 60.53% | 36.00% | 27.59% |
| meta-pipe | 31.65% | 34.25% | 48.00% | 82.76% |
| MetaRigor | 67.61% | 76.19% | 88.00% | 93.10% |
The corresponding DE and RoB results are shown below.
| Method | DE F1 ↑ | DE Recall ↑ | DE Precision ↑ | RoB Domain ↑ | RoB Judgment ↑ |
|---|---|---|---|---|---|
| Manalyzer | 48.65% | 32.31% | 98.44% | — | — |
| meta-pipe | 53.02% | 40.51% | 76.70% | 64.38% | 50.00% |
| MetaRigor | 86.29% | 87.18% | 85.43% | 71.25% | 62.50% |
MetaRigor achieves the best results across the reported EA metrics and outperforms both specialized baselines in DE F1 and the two reported RoB metrics. Although Manalyzer achieves higher DE Precision, its Recall is substantially lower. These additional comparisons provide evidence beyond the original general-purpose agent baselines.
W2. Manuscript Clarity and Presentation. We are revising the manuscript to address the readability concerns. Specifically, we are introducing key concepts and stage abbreviations more clearly at their first occurrence, clarifying the distinction between independent stage-level evaluation and error propagation across stages, and explaining SELECT2 when it first appears. We are also revising Related Work to place citations directly alongside the corresponding descriptions of individual studies and to clarify their respective contributions and differences from MetaRigor.
We are incorporating these changes into the revised manuscript, and the updated PDF will be uploaded shortly.
We appreciate the reviewer’s suggestions, which have helped us improve both the experimental comparisons and the clarity of the manuscript.
thanks for the responses. these replies address some of my concerns, but my main concerns remain: the significance of, and the difference from, existing work on evidence-based methods. The author has also acknowledged and incorporated them. I therefore maintain my original vote 2.
Initial Response and Plans for Additional Experiments
We thank the reviewer for the constructive feedback. We clarify MetaRigor's methodological design and address the concerns regarding presentation and evaluation below.
R1. Related Work and Baseline Selection. We appreciate the concern and will expand Related Work to clarify the contributions of existing systems and their differences from MetaRigor. We initially selected Codex CLI and OpenHands CLI because few existing automated systems cover the full meta-analysis workflow, and we aimed to evaluate the autonomous planning and tool-use capabilities of general-purpose agents. We agree that these comparisons alone are insufficient. Therefore, we are adding Manalyzer[1] and meta-pipe[2] as supplementary baselines to evaluate MetaRigor against specialized meta-analysis systems.
R2. Paper Readability. We apologize for the unclear definitions and terminology. We will revise how key concepts are introduced and defined throughout the manuscript to improve readability.
R3. Scientific Decision Boundary. We appreciate the request for clarification. MetaRigor does not merely constrain model behavior through prompts; it implements decision boundaries through explicit task constraints, programmatic execution, and validation. We will clarify the formal definition of the Scientific Decision Boundary, which comprises three elements: the Object of Judgment, Admissible Facts, and Decision Authority assigned to models and programs. MetaRigor operationalizes these elements through Task Contracts and corresponding program logic.
R4. Stage-Level Evaluation and Cross-Stage Propagation. We agree that Line 106 requires clarification. The two terms refer to distinct evaluation settings: Stage-level evaluation measures each stage independently with fixed upstream inputs, whereas cross-stage propagation analysis traces how result-identity mismatches or omissions affect downstream statistical inputs, risk-of-bias assessments, and final reports. We will clarify this distinction in the manuscript.
R5. Clarification of SELECT2. SELECT2 is a clinical trial of endovascular thrombectomy for large ischemic strokes reported by Sarraj et al[3]. It is included in our end-to-end analysis to illustrate a result-identity mismatch. We will clarify this at its first mention.
R6. Citation Practices. We will revise the Related Work section to associate each study's description directly with its citation, rather than grouping multiple studies under a collective citation. We will also clarify their individual contributions and relationships to MetaRigor. We apologize for the lack of clarity.
We expect to share additional results by November 16, followed by a detailed response. We appreciate the reviewer's time and continued engagement.
References
[1] Xu, W., Zhang, W., Ling, F., et al. (2025). Manalyzer: End-to-end Automated Meta-analysis with Multi-agent System. arXiv:2505.20310.
[2] Lin, H.-T., & Yeh, J.-T. (2026). meta-pipe: An LLM-agent pipeline for end-to-end automated systematic review and meta-analysis. arXiv preprint arXiv:2606.28363.
[3] Sarraj, A., Hassan, A. E., Abraham, M. G., et al. (2023). Trial of endovascular thrombectomy for large ischemic strokes. New England Journal of Medicine, 388(14), 1259–1271. https://doi.org/10.1056/NEJMoa2214403.