1. Statistical methodology and scientific validity (Clarification)
Pages 5–6, Section 4; Appendix E: The manuscript does not systematically describe the statistical methods used in the nine epidemiological reanalyses. Can the authors provide a study-by-study comparison of the original and reconstructed statistical models, including regression specifications, covariates, confounding-adjustment strategies, uncertainty estimation methods, and underlying assumptions? If causal interpretations are intended, what identification assumptions and methods support them? Without this information, how can readers determine whether the completed analyses are scientifically valid rather than merely computationally successful?
2. Preservation of the original research question and comparability of statistical results (Clarification)
Page 21, Table 6; Appendix F: Several reanalyses modify important aspects of the original studies, including study populations, exposure periods, geographic coverage, statistical specifications, and even health outcomes. How do the authors establish that these substitutions preserve the original estimands and scientific questions? Furthermore, some published statistical coefficients are transformed into multiplicative effect measures. For example, the Italian COVID-19 study uses a log-log regression model, yet its transformed result is presented as an odds ratio. The US stroke mortality study similarly converts a linear regression coefficient into a ratio-like measure. What statistical assumptions justify these transformations and comparisons? How can agreement between the reconstructed and published estimates be interpreted when the underlying models or estimands differ?
3. Independent validation of reconstructed pollution exposure (Clarification)
Page 5, Section 3.4; page 9, lines 445–450; Appendix D: The central contribution of AERIS is reconstructing pollution-exposure estimates, yet the evaluation primarily relies on agreement between downstream epidemiological results. Have the authors independently validated the reconstructed exposure concentrations against reference datasets, held-out monitoring measurements, or established exposure-assessment methods? In particular, Appendix D describes using an unweighted mean across monitoring stations for administrative-area exposure. How do the authors account for the nonuniform geographic distribution of monitoring stations, potential spatial bias, missing coverage, and exposure-measurement uncertainty? What quantitative evidence supports the accuracy of the reconstructed spatial and temporal pollution patterns?
4. Computational completion versus scientific correctness (Clarification)
Pages 6–8, Sections 4–5; Table 1; Appendix H: Autonomous AERIS completes 23 of 27 attempts, compared with 7 of 27 for the baseline, but its agreement with published results deteriorates without researcher supervision. Even with supervision, only three of the seven reconstructed confidence intervals for published positive associations exclude the null, and only four of seven published null conclusions are retained. How do the authors distinguish computational completion, agreement with published findings, and scientific validity? What criteria determine whether a completed statistical analysis is scientifically defensible? Could the evaluation reward an agent for producing a fitted but insufficiently supported result while penalizing a baseline agent for appropriately declining to perform an analysis when the available data or statistical assumptions are inadequate?
5. Potential evaluation bias and access to published results (Clarification)
Page 6, lines 309–323: The authors state that the agents do not receive the original pollution-exposure tables, but it is unclear whether they can access the published statistical effect estimates and conclusions through the supplied articles. Were these results hidden from the agents during the reanalyses? If not, how do the authors rule out the possibility that knowledge of the expected results influenced the selection of statistical models, covariates, exposure sources, or data substitutions? This is particularly important because numerical agreement with published estimates is a central evaluation metric.
6. Generalization and justification for autonomous epidemiological analysis (Clarification)
Page 9, lines 459–469; Sections 4–6: The authors acknowledge that the recovery policies were developed after observing failures in the researcher-supervised nine-study campaign, and the autonomous evaluation uses the same studies. What evidence demonstrates that these policies generalize to previously unseen studies rather than improving performance on a familiar set of research tasks? More fundamentally, what evidence supports extending AERIS from exposure reconstruction to autonomous epidemiological inference when the exposure estimates have not been independently validated and the statistical analyses may not preserve the original study designs? Would the current experimental evidence support the more limited claim that AERIS improves exposure-data reconstruction and research workflow execution, rather than the broader claim of reliable autonomous environmental-health research?
These questions are clarification requests concerning the evidence and validity of the reported contributions, rather than suggestions for future work. Their answers could materially affect my assessment, as my primary concerns involve the absence of independent scientific validation, the reliability of the statistical conclusions, and the distinction between completing an analysis and performing scientifically defensible research.
Response to Reviewer MpEi
We thank Reviewer MpEi for reviewing our work and noting the value of the end-to-end evaluation framing and real workflow artifacts.