acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Text: Verifying That Agent-Written Papers Are Backed by Their Artifacts

Abstract

Large language model agents are increasingly capable of conducting research autonomously, producing research documents alongside the code and experiments that ostensibly support them. Yet whether the reported findings are consistently supported by corresponding implementations and execution evidence remains largely unexplored. Existing review practices primarily assess textual quality and cannot reliably identify inconsistencies such as hard-coded metrics, unimplemented methods, or unsupported experimental results. We present ReAgent, an automated auditing framework for assessing the consistency between agent-generated research documents and their associated repositories. ReAgent constructs structured representations of scientific claims from research documents and uses them to guide repository analysis and evidence collection. Static auditing examines whether claimed methodologies, implementations, and experimental configurations are consistently reflected in the repository. Dynamic auditing executes relevant experiments and collects execution evidence to assess empirical findings. By combining static analysis with dynamic evidence, ReAgent identifies inconsistencies that may remain hidden under either perspective alone. The collected evidence and audit decisions are organized into a structured repository-level audit report, enabling transparent evidence traceability. We collect and manually annotate a benchmark of agent-generated research document–repository pairs, and evaluate ReAgent against representative static and reproduction-based baselines. Experimental results demonstrate that ReAgent effectively identifies inconsistencies between reported research findings and their supporting repository evidence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent System, Agentic Security

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ktstOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Paper introduces ReAgent, an automated auditor to check if documents and code produced by agents are consistent between each other. ReAgent produces structured representations of the paper claims. There are two modes for analysis: static and dynamic, and bother are LLM-agent-driven. In static, agents summarize paper and code into some structured format and "judge" the semantic alignment. In dynamic, agents review and execute the code so that's it is reproducible. Authors manually create an evaluation benchmark of 96 tasks of (paper, code repo) and find that ReAgent identifies inconsistencies.

Critical strengths
  • Authors target an important problem of faithfulness in agent outputs, that both research text in natural language and code should be consistent with each other. In fact, this is a problem beyond agent era where ideally we'd like human scientists to also do this. Paper proposes a workflow for stress-testing agent outputs.
  • The idea to standardize summaries of research papers and code into structured metadata is good, though limited to the tasks presented in the paper.
Critical weaknesses
  • Framework orchestrates several agents in Figure 2, giving them tasks that are somewhat vaguely defined and redundant e.g. "planning" "execution". Figure 2 seems to be quite a lot and yet gives loose information on what the method looks like. Paper points to Appendix A.1 with structured output schema for the agents, and it'd be better to describe them in the main text while cutting down on high-level overview.
  • I'd argue that ReAgent is not so generalizable to science domains as the paper claims. The prompts and schema outputs for static and dynamic analysis are geared for ML-heavy tasks, and so are the papers selected from AutoSOTA as a benchmark. Fundamentally, static and dynamic analysis is often not doable in other fields, where lab access is crucial. In a sense, ML papers have the extraordinary benefit of being reproducible, on machines anyone can reasonably get access it.
  • Authors can describe the manual annotation process a bit more, and how they plan on extending it. It seems quite a bit of manual effort going through all these papers and code. Oh, I see that Claude Code did this manual effort... well this is a chicken-and-egg problem: how can you trust Claude Code to read and find holes to add to benchmark tasks, when the benchmark is supposed to be used to evaluate agents in reading/finding holes?
  • This seems problematic, because it asks the static analysis agents to just say yes or no: "The static verification task is formulated as a binary classification problem, where the system predicts whether a repository contains static claim-code inconsistencies."
  • Writing is sloppy in results section, and the space can be better used. I'm not sure how to read Table 3 and 4 (static appears in both row and column, and why don't the last 2 rows of the tables match in numbers?). Paper should go through more work and experiments.
Questions for authors
  • Would you use ReAgent yourself, e.g. as a tool, rather than codex or claude code to analyze and reproduce paper results? What is the value-add of defining these prompts and structured schema, beyond better numbers on the curated benchmark?
  • Currently in my own workflow and my fellow scientists, I ask agents to implement papers and verify against the released source code/run it. So why give this workflow---that all scientists use---a name ReAgent? Due to this, I don't particularly yet see the novelty in the manuscript.
3 comments

W1: Figure 2 and Framework Description

We thank the reviewer for this constructive comment. We agree that Figure 2 currently contains a large amount of high-level information while giving only loose details about the individual agents, and that some agent tasks are described in a somewhat vague and redundant manner (e.g., “planning” and “execution”). In the updated version of the paper, we will revise Figure 2 to better clarify the division of labor among agents and remove redundant or uninformative task labels. We will also describe the agents more concretely in the main text, including their responsibilities, inputs/outputs, and the structured output schema currently provided in Appendix A.1, while cutting down the high-level overview. We believe these changes will make the method substantially clearer and more self-contained. We will make these revisions in a subsequent version update.

W2: Generalizability Boundaries

A: We agree that ReAgent is not so generalizable to science domains as the paper may have implied, and we will scope our claims accordingly. We also agree that static and dynamic analysis is often not doable in other fields, where lab access is crucial. Our work targets academic-integrity auditing of autonomous research agent systems, which typically use code for automated research; it does not claim generalizability to lab-based science. We further agree that the prompts and schema outputs for static and dynamic analysis are geared for ML-heavy tasks, and that the papers selected from AutoSOTA as a benchmark are similarly ML-heavy. ML papers do have the extraordinary benefit of being reproducible on machines that anyone can reasonably get access to, and this is precisely the setting in which our static/dynamic auditing pipeline is applicable. We will revise the abstract and introduction to avoid broad generality claims, explicitly scope ReAgent to autonomous research agent systems with executable code and ML-heavy artifacts, and list extension to other science domains—especially those requiring lab access and domain-specific schemas—as future work. We will also clarify that the AutoSOTA-based evaluation is a scoped benchmark rather than evidence of broad cross-domain generality

W3: Annotation Process and Credibility

We will describe the manual annotation process more clearly in the revised manuscript and state that Claude Code does not produce the final benchmark labels; all final labels are reviewed and adjudicated by the authors. Regarding the chicken-and-egg issue, we do not treat Claude Code as ground truth, nor do we rely on it alone to read and find holes; it is used only to generate candidate issues, and all issues claimed by Claude are reviewed again by the authors before entering the benchmark.

The specific process is as follows: for static annotation, 3 different Claude models first independently review the same batch of repositories to reduce missed detections by a single model; their claimed problematic candidates are merged, yielding about 71 flagged repositories, covering most repositories; only a small number of repositories require the authors to carefully review them line by line; each Claude claim is still reviewed by the authors in light of the code context, checking whether the risk falls in actually used code, whether parameters are hard-coded by default, whether it belongs to deprecated code, etc., and the final binary labels are adjudicated by the authors. Therefore, the benchmark labels are not the output of Claude Code, but labels after author adjudication; Claude Code only undertakes candidate generation, reducing manual effort, and does not replace ground truth.

We acknowledge that this may still fail to detect all static issues 100%, and we do not claim that Claude Code’s recall is perfect; however, independent review by three models plus author review at least provides some reference value and reduces the risk of missed detection by a single model.

W4: Static Verification Formulation

A:We agree that the original sentence was problematic and have revised it; the static verification task is not formulated as asking the static analysis agents to just say yes or no. The revised sentence now states that static verification identifies and assigns specific labels to claim-code inconsistencies, while a binary truth value is used only for subsequent state transitions and experimental evaluation.

Specifically, we have replaced the original “The static verification task is formulated as a binary classification problem...” with: “The static verification stage identifies and assigns specific labels to claim-code inconsistencies. However, for subsequent state transitions and experimental evaluation, these results are represented using a binary truth value (i.e., whether an inconsistency is detected).” We did not formulate the static verification task as a pure binary classification problem at the agent-output level. The binary representation is used because the compared baselines do not have our eight-category risk definition; retaining the specific labels in the static audit output while evaluating on a binary truth value ensures comparability. We will also clarify in the revision where the specific labels are stored/reported and that the binary value is an evaluation and state-transition representation, not the only output of the static analysis agents. This should address the concern that the formulation asks the static analysis agents to just say yes or no.

W5: Results Writing and Tables

Regarding Tables 3 and 4, the 1.04 percentage points you pointed out (82.29% → 83.33% in Table 3) is a classification metric for stage combinations within the same system; the 35.42% → 70.83% in Table 4 is a repository-level aggregate metric across different systems for “both static and dynamic correct,” used to evaluate agreement with human annotations. The two evaluate different events, so the values are not expected to be consistent.

response to W1 and W2

Thank you for agreeing with me, but I'd have liked to see an update to the Figure or a concrete plan of how you'd update it. Your plan currently is "we'll do it", but with that I don't know how to recommend for acceptance.

response to W3

Makes sense that Claude Code gives human annotators a headstart. That is a natural approach, yet human verification remains a bottleneck. I understand this bottleneck is a more general problem in the field right now. So I'm satisfied with the W3 response.

your update for W4: "The static verification stage identifies and assigns specific labels to claim-code inconsistencies. However, for subsequent state transitions and experimental evaluation, these results are represented using a binary truth value (i.e., whether an inconsistency is detected)"

The writing wasn't an issue for me, but the approach to collapse static to a binary classification task was. I think output for static should be a bit more rich, at least with a rubric for instance. This remains a weakness in the approach.

response for W5

The updated Table 3 is completely different, and you didn't explain why you replaced it. I didn't get a response for why the row and column both have "static". Still not sure how to read Table 4.

My questions are left unanswered as well. In light of lingering concerns, I'm maintaining my score and increasing my confidence. The writing in method and experiments really does need a lot of work.

Reviewer_rZDdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReAgent, a framework for auditing whether claims in agent-generated research documents are supported by their associated repositories and execution evidence. The system combines static analysis and dynamic reproduction.

The static stage extracts structured claim contracts from the report, analyzes repository metadata and code paths, binds paper-side claims to repository-side objects, and identifies possible implementation inconsistencies. The dynamic stage constructs executable verification targets, prepares an isolated environment, runs the relevant code, collects logs and artifacts, and uses an evidence audit procedure to classify claims as matched, mismatched, blocked, or not reproduced.

The authors construct a benchmark from 188 AutoSOTA repositories and retain 96 repositories after excluding cases with unavailable assets, missing reports, or missing source code. They manually annotate static risks and dynamic reproduction outcomes. ReAgent is compared with a Static-Audit baseline and a Codex baseline. It obtains 78.13% static accuracy, 90.63% dynamic accuracy, and 70.83% overall accuracy, compared with 35.42% overall accuracy for Codex.

Critical strengths
  1. The problem is relevant and well motivated. Autonomous research systems can produce a report, source code, logs, and result files that do not agree with one another. Auditing both implementation correspondence and execution evidence is a meaningful problem for trustworthy scientific automation.

  2. The system provides a useful evidence and failure taxonomy. Claim contracts, source-level evidence, execution logs, artifact provenance, and explicit blocked states are sensible engineering choices. The case studies illustrate why a single reproduction score is insufficient to characterize artifact reliability.

Critical weaknesses
  1. The paper does not establish a sufficiently distinctive methodological contribution.

The proposed framework appears to be an orchestration of existing techniques: LLM-based claim extraction, repository search, semantic matching, agent planning, sandboxed execution, logging, and rule-based evidence auditing. The paper does not introduce a new learning algorithm, verification algorithm, formal guarantee, or principled optimization objective. The main contribution is therefore a system integration and benchmark construction.

This could still be a valuable systems or benchmark contribution, but the current manuscript does not establish that the integration itself is sufficiently novel for a top machine learning conference. The benchmark also has limitations: it is derived from a single source, heavily filtered, mostly repository-level, and not accompanied by strong independent annotation statistics.

  1. The reported improvement is not cost-normalized and cannot be cleanly attributed to ReAgent’s design.

ReAgent uses approximately 10.75M tokens and 3 hours 47 minutes per repository, compared with 2.07M tokens and 46 minutes for Codex. Thus, ReAgent uses roughly 5.2 times as many tokens and 4.9 times as much time. It also has a substantially more structured protocol, multiple agents, iterative execution, evidence extraction, and a much larger execution budget.

Although ReAgent improves overall accuracy from 35.42% to 70.83%, the paper does not show whether the improvement comes from the proposed architecture or simply from giving the system more computation, more retries, and more opportunities to inspect and execute the repository. There is no compute-matched Codex baseline, no performance-versus-budget curve, no cost-normalized metric, and no complete ablation isolating the marginal contribution of claim contracts, semantic binding, iterative reproduction, and evidence auditing.

  1. The main evaluation label conflates different failure modes.

The paper maps Mismatched, Blocked, and Not Reproduced to the same negative class for dynamic accuracy. However, a measured mismatch is fundamentally different from an execution blocked by an unavailable checkpoint or dependency. The paper’s conceptual contribution depends on distinguishing these cases, but the main accuracy numbers do not preserve the distinction.

As a result, the reported dynamic and overall accuracy may partly measure the ability to identify unverifiable cases rather than the ability to detect genuine claim-code inconsistencies. Separate precision, recall, F1, and confusion matrices for Mismatched, Blocked, and Not Reproduced are needed.

  1. The benchmark construction may introduce selection bias.

The authors remove 79 repositories because external assets were unavailable, in addition to removing repositories without reports or source code. This makes the remaining benchmark more auditable, but it also removes a large class of real-world artifact failures. Missing assets, incomplete repositories, and unusable environments are themselves important reliability problems for autonomous research systems.

The paper should report results on the full 188-repository collection with incomplete or blocked cases retained as separate outcomes. Otherwise, the reported performance only characterizes a selected subset of AutoSOTA artifacts.

  1. The uncertainty and annotation reliability are insufficiently reported. The framework contains stochastic LLM decisions, repository search, execution planning, and dynamic reproduction. Nevertheless, the paper reports point estimates without standard deviations, confidence intervals, or repeated independent runs. The static ground truth is also constructed with LLM-assisted candidate discovery followed by author discussion, but no quantitative inter-annotator agreement is reported.

Without repeated runs and annotation agreement statistics, it is difficult to determine whether the reported differences are stable or whether they depend on particular model trajectories and adjudication choices.

Questions for authors
  1. Methodological Novelty & Cost Effectiveness

What specific algorithmic innovation does ReAgent’s multi-agent orchestration introduce beyond standard task decomposition, and how does it perform when compared against a compute-matched single-agent baseline with equal token and time budgets? Given its high resource cost (~10.75M tokens and ~3.8h per repo), could the authors provide performance-versus-budget curves and clarify the practical use cases that justify this expenditure?

  1. Benchmark Bias & Generalization

Does ReAgent generalize beyond the filtered subset of 96 AutoSOTA repositories, and could the authors evaluate the system on the full 188-repository set or a separate autonomous system without excluding unexecutable cases? Without ground-truth labels for the AAAI 2024 experiment, should the paper's claims be scoped specifically to this heavily filtered AutoSOTA subset?

  1. Fine-Grained Ablation Studies

Could the authors provide component-level ablations under matched compute budgets to isolate gains from individual modules, such as claim contracts, semantic binding, and iterative coordination? Furthermore, how should the minor 1.04% accuracy gain in Table 3's static-plus-dynamic ablation be reconciled with the much larger improvement over Codex in Table 4?

3 comments

Q1: Methodological Novelty & Cost Effectiveness

ReAgent’s specific algorithmic innovation beyond standard task decomposition is the structured claim–code–execution–review loop: the agents exchange explicit claim-code bindings and evidence, and the Review stage audits the outputs of Reproduce, rather than merely splitting the task among multiple agents. We have not yet established a strict compute-matched single-agent baseline with equal token and time budgets; because Codex’s token and time consumption is uncontrollable, we instead provide a billing-based cost-normalized comparison and will clearly state this boundary in the revised manuscript.

We agree that the original paper’s average of 10.75M tokens and 3 hours 47 minutes for ReAgent versus 2.07M tokens and 46 minutes for Codex is the most important fairness risk. However, Codex’s token and time consumption is uncontrollable, so we do not claim to have completed a compute-matched comparison; instead, we perform a cost-normalized comparison based on billing cost. We conducted supplementary experiments with the lower-unit-price model deepseek-flash, obtaining static 45/96, dynamic 79/96, and joint 36/96; on the same current ground truth, Codex obtains 40/96, 77/96, and 34/96. This shows that under this cost-normalized setting, ReAgent can still produce complete dynamic labels, and its current binary correctness counts are slightly higher than those of Codex. Our statistics show that after switching to this lower-unit-price model, ReAgent costs about $0.64 per repository in billing, while Codex costs about $2.12. Taking I-PERI as an example, a full run with DeepSeek-Flash uses 14.38M tokens at an estimated $0.71; among this, the Reproduce stage uses 13.06M (90.8%), code and claim extraction and binding use 0.58M (4.1%), Review uses 0.43M (3.0%), and the rest is used for evidence extraction and history summarization. This indicates that the high token/time cost mainly comes from the necessary Reproduce stage, rather than from multi-agent orchestration itself.

MetricOur systemCodex (same current human labels)
Dynamic binary correctness79/9677/96
Static correctness45/9640/96
Both simultaneously correct36/9634/96

Q2: Benchmark Bias & Generalization

We have not yet completed an evaluation with ground-truth labels on the full 188-repository set or on a separate autonomous system without excluding unexecutable cases. The reason we selected this portion of the dataset is that currently advanced autonomous research systems, when open-sourced, usually release only the papers written by the agent, without the experimental code and logs corresponding to those papers. Therefore, we have not yet tested on data outside AutoSOTA.

To address data selection bias, we supplemented the experimental results for some excluded repositories. From the 79 repositories that were flagged by the asset audit during screening as unavailable or requiring authorization, we selected 5 repositories to attempt to reproduce the final results: TokenRecycling, PIGuard, GF-OT, ChordEdit, and GraniteGuardian. Among them, 2 repositories did not complete valid metric runs and respectively output not_reproduced; the remaining 3 repositories were attempted, but ultimately produced blocked results. This supplementary result indicates that these excluded/unexecutable cases indeed lack the necessary data or materials under the current conditions, and therefore still cannot produce valid metrics.

Q3: Fine-Grained Ablation Studies

We have supplemented component-level ablations and, on the same 10-repository subset, controlled computational conditions by using the same static goals and context and reinitializing the dynamic state, so as to isolate the gains from claim contracts, semantic binding, and iterative coordination; strict equal-token/equal-time matched compute budgets are still affected by uncontrollable execution, and we will clearly state this boundary in the revised manuscript. The minor 1.04% accuracy gain in Table 3 does not contradict the much larger improvement over Codex in Table 4: the former is the marginal accuracy gain of the static-plus-dynamic ablation within ReAgent, while the latter is the end-to-end comparison between the full ReAgent and Codex; the two differ in comparison object, denominator, and the factors isolated.

Specifically, on the same 10-repository subset, the full Contract+Binding setting achieves 7/10, no Binding achieves 5/10, no Contract achieves 6/10, and neither achieves 5/10. The alternative pipelines still output static judgments normally, so these scores are not mechanical failures caused by missing fields. The dynamic-component ablation uses the same static goals and context for each of these 10 repositories and reinitializes the dynamic state; the latest valid terminal states are 7/10 for the full pipeline, 4/10 without Review, and 4/10 without Evidence. We also tested the system without Audit, which reaches 6/10, but our analysis found that the Agent guessed the final labels without sufficient evidence; although this has high agreement with the ground truth, it contradicts our goal of trustworthy reproduction verification, so we do not regard it as a valid gain.

tatic-component setting (same 10-repository subset)Score
Full Contract + Binding7/10
No Binding5/10
No Contract6/10
Neither5/10
Dynamic-component setting (same 10 repositories)Latest valid terminal score
Full pipeline7/10
Without Review4/10
Without Evidence4/10
Without Audit6/10

Regarding reconciliation: the 1.04% in Table 3 measures the marginal gain of adding the dynamic module on top of the existing static module, and its denominator and comparison object are the same set of cases within ReAgent; the static module already covers most statically determinable risks, so the marginal improvement of the dynamic module on binary accuracy is limited, and its main role is more reflected in evidence verifiability, terminal-state trustworthiness, and reducing unverified conclusions. The larger improvement over Codex in Table 4 measures the end-to-end gap between the full pipeline and the Codex baseline; its sources are claim contracts, semantic binding, iterative coordination, and the constraints imposed by Review/Evidence/Audit on execution evidence, rather than the marginal accuracy brought by the dynamic module alone. In the above ablations, the full pipeline’s 7/10 is clearly higher than 5/10 without Binding, 6/10 without Contract, 4/10 without Review, and 4/10 without Evidence, supporting the overall contribution of the module combination; the Codex baseline lacks these explicit evidence constraints and auditing, so the end-to-end improvement of the full system over Codex can be larger than the marginal gain of the dynamic module.

We will add a component-level ablation table in the revised manuscript and analyze the original Table 3 as an experimental result in the appendix.

Thank you for the detailed response and the additional experiments. While I appreciate the component ablations on the 10-repo subset and the breakdown of outcomes, my core concerns remain unresolved-the billing-normalized comparison with deepseek-flash shows that ReAgent's joint accuracy drops sharply to 36/96 , only marginally ahead of Codex's 34/96. This reinforces my concern that the headline gains in the main paper largely stem from heavy compute and token budgets rather than a architectural advantage. Moreover, the lack of full-dataset evaluation and limited generalization beyond the filtered AutoSOTA subset remain key limitations. Therefore, I will maintain my current score.

Reviewer_jt8POpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether the claims in a document written by an autonomous research agent are actually backed by the repository the agent produced. It proposes ReAgent, which first extracts structured "claim contracts" from the document, separating implementation claims (checked in code) from experimental claims (checked by execution). Static verification builds repository metadata (entry points, call and import structure, result writers), binds claims to code, and flags inconsistencies under a fixed risk taxonomy: hard-coded results, substantive implementation discrepancies, omitted or dummy implementations, fabricated datasets or hyperparameters, and label leakage. It also constructs executable targets. Dynamic verification then iteratively plans and runs the experiments, restricted to logged compatibility fixes, and a round audit assigns each target one of four outcomes: Matched, Mismatched, Blocked, or Not Reproduced.

ReAgent is evaluated on 96 agent-generated repositories from AutoSOTA (188 released, 92 excluded), with manually established ground truth, against an adapted static auditor (Static-Audit) and a Codex agent, all using GPT-5.4. It reaches 78.1% static accuracy, 90.6% dynamic accuracy and 70.8% joint accuracy (Codex: 41.7%, 79.2%, 35.4%). Of the 96 repositories, 28 contain a confirmed static issue, and 8 of the 33 whose reported metric reproduces still have unsupported implementations. The authors conclude that static and dynamic evidence are complementary, and that recovering a reported number does not show that the claimed method was implemented.

Critical strengths

Reproducing a number is shown not to be enough, and the verdict design avoids repairing the claim into agreement (§3.3, §4.3, §4.4.3). Dynamic auditing reproduces the reported metric for 33 of 96 repositories, but only 25 of those also pass the static checks. In the other eight the metric is recovered despite unsupported implementations, and static verification catches seven of them. The Reproduce Agent is limited to logged compatibility fixes (no change to algorithm, split, metric or aggregation), and Blocked, Mismatched and Not Reproduced are kept distinct, as in the invalid-checkpoint PSNR case. This is a reusable and consequential insight: it argues against judging agent-written research by rerun success alone, and it is the strongest evidence in the paper that both halves of the pipeline are needed.

The paper documents, with a full manual audit, that claim–code gaps are common in agent-written repositories (§4.1.3, §4.3, Table 2). Three authors verified code-level evidence for every one of the 96 repositories. Twenty-eight (29%) contain at least one confirmed static inconsistency, most often substantive implementation discrepancies (10) and label leakage (7), and only 25 are clean on both static and dynamic checks. Even on a filtered subset of one agent system's output, these counts motivate the task concretely and supply a usable taxonomy, and the disclosure of the exclusion criteria and selection bias (§4.1.1, §5) makes the numbers interpretable.

Critical weaknesses

The headline gain over the baselines cannot be attributed to the claim-contract design (§4.1.2, Tables 3, 4, 14, 15; App. A.2.3). On average ReAgent uses about 5× the tokens and wall-clock time of Codex (10.75M vs. 2.07M tokens; 3h47m vs. 46m; roughly 11× and 6× at the median). It is allowed up to six review–reproduce rounds and eight hours per repository, whereas Codex receives one prompt and "typically executes a single end-to-end reproduction attempt". A.2.3 says the runtime parameters determine whether methods are compared under comparable budgets, but only ReAgent's are given. The static comparison is not like-for-like either. From the reported recall, F1 and accuracy on the 28 positive repositories, Static-Audit flags about 94 of 96 repositories and Codex about 66 (implied counts), and the Codex prompt does not list the risk categories it is scored against. The only ablation (Table 3) toggles static versus dynamic verification; the paper's named contributions, claim contracts and semantic binding, are never ablated. The 35-point overall gap may therefore reflect more compute and a task-specific prompt rather than the proposed representation. The closest purpose-built systems are also missing. SciCoQA (ACL 2026) is cited only in passing, although it offers an external benchmark with independent ground truth for the static task. VERITAS (arXiv:2607.02931, July 2026) also extracts claims, executes code and judges claims against run evidence; it is recent enough that I do not count its absence heavily, but the novelty statement in §2 should address it. This is fixable but needs new experiments: a compute-matched multi-round baseline, a contract and binding ablation, and one external benchmark.

The evaluation is too small, filtered and coarsely scored to support "effectively identifies inconsistencies" (§4.1, §4.2, App. A.2–A.4).

  • Scale and selection. The 96 repositories come from a single agent system, after an LLM-run filter removed 92 of 188 (79 for unavailable assets), checked on a random sample of 30. No confidence intervals are reported. The authors acknowledge that results describe the "auditable subset".
  • Trivial baselines. Predicting "no static issue" for every repository scores 70.8% (68/96); ReAgent's 78.1% is seven repositories higher, with implied precision of about 59% (23 true and 16 false positives). Predicting "not reproducible" everywhere scores 65.6% dynamic accuracy, against ReAgent's 90.6%.
  • Metric. Mismatched, Blocked and Not Reproduced are all scored as the negative class, so dynamic accuracy largely rewards getting an experiment to run (reproduction success 76.0% vs. 60.4% for Codex). The Blocked-versus-Mismatched distinction emphasized in §4.4.3 is never scored.
  • Ground truth. Static candidates come from Claude models and the dynamic ground truth is "agent-assisted"; the paper does not say which agent built the environments or whether those runs overlap with the evaluated systems.
  • Generalization. The one experiment on human-authored papers (App. A.4) has no ground truth: of 105 claims, 23 were fully verified and 46 were "not checkable".
Questions for authors
  1. Clarification. What time, token and round budget was Codex given? Does a Codex baseline with a comparable budget (multiple rounds, around 10M tokens) and the eight risk categories in its prompt close the gap?
  2. Clarification. Can Mismatched be reported separately from Blocked and Not Reproduced (precision and recall for true inconsistencies), with confidence intervals?
  3. Clarification. Which agent built the execution environments for the dynamic ground truth, and did any of those runs overlap with ReAgent's or Codex's? Of the repositories ReAgent flagged statically but the ground truth did not (about 16 by my reckoning), were any re-examined for issues missed by the Claude-based candidate step?
  4. Clarification. How accurate is the asset filter beyond the 30-repository sample, and what does ReAgent output on the 92 excluded repositories?
  5. Suggestion. Running the static module on SciCoQA, and relating ReAgent to VERITAS, would test generalization against independent ground truth.
2 comments

Q1: Codex Budget and Baseline Gap

We did not set a predefined time, token, or round budget for Codex; we used its native /goal mode, in which Codex keeps working after the goal is set until the goal is achieved or it is blocked, and we recorded its actual token usage and wall-clock time. For the Codex baseline with a comparable budget (multiple rounds, around 10M tokens) and prompts containing the eight risk categories, we did not observe it closing the gap; in the cost-comparable settings we have run, Codex remains slightly below our system.

Thank you for accurately pointing out the value of “numerical reproduction does not equal faithful implementation” and for noting that the existing design cannot isolate the impact of additional computation. Since Reagent is also a method that requires no human interaction, we also stop interacting with Codex once it reaches a terminal state; therefore, Codex’s actual consumption is not something we can preset or control. In the original experiments, we did not count the number of rounds Codex used per repository; we reran the latest version of Codex on 3 repositories and obtained a reference average of about 3.33 rounds per repository, but this is for reference only because Codex has undergone multiple updates.

Our high token consumption and high time consumption stem from multi-agent collaboration and a staged working mode; directly reducing consumption to the same level as Codex may cause the system to complete only a single stage of analysis. Considering that excessively high consumption may indeed be unconvincing, we switched the underlying model to deepseek-flash so that the two baselines reach a similar level of billed consumption; token consumption and time consumption were not reduced as a result. In the latest experiments, dynamic binary correctness is 79/96; static correctness is 45/96; both are simultaneously correct for 36/96. On the same current human labels, Codex’s records are 77/96, 40/96, and 34/96, respectively (dynamic, static, and both simultaneously correct).

MetricOur systemCodex (same current human labels)
Dynamic binary correctness79/9677/96
Static correctness45/9640/96
Both simultaneously correct36/9634/96

Q2: Separate Reporting of Mismatched vs. Blocked/Not Reproduced with CIs

Okay. In the revised manuscript, we will report Mismatched separately from Blocked and Not Reproduced. The current 96 system outputs are: matched 22, mismatched 13, blocked 56, and not_reproduced 5; among them, there are 11 cases where the manual binary ground truth is reproducible but the system judges them as blocked. Therefore, reporting Mismatched separately from Blocked/Not Reproduced is necessary and informative.

In addition, we selected 5 repositories and ran 3 repetitions as a reference for reproducibility: 4 repositories achieved 3/3 agreement, and the remaining 1 repository achieved 2/3 agreement. We will report this reproducibility result in the revised manuscript and clearly state that it is only for reference and does not replace manual verification of true inconsistency across all 96 outputs.

CategoryCountNote
matched22
mismatched13Will be reported separately from blocked and not_reproduced
blocked56Includes 11 cases where the manual binary ground truth is reproducible but the system judges them as blocked
not_reproduced5

Q3: Execution Environment Ownership and Static-Only Flag Review

The execution environments for the dynamic ground truth were not built by a separate agent; they were the virtual environments and prepared resources built by Codex and ReAgent and reused for manual reproduction. Yes, there was overlap at the environment/resource level: we reused Codex- and ReAgent-built environments, while the dynamic ground truth runs themselves were performed manually; if neither Codex nor ReAgent produced a result, we performed manual reproduction on the already-built environment. The purpose of doing this was to reduce workload, but the dynamic ground truth itself comes from manual reproduction, rather than directly adopting the automatic run results of Codex or ReAgent.

Yes, we re-examined all about 16 repositories that ReAgent flagged statically but the ground truth did not, specifically for issues missed by the Claude-based candidate step. Of these, 10 had entered the Claude candidate union and 6 had not; we found no new static issues. These false positives mainly come from two types of situations: some parameters are hard-coded by default in the code and were not retrieved, or risks in code that is actually unused were reported (the repository appears to contain multiple code attempts released together, and some code is suspected to be deprecated).

Q4: Filter Accuracy Beyond 30 Repos; ReAgent on 92 Excluded

We currently do not have a complete estimate of asset filter accuracy beyond the 30-repository sample; we conducted a targeted audit of 5 of the 79 asset-labeled repositories, and none of these 5 produced valid reproduction metrics (2 output not_reproduced and 3 output blocked). For the 92 excluded repositories, ReAgent does not produce valid metric outputs: the outputs for the above 5 are not_reproduced or blocked. This targeted audit supports that these repositories cannot produce valid reproduction metrics under the current conditions, but it is insufficient to provide a complete accuracy rate for the asset filter beyond the 30-repository sample.

Q5: Independent Generalization Test: SciCoQA + VERITAS

We have run the static module on SciCoQA to test generalization against independent ground truth. On this independent ground truth, ReAgent achieves 28/92 (30.4%) on the 92 real differences in SciCoQA. According to the reference data in the paper, this performance is roughly between GPT-5 Codex and Gemini 2.5 Flash; this result does not show an advantage of our method, but it is overall within an acceptable range.

We have attempted to relate ReAgent to VERITAS and have read its paper to determine a reproducible way of comparison. At present, we cannot conduct a reproducible quantitative comparison: VERITAS does not appear to be open-source, so the corresponding code is unavailable, and the paper itself does not provide reference experimental results on SciCoQA. Therefore, in the revised manuscript we will add a discussion of the relationship between ReAgent and VERITAS and clarify their positioning differences, and explicitly state the above reproducibility limitations; if the VERITAS authors can provide the code or reference results on SciCoQA, we would be happy to add a quantitative comparison. Thank you for this suggestion, which helps test generalization.

W1:Contract and Binding Ablation

We have supplemented component-level ablations and, on the same 10-repository subset, controlled computational conditions by using the same static goals and context and reinitializing the dynamic state, so as to isolate the gains from claim contracts, semantic binding, and iterative coordination; strict equal-token/equal-time matched compute budgets are still affected by uncontrollable execution, and we will clearly state this boundary in the revised manuscript. The minor 1.04% accuracy gain in Table 3 does not contradict the much larger improvement over Codex in Table 4: the former is the marginal accuracy gain of the static-plus-dynamic ablation within ReAgent, while the latter is the end-to-end comparison between the full ReAgent and Codex; the two differ in comparison object, denominator, and the factors isolated.

Specifically, on the same 10-repository subset, the full Contract+Binding setting achieves 7/10, no Binding achieves 5/10, no Contract achieves 6/10, and neither achieves 5/10. The alternative pipelines still output static judgments normally, so these scores are not mechanical failures caused by missing fields. The dynamic-component ablation uses the same static goals and context for each of these 10 repositories and reinitializes the dynamic state; the latest valid terminal states are 7/10 for the full pipeline, 4/10 without Review, and 4/10 without Evidence. We also tested the system without Audit, which reaches 6/10, but our analysis found that the Agent guessed the final labels without sufficient evidence; although this has high agreement with the ground truth, it contradicts our goal of trustworthy reproduction verification, so we do not regard it as a valid gain.

tatic-component setting (same 10-repository subset)Score
Full Contract + Binding7/10
No Binding5/10
No Contract6/10
Neither5/10
Dynamic-component setting (same 10 repositories)Latest valid terminal score
Full pipeline7/10
Without Review4/10
Without Evidence4/10
Without Audit6/10

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.