- Rating
- 2
- Confidence
- 3(+1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Paper introduces ReAgent, an automated auditor to check if documents and code produced by agents are consistent between each other. ReAgent produces structured representations of the paper claims. There are two modes for analysis: static and dynamic, and bother are LLM-agent-driven. In static, agents summarize paper and code into some structured format and "judge" the semantic alignment. In dynamic, agents review and execute the code so that's it is reproducible. Authors manually create an evaluation benchmark of 96 tasks of (paper, code repo) and find that ReAgent identifies inconsistencies.
Critical strengths
- Authors target an important problem of faithfulness in agent outputs, that both research text in natural language and code should be consistent with each other. In fact, this is a problem beyond agent era where ideally we'd like human scientists to also do this. Paper proposes a workflow for stress-testing agent outputs.
- The idea to standardize summaries of research papers and code into structured metadata is good, though limited to the tasks presented in the paper.
Critical weaknesses
- Framework orchestrates several agents in Figure 2, giving them tasks that are somewhat vaguely defined and redundant e.g. "planning" "execution". Figure 2 seems to be quite a lot and yet gives loose information on what the method looks like. Paper points to Appendix A.1 with structured output schema for the agents, and it'd be better to describe them in the main text while cutting down on high-level overview.
- I'd argue that ReAgent is not so generalizable to science domains as the paper claims. The prompts and schema outputs for static and dynamic analysis are geared for ML-heavy tasks, and so are the papers selected from AutoSOTA as a benchmark. Fundamentally, static and dynamic analysis is often not doable in other fields, where lab access is crucial. In a sense, ML papers have the extraordinary benefit of being reproducible, on machines anyone can reasonably get access it.
- Authors can describe the manual annotation process a bit more, and how they plan on extending it. It seems quite a bit of manual effort going through all these papers and code. Oh, I see that Claude Code did this manual effort... well this is a chicken-and-egg problem: how can you trust Claude Code to read and find holes to add to benchmark tasks, when the benchmark is supposed to be used to evaluate agents in reading/finding holes?
- This seems problematic, because it asks the static analysis agents to just say yes or no: "The static verification task is formulated as a binary classification problem, where the system predicts whether a repository contains static claim-code inconsistencies."
- Writing is sloppy in results section, and the space can be better used. I'm not sure how to read Table 3 and 4 (static appears in both row and column, and why don't the last 2 rows of the tables match in numbers?). Paper should go through more work and experiments.
Questions for authors
- Would you use ReAgent yourself, e.g. as a tool, rather than codex or claude code to analyze and reproduce paper results? What is the value-add of defining these prompts and structured schema, beyond better numbers on the curated benchmark?
- Currently in my own workflow and my fellow scientists, I ask agents to implement papers and verify against the released source code/run it. So why give this workflow---that all scientists use---a name ReAgent? Due to this, I don't particularly yet see the novelty in the manuscript.
3 comments
W4: Static Verification Formulation
A:We agree that the original sentence was problematic and have revised it; the static verification task is not formulated as asking the static analysis agents to just say yes or no. The revised sentence now states that static verification identifies and assigns specific labels to claim-code inconsistencies, while a binary truth value is used only for subsequent state transitions and experimental evaluation.
Specifically, we have replaced the original “The static verification task is formulated as a binary classification problem...” with: “The static verification stage identifies and assigns specific labels to claim-code inconsistencies. However, for subsequent state transitions and experimental evaluation, these results are represented using a binary truth value (i.e., whether an inconsistency is detected).” We did not formulate the static verification task as a pure binary classification problem at the agent-output level. The binary representation is used because the compared baselines do not have our eight-category risk definition; retaining the specific labels in the static audit output while evaluating on a binary truth value ensures comparability. We will also clarify in the revision where the specific labels are stored/reported and that the binary value is an evaluation and state-transition representation, not the only output of the static analysis agents. This should address the concern that the formulation asks the static analysis agents to just say yes or no.
W5: Results Writing and Tables
Regarding Tables 3 and 4, the 1.04 percentage points you pointed out (82.29% → 83.33% in Table 3) is a classification metric for stage combinations within the same system; the 35.42% → 70.83% in Table 4 is a repository-level aggregate metric across different systems for “both static and dynamic correct,” used to evaluate agreement with human annotations. The two evaluate different events, so the values are not expected to be consistent.
response to W1 and W2
Thank you for agreeing with me, but I'd have liked to see an update to the Figure or a concrete plan of how you'd update it. Your plan currently is "we'll do it", but with that I don't know how to recommend for acceptance.
response to W3
Makes sense that Claude Code gives human annotators a headstart. That is a natural approach, yet human verification remains a bottleneck. I understand this bottleneck is a more general problem in the field right now. So I'm satisfied with the W3 response.
your update for W4: "The static verification stage identifies and assigns specific labels to claim-code inconsistencies. However, for subsequent state transitions and experimental evaluation, these results are represented using a binary truth value (i.e., whether an inconsistency is detected)"
The writing wasn't an issue for me, but the approach to collapse static to a binary classification task was. I think output for static should be a bit more rich, at least with a rubric for instance. This remains a weakness in the approach.
response for W5
The updated Table 3 is completely different, and you didn't explain why you replaced it. I didn't get a response for why the row and column both have "static". Still not sure how to read Table 4.
My questions are left unanswered as well. In light of lingering concerns, I'm maintaining my score and increasing my confidence. The writing in method and experiments really does need a lot of work.
W1: Figure 2 and Framework Description
We thank the reviewer for this constructive comment. We agree that Figure 2 currently contains a large amount of high-level information while giving only loose details about the individual agents, and that some agent tasks are described in a somewhat vague and redundant manner (e.g., “planning” and “execution”). In the updated version of the paper, we will revise Figure 2 to better clarify the division of labor among agents and remove redundant or uninformative task labels. We will also describe the agents more concretely in the main text, including their responsibilities, inputs/outputs, and the structured output schema currently provided in Appendix A.1, while cutting down the high-level overview. We believe these changes will make the method substantially clearer and more self-contained. We will make these revisions in a subsequent version update.
W2: Generalizability Boundaries
A: We agree that ReAgent is not so generalizable to science domains as the paper may have implied, and we will scope our claims accordingly. We also agree that static and dynamic analysis is often not doable in other fields, where lab access is crucial. Our work targets academic-integrity auditing of autonomous research agent systems, which typically use code for automated research; it does not claim generalizability to lab-based science. We further agree that the prompts and schema outputs for static and dynamic analysis are geared for ML-heavy tasks, and that the papers selected from AutoSOTA as a benchmark are similarly ML-heavy. ML papers do have the extraordinary benefit of being reproducible on machines that anyone can reasonably get access to, and this is precisely the setting in which our static/dynamic auditing pipeline is applicable. We will revise the abstract and introduction to avoid broad generality claims, explicitly scope ReAgent to autonomous research agent systems with executable code and ML-heavy artifacts, and list extension to other science domains—especially those requiring lab access and domain-specific schemas—as future work. We will also clarify that the AutoSOTA-based evaluation is a scoped benchmark rather than evidence of broad cross-domain generality
W3: Annotation Process and Credibility
We will describe the manual annotation process more clearly in the revised manuscript and state that Claude Code does not produce the final benchmark labels; all final labels are reviewed and adjudicated by the authors. Regarding the chicken-and-egg issue, we do not treat Claude Code as ground truth, nor do we rely on it alone to read and find holes; it is used only to generate candidate issues, and all issues claimed by Claude are reviewed again by the authors before entering the benchmark.
The specific process is as follows: for static annotation, 3 different Claude models first independently review the same batch of repositories to reduce missed detections by a single model; their claimed problematic candidates are merged, yielding about 71 flagged repositories, covering most repositories; only a small number of repositories require the authors to carefully review them line by line; each Claude claim is still reviewed by the authors in light of the code context, checking whether the risk falls in actually used code, whether parameters are hard-coded by default, whether it belongs to deprecated code, etc., and the final binary labels are adjudicated by the authors. Therefore, the benchmark labels are not the output of Claude Code, but labels after author adjudication; Claude Code only undertakes candidate generation, reducing manual effort, and does not replace ground truth.
We acknowledge that this may still fail to detect all static issues 100%, and we do not claim that Claude Code’s recall is perfect; however, independent review by three models plus author review at least provides some reference value and reduces the risk of missed detection by a single model.