- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors design a claim-to-control compilation path framework to verify whether machine learning outputs satisfy the claims. Their method identifies the implicit conditions for a claim to hold. Then they convert these into rules that serve as constraints. Experiments demonstrate that AI-generated results can pass evaluation.
Critical strengths
S1: The authors raise an interesting problem. They focus on an issue of practical significance: the results of agents may not fully support the claims. S2: The framework and workflow are simple and clear, allowing for verification through process evidence following the results generated by AI agents. S3: The experiments verify the existence of the latent validity issue.
Critical weaknesses
W1: The article addresses a very interesting issue, but the method appears simple. Specifically, mechanisms such as leakage prevention, provenance, and runtime enforcement are system-level implementations. To highlight the novelty, the authors should clarify the specific difficulties and challenges in addressing this problem. W2: The authors focus on a rule-deriving method. Given a research conclusion, the method can deduce the conditions that must be met for that conclusion to hold true. The method then transforms these conditions into rules to constrain the agents. The authors should explain how the correctness of these rules can be verified. W3: The method for deriving rules is the main contribution of the paper. However, the classification scheme used to find conditions is neither mutually exclusive nor exhaustive. Stating this in Appendix E does not resolve the issue. If the derived conditions are incomplete, the subsequent processing operations lack a valid basis. W4: In the main text, the authors propose a mechanism for handling rule violations, but the experiments in Appendix do not demonstrate that the guard can stop actual violations. The authors should explain why they did not design an experiment in which the AI agents could violate the rules. W5: The experiment results are inconsistent with the design. In Section 2, the authors state that a case can pass only when the process is fully documented, and otherwise it is classified as unresolved. However, Codex’s logs capture every step taken by the agents, while the logs for Claude and Gemini lack such records. As a result, the checker marks the Claude and Gemini runs as passes because it finds no violations.
Questions for authors
Q1: Could the authors explain the difficulties and challenges involved in solving this problem, to clarify the novelty of their method? Q2: How can the correctness of the rules derived from this method be verified? Q3: Given that the categories used to define conditions are neither mutually exclusive nor exhaustive, how can authors ensure that the derived rules are complete? Q4: Why did the authors not design experiments where AI agents might violate the rules, rather than conducting experiments that fail to prove whether the guard can stop violations? Q5: Why does Figure 2 regard the Claude and Gemini runs as passes, given that Section 2 states that a case can pass only when the process is fully documented?