- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper considers a problem in multi-agent systems: there may be a difference between the true state of a relevant variable (an item is refundable) and compliance with a task requirement that the state of the variable be reported to an agent that is to take action based on the variable (send a replacement item.) The paper proposes a method for checking that task completion requirements (transmitting state information to another agent who is required to have the information before acting). At least I think this is what the paper is about; it is so poorly written it is hard to tell.
Critical strengths
- This might be an important problem, of how to ensure that information claims are properly shared between agents in a multi-agent setting.
Critical weaknesses
- Terribly written and formulated with seemingly pointless formalisms and clearly AI written.
- Experiments are never properly explained
- No discussion of the real world importance of this problem of any explanation that is easy to understand of what the method is other than making checks other approaches (are they used?) don't make.
Questions for authors
- What is the point of the paper? What is the motivation for why this is important or interesting to solve?
Response to Reviewer A5Hp
Thank you for your valuable suggestions and questions.
1. What is the problem, and why does it matter?
A policy agent may know that an item is refundable while the coordinator has not received its report. The refund claim can be true, yet violate a task rule requiring receipt of the report before the claim. The introduction uses this example to explain why the required evidence must reach the agent making the claim (Section 1; Figure 1).
2. Why use the formalization?
Two traces can contain the same fact and final claim, but only one delivers the required report in time. A checker that sees only the fact and claim cannot distinguish them. Section 2.2 states the information requirement: the representation must preserve differences that change whether a task rule is satisfied. This motivates retaining report-delivery order and tool–result links.
3. How does the method work, and what do the experiments measure?
ECoCheck builds a typed graph of interaction records, retrieves relevant records, extracts facts with source references, and encodes selected task requirements for SMT checking. The resulting diagnostics guide feedback (Section 3; Figure 2). Table 1 describes how each environment uses that feedback.
The evaluation uses separate measures for checking and task performance:
4. Writing and presentation.
The introduction motivates the problem with the refund example. Sections 4–5 identify the feedback treatments, define the metrics, and explain what the results measure. The section structure, six original figures, and core formulation are retained. Appendix D places the Werewolf outcome and scheduling tables on the same pages as their respective plots. The AI-use disclosure remains as submitted.