- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ProtoGuard, a framework for coordinating LLM agents through formally validated interaction protocols. Each agent is assigned a local contract, against which its messages are checked before delivery. Protocol-aware scheduling invokes agents that are currently eligible to act, helping prevent premature decisions, missing notifications, and circular waiting.
The main contribution is the integration of static protocol validation with runtime enforcement. Multiparty session types provide the foundation for validating global protocols and deriving local contracts, which govern agent prompting, message verification, and scheduling. This makes protocols enforceable execution constraints rather than instructions that agents are expected to follow.
The authors evaluate four tasks using four LLMs across five configurations. The complete framework increases the macro-averaged completion rate from 4.1% to 98.9%, with no delivered protocol violations in the evaluated runs. The ablation results indicate that enforcement accounts for the primary reliability improvements, while scheduling reduces token consumption and execution time.
Critical strengths
The experimental design clearly distinguishes the effects of individual components across four tasks and four models. The ablations demonstrate that protocol instructions alone are insufficient. The authors also report that local contracts alone underperform global protocol context, providing a balanced assessment of the components.
Critical weaknesses
The experiments provide informative ablations but lack comparisons with established workflow orchestration or agent-governance methods. They demonstrate improvements over role-specific instructions alone and simpler protocol-based configurations, but do not establish ProtoGuard’s additional benefits over existing alternatives. Given the very low success rate of C1, a stronger external baseline would strengthen the empirical evidence.
Questions for authors
- Could the authors clarify the prompting and scheduling procedures used in C1, particularly whether the agent instructions include the coordination rules required for each task? This clarification would help explain the baseline’s low success rate.
2 comments
Thank you for the detailed response and for conducting the additional experiments. The inclusion of the new external baselines (C1-FSM, C1-MAF, and C1-LangGraph) along with the clarification on C1’s prompting, scheduling, and termination procedures thoroughly addresses my previous concerns.
Given that my questions have been resolved, I am happy to maintain my original rating of 3 (Weak Accept).
Response to Reviewer cTKR
We thank the reviewer for the constructive assessment and for recognising PROTOGUARD's static-to-runtime design and the value of our ablation studies. To address the reviewer's concerns, we conducted additional experiments and provide further clarifications below.
W1: Comparison with established workflow systems
We agree that comparisons beyond the original C1–C5 configurations are important. As detailed in our responses to Reviewer RC7s (W1/Q2) and Reviewer hWvs (W1/Q1), we added three baselines without validated workflows: C1-FSM, an LLM-generated Python state machine with deterministic control; C1-MAF (Microsoft Agent Framework); and C1-LangGraph. The latter two use LLM orchestrators. Each configuration ran 720 trials across four use cases and four models, including conditional branches and loops.
The new baselines improve on C1 but remain substantially below PROTOGUARD.
These comparisons provide evidence for the value of validated coordination over LLM-generated workflows or orchestration alone. They do not establish superiority over an equivalently specified deterministic workflow. PROTOGUARD's distinctive contribution is validating the global protocol before execution and automatically deriving compatible role-local contracts and runtime enforcement and scheduling controls.
Q1: C1 prompting and scheduling procedures
C1 included meaningful task-specific coordination instructions, rather than relying on role descriptions alone. Agents received task-specific instructions, descriptions of all team roles, role-specific skills, a terminal condition, and a required output format. The coordination information varied across benchmarks:
Thus, C1 includes meaningful coordination guidance, but does not provide a single validated global protocol or compatible projected local contracts. Its low completion rate should not be attributed to an absence of coordination instructions.
Scheduling and termination. C1 uses fixed round-robin polling, providing each selected agent with the shared interaction history. WAIT responses produce no delivered message. Attempts terminate after more than 2|R| consecutive non-progress polls (|R| = number of roles), upon reaching the terminal condition, or when the step budget is exhausted. Step budgets are 24 (Finance and AgenticPay), 44 (React 18), and 48 (SDLC). Finance and AgenticPay permit up to three attempts per trial, and React 18 and SDLC one. The new baselines do not rely on this polling: in C1-FSM the generated controller decides which roles may act, and in C1-MAF and C1-LangGraph the orchestrator chooses the next role, under the same step budgets.
C1–C4 use round-robin scheduling, whereas C5 selects roles with enabled sends using projected EFSM states. Within each benchmark, all configurations share the same step budgets. Their differences concern protocol information, enforcement, and scheduling.
The AgenticPay baseline illustrates the consequence of incompatible local instructions: 98/120 C1 trials were catastrophic, and in 93 of them an attempt froze with no delivered message. These results demonstrate the limitations of the evaluated skills-only configuration, not of all manually specified workflows.
We will incorporate the new external baselines and clarify C1's prompting, scheduling, and termination procedures in the revised manuscript. We will also provide the saved C1 system prompts for all 23 roles, their skills, and execution configurations in the supplementary material.