- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces PNYX, a coordination framework for multi-agent vision-language-action systems. It combines a structured communication protocol with task-scoped leader election. The protocol distinguishes information sharing from action negotiation, supports explicit proposal acceptance and rejection, tracks information freshness, bounds retransmissions, and terminates coordination exchanges at a deadline. The leadership mechanism uses deterministic candidate ranking, voting, heartbeats, and replacement procedures. The authors evaluate PNYX on six cooperative-driving scenarios in CARLA 0.9.14, using two VLM planners and a controlled, frozen execution interface. Each architecture–condition cell contains 240 episodes pooled across the two planners. The paper reports a nominal joint success rate of 58.3% for PNYX, compared with 51.2% for the centralized and free-form decentralized baselines, and a route completion rate of 80.7%, compared with 75.9% for the strongest baseline. Under severe degradation, PNYX achieves 55.4% joint success.
Critical strengths
Explicit coordination semantics. The paper distinguishes message receipt, action proposals, participant acceptance, and commitment. This is a meaningful design choice because exchanging intentions does not necessarily establish a mutually accepted joint action. The protocol provides a clear framework for representing these states. The paper evaluates six driving scenarios under nominal, mild, and severe degradation profiles and includes ablations of state representation, proposal rejection, and leader selection.
Critical weaknesses
-
Limited methodological novelty and unclear VLA-specific contribution. The proposed method largely combines established mechanisms from multi-agent communication and distributed systems, including FIPA-ACL-style message typing, truncated ARQ, age-of-information tracking, and Raft-inspired leader election and failure detection. The paper does not sufficiently identify what fundamentally new coordination principle or capability emerges from this combination, or clearly distinguish PNYX from existing LLM-based multi-agent planning and coordination frameworks. More importantly, the proposed mechanisms operate primarily at the level of language-mediated planning and system orchestration, rather than exploiting characteristics specific to VLA models. The evaluation uses VLM planners with a controlled, frozen execution interface, without demonstrating that the proposed design addresses challenges unique to VLA representations or action generation. Consequently, it remains unclear whether this work constitutes a substantive advance in collaborative VLA research, rather than an application of general multi-agent coordination techniques to a driving simulator.
-
The experimental evaluation lacks a coherent end-to-end demonstration of how the proposed method works. The paper reports aggregate success rates, several component ablations, and a textual failure trace, but does not provide a sufficiently clear, end-to-end case study connecting the initial scene and observations to information exchange, leader selection, proposal generation, participant acceptance, action execution, and the final outcome. Figure 7 compares free-form decentralized planning with centralized planning, rather than showing how PNYX resolves the same scenario through its proposed protocol. Moreover, the trace contains repeated discrepancies between agents' natural-language statements and the listed executor commands, and the authors explicitly acknowledge that the collision cannot be attributed solely to coordination failure. As a result, the qualitative evidence does not establish the causal mechanism behind the observed failures or demonstrate how PNYX's individual components change the decision process. The quantitative results are also presented mainly as aggregate comparisons across scenarios and degradation profiles, without a sufficiently clear connection to representative execution traces. This leaves the experimental narrative fragmented: the reader can observe reported performance differences, but cannot adequately understand when, why, and through what sequence of interactions the proposed method succeeds or fails.
-
Insufficient experimental specification and runtime characterization. The paper does not provide enough detail to independently reconstruct the evaluation, including the concrete CARLA maps and scene configurations, task prompts, planner inputs, obstacle and occlusion geometry, and complete task-level success and termination conditions. The reported distinction between “risky” and randomized trials is not sufficiently defined. In addition, although the planners are stated to operate at 2 Hz, the paper does not report the hardware configuration, per-call inference latency, end-to-end negotiation latency, communication rounds, and message counts per decision, retransmission statistics, or the frequency and cost of leader replacement. The reported route-completion time is not a substitute for coordination latency. Finally, communication and sensing factors are varied jointly across degradation profiles, preventing the reported robustness gains from being attributed specifically to communication effects. These omissions substantially limit reproducibility and make it difficult to assess whether the proposed protocol can meet the real-time constraints of the target application or whether its additional coordination overhead is justified.
Questions for authors
-
Task specification and inputs: For the overtaking scenario in Figure 7, is the overtaking objective specified by the benchmark or task prompt, or is it inferred autonomously by the planner? Please provide the actual task prompts, planner inputs, scenario initialization, and the division of responsibility between the scenario generator, planner, coordination layer, and executor.
-
Scene reproducibility: Please specify the CARLA maps, initial vehicle states, road geometry, obstacle placement, occlusion construction, and termination rules for the six scenarios(BEV View or actually car input). How are the reported “risky trials” generated, and how do they differ from randomized trials? Are scene-generation scripts and configuration files available?
-
Execution and message handling: How are planner outputs parsed and validated against the protocol? If a generated message is malformed or inconsistent with the current protocol state, is the model invoked again, is the message rejected, or is it repaired programmatically? How are these cases counted in planner calls, token usage, and coordination deadlines?
-
Runtime and communication overhead: The paper reports planners operating at 2 Hz but does not provide per-call inference latency or end-to-end negotiation latency. Please report the hardware configuration, inference latency distributions, communication rounds and messages per negotiation, retransmission counts, timeout rates, and the time from task initiation to committed action.
-
Leader election and replacement: How often does leader replacement occur in the main evaluation? What are the measured election and handoff latencies, and how much of the performance gain can be attributed to dynamic replacement rather than the structured proposal protocol? The severe-degradation election ablation performs worse with a ranked election than with a random leader, so additional evidence is needed to clarify the practical benefit of this mechanism.
-
Controlled degradation: Please provide the numerical settings for bandwidth, communication delay, packet loss, communication range, and sensing noise in the nominal, mild, and severe profiles, especially the sensing noise, as I do not see any ego view in this article. Since these factors are varied jointly, can their individual effects be isolated through controlled ablations?
-
Baselines and task difficulty: How does PNYX compare with a simple rule-based coordination strategy and existing general-purpose LLM multi-agent coordination approaches under the same initial conditions? Such comparisons would help determine whether the observed gains require the proposed framework and whether the tasks adequately test its claimed VLA-specific contribution. Also, the definition of "success" should be mentioned, as well as other detailed metrics.
5 comments
Part 2 of 4
The experimental evaluation lacks a coherent end-to-end demonstration of how the proposed method works. We have added an end-to-end case study in the revised appendix that follows one Pnyx episode on a seed where the independent, centralized, and free-form decentralized baselines all fail. It connects the initial scene and each vehicle's camera views to the state exchange, leader election, the leader's proposal, the participant's acceptance and commit, the instructions and executed commands, and the outcome. This shows the sequence of interactions through which Pnyx resolves the conflict the baselines do not, and gives a concrete trace behind the aggregate gains in that scenario. The quoted statements are messages between free-form planners, which write their peer messages and their own vehicle's instruction as separate outputs that need not agree. The remaining mismatches come from the executor's one-period lag. In steady state, every architecture's instructions are executed correctly more than 97% of the time. The disagreement between what a free-form planner tells its peer and what it instructs its own vehicle is itself the failure Pnyx's commit step prevents, where an agent's instruction follows the plan its peers approved.
Task specification and inputs. The overtaking objective is specified by the benchmark's task prompt: for example, "Overtake the stopped vehicle by passing on the left, when it is safe to do so." The planner decides only when and how to carry it out. The scenario generator (CARLA) defines the map, spawn poses, routes, hazards, task prompts, and success and termination checks. Each planner (VLM) receives its vehicle's task prompt, a text description of the sensed scene, and three camera images, and outputs a natural-language instruction. In Pnyx it also produces the content of queries, proposals, and approve/reject decisions. The coordination layer is deterministic code that handles message typing, acknowledgment and retransmission, state freshness, leader election, and commitment. The executor converts the latest instruction into atomic driving actions along the route. The revised appendix now gives every agent's task prompt in all six scenarios, the full planner prompt template, an example scene description, and the camera views each planner receives.
Scene reproducibility. Our scenarios build on the open-source TalkingVehicles benchmark, whose scene-generation scripts and configuration files are publicly available. We use its six scenarios, maps, routes, and task prompts, with three changes: the stopped truck in the overtake scenarios is placed 25 m ahead of the ego vehicle, the safe overtake trial includes both focal vehicles, and the highway-exit time limit is extended to 35 s. Highway exit and highway merge use Town06. Both overtake scenarios use Town01, and the unprotected left turn and red light use Town03. Each scenario has two configurations from TalkingVehicles, a risky one and a random counterpart. Every evaluation cell contains 10 trials of each per scenario on fixed seeds, for 120 episodes per planner. Risky trials place a conflict that the ego vehicle cannot resolve from its own observations, so an uncoordinated vehicle is likely to collide:
- in the highway exit and merge, the two vehicles start exactly side by side, so neither has priority and both must agree who goes first;
- in the overtakes, an oncoming car approaches in the passing lane, hidden from the ego vehicle by the stopped truck;
- at the left turn, an oncoming car is hidden by a stopped truck;
- at the red light, a car runs the red light at 29–31 km/h from a direction hidden by a queue of trucks.
The random trials keep the same map, routes, tasks, and occluders but remove the forced conflict. In the highway scenarios, the second vehicle and its follower are shifted 2–8 m forward or backward at random, so one vehicle has a natural lead. In the overtake, left-turn, and red-light scenarios, the hidden hazard vehicle is not spawned (TalkingVehicles' "safe" configurations). In both configurations, vehicle models and hazard speeds are also sampled per seed. An episode ends when every focal vehicle has completed its route or collided, or at the time limit (25 s; 35 s for highway exit). The simulation runs at 20 Hz. We did not record a bird's-eye view. Exact spawn poses, routes, and occluder placement follow the TalkingVehicles configuration files. We include the relevant details in the revised manuscript to improve reproducibility of our results
Part 3 of 4
Execution and message handling. We answer each individual question below and provide detailed explanations in the revised appendix.
- Parsing. every planner call uses strict JSON-schema constrained decoding. The model fills only the fields its current protocol role requires; protocol messages, checksums, and digests are built by deterministic code, so the model never writes a raw protocol message.
- Failed calls. output that does not parse is discarded, with no retry or repair. The previous instruction stays in force, and the call still counts toward planner calls, tokens, and its planner period.
- Stale outputs. output for an exchange that has since been superseded is dropped and counted as stale.
- Channel damage. damaged messages are handled by the protocol. A lost header means no acknowledgment, so the sender retransmits; a digest mismatch triggers a fallback.
- Frequency. 6 of about 28k planner calls in the main evaluation failed to parse.
Runtime and communication overhead. Experiments ran on 2× AMD EPYC 9254 CPUs with 8× NVIDIA L40S (48 GB) GPUs. Planners were served with vLLM on three GPUs, with several episodes running in parallel. This information has been added to section 6.
Mean planner latency per call (s):
| Independent | Centralized | Decentralized | Pnyx-Proto | Pnyx |
|---|---|---|---|---|
| 1.7 | 2.7 | 4.6 | 4.0 | 3.6 |
Pnyx negotiation statistics by profile:
| Metric | Nominal | Mild | Severe |
|---|---|---|---|
| Negotiations completed | 86% | 82% | 50% |
| Negotiation latency (median) | 1.25 s | 1.65 s | 5.15 s |
| Messages per negotiation, excluding ACKs (median) | 6 | 6 | 8 |
| Retransmitted share of messages | 0.5% | 3.3% | 18.1% |
| Time to first committed plan (median) | 2.0 s | 2.9 s | 9.8 s |
Model calls are locked to simulated time, meaning each result takes effect exactly one planner period after the call is made regardless of how long the call took in wall-clock time. Wall-clock latency therefore does not affect outcomes, as it is largely dependent on hardware capability and usage. Instead, we report relevant hardware-agnostic metrics, such as the number of planner calls.
Leader election and replacement. Leader replacement occurs in 5.2% of Pnyx episodes in the main evaluation. Elections take 0.3 s on average under the nominal and mild profiles. Under severe they take 3.2 s, because candidacy messages also carry each vehicle's state. The benefit of the structured proposal protocol alone shows in Pnyx-Proto, which has no leader. Adding leadership raises joint success over Pnyx-Proto at every profile (+3.7, +6.3, and +2.1 points). Within Pnyx, this benefit comes mostly from replacement. On the seeds where a handoff occurred, Pnyx succeeds in 51.1% of episodes, against 34.0% on the same seeds with a fixed leader (, paired ). On seeds without a handoff, the two perform the same (58.9% vs. 58.0%). Replacement therefore matters in the minority of episodes where the initial leader stops being effective. These situations are hard: there, the fixed leader's success falls to 34%, against 56% overall. Under severe degradation, the difference between ranked election and the random leader (53.9% vs. 56.1%) is not statistically significant ().
Part 1 of 4
Thank you for your thoughtful comments and suggestions, which have helped us significantly improve our manuscript. Below, we explain Pnyx's contribution as the composition of established mechanisms from multi-agent systems, networking, and distributed consensus, adapted for physically coupled agents acting under deadlines. We have reframed the paper away from VLA-specific claims. To address the evaluation concerns, we have added an end-to-end case study, full task prompts and scenario details, the degradation parameters, output-validation and runtime statistics, and an analysis of leader replacement showing its benefit on the episodes where it occurs. Please see the revised PDF, with changes in blue. We also directly respond to your specific points below.
Limited methodological novelty and unclear VLA-specific contribution. We build Pnyx from established mechanisms in various fields deliberately. FIPA-ACL act typing comes from agent communication languages in multi-agent systems research. Truncated ARQ comes from wireless link-layer error control. Age of information comes from status-update networking, where it was introduced for vehicular networks. Raft-style terms and leader election come from distributed consensus. None were explicitly designed for LLM agents or collaborative route planning, but each gives a well-defined solution to the problem of making the purpose of a message explicit, recovering from message loss, tracking how old shared information is, and keeping a single leader when agents fail. Reusing them gives Pnyx semantics that are already well understood instead of new mechanisms that would need their own validation. The contribution is how they are composed around the weaknesses of collaborative route planning with general LLM multi-agent frameworks and large central models, and how each one changes once agents are physically coupled and act under deadlines:
On the VLA question, please see the general response. We agree that Pnyx does not exploit VLA-specific representations, and the revised paper no longer presents it as a VLA-specific contribution. The problem it addresses sits above the executor. A language-conditioned executor acts on one instruction per decision period, so when two planners disagree, their executors carry out incompatible actions. Pnyx depends only on that interface and treats the executor as a black box, so the same layer applies to any executor that follows language instructions.