- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies the say–do gap of LLM teammates in MOBA games — what the agent says diverges from what it does. HADP is a three-stage hierarchy: Understanding grounds player requests and game state into a structured Mission; the Response Generator jointly produces action and dialogue from that Mission and an RL policy's tactical proposal; Adaptive Review screens candidates against short- and long-term memory before execution. In closed-loop Honor of Kings evaluation, HADP scores SC 91.5%, EC 86.3%, and IC 76.5%, vs. 81.3% / 67.5% / 51.5% for a budget-matched flat baseline.
Critical strengths
S1. Problem formulation — a useful diagnostic taxonomy. Decomposing the say–do gap into SC (utterance–plan), EC (plan–execution), and IC (commitments across turns) is clean and reusable, separating types of inconsistency from persona consistency and human-likeness. The taxonomy itself is a strength — the paper's most reusable contribution; W2/W4 concern its validation, not its definition.
S2. Architecture matched to the problem, corroborated by ablations. Each stage targets one failure source, and the ablations dissociate as predicted: removing Understanding hurts SC most (−18.0 pts), removing Review hurts IC most (−15.3 pts). The budget-matched baseline, under the same backbone, data, and scenarios, makes the comparison interpretable.
S3. Multi-level, practically relevant evidence. Closed-loop evaluation, offline probes, command-quality ratings, and production deployment complement each other; the deployment evidence in particular supports the practical motivation.
Critical weaknesses
W1. Mechanism not isolated. Integration is the claimed contribution, yet its parts are not separated. Review is pre-execution only, so nothing monitors EC once a plan runs; IC rests on memory-based LLM review, with no explicit pending-commitment state or trigger tracking. The RL proposal and joint action–dialogue generation are never ablated independently. Worse, conditioning both channels on one Mission allows common-mode failure: a wrong Mission can be executed consistently and still score high on SC.
W2. Consistency is not shown to imply good decisions. The gap is validation, not definition: consistency, defined independently of task success, is never reconnected to it. The Mission (player intent) and the RL proposal (tactical judgement) can conflict, yet arbitration among cooperation, refusal, and delayed cooperation is left to the SFT generator, with no criteria, conflict frequency, or outcomes reported. A compliant but tactically poor agent could therefore score highly. No closed-loop task success is given to rule this out, nor the diagnostics that would locate failures: Review pass/fail, suppression and regeneration rates, or SC/EC/IC failures split by successful vs. failed trajectories.
W3. Latency uncharacterised, despite being a hard constraint. A decision is only useful within the validity window of its input state; delay itself creates say–do inconsistency. HADP chains at least three sequential LLM calls, plus RL inference and retrieval, with regeneration adding tail latency — yet only an approximate P99 for the deployed command-channel model is reported: no end-to-end or per-module latency, no wall-clock comparison with the baseline (budget matching equalises compute, not time), and no latency vs. SC/EC/IC analysis, despite the paper noting replies going stale in team fights.
W4. Evidence is not auditable or generalisable. The closed-loop study uses a proprietary game, data, and stack over a narrow templated design (5 roles × 8 commands × 10 trials), with scenario construction unspecified and a single game/patch throughout. The supporting studies do not close this gap: the offline probe measures agreement with one logged action, not decision quality; production results omit cohort sizes, absolute rates, and uncertainty; Appendix G's key quantities — repair rate r, damage rate f — are never estimated.
Questions for authors
Q1. When Mission and RL proposal conflict, what criteria govern cooperation / refusal / delayed cooperation? How often do conflicts occur, which side prevails, and with what task outcomes?
Q2. How are SC/EC/IC adjudicated, and what is the inter-annotator agreement? Please report closed-loop task success alongside consistency.
Q3. What are the end-to-end and per-module latencies (including regeneration), how do they compare with the baseline's wall-clock time and the state-validity window, and how do SC/EC/IC degrade with latency?
Q4. How were scenarios constructed and sampled, and what protocol, data, or code can be released? Please also give cohort sizes, absolute rates, and uncertainty for the production results.
Q5. Is there an explicit pending-commitment representation, and what monitors temporal triggers and interrupted instructions? What are Review's pass/fail, suppression, and regeneration rates, and can r and f in Appendix G be estimated?