- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Compositional Coordination Confidence (CCC), an anytime-valid monitor for multi-agent LLM trajectories. An LLM judge assigns a rubric score to each turn, which is converted into a role-conditional evidence factor. Products and prespecified mixtures of these factors form test supermartingales, enabling a bound on the probability of raising an alarm before any coordination fault occurs. The paper evaluates forward and restarted-product monitors on AFTraj and Who&When and audits whether the conditional-moment assumptions required by the guarantee hold.
Critical strengths
-
The paper addresses an important operational problem: controlling false alarms over an entire multi-turn execution rather than independently at each turn.
-
The theoretical statements are generally precise, and the paper explicitly distinguishes the semantic no-fault condition from the statistical condition actually required by the theorem.
-
The authors are unusually transparent about assumption violations, weak detection results, post-registration changes, and judge sensitivity.
-
The factor-level construction is modular: once valid factors are available, several prespecified restarted, tempered, or mixed processes inherit anytime validity without separate threshold fitting.
Critical weaknesses
-
The assumption has limitation: the forward monitor consequently reaches a 15.5% false-alarm rate at a nominal 5%. Thus, the theorem does not certify the math deployment. Also, author thinks the same roles of agent will have same scoring rules, but this may not true
-
In Table2, coding's result is 0/37, Who&When is 0/126, agentic is 2/68, math is 4/58, the method can not detect errors. after reducing threshold, the coding improve but the author said the theorem may not support it. So the method may not meaningful. Also, this paper use data in Agentforesight but did not compare with them.
-
In Who&When, all trajectories in this bench failed, the author use the prefix of first note as safe data, this is not rigorous.
-
The eval data is small which only contains a small number of tasks
-
The dependence of judge is high, it has bad generalization.
-
The presentation is poor, in the main paper and appendix, there are too many different threshold and setting which is hard to understand.
Questions for authors
See weaknesses