acceptodds
Withdrawn from ICLR 2027

Compositional Coordination Confidence: Anytime-Valid Monitoring for Multi-Agent LLM Trajectories

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-agent LLM systems can carry an erroneous intermediate step many turns before a final output is inspected. We study online monitoring under an operational requirement: bound, uniformly over time, the probability of halting a trajectory before any coordination fault occurs. Compositional Coordination Confidence (CCC) calibrates per-turn evidence rather than a decision threshold: an LLM judge's rubric score becomes a role-conditional factor with null mean at most one, so products and prespecified mixtures of restarted or tempered products are test supermartingales, and one factor calibration, fit on fault-free or pre-fault turns, yields anytime-valid false-alarm control for each prespecified member of the family without refitting. Calibrating at the factor makes the guarantee's premise, a history-conditional moment restriction, a per-turn quantity that can be tested directly. On AFTraj (math, coding, agentic) and on Who&When, the primary restarted mixture has observed false-alarm rates at or below 3.5% on all five evaluation subsets at nominal 5%, with restricted post-onset detection of 25.9% on Who&When HC and 6.9% and 2.9% on AFTraj math and agentic. The premise test does not reject on coding or agentic; on math it finds a conditional factor of 1.35 under one judge and 1.24 under a second, with cluster-bootstrap intervals excluding 1 in both cases, an excess the forward product exposes on 15.5% of outcome-safe-labelled traces while the primary statistic stays below nominal. An exploratory state-conditioned recalibration reduces that excess while held-out math detection falls from 4/58 to 0/58. CCC therefore provides a reusable anytime-valid construction together with an audit that shows, per deployment, when the theorem's certificate does not apply.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
anytime-valid inference, e-values, test supermartingales, sequential hypothesis testing, multi-agent LLM systems, online monitoring, LLM-as-a-judge, calibration, failure detection, conformal risk control

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_aQR4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Compositional Coordination Confidence (CCC), an anytime-valid monitor for multi-agent LLM trajectories. An LLM judge assigns a rubric score to each turn, which is converted into a role-conditional evidence factor. Products and prespecified mixtures of these factors form test supermartingales, enabling a bound on the probability of raising an alarm before any coordination fault occurs. The paper evaluates forward and restarted-product monitors on AFTraj and Who&When and audits whether the conditional-moment assumptions required by the guarantee hold.

Critical strengths
  • The paper addresses an important operational problem: controlling false alarms over an entire multi-turn execution rather than independently at each turn.

  • The theoretical statements are generally precise, and the paper explicitly distinguishes the semantic no-fault condition from the statistical condition actually required by the theorem.

  • The authors are unusually transparent about assumption violations, weak detection results, post-registration changes, and judge sensitivity.

  • The factor-level construction is modular: once valid factors are available, several prespecified restarted, tempered, or mixed processes inherit anytime validity without separate threshold fitting.

Critical weaknesses
  • The assumption has limitation: the forward monitor consequently reaches a 15.5% false-alarm rate at a nominal 5%. Thus, the theorem does not certify the math deployment. Also, author thinks the same roles of agent will have same scoring rules, but this may not true

  • In Table2, coding's result is 0/37, Who&When is 0/126, agentic is 2/68, math is 4/58, the method can not detect errors. after reducing threshold, the coding improve but the author said the theorem may not support it. So the method may not meaningful. Also, this paper use data in Agentforesight but did not compare with them.

  • In Who&When, all trajectories in this bench failed, the author use the prefix of first note as safe data, this is not rigorous.

  • The eval data is small which only contains a small number of tasks

  • The dependence of judge is high, it has bad generalization.

  • The presentation is poor, in the main paper and appendix, there are too many different threshold and setting which is hard to understand.

Questions for authors

See weaknesses

Reviewer_T49TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies online fault monitoring of multi-agent LLM trajectories. Compositional Coordination Confidence transforms a judge's discrete rubric scores into role-conditioned factors, then combines their products through fixed restart mixtures. Experiments on AFTraj and Who&When investigate detection, calibration failures, and attempted corrections. The most informative result is that apparently acceptable false-alarm behavior of the restart mixture coexists with conditional-factor excess on outcome-safe-labeled math traces under two judges.

Critical strengths
  • The conditional-factor audit provides a useful warning against validating a sequential monitor from its aggregate false-alarm rate alone.

  • The paper carefully separates established theory from empirical operating characteristics.

Critical weaknesses
  • The claimed PAC calibration is not supported by the provided threshold implementation. Table 4 describes a distribution-free PAC coverage bound, but the designated producer, kprime_pac_inh_stage5.py, selects the floor(alpha*n)-th largest development maximum without a calibration-confidence parameter or binomial-tail correction. It also uses the same development trajectories to fit the factor reference and calibrate this cutoff. The Who&When producer calibrates and evaluates the cutoff on the same pre-onset records. These procedures do not establish the stated high-probability guarantee. This does not invalidate the Ville theorems or the descriptive AFTraj test counts, but it matters for the central claim that a calibrated alternative recovers substantial detection on coding and agentic. A corrected, independently calibrated comparison or an explicitly empirical interpretation is needed.

  • The strongest diagnostic finding hasn’t been widely proven to apply in different situations. The two-judge result involves the same math progress paths, but their outcome-safe labels don’t confirm the process-level null hypothesis. The authors’ review of annotations can’t yet determine how much this mismatch affects the results.

Questions for authors
  • On math, is the 15.5% Wfwd elevation driven by a few traces with very large factors or by broad mild miscalibration? The cluster-bootstrap CI on the conditional mean doesn't distinguish these, and it matters for whether any targeted fix is plausible.

  • What exact PAC statement, confidence level, and order-statistic rule support the thresholds in Section 5.4 and Table 4?

  • Can the existing annotation audit identify which conditional-factor exceedances and alarms occur before any genuine process fault, rather than merely on outcome-safe-labeled traces?

Reviewer_CBSHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the online monitoring of processes involving multi-agent LLM systems. The objective is to develop a monitoring mechanism capable of establishing an upper bound, consistently over time, on the probability that an execution trajectory concludes before any coordination failure occurs.

Critical strengths

S1. The paper addresses an important problem with a well-posed operational target. S2. Correct and clearly stated theory (as far as I can tell).

Critical weaknesses

The methodological novelty is limited; most of the construction is standard. Lemmas 1–2, Proposition 1, and Theorem 1 are standard closure properties of test supermartingales combined with Ville's inequality, as Remark 8 itself acknowledges. The categorical factor wi/Srw_i/S_r is a fixed-payoff bet against a known null. The restart mixture is a fixed-weight version of mixing over change points (e-detectors; Shin et al., 2023), and the good-event device behind Theorem 2 is standard (Remark 15). Relative to E-valuator (Sadhuka et al., 2025), the new elements are calibration from null data alone (with no density ratio and no failure labels), conditioning on agent role, and the premise diagnostic. These additions are reasonable but incremental, and the evidence does not show that they buy anything in practice.

The baselines are thin, and the closest prior work is not compared. Table 4 compares CCC only against a per-role likelihood ratio (SLR) and against CCC with a PAC-fitted threshold, and none of the closest prior methods is run head-to-head. E-valuator, which the paper calls "closest to our setting," appears only as a cost figure ("The MOR saving against E-valuator is 13.0%", App. I.2), with no false-alarm or detection numbers. For Schirmer et al.'s monitors, the paper says "we implement their monitors on our scores via CRC" (§2.2), but no results for them appear anywhere. AgentForesight-7B is declared out of scope (App. U), even though its code is public. App. U also notes that its held-out split sizes match this paper's in every cell, so running it seems feasible. A simple practitioner baseline is also missing: for example, raise an alarm after k consecutive low rubric scores, or after k low scores in the last n turns, with the threshold set to a 5% false-alarm rate on dev-safe traces. Without these comparisons at matched false-alarm rates, it is impossible to tell whether CCC adds anything over existing or trivial monitors.

Questions for authors

Please clarify weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.