- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to predict the final correctness of an LLM multi-agent execution from its communication history. It applies standard martingale and conditional-variance identities to describe expected uncertainty resolution and realized uncertainty increases. The proposed method, CUE, trains a shared probe on frozen-backbone features of communication prefixes using only final correctness labels. A variance regularizer encourages average uncertainty to decrease, while changes in predictions provide resolution and amplification estimates. Across four benchmarks, three orchestrations, and two backbone families, the authors report improved failure prediction over seven baselines. Additional analyses examine early prediction, message attribution, and computational cost.
Critical strengths
-
The experiments provide substantial evidence that communication history helps predict final failure. CUE achieves the highest mean AUROC in 27 of 28 configurations. Table 5 also directly tests the value of history: on Llama/MATH500, removing earlier messages reduces average AUROC from 89.2 to 79.2. This supports the main empirical motivation.
-
CUE uses only final correctness labels to train a shared predictor across prefixes. It does not require labels for individual messages. This makes the supervision practical and allows predictions before the execution finishes without training a separate model for each position.
-
The formulation clearly separates expected uncertainty reduction, realized increases, and causal effects. The decomposition does not require independence between agents, and the authors acknowledge order dependence and representation limitations. These distinctions provide a useful basis for analyzing what an observer learns from communication.
Critical weaknesses
-
The main gain seems to come from giving a correctness probe the full history. On Llama/MATH500, Table 5 shows a 10-point AUROC gain from history, but only 0.16 points from the variance regularizer. The theory’s added value to the method is therefore unclear.
-
The message-level estimates need stronger checks. Appendix D.4 assigns information gain to a vote that adds no new information to the full history. The authors explain this through the features, but it remains unclear how reliably the scores measure information from messages.
-
Table 15 compares early CUE predictions with baselines that read the completed debate. It would be more useful to also compare against a standard correctness probe trained on the same partial history.
-
CUE needs about 2,200 seconds to extract features for 500 Llama debate executions (Appendix C.4). This makes it the most expensive tested method overall. The paper does not show whether early detection can save enough computation to cover this cost.
Questions for authors
-
Does the variance regularizer help estimate correctness probabilities more accurately, even when it barely changes AUROC?
-
Could you test the message-level scores on a simple example where the true information gain is known?
-
How does CUE compare with a standard probe using the same inputs and labels, including the time spent extracting features?