- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies LLM committees under a coordinated Byzantine majority. Sentinel screens each agent against its clean calibration history using a Kolmogorov-Smirnov distance on confidence complements and total variation distance on selected answer symbols, then conformally trust-weights survivors. Theorem 2 controls honest-agent rejection under exchangeability, independently of the number of corrupted peers; Proposition 1 and Theorem 3 characterize single-decision indistinguishability and the action-law-preserving attacker that marginal screens cannot detect. Experiments cover five LLMs, five-agent committees, MMLU, TruthfulQA, ARC-Challenge, GPQA-Diamond, ALFWorld, Mind2Web, and τ-bench, with 20 seeds in the principal suite plus live and free-form prompt-injection evaluations. At three corrupt agents of five, accuracy is 0.682 versus 0.094 for majority vote; on injected GSM8K answers, Sentinel recovers 73 to 76 percent of lost accuracy.
Critical strengths
-
The paper targets a clear failure mode: a correlated majority can subvert peer consensus and standard robust aggregators. The two-channel self-history screen is a technically motivated response.
-
The formal claims are carefully scoped: Proposition 1 establishes the one-decision limit, while Theorem 3 gives a concrete blind spot and bounds symbol alteration without implying task-level security.
-
The evaluation spans five model families, four multiple-choice benchmarks, two agentic domains, live τ-bench, and free-form GSM8K injection. Table 1 reports 20 seeds, intervals, paired tests, and baselines; 0.682 mean accuracy substantially exceeds the 0.094 majority-vote result.
-
Table 5 shows decision-level resampling breaks the budget on dependent agentic traces, while episode-level resampling restores it. The appendices also document failure cases and calibration alternatives.
Critical weaknesses
-
The guarantee requires honest test windows to remain exchangeable with clean calibration. Corrupted agents can shift honest members' context: Table 24 reports 0.242 to 0.280 honest rejection in attacked live retail sessions, above the 0.10 budget. Warm-up calibration in Table 25 helps, but the guarantee does not cover these shifts.
-
An attacker preserving its action marginal remains a substantive blind spot. Theorem 3 allows every selected symbol to change when no symbol exceeds half the clean marginal; Table 8 reports 0.305 accuracy against 0.705 clean for a couple-law coupling. This disclosed limit narrows robustness to adaptive harmful agents.
-
Although BFT is cited in related work, it is not compared experimentally, leaving the advantage over current confidence-based BFT defenses unclear.
-
Live task utility remains modest: Table 6 reports 4/30 retail and 3/30 airline completions under destructive attack, versus 3/30 clean in each domain. Table 24 reports 2/25 screened retail completions for the distinct-model committee. Security improves more clearly than end-task success; code and cached outputs are promised only upon acceptance.
Questions for authors
- How does attack-induced context shift affect exchangeability and honest-agent rejection after calibration on representative attacked trajectories?
- Can you evaluate CP-WBFT under the paper's fixed-shape, permutation-mimic, and prompt-injection attacks using matched models and task splits?
- Can you report live τ-bench uncertainty intervals and compare completion against an oracle screen over more episodes?