- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Verified Plurality with Abstention (VPA), a Byzantine-robust federated-inference framework for LLMs that clusters outputs by a semantic-equivalence oracle, filters them with an external verifier, and returns the largest surviving cluster only if its support exceeds a threshold τ, with theoretical error bounds and strong results on HumanEval+ and MMLU.
Critical strengths
This paper focuses on federated inference of LLMs, a concrete scenario in large model applications, and proposes a solution to the security problems arising in this setting.
Critical weaknesses
Weakness 1: This method relies heavily on the verifier. However, if this verifier can be attacked (for example, through injected backdoors or latent bugs), the proposed method no longer holds.
Weakness 2: In addition, for natural language tasks beyond code, such as medical question answering, it is difficult to find such a verifier.
Weakness 3: The experimental coverage is too narrow. The evaluations on HumanEval+ and MMLU are not sufficient to adequately validate federated inference of LLMs.
Questions for authors
See weakness.
5 comments
Thank you for the clarifications and additional experiments. However, I still do not find sufficient experimental evidence for the claimed practical robustness. The mutant-attack evaluation is restricted to 140 tasks where correct honest answers already form a strict majority, while the CodeT attack comparison does not report the corresponding coverage. These limitations leave the robustness–availability trade-off insufficiently characterized.
Regarding W2, MedCalc-Bench evaluates structured numerical outputs for which exact matching or numerical tolerance is available. This does not fully address open-ended medical QA, where both correctness verification and semantic equivalence are difficult to establish. Reduction to plurality with abstention is a valid theoretical fallback, but does not itself demonstrate practical robustness in such settings. More extensive, more targeted adversarial evaluations are needed to substantiate the security claims.
Response
Response to Reviewer GYMu
We than the reviewer for taking the time to respond.
- On the adaptive adversary please find bellow the results on full HumanEal+. (Note that restricting to tasks where majority is correct was done to match corollary 2.9-(1) result).
Adaptive attack impact on VPA the reported results are error vpa at a given f as (error(f) − error(f=0)) to see how the adaptive adversaries impacts error compared to when no attack is performed
| Similarity | =2, =1 | =2, =2 | =2, =3 | =3, =1 | =3, =2 | =3, =3 | =6, =1 | =6, =2 | =6, =3 | =7, =1 | =7, =2 | =7, =3 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CrossHair | -0.4 | +3.9 | +6.3 | -0.4 | +0.3 | +6.5 | -0.2 | +0.8 | +1.7 | +2.5 | +3.4 | +5.4 | |
| CodeBLEU | +0.8 | +14.1 | +23.0 | +1.0 | +3.2 | +26.1 | +0.0 | +2.9 | +2.6 | +0.0 | +4.5 | +4.0 | |
| RTests | +0.6 | +1.2 | +1.8 | +0.6 | +1.8 | +2.4 | +2.1 | +3.5 | +4.7 | +0.7 | +2.3 | +3.8 | |
| Tests | +0.0 | +1.4 | +2.5 | +0.0 | +2.5 | +3.6 | +1.4 | +4.2 | +4.8 | -0.1 | +3.1 | +3.8 | |
| and V | CrossHair | +0.6 | +4.2 | +5.3 | +0.6 | +1.3 | +5.4 | +0.7 | +1.7 | +2.2 | +2.9 | +4.1 | +3.9 |
| and V | CodeBLEU | +0.9 | +11.0 | +18.4 | +1.1 | +2.3 | +20.5 | +0.0 | +3.1 | +2.9 | +0.0 | +4.8 | +4.2 |
| and V | RTests | +0.7 | +1.4 | +2.7 | +0.7 | +2.1 | +3.4 | +1.6 | +3.2 | +4.6 | +0.8 | +2.6 | +4.2 |
| and V | Tests | +0.0 | +0.8 | +1.4 | +0.0 | +1.4 | +2.1 | +0.8 | +2.4 | +3.1 | -0.1 | +1.7 | +2.6 |
- We also report the coverage with the CodeT variant abstaining the same tasks as VPA to compare at matched coverage.
Table: VPA and CodeT (with and w/o abstention) under the Byzantine adaptive adversary
| Error (%) | No abstention: All tasks | Abstain where VPA abstains: =1 | =2 | =3 | =6 | =7 |
|---|---|---|---|---|---|---|
| f=0 | ||||||
| CodeT | 4.9 [1.8, 8.5] | 4.9 [1.5, 8.6] | 4.9 [1.5, 8.6] | 4.2 [1.4, 7.7] | 1.6 [0.0, 4.0] | 1.7 [0.0, 4.3] |
| VPA (RTests, and V, abstain on ties) | -- | 4.9 [2.0, 8.6] | 4.2 [1.4, 7.7] | 3.5 [0.7, 6.7] | 0.8 [0.0, 2.4] | 1.7 [0.0, 4.3] |
| VPA − CodeT (same tasks) | -- | +0.0 [-2.1, +2.1] | -0.7 [-2.2, +0.0] | -0.7 [-2.2, +0.0] | -0.8 [-2.5, +0.0] | +0.0 [+0.0, +0.0] |
| VPA − CodeT (no abstention) | -- | +0.0 [-2.3, +2.2] | -0.7 [-2.7, +0.7] | -1.4 [-3.7, +0.4] | -4.1 [-7.3, -1.2] | -3.2 [-6.4, -0.6] |
| Coverage (%) | 100.0 | 87.2 | 87.2 | 86.6 | 78.7 | 72.6 |
| Tasks answered | 164 | 143 | 143 | 142 | 129 | 119 |
| f=1 | ||||||
| CodeT | 4.3 [1.8, 7.3] | 3.5 [0.7, 6.8] | 3.5 [0.7, 6.8] | 3.5 [0.7, 6.8] | 0.8 [0.0, 2.4] | 0.8 [0.0, 2.7] |
| VPA (RTests, and V, abstain on ties) | -- | 3.5 [0.7, 6.8] | 3.5 [0.7, 6.8] | 3.5 [0.7, 6.8] | 0.8 [0.0, 2.4] | 0.8 [0.0, 2.7] |
| VPA − CodeT (same tasks) | -- | +0.0 [+0.0, +0.0] | +0.0 [+0.0, +0.0] | +0.0 [+0.0, +0.0] | +0.0 [+0.0, +0.0] | +0.0 [+0.0, +0.0] |
| VPA − CodeT (no abstention) | -- | -0.7 [-2.7, +0.6] | -0.7 [-2.7, +0.6] | -0.7 [-2.7, +0.6] | -3.5 [-6.7, -1.0] | -3.4 [-6.7, -0.9] |
| Coverage (%) | 100.0 | 86.0 | 86.0 | 86.0 | 78.7 | 72.6 |
| Tasks answered | 164 | 141 | 141 | 141 | 129 | 119 |
| f=2 | ||||||
| CodeT | 7.3 [3.7, 11.6] | 4.4 [1.4, 8.0] | 4.4 [1.4, 8.0] | 4.4 [1.4, 8.0] | 0.8 [0.0, 2.5] | 0.8 [0.0, 2.7] |
| VPA (RTests, and V, abstain on ties) | -- | 5.1 [1.5, 9.0] | 4.4 [1.4, 8.0] | 5.1 [1.5, 9.0] | 0.8 [0.0, 2.5] | 0.8 [0.0, 2.7] |
| VPA − CodeT (same tasks) | -- | +0.7 [+0.0, +2.3] | +0.0 [+0.0, +0.0] | +0.7 [+0.0, +2.3] | +0.0 [+0.0, +0.0] | +0.0 [+0.0, +0.0] |
| VPA − CodeT (no abstention) | -- | -2.2 [-5.5, +0.8] | -2.9 [-6.0, -0.3] | -2.2 [-5.5, +0.8] | -6.5 [-10.7, -2.9] | -6.5 [-10.7, -2.9] |
| Coverage (%) | 100.0 | 83.5 | 83.5 | 83.5 | 77.4 | 72.6 |
| Tasks answered | 164 | 137 | 137 | 137 | 127 | 119 |
Making CodeT abstain in tasks VPA abstained in significantly reduces its error rates.
- On imperfect equivalene: We’d also like to highlight our Endo-Bench experiment (Sec. Appendix) where models generate bounding boxes. The equivalence is approximate. This directly tests our framework under approximate equivalence, beyond structured numerical outputs like MedCalc-Bench. Note also that even in code exact equivalence can't be established (Rice's theorem) so all the equivalence oracles we use are also approximate and are intentionally used to showcase how the equivalence impacts the results e.g., CodeBlue causing component fragmentation (reported in the paper).
Response to Reviewer GYMu
We thank the reviewer for their time and feedback. We appreciate the reviewer’s recognition that federated inference of LLMs is “a concrete scenario in large model applications” and that the paper proposes a solution to the security problems arising in this setting. We address the reviewer's concerns below:
(W1) The method relies on the verifier; if it is attacked (backdoor, latent bugs), the method no longer holds: The guarantees do not require a perfect verifier and corollary 2.9 (1) extends to the case of no verifier. We also agree that verifier are hard to find for some subsets of tasks and our experiments cover some of those settings.
Please notice that V is not one of the federated models, and in our experiments the verifier is not an LLM. It is the aggregator's own test runner, inside the same trust boundary as the aggregator.
The framework covers this case explicitly: with no verifier (α=1, β=0), \alg{} is plurality with abstention and Theorem 2.8 still applies. MMLU is this instantiation. Corollary 2.9(1) shows that without a verifier no aggregator can tolerate f≥n/2 in the worst case, so this is a limit of the problem, not of VPA. Many deployed tasks do admit verifiers: code, SQL, structured or schema-constrained outputs, numerical computations, and formal proofs. Our medical task, MedCalc-Bench (Appendix 7), shows the verifier-free case in practice: no verifier is used (α=1, β=0), the equivalence oracle is exact match or a 5% tolerance, and VPA still matches the best base model on the tasks it answers at every τ, with zero error for τ≥6, without knowing which model is best. Medical calculations can also admit cheap verifiers (recomputing the score from extracted parameters, range and unit checks, etc.).
The paper also evaluates MedCalc-Bench, a medical calculation benchmark (Appendix 7), in addition to HumanEval+ and MMLU, with 16 open-weight models from 11 developers (Table 2) and four equivalence oracles for code (CodeBLEU, Tests, RTests, CrossHair). We agree more is needed. We add EndoBench visual grounding task on 5 VLMs and extend MedCalc-Bench to cover the full test set. For ref see results from endobench in table bellow:
Table: [email protected] on EndoBench Visual Grounding: single models vs. VPA, accuracy with 95% bootstrap CIs.