- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a defense method against tolerating faulty in distributed test-time scaling paradigm. The key idea is to use an oddity statistic to identify unreliable CoTs. Experiments on several reasoning models and benchmarks demonstrate the efficacy of the proposed method.
Critical strengths
- The threat is well motivated and practically relevant.
- The proposed method is reasonable and easy to follow.
- Evaluation covers several attack scenarios, including prompt substitution, post-generation trace manipulation, and model substitution.
Critical weaknesses
- The proposed oddity metric averages token-level statistics over the entire sequence. Consequently, an attack affecting only a small number of token positions may be diluted.
- The framework assumes that all honest workers use the same model and decoding configurations. This limits applicability to heterogeneous test-time scaling scenarios, since correct heterogeneous models may be identified as attackers.
- The proposed metric is similar to existing likelihood-based statistics, and using this statistics to detect distributional inconsistencies is intuitive. The novelty appears limited.
Questions for authors
Please refer to weaknesses.
2 comments
Dear Reviewer,
Thank you once more for your insightful and constructive comments and for the time you have dedicated to reviewing our paper. As the discussion phase comes to an end, kindly let us know if you have any further questions or concerns.
Best regards,
Authors
Thank you for your valuable comments. We address all the reviewer’s questions below.
Weakness 1. An attack affecting only a small number of token positions
The original submission already evaluates attacks that modify only a small number of token positions through trace-side injection (Section 5.1 and Appendix B.3). Our experimental results show that PoT can detect these attacks by combining oddity with answer-token certainty (Section 4.4). We explain how below.
In trace-side injection, the attacker generates a valid CoT trace and replaces its final answer with a target incorrect answer. Since honest answer tokens typically have perplexity close to 1 (see Section 4.4), this replacement can increase answer-token perplexity and trigger PoT’s answer-token certainty check, whose signal is not diluted over the entire trace. To evade this check, the most effective approach among those tested replaces all occurrences of the original answer in the trace with the target answer. This lowers answer-token perplexity but can introduce locally unlikely tokens and increase oddity (Figure 10), creating a trade-off that enables the two checks to provide complementary defenses. As shown in Table 2, 31.17% of trace-side injection candidates pass the oddity check alone, and 14.83% pass the answer-token check alone. Only 6.00% pass both checks, demonstrating the benefit of combining these complementary signals.
Weakness 2. Applicability to heterogeneous test-time scaling
While the original submission considers a common model to preserve equivalence with conventional local TTS, PoT can also support heterogeneous models. The verifier assigns each worker a reference model, so it knows which model to use when verifying each returned trace. Using the corresponding model’s configuration, the verifier filters each trace separately and then pools the retained answers across models for a single vote.
To demonstrate this, we conducted an additional experiment on AIME25 using 10 Qwen3-4B workers and 10 DeepSeek-R1-8B workers per question. We used the same three attacks described in Section 5 and Appendix B of the original submission. For each attack, we corrupted three submissions from each model, yielding a pool with 30% malicious submissions, and evaluated all methods on the same subsets of n=15 submissions drawn from this pool.
Across the three attacks, PoT retained 89.0% of honest Qwen traces and 85.2% of honest DeepSeek traces, preserving the large majority of honest contributions from both models. As shown below, PoT improves accuracy over Self-Consistency by 13.8–24.6 percentage points and achieves the highest accuracy among the non-oracle methods under every attack. This is consistent with the trend observed in our homogeneous-model experiments and demonstrates that PoT’s robustness benefits extend to heterogeneous TTS.
The following table reports accuracy (%) on AIME25 for n=15 submissions drawn from the heterogeneous worker pool with 30% malicious submissions.
Weakness 3. Novelty of PoT
While oddity is derived from token probabilities, PoT does not simply identify low-likelihood traces as malicious. The experiments in Sections 3.5 and 5 show that perplexity and Self-Certainty (Kang et al., 2025) fail to reliably distinguish honest from malicious submissions in distributed TTS. These results demonstrate that straightforward likelihood-based scoring and aggregation do not provide adequate protection against the evaluated attacks.
To further evaluate PoT against existing likelihood-based statistics, we added the method of Miao & Fang (2025) [1] as a new baseline. Their method compares the submitted token’s logit with the highest logit at each position and uses the lowest-confidence window. We evaluated this baseline on AIME25 with Qwen3-4B and 30% malicious participants. PoT achieves higher average accuracy across all three attack surfaces. Specifically, under trace-side injection, PoT achieves 61.14% average accuracy, compared with 49.20% for Miao & Fang (2025), an improvement of 11.94 percentage points. These results support our contribution as a verification-and-aggregation framework designed to address different forms of adversarial manipulation, rather than the use of token probabilities alone.
[1] Qijun Miao and Zhixuan Fang. 2025. User-side Model Consistency Monitoring for Open Source Large Language Models Inference Services. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics.