- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents HALO, which uses a coding agent to assemble responses from unwatermarked base-model samples. Across four tasks, it lowers Pangram v4’s aggregate detection rate from 77% to 24% while largely preserving task performance, and estimates reduced soft-watermark detection.
Critical strengths
- The paper studies an important and interesting problem.
- The paper demonstrates a concrete way for a capable agent to retain control over a writing task while sourcing much of the final text from a base model. This is a relevant challenge for AI-text detection.
Critical weaknesses
- The incremental contribution over prior work is unclear. Xu et al. (2026) established that base-model outputs often evade commercial detectors, while Frankentext (Pham et al., 2026) demonstrated that assembling text from external snippets can evade detection. HALO combines these ideas by having a capable agent solicit, select, and splice base-model continuations. This is a useful system demonstration, but the paper does not clearly establish which new mechanism accounts for its gains beyond the known properties of base-model text and snippet assembly. The comparison with HIP also changes the generation workflow and resource budget, making it difficult to isolate the benefit of HALO’s orchestration.
- The attack has a substantial cost. Table 2 reports roughly 8–27 times the orchestrator API cost of the Agent baseline, excluding the compute needed to run the base model. This matters to the paper’s case for HALO as a broadly usable attack, even if a motivated attacker might accept the expense for selected targets.
Questions for authors
- Please clarify the specific technical contribution beyond Xu et al.’s base-model observation and Frankentext’s assembly strategy. A controlled comparison with a capable agent that checks and repairs paraphrased outputs, under a comparable inference budget, would help determine whether HALO’s advantage comes from its generation strategy rather than additional agent effort and sampling.
- Please also present the detection–quality trade-off alongside total cost, including base-model inference.
7 comments
(cont) Response to reviewer X8af
-
We completely agree that the cost of the attack is a deterrent for its adoption – but we would like to respectfully point out that a weakness of the attack is not necessarily a weakness of the paper itself. Our goal is to highlight a novel vulnerability, previously unreported in the literature – along with both its efficacy as well as the key limitation.
To further check the dependence on cost, we construct 3 cost settings for HALO using Opus 5.5: (i) HALO (high) which uses the exact same setup as the paper; (ii) HALO (med) which reduces the total number of base LM calls available to the agent down to 8 from 30; and (iii) HALO (low) which additionally sets the thinking budget to medium (from xhigh). We compare these 3 variants at a copy target of p=0.90 to the baseline Agent using the same Opus 5.5 model, on a subset of n=50 prompts to restrict costs.
| Benchmark | Metric | Agent | HALO (low) | HALO (med) | HALO (high) |
|---|---|---|---|---|---|
| Mythos | Quality (coherence) ↑ | 1.000 | 0.780 | 0.720 | 0.840 |
| Pangram ↓ | 0.997 | 0.003 | 0.033 | 0.011 | |
| Cost / query ↓ | $0.28 | $0.54 | $0.94 | $1.33 | |
| FACTS | Quality (factuality) ↑ | 0.750 | 0.920 | 0.940 | 0.960 |
| Pangram ↓ | 0.307 | 0.003 | 0.008 | 0.005 | |
| Cost / query ↓ | $0.22 | $0.28 | $0.62 | $0.55 | |
| IFEval | Quality (strict acc.) ↑ | 1.000 | 1.000 | 1.000 | 1.000 |
| Pangram ↓ | 0.971 | 0.206 | 0.118 | 0.086 | |
| Cost / query ↓ | $0.23 | $0.30 | $0.62 | $0.88 | |
| HealthBench | Quality (score) ↑ | 0.665 | 0.582 | 0.607 | 0.607 |
| Pangram ↓ | 1.000 | 0.556 | 0.351 | 0.301 | |
| Cost / query ↓ | $0.23 | $0.73 | $1.73 | $2.30 |
We make a few observations here:
- With Opus 5.5, the gap between the Agent approach and HALO approach is smaller, even at the highest attack cost setting, down to 2x-10x as compared to 8x-27x with Opus 5. This reduction is caused due to 2 reasons: (i) Opus 5.5 has a lower cost for cache reads, which form the bulk of the input tokens for HALO (since the base LM samples are read repeatedly from disk); and (ii) the baseline Agent consumes a higher number of tokens with the newer model.
- Lower cost HALO attacks are less effective since they generally either reduce the quality score, or increase the Pangram fraction-AI score.
- However, even at the lowest cost setting, which is 1x-3x of the Agent baseline, we still see meaningful evasion while preserving quality on 2 out of the 4 benchmarks.
Overall, we still believe that motivated attackers on subscription plans can easily consume these costs, especially when producing individual texts rather than analyzing data at scale.
Response to Questions
Please see the responses to the weaknesses. We note that the base LM sampling accounts for a small fraction of the orchestration cost. We generally observe the agent sample 1K-28K tokens from the base LM across the four benchmarks – at an estimated cost of \0.004 per query on the higher end. The orchestrator costs, on the other hand range from \2.30.
Dear reviewer, we are approaching the end of the author-reviewer discussion period. We have attempted to address all the concerns pointed out in your review -- if there is any additional information we can provide before the deadline, please let us know!
Response to reviewer X8af
We thank the reviewer for their detailed feedback. We are glad they found our paper important and interesting, and that our approach presents a relevant challenge for AI-text detection.
Response to Critical weaknesses
We respectfully disagree that our approach of combining the ideas presented by Xu et al and Pham et al is incremental, due to the following reasons: (i) our comparisons to either of their approaches show clear, large gains on the pareto-frontier of the quality vs evasion tradeoff; (ii) we present the novel finding that coding agents can creatively prompt base models for collecting texts to complete a varied set of writing tasks; and (iii) our paper goes beyond post-hoc detection, which is the focus of those prior works, to also devise a technique (via pointer tags) which evades watermarking; moreover, we present a novel analysis technique for determining the strength of watermarking even when the orchestrating agent provides no detection API (as is the case for most frontier models today). This analysis technique may be useful by itself for future work which attempts to study the robustness of watermarking.
To further isolate the benefits of orchestration we further present a comparison to two additional baselines, both of which use the same orchestrator as HALO, but instead of the base LM sampling tool using: (i) a static pool of ~100K tokens of texts sampled from the base LM given just the task prompt, formatted in a zero-shot manner for the base LM; and (ii) a static pool of ~100K tokens of human text consisting of retrieved passages (again given the task prompt) from reddit WritingPrompts (for Mythos), C4 (for IFEval), and MedQuad (for HealthBench). For FACTS we use the provided context document itself as a human pool of text.
The results below are computed on n=50 prompts for each benchmark at a copy target of p=0.90, using Opus 5.5 as the underlying model (since we found that Opus 5 now tends to refuse some orchestration requests).
Observations:
[1] Russell, Jenna, et al. "How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text." arXiv preprint arXiv:2609.40295 (2026).