- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper looks at when a model can stop reading a long document without hurting its answer. The idea is to stop when the model is confident and its answer has stayed stable as it reads more context. The method does not require training, and the experiments test how much computation it can save across different models and tasks.
Critical strengths
1.I like the basic idea. Being confident does not always mean that the model has read enough, so checking whether the answer stays stable makes sense to me. The method is also simple and does not require training another model, which makes it easy to try in an existing reading pipeline. 2.Some of the savings are substantial. For example, with Qwen on LongBench v2, the method saves about half the tokens while keeping similar accuracy. The authors also count the extra answer checks in the cost, which helps me judge whether the savings are useful in practice.
Critical weaknesses
1.The comparison with existing methods is the main thing missing for me. The paper discusses DCC and GRU-Mem, but does not compare with them experimentally. Most of the comparisons are between stopping rules within the same reading pipeline. I still want to know how ACS compares with these existing approaches, and what performance trade-off comes with avoiding training. 2. I am also not sure how much of the benefit comes from checking stability, and how much comes from simply reading longer before stopping. A useful comparison would be to let the simpler baselines spend the same number of tokens. ACS does not always do better than END either. With Kimi on LongBench v2, default ACS gets 51.9% accuracy and saves 3% of tokens, while END gets 52.5% accuracy and saves 46%. This leaves me unsure when I should choose ACS over the simpler rule.
Questions for authors
1.How does ACS compare with DCC or GRU-Mem in a comparable setting? I would like to see training and inference costs reported separately, so that the benefit of avoiding training is clear. 2.If the methods use the same token budget, does ACS still do better than confidence-only stopping or an END gate that requires several consecutive END decisions? This would help me understand what the stability check adds. 3.Work on confidence-based self-training has shown that confident mistakes can get reinforced. For ACS, I wonder whether the model can also stay confident and consistent about a wrong answer, then stop before reaching the information that would correct it. How often does ACS stop with a wrong answer that full reading would get right? Could you break this down by task, especially in cases where misleading information appears early and decisive evidence comes later, and show whether the stability check reduces these failures compared with confidence alone?
4 comments
Rebuttal
ACS and the END gate with Kimi K2.5 (W2). The reviewer's example is correct, and it uses the default θ. This is one of three ways to set θ: tuned on task questions when they are available, set on synthetic needles when they are not, and a default that needs no data and gives up saving but not accuracy. The table gives the three settings and the END gate on LongBench-v2, with the tokens saved and the accuracy relative to full reading.
| Stopping rule | Needs | Kimi K2.5: saving | Kimi K2.5: accuracy against full reading | Qwen3.5: saving | Qwen3.5: accuracy against full reading |
|---|---|---|---|---|---|
| ACS, θ tuned on task questions | labelled questions | 47% | +1.1 points | 53% | 0.0 points |
| ACS, θ set on synthetic needles | no labels | 49% | +1.8 points | 54% | -0.6 points |
| ACS, default θ = 0.995 | nothing | 3% | +0.2 points | 50% | +0.6 points |
| END gate | nothing | 46% | +0.8 points | 65% | -2.8 points |
With Kimi K2.5, whose confidence runs lower, the default stops less and moves toward full reading, so it gives up saving and not accuracy. Setting θ on synthetic needles needs no task labels (θ is the smallest value with no stop before the evidence, 0.92 for Kimi K2.5, Table 4), and ACS is then above the END gate in both accuracy and saving. With Qwen3.5 it is the END gate that pays, and it pays in accuracy: it saves the most and loses 2.8 points, while ACS is never more than 0.6 points below full reading. The revision will present the three settings in this way and add the two tuned ones to Table 1.
Wrong early stops (Q3). Yes, this can happen: ACS can stop on a confident and stable answer that is wrong while full reading gets it right. On LongBench-v2 it does so on 20 of the 503 questions with Qwen3.5 (4.0%) and on 12 with Kimi K2.5 (2.4%, θ set on needles). The table gives the number of such stops by task, for ACS and for confidence alone, which stops at the first confident answer.
| LongBench-v2 task | Questions | Qwen3.5: ACS | Qwen3.5: confidence alone | Kimi K2.5: ACS | Kimi K2.5: confidence alone |
|---|---|---|---|---|---|
| Code repository understanding | 50 | 5 | 5 | 3 | 4 |
| Long in-context learning | 81 | 5 | 6 | 5 | 8 |
| Long structured data understanding | 33 | 3 | 4 | 0 | 0 |
| Long-dialogue history understanding | 39 | 1 | 3 | 0 | 2 |
| Multi-document QA | 125 | 1 | 3 | 2 | 4 |
| Single-document QA | 175 | 5 | 7 | 2 | 5 |
| All | 503 | 20 (4.0%) | 28 (5.6%) | 12 (2.4%) | 23 (4.6%) |
These failures are few in every task, at most 5 questions, and they are not confined to one type of data. The stability check reduces them, from 28 to 20 with Qwen3.5 and from 23 to 12 with Kimi K2.5, and they are lower or equal in every task with both models. The case where misleading information comes early and the decisive evidence later is tested directly in the next block. We will add this table to the appendix.
Misleading information early, decisive evidence late (Q3). We built this case on the 250 S-NIAH questions of the paper: for each question, three needles of the same form for other keys are placed before the target needle, as in the multi-key task of RULER. With Qwen3-14B, ACS stops before the decisive evidence on 1 of the 250 questions (0.4%). In that one case it returns the number of a misleading needle, which is the failure the reviewer describes.
| Stopping rule | Tokens | Stops before the evidence | Accuracy |
|---|---|---|---|
| ACS | 39.1K | 1 (0.4%) | 0.984 |
| Confidence alone (first confident answer) | 29.1K | 7 (2.8%) | 0.972 |
| END gate | 23.8K | 71 (28.4%) | 0.716 |
| END gate, 4 ENDs in a row | 38.0K | 21 (8.4%) | 0.904 |
| END gate, 5 ENDs in a row | 40.9K | 16 (6.4%) | 0.924 |
| Verbalized gate | 20.8K | 101 (40.4%) | 0.596 |
| Full reading | 54.1K | 0 | 0.988 |
The stability check lowers the stops before the evidence from 7 to 1. The END gate is misled far more: it stops before the evidence on 28.4% of the questions, against 5.2% without the misleading needles (Table 2), and four or five ENDs in a row, which cost the tokens of ACS, still leave 8.4% and 6.4%. So when misleading information comes first, ACS keeps reading until the decisive evidence on all but one question, and the END gate often stops too early. We will add this experiment to Section 4.3.
Rebuttal
When to choose ACS over the END gate (W2). Taken together, the results above answer this question. Both rules need no training, and they differ in what they read: the END gate asks the model whether it has read enough, and ACS measures the answer itself. The accuracy of ACS stays close to that of full reading with both models, where the END gate loses 2.8 points with one of them, and under misleading information ACS stops before the evidence on 0.4% of the questions, against 28.4% for the END gate (Q3). Its saving depends on θ, and the three settings above say how to set it with the data at hand. In general, ACS is therefore the more stable choice, also when the documents contain misleading information. The END gate is a useful alternative when the model returns no output probabilities, which ACS needs.
I thank the authors for the extensive new experiments.These analyses should have been part of the original submission.
For this reason I raise my score only modestly, from 2 to 3.Good luck!
Rebuttal
We thank the reviewer for the constructive comments. We answer each point below.
Comparison with DCC and GRU-Mem (W1, Q1). Of the two methods the reviewer names, GRU-Mem has no public code or weights, so we ran DCC, with its official code, default settings and protocol, on the same Qwen3-14B and the same 500 RULER-HotpotQA documents as the paper (Table 10). Under this protocol, ACS matches DCC in overall accuracy without training, DCC uses fewer tokens per document, and ACS is significantly more accurate on the longer documents. The table gives accuracy by document length (100 documents each), then the training and inference costs separately.
At 8K and 16K neither method stops early, and the difference is between reading in one context (DCC) and reading in chunks (ACS). From 32K on both stop early, and ACS is the more accurate one at each length: over the 294 documents of 32K and above that both methods can read, it is 12.2 points above DCC (95% CI 6.1 to 18.4).
This gap could come from the training of the probe, because the official protocol of DCC covers training on documents of up to about 40K tokens, and our longest documents are beyond that. To make the comparison fairer, we therefore also trained the probe on all five lengths (600 labelled documents, 4.6 GPU-hours), keeping everything else as in the official code and following the DCC paper as closely as we could. The probe then stops early on fewer than 10 of the 494 documents it can read, so DCC saves no tokens and becomes full-context reading (0.781 at 48.3K tokens per document).
Taken together, the learned stopper depends on the lengths it was trained on: trained on the shorter documents it stops too early on the longer ones, and trained on all lengths it almost never stops. ACS needs no training and uses only the output probabilities of the model, so it can be applied at once to any model or API that returns them, at any document length. DCC has its own advantage: it reads the document in one context, which is more accurate on the short documents, and it uses fewer tokens per document. We will add this comparison to Section 4.
Same token budget (W2, Q2). ACS, the END gate and confidence-only stopping are all adaptive rules: each decides, document by document, how much to read. A comparison between them at the same tokens therefore compares one adaptive rule with another. To separate the idea from the amount of reading, the amount has to be given with no decision. So we read every document of LongBench-v2 (503 questions) up to a fixed share of its chunks (rounded up), with no stopping rule. The last fixed row is an oracle: its share is the average share that ACS reads.
Against fixed reading, ACS leads at the same tokens. The oracle row is told how much ACS reads in total (51% of the tokens with Qwen3.5 and 52% with Kimi K2.5), which a fixed rule could not know in advance, and reads that share of every document. It is still 3.6 points less accurate with Qwen3.5 and 5.2 points with Kimi K2.5. The right amount differs from one document to the next: ACS stops earlier than the oracle share on a third of the documents and reads further on 60% of them. Taken together, against the fixed reading controls, the adaptive compute of ACS gives both better accuracy and larger savings: it is more accurate than every control, including those that read up to twice its tokens. Against the other adaptive rules, trained and training-free, the comparison is in Q1 and in the paper: ACS matches the trained one without training, and among the training-free ones it stops before the evidence least often. We will add the fixed controls to Table 1.