- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LatentSift replaces the LLM verifier in Stage 1 of SWE-agent hybrid verification. It scores each candidate trajectory by the nearest-neighbor distance of the policy's own hidden states (reasoning, observation, and function-call channels) to banks of successful and failed training trajectories, combined with a linear probe. It cuts verification tokens by 49–62% at K=16 on SWE-bench Verified, with Best@16 matching or slightly exceeding the original cascade.
Critical strengths
Reusing hidden states the policy already computed instead of re-reading long trajectories is a clean, practical idea. The paper is clearly written and the token accounting is detailed.
Critical weaknesses
-
Can it generalize? The banks, the probe, and the three-channel design are all tied to one policy, one SWE scaffold, and SWE-bench Verified. Does a new task or domain, such as web or terminal agents, or other languages, need its own labeled bank and probe, and how does LatentSift perform on any benchmark besides SWE-bench Verified?
-
Does the bank go stale when the policy is updated? The banks hold states from the policy's own trajectories, so each new model, fine-tune, or checkpoint likely means collecting and labeling the bank again. How much does performance drop with a bank from an older checkpoint or a different policy, and must the bank be rebuilt after every policy update?
-
The bank's cost isn't counted. Each policy needs about 2,800 outcome-labeled rollouts with test execution and stored all-layer hidden states, which by a rough estimate is about 150M generation tokens. All of it is excluded from the reported savings. What is the full offline cost (tokens, test executions, storage), and how does accuracy scale with smaller banks, such as 100 or 500 trajectories?
-
Where do the banks come from? The banks are described as "collected during policy training," yet all four policies are third-party checkpoints, and every configuration has exactly 1,383 successful and 1,383 failed trajectories. How were the banks built for policies you did not train?
-
Most of the savings come from running the expensive verifier on about 4 survivors instead of 16, so any cheap Stage-1 filter would save just as much. What are Best@16 and token cost with random filtering, or with no Stage 1 at all, followed by the Stage-4 verifier?
-
The gains are small and unsupported statistically. Improvements of 0.3–1.2 points correspond to about 2–6 of 500 tasks, with no confidence intervals. On the R2EGym agent, the DeepSWE-Verifier baseline (49.70%) beats LatentSift (47.73%). Can you report bootstrap confidence intervals and explain the R2EGym result?
Questions for authors
-
Can it generalize? The banks, the probe, and the three-channel design are all tied to one policy, one SWE scaffold, and SWE-bench Verified. Does a new task or domain, such as web or terminal agents, or other languages, need its own labeled bank and probe, and how does LatentSift perform on any benchmark besides SWE-bench Verified?
-
Does the bank go stale when the policy is updated? The banks hold states from the policy's own trajectories, so each new model, fine-tune, or checkpoint likely means collecting and labeling the bank again. How much does performance drop with a bank from an older checkpoint or a different policy, and must the bank be rebuilt after every policy update?
-
The bank's cost isn't counted. Each policy needs about 2,800 outcome-labeled rollouts with test execution and stored all-layer hidden states, which by a rough estimate is about 150M generation tokens. All of it is excluded from the reported savings. What is the full offline cost (tokens, test executions, storage), and how does accuracy scale with smaller banks, such as 100 or 500 trajectories?
-
Where do the banks come from? The banks are described as "collected during policy training," yet all four policies are third-party checkpoints, and every configuration has exactly 1,383 successful and 1,383 failed trajectories. How were the banks built for policies you did not train?
-
Most of the savings come from running the expensive verifier on about 4 survivors instead of 16, so any cheap Stage-1 filter would save just as much. What are Best@16 and token cost with random filtering, or with no Stage 1 at all, followed by the Stage-4 verifier?
-
The gains are small and unsupported statistically. Improvements of 0.3–1.2 points correspond to about 2–6 of 500 tasks, with no confidence intervals. On the R2EGym agent, the DeepSWE-Verifier baseline (49.70%) beats LatentSift (47.73%). Can you report bootstrap confidence intervals and explain the R2EGym result?
6 comments
Rebuttal for reviewer T3o2(2)
Q3 (W3)
Thank you for raising this concern. We would like to clarify that bank construction introduces nearly zero additional token cost. The bank can be built online during policy training by reusing the encoding hidden states and test outcomes of training trajectories already produced by the training process. Therefore, the estimated 150M tokens should not be considered an additional cost of our method. The incremental overhead primarily comes from storing the collected hidden states. Since bank construction reuses existing training data rather than requiring additional generation, reducing the bank to 100 or 500 trajectories is not necessary for controlling generation-token cost. But we appreciate the reviewer's thought and are pleased to provide additional results with smaller banks to further investigate the impact of bank size, and we also add this part in Section B.4 in appendix of our revised paper.
Table 4. Bank-size ablation on DeepSWE at . Each pair contains one successful and one failed trajectory. Smaller banks are evaluated over 50 random seeds, with standard deviations in parentheses.
| Bank Size (pairs) | Best@16 (%) | Tokens / Task |
|---|---|---|
| 100 | 59.44 (0.66) | 384.6K |
| 500 | 59.47 (0.48) | 387.9K |
| 1,383 (Full) | 60.06 | 391.9K |
| LLM EF verifier | 59.26 | 1,035.1K |
As shown in Table 4, reducing the bank to 100 or 500 pairs yields Best@16 of 59.44% and 59.47%, respectively, compared to 60.06% with the full bank. The differences are within 0.62 percentage points, suggesting that LatentSift remains effective even with substantially smaller banks.
Q4 (W4)
Thank you for raising this question. Since our experiments use third-party checkpoints, we construct the banks by replaying training trajectories through the corresponding policy models to extract their hidden states the same as the process during model training. In practical settings, however, banks can be constructed directly during policy training by collecting hidden states from training rollouts, without additional trajectory generation or encoding cost.
We appreciate your reminder and have included this detailed implementation in the revised paper in Section 4.1.
Rebuttal for reviewer T3o2(3)
Q5 (W5)
Thank you for raising this question. Stage-1 filtering serves an important purpose in the original workflow: Stages 2–3 require executing candidate patches and running tests in real sandboxes, which is time-consuming and CPU-intensive. Therefore, the original workflow filters 16 candidates down to 8 before execution to reduce this overhead.
Simply using a cheaper filter does not guarantee comparable resolution performance. To investigate this, we evaluate two additional baselines: random Stage-1 filtering and removing Stage 1 entirely, relying on Stage 4 for final selection.
Table 5. Comparison of Stage-1 filtering strategies at . Each entry reports Best@16 (%) / verification tokens per task (K), including test generation.
| Stage-1 Filter | DeepSWE | R2EGym | CWM |
|---|---|---|---|
| LLM EF (original) | 59.26 / 1035.1K | 46.56 / 599.6K | 54.21 / 930.0K |
| No Stage 1 | 57.44 / 523.4K | 46.76 / 376.2K | 53.91 / 665.6K |
| Random filtering | 55.98 / 387.2K | 45.48 / 307.8K | 52.85 / 461.3K |
| LatentSift (ours) | 60.06 / 391.9K | 47.73 / 304.9K | 54.21 / 466.8K |
As shown in the table, random filtering reduces Best@16 by 1.36–4.08 percentage points compared to LatentSift, and removing Stage 1 results in higher token consumption (23–43%) and lower Best@16 than LatentSift.
We thank for the reviewer again for providing this meaningful suggestion and have included this in Section 4.3.2 in our revised paper.
Q6 (W6)
We would first like to clarify that the primary objective of this work is to substantially reduce LLM verification token costs while maintaining comparable resolution performance, rather than improving Best@16 over every LLM verifier. The performance differences among LLM verifiers on R2EGym primarily reflect variations in the effectiveness of the verifiers themselves. To further address reviewer's concern, we conducted paired bootstrap analysis with 10,000 task-level resamples at . The results are summarized below.
Table 6. Paired bootstrap results at with 10,000 task-level resamples. Best@16 denotes LatentSift minus baseline.
| Agent | Baseline | Best@16 (%) Ours / Baseline | ΔBest@16 (pp) [95% CI] | Token Saving (%) [95% CI] |
|---|---|---|---|---|
| DeepSWE | DeepSWE-Verifier | 60.06 / 59.26 | +0.80 [-1.01, +2.62] | 62.1 [60.7, 63.6] |
| R2EGym | R2EGym-Verifier | 47.73 / 46.56 | +1.17 [-0.87, +3.27] | 49.1 [47.6, 50.7] |
| R2EGym | DeepSWE-Verifier | 47.73 / 49.70 | -1.96 [-4.45, +0.56] | 49.1 [47.6, 50.7] |
| CWM | CWM-EF | 54.21 / 54.21 | +0.00 [-1.60, +1.61] | 49.8 [48.2, 51.4] |
As shown in Table 6, the paired confidence intervals for Best@16 differences all include zero, while token savings remain consistently substantial (49–62%). These results support our primary claim of reducing verification costs without a statistically significant change in final resolution performance. We also provide these supplementary experiments in Section 4.3.6 in our revised paper.
Rebuttal for reviewer T3o2
Q1 (W1)
Thank you for raising this question. We clarify that LatentSift is a task-adaptive framework rather than a task-agnostic scorer. Different tasks or domains require their own hidden-state banks. However, since agents are commonly adapted to specific tasks through post-training, the bank can be constructed directly during policy training, without additional trajectory generation or encoding cost. Therefore, adapting LatentSift to a new domain does not necessarily introduce additional data generation costs beyond the existing post-training process.
Cross-domain evaluation. Our paper mainly focuses on coding agent and SWE tasks, but we would also be willing to provide with LatentSift's performance on other domains. Here we give experiment results with CWM on AIME 2025, a mathematical reasoning benchmark.
On AIME 2025, the cascade retains 8 of 16 candidates at Stage 1, followed by majority voting and LLM EF tie-breaking. Only Stage 1 varies across Table 1. We sample 64 candidates per problem and average over 200 random 16-candidate subsets. LatentSift achieves 79.49% Best@16, approaching LLM EF (80.19%) while reducing verification tokens by 98.3% (4.7K vs. 284.4K). Its improvement over random filtering (78.26%) further demonstrates the effectiveness of our method beyond SWE tasks. We also add this part in Section B.5 in the appendix of our revised main paper.
Table 1. Cross-domain evaluation on AIME 2025 with the CWM policy at K=16: Best@16 (%) and verification tokens per problem (thousands).
Q2 (W2)
Thank you for raising this question. We acknowledge that hidden states stored in the bank may become outdated if the policy is retrained after deployment. However, we want to clarify that policy training pipelines will generate, evaluate, and process trajectories through policy forward passes. We can leverage the inference during training to extract hidden states and update the bank dynamically, without additional trajectory generation or encoding cost.
In our experiments, we construct the bank using the final policy checkpoint to ensure that its hidden states reflect the latest policy. To quantify the incremental overhead, we profile the runtime of a training epoch, as shown in Table 2. Hidden-state extraction accounts for only 1.01% of training time, while bank construction takes just 2.8 seconds (0.004%), demonstrating that the bank can be updated with minimal overhead during policy training. We also add this discussion in Section B.6 in the appendix of our revised paper.
Table 2. Runtime of one training epoch of CWM and the incremental cost of hidden-state extraction and bank rebuilding.
Robustness to Stale Banks. As discussed above, in practice, we can just use the newest policy to extract hidden states and build banks during training. But we appreciate the reviewer's thought about stale banks. To investigate the question proposed by the reviewer, we additionally evaluate the impact of reusing banks constructed from earlier checkpoints. We also include this part at Section B.7 in appendix of our revised paper.
In this experiment, we use the officially released CWM-SFT checkpoint (before RL post-training) as the earlier policy and the fully post-trained CWM checkpoint used in our main experiments as the current policy. We construct their respective banks by replaying training trajectories through the corresponding checkpoints to extract hidden states. The Native bank contains only current-policy representations, while the Fusion bank combines 80% current-policy and 20% earlier-policy representations.
Table 3. Robustness to earlier-policy trajectories on CWM at K=16.
As shown in Table 3, incorporating 20% hidden states from CWM-SFT changes Best@16 by only 0.26 percentage points (55.22% vs. 54.96%). These results suggest that LatentSift is robust to a moderate proportion of stale representations from an earlier training stage.