acceptodds
Under review as a conference paper at ICLR 2027

LatentSift: Policy-State Filtering for Token-Efficient Verification of Software Engineering Agents

Abstract

Test-time scaling improves software engineering agents by generating multiple candidate trajectories and selecting the best one. Verifying and selecting among these long interactions can consume as many tokens as generation itself. Existing hybrid workflows first apply an LLM-based execution-free (EF) verifier to filter candidates before running tests, which adds another model pass over every trajectory. We introduce LatentSift, a token-free and execution-free filter that replaces this first stage with hidden states the policy already produces while generating the candidates. It represents each candidate through its reasoning, observation, and function-call states, compares them with positive and negative banks of such states collected from successful and unsuccessful trajectories during policy training, and fuses the resulting distance scores with a learned linear score to retain promising candidates for the execution-based stages. On SWE-bench Verified, across three agents and two policy sizes, LatentSift cuts EF-verifier tokens by 66.6–81.0% and total verification tokens, which include test generation, by 49.1–62.1% at K=16, while hybrid Best@16 matches or improves on each agent's reference workflow, rising from 59.26% to 60.06% on DeepSWE-Preview.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large Language Model, Test Time Scaling, Token-Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_T3o2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

LatentSift replaces the LLM verifier in Stage 1 of SWE-agent hybrid verification. It scores each candidate trajectory by the nearest-neighbor distance of the policy's own hidden states (reasoning, observation, and function-call channels) to banks of successful and failed training trajectories, combined with a linear probe. It cuts verification tokens by 49–62% at K=16 on SWE-bench Verified, with Best@16 matching or slightly exceeding the original cascade.

Critical strengths

Reusing hidden states the policy already computed instead of re-reading long trajectories is a clean, practical idea. The paper is clearly written and the token accounting is detailed.

Critical weaknesses
  1. Can it generalize? The banks, the probe, and the three-channel design are all tied to one policy, one SWE scaffold, and SWE-bench Verified. Does a new task or domain, such as web or terminal agents, or other languages, need its own labeled bank and probe, and how does LatentSift perform on any benchmark besides SWE-bench Verified?

  2. Does the bank go stale when the policy is updated? The banks hold states from the policy's own trajectories, so each new model, fine-tune, or checkpoint likely means collecting and labeling the bank again. How much does performance drop with a bank from an older checkpoint or a different policy, and must the bank be rebuilt after every policy update?

  3. The bank's cost isn't counted. Each policy needs about 2,800 outcome-labeled rollouts with test execution and stored all-layer hidden states, which by a rough estimate is about 150M generation tokens. All of it is excluded from the reported savings. What is the full offline cost (tokens, test executions, storage), and how does accuracy scale with smaller banks, such as 100 or 500 trajectories?

  4. Where do the banks come from? The banks are described as "collected during policy training," yet all four policies are third-party checkpoints, and every configuration has exactly 1,383 successful and 1,383 failed trajectories. How were the banks built for policies you did not train?

  5. Most of the savings come from running the expensive verifier on about 4 survivors instead of 16, so any cheap Stage-1 filter would save just as much. What are Best@16 and token cost with random filtering, or with no Stage 1 at all, followed by the Stage-4 verifier?

  6. The gains are small and unsupported statistically. Improvements of 0.3–1.2 points correspond to about 2–6 of 500 tasks, with no confidence intervals. On the R2EGym agent, the DeepSWE-Verifier baseline (49.70%) beats LatentSift (47.73%). Can you report bootstrap confidence intervals and explain the R2EGym result?

Questions for authors
  1. Can it generalize? The banks, the probe, and the three-channel design are all tied to one policy, one SWE scaffold, and SWE-bench Verified. Does a new task or domain, such as web or terminal agents, or other languages, need its own labeled bank and probe, and how does LatentSift perform on any benchmark besides SWE-bench Verified?

  2. Does the bank go stale when the policy is updated? The banks hold states from the policy's own trajectories, so each new model, fine-tune, or checkpoint likely means collecting and labeling the bank again. How much does performance drop with a bank from an older checkpoint or a different policy, and must the bank be rebuilt after every policy update?

  3. The bank's cost isn't counted. Each policy needs about 2,800 outcome-labeled rollouts with test execution and stored all-layer hidden states, which by a rough estimate is about 150M generation tokens. All of it is excluded from the reported savings. What is the full offline cost (tokens, test executions, storage), and how does accuracy scale with smaller banks, such as 100 or 500 trajectories?

  4. Where do the banks come from? The banks are described as "collected during policy training," yet all four policies are third-party checkpoints, and every configuration has exactly 1,383 successful and 1,383 failed trajectories. How were the banks built for policies you did not train?

  5. Most of the savings come from running the expensive verifier on about 4 survivors instead of 16, so any cheap Stage-1 filter would save just as much. What are Best@16 and token cost with random filtering, or with no Stage 1 at all, followed by the Stage-4 verifier?

  6. The gains are small and unsupported statistically. Improvements of 0.3–1.2 points correspond to about 2–6 of 500 tasks, with no confidence intervals. On the R2EGym agent, the DeepSWE-Verifier baseline (49.70%) beats LatentSift (47.73%). Can you report bootstrap confidence intervals and explain the R2EGym result?

6 comments
AuthorsOpenReview ↗

Rebuttal for reviewer T3o2

Q1 (W1)

Thank you for raising this question. We clarify that LatentSift is a task-adaptive framework rather than a task-agnostic scorer. Different tasks or domains require their own hidden-state banks. However, since agents are commonly adapted to specific tasks through post-training, the bank can be constructed directly during policy training, without additional trajectory generation or encoding cost. Therefore, adapting LatentSift to a new domain does not necessarily introduce additional data generation costs beyond the existing post-training process.

Cross-domain evaluation. Our paper mainly focuses on coding agent and SWE tasks, but we would also be willing to provide with LatentSift's performance on other domains. Here we give experiment results with CWM on AIME 2025, a mathematical reasoning benchmark.

On AIME 2025, the cascade retains 8 of 16 candidates at Stage 1, followed by majority voting and LLM EF tie-breaking. Only Stage 1 varies across Table 1. We sample 64 candidates per problem and average over 200 random 16-candidate subsets. LatentSift achieves 79.49% Best@16, approaching LLM EF (80.19%) while reducing verification tokens by 98.3% (4.7K vs. 284.4K). Its improvement over random filtering (78.26%) further demonstrates the effectiveness of our method beyond SWE tasks. We also add this part in Section B.5 in the appendix of our revised main paper.

Table 1. Cross-domain evaluation on AIME 2025 with the CWM policy at K=16K=16: Best@16 (%) and verification tokens per problem (thousands).

Stage 1Best@16Tokens
LLM EF verifier80.19284.4K
Random filtering78.267.2K
LatentSift (ours)79.494.7K

Q2 (W2)

Thank you for raising this question. We acknowledge that hidden states stored in the bank may become outdated if the policy is retrained after deployment. However, we want to clarify that policy training pipelines will generate, evaluate, and process trajectories through policy forward passes. We can leverage the inference during training to extract hidden states and update the bank dynamically, without additional trajectory generation or encoding cost.

In our experiments, we construct the bank using the final policy checkpoint to ensure that its hidden states reflect the latest policy. To quantify the incremental overhead, we profile the runtime of a training epoch, as shown in Table 2. Hidden-state extraction accounts for only 1.01% of training time, while bank construction takes just 2.8 seconds (0.004%), demonstrating that the bank can be updated with minimal overhead during policy training. We also add this discussion in Section B.6 in the appendix of our revised paper.

Table 2. Runtime of one training epoch of CWM and the incremental cost of hidden-state extraction and bank rebuilding.

ComponentTime (s)
Training time of an epoch76,045
Hidden-state extraction (included in training time)765
Bank rebuilding (per epoch)2.8

Robustness to Stale Banks. As discussed above, in practice, we can just use the newest policy to extract hidden states and build banks during training. But we appreciate the reviewer's thought about stale banks. To investigate the question proposed by the reviewer, we additionally evaluate the impact of reusing banks constructed from earlier checkpoints. We also include this part at Section B.7 in appendix of our revised paper.

In this experiment, we use the officially released CWM-SFT checkpoint (before RL post-training) as the earlier policy and the fully post-trained CWM checkpoint used in our main experiments as the current policy. We construct their respective banks by replaying training trajectories through the corresponding checkpoints to extract hidden states. The Native bank contains only current-policy representations, while the Fusion bank combines 80% current-policy and 20% earlier-policy representations.

Table 3. Robustness to earlier-policy trajectories on CWM at K=16K=16.

Stage-1 FilterBankBest@16 (%)Tokens/Task
Distance-basedNative55.22463.5K
Distance-basedFusion54.96457.5K
LLM EF verifier–54.21930.0K

As shown in Table 3, incorporating 20% hidden states from CWM-SFT changes Best@16 by only 0.26 percentage points (55.22% vs. 54.96%). These results suggest that LatentSift is robust to a moderate proportion of stale representations from an earlier training stage.

AuthorsOpenReview ↗

Rebuttal for reviewer T3o2(2)

Q3 (W3)

Thank you for raising this concern. We would like to clarify that bank construction introduces nearly zero additional token cost. The bank can be built online during policy training by reusing the encoding hidden states and test outcomes of training trajectories already produced by the training process. Therefore, the estimated 150M tokens should not be considered an additional cost of our method. The incremental overhead primarily comes from storing the collected hidden states. Since bank construction reuses existing training data rather than requiring additional generation, reducing the bank to 100 or 500 trajectories is not necessary for controlling generation-token cost. But we appreciate the reviewer's thought and are pleased to provide additional results with smaller banks to further investigate the impact of bank size, and we also add this part in Section B.4 in appendix of our revised paper.

Table 4. Bank-size ablation on DeepSWE at K=16K=16. Each pair contains one successful and one failed trajectory. Smaller banks are evaluated over 50 random seeds, with standard deviations in parentheses.

Bank Size (pairs)Best@16 (%)Tokens / Task
10059.44 (0.66)384.6K
50059.47 (0.48)387.9K
1,383 (Full)60.06391.9K
LLM EF verifier59.261,035.1K

As shown in Table 4, reducing the bank to 100 or 500 pairs yields Best@16 of 59.44% and 59.47%, respectively, compared to 60.06% with the full bank. The differences are within 0.62 percentage points, suggesting that LatentSift remains effective even with substantially smaller banks.

Q4 (W4)

Thank you for raising this question. Since our experiments use third-party checkpoints, we construct the banks by replaying training trajectories through the corresponding policy models to extract their hidden states the same as the process during model training. In practical settings, however, banks can be constructed directly during policy training by collecting hidden states from training rollouts, without additional trajectory generation or encoding cost.

We appreciate your reminder and have included this detailed implementation in the revised paper in Section 4.1.

AuthorsOpenReview ↗

Rebuttal for reviewer T3o2(3)

Q5 (W5)

Thank you for raising this question. Stage-1 filtering serves an important purpose in the original workflow: Stages 2–3 require executing candidate patches and running tests in real sandboxes, which is time-consuming and CPU-intensive. Therefore, the original workflow filters 16 candidates down to 8 before execution to reduce this overhead.

Simply using a cheaper filter does not guarantee comparable resolution performance. To investigate this, we evaluate two additional baselines: random Stage-1 filtering and removing Stage 1 entirely, relying on Stage 4 for final selection.

Table 5. Comparison of Stage-1 filtering strategies at K=16K=16. Each entry reports Best@16 (%) / verification tokens per task (K), including test generation.

Stage-1 FilterDeepSWER2EGymCWM
LLM EF (original)59.26 / 1035.1K46.56 / 599.6K54.21 / 930.0K
No Stage 157.44 / 523.4K46.76 / 376.2K53.91 / 665.6K
Random filtering55.98 / 387.2K45.48 / 307.8K52.85 / 461.3K
LatentSift (ours)60.06 / 391.9K47.73 / 304.9K54.21 / 466.8K

As shown in the table, random filtering reduces Best@16 by 1.36–4.08 percentage points compared to LatentSift, and removing Stage 1 results in higher token consumption (23–43%) and lower Best@16 than LatentSift.

We thank for the reviewer again for providing this meaningful suggestion and have included this in Section 4.3.2 in our revised paper.

Q6 (W6)

We would first like to clarify that the primary objective of this work is to substantially reduce LLM verification token costs while maintaining comparable resolution performance, rather than improving Best@16 over every LLM verifier. The performance differences among LLM verifiers on R2EGym primarily reflect variations in the effectiveness of the verifiers themselves. To further address reviewer's concern, we conducted paired bootstrap analysis with 10,000 task-level resamples at K=16K=16. The results are summarized below.

Table 6. Paired bootstrap results at K=16K=16 with 10,000 task-level resamples. Δ\DeltaBest@16 denotes LatentSift minus baseline.

AgentBaselineBest@16 (%) Ours / BaselineΔBest@16 (pp) [95% CI]Token Saving (%) [95% CI]
DeepSWEDeepSWE-Verifier60.06 / 59.26+0.80 [-1.01, +2.62]62.1 [60.7, 63.6]
R2EGymR2EGym-Verifier47.73 / 46.56+1.17 [-0.87, +3.27]49.1 [47.6, 50.7]
R2EGymDeepSWE-Verifier47.73 / 49.70-1.96 [-4.45, +0.56]49.1 [47.6, 50.7]
CWMCWM-EF54.21 / 54.21+0.00 [-1.60, +1.61]49.8 [48.2, 51.4]

As shown in Table 6, the paired confidence intervals for Best@16 differences all include zero, while token savings remain consistently substantial (49–62%). These results support our primary claim of reducing verification costs without a statistically significant change in final resolution performance. We also provide these supplementary experiments in Section 4.3.6 in our revised paper.

Reviewer_4yqfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LatentSift, a lightweight filtering method for software engineering agents that uses the policy's hidden states to rank candidate trajectories without additional LLM verification tokens. It combines distances to successful and unsuccessful trajectories across three channels with a learned linear scorer. Experiments on SWE-bench Verified show substantial reductions in verification tokens while maintaining comparable patch selection accuracy.

Critical strengths
  1. Practical and important problem. Verification can be expensive for long-horizon software engineering agents. Reusing policy hidden states is a sensible approach to reducing this overhead.

  2. Encouraging efficiency results. The method reduces total verification tokens by 49.1–62.1% at K=16 while maintaining comparable performance across the evaluated agents.

  3. Useful empirical analysis. The experiments cover multiple agents and policy sizes. The oracle retention and channel ablations provide useful evidence about the filtering mechanism.

Critical weaknesses
  1. Insufficient ablation of the main components. The paper combines contrastive distance scoring with a learned linear scorer, but does not report their standalone performance. In particular, it is unclear how much the distance-based mechanism contributes beyond the linear scorer. This makes it difficult to assess the necessity of the proposed design.

  2. Incomplete efficiency evaluation. The paper focuses on token savings but does not report actual inference latency, GPU memory overhead, or computational cost. Since the method requires hidden-state extraction, nearest-neighbor retrieval, and a learned scorer, token savings alone do not establish end-to-end efficiency.

Questions for authors

Could the authors report results using only the linear scorer and only the distance-based scorer, under the same verification pipeline?

What are the actual latency and GPU memory overheads of LatentSift compared with the original verification workflow?

Could the authors provide confidence intervals or statistical significance tests for the Best@K differences, accounting for the fact that the 200 draws reuse fixed candidate pools?

1 comment
AuthorsOpenReview ↗

Rebuttal for 4yqf

Q1

Thank you for the suggestion. We conducted an ablation study on DeepSWE-Preview to evaluate the contribution of each scoring branch.

Table 1. Ablation of scoring branches on DeepSWE-Preview at K=16K=16 (%).

Stage-1 FilterBest@16Ret.
LLM EF verifier (reference)59.2697.44
Random filter55.9893.19
Distance branch only58.8596.58
Linear scorer only58.3596.30
LatentSift (distance + linear)60.0697.72

As shown in Table 1, our fusion outperforms both individual scoring branches in terms of Best@16 and oracle retention. It also outperforms the LLM EF verifier and random filtering on both metrics. We also add this part in Section 4.3.2 in our revised main paper.

Q2

Thank you for raising this question. We additionally profiled the computational overhead on a single NVIDIA B200 GPU at K=16K=16. As shown in Table 2, LatentSift requires only 72/48 ms per task for Stage-1 filtering on DeepSWE-Preview/CWM, compared to 37/22.5 s for LLM-based EF. Its peak GPU memory usage is 8.6/10.2 GB, including the hidden-state banks. Bank storage and one-time preparation costs are also reported in the table. We will also include these measurements in the revised paper at Section B.3 in appendix.

Table 2. Computational overhead at K=16K=16 on a single NVIDIA B200. LLM EF GPU memory is reported as model weights + peak KV cache. One-time preparation is bank construction for LatentSift, and vLLM setup for LLM EF.

DeepSWE-Preview
MetricLatentSiftLLM EF (14B)
Bank storage (disk)4.66 GB–
Bank footprint (GPU)7.02 GB–
Scorer parameters0.66M14B
Input per candidate1.9 + 80.7 MB51.7K tokens
One-time preparation8.8 s170–180 s
Stage-1 latency / task72 ms37 s
Peak GPU memory8.6 GB30.1 + 25.1 GB
CWM
MetricLatentSiftLLM EF (8B)
Bank storage (disk)5.62 GB–
Bank footprint (GPU)8.44 GB–
Scorer parameters0.79M8B
Input per candidate1.1 + 48.4 MB43.3K tokens
One-time preparation10.6 s140 s
Stage-1 latency / task48 ms22.5 s
Peak GPU memory10.2 GB16.8 + 21.2 GB

Q3

Thank you for raising this concern. The 200 draws correspond to random permutations of the same fixed candidate pool, following similar settings in DeepSWE: Training a Fully Open-sourced, State-of-the-Art Coding Agent by Scaling RL. The order matters when multiple candidates tie for the last available spot in Stage 1, as it determines which candidates are retained. We added this declaration in our revised paper in Section 4.1. Since candidate ordering can affect the final selection outcome, we average over these permutations to reduce order-dependent variability. We also report 95% confidence intervals using paired bootstrap resampling at the task-instance level.

Table 3. Paired bootstrap analysis at K=16K=16. Δ\DeltaBest@16 denotes LatentSift minus the original cascade (percentage points). The 95% CIs are computed by resampling task instances rather than treating the 200 draws as independent observations.

AgentBest@16 (%) Original / OursΔBest@16 (pp) 95% CIToken Saving (%) 95% CI
DeepSWE59.26 / 60.06+0.80 [-1.01, +2.62]62.1 [60.7, 63.6]
R2EGym46.56 / 47.73+1.17 [-0.87, +3.27]49.1 [47.6, 50.7]
CWM54.21 / 54.210.00 [-1.60, +1.61]49.8 [48.2, 51.4]

As shown in Table 3, all confidence intervals for Best@16 differences include zero, while LatentSift reduces verification tokens by 49–62%, without statistically significant differences in final resolution performance. We also provide these supplementary experiments in Section 4.3.6 in our revised paper.

Independent Candidate Pools. To further address the concern about fixed candidate pools, we additionally evaluate four independently sampled sets of K=8K=8 trajectories per task, using the same evaluation protocol. The results are summarized in Table 4.

Table 4. Best@8 performance and token consumption across four evaluation groups. Results are reported as mean ±\pm sample standard deviation.

MethodSample 1Sample 2Sample 3Sample 4Best@8 (%)Tokens / Issue
LLM EF Verifier52.1052.9053.6054.0053.15 ± 0.83576.3K ± 1.9K
LatentSift53.4052.1752.1753.1052.71 ± 0.63349.4K ± 3.0K

As shown in Table 4, LatentSift achieves a Best@8 of 52.71±0.63%52.71\pm0.63\%, comparable to the LLM EF verifier (53.15±0.83%53.15\pm0.83\%), while reducing verification tokens by 39.4%. These results demonstrate that LatentSift maintains stable resolution performance and consistent token savings across different candidate pools.

We also add this discussion in Section B.8 in appendix of our revised paper.

Reviewer_wsanOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

LatentSift reuses software engineering agents’ hidden states—combining success/failure distance scoring with a lightweight learned linear scorer—to replace the initial token-heavy, LLM-based execution-free (EF) filtering stage. Evaluated on SWE-bench Verified, it reduces verification tokens by approximately 49–62% while broadly maintaining final resolution rates.

Critical strengths
  • Targeted Efficiency: Directly addresses a critical bottleneck in test-time scaling—the heavy token consumption of verifying long agent trajectories. Reusing hidden states computed during generation is an elegant and efficient design strategy.
  • Rigorous Evaluation Setup: Benchmarking across multiple agent frameworks and policy sizes using fixed candidate pools helps effectively isolate the impact of candidate filtering.
  • Granular Validation: Table 3 systematically compares all one-, two-, and three-channel configurations. The full configuration achieves the highest Best@16 on both agents, supporting the joint use of reasoning, observation, and tool-call signals.
Critical weaknesses
  • Main concern: missing branch-level ablations. The experiments do not directly compare distance-only scoring, learned-only scoring, and their combination. Consequently, the contribution of each branch and the additional value over SWIFT-style learned scoring remain unclear.

  • Additional suggestion: Reporting state-extraction and retrieval runtime and memory overhead would strengthen the deployment analysis. This does not undermine the demonstrated token savings.

Questions for authors
  1. Scoring Branch Contributions: How do the individual scoring components (distance-based scoring vs. learned linear scoring) perform independently in terms of oracle retention and final resolution rate? Does the proposed fusion consistently outperform either branch used in isolation?
  2. Computational Overhead: What are the empirical runtime and peak memory requirements introduced by storing, processing, and querying the hidden-state banks during the filtering stage?
  3. Single-Oracle Retention: Could the authors report Stage-1 oracle retention on tasks with exactly one correct candidate, together with the subset size? A comparison with the existing LLM and random filters would show how reliably LatentSift preserves the only viable patch.
  4. Evaluation Stability (K=16K=16): At K=16, does each draw use all 16 distinct trajectories? If so, what varies across the 200 draws, and how are the results averaged to obtain the Best@16 values reported in Table 1?
1 comment
AuthorsOpenReview ↗

Rebuttal for reviewer wsan

Q1

Thank you for the suggestion. We conducted an ablation study on DeepSWE-Preview to evaluate the contribution of each scoring branch.

Table 1. Ablation of scoring branches on DeepSWE-Preview at K=16K=16 (%).

Stage-1 FilterBest@16Ret.
LLM EF verifier (reference)59.2697.44
Random filter55.9893.19
Distance branch only58.8596.58
Linear scorer only58.3596.30
LatentSift (distance + linear)60.0697.72

As shown in Table 1, our fusion outperforms both individual scoring branches in terms of Best@16 and oracle retention. It also outperforms the LLM EF verifier and random filtering on both metrics. We also add this part in Section 4.3.2 in our revised main paper.

Q2

Thank you for raising this question. We additionally profiled the computational overhead on a single NVIDIA B200 GPU at K=16K=16. As shown in Table 2, LatentSift requires only 72/48 ms per task for Stage-1 filtering on DeepSWE-Preview/CWM, compared to 37/22.5 s for LLM-based EF. Its peak GPU memory usage is 8.6/10.2 GB, including the hidden-state banks. Bank storage and one-time preparation costs are also reported in the table. We will also include these measurements in the revised paper. We also add this part in Section B.3 in appendix of our revised paper.

Table 2. Computational overhead at K=16K=16 on a single NVIDIA B200. LLM EF GPU memory is reported as model weights + peak KV cache. One-time preparation is bank construction for LatentSift, and vLLM setup for LLM EF.

DeepSWE-Preview
MetricLatentSiftLLM EF (14B)
Bank storage (disk)4.66 GB–
Bank footprint (GPU)7.02 GB–
Scorer parameters0.66M14B
Input per candidate1.9 + 80.7 MB51.7K tokens
One-time preparation8.8 s170–180 s
Stage-1 latency / task72 ms37 s
Peak GPU memory8.6 GB30.1 + 25.1 GB
CWM
MetricLatentSiftLLM EF (8B)
Bank storage (disk)5.62 GB–
Bank footprint (GPU)8.44 GB–
Scorer parameters0.79M8B
Input per candidate1.1 + 48.4 MB43.3K tokens
One-time preparation10.6 s140 s
Stage-1 latency / task48 ms22.5 s
Peak GPU memory10.2 GB16.8 + 21.2 GB

Q3

Thank you for the suggestion. We additionally evaluate Stage-1 oracle retention on tasks with exactly one correct candidate at K=16K=16, where Stage 1 retains 8 candidates. The results are summarized below.

Table 3. Stage-1 oracle retention (%) on tasks with exactly one correct candidate.

MethodDeepSWE-PreviewR2EGym-14B
Subset size3138
LLM EF verifier83.9 (26/31)78.9 (30/38)
Random filtering50.0 (15/31)50.0 (19/38)
LatentSift (ours)80.6 (25/31)76.3 (29/38)

As shown in Table 3, LatentSift substantially outperforms random filtering and performs comparably to the LLM EF verifier. Notably, the retention gap between LatentSift and the LLM verifier corresponds to only one task instance on each benchmark (25/31 vs. 26/31 and 29/38 vs. 30/38). These results suggest that LatentSift can effectively preserve the unique correct candidate at substantially lower verification cost. We also add this part in Section B.2 in appendix of our revised paper.

Q4

Thank you for raising this question. We first clarify that the 200 draws correspond to random permutations of the same fixed candidate pool, following similar settings in DeepSWE: Training a Fully Open-sourced, State-of-the-Art Coding Agent by Scaling RL. The order matters when multiple candidates tie for the last available spot in Stage 1, as it determines which candidates are retained. We also add this explanation in Section 4.1, the experiment setup part in our revised main paper.

To better address reviewer's concern, we will be pleased to provide with experiment results on four independently sampled sets of K=8K=8 candidate trajectories.

Table 4. Best@8 performance and token consumption across four evaluation groups. Results are reported as mean ±\pm sample standard deviation.

MethodSample 1Sample 2Sample 3Sample 4Best@8 (%)Tokens / Issue
LLM EF Verifier52.1052.9053.6054.0053.15 ± 0.83576.3K ± 1.9K
LatentSift53.4052.1752.1753.1052.71 ± 0.63349.4K ± 3.0K

As shown in Table 4, LatentSift achieves a Best@8 of 52.71±0.63%52.71\pm0.63\%, comparable to the LLM EF verifier (53.15±0.83%53.15\pm0.83\%), while reducing verification tokens by 39.4%. These results demonstrate that LatentSift maintains stable resolution performance and consistent token savings across different candidate pools.

We also add this discussion in Section B.8 in appendix of our revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.