acceptodds
Under review as a conference paper at ICLR 2027

Active Value Function Learning for Long-Horizon Language Model Agents

Abstract

Long-horizon language-model agents must compact their histories to continue beyond the context window, while sparse outcome rewards make it difficult to assign credit to earlier actions. Learned critics provide stepwise value estimates, but these estimates can be inaccurate, while estimating intermediate values with additional continuations from points within a trajectory incurs substantial costs. To mitigate these issues, we introduce FOCUS, an RL method that selectively branches from self-compacted contexts to refine the critic network's value estimates. During RL training each compacted context is treated as a branchable state. The rollout controller facilitates active learning for the critic by sampling continuations from compacted states that will most reduce the critic's estimation error. These refined value estimates then score the segment between consecutive states with a TD(λ) advantage. On legal reasoning (DeonticBench), deep search (BrowseComp-Plus), and code localization (LocBench), FOCUS consistently outperforms GRPO and VinePPO with matched training steps and sampling budget, improving held-out accuracy by 7.2 points over the strongest baseline on BrowseComp-Plus (55.4 vs. 48.2) and by 4.1 points on DeonticBench (56.5 vs. 52.4).

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning for LLM agents, long-horizon agents, credit assignment, value function estimation, compaction, active learning, tree-structured rollouts, PPO, tool-use agents, agentic search

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_S8QBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FOCUS, which selectively branches from compacted contexts to improve value estimation and credit assignment for long-horizon language-model agents. Experiments on legal reasoning, deep search, and code localization show promising performance gains. The approach has practical value, but the ablations do not sufficiently isolate the contribution of active sampling. Differences in experimental configurations and unclear checkpoint selection also require clarification, leaving the central claims insufficiently supported by the current evidence.

Critical strengths

1.The method addresses a practical challenge in long-horizon agent training. Using compacted contexts as branching points and credit-assignment boundaries provides a coherent connection between context management, value estimation, and policy optimization.

2.The results across multiple tasks are promising. Experiments cover legal reasoning, deep search, and code localization. With the 9B model, FOCUS improves over the strongest baseline by 7.2 percentage points on BrowseComp-Plus and 4.1 points on DeonticBench.

3.The paper investigates intermediate mechanisms. Beyond final performance, it examines value-estimation error, critic ranking quality, and alternative compaction strategies, providing useful evidence about the method’s behavior.

Critical weaknesses

1.The benefit of active allocation is not isolated. A controlled comparison against random or uniform branching is missing.

2.Experimental comparisons are confounded. LocBench uses different harness settings across methods, and computational costs are incompletely reported.

3.Evaluation reliability is unclear. Checkpoint selection needs clarification, and repeated runs or uncertainty estimates are missing.

Questions for authors

1.Does active allocation outperform random or uniform branching when the critic, advantage estimator, harness, and rollout budget are held fixed?

2.Were the BrowseComp-Plus checkpoints selected using the validation split or the held-out test split? If the latter, does the reported improvement persist with validation-based selection?

1 comment

We thank the reviewer for their useful comments. We have submitted a revised manuscript with additional experiments and analyses, clearer organization, and improved presentation. We highlighted additional experiments and results in purple. Below, we address your specific questions and refers back to corresponding changes in the revised manuscript.

Active-allocation Ablation (W1, Q1)

We thank the reviewer for this suggestion. We provide further ablation against a random branching + TRACE (Zou et al., 2026) branching in section 3.4 of the updated version. For TRACE, we use the Uniform STAGE 1 and Active STAGE 2 configuration of TRACE paired with our training loss, which corresponds to the reviewer’s requested same budget, same value fusion, same advantage estimator comparison. We attach the ablation result below. On DeonticBench, we show that random got 52.7 and TRACE got 55.9, vs 56.5% by FOCUS. This suggests that actively allocating a rollout budget is contributing to the effectiveness of FOCUS.

MethodAirlineSARA-NSARA-BHardOverall
Random allocation16.6746.0094.2023.9452.66
TRACE27.3342.0092.0329.5855.92
FOCUS21.352.096.426.856.5

A further discussion of the connection between TRACE and FOCUS is presented in the new Appendix I.

Matching Harness for LocBench Experiments (W2)

We thank the reviewer for pointing this out. The mismatch is due to our failure to migrate configurations during an internal harness upgrade. We have now included a matched FOCUS run and used that as our main LocBench result in table 1. FOCUS now shows performance at matched harness configurations. We also keep the original Harness with longer compaction length as an ablation over compaction threshold and the performance seems close, which we included below.

Compaction ThresholdFile@1File @5
2457672.4%80.5%
3276872.4%81.6%

Our updated Figure 4(D) now plots evals for both harness configurations throughout training, and they track each other closely. This suggests that FOCUS is not overly sensitive to the choice of segmentation window.

BrowseComp-Plus Checkpoint Selection (W3, Q2)

For each method, we select the checkpoint with the 83-task validation split and report its accuracy on the 249-task held-out test split. We have clarified our language describing the process.

Uncertainty Estimates Over Results (W3)

We thank the reviewer for asking this question. We include paired-bootstrap CI over FOCUS’ differences against the strongest baseline in the group in this updated submission. For convenience, we also extract the paired CI for each results below.

ModelBenchmark / subsetFOCUS − GRPO90% CI
Qwen3.5-9BDeonticBench: Airline+7.3[+0.67, +14.00]
Qwen3.5-9BDeonticBench: SARA-N−4.0[−16.00, +8.00]
Qwen3.5-9BDeonticBench: SARA-B+3.6[0.00, +7.25]
Qwen3.5-9BDeonticBench: Hard−1.4[−8.45, +5.63]
Qwen3.5-9BDeonticBench: Overall+4.1[+0.59, +7.69]
Qwen3.5-9BBrowseComp-Plus+7.2[+2.41, +12.05]
Qwen3.5-9BLocBench: File@1+2.3[−1.15, +8.05]
Qwen3.5-9BLocBench: File@50.0[−2.30, +2.30]
Qwen3.5-27BDeonticBench: Airline+2.0[−4.67, +8.67]
Qwen3.5-27BDeonticBench: SARA-N+2.0[−6.00, +10.00]
Qwen3.5-27BDeonticBench: SARA-B+2.2[+0.72, +4.35]
Qwen3.5-27BDeonticBench: Hard0.0[−7.04, +7.04]
Qwen3.5-27BDeonticBench: Overall+2.0[−1.18, +5.33]

References

[1] TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning (Zou et al., EMNLP 2026)

Reviewer_dFmBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FOCUS, a reinforcement learning method for long-horizon language model agents that treat compacted contexts as branchable checkpoints for value estimation and credit assignment. FOCUS outperforms GRPO and VInePPO on several benchmarks, showing solid result.

Critical strengths
  1. The paper addresses an interesting problem in long-horizon agentic RL. Reusing context compaction as the point at which trajectories are segmented is a natural and motivated idea.
  2. The experiments are thorough and solid, showing gains across several long-horizon benchmarks.
Critical weaknesses
  1. The method segments the context by the context window limitation, but the segmentation may not retain enough information for the task or may not be a reliable segmentation boundary.
  2. The method works on textual state and may not work in more complex states, such as an embodied agent.
Questions for authors
  1. How sensitive is FOCUS to the compaction segmentation? For example, if compactions are segmented earlier or at different intervals, do the learned values and downstream performance remain stable?
1 comment

We thank the reviewer for their useful comments. We have submitted a revised manuscript with additional experiments and analyses, clearer organization, and improved presentation. We highlighted additional experiments and results in purple. Below, we address your specific questions and refers back to corresponding changes in the revised manuscript.

Context Segmentation Ablation (W1, Q1)

We thank the reviewer for raising this question. In our newly uploaded version, we include a new ablation study regarding compaction threshold. On LocBench, we run FOCUS with a longer compaction threshold (24576 vs 32768) and the File@1 and File@5 differences are minor, as shown below. In fact, in our updated Figure 4(d), evals for both harness configurations throughout training track each other closely.

Compaction ThresholdFile@1File @5
2457672.4%80.5%
3276872.4%81.6%

This suggests that FOCUS is not overly sensitive to the choice of segmentation window. This is consistent with Salle et al. (2026), who find that agent performance is largely insensitive to the compaction threshold.

Limited Scope (W2)

We thank the reviewer for this point. Our scope is long-horizon language-model agents, as reflected in the title, which we believe is an important domain in its own right. Closely related methods (VinePPO, TreePO, TRACE) are likewise developed and evaluated on language tasks. Within this domain, FOCUS covers diverse tasks: legal reasoning, deep research, and code localization. That said, FOCUS’s acquisition rule and value refinement require only critic values at resumable states, which simulators with snapshot/restore also provide. Extending FOCUS to embodied agents is therefore an interesting direction for future work.

References

[1] Salle, Jeanne, et al. "Benchmarks are surprisingly compaction-friendly." arXiv preprint arXiv:2610.31778 (2026) on many modern benchmarks.

Reviewer_zuvLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FOCUS, which selectively branches from compacted agent contexts (checkpoints) to adapt critic estimates through active rollout allocation. The refined values then provide segment-level TD(λ) advantages for policy optimization. The experiments demonstrate improvements on legal reasoning, deep search, and code localization.

Critical strengths
  1. The paper utilizes compaction checkpoints to provide a useful connection between context management and long-horizon credit assignment.
  2. The method achieves improvements across three domains, particularly on BrowseComp-Plus, are supported by critic diagnostics and compaction ablations.
Critical weaknesses
  1. Although the no-refinement ablation supports continuation-based value correction, it does not isolate the allocation rule’s benefit over random or uniform branching with other components fixed.
  2. The baseline comparison is somewhat limited. Comparing with other agentic RL methods such as TRACE [1] would help clarify whether reducing critic error is a better way to allocate rollouts than increasing reward contrast.
  3. LocBench uses different compaction thresholds, response limits, and interaction horizons across methods, confounding attribution of the gains.

[1] TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning

Questions for authors
  1. If the critic, value fusion, advantage estimator, and total sampling budget are kept the same, does FOCUS still outperform random or uniform branching on final task performance?
  2. Tables 8–10 show different compaction thresholds, response limits, and interaction horizons for FOCUS and the baselines on LocBench. Why were these settings chosen, and do the gains hold when these settings are matched?
1 comment

We thank the reviewer for their useful comments. We have submitted a revised manuscript with additional experiments and analyses, clearer organization, and improved presentation. We highlighted additional experiments and results in purple. Below, we address your specific questions and refers back to corresponding changes in the revised manuscript.

Branching Ablation (W1, W2, Q1)

We thank the reviewer for suggesting this highly relevant concurrent work. We provide further ablation against a random branching + TRACE branching in section 3.4. For TRACE, we use the Uniform STAGE 1 and Active STAGE 2 configuration of TRACE paired with our training loss, which corresponds to the reviewer’s requested same budget, same value fusion, same advantage estimator comparison. We attach the ablation result below. On DeonticBench, we show that random got 52.7% and TRACE got 55.9%, vs 56.5% by FOCUS. This suggests that actively allocating a rollout budget is contributing to the effectiveness of FOCUS.

MethodAirlineSARA-NSARA-BHardOverall
Random allocation16.6746.0094.2023.9452.66
TRACE27.3342.0092.0329.5855.92
FOCUS21.352.096.426.856.5

Interestingly, while TRACE and FOCUS motivate the budget allocation problem from different objectives (FOCUS allocates rollouts to learn a critic while TRACE allocates rollout for contrasts), we observe that the first round of two allocators coincides exactly (we have included a discussion in the updated version of the paper as well, appendix I). FOCUS mainly differs in the following three perspectives:

  • Sequential Updating. FOCUS greedily assigns a small number of continuations and updates the Beta posterior after every completed continuation while TRACE allocates its budget at once. This leads to FOCUS prioritizing underexplored in later rounds.
  • Deeper Branching. States created by a FOCUS continuation become candidates themselves, so FOCUS can branch below a branch. This requires a different mean outcome estimator Gˉ\bar{G} as discussed in section 3.3.
  • FOCUS is a critic training framework. Beyond the allocation criteria, FOCUS also proposes bias-corrected value refinement from selectively expanded trees, a reusable critic trained on those targets, and segment-level TD advantages over states induced by context compaction.

Please check appendix I for more in-depth discussion on this topic.

Matching Harness for LocBench Experiments (W3, Q2)

We thank the reviewer for pointing this out. The mismatch is due to our failure to migrate configurations during an internal harness upgrade. We have now included a matched FOCUS run and used that as our main LocBench result in table 1. FOCUS now shows performance at matched harness configurations. We also keep the original Harness with longer compaction length as an ablation over compaction threshold and the performance seems close, which we included below.

Compaction ThresholdFile@1File @5
2457672.4%80.5%
3276872.4%81.6%

Our updated Figure 4(d) now plots evals for both harness configurations throughout training, and they track each other closely. This suggests that FOCUS is not overly sensitive to the choice of segmentation window.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.