acceptodds
Under review as a conference paper at ICLR 2027

Clear the Bottleneck: Learning When Language Agents Are Ready to Act

Abstract

Long-horizon language agents often fail by acting before task-enabling prerequisites have been resolved: answering before evidence is sufficient, executing before the relevant state is known, or committing before required verification is complete. We study these failures through readiness bottlenecks, route-dependent prerequisites whose completion can substantially change the value of the remaining trajectory. The most faithful signal requires actually executing each candidate completion and rolling out the resulting context, which is too expensive for dense training. We therefore introduce a readiness certificate: a compact, grounded semantic representation of a prerequisite's completed state. The certificate acts as a surrogate for the full post-completion context, and Readiness-to-Action (RTA) measures the teacher's induced high-level action response to this surrogate. Its validity is tested against real execution rather than assumed: certificate-induced action shifts align with real-completion shifts (cosine 0.78), and RTA sensitivity correlates with downstream Prerequisite Completion Gain (Spearman across 2,643 audited state–condition pairs). Across the six prerequisite families, real completion yields a +14.8-point macro-average gain in normalized continuation return. We aggregate calibrated RTA signals into a readiness-aware action advantage and distill the resulting KL-regularized policy update alongside outcome RL. The resulting Bottleneck Distillation improves average task success from 42.6% to 46.1% over the strongest non-ours baseline while reducing premature commitment from 9.8% to 5.9% across procedural, stateful, evidence-intensive, compliance, and scientific agent tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language Agents, Agentic Reinforcement Learning, On-Policy Distillation, Long-Horizon Decision Making

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EmzMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper identifies and attacks an interesting, quite niche but application-general problem in LLMS, whereby they take decisive action without gathering all the useful information first. The authors basically claim that this is inefficient to address as a side-effect of standard post-training and can benefit from targeted intervention, and this is the subject of the rest of the paper. They start by formalising the problem, and proposing a “true” way of measuring the usefulness of gathering information about a particular precondition: by rolling out multiple times with and without the precondition having been checked. Then they propose a cheaper “proxy” method for this: just insert a statement that the condition has/hasn’t been checked into the LLM’s context and see how much this influences its next action. Finally, armed with this cheap method, they propose an augmentation of GRPO that additionally rewards taking actions that are identified as useful by this cheap method. This method is quite similar to a type of self-distillation policy optimisation. The authors show that their method helps somewhat in various settings with a small model, and provide various supporting empirical evidence for all of the methods they propose.

Critical strengths
  • The problem is relevant and real, and an instance of a particular behaviour that i may be desirable to correct for in post-training.
  • The authors have correctly identified that some kind of process reward / denser feedback signal would be helpful here (as is generally the case across all sorts of post-training problems today).
  • The experimental sequence is generally very strong and the authors have provided substantial empirical evidence for the validity of most of their claims (albeit at a toy scale).
  • The method proposed, although there are some aspects that feel brittle (e.g. the extraction and identification of particular candidate prerequisites), is generally quite compatible with successful post-training techniques recently (most notably SDPO). The authors have nicely framed their training procedure using an additional KL-divergence loss term.
  • The results are quite promising, though it is hard to tell how significant they are.
Critical weaknesses
  • The presentation of the paper is quite laden with terminology, and therefore dense and difficult to read. It would be an easier read if the authors leant into “simple English” and using the least possible additional buzzwords to describe their methods.
  • I am not convinced about the usefulness of the formalisation. Probably this is not useful as a standalone contribution and is mainly an instrument for conveying the problem and methods to the reader, and I generally find plain English plus some minimal notation is more effective at this.
  • The experiments are somewhat toy, though this may be due to resource constraints.
  • The related work section is currently very brief and does not, in my view, adequately address the relation of these techniques to broader self-distillation literature.
  • The measurement of the action distribution probabilities (length-normalised log-probs) seems a bit iffy to me – I would be interested in more discussion or evidence of the reliability of this method.
  • There is not much discussion, or indeed a limitations section, and I believe this is lacking and the work could be made stronger by being placed into context a bit better.
Questions for authors
  1. Do you think the presentation of the paper could be made more effective with more context and discussion and less dense, coined terminology?
  2. What are your thoughts on the robustness of measuring action probabilities using length-normalised log-probs in the way you do? Did you try any alternatives / do you have any empirical evidence here?
Reviewer_VcxuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Bottleneck Distillation, a method for teaching language agents when to prepare and when to act. It introduces readiness certificates to represent task prerequisites and uses Readiness-to-Action (RTA) signals for policy optimization. Experiments across nine agent benchmarks show improvements in task success and reduced premature actions.

Critical strengths
  • Interesting Problem: The paper addresses an important challenge in long-horizon agents: deciding when prerequisites are satisfied before taking actions.
  • Well-Motivated Method: Readiness certificates provide an effective way to approximate prerequisite completion without expensive environment rollouts during training.
  • Strong Experimental Results: The method achieves consistent improvements across different agent benchmarks and model settings.
  • Detailed Analysis: The paper includes real-completion validation, compute-matched comparisons, and OOD evaluations.
Critical weaknesses
  • Dependency on Structured Prerequisites: The method relies on predefined conditions and executable checkers. Although automatic construction is evaluated, its generalization to open-ended tasks remains unclear.
  • Limited Action Representation: The method operates on predefined high-level actions (Appendix F), which may limit its effectiveness on tasks requiring more fine-grained decisions.
  • Limited Generalization of Readiness Certificates: The approach assumes that prerequisite completion can be represented by changes in certificate states. Its reliability in tasks with complex dependencies remains unclear.
  • Incomplete Efficiency Analysis: The paper reports relative FLOPs but lacks end-to-end training time and a complete analysis of prerequisite construction costs.
Questions for authors
  1. How does the method generalize to open-ended environments without explicit prerequisites or checkers?
  2. Can the authors evaluate more fine-grained action representations?
  3. How does the method handle complex dependencies or inaccurate readiness certificates?
  4. What are the end-to-end training costs, including prerequisite construction?

I am willing to adjust my score according to the rebuttal.

Reviewer_FGEsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on agents which act before the key prerequisites have been fulfilled. It considers a readiness bottleneck to be a prerequisite that is specific to a route and measures its value by creating a fork of the environment, satisfying the prerequisite in fact, and then comparing it with a control that has been given an equal budget. Since this approach is too costly for training, it instead uses a textual "readiness certificate" and employs the log-ratio of the policy's action distribution under the two certificates as a signal, this signal being converted into a KL-regularized target and trained together with GRPO. The authors state that the policy shifts induced by the certificate and those that are real align, that the magnitude of RTA correlates with PCG, and that average success increases from 42.6 to 46.1 whilst premature commitment falls from 9.8% to 5.9% across nine benchmarks.

Critical strengths

The real completion audit does represent a sound methodology since when you branch from a checkpoint and compare actual completion with a matched control you obtain a clear ground truth regarding whether or not it is worthwhile to complete a prerequisite, and the paper deals separately with two questions which are frequently mixed up, namely, whether the surrogate causes the policy to move in the same direction and whether its sensitivity forecasts value.

Critical weaknesses
  1. The surrogate validation is probably to some extent circular and is mainly limited by noise. Table 2 shows the real completion branch with the certificate state SAT together with the control having the state UNSAT. When the teacher sees the updated certificate table, the shift in the real case already includes the certificate shift, and the cosine value of 0.78 is therefore artificially inflated. There is also no null value for the cosine: in a small canonical action space, any typical "ready → commit" shift will obtain a high score. With regard to value fidelity, since the outcomes are binary and there are M = 8 per branch, the per pair PCG standard error is approximately ±25 points relative to a mean of 14.8, which means that a ρ value of 0.66 would imply a very large amount of true heterogeneity.

  2. It is difficult to assign the improvements seen in the headline. The certificates are based on checkers that are constructed from the policy text, the tool schemas, and the workflow documentation, and on tasks similar to those used in τ-bench/MAC-Bench this overlaps with the kind of things that the grader examines. It is not mentioned whether the baseline values in Table 4 include this table.

Questions for authors
  1. Does the teacher see the updated (SAT) certificate table in the real-completion branch? Please provide the cosine value when the table is held at UNSAT and in the case where there is no table, together with a mismatched-condition null (that is, the certificate shift for c_j compared with the real shift for c_i).
  2. What is the split-half reliability of PCG when using four versus four continuations, and what is the correlation between the disattenuated RTA and the PCG? Was the audit rerun at each of the checkpoints listed in Table 14?
  3. Did the baseline entries in Table 4 receive the same condition table? How was the split between training and test tasks carried out for each benchmark?
  4. What is the source of the figure 46.1 (table 12 gives the values 46.5 and 45.8)? Table 8 shows a value of P@3 equal to 0.77 for the setting, whereas Table 13 states 0.79. 5. According to equation 22, the centered RTA is the Q-difference between two hypothetical states, not an advantage at the current state. In what way is equation 14 a policy-improvement step?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.