acceptodds
Under review as a conference paper at ICLR 2027

Think or Call? Counterfactual Necessity Optimization for Software Agents

Abstract

Software agents must repeatedly decide whether to invoke a tool or continue reasoning with the evidence already available. Terminal task rewards provide limited credit for individual invocations: successful trajectories may contain redundant calls, while failed trajectories may contain essential ones. We introduce Counterfactual Necessity Optimization (), a method that learns tool routing by comparing the downstream value of these two choices. At selected decision states, restores the same agent and repository state into two isolated branches, executes a proposed tool call in one and a reasoning step in the other, and completes both under a shared frozen policy. Their return difference, accounting for task completion, protocol compliance, and optional-call cost, supplies a soft necessity target. A necessity head learns these targets jointly with the policy and guides routing at deployment without counterfactual rollouts. An obligation mask enforces required interactions during training and inference. We establish conditions for unbiased paired estimation and show that uniform error in the optimal tool-versus-reason value gap yields routing regret of at most over horizon . On SWE-bench Verified, improves resolution from 38.8% to 45.8% over outcome-only reinforcement learning while reducing optional calls per resolved issue by 39.1%. Further evaluation on SWE-Bench Pro shows consistent resolution gains. Routing diagnostics show fewer unnecessary invocations and skipped necessary calls, with protocol validity close to mask-only training. These results support invocation-level counterfactual credit for improving task completion and tool efficiency under explicit interaction constraints.

Primary area:
causal reasoning
Keywords:
Reasoning, Agent, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JSrHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when software agents should call a tool rather than continue reasoning. It proposes Counterfactual Necessity Optimization (CNO), which compares a concrete tool call with a reasoning step from the same saved state. Both branches then finish under a frozen continuation policy, and their return difference is used to train a necessity head and guide tool routing. An obligation mask enforces required interactions. On SWE-bench Verified, the paper reports 45.8% resolution compared with 38.8% for outcome-only RL, and 39.1% fewer optional calls per resolved issue.

Critical strengths

(1) The problem is interesting and well motivated. Final task rewards do not directly show whether individual tool calls are useful, and the paired comparison provides a concrete way to study this.

(2) The reported results show higher issue resolution and fewer optional calls per resolved issue on Verified. The mask-only baseline and component ablations are useful for understanding the method.

Critical weaknesses

(1) Some experimental settings are missing.

(2) It is not clear whether the training comparison is fair. CNO uses additional complete rollouts, but only the number of training instances is matched.

(3) The SWE-Bench Pro setup also needs clarification: which version and subsets were evaluated? The reported 1,865 issues include non-public subsets in the original benchmark; how were these evaluated, and what are the per-subset results?

(4) Could you provide a breakdown of call counts and costs by tool type (e.g., search, file reading, editing, testing)? If most of the reduction comes from cheap calls, the cost savings might be limited.

Questions for authors

Please refer to the weakness section.

1 comment

We thank the reviewer for the valuable feedback.

It is not clear whether the training comparison is fair. CNO uses additional complete rollouts, but only the number of training instances is matched.

CNO’s intended benefit is more informative credit assignment per unit of rollout. At a selected state, both branches share the same context and budget. Their return difference directly supervises whether the proposed tool call improves the continuation relative to reasoning.

To test whether this mechanism provides benefits beyond additional sampling, we further conduct comparison that match total generated rollout tokens (original budget 1x together with paired suffix rollouts 2x).

MethodTotal rollout tokensTotal environment interactionsVerified resolved
Outcome RL, original budget1.00×1.00×38.8%
Outcome RL, increased budget3.00×3.24×42.6%
TRM, increased budget3.00×3.22×43.6%
CNO with unpaired value targets3.00×3.21×43.0%
Full CNO3.00×3.20×45.8%

From the results, additional rollouts improve Outcome RL by 3.8%, while CNO retains a 3.2% advantage at the same total rollout-token budget. CNO also exceed TRM by 2.2% and the unpaired-target ablation by 2.8%, without using more environment interactions. This pattern supports the interpretation that CNO benefits from how it allocates rollouts and constructs state-specific supervision, beyond the benefit of additional sampling alone.

The SWE-Bench Pro setup ...

The 1865 refers to the complete original benchmark, whereas our evaluation uses only the original public test set (731 samples), and we will correct the dataset description and label the corresponding results as SWE-Bench Pro Public V1. We will include our recipe for other settings in the revised manuscript.

Could you provide a breakdown of call counts and costs by tool type (e.g., search, file reading, editing, testing)? If most of the reduction comes from cheap calls, the cost savings might be limited.

CNO aims to improve when an agent calls a tool by estimating how the call affects downstream task outcomes. This can reduce redundant calls and recover useful calls that the agent would otherwise skip. On SWE-bench Verified, resolution rises from 38.8% to 45.8%, while optional calls per resolved issue fall by 39.1%. Here an training example from SWE-Gym dask__dask-10422 shows how CNO can identify and avoid expensive, low-value execution calls rather than merely eliminating cheap inspection calls.

Case dask__dask-10422. The agent had written a script to reproduce a bug in Dask’s distributed to_zarr path. The script started Client() at module scope, so spawned workers would try to start new processes during their own initialization.

StageCall-first branchReason-first branch
First decisionRun python3 reproduce_error.py immediately.Continue reasoning. Inspect the script already in context and recognize that Client() needs to run under an if __name__ == "__main__": guard.
ExecutionWorkers repeatedly fail to start. The call times out after 300 seconds and adds 30,121 characters of tool output to the transcript without reaching the target bug.Add the guard before executing the script. The guarded script reaches to_zarr and reproduces the target error: Dask rejects a disk-backed Zarr store as an in-memory store.
CNO training signalThe initial call spends time and context on a startup failure that can be inferred from the script.Prefer reasoning. Avoiding the failed optional execution gives CNO a training signal to favor reasoning at this state.
Reviewer_GzwNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Counterfactual Necessity Optimization (CNO) to train software agents to choose between calling a tool and continuing reasoning. At selected states, it creates two branches from the same saved agent and repository state. One branch executes a proposed tool call, and the other takes a reasoning step. A frozen policy completes both branches, and their return difference provides a training target for the necessity head and routing policy. An obligation mask enforces required tool interactions. Experiments on SWE-bench Verified and SWE-Bench Pro report higher resolution rates and fewer optional calls per resolved issue.

Critical strengths
  1. Comparing the two choices from the same state is a useful way to study whether a tool call is needed. Completing both branches also accounts for its effects on later edits and tests.
  2. The results show improvements in both success and tool use. On Verified, resolution increases from 38.8% with outcome-only RL to 45.8%. The method also improves resolution on Pro.
Critical weaknesses
  1. Each target requires counterfactual suffix rollouts for both choices. Matching training instances or primary interactions does not account for this cost, so the comparison needs a matched total rollout budget.
  2. Theorem 1 assumes a uniform error bound against the optimal value gap. The training target instead uses a frozen suffix policy. The theorem therefore does not directly explain the behavior of the learned estimator, and the paper does not measure this gap.
  3. There is no paired trajectory showing how the call and reasoning branches lead to different repair outcomes. The aggregate plots do not explain which decisions improve, which calls remain unnecessary, or why routing sometimes fails.
  4. The paper reports calibration improvements, but does not show reliability diagrams or predicted-versus-reference gap plots. These would help readers see where the estimates are accurate and where they remain unreliable.
  5. All experiments use Qwen3.5-35B-A3B on repository-repair tasks. The value of a call depends on the model and its proposed action, so the results give limited evidence about whether the gain holds with another backbone.
  6. Sensitivity to K, call cost, and the deployment routing adjustment is not reported.

After rebuttal: The new comparison at the same rollout budget addresses my main concern. CNO still improves over the baselines on Verified after counting the extra rollouts, so I raise my rating to 3. The gains are still shown on one backbone, and their sensitivity to the routing settings remains unclear.

Questions for authors

see weaknesses

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the valuable feedback.

Each target requires counterfactual suffix rollouts for both choices ...

CNO’s intended benefit is more informative credit assignment per unit of rollout. At a selected state, both branches share the same context and budget. Their return difference directly supervises whether the proposed tool call improves the continuation relative to reasoning.

To test whether this mechanism provides benefits beyond additional sampling, we further conduct comparison that match total generated rollout tokens (original budget together paired suffix rollouts).

MethodTotal rollout tokensTotal environment interactionsVerified resolved
Outcome RL, original budget1.00×1.00×38.8%
Outcome RL, increased budget3.00×3.24×42.6%
TRM, increased budget3.00×3.22×43.6%
CNO with unpaired value targets3.00×3.21×43.0%
Full CNO3.00×3.20×45.8%

From the results, additional rollouts improve Outcome RL by 3.8%, while CNO retains a 3.2% advantage at the same total rollout-token budget. CNO also exceed TRM by 2.2% and the unpaired-target ablation by 2.8%, without using more environment interactions. This pattern supports the interpretation that CNO benefits from how it allocates rollouts and constructs state-specific supervision, beyond the benefit of additional sampling alone.

Theorem 1 assumes a uniform error bound against the optimal value gap. The training target instead uses a frozen suffix policy. The theorem therefore does not directly explain the behavior of the learned estimator, and the paper does not measure this gap.

The distinction between the optimal gap and the frozen-policy gap can be addressed by adding a policy-improvement result directly matched to CNO’s training target. The optimal gap is relevant to comparison against an optimal router, whereas the frozen-policy gap is the quantity for improving the current router.

Consider the same binary routing abstraction as Theorem 1, with a fixed tool proposal, fixed within-route generation kernels, and a shared reward function and obligation mask. Let π0\pi_0 be the frozen reference policy and π1\pi_1 an updated router. Write pti(s)p_t^i(s) for their tool-selection probabilities and

Δtπ0(s)=Qtπ0(s,u)−Qtπ0(s,reason).\Delta_t^{\pi_0}(s)=Q_t^{\pi_0}(s,u)-Q_t^{\pi_0}(s,\mathrm{reason}).

Let J(π)J(\pi) denote expected return from the common initial-state distribution. The finite-horizon performance-difference identity gives

J(π1)−J(π0)=∑t=0H−1Es∼dtπ1[(pt1(s)−pt0(s))Δtπ0(s)].J(\pi_1)-J(\pi_0)=\sum_{t=0}^{H-1}\mathbb E_{s\sim d_t^{\pi_1}}\left[\bigl(p_t^1(s)-p_t^0(s)\bigr)\Delta_t^{\pi_0}(s)\right].

Note that the expected reference-policy advantage under the updated router is exactly

Ea∼π1[Atπ0(s,a)]=(pt1−pt0)Δtπ0.\mathbb E_{a\sim\pi_1}[A_t^{\pi_0}(s,a)]=(p_t^1-p_t^0)\Delta_t^{\pi_0}.

Summing these advantages along the updated policy’s trajectories proves the identity. Thus, CNO’s counterfactual target directly characterizes the return consequences of changing the routing probabilities. As such, a useful consequence is a robust guarantee for an explicitly specified signed-logit routing update. Let ft(s)f_t(s) be the learned gap estimate and consider pt1(s)=σ ⁣(logit⁡pt0(s)+αft(s)),α≥0.p_t^1(s)=\sigma\!\left(\operatorname{logit}p_t^0(s)+\alpha f_t(s)\right),\alpha\geq0. In the runtime forced routes remain unchanged. If ft=Δtπ0f_t=\Delta_t^{\pi_0}, every term in the performance-difference identity is nonnegative, giving J(π1)≥J(π0)J(\pi_1)\geq J(\pi_0), without requiring π0\pi_0 to be near-optimal.

More generally, define et(s)=ft(s)−Δtπ0(s).e_t(s)=f_t(s)-\Delta_t^{\pi_0}(s). Then J(π1)−J(π0)≥−α16∑t=0H−1Es∼dtπ1[et(s)2].J(\pi_1)-J(\pi_0)\geq-\frac{\alpha}{16}\sum_{t=0}^{H-1}\mathbb E_{s\sim d_t^{\pi_1}}[e_t(s)^2]. To see this, a negative contribution is possible only when ftf_t and Δtπ0\Delta_t^{\pi_0} have opposite signs. Since the sigmoid is 1/41/4-Lipschitz, ∣pt1−pt0∣≤α4∣ft∣.|p_t^1-p_t^0|\leq \frac{\alpha}{4}|f_t|. On such a sign disagreement, ∣ft∣ ∣Δtπ0∣≤(∣ft∣+∣Δtπ0∣)24=et24.|f_t|\,|\Delta_t^{\pi_0}|\leq\frac{(|f_t|+|\Delta_t^{\pi_0}|)^2}{4}=\frac{e_t^2}{4}. Combining these inequalities proves the bound. This result accommodates stochastic routing and measures error against the frozen-policy gap actually targeted by counterfactual supervision, without any optimal-policy oracle or a uniform error bound over all reachable states.

The connection to the original optimality statement can also be made explicit. For the sign-greedy router πf\pi_f, define

εt=∥ft−Δtπ0∥∞,βt=∥Δtπ0−Δt∗∥∞.\varepsilon_t = \left\lVert f_t - \Delta_t^{\pi_0} \right\rVert_{\infty}, \beta_t = \left\lVert \Delta_t^{\pi_0} - \Delta_t^* \right\rVert_{\infty}.

Theorem 1 then yields

J(π∗)−J(πf)≤∑t=0H−1(εt+βt).J(\pi^*) - J(\pi_f) \leq \sum_{t=0}^{H-1} (\varepsilon_t + \beta_t).

The second term has a precise interpretation. Writing ρt(s,a)=Qt∗(s,a)−Qtπ0(s,a)\rho_t(s,a)=Q_t^*(s,a)-Q_t^{\pi_0}(s,a),

Δt∗(s)−Δtπ0(s)=ρt(s,u)−ρt(s,reason).\Delta_t^*(s)-\Delta_t^{\pi_0}(s)=\rho_t(s,u)-\rho_t(s,\mathrm{reason}).

Therefore, the discrepancy depends on the difference in continuation suboptimality between the two branches.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

There is no paired trajectory showing how the call and reasoning branches lead to different repair outcomes. The aggregate plots do not explain which decisions improve, which calls remain unnecessary, or why routing sometimes fails.

The trajectory-level question is whether a proposed call supplies evidence that changes the repair, repeats evidence already available, or is incorrectly skipped because existing checks appear sufficient. We propose a case-study that exposes the common decision state, the first decision, the resulting patch, and the final preference.

  1. Project-MONAI__MONAI-907, reason first to avoid an expensive test.
StageCall-first branchReason-first branch
Shared stateThe agent has patched sliding-window inference and is about to test a 450 × 344 × 450 input with roi_size=(96, 96, 1).Same state.
First decisionObserved: Run the original reproducer immediately.Counterfactual: Notice that the regression hinges on the size-one ROI axis. Keep roi_size=(96, 96, 1) and reduce the input to 150 × 100 × 150.
VerificationThe first run times out after 300 seconds. The agent then reduces the input and reruns the script, which returns the expected output shape.Run the smaller reproducer first. In the continuation, it returns the same expected shape and the agent submits the same patch.
CNO preferenceThe original run adds an optional execution before the useful check.Prefer reasoning. The difference is the avoidable timeout.
  1. conan-io__conan-13721, call first to resolve uncertainty
StageCall-first branchReason-first branch
Shared stateThe agent has added profile_name to Jinja profile rendering and written a test that uses it as a Conan setting.Same state.
First decisionObserved: Run test_profile_jinja.py.Counterfactual: Continue reasoning about whether the renderer exposes profile_name correctly.
Repair and verificationPytest reports 1 failure and 6 passes. The error identifies an invalid setting. The agent moves the check to conf, then reruns the file and gets 7 passes.A failure path is to attribute the problem to the renderer and spend the remaining repair budget editing it. The original test still uses an invalid setting, so the regression check remains failing.
CNO preferencePrefer calling.Reasoning can still find the mismatch from code. The failure mode is continuing to speculate when a short test can distinguish a test configuration error from an implementation error.
AuthorsOpenReview ↗

Official Comment by Authors (3/3)

  1. pandas-dev__pandas-47927, reason first when verification is blocked
StageCall-first branchReason-first branch
Shared stateThe patch is in place. A foreground rebuild has timed out after 300 seconds, and a background rebuild request has returned HTTP 503.Same state.
First decisionObserved: Retry the rebuild with nohup in the same runtime.Counterfactual: Recognize that changing how the process is launched does not address the runtime’s 503. Preserve the patch and assess whether the runtime can be recovered.
Terminal outcomeThe retry returns 503 again. The trajectory ends without testing the rebuilt extension.With no healthy runtime available, stop retrying and record verification as blocked. The patch remains unverified, and the second failed request is avoided.
CNO preferenceA retry is useful if the service has recovered.Prefer reasoning. An infrastructure failure can limit both branches even when the routing decision improves.

The paper reports calibration improvements, but does not show reliability diagrams or predicted-versus-reference gap plots. These would help readers see where the estimates are accurate and where they remain unreliable.

At test time, the point of comparison is the shared state just before CNO and the base policy disagree on whether to call a tool. Once they take different actions, the repository state and context can diverge, and stochastic continuations can amplify those differences. To isolate the effect of a single routing decision, we can restore that state, execute the proposed call in one branch and a reasoning step in the other, and complete both under the same frozen policy.

Our paper evaluates the practical effect of learned routing at the policy level. On SWE-bench Verified, CNO increases resolution from 38.8% to 45.8% and reduces optional calls per resolved issue by 39.1%. On the routing diagnostics, unnecessary calls fall from 31.81% to 7.02%, and skipped necessary calls fall from 10.17% to 6.78%. Replacing paired supervision with unpaired value targets lowers resolution to 41.4%.

For calibration, the held-out branch states provide a reference gap under a fixed continuation policy. Relative to the unpaired head, CNO reduces ECE from 0.0949 to 0.0462 and gap MAE from 0.2446 to 0.1383.

All experiments use Qwen3.5-35B-A3B on repository-repair tasks ... & Sensitivity to K, call cost, and the deployment ...

CNO’s training objective is independent of the backbone architecture. It estimates the value of the tool call that the current model proposes, relative to continuing to reason from the same state. Changing the backbone changes both the proposed calls and their downstream value, so the training targets are estimated anew for each model. Thus, we expect the method to generalize to other backbones for SWE agents. The size of the gain remains an empirical question.

Cross-backbone comparisons and training-time sweeps are computationally expensive because they require repeated RL runs and paired rollouts on models with sufficient repository-repair capability. We would like to leave this broader evaluation to future work. The sensitivity analysis should distinguish three effects: KK controls the trade-off between estimation precision and rollout cost, the call penalty changes how training weighs optional calls against repair success, and the deployment routing adjustment shifts the operating point of an already trained model. The last can be swept on a fixed checkpoint without retraining.

We are pleased that your concerns have been addressed, and we will make the necessary revisions to the manuscript.

Thank you for your comments!

Reviewer_zdBoOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces Counterfactual Necessity Optimization (CNO), a method that learns tool routing by comparing the downstream value of these two choices. The main contribution is a paired estimator of tool necessity at repeated states of a software-agent trajectory, together with a training objective that combines necessity distillation, policy optimization, and obligation masking. A finite-horizon analysis connects value-gap accuracy to routing regret. The evaluation measures resolution, both directions of routing error, protocol validity, and calibration, with ablations that isolate the method’s components.

Critical strengths
  1. The paper targets a fundamental challenge in training tool-using agents that the terminal outcome rewards provide insufficient credit assignment for individual tool invocations. Successful trajectories may contain redundant calls, while failed trajectories may include genuinely useful interactions.

  2. The paper evaluates CNO on both SWE-bench Verified and SWE-Bench Pro, demonstrating improvements over several relevant baselines.

  3. Many tool-efficiency approaches emphasize reducing the number of invocations, which can obscure whether the agent is becoming overly conservative. This paper explicitly measures both directions of error.

Critical weaknesses
  1. CNO requires two complete suffix rollouts for every sampled decision state, with up to four sampled states per trajectory and (K) samples per branch. This can introduce substantial additional environment interaction and model inference. The experimental setup states that methods share the same training-instance budget, rather than demonstrating that they receive equal total training FLOPs, model-generated tokens, or environment interactions.

  2. The paper includes an unpaired value-head baseline and ablations removing exact state restoration, matched suffix sampling, and full completion. These are useful, but they do not adequately isolate the benefit of the specific CNO learning objective.

  3. The paper writing is tough to parse and the paper can benefit from a structural revision to make it more readable.

Questions for authors
  1. CNO requires two complete suffix rollouts for every sampled decision state, with up to four sampled states per trajectory and (K) samples per branch. This can introduce substantial additional environment interaction and model inference. The experimental setup states that methods share the same training-instance budget, rather than demonstrating that they receive equal total training FLOPs, model-generated tokens, or environment interactions.

  2. The paper includes an unpaired value-head baseline and ablations removing exact state restoration, matched suffix sampling, and full completion. These are useful, but they do not adequately isolate the benefit of the specific CNO learning objective.

  3. The paper writing is tough to parse and the paper can benefit from a structural revision to make it more readable.

1 comment

We thank the reviewer for the valuable feedback.

CNO requires two complete suffix rollouts for every sampled decision state ...

With m≤4m\leq 4 sampled states per trajectory and KK samples per branch, CNO requires 2mK2mK additional suffix completions. Their generation and environment execution incur real costs, irrespective of whether the complete trajectories are retained. The efficiency consideration is that counterfactual evaluation is conditioned on an already-reached state, since both branches inherit the same context and remaining budgets, and continue under a shared frozen policy. Thus, the quantity being estimated is a local tool vs reason value difference. However, we can derive that the number of CNO training samples required exhibits a decreasing trend as training progresses.

We first characterize when this local estimate stabilizes with relatively few samples. For a fixed state ss and frozen policy π\pi, let

\Delta^\pi(s)=\mathbb{E}[D_k(s)\mid s].$$ The conditional variance of a paired difference is $$v_s=\operatorname{Var}(D_k\mid s) =\sigma_u^2(s)+\sigma_r^2(s) -2\operatorname{Cov}(G_k^u,G_k^r\mid s).$$ Holding the initial state fixed removes variation arising from different starting states. When matched randomness induces positive return covariance, pairing further reduces $v_s$ relative to independently sampled branches with the same marginals. More precisely, assume that the pairs are conditionally i.i.d. across $k$, have the correct branch marginals, and satisfy $\lvert D_k(s)-\Delta^\pi(s)\rvert\leq B_s$. For a nonzero value gap $\gamma_s=\lvert\Delta^\pi(s)\rvert$, Bernstein’s inequality implies that the sample mean identifies its sign with probability at least $1-\delta$ whenever $$K\geq \left( \frac{2v_s}{\gamma_s^2} +\frac{2B_s}{3\gamma_s} \right)\log\frac{2}{\delta}.$$ This follows because an incorrect sign requires an estimation error of at least $\gamma_s$. Consequently, states with low paired-return variance and a clearly separated route preference require fewer samples to obtain reliable routing evidence. > The paper includes an unpaired value-head baseline and ablations removing exact state restoration, matched suffix sampling, and full completion. These are useful, but they do not adequately isolate the benefit of the specific CNO learning objective. CNO’s intended benefit is more informative credit assignment per unit of rollout. At a selected state, both branches share the same context and budget. Their return difference directly supervises whether the proposed tool call improves the continuation relative to reasoning. To test whether this mechanism provides benefits beyond additional sampling, we further conduct comparison that match total generated rollout tokens (original budget 1x together with paired suffix rollouts 2x). | Method | Total rollout tokens | Total environment interactions | Verified resolved | | ------------------------------- | -------------------: | -----------------------------: | ----------------: | | Outcome RL, original budget | 1.00× | 1.00× | 38.8% | | Outcome RL, increased budget | 3.00× | 3.24× | 42.6% | | TRM, increased budget | 3.00× | 3.22× | 43.6% | | CNO with unpaired value targets | 3.00× | 3.21× | 43.0% | | Full CNO | 3.00× | 3.20× | 45.8% | From the results, additional rollouts improve Outcome RL by 3.8%, while CNO retains a 3.2% advantage at the same total rollout-token budget. CNO also exceed TRM by 2.2% and the unpaired-target ablation by 2.8%, without using more environment interactions. This pattern supports the interpretation that CNO benefits from how it allocates rollouts and constructs state-specific supervision, beyond the benefit of additional sampling alone. > The paper writing is tough to parse and the paper can benefit from a structural revision to make it more readable. We will make the necessary revisions to the manuscript, and include the experiments (matched budget evaluation) and clarifications (efficient samples in CNO) in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.