acceptodds
Under review as a conference paper at ICLR 2027

What Algorithm Can Transformers Learn in Closed-Loop Interaction?

Abstract

The learning algorithm view of in-context learning (ICL) studies how fixed-parameter Transformers learn from supplied examples. Language model agents extend this setting through interaction: their actions determine what evidence enters context next. We call this regime Agentic In-Context Learning (Agentic ICL). Existing studies largely depend on externally supplied contexts or reward-bearing interaction histories, while work on general tool-using agents lacks a learning algorithm view. We represent an interactive learner by its state, update, and decision rule, covering active learning, bandits, and online reinforcement learning (RL). For expressivity, we construct causal Transformers that approximate these algorithms over a finite horizon, and bound the resulting loss in return. For training, we characterize when optimizing process rewards preserves behavior that maximizes the final outcome. Distillation experiments show that Transformers can learn to execute three representative recursions. We then train models from outcome or process rewards without oracle action or state targets, as in RL post-training. Different rewards yield distinct decision policies. Improving the alignment of process supervision with outcome-optimal action preferences shifts learned decisions accordingly. Such behavioral differences persist in a pretrained language model across textual paraphrases.

Primary area:
foundation or frontier models, including LLMs
Keywords:
In-Context Learning, LLM Agent, Transformer Expressivity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qyecOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how Transformers learn from feedback in closed-loop interactions and how training rewards shape their decisions. The authors propose the Agentic ICL framework, which unifies active learning, bandits, and online reinforcement learning through a state-update-decision decomposition. They prove that Transformers can approximate finite-horizon interactive algorithms under regularity conditions and provide a bound on the resulting return error. Experiments show that supervised distillation enables models to imitate three representative algorithms.

Critical strengths
  1. The paper clearly defines the problem and distinguishes the feedback available to the model from the rewards used for training.
  2. The experiments are thorough. In particular, the extensive reward interventions show interpretable changes in decision-making as rewards change. The diagnostics and negative controls are comprehensive: the paper examines state information, update consistency, and the effects of resource changes on decisions, and includes controls that disrupt the correspondence between information and its associated history or target.
  3. The theoretical proofs are rigorous and well developed.
Critical weaknesses
  1. The experiments remain removed from real-world tasks. The primary environment contains only four targets and six tools, and the tools' outcome probabilities are explicitly provided to the model. The natural-language experiments use the same task, so they do not establish applicability to real-world search or coding.
  2. The theory establishes existence but does not show whether standard training converges to such solutions.
  3. The evidence for long-interaction capabilities is limited. Even when six tool calls are allowed, the model makes only 2.73 calls on average, and only 5% of episodes exceed four calls. Thus, the evaluation provides limited coverage of long-horizon decision-making.
Questions for authors
  1. Has the model learned multi-step planning, or a shortcut that approximates the results of planning? Beyond comparing candidate policies and decoding posterior beliefs, could the authors design new tasks or interventions on internal states to rule out simple heuristic strategies? The current evidence primarily supports behavior that is close to Bayes DP.
  2. Can the model solve tasks that require longer interactions? In the current six-call evaluation, even the optimal policy makes only 2.88 tool calls on average. Could the authors construct tasks in which achieving a high return requires interactions longer than those seen during training, to assess genuine long-horizon decision-making?
1 comment

Response to Reviewer qyec

Thank you for the positive assessment of the proofs, reward controls, and diagnostic controls. We address your questions through the additional unknown-dynamics and long-interaction evaluations.

W1. Distance from real-world tool use

We added two evaluations beyond the original BTS protocol. E1 uses reward-only training in a small unknown-transition MDP, with no sampled transition probabilities, posterior states, or teacher actions supplied to the policy. E2 tests autonomous interaction on tasks where achieving the full oracle value requires more than the training maximum of four calls. Across these evaluations, outcome and outcome-preserving training retain greater alignment with the outcome objective, while training on local process rewards produces different decisions and lower outcome returns. The setups and results are reported in E1–E2 below.

Q1. Multi-step planning versus a shortcut

There is already evidence against particular simple explanations. Receding two-query lookahead is included in Table 8. In Table 10, changing only future horizon or budget substantially changes the outcome model's decisions while leaving current evidence and legal actions unchanged. A policy that depends only on the posterior and current action availability cannot account for that response.

E2 adds new tasks and matched histories that distinguish full-horizon decisions from KG, receding two-step lookahead, entropy-based acquisition, and open-loop acquisition. Paired prompts retain the same evidence and current actions while changing future resources, testing heuristics that can also read the resource fields. On full-DP/two-step disagreement histories, outcome and preserving models agree with full DP on 75.375.3\\% and 74.774.7\\% of histories, compared with 26.526.5\\% for the one-step process model. Their mean correct-direction resource responses are 0.1350.135 and 0.1310.131, compared with 0.0150.015 for the one-step model. These measurements show movement toward the oracle's preferred action, beyond sensitivity to a changed resource field. The E2 heuristic table reports the comparisons and five-seed variation.

W3 and Q2. Tasks that require longer interaction than training

E2 selects tasks from the original BTS generator whose exact full-horizon value exceeds the best value achievable with at most four calls by at least 0.030.03. The models start from the initial prompt and choose all actions themselves, with no supplied prefix or forced calls. Each task is evaluated under both the four-call cap and the longer horizon, with the original likelihood-generation law.

At horizon eight, outcome and preserving training achieve mean returns of 0.6010.601 and 0.6020.602, exceeding the four-call oracle value of 0.5470.547, while 87.687.6\\% and 86.886.8\\% of their episodes use more than four calls. The one-step process model returns 0.5240.524, below the capped optimum. Thus, the outcome and preserving models use longer autonomous interactions to attain returns beyond the best four-call policy on this family. The E2 table also reports the horizon-six results and paired return gains.

Reviewer_t829OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies which learning algorithms fixed-parameter Transformers can implement during closed-loop interaction. It constructs causal Transformers that approximate finite-horizon update rules, relates action-distribution error to autonomous return loss, and analyzes when process rewards preserve outcome-optimal actions. Experiments examine supervised execution of representative algorithms and reward-trained behavior in BTS, including a language-model variant.

Critical strengths

S1. The theory connects algorithm execution with autonomous performance. The constructive realization shows how causal Transformers approximate regular finite-horizon recursions. The action-TV-to-return bound then relates policy approximation error to autonomous return loss under the stated assumptions.

S2. The policy comparisons are informative. Comparisons on histories where Bayes DP and KG disagree reveal which candidate better matches learned decisions. Controlled reward interventions test whether that alignment changes with the objective.

Critical weaknesses

W1. The reward-trained evidence is specific to BTS. UCB and Q-learning test supervised execution, while the reward-trained experiments, including the language variant, remain within BTS. My concern is whether the same objective effects hold for different update rules or unknown transition dynamics.

W2. Autonomous length extrapolation remains less well tested. Most longer-horizon test episodes stay within the training interaction length. Deeper supplied-prefix tests evaluate continuation from given histories, which need not be reached autonomously.

Questions for authors

Q1. Could you include a small unknown-transition MDP experiment using reward-only training, without oracle action/state targets?

Q2. How do autonomous return/regret and candidate-policy agreement vary across reward objectives in that setting?

2 comments

Response to Reviewer t829

Thank you for the positive assessment of the connection between execution and autonomous performance, and for the concrete suggestion of an unknown-transition MDP.

W1 and Q1. Reward-only learning beyond BTS

We conducted the requested unknown-transition experiment in E1. A small three-state, two-action MDP has transition probabilities drawn anew for each task and fixed within the interaction. The policy observes its states and actions, but receives neither the sampled transition probabilities nor an estimated model or sufficient-state target. Its parameters remain fixed during each test interaction. We compare terminal outcome reward, an immediate progress score, and outcome-preserving dense reward using matched architectures, training budgets, and five seeds.

This setting requires accumulating state–action transition evidence, in addition to tracking the current physical state. An exact Bayes-adaptive planner is used for evaluation only. The experiment therefore tests objective effects when the observation mechanism must be learned through interaction, rather than supplied as BTS likelihood cards.

Q2. Autonomous performance and candidate-policy agreement in that setting

E1 shows that the objective-dependent contrast persists with unknown transition dynamics. Outcome reward and outcome-preserving dense reward achieve mean autonomous returns of 0.1400.140 and 0.1390.139, compared with 0.1340.134 for immediate progress and a Bayes-adaptive value of 0.1460.146. Their mean Bayes-adaptive agreements on the shared BA/immediate disagreement set are 91.491.4\\% and 90.690.6\\%, compared with 9.39.3\\% for immediate-progress training; the latter agrees with the immediate-progress optimum on 90.790.7\\% of those histories. Thus, outcome-like decisions persist under dense supervision when outcome-optimal choices are preserved, while the local objective produces a different interaction strategy. The E1 table reports five-seed variation, gaps to the Bayes-adaptive optimum, and each condition's own-objective regret in its respective units.

W2. Autonomous length extrapolation

E2 selects tasks from the original BTS generator whose exact full-horizon value exceeds the optimum over all policies restricted to four calls by at least 0.030.03. Models start from the initial prompt, answers remain legal throughout, and every action is chosen autonomously. At horizons six and eight, outcome training achieves mean returns of 0.5800.580 and 0.6010.601, exceeding the respective four-call oracle values of 0.5490.549 and 0.5470.547; 79.379.3\\% and 87.687.6\\% of its episodes exceed four calls. Outcome-preserving training performs similarly, whereas the one-step process model remains below the four-call oracle at both horizons. These results demonstrate use of longer autonomous interactions to improve return on the evaluated family. The E2 table includes all reward conditions, paired capped/full-horizon returns, and call-count summaries.

Thank you for the response and additional experiments. My main concerns have been addressed, and I have no further questions. I will keep my original score.

Reviewer_qKo5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies fixed-parameter Transformers interacting with an environment whose feedback depends on their actions. It formalizes interactive learners through a state–update–decision interface and constructs causal Transformers that approximate regular finite-horizon recursions, with a bound on autonomous return loss. Experiments examine supervised algorithm distillation and reward-driven learning in a structured Bayesian tool-selection environment. Behavioral comparisons and controlled interventions investigate how training objectives affect decision policies. A further experiment fine-tunes Qwen2.5-1.5B-Instruct on a textual version of the environment.

Critical strengths
  1. The state–update–decision framework is a useful organizing abstraction, and the controlled environment makes it operational. Separating evidence updates from action selection provides a common language for active learning, bandits, and online RL. The exactly solvable BTS environment then supports comparisons that aggregate task scores alone cannot provide: prespecified disagreement histories and interventions on future resources, with evidence and current legal actions held fixed, help distinguish decision behavior. This combination of a clear framework and actionable diagnostics is the paper’s strongest contribution.
  2. The empirical controls strengthen the interpretation within the studied setting. The paper includes verifier-specification sweeps, a fixed-reward credit-discount sweep, an outcome-preserving shaping control, and a pretrained language-model experiment with held-out paraphrases. It also appropriately acknowledges that candidate-relative behavioral similarity and state probes do not identify a unique internal algorithm.
Critical weaknesses
  1. Limited theoretical novelty. The proposed framework largely reformulates general history-dependent interactive decision processes. The expressivity result builds on existing approximation techniques, while the process-reward optimality condition is essentially an inclusion of optimal-action sets. Potential-based reward shaping is also well established. Consequently, the theoretical contributions appear relatively incremental.

  2. Restrictive theoretical assumptions. The Transformer construction assumes finite horizons, compact fixed-width representations, smooth decision rules, and finite action spaces with externally provided legality masks. Moreover, the result establishes representational capacity rather than learnability or optimization guarantees, limiting its implications for practical LLM agents.

  3. Limited experimental generality. Most experiments rely on small, synthetic environments with known observation models and short interaction horizons. Even the pretrained LLM experiments use the same structured Bayesian tool-selection task with explicitly provided likelihoods and constrained action outputs. It remains unclear whether the findings transfer to realistic, open-ended agentic tasks with complex tools and feedback.

  4. Limited algorithmic identification and reward generality. Behavioral comparisons are restricted to a predefined candidate family, and the diagnostic histories do not certify the proposed separation condition. State probes provide supporting evidence but cannot establish which internal algorithm is implemented. Furthermore, the process-reward findings rely on hand-designed verifiers and analytically outcome-preserving shaping, leaving robustness to learned or imperfect verifiers insufficiently explored.

Questions for authors
  1. What is the precise technical advance of Theorem 4.1 over the closest existing Transformer algorithm-execution constructions? In particular, can the authors clarify the total resource requirements, including the feed-forward approximators, and identify a restriction or difficulty that prior results do not address?
  2. What new understanding is enabled by the unified framework and behavioral diagnostics? After expressing interactive learning through the state–update–decision interface, which concrete conclusion does the paper establish that existing ICL/ICRL analyses do not readily provide? For example, do the resource interventions or state probes reveal model behavior that cannot be anticipated solely from the optimal policies of the specified training objectives? Please identify the relevant result and explain its significance. This clarification could affect my assessment of the contribution.
6 comments

Response to Reviewer qKo5

Thank you for recognizing the usefulness of the framework, controlled diagnostics, and reward interventions. Your questions help us identify where the paper needs a more precise account of its contribution.

We have added relevant theoretical works listed below to the bibliography and cited them in Section 2 (Related Work), in the paragraph "Transformer expressivity for recursive algorithms." The revised paragraph briefly summarizes their results on universality, fixed-weight algorithm execution and composition, and iterative in-context learning.

W1 and Q1. Theoretical contribution and the Agentic ICL setting

Our paper analyzes the reward-optional execution setting of LLM agents using established Transformer expressivity tools. The theoretical contribution concerns how a fixed-parameter learner acquires and uses evidence through its own actions when training rewards need not appear in context. We do not claim a new general-purpose theoretical tool for Transformers or a new universal-approximation principle. We will further clarify this positioning in the contribution statement.

Several theoretical results provide important precedents for the computational ingredients:

The revised Section 2 now cites all six works and briefly summarizes their results. Our construction implements a specified learner recursion with weights that remain fixed during interaction, and relates its execution accuracy to autonomous performance.

The important distinction from reward-observing ICRL is the agent's information interface. Algorithm Distillation and Decision-Pretrained Transformers condition decisions on state/observation–action–reward histories; Liang and Lai (2026) construct reward-dependent SARSA and actor-critic updates. These formulations analyze adaptation with observed rewards. Their guarantees do not directly address the deployment interface in which a model receives tool evidence while the reward used to train or evaluate it remains outside its context. Agentic ICL makes this separation explicit: returned records can be search results, compiler diagnostics, or measurements, and their distribution may depend on the full interaction history. Reward-observing ICRL is recovered when rewards are included in those records.

This interface captures a central feature of deployed LLM agents: a coding agent uses returned errors to revise its next command, and a search agent uses retrieved content to choose its next query, as illustrated by ReAct and Reflexion. A scalar post-training reward need not be available during this adaptation. Theorem 4.1 constructs a causal implementation of a specified regular state–update–decision recursion under this interface: it routes the previous learner state, emitted action, and returned observation into a shared update module, respects cross-turn dependencies, and approximates the complete legal action distribution uniformly over legal histories. Since an action error can change subsequent evidence, a standard coupling argument then connects this uniform action error to autonomous return error, bounded by BHϵBH\epsilon for a terminal return in [0,B][0,B].

Regarding total resources, the O(H)O(H) statement counts shared update blocks; feed-forward size, internal depth, and weight precision also depend on the learner maps, horizon, and accuracy. We will clarify this dependence beside Theorem 4.1 and in Appendix D.

Q2. What new understanding does Agentic ICL provide?

Agentic ICL separates the feedback that changes the agent's context from the objective that trains its parameters. This distinction matters for LLM agents trained by outcome or process rewards but deployed with tool observations rather than reward annotations. The reward-observing ICRL formulations discussed above do not directly establish how changes in a trainer-side objective affect adaptation under this information interface. Using RL for post-training does not imply that its reward is available to the model during in-context adaptation.

With the task and evidence held fixed, we ask whether changing the training objective changes the representation of accumulated evidence, its use for decisions, or both. This includes evidence-acquisition and stopping decisions that do not arise in supplied-context ICL, where the examples are given. In our reward-trained experiments, no scalar training reward, oracle action, or sufficient-state target is supplied in the policy context.

The trained models make different decisions while retaining similarly decodable evidence and update information. In our controlled Agentic ICL setting, outcome training produces greater sensitivity to future interaction opportunities, while one-step process training produces more myopic decisions. Outcome-preserving dense supervision retains outcome-like behavior. Analytically optimal policies specify preferred actions; they do not determine whether finite training produces this combination of evidence representation and decision behavior. This finding is supported by the matched-history comparison and shaping control (Table 6), paired resource interventions (Table 10), and evidence-related diagnostics (Table 11).

Representing tool evidence does not by itself ensure outcome-aligned interaction. For search or coding agents, the next query, tool call, or stopping decision determines what evidence becomes available later. Our findings motivate evaluating sensitivity to these future opportunities and assessing process supervision by its cumulative action preferences, beyond local verifier accuracy.

W3. Experimental generality and realistic agent tasks

To test whether the objective-dependent findings extend beyond known-likelihood BTS, we conducted reward-only training in a small unknown-transition MDP (E1). The policy receives no sampled transition probabilities, posterior states, or teacher actions. We compare outcome reward, immediate progress reward, and outcome-preserving dense reward; the setup and results are reported below.

W4. Candidate-relative identification, diagnostic separation, and imperfect verifiers

E2 adds diagnostic histories that distinguish full planning from KG, two-step lookahead, entropy acquisition, and open-loop acquisition. On full-DP/two-step disagreement histories, outcome training agrees with full DP on 75.375.3\\% of histories, compared with 26.526.5\\% for one-step process training. These comparisons measure candidate-relative behavioral alignment; Proposition 5.2 additionally requires verified aggregate separation between candidate policies.

E3 uses a separately trained correctness verifier, held fixed across the two credit conditions. Extending credit from zero to one raises mean outcome return from 0.6980.698 to 0.7370.737 and reduces ordering violations from 40.040.0\\% to 8.98.9\\%. Thus, longer credit improves alignment even with an imperfect learned verifier, while the remaining violations quantify its residual objective mismatch.

Thanks for the detailed response, especially the extension to more agent-native tasks. But I am still concerned about the theoretical contribution and the strong assumptions it has to take.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to all reviewers

Response to all reviewers

We thank the reviewers for their careful reading and constructive suggestions.

We conducted three additional experiments: reward-only learning in a small unknown-transition MDP (E1), an autonomous long-interaction evaluation with tests of specified heuristics (E2), and a learned-verifier control (E3). Their setups and results appear below.

AuthorsOpenReview ↗

E1. Reward-only learning with unknown transition dynamics

E1. Reward-only learning with unknown transition dynamics

Setup. We used a three-state, two-action MDP. Actions at s0s_0 either stay there or advance to s1s_1; actions at s1s_1 either return to s0s_0 or reach the goal gg, which resets to s0s_0. Transition probabilities were drawn from public Beta priors and fixed within each task. The policy observed the prior, state roles, current state, horizon, and action–transition history. Sampled probabilities and posterior parameters were excluded, and training used no oracle actions or learner-state targets. Models were trained at T∈{4,6,8}T\in\{4,6,8\} and evaluated at T=8T=8, with return R=T−1∑t=1T1{St=g}R=T^{-1}\sum_{t=1}^T\mathbf1\{S_t=g\}.

Training objectives. We used the symbolic architecture and PPO budget of Appendix G.5 with five paired initialization seeds:

  1. Terminal outcome reward RR, with credit discount one.
  2. Immediate progress f(St)−f(St−1)f(S_t)-f(S_{t-1}), where f(s0)=0f(s_0)=0, f(s1)=1/2f(s_1)=1/2, and f(g)=1f(g)=1, with credit discount zero.
  3. Outcome-preserving dense reward, adding Φ(ht)−Φ(ht−1)\Phi(h_t)-\Phi(h_{t-1}) to the terminal-outcome stream, where Φ(h)=f(S(h))/T\Phi(h)=f(S(h))/T on nonterminal histories and zero at termination, with credit discount one.

Rewards were used for training and were not appended to the transcript.

Evaluation. We evaluated autonomous return on 8,192 fresh tasks per seed and action agreement on a shared set of 1,024 independently selected histories with unique, different Bayes-adaptive (BA) and immediate-progress optima. The gap is VBA−VπV_{\mathrm{BA}}-V_\pi under the task prior. On this diagnostic subset, posterior-mean planning, two-step Bayes-adaptive lookahead, and immediate progress choose the same action; their agreements are combined in the local-reference column.

Policy / training objectiveAutonomous return ↑Gap to BA ↓BA agreement ↑Local-reference agreementOwn-objective optimal mass ↑Own-objective regret ↓
Exact BA reference0.1460.000100.0%0.0%1.0000.000
Outcome reward0.140 ± 0.0020.006 ± 0.00291.4 ± 1.5%8.6 ± 1.5%0.823 ± 0.0190.002 ± 0.001
Immediate progress0.134 ± 0.0020.012 ± 0.0029.3 ± 1.5%90.7 ± 1.5%0.853 ± 0.0170.007 ± 0.001
Outcome-preserving dense reward0.139 ± 0.0020.007 ± 0.00290.6 ± 1.8%9.4 ± 1.8%0.821 ± 0.0180.002 ± 0.001

Results. The objective-dependent contrast persists with unknown dynamics: outcome and preserving training favor the BA reference, while immediate-progress training favors the local reference.

AuthorsOpenReview ↗

E2. Necessary long interaction and tests of specified heuristics

E2. Necessary long interaction and tests of specified heuristics

Task selection. We drew candidate tasks from the original BTS generator using its original prior, likelihood, cost, and discount sampling rules, restricted to the feasible-long resource stratum q≥Hq\geq H. We evaluated horizons H=6H=6 and H=8H=8. Before model evaluation, we selected 2,048 tasks per horizon using only oracle values, requiring

VH∗(x0)−V≤4∗(x0)≥0.03,V_H^{\ast}(x_0)-V_{\leq 4}^{\ast}(x_0)\geq 0.03,

where V≤4∗V_{\leq 4}^{\ast} is the optimum over all adaptive policies allowed at most four calls on the same task.

Autonomous evaluation. We reused the existing outcome, one-step process, and outcome-preserving checkpoints without further training. Each model started from the initial prompt and chose every action, with answers legal throughout and no supplied prefixes or forced calls. We evaluated the same tasks under both the four-call cap and the full horizon; task selection was independent of model performance.

Test horizonPolicy / training objectiveReturn with ≤4 calls ↑Full-horizon return ↑Paired return gain ↑Gap to full oracle ↓Mean callsP(Ncalls>4)P(N_{\mathrm{calls}} > 4)
6Exact full-horizon Bayes DP0.5490.6110.0620.0005.45590.9%
6Outcome reward0.526 ± 0.0070.580 ± 0.0090.054 ± 0.0050.031 ± 0.0095.093 ± 0.03179.3 ± 1.0%
6One-step process reward0.514 ± 0.0070.523 ± 0.0050.009 ± 0.0020.088 ± 0.0053.180 ± 0.01713.3 ± 0.4%
6Outcome-preserving reward0.525 ± 0.0070.578 ± 0.0110.053 ± 0.0060.033 ± 0.0115.055 ± 0.01977.7 ± 0.8%
8Exact full-horizon Bayes DP0.5470.6390.0920.0007.11598.4%
8Outcome reward0.523 ± 0.0080.601 ± 0.0120.078 ± 0.0050.038 ± 0.0126.178 ± 0.02887.6 ± 0.3%
8One-step process reward0.513 ± 0.0100.524 ± 0.0110.011 ± 0.0020.115 ± 0.0113.326 ± 0.03817.5 ± 1.1%
8Outcome-preserving reward0.525 ± 0.0090.602 ± 0.0120.077 ± 0.0050.037 ± 0.0126.149 ± 0.01186.8 ± 0.2%

Heuristic comparisons. We formed a separate diagnostic set for each alternative—KG, receding two-query lookahead, entropy acquisition, or open-loop acquisition—using independent histories where it and full Bayes DP had unique, different maximizing actions with margins of at least 0.010.01. The two-step comparison used histories with at least three calls remaining. Resource interventions changed future horizon or budget while fixing evidence, transcript length, and current legal actions. For pairs whose full-DP optimum switched, the correct-direction response is the increase in probability assigned to the action preferred after the intervention. Pairs with unchanged oracle actions served as controls.

Policy / training objectiveFull-DP agreement on DP/KG disagreement ↑Full-DP agreement on DP/2-step disagreement ↑Full-DP agreement on DP/entropy disagreement ↑Full-DP agreement on DP/open-loop disagreement ↑Correct-direction resource response ↑TV on unchanged-oracle-action control
Outcome reward83.1 ± 2.1%75.3 ± 2.3%86.2 ± 2.4%88.4 ± 2.3%0.135 ± 0.0160.021 ± 0.004
One-step process reward11.4 ± 2.5%26.5 ± 2.1%54.9 ± 2.2%61.2 ± 2.6%0.015 ± 0.0060.018 ± 0.004
Outcome-preserving reward82.9 ± 2.4%74.7 ± 2.3%84.9 ± 2.6%88.4 ± 2.3%0.131 ± 0.0180.024 ± 0.004

Results. Outcome and preserving models exceeded the four-call optimum while using longer autonomous interactions. Their decisions also aligned more closely with full DP on the tested disagreement sets, and resource changes shifted action probabilities toward the oracle's preferred action. These are behavioral comparisons within the tested candidate family.

AuthorsOpenReview ↗

E3. A learned, imperfect local verifier

E3. A learned, imperfect local verifier

Setup. We trained a correctness predictor v^ψ(h,g)∈[0,1]\widehat v_\psi(h,g)\in[0,1] with binary cross-entropy on independent BTS tasks, using random candidate answers and realized correctness labels. Inputs were public task fields, the transcript, resources, and the candidate answer; posterior values and DP scores were excluded. We selected the verifier checkpoint by validation loss and froze it before actor training. Each of five paired verifier/actor seeds used the same verifier for credit discounts zero and one, with the actor architecture and PPO budget matched to Appendix G.5.

Reward and evaluation. With uψ(h)=max⁡gv^ψ(h,g)u_\psi(h)=\max_g\widehat v_\psi(h,g), we used

=\delta^{C(h')}u_\psi(h')-\delta^{C(h)}u_\psi(h),$$ $$\widehat r_\psi(h,\operatorname{answer}(g)) =\delta^{C(h)}\widehat v_\psi(h,g)-\delta^{C(h)}u_\psi(h).$$ Scalar rewards were used only for actor training and were not appended to the policy context. At credit one, the objective is the cost-discounted learned terminal correctness score minus the initial baseline. We computed its optimal action values under the true environment law on frozen $D$ and $D_{BK}$, audited outcome-ordering violations, and evaluated actor alignment and autonomous return. The analytic-verifier endpoints provide references. | Verifier / credit condition | Correctness Brier score ↓ | Outcome-ordering violations on $D$ ↓ | Own-objective optimal mass ↑ | Outcome return ↑ | Bayes-score TV ↓ | KG-score TV ↓ | | --- | --- | --- | --- | --- | --- | --- | | Analytic calibrated verifier, credit 0 | 0.117 | 37.598% | 0.830 ± 0.018 | 0.711 ± 0.009 | 0.218 ± 0.012 | 0.038 ± 0.006 | | Analytic calibrated verifier, credit 1 | 0.117 | 0.0% | 0.796 ± 0.020 | 0.750 ± 0.011 | 0.049 ± 0.007 | 0.178 ± 0.014 | | Learned verifier, credit 0 | 0.136 ± 0.004 | 40.0 ± 1.6% | 0.805 ± 0.022 | 0.698 ± 0.009 | 0.230 ± 0.011 | 0.064 ± 0.008 | | Learned verifier, credit 1 | 0.136 ± 0.004 | 8.9 ± 1.3% | 0.784 ± 0.023 | 0.737 ± 0.013 | 0.071 ± 0.011 | 0.168 ± 0.014 | **Results.** With verifier prediction quality held fixed, full credit increased mean outcome return from $0.698$ to $0.737$ and reduced ordering violations from $40.0\\%$ to $8.9\\%$. The residual violations show that verifier errors still affect optimal action preferences under the learned objective.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.