acceptodds
Under review as a conference paper at ICLR 2027

Large Language Models for Sequential Decision-Making: Improving In-Context Learning via Supervised Fine-Tuning

Abstract

Large language models (LLMs) have shown remarkable in-context learning (ICL) capabilities, yet their potential for sequential decision-making remains underexplored. In this paper, we study the ICL capabilities of LLMs in sequential decision-making settings, including Markov Decision Processes (MDPs), Partially Observable MDPs (POMDPs), and Ambiguous POMDPs (APOMDPs). We fine-tune pretrained LLMs to perform few-shot decision-making directly from offline, oracle-labeled trajectories. Our framework enables flexible imitation of policies through supervised fine-tuning (SFT). Theoretically, we focus on linear MDPs and interpret a fine-tuned attention layer as implicitly estimating optimal Q-functions from in-context data. Building on this interpretation, we derive an end-to-end suboptimality bound for the induced policy that separates the in-context estimation error from the training-length bias. Empirically, across synthetic MDP, POMDP, and APOMDP settings, we find that fine-tuned LLMs achieve substantially smaller optimality gaps than in-context-only and random baselines, with especially large gains in longer-horizon, partially observed, and model-ambiguous environments. Together, these results show that supervised fine-tuning provides an effective route to endowing pretrained LLMs with sequential decision-making capabilities from offline data, which is an important advantage in domains such as healthcare where offline data are abundant.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
sequential decision making, large language model, supervised fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pL7SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies supervised fine-tuning of pretrained large language models for sequential decision-making in MDPs, POMDPs, and ambiguous POMDPs. The proposed approach fine-tunes Llama-2-7B using QLoRA on oracle-generated trajectories and conditions action generation on demonstrations from a new task at evaluation time. The theoretical analysis combines an existing prediction-error guarantee for linear self-attention with a performance-difference argument to obtain a policy suboptimality bound. Experiments evaluate the approach on energy-management environments and a Darkroom benchmark.

However, I have substantial concerns about the connection between the theory and the implemented method, the strength of the empirical evidence for the claimed contributions, and several ambiguities in the experimental protocol.

Critical strengths
  1. The paper addresses an important problem: adapting pretrained models to sequential decision-making through supervised learning from demonstrations. Examining partial observability and mode ambiguity within a common experimental framework is useful.

  2. The reported improvements over the unfine-tuned model are substantial in several settings. For example, the MDP optimality gap at horizon T=15T=15 decreases from approximately 43% to 15%.

  3. The appendices provide additional evaluations beyond the main experiments, including comparisons with DPT, distribution shifts, changes in demonstration quality, and Darkroom tasks.

Critical weaknesses
  1. I find that the theoretical analysis has some mismatches with the proposed learning procedure: The analysis assumes access to exact optimal action-value labels, yi=Qτ∗(si,ai)y_i=Q_\tau^*(s_i,a_i), independent Gaussian features, and population squared-loss training of a single linear self-attention layer. Meanwhile, the implemented method instead minimizes an autoregressive likelihood loss on observation--action--reward trajectories and predicts actions directly. Thus, the supervision, objective, and decision rule differ substantially between the theoretical and empirical settings.

In addition, this work does not establish a reduction connecting trajectory-based action imitation to the analyzed Q-value regression problem. Similarly, identifying KTKT observations from sequential trajectories with KTKT independent regression samples requires assumptions beyond those ordinarily available in an MDP. The analysis is therefore best viewed as an illustrative surrogate model rather than for the proposed SFT procedure.

  1. Some theoretical contributions are limited and rely on a strong transfer assumption: Lemma 1 is essentially a restatement of results from Zhang et al. (2024a). Proposition 1 then combines this prediction guarantee with greediness, the performance-difference identity, and Cauchy--Schwarz. Although this is a useful observation, the resulting argument is relatively straightforward.

In my opinion, for the problem, the main challenge is controlling prediction error on states visited by the learned policy and on actions selected by optimization. Equation (7) assumes this control through ConC_{\mathrm{on}}, without deriving a quantitative bound or providing readily verifiable sufficient conditions. Matching the marginal feature distribution alone does not establish this condition, since action selection and policy-dependent visitation can alter the relevant query distribution. Consequently, the result provides limited insight into when the desired policy guarantee holds.

  1. The training-length bound does not explain the main training-data scaling experiments: Theoretically, N=KTN=KT denotes the number of examples within a training prompt, while optimization is performed over a population of tasks. In the experiments, the principal learning curves vary the number of training tasks while keeping the number of trajectories per task fixed. These are different sources of statistical improvement. This paper leaves finite-task generalization error unanalyzed, so the N−2N^{-2} term does not directly explain the reported dependence on the number of training tasks.

In addition, for isotropic covariance Λ=cId\Lambda=cI_d,

= c\left(1+\frac{d+1}{N}\right)I_d.$$ For fixed support data, varying $N$ therefore rescales all predicted Q-values by the same positive factor and leaves the induced greedy policy unchanged. This does not contradict the upper bound, but it shows that reducing the stated training-length bias need not improve policy performance. The interpretation of this term should be discussed in more detail. 4. The experiments do not sufficiently isolate the benefits of language pretraining or in-context adaptation: The main benchmark has ten states, three action labels corresponding to two distinct controls, and a relatively narrow family of dynamics. The DPT comparison is relevant, but differs in architecture, capacity, input representation, and training configuration prevent it from isolating the contribution of language pretraining, as the paper itself acknowledges in Appendix~A. I suggest the author(s) add more baselines to determine whether the reported performance requires a pretrained LLM. Particularly informative comparisons would include a small recurrent or transformer policy trained on the same data, a pooled policy without support demonstrations, and a model-estimation-and-planning baseline. Wrong-task or shuffled demonstrations would also help establish whether the model performs meaningful task-specific adaptation. The Darkroom result is promising, but comparison only against random and oracle policies is insufficient. Since oracle demonstrations provide substantial information about the goal, a simple goal-inference-and-navigation baseline would help contextualize the result.
Questions for authors
  1. Does each SFT training sequence contain multiple support trajectories followed by a query trajectory? Please provide an explicit example.

  2. Does the POMDP oracle condition its belief updates on observed rewards?

  3. Which theoretical quantity corresponds to the number of training tasks varied in the experimental learning curves?

2 comments

Critical Weakness 1

Response. Thank you for your great suggestion. We agree that Proposition 1 does not directly analyze the implemented QLoRA procedure. The theoretical model receives independent Gaussian features with exact oracle-QQ labels, trains a single linear-attention layer using a population squared-loss objective, and forms a greedy policy from the predicted Q-values. In contrast, the empirical model receives separate serialized observation–action–reward trajectories, optimizes a whole-sequence autoregressive token loss, and predicts actions directly. The theoretical analysis deliberately uses a tractable setting to study how Q-prediction error can translate into policy-value loss, providing a conceptual reference for the more complex setting studied empirically.

We also agree that KTKT is not a collection of independent regression samples in the implemented method. For fixed-length charge trajectories, it counts decision points distributed across KK separate sequential trajectories; for early-terminating diagnostic trajectories, it is only an upper bound on that count. Those observations are temporally dependent, and the dataset contains oracle actions rather than oracle-QQ labels.

We will therefore present Proposition 1 explicitly as an illustrative surrogate or an idealized reference model. We will state that its limited role is conditional: under the linear-MDP, Gaussian-feature, population-training, and on-policy calibration assumptions, a Q-prediction guarantee can be translated into an oracle-relative policy-value bound. We will remove language suggesting that the proposition directly characterizes the QLoRA policy or explains the empirical dependence on the number of training tasks. We will make this qualification explicit in the abstract, the opening of the theoretical analysis section, and the discussion, and clarify that the training and evaluation inputs share a serialization schema while only evaluation concatenates support and query trajectories. The empirical results will be presented as separate evidence rather than as validation of the proposition.

Critical Weakness 2

Response. Thank you for your great insight. Lemma 1 is inherited from Zhang et al. (2024a), and Proposition 1 provides a conditional conversion from their Q-prediction guarantee to a policy-value bound, rather than resolving the on-policy distribution-shift problem. Equation (7) explicitly assumes that prediction error remains controlled on the states and actions selected by the learned policy.

Our training and held-out tasks are sampled from the same task distribution, which controls task-level distribution shift and partially motivates the assumption. However, we agree that matched task sampling does not control the policy-induced state–action distribution: training trajectories are oracle-generated, whereas evaluation trajectories depend on the learned policy’s actions. We therefore do not claim that the empirical generator establishes Equation (7) or that Con=1C_{\mathrm{on}}=1 applies to our experiments.

In the revised manuscript, we will present Proposition 1 only as an idealized reference model showing what additional on-policy control would be sufficient to translate Q-prediction accuracy into policy performance. We will explicitly identify the inherited and assumed components of the result, remove language suggesting that it explains the empirical task-scaling curves or directly guarantees the implemented QLoRA policy, and identify the derivation of verifiable coverage or concentrability conditions as future work.

Critical Weakness 3

Response. Thank you for these observations. We agree that the theoretical N=KTN=KT is different from the generated task count EE varied in our experiments, so the N−2N^{-2} term does not explain the empirical task-scaling curves. We also agree that, under isotropic covariance, changing NN can rescale all predicted Q-values without changing the induced greedy policy. We will therefore describe the N−2N^{-2} term only as a prediction-level bias, remove claims that it necessarily implies policy improvement or explains the empirical scaling results, and present the theory as an idealized reference rather than an explanation of the implemented SFT procedure.

Critical Weakness 4

Response. Thank you for raising these great questions. First, our current experiments do not isolate the effect of language pretraining. A clean comparison would need to hold architecture, capacity, input representation, and downstream training fixed while varying whether the model was pretrained. DPT and the suggested smaller policies are useful performance baselines, but they do not provide this matched comparison. We will therefore make no causal claim that language pretraining is responsible for the observed performance. Our comparison with the same pretrained LLM without SFT isolates the incremental effect of fine-tuning conditional on that pretrained model, not the effect of pretraining itself.

Second, the new diagnostic support ablations provide direct evidence about in-context adaptation. In diagnostic POMDP, removing support, using support from a mismatched task, or using same-task random support increases the absolute gap by 0.204, 0.135, and 0.139, respectively, relative to matched two-shot oracle support; all paired 95% confidence intervals exclude zero. This result shows that the POMDP policy benefits from matched, informative task context.

Third, we have conducted additional experiments on the diagnostic testing–intervention benchmark to reduce reliance on the original charge setting. It requires history-dependent information gathering, intervention, and safe stopping under partial observability and model ambiguity. At E=3200E=3200, SFT reduces the mean absolute gap relative to the same pretrained LLM without fine-tuning by 0.180 and 0.210 reward units for POMDP and APOMDP, respectively, with both improvements supported by paired comparisons. We will present Darkroom only as supplementary evidence rather than as a central validation result.

Question 1

Response. Thank you for this question. Each SFT dataset row contains one complete oracle trajectory, not multiple support trajectories followed by a query. For example, an actual charge-MDP training row has the form

<O_0> 2, <A_0> 0, <R_0> -0.02, <O_1> 3, <A_1> 0, <R_1> -0.02, <O_2> 4, <A_2> 1, <R_2> 0.44, <O_3> 3, <A_3> 1, <R_3> 0.33, <O_4> 2, <A_4> 1, <R_4> 0.22

The complete learner-visible observation–action–reward sequence is tokenized and used with a causal next-token loss. Each training task generates K=15K=15 oracle trajectories, which become 15 separate dataset rows; they are not concatenated during SFT. MDP observations reveal the state; in POMDPs and APOMDPs, no additional latent-state information is supplied. Belief vectors, numerical task parameters, and APOMDP model identities are not included.

Question 2

Response. Thank you for raising this point. The POMDP oracle uses a transition–observation Bayesian filter to update its belief; the realized reward is not included in this update.

Question 3

Response. Thank you for this question. The generated task count EE, split 90%/10% between SFT training and validation, determines the number of tasks used in the empirical training objective. That number would naturally enter through a finite-task statistical error measuring how well the empirical training objective approximates the population objective. Our current analysis works directly at the population level and does not derive this additional error term. The theoretical N=KTN=KT instead counts within-task regression examples and should not be identified with EE.

Reviewer_Y1YWOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies supervised fine-tuning on improving the in-context learning abilities for POMDP and APOMDP on Llama-2-7B using QLoRA on solver-generated trajectories. Theoretically, we focus on linear MDPs and interpret a fine-tuned attention layer as implicitly estimating optimal Q-functions from in-context data.

Critical strengths
  1. The reported improvements look promising. For the MDP with horizon 15, fine-tuning reduces the optimality gap from approximately 43.2% for the base model to 15%.
  2. The paper clearly acknowledges its main weakness, depending on the oracle supervision.
  3. The provide careful confidence intervals across test tasks are helpful to understand the information gains.
Critical weaknesses
  1. The detailed oracle construction is not well specified.
  2. The contributions seem to be incremental, compared with EVOLVE paper.
Questions for authors
  1. What is an exact fine-tuning example? Can you provide a detailed prompt example used in the whole Are multiple trajectories from the same task concatenated during training?
  2. What is the real difference between the current method and EVOLVE, other than different sequential decision making environment, they consider?
2 comments

Critical Weakness 1

Response. Thank you for your great suggestion. In the final manuscript, we will add more discussion in the appendix to distinguish finite-horizon dynamic programming for charge MDPs, belief-grid PBVI for charge POMDPs, belief-grid α\alpha-MEU backward induction for charge APOMDPs, and exact finite-history recursion for diagnostic POMDPs and APOMDPs. We will also specify initial-observation conditioning, belief updates, solver approximation, and rollout evaluation in the appendix.

Critical Weakness 2

Response. Thank you for your great question. We agree that the two methods share the central training recipe of using oracle-generated behavior for SFT, and we do not claim this recipe as a new algorithm. EVOLvE uses behavior generated by exploration algorithms in bandit-style problems and primarily studies exploration and regret. Our question is whether one trajectory-only pretrained-LLM interface can distill solver policies defined under different information structures and decision criteria: state-based finite-horizon MDP policies, belief-based POMDP policies, and ambiguity-aware α\alpha-MEU APOMDP policies. In our work, the LLM receives only few-shot observation–action–reward trajectories and the current observable history at evaluation. MDP observations reveal the state; in POMDPs and APOMDPs, no additional latent-state, belief, or realized-model information is supplied. We evaluate closed-loop oracle-relative performance while varying horizon, observation quality, ambiguity-set size, and ambiguity attitude. Following your feedback, we have conducted additional experiments on the diagnostic benchmark, which requires history-dependent testing, stopping, and intervention decisions. We will therefore frame the contribution as a controlled study of history-dependent and ambiguity-aware policy distillation, and revise the paper to avoid claiming a fundamentally new SFT method.

Question 1

Response. Thank you for your great question. Each SFT dataset row contains one complete oracle trajectory serialized as alternating observation, action, and realized-reward fields. For example, an actual charge-MDP training row has the form

<O_0> 2, <A_0> 0, <R_0> -0.02, <O_1> 3, <A_1> 0, <R_1> -0.02, <O_2> 4, <A_2> 1, <R_2> 0.44, <O_3> 3, <A_3> 1, <R_3> 0.33, <O_4> 2, <A_4> 1, <R_4> 0.22

Here, OtO_t, AtA_t, and RtR_t denote the learner-visible observation, oracle action, and realized reward at decision tt. MDP observations reveal the state; in POMDPs and APOMDPs, no additional latent-state information is supplied. Belief vectors, numerical task parameters, and APOMDP model identities are never included. The 15 trajectories from each task are separate SFT examples.

Question 2

Response. Thank you for this question. At the level of the learning algorithm, the methods are indeed closely related: both use oracle-generated behavior and supervised fine-tuning, and we do not claim a fundamentally different SFT objective or optimizer. The distinction is in what behavior is distilled and how it is evaluated. EVOLvE distills behavior generated by exploration algorithms in bandit-style interactions and focuses on exploration and regret. Our supervision is generated by finite-horizon dynamic-programming, POMDP, and α\alpha-MEU APOMDP solvers. In the partially observed and ambiguous settings, the target action is history-dependent and is defined by a belief-state or ambiguity-aware planning objective, while the LLM in our setting receives only observation–action–reward trajectories and never receives the belief vector, latent state, or realized model identity. We then evaluate the learned policy in closed loop against the formulation-specific oracle and vary observation quality, ambiguity-set size, and ambiguity attitude. The new diagnostic benchmark makes this distinction explicit by requiring the policy to decide when to gather information, when to intervene, and when to terminate safely from its observable history. Thus, our contribution is a controlled study of whether a common pretrained-LLM SFT interface can distill these history-dependent and ambiguity-aware policy classes, not a new general SFT algorithm. We will revise the novelty claim accordingly and do not claim that our experiments isolate a training-recipe advantage over EVOLvE.

Reviewer_Y1YWOpenReview ↗

Thank you for Response

Thank you for the response. I will maintain my score.

Reviewer_AFSkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates supervised fine-tuning (SFT) of pretrained language models for few-shot sequential decision-making. The authors fine-tune Llama-2-7B using QLoRA on serialized trajectory data generated by optimal or near-optimal solvers. At test time, the model performs in-context decision-making without parameter updates, predicting actions based on task-specific support trajectories and the current interaction history.

The approach is evaluated on a synthetic energy-management benchmark across MDP, POMDP, and ambiguous POMDP (APOMDP) formulations. Results demonstrate that the fine-tuned model outperforms both the frozen LLM baseline and an existing Decision Pretrained Transformer (DPT) implementation.

The paper also provides a theoretical analysis of an idealized single linear self-attention layer trained with Gaussian features and oracle optimal Q-values. By synthesizing existing in-context linear regression bounds with an on-policy calibration assumption, the authors establish an upper bound on policy suboptimality.

Critical strengths

The paper clearly presents its core idea and the paper is not hard to read.

Critical weaknesses
  1. Limited Conceptual Novelty

The underlying framework combines standard ingredients: supervised fine-tuning (SFT), in-context learning (ICL), and offline dataset generation from oracle solvers, that are already well-established in the literature. The paper does not sufficiently engage with this prior work to clarify its actual technical novelty. Distilling policies from partially observed and ambiguous solvers serves as a neat empirical demonstration, but it feels like a routine application rather than a fundamental methodological advance.

  1. Weak Connection Between Theory and Practice

The theoretical analysis does not provide meaningful insight into the paper's actual framework. The bounds build directly on standard results in prior work, and the theoretical model relies on restrictive idealized assumptions that do not reflect the action-token SFT training used in practice. Because the theory models prompt length rather than task diversity, it fails to explain why fine-tuning a pretrained LLM actually helps.

  1. Narrow Empirical Scope

The empirical evaluation relies heavily on a small, synthetic domain with a tiny state-action space, making it hard to judge whether the approach scales or generalizes to more complex, realistic decision-making tasks. Without broader environments and proper control baselines, it is difficult to determine what is truly driving the observed gains.

  1. Mismatch Between Evaluation and Methodology

There is a clear disconnect between how the theoretical framework is framed and how it is evaluated. For ambiguous environments, the paper introduces an ambiguity-dependent objective, yet the experiments simply report standard averaged rollout rewards without explaining how these rollout metrics actually align with the targeted objective.

Questions for authors
  1. Could the author discuss more about the core contribution of the paper and how it compares to other related works? This should include the theory, methodology, and empirical findings.

  2. Could the author provide results on more familiar tasks to make the results more convincing?

  3. Which empirical parameter corresponds to the theoretical training-length parameter (NN), and under what concrete data-generation setup is the on-policy calibration assumption in Eq. (7) theoretically justified? Furthermore, is there a direct link between the oracle-QQ regression analysis and the empirical action-token loss in Eq. (1), or should Proposition 1 be viewed purely as an idealized reference model?

3 comments

Critical Weakness 1: Limited Conceptual Novelty

Response. Thank you for this question. Our contribution is a controlled study of the sequential decision-making capabilities of a pretrained transformer in settings with partial observability and model ambiguity. We use a common trajectory interface to distill policies defined by MDP, POMDP, and α\alpha-MEU APOMDP solvers, varying the planning horizon, observation quality, ambiguity-set size, and ambiguity attitude. We agree that oracle-trajectory SFT is not a new training algorithm. We will narrow the novelty claim and consolidate the existing related-work discussion, making the similarities and differences from EVOLvE, DPT, meta-RL, and offline RL more explicit in terms of supervision, policy class, input representation, and evaluation.

Critical Weakness 2: Weak Connection Between Theory and Practice

Response. Thank you for your thoughtful feedback. We agree that the theoretical analysis concerns population Q-value regression by one linear attention layer, not QLoRA action-token SFT. This is an intentional modeling choice, made to preserve tractability while gaining clear insight into how Q-prediction error relates to policy performance under the stated assumptions. We will present the analysis as an idealized reference model and explicitly state that it does not provide a finite-task generalization guarantee for the implemented SFT procedure.

Critical Weakness 3: Narrow Empirical Scope

Response. Thank you for your great suggestion. Following your suggestion, we have conducted additional experiments on a distinct diagnostic testing–intervention benchmark covering MDP, POMDP, and APOMDP settings. Each task contains four latent condition types, and the policy must decide whether to purchase one of two imperfect diagnostic tests, select one of two terminal interventions whose success depends on the latent type, or terminate with a safe action. The MDP setting reveals the latent type, the POMDP setting requires inference from noisy diagnostic observations, and the APOMDP setting additionally introduces ambiguity over intervention-success models. The benchmark is designed so that optimal testing, stopping, and intervention decisions vary with both task parameters and observable history. On 100 canonical held-out tasks at E=3200E=3200, relative to the same pretrained LLM without fine-tuning but given identical support trajectories, SFT reduced the mean absolute optimality gap by 0.098, 0.180, and 0.210 reward units for MDP, POMDP, and APOMDP, respectively. Paired comparisons support the POMDP and APOMDP improvements. These results provide evidence of SFT gains relative to ICL in a second controlled domain.

Critical Weakness 4: Mismatch Between Evaluation and Methodology

Response. Thank you for your great suggestion. Below, we clarify the evaluation metric. We will incorporate this clarification into the final manuscript. In the charge benchmark, the learned policy and oracle use the same fixed episode plan. For MDPs and POMDPs, we estimate each policy's value by its mean discounted rollout return. For APOMDPs, we allocate an equal number of episodes to every candidate model, compute the mean return J^m(π)\widehat J_m(\pi) within each model, and aggregate these returns as

V^αroll(π)=αmin⁡mJ^m(π)+(1−α)max⁡mJ^m(π).\widehat V_{\alpha}^{\mathrm{roll}}(\pi) =\alpha\min_m\widehat J_m(\pi) +(1-\alpha)\max_m\widehat J_m(\pi).

The primary charge metric is a relative oracle-reference rollout gap, with the reference policy and learned policy evaluated using this same score. For charge APOMDPs, this is an episode-level ambiguity-sensitive rollout score rather than the recursively defined α\alpha-MEU value used to construct the oracle policy; the recursive planner value is retained as a separate diagnostic. In the diagnostic benchmark, we instead query the LLM at every reachable observable history and exactly integrate the resulting deterministic policy over the complete history tree; the oracle is evaluated on the same tree. For diagnostic APOMDPs, this exact evaluation applies the recursive α\alpha-MEU criterion at every reachable history, and we report the absolute oracle-relative gap because the Safe action can make the oracle value zero. Both criteria reflect the targeted ambiguity attitude: α=1\alpha=1 emphasizes worst-model performance, α=0\alpha=0 emphasizes best-model performance, and intermediate values interpolate between the two rather than averaging uniformly across models.

Question 1

Response. Thank you for your great suggestion. Below, we summarize the main contributions of our paper.

  1. Methodology. We use a pretrained transformer together with a common observation–action–reward trajectory interface and the same autoregressive SFT objective to distill oracle policies for finite-horizon MDPs, POMDPs, and α\alpha-MEU APOMDPs. At evaluation, the fixed model receives few-shot support trajectories and the current observable history. MDP observations reveal the state; in POMDPs and APOMDPs, no additional latent-state, belief, or realized-model information is supplied. This approach allows us to study how the same distillation procedure behaves as the information structure and decision criterion change.

  2. Relation to prior work. EVOLvE also uses oracle-generated behavior and SFT, but focuses on exploration-generated behavior in bandit-style problems; our focus is the sequential decision-making capabilities of pretrained transformers in finite-horizon tasks under partial observability and model ambiguity. DPT is a close baseline because both methods use supervised learning to imitate oracle actions from in-context trajectory data. However, DPT trains a smaller transformer from scratch on structured numerical inputs, whereas our method fine-tunes a pretrained LLM on text-serialized trajectories. Unlike meta-RL, we do not optimize the policy with an RL objective across task episodes or perform gradient-based test-time adaptation; unlike offline RL, we do not optimize a value or policy objective directly from potentially suboptimal logged behavior. Instead, we assume access to solver-generated supervision and study its distillation into a pretrained LLM. We will make these distinctions explicit without claiming a fundamentally new SFT recipe.

  3. Theory. The analysis builds on the population linear-regression result of Zhang et al. (2024a). Our added step is to translate prediction error into an oracle-relative policy-value bound under an explicit on-policy calibration assumption. It is an idealized single-layer linear-attention reference model: no reduction connects it to the implemented QLoRA action-token loss, and it does not explain finite-task generalization or the benefit of language pretraining.

  4. Empirical findings. The original charge experiments examine how SFT performance changes with horizon, observation quality, ambiguity-set size, and ambiguity attitude. To test whether the findings are specific to that domain, we have conducted additional experiments on a structurally distinct diagnostic testing–intervention benchmark with history-dependent testing, stopping, and intervention decisions. On 100 held-out tasks at E=3200E=3200, SFT reduces the mean absolute gap relative to the same pretrained LLM without fine-tuning but given identical support trajectories by 0.098, 0.180, and 0.210 reward units for MDP, POMDP, and APOMDP, respectively. Paired comparisons support the POMDP and APOMDP improvements. The diagnostic POMDP also gives the clearest evidence of support use: removing support, using support from a mismatched task, or using same-task random support increases the absolute gap by 0.204, 0.135, and 0.139, respectively, relative to matched two-shot oracle support, with all paired 95% confidence intervals excluding zero. These results support SFT gains relative to ICL in controlled synthetic task families.

Question 2

Response. Thank you for this suggestion. Following your suggestion, we have conducted additional experiments on a synthetic diagnostic testing–intervention benchmark that is structurally distinct from the original charge domain. Each task contains four latent condition types, and the policy must decide whether to purchase one of two imperfect diagnostic tests, select one of two terminal interventions, or terminate safely. We instantiate this benchmark as an MDP, POMDP, and APOMDP and evaluate each formulation on 100 held-out tasks. At E=3200E=3200, relative to the same pretrained LLM without fine-tuning but with identical support trajectories, SFT reduces the mean absolute optimality gap by 0.098, 0.180, and 0.210 reward units for MDP, POMDP, and APOMDP, respectively. Paired comparisons support the POMDP and APOMDP improvements. These results provide extra evidence of SFT gains relative to ICL beyond the original charge domain and Darkroom evaluation.

Question 3

Response. Thank you for these questions. First, NN in the theory is the number of labeled regression pairs within a task-level training prompt. For fixed-length charge trajectories, its bookkeeping analogue is KTKT, with K=15K=15 oracle trajectories of length TT per task. Those decision points are distributed across separate trajectory examples, whereas the theoretical parameter NN counts independent oracle-QQ regression pairs within one prompt. Diagnostic trajectories can terminate early, so KTKT is only an upper bound on their total decision-point count per task. The experimental scaling variable EE instead denotes the generated task pool, split 90%/10% between SFT training and validation.

Second, Equation (7) is the additional assumption needed to convert an average Q-prediction guarantee into a policy-performance guarantee. Our training and held-out tasks are sampled from the same task distribution, which controls task-level distribution shift and provides partial motivation for this assumption. Because our training trajectories are oracle-generated while evaluation trajectories depend on the learned policy's actions, matched task sampling alone does not establish this assumption. We therefore treat ConC_{\mathrm{on}} as an assumption rather than claiming Con=1C_{\mathrm{on}}=1.

Third, we do not have a reduction connecting the two learning procedures. The theory assumes exact oracle-QQ labels, population squared loss, and a single linear-attention layer, whereas the implemented method uses serialized observation–action–reward trajectories, autoregressive token loss, and QLoRA fine-tuning of a pretrained LLM. We will therefore present Proposition 1 as an idealized reference model and remove language suggesting that it explains finite-task scaling or directly guarantees the implemented SFT policy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.