- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies supervised fine-tuning of pretrained large language models for sequential decision-making in MDPs, POMDPs, and ambiguous POMDPs. The proposed approach fine-tunes Llama-2-7B using QLoRA on oracle-generated trajectories and conditions action generation on demonstrations from a new task at evaluation time. The theoretical analysis combines an existing prediction-error guarantee for linear self-attention with a performance-difference argument to obtain a policy suboptimality bound. Experiments evaluate the approach on energy-management environments and a Darkroom benchmark.
However, I have substantial concerns about the connection between the theory and the implemented method, the strength of the empirical evidence for the claimed contributions, and several ambiguities in the experimental protocol.
Critical strengths
-
The paper addresses an important problem: adapting pretrained models to sequential decision-making through supervised learning from demonstrations. Examining partial observability and mode ambiguity within a common experimental framework is useful.
-
The reported improvements over the unfine-tuned model are substantial in several settings. For example, the MDP optimality gap at horizon decreases from approximately 43% to 15%.
-
The appendices provide additional evaluations beyond the main experiments, including comparisons with DPT, distribution shifts, changes in demonstration quality, and Darkroom tasks.
Critical weaknesses
- I find that the theoretical analysis has some mismatches with the proposed learning procedure: The analysis assumes access to exact optimal action-value labels, , independent Gaussian features, and population squared-loss training of a single linear self-attention layer. Meanwhile, the implemented method instead minimizes an autoregressive likelihood loss on observation--action--reward trajectories and predicts actions directly. Thus, the supervision, objective, and decision rule differ substantially between the theoretical and empirical settings.
In addition, this work does not establish a reduction connecting trajectory-based action imitation to the analyzed Q-value regression problem. Similarly, identifying observations from sequential trajectories with independent regression samples requires assumptions beyond those ordinarily available in an MDP. The analysis is therefore best viewed as an illustrative surrogate model rather than for the proposed SFT procedure.
- Some theoretical contributions are limited and rely on a strong transfer assumption: Lemma 1 is essentially a restatement of results from Zhang et al. (2024a). Proposition 1 then combines this prediction guarantee with greediness, the performance-difference identity, and Cauchy--Schwarz. Although this is a useful observation, the resulting argument is relatively straightforward.
In my opinion, for the problem, the main challenge is controlling prediction error on states visited by the learned policy and on actions selected by optimization. Equation (7) assumes this control through , without deriving a quantitative bound or providing readily verifiable sufficient conditions. Matching the marginal feature distribution alone does not establish this condition, since action selection and policy-dependent visitation can alter the relevant query distribution. Consequently, the result provides limited insight into when the desired policy guarantee holds.
- The training-length bound does not explain the main training-data scaling experiments: Theoretically, denotes the number of examples within a training prompt, while optimization is performed over a population of tasks. In the experiments, the principal learning curves vary the number of training tasks while keeping the number of trajectories per task fixed. These are different sources of statistical improvement. This paper leaves finite-task generalization error unanalyzed, so the term does not directly explain the reported dependence on the number of training tasks.
In addition, for isotropic covariance ,
= c\left(1+\frac{d+1}{N}\right)I_d.$$ For fixed support data, varying $N$ therefore rescales all predicted Q-values by the same positive factor and leaves the induced greedy policy unchanged. This does not contradict the upper bound, but it shows that reducing the stated training-length bias need not improve policy performance. The interpretation of this term should be discussed in more detail. 4. The experiments do not sufficiently isolate the benefits of language pretraining or in-context adaptation: The main benchmark has ten states, three action labels corresponding to two distinct controls, and a relatively narrow family of dynamics. The DPT comparison is relevant, but differs in architecture, capacity, input representation, and training configuration prevent it from isolating the contribution of language pretraining, as the paper itself acknowledges in Appendix~A. I suggest the author(s) add more baselines to determine whether the reported performance requires a pretrained LLM. Particularly informative comparisons would include a small recurrent or transformer policy trained on the same data, a pooled policy without support demonstrations, and a model-estimation-and-planning baseline. Wrong-task or shuffled demonstrations would also help establish whether the model performs meaningful task-specific adaptation. The Darkroom result is promising, but comparison only against random and oracle policies is insufficient. Since oracle demonstrations provide substantial information about the goal, a simple goal-inference-and-navigation baseline would help contextualize the result.Questions for authors
-
Does each SFT training sequence contain multiple support trajectories followed by a query trajectory? Please provide an explicit example.
-
Does the POMDP oracle condition its belief updates on observed rewards?
-
Which theoretical quantity corresponds to the number of training tasks varied in the experimental learning curves?
2 comments
Critical Weakness 4
Response. Thank you for raising these great questions. First, our current experiments do not isolate the effect of language pretraining. A clean comparison would need to hold architecture, capacity, input representation, and downstream training fixed while varying whether the model was pretrained. DPT and the suggested smaller policies are useful performance baselines, but they do not provide this matched comparison. We will therefore make no causal claim that language pretraining is responsible for the observed performance. Our comparison with the same pretrained LLM without SFT isolates the incremental effect of fine-tuning conditional on that pretrained model, not the effect of pretraining itself.
Second, the new diagnostic support ablations provide direct evidence about in-context adaptation. In diagnostic POMDP, removing support, using support from a mismatched task, or using same-task random support increases the absolute gap by 0.204, 0.135, and 0.139, respectively, relative to matched two-shot oracle support; all paired 95% confidence intervals exclude zero. This result shows that the POMDP policy benefits from matched, informative task context.
Third, we have conducted additional experiments on the diagnostic testing–intervention benchmark to reduce reliance on the original charge setting. It requires history-dependent information gathering, intervention, and safe stopping under partial observability and model ambiguity. At , SFT reduces the mean absolute gap relative to the same pretrained LLM without fine-tuning by 0.180 and 0.210 reward units for POMDP and APOMDP, respectively, with both improvements supported by paired comparisons. We will present Darkroom only as supplementary evidence rather than as a central validation result.
Question 1
Response. Thank you for this question. Each SFT dataset row contains one complete oracle trajectory, not multiple support trajectories followed by a query. For example, an actual charge-MDP training row has the form
<O_0> 2, <A_0> 0, <R_0> -0.02, <O_1> 3, <A_1> 0, <R_1> -0.02, <O_2> 4, <A_2> 1, <R_2> 0.44, <O_3> 3, <A_3> 1, <R_3> 0.33, <O_4> 2, <A_4> 1, <R_4> 0.22
The complete learner-visible observation–action–reward sequence is tokenized and used with a causal next-token loss. Each training task generates oracle trajectories, which become 15 separate dataset rows; they are not concatenated during SFT. MDP observations reveal the state; in POMDPs and APOMDPs, no additional latent-state information is supplied. Belief vectors, numerical task parameters, and APOMDP model identities are not included.
Question 2
Response. Thank you for raising this point. The POMDP oracle uses a transition–observation Bayesian filter to update its belief; the realized reward is not included in this update.
Question 3
Response. Thank you for this question. The generated task count , split 90%/10% between SFT training and validation, determines the number of tasks used in the empirical training objective. That number would naturally enter through a finite-task statistical error measuring how well the empirical training objective approximates the population objective. Our current analysis works directly at the population level and does not derive this additional error term. The theoretical instead counts within-task regression examples and should not be identified with .
Critical Weakness 1
Response. Thank you for your great suggestion. We agree that Proposition 1 does not directly analyze the implemented QLoRA procedure. The theoretical model receives independent Gaussian features with exact oracle-Q labels, trains a single linear-attention layer using a population squared-loss objective, and forms a greedy policy from the predicted Q-values. In contrast, the empirical model receives separate serialized observation–action–reward trajectories, optimizes a whole-sequence autoregressive token loss, and predicts actions directly. The theoretical analysis deliberately uses a tractable setting to study how Q-prediction error can translate into policy-value loss, providing a conceptual reference for the more complex setting studied empirically.
We also agree that KT is not a collection of independent regression samples in the implemented method. For fixed-length charge trajectories, it counts decision points distributed across K separate sequential trajectories; for early-terminating diagnostic trajectories, it is only an upper bound on that count. Those observations are temporally dependent, and the dataset contains oracle actions rather than oracle-Q labels.
We will therefore present Proposition 1 explicitly as an illustrative surrogate or an idealized reference model. We will state that its limited role is conditional: under the linear-MDP, Gaussian-feature, population-training, and on-policy calibration assumptions, a Q-prediction guarantee can be translated into an oracle-relative policy-value bound. We will remove language suggesting that the proposition directly characterizes the QLoRA policy or explains the empirical dependence on the number of training tasks. We will make this qualification explicit in the abstract, the opening of the theoretical analysis section, and the discussion, and clarify that the training and evaluation inputs share a serialization schema while only evaluation concatenates support and query trajectories. The empirical results will be presented as separate evidence rather than as validation of the proposition.
Critical Weakness 2
Response. Thank you for your great insight. Lemma 1 is inherited from Zhang et al. (2024a), and Proposition 1 provides a conditional conversion from their Q-prediction guarantee to a policy-value bound, rather than resolving the on-policy distribution-shift problem. Equation (7) explicitly assumes that prediction error remains controlled on the states and actions selected by the learned policy.
Our training and held-out tasks are sampled from the same task distribution, which controls task-level distribution shift and partially motivates the assumption. However, we agree that matched task sampling does not control the policy-induced state–action distribution: training trajectories are oracle-generated, whereas evaluation trajectories depend on the learned policy’s actions. We therefore do not claim that the empirical generator establishes Equation (7) or that Con=1 applies to our experiments.
In the revised manuscript, we will present Proposition 1 only as an idealized reference model showing what additional on-policy control would be sufficient to translate Q-prediction accuracy into policy performance. We will explicitly identify the inherited and assumed components of the result, remove language suggesting that it explains the empirical task-scaling curves or directly guarantees the implemented QLoRA policy, and identify the derivation of verifiable coverage or concentrability conditions as future work.
Critical Weakness 3
Response. Thank you for these observations. We agree that the theoretical N=KT is different from the generated task count E varied in our experiments, so the N−2 term does not explain the empirical task-scaling curves. We also agree that, under isotropic covariance, changing N can rescale all predicted Q-values without changing the induced greedy policy. We will therefore describe the N−2 term only as a prediction-level bias, remove claims that it necessarily implies policy improvement or explains the empirical scaling results, and present the theory as an idealized reference rather than an explanation of the implemented SFT procedure.