acceptodds
Under review as a conference paper at ICLR 2027

Partial Policy Gradients for Conversational RL in LLMs

Abstract

Learning to act sequentially in an unknown environment can be formulated using reinforcement learning and solved by policy gradients. We propose a natural approach for trading off the bias and variance in policy gradients. The key idea is to optimize for a subset of future rewards: smaller subsets represent simpler policies, which can be learned more efficiently because their empirical gradient estimates have lower variance. Our approach encompasses various policy classes, such as full planning, greedy, and -step lookahead policies. We evaluate the policies empirically in the offline setting on conversational tasks of persona alignment and agentic tool-calling, and directly measure gradient variance across lookahead horizons. Different policies excel in different problems, reflecting their different characteristics and highlighting the importance of our studied trade-off.

Primary area:
reinforcement learning
Keywords:
partial policy gradients, credit assignment, instruction-following, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_P3kyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Partial Policy Gradients (PPG), a framework for controlling the temporal scope of credit assignment in reinforcement learning for multi-turn LLM interactions. Instead of attributing the full trajectory reward to each action, PPG updates the policy using a selected subset of future rewards. The framework encompasses full policy gradients, greedy policy gradients, and K-step lookahead policy gradients. The authors provide a theoretical analysis of gradient concentration and empirically evaluate the approach on persona-consistent dialogue generation and agentic tool-calling tasks. The results suggest that intermediate lookahead horizons can improve performance over full policy gradients and greedy approaches, while reducing gradient estimation variance.

Critical strengths
  1. A unified framework for temporal credit assignment. The paper presents a coherent formulation that unifies full, greedy, and K-step policy gradients by varying the subset of future rewards assigned to each action. The theoretical concentration analysis provides a useful perspective on the trade-off between statistical efficiency and the scope of credit assignment.
  2. Promising empirical results supported by variance measurements. The experiments cover multiple conversational domains and two LLM architectures. The authors evaluate both rollout-level and step-level consistency and provide direct measurements of gradient variance across different lookahead horizons. The results demonstrate that intermediate horizons can substantially outperform full policy gradients in several settings.
Critical weaknesses
  1. Limited methodological novelty and insufficient comparison with closely related approaches. The central idea of restricting the temporal scope of reward attribution is closely related to existing work on multi-step returns and temporal credit assignment. In particular, Segment Policy Optimization (SPO) has already explored segment-level credit assignment for LLM reinforcement learning. Although PPG provides a more general formulation and introduces overlapping K-step reward windows, the empirical evaluation does not include a direct comparison with SPO or other closely related temporal credit assignment methods. This makes it difficult to determine whether the proposed formulation offers substantial advantages beyond existing approaches.
  2. Insufficient evidence for the proposed bias–variance trade-off and its generalizability. The paper attributes much of the performance improvement to reduced gradient variance, but the empirical variance analysis is conducted at a single training checkpoint. Since changing K also changes the optimization objective, improved downstream performance cannot be conclusively attributed to variance reduction alone. Furthermore, the main experiments focus on tasks with decomposable per-step consistency rewards and use only one random seed. The K-step trade-off is evaluated primarily in offline training, while PPO is trained online, limiting the fairness of that comparison. Additional evaluations on tasks involving delayed or sparse rewards would strengthen the claim that the proposed approach generalizes to broader long-horizon agent settings.
Questions for authors
  1. How does K-Step-PG compare with Segment Policy Optimization and other temporal credit assignment methods under matched models, datasets, and training budgets?
  2. Can the authors provide additional experiments isolating the effects of gradient variance reduction from the changes in the optimization objective introduced by varying K? For example, measurements across multiple training checkpoints would be informative.
  3. How does PPG perform when early actions strongly influence rewards appearing many steps later, particularly in tasks with delayed or sparse rewards? Would the optimal K differ substantially in these settings?
  4. Can the authors provide multi-seed evaluations and a more controlled comparison between online and offline policy optimization?
2 comments

We thank the reviewer for the constructive assessment and for recognizing the value of the unified temporal credit-assignment framework, the concentration analysis, and the empirical improvements across conversational domains and model architectures.

1. Comparison with Segment Policy Optimization and related methods

We agree that Segment Policy Optimization (SPO; Guo et al., 2025) is closely related to our work and warrants a precise comparison.

Our manuscript already discusses segment-based policy optimization in Section 6 and presents a segment-policy instantiation of the reward-subset framework in Appendix A.5. However, we agree that formal representability is not equivalent to a controlled empirical comparison. There is an important distinction in the structure of temporal credit assignment. In segment-based attribution, rewards are assigned to actions within a designated segment, with the partition boundaries determining which rewards contribute to each action. In contrast, K-Step-PG uses overlapping, sliding reward windows, where each action receives credit from up to K subsequent rewards, independently of a predefined segmentation. For example, adjacent actions can share several future rewards under K-Step-PG, whereas a fixed-segment policy changes its attribution pattern at segment boundaries.

Furthermore, the full SPO training procedure involves optimization and segmentation choices beyond selecting a reward-dependency set. Our conversational trajectories do not provide a canonical semantic partition. Consequently, a matched SPO comparison would require introducing and controlling an additional segmentation procedure, while also aligning the optimizer, training data, and experimental budget.

2. Separating gradient variance reduction from changes in the optimization objective

We agree that modifying K changes both the temporal scope of the optimization objective and the corresponding stochastic gradient estimator. Consequently, improved downstream performance cannot be attributed exclusively to variance reduction. Our framework studies precisely this trade-off: restricting future-reward dependencies may yield statistically easier-to-estimate gradients, while potentially sacrificing useful longer-range dependencies. Theorem 4 characterizes the associated worst-case concentration behavior under stated assumptions. It does not establish that every shorter-horizon policy is superior or that the measured performance gains are caused solely by lower variance.

To strengthen the empirical evidence, we extended the gradient-variance measurements to multiple checkpoints. Using Qwen3-8B on Therapy, we evaluated steps 100, 250, and 400 following the existing Appendix D.7 variance-estimation protocol.

PolicyStep 100Step 250Step 400
GreedyPG (K=1)0.063 ± 0.0040.044 ± 0.0030.038 ± 0.002
2-Step-PG0.095 ± 0.0080.087 ± 0.0050.078 ± 0.004
3-Step-PG0.181 ± 0.0160.153 ± 0.0120.122 ± 0.013
4-Step-PG0.244 ± 0.0210.199 ± 0.0180.164 ± 0.017
5-Step-PG0.319 ± 0.0270.283 ± 0.0220.268 ± 0.022
Full PG0.590 ± 0.0410.505 ± 0.0320.463 ± 0.025

At every checkpoint, the measured variance increases with the temporal attribution horizon. These results extend the original Table 6 finding beyond a single training checkpoint and demonstrate that the empirical variance–horizon relationship persists across the evaluated training stages. However, because the analysis uses checkpoints from the respective trained policies, it does not isolate the attribution horizon while holding model parameters fixed. We therefore view the new results as evidence of a persistent empirical association, rather than definitive causal identification of the source of downstream improvements. We will add this analysis to Appendix D.1, clarify the protocol in Appendix D.7, and revise Section 5.2 to distinguish the concentration mechanism from the effects of optimizing different reward objectives.

3. Generalization to delayed and sparse rewards

We agree that environments with sparse or delayed rewards are an important setting for evaluating temporal credit assignment. Our current benchmarks provide dense, decomposable per-turn prompt-consistency rewards, and we do not claim that their results directly establish performance under arbitrary delayed terminal rewards. If an action has an important consequence that is observed only many turns later, a hard K-step window shorter than that delay may exclude the relevant future reward from the action's direct training signal. In such environments, longer-horizon policies or different attribution mechanisms may be preferable, despite the statistical-efficiency advantages associated with shorter reward windows. This is consistent with our empirical observation that the best tested horizon depends on the application.

In the original experiments, 3-Step-PG performs best on Therapy rollout consistency, whereas 5-Step-PG performs best on agentic tool-calling rollout consistency. Full PG is also preferable in some additional settings, including the Education domain in Table 7. These findings support task-dependent horizon selection. However, they are not substitutes for a controlled sparse-reward or delayed-outcome benchmark. We will clarify this boundary in Section 5.2 and the limitations section, and identify delayed-reward evaluation as an important extension.

4. Independent seeds and controlled online/offline comparisons

We agree that additional independent-seed experiments would strengthen reproducibility, particularly for small performance differences. Our original experiments use fixed seed 123. As documented in the paper, the complete training and evaluation pipeline was repeated three times with the same seed to characterize execution nondeterminism. A comprehensive independent-seed sweep spanning two 8B-parameter models, multiple domains, and all credit-assignment horizons was not feasible within the available resources.

We also agree that PPO is not a fully controlled comparator because the PPO baseline is trained online, while our PG-family experiments use offline training. However, our central controlled comparison is among GreedyPG, K-Step-PG, and Full PG, which share the same logged trajectories, reward labels, backbone, optimizer, training budget, and evaluation protocol. PPO is included as an external reference rather than a regime-matched baseline. We will clarify the distinction between controlled within-family comparisons and external PPO results in Sections 5.1–5.2.

Reviewer_LqMAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Partial Policy Gradients (PPG), an approach to improve policy gradient algorithms. It multiplies the score function of each turn's action by the sum of future rewards within a time window. The method can be viewed as providing better temporal credit assignment within policy gradient family estimators. Experiments train Llama-3.1-8B and Qwen3-8B on two multi-turn conversational datasets: persona consistency (Therapy and Chatting from Consistent-LLMs) and policy adherence in customer-service tool-calling dialogues (Nemotron-Agentic-v1). The per-turn reward is a binary GPT-4o-mini judgment of whether the turn is consistent with the persona or policy. The results show that a K-step lookahead policy outperforms PPO in each experiment.

Critical strengths
  1. A simple and general experiment framework that unifies full-horizon, greedy, segment-level, and K-step lookahead policy gradients through the choice of reward index set R_t.
  2. A practically relevant empirical finding: in long multi-turn conversations (up to 60 turns), intermediate credit-assignment horizons can substantially outperform full-horizon PG, with large gains in persona consistency (e.g., 0.985 vs. 0.867 for PG in Qwen Chatting rollouts).
  3. The paper is well written overall, and the method and experimental setup are presented clearly.
Critical weaknesses
  1. Limited novelty. The estimator is a truncated reward-to-go, and the theoretical results are direct applications of standard tools. The missing comparison to γ-discounting, the most standard way to trade horizon for variance, is a significant gap.
  2. The evaluation scope is narrow relative to the claims. Everything is LLM-simulated (data, user/environment, judge). The "agentic tool-calling" task is evaluated only by turn-level consistency with a policy document, not by task success in an executable environment. The reward appears to be local to each utterance, which favors small K.
  3. The theory offers little support for the main claim: Theorem 4 analyzes only variance, ignores the truncation bias, and is vacuous at LLM scale, while Lemma 3's guarantee assumes non-negative rewards and thus does not cover the signed advantages used in practice.
Questions for authors
  1. How does K-Step-PG compare with γ-discounted reward-to-go, for a few values of γ chosen so the effective horizon matches K? If the two perform similarly, what is the advantage of a hard window?
  2. Does the GPT-4o-mini judge see the dialogue history? The template in Appendix B.1 includes only the scenario, role, backstory, and current line. If the history is not provided, how can the reward capture contradictions with earlier statements, or precondition violations in tool calling (e.g., calling process_refund before eligibility is checked), as claimed in Appendices E.13 and E.14?
  3. In the tool-calling rollouts, how are tool outputs produced? Is there real execution, or is the Llama-3.1-70B simulator generating them? Can you report task-level success, such as correct final state or correct arguments, in addition to PC?
  4. Have you checked whether higher PC comes with more generic, shorter, or repetitive responses? Response length, diversity, or a small human study would help rule out judge exploitation.
2 comments

1. Novelty and comparison with discounted reward-to-go

We agree that reward-to-go estimators, exponential discounting, and concentration inequalities are established techniques. Our contribution is not the introduction of these individual primitives.

Rather, we present a general framework for assigning subsets of future rewards to actions in sequential interaction with observations. The framework incorporates full planning, greedy optimization, K-step lookahead, and segment-based credit-assignment policies, together with online and offline gradient formulations and a concentration analysis.

To address the missing baseline, we trained discounted policy-gradient variants for γ ∈ {0.5, 0.9} on Qwen3-8B across both Therapy and agentic tool-calling.

The discounted return is defined as:

Gtγ=∑j=tnγj−trj.G_t^{\gamma} = \sum_{j=t}^{n} \gamma^{j-t} r_j.

We retained the offline training framework, logged datasets, reward labels, optimization budget, advantage-normalization convention, and evaluation protocols used for the original K-Step-PG experiments.

The chosen discounts cover approximate infinite-horizon weight masses of two, four, and ten steps, respectively. These provide comparisons across short, intermediate, and longer temporal scales, although exponential discounting is not mathematically equivalent to a hard K-step cutoff.

Additional experimental results:

PolicyTherapy (Rollout)Therapy (Step)Agentic Tool-calling (Rollout)Agentic Tool-calling (Step)
GreedyPG (K=1)0.8400.8820.4890.657
2-Step-PG0.8540.9360.5040.678
3-Step-PG0.9370.9400.5350.693
4-Step-PG0.8350.8300.5810.712
5-Step-PG0.8690.8900.7050.629
Full PG0.7850.8150.5130.702
Discounted PG (γ=0.5)0.7600.7860.4930.502
Discounted PG (γ=0.9)0.7920.8250.6950.702

K-Step-PG uses a hard cutoff, explicitly restricting reward attribution to a selected number of future steps. Discounted rewards-t-go instead retain all future rewards with exponentially decreasing weights.

The results reinforce the importance of controlling temporal credit assignment, where value of K / discounted factor (γ) needs to be based on the domain. We used traditionally popular discount factors of 0.9 and 0.5.

We will incorporate these experiments into Section 5.2 and expand the discussion of discounted returns in Section 6.

2. Evaluation scope and decomposable rewards

We agree that our evaluation does not establish unrestricted generalization to all agentic environments. Our experiments cover persona alignment and operational-policy adherence using simulated multi-turn conversations and dense per-turn consistency rewards. Although rewards are evaluated at individual turns, the policies interact sequentially with the environment. Actions affect later conversation histories, which in turn influence subsequent actions and reward outcomes. Thus, the temporal attribution problem remains meaningful. Nevertheless, we agree that these tasks differ from environments with sparse terminal rewards or delayed execution outcomes. We will explicitly restrict our empirical conclusions to the evaluated conversational settings and distinguish policy adherence from execution-verified task completion.

3. Judge history and tool-calling execution semantics

We thank the reviewer for asking us to clarify what information is available to the evaluator.

Appendix B.1 specifies that the GPT-4o-mini judge receives the scenario, speaker role, speaker backstory or operational policy, and the utterance being evaluated. It also by default receives the full preceding dialogue history.

The current prompt-consistency metric evaluates adherence to the prompt specified persona or operational policy. The judge is provided by past chat turns that helps it verify whether an utterance contradicts facts newly introduced earlier in the conversation. We will make this explicit in Section 5.1, Appendix B.1, and the descriptions of the qualitative examples.

4. Could the consistency metric favor generic, short, or repetitive responses?

We agree that a model could potentially achieve favorable consistency scores by producing generic, short, repetitive, or excessively cautious responses. However, our work expands prior foundational work of Abdullah et al.(https://arxiv.org/abs/2511.00222) for persona simulation and Nemotron Agentic-v1 for agentic policy adherence, where the LLM judge prompts calibrate against human preferences for different domains. On conducting a manual human verification of 100 model responses across all 3 tasks, we did not find such reward-hacking in any of the rollouts. However, for abundance of caution, we will add this as a potential limitation in the draft to caution future research on such possibilities.

5. Theoretical support, truncation bias, and signed advantages

We agree that Theorem 4 analyzes the concentration of empirical gradients rather than providing a complete bias–variance decomposition of the full-return estimator. Our framework deliberately considers policies optimized using different reward-dependency structures. Restricting these dependencies can improve the worst-case concentration bound, but may omit important long-range rewards and therefore change the optimal policy.

The resulting trade-off is task-dependent, as illustrated by the differing preferred horizons in Therapy and tool-calling. We also agree that the lower-bound interpretation for nonnegative offline rewards should be distinguished from the signed, standardized advantages employed in the implementation. We will clarify these assumptions in Sections 4.1–4.2 and Appendix B.2, without claiming an unconditional improvement guarantee for arbitrary signed advantages.

Reviewer_PHB4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigated how to trade off bias and variance in policy gradients for LLM reinforcement learning. Standard policy gradients credit every action with the full trajectory reward, which is noisy. Long-horizon conversational tasks make this especially hard. The authors propose partial policy gradients. The method optimizes only a chosen subset of future rewards. Smaller subsets give simpler policies with lower gradient variance. The framework covers full planning, greedy, and K-step lookahead policies. The authors evaluate it offline on persona alignment and agentic tool-calling. The contributions include: A general way to trade bias for variance by optimizing a subset of future rewards; Online and offline variants, with smaller subsets giving simpler policies that are more sample-efficient; Recovery of classic policy gradients and greedy policies as special cases, plus the first K-step lookahead policies for LLMs; Experiments showing K-step lookahead improves results by 10 to 19% on persona and tool-calling tasks. The best K depends on task complexity and data size.

Critical strengths
  1. The investigated problem is well motivated and timely. It is commonly-seen that multi-turn LLM agents drift from assigned personas and break operational rules over long conversations. Standard policy gradients credit every action with the full trajectory reward, so the gradient gets noisier as dialogues grow longer.

  2. The framework unifies several methods, e.g., full PG, greedy PG, and K-step lookahead.

  3. There is a formal variance argument. Theorem 4 gives a concentration bound that is tighter for smaller reward subsets. Smaller subsets therefore have lower variance in the worst case.

  4. The experimental results are consistent. The authors uses two tasks, two models (Qwen and Llama), and two evaluation protocols (rollout and step-level). The best K-step policies beat PPO in every setting in Table 1.

  5. The authors moreover check their own method. In Figure 2, larger K gives smoother consistency residuals over time. Table 6 measures gradient variance and shows it rises with K, with full planning about 11 to 13 times greedy.

Critical weaknesses
  1. In Theorem 4, there is a factor of d, the parameter dimension, which the authors admit is vacuous for LLMs. So the bound is weak at LLM scale.

  2. The variance evidence is insufficient. Table 6 is measured at one checkpoint, and the authors also say it does not prove the variance explains the gains.

  3. All results use a single seed, and multi-seed runs were not done. Some gaps are large, but a few are small, such as 2-Step-PG versus GreedyPG on Qwen therapy, so seed noise could matter there.

  4. The PPO comparison is under a weak controlled setting. PPO is trained online while the PG variants are trained offline. The K trade-off is shown only offline, so the "policies beat PPO" claim mixes two settings.

  5. There is a unclear point about the offline objective. Lemma 3 gives a lower bound on the true objective for non-negative rewards, and a constant C(π, π0) controls the gap. Section D also says guarantees hold for negative rewards, which conflicts with the non-negative assumption in Section 4.1. The authors should clarify this.

Questions for authors

See weaknesses part.

2 comments

We thank the reviewer for the thoughtful assessment of our work and for recognizing the motivation, unified formulation of full, greedy, and K-step policy gradients, and experimental findings. We appreciate the opportunity to strengthen the empirical evaluation and clarify the precise scope of our theoretical claims.

1. Theoretical concentration bound and parameter dimension

We agree that the parameter-dimension dependence in Theorem 4 produces a loose numerical bound at the scale of billion-parameter LLMs.

Our theoretical contribution is a structural, worst-case concentration comparison between different reward-attribution schemes, rather than a tight numerical prediction of gradient variance for an 8B-parameter model.

Under the stated boundedness assumptions, reducing the set of reward terms entering each action's gradient improves the corresponding worst-case concentration bound. This provides a formal motivation for restricting temporal credit assignment. Importantly, the theorem does not establish a universal ordering of realized gradient covariances or guarantee superior downstream performance for every horizon and task. We will clarify this distinction and the dependence on parameter dimensionality in Section 4.2, separating the analytical result from the empirical observations.

2. Gradient variance measured at only one checkpoint

We agree that the original Table 6, measured at training step 250, provides incomplete evidence about the persistence of the variance–horizon relationship.

To address this concern, we extended the variance analysis to three training checkpoints: steps 100, 250, and 400, using Qwen3-8B on the Therapy domain.

We followed the variance-estimation protocol of Appendix D.7, computing each estimate from 50 independently sampled minibatches and reporting the mean and standard error across 10 repetitions.

PolicyStep 100Step 250Step 400
GreedyPG (K=1)0.063 ± 0.0040.044 ± 0.0030.038 ± 0.002
2-Step-PG0.095 ± 0.0080.087 ± 0.0050.078 ± 0.004
3-Step-PG0.181 ± 0.0160.153 ± 0.0120.122 ± 0.013
4-Step-PG0.244 ± 0.0210.199 ± 0.0180.164 ± 0.017
5-Step-PG0.319 ± 0.0270.283 ± 0.0220.268 ± 0.022
Full PG0.590 ± 0.0410.505 ± 0.0320.463 ± 0.025

Three observations follow from this analysis.

  1. Consistent ordering across horizons: At all three checkpoints, empirical gradient variance increases from GreedyPG through intermediate K-step horizons to Full PG.
  2. Substantially higher variance under full planning: Full PG exhibits 9.37×, 11.48×, and 12.18× the variance of GreedyPG at checkpoints 100, 250, and 400, respectively.

Moreover, as in the original protocol, measurements from policy-specific checkpoints characterize gradients encountered by the trained methods; they do not isolate the effect of K at identical model parameters. We therefore interpret these findings as strengthened empirical evidence for the variance–horizon relationship, rather than a causal proof that variance reduction alone explains the downstream performance gains.

We will add this table to Appendix D.1, expand the protocol description in Appendix D.7, and update the discussion in Section 5.2.

3. Single-seed experiments and reproducibility

As documented in Sections 5.1 and B.2, all experiments use fixed seed (123), following the benchmark protocol. We additionally repeated the complete training and evaluation pipeline three times using that fixed seed to characterize variability arising from nondeterministic hardware execution. The repetitions measure execution non-determinism variation was small. We report the mean of these runs.

A comprehensive independent-seed sweep across two 8B-parameter models, multiple conversational domains, and six credit-assignment policies would require substantial additional training resources.

However, several differences in the original experiments are substantially meaningful. For example, 3-Step-PG achieves 0.937 rollout PC on Qwen/Therapy, compared with 0.785 for Full PG. The additional multi-checkpoint variance experiment strengthens the evidence for the proposed gradient-estimation mechanism, although it does not replace independent-seed validation. We will clarify this explicitly in the manuscript.

4. Fairness of the PPO comparison

We thank the reviewer for raising this concern and would like to clarify the role of PPO in our experimental evaluation.

Our primary contribution is the study of temporal credit assignment within the Partial Policy Gradient (PPG) framework, rather than a comparison between offline and online RL algorithms. To isolate the effect of credit assignment, all policies within the PPG family—GreedyPG, K-Step-PG, and Full PG—are trained on the same offline trajectories, using identical reward labels, model architectures, optimizer, learning-rate schedule, batch construction, training budget, and evaluation protocol. The only component that changes is the subset of future rewards used in the gradient estimator, allowing us to isolate the effect of temporal credit assignment under otherwise controlled conditions.

PPO is included as an external reference baseline, as it is a widely adopted online RL algorithm and was previously evaluated on these conversational benchmarks. Importantly, PPO was independently tuned under its own training setup rather than being forced to share hyper-parameters designed for offline PPG. As detailed in Appendix B.2, we performed a hyper-parameter search over actor learning rates, KL penalties, and training sample budgets, and report the selected configuration in Table 2. This ensures that PPO serves as a meaningful reference for established RL approaches rather than an intentionally disadvantaged baseline.

We agree that the online PPO and offline PPG results should not be interpreted as a strictly controlled comparison of credit-assignment strategies, given their different training regimes. However, this distinction does not undermine our central empirical findings, which are established through controlled comparisons within the PPG family, where all training conditions are held fixed except the reward-attribution horizon. We will make the role of PPO as an external baseline more explicit in Sections 5.1–5.2 and clarify that our empirical conclusions regarding the effect of varying K are based on the controlled offline PPG experiments.

5. Offline objective: nonnegative rewards and signed advantages

We appreciate the reviewer identifying this ambiguity. Lemma 3 does not need any assumption on non-negative rewards. The follow-up discussion considers two cases:

  • When rewards are non-negative, we optimize a lower bound on the online objective.

  • When rewards can be negative, the optimized objective is within C(π,π0)C(\pi, \pi_0) of the online objective.

We distinguish these two cases because the first bound is stronger, since optimizing a lower bound on an objective provably optimizes that objective. We will revise Section 4.1 and Appendix B.2 to state the relevant assumptions precisely, clarify the role of the approximation term C(π,π₀).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.