acceptodds
Under review as a conference paper at ICLR 2027

SegBench-GC: Testing Segmentation Invariance in Multi-Step Offline Goal-Conditioned Reinforcement Learning

Abstract

Offline goal-conditioned reinforcement learning (GCRL) often uses trajectory structure for future-goal sampling and multi-step targets, yet logged trajectories may be partitioned for administrative reasons that do not correspond to termination. We introduce SegBench-GC, a controlled stress test of segmentation invariance that holds transitions, source trajectories, goal sampling, optimization settings, and evaluation fixed while varying only artificial backup boundaries and whether those boundaries retain continuation value. Continuation-valid targets (CVT) provide the segmentation-consistent control: reward accumulation stops at an artificial cut, but the target bootstraps from its stored successor. In a matched-count PointMaze study with 35,000 artificial cuts, three segmentation realizations, and three optimization seeds, final 50-episode-per-task success is 50.5% uncut, 39.1% with CVT, and 19.1% when the same cuts are treated as absorbing; across segmentation realizations, naive mean success ranges from 4.8% to 31.9%. An independent published n-step baseline (n=25) from the Decoupled Q-Chunking codebase shows the same failure on Puzzle-4x5: 47.2% uncut, 58.5% CVT, and 0.27% naive across three optimization seeds. A target-level diagnostic verifies the analytic target difference to numerical precision, and learned-critic diagnostics show a large optimistic shift under naive handling while CVT remains approximately aligned with the uncut critic. CVT applies standard continuation bootstrapping rather than a new Bellman rule; the contribution is the controlled benchmark, failure isolation, and cross-learner evidence that administrative segmentation can materially change multi-step offline GCRL.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, goal-conditioned reinforcement learning, segmentation invariance, trajectory truncation, multi-step targets, reinforcement learning benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_q2BMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper keeps the offline data and the goals fixed, and changes only what a multi-step backup does at a cut that is not a real ending. One version still uses the value of the next state. The other treats the cut as the end. Treating the cut as the end lowers success. A learner that backs up only one step does not read the cuts.

Critical strengths

The contribution is a comparison that changes only whether a nonterminal cut keeps the next state's value. Bootstrapping keeps more of the task success. Treating the cut as an ending loses it. The same cuts do not affect a one-step learner.

Critical weaknesses

Gymnasium, OGBench masks, and D4RL timeouts already separate a cut from true termination. Official IQL bootstraps with masks. One-step methods that follow these interfaces are unaffected. The drop appears only when a multi-step backup treats the cut as an ending.

I would raise the score if the authors explain what is new and how many current offline RL methods are actually affected.

Questions for authors

What should other papers take from this work beyond the known bootstrap rule?

Which current methods, besides the two tested here, treat a nonterminal cut as an ending? Are one-step OGBench or D4RL implementations affected?

Reviewer_8RJYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper deals with the segmentation and termination in offline Reinforcement Learning. In RL, it is standard practice to artificially truncate some episodes, usually (but not always) at some fixed episode length. The authors ask how much the placement of the segmentation affects the final policy performance. To test this, the authors construct a protocol called SegBench-GC that compares policies trained under different cut regimes, revealing noticeable differences. The protocol evaluates original trajectories with backups, without artificial cuts, backups truncated at artificial boundaries without bootstrapping, and backups truncated at those same boundaries, but with bootstrapping (CVT). The authors work on two RL environments: PointMaze and Puzzle-4x5.

Critical strengths
  • The authors very clearly identify a problem and focus on it. The problem can potentially occur in many RL settings, as episode truncation is common and requires careful handling.
  • The protocol appears to test what the authors promise. It is quite simple, easy to understand, and intuitive. It also does not require substantial effort to implement in other environments and algorithms.
  • The results show the gap that is formed by artificial episode truncation being treated as terminal states, supporting the usefulness of SegBench (but see Weaknesses).
  • The authors verify that their manipulation of the dataset (artificial cuts) does not interfere with the rest of the data by using a one-step GCIQL learner. This is a sanity check, which I find is very good practice.
Critical weaknesses
  • The main weakness in the paper is the experimental side. The authors test only two multi-step algorithms, both on distinct environments. Each algorithm is run on only 3 optimization seeds. In the PointMaze environment, each of the three optimization seeds is evaluated on 3 segmentation seeds. I do not think this is enough. It is known for quite a while that RL runs can have substantial seed variations; the authors cite a well-known paper, “Deep reinforcement learning at the edge of the statistical precipice”, that deals with exactly this matter. This is especially important, given how much the reported performance varies between seeds (for example, CVT on the Puzzle environment varies from ~40% to ~80% success rate. I appreciate that the authors test one algorithm on one environment and then use another algorithm from a different library as a cross-implementation validation. However both algorithms should be run on both of the environments. And to add to that, there should be more than two environments tested.
  • I think that the paper could really use a bit better presentation. While it is very much possible to understand what the point of the paper is, I think that a simple figure that illustrates where segmentation occurs and why this can be a problem, as well as explaining the three regimes of cutting (original, CVT, Naive) would greatly benefit the paper. After adding more environments or additional algorithms, it also would not hurt to have a plot with the main results, as currently all the results are presented in a simple table.
Questions for authors

I think that this paper does investigate an interesting problem, and has an elegant way to test the central premise. Where it greatly lacks, in my opinion, is the experimental side. As it stands, I cannot recommend acceptance; however, I am willing to increase my rating if the authors provide additional experiments involving more seeds for increased result reliability and more environments to truly test how prevalent the gap in performance caused by segmentation is. Other than those remarks, my questions to the authors are:

  • Do the authors have theories on what type of environments and algorithms are more vulnerable to the performance degradation described in the paper?
  • For all experiments, the authors use the same cut density of 3.5%. Did the authors test any other densities? What was the reasoning behind choosing this particular density for all experiments?
Reviewer_u5ysOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces SegBench-GC, a benchmark for evaluating the sensitivity of multi-step offline goal-conditioned reinforcement learning (GCRL) to artificial trajectory segmentation. The authors demonstrate that incorrectly treating artificial trajectory boundaries as terminal states can significantly degrade performance by removing bootstrapped continuation values. To address this issue, they employ Continuation-Valid Targets (CVT), which preserve bootstrapping at nonterminal boundaries. Experiments on PointMaze and Puzzle-4x5 show that CVT substantially outperforms naive terminal handling. The main contribution is a controlled evaluation protocol highlighting the importance of correctly handling trajectory boundaries in multi-step offline GCRL.

Critical strengths

None

Critical weaknesses

In RL, bootstrapping from a value function VV is a standard and well-established technique. For example, the nn-step TD target is defined as $ y_t = \sum_{i=0}^{n-1} \gamma^i r_{t+i} + \gamma^n V(s_{t+n}). $

If a trajectory is truncated after k<nk < n steps without actual episode termination, the target should retain the continuation value: $ y_t = \sum_{i=0}^{k-1} \gamma^i r_{t+i} + \gamma^k V(s_{t+k}). $

In contrast, if the trajectory ends at a true terminal state, the bootstrapped value should be omitted. This distinction between truncation and termination has already been extensively discussed in prior work, particularly in Time Limits in Reinforcement Learning (Pardo et al., 2018).

Given that the proposed CVT approach essentially follows this well-established principle, I struggle to identify a substantial novel contribution in this paper. While the authors empirically demonstrate the consequences of incorrectly treating artificial trajectory boundaries as terminal states, the underlying issue and its correction are already well understood in the RL literature.

Furthermore, the paper is difficult to follow due to a lack of clarity in its presentation, and the experimental evaluation is limited to a narrow range of environments and algorithms. Overall, I find the novelty and empirical scope insufficient to support the paper's contributions.

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.