acceptodds
Under review as a conference paper at ICLR 2027

Must Decision Making Be Trajectory-Centered? Behavioral Structure as a Relation-Centered Alternative

Abstract

Reinforcement learning is commonly learned from trajectories: finite sequences generated by interactions between an agent and its environment. Yet trajectories are samples of realized behavior, not necessarily the underlying structure of what behaviors are possible. We ask whether decision making must be organized around observed trajectories, or can instead be organized around the behavioral relations revealed by them. We introduce Behavioral Structure, a relation-centered framework that represents action-conditioned relations among states and composes them directly for decision making. On sparse long-horizon goal-directed control, Behavioral Structure reaches 32% success versus 17% for goal-conditioned Decision Transformer and 13% for Decision Transformer on Ant. When training trajectories are deliberately fragmented across withheld behavioral boundaries, Behavioral Structure achieves 32% versus 2% for endpoint-conditioned Decision Diffuser, while retaining 36% success at composition depths 4–5 versus 0%. These results support a view in which trajectories describe realized experience while behavioral relations provide an alternative substrate for organizing decisions.

Primary area:
reinforcement learning
Keywords:
Relation-Centered Decision Making, Behavioral Structure, Compositional Decision Making, Decision Representation, Reinforcement Learning, Long-Horizon Decision Making

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_HkzjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents behavior structure, an approach that treats action-conditioned relations between states as representation units for decision making and solving tasks by composing these local relations. Instead of modeling behavior from a trajectory-centered perspective, behavior structure exploits reusable local structures and builds a weighted graph structure to facilitate goal-directed planning. The experiments show that the proposed method outperforms the baselines on two simulated tasks and remains effective under withheld-boundary evaluations. Ablation studies further stress the importance of modeling action-conditioned relation costs for shortest-path planning over the behavior structure.

Critical strengths
  1. The paper proposes a new modeling perspective for offline decision making. Rather than treating complete trajectories as the primary units, behavior structure instead builds on action-conditioned local relations between states and enables graph-based planning using learned relation costs as edge weights. This formulation could construct behaviors that are not observed as complete trajectories.
  2. Withheld-boundary experiments demonstrate the effectiveness of the methods in utilizing local behavior for planning.
Critical weaknesses
  1. Several important details are missing in the method and require further elaboration. (1) how is fθf_{\theta} trained to obtain transition-supervised representation? Is it trained separately via contrastive learning or end-to-end with qϕq_{\phi}? (2) Are the 12 nearest neighbors selected in Section 3.2 treated as positive targets for qϕq_{\phi} training in Section 3.3, or can they be incidentally sampled as negative targets? (3) How are the upper and lower bounds (i.e., 0 and 3) decided for the clip function in Section 3.3?
  2. The standard goal-directed evaluation in Section 4.2 focuses on only two simulated tasks with low-dimensional state input, and compares against one baseline and its goal-conditioned variant. The controlled composition test instead compares against another baseline method. Consistent evaluation settings and additional results on more challenging tasks and more recent baselines (e.g., models that stitch sub-trajectories via RL or generative models) are critical to strengthen the experiments.
  3. See the questions below.
Questions for authors
  1. The standard evaluation tests the method at different target horizons. However, Table 1 only provides aggregated results. How do the proposed methods and the baselines perform under different target horizons? Would the method be able to tackle more complicated tasks with target horizons beyond the evaluated range?
  2. How does the number of sampled nearest-neighbor states (e.g., K=12 used in the method) affect the performance?
  3. What’s the inference cost for running Dijkstra over the constructed graph? How does it compare to baseline methods that do not rely on graph-based planning?
Reviewer_pZMuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper attempts to recover the environment structure by repeatedly sampling <state, action, next-state> triples and use the recovered structure to learn behavior.

Critical strengths

I do not see any strengths.

Critical weaknesses

The paper does not introduce a new insight. The approach attempts to recover the environment structure by repeatedly sampling <state, action, next-state> triples and use the recovered structure to learn how to act. This general approach is widely known and used for decades, for example, in model-based reinforcement learning.

The paper does not state its assumptions; furthermore, it does not state whether the same restrictive assumptions are also made by algorithms they use as baselines. Critically, the proposed approach assumes that multiple <state, action, next-state> samples can be generated from any state, upon request. This assumption will not generally hold. An agent interacting with its environment will generally be able to produce only one such sample, and that will be from the state that it is currently in. It is unclear whether any additional assumptions are being made, for example, regarding Markovian dynamics or stochasticity.

Questions for authors

I have no questions for the authors

Reviewer_m9kbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper questions whether decision-making representations must be organized around complete trajectories and proposes Behavioral Structure, a relation-centered alternative for offline goal-directed control. The method learns transition-informed state embeddings, constructs a sparse k-nearest-neighbor graph, assigns action-conditioned costs to candidate edges, and uses shortest-path planning to compose local behavioral relations into executable action sequences.

The authors evaluate the approach on Ant and Hopper and introduce a withheld-boundary benchmark where trajectories are fragmented to test cross-fragment composition. Behavioral Structure achieves 32% success on Ant, compared with 17% for goal-conditioned Decision Transformer. In the fragmented setting, it achieves 32% versus 2% for Decision Diffuser. Additional graph controls and edge-sparsification experiments investigate the contribution of learned relation weights.

Critical strengths
  1. An interesting perspective on behavioral composition.

The distinction between trajectories as observation units and local relations as composition units is conceptually reasonable. The paper correctly recognizes that useful decisions may be constructed by recombining experiences that were not observed together in complete trajectories.

  1. A targeted evaluation of cross-fragment composition.

The withheld-boundary experiment in Section 4.3 is a useful attempt to isolate compositional generalization. By removing transitions between trajectory fragments while preserving local behavioral evidence, the authors construct a setting that directly tests the proposed representation hypothesis.

  1. Evidence that learned relation weighting can improve graph-based execution.

Tables 2–4 show that Behavioral Structure substantially outperforms geometric graph controls. The edge-sparsification experiment is particularly informative: retaining edges according to learned scores preserves considerably higher execution success than random retention at comparable graph densities.

Critical weaknesses
  1. The methodological novelty is substantially overstated.

Despite the conceptual framing, the implemented approach consists primarily of transition-supervised representation learning, kNN graph construction, learned edge costs, and Dijkstra planning. These components are closely related to existing graph-based planning and trajectory-stitching methods, including SoRB, LEAP, and Graph-Assisted Stitching (GAS). Attaching observed actions to graph edges is a potentially useful distinction, but the paper does not establish why this constitutes a fundamentally different decision-making framework rather than a variant of experience-graph planning.

  1. The learned behavioral relations do not establish executable transitions.

In Section 3.2, every candidate edge originating from state sis_i inherits the same observed action aia_i, regardless of its destination sjs_j. However, executing aia_i does not necessarily lead to sjs_j, particularly for kNN edges that were never observed as transitions. The discriminative relation model estimates compatibility relative to randomly sampled negatives, not transition probabilities or guaranteed reachability. The discrepancy penalty in Eq. (4) does not resolve this issue. Consequently, the validity of composing these candidate edges into executable plans remains insufficiently justified.

  1. The experimental baselines do not adequately support the central contribution.

The main comparisons focus on Decision Transformer, goal-conditioned Decision Transformer, and Decision Diffuser. However, these are not the closest alternatives to the proposed graph-based planning mechanism. The paper discusses SoRB, LEAP, and GAS but does not evaluate them. The geometric graph controls are also insufficient to isolate the proposed novelty. In particular, a simpler action-conditioned graph constructed directly from observed transitions would be an important baseline for determining whether the learned relation model is necessary.

  1. The withheld-boundary experiment does not sufficiently characterize successful composition.

Section 4.3 removes 112 boundary transitions from 28 Ant trajectories, producing 2,408 training transitions. However, evaluation queries are constructed from fragments of the original trajectories, and the method can introduce inferred edges between nearby observed states. It is unclear how often successful plans actually rely on valid cross-boundary transitions rather than local geometric proximity. The paper does not report the proportion of inferred edges used, their execution accuracy, or how errors accumulate across composed relations. These measurements are important for substantiating the proposed mechanism.

  1. The experimental evidence is limited and lacks statistical rigor.

The experiments cover only Ant and Hopper, with relatively low absolute success rates of 32% and 14% in standard evaluation. The principal compositional comparison uses only 100 queries, with 25 per depth, and no confidence intervals or variation across independently trained models are reported. Moreover, the 32% success in Table 2 and 26.7% at full graph density in Table 4 require clarification regarding their evaluation protocols. The current evidence is insufficient to establish reliable improvements or generalization across environments.

  1. Reproducibility and computational scalability are insufficiently addressed.

Section 3.1 describes transition-supervised representation learning but does not specify its complete training objective or optimization procedure. Important implementation details, including training budgets, model-selection criteria, and computational costs, are missing. Furthermore, the graph contains individual observed state instances and requires nearest-neighbor search, relation scoring, and shortest-path planning. The paper provides no systematic analysis of memory requirements, planning latency, or scaling with dataset size, making the practicality of the proposed framework difficult to assess.

Questions for authors
  1. Relation validity: How is an inferred edge (si,ai,sj)(s_i,a_i,s_j) considered executable when aia_i was observed to produce a different successor? Can the authors report one-step transition errors for inferred edges and the proportion of successful plans relying on such edges?

  2. Methodological novelty: What distinguishes Behavioral Structure from an action-conditioned experience graph using observed transitions and standard shortest-path planning? Is there evidence that the learned relation model provides substantial benefits over this simpler alternative?

  3. Baseline comparisons: Why are SoRB, LEAP, and GAS not included in the experiments, despite being acknowledged as closely related approaches? Are comparable results available under the same dataset and evaluation protocol?

  4. Experimental reliability: Were the reported results obtained across multiple independently trained models or using a single training run? Can the authors clarify the difference between the 32% success in Table 2 and the 26.7% full-density result in Table 4?

5.Reproducibility: What exact objective is used to train the transition-supervised encoder in Section 3.1? What are the computational costs of graph construction, relation scoring, and planning as the dataset grows?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.