acceptodds
Under review as a conference paper at ICLR 2027

Learning Task Representations from Sparse Evidence in Offline Meta-Reinforcement Learning

Abstract

Context-based offline meta-reinforcement learning (COMRL) trains a context encoder to infer a latent task from collected trajectories. Existing methods commonly treat task-relevant evidence as broadly distributed across the context, or isolate a few individually informative transitions. This leaves open the case in which only a small fraction of interactions carries task-dependent information, while the rest is valid but weakly informative experience. A trajectory may still contain sufficient task information, but the informative feedback can appear in brief intervals of unknown location and duration. Learning a reliable task representation then requires combining these sparse clues while limiting interference from the rest of the context. We study this sparse-evidence regime and show that task inference requires preserving weak, collectively informative responses before determining their relevance. Based on this observation, we introduce STeR (Slot-Based Task Representation Learning from Sparse Evidence in Offline Meta-Reinforcement Learning), a structured context encoder that performs accumulation before selection. Transition features are softly assigned to content-based latent slots, each slot is normalized independently, and a task-level readout selects the summaries that matter. The organization is learned from trajectory-level task labels alone, with no evidence masks or temporal localization supervision. We evaluate STeR on controlled offline meta-reinforcement learning environments where task information appears through sparse rewards, transient motion responses, and hidden contact dynamics. Across variations in evidence duration and onset, STeR improves task identification and downstream adaptation while preserving performance when task evidence becomes sparse. The results suggest that structured accumulation is a useful inductive bias for task representations drawn from incomplete, noisy offline experience.

Primary area:
reinforcement learning
Keywords:
Meta-Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VEHmOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes STeR for offline meta-RL tasks where only a small part of the context carries task information. STeR groups transitions into slots, computes normalized mean–variance summaries, and learns how to combine them into a task embedding. Training uses task labels, auxiliary losses for initial slot formation, and subsequent readout refinement. The analysis gives conditions for identifiability and bounds background contamination of slot means. Experiments on ADSE, HCSE, and DDE evaluate task identification and downstream TD3+BC control as evidence duration, onset, and task parameters change.

Critical strengths
  1. COMRL with sparse task evidence is a pratical problem. Grouping related transitions using slot representations before deciding their task relevance is a sensible approach. Figure 5 supports both slot grouping and learned routing on ADSE, particularly when evidence is short.

  2. Extensive experiments. The normal-condition gains are large across all three benchmarks; for example, DDE OOD success improves from the best baseline's 79.7% to 90.3%. Using five seeds and a common controller makes these comparisons more useful. The evidence-removal and replacement tests also help establish what the representation relies on.

Critical weaknesses
  1. The experiments leave the central explanation unresolved. The key test would hold informative transitions fixed and add background. Changing evidence duration or randomly retaining transitions also changes the available evidence. In ADSE/HCSE, nonzero rewards already identify candidate evidence transitions, so the results do not establish that the clues are individually weak or require accumulation before selection. The ablations are limited to ADSE identification, do not isolate separate normalization, and give too little detail about “Pure attention” to assess the comparison. Contextual attention, such as Set Transformer and Non-Parametric Transformers, can also combine information before pooling.

    The collector (i.e., behavior policy) assumption in line 106 needs clarification too. HCSE does not specify the collector mixture for each task, and DDE's “matched” branch could be problematic if its controller uses the true damping. I cannot tell from the description whether this happens. Equal branch counts and withholding collector labels would not rule out task cues in the actions. Removing whole evidence transitions removes those actions as well, so the attribution tests do not settle this concern.

  2. High identification accuracy does not translate into robust control in an important setting. ADSE OOD return drops from 379.0 under normal evidence to −293.4 with delayed onset and −293.0 with long evidence (Tables 5/16), despite identification accuracies of 97.9% and 99.9% (Table 9). In contrast, FOCAL++ achieves a positive return of 60.8 in the long condition. This is a substantial failure under the shifts the paper aims to handle, yet it receives little discussion. The authors need to explain the discrepancy and qualify the claims about downstream adaptation.

Questions for authors
  1. Collector construction: Is the distribution over actual collectors identical for every task? What changes between DDE's matched, nominal, and mismatched branches, and does any controller use the true task damping?

  2. Control failure: Do the ADSE identification and control results use the same frozen encoder checkpoint? What explains the OOD return collapse despite high identification accuracy?

  3. Model details: How is “Pure attention” implemented and trained relative to STeR? Please also reconcile Table 3's four total slots with Table 20's five HCSE slots, and explain Appendix F's ADSE plus/minus reward views.

  4. Additional evidence: A comparison with fixed evidence and increasing background would directly test the main motivation. Identification from one versus several evidence transitions, and pooling transitions selected by nonzero reward, would help distinguish weak collective clues from readily identifiable sparse cues.

  5. Morivation: COMRL with sparse evidence is not a new topic. I believe prior works such as FOCAL++ specifically targets the same problem and propose effective solutions. I suggest the authors to properly discuss it in the related work section.

  6. Clarification on slot attention: To compute slot attention in eqn 3, the latent dimension of slot queries dsd_s and key values dgd_g should be identical in Figure 2?

Remark: I'm willing to adjust my score if the authors can properly address my concerns above.


Post-rebuttal comments: I appreciate authors' efforts for providing extensive experiments to address my questions. I think most of my concerns regarding experimental results are resolved. However, frankly, as someone who has done research closely related to the topic, I personally feel that the methodology of COMRL is somewhat obsolete in the large-model era, and in-context RL with transformer architecture might be more promising to scale. Therefore, my overall assessment is that the method and experiments of this paper are solid, but the significance/novelty of the topic is somewhat limited. This work is therefore borderline to me, and I have increased my score to 3 to lean slightly towards acceptance.

3 comments
AuthorsOpenReview ↗

Response to Reviewer VEHm (Part 1)

We thank the reviewer for the detailed assessment. We address collection and control in Part 1, architecture and evidence controls in Part 2, and reward cues, mechanism tests and related work in Part 3. The corresponding weaknesses are addressed alongside the questions.

Q1 / W2. Collector quotas, damping access and action cues

We thank the reviewer for asking about collector distributions and task information in actions. Within each dataset, every task has identical per-collector quotas: ADSE uses eight equally weighted collectors; HCSE uses the same nonuniform weights over 11 collectors, chosen from pilot forward speeds to cover target speeds. Integer rounding can slightly change proportions between datasets. Training uses 240 ADSE and 437 HCSE trajectories per task; checkpoint selection uses 32 supports/16 queries, and final evaluation 64 supports/64 normal queries per task.

DDE collects three paired trajectories with the same drawer, physical damping, initial state and pulling controller, changing only the reference damping used to compute actions:

  • Matched: the current task's true damping. Yes, this controller accesses true damping.
  • Nominal: one fixed reference shared by all tasks, without reading current-task damping.
  • Mismatched: the paired adjacent training task's damping reference.

The purpose is to expose the same physical task under different action patterns, encouraging inference from dynamical responses rather than one collection template. Each of ten training tasks contributes 160 triplets; each reference contributes one third of every task's data. The 1,600 triplets share their first 24 approach/descent actions. True damping, reference damping and branch labels are excluded from encoder inputs. Episode seeds do not overlap training, validation and test. Encoders train on all three references; frozen-encoder TD3+BC training, validation and final evaluation use matched collection. We will correct line 106 to distinguish shared collector mixtures in ADSE/HCSE from DDE's paired references.

We further evaluate task inference when the controller uses a common reference. On five DDE OOD onset tasks, with eight disjoint support/query trajectories per task, STeR achieves 91.25% without added background and 90.83% with 64 added background transitions, averaging two models and three draws. Thus, task identification remains effective without the collection controller knowing current-task damping.

Table 1. DDE OOD normal identification across references (%), averaging two frozen models with 16 disjoint trajectories per task per set.

Support referenceTrue-reference queryFixed-reference query
True100.0021.88
Fixed28.1399.38

Paired mixing reduces reliance on a single action template, but cross-reference matching remains sensitive to collection behavior. We will add the data construction and both controls to make this distinction explicit.

Q2 / W3. Frozen encoders and the ADSE control results

We thank the reviewer for identifying this discrepancy. Each seed uses the same frozen encoder for task identification and policy input, and the same selected policy checkpoints across all six evidence conditions. The negative ADSE OOD returns in the five non-normal conditions are sign typos; magnitudes and standard deviations are unchanged.

Table 2. STeR ADSE OOD identification and corrected return.

ConditionIdentification (%)Corrected return
normal97.75 ± 0.52379.01 ± 18.58
onset97.91 ± 0.84293.42 ± 62.69
short76.05 ± 2.27247.47 ± 55.98
long99.90 ± 0.10292.98 ± 65.29
onset-short77.68 ± 2.40265.76 ± 39.56
onset-long99.58 ± 0.43294.37 ± 61.08

On onset and long, STeR's returns are 293.42 and 292.98, compared with 20.71 and 60.77 for FOCAL++; STeR identification is 97.91% and 99.90%. Its task representations therefore support effective control in both conditions, although returns remain below normal's 379.01. Identification uses cosine matching to support prototypes, while the policy directly uses the continuous embedding. Evidence changes can affect actions even when the task label remains correct. We will correct the signs in manuscript Tables 5/16 and add this explanation and the FOCAL++ comparison.

AuthorsOpenReview ↗

Response to Reviewer VEHm (Part 2)

Q3. Pure attention, slot counts and reward views

We thank the reviewer for requesting these details. Pure attention scores each transition key with one learned query, applies softmax over all valid transitions, and projects the weighted sum. STeR instead groups transitions by content, independently normalizes within-slot means/variances, then routes the summaries.

Pure attention's feature encoder, normalization, key projection, query and output projection train end to end for all 200k updates. Mean pool and GRU likewise remain trainable throughout. The 60k initial stage followed by readout refinement applies only to STeR; we will correct Appendix D.5.

In the paper, Table 3 uses ADSE's three foreground plus one background slot; Table 20 uses HCSE's four foreground plus one background slot. We will label the environments. ADSE views share states, actions, next states, evidence windows and encoder weights; only rr becomes −r-r before shared normalization. The 64-dimensional readout has view-exchange-invariant and sign-changing components. Appendix F averages attribution computed separately for the two views. We will clarify these definitions.

Q4 / W1. Fixed evidence and fixed background

We thank the reviewer for proposing these controls. All five methods use the same support/query trajectories. STeR uses its complete model; four pooling baselines freeze its transition features and train for 2,000 updates on identical batches. Slot pooling also reuses its slot assignments. Results average two models and three draws.

1. Fix evidence; add background. ADSE/HCSE use OOD normal, with 64 disjoint support/query trajectories per task (640/320 per set). DDE uses OOD onset, with eight per task across five tasks. Evidence and native rewards remain fixed.

Table 3. ADSE OOD normal identification (%).

Method0 background64 background
STeR96.8896.95
Slot statistics pooling (mean and variance)74.6970.26
Slot statistics pooling (mean only)54.3847.40
Direct attention43.8334.58
Global mean46.7231.59

Table 4. HCSE OOD normal identification (%).

Method0 background64 background
STeR73.5977.50
Slot statistics pooling (mean and variance)63.5960.47
Slot statistics pooling (mean only)66.4162.19
Direct attention69.6968.13
Global mean76.2545.36

Table 5. DDE OOD onset identification (%).

Method0 background64 background
STeR91.2593.33
Slot statistics pooling (mean and variance)92.5089.58
Slot statistics pooling (mean only)85.0086.25
Direct attention86.2580.42
Global mean85.0052.50

STeR preserves identification as background increases in all three environments. Direct attention normalizes individual responses over the whole context; weak evidence competes with background. STeR first aggregates similar responses within independently normalized slots, reducing that interference before summary selection.

2. Fix background; vary evidence count. ADSE/HCSE retain 32 background transitions using the same trajectories as above. DDE uses OOD normal, 16 fixed background transitions, and 80 disjoint trajectories per support/query set.

Table 6. ADSE OOD normal identification (%).

Method1 evidence4 evidenceAll evidence
STeR33.4967.5896.95
Slot statistics pooling (mean and variance)25.8145.3667.86
Slot statistics pooling (mean only)24.8434.9045.96
Direct attention20.7825.7636.09
Global mean18.6723.4634.95

Table 7. HCSE OOD normal identification (%).

Method1 evidence4 evidenceAll evidence
STeR44.1152.7675.94
Slot statistics pooling (mean and variance)30.6335.5759.22
Slot statistics pooling (mean only)31.4138.3961.15
Direct attention32.0837.5068.33
Global mean18.3321.9849.90

Table 8. DDE OOD normal identification (%).

Method1 evidence4 evidenceAll evidence
STeR36.6760.2199.79
Slot statistics pooling (mean and variance)38.7543.5496.46
Slot statistics pooling (mean only)34.1737.9288.75
Direct attention20.2122.7190.21
Global mean21.8820.8365.63

With four evidence transitions, STeR exceeds direct attention in all three environments. On DDE, STeR reaches 60.21%, versus attention's 22.71%; mean-and-variance slot pooling also reaches 43.54% on the same features. We will add these controls supporting accumulation before selection.

AuthorsOpenReview ↗

Response to Reviewer VEHm (Part 3)

Q4 / W1. Reward cues and statistical aggregation

We thank the reviewer for asking about reward selection and contextual attention. Nonzero-reward mean filters by raw reward; absolute-reward weighting uses its magnitude. Their encoders train end to end for 80k updates, then TD3+BC for 200k, with independent validation selection. Tables 9–10 average six OOD conditions; baseline control uses seed 101.

Table 9. ADSE OOD reward-pooling comparison.

MethodIdentification (%)ReturnIdentification with overlap (%)
STeR91.48295.5081.90
Nonzero-reward mean54.3585.9011.02
Absolute-reward weighting62.53102.2411.51

Table 10. HCSE OOD reward-pooling comparison.

MethodIdentification (%)ReturnIdentification with overlap (%)
STeR67.18−128.5360.78
Nonzero-reward mean51.98−285.0826.04
Absolute-reward weighting53.39−208.6530.47

For overlap, trajectories, task parameters, evidence windows and within-window rewards stay fixed. Only outside-window rewards are replaced with task-independent training reward segments, sampled separately for support/query. All methods receive 80k updates; STeR retains 60k initial training plus 20k refinement. Reward pooling now includes rewarded background; STeR selects content-based summaries and retains its advantage.

DDE rewards are nonzero in both background and contact transitions; 94.82% of query contact rewards fall within the background range. Using five OOD normal tasks, 16 disjoint trajectories per task per set, two frozen models and 2,000-update controls gives:

Table 11. DDE OOD normal identification with native rewards (%).

MethodIdentification
STeR99.38
Direct attention83.75
Nonzero-reward mean (global mean)71.88
Absolute-reward weighting86.88

Variance and independent normalization. In Table 8, mean-and-variance pooling exceeds mean-only pooling with the same features and training budget. We additionally fix two complete STeR models, slot assignments and normal supports, then remove variance or share the slot-mean denominator and recompute routing:

Table 12. Fixed-model interventions on DDE OOD identification (%).

Settingonsetonset-shortlong
STeR59.0653.4458.13
Remove variance39.5331.8843.91
Shared mean denominator31.7237.8145.94

Both interventions reduce accuracy in every condition, supporting variance information and normalization by each slot's assigned weight.

Contextual attention. Set Transformer uses two SAB layers and one PMA layer; NPT alternates transition/feature attention. Both share the frozen features, batches and 2,000-update budget of the pooling baselines.

Table 13. DDE OOD normal evidence-count comparison (%), matching Table 8.

Method1 evidence4 evidenceAll evidence
Set Transformer20.4223.7594.79
NPT27.9236.6774.38

With four evidence transitions, STeR reaches 60.21%, above both contextual-attention methods; mean-and-variance slot pooling reaches 43.54%. With 64 background transitions in the onset setting of Table 5, Set Transformer/NPT reach 80.42/76.25%, compared with STeR's 93.33%. Contextual attention exploits abundant evidence, while slot statistics make limited evidence more useful amid background. We will add these comparisons and analyses.

Q5. Relation to FOCAL++

We thank the reviewer for pointing out this connection. We will add the following to related work:

FOCAL++ addresses context-based offline meta-RL under sparse rewards and distribution shift. It combines batch-wise gated attention, sequence-wise self-attention over transition components, and inter-task matrix momentum contrast. STeR shares the task-inference goal and groups transitions into content-based slots, independently computes within-slot means/variances, then routes summaries. Multiple weak responses can thereby form distinguishable statistics before task information is selected.

Q6. Query, key and value dimensions

We thank the reviewer for identifying the ambiguity. Slot queries and transition keys are both projected to dgd_g, making κt⊤qk(ℓ)/dg\kappa_t^{\top}q_k^{(\ell)}/\sqrt{d_g} dimensionally consistent. dsd_s is the slot-state width before projection. Values are also projected to dgd_g; their weighted aggregate enters the GRU to update the slot state. We will distinguish these widths in Figure 2.

Reviewer_UrP8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates offline meta-RL under sparse task evidence, where useful task cues only exist within an unknown short segment of trajectories. Prior context encoders score individual transitions first and struggle with faint scattered evidence.

The authors propose STeR, a slot-based encoder adopting an accumulation-before-selection design. It softly groups transitions into slots, aggregates mean-variance statistics inside each slot to gather weak evidence, and leverages a router to produce task embeddings, with a dedicated background slot for irrelevant transitions. Two-stage training and regularizers prevent slot collapse.

Three custom simulation benchmarks (ADSE, HCSE, DDE) are built with behavior policies decoupled from tasks to exclude context shift. Evaluations combine 1-NN task identification and frozen-encoder TD3+BC downstream control. STeR surpasses baselines, especially on hard delayed/short evidence settings.

Critical strengths
  1. It identifies a meaningful, understudied problem: sparse-evidence offline meta-RL, where task-relevant cues only appear in unknown short trajectory segments.
  2. The core inductive bias (accumulation-before-selection) is well-motivated: aggregating weak evidence within slots before selection avoids the failure mode of per-step scoring on faint signals.
  3. The proposed slot encoder captures both mean and variance statistics of transitions, which is effective for extracting subtle dynamical cues. A dedicated background slot naturally suppresses irrelevant interactions.
  4. Comprehensive evaluations: combining task identification and downstream offline control, plus diagnostic ablation and attribution experiments to verify model behavior.
Critical weaknesses
  1. The synthetic datasets are collected by task-agnostic behavior policies, eliminating context shift for isolated study of sparse-evidence meta-RL. However, real-world offline datasets typically come from task-dependent behavior policies and inherently contain context shift. Many competing baselines are designed exactly to tackle context shift; their poor performance on the proposed task-agnostic data may not reflect their practical capability. The effectiveness of STeR under realistic context shift remains unvalidated.
  2. Two of the three environments (ADSE and HCSE) rely on artificial reward masking to create sparse evidence windows. This construction is highly contrived, and the practical necessity of such scenarios is unclear. Only the DDE benchmark constructs sparse evidence naturally through contact dynamics without reward manipulation.
Questions for authors

none

2 comments
AuthorsOpenReview ↗

Response to Reviewer UrP8 (Part 1)

We thank the reviewer for the suggestions on collection shift and sparse-evidence benchmarks. We address W1 here and W2 in Part 2.

W1. Task-dependent collection and context shift

We thank the reviewer for highlighting this issue. STeR retains a strong identification advantage under changed collection behavior on ADSE and HCSE IID/Unseen tasks. We also report where combining task extrapolation with collection shift remains difficult.

1. Controlled changes in collection behavior. We hold physical tasks and labels fixed and change support/query collection at evaluation. ADSE splits collectors 0–3 and 4–7 into groups A/B, assigning tasks alternately to groups in task-ID order. Supports and unshifted queries use the assigned group; shifted queries use the other, reversing the task–collector association. HCSE splits collectors 0–5 and 6–10 and tests both association reversal and cross-group shift. The latter averages A→A/B→B without shift and A→B/B→A with shift. DDE changes between the true task damping reference and one common fixed reference.

All comparisons use normal trajectories, disjoint physical support/query episodes, and unchanged encoders and identification rules. ADSE/HCSE match evidence duration in five-step bins within each task and average two seeds and three draws. Per-task trajectories on each side are 17/18/18 for ADSE IID/Unseen/OOD and 13/12/15 for HCSE. DDE uses five OOD tasks, 16 trajectories per task per side, and two models. Table 1 arrows indicate support-to-query references; unshifted queries use the support reference.

Table 1. Identification before and after collection shift (%).

EnvironmentShiftMethodSplitNo shiftWith shift
ADSEAssociation reversalSTeRIID99.5198.63
ADSEAssociation reversalSTeRUnseen99.4499.91
ADSEAssociation reversalSTeROOD96.3095.74
ADSEAssociation reversalPure attentionIID76.1871.13
ADSEAssociation reversalPure attentionUnseen83.6179.07
ADSEAssociation reversalPure attentionOOD60.2839.54
ADSEAssociation reversalMean poolIID76.3234.95
ADSEAssociation reversalMean poolUnseen89.2666.39
ADSEAssociation reversalMean poolOOD56.3012.69
HCSECross-groupSTeRIID99.7799.77
HCSECross-groupSTeRUnseen98.7597.78
HCSECross-groupSTeROOD71.4539.11
HCSECross-groupUNICORNIID30.657.93
HCSECross-groupUNICORNUnseen45.8320.83
HCSEAssociation reversalSTeROOD90.005.78
DDETrue→FixedSTeROOD100.0021.88
DDEFixed→TrueSTeROOD99.3828.13

2. Relative performance under collection shift. After ADSE association reversal, STeR reaches 95.74% OOD identification, versus 39.54% for Pure attention and 12.69% for Mean pool. On HCSE cross-group IID/Unseen, STeR reaches 99.77/97.78%, versus UNICORN's 7.93/20.83%. These comparisons show that STeR can extract task-discriminative information even when collector groups or task–collector associations change.

3. Combined task extrapolation and collection shift. HCSE OOD falls to 39.11% under cross-group shift and 5.78% under association reversal. DDE reaches 100.00/99.38% within the two reference settings but 21.88/28.13% across them. Using supports with the same reference restores high accuracy on the same queries. Thus, STeR retains identifiable task information under both collection schemes, while matching representations across references remains sensitive to collection behavior. We will add these controls, clarify that task-independent collection isolates sparse-evidence effects, and discuss the observed strengths and remaining difficulties under collection shift.

AuthorsOpenReview ↗

Response to Reviewer UrP8 (Part 2)

W2. Reward masking and naturally sparse dynamics evidence

We thank the reviewer for asking how the benchmarks relate to practical problems. STeR's benefits extend beyond zero rewards marking evidence: it also uses limited native contact evidence more effectively than direct attention.

Drawer damping becomes observable through contact and pulling responses; pre-contact motion does not provide equivalent information. ADSE/HCSE control the location and duration of reward information on the same physical trajectories, isolating delayed evidence, evidence count and background interference. DDE tests these issues through native contact dynamics. We will clarify this experimental purpose and connection in the revision.

1. Reward-location cues. We keep physical trajectories, task parameters, evidence windows and within-window rewards fixed. Only outside-window rewards are replaced with task-independent training reward blocks, sampled independently for support/query, so background and evidence reward ranges overlap. Each method receives 80k updates with independent validation selection: STeR uses 60k initial training plus 20k readout refinement; the two reward-pooling encoders train end to end. Results average six OOD conditions equally.

Table 2. Identification with original and overlapping background rewards (%).

Environment and rewardsSTeRNonzero-reward meanAbsolute-reward weighting
ADSE original rewards91.4854.3562.53
ADSE overlapping background81.9011.0211.51
HCSE original rewards67.1851.9853.39
HCSE overlapping background60.7826.0430.47

STeR retains 81.90/60.78% on ADSE/HCSE when background rewards also become nonzero. Reward-based pooling then includes rewarded background together with evidence; STeR instead groups responses by content and selects task-relevant slot summaries.

2. Fixed contact evidence with added background. On DDE OOD onset, we retain native rewards and the same contact evidence, adding only pre-contact transitions. Support/query each contain eight trajectories per task across five tasks, with disjoint physical episodes and collection using the true damping reference. STeR uses its complete model; all four controls share frozen transition features, training batches and 2,000 updates. Results average two seeds and three draws.

Table 3. DDE OOD onset identification with fixed evidence (%).

Method0 added background64 added background
STeR91.2593.33
Slot statistics pooling (mean and variance)92.5089.58
Slot statistics pooling (mean only)85.0086.25
Direct attention86.2580.42
Global mean85.0052.50

With added background, STeR retains 93.33%, versus 80.42% for direct attention. Slot statistics pooling with mean and variance also retains 89.58% on the same features, supporting aggregation within slots before selecting task information.

3. Fixed background with limited evidence. On DDE OOD normal, we retain 16 background transitions and vary contact-evidence count. Support/query each contain 80 disjoint trajectories; method settings, reference, model seeds and draws follow Table 3.

Table 4. DDE OOD normal identification with 16 background transitions (%).

Method1 evidence4 evidence16 evidenceAll evidence
STeR36.6760.2168.9699.79
Slot statistics pooling (mean and variance)38.7543.5452.5096.46
Slot statistics pooling (mean only)34.1737.9243.3388.75
Direct attention20.2122.7137.0890.21
Global mean21.8820.8328.7565.63

From one to four evidence transitions, STeR improves from 36.67% to 60.21%, while direct attention improves from 20.21% to 22.71%. Direct attention scores each transition against the whole context, so weak responses compete with background. STeR first accumulates similar responses into distinguishable summaries. With abundant evidence, direct attention recovers to 90.21%, while STeR reaches 99.79%; its advantage is already substantial when evidence is scarce. ADSE shows the same trend: with 32 background transitions, STeR rises from 33.49% with one evidence transition to 96.95% with all evidence. We will add these controls and analyses to the revision.

Reviewer_8EbtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies offline meta-RL when task information appears in only a small part of a context trajectory. STeR softly groups transitions into slots, summarizes each slot with a weighted mean and variance, and combines the summaries into a task embedding. The idea is to accumulate weak evidence before deciding which parts matter. Training uses task labels, without annotations of where the evidence appears.

Experiments cover three controlled benchmarks: Ant direction, HalfCheetah target speed and a drawer with hidden damping. Against five representation-learning baselines using the same TD3+BC controller, STeR reports large gains on the two reward-masked benchmarks, including Ant return of 483 versus 265. On the drawer task, success improves from 83% to 98%. Results are reported over five seeds with standard deviations, alongside ablations of grouping, slot statistics, routing and training choices.

Critical strengths
  1. The sparse-evidence setting is concrete and worth studying. The experiments separately vary when evidence appears and how long it lasts, making it possible to examine these effects directly. The drawer benchmark is particularly useful: hidden damping and dense rewards extend the evaluation beyond artificially masking reward signals.

  2. The main comparisons are well controlled. Methods share the task-identification readout, downstream TD3+BC controller and training budget, with five-seed means and standard deviations. The improvements also carry over to control, including drawer success rising from 83% to 98%, so the benefit goes beyond separating task embeddings. The attribution diagnostics also report imperfect evidence/background separation, which helps clarify the method’s limits.

Critical weaknesses
  1. The reward channel makes evidence much easier to locate on two benchmarks. In ADSE and HCSE, context rewards are exactly zero outside the evidence window, giving the encoder a strong cue about which transitions matter. The paper does not compare against a simple encoder that filters transitions with nonzero raw reward or weights them by reward magnitude. Without this control, the large gains do not establish how much slots contribute beyond exploiting the reward cue.

  2. The baseline results need stronger validation. Several established methods perform near chance even on IID tasks: ContraBAR reaches 5.23% on ADSE, where chance is 5%, and 9.01% on HCSE, where chance is about 9%. These results warrant implementation checks, baseline-specific hyperparameters and a clear tuning protocol. Reproducing their performance on ordinary dense-reward tasks would help establish that the comparison reflects sparse-evidence difficulty rather than unsuitable training choices.

  3. Robustness is less consistent on the hidden-dynamics benchmark. On DDE OOD with long evidence, STeR’s task-identification accuracy is 52.7±23.3%, versus BOReL’s 78.9±5.8%; downstream success also falls behind, at 86.5% versus 92.6%. STeR leads in most DDE conditions, but these reversals and large variability matter for the claimed robustness. The main text gives them little attention and should explain where the method struggles.

  4. The experiments do not fully isolate the proposed mechanism. The main structural ablations are confined to ADSE, where reward already helps locate evidence, and the mean–variance readout is not separately compared with a mean-only version. On HCSE, removing background transitions lowers task identification from 99.2% to 64.7%. Because support prototypes still come from complete contexts, this also introduces a support–query mismatch. The results therefore leave unclear how reliably slots separate useful evidence from background.

  5. Relevant representation-learning baselines are missing. Meta-DT is cited and explicitly learns task representations through a context-aware world model, yet it is not compared. A shared TD3+BC controller does not rule out evaluating its encoder under the same downstream protocol. T2DA also learns dynamics-aware representations and deserves discussion, with its additional language supervision clearly accounted for. These omissions leave the advantage over other representation-learning approaches insufficiently tested.

Questions for authors

Q1 Critical control. Please compare STeR with an end-to-end mean-pool encoder using only transitions with nonzero raw reward, and separately with pooling weighted by absolute raw reward. How do task identification and downstream control compare across the six ADSE/HCSE conditions? This would directly test whether slots add value beyond the reward cue.

Q2 Baseline clarification. Please identify the code sources, modifications, training objectives, hyperparameters and tuning protocol for each baseline, including what “FOCAL++” and “ContraBAR (sup.)” mean. What checks establish that these implementations recover reasonable performance on standard dense-reward tasks? Please also provide the seed-level results behind CTMRL’s 42.6±44.9% DDE accuracy.

Q3 Training clarification. For Mean pool, GRU and Pure attention, which parameters are updated before and after the 60k-step boundary? Are any feature encoders frozen during readout refinement? If so, how do these architectures perform when trained end-to-end on the task loss under the same total budget?

Q4 Critical benchmark control. Do the ADSE/HCSE gains survive when task-independent background rewards overlap the informative rewards, so that nonzero reward no longer identifies the evidence window? Please keep the physical trajectories and task parameters fixed. Without this control, how much of the claimed unknown-location challenge is actually tested?

Q5 Attribution clarification and control. In HCSE, are behavior-source assignment and onset thresholds sampled independently of target speed? Please state the sampling procedure. Does the 99.2% → 64.7% drop after background removal persist when support contexts and query contexts undergo the same intervention, rather than retaining full-context support prototypes?

Q6 DDE clarification. What explains STeR falling behind BOReL on IID and OOD short/long task identification, particularly OOD long at 52.7±23.3% versus 78.9±5.8%? Why does delayed onset reduce IID accuracy from 100% to 76.6%, while downstream success remains high? How should these results qualify the robustness claim?

Q7 Missing comparison. Meta-DT explicitly learns task representations through a context-aware world model. Why is its encoder absent from the comparison, and can it be evaluated with the same context budget and TD3+BC controller? Please also explain the relation to T2DA and which comparisons would require additional language supervision.

Q8 Mechanism isolation. With slots, router and training budget fixed, how much does adding variance improve over a mean-only slot readout? Do the structural gains over direct attention and global pooling also hold on DDE, where evidence is not marked by zero rewards? These comparisons would help isolate the contribution of accumulation before selection.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 8Ebt (Part 1)

We thank the reviewer for the detailed assessment. We address Q1–Q3 below, Q4–Q6 in Part 2, and Q7–Q8 in Part 3; the corresponding weaknesses are addressed alongside these questions.

Q1 / W1. Do slots add value beyond reward cues?

We thank the reviewer for suggesting these controls. Nonzero-reward mean averages features only where raw reward is nonzero; absolute-reward weighting uses normalized absolute raw rewards. Both encoders train end to end for 80k updates, with independent validation selection.

Table 1. ADSE OOD identification (%).

ConditionSTeRNonzero meanAbsolute weighting
normal97.7555.6266.72
onset97.9154.3862.19
short76.0547.8159.38
long99.9060.4765.31
onset-short77.6846.8857.97
onset-long99.5860.9463.59

The six-condition means are 91.48/54.35/62.53% on ADSE and 67.18/51.98/53.39% on HCSE, in the same method order. With the common TD3+BC controller, mean OOD returns are 295.50/41.74/78.07 on ADSE and −128.53/−268.60/−227.73 on HCSE. STeR has higher control return in all six conditions of both environments. Thus, reward filtering alone does not replace STeR's organization and aggregation of task responses. Q4 tests this further when background rewards overlap with evidence rewards.

Q2 / W2. Baseline implementations and validation

We thank the reviewer for requesting these details. We retain each method's representation objective, adapt its data interface, freeze its encoder, and attach the common TD3+BC controller. Table 2 links the code sources and lists the principal fixed settings.

Table 2. Representation objectives and configurations.

Method / codeObjective and task-ID usePrincipal settings
UNICORNReconstruction and task-ID metric lossLatent 5; hidden 64; 16 tasks/batch; metric weight 0.15; LR 3e−4
BOReLVAE and KL; task-balanced samplingLatent 5; GRU 128; 4 tasks × 8 trajectories; KL weight 0.05; LR 3e−4
CTMRLTask metric, CLUB, quantization and contrastLatent 20; hidden 200; codebook 4; 100 context + 256 reconstruction transitions/task; LR 3e−4
FOCAL++Within-task attention and inter-task matrix momentum contrastEmbedding 20; hidden 128; 3 blocks, 8 heads; 512 transitions/task; LR 3e−3
ContraBAR (sup.)CPC; task IDs exclude same-task negativesADSE latent/GRU 512, HCSE/DDE 50; 10/20 trajectories/batch; lookahead 10; 15 negatives; LR 1e−4 → 5e−5 at update 1200

FOCAL++ is a paper-based reproduction using reusable UNICORN network modules. In ContraBAR, sup. means supervised: we use the released precise negative sampler with task IDs; positives remain future transitions from the same trajectory. BOReL concatenates posterior mean and log variance for policy input.

Configurations are fixed before runs and shared across evidence conditions. The main comparison uses 200k representation updates. Every 10k updates, independent validation ranks checkpoints by identification, negative log-likelihood, then earlier step; four pre-final checkpoints and the final checkpoint are retained. Policy checkpoints are also validation-selected; test results do not select parameters.

Dense ADSE/HCSE validation removes reward masking while retaining trajectories and task splits (20k encoder/50k policy updates). CTMRL and BOReL achieve 94.79/79.82% ADSE IID identification and 448.95/373.09 return. ContraBAR reaches 13.10% on ADSE and 89.58% on HCSE: it learns task information, although ADSE identification remains difficult. STeR reaches 99.53/95.00% OOD identification; its ADSE OOD return of −163.53 also shows that high identification does not automatically imply high control return.

The five CTMRL DDE IID runs are approximately 10.2, 9.1, 10.3, 90.9, 92.5%, giving 42.6 ± 44.9%: three runs remain near chance, while two reach about 91%. We will add these implementation, selection, and seed-level details.

Q3. Training before and after 60k

We thank the reviewer for identifying the ambiguity. Mean pool, GRU and Pure attention train their feature encoders and readout modules end to end for all 200k updates; no module is frozen at 60k. The 60k initial stage followed by readout refinement applies only to STeR. We will correct Appendix D.5. Under the same 200k budget, STeR exceeds all three variants in the six ADSE OOD conditions in Figure 5.

AuthorsOpenReview ↗

Response to Reviewer 8Ebt (Part 2)

Q4 / W1. Overlapping background rewards

We thank the reviewer for proposing this control. STeR still identifies tasks when nonzero rewards alone cannot locate the evidence. We keep physical trajectories, task parameters, evidence windows and within-window rewards fixed. Only outside-window rewards are replaced with short, task-independent training-data reward blocks, independently sampled for support and query, so background and evidence reward ranges overlap.

All methods receive 80k representation updates and independent validation selection. STeR retains its 60k initial stage plus 20k readout refinement; the reward-pooling encoders train end to end.

Table 3. Six-condition mean OOD identification with overlapping rewards (%).

EnvironmentSTeRNonzero meanAbsolute weighting
ADSE81.9011.0211.51
HCSE60.7826.0430.47

Reward filtering and magnitude weighting now include rewarded background together with evidence. STeR instead groups responses by transition content, accumulates them within slots, and selects task-relevant summaries. Its strong advantage in both environments shows that the gains extend beyond nonzero-reward localization. We will add this control and analysis to the revision.

Q5 / W4. HCSE collection and symmetric deletion

We thank the reviewer for identifying the support–query mismatch. Every HCSE task uses the same quotas over 11 collectors, with one collector generating each trajectory. The target velocity determines reward; the collector and evidence condition determine a cumulative forward-displacement threshold. Evidence starts at the first of 30 consecutive steps at or above that threshold, with subsequent stability checked. Thus, collector assignment and the onset rule do not use the task's target velocity.

The earlier deletion experiment removed background only from queries while retaining full supports. We corrected this by applying the same deletion to both sets, holding the three frozen models, physical support/query episodes, normalization and identification rule fixed. Support and query episodes are disjoint. Results average three seeds and the six conditions equally.

Table 4. HCSE identification under background deletion (%).

SplitFull support/queryDelete query background onlyDelete support/query background
IID99.2064.7396.50
Unseen98.3976.5696.56
OOD73.3332.0070.45

Symmetric deletion restores most of the lost accuracy; the remaining gap is below three percentage points in every split. STeR still identifies tasks from evidence alone. The earlier large drop mainly reflected inconsistent support/query treatment. We will correct the evaluation and its interpretation in the revision.

Q6 / W3. DDE evidence duration and control

We thank the reviewer for highlighting these reversals. On DDE OOD long, STeR's identification is 52.7 ± 23.3%, below BOReL's 78.9 ± 5.8%. STeR summarizes response distributions within slots; changing evidence duration changes these summaries and their distances to normal-condition support prototypes.

To examine this, we fix queries and compare 32 normal supports per task with 32 same-condition supports, using disjoint physical episodes. In two frozen STeR models, OOD long identification rises from 83.75% and 39.38% to 100.00% in both. Same-condition support also gives 98.13–100.00% on IID short, IID long and OOD short. The queries retain identifiable damping information, and matching support/query evidence conditions substantially improves identification.

Identification and control also use representations differently. On IID normal versus onset, identification is 100.0% versus 76.6%, while success is 98.08% versus 98.00%. Cosine 1-NN assigns a discrete task label; TD3+BC directly receives the continuous embedding and current state, without using that label. Crossing a classification boundary therefore need not cause control failure.

OOD long nevertheless remains weaker in both metrics: control success is 86.50 ± 2.80%, versus 92.60 ± 3.30% for BOReL. We will add the support-condition controls and clarify STeR's sensitivity to evidence-duration changes, discussing identification and control separately.

AuthorsOpenReview ↗

Response to Reviewer 8Ebt (Part 3)

Q7 / W5. Meta-DT and T2DA

We thank the reviewer for suggesting this comparison. We evaluate the Meta-DT encoder with the same context budget and TD3+BC controller. ADSE/HCSE use its recurrent context encoder and reward decoder; DDE uses its encoder and next-state decoder. Each encoder trains for 20k updates, with validation prediction error checked every 1k. Across three seeds, the best checkpoints occur at 10k–18k, 18k–19k and 13k–19k, respectively. Each run uses its lowest-validation-error encoder.

Identification uses at most 200 transitions, averages causal four-step window embeddings, and applies cosine 1-NN. The prediction target is excluded from its input history. Encoders are then frozen for 200k updates of the common TD3+BC policy; policy checkpoints are selected independently on validation data.

Table 5. OOD task identification and downstream performance. ADSE/HCSE average six conditions; DDE uses normal. Downstream means return for ADSE/HCSE and success (%) for DDE.

EnvironmentSTeR identification (%)Meta-DT identification (%)STeR downstreamMeta-DT downstream
ADSE91.4815.98 ± 0.08295.5027.12 ± 15.83
HCSE67.1827.27 ± 2.32−128.53−173.90 ± 5.96
DDE99.30 ± 0.7061.25 ± 25.6890.31 ± 1.610.00 ± 0.00

STeR has higher identification and return in all six ADSE/HCSE conditions. Meta-DT learns predictive information: all three ADSE models outperform constant-zero reward prediction. On DDE, zeroing or permuting the latent increases prediction error, yet substituting another episode's latent from the same or a different task has similar effects. Accurate local prediction does not necessarily yield embeddings that reliably match tasks across trajectories. STeR directly optimizes this task discrimination and aggregates dispersed responses before selecting task information; its advantage also carries over to control.

T2DA pairs trajectories with natural-language task descriptions and trains a text-conditioned policy. Such a comparison adds language supervision; STeR uses transitions and task IDs without task descriptions. We will add the Meta-DT comparison and clarify this distinction.

Q8 / W4. Variance and accumulation before selection

We thank the reviewer for requesting these mechanism controls on native DDE rewards, where zero rewards do not mark evidence.

Table 6. DDE OOD normal identification (%), with 16 background transitions fixed.

Method1 evidence4 evidence16 evidenceAll evidence
STeR36.6760.2168.9699.79
Slot statistics pooling (mean and variance)38.7543.5452.5096.46
Slot statistics pooling (mean only)34.1737.9243.3388.75
Direct attention20.2122.7137.0890.21
Global mean21.8820.8328.7565.63

The four pooling baselines use the same frozen transition features. The slot variants also share slot assignments, router/output-projection architectures, training batches and a 2k-update budget; they differ in the variance branch. Support/query each contain 80 disjoint trajectories. Results average two model seeds and three draws.

Adding variance improves identification from 37.92% to 43.54% with four evidence transitions and from 88.75% to 96.46% with all evidence. A separate intervention in the fixed complete STeR model, using normal supports and averaging two seeds, confirms its use of these statistics:

Table 7. Fixed-model interventions on DDE OOD identification (%).

Settingonsetonset-shortlong
STeR59.0653.4458.13
Remove variance39.5331.8843.91
Shared mean denominator31.7237.8145.94

With six onset contact transitions fixed, adding 64 background transitions changes STeR from 91.25% to 93.33%, direct attention from 86.25% to 80.42%, and global mean from 85.00% to 52.50%. This control uses 40 disjoint trajectories each for support/query, two models and three draws. Direct attention scores individual transitions against the entire context, so weak evidence competes with background. STeR first accumulates similar responses within independently normalized slots, then selects summaries. This explains its stronger use of limited evidence and resistance to added background. We will add these controls and analysis to the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.