- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes STeR for offline meta-RL tasks where only a small part of the context carries task information. STeR groups transitions into slots, computes normalized mean–variance summaries, and learns how to combine them into a task embedding. Training uses task labels, auxiliary losses for initial slot formation, and subsequent readout refinement. The analysis gives conditions for identifiability and bounds background contamination of slot means. Experiments on ADSE, HCSE, and DDE evaluate task identification and downstream TD3+BC control as evidence duration, onset, and task parameters change.
Critical strengths
-
COMRL with sparse task evidence is a pratical problem. Grouping related transitions using slot representations before deciding their task relevance is a sensible approach. Figure 5 supports both slot grouping and learned routing on ADSE, particularly when evidence is short.
-
Extensive experiments. The normal-condition gains are large across all three benchmarks; for example, DDE OOD success improves from the best baseline's 79.7% to 90.3%. Using five seeds and a common controller makes these comparisons more useful. The evidence-removal and replacement tests also help establish what the representation relies on.
Critical weaknesses
-
The experiments leave the central explanation unresolved. The key test would hold informative transitions fixed and add background. Changing evidence duration or randomly retaining transitions also changes the available evidence. In ADSE/HCSE, nonzero rewards already identify candidate evidence transitions, so the results do not establish that the clues are individually weak or require accumulation before selection. The ablations are limited to ADSE identification, do not isolate separate normalization, and give too little detail about “Pure attention” to assess the comparison. Contextual attention, such as Set Transformer and Non-Parametric Transformers, can also combine information before pooling.
The collector (i.e., behavior policy) assumption in line 106 needs clarification too. HCSE does not specify the collector mixture for each task, and DDE's “matched” branch could be problematic if its controller uses the true damping. I cannot tell from the description whether this happens. Equal branch counts and withholding collector labels would not rule out task cues in the actions. Removing whole evidence transitions removes those actions as well, so the attribution tests do not settle this concern.
-
High identification accuracy does not translate into robust control in an important setting. ADSE OOD return drops from 379.0 under normal evidence to −293.4 with delayed onset and −293.0 with long evidence (Tables 5/16), despite identification accuracies of 97.9% and 99.9% (Table 9). In contrast, FOCAL++ achieves a positive return of 60.8 in the long condition. This is a substantial failure under the shifts the paper aims to handle, yet it receives little discussion. The authors need to explain the discrepancy and qualify the claims about downstream adaptation.
Questions for authors
-
Collector construction: Is the distribution over actual collectors identical for every task? What changes between DDE's matched, nominal, and mismatched branches, and does any controller use the true task damping?
-
Control failure: Do the ADSE identification and control results use the same frozen encoder checkpoint? What explains the OOD return collapse despite high identification accuracy?
-
Model details: How is “Pure attention” implemented and trained relative to STeR? Please also reconcile Table 3's four total slots with Table 20's five HCSE slots, and explain Appendix F's ADSE plus/minus reward views.
-
Additional evidence: A comparison with fixed evidence and increasing background would directly test the main motivation. Identification from one versus several evidence transitions, and pooling transitions selected by nonzero reward, would help distinguish weak collective clues from readily identifiable sparse cues.
-
Morivation: COMRL with sparse evidence is not a new topic. I believe prior works such as FOCAL++ specifically targets the same problem and propose effective solutions. I suggest the authors to properly discuss it in the related work section.
-
Clarification on slot attention: To compute slot attention in eqn 3, the latent dimension of slot queries and key values should be identical in Figure 2?
Remark: I'm willing to adjust my score if the authors can properly address my concerns above.
Post-rebuttal comments: I appreciate authors' efforts for providing extensive experiments to address my questions. I think most of my concerns regarding experimental results are resolved. However, frankly, as someone who has done research closely related to the topic, I personally feel that the methodology of COMRL is somewhat obsolete in the large-model era, and in-context RL with transformer architecture might be more promising to scale. Therefore, my overall assessment is that the method and experiments of this paper are solid, but the significance/novelty of the topic is somewhat limited. This work is therefore borderline to me, and I have increased my score to 3 to lean slightly towards acceptance.
3 comments
Response to Reviewer VEHm (Part 2)
Q3. Pure attention, slot counts and reward views
We thank the reviewer for requesting these details. Pure attention scores each transition key with one learned query, applies softmax over all valid transitions, and projects the weighted sum. STeR instead groups transitions by content, independently normalizes within-slot means/variances, then routes the summaries.
Pure attention's feature encoder, normalization, key projection, query and output projection train end to end for all 200k updates. Mean pool and GRU likewise remain trainable throughout. The 60k initial stage followed by readout refinement applies only to STeR; we will correct Appendix D.5.
In the paper, Table 3 uses ADSE's three foreground plus one background slot; Table 20 uses HCSE's four foreground plus one background slot. We will label the environments. ADSE views share states, actions, next states, evidence windows and encoder weights; only becomes before shared normalization. The 64-dimensional readout has view-exchange-invariant and sign-changing components. Appendix F averages attribution computed separately for the two views. We will clarify these definitions.
Q4 / W1. Fixed evidence and fixed background
We thank the reviewer for proposing these controls. All five methods use the same support/query trajectories. STeR uses its complete model; four pooling baselines freeze its transition features and train for 2,000 updates on identical batches. Slot pooling also reuses its slot assignments. Results average two models and three draws.
1. Fix evidence; add background. ADSE/HCSE use OOD normal, with 64 disjoint support/query trajectories per task (640/320 per set). DDE uses OOD onset, with eight per task across five tasks. Evidence and native rewards remain fixed.
Table 3. ADSE OOD normal identification (%).
| Method | 0 background | 64 background |
|---|---|---|
| STeR | 96.88 | 96.95 |
| Slot statistics pooling (mean and variance) | 74.69 | 70.26 |
| Slot statistics pooling (mean only) | 54.38 | 47.40 |
| Direct attention | 43.83 | 34.58 |
| Global mean | 46.72 | 31.59 |
Table 4. HCSE OOD normal identification (%).
| Method | 0 background | 64 background |
|---|---|---|
| STeR | 73.59 | 77.50 |
| Slot statistics pooling (mean and variance) | 63.59 | 60.47 |
| Slot statistics pooling (mean only) | 66.41 | 62.19 |
| Direct attention | 69.69 | 68.13 |
| Global mean | 76.25 | 45.36 |
Table 5. DDE OOD onset identification (%).
| Method | 0 background | 64 background |
|---|---|---|
| STeR | 91.25 | 93.33 |
| Slot statistics pooling (mean and variance) | 92.50 | 89.58 |
| Slot statistics pooling (mean only) | 85.00 | 86.25 |
| Direct attention | 86.25 | 80.42 |
| Global mean | 85.00 | 52.50 |
STeR preserves identification as background increases in all three environments. Direct attention normalizes individual responses over the whole context; weak evidence competes with background. STeR first aggregates similar responses within independently normalized slots, reducing that interference before summary selection.
2. Fix background; vary evidence count. ADSE/HCSE retain 32 background transitions using the same trajectories as above. DDE uses OOD normal, 16 fixed background transitions, and 80 disjoint trajectories per support/query set.
Table 6. ADSE OOD normal identification (%).
| Method | 1 evidence | 4 evidence | All evidence |
|---|---|---|---|
| STeR | 33.49 | 67.58 | 96.95 |
| Slot statistics pooling (mean and variance) | 25.81 | 45.36 | 67.86 |
| Slot statistics pooling (mean only) | 24.84 | 34.90 | 45.96 |
| Direct attention | 20.78 | 25.76 | 36.09 |
| Global mean | 18.67 | 23.46 | 34.95 |
Table 7. HCSE OOD normal identification (%).
| Method | 1 evidence | 4 evidence | All evidence |
|---|---|---|---|
| STeR | 44.11 | 52.76 | 75.94 |
| Slot statistics pooling (mean and variance) | 30.63 | 35.57 | 59.22 |
| Slot statistics pooling (mean only) | 31.41 | 38.39 | 61.15 |
| Direct attention | 32.08 | 37.50 | 68.33 |
| Global mean | 18.33 | 21.98 | 49.90 |
Table 8. DDE OOD normal identification (%).
| Method | 1 evidence | 4 evidence | All evidence |
|---|---|---|---|
| STeR | 36.67 | 60.21 | 99.79 |
| Slot statistics pooling (mean and variance) | 38.75 | 43.54 | 96.46 |
| Slot statistics pooling (mean only) | 34.17 | 37.92 | 88.75 |
| Direct attention | 20.21 | 22.71 | 90.21 |
| Global mean | 21.88 | 20.83 | 65.63 |
With four evidence transitions, STeR exceeds direct attention in all three environments. On DDE, STeR reaches 60.21%, versus attention's 22.71%; mean-and-variance slot pooling also reaches 43.54% on the same features. We will add these controls supporting accumulation before selection.
Response to Reviewer VEHm (Part 3)
Q4 / W1. Reward cues and statistical aggregation
We thank the reviewer for asking about reward selection and contextual attention. Nonzero-reward mean filters by raw reward; absolute-reward weighting uses its magnitude. Their encoders train end to end for 80k updates, then TD3+BC for 200k, with independent validation selection. Tables 9–10 average six OOD conditions; baseline control uses seed 101.
Table 9. ADSE OOD reward-pooling comparison.
| Method | Identification (%) | Return | Identification with overlap (%) |
|---|---|---|---|
| STeR | 91.48 | 295.50 | 81.90 |
| Nonzero-reward mean | 54.35 | 85.90 | 11.02 |
| Absolute-reward weighting | 62.53 | 102.24 | 11.51 |
Table 10. HCSE OOD reward-pooling comparison.
| Method | Identification (%) | Return | Identification with overlap (%) |
|---|---|---|---|
| STeR | 67.18 | −128.53 | 60.78 |
| Nonzero-reward mean | 51.98 | −285.08 | 26.04 |
| Absolute-reward weighting | 53.39 | −208.65 | 30.47 |
For overlap, trajectories, task parameters, evidence windows and within-window rewards stay fixed. Only outside-window rewards are replaced with task-independent training reward segments, sampled separately for support/query. All methods receive 80k updates; STeR retains 60k initial training plus 20k refinement. Reward pooling now includes rewarded background; STeR selects content-based summaries and retains its advantage.
DDE rewards are nonzero in both background and contact transitions; 94.82% of query contact rewards fall within the background range. Using five OOD normal tasks, 16 disjoint trajectories per task per set, two frozen models and 2,000-update controls gives:
Table 11. DDE OOD normal identification with native rewards (%).
| Method | Identification |
|---|---|
| STeR | 99.38 |
| Direct attention | 83.75 |
| Nonzero-reward mean (global mean) | 71.88 |
| Absolute-reward weighting | 86.88 |
Variance and independent normalization. In Table 8, mean-and-variance pooling exceeds mean-only pooling with the same features and training budget. We additionally fix two complete STeR models, slot assignments and normal supports, then remove variance or share the slot-mean denominator and recompute routing:
Table 12. Fixed-model interventions on DDE OOD identification (%).
| Setting | onset | onset-short | long |
|---|---|---|---|
| STeR | 59.06 | 53.44 | 58.13 |
| Remove variance | 39.53 | 31.88 | 43.91 |
| Shared mean denominator | 31.72 | 37.81 | 45.94 |
Both interventions reduce accuracy in every condition, supporting variance information and normalization by each slot's assigned weight.
Contextual attention. Set Transformer uses two SAB layers and one PMA layer; NPT alternates transition/feature attention. Both share the frozen features, batches and 2,000-update budget of the pooling baselines.
Table 13. DDE OOD normal evidence-count comparison (%), matching Table 8.
| Method | 1 evidence | 4 evidence | All evidence |
|---|---|---|---|
| Set Transformer | 20.42 | 23.75 | 94.79 |
| NPT | 27.92 | 36.67 | 74.38 |
With four evidence transitions, STeR reaches 60.21%, above both contextual-attention methods; mean-and-variance slot pooling reaches 43.54%. With 64 background transitions in the onset setting of Table 5, Set Transformer/NPT reach 80.42/76.25%, compared with STeR's 93.33%. Contextual attention exploits abundant evidence, while slot statistics make limited evidence more useful amid background. We will add these comparisons and analyses.
Q5. Relation to FOCAL++
We thank the reviewer for pointing out this connection. We will add the following to related work:
FOCAL++ addresses context-based offline meta-RL under sparse rewards and distribution shift. It combines batch-wise gated attention, sequence-wise self-attention over transition components, and inter-task matrix momentum contrast. STeR shares the task-inference goal and groups transitions into content-based slots, independently computes within-slot means/variances, then routes summaries. Multiple weak responses can thereby form distinguishable statistics before task information is selected.
Q6. Query, key and value dimensions
We thank the reviewer for identifying the ambiguity. Slot queries and transition keys are both projected to , making dimensionally consistent. is the slot-state width before projection. Values are also projected to ; their weighted aggregate enters the GRU to update the slot state. We will distinguish these widths in Figure 2.
Response to Reviewer VEHm (Part 1)
We thank the reviewer for the detailed assessment. We address collection and control in Part 1, architecture and evidence controls in Part 2, and reward cues, mechanism tests and related work in Part 3. The corresponding weaknesses are addressed alongside the questions.
Q1 / W2. Collector quotas, damping access and action cues
We thank the reviewer for asking about collector distributions and task information in actions. Within each dataset, every task has identical per-collector quotas: ADSE uses eight equally weighted collectors; HCSE uses the same nonuniform weights over 11 collectors, chosen from pilot forward speeds to cover target speeds. Integer rounding can slightly change proportions between datasets. Training uses 240 ADSE and 437 HCSE trajectories per task; checkpoint selection uses 32 supports/16 queries, and final evaluation 64 supports/64 normal queries per task.
DDE collects three paired trajectories with the same drawer, physical damping, initial state and pulling controller, changing only the reference damping used to compute actions:
The purpose is to expose the same physical task under different action patterns, encouraging inference from dynamical responses rather than one collection template. Each of ten training tasks contributes 160 triplets; each reference contributes one third of every task's data. The 1,600 triplets share their first 24 approach/descent actions. True damping, reference damping and branch labels are excluded from encoder inputs. Episode seeds do not overlap training, validation and test. Encoders train on all three references; frozen-encoder TD3+BC training, validation and final evaluation use matched collection. We will correct line 106 to distinguish shared collector mixtures in ADSE/HCSE from DDE's paired references.
We further evaluate task inference when the controller uses a common reference. On five DDE OOD onset tasks, with eight disjoint support/query trajectories per task, STeR achieves 91.25% without added background and 90.83% with 64 added background transitions, averaging two models and three draws. Thus, task identification remains effective without the collection controller knowing current-task damping.
Table 1. DDE OOD normal identification across references (%), averaging two frozen models with 16 disjoint trajectories per task per set.
Paired mixing reduces reliance on a single action template, but cross-reference matching remains sensitive to collection behavior. We will add the data construction and both controls to make this distinction explicit.
Q2 / W3. Frozen encoders and the ADSE control results
We thank the reviewer for identifying this discrepancy. Each seed uses the same frozen encoder for task identification and policy input, and the same selected policy checkpoints across all six evidence conditions. The negative ADSE OOD returns in the five non-normal conditions are sign typos; magnitudes and standard deviations are unchanged.
Table 2. STeR ADSE OOD identification and corrected return.
On onset and long, STeR's returns are 293.42 and 292.98, compared with 20.71 and 60.77 for FOCAL++; STeR identification is 97.91% and 99.90%. Its task representations therefore support effective control in both conditions, although returns remain below normal's 379.01. Identification uses cosine matching to support prototypes, while the policy directly uses the continuous embedding. Evidence changes can affect actions even when the task label remains correct. We will correct the signs in manuscript Tables 5/16 and add this explanation and the FOCAL++ comparison.