acceptodds
Under review as a conference paper at ICLR 2027

Temporal Self-Imitation Learning

Abstract

Long-horizon policies trained with reinforcement learning can still achieve high return through inefficient interactions, while rare efficient behaviors discovered during training may be forgotten. We argue that temporal efficiency itself provides a source of self-supervision for reinforcement learning. We introduce Temporal Self-Imitation Learning (TSIL), a reinforcement learning framework that mines temporally efficient successful trajectories generated during learning and converts them into reusable supervision for future policy improvement. TSIL progressively refines learning using configuration-conditioned adaptive temporal targets derived from fast successful trajectories, while preserving and replaying efficient behaviors through efficiency-weighted self-imitation learning. Across 30 long-horizon tasks spanning robot manipulation and interactive navigation, TSIL consistently improves learning efficiency, behavioral efficiency, revisitation of fast successful behaviors, and robustness to unstable training conditions.

Primary area:
reinforcement learning
Keywords:
Robot Learning, Reinforcement Learning, Self-Imitation Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xD71OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Temporal Self-Imitation Learning (TSIL), which provides additional self-supervisory signals for policy optimization by exploiting temporally efficient successful trajectories discovered during reinforcement learning. It first proposes adaptive temporal target learning (ATTL), which tightens temporal targets using the fastest successful trajectories so far and adds a target-conditioned temporal bonus to the success reward. It also proposes weighted self-imitation, which prioritizes fast successful trajectories and applies an efficiency-weighted self-imitation objective to preserve and reproduce effective behaviors. Experiments on MiniGrid and Meta-World show that TSIL achieves favorable performance and robustness.

Critical strengths

The problem addressed by TSIL is valuable and practically relevant. Recording successful trajectories and extracting temporal-efficiency supervision are particularly relevant for long-horizon manipulation.

The main results are interetsting. The authors compare TSIL with other baseline methods, and separately discuss the role of the ATTL component. For example, on MiniGrid, ATTL alone reaches an AUC of 73, higher than the other baselines, and the full TSIL reaches 88.2, showing that both ATTL and replay are effective.

Critical weaknesses

It remains unclear whether TSIL offers substantial novelty beyond a combination of SIL and temporal components. The paper evaluates several individual baselines but does not examine relevant combinations, such as SIL augmented with Step-cost or ATTL combined with standard SIL. The ablation studies also do not sufficiently establish the independent contributions of the proposed components, so the novelty claim is not yet well supported.

For unseen MiniGrid evaluation configurations, the authors infer the temporal target by averaging the targets from the top-5 nearest successful training configurations. This design assumes that configuration distance somehow reflects similarity in task completion time or learning difficulty. However, this assumption has not been sufficiently validated. In particular, it remains unclear whether configuration distance correlates with actual completion times and how sensitive the results are to the number of nearest neighbors and the choice of distance metric. For Meta-World, however, the authors do not describe the corresponding configurations in detail.

Questions for authors

Could the authors provide more comprehensive ablation studies, including comparisons between ATTL combined with standard SIL, ATTL combined with fast-success replay without efficiency weighting, and variants that remove temporal conditioning or temporal reward shaping?

Is configuration similarity significantly correlated with task completion time? Does changing the number of nearest neighbors or the distance metric significantly affect performance on unseen configurations?

The authors report ATTL alone in the main table, but do not report the replay strategy alone. Could they also test a model that uses only the replay strategy as a baseline?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 1/4

We thank the reviewer for their thoughtful and constructive comments. We address each comment point by point below.

W1. It remains unclear whether TSIL offers substantial novelty beyond a combination of SIL and temporal components. The paper evaluates several individual baselines but does not examine relevant combinations, such as SIL augmented with Step-cost or ATTL combined with standard SIL. The ablation studies also do not sufficiently establish the independent contributions of the proposed components, so the novelty claim is not yet well supported.

Q1. Could the authors provide more comprehensive ablation studies, including comparisons between ATTL combined with standard SIL, ATTL combined with fast-success replay without efficiency weighting, and variants that remove temporal conditioning or temporal reward shaping?

Response. We would like to clarify that our key contribution is to turn self-discovered fast successes into reusable temporal supervision: their completion times define adaptive temporal targets, and their trajectories are relabeled under the current targets for efficiency-weighted fast-success replay.

We thank the reviewer for pointing out additional ablations and baselines to better understand the benefits of each component of TSIL. Here, we provide additional baselines for both benchmarks. The training and evaluation settings and metric definitions remain the same as those in our paper. TSIL (w/o EW) removes efficiency weighting, making it ATTL + fast-success replay. TSIL (no time inputs) removes both elapsed time and the adaptive temporal target from the policy and value inputs. TSIL (fixed bonus) replaces the temporally conditioned success reward with a constant reward. We use the TSIL training logs to calibrate the magnitude of the success reward. We use 186.83 for MiniGrid and 184.08 for Meta-World.

MiniGrid

MethodAUC ↑\uparrowTrain success ↑\uparrowNorm. completion time ↓\downarrowSuccess count ↑\uparrowEval success ↑\uparrow
Step-cost + SIL49.05 ± 0.1977.21 ± 10.1429.46 ± 9.3926.62 ± 2.5950.76 ± 7.38
ATTL + SIL80.62 ± 11.6489.90 ± 14.1114.54 ± 13.3466.60 ± 4.2685.37 ± 13.43
TSIL (w/o EW)82.71 ± 7.7693.44 ± 9.1611.05 ± 9.1067.25 ± 3.2787.86 ± 9.83
TSIL (no time inputs)71.50 ± 4.9286.31 ± 0.0918.24 ± 0.1562.68 ± 2.0565.20 ± 1.08
TSIL (fixed bonus)84.24 ± 6.9596.62 ± 4.778.00 ± 4.5866.86 ± 2.5282.05 ± 5.01
TSIL88.66 ± 1.3999.96 ± 0.004.69 ± 0.1569.58 ± 0.0893.46 ± 1.29

Meta-World

MethodAUC ↑\uparrowTrain success ↑\uparrowNorm. completion time ↓\downarrowSuccess count ↑\uparrowEval success ↑\uparrow
Step-cost + SIL19.26 ± 3.1529.42 ± 8.0780.14 ± 5.02165.94 ± 34.9129.40 ± 8.27
ATTL + SIL58.26 ± 5.1385.90 ± 6.6839.29 ± 4.44671.16 ± 50.4583.47 ± 4.28
TSIL (w/o EW)58.72 ± 7.0591.75 ± 0.4834.63 ± 0.64670.01 ± 80.1887.69 ± 0.52
TSIL (no time inputs)51.40 ± 1.3181.11 ± 3.9041.72 ± 3.28577.17 ± 22.5977.44 ± 3.95
TSIL (fixed bonus)59.49 ± 1.2186.77 ± 3.1039.39 ± 1.45651.39 ± 11.5386.04 ± 2.24
TSIL61.05 ± 2.7795.14 ± 4.3932.42 ± 3.92704.07 ± 26.3391.37 ± 3.94

We clearly see that, under the same SIL replay method, our ATTL method outperforms the step-cost method in both benchmarks. Replacing SIL with fast-success replay (TSIL (w/o EW)) further improves the success rate. Without temporal observations or with a constant bonus, TSIL experiences a performance drop. On the Meta-World benchmark in particular, the success rate drops from 95.14% to 81.11% and 86.77%, respectively.

We will add these comparisons and variant definitions to the revision.

AuthorsOpenReview ↗

Response to Reviewer 2/4

W2. For unseen MiniGrid evaluation configurations, the authors infer the temporal target by averaging the targets from the top-5 nearest successful training configurations. This design assumes that configuration distance somehow reflects similarity in task completion time or learning difficulty. However, this assumption has not been sufficiently validated. In particular, it remains unclear whether configuration distance correlates with actual completion times and how sensitive the results are to the number of nearest neighbors and the choice of distance metric. For Meta-World, however, the authors do not describe the corresponding configurations in detail.

Q2. Is configuration similarity significantly correlated with task completion time? Does changing the number of nearest neighbors or the distance metric significantly affect performance on unseen configurations?

Response. We now provide the detailed definitions of the distance between configurations for each benchmark. We further examine whether similar configurations improve temporal-target estimation and whether the inference choices affect unseen-configuration performance.

Configurations and distances. All comparisons are within the same task. For Meta-World, let zz contain the initial object and goal/fixture coordinates (e.g., peg-start and fixture-center xyz), and let rjr_j be coordinate jj's range across training configurations. The default Euclidean and alternative Manhattan distances are

\qquad d_{\mathrm{MW}}^{\mathrm{alt}}(z,z')=\sum_j\frac{|z_j-z_j'|}{r_j},$$ with constant coordinates contributing zero. For MiniGrid, configurations contain the initial symbolic grid $G$, agent position $(x,y)$, orientation $\theta$, and mission $m$. For a $W\times H$ grid, define $$L=\frac{1}{WH}\sum_{u,v}\mathbf{1}[G_{uv}\ne G'_{uv}], \qquad P=\frac{|x-x'|}{W}+\frac{|y-y'|}{H}, \qquad O=\mathbf{1}[\theta\ne\theta'], \qquad M=\mathbf{1}[m\ne m'].$$ Here, grid cells encode object type, color, and state. The default distance sums the four components. The alternative equally weights the four normalized components: $$d_{\mathrm{MG}}=L+P+O+M, \qquad d_{\mathrm{MG}}^{\mathrm{alt}}=\frac{L+P/2+O+M}{4}.$$ **Temporal target estimation accuracy.** We predict each training configuration's learned temporal target—the shortest successful duration recorded up to the checkpoint—from the other successful training configurations, excluding the query itself. We use all available training labels from 32 configurations per MiniGrid task and 6,400 per Meta-World task. We compare KNN with $K=1,3,5$, five randomly selected targets without replacement, $K=5$ under the alternative distance, and the mean target across all training configurations. **Policy sensitivity.** We evaluate the same six estimators using fixed TSIL checkpoints, unchanged evaluation configurations and resets, and 2,000 episodes per task and seed. Results are task averages, reported as mean ± SD across three seeds. Configurations without successful target labels are excluded from MAE. MG = MiniGrid, MW = Meta-World. MAE is measured in steps for MG and seconds for MW. SR is evaluation success (%). | Estimator | MG MAE $\downarrow$ | MW MAE $\downarrow$ | MG SR $\uparrow$ | MW SR $\uparrow$ | | --- | --- | --- | --- | --- | | $K=1$ | 2.766 ± 0.002 | 0.0189 ± 0.0051 | 93.30 ± 0.75 | 91.31 ± 3.98 | | $K=3$ | 2.287 ± 0.031 | 0.0166 ± 0.0044 | 93.14 ± 0.48 | 91.40 ± 3.96 | | $K=5$ | **2.203 ± 0.039** | **0.0160 ± 0.0042** | 93.28 ± 0.63 | 91.37 ± 3.94 | | $K=5$ (random) | 2.599 ± 0.030 | 0.0541 ± 0.0022 | 93.34 ± 0.77 | 91.36 ± 3.84 | | $K=5$ (alt. dist.) | 2.327 ± 0.025 | 0.0160 ± 0.0042 | 93.28 ± 0.77 | 91.35 ± 3.86 | | All-source mean | 2.298 ± 0.034 | 0.0494 ± 0.0016 | 93.34 ± 0.75 | 91.41 ± 3.93 | $K=5$ estimation improves MAE over random-five and the task-wide mean, particularly in Meta-World. Evaluation success varies by only 0.20 percentage points in MiniGrid and 0.10 in Meta-World across the six choices. Thus, policy performance is robust to these inference choices.
AuthorsOpenReview ↗

Response to Reviewer 3/4

Distance-metric-free deployment. Using the same trained TSIL checkpoints, we replace the inferred temporal target with the mean temporal target across all training configurations, zero, or the maximum episode horizon HH. This tests simpler temporal targets for unseen configurations without distance metrics or KNN estimation. Results use the same three seeds and evaluation protocol. NCT denotes evaluation normalized completion time (%), assigning the full horizon to failed episodes as in the paper.

Evaluation targetMG SR ↑\uparrowMG NCT ↓\downarrowMW SR ↑\uparrowMW NCT ↓\downarrow
All-source mean93.34 ± 0.7523.35 ± 0.0491.41 ± 3.9335.16 ± 3.53
D=0D=091.88 ± 0.4124.68 ± 1.0888.08 ± 3.5838.81 ± 2.32
D=HD=H93.53 ± 0.5825.75 ± 0.6776.25 ± 1.0748.22 ± 0.07

The all-source mean provides a distance-free inference option with success rates comparable to K=5K=5 KNN. Compared with the all-source mean, setting D=0D=0 reduces success in both benchmarks, and setting D=HD=H substantially reduces Meta-World success and worsens NCT in both benchmarks. These results suggest that suitable temporal targets help the policy maintain success and efficiency during evaluation, even without precise configuration-specific estimates. If no distance metric is available, the mean temporal target across all training configurations can serve as a substitute. A suitable temporal target is still preferred, as a large mismatch reduces deployment performance.

Reviewer_85LuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Temporal Self-Imitation Learning (TSIL), which augments PPO with a configuration-specific completion-time target and an auxiliary buffer of fast successful trajectories. The target conditions the policy input and shapes the terminal success reward; stored trajectories are relabelled under the current target and replayed using a positive return-value gate and efficiency weighting.

Critical strengths

(i)The paper identifies a real failure mode of long-horizon PPO and turns it into a simple, coherent training mechanism. Adaptive temporal supervision and fast-success replay address reward distraction and forgetting in complementary ways. (ii)The empirical scope is strong. Thirty tasks span sparse and dense rewards, image and state observations, and discrete and continuous control. TSIL improves success, AUC, and the composite completion metric in both domains, with per-task curves, disturbed-training tests, and component analyses in the appendix.

Critical weaknesses

(i)The novelty positioning and the central replay claim need sharper evidence. TSIL adds configuration-conditioned adaptive targets, relabelling, and PPO integration, but the paper lacks an ATTL plus high-return SIL baseline that would isolate fast-success selection from the other additions. (ii)Normalized completion time assigns a full horizon to every failure, so it mixes success probability with speed among successful episodes. The aggregate Meta-World means in Figure 4 imply a success-weighted completion time of about 30.0% for TSIL and 29.1% for ATTL; TSIL's better composite value there mainly comes from higher success. The evidence supports reliability more strongly than faster successful behavior, and three seeds with cross-task aggregation leave uncertainty about task-level consistency.

Questions for authors

(i) Can the authors compare high-return replay and fast-success replay on the same ATTL backbone, with identical observations, target-shaped rewards, replay budget, and optimizer, and clarify TSIL's difference from XTX and MobileRL? (ii) Can the authors report success-conditioned completion time and paired task-level confidence intervals, and include a reward-magnitude-matched control for the target-conditioned terminal bonus?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1/4

We thank the reviewer for their thoughtful and constructive comments. We address each comment point by point below.

W1. The novelty positioning and the central replay claim need sharper evidence. TSIL adds configuration-conditioned adaptive targets, relabelling, and PPO integration, but the paper lacks an ATTL plus high-return SIL baseline that would isolate fast-success selection from the other additions.

Q1. Can the authors compare high-return replay and fast-success replay on the same ATTL backbone, with identical observations, target-shaped rewards, replay budget, and optimizer, and clarify TSIL's difference from XTX and MobileRL?

Response. Thanks for raising this point. We have now followed the reviewer's suggestion to include the requested comparisons.

Fast-success replay improves all five metrics in MiniGrid and four of five in Meta-World compared with high-return replay on the ATTL backbone. ATTL + SIL replays high-return trajectories, while TSIL (w/o EW) replays fast successful trajectories without efficiency weighting. All methods below use the same observations and target-conditioned rewards. The replay variants also use the same replay budget.

MiniGrid

MethodAUC ↑\uparrowTrain success ↑\uparrowNorm. completion time ↓\downarrowSuccess count ↑\uparrowEval success ↑\uparrow
ATTL72.78 ± 3.6479.71 ± 0.0624.22 ± 0.1056.77 ± 1.1461.13 ± 3.66
ATTL + SIL80.62 ± 11.6489.90 ± 14.1114.54 ± 13.3466.60 ± 4.2685.37 ± 13.43
TSIL (w/o EW)82.71 ± 7.7693.44 ± 9.1611.05 ± 9.1067.25 ± 3.2787.86 ± 9.83
TSIL88.66 ± 1.3999.96 ± 0.004.69 ± 0.1569.58 ± 0.0893.46 ± 1.29

Meta-World

MethodAUC ↑\uparrowTrain success ↑\uparrowNorm. completion time ↓\downarrowSuccess count ↑\uparrowEval success ↑\uparrow
ATTL55.73 ± 3.1484.68 ± 1.2439.95 ± 1.92648.72 ± 39.5084.91 ± 0.58
ATTL + SIL58.26 ± 5.1385.90 ± 6.6839.29 ± 4.44671.16 ± 50.4583.47 ± 4.28
TSIL (w/o EW)58.72 ± 7.0591.75 ± 0.4834.63 ± 0.64670.01 ± 80.1887.69 ± 0.52
TSIL61.05 ± 2.7795.14 ± 4.3932.42 ± 3.92704.07 ± 26.3391.37 ± 3.94

Full TSIL with efficiency weighting further improves all five metrics and achieves the best performance in both benchmarks. Our conclusions hold.

We would like to distinguish TSIL with XTX and MobileRL in three aspects:

  • Motivation: XTX addresses exploration bottlenecks caused by sparse rewards and large action spaces. MobileRL addresses sparse success signals, uneven task difficulty, and costly sampling in mobile environments. TSIL addresses reward distraction and forgetting of efficient successful behavior discovered during learning. Adaptive temporal targets encourage efficient task completion, while replay helps preserve the discovered behavior.

  • Mechanism: XTX samples trajectories by score, favors shorter trajectories at the same score, and switches between exploitation and exploration within each episode. MobileRL combines shortest-path reward adjustment and difficulty-adaptive positive replay. TSIL uses adaptive temporal targets to augment policy and value inputs and shape success rewards. As faster successes are discovered by the policy, the targets tighten, stored trajectories are relabeled, and replay updates are weighted by efficiency relative to the current target. TSIL integrates these updates into policy updates and trains from scratch using self-generated experience, without MobileRL’s demonstration-based supervised warm-up.

  • Evaluation domains: XTX studies text-adventure games, while MobileRL studies mobile GUI interaction. We evaluate TSIL on 30 MiniGrid and Meta-World tasks with image or state observations, discrete or continuous control, and sparse or dense rewards under both standard and disturbed training settings.

We will add these comparisons and analyses to the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 2/4

W2. Normalized completion time assigns a full horizon to every failure, so it mixes success probability with speed among successful episodes. The aggregate Meta-World means in Figure 4 imply a success-weighted completion time of about 30.0% for TSIL and 29.1% for ATTL. TSIL's better composite value there mainly comes from higher success. The evidence supports reliability more strongly than faster successful behavior, and three seeds with cross-task aggregation leave uncertainty about task-level consistency.

Q2. Can the authors report success-conditioned completion time and paired task-level confidence intervals, and include a reward-magnitude-matched control for the target-conditioned terminal bonus?

Response. Our normalized completion time intentionally combines reliability and speed by assigning the full horizon to failed episodes, including timeouts. This avoids favoring methods that solve only easy configurations. Comparing success-conditioned completion time separately for each method can mix differences in speed with differences in which configurations it solves. Requiring success shared by all methods can instead leave a smaller comparison subset as more methods are included. We therefore report pairwise shared-success completion time for TSIL versus ATTL and TSIL versus SIL as additional evidence of behavioral efficiency.

For each task, we evaluate each method for 2,000 episodes per seed on the same evaluation configurations. We select shared-success configurations separately for TSIL–ATTL and TSIL–SIL: a configuration is retained only if both methods in the pair succeed on it at least once in every seed. For each retained configuration and seed, we use only each method's successful episodes, divide their durations by the task horizon, and average them. We then average these configuration-level times within each task and across supported tasks, and report the mean and SD across three seeds. Thus, both methods are compared on the same configurations, while their successful episodes may differ. Each pair has its own shared-success subset, so TSIL's reported time can differ between the two comparisons.

Differences are TSIL − ATTL or TSIL − SIL, as indicated for each pair, in percentage points. We report mean ± SD across three seeds and paired 95% Student's t confidence intervals, computed as dˉ±t0.975,2sd3\bar{d} \pm t_{0.975,2}\frac{s_d}{\sqrt{3}}, where dˉ\bar{d} and sds_d are the mean and sample SD of the three paired seed differences. Success-rate differences use all evaluation configurations. Time differences use successful episodes on each pair's shared-success configurations. Episode counts include all attempts on the retained configurations, but failed episodes do not enter the time averages. HH denotes the task horizon. Supported tasks have at least one shared-success configuration.

In the tables below, times are percentages of HH, and Δ denotes TSIL minus the paired baseline. Task-level differences are shown as mean ± SD [95% CI]. Shared Configs and Shared Episodes refer to retained configurations and all attempts on them. Episode counts are per method and seed. Tasks = supported tasks.

TSIL versus ATTL.

BenchmarkATTL Time ↓\downarrowTSIL Time ↓\downarrowShared ConfigsTasksShared Episodes
MiniGrid12.63 ± 1.799.70 ± 0.3924511/1515,313
Meta-World28.39 ± 0.7927.99 ± 0.8222,46012/1522,460
BenchmarkΔ Time95% CI
MiniGrid-2.93 ± 2.18[-8.35, +2.49]
Meta-World-0.40 ± 0.03[-0.47, -0.33]

MiniGrid

TaskΔ SR ↑\uparrowΔ Time ↓\downarrowShared ConfigsShared Episodes
T000.00 ± 0.00 [+0.00, +0.00]-0.23 ± 0.11 [-0.50, +0.04]322,000
T010.14 ± 2.80 [-6.82, +7.10]-4.25 ± 2.60 [-10.71, +2.21]13812
T0318.30 ± 18.18 [-26.86, +63.46]-5.17 ± 15.62 [-43.97, +33.63]181,126
T048.20 ± 29.71 [-65.60, +82.00]-1.51 ± 2.17 [-6.90, +3.88]211,314
T054.51 ± 6.59 [-11.86, +20.88]-0.18 ± 0.14 [-0.53, +0.17]311,938
T062.82 ± 5.63 [-11.17, +16.81]-2.80 ± 2.06 [-7.92, +2.32]291,811
T074.11 ± 9.14 [-18.60, +26.82]-3.02 ± 1.46 [-6.65, +0.61]271,686
T084.96 ± 2.81 [-2.02, +11.94]-2.77 ± 4.99 [-15.17, +9.63]301,875
T0932.36 ± 35.17 [-55.01, +119.73]-7.60 ± 10.11 [-32.71, +17.51]15938
T134.60 ± 8.35 [-16.14, +25.34]-4.16 ± 1.57 [-8.06, -0.26]12752
T149.18 ± 6.50 [-6.97, +25.33]-0.51 ± 0.46 [-1.65, +0.63]171,061
AuthorsOpenReview ↗

Response to Reviewer 3/4

Meta-World

TaskΔ SR ↑\uparrowΔ Time ↓\downarrowShared ConfigsShared Episodes
T00-1.77 ± 2.16 [-7.14, +3.60]-3.44 ± 2.83 [-10.47, +3.59]1,7001,700
T010.17 ± 0.74 [-1.67, +2.01]-0.36 ± 1.58 [-4.28, +3.56]1,9011,901
T030.35 ± 0.00 [+0.35, +0.35]-0.12 ± 0.95 [-2.48, +2.24]1,9081,908
T14-0.02 ± 0.04 [-0.12, +0.08]-0.02 ± 0.03 [-0.09, +0.05]1,9991,999
T21-1.25 ± 1.63 [-5.30, +2.80]1.37 ± 1.23 [-1.69, +4.43]1,9301,930
T22-3.60 ± 0.21 [-4.12, -3.08]-0.60 ± 0.60 [-2.09, +0.89]1,6511,651
T270.00 ± 0.07 [-0.17, +0.17]-0.99 ± 1.44 [-4.57, +2.59]1,9981,998
T28-0.03 ± 0.04 [-0.13, +0.07]0.18 ± 0.34 [-0.66, +1.02]1,9691,969
T290.02 ± 0.04 [-0.08, +0.12]0.48 ± 0.33 [-0.34, +1.30]1,9991,999
T30-0.10 ± 0.14 [-0.45, +0.25]1.31 ± 1.83 [-3.24, +5.86]1,9961,996
T430.13 ± 0.53 [-1.19, +1.45]-0.44 ± 0.42 [-1.48, +0.60]1,9061,906
T46-1.55 ± 7.99 [-21.40, +18.30]-2.19 ± 4.05 [-12.25, +7.87]1,5031,503

On MiniGrid, TSIL has an equal or higher mean evaluation success rate on all 11 reported tasks and lower mean success-conditioned completion time on all 11 tasks with shared-success configurations. On Meta-World, TSIL and ATTL show similar behavioral efficiency on this shared-success subset.

TSIL versus SIL.

BenchmarkSIL Time ↓\downarrowTSIL Time ↓\downarrowShared ConfigsTasksShared Episodes
MiniGrid11.48 ± 3.229.21 ± 0.4327013/1518,684
Meta-World39.41 ± 1.7528.81 ± 0.889,2078/159,207
BenchmarkΔ Time95% CI
MiniGrid-2.27 ± 3.50[-10.96, +6.42]
Meta-World-10.60 ± 2.47[-16.73, -4.46]

MiniGrid

TaskΔ SR ↑\uparrowΔ Time ↓\downarrowShared ConfigsShared Episodes
T001.09 ± 1.76 [-3.29, +5.47]-0.03 ± 0.16 [-0.42, +0.37]311,937
T011.20 ± 5.10 [-11.46, +13.86]0.71 ± 1.62 [-3.31, +4.73]15939
T0313.94 ± 17.69 [-30.00, +57.89]-9.54 ± 8.08 [-29.62, +10.55]181,125
T042.55 ± 3.41 [-5.91, +11.00]1.14 ± 4.13 [-9.13, +11.40]261,624
T055.58 ± 7.96 [-14.20, +25.37]-0.00 ± 0.00 [-0.01, -0.00]281,751
T06-1.72 ± 2.92 [-8.99, +5.54]0.45 ± 1.22 [-2.58, +3.48]311,937
T07-0.21 ± 10.13 [-25.39, +24.97]2.67 ± 2.77 [-4.21, +9.55]261,623
T080.95 ± 1.99 [-3.98, +5.89]0.35 ± 0.65 [-1.26, +1.95]311,937
T092.70 ± 5.87 [-11.89, +17.28]-1.99 ± 3.57 [-10.86, +6.88]322,000
T110.00 ± 0.00 [+0.00, +0.00]-1.38 ± 2.38 [-7.29, +4.53]32,000
T1232.66 ± 53.19 [-99.48, +164.80]-22.17 ± 37.67 [-115.73, +71.40]2125
T130.24 ± 14.92 [-36.81, +37.29]0.24 ± 4.78 [-11.63, +12.11]10625
T14-9.00 ± 19.26 [-56.84, +38.84]0.08 ± 0.90 [-2.15, +2.30]171,061

Meta-World

TaskΔ SR ↑\uparrowΔ Time ↓\downarrowShared ConfigsShared Episodes
T011.55 ± 3.99 [-8.35, +11.45]-2.18 ± 0.94 [-4.52, +0.17]1,7721,772
T0382.45 ± 23.52 [+24.02, +140.88]-57.32 ± 9.45 [-80.80, -33.85]11
T2123.62 ± 21.66 [-30.19, +77.42]-5.15 ± 4.17 [-15.52, +5.23]874874
T22-1.22 ± 1.60 [-5.20, +2.77]-0.93 ± 1.12 [-3.72, +1.86]1,5131,513
T270.02 ± 0.03 [-0.06, +0.09]-1.73 ± 2.57 [-8.12, +4.66]1,9971,997
T4312.32 ± 5.13 [-0.43, +25.06]-6.86 ± 2.58 [-13.26, -0.46]1,2711,271
T44-0.05 ± 3.13 [-7.83, +7.73]-0.40 ± 1.36 [-3.78, +2.98]1,7341,734
T4654.08 ± 14.08 [+19.10, +89.06]-10.23 ± 9.28 [-33.27, +12.81]4545

On MiniGrid, TSIL has a lower mean shared-success completion time than SIL, reducing the horizon by 2.27 percentage points, although task-level differences vary. On Meta-World, TSIL has lower mean completion time on all eight supported tasks, with a reduction of 10.60 percentage points. Thus, TSIL shows better average behavioral efficiency than SIL on both shared-success subsets.

Reviewer_pwyuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Temporal Self-Imitation Learning (TSIL), a reinforcement learning framework designed to address reward distraction and forgetting of previous efficient behaviours, improving the learning and behavioural efficiency of long-horizon policies. TSIL progressively refines learning using configuration-conditioned adaptive temporal targets derived from fast successful trajectories, while preserving and replaying efficient behaviours through efficiency-weighted self-imitation learning. The experimental results show that TSIL improves the success rate, learning efficiency and behavioural efficiency across diverse tasks in terms of AUC and normalised completion time.

Critical strengths
  • The authors recognise the problem that maximising return does not necessarily lead to efficient learning and behaviour and that successful behaviour can be forgotten after discovering it. The proposed method is well-motivated.
  • The experiments cover 30 tasks with dense and sparse rewards, state-based or image-based observations, and continuous or discrete control. TSIL consistently achieves improvements in learning efficiency, behavioural efficiency, and robustness to unstable training conditions overall.
Critical weaknesses

None

Questions for authors
  • Although the paper includes comparisons between SIL, ATTL, and TSIL, and also the ablation of efficiency-weighting, it is still unclear what the isolated effect of the replay buffer with fastest trajectories is compared with highest-return trajectories. Could the authors additionally compare TSIL without efficiency-weighting against SIL with adaptive temporal target learning to further isolate the effect of different replay buffers in self-imitation learning?
  • The reported normalised completion time mixes successful and failed trajectories, and its lower value could partly result from a high success rate. Could the authors additionally report the normalised completion time on successful trajectories only to more directly assess the behavioural efficiency?
  • In the experiments, each parallel training environment maintains a fixed task configuration across episode resets. Is it possible to extend TSIL to settings where task configurations are randomly resampled across episodes?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 1/4

We thank the reviewer for their thoughtful and constructive comments. We address each comment point by point below.

Q1. Although the paper includes comparisons between SIL, ATTL, and TSIL, and also the ablation of efficiency-weighting, it is still unclear what the isolated effect of the replay buffer with fastest trajectories is compared with highest-return trajectories. Could the authors additionally compare TSIL without efficiency-weighting against SIL with adaptive temporal target learning to further isolate the effect of different replay buffers in self-imitation learning?

Response. Thanks for raising this point. We have now followed the reviewer's suggestion to include the requested comparisons.

Specifically, fast-success replay improves all five metrics in MiniGrid and four of five in Meta-World compared with high-return replay on the ATTL backbone. ATTL + SIL replays high-return trajectories, while TSIL (w/o EW) replays fast successful trajectories without efficiency weighting. All methods below use the same observations and target-conditioned rewards. The replay variants also use the same replay budget. Full TSIL with efficiency weighting further improves all five metrics and achieves the best performance in both benchmarks. As the results show, our conclusions remain the same. We will add these comparisons to the revision.

MiniGrid

MethodAUC ↑\uparrowTrain success ↑\uparrowNorm. completion time ↓\downarrowSuccess count ↑\uparrowEval success ↑\uparrow
ATTL72.78 ± 3.6479.71 ± 0.0624.22 ± 0.1056.77 ± 1.1461.13 ± 3.66
ATTL + SIL80.62 ± 11.6489.90 ± 14.1114.54 ± 13.3466.60 ± 4.2685.37 ± 13.43
TSIL (w/o EW)82.71 ± 7.7693.44 ± 9.1611.05 ± 9.1067.25 ± 3.2787.86 ± 9.83
TSIL88.66 ± 1.3999.96 ± 0.004.69 ± 0.1569.58 ± 0.0893.46 ± 1.29

Meta-World

MethodAUC ↑\uparrowTrain success ↑\uparrowNorm. completion time ↓\downarrowSuccess count ↑\uparrowEval success ↑\uparrow
ATTL55.73 ± 3.1484.68 ± 1.2439.95 ± 1.92648.72 ± 39.5084.91 ± 0.58
ATTL + SIL58.26 ± 5.1385.90 ± 6.6839.29 ± 4.44671.16 ± 50.4583.47 ± 4.28
TSIL (w/o EW)58.72 ± 7.0591.75 ± 0.4834.63 ± 0.64670.01 ± 80.1887.69 ± 0.52
TSIL61.05 ± 2.7795.14 ± 4.3932.42 ± 3.92704.07 ± 26.3391.37 ± 3.94
AuthorsOpenReview ↗

Response to Reviewer 2/4

Q2. The reported normalised completion time mixes successful and failed trajectories, and its lower value could partly result from a high success rate. Could the authors additionally report the normalised completion time on successful trajectories only to more directly assess the behavioural efficiency?

Response. Our normalized completion time intentionally combines reliability and speed by assigning the full horizon to failed episodes, including timeouts. This avoids favoring methods that solve only easy configurations. Comparing success-conditioned completion time separately for each method can mix differences in speed with differences in which configurations it solves. Requiring success shared by all methods can instead leave a smaller comparison subset as more methods are included. We therefore report pairwise shared-success completion time for TSIL versus ATTL and TSIL versus SIL as additional evidence of behavioral efficiency.

For each task, we evaluate each method for 2,000 episodes per seed on the same evaluation configurations. We select shared-success configurations separately for TSIL–ATTL and TSIL–SIL: a configuration is retained only if both methods in the pair succeed on it at least once in every seed. For each retained configuration, we use only each method's successful episodes, divide their durations by the task horizon, and average them. We then average these configuration-level times within each task and across supported tasks, and report the mean and SD across three seeds. Thus, both methods are compared on the same configurations, while their successful episodes may differ. Each pair has its own shared-success subset, so TSIL's reported time can differ between the two comparisons.

Differences are TSIL − ATTL or TSIL − SIL, as indicated for each pair, in percentage points. We report mean ± SD across three seeds and paired 95% Student's t confidence intervals, computed as dˉ±t0.975,2sd3\bar{d} \pm t_{0.975,2}\frac{s_d}{\sqrt{3}} and sds_d are the mean and sample SD of the three paired seed differences. Success-rate differences use all evaluation configurations. Time differences use successful episodes on each pair's shared-success configurations. Episode counts include all attempts on the retained configurations, but only successful episodes enter the time averages. Supported tasks have at least one shared-success configuration.

In the tables below, HH denotes the task horizon, times are percentages of HH, and Δ denotes TSIL minus the paired baseline. Task-level differences are shown as mean ± SD [95% CI]. Shared Configs and Shared Episodes refer to retained configurations and all attempts on them. Episode counts are per method and seed. Tasks = supported tasks.

TSIL versus ATTL.

BenchmarkATTL Time ↓\downarrowTSIL Time ↓\downarrowShared ConfigsTasksShared Episodes
MiniGrid12.63 ± 1.799.70 ± 0.3924511/1515,313
Meta-World28.39 ± 0.7927.99 ± 0.8222,46012/1522,460
BenchmarkΔ Time95% CI
MiniGrid-2.93 ± 2.18[-8.35, +2.49]
Meta-World-0.40 ± 0.03[-0.47, -0.33]

MiniGrid

TaskΔ SR ↑\uparrowΔ Time ↓\downarrowShared ConfigsShared Episodes
T000.00 ± 0.00 [+0.00, +0.00]-0.23 ± 0.11 [-0.50, +0.04]322,000
T010.14 ± 2.80 [-6.82, +7.10]-4.25 ± 2.60 [-10.71, +2.21]13812
T0318.30 ± 18.18 [-26.86, +63.46]-5.17 ± 15.62 [-43.97, +33.63]181,126
T048.20 ± 29.71 [-65.60, +82.00]-1.51 ± 2.17 [-6.90, +3.88]211,314
T054.51 ± 6.59 [-11.86, +20.88]-0.18 ± 0.14 [-0.53, +0.17]311,938
T062.82 ± 5.63 [-11.17, +16.81]-2.80 ± 2.06 [-7.92, +2.32]291,811
T074.11 ± 9.14 [-18.60, +26.82]-3.02 ± 1.46 [-6.65, +0.61]271,686
T084.96 ± 2.81 [-2.02, +11.94]-2.77 ± 4.99 [-15.17, +9.63]301,875
T0932.36 ± 35.17 [-55.01, +119.73]-7.60 ± 10.11 [-32.71, +17.51]15938
T134.60 ± 8.35 [-16.14, +25.34]-4.16 ± 1.57 [-8.06, -0.26]12752
T149.18 ± 6.50 [-6.97, +25.33]-0.51 ± 0.46 [-1.65, +0.63]171,061
AuthorsOpenReview ↗

Response to Reviewer 3/4

Meta-World

TaskΔ SR ↑\uparrowΔ Time ↓\downarrowShared ConfigsShared Episodes
T00-1.77 ± 2.16 [-7.14, +3.60]-3.44 ± 2.83 [-10.47, +3.59]1,7001,700
T010.17 ± 0.74 [-1.67, +2.01]-0.36 ± 1.58 [-4.28, +3.56]1,9011,901
T030.35 ± 0.00 [+0.35, +0.35]-0.12 ± 0.95 [-2.48, +2.24]1,9081,908
T14-0.02 ± 0.04 [-0.12, +0.08]-0.02 ± 0.03 [-0.09, +0.05]1,9991,999
T21-1.25 ± 1.63 [-5.30, +2.80]1.37 ± 1.23 [-1.69, +4.43]1,9301,930
T22-3.60 ± 0.21 [-4.12, -3.08]-0.60 ± 0.60 [-2.09, +0.89]1,6511,651
T270.00 ± 0.07 [-0.17, +0.17]-0.99 ± 1.44 [-4.57, +2.59]1,9981,998
T28-0.03 ± 0.04 [-0.13, +0.07]0.18 ± 0.34 [-0.66, +1.02]1,9691,969
T290.02 ± 0.04 [-0.08, +0.12]0.48 ± 0.33 [-0.34, +1.30]1,9991,999
T30-0.10 ± 0.14 [-0.45, +0.25]1.31 ± 1.83 [-3.24, +5.86]1,9961,996
T430.13 ± 0.53 [-1.19, +1.45]-0.44 ± 0.42 [-1.48, +0.60]1,9061,906
T46-1.55 ± 7.99 [-21.40, +18.30]-2.19 ± 4.05 [-12.25, +7.87]1,5031,503

On MiniGrid, TSIL has an equal or higher mean evaluation success rate on all 11 reported tasks and lower mean success-conditioned completion time on all 11 tasks with shared-success configurations. On Meta-World, TSIL and ATTL show similar behavioral efficiency on this shared-success subset.

TSIL versus SIL.

BenchmarkSIL Time ↓\downarrowTSIL Time ↓\downarrowShared ConfigsTasksShared Episodes
MiniGrid11.48 ± 3.229.21 ± 0.4327013/1518,684
Meta-World39.41 ± 1.7528.81 ± 0.889,2078/159,207
BenchmarkΔ Time95% CI
MiniGrid-2.27 ± 3.50[-10.96, +6.42]
Meta-World-10.60 ± 2.47[-16.73, -4.46]

MiniGrid

TaskΔ SR ↑\uparrowΔ Time ↓\downarrowShared ConfigsShared Episodes
T001.09 ± 1.76 [-3.29, +5.47]-0.03 ± 0.16 [-0.42, +0.37]311,937
T011.20 ± 5.10 [-11.46, +13.86]0.71 ± 1.62 [-3.31, +4.73]15939
T0313.94 ± 17.69 [-30.00, +57.89]-9.54 ± 8.08 [-29.62, +10.55]181,125
T042.55 ± 3.41 [-5.91, +11.00]1.14 ± 4.13 [-9.13, +11.40]261,624
T055.58 ± 7.96 [-14.20, +25.37]-0.00 ± 0.00 [-0.01, -0.00]281,751
T06-1.72 ± 2.92 [-8.99, +5.54]0.45 ± 1.22 [-2.58, +3.48]311,937
T07-0.21 ± 10.13 [-25.39, +24.97]2.67 ± 2.77 [-4.21, +9.55]261,623
T080.95 ± 1.99 [-3.98, +5.89]0.35 ± 0.65 [-1.26, +1.95]311,937
T092.70 ± 5.87 [-11.89, +17.28]-1.99 ± 3.57 [-10.86, +6.88]322,000
T110.00 ± 0.00 [+0.00, +0.00]-1.38 ± 2.38 [-7.29, +4.53]32,000
T1232.66 ± 53.19 [-99.48, +164.80]-22.17 ± 37.67 [-115.73, +71.40]2125
T130.24 ± 14.92 [-36.81, +37.29]0.24 ± 4.78 [-11.63, +12.11]10625
T14-9.00 ± 19.26 [-56.84, +38.84]0.08 ± 0.90 [-2.15, +2.30]171,061

Meta-World

TaskΔ SR ↑\uparrowΔ Time ↓\downarrowShared ConfigsShared Episodes
T011.55 ± 3.99 [-8.35, +11.45]-2.18 ± 0.94 [-4.52, +0.17]1,7721,772
T0382.45 ± 23.52 [+24.02, +140.88]-57.32 ± 9.45 [-80.80, -33.85]11
T2123.62 ± 21.66 [-30.19, +77.42]-5.15 ± 4.17 [-15.52, +5.23]874874
T22-1.22 ± 1.60 [-5.20, +2.77]-0.93 ± 1.12 [-3.72, +1.86]1,5131,513
T270.02 ± 0.03 [-0.06, +0.09]-1.73 ± 2.57 [-8.12, +4.66]1,9971,997
T4312.32 ± 5.13 [-0.43, +25.06]-6.86 ± 2.58 [-13.26, -0.46]1,2711,271
T44-0.05 ± 3.13 [-7.83, +7.73]-0.40 ± 1.36 [-3.78, +2.98]1,7341,734
T4654.08 ± 14.08 [+19.10, +89.06]-10.23 ± 9.28 [-33.27, +12.81]4545

On MiniGrid, TSIL has a lower mean shared-success completion time than SIL, reducing the horizon by 2.27 percentage points. On Meta-World, TSIL has lower mean completion time on all eight supported tasks, with a reduction of 10.60 percentage points. Thus, TSIL shows better average behavioral efficiency than SIL on both shared-success subsets. The MiniGrid confidence interval includes zero, while the Meta-World interval remains below zero.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.