- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Temporal Self-Imitation Learning (TSIL), which provides additional self-supervisory signals for policy optimization by exploiting temporally efficient successful trajectories discovered during reinforcement learning. It first proposes adaptive temporal target learning (ATTL), which tightens temporal targets using the fastest successful trajectories so far and adds a target-conditioned temporal bonus to the success reward. It also proposes weighted self-imitation, which prioritizes fast successful trajectories and applies an efficiency-weighted self-imitation objective to preserve and reproduce effective behaviors. Experiments on MiniGrid and Meta-World show that TSIL achieves favorable performance and robustness.
Critical strengths
The problem addressed by TSIL is valuable and practically relevant. Recording successful trajectories and extracting temporal-efficiency supervision are particularly relevant for long-horizon manipulation.
The main results are interetsting. The authors compare TSIL with other baseline methods, and separately discuss the role of the ATTL component. For example, on MiniGrid, ATTL alone reaches an AUC of 73, higher than the other baselines, and the full TSIL reaches 88.2, showing that both ATTL and replay are effective.
Critical weaknesses
It remains unclear whether TSIL offers substantial novelty beyond a combination of SIL and temporal components. The paper evaluates several individual baselines but does not examine relevant combinations, such as SIL augmented with Step-cost or ATTL combined with standard SIL. The ablation studies also do not sufficiently establish the independent contributions of the proposed components, so the novelty claim is not yet well supported.
For unseen MiniGrid evaluation configurations, the authors infer the temporal target by averaging the targets from the top-5 nearest successful training configurations. This design assumes that configuration distance somehow reflects similarity in task completion time or learning difficulty. However, this assumption has not been sufficiently validated. In particular, it remains unclear whether configuration distance correlates with actual completion times and how sensitive the results are to the number of nearest neighbors and the choice of distance metric. For Meta-World, however, the authors do not describe the corresponding configurations in detail.
Questions for authors
Could the authors provide more comprehensive ablation studies, including comparisons between ATTL combined with standard SIL, ATTL combined with fast-success replay without efficiency weighting, and variants that remove temporal conditioning or temporal reward shaping?
Is configuration similarity significantly correlated with task completion time? Does changing the number of nearest neighbors or the distance metric significantly affect performance on unseen configurations?
The authors report ATTL alone in the main table, but do not report the replay strategy alone. Could they also test a model that uses only the replay strategy as a baseline?
7 comments
Response to Reviewer 2/4
W2. For unseen MiniGrid evaluation configurations, the authors infer the temporal target by averaging the targets from the top-5 nearest successful training configurations. This design assumes that configuration distance somehow reflects similarity in task completion time or learning difficulty. However, this assumption has not been sufficiently validated. In particular, it remains unclear whether configuration distance correlates with actual completion times and how sensitive the results are to the number of nearest neighbors and the choice of distance metric. For Meta-World, however, the authors do not describe the corresponding configurations in detail.
Q2. Is configuration similarity significantly correlated with task completion time? Does changing the number of nearest neighbors or the distance metric significantly affect performance on unseen configurations?
Response. We now provide the detailed definitions of the distance between configurations for each benchmark. We further examine whether similar configurations improve temporal-target estimation and whether the inference choices affect unseen-configuration performance.
Configurations and distances. All comparisons are within the same task. For Meta-World, let contain the initial object and goal/fixture coordinates (e.g., peg-start and fixture-center xyz), and let be coordinate 's range across training configurations. The default Euclidean and alternative Manhattan distances are
\qquad d_{\mathrm{MW}}^{\mathrm{alt}}(z,z')=\sum_j\frac{|z_j-z_j'|}{r_j},$$ with constant coordinates contributing zero. For MiniGrid, configurations contain the initial symbolic grid $G$, agent position $(x,y)$, orientation $\theta$, and mission $m$. For a $W\times H$ grid, define $$L=\frac{1}{WH}\sum_{u,v}\mathbf{1}[G_{uv}\ne G'_{uv}], \qquad P=\frac{|x-x'|}{W}+\frac{|y-y'|}{H}, \qquad O=\mathbf{1}[\theta\ne\theta'], \qquad M=\mathbf{1}[m\ne m'].$$ Here, grid cells encode object type, color, and state. The default distance sums the four components. The alternative equally weights the four normalized components: $$d_{\mathrm{MG}}=L+P+O+M, \qquad d_{\mathrm{MG}}^{\mathrm{alt}}=\frac{L+P/2+O+M}{4}.$$ **Temporal target estimation accuracy.** We predict each training configuration's learned temporal target—the shortest successful duration recorded up to the checkpoint—from the other successful training configurations, excluding the query itself. We use all available training labels from 32 configurations per MiniGrid task and 6,400 per Meta-World task. We compare KNN with $K=1,3,5$, five randomly selected targets without replacement, $K=5$ under the alternative distance, and the mean target across all training configurations. **Policy sensitivity.** We evaluate the same six estimators using fixed TSIL checkpoints, unchanged evaluation configurations and resets, and 2,000 episodes per task and seed. Results are task averages, reported as mean ± SD across three seeds. Configurations without successful target labels are excluded from MAE. MG = MiniGrid, MW = Meta-World. MAE is measured in steps for MG and seconds for MW. SR is evaluation success (%). | Estimator | MG MAE $\downarrow$ | MW MAE $\downarrow$ | MG SR $\uparrow$ | MW SR $\uparrow$ | | --- | --- | --- | --- | --- | | $K=1$ | 2.766 ± 0.002 | 0.0189 ± 0.0051 | 93.30 ± 0.75 | 91.31 ± 3.98 | | $K=3$ | 2.287 ± 0.031 | 0.0166 ± 0.0044 | 93.14 ± 0.48 | 91.40 ± 3.96 | | $K=5$ | **2.203 ± 0.039** | **0.0160 ± 0.0042** | 93.28 ± 0.63 | 91.37 ± 3.94 | | $K=5$ (random) | 2.599 ± 0.030 | 0.0541 ± 0.0022 | 93.34 ± 0.77 | 91.36 ± 3.84 | | $K=5$ (alt. dist.) | 2.327 ± 0.025 | 0.0160 ± 0.0042 | 93.28 ± 0.77 | 91.35 ± 3.86 | | All-source mean | 2.298 ± 0.034 | 0.0494 ± 0.0016 | 93.34 ± 0.75 | 91.41 ± 3.93 | $K=5$ estimation improves MAE over random-five and the task-wide mean, particularly in Meta-World. Evaluation success varies by only 0.20 percentage points in MiniGrid and 0.10 in Meta-World across the six choices. Thus, policy performance is robust to these inference choices.Response to Reviewer 3/4
Distance-metric-free deployment. Using the same trained TSIL checkpoints, we replace the inferred temporal target with the mean temporal target across all training configurations, zero, or the maximum episode horizon . This tests simpler temporal targets for unseen configurations without distance metrics or KNN estimation. Results use the same three seeds and evaluation protocol. NCT denotes evaluation normalized completion time (%), assigning the full horizon to failed episodes as in the paper.
| Evaluation target | MG SR | MG NCT | MW SR | MW NCT |
|---|---|---|---|---|
| All-source mean | 93.34 ± 0.75 | 23.35 ± 0.04 | 91.41 ± 3.93 | 35.16 ± 3.53 |
| 91.88 ± 0.41 | 24.68 ± 1.08 | 88.08 ± 3.58 | 38.81 ± 2.32 | |
| 93.53 ± 0.58 | 25.75 ± 0.67 | 76.25 ± 1.07 | 48.22 ± 0.07 |
The all-source mean provides a distance-free inference option with success rates comparable to KNN. Compared with the all-source mean, setting reduces success in both benchmarks, and setting substantially reduces Meta-World success and worsens NCT in both benchmarks. These results suggest that suitable temporal targets help the policy maintain success and efficiency during evaluation, even without precise configuration-specific estimates. If no distance metric is available, the mean temporal target across all training configurations can serve as a substitute. A suitable temporal target is still preferred, as a large mismatch reduces deployment performance.
Response to Reviewer 1/4
We thank the reviewer for their thoughtful and constructive comments. We address each comment point by point below.
Response. We would like to clarify that our key contribution is to turn self-discovered fast successes into reusable temporal supervision: their completion times define adaptive temporal targets, and their trajectories are relabeled under the current targets for efficiency-weighted fast-success replay.
We thank the reviewer for pointing out additional ablations and baselines to better understand the benefits of each component of TSIL. Here, we provide additional baselines for both benchmarks. The training and evaluation settings and metric definitions remain the same as those in our paper. TSIL (w/o EW) removes efficiency weighting, making it ATTL + fast-success replay. TSIL (no time inputs) removes both elapsed time and the adaptive temporal target from the policy and value inputs. TSIL (fixed bonus) replaces the temporally conditioned success reward with a constant reward. We use the TSIL training logs to calibrate the magnitude of the success reward. We use 186.83 for MiniGrid and 184.08 for Meta-World.
MiniGrid
Meta-World
We clearly see that, under the same SIL replay method, our ATTL method outperforms the step-cost method in both benchmarks. Replacing SIL with fast-success replay (TSIL (w/o EW)) further improves the success rate. Without temporal observations or with a constant bonus, TSIL experiences a performance drop. On the Meta-World benchmark in particular, the success rate drops from 95.14% to 81.11% and 86.77%, respectively.
We will add these comparisons and variant definitions to the revision.