acceptodds
Withdrawn from ICLR 2027

Reachable Information Gain: Task-Relevant Goal Curricula for Sparse-Reward Long-Horizon Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sparse-reward long-horizon reinforcement learning requires agents to discover intermediate behaviors that make distant rewards reachable. Existing methods often select subgoals by novelty, controllability, or reachability, but such signals may not reveal progress toward final success, especially in tasks with distractors. To address this limitation, we introduce Reachable Information Gain (RIG), an evaluation criterion for candidate progress signals in sparse-reward long-horizon reinforcement learning. RIG is motivated by an ideal latent progress criterion, but uses observable multi-step progress over task predicates or goal potentials because latent task progress is not directly identifiable from observations alone. We instantiate RIG as an online teacher for goal-conditioned off-policy learning. The teacher learns context-dependent utilities from delayed multi-step evidence and supplies curriculum rewards and replay priorities without requiring successful trajectories. Experiments across sparse-reward manipulation, navigation, and discrete-control benchmarks show that RIG improves both sample efficiency and success rate over strong baselines, demonstrating its effectiveness across diverse long-horizon tasks. The code is available at https://anonymous.4open.science/r/RIG.

Primary area:
reinforcement learning
Keywords:
Long-horizon reinforcement learning, goal-conditioned learning, sparse-reward

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_795WOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a new method to improve exploration and sample efficiency in sparse-reward environments. The main idea is to incorporate a notion of task progress or relevance into the reward function and replay buffer priorities, encouraging the agent to visit states that contribute to reaching the task goal.

Critical strengths

Please refer to the weaknesses section.

Critical weaknesses
  1. My main concern is the clarity of the paper. There is no clear explanation of how the progress signal zz is determined in each environment. Is this signal manually defined for each environment? If so, this raises concerns about the generalization of the proposed method, as there is no clear procedure for defining zz in new environments.

  2. The literature review is very limited, and the paper could have discussed more existing works on improving exploration in sparse-reward settings.

  3. I had a difficult time understanding the paper and could not clearly identify the motivation behind the proposed method. For instance, what is Figure 2 actually illustrating?

  4. Based on Figure 3, the proposed method outperforms the baselines in only 3 out of 12 environments. In the remaining environments, all methods achieve approximately the same final success rates.

  5. Another concern is the motivation behind the proposed method. In environments with distractors, the policy should learn to avoid those distractors and continue toward the goal. However, the proposed method appears to avoid the states where distractors are located rather than learning how to act appropriately in those states. As a result, if the policy is initialized in one of these states, it may not be able to select an appropriate action.

Questions for authors

Please refer to the Weaknesses section.

Reviewer_X2AEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper scores candidate progress signals by how much their hand-specified potentials increase over a short window, and uses a context-conditioned bandit teacher to pick which signal becomes the curriculum reward and to prioritize replay. It reports faster learning on manipulation, navigation and grid-world tasks, including settings with distractor objects.

Critical strengths

The distractor variants of the manipulation task are a useful testbed, and the finding that progress-based replay prioritization alone helps a lot is interesting. The paper is also transparent about the manual interface it relies on.

Critical weaknesses
  1. The method is close to prior work. It combines hand-designed stage rewards, a learned scheduler and progress-based prioritization, much like SAC-X, teacher-student curriculum learning (TSCL) and energy-based hindsight experience prioritization, none of which is discussed. The information-theoretic quantity that motivates the paper is not what the algorithm computes. SAC-X: M. Riedmiller et al. Learning by Playing: Solving Sparse Reward Tasks from Scratch. ICML, 2018. TSCL: T. Matiisen et al. Teacher-Student Curriculum Learning. IEEE Transactions on Neural Networks and Learning Systems, 2020. Energy-based hindsight experience prioritization: R. Zhao and V. Tresp. Energy-Based Hindsight Experience Prioritization. CoRL, 2018.
  2. The theory adds little. The non-identifiability result is a standard construction, the information bound is the data processing inequality, and the ranking guarantee rests on a calibration assumption that, by the paper's own theorem, cannot be checked from observations.
  3. The teacher is not really a bandit, and the comparisons are confounded. Candidates are scored on data that was already collected, so the exploration bonus gathers no information. RIG also gets dense hand-designed rewards and oracle stage information that the sparse-reward baselines never see, and the obvious control, the same potentials used as a fixed dense reward, is missing.
  4. Reporting is inconsistent. The same configuration gives noticeably different results across the ablation tables, the promised confidence intervals and extra controls are absent, and some navigation curves are cut off long before the end of training.
Questions for authors
  1. Could you add, on Panda-Gym and the distractor tasks, (a) DDPG+HER with the same potentials as a fixed dense reward (summed, or panda-gym's built-in dense reward), (b) a fixed context-to-next-candidate curriculum, (c) a SAC-X or TSCL style scheduler, and (d) energy-based prioritization instead of RIG-P? If (a) or (b) comes close to Online-RIG, the learned selection is not what drives the results.
  2. What are ρ, C, Z and Φ_z for AntMaze, and do they encode bottleneck or key locations? Does Online-RIG use SSE's graph machinery there (L1015 mentions initial graph coverage)? What are the stage rules and valid subgoals in MiniGrid, and which reported results use context masks (L1069)?
  3. Why update only the selected arm when u^H can be computed for all candidates on the same segment? How does a variant that updates all arms and selects greedily perform, how are indicator and continuous potentials made comparable, and how do the distractor estimates evolve early in training on the biased layout?
  4. Please give the budget, number of seeds and variance for Tables 1-4, explain the spread of AUC values for what appears to be the same configuration, add the promised confidence intervals and landmark/random-feature results, and report final success at the full 5M-step budget for DoubleBottleneck and AntKeyChest.
  5. Given Theorem 1, how could Assumption 1 ever be verified? Could you relate u^H to an identifiable quantity, for example the information that candidate evidence carries about eventual success, or at least compare u^H (rather than RIG_MI) with CIG in the tabular model?
Reviewer_kTv5OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article proposes a method called RIG, used to evaluate subgoals in sparse-reward long-term tasks.

Critical strengths

The article is very well organised and logical.

Critical weaknesses

(Honestly, I did not really understand the paper)

  1. The writing of this paper is not easy to follow.
  2. It is not very clear how the RIG evaluation criterion is applied to the RL model.
  3. It is also not very clear how the proposed method is trained.
Questions for authors
  1. Is this RIG plug-and-play?
  2. How is RIG trained and how is it applied to the RL model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.