acceptodds
Withdrawn from ICLR 2027

TopoExplore: Homology as an Exploration Signal

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Exploration signals in reinforcement learning are currently computed from what an agent has seen: visitation counts, density estimates, or a model's prediction error at individual states. None of these report the holes in the visitation space. To capture them, we compute the persistent homology of the agent's own archive of visited states during exploration and turn each detected class—an enclosed region the archive surrounds but has not entered—into a selection bonus concentrated on the archived states from which entry is possible, gated by an attempt counter that retires candidate entrances and prunes sealed structures. The method, TopoExplore, is Go-Explore plus one additive term, so the comparison with Go-Explore isolates that term. On the held-out worlds of the open-source TopoGym benchmark of topologically varied hard-exploration environments, TopoExplore finds the goal in worlds against for Go-Explore and, where both find it, with a median k steps against k. On an environment that stress-tests far-apart chambers that create holes in the archive space, TopoExplore enters all six or eight chambers across seeds, against and of seeds for Go-Explore. On Montezuma's Revenge, built over the room graph, the term helps whenever the archive surrounds a room it has not entered, puts –% of the selections on the surrounding rooms while it is active, and enters the room sooner in five of six cases; such enclosures are rare on this game, so the score is unchanged. Beyond discrete, restorable state spaces, the same signal drives an embodied explorer that walks to every target it selects and explores from depth alone; a vision-language-action navigation policy fine-tuned on its trajectories succeeds % more often on unseen scenes than one fine-tuned on shortest-path oracle demonstrations.

Primary area:
reinforcement learning
Keywords:
exploration, reinforcement learning, persistent homology, topological data analysis, Go-Explore, archive-based exploration, hard-exploration, Montezuma's Revenge, embodied navigation, topology

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4DDiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces an extension of Go-Explore that uses the topology of previously visited states to guide exploration. Specifically, it identifies enclosed but unexplored regions and increases the probability of selecting states along their boundaries, while gradually deprioritising unsuccessful exploration attempts. The method improves exploration efficiency on the authors' TopoGym benchmark, but shows no meaningful improvement over Go-Explore on Montezuma's Revenge without additional heuristic signals. The authors also demonstrate that trajectories generated by their method can improve the performance of a fine-tuned navigation policy, although the gains are relatively modest and limited to unseen environments rather than unseen goal categories.

Critical strengths
  • The central idea of using the structure of previously visited states to guide exploration is intuitive and reasonably well motivated.

  • The method builds directly on Go-Explore, making it relatively straightforward to isolate the contribution of the additional exploration signal.

  • The results on TopoGym show consistent improvements in both exploration efficiency and the number of environments solved.

  • The paper provides extensive theoretical analysis and diagnostic experiments, including investigations of where the proposed signal is effective and where it fails.

Critical weaknesses
  • The strongest results come from TopoGym, a benchmark developed by the authors that specifically includes environments with the topological structures their method is designed to exploit. It remains unclear how well the approach generalises to less structured environments.

  • The experiments on Montezuma's Revenge are not particularly convincing. The original method does not improve performance, while the only statistically significant improvement comes from adding texture-based heuristics. Without a Go-Explore + texture baseline, it is impossible to attribute this improvement to the proposed topological signal.

  • Several of the largest improvements in the synthetic experiments come from the additional Ricci curvature signal rather than persistent homology itself. This makes it difficult to assess how much of the reported benefit comes from the paper's central contribution.

  • The embodied navigation results are relatively weak. Although the method improves success from 15% to 18% compared with frontier exploration on unseen scenes, the absolute success rate remains low.

  • Despite the extensive mathematical formulation, it is unclear whether persistent homology offers a meaningful advantage over simpler exploration heuristics. In particular, the TopoGym implementation already uses information about observed but unvisited free space, yet the paper does not compare against a simpler frontier-based method exploiting the same information.

Questions for authors
  1. How much of the improvement on TopoGym specifically comes from identifying enclosed regions, rather than simply prioritising states adjacent to observed but unexplored free space? Could a simpler frontier-based heuristic achieve similar results?

  2. Why is there no Go-Explore + texture baseline for Montezuma's Revenge? Given that the authors attribute most of the improvement to the additional door and key features, this seems essential for isolating the contribution of persistent homology.

  3. How well would the proposed approach generalise to environments where unexplored regions do not form enclosed structures? Is there evidence that the topological signal offers an advantage beyond the environment families explicitly considered in the paper?

Reviewer_5AsROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Efficient exploration is challenging in sparse reward environments, especially hard explore tasks. Go explore may under-estimate exploration signals for regions that are surrounded by visited states. The paper proposed a method that use original go explore with bonus that prioritizes candidate entrances to enclosed regions and retires them after repeated attempts. Experiments on TopoGym, Atari, and simulated embodied navigation show improved exploration on some benchmarks.

Critical strengths
  1. Clearly identified a failure mode of existing, influential method. The paper clearly identified that go-explore may fail under multi-chamber environment in theory and in empirical environment(figure 2).
  2. The paper proposed a clean design to solve the identified problem, supported by theoretical analysis under strong assumptions.
Critical weaknesses
  1. Strong state return assumption. This method strongly relies on exact state restoration, which seems only possible in a simulated environment. And the selected environments are free of any kind of randomness. I wonder how this method will perform under a random environment or real world environment where exact restoration is no longer guaranteed.
  2. Limited variety of experiment environments. Although three environments (gridworld, Atari, and embodied) covers different observation space complexity, the action space remains small and discrete. More experiments on large discrete or continuous action space should be conducted to proof the generalization of the proposed method.
  3. Limited baseline coverage, the main exploration baselines are PPO with RND or ICM (beyond go explore which the method build upon), which are over 7 years ago and a large number of RL exploration was introduced for hard exploration task (which go explore was focusing when proposed). And even with these out dated baselines, the performance gain remains limited.
Questions for authors

Please try to address my listed weakness above.

Reviewer_mZSvOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Paper is an interesting attempt for the selection bonuses in Go Explore. This one is based on topology. For example, a similar one is based on contrastive learning in a probability space to create a different kind of geometry https://arxiv.org/pdf/2101.02055

In this paper, the authors look for a region enclosed by visited states and then prioritize exploration from their boundaries. There is a special way to retire candidates where they were repeatedly unsuccessful, and an archive-dependent waiting prevents the bonus from being diluted.

Critical strengths

I really like the idea of a global structure of the archive rather than one novelty score. In the go-explore comparison, it nicely utilizes the added selection term.

The embedded comparison is also encouraging.

Critical weaknesses

The thing is, the biggest concern of the paper is guaranteeing the theoretical average. In Theorem 3, the author assumes that exploring chambers are free of interior obstacles. Should there be a remaining obstacle or wall that can prevent the claim from filling

I think the final finite fail terms do not certify that entrance is sealed because, in equation 20, you can explicitly permit prune during entry.

Small detailed interim 2: The paper claims they are independent from world size, but at the same time, required budget depends on log n.

Questions for authors

My main question is: how does this Theorem 3 handle a chamber that encloses a class that survives after you have all interior cells that are free visited?

Do the authors have any evidence that distinguishes a real entrance from one exhausted by random attempts? Can TM2 make all dependents explicit, including through the attempt budget? I'd like to see anything that depends on the final guarantee.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thanks To All

We thank all three reviewers for careful and constructive reviews. Three concerns came up in more than one review, and we address them together here.

First, whether persistent homology itself drives the gains. Reviewers 4DDi and mZSv asked how much of the improvement comes from detecting enclosed regions rather than from simpler signals. We agree this is the central question. The revised version will add three ablations: a frontier-based baseline that prioritizes archived states adjacent to observed but unvisited free space, using the same information TopoExplore has in TopoGym; Go-Explore with the texture signal and no homology term; and a separation of the homology term from the Ricci curvature term in every synthetic experiment.

Second, the theoretical guarantees. Reviewer mZSv identified real gaps in the statement of Theorem 3 and in what the attempt budget certifies. We will restate the theorem with its assumptions explicit and narrow the claims to what the proof supports. Details are in our reply to mZSv.

Third, benchmark and baseline coverage. Reviewers 5AsR and 4DDi noted that the strongest results come from TopoGym, which we built, and that the RND and ICM baselines are dated. The revision will evaluate on publicly maintained environments, including continuous-control mazes, and compare against current hard-exploration methods.

These changes are too substantial for the discussion period, and we will incorporate them in a revised submission. We are grateful for reviews that make the path forward clear.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.