- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors argue that, during GFlowNet training, one should acquire experiences that are dissimilar to those already available in the replay buffer, rather than sampling trajectories simply because they are useful for learning. To this end, they propose Geometry-guided Exploration with Adaptive Termination (GEAT). GEAT derives a novelty score by differentiating a geometry-aware Shannon entropy at an empirical reference distribution uniformly sampled from the replay buffer. This score is used both to select among candidate terminal states generated by backtracking and reconstruction, and as an intrinsic reward for training the exploration GFlowNet. Across a range of benchmarks, GEAT demonstrates improved exploration efficiency.
Critical strengths
The motivation is clearly stated, and experiments across multiple benchmarks support the effectiveness of the method, particularly in terms of mode discovery. The authors identify a potentially overlooked opportunity to improve current exploration practice.
Critical weaknesses
The contribution appears to be a strategic combination of existing components: LSGFN, PER, and an exploration GFlowNet (as in SA-GFN), together with a novelty score computed from the replay buffer to guide both the local search and the exploration GFlowNet. Notably, in the ablation study, the NAT-only variant performs almost identically to LSGFN. While the combination works well empirically, the connection between these design choices is not well established, which makes the novelty of the contribution unclear. If the authors can adequately address this point and Q1, and the additional results requested in Q2 and Q3 do not change the conclusions of the experiments, I would be willing to raise my rating.
Questions for authors
- The paper argues that exploration should acquire experiences dissimilar to those already in the replay buffer, and the experiments use PER for replay. However, GEAT decouples the novelty computation from the PER rule: the reference set is sampled from the replay buffer uniformly rather than according to the PER distribution. This raises two concerns. First, a trajectory that remains in the buffer but has low PER priority is rarely sampled for training, yet it is still treated as "covered" by the novelty score. Second, if the buffer has finite capacity, an evicted sample will be treated as "uncovered" again, even if it has already been well learned. The buffer size and eviction policy are also not reported. Could the authors clarify the relationship between the novelty score and the PER rule?
- All baselines are run with PER, although many of them, unlike GEAT, do not necessarily require a replay buffer. Could the authors also report the performance of the baselines without a replay buffer?
- GEAT relies on hand-designed, domain-specific kernels, whereas SA-GFN, the only baseline with an intrinsic novelty signal, uses RND. Could the authors report results for the baselines equipped with the same kernel on each task (e.g., SA-GFN with RND replaced by the kernel-based novelty)?
5 comments
The Separation Between Novelty Computation and PER Is Intentional
Novelty measures what has already been collected, while PER prioritizes which stored experiences need further learning. This separation prevents learning priorities from driving unnecessary data collection.
- Low-priority samples are correctly treated as covered. Under loss-based PER, low priority may indicate a small TB residual. Regardless of training priority, stored experiences can be reused through replay without additional reward evaluations.
- PER-weighted references would make covered regions appear novel. Previously collected, low-priority samples would be underrepresented in the reference set, appear novel, and attract repeated acquisition, which is exactly the redundancy GEAT aims to avoid.
Our additional experiments support this distinction. As illustrated here, PER-weighted reference sampling can make previously visited, low-priority states appear more novel.
To examine this effect, we compared uniform and PER-weighted references using identical replay buffers and candidate sets. We focused on previously visited, low-reward terminal states in the bottom 25% of PER priorities, averaging results over training iterations 500–2,000.
| Metric | Uniform References | PER-Weighted References |
|---|---|---|
| Reference Inclusion Rate | 15.61% | 0.23% |
| Geometric Novelty | 1.45 | 2.57 |
| Selection Rate | 1.49% | 3.43% |
PER-weighted references substantially increase both the estimated novelty and selection rate of previously collected, low-priority states. This indicates that incorporating learning priorities into novelty estimation can redirect exploration toward experiences assigned relatively low priority for further training.
We also compared the final performance of uniform and PER-weighted reference sampling:
| Reference Sampling | TV ↓ | Discovered Modes ↑ |
|---|---|---|
| Uniform GEAT | 0.223 ± 0.063 | 2,574 ± 8 |
| PER-Weighted GEAT | 0.279 ± 0.062 | 2,388 ± 158 |
Mean ± sample standard deviation over three seeds.
Uniform GEAT achieves better average final performance in these runs. Together, these results support separating novelty-based coverage estimation from PER-based learning priorities.
Effect of Buffer Eviction
GEAT maintains comparable mode coverage across buffer sizes, while removing task-policy sampling substantially reduces mode discovery. The task GFlowNet continues to sample from its learned reward-proportional distribution and replenish the shared buffer, so eviction does not necessarily leave previously explored high-reward regions uncovered.
Buffer configuration and ablation. All main experiments use a shared FIFO replay buffer with a capacity of 6,553 unique terminal states, as now reported in App. D. To assess sensitivity to buffer eviction, we evaluated GEAT on QM9 with three buffer capacities (0.25×, 1×, and 4×), with and without task-policy sampling.
Both settings use the same budget of 64 fresh samples per iteration: 32 task and 32 exploration samples when task sampling is enabled, versus 64 exploration samples otherwise. The task GFlowNet remains trained off-policy in both settings.
QM9: Buffer-Size and Task-Sampling Ablation
| Buffer Size | Task Sampling | Modes ↑ | Top-100 ↑ | FCS ↓ |
|---|---|---|---|---|
| 1,638 (0.25×) | ON | 734.3 ± 8.1 | 9.256 ± 0.133 | 0.0879 ± 0.0028 |
| OFF | 514.7 ± 5.0 | 9.349 ± 0.059 | 0.1219 ± 0.0064 | |
| 6,553 (1×) | ON | 739.7 ± 3.5 | 9.208 ± 0.113 | 0.0922 ± 0.0079 |
| OFF | 530.3 ± 8.6 | 9.322 ± 0.201 | 0.1203 ± 0.0120 | |
| 26,212 (4×) | ON | 738.0 ± 5.3 | 9.293 ± 0.218 | 0.0929 ± 0.0056 |
| OFF | 542.7 ± 7.2 | 9.311 ± 0.051 | 0.1091 ± 0.0037 |
Mean ± sample standard deviation over three seeds.
Results. With task-policy sampling, GEAT maintains comparable mode coverage across all buffer sizes. In contrast, removing task-policy sampling reduces mode discovery by 26–30% and increases FCS across all buffer sizes, while Top-100 rewards remain comparable.
These results support GEAT's robustness to buffer capacity and highlight the importance of continued task-policy sampling for maintaining mode coverage.
Fairness of Kernel-Based Novelty Comparison
GEAT's mode-discovery improvements cannot be attributed solely to the geometric novelty kernel or its alignment with the evaluation metric. We conducted two additional comparisons to examine these potential concerns.
1. Comparison Using the Same Geometric Novelty
GEAT discovers more high-reward modes than SA-GFN even when both use the same geometric novelty. To isolate the effect of the novelty kernel, we evaluated SA-GFN + Geo, replacing SA-GFN's RND intrinsic reward with GEAT's geometric novelty while keeping all other components unchanged.
Number of Discovered Modes ↑
| Benchmark | SA-GFN (RND) | SA-GFN + Geo | GEAT |
|---|---|---|---|
| Bit Sequence (K=8) | 35.7 ± 2.5 | 35.0 ± 3.5 | 55.7 ± 2.5 |
| Bio (AMP) | 300.3 ± 16.0 | 301.3 ± 18.8 | 442.0 ± 11.1 |
Mean ± sample standard deviation over three seeds.
These results indicate that GEAT's improvements cannot be explained solely by its choice of geometric novelty.
2. Evaluation Using Alternative Mode Definitions
GEAT maintains the highest average mode discovery even when the evaluation distances differ from those used in its novelty kernel.
GEAT originally uses Levenshtein-distance-based geometric novelty for Bit Sequence and ProtAlbert embedding-based L2 geometric novelty for AMP. The original mode-discovery criteria also rely on these respective distances.
To assess whether this alignment favors GEAT, we kept the novelty kernels unchanged and reevaluated mode discovery using:
- Bit Sequence: Hamming and Jensen–Shannon (JS) distances.
- AMP: Hamming and Levenshtein distances.
We tested three thresholds for each distance metric.
Bit Sequence: Number of Discovered Modes ↑
| Method | Hamming 0.26 | Hamming 0.275 | Hamming 0.29 | JS 0.58 | JS 0.60 | JS 0.62 |
|---|---|---|---|---|---|---|
| TB | 2.3 ± 1.2 | 16.7 ± 3.8 | 32.7 ± 4.0 | 22.0 ± 8.0 | 39.0 ± 14.0 | 50.7 ± 10.2 |
| Teacher | 5.0 ± 1.7 | 28.3 ± 1.5 | 48.3 ± 0.6 | 22.7 ± 7.0 | 41.3 ± 10.8 | 54.0 ± 5.2 |
| SA-GFN | 3.3 ± 2.1 | 27.3 ± 0.6 | 46.3 ± 1.5 | 22.0 ± 8.0 | 40.0 ± 12.3 | 52.3 ± 8.1 |
| ACE | 3.3 ± 2.1 | 27.3 ± 3.2 | 46.7 ± 2.3 | 17.7 ± 15.0 | 35.0 ± 21.7 | 51.0 ± 11.3 |
| GEAT | 19.0 ± 5.6 | 52.7 ± 2.3 | 57.7 ± 1.5 | 59.3 ± 0.6 | 59.7 ± 0.6 | 60.0 ± 0.0 |
Mean ± sample standard deviation over three seeds. JS denotes Jensen–Shannon.
AMP: Number of Discovered Modes ↑
| Method | Hamming 0.75 | Hamming 0.80 | Hamming 0.85 | Levenshtein 0.75 | Levenshtein 0.80 | Levenshtein 0.85 |
|---|---|---|---|---|---|---|
| TB | 373.3 ± 16.3 | 364.7 ± 14.2 | 157.3 ± 7.8 | 336.0 ± 17.7 | 156.0 ± 4.6 | 18.3 ± 1.2 |
| Teacher | 462.7 ± 29.3 | 450.7 ± 27.5 | 176.3 ± 12.6 | 409.7 ± 18.1 | 177.7 ± 5.5 | 19.7 ± 1.2 |
| SA-GFN | 605.3 ± 19.4 | 579.0 ± 19.9 | 198.7 ± 8.0 | 518.0 ± 4.0 | 199.7 ± 10.1 | 20.3 ± 0.6 |
| ACE | 615.0 ± 41.6 | 588.3 ± 35.9 | 197.3 ± 5.7 | 517.0 ± 24.3 | 205.3 ± 10.6 | 22.3 ± 0.6 |
| GEAT | 872.3 ± 21.5 | 820.3 ± 22.9 | 237.7 ± 6.7 | 702.0 ± 22.0 | 239.0 ± 14.4 | 24.3 ± 1.5 |
GEAT achieves the highest average mode discovery across all twelve alternative evaluation settings, supporting that its advantage is not merely due to alignment between the novelty kernel and evaluation metric.
We appreciate your time and feedback on our paper. Below are our responses to your comments.
Justification for Component Connection
We appreciate the opportunity to further clarify the rationale behind GEAT's design.
We have revised the manuscript to explain why GEAT combines geometric novelty and NAT: expanding exploration coverage requires both reaching less-explored regions and selecting novel terminal states within them. These two components address complementary decisions.
The contribution lies in connecting these two stages to improve exploration coverage. We have incorporated this discussion and supporting experimental evidence into Section 5 (Experiments). An additional visualization, available here, compares the baseline, NAT-only, geometric-only, and full GEAT: (a) shows cumulative explorer terminal visits across training steps, while (b) tracks the number of high-reward basins reached during training.
Baselines without Replay
Removing replay generally reduces baseline performance, and none of the replay-free baselines outperforms GEAT in mode discovery. This suggests that our shared replay protocol does not systematically disadvantage the baselines.
To verify this, we evaluated TB, SA-GFN, Adaptive Teacher, and ACE with and without replay on HyperGrid, Bit Sequence (K=8), and QM9. Without replay, each baseline is trained only on newly collected experiences, retaining its original exploration mechanism and sampling budget. Replay-enabled configurations use a shared buffer capacity of 6,553 unique terminal states.
All values are reported as Replay ON → OFF, with mean ± sample standard deviation over three seeds.
HyperGrid (D=2, H=256)
Removing replay substantially worsens TV distance for all baselines, even when mode discovery increases, as observed for Teacher.
Bit Sequence (K=8)
Replay has a relatively limited effect on Bit Sequence, with comparable mode discovery and Top-100 rewards across configurations.
Bio (QM9)
Removing replay reduces mode discovery by 9.4–21.8% and lowers Top-100 rewards across all baselines, although its effect on FCS is mixed.
All results are reported as mean ± sample standard deviation over three seeds. HyperGrid and QM9 use 2,000 training iterations; Bit Sequence uses 20,000.
Overall, replay improves baseline performance on HyperGrid and QM9, while its effect on Bit Sequence is relatively small. These results provide no evidence that our shared replay protocol systematically disadvantages the baselines.