- A significant mathematical gap in the policy invariance proof.
The theoretical analysis in Section 4.3 and Appendix A relies on the convergence of graph edge weights. Specifically, Appendix A.2 claims that the fixed-step-size exponential moving average in Eq. (3) converges almost surely to the expected environment reward under a stationary visitation distribution.
This conclusion does not generally hold. For a stationary sequence of stochastic rewards with nonzero variance, an EMA with constant coefficient α typically retains nonzero asymptotic variance rather than converging almost surely to its expectation. Stationarity of the visitation distribution is insufficient to establish Eq. (12) in Appendix A.
Consequently, the subsequent convergence claims for the graph potentials and the limiting shaped reward are not justified under the stated assumptions. Since asymptotic policy preservation is a central theoretical contribution, this is a substantive concern rather than a minor proof omission.
- The graph potential may not accurately represent achievable long-term returns.
Equation (6) computes node potentials through a max-based value iteration over observed graph edges. However, these edges store averaged immediate rewards without explicitly modeling action-conditioned transition probabilities.
In stochastic environments, an observed transition does not necessarily correspond to an outcome that the agent can reliably select. Maximizing over observed successors may therefore overestimate achievable returns, especially when several actions or stochastic outcomes are aggregated into the same edge.
The paper does not sufficiently discuss the assumptions under which this graph-based propagation provides a meaningful approximation of long-term value. Additionally, Figure 1 depicts an edge update using Q predictions, whereas Eq. (3) and Algorithm 1 use environment rewards. This inconsistency should be clarified.
- The reward-gated mechanism provides limited evidence of genuine cost awareness.
The proposed reward gate in Eq. (4) is defined as σ(r_env), making its effect highly dependent on the numerical scale of the environment reward.
For instance, Appendix B.2 specifies an action penalty of -0.1a² for MountainCar, with a ∈ [-1, 1]. Consequently, the sigmoid gate varies only between approximately 0.475 and 0.5 for ordinary non-terminal transitions. Such a narrow range raises questions about whether the gate can meaningfully distinguish actions with different costs.
More importantly, the method does not explicitly separate action costs from task rewards. It remains unclear whether the proposed gate offers a principled mechanism for cost-aware exploration or simply rescales the conventional count-based bonus.
- The methodological novelty is limited, and the experimental attribution is incomplete.
The main components of GIR, including visitation-based exploration, potential-based reward shaping, and graph-based value propagation, are individually well-established. The primary contribution appears to be their integration with a reward-dependent modulation mechanism.
Although Figure 5 evaluates several ablations, the paper does not compare GIR against a straightforward combination of count-based exploration and potential-based shaping without the proposed graph formulation.
Such a comparison is important for determining whether the performance gains arise from the particular GIR design or primarily from combining complementary intrinsic reward signals. Without this control, the incremental contribution over existing exploration and reward shaping approaches remains difficult to assess.
- The experimental evaluation does not sufficiently isolate cost-aware exploration.
The main experiments use conventional sparse-reward benchmarks with action penalties. While these environments are suitable for evaluating exploration, they do not fully establish the effectiveness of GIR across different cost regimes.
In particular, the paper does not systematically vary the action-cost coefficient or report how goal-reaching success, accumulated action cost, and exploration efficiency change as costs increase. The learning curves primarily report cumulative rewards, making it difficult to distinguish improvements in task success from reductions in exploration cost.
Furthermore, Figure 4 evaluates fixed start-goal configurations selected to create difficult exploration tasks. Results across additional configurations would help establish whether the observed improvements generalize beyond these specific layouts.
- Scalability and sensitivity to graph construction remain insufficiently characterized.
GIR relies on online state clustering, graph updates, and repeated potential propagation. Its performance may therefore depend considerably on the clustering threshold, state representation, and graph size.
Although Figure 5 includes a graph-size ablation, the analysis does not systematically characterize the sensitivity of the method across environments. Table 4 reports increasing computational overhead as the graph grows: the total time rises from approximately 4.17 ms per step for TD3 to 6.02 ms for GIR with 1000 nodes.
This overhead may be acceptable in the evaluated tasks, but the experiments do not establish scalability to substantially larger graphs or higher-dimensional continuous state spaces. The broad plug-and-play and scalability claims should therefore be interpreted more cautiously.