- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Self-Tuned Q-learning (ST-Q-learning) and Self-Tuned SARSA (ST-SARSA), which use an implicit/fixed-point update to transform the original step size into an adaptive effective step size related to the feature norm. The authors further derive finite-time error bounds for projected linear Q-learning and SARSA, and claim that the self-tuned variants allow arbitrarily large initial step sizes. Experiments cover tabular settings, linear function approximation, and simple deep Q-learning scenarios.
Critical strengths
- The problem is clear and practically relevant. Step-size sensitivity is a practical issue in Q-learning/SARSA, while the proposed method only modifies the update rule without introducing complex additional modules, making the overall formulation relatively simple.
- The paper not only provides a closed-form expression for the self-tuned update, but also gives finite-time analyses for both Q-learning and SARSA with a time-varying policy.
Critical weaknesses
-
The main experimental comparisons do not fully demonstrate the benefit of "self-tuning" itself. According to Proposition 3.1, the actual effective step size is . For many tabular experiments using one-hot features, , so this becomes . In other words, even when the nominal is set to 100, 300, or larger, the actual update step size is automatically reduced below 1. The current experiments mainly compare ST methods with standard Q-learning/SARSA using the same nominal , but do not include a matched baseline where the standard algorithms use the same transformed/effective step size. Therefore, part of the observed advantage in "large-step-size robustness" may directly result from this explicit step-size shrinkage.
-
Section 2 assumes , i.e., non-negative rewards. However, Cliff Walking uses rewards of and , and GridWorld also uses negative rewards. The paper does not clarify whether the theory actually only requires , or whether reward shifting is applied in the experiments.
-
The deep extension directly replaces in the linear setting with , but no corresponding theoretical derivation is provided.
-
There are still some basic issues in the references and presentation. For example, capitalization of journal and conference names is inconsistent in the bibliography, such as "nature," "Machine learning," "Springer nature," and "International conference on machine learning." The main text also contains unnatural citation rendering such as
(Zou et al., 2019b;a). These issues do not affect the core results, but should be standardized in the final version.
Questions for authors
See Weaknesses.