- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors single out a class of bilevel problems in which the upper level has to drive a residual to zero. Examples include temperature tuning in SAC, the penalty weight in WGAN-GP, the KL penalty in RLHF, and the SimCLR temperature. They propose solving such problems with two-timescale stochastic approximation (TTSA) applied directly to , without hypergradients. The paper gives convergence rates under strong convexity and the PL condition, together with a variance comparison against minimizing . Experiments cover synthetic tasks, control of a stochastic ODE, SAC (Pendulum, HalfCheetah), a multi-agent game, WGAN-GP and SimCLR.
Critical strengths
-
- The unifying observation is helpful. Many procedures that tune a hyperparameter to hit a target statistic are essentially root-finding problems, and reframing them as minimization of with implicit differentiation is often unnecessary.
-
- The method is straightforward to implement, adds almost no computational overhead, and applies to a broad range of problems.
Critical weaknesses
-
- In my view the theory needs both revision and more careful positioning. The algorithm is the classical TTSA scheme [1], and SAC's automatic temperature adjustment [2] already works this way, which the authors acknowledge. The main text states the results for Markovian noise and calls them the first non-asymptotic analysis in that setting. The proofs, however, rely on conditionally unbiased estimates (Assumption B.5), i.e. martingale-difference noise, and the Markovian case is not treated separately. Non-asymptotic bounds for nonlinear TTSA are already available [3], including under Markovian noise [4,5]. The proof of Theorem 5.5 arrives at two different rates: with in Eq. (19), and then ; the two disagree for . Assumption B.6 (strong monotonicity is considerably stronger than Assumption 5.3 as stated in the main text, and it effectively contains the main difficulty for vector-valued . Proposition 5.4 compares upper bounds on the variances of two updates that live on different scales, so on its own it does not establish the variance trap.
-
- The experiments do not yet fully support the conclusions. On HalfCheetah, RF-BO ends with a return of 45.5 on one of the five seeds, although the text states that it avoids divergence on all seeds; moreover, returns for all methods (at most about 1200) are several times lower than what SAC typically reaches at 200k steps [2]. On Pendulum, RF-BO's success rate is lower than that of standard SAC (52% vs. 68%), and the final returns agree within the reported spread. SimCLR is trained for 50 epochs on 3 seeds, and the best epoch for every method is close to the last one, so training has not converged; WGAN-GP is evaluated on a one-dimensional mixture of two Gaussians. The paper does include dual-type baselines (standard SAC, and Dual Adam for WGAN-GP), and they update the parameter using the same quantity , but with a different optimizer and step-size schedule. It is therefore unclear whether the gains come from the root-finding formulation or from the step-size choice. An ablation that separates the two, and a comparison with PID Lagrangian methods [6], are missing. The motivating RLHF KL-penalty example [7] is not tested experimentally.
References:
-
- V.R.Konda, J.N.Tsitsiklis. Actor-Critic Algorithms. // Advances in Neural Information Processing Systems (NIPS), 1999.
-
- T.Haarnoja, A.Zhou, K.Hartikainen, G.Tucker, S.Ha, J.Tan, V.Kumar, H.Zhu, A.Gupta, P.Abbeel, S.Levine. Soft Actor-Critic Algorithms and Applications. // arXiv:1812.05905, 2018.
-
- T.T.Doan. Nonlinear Two-Time-Scale Stochastic Approximation: Convergence and Finite-Time Performance. // IEEE Transactions on Automatic Control, 68(8):4695--4705, 2023.
-
- T.T.Doan. Finite-Time Convergence Rates of Nonlinear Two-Time-Scale Stochastic Approximation under Markovian Noise. // arXiv:2104.01627, 2021.
-
- M.Kaledin, E.Moulines, A.Naumov, V.Tadic, H.-T.Wai. Finite Time Analysis of Linear Two-Timescale Stochastic Approximation with Markovian Noise. // Conference on Learning Theory (COLT), 2020.
-
- A.Stooke, J.Achiam, P.Abbeel. Responsive Safety in Reinforcement Learning by PID Lagrangian Methods. // International Conference on Machine Learning (ICML), 2020.
-
- L.Ouyang et al. Training Language Models to Follow Instructions with Human Feedback. // Advances in Neural Information Processing Systems (NeurIPS), 2022.
Questions for authors
-
- How do the proofs account for Markovian noise, given that Assumption B.5 requires conditionally unbiased estimates? Which of the two rates in the proof of Theorem 5.5 is correct, and what is new in the analysis compared with [3,4,5]?
-
- Which conditions on guarantee Assumption B.6 when is a vector?
-
- Is the clipping operator from Algorithm 1 used in the experiments, and is the same clipping applied to the baselines? Could you compare RF-BO with standard SAC and Dual Adam under the same optimizer and step-size schedule, and with PID Lagrangian methods [6]?
-
- How exactly is the residual defined for SimCLR (Table 12 only lists a similarity target of 0.6), and why is this the right target for the temperature ? What explains the 3.3-point gap over Fixed-Temp when the alignment and uniformity values are nearly identical?
-
- How does this work relate to the concurrent submission Authors (2026) on quantile-guided Huber updates in TTSA, given that Section 3.3 describes normalizing updates by dynamic quantiles?