- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper addresses the problem of recovering exact inaction regions in continuous control with linear action costs, where standard neural policies produce small, non-zero "leaky" actions instead of exact zeros. Rather than applying heuristic cutoffs to raw policy outputs, the authors use a frozen neural policy to run full-horizon Monte Carlo rollouts at test time and estimate state-value sensitivities via backpropagation. These marginal sensitivities are passed through a soft-thresholding operator (Stage III) and iteratively refined across virtual state updates (Stage IV) to output a net control action. The method is mainly evaluated on dynamic portfolio management up to 50 assets, alongside low-dimensional control tasks.
Critical strengths
- Quantitative improvements compared with raw neural policies and magnitude-based heuristics. Table 1 shows that raw neural policies fail at inaction detection (0% hold recall), while naive magnitude thresholding cannot decouple hold recall from action recall. Comparatively, the proposed method achieves higher action and hold recall rates.
- Accurate Low-Dimensional Boundary Recovery. In 1D to 3D settings where dynamic programming ground truth is available, the sensitivity-based detector accurately recovers the true free-boundary geometry, including orientation shifts caused by asset correlations (Figures 3–6).
Critical weaknesses
- Requires a differentiable simulator during inference, and induces massive computational overhead. To make a single decision, the online controller must simulate many full trajectories and backpropagate through environment dynamics, and repeats this loop up to 500 times per-step, which is drastically slower compared to the base policy (Table 13, 14). While the authors distill the detector into a fast student network to fix this latency, performance drops sharply (Table 1 line 452 vs. 451) and selective inaction can be completely lost (Figure 7).
- Insufficient empirical justification for stages III and IV. The results in Table 1 (line 449) do not show that the Yosida soft-thresholding map (Stage III) and iterative refinement loop (Stage IV) improve upon simply using the BPTT sensitivity as a binary mask. Similarly, in Figure 7, the held-out utility of the full multi-stage refinement pipeline is indistinguishable from the simple hold gate baseline (and even from naive magnitude thresholding) within standard error bars across all tested dimensions.
- Disconnect between theory and practice. The main theoretical guarantees (Theorems 3 & 4) rely on overly restrictive oracle assumptions. Theorem 3 only guarantees improvement on an artificially regularized objective rather than the true objective, conditioned on BPTT estimation errors already being smaller than the policy gap and assuming strictly positive cash marginals everywhere. Theorem 4 relies on an idealized oracle strategy that the authors explicitly admit stage IV does not construct (line 264).
- Lack of competitive baselines and relevant literatures, which makes it hard to evaluate the paper's contribution. There are no citations after 2020, and authors only compare against naive post-hoc thresholding and a PINN baseline. Although I am unsure of the exact recent baseline methods in finance, deciding when "not to act" is actively studied in the reinforcement learning/control regime [1, 2]. The paper's current evaluation omits competitive end-to-end RL methods that achieve exact inaction in a single forward pass.
- Unnecessarily dense mathematical presentation. The paper's core ideas are obscured by heavy mathematical notation. For instance, the "Yosida approximation of the Euclidean normal cone" (Eq. 9) is a coordinate-wise soft-thresholding function, and Proposition 1 follows directly from maximizing a basic scalar quadratic objective. Streamlining this formalism into clear algorithmic intuition would vastly improve readability.
References
[1] Treven, Lenart, et al. "When to sense and control? a time-adaptive approach for continuous-time rl." NeurIPS 2024.
[2] Nie, Buqing, Yangqing Fu, and Yue Gao. "Select before Act: Spatially Decoupled Action Repetition for Continuous Control." ICLR 2025.
Questions for authors
- Following weakness point 2, can authors provide additional evidence to justify the choice of stage III and IV?
- How does the method compare against more recent end-to-end RL architectures that directly learn act-or-hold gates [2] or holding durations [1] without test-time rollouts (choose one baseline method without post-hoc refinement for comparison)?
- Can authors explain why does the distilled student suffer a >14% drop in hold recall in Table 1 and degenerate to 100% active trading at N=20 in Figure 7? Since online BPTT rollouts are impractical for real-time control, deployment relies on the distilled student network, so please provide more quantitative analysis regarding failure modes of student network.