acceptodds
Under review as a conference paper at ICLR 2027

WHEN AND HOW DOES A REINFORCEMENT LEARNING OPTIMIZER HELP OUT-OF-DISTRIBUTION DETECTION? A THEORETICAL STUDY

Abstract

Out-of-distribution (OOD) detection in dynamic open-world environments requires a model to continually adapt to evolving data distributions while generalizing to covariate-shifted inputs and rejecting semantic-shifted OOD examples. Most existing OOD detection methods optimize only the current-step objective and do not explicitly account for how post-deployment environment changes affect future OOD behavior. In this paper, we establish a theoretical framework that explains how and when reinforcement learning (RL)-guided update improves dynamic OOD detection. Our RL-guided optimizer explicitly favors updates that reduce the semantic OOD false positive rate over time. We analyze how the value function shapes the updates and affects the training trajectory, and derive a temporal error decomposition that separates the change in evaluation risk into a model-change term, controlled by the optimizer, and an environment-change term, derived by distribution shift. This decomposition yields measurable conditions under which the RL-guided optimizer lowers the semantic-OOD false positive rate relative to GD without degrading classification generalization error. Experimental evidence support better performance of the RL-guided optimizer over a GD optimizer.

Primary area:
learning theory
Keywords:
Out of distribution detection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FZrjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies dynamic OOD detection under evolving environments and investigates whether RL can improve adaptation compared with conventional gradient descent. The authors propose an RL-guided optimizer that augments standard gradient updates with a correction derived from a temporal-difference-trained value function. The value function operates on a low-dimensional state containing energy-distribution statistics, a differentiable semantic-OOD false-positive-rate surrogate, and classification information.

Critical strengths
  1. Dynamic OOD detection is a challenging setting because adaptation must simultaneously preserve known-class classification performance and maintain the ability to reject unseen semantic categories.
  2. The risk-specific decomposition in theorem 4.2 provides useful insights into the adaptation process.
  3. The empirical evaluation covers diverse temporal environments and feature encoders.
Critical weaknesses
  1. The most important issue: there is an important inconsistency between the proposed optimizer and the optimization objective used in the theoretical analysis. Equation (2) defines the RL-guided update as
=\theta_k-\eta\nabla L_{\mathrm{OW}}(\theta_k) +\eta_{\mathrm{RL}}\nabla_\theta V_\phi(s_k).$$ However, Definition 5.5 defines the corresponding RL optimization objective as $L_G=L_{\mathrm{OW}}+\eta_{\mathrm{RL}}V(s)$. 2. The contribution of temporal value learning is not sufficiently isolated from simpler gradient-based regularization. A learned value function may improve optimization simply by introducing an additional task-dependent gradient, without requiring meaningful prediction of future returns. 3. The main experiments adapt only a linear classification head on top of frozen pretrained features. The paper explicitly reports that RL no longer improves upon GD when the feature encoder is fine-tuned. 4. Additionally, the improvements are modest or absent on several benchmarks, e.g., domainNet.
Questions for authors
  1. Plz reconcile Equation (2) with the objective defined in Definition 5.5.
  2. Could you provide an ablation that isolates the contribution of temporal-difference learning?
  3. How sensitive is the method to errors in pseudo-OOD selection?
1 comment
AuthorsOpenReview ↗

TD Ablation and Pseudo-OOD Sensitivity

We thank the reviewer for the careful review.

W1.1&Q1. Consistency of the update rule and objective. The implemented update corresponds to the locally frozen objective LOW−ηRLηVϕ(s)L_\text{OW}-\frac{\eta_\text{RL}}{\eta}V_\phi(s) (Section 3). The effective gradient flow is now reported for the objective the update actually descends (Appendix D.10). Theorem 4.2, the main result, uses the update rule directly and is unaffected.

W1.2&Q2. Contribution of TD learning. The TD ablation confirms the role of the critic. Without bootstrapping (discount 0), RL loses its advantage on all three gradual streams (by 2.4, 1.5, and 1.0 points, all significant), and a larger step does not restore it. Bootstrapping from a random earlier state instead of the next one changes nothing, so the gain comes from the bootstrapped value estimation rather than transition order (Section 6.5).

W1.3&Q3: Sensitivity to pseudo-OOD selection. We varied the pseudo-OOD percentile and flipped 10% or 20% of the pseudo-labels for both optimizers (Appendix D.17). RL's advantage in FPR@95 at percentiles 0.90, 0.95, and 0.99 is 1.8, 2.0, and 1.1 points on Core50 and 1.8, 1.9, and 1.8 points on iCubWorld. Given 10% and 20% of the labels flipped, it is 4.0 and 3.8 points on iCubWorld, and almost zero on Core50 and OpenLORIS. The gain holds across thresholds on Core50 and iCubWorld, and on iCubWorld it grows under label noise. Core50 shows that pseudo-label quality is a lever for further gains.

Modest gains and DomainNet. Theorem 4.2 explains where gains appear. On DomainNet the measured value term is 10x smaller than on Core50, which accounts for the small difference we observe. We also extended the OOD-weight sweep: RL stays lower at matched weight on iCubWorld throughout and on Core50 up to a weight of 6, and we now state the gains at matched weight.

Linear head. The frozen-feature setting is where Proposition 4.3 makes the constants explicit. The fine-tuning results confirm this boundary and point to step-size control as a route to larger models (Section 6.4).

Reviewer_GV6COpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies out-of-distribution (OOD) detection in dynamic open-world environments, where a model must continually adapt while maintaining both known-class classification under covariate shift and rejection of semantic OOD samples. The authors argue that standard gradient descent (GD) optimizes only the current objective and may therefore select updates that achieve a low immediate loss but degrade subsequent OOD performance. They propose an RL-guided optimizer that first extracts a pseudo-semantic-OOD subset from unlabeled wild data using an ID-calibrated energy threshold, defines the reward as the negative pseudo-OOD soft-FPR, trains a value function through temporal-difference (TD) learning, and uses the value gradient to modify updates to the linear classification head. The theoretical analysis decomposes the difference between the RL and GD trajectories into the accumulated value correction, the induced drift in subsequent base updates, and a curvature remainder. It further provides sufficient conditions under which the RL-guided trajectory reduces held-out semantic-OOD FPR while controlling known-class classification risk.

Critical strengths
  • The problem is practically relevant. The paper highlights that an adaptation update that improves current classification performance may simultaneously distort the energy landscape and weaken OOD rejection in later environments. Studying the long-term effect of sequential parameter updates is meaningful compared with evaluating OOD detection only under a static test distribution.
  • The paper distinguishes several forms of distribution shift. The experiments separately consider covariate shift, gradual drift, abrupt drift, and saturated settings, rather than treating all dynamic distribution changes as equivalent.
  • The theory offers a useful framework for analyzing optimization trajectories. The decomposition separates the accumulated value correction, the change in subsequent base gradients induced by earlier corrections, and the local curvature term. This provides a more informative view of how a correction at one step can influence the final parameters through later updates than a purely one-step gradient analysis.
Critical weaknesses
  • The TD transitions mainly describe optimization within the current environment rather than genuine transitions to future domains. The value function is trained on transitions of the form st,k→st,k+1s_{t,k}\rightarrow s_{t,k+1}, where kk indexes inner optimization steps within the same environment. Consequently, the critic can most directly learn how energy statistics change after a local parameter update, but it does not observe the unknown transition from the current environment to the next domain. Describing this signal as accounting for “future-domain consequences” therefore appears stronger than what the training protocol supports. The absence of an improvement under abrupt drift is also consistent with this limitation.
  • The necessity of reinforcement learning is not sufficiently established. The base objective optimized by both GD and RL already contains a pseudo-OOD soft-FPR term, while the critic reward is simply the negative of the same quantity. If the critic mainly learns a mapping such as V(s)≈−aFPR^softV(s)\approx-a\widehat{\mathrm{FPR}}^{\mathrm{soft}}, the value correction may effectively reduce to an adaptive reweighting of the existing OOD loss. Although the appendix compares against a sweep over fixed OOD-loss weights, the paper does not evaluate stronger non-RL alternatives such as state-dependent loss weighting, a moving-average penalty, direct multi-step soft-FPR optimization, an MLP trained without TD bootstrapping, or a critic whose only input is the soft-FPR. These controls are needed to determine whether TD learning is essential or whether a simpler adaptive regularizer would provide the same benefit.
  • The empirical gains are modest and limited to a relatively narrow setting. In the main multimodal CLIP-B/16 experiment, RL reduces FPR@95 relative to GD by approximately 2.0, 1.9, and 0.3 percentage points on Core50, iCubWorld, and OpenLORIS, respectively. It provides no consistent advantage on DomainNet, CLEAR, Terra Incognita, or Century-of-Portraits. In the covariate-shift experiment, known-class accuracy sometimes improves, but semantic-OOD FPR is not consistently reduced. More importantly, RL no longer outperforms GD when the encoder and classification head are fine-tuned jointly, suggesting that the demonstrated benefit is largely restricted to adaptation of a linear head over frozen features.
Questions for authors
  • How can the method recover difficult OOD samples that are excluded by its own pseudo-labeling rule? A genuine false-positive OOD sample satisfies Eθ(x)≤τtE_\theta(x)\leq\tau_t and may therefore be excluded from the pseudo-semantic-OOD subset, which is formed from samples above the threshold. Does the algorithm contain any mechanism that can later identify these missed OOD samples, or does it primarily push already detectable, relatively easy pseudo-OOD samples farther from the decision threshold?
  • Why does the RL-guided optimizer cease to help when the encoder is fine-tuned? Does this result indicate that the chosen state representation, value-gradient correction, or theoretical analysis is valid mainly for low-dimensional linear-head adaptation over frozen features? A more detailed diagnosis of the full-model setting would clarify the applicability of the proposed optimizer.
2 comments
AuthorsOpenReview ↗

Mechanism and Necessity of the Learned Value

We thank the reviewer for the thoughtful review.

Within-environment transitions. The new controls clarify the mechanism. Bootstrapping from a random earlier state of the same environment gives the same FPR@95 as our critic, so the benefit comes from bootstrapped value estimation within the adaptation window, which needs no cross-environment credit assignment. Its effect is carried to later environments through the parameters, and Theorem 4.2 measures this directly as the drift term, which accounts for most of the change in soft FPR on Core50.

Necessity of RL. In FPR@95 on Core50, iCubWorld, OpenLORIS, and DomainNet, GD reaches 62.7, 87.6, 49.5, and 54.4, and RL 60.7, 85.7, 49.2, and 54.6. A critic without bootstrapping (discount 0) reaches 63.1, 87.2, 50.2, and 54.6, a critic that sees only the soft FPR 60.1, 87.5, 48.1, and 54.0, and a moving-average penalty with no critic 57.2, 87.1, 44.0, and 52.8. Bootstrapping is essential: without it the gain disappears, and a 10x larger step does not bring it back. The moving-average penalty works by raising the effective OOD weight, so we compared RL and GD at matched weights. RL is lower at every weight on iCubWorld, the hardest stream, and on Core50 up to a weight of 6, while very large weights let a fixed penalty catch up on Core50 and OpenLORIS. We now state RL's advantage at matched weight. Theorem 4.2 covers both kinds of correction, so it explains when each helps.

Q1. RL selects more true OOD samples than GD on every gradual stream (recall on iCubWorld 0.128 to 0.138). Improving recall below the threshold is a natural extension, now discussed (Section 6.6 and Appendix D.17).

Q2. The fine-tuning results match the theory's curvature term. RL returns to GD as the correction step shrinks (Core50: GD 46.3, RL 47.0 at step 0.001 and 84.4 at step 0.02), which marks where the frozen-feature analysis applies.

Reviewer_GV6COpenReview ↗

Response to Author

Thank you very much for the author's reply, which has resolved most of my doubts. I'm willing to improve my score.

Reviewer_KKvBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether a reinforcement-learning-guided optimizer can improve out-of-distribution detection under sequential distribution shifts. The proposed method augments gradient descent on an OOD-aware objective with the gradient of a TD-trained value function. The critic receives summary statistics of ID and pseudo-OOD energies, a soft-FPR surrogate, and classification performance, with reward defined as negative pseudo-OOD soft FPR. The theoretical analysis first derives a finite-window decomposition of the difference between the GD and value-guided trajectories and projects this difference onto semantic-OOD and classification-risk gradients. It then presents additional loss-gap and effective-gradient-flow bounds. Experiments adapt a linear head on frozen features across multiple sequential benchmarks and encoders. The reported gains are concentrated on several gradual-drift streams, while abrupt-drift, saturated, and covariate-shift settings show more limited or mixed detection benefits.

Critical strengths
  1. The trajectory identity explicitly separates accumulated value corrections, induced changes in subsequent base updates, and inherited parameter differences. The subsequent risk expansion distinguishes the effects on semantic-OOD rejection and known-class classification. Under the stated smoothness assumptions, these derivations are valid, and the frozen-feature linear-head specialization provides explicit gradients and curvature bounds. This is a constructive way to analyze how an additional update direction affects different evaluation risks.

2.GD and the proposed optimizer share the initial head, batches, and base objective. The experiments cover multiple encoders and report five seeds, paired bootstrap analyses, penalty-weight sweeps, inner-step ablations, modality ablations, and a frozen-head control. Table 1 reports FPR@95 improvements of 2.0 and 1.9 percentage points on Core50 and iCubWorld, respectively, for the main configuration. The paper also reports settings with negligible or unfavorable gains, which helps delimit the empirical scope of the method.

Critical weaknesses
  1. The principal bounds in Section 5 are not established by the provided proofs, and parts of the analysis do not match the implemented optimizer.

    In Appendix D.1, Eq. (53) bounds the squared norm of a sum by the sum of squared norms without cross terms or additional factors. This inequality is false in general: taking scalar values a=b=1 and c=0 gives 4 on the left and 2 on the right. Equation (56) similarly bounds the squared norm of accumulated gradient increments by the sum of their squared norms without the required factor or an orthogonality assumption. Consequently, Theorem 5.3 and its dependent results do not follow from the stated assumptions.

    A separate problem occurs in Eqs. (49)–(50), where a second-order approximation is promoted to a strict upper bound using only the Hessian at the GD endpoint. Smooth gradients and bounded iterates do not justify this step. For example, f(x)=(1−cos x)^2 has a globally Lipschitz gradient and satisfies f′(0)=f″(0)=0, but f(δ)>0 for sufficiently small nonzero δ. Thus, the endpoint Hessian alone cannot provide the claimed bound. The proof also treats the finite-K GD endpoint as stationary without establishing this condition. Appendix D.13 acknowledges that the experimental endpoint is not stationary and that the corresponding quadratic estimate frequently fails to upper-bound the measured gap.

    There are also substantive inconsistencies between the analyzed and implemented quantities. Equation (2) corresponds to descent on L_OW−(η_RL/η)V, whereas Definition 5.5, Eq. (63), and Appendix D.10 use L_OW+η_RL V. Moreover, Appendix D replaces the implemented six-dimensional critic state with a scalar semantic-drift statistic involving the next environment’s semantic distribution and uses them interchangeably without establishing the necessary relationship. These issues undermine the claimed loss-gap and gradient-flow explanation of the actual optimizer.

2.The remaining valid theory and empirical diagnostics do not establish the claimed connection between temporal value learning and improved detection at the reported operating point.

Theorem 4.2 concerns soft FPR at a fixed threshold, whereas the principal experimental metric is FPR@95 with a threshold calibrated to ID acceptance. These are materially different quantities. A common downward shift of all class logits leaves softmax probabilities and classification cross-entropy unchanged while increasing every energy score by the same constant. This lowers fixed-threshold soft FPR, yet leaves FPR@95 and AUROC unchanged after threshold recalibration. Therefore, even a simultaneous decrease in the theorem’s soft-FPR risk and preservation of classification risk does not establish improved detection at controlled ID acceptance.

In addition, the finite-window identities hold for arbitrary additive corrections and do not use the TD or Bellman structure to establish the required alignment. Table 7 reports critic sensitivity and mean semantic-OOD energy changes, but these do not verify the complete alignment, induced-drift, and curvature conditions in Theorem 4.2. The penalty sweep in Appendix D.11 is useful, but it does not isolate temporal bootstrapping from an adaptive instantaneous regularizer. A matched myopic critic or another temporal-control experiment is needed to substantiate the proposed explanation of why RL helps.

Questions for authors

1.Can the authors provide corrected derivations for Eqs. (53) and (56), retain or rigorously bound the Taylor remainder in Eqs. (49)–(50), and state the conditions under which the GD endpoint is stationary? How do the corrected bounds depend on K, and which of their assumptions hold in the K=30 online protocol? 2.Which effective objective was actually used to compute the EGF quantities in Table 8: L_OW−(η_RL/η)V or L_OW+η_RL V? What relationship permits replacing the implemented six-dimensional state with the scalar drift statistic in Appendix D? How are changes in the TD-trained critic parameters accounted for when bounding consecutive value-gradient differences? 3. How does the fixed-threshold soft-FPR result translate to FPR@95 with a model-dependent calibrated threshold? Can the authors report the actual alignment, induced-drift, and curvature terms from Theorem 4.2 alongside fixed-threshold soft/hard FPR and recalibrated FPR@95, so that the claimed mechanism can be distinguished from a common energy-score shift? 4. Does a matched γ=0 critic, or a control that removes temporal ordering, obtain similar gains? Also, Algorithm 1 updates the critic before updating the detector, while the TD target requires the next state. Which already-observed transition is used for this update, and how is bootstrapping handled at environment boundaries?

1 comment
AuthorsOpenReview ↗

Theory-Experiment Connection and TD Controls

We thank the reviewer for the careful review. W1 (Section 5 derivations). We appreciate the close reading of these steps. Section 5 complements the main result: Theorem 4.2, Proposition 4.3, and the experiments of Section 6 do not depend on it. For the step in Eqs. 49 to 50 that bounds the curvature by the Hessian at the GD endpoint, Appendix D.13 now tests it directly on the implemented objective, which shows when that step is valid and which constant can replace it (Q1 below).

Q1. The bound is well supported empirically once the endpoint Hessian is replaced by a global smoothness constant. The curvature along the segment between the GD and RL endpoints never exceeds the global constant of Proposition 4.3, and with this constant the bound holds in 91% to 100% of environments, and in 100% once the threshold and pseudo-OOD mask are held fixed as the analysis assumes. The remaining failures are therefore due to the threshold being recomputed, not to curvature (Appendix D.13).

Q2. Table 8 of the submission (Table 12 in the revision) followed Definition 5.5 as written in the submission. We now report the effective gradient flow of the objective the update actually descends, which is the more meaningful quantity (Appendix D.10). The main result, Theorem 4.2, uses the update rule directly and is unaffected.

Q3. This question led to a useful new diagnostic. Theorem 4.2 uses the fixed-threshold soft FPR because it is a differentiable risk for analyzing the value-guided update, while FPR@95 evaluates the resulting detector at an ID-calibrated threshold. We use the theorem to analyze the update mechanism and calibrated metrics to evaluate the detector, and Appendix D.10 now makes this distinction explicit. Since RL raises ID and OOD energies together, most of the change in soft FPR at a fixed threshold is a common shift, which is exactly why the paper reports threshold-recalibrated metrics. On Core50, RL increases the mean semantic-OOD energy by 0.36 and the mean ID energy by 0.33. Beyond this common shift, the recalibrated soft FPR decreases from 0.582 to 0.570, FPR@95 from 62.7 to 60.7, and the separability index d' (the ID-OOD energy gap divided by their pooled standard deviation) increases from 2.22 to 2.27, which indicates improved ID-OOD separation beyond a uniform energy shift. We also measured the alignment, induced-drift, and the curvature terms of Theorem 4.2 over 150 adaptation windows. The sufficient condition never held without the soft FPR decreasing, and when both optimizers share the starting point, it holds in 95% of Core50 windows, with the measured change equal to the predicted value term. These quantities are now reported in Section 6.6 (Table 4), with full tables in Appendix D.10, which makes the connection between the theorem and the experiments explicit.

Q4. The critic is updated on (sk−1,rk−1,sk)(s_{k-1}, r_{k-1}, s_k) with sks_k computed before the head step and no bootstrapping across environments (Appendix D.6).

Second weakness. Theorem 4.2 is deliberately formulated for a general additive correction, with the RL-guided update as one instantiation. It characterizes how such a correction affects the two evaluation risks, and the ablations of Section 6.5 then isolate what the TD critic contributes.

As shown in Table 3, a critic without bootstrapping (discount 0) is 2.4, 1.5, and 1.0 FPR@95 points worse than RL on Core50, iCubWorld, and OpenLORIS, respectively. Increasing the correction step by 10×10\times does not recover the gain. The random-earlier-state control matches RL on all four streams, so the precise ordering of adjacent states does not appear essential. A moving-average penalty performs better on some datasets. Therefore, we make the narrower claim that TD-based correction helps in gradual-drift settings at a matched penalty weight, rather than claiming universal superiority over non-RL alternatives (Section 6.5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.