acceptodds
Under review as a conference paper at ICLR 2027

Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning

Abstract

Reinforcement learning (RL) fine-tuning is widely used in language model training to improve model performance on a target task while limiting drift from a reference policy. A standard way to balance this trade-off is via a KL-regularized RL objective, although this formulation does not by itself provide a principled way to set the regularization coefficient. In practice, the coefficient is typically chosen heuristically or via hyperparameter search, which can lead to unnecessary overhead in training cost or undesirable reward–retention trade-offs. We instead propose a game-theoretic framework that gives this trade-off an explicit statistical interpretation. Specifically, we study a sequential game in which an agent chooses a policy to maximize cumulative reward while a monitor observes policy outputs over time and tests for deviations from the reference policy. Although not originating from the same perspective, we show that the resulting equilibrium policy can nonetheless be expressed as the solution to a KL-regularized RL problem for an optimal regularization parameter that can be viewed as maximizing reward per unit of statistical distinguishability. Drawing on classical results from concave-convex fractional programming, we provide a principled method for learning this equilibrium coefficient via reduction to the KL-regularized RL objective, thus allowing for flexible integration into standard fine-tuning pipelines. In experiments with Qwen3-8B and Llama-3.2-1B, we demonstrate that our methods result in competitive reward-retention trade-offs in a continual learning setting, and illustrate how our framework may be used to audit API providers serving open-source models.

Primary area:
learning theory
Keywords:
game theory, KL-regularized RL, fractional programming, Dinkelbach

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_rWTwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a sequential detection game in which an agent maximizes reward accumulated before a monitor detects deviation from a reference policy, subject to a type-I error constraint. In the high-confidence asymptotic regime, the analysis connects reward per unit KL divergence to KL-regularized RL with an endogenous regularization coefficient. A stochastic bisection procedure estimates this coefficient through RL-oracle calls. Experiments with Qwen3-8B and Llama-3.2-1B illustrate length-coherence trade-offs from a fixed story-completion prompt and model auditing with an independently trained monitor policy.

Critical strengths
  1. Relating the agent’s accumulated reward to the statistical evidence needed for detection gives a concrete meaning to the reward-to-KL ratio and the resulting regularization coefficient in Section 3. The paper explicitly limits its equilibrium characterization to leading order as α\alpha tends to zero (p. 4, lines 176–184), which helps delimit the theoretical contribution.
  2. The auditing experiment distinguishes a detector using the true deployed policy from a monitor using an independently trained policy, and evaluates the detectors on the same observed completions (Section 5.2, p. 8, lines 410–425). This is a useful design for studying the gap between the proposed idealization and a monitor without access to the agent’s realized policy.
Critical weaknesses

1.What stopping or interval-update rule handles an exact or statistically unresolved root hit in Algorithm 1, and what coefficient-error and complexity guarantee holds for that rule?

2.Does Appendix B.1 intend only prescribed-policy utility comparisons, or an approximate Nash guarantee? For a lower-coefficient monitor �1��1�, the drift under an agent policy with mean reward �2��2� is $

μ−M(b)b\frac{\mu-M(b)}{b}

$.

3.Since �3��3�, feasible policies with $ μ=M(b)+η\mu=M(b)+\eta$ can yield unbounded gains as �5��5� approaches zero from above. Such policies exist in the bounded binary example above. The displayed B.8–B.9 inequalities do not rule out these deviations. Please clarify the intended scope. This observation does not contradict those narrower inequalities or the exact-root equilibrium.

4.What explains the grid mixture’s excess null rejections? Please specify the null-trial count and independence, monitoring horizon, treatment of non-detections, and how sequence likelihoods account for sampling, EOS and truncation. Reconcile the existing null results before interpreting detection times.

5.What operational preference or per-round cost determines the calibration margin �6��6�, and how is it chosen before examining coherence? The reward shift preserves the optimizer for a fixed �7��7�, but changes �8��8� and hence the selected root (p. 7, lines 337–342). Please also specify whether the reward uses characters or tokens and give the normalization units needed to interpret �9��9� and �10��10�.

Questions for authors

See in weakness.

2 comments

Thank you for taking the time to review our submission. Our responses to your questions and comments are below.

What stopping or interval-update rule handles an exact or statistically unresolved root hit in Algorithm 1, and what coefficient-error and complexity guarantee holds for that rule?

In Section 4 (L268) we mention that Algorithm 1 needs O(M(β^)−2)O(M(\hat{\beta})^{-2}) samples per iteration, so the sample count will grow as β^→β∗\hat{\beta} \rightarrow \beta^*. As a practical relaxation, in our experiments we skip the confidence sequence and simply bisect based on the sign of M^n(β^)\hat{M}_n(\hat{\beta}) using a fixed, large n (and terminate if it equals zero exactly). While this approach does not come with coefficient-error guarantees, we found that it worked well in our experiments.

Does Appendix B.1 intend only prescribed-policy utility comparisons, or an approximate Nash guarantee? For a lower-coefficient monitor b<βb < \beta, the drift under an agent policy with mean reward μ\mu is μ−M(b)b\frac{\mu - M(b)}{b}.

And

Since M(b)>0M(b) > 0, feasible policies with μ=M(b)+η\mu = M(b) + \eta can yield unbounded gains as η\eta approaches zero from above. Such policies exist in the bounded binary example above. The displayed B.8–B.9 inequalities do not rule out these deviations. Please clarify the intended scope. This observation does not contradict those narrower inequalities or the exact-root equilibrium.

Thanks for raising this point and giving us the opportunity to clarify. As written, Appendix B.1 is a prescribed-policy utility comparison which shows that their utilities are close to the equilibrium values. However we believe that it is possible to get an approximate Nash equilibrium guarantee and rule out the unbounded deviation gains in your example if the monitor instead uses βhigh\beta_{high} while the agent plays πβlow\pi_{\beta_{low}}. The monitor’s side of this calculation carries over directly from Theorem B.9 (part 2), and the agent side requires a short corollary showing that no deviation earns more than βhighlog⁡(1/α)\beta_{high}\log(1 / \alpha). We’ll be sure to clarify this in the revision.

What explains the grid mixture’s excess null rejections? Please specify the null-trial count and independence, monitoring horizon, treatment of non-detections, and how sequence likelihoods account for sampling, EOS and truncation. Reconcile the existing null results before interpreting detection times.

We rechecked our code and found that this behavior was coming from some oddities in the Tinker API we used for model fine-tuning and serving. In short, we were computing the probabilities of token sequences under the null hypothesis model by using an untrained Tinker “training client”, which should have been fine in theory, but introduced some non-determinism in practice. We fixed this by using a Tinker “sampling client” instead. After re-running, the high-level takeaway is the same: our SPRT comes close to the skyline performance while significantly outperforming the grid mixture baseline. The updated results for Qwen are below. (Unfortunately Tinker has deprecated the Llama endpoint we were using, so we could not rerun those results.)

We used 200 independent null trials, each with a fresh completion from πref\pi_{ref}. A completion’s log-likelihood is the sum of the log-probabilities of its generated tokens (including EOS). Completions truncated at the 2048 token cap are scored by the probability of the observed sequence.

MethodMedian detection tokenHonest FPR
Trained monitor654.0%
Agent alternative (oracle)314.5%
Grid mixture1611.0%

What operational preference or per-round cost determines the calibration margin ρ\rho, and how is it chosen before examining coherence? The reward shift preserves the optimizer for a fixed β\beta, but changes M(β)M(\beta) and hence the selected root (p. 7, lines 337–342). Please also specify whether the reward uses characters or tokens and give the normalization units needed to interpret ρ\rho and β\beta.

This is a good point worth clarifying: ρ\rho is the empirical counterpart to the constant cc in our discussion of Assumption 3.1 (L168). There, cc is interpreted as the minimum improvement in expected reward required for the agent to want to update the reference policy. In practice, cc could be a proxy for the (monetary, time, effort) costs associated with model training. In our experiments, we chose the values of ρ\rho to span small-to-moderate costs relative to the reward scale, and did not tune them.

Reward uses characters not tokens in our experiments. We will clarify this in the revision. ρ\rho is in normalized reward units, i.e. one unit corresponds to 2048 tokens. β\beta is in normalized reward units per nat of KL.

Thank you for answering my question.

Reviewer_YLBFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a two-player "sequential detection game" between an agent that deploys a fine-tuned policy and a monitor that runs an correct sequential test. The agent maximizes expected cumulative reward before detection while the monitor minimizes it. Using (i) the Robbins–Siegmund lower bound, (ii) Wald's identity, and (iii) Dinkelbach fractional programming, the authors argue that the equilibrium agent policy is the KL-regularized tilt and give a stochastic bisection algorithm (Alg. 1). Experiments on Qwen3-8B / Llama-3.2-1B show bisection lands near the "elbow" of the reward–coherence frontier, and an auditing experiment where an independently trained monitor's SPRT detects a strategically fine-tuned model faster than a grid-mixture baseline at controlled type-I error.

Critical strengths

This is a well-written paper with a genuinely attractive idea: deriving the KL-regularized RL objective and its coefficient from a detection game. The core derivation chain is largely correct as far as I can verify. Experiments on Qwen3-8B / Llama-3.2-1B show bisection lands near the "elbow" of the reward–coherence frontier, and an auditing experiment where an independently trained monitor's SPRT detects a strategically fine-tuned model faster than a grid-mixture baseline at controlled type-I error.

Critical weaknesses
  • The central "Nash equilibrium" claim is never stated as a theorem, and at the exact level it is false: it holds only at leading order α→0\alpha \rightarrow 0, which the abstract/intro do not convey;
  • The "principled, non-heuristic" coefficient β\beta in fact depends on an arbitrary calibration margin ρ that the experimenter picks, and the paper's own Lemma B.3 shows β⋆ ≲ σ²/(2ρ);
  • The monitor's utility is -inf under honest deployment for any power-one test, making the normal-form game ill-posed as written;
  • The mixture-test baseline in Table 1 empirically violates Ville's inequality (FPR 0.09–0.20 at nominal α = 0.05), which suggests an implementation error that undermines the comparison.
Questions for authors

The reward is the number of characters/tokens in a completion. Completion lengths under LLM policies are empirically heavy-tailed (mixture of short and cap-limited 2048-token completions). What is the motivation of choosing this reward?

In the abstract: "our methods result in competitive reward-retention trade-offs"; but l. 79–80 says "yields better reward-retention trade-offs than compute-matched manual tuning," while the figures show bisection points landing on the elbow of the curve traced by the grid. With only two bisection runs per setting and a single prompt, the evidence is thin for a general claim. Could you compare with SoTA RL methods on standard benchmarks such as AIME?

3 comments

Thanks for taking the time to review our submission. Our replies to your questions and comments are below.

The central NE claim is never stated as a theorem, and at the exact level it is false: it holds only at leading order, which the abstract/intro do not convey

We agree that the precise equilibrium we consider is first-order asymptotic (as α→0\alpha \rightarrow 0). The reason for this is that we are operating in discrete time and the test statistic is updated in discrete jumps. It is generally challenging to compute exact expected stopping times for crossing events of discrete-time processes. We do mention this in Section 3, but we’d be happy to propagate this up into the abstract/intro as well. The Nash equilibrium is stated on line 213 and follows immediately from the preceding discussion (L186-L212). That being said, we’d be happy to state this as a formal result in the next revision.

The "principled, non-heuristic" coefficient in fact depends on an arbitrary calibration margin ρ that the experimenter picks, and the paper's own Lemma B.3 shows β ≲ σ²/(2ρ);

ρ\rho is the empirical counterpart of the constant cc in our discussion of Assumption 3.1 (L168). There, cc has the interpretation as the minimum improvement in expected reward required for the agent to want to update the reference policy. In practice, cc could be a proxy for the (monetary, time, effort) costs associated with model training. In summary, ρ\rho is not chosen heuristically, but instead can be thought of as an estimate of c.

The monitor's utility is -inf under honest deployment for any power-one test, making the normal-form game ill-posed as written;

A simple fix is to define the monitor’s utility to be 0 under honest deployment. That being said, this never arises at equilibrium since under Ass. 3.1, honest deployment gives the agent unboundedly negative reward.

The mixture-test baseline in Table 1 empirically violates Ville's inequality (FPR 0.09–0.20 at nominal α = 0.05), which suggests an implementation error that undermines the comparison.

Thanks for pointing this out. We rechecked our code and found that this behavior was coming from some oddities in the Tinker API we used for model fine-tuning and serving. In short, we were computing the probabilities of token sequences under the null hypothesis model by using an untrained Tinker “training client”, which should have been fine in theory, but introduced some non-determinism in practice. We fixed this by using a Tinker “sampling client” instead. After re-running, the high-level takeaway is the same: our SPRT comes close to the skyline performance while significantly outperforming the grid mixture baseline. The updated results for Qwen are below. (Unfortunately Tinker has deprecated the Llama endpoint we were using, so we could not rerun those results.)

MethodMedian detection tokenHonest FPR
Trained monitor654.0%
Agent alternative (oracle)314.5%
Grid mixture1611.0%

The reward is the number of characters/tokens in a completion. Completion lengths under LLM policies are empirically heavy-tailed (mixture of short and cap-limited 2048-token completions). What is the motivation of choosing this reward?

Length is a natural reward proxy for our auditing setting. API providers that bill per token have an incentive to inflate response length while still appearing to serve the advertised model. (See Section 5.2, as well as Velasco et al. and Cao et al.). It also doubles as a clean testbed for our continual learning experiments, as it trades off directly against coherence and was affordable within our limited compute budget.

In the abstract: "our methods result in competitive reward-retention trade-offs"; but l. 79–80 says "yields better reward-retention trade-offs than compute-matched manual tuning," while the figures show bisection points landing on the elbow of the curve traced by the grid. With only two bisection runs per setting and a single prompt, the evidence is thin for a general claim. Could you compare with SoTA RL methods on standard benchmarks such as AIME?

We agree that further empirical investigation is an important future direction. That being said, we view our primary contribution as theoretical and we had a limited compute budget to run fine-tuning experiments with.

We would like to clarify that our Alg 1 is not an alternative to any SoTA RL methods. It is a way of choosing the KL coefficient that works with any of them, since it only needs an oracle for the KL-regularized objective. Any method optimizing this objective could replace GRPO as this oracle. The relevant comparison is therefore between different ways of choosing β\beta with the oracle held fixed, which is what our grid-search baseline does. We are not aware of any other theoretically grounded baseline methods for choosing β\beta, as this question has not been addressed by the literature.

Thanks for taking the time to review our submission. We’d appreciate it if you could take the time to read our rebuttal and respond, as we believe that our response addresses the concerns raised.

Thank you for your response. My questions have been addressed, and I have raised my score.

Reviewer_Fue7OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Paper introduces a game theory based framework to motivate RL finetuning and show how it (asymptotically) recovers the widely used soft-RL objective and also gives meaning to \beta. Proposed framework also provides tools for estimating \beta in presence of oracle, and a auditor to detect deviations from a reference model

Critical strengths

The paper is quite well written, and the presentation and motivation sparked a lot of other thoughts and connections in my mind. To me, that is a hallmark of a good paper and an interesting framework/way of looking at finetuning. The framework connects the RL-finetuning objective to the statistical difficulty of distinguishing 2 policies. This gives a way to estimate the optimal \beta and also audit implementations/offerings of advertised policies. While I don’t expect practitioners to use the suggested procedure to choose β, it is valuable that the framework gives meaning to something left as a hyperparameter.

There seems to be some shared structure with work on preventing distillation (arxiv 2605.22737) . This formulation also seems to suggest a natural way(?) for thinking about privacy. DP-SGD (and variants) provide indistinguishability at a parameterisation level, some version of this framework might instead be useful for studying or auditing indistinguishability at the policy(/routput/action) level, when that is all an observer has access to. These are possible extensions, but I think it is a strength of the paper and the framework, that interesting directions beyond the ones already explored naturally come to mind.

Critical weaknesses

The role and choice of ρ /“calibration margin” in Appendix C.1 isnt clear and ould be better explaind and introduced earlier since it affects the selected β. Some explanation of how the experimental values were chosen, whats the sensitivity of β wrt ρ , and how one should choose ρ for a new task would be nice.

Questions for authors

paper is primarily theoretical, and it is always tempting to ask for larger-scale/“realistic” experiments, but one thing I am curious about is whether β estimated using smaller models transfer to larger models? Since bisection requires several RL oracle queries, would be interesting if RL-ing smaller models could be good enoughfor finding beta @ larger scales

1 comment

Thank you for your positive feedback and review. We agree that there are some potentially interesting connections to areas like antidistillation and privacy, and exploring this shared structure is an important direction for future research.

The role and choice of ρ /“calibration margin” in Appendix C.1 isnt clear and could be better explaind and introduced earlier since it affects the selected β. Some explanation of how the experimental values were chosen, whats the sensitivity of β wrt ρ , and how one should choose ρ for a new task would be nice.

While ρ\rho is introduced in Section 5 (L337), we agree that it could be better explained. ρ\rho is the empirical counterpart of the constant cc in our discussion of Assumption 3.1 (L168). There, cc has the interpretation as the minimum improvement in expected reward required for the agent to want to update the reference policy. In practice, cc could be a proxy for the (monetary, time, effort) costs associated with model training. We will make this connection more clear in the revision.

We chose the values of ρ\rho to span small-to-moderate costs relative to the reward scale, and did not tune them. Regarding the sensitivity of β\beta with respect to ρ\rho, consider β∗(ρ)=max⁡π(E[r]−ρ)/E[KL(π,πref)]\beta^* (\rho) = \max_{\pi} (\mathbb{E}[r] - \rho) / \mathbb{E}[KL(\pi, \pi_{ref})]. By Danskin’s theorem, dβ∗/dρ=−1/E[KL(πβ∗,πref)]d \beta^* / d \rho = -1 / \mathbb{E} [KL(\pi_{ \beta^* }, \pi_{ref})]. So β∗\beta^* is most sensitive to ρ\rho whenever the equilibrium policy stays close to πref\pi_{ref}.

one thing I am curious about is whether β estimated using smaller models transfer to larger models? Since bisection requires several RL oracle queries, would be interesting if RL-ing smaller models could be good enough for finding beta @ larger scales

Thank you for the interesting question. Our preliminary empirical results suggest that $\beta^* $ does not directly transfer across models. For example, at ρ=1\rho = 1, the β∗\beta^* values for Llama-3.2-1B and Qwen3-8B are roughly an order of magnitude apart. It would be interesting to explore this further to see if there is better transfer between models in the same family (e.g. Qwen3-0.6B and Qwen3-8B).

With that being said, a small model estimate can still reduce training costs even when it is imprecise, by using the warm-start procedure from Section 4 (L301). If the warm start proxy is off by a factor of kk, then this only costs about log⁡2k\log_2 k extra RL oracle calls on the large model. For a 10x error like the one above, this is roughly 3 extra calls.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.