- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
I understand that the authors are investigating how much additional regret is generated in KL-regularized finite-horizon control when a policy is first solved in an abstracted/compressed MDP and then lifted back to the original environment for execution. The paper decomposes this loss into a four-term signed identity and compares two types of posterior certification intervals: independent and coupled. I believe precise error accounting is valuable, but it is not the same as a practical policy selection method.
Critical strengths
S1/S2: I prefer the treatment of signed regret accounting. Simply taking the absolute values of the solution error, abstraction error, and policy-lifting error and summing them may obscure the offsetting effects between them; the authors’ precise decomposition clarifies these offsets. The zero-width condition for the coupled certificate is not just a vague claim that “the bound is tighter,” but rather the result of explicit structural assumptions.
Critical weaknesses
W1: My first concern is that the paper does not clearly articulate the theoretical contributions relative to existing results on KL-control, Gibbs’ identities, and potential-based reward invariance. I’d like to know exactly what new theorems the quotient model and policy lifting yield—beyond simply rearranging a few familiar control identities.
W2: I have questions about the practical utility of the certificate. Even if the coupled interval is narrower than the independent interval, this does not mean it can select high-reward policies faster. I would prefer to see, given the same set of policy candidates and tolerance, exactly how many more feasible policies it certifies, how many fewer candidates it retains, and how much solution time it saves; if direct target policy evaluation is faster, the practical advantage of the certificate method needs to be redefined.
Questions for authors
Q1: I hope the authors will specify which of the new theorems relies on quotient-policy lifting, relative to existing KL-regularized control and reward invariance theories.
Q2: I request a comparison of the coupled/independent certification methods with direct target interval filtering, using the same candidate pool, the same threshold, and the same computational cost.
3 comments
Assessment after the coupled comparison
The coupled comparison answers my earlier concern about missing evaluation. At tolerance 0.01, certifying 83 of 160 inventory candidates rather than 65 is a useful improvement over the separate envelope. The signed ledger and the zero-width condition also have clear explanatory value, and the discussion of prior work is much clearer now. I still find the case for acceptance borderline, and will keep my score at 2. In the matched study, both regret screens retain every candidate, while direct evaluation makes a unique choice in all 37 pools. For absolute regret, exact evaluation is also cheaper even after including the shared fine-optimum solve: 5.27 versus 24.7 seconds in the energy study. All 160 threshold-test candidates already meet the strictest tolerance, so that experiment shows better certification of acceptable policies but leaves the wider decision benefit open. I appreciate the more focused claim about interpretation and exactness; I just don't yet find that contribution strong enough, together with the current results, to recommend acceptance.
My concerns regarding whether to accept the paper have not yet been resolved; I still need to continue discussing this with the other reviewers and the AC.
The specific contribution beyond stochastic-control corrections and reward invariance
We thank you for recognizing the exactness and interpretability of the signed regret decomposition, including its separation of deployment mismatches while preserving cancellation. We also appreciate your positive assessment of the structural characterization of certificate tightness through dynamical feasibility, error cancellation, and the Bellman-potential condition for zero width.
Thank you as well for your constructive criticism concerning the focus and novelty of the contribution, its relationship to prior work, and the practical significance and empirical evaluation of the certificates.
Your suggestions helped us sharpen the paper's focus and identify the comparisons needed to assess its contribution. We have clarified the distinction between our results and the stochastic-control and reward-transformation principles they build on, including a more explicit discussion of Arriojas et al. (Sections 1 and 8). We have also added matched evaluations of the coupled and separate certificates, direct objective screening, and exact regret on additional energy and inventory tasks, reporting interval widths, certification outcomes, and computational costs (Sections 7.2–7.4; Appendix J.4). We hope these additions make the contribution and its measured benefits and limitations clearer as we address your questions below.
Q1. Which theoretical results are genuinely new relative to existing work on KL-regularized control, stochastic dynamics corrections (e.g., Arriojas et al., UAI 2023), and potential-based reward invariance, and what fundamentally new insight does the quotient-model construction provide? (W1: Insufficiently established novelty, unfocused contribution, and incomplete related work)
We revised Sections 1 and 8 to distinguish the established foundations from the specific contribution. Arriojas et al. (UAI 2023) address fixed versus unconstrained stochastic dynamics through a transformed inference problem with the same optimal policy. Gibbs/KL identities, corrective soft tasks, and potential-based invariance are principles we build on.
We connect those principles to the coarse model we solved and the policy lift we used and a regret breakdown that keeps the real dynamics and lets positive and negative terms cancel. Theorems 3.1–3.2 identify signed prior and continuation-value witnesses for the actual coarse solve and lift that keeps the prior's relative action probabilities within each group, evaluated under the original dynamics. Theorems 4.1–4.2 connect the projection that keeps the environment's transition probabilities fixed to a joint residual that adds the accumulated information premium to the evaluation-cost difference.
Theorem 5.2 identifies the raw coupled width as:
F_C &= H+(C-\widetilde C),\\\\ W_{\mathrm{cpl}} &= \mathbb E_{Q^\rho}F_C-\min_{Q\in\mathcal Q_D}\mathbb E_QF_C. \end{aligned}$$ The expectation follows the projected policy, and the minimum keeps the original dynamics fixed. Their difference is the projected policy's extra expected residual cost above the best feasible value, without the KL penalty. The projected policy's positive visit probability for each allowed, reachable state–action pair makes zero raw width equivalent to the expected residual cost being the same for every allowed policy, or the stated Bellman-potential condition. Thus the exactness criterion concerns the controllable part of the combined residual. To show why this distinction matters, Appendix E (p. 17) fixes the surrogate and chooses the evaluation-cost difference to cancel the information premium. The coupled residual is then zero and the coupled certificate is exact, while separate mismatch budgets can remain positive. This is an analytical example, not a benchmark result. The quotient records which model was solved and how its policy was lifted. Identical state/action maps can therefore lead to different regret when the declaration changes. The residual and invariance principles do not themselves require a quotient; our claim concerns their connection to the deployment account and this certificate's exactness criterion.