acceptodds
Under review as a conference paper at ICLR 2027

Rank Is Not Enough: Adaptive Policy Selection at Singular Ties

Abstract

Rank is not enough to predict the difficulty of selecting among tied policies. We construct finite MDPs with identical state and action spaces, transition-rank budgets, symmetry multiplicities, Fisher information, exact policy tie, and comparison-gradient norm, yet whose fixed-radius local minimax regrets differ by a factor . The governing object is the Fisher-relative geometry between the contingent cone of feasible transition perturbations and the polytope of tied policy gradients. For equivariant sparse low-rank transition models, we first prove that the oracle-local root- decision problem converges exactly to the Gaussian game determined by . We then construct one unknown-center rule that attains this oracle value uniformly on quantitative stable support–rank–policy strata. At singular rank intersections, however, we prove a positive adaptation tax: no branch-agnostic procedure can simultaneously attain the neighboring oracle risks. A commitment lift realizes the rank-diffusion gap in the complete deterministic policy class. Finally, we give a certified finite algorithm for the Gaussian game and finite-sample categorical experiments that recover adaptive attainment, adaptation failure, and the predicted scaling.

Primary area:
learning theory
Keywords:
reinforcement learning theory, Markov decision processes, adaptive policy selection, local asymptotic minimax theory, singular statistical models, policy ties, Le Cam theory, Gaussian decision theory, low-rank transition models, equivariant reinforcement learning, tangent cones, minimax regret

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_BbbhOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the statistical difficulty of selecting an optimal policy in a structured Markov decision process, particularly when multiple policies are tied and the transition model has low-rank singularities.

Critical strengths

Due to the poor presentation, I cannot fully understand this paper and verify the correctness and soundness of this paper. If we assume that the results are correct, then the possible strength is: this paper establishes an exact local asymptotic minimax characterization of structured MDP policy selection through a Gaussian decision problem, showing that difficulty depends on the alignment between feasible transition perturbations and policy-value gradients, rather than transition rank alone.

Critical weaknesses

The presentation of this paper is extremely poor and needs another round of revision. This makes this paper's argument untenable.

  • Many notations are not defined when used. For example, the abstract never defines what are dd, rr, and K0\mathcal{K}_0. In the introduction, what are the definitions of μ,r,Π\mu,r,\Pi?

  • The introduction is poorly written since the studied problems are not well-motivated, and the authors do not clearly contrast this paper with previous papers.

  • Throughout the paper, many sentences are claimed without any motivation, which makes readers lost.

Overall, the current version is hard to be understood by human.

Questions for authors

Please rewrite the paper in a more interpretable way.

Reviewer_FiTLOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies local policy-selection difficulty through the Fisher-relative geometry of a feasible transition tangent cone and the convex hull of tied-policy value gradients. This distinguishes statistical distinguishability from the consequences of choosing the wrong policy. For finite, structured MDPs observed through independent categorical transition samples, the paper proves an exact fixed-radius local minimax Gaussian decision limit. Then, this work constructs a center-independent procedure attaining the oracle value uniformly on quantitative stable support–rank–policy strata. A separate construction establishes a strictly positive cost of hiding which of two noncollinear one-dimensional rank-one branches generated the data. Finally, this work proposes a commitment lift realizes a concentrated-versus-diffuse gradient separation of order ΘR(d/r)\Theta_R(\sqrt{d/r}) when decisions range over the complete deterministic nonstationary Markov policy class. This work also provides certified Gaussian-game approximations and categorical-data simulations to support its results.

Critical strengths
  1. A coherent decision-theoretic treatment of structured local control. The paper connects nonconvex sparse-determinantal geometry to the actual policy-regret loss and proves matching decision limits.

  2. A convincing full-policy realization of the geometric separation. The commitment construction controls the entire deterministic policy class, gives all other policies a uniform positive base-gap lower bound, and matches the complete base-policy value table across the concentrated and diffuse tasks.

Critical weaknesses
  1. My main concern is the presentation. This work uses too many paragraphs in the main content. For example, in the introduction, there are 5 paragraphs. For the table (line 378 - 408), the tables do not have caption to explain the role of this table.
Questions for authors

Please see the weakness part.

Reviewer_oyigOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary
  • This work studies policy selection in structured finite-horizon MDPs. It shows that rank constraints alone do not determine the difficulty of policy selection and provides an adaptive procedure that attains the oracle local minimax regret under stability conditions.
Critical strengths
  • This work shows that rank alone does not determine how difficult it is to select a good policy in an MDP.
  • This work proposes a method that learns the unknown model structure and shows that, under stability conditions, this learning does not increase the leading regret term.
Critical weaknesses
  • The paper is very difficult to follow. For example, in the abstract, it uses expressions such as “transition-rank budgets,” “symmetry multiplicities,” “comparison-gradient norm,” and “fixed-radius local minimax regrets” without explanation. It is unclear even what rank refers to. The notation ΘR(d/r)\Theta_R(\sqrt{d/r}) is also introduced without explaining what RR and ΘR\Theta_R mean. Similarly, expressions such as “oracle-local root-n decision problem,” “quantitative stable support–rank–policy strata,” and “rank-diffusion gap” are difficult to interpret. This makes the problem setting and main contributions difficult to understand.

  • It also occurs in the main text. Key quantities and technical expressions are used without sufficient explanation. To name a few, Θn,R(P0)\Theta_{n,R}(P_0) is not defined in the problem formulation (Eq. (4)), the algorithm refers to a “safe tie patch” without explaining it (Section 3, line 205), and the “restricted decision width” ωK(g)\omega_K(g) is used without defining it (Figure 1, Section 5, lines 310-316). Their definitions are deferred to the appendix, making the problem setting, algorithm, and theoretical results difficult to follow. See the question below.

Questions for authors
  • Could you revise the paper to make this easier to follow? Please explain the problem setting, main idea, and results more clearly, and define the key terms and quantities when they are first introduced.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.