acceptodds
Withdrawn from ICLR 2027

What Action Geometry Buys: A Design Space for Optimal-Transport Bellman Backups

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Many Reinforcement Learning (RL) action spaces carry geometry that maximum-entropy backups ignore. Laplacian optimal transport (LOT) regularization admits a closed form—an average of cost-shifted local softmax—governed by one anchor-normalized action kernel; we turn this closed form into a design space, proving which kernel property buys which guarantee. Concentration buys planning: exact pruning error and a localized planner replace branching over actions by an effective count , still targeting the unpruned regularized value. Uniform cost stability buys learning: reusable reward-free geometry and same-stream tabular Q-learning. Poisson truncation buys local computation: a graph-local kernel targets the exact heat-kernel value with no explicit dependence under polynomial growth. Strictly positive couplings buy interpretation: inverse policy curvature is effective resistance on a value-dependent transport graph, reducing to Fisher–Rao at zero cost. Matched obstructions show these properties are not free; choosing a geometry is choosing a row of our map.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, optimal transport, regularized MDPs, Bellman operators, action geometry, graph Laplacians, local planning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_F6zsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Building on the closed-form semi-relaxed OT Bellman backup of Dam (2026), the paper studies Laplacian OT (LOT) regularization, whose backup is a mixture of cost-shifted local softmaxes indexed by an anchor-normalized action kernel. It then catalogues which kernel properties yield which guarantees: concentration gives a pruning bound and a localized planner (LocalLOTPlan) whose complexity depends on an effective action count dker⁡d_{\ker} rather than KK; a Poisson-truncated heat kernel gives local computation; uniformly stable costs give reward-free geometry learning and tabular Q-learning convergence; and strictly positive couplings let inverse policy curvature be read as effective resistance on a value-dependent graph. The submission spans 151 pages with 23 theorems. Validation is limited to synthetic numerical illustrations on path, cycle, and ring graphs with up to K=512K=512 actions and small tabular MDPs, all reported in the appendix.

Critical strengths

None. The closed-form analysis is internally consistent, but I could not identify a strength that bears on acceptance at ICLR given the concerns below.

Critical weaknesses

Format and verifiability: The paper is a 151-page monograph with 23 theorems and roughly 70 supporting results, whose main text largely states results and defers definitions, algorithms, and all figures to the appendix. Its correctness cannot be reasonably verified within a conference review, and the 9-page body is not self-contained.

No evidence of relevance to RL practice: The only experiments are controlled illustrations on synthetic graphs and toy tabular MDPs with known transition kernels. No learning agent, standard benchmark, or comparison with existing RL methods is reported, so it remains unknown whether action geometry "buys" anything in practice.

Limited scope of the guarantees: By the authors' own statements, planning bounds target a scalar regularized root value rather than policy return, learning results are tabular, and the effective-resistance identity yields neither an algorithm nor a rate. Combined with the reliance on Dam (2026) for the core closed form, the contribution reduces to a catalogue of properties.

Questions for authors
  • Beyond Lemma 1 and Theorem 1, which results are new relative to Dam (2026)? Please identify the one or two theorems you consider the core contribution.

  • Can you show on any standard discrete-action RL benchmark (e.g., discretized control or large-action recommendation tasks) that an LOT-based agent improves return or sample efficiency over MaxEnt baselines?

  • All experiments assume a known kernel PP and exact expected-QQ evaluations. How do the planning guarantees behave with function approximation or sampled transitions?

  • Assumption 2 requires local kernel access in O~(∣Is,j∣)\tilde O(|I_{s,j}|) time. In which practical action spaces is such access available without scanning all KK actions?

  • Given its length, the work may be better suited to a journal (e.g., JMLR), where the full proofs can be properly refereed.

Reviewer_J6WdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper deals with reinforcement learning with discrete action spaces and how to utilize geometry of action spaces. Here by "geometry" the authors mean a notion of action similarity that is captured through a kernel in action space. Given a discounted MDP, consider its QQ function, then the max over actions in the Bellman backup max⁡aQ(s,a)\max_a Q(s, a) is replaced by a regularized soft max based on (Dam, 2026).

Critical strengths

Exploiting the mathematical structure of the action space is a very natural approach and interesting in its very idea. In particular, when actions that are close to each other also cause similar effects in the RL environment, it is natural to exploit this problem structure.

Critical weaknesses

It was extremely difficult to read this paper, and it is actually challenging to organize the critique of the paper in a meaningful order.

Abstract

The abstract relies on concepts developed within the paper or developed in very specific prior work. Without reading the paper first, the abstract cannot be meaningfully followed.

Organization

The paper lacks the typical organization, i.e., high-level motivation, briefly building up context and core concepts, naming the research gaps in the prior state of the art, and giving an overview of the contribution to fill in the gap.

In order to understand the very first paragraph of the paper, very specific prior publications need to be known, which are however not cited at this point for the reader. The paper lacks a gentle introduction of various notions and terms, including the citation of the original work. For instance, a reader of the paper can hardly be expected to know what "anchor-normalized kernel w(s,j)w^{(s, j)} in (14)" should mean in the second paragraph of the paper, without reading all technical details until the middle of the paper.

Although the aspect of "geometry" is central for the narrative of the paper, the authors do not deliver early on what they mean by "geometry", and certainly it is not meant in the strict sense of the word. Likewise, it is unclear what the terms "deterministic action sets", "LocalLOTPlan", "Uniform MaxEnt", "worst-anchor count dkerd_{ker}", "asymmetric child accuracies", and so on and so forth should mean.

The second paragraph of the introduction reads like a sequence of uncoupled sentences. A clear thread of thoughts missing, terms and notions are not introduced.

Related work on the big picture

Of course, the very basic idea, namely exploiting action space structure, goes back to classic references, like (Dulac-Arnold et al., 2015) or (Chandak et al., 2019). However, this would require the authors of this submission to put their work and contribution into a larger conceptual context, which is missing.

Also concerning the bigger picture: The main results concern "planning" (Local Planning from Action Geometry), but this problem setting is not clearly defined and the paper is primarily framed as RL. On the other hand, also section 7 actually does not discuss a RL task and we miss comparisons with baselines.

Readability

The whole submission, starting from the abstract, is extremely difficult to read. Take for instance the preliminaries given in section 2. It reads almost like a brief cheat sheet of notions, without citations. A reader can only follow if they already know what is written beforehand.

All notions solely rely on brief mathematical expressions, largely skipping prose text explanations. While this is hard to handle for any reader, it also requires you to be absolutely precise on the formal language. In particular, you should then avoid overloading notions, e.g., for λ\lambda.

This also increases pressure on the precision of the usage of notions. For instance, if you write "1/(2τλ)1/(2 \tau \lambda)-smooth" then you need to define what notion of smoothness you are using here.

And gERg_{ER} and gtg_t have not been defined, but appear in statements in section 2. GG is used in section 2, but only defined in section 4.

Or in what terms is Algorithm 1 "numerically stable"?

Excessive size of the paper

The whole submission has 151 pages, which cannot possibly be checked within the reviewing time frame given.

On the other hand, the actual content of the paper starts with section 3, from which on the paper only lists a sequence of results, without much of an explanation or intuition or a larger conceptual context, frequently simply referring to the appendix for the details.

Given that the proofs span an order of 100 pages, it is not possible to check the soundness of this paper.

Even the final discussion (Scope and Conclusion) refers to tables in the appendix.

In fact, it makes it effectively impossible to me to give a numerical value for the soundness of the paper.

Questions for authors
  • The abstract does not mention that your approach is limited to discrete action spaces, but the introduction suggests so. First, can you make this explicit early on. Second, can you discuss how your approach is really bound to this setting.

  • Will code be provided to the public?

Reviewer_q5rjOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper develops a framework for localized optimal-transport (LOT) regularized Bellman operators over structured finite action spaces. Using a normalized-kernel representation, the authors express the Bellman backup as a mixture of anchor-dependent log-sum-exp operators. The paper establishes a sharp characterization of admissible action pruning (Lemma 2) and develops a recursive localized planning algorithm with theoretical complexity guarantees (Theorem 2). It further introduces truncated graph heat-kernel constructions for efficient local computation (Theorem 3 and Corollary 1), studies reward-free action-geometry estimation and its integration into online tabular learning (Theorem 4), and connects the induced dual policy geometry to effective resistance (Theorem 5). The analysis also includes lower bounds and structural limitations that clarify the conditions under which these computational and statistical benefits can be achieved.

Overall, the work presents an interesting theoretical synthesis of optimal transport, spectral diffusion, and recursive planning.

Critical strengths
  1. The normalized-kernel formulation provides a clear connection between LOT Bellman backups, localized action supports, and entropy-regularized Bellman operators.

  2. The sharp worst-case pruning characterization in Lemma 2, together with the constructive heat-kernel approximation, establishes useful theoretical connections between kernel concentration, localization accuracy, and effective action-space complexity.

  3. The paper carefully distinguishes statistical complexity arising from transition sampling from arithmetic complexity associated with local-kernel access. It also discusses structural requirements and limitations of different kernel constructions.

  4. The reward-free action-geometry learning framework, together with its statistical guarantees and effective-resistance interpretation, provides an interesting theoretical extension of the localized planning framework.

Critical weaknesses

W1: Does the learned action geometry preserve behaviorally meaningful distinctions between actions?

The reward-free geometry in Eq. (34) represents each action through its effect on a single probe distribution ν\nu, rather than its full state-conditional transition kernel. Consequently, distinct actions may receive identical signatures despite having substantially different transition behavior. For example, consider a two-state MDP with a uniform probe distribution, where action aa preserves the current state and action bb deterministically switches between states. Both actions leave the uniform distribution unchanged, implying za=zb=0z_a=z_b=0 and hence Cabt=0C_{ab}^{t}=0 for every t≥0t\geq 0, although their state-conditional transition kernels differ at both states. More generally, Pa⊤ν=Pb⊤νP_a^\top\nu=P_b^\top\nu is sufficient for this ambiguity.

This highlights an important distinction between accurately estimating the proposed population geometry and establishing its relevance for downstream decision-making. Although Theorem 4 provides guarantees for learning the defined cost, these results do not necessarily imply that geometrically similar actions have similar state-conditional effects or induce comparable action values. The probe-related degeneracy discussed in Appendix H.4 further motivates clarifying the conditions under which the learned representation remains informative. Moreover, the geometry experiments primarily evaluate estimation accuracy relative to the oracle geometry, without demonstrating whether the learned geometry improves downstream planning or control compared with a geometry-free baseline.

W2: When do the localized planning guarantees yield meaningful computational advantages?

Although Theorems 2–3 establish favorable complexity guarantees for localized planning, their practical significance depends on the required localization radius and the associated structural constants. In particular, Lemma 2 shows that the admissible discarded mass can become exponentially small when the value span is large relative to T0T_0, potentially requiring the retention of a substantial portion of the action space. Consequently, the sufficient localization radius in Corollary 1 may be considerably more conservative than the neighborhood sizes used in practice. The experiments in Appendix H.3 employ much smaller, empirically selected neighborhoods, but do not establish how these choices relate to the certified approximation requirements. Furthermore, although Propositions 3–4 separately characterize regularization bias, the resulting end-to-end accuracy–complexity trade-off relative to the original unregularized MDP remains insufficiently explored.

A further limitation concerns the experimental validation of the stochastic recursive estimator analyzed in Theorem 2. The ring-control experiments use known transition probabilities and exact expected action values rather than sampled transitions from a generative model. While these results demonstrate the computational benefits of localized action evaluation, they do not directly validate the estimator's transition-sample complexity or nonlinear bias correction. Additionally, the use of a strongly goal-directed anchor prior and potentially different pruning-calibration procedures raises questions about the robustness and comparability of the reported improvements. Experiments using sampled transitions, matched approximation-error budgets, and alternative anchor priors would provide stronger empirical support for the theoretical contributions.

Questions for authors

Question-W1

  • Under what assumptions on the transition kernels and probe distribution does a small geometric cost imply meaningful state-conditional or value-based similarity between actions?

  • Could multiple probe distributions or state-conditioned signatures mitigate this ambiguity? What additional statistical or computational requirements would such extensions introduce?

  • Could the authors provide downstream planning or control experiments comparing the learned geometry against a geometry-free baseline, including sensitivity to the probe distribution and reward function?

Questions-W2

  • Under what conditions on temperature, diffusion time, and action-graph structure does the certified localization radius yield a genuine sub-KK computational advantage? How do the hidden complexity constants depend on these parameters?

  • Could the authors clarify the end-to-end accuracy--complexity trade-off when accounting for both localization error and regularization bias?

  • Could the actual stochastic recursive estimator be evaluated using sampled transitions, with estimation error, generative-model calls, and computational time reported across different accuracy requirements and action-space sizes?

  • Could the localization methods be compared under matched approximation-error budgets, including sensitivity to the anchor prior?

Comments on the paper

1 comment

We have decided to withdraw our submission from ICLR 2027 to revise the presentation, clarify the scope of our guarantees, and strengthen the empirical evaluation. We thank the reviewers and area chair for their time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.