- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes the Bidirectional Local Path-Transition Kernel (BLPTK), which uses the forward and reverse conditional distributions of local trajectory segments to identify local multi-path structures such as splits and merges in policy rollouts. The method first generates candidate partitions through spectral clustering, then explicitly validates them using mass and conductance criteria, and finally uses the resulting regions to impose a path-diversity constraint during reinforcement learning fine-tuning. Experiments cover synthetic graph structures, D3IL-Avoid, and a simulated 7-DoF robotic manipulation task.
Critical strengths
- The paper clearly distinguishes outgoing splits from incoming merges, and points out that endpoint-based or forward-only information is insufficient to fully characterize such directed local path structures. The research question itself is clearly defined.
- The authors specifically compare against a time-only clock baseline and same-feature clustering baselines, and explicitly report cases where BLPTK is outperformed by other methods rather than avoiding negative results.
- In the simulated robotic task, constraining branch mass using the frozen local partitions preserves both demonstrated routes across five seeds, showing that the detected structures have at least some downstream utility.
Critical weaknesses
- The main components of BLPTK, including conditional kernels, kernel regression, normalized spectral clustering, and conductance tests, are all based on existing techniques. The authors also explicitly acknowledge that they do not introduce a new transition operator, coherent-set decomposition, or statistical theorem. The main contribution therefore appears closer to a specific combination of existing tools together with a reformulation of the problem.
- Across the 14 controlled cells, BLPTK is tied for the best result in only 5 cases and is outperformed by other methods in the remaining 9. In addition, in 9 of the 14 settings, its difference from the bare eigencount baseline is no more than 0.005.
- The downstream evaluation contains only one simulated 7-DoF task, one geometry, and five seeds, and does not compare against existing diversity-preserving approaches such as trajectory entropy, mutual-information objectives, mixture policies, or behavior-regularized RL.
- The references contain clear formatting and completeness issues. For example, Ref. [19] starts directly with the paper title and omits the author information. Conference names are also presented in inconsistent formats, including “International Conference on Learning Representations,” “ICLR,” “ICML,” and “IJCAI,” and the bibliography is not consistently formatted.
- Most of the experiments are placed in the appendix. Based on the main paper alone, it is difficult to independently assess whether the method actually outperforms the baselines, under which conditions it fails, and whether the results are stable.
- The template also appears not to follow the standard ICLR 2027 format. For example, the font differs from the Times-style font used by the official template.
- The title and first-level section headings also do not match the small-caps style of the official template.
Questions for authors
See Weaknesses.
2 comments
Response to Reviewer oRXk (2/2): baselines (W3), presentation and formatting (W4–W7)
W3 (one task, no comparison with existing diversity methods). One task, one geometry and five seeds remain true for the RL result, and the revised paper says so. The downstream probe is there to show that the formulation can be used, by a plain penalty on detected regions. We do not offer it as a new diversity method. We ran the families you name where we could, end to end, reading reward first, because a policy that stays near behaviour cloning keeps its routes without fine-tuning. Behaviour cloning itself fails our gate on collisions in every seed.
- Behaviour-regularised RL, full-policy KL to the cloned policy: doses 1, 0.3, 0.17 and 0.1 pass 0, 0, 0 and 2 of 5 seeds, against 4/5 for the local constraint. At dose 0.1, the closest in reward, its mean minority share among on-route rollouts is 17% against 37% for ours. Run to 3200 iterations, KL at dose 0.3 passes 1/5 and keeps 39% of the reward gain, where the local constraint passes 5/5 and keeps 79%. We make no claim about untried doses.
- Trajectory entropy: an unlocalised bonus on whole paths passes 4/5 at dose 0.3 and diverges at doses 1 and 3. With a 3200-iteration minimum and the same stopping rule, both it and our constraint pass 5/5. Entropy keeps the routes more even (minority share 0.46 against 0.32) and ours keeps more of the reward gain (0.79 against 0.69). Neither dominates, and this comparison does not show that localising is what helps.
- Mutual information: a DIAYN-style head on the diffusion policy passes 0/5 at doses 0.3, 1 and 3. In our first runs the declared validity check failed (discriminator accuracy 0.26 to 0.30 at the end of warm-up, chance 0.25). We then declared and ran a second version with a warm-up four times longer, run to convergence. Its discriminator identifies the four-way latent perfectly in all 15 runs, and every run still ends on one route while keeping 0.97 to 1.02 of plain fine-tuning's reward gain on average across doses. The warm-up left only 20 to 46% of rollouts on a route; we did not measure which features identified the codes. This is one implementation on one task, but it is a valid test, and it did not keep the routes.
- Behavioral Mode Discovery: we reimplemented it from its paper, as we could not find released code. The paper uses two-step DDIM, deterministic for the frozen policy and stochastic during fine-tuning. Deterministic short-step sampling leaves the routes on our arm, so we used a stochastic 10-step sampler in both stages, which is a departure from it. The discriminator stayed near chance at the end of discovery, failing the same declared check. This is an attempt we could not make work on this arm, and we do not count it as a result against that method.
- Mixture policies: we report no mixture baseline, because we found no published fine-tuning method of that form to reproduce.
These objectives act on the policy over all states, and none of them says where a rollout set divides or rejoins. That statement is what the formulation provides, and a detector can supply it for any of them. We do not show that using it improves on them.
W4, W6, W7 (references, font, small caps). You were right. Our build silently substituted the font and dropped the small caps. The revision uses the official style with a Times face and a consistent bibliography.
W5 (experiments in the appendix). The rebuttal limit is 10 pages. The revision moves the claim map and a compact downstream table into the main text, and adds a schematic of the formulation as Figure 1. The natural-merge results are in the main text, with their rollout figure in the appendix (Figure 5).
Response to Reviewer oRXk (1/2): the formulation (W1) and the toy comparison (W2)
Thank you for the review. New results below were declared before they ran.
W1 (combination of existing tools). We agree that every component is standard, and that the contribution is a reformulation of the problem. That reformulation is what we propose: where a set of rollouts divides and rejoins, posed on a pair of directed local path laws and read out as counts at a declared resolution. The revised paper adds a single schematic of this formulation as Figure 1 (page 3): the forward and backward path laws at an anchor, the count read from them, and fine-tuning posed as a constraint on the same kernels at the detected regions. The same estimator on the same anchors classifies 361/361 probes with path features and 255/361 with endpoints. A discrete-time process is specified by its initial law and its conditional laws given the past. A policy is a probability kernel from its available information to actions; for a Markov policy that information is the state, augmented with memory or history when required. Together with the dynamics and the start distribution it induces a law over paths, regardless of whether its action density is tractable. Our object is that induced law at the level of local behaviour, namely the distribution of a path feature given the local context, in each direction. These conditional path-feature kernels generally depend on the policy parameters, including through rollout occupancy. At contexts where conditioning is well defined, a fixed measurable partition therefore defines policy-dependent masses; their gradients require differentiability and justification for differentiating the conditional expectations. These laws provide a natural object for comparing local behaviour across policies. Given samples from that law, single linkage uses only their pairwise distances and the total mass of each block, so it is one possible readout of the empirical kernel. The kernel is the more general object because it also supports readouts that depend on how probability is distributed within a connected set of paths, such as conductance. Here conductance refers to the graph induced by the declared affinity and path law, not to the policy's transition probabilities. A new test, declared before it ran, shows what this means on real rollouts. The cloned arm policies leave both routes in 9 to 15% of rollouts, and these strays run between the routes. Growing the rollouts sixteenfold, on anchors that are two routes at both sizes, single linkage's detection falls from 0.61 to 0.04, while the readouts that use probability, ours and DBSCAN with a mass threshold, roughly halve. Ours also fails the stability gate we declared, because at our tolerance of 0.005 the strays eventually make the route split too leaky to accept. We claim this generality only for what the object can express; on detection accuracy, strong clustering baselines on the same features are close on the arm and can be substantially more accurate on the toys. Single-linkage regions also pass as many seeds downstream, so this comparison establishes no downstream advantage specific to the BLPTK estimator. The revised paper makes the link to fine-tuning explicit: reward maximisation subject to preserving local kernel masses at regions detected from base rollouts, with the gradient of such a mass stated and proved, and the sense in which our penalty is only a biased surrogate stated beside it. This differs in kind from Behavioral Mode Discovery, which conditions a steering policy on a latent throughout and adds an information reward, with no step that locates regions. We also compare the kernels of two policies directly, with no partition. That map shows where plain fine-tuning changed behaviour, including at contexts where the count did not change, though our declared test fails for the constrained policy, whose change stays near the noise floor at this sample size. How best to use detected regions remains open, and our penalty is a naive first probe.
W2 (5 ties, 9 losses, 9 of 14 within 0.005 of the eigencount). These numbers are correct, and we do not argue for the method on accuracy. Some context on the toy cells: two have no branchable anchors, so a constant predictor scores perfectly there, and BLPTK beats that predictor in the other 12. The toy truth is a single-linkage count, so agglomerative clustering is graded by its own algorithm, and BLPTK ties for best in most staggered cells. This metric does not score what is specific to our estimator, the acceptance test that returns a feasibility witness. Every method in these cells also runs on our local directed path features at the declared resolution, so the cells compare readouts within the formulation. We test the formulation itself against endpoint features on identical anchors: path features classify 361 of 361 probes, endpoints 255.