- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a MoMo method to alleviate the problem of long-horizon planning reduction via latent structures. The idea constitutes preference conditioning with feature-wise linear modulation (FiLM) and low-rank neural modulation, thereby preserving probabilities ration in latent space without significant inference overhead. The method is tested on six environments with demonstration of empirical improvement of temporal consistency for longer horizons.
Critical strengths
- the geometry during decoding is preserved as to representations of candidates and anchors that justifies the theoretical propositions and intuition for integrating the modulation direction, the result is consistent across environments in the fact MoMo's waypoint reconstruction on the plain level of order is far better than state-augmentation baselines
- the main empirical is simple and consistently stable across reliable tasks with multi-seed evaluation with additional checking of spearman correlation of positive mean curve (Appendix B.3)
Critical weaknesses
- Equation (11), formulas from Appendix A do look like separate instances, though it seems they try to tell about one thing which is about ratio of unconditioned equation similar to (4). Maybe they tell about the same thing but from this paper's part the soundness is not so clear to fully understand the connection: how ratio relates to waypoints and linear gaussian transition clearer than it is described at the moment
Questions for authors
- which density ratio is the one of the loss that was used for training - equation (11), equation (17) in the appendix, or something else? Does the second term of (10) use or ?
- how often is the cost monotonic in ? Is one of the discrete collection levels held out during training?
- related to a more general sense of evaluation: is the perfect correlation meant as a success of the model or as a check that the axes are the same quantity?
5 comments
Cost monotonicity and preference-level holdout
We evaluate preference modulation using 11 uniformly spaced values of , with the same start-goal queries at each value. Across 8 training seeds and 6 environments, MoMo achieves a positive endpoint cost change and positive Spearman correlation in all 48 seed-environment evaluations. In particular, the correlation is at least for every seed on Point Four Obstacles, Point Contour, Point Habitat, and UR5. We provide these results under “More training seeds” and “Seeds and recovering cost across preference levels” comments in our response to Reviewer qGmb.
To directly assess monotonicity, we additionally examine all 10 consecutive preference intervals in each seed-specific mean cost curve. MoMo's cost is non-decreasing across of these intervals, compared with for IvI-SA, for PCA-SA, and for VIP-SA. The entire cost curve is non-decreasing across all 11 preference values in 8 of MoMo's 48 evaluations, including 6 of 8 UR5 seeds, compared with only 1 of the 144 baseline curves. Thus, while the response is not strictly monotonic in every setting, MoMo produces substantially more consistent preference ordering than the baselines. This is also reflected in its lowest mean normalized curve roughness in five of six environments (Table 6).
At the individual-query level, MoMo's endpoint cost increases with for , , and of queries on Point Four Obstacles, Point Contour, and Point Habitat, respectively, and and on Drone and Ant Habitat. We further verify preference ordering during closed-loop execution on UR5, where the crossing height decreases monotonically with for 151 of 160 query-seed pairs, and every pair achieves a Spearman correlation of at most . These results demonstrate that MoMo's preference response is consistently directed across environments and remains strongly ordered during execution.
To address the reviewer's question about generalization to unseen preferences, we introduce an additional UR5 experiment where all trajectories with are removed from both the training and decoding datasets. Despite the absence of these preference levels, MoMo achieves a mean crossing-height error of only cm against validation trajectories, places of withheld preferences between the observed boundary preferences, and maintains closed-loop success. In comparison, IvI-SA incurs a cm error and achieves success. We provide the full experimental protocol and results under "Generalization to unseen preferences" in our response to Reviewer TqsC.
Correlation representing model success
The correlation between the queried preference level and the realized waypoint cost measures whether the planner produces behavior ordered correctly by the requested preference. The horizontal axis is supplied before planning. For a fixed start and goal, we query different values of . We then compute the cumulative cost of the inferred waypoint sequences, which is represented by the vertical axis. We repeat this evaluation across held-out start-goal pairs to assess whether the preference response generalizes across different planning queries. In this way, the two are coupled only through the planner's ability to modulate plans effectively as a function of the queried preference.
The conditioning variable , which is a label during training and part of the query at inference, and the evaluation cost thus share an intended notion of risk, but their relation must be learned. This is the fundamental problem of preference-conditioning, whereby a plan must reflect a user-specified preference. The alignment of the plan with the preference is measured via the cumulative cost.
Accordingly, a perfect correlation indicates an exact rank ordering of the mean waypoint cost across the queried preference values. This is indicative of successful preference modulation. We also measure waypoint reconstruction error against feasible held-out trajectories to assess whether the inferred waypoints preserve the temporal relationships required for planning. This is done by querying the planner with a held-out trajectory's start, goal, and preference label, and observing whether it recovers its intermediate states. Therefore, both the preference modulation curve and the waypoint reconstruction plots must be viewed together to evaluate the success of MoMo and baselines. For instance, IvI-SA returns a larger difference in endpoint costs, , than MoMo in the Ant Habitat environment; however, it has a reconstruction error over the IvI baseline. Together with its substantially higher cost-curve roughness (Table 6), this shows that IvI-SA's larger cost range does not translate into equally reliable preference-conditioned waypoint predictions. In conclusion, and in response to the reviewer's third question, recovering the requested behavioral trend is therefore an intended model outcome.
Thank you for your time in reviewing our paper and for your precise feedback. We appreciate the opportunity to address the questions raised and respond to each point below.
Density ratio encoding
The relation between density-ratio encoding, the loss function, and Gaussian transitions Equations (4) and (11), and the derivation in Appendix A.1 describe the density-ratio property that underpins inference-driven planning. In the unconditioned case, this property states that learned logits encode the ratio of the probability of seeing a certain future state given an anchor state, over the probability of seeing that future state at random when drawing from the marginal state distribution. Equation (4), S⋆(s,s+)∝p(s+)pγ(s+∣s), describes this first density-ratio for unconditioned learning. In the conditioned case, this ratio changes to the conditional probability of seeing a future state given both an anchor state and preference context. This is described in Equation (11), SΔ⋆(s,s+)∝p(s+)pγ(s+∣s,Δ), such that Equations (4) and (11) are the unconditioned and conditioned analogs of one another, respectively.
Appendix A.1 derives these results via a logit function optimization problem. This starts with the unconditioned case [1], which derives the optimal logit function g∗(s,s+) in Equation (16) as equal to relation given in Equation (4). The conditioned case extends this derivation under the conditioned loss function of Equation (10) to yield the optimal conditioned logit function (g^)∗(s,s+;Δ) in Equation (17), equal to the relation from Equation (11). In this way, Equations (16) and (17) are the results of the derivations proving the analogous statements of Equations (4) and (11), respectively.
Under the Gaussian marginal assumption discussed in Appendix A.2, the density-ratio property yields the Gauss-Markov latent transitions used for inference in Section 3.2 and Appendix A.3. In the conditioned case, for a given Δ, let z=ψΔ(s) and z′=ψΔ(s+). Let pΔ(z′) denote the distribution obtained by encoding marginally sampled future states under this preference. Using the conditioned logit from Equation (6), the density-ratio relation gives
p(z′∣z,Δ)∝exp(−21∥AΔz−z′∥22)pΔ(z′).
Under the Gaussian marginal assumption pΔ(z′)=N(0,cI), this becomes
\exp\left( -\frac{1}{2}\lVert A_\Delta z-z'\rVert_2^2 -\frac{1}{2c}\lVert z'\rVert_2^2 \right).$$ Completing the square gives the conditioned version of Equation (5): $$\mathcal N\left( \frac{c}{c+1}A_\Delta z, \frac{c}{c+1}I \right).$$ Thus, the density-ratio and Gaussian marginal together produce a Gaussian transition with a mean that is linear in the current representation. This transition reflects the discounted future distribution used to sample training pairs. Under the Markov-chain model, conditioning these transitions on the encoded start and goal gives $$p(z_{1:n}\mid z_0,z_g,\Delta) \propto p(z_g\mid z_n,\Delta) \prod_{k=1}^{n} p(z_k\mid z_{k-1},\Delta),$$ such that the posterior is Gaussian. Its block tridiagonal matrix and endpoint terms are given by Equations (21) and (22), with $A_\Delta$ replacing $A$. Solving $\Lambda_\Delta\mu=\eta_\Delta$ produces the latent waypoint means. Nearest-neighbor retrieval under $\psi_\Delta$ then converts them into state-space waypoints. This derivation therefore depends directly on the optimal density-ratio encoding. In summary, and in response to the reviewer's first question, Equations (11) and (17) are equivalent. Equation (17) derives the optimal conditioned density-ratio relation, while Equation (11) states this result for the conditioned similarity scores. Equation (6) defines the exponential similarity function used to approximate this relation during training. Regarding the loss function of Equation (10), the second term uses $\Delta_i$ in the numerator and $\Delta_j$ in the denominator so that the logits are always conditioned on the preference context of the anchor state. The numerators of the log terms are diagonal terms corresponding to positive pairs, so the preference of the anchor and future are equal, as they come from the same trajectory. In the first denominator, we sum over futures indexed $j$ for a given anchor, so $\Delta_i$ of the anchor is used for conditioning. In the second denominator, we sum over anchors indexed $j$ for a given future, so $\Delta_j$ of the anchors is used for conditioning. This allows for the loss function to remain symmetrized such that Equation (17) holds for both terms in the loss function. [1] Eysenbach, B., Myers, V., Salakhutdinov, R., and Levine, S. (2025). Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inference. https://arxiv.org/abs/2403.04082