acceptodds
Under review as a conference paper at ICLR 2027

PriGo: Primitive Guidance to Diffusion and Flow Policies for Adaptive Robotic Manipulation

Abstract

Imitation learning has enabled remarkable progress in robotic manipulation, especially with diffusion- and flow-based policies that generate complex visuomotor behaviors directly from demonstrations. Despite their strong performance, these policies often struggle to generalize across tasks and environments. A key aspect that remains underexplored is the local motion structure inherent in manipulation behaviors, which is not explicitly captured by continuous action prediction. Inspired by the compositional structure of human behaviors, we propose PriGo, a primitive-guided test-time adaptive framework for robust robotic manipulation. PriGo introduces PANet, a lightweight primitive prediction module that infers primitive distributions directly from observations. We further propose a differentiable primitive guidance mechanism that refines generated actions during inference, steering trajectories toward semantically consistent behaviors. Unlike prior primitive-conditioned approaches, PriGo operates exclusively at test time and can be seamlessly integrated with pretrained diffusion- and flow-based policies without retraining the backbone policies. Extensive experiments on LIBERO, CALVIN, SIMPLER, and real-world robotic tasks demonstrate that PriGo consistently improves robustness and generalization across both diffusion- and flow-based policies.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotic Manipulation, Imitation learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hYH5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PriGo, a primitive-guided test-time refinement framework for diffusion- and flow-based robotic manipulation policies. It trains a primitive prediction network (PANet) using rule-based labels derived from expert actions and introduces a differentiable primitive classifier to guide action generation toward predicted primitive distributions. The method does not require retraining the backbone policies. Experiments on LIBERO, SIMPLER, CALVIN, and real-world manipulation tasks demonstrate consistent performance improvements.

Critical strengths
  1. PriGo provides a simple, plug-and-play approach that consistently improves multiple diffusion- and flow-based policies without retraining the backbones.

  2. Extensive experiments on LIBERO, SIMPLER, CALVIN, and real-world robotic tasks demonstrate promising improvements in robustness and generalization.

Critical weaknesses
  1. The introduction lacks a clear logical flow, and some key arguments are not well justified. For example, the paper claims that continuous action prediction fails to capture motion structures, but does not explain why diffusion/flow policies, which already model structured action trajectories, suffer from this limitation. It also criticizes existing works for limiting primitives to basic actions, while the proposed method similarly relies on only eight hand-crafted, low-level action primitives.

  2. The mathematical formulation is not clearly explained, and several notations are introduced without proper definitions. In Eq. (5), the clean-action estimation mixes quantities from different timesteps without justification. In Eqs. (5) and (6), it is unclear why gradients computed at the estimated final actions can be directly applied to intermediate states. Furthermore, the guided velocity formula following Eq. (6) appears algebraically inconsistent with the preceding update.

  3. The experimental evaluation leaves several important questions unanswered. The training and test splits for PANet are not clearly described. It appears that PANet is evaluated only on held-out trajectories, without a direct analysis of its classification accuracy on unseen tasks or environments. The relationship between PANet's prediction accuracy and downstream performance improvements, particularly under incorrect primitive predictions, is not investigated. Additionally, the real-world long-horizon experiments report subtask success rates rather than full-task completion rates.

Questions for authors
  1. Could the authors clarify why existing diffusion/flow policies lack the necessary motion structures, and how the proposed eight primitives address this limitation compared to existing primitive-based approaches?

  2. Could the authors provide more detailed explanations of Eqs. (5) and (6), particularly the clean-action estimation, the application of guidance gradients to intermediate states, and the apparent inconsistency in the guided velocity formulation?

  3. Could the authors clarify the training/test splits used for PANet? Does the evaluation include unseen tasks or environments?

  4. How does PANet's prediction accuracy affect downstream task performance? In particular, what happens when PANet predicts incorrect or conflicting primitives?

  5. Could the authors report full-task completion rates for the real-world long-horizon experiments, in addition to subtask success rates?

  6. Could the authors revise the citation formatting throughout the paper? Many references use narrative citation formats in places where parenthetical citations would be more appropriate.

4 comments
AuthorsOpenReview ↗

Response Part1

We thank the reviewer for the detailed feedback. It helped us separate what diffusion/flow policies may learn implicitly from what PriGo explicitly constrains, and identify ambiguities in the update equations and PANet evaluation. We will revise both the claims and the presentation accordingly.

W1: Motivation and Logical Flow

Response: We do not claim that diffusion and flow policies are unable to model complex, structured action trajectories. On the contrary, these models can learn rich temporal patterns from demonstrations. However, fitting the training distribution well does not guarantee that the generated actions will follow the appropriate local motion pattern under distribution shifts. This motivates PriGo to predict which motion primitive is appropriate for the current visual observation and task instruction, and use that prediction to guide action generation at test time. The goal is to improve generalization while retaining the pretrained policy’s ability to generate fine-grained continuous actions.

We agree that the eight hand-designed categories are not substantially richer than prior primitives. The novelty we claim is the observation-conditioned predictor, differentiable action-to-category relaxation, and integration with frozen diffusion and flow policies. We will rewrite the Introduction for clearer logic.

W2: Mathematical Formulation

Response: The reviewer is correct. In Eq. (5), the clean action is estimated from the current at\mathbf{a}_t; using a~t−1\tilde{\mathbf{a}}_{t-1} in the numerator is a notational error. The implementation evaluates the primitive loss at the estimated clean/endpoint action but differentiates the composed loss with respect to the current intermediate state, i.e., ∇atLPG(a^(at),O)\nabla_{\mathbf{a}_t}\mathcal L_{\mathrm{PG}}(\hat{\mathbf{a}}(\mathbf{a}_t),\mathbf{O}). Automatic differentiation therefore includes (∂a^/∂at)⊤(\partial\hat{\mathbf{a}}/\partial\mathbf{a}_t)^\top. We will replace the incorrect ∇a0\nabla_{\mathbf{a}_0} and ∇a1\nabla_{\mathbf{a}_1} notation in Eqs. (5)–(6) with ∇at\nabla_{\mathbf{a}_t}, correct the clean-action estimate, and define all variables explicitly.

For flow Euler step δ\delta, at+δ=at+δvθ−η∇atLPG(a^1(at),O)\mathbf{a}_{t+\delta}=\mathbf{a}_t+\delta\mathbf{v}_\theta-\eta\nabla_{\mathbf{a}_t}\mathcal L_{\mathrm{PG}}(\hat{\mathbf{a}}_1(\mathbf{a}_t),\mathbf{O}); hence the equivalent effective velocity is vθguided=vθ−(η/δ)∇atLPG\mathbf{v}_\theta^{\mathrm{guided}}=\mathbf{v}_\theta-(\eta/\delta)\nabla_{\mathbf{a}_t}\mathcal L_{\mathrm{PG}}. We will also distinguish this discrete effective velocity from a separately learned continuous field.

W3: Experimental Evaluation

Response: PANet is trained on 90% of the LIBERO episodes, 20 successful CogACT rollouts per SIMPLER task (10 Visual Matching and 10 Variant Aggregation), and a random 50% of CALVIN A–C episodes. Accuracy is 94% and 95% on held-out LIBERO and SIMPLER trajectories, respectively, and 89% on unseen CALVIN environment D.

We conducted a new random-primitive ablation, reported in Table R1 of our response. Replacing PANet predictions with random primitive targets reduces policy success rates from 87.8 to 80.2, whereas PANet-guided refinement improves performance to 92.4. This shows that target content matters. Online rollouts lack state-matched expert actions, so we cannot measure PANet accuracy on those visited states from the present data. We will state both the evidence and its limits.

Finally, full-task success for the three real-world tasks improves from 10% to 40%, 20% to 50%, and 10% to 30%. We will add these alongside subtask rates.

Table R1. Random-target ablation on LIBERO. Random guidance replaces PANet predictions with randomly sampled primitive targets while keeping the backbone and guidance configuration unchanged.

MethodSpatialObjectGoalLongAvg.
CogACT (Baseline)91.192.091.576.487.8
+ Random Primitive Guidance83.7 ± 1.586.2 ± 1.785.5 ± 2.265.3 ± 1.580.2 ± 1.7
+ PriGo (Ours)95.3 ± 1.097.9 ± 0.595.2 ± 0.681.1 ± 0.892.4 ± 0.6
AuthorsOpenReview ↗

Response Part2

Q1: Why Do Existing Diffusion and Flow Policies Need Explicit Motion-Structure Guidance?

Response: We do not claim that diffusion and flow policies are unable to model complex, structured action trajectories. On the contrary, these models can learn rich temporal patterns from demonstrations. However, fitting the training distribution well does not guarantee that the generated actions will follow the appropriate local motion pattern under distribution shifts. This motivates PriGo to predict which motion primitive is appropriate for the current visual observation and task instruction, and use that prediction to guide action generation at test time. The goal is to improve generalization while retaining the pretrained policy’s ability to generate fine-grained continuous actions.

Q2: Could the Authors Clarify Eqs. (5) and (6)?

Response: For diffusion, a^0\hat{\mathbf{a}}_0 is estimated from current at\mathbf{a}_t, ϵθ(O,at,t)\epsilon_\theta(\mathbf{O},\mathbf{a}_t,t), and αˉt\bar\alpha_t; a~t−1\tilde{\mathbf{a}}_{t-1} in Eq. (5) is a notation error. For flow, a^1=at+(1−t)vθ(at,O,t)\hat{\mathbf{a}}_1=\mathbf{a}_t+(1-t)\mathbf{v}_\theta(\mathbf{a}_t,\mathbf{O},t). In both cases, the implementation differentiates the composed loss with respect to at\mathbf{a}_t:

=\left(\frac{\partial\hat{\mathbf{a}}}{\partial\mathbf{a}_t}\right)^\top \nabla_{\hat{\mathbf{a}}}\mathcal L_{\mathrm{PG}}(\hat{\mathbf{a}},\mathbf{O}).$$ Thus, the clean/endpoint estimate is where the loss is evaluated, whereas the current intermediate state $\mathbf{a}_t$ is the differentiation variable; automatic differentiation includes the estimator Jacobian. The next state $\mathbf{a}_{t-1}$ for diffusion, or $\mathbf{a}_{t+\delta}$ for flow, is the output to which this guidance correction is applied, not the variable with respect to which the loss is differentiated. For flow policies, our guided Euler update is $$\mathbf{a}_{t+\delta} =\mathbf{a}_t+\delta\mathbf{v}_\theta(\mathbf{a}_t,\mathbf{O},t) -\eta\nabla_{\mathbf{a}_t}\mathcal L_{\mathrm{PG}}(\hat{\mathbf{a}}_1(\mathbf{a}_t),\mathbf{O}).$$ Rewriting this update as $\mathbf{a}_{t+\delta}=\mathbf{a}_t+\delta\mathbf{v}_\theta^{\mathrm{guided}}$ directly yields $$\mathbf{v}_\theta^{\mathrm{guided}} =\mathbf{v}_\theta-\frac{\eta}{\delta} \nabla_{\mathbf{a}_t}\mathcal L_{\mathrm{PG}}(\hat{\mathbf{a}}_1(\mathbf{a}_t),\mathbf{O}).$$ Thus, $1/\delta$ follows algebraically when the discrete correction is expressed as an effective velocity. We agree that calling it simply a “guided velocity field” was misleading: it is the effective velocity of this Euler update, not a separately learned continuous field. We will use that terminology and show the derivation adjacent to Eq. (6). ## Q3: What Are the PANet Training/Test Splits, and Are Unseen Tasks Evaluated? **Response:** For CALVIN, PANet trains on a random 50% of A–C episodes and reaches 89% on the entirely unseen environment D. For LIBERO, PANet trains on 90% of the LIBERO episodes and reaches 94% on excluded trajectories. For SIMPLER, PANet trains on 20 successful CogACT rollouts per task (10 per evaluation setting) and reaches 95% on excluded trajectories. Thus, LIBERO/SIMPLER test held-out trajectories, while CALVIN tests an unseen environment. ## Q4: How Does PANet Prediction Accuracy Affect Downstream Performance? **Response:** Replacing PANet’s predictions with random primitive targets reduces average LIBERO success from 87.8% without guidance to 80.2% with random guidance (Table R1). This shows that poorly chosen targets can hurt downstream performance. As shown in Fig. 3, PANet’s errors often occur between related categories, such as *rotation* and *push+rotation*, which share some kinematic conditions. Because PriGo uses the full predicted distribution, an incorrect top-ranked primitive can still leave probability mass on a more suitable category. When the predicted distribution assigns substantial mass to conflicting primitives, their guidance contributions can compete, potentially weakening or misdirecting the correction. Since guidance adjusts the actions proposed by the frozen backbone rather than imposing a hard primitive choice, an incorrect prediction does not necessarily lead to task failure. ## Q5: Can the Authors Report Full-Task Completion Rates? **Response:** Yes. Across the three real-world tasks, full-task success changes from 10% to 40%, 20% to 50%, and 10% to 30%, respectively. ## Q6: Could the Authors Revise the Citation Formatting? **Response:** Thank you for noting this presentation issue. We will audit all citations and use narrative citations only when the cited authors are grammatical subjects.

I appreciate the authors' clarifications and additional experiments, which address several of my initial concerns. However, the random-target ablation does not fully establish how errors in PANet predictions affect downstream performance. Overall, I am not fully convinced that the limited set of eight hand-crafted primitives adequately supports the paper's claims about local motion structure and improved generalization. My overall assessment therefore remains unchanged.

AuthorsOpenReview ↗

Follow-up Response

Response: Thank you for recognizing the clarifications and additional experiments, and for identifying what remained unresolved. We agree that the random-target ablation is an extreme intervention and does not by itself characterize sensitivity to progressively less accurate PANet targets.

We therefore add a controlled target-corruption experiment on the unseen CALVIN-D environment. With probability qq at each replanning step, we replace the current PANet prediction with a state-shuffled prediction from another timestep of the same task. The intervention preserves PANet's task-conditioned marginal output distribution while progressively removing alignment with the current state; the backbone, 1,000 evaluation sequences, guidance strength, and inference steps remain fixed.

MethodCorruption qq1 task3 tasks5 tasksAvg. Len.
3DDA (no guidance)--93.866.241.23.35
PriGo0.0098.187.659.94.18
PriGo + shuffled targets0.2590.762.836.93.23
PriGo + shuffled targets0.5086.256.331.12.93
PriGo + shuffled targets0.7580.550.925.92.66
PriGo + shuffled targets1.0072.139.620.32.22

Performance decreases monotonically with increasing corruption. Fully shuffled targets reduce five-task completion from 59.9% to 20.3% and average length by 1.96; even q=0.25q=0.25 falls below unguided 3DDA (3.23 versus 3.35). This provides a graded, causal measurement of how target misalignment affects downstream performance beyond the binary random-target comparison. We do not claim that it reproduces every naturally occurring PANet error, but it directly establishes that the benefit depends on state-aligned predictions and that increasingly inaccurate targets lead to increasingly worse outcomes.

We also agree that eight hand-crafted categories do not constitute a complete or universally optimal representation of manipulation structure. We have revised the paper to call them coarse local-motion categories rather than semantic skills and removed claims that they fully model action structure. Their role is deliberately limited: they constrain a compact set of translational, rotational, and gripper patterns while leaving exact pose, contact, object identity, rotation axis, and continuous magnitudes to the pretrained policy. Table 5 in our manuscript shows that merging directional categories or removing categories that combine translation and rotation reduces task success, supporting the value of these distinctions within PriGo. Supplementary Table 10 further shows that guiding all primitive categories achieves 92.4% success, compared with 91.0% when guiding only grasp, the best single-category variant.

Reviewer_EG9HOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
3(+2)
Read review
Paper summary

The paper presents an action-refinement framework for robot policy execution. Specifically, the authors introduce a rule-based way to discretize 7-DoF action increments into eight semantic "primitives" based on predefined thresholds (‖x‖, ‖r‖, and Δw) and the sign of x_1. A classifier network (PANet) is trained on these pseudo-labels to predict the primitive distribution from visual observations. At inference time, the discrete mapping Φ is relaxed via a sigmoid gate into a soft action distribution p(a). The gradient of the cross-entropy loss with respect to the initial estimate x_0 is then used to refine action trajectory after each denoising/integration step. Extensive evaluations across simulator benchmarks (LIBERO, SIMPLER, CALVIN) and real-world robot tasks demonstrate noticeable performance gains.

Critical strengths

PriGo requires no architectural modification or retraining of the underlying backbone policy; instead, it applies test-time gradient guidance to refined actions during sampling. This formulation naturally fits both diffusion-based and flow-based architectures. Crucially, because guidance gradients come from an analytical soft classifier rather than backpropagating through the policy network, deployment overhead remains minimal.

Critical weaknesses

1 The Most Critical Weakness: Experimental evidence cannot attribute performance gains to the “primitive alignment” mechanism. All three benchmarks lack random primitive targets and gradient-free baselines, making it impossible to isolate the claimed mechanism. SIMPLER has an undisclosed data advantage for PANet. LIBERO is in-distribution only, with gains within statistical noise and no standard deviations. CALVIN’s 53.3%→80.4% jump is unexplained. The experiments only prove that adding PANet raises success rate — not that primitive alignment is the cause.

2 The Guidance Mechanism’s Actual Capabilities Are Far Narrower Than Its Semantic Claims. Definition 1 classifies actions by amplitude thresholds and the sign of an undefined x₁ — no contacts, objects, or temporal structure. These are action discretizations, not semantic primitives, as the authors admit in L185–188, yet the paper claims “semantically coherent” behavior throughout. Since L_PG depends only on ‖x‖, x₁, ‖r‖, and Δw, guidance cannot alter rotation axis, y/z translation ratios, or grasp location. The paper’s own case study attributes a grasp-location error to primitive misalignment — impossible under Definition 1, where both grasps share the same primitive.

Questions for authors

Major questions: 1 The gradient guidance in Eq. 5–6 coincides with classifier guidance and MPGD’s x̂₀ shortcut, yet neither is cited. What is the precise technical novelty beyond the primitive taxonomy and PANet? 2 Inference-time filtering methods such as RoboMonkey are direct alternatives to PriGo but are neither discussed nor benchmarked. Why are these omitted from the related work and experimental comparison? 3 The thresholds τ_trans, τ_rot, τ_w, β, and τ are never specified anywhere, making results unreproducible. Why are these values absent, and how can the reported η range in Table 6 generalize across configurations? 4 Since each primitive region is defined by simple thresholds, closed-form projection is directly applicable without softening. Why is this gradient-free alternative not compared, and how is the unequal guidance strength across categories — caused by varying geometric-mean roots — addressed? 5 L_PG depends on rotation only through ‖r‖, so any equal-magnitude rotation yields zero gradient, violating the uniqueness assumption required by Prop. 1. How does this proposition apply to the actual multi-step denoising algorithm, where corrections are applied to noisy intermediate states rather than a fixed point?

Minor questions:

  1. Appendix B (L818) states that PANet uses "20 demonstrations per task from SIMPLER," yet SIMPLER itself does not provide training demonstrations. Please clarify: Were these demonstrations collected from SIMPLER simulations, or were they derived from real-world Fractal data? Do they originate from the "Visual Matching" or "Variant Aggregation"? The results for CogACT in Table 2 are cited directly from the original paper (L348); were they re-run using the same evaluation environment and random seeds as those used for PriGo?

  2. All key parameters of the guidance module. Please provide the values of: threshold τtrans, τrot, τ_w (L172), smoothing parameter β (L203), and temperature τ (L220); Which axis of which coordinate frame does x₁ denote?

  3. Eq. 4 requires ŷᵏ = PANet(Oᵏ), but future observations cannot be obtained during inference. CogACT outputs 16 steps at a time and π0 uses a chunk length of 50 in its original paper. How is the ŷ value determined for each step within the action chuck? Which wᵏ⁻¹ does the header's Δw depend on?

  4. Also using 3DDA as the backbone, ADPro brings +0.29, PriGo brings +0.83, and the success rate of the fourth task rises from 53.3% to 80.4%. Please report: PANet’s classification accuracy on unseen environment D; The number of random seeds and sequences used in the evaluation; Classification statistics of failure modes, indicating which types of errors were specifically corrected by the improvement. In addition, the three rows of numbers for DP3, 3DDA, and ADPro in the table are consistent with the original text of ADPro. Please indicate the source.

  5. The training data for PANet listed in Appendix B does not include data from real-world hardware. What data is used to train PANet for deployment on real-world hardware? What is the accuracy? Have the thresholds been readjusted to account for the action scales encountered in the real-world environment?

  6. The text (L881–886) refers to " grasps the inner wall rather than the handle" as a "primitive-level error," but according to Def. 1, both belong to grasp, the only difference being the grasping position. ∇L_PG can only change ‖x‖, x₁, ‖r‖ and Δw. Please explain how PG corrects this type of error and provide quantitative statistics on the proportion of each type of failure.

  7. Eq. 5 estimates â₀ from the already-updated ã{t−1} while using εθ(a_t, t) and ᾱt, whereas the flow counterpart (Eq. 6) correctly starts from a_t, and L251 states that â₀ is obtained “through the pre-defined schedule”, which holds only for a_t. Which variable does the code actually use?

  8. Keeping all other settings constant, report the following—at least for LIBERO (CogACT) and CALVIN: (a) Results using primitive targets that have been randomized or shuffled; (b) Results using the Φ (phi) values derived from demonstration actions as "oracle" targets; (c) Performance curves obtained after artificially reducing the accuracy of PANet. If (a) also leads to a comparable improvement, the "primitive alignment" explanation breaks down; if (b) is not significantly better than PANet, the gain is independent of the quality of the primitive predictions.

  9. Please compare them with the following approaches: Closed-form projection: rescaling the translation and rotation magnitudes along their current directions, flipping the sign of x₁, and adjusting the gripper. If "softening" combined with CE-guidance does not offer a clear advantage over this approach, then Def. 2 is not a necessary design.

  10. PANet considers only the current single frame, making it unable to distinguish between stages that depend on temporal sequence—such as the states "before turning the key" and "after turning the key". One could consider incorporating historical observations or defining primitives that span multiple time steps.

  11. One could explore integrating primitive prediction with end-to-end policy training—for instance, by using it as an auxiliary prediction head—and compare this approach directly against "inference-time guidance" methods.

7 comments
AuthorsOpenReview ↗

Response Part1

We sincerely thank the reviewer for the detailed technical assessment. It exposed important gaps in causal attribution, claim scope, related work, and reproducibility. We address each point directly below and will revise the paper so that the claims match the evidence.

W1: Causal Attribution of Performance Improvements

Response: We agree that the original experiments did not isolate primitive alignment. We therefore added two controls on LIBERO with the same CogACT backbone: random targets reduce average success from 87.8 to 80.2, whereas PANet targets raise it to 92.4 (see Table R1); closed-form projection reaches 90.0 (see Table R2). These results show that arbitrary guidance is not sufficient and that the differentiable method outperforms one hard alternative.

PANet was trained on 90% of the LIBERO episodes, 20 episodes per SIMPLER task, and a 50% subset of CALVIN episodes in environments A, B, and C. We re-evaluated the policy on LIBERO and report the standard deviations in Table R1.

The increase from 53.3% to 80.4% measures completion of the first four subtasks in sequence, not success on the fourth subtask in isolation. Success on the first two subtasks also rose from 80.3% to 92.1%. Since only rollouts that complete the earlier subtasks can succeed on the full sequence, early gains can carry forward into later completion rates. This pattern is consistent with our motivation that primitive guidance can be particularly useful in long-horizon tasks.

Table R1. Random-target ablation on LIBERO. Random guidance replaces PANet predictions with randomly sampled primitive targets while keeping the backbone and guidance configuration unchanged.

MethodSpatialObjectGoalLongAvg.
CogACT (Baseline)91.192.091.576.487.8
+ Random Primitive Guidance83.7 ± 1.586.2 ± 1.785.5 ± 2.265.3 ± 1.580.2 ± 1.7
+ PriGo (Ours)95.3 ± 1.097.9 ± 0.595.2 ± 0.681.1 ± 0.892.4 ± 0.6

W2: Representational Limitations and Semantic Claims

Response: We agree that the training labels of our eight primitives are derived from low-level kinematic properties. However, we would like to distinguish between how these labels are constructed and how they are predicted. Although the labels are automatically generated from action kinematics, PANet predicts primitive distributions directly from visual observations and language instructions, without access to ground-truth actions at inference time. Therefore, PANet must learn task-relevant associations between visual-language context and local manipulation behaviors, rather than simply applying predefined kinematic thresholds to observed actions.

We agree that grasping the cup’s inner wall rather than its handle is a spatial execution error. Both actions can receive the same grasp label under Definition 1, so the observed failure does not necessarily result from primitive misalignment at that moment. It may instead be a downstream consequence of incorrect primitive choices at preceding steps, because earlier motion corrections can affect the robot’s approach and eventual grasp. Thus, the qualitative outcome may reflect the downstream effect of earlier primitive alignment rather than a direct correction of grasp location.

Major Q1: Novelty Beyond Classifier Guidance and MPGD

Response: We agree that generic classifier guidance is not our technical novelty. We will add the missing classifier-guidance and MPGD citations. Our contribution is the combination of an observation-conditioned local-motion predictor, a differentiable continuous-action-to-category relaxation, and a compatibility objective integrated into frozen robotic diffusion and flow backbones. We will state this precisely and remove any wording that implies a new generic guidance algorithm.

Major Q2: Comparison With RoboMonkey

Response: Thank you for pointing it out. RoboMonkey is a relevant inference-time alternative. It samples and verifies candidate actions, whereas PriGo modifies each generative trajectory with a differentiable category-consistency loss; the approaches are potentially complementary. We will add it to related work and clarify the distinction between sampling-and-verification and gradient-based action refinement.

Major Q3: Missing Hyperparameters and Reproducibility

Response: We used τtrans=τrot=τw=0.1\tau_{\mathrm{trans}}=\tau_{\mathrm{rot}}=\tau_w=0.1, smoothing β=0.1\beta=0.1, temperature τ=1\tau=1, and guidance strength η=0.002\eta=0.002 in all reported experiments. PriGo uses the same primitive definitions and loss settings across configurations. With comparable action scales, similar deviations can yield corrections of comparable magnitude. We selected η=0.002\eta=0.002 on LIBERO and used it unchanged on the other benchmarks, where PriGo improved performance without retuning η\eta. We do not claim that this value is optimal for every benchmark or policy architecture.

AuthorsOpenReview ↗

Response Part2

Major Q4: Closed-Form Projection and Unequal Category Gradients

Response: Thank you for suggesting this strong control. On LIBERO, with the same backbone, projection improves the average from 87.8 to 90.0, while PriGo reaches 92.4 (Table R2). This supports a benefit for soft guidance in this setting.

The geometric mean reduces this imbalance: taking the root makes the log compatibility score the average, rather than the sum, of the log gate scores. This removes systematic scaling with the number of conjunctive conditions. Gradient magnitudes can still differ across categories depending on how close each gate is to its threshold, because saturated gates contribute vanishing gradients. Furthermore, we set the temperature parameter τ=1\tau=1 in Eq. (2) across all experiments, so the primitive distribution is obtained by directly normalizing the compatibility scores.

Major Q5: Applicability of Proposition 1

Response: We agree that the primitive loss cannot distinguish rotation increments with the same magnitude but different axes. Equal magnitudes imply equal loss values, but the radial gradient can still be nonzero. If the reference action is stationary and has a nonzero rotation increment, changing only its rotation axis yields close stationary actions. Moreover, since any rotation of the same magnitude yields a zero gradient, the conclusion of Prop. 1 still holds. During denoising, precisely because we operate on noisy intermediate actions, the set of rotations with the same magnitude is essentially of measure zero. Therefore, our guidance is effective overall.

Minor Q1: Source of SIMPLER Demonstrations and Evaluation Protocol

Response: Thank you for catching this ambiguity. The 20 trajectories per task were successful CogACT rollouts that we collected in SIMPLER: 10 under Visual Matching and 10 under Variant Aggregation. The unguided CogACT numbers in Table 2 of the manuscript were copied from the CogACT paper. We evaluated only CogACT with PriGo, following the evaluation protocol of CogACT and using SimplerEnv’s default random seed.

Minor Q2: Guidance Hyperparameters and Coordinate Frame

Response: We used τtrans=τrot=τw=0.1\tau_{\mathrm{trans}}=\tau_{\mathrm{rot}}=\tau_w=0.1, smoothing β=0.1\beta=0.1, temperature τ=1\tau=1, and guidance strength η=0.002\eta=0.002 in all reported experiments. Actions are expressed in the robot-base frame; x1x_1 is the first translational component along the robot-base x-axis.

Minor Q3: Action-Chunk Alignment and Gripper Differences

Response: PANet uses the current observation and instruction to produce one soft target, shared by all steps of the predicted chunk; no future observation is used. A new target is predicted when the policy replans. For Δwk=wk−wk−1\Delta w^k=w^k-w^{k-1}, the first step uses the last executed gripper command before the new chunk, and later steps use the preceding command in that chunk. We will revise Eq. (4) to show both conventions explicitly.

Minor Q4: CALVIN Results, Unseen-Environment Accuracy, and Baseline Sources

Response: PANet achieves 89% primitive classification accuracy on unseen CALVIN environment D. Under the official evaluation protocol, trajectories are generated by rolling out the policy online in simulation. Although we can infer primitive types from the executed actions, we do not have ground-truth action or primitive labels. We therefore cannot directly quantify primitive correctness during testing and its effect on the policy's performance. Our policy result uses seed 0 and 1,000 task sequences, following 3DDA's and ADPro's default protocol. The DP3, 3DDA, and ADPro rows were copied from the ADPro paper; only PriGo was run by us. The 53.3% to 80.4% fourth-task result should therefore be treated as an aggregate improvement.

Minor Q5: Real-World PANet Training Data and Action Normalization

Response: For hardware deployment, we fine-tuned PANet on the same 50 real-robot demonstrations used for the backbone; PANet achieves 92% classification accuracy on the real-robot data. We retained the simulation thresholds without hardware-specific retuning.

Minor Q6: Grasp-Location Errors and Primitive Guidance

Response: We agree that grasping the cup’s inner wall rather than its handle is a spatial execution error. Both actions can receive the same grasp label under Definition 1, so the observed failure does not necessarily result from primitive misalignment at that moment. It may instead be a downstream consequence of incorrect primitive choices at preceding steps, because earlier motion corrections can affect the robot’s approach and eventual grasp. Thus, the qualitative outcome may reflect the downstream effect of earlier primitive alignment rather than a direct correction of grasp location.

AuthorsOpenReview ↗

Response Part3

Minor Q7: Clean-Action Estimation in Eq. (5)

Response: The code uses the current state at\mathbf{a}_t. The a~t−1\tilde{\mathbf{a}}_{t-1} numerator in Eq. (5) is a notation error. The corrected estimate is a^0=(at−1−αˉt ϵθ(O,at,t))/αˉt\hat{\mathbf{a}}_0=(\mathbf{a}_t-\sqrt{1-\bar\alpha_t}\,\epsilon_\theta(\mathbf{O},\mathbf{a}_t,t))/\sqrt{\bar\alpha_t}, consistent with estimating the flow endpoint from its current at\mathbf{a}_t. We will correct the equation and surrounding description.

Minor Q8: Randomized, Oracle, and Corrupted Primitive Targets

Response: These are valuable tests. On LIBERO, random targets reduce average success from 87.8 to 80.2, while PANet targets reach 92.4, showing that arbitrary guidance does not explain the gain. An online oracle is unavailable from the existing demonstrations: policy rollouts visit states that generally have no paired expert action, so applying Φ\Phi to demonstration actions would not yield state-matched targets.

Minor Q9: Comparison With Closed-Form Projection

Response: We added a closed-form projection baseline on LIBERO using the same pretrained CogACT backbone (Table R2). Projection improves average success from 87.8% to 90.0%, while PriGo reaches 92.4%, outperforming projection on all four suites. The largest gain over projection is 4.0 percentage points on LIBERO-Long. These results support the practical benefit of soft cross-entropy guidance over hard projection in this setting.

Table R2. Closed-form primitive projection versus differentiable primitive guidance on LIBERO. All methods use the same pretrained backbone.

MethodSpatialObjectGoalLongAvg.
CogACT (Baseline)91.192.091.576.487.8
+ Closed-form Projection93.6 ± 1.196.3 ± 0.993.0 ± 0.977.1 ± 1.390.0 ± 1.0
+ PriGo (Ours)95.3 ± 1.097.9 ± 0.595.2 ± 0.681.1 ± 0.892.4 ± 0.6

Minor Q10: Lack of Temporal History in PANet

Response: We agree. A single frame cannot, in general, disambiguate visually similar states with different histories, such as before versus after turning a key. The language instruction and current image may help in some cases, but they do not solve partial observability. The present method shares one prediction across a chunk only to encourage local consistency. We will add this limitation and identify observation history or temporally extended primitives as the natural extension.

Minor Q11: End-to-End Primitive Prediction Versus Inference-Time Guidance

Response: Thank you. End-to-end auxiliary prediction updates the policy parameters during training, whereas PriGo updates generated actions at inference with a frozen backbone. This distinction motivates our plug-in setting but does not make the end-to-end baseline irrelevant. We will frame it as a complementary alternative and avoid implying that frozen-backbone convenience establishes better effectiveness.

Reviewer_PteaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adds a guidance mechanism to the inference process of Diffusion Policies and Flow Policies for robot manipulation.

The authors first use some hand-designed thresholds on translation, rotation, gripper changes, and the sign of one translation component to divide actions from expert demonstrations into eight primitive categories. They then train a vision-language classifier, PANet, to predict the probabilities of these primitives from the current observation and language instruction.

At test time, the authors turn the original rule-based classification into a differentiable soft version. This lets them map a generated continuous action back to a primitive distribution. They then compute a cross-entropy loss between this distribution and the one predicted by PANet, and use its gradient to guide diffusion denoising or flow integration. The pretrained policy backbone stays frozen.

The method is evaluated on LIBERO, SIMPLER, CALVIN, and three real-robot tableware tasks. On LIBERO, the four backbones gain 2.1–4.5 percentage points in average success rate. On SIMPLER, CogACT improves by 5.1 and 7.5 points in the two settings. On CALVIN, the average number of consecutively completed tasks goes from 3.35 to 4.18 compared with 3DDA. The real-robot experiments report 22–29-point improvements in subtask completion rate. The paper also studies different primitive sets and guidance strengths. The appendix reports roughly 15% extra inference latency.

Critical strengths

1. Evaluation across different policies and settings

The authors evaluate PriGo on LIBERO, SIMPLER, CALVIN, and real-robot tasks, with both diffusion- and flow-based policies. I think this is a strong point of the work.

Most importantly, the gains are not limited to just one backbone. The improvements on CALVIN's long-horizon tasks and on real-robot subtask completion are also fairly noticeable. So I do think the experiments support the claim that this extra guidance can be useful in practice.

2. Easy to add to an existing policy

Another thing I like is that the method can be added at inference time without modifying or retraining the policy backbone. Since the guidance is defined in continuous action space, the same general idea can be used with both DP and Flow Policy. That is attractive if a policy has already been trained and is expensive to retrain.

The authors also report the extra inference cost. For CogACT, latency goes from 181 ms to 208 ms, about a 15% increase. PANet itself still needs separate training, so this is not a training-free method overall, but the cost of attaching it to a pretrained policy seems fairly manageable.

Critical weaknesses

1. I am not fully convinced that the discrete primitive representation is needed

This is probably my main concern with the paper.

The authors use eight primitives, but when I look at their actual definitions, they are mostly based on manually selected thresholds for translation, rotation, and gripper movement. For example, Push and Pull are separated by the sign of one translation component. This makes me wonder whether these are really semantic manipulation primitives, or just labels for a few low-level motion patterns.

To be clear, there is nothing wrong with using simple motion patterns as a prior. They may well be useful. But a label such as Push does not necessarily mean that the robot is pushing an object. The definition does not check object contact or the manipulation goal. It also does not explicitly capture prerequisites or the order in which different skills should happen.

I also kept coming back to the way the guidance is constructed. The method starts with continuous actions, turns them into eight discrete labels, uses smooth gates to turn those labels back into a differentiable distribution, and finally uses the gradient to update continuous actions. I understand how this works mathematically, but I am not yet sure what the intermediate classification step buys us.

For instance, why not predict translation direction, rotation magnitude, or gripper movement directly from the observation and guide the action using those quantities? A simpler continuous kinematic loss seems like it could provide a similar correction. Maybe the discrete categories make prediction more stable or help generalization, but that is exactly what I would want the paper to demonstrate.

The current experiments do not include a direct comparison with continuous kinematic guidance, simple action-smoothness guidance, or comparable action-level regularization under a similar compute budget. Because of this, I find it hard to tell how much of the improvement comes from the primitive representation itself, rather than from adding a second prediction model and a gradient correction. This also leaves the methodological novelty somewhat unclear to me.

2. The gradient updates and the theoretical result need more clarification

I had a separate concern when reading Equations (5) and (6). The authors estimate a noise-free or final action, compute the primitive loss at that estimate, and use the gradient to change an intermediate diffusion or flow state.

What I could not tell from the equations is whether they backpropagate through the full mapping from the intermediate state to the estimated final action. In general, that mapping has a Jacobian. Or do they compute the gradient in final-action space and directly apply it to the intermediate state as an approximation? Either might be an implementation choice, but I think it should be stated clearly. If it is an approximation, I would also like to understand why it is reasonable here.

I looked at the error bound in Appendix A as well. The proof assumes strong monotonicity and Lipschitz continuity of the primitive-guidance gradient field. Under those assumptions, the bound itself makes sense. What is missing for me is an argument that the particular primitive loss used in this paper actually satisfies them.

The loss only uses fairly coarse kinematic features. For example, two rotations around different axes could have the same rotation magnitude, and thus the same primitive scores and loss. That makes it hard to see how this loss, on its own, would necessarily pull an action toward one particular correct expert action.

So I would not say the theorem is wrong. Rather, I think its practical meaning is limited unless the authors justify the assumptions for their actual objective. As written, it does not directly establish that the proposed update improves action accuracy or semantic consistency.

Questions for authors

1. Could the authors compare discrete primitive guidance with direct continuous kinematic guidance?

This is the experiment I would most like to see. Using the same backbone and roughly the same inference budget, could the authors predict continuous features such as translation direction, rotation magnitude, or gripper changes, and use those directly for guidance? A simple action-smoothness or kinematic-regularization baseline would also help.

Since the primitive labels are derived from these features in the first place, I think this comparison is important. What does discretizing them into eight classes give us that a direct continuous objective does not? If the simpler approaches work just as well, it would change how I view the main contribution.

2. How exactly is the primitive prediction aligned with an action chunk?

Appendix B says PANet predicts the primitive of the next action, while Figure 2 shows a guidance loss summed over actions in a predicted chunk. I may be missing an implementation detail here, but I could not figure out how those two descriptions fit together.

Is the same eight-way distribution applied to every action in the chunk? Does PANet predict a different distribution for each future action? Or is only the next action to be executed guided?

This matters because one chunk can contain several motion patterns, such as approaching, grasping, and lifting an object. The paper also emphasizes smooth transitions between primitives, so I think it would help to explain how such transitions are handled within a chunk.

3. Could the authors clarify the gradient calculation and give more direct evidence of improved action structure?

For Equations (5) and (6), is the primitive gradient backpropagated through the complete final-action estimation map, or is it applied directly to the intermediate state? If the latter, how much does that approximation matter in practice?

Also, is there evidence that the strong-monotonicity assumption in Appendix A holds for the actual primitive loss, even locally?

Finally, I would be interested in metrics that are closer to the motivation of the paper: for example, correctness of primitive transitions, frequency of invalid action ordering, or trajectory smoothness. I realize task success rate is the ultimate metric, but it does not tell us why performance improves. It could be due to more stable local actions, even without learning a better sequential primitive structure.

4 comments
AuthorsOpenReview ↗

Response Part1

We thank the reviewer for the constructive feedback, which helped us clarify what PriGo’s primitives capture and what our experiments show about their role in guidance.

W1: Necessity and Novelty

Response: We agree with the reviewer that the eight labels are automatically derived from translation, rotation, and gripper kinematics; they do not encode contact or object identity. We will therefore replace claims of “semantic skills” with “observation-conditioned local motion categories/consistency.”

The labels nevertheless serve a different role from applying thresholds to a known action. During inference, PANet receives only the current image and language instruction and predicts a distribution over local motion categories. This requires observation-conditioned visual-language reasoning rather than simply applying kinematic thresholds to known actions.

An important consideration for continuous kinematic guidance is how to obtain reliable targets at inference time. If precise full-dimensional action targets were already available, the need for a separate generative policy would be substantially reduced.

PriGo deliberately separates motion-type prediction from fine-grained continuous action generation. PANet predicts a compact distribution over local motion primitives, while the pretrained diffusion or flow policy determines the precise action parameters.

Our random-primitive ablation in Table R1 further demonstrates that meaningful primitive predictions are important for downstream performance. We acknowledge that a matched-budget continuous guidance baseline would help quantify the specific advantage of this representation, but our central contribution is a practical, differentiable primitive-guidance framework that improves pretrained generative policies without backbone retraining.

W2: Theoretical Justification and Gradient Updates

Response: The reviewer is correct that the original notation was ambiguous. In the implementation, the primitive loss is evaluated at the estimated clean/endpoint action, but its gradient is taken with respect to the current intermediate state:

=\left(\frac{\partial\hat{\mathbf{a}}}{\partial\mathbf{a}_t}\right)^\top \nabla_{\hat{\mathbf{a}}}\mathcal L_{\mathrm{PG}}(\hat{\mathbf{a}},\mathbf{O}).$$ Thus, gradients are propagated through the clean-action estimator, and automatic differentiation includes its Jacobian. The correct subscript is $\mathbf{a}_t$, because this is the input variable from which the clean/endpoint estimate is computed; $\tilde{\mathbf{a}}_{t-1}$ is the subsequent state to which the resulting guidance correction is applied. The correction $\mathbf{a}_{t-1}=\tilde{\mathbf{a}}_{t-1}-\eta\mathbf{g}_t$ for diffusion, and its flow analogue, therefore uses a guidance gradient defined at the current diffusion/flow state rather than directly reusing $\nabla_{\hat{\mathbf{a}}}\mathcal L_{\mathrm{PG}}$. We will correct Eqs. (5)–(6) and make both the differentiation variable and the dependency $\hat{\mathbf{a}}(\mathbf{a}_t)$ explicit. We also agree that Proposition 1 is conditional. Its strong-monotonicity and Lipschitz assumptions are not automatically satisfied by the proposed primitive loss. We will clarify this scope and avoid interpreting the bound as a general proof of improved action accuracy. ## Q1: Why Are Discrete Primitives Necessary Compared With Continuous Kinematic Guidance? **Response:** A categorical target is attractive because it constrains only a coarse local behavior and leaves magnitudes, axes, and spatial targets to the pretrained policy; a continuous target predictor would have to estimate these values accurately at inference time. If precise continuous action targets were already available, the need for additional action refinement would be substantially reduced. We would like to emphasize that the contribution of our eight-class representation is not merely to introduce an additional guidance loss, but to provide interpretable, motion-level constraints without prescribing precise continuous actions. Unlike direct continuous kinematic guidance, which requires predicting specific motion values, PriGo only needs to infer the appropriate motion category from visual-language observations, while leaving fine-grained action parameters to the pretrained policy. The primitive-level ablation shows that removing directional or combined primitives degrades performance, while our additional random-primitive experiment (Table R1) demonstrates that arbitrary guidance targets cannot reproduce the improvements achieved by PANet-guided refinement. Together, these results indicate that the effectiveness of PriGo depends on meaningful, structured primitive information rather than simply adding an arbitrary gradient correction.
AuthorsOpenReview ↗

Response Part2

Table R1. Random-target ablation on LIBERO. Random guidance replaces PANet predictions with randomly sampled primitive targets while keeping the backbone and guidance configuration unchanged.

MethodSpatialObjectGoalLongAvg.
CogACT (Baseline)91.192.091.576.487.8
+ Random Primitive Guidance83.7 ± 1.586.2 ± 1.785.5 ± 2.265.3 ± 1.580.2 ± 1.7
+ PriGo (Ours)95.3 ± 1.097.9 ± 0.595.2 ± 0.681.1 ± 0.892.4 ± 0.6

Q2: How Are Primitive Targets Temporally Aligned With Action Chunks?

Response: Thank you for identifying this ambiguity. In our implementation, PANet predicts a single soft primitive distribution conditioned on the current visual observation and language instruction, and the same distribution is applied to all predicted actions within the corresponding action chunk. PANet does not independently predict a separate primitive distribution for each future action.

This design serves two purposes. First, applying a shared primitive distribution within each action chunk encourages all generated actions to follow a consistent local motion pattern, reducing potentially conflicting primitive preferences within the chunk. Second, as the policy receives new observations and generates subsequent action chunks, PANet updates its predicted primitive distribution accordingly. Since PANet outputs a probability distribution rather than a hard primitive label, neighboring chunks can assign non-negligible probabilities to multiple related primitives during transitions, allowing the guidance objective to change gradually instead of imposing abrupt categorical switches. Therefore, PriGo promotes intra-chunk primitive consistency and can reduce inter-chunk abrupt changes through shared, probabilistic primitive guidance.

Q3: What Supports the Gradient Updates, Theoretical Assumptions, and Structural Improvements?

Response: The implementation computes ∇atLPG(a^(at),O)\nabla_{\mathbf{a}_t}\mathcal L_{\mathrm{PG}}(\hat{\mathbf{a}}(\mathbf{a}_t),\mathbf{O}), so gradients are propagated through the estimated clean/endpoint-action mapping and its Jacobian is included by automatic differentiation. The notation ∇a0\nabla_{\mathbf{a}_0} and ∇a1\nabla_{\mathbf{a}_1} in Eqs. (5)–(6) is therefore incorrect and will be replaced by ∇at\nabla_{\mathbf{a}_t}. The subscript is at\mathbf{a}_t, rather than at−1\mathbf{a}_{t-1}, because the loss is differentiated through the estimate constructed from the current state at\mathbf{a}_t; at−1\mathbf{a}_{t-1} is the output of the guided update. Proposition 1 remains conditional; it does not establish its assumptions for the actual loss. We will revise the theoretical analysis to clarify these conditions and the scope of the resulting guarantee.

For evidence, the new LIBERO control gives 87.8 baseline, 80.2 random-target guidance, and 92.4 PANet guidance. This supports the narrower conclusion that target content matters. Under our current evaluation protocol, trajectories are generated by rolling out the policy online in simulation. Although we can infer primitive types from the executed actions, we do not have ground-truth actions or labels for which primitive should be executed at each visited state or in what order. We therefore cannot directly quantify primitive-transition correctness or identify invalid action ordering from the existing rollouts alone.

I thank the authors for the response and most of my concerns are addressed. I am raising the score to 3.

Thank you for taking the time to read our responses and update your recommendation. We are glad we could address your concerns, and we are grateful for the care you put into your review. Your feedback helped us make the manuscript clearer and stronger.

Comments on the paper

1 comment

We sincerely thank the reviewers for their constructive comments and valuable suggestions. We have revised the manuscript accordingly, and all changes are highlighted in blue in the revised PDF. For further and more specific changes made, please refer to the detailed responses to the reviewers below as well as the duly revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.