- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PriGo, a primitive-guided test-time refinement framework for diffusion- and flow-based robotic manipulation policies. It trains a primitive prediction network (PANet) using rule-based labels derived from expert actions and introduces a differentiable primitive classifier to guide action generation toward predicted primitive distributions. The method does not require retraining the backbone policies. Experiments on LIBERO, SIMPLER, CALVIN, and real-world manipulation tasks demonstrate consistent performance improvements.
Critical strengths
-
PriGo provides a simple, plug-and-play approach that consistently improves multiple diffusion- and flow-based policies without retraining the backbones.
-
Extensive experiments on LIBERO, SIMPLER, CALVIN, and real-world robotic tasks demonstrate promising improvements in robustness and generalization.
Critical weaknesses
-
The introduction lacks a clear logical flow, and some key arguments are not well justified. For example, the paper claims that continuous action prediction fails to capture motion structures, but does not explain why diffusion/flow policies, which already model structured action trajectories, suffer from this limitation. It also criticizes existing works for limiting primitives to basic actions, while the proposed method similarly relies on only eight hand-crafted, low-level action primitives.
-
The mathematical formulation is not clearly explained, and several notations are introduced without proper definitions. In Eq. (5), the clean-action estimation mixes quantities from different timesteps without justification. In Eqs. (5) and (6), it is unclear why gradients computed at the estimated final actions can be directly applied to intermediate states. Furthermore, the guided velocity formula following Eq. (6) appears algebraically inconsistent with the preceding update.
-
The experimental evaluation leaves several important questions unanswered. The training and test splits for PANet are not clearly described. It appears that PANet is evaluated only on held-out trajectories, without a direct analysis of its classification accuracy on unseen tasks or environments. The relationship between PANet's prediction accuracy and downstream performance improvements, particularly under incorrect primitive predictions, is not investigated. Additionally, the real-world long-horizon experiments report subtask success rates rather than full-task completion rates.
Questions for authors
-
Could the authors clarify why existing diffusion/flow policies lack the necessary motion structures, and how the proposed eight primitives address this limitation compared to existing primitive-based approaches?
-
Could the authors provide more detailed explanations of Eqs. (5) and (6), particularly the clean-action estimation, the application of guidance gradients to intermediate states, and the apparent inconsistency in the guided velocity formulation?
-
Could the authors clarify the training/test splits used for PANet? Does the evaluation include unseen tasks or environments?
-
How does PANet's prediction accuracy affect downstream task performance? In particular, what happens when PANet predicts incorrect or conflicting primitives?
-
Could the authors report full-task completion rates for the real-world long-horizon experiments, in addition to subtask success rates?
-
Could the authors revise the citation formatting throughout the paper? Many references use narrative citation formats in places where parenthetical citations would be more appropriate.
4 comments
Response Part2
Q1: Why Do Existing Diffusion and Flow Policies Need Explicit Motion-Structure Guidance?
Response: We do not claim that diffusion and flow policies are unable to model complex, structured action trajectories. On the contrary, these models can learn rich temporal patterns from demonstrations. However, fitting the training distribution well does not guarantee that the generated actions will follow the appropriate local motion pattern under distribution shifts. This motivates PriGo to predict which motion primitive is appropriate for the current visual observation and task instruction, and use that prediction to guide action generation at test time. The goal is to improve generalization while retaining the pretrained policy’s ability to generate fine-grained continuous actions.
Q2: Could the Authors Clarify Eqs. (5) and (6)?
Response: For diffusion, is estimated from current , , and ; in Eq. (5) is a notation error. For flow, . In both cases, the implementation differentiates the composed loss with respect to :
=\left(\frac{\partial\hat{\mathbf{a}}}{\partial\mathbf{a}_t}\right)^\top \nabla_{\hat{\mathbf{a}}}\mathcal L_{\mathrm{PG}}(\hat{\mathbf{a}},\mathbf{O}).$$ Thus, the clean/endpoint estimate is where the loss is evaluated, whereas the current intermediate state $\mathbf{a}_t$ is the differentiation variable; automatic differentiation includes the estimator Jacobian. The next state $\mathbf{a}_{t-1}$ for diffusion, or $\mathbf{a}_{t+\delta}$ for flow, is the output to which this guidance correction is applied, not the variable with respect to which the loss is differentiated. For flow policies, our guided Euler update is $$\mathbf{a}_{t+\delta} =\mathbf{a}_t+\delta\mathbf{v}_\theta(\mathbf{a}_t,\mathbf{O},t) -\eta\nabla_{\mathbf{a}_t}\mathcal L_{\mathrm{PG}}(\hat{\mathbf{a}}_1(\mathbf{a}_t),\mathbf{O}).$$ Rewriting this update as $\mathbf{a}_{t+\delta}=\mathbf{a}_t+\delta\mathbf{v}_\theta^{\mathrm{guided}}$ directly yields $$\mathbf{v}_\theta^{\mathrm{guided}} =\mathbf{v}_\theta-\frac{\eta}{\delta} \nabla_{\mathbf{a}_t}\mathcal L_{\mathrm{PG}}(\hat{\mathbf{a}}_1(\mathbf{a}_t),\mathbf{O}).$$ Thus, $1/\delta$ follows algebraically when the discrete correction is expressed as an effective velocity. We agree that calling it simply a “guided velocity field” was misleading: it is the effective velocity of this Euler update, not a separately learned continuous field. We will use that terminology and show the derivation adjacent to Eq. (6). ## Q3: What Are the PANet Training/Test Splits, and Are Unseen Tasks Evaluated? **Response:** For CALVIN, PANet trains on a random 50% of A–C episodes and reaches 89% on the entirely unseen environment D. For LIBERO, PANet trains on 90% of the LIBERO episodes and reaches 94% on excluded trajectories. For SIMPLER, PANet trains on 20 successful CogACT rollouts per task (10 per evaluation setting) and reaches 95% on excluded trajectories. Thus, LIBERO/SIMPLER test held-out trajectories, while CALVIN tests an unseen environment. ## Q4: How Does PANet Prediction Accuracy Affect Downstream Performance? **Response:** Replacing PANet’s predictions with random primitive targets reduces average LIBERO success from 87.8% without guidance to 80.2% with random guidance (Table R1). This shows that poorly chosen targets can hurt downstream performance. As shown in Fig. 3, PANet’s errors often occur between related categories, such as *rotation* and *push+rotation*, which share some kinematic conditions. Because PriGo uses the full predicted distribution, an incorrect top-ranked primitive can still leave probability mass on a more suitable category. When the predicted distribution assigns substantial mass to conflicting primitives, their guidance contributions can compete, potentially weakening or misdirecting the correction. Since guidance adjusts the actions proposed by the frozen backbone rather than imposing a hard primitive choice, an incorrect prediction does not necessarily lead to task failure. ## Q5: Can the Authors Report Full-Task Completion Rates? **Response:** Yes. Across the three real-world tasks, full-task success changes from 10% to 40%, 20% to 50%, and 10% to 30%, respectively. ## Q6: Could the Authors Revise the Citation Formatting? **Response:** Thank you for noting this presentation issue. We will audit all citations and use narrative citations only when the cited authors are grammatical subjects.I appreciate the authors' clarifications and additional experiments, which address several of my initial concerns. However, the random-target ablation does not fully establish how errors in PANet predictions affect downstream performance. Overall, I am not fully convinced that the limited set of eight hand-crafted primitives adequately supports the paper's claims about local motion structure and improved generalization. My overall assessment therefore remains unchanged.
Follow-up Response
Response: Thank you for recognizing the clarifications and additional experiments, and for identifying what remained unresolved. We agree that the random-target ablation is an extreme intervention and does not by itself characterize sensitivity to progressively less accurate PANet targets.
We therefore add a controlled target-corruption experiment on the unseen CALVIN-D environment. With probability at each replanning step, we replace the current PANet prediction with a state-shuffled prediction from another timestep of the same task. The intervention preserves PANet's task-conditioned marginal output distribution while progressively removing alignment with the current state; the backbone, 1,000 evaluation sequences, guidance strength, and inference steps remain fixed.
| Method | Corruption | 1 task | 3 tasks | 5 tasks | Avg. Len. |
|---|---|---|---|---|---|
| 3DDA (no guidance) | -- | 93.8 | 66.2 | 41.2 | 3.35 |
| PriGo | 0.00 | 98.1 | 87.6 | 59.9 | 4.18 |
| PriGo + shuffled targets | 0.25 | 90.7 | 62.8 | 36.9 | 3.23 |
| PriGo + shuffled targets | 0.50 | 86.2 | 56.3 | 31.1 | 2.93 |
| PriGo + shuffled targets | 0.75 | 80.5 | 50.9 | 25.9 | 2.66 |
| PriGo + shuffled targets | 1.00 | 72.1 | 39.6 | 20.3 | 2.22 |
Performance decreases monotonically with increasing corruption. Fully shuffled targets reduce five-task completion from 59.9% to 20.3% and average length by 1.96; even falls below unguided 3DDA (3.23 versus 3.35). This provides a graded, causal measurement of how target misalignment affects downstream performance beyond the binary random-target comparison. We do not claim that it reproduces every naturally occurring PANet error, but it directly establishes that the benefit depends on state-aligned predictions and that increasingly inaccurate targets lead to increasingly worse outcomes.
We also agree that eight hand-crafted categories do not constitute a complete or universally optimal representation of manipulation structure. We have revised the paper to call them coarse local-motion categories rather than semantic skills and removed claims that they fully model action structure. Their role is deliberately limited: they constrain a compact set of translational, rotational, and gripper patterns while leaving exact pose, contact, object identity, rotation axis, and continuous magnitudes to the pretrained policy. Table 5 in our manuscript shows that merging directional categories or removing categories that combine translation and rotation reduces task success, supporting the value of these distinctions within PriGo. Supplementary Table 10 further shows that guiding all primitive categories achieves 92.4% success, compared with 91.0% when guiding only grasp, the best single-category variant.
Response Part1
We thank the reviewer for the detailed feedback. It helped us separate what diffusion/flow policies may learn implicitly from what PriGo explicitly constrains, and identify ambiguities in the update equations and PANet evaluation. We will revise both the claims and the presentation accordingly.
W1: Motivation and Logical Flow
Response: We do not claim that diffusion and flow policies are unable to model complex, structured action trajectories. On the contrary, these models can learn rich temporal patterns from demonstrations. However, fitting the training distribution well does not guarantee that the generated actions will follow the appropriate local motion pattern under distribution shifts. This motivates PriGo to predict which motion primitive is appropriate for the current visual observation and task instruction, and use that prediction to guide action generation at test time. The goal is to improve generalization while retaining the pretrained policy’s ability to generate fine-grained continuous actions.
We agree that the eight hand-designed categories are not substantially richer than prior primitives. The novelty we claim is the observation-conditioned predictor, differentiable action-to-category relaxation, and integration with frozen diffusion and flow policies. We will rewrite the Introduction for clearer logic.
W2: Mathematical Formulation
Response: The reviewer is correct. In Eq. (5), the clean action is estimated from the current at; using a~t−1 in the numerator is a notational error. The implementation evaluates the primitive loss at the estimated clean/endpoint action but differentiates the composed loss with respect to the current intermediate state, i.e., ∇atLPG(a^(at),O). Automatic differentiation therefore includes (∂a^/∂at)⊤. We will replace the incorrect ∇a0 and ∇a1 notation in Eqs. (5)–(6) with ∇at, correct the clean-action estimate, and define all variables explicitly.
For flow Euler step δ, at+δ=at+δvθ−η∇atLPG(a^1(at),O); hence the equivalent effective velocity is vθguided=vθ−(η/δ)∇atLPG. We will also distinguish this discrete effective velocity from a separately learned continuous field.
W3: Experimental Evaluation
Response: PANet is trained on 90% of the LIBERO episodes, 20 successful CogACT rollouts per SIMPLER task (10 Visual Matching and 10 Variant Aggregation), and a random 50% of CALVIN A–C episodes. Accuracy is 94% and 95% on held-out LIBERO and SIMPLER trajectories, respectively, and 89% on unseen CALVIN environment D.
We conducted a new random-primitive ablation, reported in Table R1 of our response. Replacing PANet predictions with random primitive targets reduces policy success rates from 87.8 to 80.2, whereas PANet-guided refinement improves performance to 92.4. This shows that target content matters. Online rollouts lack state-matched expert actions, so we cannot measure PANet accuracy on those visited states from the present data. We will state both the evidence and its limits.
Finally, full-task success for the three real-world tasks improves from 10% to 40%, 20% to 50%, and 10% to 30%. We will add these alongside subtask rates.
Table R1. Random-target ablation on LIBERO. Random guidance replaces PANet predictions with randomly sampled primitive targets while keeping the backbone and guidance configuration unchanged.