acceptodds
Under review as a conference paper at ICLR 2027

Factored Diffusion Policies: Compositionally Generalized Robot Control with a Single Score Network

Abstract

Robotic tasks are typically specified by a tuple of factors, such as the goal to be reached, the obstacles to be avoided, and the appearance of the scene, and collecting expert demonstrations for every combination of factor values grows combinatorially. We present factored diffusion policies: a single diffusion network trained with per-factor null-token dropout, whose score decomposes additively across factors. When the factors are approximately conditionally independent given the action and observation, the composed score approximates the joint score uniformly over all factor combinations, so a training set in which every factor value appears at least once suffices: the task budget grows with the sum of the factor cardinalities rather than their product. The decomposition residual equals the action-gradient of the factors' interaction log-ratio and is measurable from forward passes, so the assumption can be checked per task, and the same structure yields a closed-loop trajectory-tube certificate. We evaluate our approach using drone racing simulations where a drone must navigate through gates while balancing speed and path curvature against tracking overshoot which can cause gate crashes. Trained on such a cover, the factored network passes several times more held-out gates than the same architecture without dropout, replicated across disjoint covers, seeds, and two and three factors, while the same composition rule over separately trained networks generalizes far worse. Injected factor interaction raises the measured residual and degrades composed inference, and our experiments indicate that the gain comes from the factored training, and the decomposition supplies the task budget, the certificate, and the approximate conditional independence check.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Generalization, multi-task learning, compositional diffusion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GijrOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies compositional generalization in robot control: learning from a limited set of task combinations and transferring to unseen combinations of familiar factors. The proposed method uses a shared diffusion network trained with independent dropout of task factors and composes factor contributions during inference. Under approximate conditional independence and additional regularity assumptions, the paper analyzes score decomposition error and its propagation to closed-loop trajectory deviations. Experiments in simulated drone racing show that factor dropout improves generalization when training coverage is sparse. However, composed inference does not outperform joint conditioning of the same network, and the computed trajectory bounds do not establish task success in the evaluated settings.

Critical strengths
  1. The paper addresses a meaningful problem. Collecting demonstrations for every combination of task factors becomes expensive as the number of factors grows. Exploiting task structure within a shared policy is a relevant direction for improving data efficiency in robot learning.

  2. Some controlled experiments provide useful evidence about the proposed training procedure. Table 2 evaluates sparse task coverage using two covers and two training seeds, showing benefits from factor dropout. The controlled factor-interaction experiment also helps characterize when composed inference deteriorates. These results provide insight beyond a comparison against an unconditioned baseline.

Critical weaknesses
  1. The experimental evidence does not sufficiently establish the practical advantage of the complete method over simpler alternatives and representative existing approaches.

Most comparisons involve variants of the same diffusion architecture. The K-network baseline is a PoCo/DSE-style adaptation to the authors' factor decomposition, rather than separate evaluations of those methods. Its representativeness therefore requires further justification. The baseline without factor inputs also receives less task information, so improvements over that baseline alone do not isolate the value of the proposed mechanism.

Table 1 reports 87% held-out gate passage for composed inference, compared with 97% for joint inference using the same trained network. Joint inference without factor dropout also reaches 97%. Table 2 provides evidence for factor dropout under sparse coverage, but does not establish an advantage from explicit composition. For two factors, composed inference additionally requires three network evaluations per denoising step, compared with one for joint inference. The authors acknowledge that composition primarily provides analytical structure. However, the resulting rigorous trajectory bounds do not certify gate passage in the experiments, leaving the practical benefit of this additional computation insufficiently demonstrated.

The validation scope is also limited. All experiments concern simulated drone flight, without evaluation in another robotic domain or on physical hardware. Importantly, Appendix C states that the main visual checkpoints were trained without factor dropout and do not exercise the proposed composition mechanism. Those results therefore do not directly validate the complete method. Table 1 reports a single evaluation with gates scored independently, which does not establish full-race completion rates or robustness across independent training runs.

  1. The presentation makes it unnecessarily difficult to distinguish the central contribution, supporting evidence, and scope of the claims.

The introduction introduces score decompositions, error bounds, trajectory radii, and ODE sensitivity constants before sufficiently developing the problem and methodological intuition. It combines motivation, technical analysis, and detailed experimental interpretation, making the main argument difficult to follow. A clearer introduction would retain the essential intuition and move detailed mathematical conditions and bounds to the method section.

Although relevant references are included, the introduction does not sufficiently explain the precise limitations of closely related approaches or distinguish the contribution from existing score composition, condition dropout, and task-conditioned diffusion policies. The abstract reports qualitative improvements without concrete numerical results and evaluation conditions, making the magnitude of the empirical benefit difficult to assess.

Figure 1 requires substantial effort to relate multiple tasks, spatial views, and speed profiles. Table 1 highlights the proposed methods in bold even when their results are not the best, without clearly explaining the emphasis convention. Finally, the paper lacks a concise concluding synthesis that separates experimentally supported findings from conditional theoretical results. Together, these issues weaken the clarity of the contribution and its evidential support.

Questions for authors
  1. Please clarify the differences between the implemented K-network baseline and the original PoCo/DSE methods. How do these adaptations affect its representativeness, data utilization, and optimization difficulty?

  2. Beyond providing analytical structure, what demonstrated practical benefit does composed inference offer over joint inference using the same network? Given that the rigorous bounds do not certify gate passage, what concrete decision can the current diagnostic support?

  3. Table 15 reports 36.7% success and 15.0% crashes for zero-shot visual transfer, whereas the “released (paper)” checkpoint in Table 16 reports 42.5% success and 5.8% crashes. Do these results use the same checkpoint and evaluation protocol? Please explain any differences in model versions or evaluation conditions.

  4. Are results from independent training runs or full-race completion evaluations already available for the comparisons in Table 1? If so, do they preserve the reported performance relationships? This question concerns existing evidence rather than a request for an extensive new experimental study.

2 comments
AuthorsOpenReview ↗

Title: Response to Reviewer Gijr

Thank you for the review. The revision answers each point below with new evidence: two added domains exercise the complete method, the same controlled comparison gives the same answer in all three, and the introduction, abstract and conclusion are rewritten.

Weakness 1. The claims rest on the same network with and without factor dropout, and that comparison gives the same answer in all three domains. We agree that the baseline receives less information and that the K-network row is confounded. The comparison we rest on is the same network with and without per-factor dropout at sum-sized coverage, on two disjoint covers and two training seeds per domain: drone racing 3-16% to 36-50% held-out gates; tabletop 52-71% to 96-99%; LiDAR racing 26-81% to 88-97%, crashes 18-28% to 2-9%; with a third latent factor (wheelbase, 72 tasks) a cover of 8 tasks gives 85-92% against 61% without dropout and 30% for the baseline. We keep the vision study in the appendix as preliminary. The composed-versus-joint half of this weakness is answered under Q2.

Weakness 2. Introduction, abstract, Figure 1, Table 1 and conclusion are rewritten as asked. The introduction now states the problem, gives the method with the additive-score intuition, lists four findings labeled F1 to F4, each tagged on the paragraph of Section 4 that establishes it, and names the gap to score composition, condition dropout and task-conditioned policies. The abstract carries the numbers; Table 1 bolds the best entry per column; Figure 1 shows one held-out task per domain; a conclusion separates experimental findings from theoretical results.

Q1. The K-network row is PoCo/DSE-style composition over named task factors with everything else matched; it is not the comparison our claims rest on. PoCo composes policies trained on different data sources by weighting their scores at inference; DSE composes a base skill model with task-specific experts. Both compose separately trained networks with the additive rule we use. Our row keeps that rule and training scheme (12 networks: one unconditional, eight track-only, three aperture-only, each trained on the tasks carrying its value) and changes only what is composed: named task factors. Nothing makes separately fit scores additively compatible, so their sum is unconstrained on unseen pairs; and the row changes parameter sharing, data allocation and optimization at once. We say so in the revision and report three seeds (0-6 of 30 held-out gates, no held-out race).

Q2. No gain in success rate over joint readout, and the paper never claimed one; the demonstrated benefits are a rule for which corrections to compose when a factor is observable, and a per-task residual that predicts composition failure. First, the decomposition says which readout to use when a factor is determined by the observation. In LiDAR racing (Section 4.3) the scan already identifies the circuit. On a sum-sized cover each track is trained at one speed level, so the track token becomes a proxy for that level, and joint readout on held-out combinations is erratic (34-87%). Dropping the track correction leaves a single-factor conditional that completes 88-97% of held-out laps with 2-9% crashes, at one forward pass per step; a jointly conditioned network has no term to drop. Second, the residual eps_D predicts composition failure per task. Across all twelve factored LiDAR checkpoints (Appendix D.2), on the six dropout-trained ones, every task composed inference passes has eps_D at most 0.29 and 39 of the 42 it fails have eps_D between 0.47 and 2.4; the three exceptions are tasks where the joint readout also fails, so the composed score agrees with a wrong joint score. Without dropout the residual predicts nothing. The decision it supports is concrete: drop a correction whose factor the observation fixes, and compute the residual of a new combination before deploying; at the training-task level (0.1-0.2) composed inference passed every task, above 0.47 none. The tube certificate ranks held-out tasks by closed-loop sensitivity, and we present it only as that ranking.

Q3. Same checkpoint, two evaluation runs with different spawn draws. Table 15 of the submission (now Table 17) is the original evaluation; the released row of Table 16 (now 18) re-evaluates it under the retraining study's spawn seed. We say so in the caption of Table 18.

Q4. Yes. Four training seeds and race-level completion are in Table 1, and every ordering holds. Four training seeds for the factored and no-dropout rows, three for the K-network: factored joint 101-102 of 104 gates and 29 of 30 held out on every seed; composed 97-98 and 25-26; no-dropout joint 101-102 and 28-29. Races (all gates, no crash): factored joint 17-18 of 20 and 5 of 6 held out on every seed; composed 14-15 and 2-3; oracle 19 and 6; baseline 3 and 1; K-network 8-10 and 0. Baseline and oracle are single runs.

Reviewer_GijrOpenReview ↗

Follow-up on the Author Response

Thank you for the detailed response and the substantial revision. The new domains, additional training seeds, race-level results, and the rewritten introduction address most of my concerns. I will raise my score to 2. My remaining concern is that composed inference still shows no practical advantage over joint readout, so the main empirical contribution rests on factor-dropout training.

Reviewer_SPRJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Factored Diffusion Policies, a compositional diffusion policy for robot control with a single shared score network. The method assumes that a task can be described by multiple factors, such as track identity and gate aperture in drone racing.

Critical strengths
  1. The paper presents a novel and conceptually coherent formulation of compositional policy learning. Adapting score-space composition to task factors in robot control is interesting, and the use of a single shared diffusion network with per-factor null-token dropout provides a simple and elegant alternative to separately training multiple policies. The method targets a practically relevant problem: reducing the need to collect demonstrations for every combination of task factors.

  2. The theoretical analysis gives an interpretable explanation of when score composition should work. The identity relating the decomposition residual to the action gradient of the factor-interaction log-ratio is particularly useful. It provides a concrete way to reason about the effect of conditional dependence among task factors, rather than treating compositional generalization as a purely empirical phenomenon. The trajectory-level analysis also attempts to connect score approximation errors with closed-loop control behavior, which is valuable for robotic applications.

  3. The experiments include several informative controls and mechanism-oriented analyses. The comparison between dropout and no-dropout training, joint and composed readout, shared and separately trained networks, sparse task coverage, multiple numbers of factors, and manually injected factor interactions provides more insight than a single headline comparison. The strong improvements under sparse task coverage are promising and broadly consistent with the proposed motivation.

Critical weaknesses
  1. The theoretical assumptions do not clearly match the actual task distribution used in the experiments. The analysis assumes a factorized task-generation distribution pgen(z)=∏ipgen(zi)p_{\mathrm{gen}}(z)=\prod_i p_{\mathrm{gen}}(z_i) over the full product space. However, the drone-racing benchmark removes several track–gate combinations because they are infeasible, leaving a structured feasible subset rather than a complete Cartesian product. This filtering induces dependencies between the factors and makes it unclear whether the marginal scores used in the theoretical decomposition are the same as those learned from the sparse demonstrations. In addition, the trajectory-tube theorem compares the learned composed score for a test task with the true score of a nominal training task, without explicitly controlling the difference between the true score of the test task and that nominal score. Thus, the current theorem does not directly establish task success on an unseen factor combination.

  2. The experiments do not isolate a practical benefit of the proposed composed inference. On the main split, factored joint readout achieves 29/30 held-out gates, whereas factored composed readout achieves 26/30. Under sparse coverage, the reported Table 2 results primarily use joint readout, while composed readout is weaker and requires K+1K+1 network evaluations per diffusion step. This makes it difficult to determine whether the main benefit comes from additive score composition or simply from factor conditioning and null-token dropout as a regularizer. The separately trained K-network comparison also changes parameter sharing, data allocation, and optimization difficulty simultaneously, so it does not provide a fully controlled comparison.

  3. The empirical validation is too limited to support the broader claims. The main results come from one simulated drone-racing domain with a small number of fixed task splits and limited random seeds. The headline metric counts gates independently, even though the task definition requires completing an entire multi-gate race; therefore, gate-level success rates may not reflect race-level task success. The trajectory certificate is highly conservative, relies on partially defaulted closed-loop constants, and does not clear the physical gate margins. The visual experiment is explicitly preliminary and does not use the proposed factor-dropout composition mechanism, while no real-robot or manipulation experiments are provided.

Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer SPRJ

Thank you for the review. We respond to the three weaknesses in turn.

Theory versus the feasible subset. The feasible set is 20 of the 24 (track, aperture) pairs, so the task distribution is not a full product. Theorem 1 does not need a product distribution: the decomposition bound is pointwise in z and holds for every task in the product space at which the per-factor scores are defined, and Assumption 1 is a condition on the action-conditional density, not on the task prior. What the feasible subset affects is which per-factor scores the training data identify: a factor value appearing in no feasible training task gives no correction to compose. Every value appears in at least three training tasks of the 14-task split and at least once in every cover. We say this in Section 2 where p_gen is introduced and removed the product notation there. The second point is right, and we made it explicit: Theorem 2 bounds the distance from the composed policy's trajectory on the test task to the nominal trajectory of a training task, and the nominal-to-test gap enters through the per-factor extrapolation term, which is the distance from each test factor value to its nearest training value times the factor's Lipschitz constant. For a held-out combination of seen values that term is zero and the tube is centered on the test task's own nominal trajectory, which is what we evaluate. The revised statement of Theorem 2 names the center.

No practical benefit of composed inference over joint readout. Correct, and we have never claimed one; the revision states this from the abstract on. What we claim, and now show in three domains, is the training effect: the same network without dropout collapses from sum-sized coverage (3-16% to 36-50% held-out gates with two factors, 5-11% to 25-32% with three), and the new tabletop domain reproduces this (52-71% to 96-99%, Section 4.2), as does the new LiDAR racing domain (26-81% to 88-97%, Section 4.3). Composed inference is the object the analysis is about, and in the tabletop domain it matches joint readout within sampling error. Where it does fail, the measured residual says so in advance: in the LiDAR domain, across six dropout-trained checkpoints and 80 held-out tasks, every task composed inference passes has eps_D at most 0.29 and 39 of the 42 it fails have eps_D at least 0.47, the other three being tasks where joint readout fails too (Appendix D.2). We agree the K-network comparison is confounded; we label it as such and rest the mechanism argument on the no-dropout control.

Limited validation. The revision adds the three things the review asks for. (i) Two further domains: tabletop manipulation, with its own success criterion (cube placed, released, no wall contact), replicating the coverage result across two covers, two training seeds and three evaluation seeds; and LiDAR-based racing on the F1TENTH car model, closed loop from the scan, with a lap- time criterion, three training seeds on the dense split and two covers, plus a third latent factor (the car's wheelbase, 72 tasks) on which a sum-sized cover of 8 tasks gives 85-92% held-out laps against 30% for the baseline (Appendix D.1). (ii) Four training seeds for the factored and no-dropout rows of Table 1 and three for the K-network row; all orderings hold. (iii) A race-completion column in Table 1: a race counts only if every gate is passed and the drone does not crash. Factored joint readout completes 5 of 6 held-out races on all four seeds, the oracle 6, the baseline 1, the K-network 0. We agree the certificate is conservative; we present it as a robustness diagnostic and have not strengthened that claim. We keep the vision study in the appendix as preliminary, with its limitations stated first.

Thank you for your detailed rebuttal and the revisions. Your response to W2 clarifies that the main contribution lies in factor-dropout training rather than composed inference.

However, after carefully examining your revised manuscript, I noticed several formatting issues that should be addressed. Both Table 1 and Table 5 extend beyond the page margins.

Did you carefully review the entire manuscript after making these revisions?

Thank you for pointing out the formatting issues. This was an oversight on our part. We uploaded a revised PDF that fixes formatting (tables and one displayed derivation that ran past the margin, and the spacing on page 3; no text, results, or claims have changed.

AuthorsOpenReview ↗

Response to contributions and weaknesses

Thank you also for confirming that the response to W2 clarifies the contribution. If anything in our answers on the theory (W1) or the added validation (W3) remains unclear, we would be glad to address it before the discussion closes.

Reviewer_jtxXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This work is a framework designed to enable compositional generalization in robotic control without requiring expert demonstrations for every combination of task factors. The authors claim that when task factors are approximately conditionally independent, score density factorization translates to additive addition in score space. Thus, one can learn the effect of each factor from data covering these individual factors. Authors also bound the error when factors are approximately conditionally independent. Results are shown on drone racing tasks in simulation, with task factors such as track geometry, desired speed, etc.

Critical strengths

well motivated. good theory on breaking down conditionally (nearly) independent factors.

Critical weaknesses
  1. Suboptimal ablations: Authors want to show that a single shared network is superior to training individual networks for each factor. For this, the authors compare their method against a 12-network baseline. However, this baseline changes multiple items simultaneously (parameter count, data sharing across networks, and optimization dynamics, etc.), discussed in appendix B.6.

  2. From Table 1, it looks like the gains actually come from training a single shared network with per-factor dropout, not from test-time score composition (definition 1).

  3. real robot expts are always a plus. It would help to have those.

Questions for authors

The crash-rate figures in the table should not be read as a property of the method: they depend strongly on how long the policy is trained.

Could the authors explain or hypothesize why we would see this phenomenon? On table 16, I notice that the best epoch is not always near the end of training.

2 comments
AuthorsOpenReview ↗

Response to Reviewer jtxX

Thank you for the review.

The 12-network baseline changes several things at once. We agree, and we say so where the row is introduced. The controlled comparison in the paper is the no-dropout network, which differs from ours only in the dropout probability; the K-network row is kept because it is the closest analogue of composing separately trained policies (PoCo, DSE) and because reviewers of the earlier version asked for it. We now report it over three seeds (60-68 of 104 gates, 0-6 of 30 held out) with the caveat in the caption and the mechanism paragraph.

The gains come from training, not from test-time composition. Yes, and it is the paper's stated finding, now in the abstract, the introduction and Section 4.1, and replicated in three domains. What composition contributes is the analysis: the sum-sized budget, the per- task residual diagnostic and the certificate are all derived through the composed score. The new tabletop and racing domains (Sections 4.2 and 4.3) show the same training effect; in the tabletop domain composed inference matches joint readout, and in the racing domain the composed score without the observable factor's correction is the readout that makes the sum-sized cover work, with two factors and with a third latent factor (8 of 72 tasks, 85-92% held out). The residual the theory bounds is also now a working per-task diagnostic there: it separates the held- out tasks composed inference fails from those it passes on every dropout-trained checkpoint (Appendix D.2).

Real-robot experiments. Not in this revision; we added two simulated domains within the discussion period: tabletop pick-and-place with a different embodiment and success criterion, and closed-loop LiDAR racing on the F1TENTH car model, the platform closest to hardware we could reach, with a receding-horizon policy driving from the raw scan.

Why does the best epoch not fall at the end of training, and why do crash rates depend on training length? The vision policies are trained with a fixed 50-epoch cosine schedule and the checkpoint is selected by validation denoising loss, which bottoms out early (epoch 7 to 12) and then rises as the model overfits the 8,550 training rollouts. Across the six retrained models in Table 18 of the revision (Table 16 of the submission) the crash rate rises as the validation loss falls: a tighter fit to the training venues produces more confident, faster approaches that turn misses into crashes on the held-out venue. We therefore report success only and treat crash rate as a training-length property. We added the matched early-stopping pair (both models stopped at epoch 7): with dropout off, 44.2% success and 15.0% crashes; with dropout on, 30.8% and 14.2%. Dropout does not change the crash rate at matched training length. That sensitivity is why the two new domains use state and LiDAR inputs, where the closed loop is deterministic given the policy and the training-rule question can be isolated.

Thank you for your response and updating the messaging of the paper. I acknowledge that I have read the rebuttal.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion for the Area Chair

Dear Area Chair,

Thank you for handling our submission. This is a brief summary of where the discussion stands.

The revision adds two domains, tabletop manipulation and LiDAR-based car racing, and the central result holds in all three: with per-factor dropout, one shared network generalizes from a training set that grows with the sum of the factor cardinalities, not their product. Same network without and with dropout: 3-16% to 36-50% of held-out gates (drone racing), 52-71% to 96-99% of held-out trials (tabletop), 26-81% to 88-97% of held-out laps (LiDAR f1tenth car racing), each on two disjoint covers and two training seeds.

All three reviewers noted that composed inference does not outperform joint readout. We agree, and the revision says so from the abstract on; the claim is the training result above. After the revision, Reviewer Gijr raised their rating from 1 to 2 and wrote that it addresses most of their concerns, and Reviewer SPRJ wrote that our response clarifies the contribution. The remaining question, as we read it, is whether the training result is a sufficient contribution without an inference-time gain. We believe it is, since the cost of demonstrations for every factor combination is the practical problem.

The tables Reviewer SPRJ found extending past the margins are corrected in the PDF uploaded on 18 Nov.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.