acceptodds
Under review as a conference paper at ICLR 2027

DreamCloth : Structural Counterfactual Score Factorization for Behavioral System Identification

Abstract

Frozen generative models are increasingly used as differentiable objectives for inverse problems, but their scores often mix together multiple changes in a scene. This makes it difficult to guide an inverse solver when we want to control a specific factor, such as garment appearance or body motion. We introduce **structural counterfactual score factorization**, which addresses this problem by rendering a set of matched counterfactual scenes in which the relevant factors are changed independently. A frozen video model then scores all of these scenes using the same stochastic draw. By comparing the four resulting scores, we cancel the effects of each factor in isolation and retain the part of the score that reflects how the two factors interact. This procedure does not require the frozen model to be linear or its internal representations to be semantically disentangled. We apply this idea in **DreamCloth**, where garment appearance and body motion are the two factors of interest. The resulting interaction signal is used to guide a differentiable cloth simulator and recover material parameters. We further distinguish behavioral identification from unique physical-parameter identification: successful held-out rollouts support the former, but do not by themselves establish the latter. Experiments on ActorsHQ and 4D-DRESS show that factorized guidance improves held-out geometry and rendering compared with direct SDS, while our analyses reveal when the objective provides reliable guidance and when the available evidence is insufficient to uniquely determine the underlying parameters. Overall, our approach provides a way to isolate garment–motion interactions from frozen generative models and use them to guide cloth simulation.

Primary area:
generative models
Keywords:
score distillation, video diffusion prior, differentiable simulation, material point method, cloth simulation, inverse problems, system identification, identifiability, physics-based generative priors, variance reduction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7xctOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DreamCloth, which uses a pretrained video model to fit cloth simulation parameters from multi-view images. It combines guidance from four rendered scene variants to emphasize the interaction between clothing and body motion. Experiments show improvements over direct video-model guidance, but the physical accuracy of the estimated parameters and reliability across repeated runs need stronger validation.

Critical strengths
  1. The proposed four-point interaction contrast constructs model guidance without requiring the video model to internally separate appearance and motion.

  2. The paper distinguishes accurate simulated motion from correct material recovery. The gradient checks, parameter-boundary analysis, and optimization landscape study in Sec. 4.3 show that accurate cloth motion does not necessarily mean the recovered material parameters are correct.

Critical weaknesses
  1. It is unclear how much of the remaining guidance describes cloth physics. The subtraction correctly removes the individual effects defined by the four rendered clips. However, the remaining signal may still reflect the video model’s visual preferences. For example, it might favor a wrinkle pattern because that pattern is common in training videos. The improved geometric accuracy shows that the combined guidance helps, but does not fully explain why it helps. I suggest conducting a controlled test that changes only texture or lighting to measure how much unwanted appearance influence remains.

  2. The experiments do not show whether the improvements are reliable across repeated runs. Each sequence is optimized only once and the results may depend on the starting material values and the random noise used during optimization. The existing 25-run experiment uses a different geometric objective, so it does not test FSD’s reliability. I suggest repeating the full FSD optimization several times for each sequence, with different random seeds and starting material values.

Questions for authors

How stable are the held-out results and fitted parameters across repeated full-FSD runs? Please report variation and boundary-hit rates, and extend the fourth-corner ablation beyond s190.

1 comment

Thank you for a constructive review. You asked for the two experiments we were most worried about. The revision adds one of them in full and part of the other. New text in the revised PDF is in blue.

Are the results stable across repeated runs?

We repeated the full optimization five times on s190, changing the initialization, the sampled timesteps and the noise draws, for three objectives with the same optimizer (new Table 3):

Objective (5 seeds, s190)ρκ_s (Pa)h
Plain SDS1.526 ± 0.1881076 ± 4490.947 ± 0.081
CFG-style score1.522 ± 0.1251157 ± 5450.837 ± 0.047
FSD (ours)1.592 ± 0.202946 ± 5230.990 ± 0.100

Under FSD the stiffness is the least stable parameter (coefficient of variation about 55%), while density and thickness move much less (about 13% and 10%). We don't think this is the optimizer misbehaving. The 25-start landscape study (Table 4) shows a long valley on s190 along which density and stiffness trade off with almost no change in the rollout, and the seeds spread along that valley. The supervision simply does not pin down that direction, and Section 4.3 says so.

Thickness is where the objectives differ. The CFG-style score drifts toward the lower bound (0.837 ± 0.047), below the range that FSD stays in.

We do not yet have held-out error across seeds, or boundary-hit rates, for every sequence. Those runs are not finished, and we would rather add them to the camera-ready than post partial numbers.

The fourth-corner ablation beyond s190

Table 2 now covers three 4D-DRESS sequences with matched runs. Removing the static garment-free render makes the held-out error worse on all three:

SequenceFour rendersThree rendersChange
s1900.3620.422+17%
s1910.2240.278+24%
s1700.3540.433+22%

Without the fourth render, the residual of the static body, including its share of the noise, stays in the objective (Section 3.6), and these runs show that it hurts in practice. They are single-seed runs that end on a box constraint, so they compare objectives rather than recovered materials.

Is the remaining signal still partly appearance?

It can be, and your wrinkle example is exactly the right worry. The four renders cancel the effect of appearance on its own and of body motion on its own. A learned preference for certain wrinkle patterns on moving cloth belongs to the garment–motion interaction, so it survives. Section 3.6 states this, and the paper's claims rest on held-out behavior rather than on the signal being purely physical.

The sign-agreement check in Table S1 helps only a little. The FSD update points the same way as the update from an appearance-free geometric supervisor 68 to 77% of the time, depending on the video model, but that says nothing about how much appearance leaks in.

Your proposed control is the direct test: change only the texture or the lighting, identically in all four renders, and measure how much the score and the fitted parameters move. It is not in this revision. The Conclusion lists it among the next experiments, and we will include it in the camera-ready.

Physical accuracy

The revision now states explicitly that we do not claim to recover the physical parameters exactly; we report parameters that reproduce held-out motion. Section 4.3 shows where 25 runs of an auxiliary geometric optimizer agree (s170 and s185) and where they do not (s190 spreads along the valley, and s191 sits on a bound). A test on synthetic cloth with known parameters is the direct check, and we list it as future work.

Reviewer_a1L4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

DreamCloth estimates garment material parameters for codimensional MPM cloth sim from frozen video generation models. DreamCloth renders four variants of a scene, with simulated clothed, static clothed, moving bare body and static bare body, and backprops the 2x2 interaction contrast of their scores with the goal of cancelling out appearance and body motion components of the score-distillation signal.

Critical strengths

The base problem, that a video prior mixes appearance and body motion, is a real problem, and has applications beyond clothing.

Critical weaknesses

The paper is obtuse and hard to follow with very dense notation. For example, Abstract and Introduction do use lots of jargon to describe the theoretical part of the paper before introducing the actual problem. In general, it feels that the jargon is doing a lot of the heavy lifting: I.e. if we reframe “structural counterfactual score factorization” and ”Moebius/inclusion-exclusion contrast”, than the estimator in Eq 6 is just the 2x2 difference-in-differences of four network outputs with shared noise, which is very similar to DreamPhysics.

Figure 1 is unclear: Monocular 3D Pipeline has two outputs (teal 3d human surface and garment mesh - in “Stage 2” the colors flip - it is unclear what is happening here. What makes this more confusing is that the paper claims that the method can be used monocular (L085) which is not evaluated. It looks almost like Figure 1 describes an earlier single-image design.

The authors should make the difference to Delta Denoising Score more clear, i.e. in L126 the say that “... — FSD instead subtracts those residuals explicitly under shared noise —” - which is what, to my understand, Delta Denoising Score also does. This needs to be differentiated more clearly, or credited here explicitly.

Where do the numbers in Table 1 come from, does the protocol follow MPMAvatar? It seems that all baseline numbers are from MPMAvatar but shifted by ~2% - are those experimental re-runs?

The method heavily relies on stochastic elements, i.e. SFD gradient step is a random sample, SPSA step for madras random perturbations, and the tilmestep schedule is sampled. This requires the experiments to run multiple time and to report mean + variance, to get a better understanding of the expected performance.

This submission has no qualitative videos as result which makes judging the results difficult.

I suggest to soften some of the claims, i.e.

  • Ll040-041: “... an inverse solver often controls only one factor of the scene” —> this statement is inaccurate, inverse solvers often solve many factors of a scene jointly, i.e. geometry + appearance. Maybe re-write it as “... the solver’s free variables cover only part of what the score wants to change.”?
Questions for authors

What differentiating method was used? The authors claim reverse-mode differentiation (L254) through an “unmodified” MPMAvatar solver (L252), but say in Appendix D “500 finite-difference iterations” (L1121). On top of that, MPMAvatar code uses finite differences.

1 comment

We thank Reviewer a1L4 for the careful reading, in particular for catching the inconsistency in the differentiation method. Each answer below says where the revised PDF addresses the point; new and changed text is in blue.


Q1. Which differentiation method is used?

You were right: the submission was inconsistent. Every result uses one-sided finite differences, not reverse-mode differentiation.

  • At each step we run one baseline rollout and one perturbed rollout per parameter (five in total) and form (L(θ+δ) − L(θ))/δ for ρ, κ_s, h and μ.
  • As in SDS, the FSD term enters the loss as the stop-gradient surrogate ⟨sg(w(τ)û_int), V_sim⟩. The four-branch residual is computed once per step from a single (τ, ε) draw and reused for the four perturbed rollouts, so the finite difference approximates Eq. 7 and the video model is queried only on the baseline rollout.
  • ρ, κ_s and h are updated with Adam, μ with a plain gradient step, all in log space and clamped to fixed bounds.
  • The released MPMAvatar solver (NVIDIA Warp) is used for the forward simulation only; no gradient tape is recorded.

"500 finite-difference iterations" now reads "500 optimization iterations". The method is stated consistently in Sec. 3.4, Sec. 3.7, Algorithm 1, App. B.5 and Table S7 (estimator and update rule per parameter). The cheaper two-sided SPSA estimator appears only in the Table S2(b,c) ablations (Eq. S15–S18).


Q2. Relation to Delta Denoising Score and DreamPhysics

We agree that the estimator is a 2×2 difference-in-differences of four network outputs under shared noise, and Sec. 3.6 now introduces it in those words; the Möbius/inclusion–exclusion reading is a footnote.

  • DDS is credited in Sec. 2 and Sec. 3.6 as the two-point case. Subtracting one appearance reference cancels the appearance effect but leaves the body-motion effect; the two extra renders of FSD remove that as well.
  • Measured evidence. Removing the fourth render raises held-out Chamfer distance on all three 4D-DRESS sequences where we ran the ablation, by 17–24% (Table 2). Against DDS itself, FSD is 19% lower on s191 in matched runs. In the SPSA ablation runs of Table S2(c) it is 10% lower on s170 and slightly higher on s190 (0.406 vs 0.399). The DDS comparison is therefore favorable but not uniform, and the paper reports it that way.
  • DreamPhysics keeps a single score branch, reweights it over time to counter the color bias of SDS, and fits volumetric visco-elastic Gaussians. FSD renders four counterfactual scenes and takes their interaction, and fits a codimensional thin-shell cloth model (Sec. 2).

Q3. Where do the Table 1 numbers come from?

Table 1 now marks the capture-based baselines and states the protocol behind each row. [CONFIRM BEFORE POSTING: list which rows were re-run under our rendering and SAM 3 garment-mask protocol and which are taken from the original papers.] The MPMAvatar numbers differ from its paper because our split and garment-mask evaluation differ from theirs.


Stochasticity and repeated runs

Table 3 (new) repeats the optimization five times on s190, varying the initialization, sampled timesteps and noise draws, for plain SDS, a CFG-style two-branch score and FSD. Under FSD, stiffness varies most (coefficient of variation ≈ 55%), density and thickness less (≈ 13% and ≈ 10%). This matches the density–stiffness basin of the 25-initialization study (Table 4). Held-out error across seeds is not yet in the paper; we will add mean ± sd for every sequence in the camera-ready.


Qualitative results

Figure 3 (new, main text) compares held-out rollouts with ground truth over five frames for four garments, and Figure S5 adds zoomed views.


Presentation and claims

  • The Abstract and Introduction now lead with the problem and the confound; the method is explained in plain terms before it is named.
  • L040–041 uses your wording: "the solver's free variables cover only part of what the score wants to change."
  • The single-image claim is removed. All experiments are stated as multi-view, and the single-image setting appears only as future work.
  • Figure 1 is updated for the evaluated multi-view pipeline with corrected labels, and its caption states which paths the experiments use.
  • We do not claim exact recovery of material parameters. The paper reports behaviorally consistent parameters and maps separately where the data do not determine them (Sec. 4.3).
Reviewer_GuV6OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This papers considers video rendering of a person and their garment. The method starts from a set of clothed person images, for which cloth material features are estimated. From this, a person and the garment are reconstructed in 3D and used to initialize Material Point Method simulator which is differentiable. The simulator is used to generate 4 different variants of a video (combinations of body and garment moving and garment being present). The videos are embedded with a video generation model which estimates the direction of change towards a more realistic video. Aggregation of the 4 variants isolates the moving garment. Since the pipeline is differentiable, garment properties may be modified to improve garment movement realism.

Critical strengths

[S1] The idea might be novel

Critical weaknesses

[W1] The paper is very difficult to follow and not ready for publication. E.g. the introduction goes quickly to implementation details without establishing the motivation. The domain or application are not established. There are sentences that disrupt the flow of writing (e.g "Our goal is not to replace those measurements with a generative model and then call the result metorology", "Gradient sign agreement, boundary saturation, and an auxiliary 25-initialization geometric-optimizer study diagnose pieces of the latter but do not prove it. This distinction turns negative cases into useful evidence rather than hiding them.")

[W2] Mixed quantitative results

Questions for authors

N/A

1 comment

Thank you for the review. You were right that the submission was hard to follow, and the front of the paper was the first thing we changed. New text in the revised PDF is in blue.

What is new

Since the review found little that was new, here is the contribution in three points.

  1. We render the scene four ways, with and without the garment and with and without motion. We then combine the four video-model scores so that the effect of appearance alone and the effect of body motion alone cancel, without assuming the model separates them internally. Earlier methods either reweight a single score (DreamPhysics) or subtract one reference (Delta Denoising Score), which leaves the body-motion effect in place.
  2. We use the remaining signal to fit the material of a codimensional cloth simulator from multi-view video, without tracking the garment. To our knowledge, no earlier video-prior method does this for cloth.
  3. We show that matching held-out motion does not pin down the material. Section 4.3 maps where the data determine the parameters and where many settings fit equally well.

Readability

The abstract and introduction now start from the application. To simulate how someone's clothes move you need the garment's material properties, and today these come from calibrated capture with the garment tracked in every frame. We then ask whether a video model can replace that tracking and explain why it cannot be used directly. Only after that do we describe the method, in plain words before any notation.

The contributions are now three short points, and both sentences you quoted are gone. Figure 1 now matches the pipeline we evaluate, and a new Figure 3 shows held-out rollouts over time.

Mixed results

The results are mixed, and the revision says where.

  • Against plain score distillation through the same simulator, FSD lowers held-out Chamfer distance by 19 to 20% on both datasets, or 23 to 26% with timestep annealing (Table 1). Against plain SDS with its best timestep setting on 4D-DRESS, the gain is 5%.
  • The fourth render matters. Removing it makes the results worse on all three sequences where we tested it (Table 2).
  • Against methods that track the garment, DreamCloth is behind MPMAvatar on both datasets and behind PhysAvatar on 4D-DRESS, but ahead of ARAH, TAVA, GaussianAvatar and PhysAvatar on ActorsHQ geometry. Those methods fit tracked garment geometry in every frame, so we see them as a reference with stronger supervision rather than as competitors.
  • In the ablations of Table S2(c), some simpler variants match or beat FSD on individual sequences, and we left those results in.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response To All Reviewers and The Area Chair

Dear reviewers and area chair,

Thank you for the careful reviews. All three of you found the paper harder to read than it should have been, and that was fair. We have uploaded a revised PDF with new and changed text marked in blue. Below is a short summary; the individual replies answer each question in detail.

Readability. We rewrote the abstract and introduction so that they start from the problem. To simulate how someone's clothes move you need the garment's material properties, and today these come from calibrated capture with the garment tracked in every frame. The paper now explains why a pretrained video model cannot simply replace that tracking, and what we do about it, before any formal notation appears. The contributions are three short points, a two-line protocol note says what we do and do not claim, and the sentences GuV6 quoted are gone. Figure 1 has been corrected to match the multi-view pipeline we actually evaluate, and a new Figure 3 shows held-out rollouts next to ground truth.

Method description. a1L4 caught an inconsistency in how we described the gradients. Every result uses one-sided finite differences; we never backpropagate through the simulator. Sections 3.4 and 3.7, a new Algorithm 1, Appendix B.5 and Table S7 now all describe the same procedure.

Related work. Delta Denoising Score is now credited explicitly as the two-point version of our construction, and Section 2 states how we differ from DreamPhysics.

New results. The fourth-corner ablation now covers three sequences instead of one, and removing the fourth render hurts on all three (17 to 24% higher held-out error, Table 2). Table 3 repeats the optimization with five seeds on s190 for plain SDS, a CFG-style score and FSD.

Claims. We now say plainly what the numbers show. FSD lowers held-out error by 19 to 20% against plain score distillation through the same simulator (23 to 26% with timestep annealing). Against plain SDS with its best timestep setting on 4D-DRESS the gain is only 5%, and DreamCloth does not reach the strongest capture-based method. We also no longer suggest that the material parameters are recovered exactly. The paper reports parameters that reproduce held-out motion, and Section 4.3 shows where the data pin them down and where they do not.

Still missing. Two requests need runs we have not finished: held-out error across seeds for every sequence (7xct, a1L4), and the texture and lighting control that 7xct proposed. We would rather add both to the camera-ready than post partial numbers now.

We are happy to answer further questions during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.