DreamCloth : Structural Counterfactual Score Factorization for Behavioral System Identification

Frozen generative models are increasingly used as differentiable objectives for inverse problems, but their scores often mix together multiple changes in a scene. This makes it difficult to guide an inverse solver when we want to control a specific factor, such as garment appearance or body motion. We introduce **structural counterfactual score factorization**, which addresses this problem by rendering a set of matched counterfactual scenes in which the relevant factors are changed independently. A frozen video model then scores all of these scenes using the same stochastic draw. By comparing the four resulting scores, we cancel the effects of each factor in isolation and retain the part of the score that reflects how the two factors interact. This procedure does not require the frozen model to be linear or its internal representations to be semantically disentangled. We apply this idea in **DreamCloth**, where garment appearance and body motion are the two factors of interest. The resulting interaction signal is used to guide a differentiable cloth simulator and recover material parameters. We further distinguish behavioral identification from unique physical-parameter identification: successful held-out rollouts support the former, but do not by themselves establish the latter. Experiments on ActorsHQ and 4D-DRESS show that factorized guidance improves held-out geometry and rendering compared with direct SDS, while our analyses reveal when the objective provides reliable guidance and when the available evidence is insufficient to uniquely determine the underlying parameters. Overall, our approach provides a way to isolate garment–motion interactions from frozen generative models and use them to guide cloth simulation.