MatDuet: Agentic Procedural Material Extraction from Generative Video Models

Procedural materials are editable programs that generate surface appearance at arbitrary resolution, but authoring them requires substantial expertise. Generating them automatically is a challenging problem that involves both discrete program design and continuous parameter fitting. We introduce MatDuet, which generates executable procedural shader programs from an image of a material and/or a text prompt. In our full pipeline, a finetuned video generator acts as a "virtual gonioreflectometer", converting the image or text prompt into an 81-frame sequence under prescribed illuminations and viewpoints. From this sequence, we select six frontal relightings and six oblique views that instantiate a plausible target material. MatDuet then alternates discrete and continuous optimization. A coding agent writes an executable Blender program and declares its continuous parameters, which a differentiable renderer fits to the virtual gonioreflectometer relightings while holding the program fixed. The resulting program, its rendered images, matching scores, and visual critique are fed back to the coding agent, allowing fitting residuals to guide subsequent program revisions. The output is resolution-independent Blender source code whose structure and parameters remain available for rendering and editing. Across synthetic materials, authored Blender and Substance materials, and real photographs, MatDuet reduces LPIPS by 9.3–26.1% relative to the strongest baseline on each dataset. Ablations analyze input augmentation, agentic search, differentiable refinement, and test-time scaling. Our code and data will be publicly released. Project page: https://matduet-agent.github.io/