acceptodds
Under review as a conference paper at ICLR 2027

MatDuet: Agentic Procedural Material Extraction from Generative Video Models

Abstract

Procedural materials are editable programs that generate surface appearance at arbitrary resolution, but authoring them requires substantial expertise. Generating them automatically is a challenging problem that involves both discrete program design and continuous parameter fitting. We introduce MatDuet, which generates executable procedural shader programs from an image of a material and/or a text prompt. In our full pipeline, a finetuned video generator acts as a "virtual gonioreflectometer", converting the image or text prompt into an 81-frame sequence under prescribed illuminations and viewpoints. From this sequence, we select six frontal relightings and six oblique views that instantiate a plausible target material. MatDuet then alternates discrete and continuous optimization. A coding agent writes an executable Blender program and declares its continuous parameters, which a differentiable renderer fits to the virtual gonioreflectometer relightings while holding the program fixed. The resulting program, its rendered images, matching scores, and visual critique are fed back to the coding agent, allowing fitting residuals to guide subsequent program revisions. The output is resolution-independent Blender source code whose structure and parameters remain available for rendering and editing. Across synthetic materials, authored Blender and Substance materials, and real photographs, MatDuet reduces LPIPS by 9.3–26.1% relative to the strongest baseline on each dataset. Ablations analyze input augmentation, agentic search, differentiable refinement, and test-time scaling. Our code and data will be publicly released. Project page: https://matduet-agent.github.io/

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
procedural material, appearance modeling, program synthesis, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fUxNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an agentic framework for procedural material generation conditioned on image and text, i.e., the system input is a reference image or text description, and the output is a Blender program that matches the reference as close as possible. The pipeline is divided into 2 main parts: first, a video generation model converts the input to multiple images from different lighting and camera pose conditions. Then, a loop iterates between discrete compute graph generation using an agent and a differentiable renderer for parameter optimization, with error feedback to the agent. The authors tested their systems on existing benchmarks (both synthetic and real images), and showed improvements over baselines.

Critical strengths
  1. The quality of the procedural material looks impressive, faithfully capturing the structure and details of the input images. In the appendix, the authors also show meaningful editing by controlling the interpretable procedural parameters, making the system practically useful.
  2. The paper is well-presented and easy to understand. The overall design of separating the discrete and continuous optimization makes sense.
Critical weaknesses
  1. My main concern is whether and how much the system is better than baselines, and under what conditions one would perform better than the other. While the supplementary websites provide a lot of visualizations, there are no direct comparisons with baselines. I find the media format of the website is much easier to judge and provides much more diversity than the few figures shown in the pdf, so it would be helpful to provide comparisons with baselines in this format. This is especially important because, numerically, the method is not strictly better than baselines (Tab1, on real photos, the proposed method is ranked 3rd under the VLM judge).
  2. The novelty of this method is somewhat limited. Both stages of the method have been presented in previous works: VideoNeuMat uses the exact same method to promote input images into videos, and the 2nd stage of agent-based procedural material generation is not new either. It is not surprising that stitching these two together will work, especially given the capability of top LLM agents today. I think this paper brings limited new knowledge to the community, although it may be useful as a well-engineered system.
  3. The meaning of the ablation terms is vague and is not explained in this paper. What does Pure-Agent, No diff-opt, Post-opt mean and how are they different?
  4. The paper emphasizes 81-frame video since the abstract, but then L218 says you use 12 reference images. This is confusing in 1) Why does 81-frame worth emphasizing, and 2) how do you select 12 frames from 81?
Questions for authors
  1. Is the system able to handle more complex materials such as trasparent/traslucent materials (e.g., glass, skin)? I don't think there is a fundamental blocker from my understanding of the system, so it would be interesting to see some examples.
7 comments
AuthorsOpenReview ↗

Author Responses (1/2)

We thank the reviewer for the careful review and constructive feedback. We particularly appreciate the positive assessment of the visual quality of our materials, the practical usefulness of interpretable parameter editing, and the clarity of the paper. Below, we address the concerns about baseline comparisons, methodological novelty, ablation definitions, and frame selection, followed by the question about more complex materials.

W1. Direct visual comparisons with baselines

Thanks for this suggestion, we have updated these results to the website.

W2. Novelty beyond combining existing components

Our contribution is a unified framework that combines video generation, agent-based program synthesis, and differentiable parameter optimization for editable procedural material generation:

  1. Three complementary components. The video model provides references under multiple lighting and viewing conditions; the agent generates and revises program structure; and differentiable optimization fits continuous parameters. To our knowledge, prior work has not combined these three components through the proposed interleaved feedback mechanism to generate executable, editable Blender material programs.

  2. Interleaved feedback between the agent and differentiable optimization. After each optimization round, the selected program, rendered images, and remaining errors are fed back to the agent to guide subsequent program revisions. We implement Blender-compatible differentiable operators and a rendering proxy that enable parameter fitting for agent-generated shaders and write the fitted values back into their source code. This allows the outcome of continuous fitting to inform subsequent discrete program search.

  3. Ablation evidence for the value of this interaction. Post-opt optimizes candidates from all rounds after agent search, matching the full pipeline’s numbers of agent rounds and optimization calls. On MatSynth-I2V, the full method outperforms Post-opt on all five metrics, reducing LPIPS from 0.3570 to 0.3195. This supports the value of interleaved feedback: parameter optimization can both improve individual candidates and provide useful information for subsequent program revisions.

W3. Definitions and differences of the ablation settings

Thanks for pointing this out. As described in the paper’s ablation experiments:

  1. Pure-Agent: A single continuous coding-agent session for generating and revising the program, without round-based feedback or differentiable optimization.
  2. No diff-opt: Our full pipeline with differentiable optimization disabled.
  3. Post-opt: Run No diff-opt, then optimize the candidates from all rounds after the search to match the full pipeline’s optimization budget, and select the best result.
  4. Full: Our complete pipeline, which interleaves agent-based program generation and revision with differentiable parameter optimization.
AuthorsOpenReview ↗

Author Responses (2/2)

W4. The 81-frame video and selection of 12 reference images

The 81 frames and 12 reference images correspond to video generation and material reconstruction, respectively. These stages use different sampling densities to serve their respective purposes:

  1. Video generation uses 81 frames to represent continuous changes in lighting and viewpoint. We adopt the 81-frame output configuration of the pretrained VideoNeuMat model. Dense sampling along the prescribed trajectories produces gradual appearance changes between neighboring frames, allowing us to exploit the video model’s temporal coherence and material appearance priors to generate related observations under different lighting and viewing conditions. The specific sequence length of 81 frames follows the pretrained model’s configuration.

  2. Material reconstruction uses 12 sparsely sampled reference images to balance coverage and computational cost. Adjacent video frames have similar lighting, viewpoints, and appearance, so their information substantially overlaps. We therefore select six frames distributed across each of the two trajectories. This sampling covers different lighting and viewing directions while reducing redundant image processing and the costs of agent image inputs, rendering, and evaluation.

As detailed in the camera and lighting configuration section of the supplementary material, we apply the same fixed frame-selection rule to all samples, using zero-based frame indices:

  • Lighting phase, frames 0-39: The camera remains at a fixed frontal viewpoint while the light orbits the material. We select frames 0, 7, 13, 20, 26, and 33, yielding six frontal reference images under different lighting conditions.

  • Viewing phase, frames 40-80: The light remains fixed above the material while the camera orbits it. We select frames 40, 47, 53, 60, 67, and 73, yielding six oblique reference images from different viewing directions.

  • Both phases are sampled at approximately 60° azimuth intervals. Frames 39 and 80 close their respective orbits and are excluded to avoid repeating the initial poses.

Q1. Support for transparent and translucent materials

Our existing results already include materials with translucency and subsurface scattering effects, such as beeswax and a jelly-like material. We would be happy to add further examples, including rough glass and skin.

Thank the authors for the additional comparisons and explanation of the details. The qualitative comparisons shown in the website are helpful; now I'm convinced that the proposed method generates higher-quality materials than the baselines. My concern about the novelty is not fully addressed. The authors claim their contribution to be a "unified framework", which is more a systems engineering work that chains existing models and systems together rather than interesting research findings. The interleaved feedback makes sense and could be a contribution, but it sounds like a standard way to formulate this problem and is not that exciting. "Ablation evidence" can barely be called a contribution; every paper should do experiments to prove it works. Considering all the factors, I think this paper has good value as an engineering work that provides good results. So I keep weak accept.

Reviewer_ZoizOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper generates materials from an image and/or a text prompt. First, a video model generates 6 relit views and 6 oblique views as input for the writer, a coding agent. Then, in each round, the writer writes or revises the node graph and declares parameters, and a differentiable renderer optimizes those parameters. The optimized program is kept only if it lowers LPIPS. The renders, scores, and text feedback are sent back to the writer. After a fixed number of rounds, the program with the lowest LPIPS is chosen. The paper (1) formulates the problem as reconstruction from n >= 1 images, (2) develops the method as an interleaved two-level search over program structure and parameters, and (3) shows 9.3-26.1% lower LPIPS than the baselines on each of MatSynth, Blender, Substance, and real photos. The ablations compare single to augmented input, and interleaved fitting to fitting only after the agent search.

Critical strengths

The full pipeline shows a reasonable design with per-round feedback to address the unrecoverable error when fitting is applied only after a program has been generated. Post-opt uses the same number of agent rounds and optimization runs but fits only after the agent search, and the paper show that MatDuet has lower LPIPS, DreamSim, style loss, and SWD and higher CLIP.

Critical weaknesses

The source of the gain over baselines is unclear. MatDuet has video augmentation and the agent-gradient loop. On MatSynth, removing augmentation (Single input) raises LPIPS from 0.3195 to 0.3563, while removing the loop and fitting (Pure-Agent, L457-459) raises it to 0.3759 (Table 2). Both are still below the best baseline (0.4323, Table 1). If Pure-Agent uses the augmented views, Table 2 does not include a variant that removes both. The agentic baselines use a single image, but they are in their own frameworks (L362-363). On MatSynth, MatDuet gains over Pure-Agent mainly in LPIPS, the selection metric (L299-301, L314-315), not in DreamSim or CLIP (Table 2). Therefore, it is unclear whether the writer with a single image and no loop would already outperform the baselines, and how much of the 9.3-26.1% improvement (L416-417) is from the proposed method.

Questions for authors
  1. What inputs, budget, and tools does Pure-Agent have? My reading assumes it uses the augmented views and no fitting, but the paper does not state this.
  2. Can you report Single input results on all datasets, and Pure-Agent with a single input image? The latter has no augmentation and the loop, so it would show whether the gain is from the writer alone.
  3. Appendix E.3 attributes the lack of gain in DreamSim and CLIP to the fitting loss, but No diff-opt, which has no fitting, is worse than Pure-Agent in DreamSim (Table 2). Is MatDuet still better than Pure-Agent if programs are selected by a metric other than LPIPS?
3 comments

We thank the reviewer for the careful evaluation and constructive feedback. We particularly appreciate the recognition that interleaving program revisions with differentiable parameter optimization addresses limitations of fitting only after agent search. Below, we clarify the Pure-Agent setting, report the additional single-input ablation, and address candidate selection.

W1 and Q2. Source of the improvements and single-input ablation

  1. We agree that the original ablations lacked a Pure-Agent variant that removes both video augmentation and the agent-gradient loop. We therefore add this variant on fixed subsets of 10 cases each from MatSynth and VideoNeuMat. It receives only one reference image, without augmentation, the MatDuet outer loop, or differentiable fitting.

  2. The table below compares the new variant with multi-input Pure-Agent and the reported Full results on the same cases. Multi-input Pure-Agent achieves better mean scores than single-input Pure-Agent on all five metrics on both subsets, supporting the usefulness of our observation augmentation even without the agent-gradient loop. Full adds our agent-gradient loop and provides further improvements. Together with the existing Full vs. Post-opt ablation, these results support video augmentation and interleaved optimization as useful components of MatDuet.

    DatasetMethodLPIPS↓DreamSim↓Style↓SWD↓CLIP↑
    MatSynthPure-Agent, single input0.42940.23520.02860.03760.9200
    MatSynthPure-Agent, multi-input0.30290.11890.02170.02560.9481
    MatSynthFull0.30270.12770.01970.02530.9259
    VideoNeuMatPure-Agent, single input0.39960.23870.04020.05060.8736
    VideoNeuMatPure-Agent, multi-input0.37630.20730.03240.04340.8777
    VideoNeuMatFull0.31420.19160.03310.03180.8803

Q1. Inputs, budget, and tools of Pure-Agent

  1. Inputs. The original Pure-Agent receives the same twelve reference observations as Full: six frontal images under different lighting conditions and six oblique views. The additional single-input variant receives only the canonical reference image. Both use Claude Code with Opus 5 at the xhigh reasoning setting.

  2. Budget. The original multi-input Pure-Agent runs as one continuous coding-agent session with a maximum duration of 4.5 hours, matching the loop-based pipeline’s total writer-time allowance.

  3. Tools. Pure-Agent can write and execute Python code, construct and revise Blender shaders, render candidates, and inspect the rendered images visually. It does not receive MatDuet’s external critic feedback or differentiable optimization tools.

Q3. Metric differences and alternatives to LPIPS selection

We agree that the fitting loss alone cannot explain this observation, since No diff-opt also trails Pure-Agent in DreamSim. We will revise Appendix E.3 accordingly. Our experiments focus on video augmentation and interleaved agent–gradient refinement, using LPIPS as a simple, fixed criterion for candidate selection. We have not evaluated alternative selection criteria. Exploring selection criteria that better balance perceptual metrics is a useful direction for future work.

Thank you for the results. I went to check Table 1 and Table 5. The single-input Pure-Agent already has comparable LPIPS on MatSynth (0.4294 vs. 0.4323) and lower LPIPS on VideoNeuMat (0.3996 vs. 0.5016), although they use different case sets. However, the agent-gradient loop improves the results on VideoNeuMat (LPIPS 0.3763 to 0.3142) but not on MatSynth. Multi-input Pure-Agent and Full have nearly equal LPIPS (0.3029 vs. 0.3027), but Full is worse in DreamSim (0.1189 to 0.1277) and CLIP (0.9481 to 0.9259). The gain, to me now, mainly comes from a better writer on VideoNeuMat and from more views on MatSynth, which limits the contribution of the agent-gradient loop. The authors also acknowledge that they have not evaluated selection criteria besides LPIPS. Nevertheless, I appreciate the idea that interleaving the fitting with the agent rounds avoids the errors of fitting after the agent search. I keep my rating.

Thank you for the thoughtful follow-up and for appreciating the idea behind our interleaved optimization approach. We will clarify how the different components contribute to the observed gains in the revision.

Reviewer_vN9jOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MATDUET targets the task of generating editable Blender procedural materials from text or image inputs. It leverages the frozen VideoNeuMat video model as a virtual gonioreflectometer to expand single‑source inputs into multi‑illumination and multi‑view observation samples, and establishes an iterative agent‑gradient closed loop alternating between a coding agent and differentiable optimization. The outer multimodal coding agent writes and revises discrete shader program structures and declares optimizable parameters, while the inner differentiable rendering proxy performs gradient‑based fitting for continuous parameters with fixed graph topology. Quantitative metrics from authentic Cycles rendering and VLM‑produced visual defect feedback are fed back to the agent for multi‑round iterations, ultimately outputting resolution‑agnostic, parameter‑editable executable Blender shader code. Nevertheless, this work has notable limitations. The full pipeline heavily relies on closed‑source commercial large‑model APIs such as Claude and Gemini, incurring substantial API costs; model version drift may degrade output quality, and end‑to‑end validation with open‑source models is absent. Physically inconsistent samples yielded by the upstream video generation model propagate downstream errors. The system selects candidates purely based on front‑lighting performance, with oblique‑view surface geometry constrained only indirectly via textual feedback. Its lengthy pipeline exhibits considerable non‑determinism and heavy computational overhead. Some output shader node graphs become excessively bloated, and bitmap texture embedding is unsupported. Overall, its closed‑loop paradigm combining agents and differentiable optimization offers high reference value, yet practical real‑world deployment remains challenging due to external dependencies, computational costs and evaluation constraints.

Critical strengths

The paper introduces an iterative workflow that interleaves LLM‑driven discrete shader‑program editing with differentiable continuous‑parameter fitting. Unlike prior methods that generate a full procedural program and only tune parameters afterwards, reconstruction residuals from differentiable optimization are fed back to the coding agent to revise program structure. This addresses a core pain point in inverse procedural modeling: parameter‑only tuning cannot fix structural defects in material graphs. This design represents a meaningful conceptual advance for agent‑based graphics asset generation and strongly supports the paper’s acceptance.

Critical weaknesses
  1. This whole system needs commercial AI APIs to write shader code, check visual results, and score outputs. The authors only tried one open‑source model for code writing. Other key parts still depend on paid black‑box AI services. Different API versions or random outputs from these paid tools can change final results. High money cost and poor repeatability hurt how useful this method is for other researchers. This is an important reason against acceptance.

  2. The system picks good materials only by checking images under front‑facing light. It has no strict numerical check for surface shape when viewed from side angles; it only uses text feedback for these cases. Besides, the multi‑view input images come from a generative video model. These generated pictures may break real‑world physical rules, and wrong information will pass into later steps. Good test scores do not promise realistic material performance under new views or lighting. This lowers trust in the experiment results and real‑world usability.

Questions for authors
  1. In your main experiments, all key parts except the coding agent use closed‑source commercial VLM APIs. Can you share results when you replace every VLM‑based component (visual critic and VLM‑Judge) with open‑source models? These results will help us know how well your method works without paid commercial AI tools.

  2. The multi‑view input images are generated by VideoNeuMat, and these generated samples may be physically inconsistent. Do you have any simple way to detect or filter these bad generated samples before feeding them into MATDUET? If no, why not?

  3. Since LLM sampling creates randomness, when you run MATDUET multiple times on one same input, how large is the typical spread of final LPIPS scores? Please report mean and standard deviation over repeated runs for a small subset of test cases. This information is important to judge result stability and reproducibility.

6 comments
AuthorsOpenReview ↗

Author Responses (1/3)

We thank the reviewer for the careful evaluation and constructive feedback. We particularly appreciate the recognition that interleaving shader-program revisions with differentiable parameter optimization addresses structural defects that parameter-only tuning cannot resolve. Below, we respond to the concerns about external dependencies, observation reliability, and reproducibility, and report additional open-source and repeated-run experiments.

AuthorsOpenReview ↗

Author Responses (2/3)

W1. Commercial API dependence and reproducibility

  1. We selected the commercial models used in our experiments for their strong code-generation and visual-reasoning capabilities at the time; there is no fundamental dependence on them. Our framework allows alternative models to implement these roles as suitable open-source models become available. Use of SOTA commercial models is established practice in procedural material generation and related 3D graphics research: BlenderAlchemy (ECCV 2024), Material Apprentice (ECCV 2026), and SceneCraft (ICML 2024) all invoke proprietary models within their generation and visual-feedback pipelines. Commercial models and their associated costs are therefore a practical decision shared with related works that should be assessed alongside the methodological contribution and empirical evidence.

  2. Regarding API drift: AutoFigure and pySpatial, both accepted at ICLR 2026, directly invoke proprietary models within their inference pipelines. Their released implementations use undated identifiers, including gemini-2.5-pro and gpt-4.1, rather than date-pinned snapshots. These precedents indicate that date-pinned API snapshots are not a universal prerequisite for acceptance. The possibility of future provider-side changes alone does not invalidate our reported comparisons or methodological conclusions, and we respectfully disagree that specific APIs constitute a major part of our method.

  3. We are aware of these reproducibility concerns and have taken practical steps to mitigate their impact within our API-cost budget. In particular, our VLM-based scoring follows an LLM-as-a-Verifier approach, using repeated sampling to reduce the influence of individual stochastic responses on the final score. This design balances scoring stability with evaluation cost.

  4. In fact, our work takes a concrete step in the direction of reducing dependence on proprietary software and making procedural material research more accessible. To our knowledge, we are the first to enable differentiable parameter optimization of procedural material graphs in the Blender ecosystem, addressing a capability gap explicitly identified by VLMaterial. Our implementation extends this capability beyond Substance-based workflows. We will publicly release the complete differentiable optimization implementation, enabling researchers to reproduce, extend, and build upon this capability without Adobe Substance Designer or its Automation Toolkit. Our goal is to provide open research infrastructure that lowers the barrier to entry for the broader community.

W2. Multi-view evaluation and physical consistency

  1. We would like to point out that, as explicitly stated in Section 3.2, our method uses “six frontal images under different lighting conditions.” Here, “frontal” refers to the camera viewpoint, while the lighting varies across six different positions. Candidate selection is based on the mean LPIPS across all six lighting conditions.

  2. We would like to clarify that we use a fixed frontal camera only for the differentiable optimization step, for simplicity and computational efficiency, while fitting across six different lighting conditions that provide cues about surface relief. We compute mean LPIPS over six oblique views and provide this numerical feedback alongside visual critique to guide subsequent agent revisions. The oblique-view score serves as an auxiliary diagnostic rather than a hard selection criterion.

  3. We agree that generated views may contain physical inconsistencies that propagate to downstream reconstruction. Thus, to isolate the confounding effect of video-generation errors, we conduct an Oracle ablation in the supplementary material (Sec. D.3.2) on 89 MatSynth materials, replacing generated videos with rendered GT observations  under multiple lighting and viewing conditions. While GT observations yield better reconstruction metrics, our pipeline outperforms competing methods using both generated and GT video inputs. On MatSynth, it achieves an LPIPS of 0.3195, compared with 0.4323 for the strongest baseline, with both evaluated against independent source-material ground truth. These results demonstrate that generated observations remain useful for material reconstruction despite their imperfections.

  4. We agree that numerical scores alone do not fully characterize material behavior under new views and lighting. Therefore, our assessment also includes extensive visualizations on the anonymous project page, showing reconstructed materials under varied viewpoints and illumination. We additionally demonstrate the recovered materials in a complete room scene, providing a concrete example of their integration into a practical 3D modeling and rendering workflow. These demonstrations offer direct qualitative evidence of material behavior in context and support the practical usability of our outputs.

AuthorsOpenReview ↗

Author Responses (3/3)

Q1. Open-source alternatives for the visual critic and VLM-Judge

  1. Our method can be instantiated with a fully open-source model-and-harness stack. Appendix D.5 was intended to demonstrate this flexibility by replacing the Opus 5 writer and Claude Code harness with GLM-5.3 and pi. We thank the reviewer for pointing out that this experiment still retained a Gemini visual critic. We have now replaced that critic with Kimi K3 as well. The resulting GLM-5.3 + Kimi K3 + pi pipeline completed all 10 cases in a fixed MatSynth-I2V cohort and improved all five offline metrics over the earlier GLM-5.3 + pi configuration with Gemini, as shown below.

    ConfigurationCasesLPIPS ↓DreamSim ↓Style ↓SWD ↓CLIP ↑
    Full (Opus 5 + Gemini + Claude Code)100.30270.12770.01970.02530.9259
    GLM-5.3 + Gemini + pi (Appendix D.5)100.35170.23180.02480.03090.9110
    GLM-5.3 + Kimi K3 + pi100.33780.17200.02350.02420.9187
  2. We additionally clarify that VLM-Judge is a separate offline evaluation tool: it does not participate in shader generation, parameter optimization, or candidate selection. The five metrics reported above are independent of VLM-Judge.

  3. We additionally evaluated judge sensitivity on the same 10 MatSynth-I2V materials, scoring the fixed outputs with both GPT-5.6-sol and Kimi K3 under the same rubric and image inputs, with three sampled ratings per output. The Spearman correlations between the two judges’ case rankings are 0.9636 for GLM-5.3 + Gemini + pi and 0.8960 for GLM-5.3 + Kimi K3 + pi. The two judges also agree on the relative ordering of the two configurations in 9 out of 10 cases, indicating substantial agreement under judge replacement on this cohort.

Q2. Detecting and filtering inconsistent generated observations

  1. Our current pipeline does not apply a dedicated consistency-based pre-filter. Such filtering could be added, but our results suggest that effective reconstruction is already possible without it. VideoNeuMat’s Large Reconstruction Model (LRM) likewise demonstrates reconstruction from imperfect generated observations. In MatDuet, we limit reliance on potentially less reliable oblique views by using them as auxiliary feedback, while frontal relightings drive continuous optimization and candidate selection.

  2. Our ablations further support the effectiveness of the agent–optimization loop with and without video-generation errors. In MatSynth-Oracle, generated observations are replaced with consistent source-material renders, and the full method improves LPIPS over Post-opt from 0.2953 to 0.2568. With generated inputs in MatSynth-I2V, it similarly improves LPIPS from 0.3570 to 0.3195. Both settings evaluate final shaders against source-material ground truth. These results suggest that interleaving agent revisions with parameter optimization benefits reconstruction under both consistent and potentially imperfect inputs.

  3. For screening, VideoNeuMat’s Residual Parallax Coherence (RPC) metric checks whether residual parallax after image rectification varies consistently with camera motion. It could help flag problematic sequences, although it does not establish full consistency of geometry, lighting, and reflectance. We have not yet validated a rejection threshold or its downstream benefit, so we view consistency-guided filtering or regeneration as a possible refinement to the current pipeline.

Q3. Run-to-run variability in final LPIPS

We evaluated the full pipeline three times on 10 cases per dataset. The subset-averaged final LPIPS was 0.3211 ± 0.0054 on MatSynth-I2V and 0.3448 ± 0.0151 on VideoNeuMat, reported as mean ± std across the three runs.

Comments on the paper

1 comment

Dear reviewers,

Thanks for your continued efforts. The authors have provided a rebuttal to the initial reviews. Please review it carefully, along with all other reviews and the corresponding author responses, and consider whether you have any further questions for the authors or whether the rebuttal has addressed your concerns. There are only a few days remaining before the discussion period closes, so please use this time to seek any further clarifications or raise any additional questions about this paper.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.