acceptodds
Under review as a conference paper at ICLR 2027

MatLoom: Layered Text-to-Material Generation in a Compact Program Space

Abstract

Material generation should produce not only an appearance, but also the rules that construct it. We introduce MatLoom, a compact, layer-oriented language for text-to-material generation with pretrained language models. Each program composes alpha-masked layers whose shared spatial expressions define coverage and physically based rendering (PBR) channels, making dependencies between patterns, color, and relief explicit. A standalone interpreter evaluates the program into material maps, while the source retains named fields and layer parameters for subsequent authoring. Without task-specific fine-tuning, our pipeline uses parser-guided repair and preview-based critique to revise material designs, then searches noise seeds while keeping each candidate's remaining source fixed. On a curated benchmark of 141 prompts evaluated with six backbones, our best-performing configuration achieves higher mean scores than three diffusion baselines on all four flat-layout prompt-alignment metrics. Its initial programs already exceed all three baselines on mean BLIPScore, before critique or seed search. Retained programs have a median length of 21 lines when pooled across backbones. In a blind four-way comparison involving 30 participants and 20 prompts, our renders receive 59.2% of choices, compared with 19.3% for the most-preferred baseline. Compact executable programs thus offer a way to generate prompt-aligned materials while retaining their construction as part of the asset.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
material generation, procedural materials, physically based rendering, domain-specific languages, program synthesis, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KGqvOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MatLoom, a compact, layer-oriented language for text-to-material generation. Shared spatial expressions define layer coverage and material channels such as base color, roughness, and height, while a standalone interpreter evaluates programs into renderable material maps. The synthesis pipeline uses pretrained language models without task-specific fine-tuning, combining parser-guided repair, render-based refinement, and text–image scoring for program selection and noise-seed search.

The authors evaluate six language-model backbones on 141 material descriptions against three diffusion-based material generators. The strongest configuration achieves the highest mean on all four flat-layout alignment metrics and receives 59.2% of choices in a blind study involving 30 participants and 20 prompts. A manual editing case additionally illustrates parameter control and explicit cross-channel dependencies.

Critical strengths

The representation is coherent and retains meaningful program-level control. MatLoom organizes layers, shared spatial expressions, and material channels into a compact executable representation. Dependencies between coverage, color, and relief remain explicit in the source. For example, a shared tile mask can control both exposed grout regions and relief boundaries. This provides a concrete mechanism for inspection, parameter modification, and procedural resampling. The editing example in Section 5.4 verifies these mechanisms for the demonstrated material.

The complete system’s appearance results are supported by both automated evaluation and a human study, with appropriately qualified reporting. The evaluation covers 141 prompts, six language-model backbones, and three diffusion baselines. Beyond automated metrics, the blind study with 30 participants provides evidence of appearance preference on the evaluated subset. The paper also reports variation across backbones and prompt sources, disagreement between selection and evaluation metrics, and a nonsignificant comparison. These analyses help readers assess the scope of the demonstrated gains.

Critical weaknesses

The practical value of the proposed representation relative to existing procedural representations is insufficiently established. The central contribution is a restricted, layer-oriented field language, but the main quantitative comparisons are against diffusion models that output raster maps. These comparisons establish the competitiveness of the complete system, but do not determine whether the proposed representation is easier for language models to generate correctly, or more useful for subsequent editing, than existing procedural material approaches or direct Blender program generation. The related-work discussion and Appendix A already acknowledge these distinctions and the absence of direct controls; nevertheless, its practical advantages over alternative procedural representations remain insufficiently established. Furthermore, Section 5.4 demonstrates manual parameter edits for only one material, which is insufficient to assess editing utility across a broader set of generated programs. Given substantial prior work on layered materials and procedural generation, this gap limits the assessment of the contribution’s incremental significance.

Uncontrolled inference budgets and rendering differences limit the interpretation of the baseline comparisons. MatLoom uses five refinement rounds and seed search over up to 3,003 candidates, while the diffusion baselines do not receive a matched search budget. The rendering routes also differ materially: MatLoom’s height drives geometric displacement, StableMaterials’ height drives bump shading, the other two baselines provide no height map, and only MatLoom uses transmission. Consequently, some appearance gains may reflect a combination of generation quality, inference expenditure, and rendering treatment. The initial programs already outperform the baselines on some metrics, and the human study supports the complete pipeline, so these concerns do not invalidate the results. However, the current experiments do not establish how much of the advantage remains under comparable computational budgets and shared rendering channels. Section 5.6 transparently acknowledges this limitation, but its impact is not experimentally quantified.

Questions for authors

The main quantitative comparisons focus on diffusion-based material-map generators. Why were closer procedural material-generation methods, particularly Material Apprentice, which also supports text-conditioned generation, not included? The related-work discussion explains differences in representation and execution, but what practical advantages do these differences provide—for example, higher program-generation success, a better quality–cost trade-off, or more reliable editing control? Could the authors provide a direct comparison or other evidence supporting these benefits?

How much of the performance gap remains when comparable material channels and height treatments are used? Could the authors report quality–cost curves using the final evaluation metrics, rather than only improvements in the internal selection score as the seed-search budget increases?

4 comments
AuthorsOpenReview ↗

Response to Reviewer KGqv (1/4): points 1–2 (Material Apprentice, generation success)

Thank you for the detailed and fair review. Your two main points were that the value of the representation against other procedural approaches was not established, and that Sec. 5.6 acknowledged the budget and rendering confounds without quantifying them. We agree with both. Below we report the experiments we ran in response, including where the outcome is less favorable to us. Our answers are numbered 1 to 6 across four comments: (1) Material Apprentice, (2) program-generation success, (3) quality and cost, (4) editing control, (5) channels and height treatment, (6) quality against cost in the final metrics.

1. Why Material Apprentice was not included, and the comparison now (Q1)

"Why were closer procedural material-generation methods, particularly Material Apprentice … not included?"

Material Apprentice (MA) was published at ECCV 2026 and we treated it as concurrent work. The submission discussed it in Sec. 2 and Appendix A but did not run it. We have now run its released code (commit 6811f66, default gpt-5 configuration) and rendered and scored its outputs with our stack. One MA run takes about an hour and 1.5M prompt tokens, so in the rebuttal period we could cover a stratified 45-prompt subset, not all 141 prompts.

SystemExecutedFlat BLIPFlat CLIPFlat VQALLM calls per promptMinutes per prompt
Material Apprentice (text-only, gpt-5)45/4517.9821.9732.68112 (84 to 123)41 to 84 (median 61)
MatLoom (gemini-3.6-flash)45/4558.9429.1257.22124.6 (median)
StableMaterials45/4526.6925.3843.630under 0.1

Two caveats matter for reading this table, and App. U spells out both. First, MA was designed and evaluated with an image plus text as input. We run its text-only mode, in which it first generates its own reference image, so this is outside the setting its authors evaluated. Second, our API proxy does not support the image tool that MA calls, so we rerouted that call to the equivalent images endpoint with the same parameters and capped concurrency at two. For both reasons we treat MA's appearance scores as a lower bound on what the system can do and do not use them as a headline. For completeness, the paired difference is +41.0 flat BLIPScore, and +40.1 judge points within one judge session.

What we rely on is execution and cost. Both systems completed all 45 prompts, and MA needed roughly nine times as many LLM calls per material. Its cost comes from its multi-module design (retrieval, compilation, three rounds of best-of-10 sampling) and not from failed runs.

MatLayerNet and MultiMat have not released code, and MultiMat is not text-conditioned, so we could not run them.

2. Practical advantages: program-generation success (Q1)

"…what practical advantages do these differences provide—for example, higher program-generation success, a better quality–cost trade-off, or more reliable editing control?"

We measured all three; they are points 2, 3, and 4. To isolate the representation, we ran a controlled swap of the output language: same backbone (gemini-3.6-flash), same 45 prompts, repair budget, 5 critique rounds, selection rule, renderer, and metrics, with seed search off on both sides. The model writes either MatLoom or Blender-Python. For Blender we tested a scaffolded variant (the model fills in a build_material function on our scene) and an unrestrained one (it writes the whole script, 30 prompts). The Blender reference document in the prompt is 24k characters, at least as long as our DSL prompt.

Program-generation success turned out not to be the advantage.

MatLoomBlender-Python (scaffolded)Blender-Python (unrestrained)
Runs ending with a valid render45/4543/4530/30
First attempt valid(a)36/45 (80%)17/30 (57%)
Generation attempts that execute(a)93.3% of 29786.2% of 210
Runs recovered by repair after a failed first attempt(a)8 of 913 of 13
Final renders with visible texture (b)40/45 (89%)26/43 (60%)20/30 (67%)

(a) Per-attempt statistics were recorded for the Blender variants; a MatLoom program that parses always executes, and initial generations over the same 45 prompts parse on the first attempt in 37/45 (82%, 95% CI 69-91). (b) Pixel standard deviation of the flat render of at least 10, computed after the runs.

Blender first attempts fail in 20% and 43% of runs, in line with the roughly 30% first-shot failure rate that MA reports, but the repair loop recovers nearly all of them. A comparison of execution rates would therefore miss the difference, which is in what the successful programs produce (point 3).

AuthorsOpenReview ↗

Response to Reviewer KGqv (2/4): points 3–4 (quality and cost, editing control)

3. Practical advantages: quality and cost per successful program (Q1)

This uses the same controlled swap between MatLoom and Blender-Python (BP) as point 2.

MetricMatLoomBP scaffoldedBP unrestrainedMatLoom − scaffolded, paired [95% CI]
Flat BLIP47.628.815.6+19.7 [+8.1, +31.1]
Flat CLIP29.424.722.5+4.9 [+3.4, +6.5]
Flat VQA54.042.135.7+12.6 [+6.8, +18.5]
Flat judge76.547.140.4+29.3 [+20.6, +38.3]
Staged BLIP27.67.0(c)+21.4 [+12.6, +31.2]
Staged CLIP25.020.8(c)+4.5 [+2.9, +6.2]
Staged VQA52.035.6(c)+16.9 [+12.0, +22.1]
Staged judge61.330.7(c)+30.0 [+21.7, +38.7]
Mean program size31.3 lines, 2,559 bytes88.4 lines, 3,541 bytes2,729 bytes
API cost per successabout 0.08 USD (est.)0.09 USD0.06 USD

Means are over the runs that delivered a render. All paired differences have p < 0.001 except flat BLIP (p = 0.001). The judge columns were scored in one session. (c) The unrestrained variant renders its own scene, so it has no comparable staged render.

API cost per material is about the same on both sides. The differences are quality, program length (about a third as long), and preview time: about 70 ms in our interpreter against about 12.5 s for a Blender attempt. The scope is one backbone and one seed, because the second backbone did not finish before we froze scoring.

4. Practical advantages: editing control (Q1)

Sec. 5.4 showed one material, as you noted. We built a benchmark of 172 edit tasks on 40 generated materials. A task succeeds only if the requested change is made and every channel outside the intended change stays within 1/255. The task list, tolerances, and masks were fixed before any editor ran. An LLM edits the MatLoom program (2.6 calls per task on average). MatFuse and StableMaterials are re-prompted with the edit instruction and keep the best of three seeds.

Success rate in %, with the rate of making the requested change (ignoring preservation) in parentheses:

Edit typenMatLoom program editMatFuse re-promptStableMaterials re-prompt
Roughness shift2580.0 (84.0)0.0 (84.0)0.0 (76.0)
Base-color shift3884.2 (84.2)0.0 (57.9)0.0 (78.9)
Widen a shared mask feature (e.g. grout)3180.6 (83.9)54.8 (61.3)22.6 (61.3)
Noise-seed swap4077.5 (100.0)n/an/a
Layer removal3897.4 (100.0)n/an/a
All tasks17284.3 (91.3)9.9 (36.0)4.1 (39.5)
First three types only9481.9 (84.0)18.1 (66.0)7.4 (72.3)
Mean change on channels that should not move (of 255 levels)0.0834.868.1

Seed swap and layer removal are defined on the program and count as failures for the raster baselines in the "All tasks" row, so the fair comparison is the "first three types" row. Re-prompting often makes the requested change but regenerates everything else. In all 145 successful MatLoom edits the untouched channels are byte-identical. The original benchmark compared only raster re-prompting, and its criteria are computed on the maps, so they measure exactness and not perceived quality.

We have also run the Blender-Python editing comparison: the same five edit types frozen over the 40 matched Blender-Python materials (199 tasks), each arm editing its own program under the matched backbone, judged by one before/after MLLM. Joint success is 54.8% (MatLoom) vs. 44.7% (Blender-Python), paired p = 0.057 (48 vs. 28 discordant successes), execution 98.5% vs. 92.0%, and the largest gap is on the parameter edit (widen: 45.0% vs. 17.5%). The visual judge is stricter than the map-level checks above, so the 84.3% and the 54.8% are not comparable levels. This is now App. R of the revised PDF.

AuthorsOpenReview ↗

Response to Reviewer KGqv (3/4): point 5 (channels and height treatment)

5. Comparable channels and height treatment (Q2)

"How much of the performance gap remains when comparable material channels and height treatments are used?"

We separated two things that the submission mixed together: which channels a system produces, and how a height map is used at render time.

5.1 Channels. We re-rendered the stored outputs of all four systems through one path with only the channels every system produces: base color, roughness, and normal. MatLoom gives up height, transmission, metallic, ambient occlusion, IOR, and emission; StableMaterials and IntrinsiX give up metallic; MatFuse gives up specular. Nothing was regenerated. Each entry is the shared-channel score, with the loss relative to that system's native render in parentheses (negative means the shared render scored higher).

MethodFlat BLIPFlat CLIPFlat VQAFlat judgeStaged BLIPStaged CLIPStaged VQAStaged judge
IntrinsiX28.80 (1.1)24.06 (0.2)43.09 (0.5)38.13 (0.9)19.40 (1.6)21.36 (0.4)41.25 (0.7)31.55 (0.5)
MatFuse8.23 (0.0)20.09 (0.0)30.57 (0.0)19.66 (1.9)8.61 (0.0)19.19 (0.1)30.84 (−0.3)17.41 (−0.1)
StableMaterials27.77 (0.8)25.36 (0.3)44.96 (0.4)43.82 (0.4)20.07 (1.3)23.34 (0.3)44.45 (1.1)37.92 (2.1)
MatLoom50.55 (5.5)27.74 (1.1)51.77 (2.9)66.73 (6.1)35.54 (0.6)25.60 (−0.3)52.64 (1.6)58.35 (3.2)

So the gap that remains is large: +22.8 flat and +15.5 staged BLIPScore over StableMaterials, with MatLoom ahead on every metric in both layouts and the contrasts against the baselines significant after Holm correction (p < 0.001). The extra channels do contribute. Removing them costs MatLoom 5.5 flat BLIPScore points and, on the judge, 6.1 flat (p < 0.001) and 3.2 staged (p = 0.057). No baseline loses more than 2.1.

A caution on the judge columns. They come from a new scoring session, in which StableMaterials' flat renders score about 13 points lower than in Table 1 and ours about 6 points higher, with or without the channel restriction. This is the session effect described in point 4 of the general response. Judge differences inside this table are valid, but the judge gap between systems is larger here than in the original evaluation, so we put more weight on the three deterministic metrics.

5.2 Height treatment. We rendered MatLoom with its height as bump (the StableMaterials default: normalized height, distance 0.1) and StableMaterials with its height as true displacement (midlevel 0.5, scale 0.1).

ConditionFlat BLIPFlat CLIPFlat VQAStaged BLIPStaged CLIPStaged VQA
MatLoom, displacement (native)56.0628.8054.7136.1425.3054.26
MatLoom, bump48.6027.5453.0134.9824.6053.64
StableMaterials, bump (native)28.5725.6645.3621.3323.5945.53
StableMaterials, displacement30.7925.8446.3121.2123.5946.78

Displacement helps MatLoom (7.5 flat BLIPScore, and 8.2 flat and 5.1 staged judge points within one session) and changes StableMaterials by about 2 BLIPScore and 1 judge point. With bump shading on our side and displacement on theirs, MatLoom still leads on every metric, by 19.2 flat and 16.6 staged judge points within one session. No swap changes the ranking. The full table with judge columns is Table 26. This control covers our best configuration only, and displacement at scale 0.1 is not StableMaterials' released setting, so we do not interpret its displacement row further.

AuthorsOpenReview ↗

Response to Reviewer KGqv (4/4): point 6 (quality against cost)

6. Quality–cost curves in the final metrics (Q2)

"Could the authors report quality–cost curves using the final evaluation metrics, rather than only improvements in the internal selection score…?"

Yes. Fig. 7 in the revision plots final metrics against budget, and nothing in it uses the internal selector score. The same data as tables (flat layout, means over 141 prompts and 3 runs):

6.1 Operating points

Operating pointLLM callsCandidates scoredBLIPCLIPVQAJudge
MatLoom, first program1148.3127.7053.3461.35
MatLoom, after 5 critique rounds11651.8628.2552.9367.61
MatLoom, selected program12654.1728.7153.9766.56
MatLoom, with seed search (full)126 + up to 3,00356.0628.8054.7167.01
StableMaterials, 1 sample0028.5725.6645.3657.41
StableMaterials, best of 30336.2127.0247.9162.09
StableMaterials, best of 1001042.6328.1852.37(a)
IntrinsiX, 1 sample0029.9424.2943.6156.10
IntrinsiX, best of 30340.3926.3349.2364.70
MatFuse, 1 sample008.2720.1230.6136.36
MatFuse, best of 90910.0721.7034.4745.16

Baseline best-of-N is selected by the same MobileCLIP2 scorer our pipeline uses. (a) Judged in a separate session; see 6.3.

6.2 Our seed search at truncated budgets (K seed variants per candidate, winners re-scored on final metrics)

KFlat BLIPFlat VQAFlat judge (one session)Share of wall-clock time
155.4753.9770.811%
1056.4454.1770.353%
5055.9953.9470.2213%
10056.3554.3770.9523%
25056.4254.3671.2243%
50056.3754.7170.2560%
1000 (full)56.0654.71n/a75%

The curve is flat. One variant per candidate already gives 99% of the full search's BLIPScore, and the judge stays within one point from K = 1 to 500. Larger K keeps improving the internal selector score, which is what the submission reported, but on final metrics it adds about 0.6 BLIPScore. Nearly all of the measured quality comes from the first call, the critique rounds, and a minimal search.

6.3 Paired differences against StableMaterials best-of-N (MatLoom full pipeline minus baseline)

Selection ruleFlat BLIPFlat CLIPFlat VQAFlat judgeStaged BLIPStaged CLIPStaged VQAStaged judge
Best of 3, MobileCLIP2+19.8***+1.8***+6.8***+4.9 [+0, +10]+9.9***+0.8 [−0, +2]+6.3***−1.1 [−6, +3]
Best of 10, MobileCLIP2+13.4***+0.6 [−0, +1]+2.3 [−1, +5]+15.6*** (b)+5.3 [−0, +11]−0.2 [−1, +1]+3.1 [+0, +6]+12.2*** (b)
Best of 3, per-metric oracle+14.7***+1.2*+2.7*−3.5 [−8, +1]+4.0 [−2, +10]−0.1 [−1, +1]+2.3 [−1, +5]−9.2***
Best of 10, per-metric oracle+0.9 [−5, +7]−0.4 [−1, +0]−3.5**+6.1* (b)−6.9**−1.7***−4.9***+1.2 [−3, +6] (b)

Stars: * p < 0.05, ** p < 0.01, *** p < 0.001 (two-sided Wilcoxon). Brackets: 95% bootstrap intervals (not significant). (b) Scored in a separate judge session together with our re-judged outputs.

With a deployable selector, the flat BLIPScore lead holds at ten samples, and CLIPScore, VQAScore, and staged BLIPScore become ties. The best-of-10 judge entries favor us but come from the later session discussed in point 5, so we read them next to the best-of-3 judge entries, which are ties. The oracle rows pick the baseline's best sample separately for each metric using that metric. No real selector can do this, so it is an upper bound: at ten samples, flat BLIPScore is a tie and four entries move to StableMaterials. The abstract and conclusion now state the feasible result together with this bound. Tables 28 to 30 give the same analysis for IntrinsiX and MatFuse.

6.4 Dependence on the backbone. At one call, three of the five other backbones beat StableMaterials on flat BLIPScore (by 11.6 to 12.7 points), and Gemma and Qwen fall below it (Table 27).

Two of these results changed our own view of the system. Scaling the seed search from 1 to 1,000 variants, which is where 75% of our wall-clock time goes, adds about 0.6 BLIPScore on final metrics. And the same loop writing Blender-Python executes almost as reliably as MatLoom yet produces much weaker materials, so execution rate alone says little about a program target. Both findings are now in the paper, and every limitation that Sec. 5.6 listed is still there, with the number that measures it. If part of Q2 is still open, please tell us which comparison would settle it and we will try to run it. We would be grateful if you would reconsider your rating in light of these results.

Reviewer_eastOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a compact, layer-oriented language for text-to-material generation. It also proposes a complete workflow that includes parser-guided repair, preview-based critique and revision, and seed search. On a curated benchmark of 141 prompts evaluated with six backbones, the best-performing configuration achieves higher mean scores than three diffusion baselines.

Critical strengths
  1. The paper introduces a compact, layer-oriented language for text-to-material generation, which allows users to explicitly modify material attributes. It also proposes a three-stage pipeline to generate material programs.

  2. The experiments are comprehensive, evaluating six different LLM backbones on 141 prompts and comparing against three diffusion-based baselines. The results demonstrate strong prompt alignment across multiple evaluation metrics, further supported by a human preference study.

Critical weaknesses
  1. Limited novelty: The proposed approach largely follows a standard LLM-based asset generation pipeline, where an LLM generates executable code, receives feedback from rendered outputs, and iteratively refines the code.

  2. The main distinction appears to be the introduction of a material-specific language, but the paper does not sufficiently demonstrate why this representation offers advantages.

  3. The proposed pipeline requires multiple rounds of LLM-based generation and critique, followed by an extensive noise-seed search. The diffusion baselines are not evaluated under matched computational budgets, making it difficult to assess whether the improvements come from the proposed representation / workflow or the larger inference-time budget.

Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to Reviewer east (1/2): points 1–2 (novelty, value of the representation)

Thank you for the review. We answer your three weaknesses as points 1 to 3. Point 3 is in the next comment.

1. Novelty of the pipeline (W1)

"The proposed approach largely follows a standard LLM-based asset generation pipeline…"

We agree, and the paper does not claim the loop as a contribution. Sec. 2 lists program repair, iterative critique, and rendered feedback as established components. What we propose is the language the LLM writes in: a small layer-based language in which one named field, such as a tile mask, drives coverage, color, and height together, with a standalone interpreter. The question we ask is whether this choice of target matters when the loop stays the same. The submission did not test that directly, and you were right to point it out. The revision tests it (below), and the contributions in Sec. 1 now say that the claim is about the representation and that the loop is borrowed.

Recent systems suggest that the target matters. Material Apprentice reports that generating shader graphs directly fails in about 30% of cases, and it reaches 100% execution through best-of-10 compiler sampling at about 2 USD and 30 minutes per material. VLMaterial fine-tunes a vision-language model to write Blender programs. ShaderAgent (SIGGRAPH Asia 2026) independently writes a compact validated DSL before executing in Blender. All of them surround generation with repair or validation. The size of the target language is the variable we measure.

2. Why this representation offers advantages (W2)

"…the paper does not sufficiently demonstrate why this representation offers advantages."

We now have three measurements.

2.1 Better materials from the same loop. We kept the backbone (gemini-3.6-flash), prompts, repair budget, critique rounds, selection, renderer, and metrics fixed, turned seed search off on both sides, and changed only the language the model writes: MatLoom, or Blender-Python with or without a scene scaffold. The Blender reference document in the prompt is at least as long as ours.

MatLoomBlender-Python (scaffolded)Blender-Python (unrestrained)
Runs ending with a valid render45/4543/4530/30
Flat BLIPScore47.628.815.6
Flat CLIPScore29.424.722.5
Flat VQAScore54.042.135.7
Flat judge (one session)76.547.140.4
Final renders with visible texture89%60%67%
Mean program length31.3 lines88.4 linesnot reported

Blender-Python executes nearly as often as MatLoom once the repair loop is in place, so execution rate is not the advantage. The renders are what differ: the paired gap to the scaffolded variant is +19.7 flat BLIPScore (p = 0.001) and +29.3 judge points (p < 0.001), and all 16 paired metric comparisons favor MatLoom (Table 22). This control uses one backbone and one seed.

2.2 Edits after generation. On 172 edit tasks over 40 generated materials, an edit counts as successful if the requested change is made and all other channels stay within 1/255. Diffusion baselines are re-prompted with the edit instruction and keep the best of three seeds.

MatLoom program editMatFuse re-promptStableMaterials re-prompt
Success, all 172 tasks84.3%9.9%4.1%
Success, the 94 tasks a raster model can express81.9%18.1%7.4%
Mean change on channels that should not move (of 255 levels)0.0834.868.1

2.3 Cheaper iteration. A MatLoom preview takes about 70 ms in the standalone interpreter, and a Blender attempt takes about 12.5 s. We also ran the released Material Apprentice code on 45 prompts. Both systems executed every prompt. MA used on average 112 LLM calls and a median of 61 minutes per prompt, and MatLoom used 12 calls and 4.6 minutes. MA ran in its text-only mode, which its authors did not evaluate, so we compare cost and do not headline the appearance gap (App. U).

AuthorsOpenReview ↗

Response to Reviewer east (2/2): point 3 (matched budgets)

3. Matched computational budgets (W3)

"The diffusion baselines are not evaluated under matched computational budgets…"

This is correct, and the gap is large. Per generated material:

MethodLLM callsCandidates scoredMean time per run
MatLoom (gemini-3.6-flash)12up to 3,003381 s (median 4.6 min)
IntrinsiX0044.4 s
MatFuse0012.9 s
StableMaterials000.5 s

Timings were taken on different hardware (ours on CPU with API calls, the baselines on GPUs), so they describe budgets, not a speed ranking. We ran three controls.

3.1 Our first program only. One LLM call, no critique, no seed search (new Table 2). Entries are paired mean differences, MatLoom minus baseline, over 141 prompts.

BaselineFlat BLIPFlat CLIPFlat VQAFlat judgeStaged BLIPStaged CLIPStaged VQAStaged judge
IntrinsiX+18.4***+3.4***+9.7***+5.2 [+0, +10]+10.0***+2.6***+10.7***+3.4 [−1, +8]
MatFuse+40.0***+7.6***+22.7***+25.0***+22.4***+5.1***+22.1***+19.4***
StableMaterials+19.7***+2.0***+8.0***+3.9 [−1, +9]+9.6***+0.8*+7.1***−1.3 [−6, +3]

Stars: * p < 0.05, ** p < 0.01, *** p < 0.001 (two-sided Wilcoxon). Brackets: 95% bootstrap intervals (not significant).

To your question of whether the gains come from the representation and workflow or from the budget: on BLIP/CLIP/VQAScore they are present with a single call. The judge lead over StableMaterials and IntrinsiX is not; those four entries are ties at one call, and the lead in Table 1 comes from the critique rounds. The one-call result depends on the backbone: gpt-5.6-luna, glm-5.2, and deepseek-v4-flash also beat StableMaterials on flat BLIPScore (by 11.6 to 12.7 points), while Gemma and Qwen fall below it (Table 27).

3.2 More samples for the baselines. Each baseline keeps its best of N samples, chosen by the same MobileCLIP2 scorer our pipeline uses. Entries are MatLoom full pipeline minus baseline.

BaselineFlat BLIPFlat CLIPFlat VQAFlat judgeStaged BLIPStaged CLIPStaged VQAStaged judge
IntrinsiX, best of 3+15.7***+2.5***+5.5**+2.3 [−3, +8]+9.9***+1.8***+7.2***+1.9 [−3, +7]
MatFuse, best of 9+46.0***+7.1***+20.2***+21.6***+25.4***+5.0***+20.4***+19.7***
StableMaterials, best of 3+19.8***+1.8***+6.8***+4.9 [+0, +10]+9.9***+0.8 [−0, +2]+6.3***−1.1 [−6, +3]
StableMaterials, best of 10+13.4***+0.6 [−0, +1]+2.3 [−1, +5]+15.6*** (a)+5.3 [−0, +11]−0.2 [−1, +1]+3.1 [+0, +6]+12.2*** (a)

(a) Scored in a separate judge session together with our re-judged outputs (see point 4 of the general response).

Against StableMaterials best-of-10, the flat BLIPScore lead holds, while CLIPScore, VQAScore, and staged BLIPScore become ties. The best-of-10 judge entries favor us but come from a separate session, so we read them next to the best-of-3 judge entries, which are ties. We also report an oracle that gives the baseline, for each metric, its best sample according to that metric (Table 29). No real selector can do this. Against the ten-sample oracle, flat BLIPScore is a tie and four entries move to StableMaterials (flat VQAScore −3.5; staged BLIPScore −6.9, CLIPScore −1.7, VQAScore −4.9). The abstract now states the feasible result and this bound.

3.3 How much our own search buys. We replayed the seed search at smaller budgets and re-scored on final metrics. One seed variant per candidate, in place of 1,000, gives 55.47 flat BLIPScore against 56.06 for the full search (Table 5, Fig. 7). Raising the count from 1 to 1,000 is what takes the search to 75% of our wall-clock time, and it adds about 0.6 points. Most of the candidate budget you asked about therefore buys very little on final metrics, and the revision says so.

If there is a budget-matched comparison you would find more convincing than these, we would be glad to run it during the discussion. We would be grateful if you would reconsider your rating in light of the new controls.

Reviewer_eastOpenReview ↗

Official Review

Thank you for your response. I remain concerned about the paper’s novelty and the marginal improvements under matched budgets. I therefore maintain my initial rating.

Reviewer_mrvtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MatLoom, a layered programming language for text-to-material generation. The system combines program repair, visual feedback, and noise-seed search to generate editable materials. Experiments demonstrate strong text alignment and generation quality.

Critical strengths
  1. The layered programming language supports shared spatial fields and parameter editing, improving the interpretability and controllability of generated materials.
  2. The experimental results are strong. The method achieves competitive performance on 141 prompts and receives 59.2% of preferences in the blind user study.
Critical weaknesses
  1. Direct comparisons with existing procedural material generation methods are missing. The benefits of the layered representation for generation quality and editability remain insufficiently validated.
  2. Search budgets and rendering settings differ across methods, affecting the fairness of comparisons. The ablation experiments also use inconsistent sampling settings.
Questions for authors
  1. Could the authors provide fair comparisons with procedural representations such as Blender Python and quantitatively evaluate editing success rates?
  2. Could the authors standardize search budgets and rendering settings, and re-evaluate the contributions of individual components under consistent conditions?
3 comments
AuthorsOpenReview ↗

Response to Reviewer mrvt (1/3): point 1 (Blender Python)

Thank you for the review. Both of your questions ask for specific experiments. We ran them, and the results are below and in the revised PDF. Our answers are numbered 1 to 5 across three comments: (1) Blender Python, (2) editing success rates, (3) search budget, (4) rendering settings, (5) ablations.

1. Fair comparison with Blender Python (W1, Q1)

"Could the authors provide fair comparisons with procedural representations such as Blender Python…"

We ran this as a controlled swap of the output language. Both sides use the same backbone (gemini-3.6-flash), the same 45 prompts, the same repair budget (3 retries), 5 critique rounds, and the same selection rule, renderer, and metrics. Seed search is off on both sides, so neither gets extra search. For Blender we tested two variants. In the scaffolded one, the model writes the body of a build_material function and we provide the scene. In the unrestrained one, it writes the complete script (30 prompts). The Blender reference document given to the model is 24k characters, at least as long as our DSL prompt. We tuned it on 5 pilot prompts that are excluded from all results, and we froze the protocol before running the test prompts.

1.1 Execution, size, and cost

MatLoomBlender-Python (scaffolded)Blender-Python (unrestrained)
Runs ending with a valid render45/4543/4530/30
First attempt valid(a)36/45 (80%)17/30 (57%)
Generation attempts that execute(a)93.3% of 29786.2% of 210
Final renders with visible texture (b)40/45 (89%)26/43 (60%)20/30 (67%)
Mean program size31.3 lines, 2,559 bytes88.4 lines, 3,541 bytes2,729 bytes
API cost per successful materialabout 0.08 USD (c)0.09 USD0.06 USD

(a) Per-attempt statistics were recorded for the Blender variants; a MatLoom program that parses always executes, and initial generations over the same 45 prompts parse on the first attempt in 37/45 (82%, 95% CI 69-91). (b) Pixel standard deviation of the flat render of at least 10, computed after the runs. (c) Estimated from the scaffolded variant's token rates.

1.2 Quality of the final render (means over the runs that delivered a render)

MetricMatLoomBP scaffoldedBP unrestrainedMatLoom − scaffolded, paired [95% CI]
Flat BLIP47.628.815.6+19.7 [+8.1, +31.1]
Flat CLIP29.424.722.5+4.9 [+3.4, +6.5]
Flat VQA54.042.135.7+12.6 [+6.8, +18.5]
Flat judge76.547.140.4+29.3 [+20.6, +38.3]
Staged BLIP27.67.0(d)+21.4 [+12.6, +31.2]
Staged CLIP25.020.8(d)+4.5 [+2.9, +6.2]
Staged VQA52.035.6(d)+16.9 [+12.0, +22.1]
Staged judge61.330.7(d)+30.0 [+21.7, +38.7]

All paired differences have p < 0.001 except flat BLIP (p = 0.001). The three judge columns were scored in one session. (d) The unrestrained variant renders its own scene, so it has no comparable staged render.

Three things stand out. First, execution is not where the two targets differ. With the repair loop, almost every Blender run ends in a valid render, and the paper says so. Second, the renders differ a lot. On the 43 prompts where both sides deliver, MatLoom is ahead by 19.7 flat BLIPScore and 29.3 judge points, and counting failed runs as zero only widens the gap. Many Blender materials execute but come out nearly uniform, which is what row (b) measures. Third, MatLoom programs are about a third as long, and a preview takes about 70 ms in our interpreter against about 12.5 s for a Blender attempt. That speed is what makes render-based critique affordable.

The scope is one backbone and one seed, because the second backbone did not finish before we froze scoring. App. Q states this.

We also ran the released Material Apprentice system on a 45-prompt subset (App. U, Table 32). It executed every prompt, using on average 112 LLM calls and a median of 61 minutes per prompt. MatLoom uses 12 calls and a median of 4.6 minutes.

AuthorsOpenReview ↗

Response to Reviewer mrvt (2/3): points 2–3 (editing, search budget)

2. Editing success rates (Q1)

"…and quantitatively evaluate editing success rates?"

We built an editing benchmark of 172 tasks on 40 materials. It covers five edit types (rows below). A task succeeds only if the requested change is made and every other channel stays within 1/255. The task list, tolerances, and masks were fixed before any editor ran. For MatLoom, an LLM (gpt-5.6-luna, 2.6 calls per task) edits the program. A diffusion model has no program to edit, so MatFuse and StableMaterials are re-prompted with the instruction, keeping the best of three seeds.

Success rate in %, with the requested-change rate (ignoring preservation) in parentheses.

Edit typenMatLoom program editMatFuse re-promptStableMaterials re-prompt
Roughness shift2580.0 (84.0)0.0 (84.0)0.0 (76.0)
Base-color shift3884.2 (84.2)0.0 (57.9)0.0 (78.9)
Widen a shared mask feature (e.g. grout)3180.6 (83.9)54.8 (61.3)22.6 (61.3)
Noise-seed swap4077.5 (100.0)n/an/a
Layer removal3897.4 (100.0)n/an/a
All tasks17284.3 (91.3)9.9 (36.0)4.1 (39.5)
First three types only9481.9 (84.0)18.1 (66.0)7.4 (72.3)
Mean change on channels that should not move (of 255 levels)0.0834.868.1

Seed swap and layer removal are program-level, so a regenerated map set cannot satisfy them (failures in the "All tasks" row). The fair comparison is the "first three types" row: 81.9% against 18.1% and 7.4%. Re-prompting often makes the requested change but regenerates everything else too. In all 145 successful MatLoom edits the untouched channels are byte-identical. One limit: success is measured on the material maps, so it captures exactness, not perceived quality. We also ran these edits against Blender-Python programs on the matched materials (199 tasks, one before/after judge): joint success 54.8% vs 44.7%, paired p = 0.057, execution 98.5% vs 92.0% (App. R).

3. Search budget (W2, Q2)

"Search budgets and rendering settings differ across methods…"

3.1 Removing our budget. The new Table 2 compares each baseline with our first program (one LLM call, no critique, no search). Entries are paired mean differences, MatLoom minus baseline, over 141 prompts.

BaselineFlat BLIPFlat CLIPFlat VQAFlat judgeStaged BLIPStaged CLIPStaged VQAStaged judge
IntrinsiX+18.4***+3.4***+9.7***+5.2 [+0, +10]+10.0***+2.6***+10.7***+3.4 [−1, +8]
MatFuse+40.0***+7.6***+22.7***+25.0***+22.4***+5.1***+22.1***+19.4***
StableMaterials+19.7***+2.0***+8.0***+3.9 [−1, +9]+9.6***+0.8*+7.1***−1.3 [−6, +3]

Stars: * p < 0.05, ** p < 0.01, *** p < 0.001 (two-sided Wilcoxon). Brackets: 95% bootstrap intervals (not significant).

The BLIP/CLIP/VQAScore advantage is already there at one call, so it does not come from the search budget. The judge advantage over StableMaterials and IntrinsiX does: at one call those four entries are ties, and the lead in Table 1 comes from the critique rounds (Sec. 5.3 now says this).

3.2 Giving the baselines more samples. Each baseline keeps its best of N samples, chosen by the same MobileCLIP2 scorer our pipeline uses. Entries are MatLoom full pipeline minus baseline.

BaselineFlat BLIPFlat CLIPFlat VQAFlat judgeStaged BLIPStaged CLIPStaged VQAStaged judge
IntrinsiX, best of 3+15.7***+2.5***+5.5**+2.3 [−3, +8]+9.9***+1.8***+7.2***+1.9 [−3, +7]
MatFuse, best of 9+46.0***+7.1***+20.2***+21.6***+25.4***+5.0***+20.4***+19.7***
StableMaterials, best of 3+19.8***+1.8***+6.8***+4.9 [+0, +10]+9.9***+0.8 [−0, +2]+6.3***−1.1 [−6, +3]
StableMaterials, best of 10+13.4***+0.6 [−0, +1]+2.3 [−1, +5]+15.6*** (e)+5.3 [−0, +11]−0.2 [−1, +1]+3.1 [+0, +6]+12.2*** (e)

(e) Scored in a separate judge session with our re-judged outputs (see general response, point 4).

Against StableMaterials best-of-10, the flat BLIPScore lead holds, while CLIPScore, VQAScore, and staged BLIPScore become ties. The best-of-10 judge entries favor us but come from a separate session, so we read them next to the best-of-3 judge entries, which are ties.

3.3 How much our own search buys. We replayed the seed search with 1 variant per candidate instead of 1,000 and re-scored on final metrics: flat BLIPScore is 55.47 against 56.06 for the full search (Table 5). Going from 1 to 1,000 variants takes the search from 1% to 75% of our wall-clock time and adds about 0.6 points, so nearly all of the result is available at a fraction of the budget.

AuthorsOpenReview ↗

Response to Reviewer mrvt (3/3): points 4–5 (rendering, ablations)

4. Rendering settings (W2, Q2)

"Could the authors standardize … rendering settings…"

You are right that the routes differed. Our height map drove displacement, StableMaterials' drove bump shading, the other two baselines have no height map, and only MatLoom has transmission. We re-rendered the stored outputs of all four systems through one path, using only the channels that every system produces: base color, roughness, and normal. Here MatLoom gives up height, transmission, metallic, ambient occlusion, IOR, and emission. Nothing was regenerated.

Shared channels only (means over 141 prompts and 3 seeds):

MethodFlat BLIPFlat CLIPFlat VQAFlat judgeStaged BLIPStaged CLIPStaged VQAStaged judge
IntrinsiX28.8024.0643.0938.1319.4021.3641.2531.55
MatFuse8.2320.0930.5719.668.6119.1930.8417.41
StableMaterials27.7725.3644.9643.8220.0723.3444.4537.92
MatLoom (gemini-3.6-flash)50.5527.7451.7766.7335.5425.6052.6458.35

The judge columns come from a new scoring session, so their level is not comparable with Table 1 (see point 4 of the general response).

MatLoom remains ahead on every metric in both layouts. Compared with each system's own native render, the restriction costs MatLoom 5.5 flat BLIPScore and 6.1 flat judge points, and it costs each baseline at most 2.1 on any metric (Table 25). So the extra channels help us by a measurable amount, but the lead does not come from them. We also swapped the height treatment, rendering ours as bump and StableMaterials as displacement. No swap changes the ranking on any metric (Table 26).

5. Component contributions under consistent settings (W2, Q2)

"…and re-evaluate the contributions of individual components under consistent conditions?"

You are right about the ablations as well. In the submission, the full reference ran at temperature 0.7 while the variants used the provider default, with one run per prompt. We re-ran six configurations with temperature 0.7 everywhere and three seeds each, on a stratified 40-prompt subset (720 runs, generator gpt-5.6-luna). Entries below are changes relative to the full system, with 95% bootstrap intervals.

ConfigurationFlat BLIPFlat CLIPFlat VQAStaged BLIPStaged CLIPStaged VQA
Full system (absolute)52.8327.7250.2731.2824.4949.02
w/o playbook−4.8 [−9, −0]−0.6 [−1, −0]−0.4 [−3, +3]−5.7 [−10, −2]−0.9 [−2, −0]+0.1 [−3, +3]
w/o few-shot examples−1.9 [−7, +3]−0.1 [−1, +0]+0.4 [−2, +3]−2.9 [−8, +2]−0.4 [−1, +0]+1.9 [−1, +5]
w/o render critique−3.1 [−9, +3]−1.2** [−2, −1]−0.5 [−3, +2]−1.1 [−6, +4]−0.5 [−1, +0]+2.3 [−1, +5]
w/o code critique+1.5 [−4, +7]+0.2 [−0, +1]+2.3 [−1, +6]−1.2 [−5, +3]+0.0 [−1, +1]+2.7 [−1, +7]
w/o statistics panel−5.8 [−11, −1]−0.3 [−1, +0]−1.4 [−5, +2]−7.3*** [−11, −4]−0.6 [−1, −0]−0.9 [−4, +2]
w/o code critique and statistics panel−0.8 [−6, +5]+0.6 [−0, +1]+1.6 [−2, +5]−4.3 [−9, +1]−0.4 [−1, +0]−0.1 [−4, +3]

Stars mark entries that remain significant after Holm correction (** p < 0.01, *** p < 0.001). Judge differences are omitted because the variants and the reference were judged in different sessions.

Under matched settings, the playbook and the statistics panel carry the largest effects, about 5 to 7 BLIPScore points. After Holm correction, only the statistics panel's staged BLIPScore effect and the render critique's flat CLIPScore effect remain significant, so we treat the rest as trends. Two flat-BLIPScore effects from the original table were produced by the temperature mismatch: removing the render critique went from −6.8 to −3.1, and the odd +7.0 gain from removing the code critique went to +1.5. Neither is significant now. Sec. 5.3 reports the matched results, and App. G keeps the old table, labeled as exploratory.

If you had a different procedural target in mind for Q1, such as MDL or MaterialX, please let us know and we will try to run a smaller version of the same control during the discussion. We hope these experiments answer your questions, and we would be grateful if you would reconsider your rating in light of them.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (to all reviewers)

Summary of the new experiments and the revision

We thank the three reviewers for their time and for being specific about what was missing. The reviews agree with each other, and we agree with them. The submission compared MatLoom only with diffusion models, gave our pipeline a larger inference budget and a richer rendering route, and showed editing on a single material. Sec. 5.6 listed these as limitations without measuring them. During the rebuttal period we ran the missing experiments. They are in the revised PDF (new Table 2, Appendices Q to U) and summarized here. The full tables are in each reviewer's thread.

1. New experiments

ConcernWhat we ranResultWhere
No comparison with another procedural representation (mrvt, east, KGqv)Same LLM, same repair and critique loop, same renderer; the model writes Blender-Python instead of MatLoom (45 prompts)Both almost always execute (45/45 vs. 43/45 runs). MatLoom renders score +19.7 flat BLIPScore (p = 0.001), with programs of 31 vs. 88 linesApp. Q, Tables 20–22
Closest prior system not run (KGqv)Material Apprentice, released code, 45 promptsBoth execute 45/45. MA uses about 112 LLM calls and a median of 61 min per prompt; MatLoom uses 12 calls and 4.6 minApp. U, Table 32
Editing shown on one material (mrvt, KGqv)172 edit tasks on 40 generated materials, 5 edit typesRequested change made and everything else preserved: 84.3% for MatLoom programs; 9.9% and 4.1% when re-prompting MatFuse and StableMaterialsApp. R, Table 23
Unmatched inference budget (all three)(i) our first program only: 1 LLM call, no critique, no search; (ii) best-of-N for each baseline; (iii) our seed search replayed at smaller budgets and scored on final metrics(i) ahead of every baseline on BLIP/CLIP/VQAScore, judge tied with StableMaterials and IntrinsiX; (ii) a flat BLIPScore lead of +13.4 remains against StableMaterials best-of-10; (iii) one seed variant gives 99% of the final BLIPScoreSec. 5.3, Tables 2 and 5; App. T, Tables 27–31, Fig. 7
Different rendering routes (mrvt, KGqv)All four systems re-rendered with base color, roughness, and normal only; height routing swappedMatLoom stays ahead on every metric in both layouts. Its extra channels are worth about 6 flat judge pointsApp. S, Tables 24–26
Ablations with mismatched sampling (mrvt)6 configurations re-run at temperature 0.7, 3 seeds, 40 promptsThe playbook and the statistics panel matter most (5 to 7 BLIPScore points); two effects in the old table were temperature artifactsApp. G, Table 10

2. Results that do not favor us

The revision reports these as well.

  • With one LLM call, our judge score is statistically tied with StableMaterials and IntrinsiX. The flat judge lead in Table 1 comes from the critique rounds.
  • With one LLM call, the two weakest backbones (Gemma, Qwen) fall below StableMaterials.
  • If a baseline may choose, for each metric, the best of its samples according to that same evaluation metric, several of our margins shrink or reverse. No deployable selector has this oracle; we report it as an upper bound.
  • The Blender-Python control covers one backbone and one seed.

3. Other changes to the paper

The abstract and conclusion now state the budget-matched result with its exceptions. The contributions (Sec. 1) are rewritten around the representation, since the loop is standard. Sec. 2 adds ShaderAgent and says which prior systems have released code. Each limitation in Sec. 5.6 now cites the experiment that measures it. Numbers in the original tables are unchanged, but table numbers after Table 1 have shifted because of the insertions.

4. A note on the judge metric

The MLLM judge's absolute level is not stable between scoring sessions. In later sessions, unchanged StableMaterials renders scored up to 13.8 points lower than in the original evaluation, and our re-judged outputs scored about 6 points higher (App. T). We therefore compute every judge difference between renders scored in the same session, which is why judge levels in some new tables differ from Table 1. BLIPScore, CLIPScore, and VQAScore are deterministic, so we rely on them first.

5. Human study in progress

We are running a blind study with the protocol of Sec. 5.5 that compares MatLoom with the two Blender-Python variants and with Material Apprentice. We will post the results in this thread as soon as it closes.

AuthorsOpenReview ↗

Follow-up to all reviewers: results of the second blind study

In point 5 of our general response, we said that a second blind study was running. It compares MatLoom with the two Blender-Python variants of App. Q and with Material Apprentice (MA) from App. U. The study has now closed, and the results are below. They are also in App. J of the revised PDF. These results complete the human evaluation for the procedural comparisons: mrvt point 1, east point 2, and KGqv points 1–3. We are glad to answer any remaining questions before the discussion closes.

1. Protocol

The study uses the same instrument as the first one (Sec. 5.5). In each trial, participants see unlabeled flat-layout renders for one text prompt. They pick the render that best matches the text, then rate each render's fidelity on a 1–7 scale. Display positions are balanced across methods, and trial order is randomized for each participant. An attention check sits between the two blocks. All 21 participants completed the study and passed the attention check, so none was excluded.

BlockTrialsCompared systemsChoices
A12 three-wayMatLoom vs. Blender-Python scaffolded vs. Blender-Python unrestrained (same backbone and loop, App. Q)252
B8 two-wayMatLoom (full pipeline, final program) vs. MA in text-only mode (App. U)168

Both blocks use only prompts on which the comparison system delivered a render. For MA, we further required a visibly textured render. This conditioning excludes the failures of the other systems, so it works in their favor.

2. Results against Blender-Python (block A)

MethodChoice share [95% CI]Mean fidelity (1–7)
MatLoom65.5% [59.9, 70.6]4.8
Blender-Python, unrestrained21.0% [15.5, 26.6]3.5
Blender-Python, scaffolded13.5% [9.5, 17.5]3.2

Every share differs from the 1/3 chance level (p ≤ 0.003). MatLoom is above chance, and both Blender-Python variants are below it. Within a trial, MatLoom's rating exceeds the unrestrained variant's by 1.3 points and the scaffolded variant's by 1.6 points (both p < 0.0001).

3. Results against Material Apprentice (block B)

MethodChoice share [95% CI]Mean fidelity (1–7)
MatLoom83.3% [77.4, 89.3]4.9
Material Apprentice (text-only)16.7% [10.7, 22.6]3.0

Within a trial, MatLoom's rating is 1.9 points higher (p < 0.0001).

4. How we read these results

  • Human raters agree with the automated metrics in the Blender-Python control. App. Q found that the same loop writing Blender-Python executes nearly as reliably as MatLoom (43/45 and 30/30 runs) but scores lower on every metric. Participants now show the same pattern: with two Blender-Python alternatives on screen, they chose MatLoom in about two of every three trials. This addresses the concern that the metric gaps in App. Q might not reflect what people see.
  • One result runs against the metrics. Participants slightly preferred the unrestrained variant over the scaffolded one (21.0% vs. 13.5% of choices; ratings of 3.5 vs. 3.2), whereas all four automated flat metrics rank the scaffolded variant higher. A likely factor is presentation. The unrestrained variant builds its own scene, so its images come from its own lighting setup rather than our shared flat swatch (App. Q). The difference between the two variants should therefore not be over-read. MatLoom's lead over both variants holds either way.
  • The MA comparison stays a lower bound on MA. We ran MA in its text-only mode with a transport shim, outside the image-plus-text setting its authors evaluated. The conditioning on visible renders also favors MA. Within those limits, participants chose MatLoom in five of every six trials. MatLoom also used about nine times fewer LLM calls (12 vs. about 112 per prompt) and a median of 4.6 vs. 61 minutes per prompt.
  • Scope. This study has 21 participants and 20 prompts and covers the flat layout only. Like the first study, it measures prompt-conditioned appearance preference, not editing utility or physical accuracy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.