acceptodds
Under review as a conference paper at ICLR 2027

Tuning Multimodal Large Language Model for Microstructure Generation

Abstract

Although 3D material microstructure generation has achieved notable progress, existing works typically rely on domain-specific constraints, leaving research oriented to natural languages largely unexplored. Inspired by the recent success of Large Language Models (LLMs) across diverse domains, this work introduces a multi-conditional generation framework that supports text-, sketch-, and property-driven microstructure design. The proposed method employs a Vector Quantized Variational AutoEncoder (VQ-VAE) to encode microstructural representations and a visual encoder to process sketches, allowing multi-modal alignment between text, sketch and structure within an LLM-based architecture. Moreover, to ensure that the generated microstructures satisfy desired physical properties, property constraints are incorporated into the optimization process. Experimental results demonstrate that the proposed framework can produce high-quality and semantically consistent microstructures, suggesting its potential to support more intuitive and accessible microstructure design workflows.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
3D material generation, microstructure generation, text-to-microstructure

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cmhQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multimodal inverse design framework for 3D microstructure synthesis, capable of generating designs based on textual descriptions and hand-drawn sketches. The workflow integrates a VQ-VAE quantizes 64364^3 voxel microstructures into 512 structural tokens for structural discretization, GPT-2 for generating structure tokens, and a Property-Guided Adaptation module for performance optimization.

Critical strengths

1.This paper combines textual descriptions with hand-drawn three-view sketches, offering non-expert users an intuitive and flexible approach to microstructure design, treating sketches as optional controls rather than mandatory inputs is reasonable from an interaction design perspective.

2.This paper constructs the large-scale GenText-Microstruct dataset comprising 14,000 samples and establishes a multidimensional evaluation framework incorporating FID, CLIP score, Chamfer Distance and R2R^2, providing a reference benchmark for research in this field. The authors clearly state that established benchmarks are lacking in this direction, and therefore this contribution has its own value.

3.Some reports are transparent. The author fairly discussed the inference cost and the "not strictly fair" compared to 3D-CGAN, as well as the potential losses caused by attribute optimization, thereby enhancing the credibility of the article.

Critical weaknesses

1.The significant performance drop in the "w/o Sketch" variant compared to the baseline suggests that the MLLM’s structural guidance is weaker than the baseline’s latent diffusion. Under the same evaluation criteria, pure-text LLMs without sketches underperform earlier baselines on both datasets: on Geometries 2000, FID is 97.68 vs. 77.80 for Txt2Microstruct-Net; on GenText, FID 114.17 / CD 0.1355 vs. diffusion baseline FID 84.46 / CD 0.0666. Only when sketches are included does the model surpass them (65.55, 55.47). An explanation for this performance gap is required.

2.In Table 1, several sub-optimal metrics are bolded, which undermines the clarity of the performance hierarchy. Why 3D-CGANtrained on a single class of Voronoi-based open-cell foams is the reason of lower FID? T2M-LLM's CD of 0.1352 is worse than Txt2Microstruct-Net’s 0.1162, and its Young’s modulus R2R^2 of 0.713 is also lower than 0.757,both values are still bolded.

3.The manuscript relies exclusively on quantitative metrics. Lack qualitative comparison with competing methods. Figures 6, 7, and 10 show only side-by-side generation and optimization results from T2M-LLM, with no visual comparisons against Txt2Microstruct-Net, 3D-CGAN, or diffusion baselines.

4.The core technical approach of the paper follows relatively mature methods in the cross-modal generation domain (e.g., Text-to-Image/3D), lacking significant innovation. The combination of VQ-VAE, expanded vocabulary, LoRA, and latent optimization at test time is a well-established approach in cross-modal generation. Although the authors applied the aforementioned approach to materials science, they did not propose fundamental modeling innovations specific to structures in this field, leaning more towards an engineering integration of existing techniques.

Questions for authors
  1. In the plain text setting, this method is on par or slightly superior in terms of Accuracy (0.8600 vs 0.8200) and CLIP (0.5269 vs 0.5238), but significantly lags behind in FID and CD. Do you agree that a more accurate statement would be "LLM mainly contributes semantic and category alignment rather than geometric fidelity"? If so, do the wording "Significant improvement in generation quality" in the main text and abstract need to be narrowed?

  2. If the bottleneck of generation quality is "insufficient semantic understanding", then giving the model more semantic capabilities (switching to LLM) should lead to an improvement in text-only mode, the results shows the opposite. The assumption seems not to hold.

  3. Please explain the bolding rule for the table. Is it the optimal performance value that is bolded?

4 comments

Weakness 1 & Question1 & Question 2: The main performance improvements appear to stem primarily from the sketch modality, while evidence supporting the necessity of the LLM remains insufficient.

Response: First, to better characterize the performance of text-only generation, we compare T2M-LLM with the text-based baseline on Geometries-2000. As shown in Table 1, T2M-LLM achieves higher Accuracy (0.8600 vs. 0.8200) and CLIP (0.5269 vs. 0.5238) than Txt2Microstruct-Net. However, its FID is higher (97.68 vs. 77.80). These results suggest that the text-only model provides certain advantages in semantic category alignment, while its generative distribution quality remains limited.

Table 1. Comparison between T2M-LLM and the text-based baseline under text-only conditioning on Geometries-2000.

MethodAccuracy ↑FID ↓CLIP ↑
Txt2Microstruct-Net0.820077.800.5238
T2M-LLM (Text only)0.860097.680.5269
Difference (Ours − Baseline)+0.0400+19.88+0.0031

Second, to investigate the contribution of sketch conditioning, we compare the model's performance before and after incorporating sketches. As shown in Table 2, adding sketch conditions substantially improves generation performance on Geometries-2000, particularly in terms of distribution quality and semantic alignment. This indicates that sketches provide complementary spatial and geometric priors that are difficult to capture through textual descriptions alone.

Table 2. Ablation study of sketch and property-guided adaptation on Geometries-2000.

ConfigurationAccuracy ↑FID ↓CLIP ↑
Text only0.860097.680.5269
Text + Property0.895099.740.5242
Text + Sketch0.915065.550.5463

Furthermore, as shown in Table 3, removing sketch conditioning from T2M-LLM on GenText-Microstruct leads to a substantial deterioration in all three metrics: FID increases from 55.47 to 114.17, CLIP decreases from 0.4200 to 0.3856, and CD increases from 0.0589 to 0.1355. This further demonstrates the important role of sketch conditions in improving the quality and geometric fidelity of generated structures.

Table 3. Comparison between the full model and its sketch ablation on GenText-Microstruct.

MethodFID ↓CLIP ↑CD ↓
Full T2M-LLM55.470.42000.0589
T2M-LLM w/o Sketch114.170.38560.1355
Change after Removing Sketch+58.70−0.0344+0.0766

Taken together, these results suggest that the different components play complementary roles: the pretrained language model establishes semantic mappings between textual descriptions and structural tokens; sketch conditioning provides spatial and geometric priors; and property-guided adaptation further refines the generated structures toward the target properties. Importantly, we do not attribute the overall performance gains solely to the LLM backbone. Rather, our contribution lies in integrating linguistic and visual information within a unified token-based generation framework.

Finally, we acknowledge the reviewer's suggestion to include a diffusion-based baseline under the same Text+Sketch conditioning setting. Accordingly, we provide an additional comparison with PropDiff-TMG (2026). As shown in Table 4, T2M-LLM achieves better Accuracy, FID, and CLIP scores, while PropDiff-TMG performs better in terms of CD. These results further support the advantages of T2M-LLM in semantic alignment and distribution-level generation quality, while indicating that geometric fidelity remains an area for further improvement.

Table 4. Comparison between T2M-LLM and PropDiff-TMG under Text+Sketch conditioning.

MethodAccuracy ↑FID ↓CLIP ↑CD ↓
PropDiff-TMG (2026)0.750073.550.50320.0838
T2M-LLM (Ours)0.915065.550.54630.1352

Weakness 2 & Question 3: The meaning of bolding

Response: We have bolded the optimal results and the experimental results of the methods presented in this paper. Specifically, if there are two bolded entries in the same column, it indicates that our results are not the optimal ones.

Weakness 3: The qualitative results lack visual comparisons with other methods, particularly in cases where geometric fidelity metrics are poor.

Response: We thank the reviewer for pointing out this deficiency in our qualitative experiments. We agree that presenting only the results generated by our own method is insufficient to highlight the differences between various generative approaches regarding the geometric morphology and structural details of microstructures. We have included additional qualitative comparative experiments in the revised paper.

Weakness 4: The methodological novelty mainly lies in combining existing modules, with limited material-specific modeling innovations.

Response: We appreciate the reviewer's comments and would like to further clarify the scope and contributions of our work. This study explores a multimodal conditional generation paradigm for 3D metamaterial microstructure design, aiming to provide a natural-language-driven approach to 3D microstructure generation. Rather than introducing an entirely new generative architecture, our contributions lie in the integration and adaptation of language modeling, multimodal conditioning, and property-guided optimization for this task. Specifically, they comprise three aspects:

  1. Discrete Microstructure Representation and Autoregressive Generation: We employ VQ-VAE to encode 3D voxel-based microstructures into discrete tokens, enabling a pretrained language model to learn the correspondence between textual descriptions and microstructure representations through autoregressive sequence modeling.

  2. Multimodal Conditional Control: We integrate textual descriptions and multi-view sketches into a unified generation framework, allowing the model to leverage both semantic information and spatial-structural priors. This helps alleviate structural ambiguity inherent in text-only generation.

  3. Property-Guided Latent Space Optimization: During inference, we leverage a learned property predictor to optimize the latent representation of generated structures, enabling controllable generation toward target physical properties.

We acknowledge that material-specific constraints, such as periodicity, connectivity, and manufacturability, are important for practical metamaterial design. However, these constraints are not explicitly implemented as mechanisms in the current framework, and we therefore do not claim them as established technical contributions. We have also strengthened the related-work discussion to better contextualize our approach with respect to existing LLM-based generation methods.

Thanks for the authors' response. I appreciate the effort made to help me better understand the paper.

That said, I still find the novelty of the proposed method limited, and the baselines somewhat outdated. However, I consider the additional ablation experiments provided by the authors to be meaningful, and I recommend including them in the paper or the supplementary material. I also suggest that the authors explain what the bold font denotes in the paper, since its current usage may not be easy to interpret.

I have also seen the qualitative experiments added in the revision. In the first row ("Microstructure with finely spaced perforations. It has pentagon pores."), the proposed method does show a clear improvement over Propdiff-TMG.

For these reasons, I am raising my score from 2 to 3. However, I would like to note that, in my view, this paper remains borderline. If the AC or the other reviewers lean toward rejection, I will have no objection.

Thank you for your careful consideration of our response and for raising your rating. We appreciate your recognition of our ablation experiments experiments and are glad that they have addressed most of your concerns. Thank you again for your time and constructive feedback.

Reviewer_iJ51OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a multimodal framework for 3D material microstructure generation conditioned on text descriptions, sketches, and physical properties. The method uses a VQ-VAE to convert 3D microstructures into discrete tokens and fine-tunes GPT-2 with LoRA for autoregressive generation. Multi-view sketches are incorporated through a visual encoder to provide additional geometric guidance, while a property-guided optimization strategy refines the generated structures toward target physical properties. The authors also construct a dataset for text-to-microstructure generation by extending existing data with text descriptions and sketches. Experiments on Geometries 2000 and GenText-Microstruct demonstrate improvements in generation quality and semantic alignment over selected baselines, with ablation studies evaluating the effects of different conditioning modalities.

Critical strengths
  • Practical multimodal generation framework. The paper presents a unified framework that combines text descriptions, sketches, and physical property constraints for 3D microstructure generation. This is a meaningful application of multimodal generative models to material design, allowing users to specify both semantic and geometric requirements.

  • Promising quantitative and qualitative results. The proposed method achieves improvements in FID, CLIP score, and classification accuracy over selected baselines. In particular, incorporating sketch guidance significantly improves generation quality, demonstrating the benefit of combining textual and visual conditions.

  • Property-guided refinement. The paper introduces a test-time optimization strategy to adjust generated structures toward desired physical properties. The qualitative examples show that this refinement can improve agreement with target property values while largely preserving the original structures, providing additional controllability beyond text- and sketch-conditioned generation.

Critical weaknesses
  • Limited methodological novelty and insufficient controlled comparisons. The proposed framework mainly combines existing techniques, including VQ-VAE tokenization, GPT-2 with LoRA, visual feature injection, and gradient-based property optimization. The paper does not clearly demonstrate which advantages come from using a pretrained LLM rather than a conventional generative model. In Table 2, the text-only variant performs substantially worse than the diffusion baseline in FID and CD, while most improvements come from adding sketch guidance. A comparison with a similarly conditioned diffusion model or a Transformer trained from scratch would help establish the contribution of the LLM.

  • Limited dataset scale and lack of generalization evaluation. The experiments rely on relatively small datasets, with only 1,600 training samples in Geometries 2000 and approximately 14,000 in GenText-Microstruct. Moreover, the text descriptions and sketches are largely generated from existing structures through automated pipelines, potentially limiting the diversity of training conditions. The evaluation focuses on random train/test splits without testing generalization to unseen material categories, novel structural topologies, or real user-provided text and sketches. It is therefore unclear how well the model generalizes beyond the training distribution.

  • Insufficient validation of physical properties. The property-guided optimization relies on a learned property predictor, while the evaluation of physical consistency also uses learned predictors rather than independent physical simulations. It is unclear whether the optimized structures actually satisfy the target mechanical properties or simply achieve better predictions from the surrogate model. Moreover, the paper primarily provides qualitative examples of property optimization without a systematic quantitative evaluation of property errors before and after refinement. Independent simulation-based validation would make the claims about physical controllability more convincing.

  • Missing related work. The related work could also discuss recent methods for generating complex 3D surfaces with flexible topologies, particularly those based on unsigned distance fields (UDFs). Relevant examples include Surf-D: Generating High-Quality Surfaces of Arbitrary Topologies Using Diffusion Models (ECCV 2024), UDiFF: Generating Conditional Unsigned Distance Fields with Optimal Wavelet Diffusion (CVPR 2024), and LoG3D: Ultra-High-Resolution 3D Shape Modeling via Local-to-Global Partitioning (CVPR 2026). These works provide useful context on alternative 3D representations for handling open surfaces, complex topologies, and fine geometric details.

Questions for authors
  • Contribution of the LLM. In Table 2, the text-only variant performs worse than the diffusion baseline in FID and CD, while adding sketch guidance leads to substantial improvements. How much of the improvement comes from the pretrained LLM itself? Could the authors provide a comparison with a model trained from scratch or a diffusion baseline using the same text and sketch conditions?

  • Generalization to unseen structures and user inputs. Both datasets are relatively small, and the text descriptions and sketches are largely generated from existing structures. How well does the model generalize to unseen material categories, structural topologies, or text descriptions outside the training distribution? In particular, can the authors evaluate the model using real hand-drawn sketches rather than sketches automatically derived from ground-truth structures?

  • Physical property validation. The property-guided optimization relies on a learned property predictor. Are the optimized physical properties verified through independent simulations or other ground-truth measurements? Could the authors provide quantitative property errors before and after optimization to demonstrate that the refinement consistently improves physical accuracy rather than simply fitting the predictor?

  • Clarification of the optimization formulation. In Section 3.3, Equation (7) defines zz as the decoded microstructure, while z0z_0 represents the VQ-VAE latent representation. However, Equation (8) directly penalizes the difference between zz and z0z_0. Could the authors clarify which variables are actually optimized and how this regularization term is computed?

4 comments

Weakness 1: The methodological innovation is limited, and there is a lack of controlled experiments to distinguish the contributions of the pre-trained LLM from those of other generative models.

Response: To further clarify the methodological contributions and the respective roles of different components, we have supplemented our experiments with comparisons against recent diffusion-based methods and additional ablation studies. First, we compare T2M-LLM with the recent diffusion-based method PropDiff-TMG (2026) under joint Text+Sketch conditioning. As shown in Table 1, T2M-LLM achieves higher Accuracy (0.9150 vs. 0.7500), lower FID (65.55 vs. 73.55), and higher CLIP (0.5463 vs. 0.5032). However, its CD remains higher than that of PropDiff-TMG, indicating that improved semantic alignment does not necessarily translate into better geometric fidelity. This comparison provides evidence of the effectiveness of our LLM-based framework in semantic category alignment and distribution modeling, while also highlighting its limitations in local geometric reconstruction.

Table 1: Comparison with PropDiff-TMG under joint Text+Sketch conditioning.

MethodAccuracy ↑FID ↓CLIP ↑CD ↓
PropDiff-TMG (2026)0.750073.550.50320.0838
T2M-LLM (Ours)0.915065.550.54630.1352

Second, we conduct ablation experiments on Geometries-2000 to investigate the effects of different conditioning modalities. As reported in Table 2, incorporating sketches improves Accuracy from 0.8600 to 0.9150, reduces FID from 97.68 to 65.55, and increases CLIP from 0.5269 to 0.5463. Further applying property-guided adaptation reduces FID to 64.66 and increases CLIP to 0.5486, although Accuracy decreases slightly. In contrast, text-plus-attribute conditioning improves Accuracy but does not yield consistent gains across the other metrics. These results demonstrate the substantial contribution of sketch-based structural information and the complementary role of property-guided adaptation.

Table 2: Ablation results under different conditioning settings on Geometries-2000.

ConditioningAccuracy ↑FID ↓CLIP ↑
Text only0.860097.680.5269
Text + Attributes0.895099.740.5242
Text + Sketch0.915065.550.5463
Text + Sketch + Property-guided adaptation0.905064.660.5486

Furthermore, we compare T2M-LLM with the diffusion baseline (yang2024guided) on the larger-scale GenText-Microstruct dataset. As shown in Table 3, T2M-LLM achieves better performance across FID, CLIP, and CD. Removing the sketch condition substantially degrades all three metrics, further confirming the importance of visual structural priors in the generation process.

Table 3: Comparison with the diffusion baseline and the effect of sketch conditioning on GenText-Microstruct.

MethodFID ↓CLIP ↑CD ↓
Diffusion Baseline (yang2024guided)84.460.32810.0666
T2M-LLM w/o Sketch114.170.38560.1355
T2M-LLM (Ours)55.470.42000.0589

Overall, these results suggest that the different components play complementary roles: the pre-trained LLM establishes the semantic mapping between textual descriptions and structural tokens, sketch conditioning provides spatial and geometric priors, and property-guided adaptation further refines the generated structures toward target attributes. Importantly, we do not attribute the overall performance improvements solely to the LLM backbone. Rather, our contribution lies in integrating linguistic and visual information within a unified token-based generation framework.

Weakness 2 & Question 2: The dataset size is limited, and there is a lack of validation regarding generalization to unseen classes, topological structures, and inputs from actual users.

Response: We evaluate the proposed framework on two complementary datasets with different scales and experimental objectives. Geometries-2000 contains 2,000 3D microstructure samples (1,600 for training and 400 for testing), supporting the analysis of different conditional modalities. GenText-Microstruct includes over 14,000 training samples and 2,000 evaluation samples, enabling larger-scale multimodal conditional generation and comparisons with diffusion-based methods. Both datasets provide paired textual descriptions, sketches, and corresponding 3D structures, enabling cross-modal association learning. To further examine the applicability of the framework to user-provided inputs, Figure 11 in the supplementary material presents qualitative results generated from hand-drawn sketches. These results demonstrate the framework's capability to utilize sketches created by human users as structural guidance.

Weakness 3 & Question 3: The optimization of physical properties lacks independent simulation-based verification; furthermore, no systematic error analysis or quantitative assessment comparing the results before and after optimization is provided.

Response: Our property-guided adaptation module optimizes the latent representations using feedback from a differentiable property predictor, while a latent consistency regularization term helps preserve the original structural characteristics. As reported in Table 4, incorporating this module improves FID from 65.55 to 64.66 and CLIP from 0.5463 to 0.5486, while Accuracy decreases slightly from 0.9150 to 0.9050. These results indicate improvements in distributional quality and semantic consistency, accompanied by a modest trade-off in category-level accuracy. Furthermore, due to the practical constraints of large-scale simulation, we provide qualitative comparisons before and after property-guided optimization in Figure 7 of the main paper and Figure 12 of the supplementary material. The visual results show observable structural changes, with volume fraction (VF) exhibiting encouraging optimization trends. Together, the quantitative and qualitative results provide complementary evidence of the effects of property-guided adaptation.

Table 4: Ablation results under property-guided adaptation optimization on Geometries-2000.

ConditioningAccuracy ↑FID ↓CLIP ↑
Text only0.860097.680.5269
Text + Sketch0.915065.550.5463
Text + Sketch + Property-guided adaptation0.905064.660.5486

Weakness 4: The "Related Work" section omits methods for generating complex 3D surfaces based on UDFs (Unsigned Distance Fields).

Response: We appreciate the reviewer for highlighting these relevant advances in UDF-based 3D generation. We have extended the Related Work section to discuss three representative methods: Surf-D (ECCV 2024), the first latent diffusion framework based on UDF representations for generating surfaces with arbitrary topologies; UDiFF (CVPR 2024), which learns optimal wavelet filters to compress UDF representations and performs conditional diffusion in the joint spatial-frequency domain; and LoG3D (CVPR 2026), which introduces a local-to-global UDF modeling architecture and pushes 3D VAE reconstruction resolution to 2048³. While existing works primarily focus on generating complex 3D surfaces and geometric representations, this paper centers on the generation of 3D metamaterial microstructures using voxel-based representations, incorporating multi-conditional control via text, sketches, and target physical properties. Consequently, we clarify the distinctions between these two categories of research—specifically regarding task objectives and structural representations—and expand our literature review to avoid overly broad generalizations about the current state of 3D generation research.

Question 1: To what extent is the performance improvement attributable to the pre-trained LLM itself? Could you provide a comparison with a diffusion-based baseline model using the same text and sketch conditions?

Response: As reported in Table 2, under text-only conditioning, T2M-LLM achieves an accuracy of 0.8600, outperforming Txt2Microstruct-Net (0.8200), and yields a modest improvement in CLIP, although it does not achieve better FID. This suggests that the pre-trained LLM contributes to capturing semantic relationships between textual descriptions and structural categories, but semantic understanding alone does not necessarily translate into improved geometric fidelity. With sketch conditioning, T2M-LLM achieves substantial improvements in accuracy, FID, and CLIP, highlighting the importance of visual structural priors in complementing the spatial information conveyed by text. To provide a more direct comparison under multimodal conditions, we further include PropDiff-TMG (2026), a recent diffusion-based method, using both text and sketches as inputs. As shown in Table 1, T2M-LLM achieves better accuracy, FID, and CLIP, while PropDiff-TMG performs better in terms of Chamfer Distance (CD). These results suggest that the proposed framework benefits from combining language-based semantic modeling with visual structural guidance, while geometric fidelity remains an area for further improvement.

Question 4: Clarification of the optimization formulation.

Response: We thank the reviewer for pointing out this ambiguity. We optimize the continuous VQ-VAE latent z\mathbf{z}, not the decoded microstructure. Here, z0\mathbf{z}_0 is the initial latent, and z\mathbf{z} is its optimized counterpart. Decoding and prediction are defined as x=Ds(z)\mathbf{x}=D_{\mathrm{s}}(\mathbf{z}) and pred=Pprop(x){\mathbf{pred}}=P_{\mathrm{prop}}(\mathbf{x}). We have revised Equations (7) to clarify the notation and align the formulation with the implementation.

Thanks for the detailed explanation! That helps me understand the work better. I will keep my score.

Thank you for your careful consideration of our response and for maintaining your positive assessment. We are glad that the rebutal have addressed your concerns.

Thank you for your suggestions for improving our paper. We will incorporate these changes into the final proof.

Reviewer_kf18OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed T2M-LLM, a unified LLM-based framework for inverse design of 3D metamaterial microstructures conditioned on natural language descriptions, hand-drawn sketches, and target property constraints. The authors also extend existing datasets, construct relevant benchmarks. Extensive experiments demonstrate that T2M-LLM can achieve significant improvements over existing methods.

Critical strengths
  1. The research question has practical value and the overall framework is coherent with clearly defined module responsibilities.
  2. Designing a novel multimodal material inverse design paradigm to synthesize 3D microstructures from textual descriptions and hand-drawn sketches.
  3. Extending existing datasets and establishing a large-scale benchmark for text-to-microstructure generation.
Critical weaknesses
  1. This paper claims to use a “multimodal large language model”, but the underlying model actually used is GPT-2 (released in 2019). Meanwhile, the baselines listed in Table 1 are 3D-CGAN (2023) and Txt2Microstruct-Net (2024). They are severely outdated for a submission to ICLR 2027.
  2. In Appendix A.2, the inference time of the proposed model is approximately 7.3 seconds per sample, while the base method (VAE+MLP) is approximately 2.9 seconds per sample. The inefficient inference poses a significant bottleneck for application. This paper lacks discussion on how to mitigate this issue.
  3. In Table 1, the authors explicitly state that the 3D-CGAN baseline was trained on a single class of Voronoi-based open-cell foams. What about other methods? How to ensure a fair comparison?
Questions for authors
  1. Why choose GPT-2 rather than other more powerful modern LLMs? May be you can observe better scalability or reasoning capabilities when using larger backbone networks.
  2. Why choose the 3D-CGAN model (Zheng et al., 2023) and Txt2Microstruct-Net (Zheng et al., 2024) as the comparison baselines? Are there any more recent methods that could serve as baselines for comparison?
  3. How to ensure the fairness of the comparison mentioned in the Weaknesses.
  4. Given the inefficient inference, the authors can discuss how to further accelerate inference and improve the practical usability. The answers for these questions in the rebuttal are crucial to the final decision.
3 comments

Weakness 1: The backbone used in the model is somewhat outdated, and the baseline methods lack contemporary relevance.

Response: First, we would like to clarify that the primary contribution of this work is not the development of a generative model tied to a specific backbone, but rather the exploration of a multimodal conditional generation paradigm that integrates discrete 3D microstructure tokens into an LLM-based sequence modeling framework. By representing 3D microstructures as discrete tokens within a unified sequence space, the framework establishes a connection between natural language semantics and structural representations. Furthermore, incorporating sketch-based conditioning enables 3D microstructure generation guided jointly by textual descriptions and visual sketches. GPT-2 is adopted as an initial backbone to demonstrate the feasibility of this paradigm, rather than as an indispensable architectural component. The framework is inherently backbone-agnostic and can, in principle, be extended to more advanced language models, such as LLaMA and Qwen.

To address the concern regarding the recency of the baselines, we further include a comparison with the recent diffusion-based method PropDiff-TMG (2026) under the joint Text+Sketch conditioning setting:

Table 1: Comparative results based on the diffusion baseline

MethodAccuracy ↑FID ↓CLIP ↑CD ↓
PropDiff-TMG (2026)0.750073.550.50320.0838
T2M-LLM (Ours)0.915065.550.54630.1352

As shown in Table 1, T2M-LLM outperforms PropDiff-TMG in Accuracy, FID, and CLIP, suggesting its advantages in semantic category alignment and generative distribution modeling under joint text and sketch conditioning. Nevertheless, its higher CD indicates that geometric fidelity remains an area for improvement. These results provide encouraging evidence for the effectiveness of the proposed LLM-based multimodal generation paradigm, while also highlighting its current limitations.

Weakness 2: The model's inference speed is relatively slow, and there is a lack of discussion regarding inference efficiency.

Response: T2M-LLM first encodes each 3D microstructure into 512 discrete tokens and then generates the structural tokens autoregressively. Consequently, its inference cost is mainly attributed to the sequential token generation process, whereas the VAE+MLP baseline directly maps inputs to outputs without autoregressive decoding. As reported in Sec. 4.2, T2M-LLM requires approximately 7.3 seconds per sample, compared with 2.9 seconds for VAE+MLP. This additional overhead arises from autoregressive generation and multimodal conditioning. To improve inference efficiency, we have discussed three potential optimization strategies in Sec. 4.2:

  1. Autoregressive acceleration: Employ KV caching and speculative decoding to reduce redundant computations.
  2. Token sequence compression: Reduce the number of VQ-VAE tokens while preserving essential geometric information.
  3. Conditional feature caching: Reuse encoded text and visual features to avoid redundant computations.

Table 2: Experiments on GenText-Microstruct

MethodFID ↓CLIP ↑CD ↓
Diffusion Baseline84.460.32810.0666
T2M-LLM (Ours)55.470.42000.0589
T2M-LLM w/o Sketch114.170.38560.1355

Table 3: Experiments on Geometries-2000

MethodAccuracy ↑FID ↓CLIP ↑
Txt2Microstruct-Net0.820077.800.5238
Text (Ours)0.860097.680.5269
Text + Attributes (Ours)0.895099.740.5242
Text + Sketch (Ours)0.915065.550.5463

Weakness 3: Inconsistencies in training data and task configurations across different baselines may compromise the fairness of the experimental comparisons.

Response: First, on the GenText-Microstruct dataset, we compare T2M-LLM with the diffusion-based baseline, Guided (2024). Both methods use the same dataset and evaluation protocol, which comprises over 14,000 training samples and 2,000 evaluation samples. Therefore, the results in Table 2 provide a consistent comparison under the same data and evaluation settings. Second, for the Geometries 2000 dataset, all methods reported in Table 1 and Table 3 are evaluated under the same experimental setup, ensuring consistency when comparing the effects of different conditioning strategies. Finally, as clarified in the corresponding table captions, 3D-CGAN is trained on a single class of Voronoi-based open-cell foams. Its results are therefore provided only as a task-specific reference to illustrate the performance of generation methods under different experimental settings, rather than as a directly comparable baseline for quantitative ranking against multi-class methods.

Question 1: Why choose GPT-2 instead of a more powerful, modern LLM? Do larger backbones offer better scalability and reasoning capabilities?

Response: Our work explores a multimodal conditional generation paradigm that maps textual and visual conditions, together with 3D microstructure representations, into a unified discrete token space, enabling structural generation through an LLM-based sequence modeling framework and a VQ-VAE decoder. GPT-2 is selected as an initial backbone because its pre-trained language representations are sufficient for the semantic modeling requirements of this task, while its moderate model size helps control the computational overhead of autoregressive generation. In addition, LoRA enables parameter-efficient domain adaptation, which is particularly suitable for the limited availability of material data.

Question 2: Why were 3D-CGAN and Txt2Microstruct-Net chosen as baselines? Are there more recent methods available for comparison?

Response: We selected 3D-CGAN and Txt2Microstruct-Net because they represent established approaches to 3D microstructure generation and text-conditioned microstructure generation, respectively, providing relevant reference points for evaluating our method. To strengthen the comparison with recent approaches, we further included PropDiff-TMG (2026) under the joint Text+Sketch conditioning setting. We also incorporated the results of yang2024guided on the GenText-Microstruct dataset. Specifically, Table 1 presents the comparison with PropDiff-TMG, while Table 2 reports the comparison with yang2024guided, where T2M-LLM achieves better performance in FID, CLIP, and CD. These additional experiments provide a broader evaluation across different datasets and conditioning settings.

Question 3: How is the fairness of the experiments?

Response: We ensure experimental consistency primarily in terms of datasets and evaluation protocols. For the comparison between T2M-LLM and yang2024guided on GenText-Microstruct, both methods are evaluated using the same dataset and evaluation protocol, enabling a direct comparison of FID, CLIP, and CD, as reported in Table 2. On Geometries-2000, the experiments in Table 3 employ a consistent experimental setup to investigate the effects of different conditioning modalities. Regarding 3D-CGAN, its training data consists exclusively of a single category of Voronoi-based open-cell foams, which differs from our multi-category generation setting. Therefore, we include its results only as a task-specific reference rather than as a directly comparable baseline for quantitative ranking.

Question 4: How can inference be further accelerated to enhance practical utility?

Response: The main computational bottleneck arises from the autoregressive generation of 512 structural tokens, which requires sequential decoding. Several strategies could improve inference efficiency. First, KV caching and speculative decoding can reduce redundant computations during autoregressive generation. Second, optimizing the VQ-VAE latent representation could reduce the number of tokens required while preserving essential geometric information. Finally, caching text and visual conditional features can avoid repeated encoding computations. Attribute-guided optimization can also be selectively enabled according to task requirements, reducing unnecessary computational overhead. These strategies provide promising directions for improving the practical efficiency of T2M-LLM.

As demonstrated in Tables 1–3 of the supplementary material, GPT-2-based T2M-LLM achieves competitive semantic alignment, although its geometric fidelity remains inferior to that of some specialized generative methods. Importantly, GPT-2 serves as a backbone for validating the proposed framework rather than a fundamental architectural limitation. Larger and more advanced LLMs, such as LLaMA and Qwen, could be incorporated into the same framework to explore potential improvements in semantic understanding, scalability, and generation quality.

Dear Reviewer kf18,

Thank you again for your follow-up comments. We have updated our responses to address the points you raised. We would greatly appreciate it if you could take another look at our updated responses and additional experiments. Please let us know if any concerns remain or if further clarification would be helpful.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We sincerely thank the Area Chair (AC) and the reviewers for their valuable feedback. We respectfully highlight the post-rebuttal discussion outcomes as follows:

  • We have further clarified the methodological contributions of this work and added comparisons with recent generative approaches (addressing comments from reviewers kf18 [W1, Q1, Q2], iJ51 [W1, Q1], and cmhQ [W1, Q1, Q2, W4]). We emphasize that the core contribution of T2M-LLM lies in integrating discrete 3D microstructure tokens into a multimodal generative framework based on a Large Language Model (LLM). This approach is not dependent on any specific backbone network, nor do we attribute performance gains solely to the LLM itself. In Table 5 of the revised paper, we have included a comparison with PropDiff-TMG (2026) under "Text+Sketch" input conditions. The results show that T2M-LLM performs better in terms of accuracy (0.9150 vs. 0.7500), FID (65.55 vs. 73.55), and CLIP score (0.5463 vs. 0.5032), although a gap remains regarding the CD metric (0.1352 vs. 0.0838). Furthermore, on the GenText-Microstruct dataset, T2M-LLM outperforms the "Guided" (2024) method in terms of FID (55.47 vs. 84.46), CLIP score (0.4200 vs. 0.3281), and CD (0.0589 vs. 0.0666), which further validates the effectiveness of our proposed method.

  • We have conducted a more in-depth analysis of our existing conditional ablation experiments to clarify the roles of different modalities and their complementarity (addressing comments from iJ51 [W1, W2, Q1, Q2] and cmhQ [W1, Q1, Q2, W3]). On the Geometries-2000 dataset, the introduction of the sketch condition improved accuracy from 0.8600 to 0.9150 and reduced the FID score from 97.68 to 65.55. On the GenText-Microstruct dataset, removing the sketch condition caused the FID to rise from 55.47 to 114.17, while the CLIP score dropped from 0.4200 to 0.3856. These experiments demonstrate that the geometric priors provided by sketches are complementary to the semantic understanding capabilities of Large Language Models (LLMs). Furthermore, we have added experimental results using real hand-drawn sketches in Figure 11 of the supplementary material, which showcase the model's generalization ability.

  • We have clarified the mathematical formulation of "property-guided optimization" and included additional quantitative ablation experiments (addressing comments from iJ51 [W3, Q3, Q4]). We explicitly state that the optimization variables are the continuous latent representations of the VQ-VAE (rather than the decoded microstructures) and have revised the relevant formulas accordingly. Additionally, new quantitative experiments show that property-guided optimization improves the FID from 65.55 to 64.66 and the CLIP score from 0.5463 to 0.5486. By comparing visualizations before and after optimization (see Figure 8 in the main text and the supplementary material), we further illustrate the impact of this optimization process on the generated structures, thereby confirming the effectiveness of the optimization module.

  • We have refined our experimental descriptions and expanded the discussion on related work (addressing comments from kf18 [W2, W3, Q3, Q4], iJ51 [W4], and cmhQ [W2, Q3]). We clarify that 3D-CGAN was trained exclusively on a single category (Voronoi open-cell foam structures); it therefore serves as a task-specific reference rather than a multi-category baseline suitable for direct, broad-spectrum comparison. Furthermore, we have addressed the computational overhead inherent in autoregressive generation by proposing several acceleration strategies, including KV caching, speculative decoding, token sequence compression, and conditional feature caching. We have also specified the criteria for the bolded entries in our tables, added qualitative comparative analyses, and expanded the "Related Work" section to include methods based on Signed Distance Fields (SDFs)—such as Surf-D (ECCV 2024), UDiFF (CVPR 2024), and LoG3D (CVPR 2026).

Finally, we thank the reviewers for their further evaluation of our responses. Reviewer cmhQ acknowledged our additional ablation studies and qualitative comparisons (noting that T2M-LLM outperforms PropDiff-TMG in certain examples) and subsequently raised their score from 2 to 3. Reviewer iJ51 expressed satisfaction with our clarifications and maintained their original assessment. We believe these additional experiments, analyses, and revisions have further strengthened our research. We once again thank the AC and all the reviewers for their valuable suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.