acceptodds
Under review as a conference paper at ICLR 2027

Vision-Language Models as End-to-End Combinatorial Optimization Solvers

Abstract

Large language models (LLMs) have shown promise for automating combinatorial optimization (CO), yet existing approaches rely on textual representations of problem instances. Many real-world CO problems are naturally presented in visual forms, and text-based encodings may lead to spatial information loss and limit spatial reasoning capabilities. In this paper, we introduce a novel framework that empowers vision-language models (VLMs) to serve as end-to-end CO solvers by directly processing visual problem instances. We design visual-attributed instances that encode problem structures and basic features into images, and develop selfevolutionary visual prompt optimization to automatically discover effective visual designs. Our two-stage training strategy first uses supervised fine-tuning (SFT) to learn solution patterns from domain-specific solvers, followed by reinforcement learning to reduce constraint violations and improve solution quality. Additionally, we introduce inference-time visual augmentation that exploits structure-preserving transformations to generate diverse candidate solutions. Experiments on six NPhard CO problems show that our method achieves 100% feasibility, with optimality gaps ranging from 1.78% to 10.93%. It improves average feasibility over advanced reasoning models, including GPT-6-Astra and Gemini-2.5-Pro, by 26.5% and 44.3%, respectively, while also outperforming specialized heuristics.

Primary area:
optimization
Keywords:
Vision-Language Models, End-to-End, Combinatorial Optimization

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_eD5DOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the application of vision-language models to end-to-end combinatorial optimization, covering six problem types: TSP, CVRP, MIS, MVC, PFSP, and JSSP. The proposed method renders structured problem instances as images and optimizes their visual representation via evolutionary search. It employs supervised fine-tuning and reinforcement learning based on verifiable rewards for the Qwen2.5-VL-7B model, while enhancing solution quality through multi-candidate sampling and visual augmentation. Experimental results demonstrate effective constraint satisfaction capabilities.

Critical strengths
  1. The research direction is significant, and the task coverage is broad. Directly mapping visual inputs to combinatorial optimization solutions is a valuable area of exploration; the framework encompasses routing, graph optimization, and scheduling tasks without requiring architectural modifications for different problems.
  2. The training process is supported by empirical evidence. Table 1 demonstrates that reinforcement learning significantly improves feasibility rates across multiple tasks—for instance, the CVRP feasibility rate rose from 62% (with SFT) to 88%, reaching 100% when combined with the "Best-of-8" strategy—while ablation studies confirm the practical utility of visual prompts.
  3. The experimental scope is comprehensive. In addition to general-purpose models and traditional heuristics, the study incorporates text-based solvers, various VLM backbones, joint training, and standard benchmarks, facilitating a better understanding of the method's applicability.
Critical weaknesses
  1. There are inconsistencies between the numerical values and the metric definitions regarding key results. In Table 2, the average BoN gap for large-scale MIS is 3.29%, yet the Gap@1/@5/@10 figures are 84%, 94%, and 100%, respectively; mathematically, these proportions imply an average gap of less than 1.94%, meaning the two sets of figures are mutually incompatible. Furthermore, while §3 defines the gap of infeasible solutions as infinite, Table 1 reports finite mean values for methods with feasibility rates below 100%, and the evolutionary evaluation in Equation (6) adopts this same metric; the actual calculation method requires clarification.
  2. The independent benefits of the core contribution are not sufficiently substantiated. The ablation study on visual prompts primarily compares "with prompts" versus "without prompts" but lacks comparisons against manual designs or random search under equal computational budgets, making it difficult to confirm that the gains stem from the evolutionary search process itself. Additionally, Table 3 shows that the visual-based approach yields higher gaps than the text-based approach across all six tasks (e.g., 5.97% vs. 1.07% for TSP); consequently, its value requires further justification through an analysis of the trade-off between efficiency and solution quality.
  3. Conclusions regarding out-of-distribution (OOD) generalization lack clear support. The "OOD Uniform" and "OOD Gaussian" distributions mentioned in Appendix E.7 utilize parameters that already appear in the training data generation settings in Appendix A.1.2, meaning they cannot serve as valid evidence for OOD generalization. The scope of the standard benchmarks is also limited, with gaps exceeding 30% on larger JSSP instances.
  4. There are conflicting descriptions regarding data and training settings that hinder reproducibility. The main text states that 500,000 SFT instances are used per task, whereas Appendix C.2 specifies 50,000; furthermore, Appendix A.2.2 sets the CVRP capacity equal to the average customer demand, a configuration under which customers with demands exceeding the average cannot be served. The actual data generation and training configurations need to be clarified.
Questions for authors

See Weaknesses.

3 comments

Q1:Thank you for your question, which helps us further improve the paper. We apologize for the error in our statistical analysis. We have carefully rechecked all other results in the paper to prevent similar errors, rerun the evaluation, and repeated the analysis across instance-size ranges (10–30, 40–60, and 70–100 nodes). The corrected results are shown below:

Ours (BoN)

Node rangeMean gap (%)Std. (%)Gap < 1%Gap < 5%Gap < 10%Max gap (%)
10–301.3253.5240 (87%)40 (87%)42 (91%)12.50
40–602.2384.0222 (73%)23 (77%)28 (93%)13.33
70–1003.3185.6815 (62%)18 (75%)20 (83%)18.75
All2.0774.3077 (77%)81 (81%)90 (90%)18.75

Ours (VA)

Node rangeFeasible instancesMean gap (%)Std. (%)Gap < 1%Gap < 5%Gap < 10%Max gap (%)
10–3046 (100%)1.333.5240 (87%)40 (87%)42 (91%)12.50
40–6029 (97%)2.324.0721 (72%)22 (76%)27 (93%)13.33
70–10020 (83%)3.245.1912 (60%)15 (75%)16 (80%)16.67
All95 (95%)2.034.1173 (77%)77 (81%)85 (89%)16.67
We have corrected this in the revised manuscript.

Q2: Thank you for this insightful question. “Without visual prompts” refers to the original, manually designed images, while “with visual prompts” refers to the evolved images. Please refer to our response to Reviewer s9yG for detailed explanations and experimental results. In particular, Experimental Section 5.5 provides a detailed analysis of how visual evolution reduces constraint violations across problems. Figure 12 in Appendix C.1 of the revised manuscript also illustrates examples of the image evolution process for CVRP, MIS, and JSSP. In Section 5.5, we compare the performance of our visual solver with that of the text-based solver. Although our visual solver has a slight gap in solution quality, it achieves faster inference. Some experimental results are also provided in our response to Reviewer s9yG. We hope these clarifications address your concerns.

Q3: Thank you for your question. Since our training data combine multiple distributions, we also evaluate our model on purely uniform and Gaussian distributions. In addition, we introduce a new out-of-distribution test set with a mixed distribution, combining uniform and clustered characteristics by sampling half of the nodes uniformly and the other half from the clustered distribution. The results are shown below:

ProblemMethodOOD (Clustered): FeasibilityOOD (Clustered): GapOOD (Mixed): FeasibilityOOD (Mixed): Gap
V-TSPOurs (BoN)100%5.83%100%6.34%
V-TSPOurs (VA)100%5.57%100%6.02%
V-CVRPOurs (BoN)100%10.71%100%9.85%
V-CVRPOurs (VA)98%10.66%100%9.37%
We have revised Appendix E.7 accordingly.
To explain the challenges posed by the larger JSSP instance ta21 (20 × 20), we first summarize the instances of different sizes below:
InstanceSizeNumber of operationsOptimal makespan
---------:---:
ta0115 × 152251231
ta1120 × 153001357
ta2120 × 204001642
As the numbers of jobs and machines increase, the space of possible machine-wise job orderings grows factorially, as (n!)m(n!)^m, where nn and mm denote the numbers of jobs and machines, respectively. Visual models face an additional challenge under a fixed pixel budget: in a 20 × 20 JSSP Gantt chart, each operation occupies a smaller pixel area, while information density and output length increase substantially. Results from other algorithms further indicate that larger instances are more challenging not only for visual solvers but also for text-based solvers and heuristic methods.

Q4: Thank you for your questions and suggestions. Our approach uses two-stage learning: SFT followed by RL. During SFT, 500,000 instances per problem are used for training, and “5,000” refers to the separate visual self-evolution training stage. We have corrected this in Appendix C.2 of the manuscript.

For CVRP, vehicle capacity is determined by multiplying the mean customer demand by a random factor:

Q=⌊r⋅qˉ⌋,r∼Triangular(3,6,25)Q = ⌊r · q̄⌋, r ∼ Triangular(3, 6, 25)

where q̄ is the mean demand across all customers in the instance, and r follows a triangular distribution with a lower bound of 3, a mode of 6, and an upper bound of 25. The floor operation rounds the result down to the nearest integer. This setting is identical to that used in the CVRP dataset for the text-based solver.

We have updated the Appendix C.2 of the revised manuscript accordingly and highlighted the changes in blue.

Thank you for your response. I will maintain a positive score.

Thank you for your response and maintain a positive score. We deeply appreciate the time and effort you have devoted to reviewing our work. If you have any additional questions or need further clarification, please let us know. We will do our best to address them promptly. Thank you again for your thoughtful feedback.

Reviewer_s9yGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work first creates a combinatorial optimization QA dataset. It also proposes a training method that enables vision-language models (VLMs) to achieve the same feasibility rates as their textual counterparts.

Critical strengths

The baselines include both frontier LLMs and problem-specific methods. The proposed method demonstrates good performance against several of these baselines.

Critical weaknesses

Value of evolutionary renderer search: While the authors demonstrate that adding visual cues improves training results, this does not isolate the benefit of evolutionary search over reasonable fixed cues. It would be meaningful to compare a renderer with manually specified cues against an evolved renderer initialized from those same cues.

Failure attribution for the VLM baselines: It would be insightful for the authors to analyze the failure modes of VLM baselines. Distinguishing errors would help identify which capabilities the proposed training method improves.

Practical benefit of the VLM reasoning approach: In Table 3, the text-based solver achieves better solution quality. The authors are encouraged to test a two-step approach consisting of a) extracting structured data from each image and b) passing it to the trained text-based solver. Comparing this pipeline with direct visual solving would help assess whether the end-to-end approach offers an advantage in solution quality or runtime.

Scalability: The routing and graph experiments are limited to at most 100 nodes. How does performance change on substantially larger instances, where visual crowding and longer input/output sequences may become bottlenecks? Comparing visual, text-based, and conventional solvers across increasing instance sizes would clarify the method’s practical limits.

Questions for authors

N/A

3 comments

Q1:Thank you for your question. We apologize for the unclear wording. In our paper, “without visual prompts” refers to the manually designed initial images, while “with visual prompts” refers to the images obtained through visual evolution. In the revised manuscript, we present the image evolution process in Appendix C.1 and provide a detailed performance comparison between the original and evolved images, along with an analysis of the reductions in constraint violations, in Section 5.5. Some experimental results are shown below:

ProblemWithout visual prompts: Gap (%)FeasibilityWith visual prompts: Gap (%)Feasibility
CVRP16.54 ± 11.883%9.07 ± 6.3100%
TSP6.28 ± 3.9100%5.97 ± 4.1100%
MIS2.23 ± 4.2100%2.08 ± 4.3100%
MVC2.05 ± 3.4100%1.98 ± 3.1100%
PFSP2.15 ± 3.0100%1.85 ± 2.5100%
JSSP12.63 ± 11.0100%10.93 ± 8.9100%

Q2: Thank you for this suggestion, which helps us improve the paper. Figure 14 in Appendix E.2 presents “Statistics on Constraint Violations by baseline GPT-6-Astra.” The specific constraint violations of the GPT-6-Astra baseline across the six problems are summarized below::

ProblemConstraint violationCount
TSPnot_a_permutation17
TSPtour_not_closed10
TSPnode_out_of_range6
TSPno_route_parsed4
TSP******37
CVRPcapacity_exceeded12
CVRPcustomer_unvisited10
CVRPcustomer_visited_twice3
CVRProute_not_closed3
CVRPno_routes_parsed2
CVRP******30
MISendpoints_share_edge2
MISno_set_parsed1
MIS******3
MVCedge_not_covered18
MVCno_set_parsed5
MVC******23
FSSPinvalid_permutation2
FSSPno_order_parsed1
FSSP******3
JSSPmachine_order_violated45
JSSPinvalid_machine_sequence14
JSSPwrong_machine_count6
JSSPno_schedule_parsed5
JSSP*****70

These results show that the observed constraint violations align with the issues targeted by our RL reward design and further motivate the heuristic cues incorporated into our visual representations.

Q3: Thank you for your suggestions. In Table 3, both solvers use the same underlying instances from a shared structured numerical dataset, supporting a fair comparison. Figure 3 further compares token consumption and inference time. The result is as follows:

Inference Time

MethodCVRPTSPMISMVCPFSPJSSP
Text-based solver11.7 s11.3 s4.0 s7.1 s3.1 s20.7 s
Ours (SFT+RL+BoN)9.4 s8.6 s3.8 s6.3 s2.5 s15.1 s

Token Counts: CVRP

Input modality20 nodes40 nodes60 nodes80 nodes100 nodes
Text1,1472,2713,3924,5145,640
Image1,0561,0561,0561,0561,056

Token Counts: MIS

Input modality20 nodes40 nodes60 nodes80 nodes100 nodes
Text5821,4852,4824,0505,836
Image7741,0981,2451,4421,722

Although our visual solver has a slight gap in solution quality, it achieves faster inference. This gap may arise from two factors. First, image evolution remains constrained by a manually designed framework rather than fully autonomous LLM-based design. Second, CVRP images provide no explicit numerical coordinates, requiring the VLM to infer customer locations visually, potentially introducing perception errors. In particular, our visual solver could potentially eliminate explicit numerical representations altogether by encoding problem information visually. For example, customer demands in CVRP could be represented through colors or node sizes.

Q4: As the number of nodes increases, the problems become more challenging, and Table 2 shows that our visual solver’s performance gradually declines. Similar trends are observed for the text-based and traditional solvers. Our model is trained on instances with 10–100 nodes and has not been trained on larger instances. We apologize that time constraints currently prevent us from conducting additional training at larger scales. As discussed in our response to Reviewer 7u26, our approach differs from text-based solvers:1)Different input modalities: Directly uses images without requiring structured numerical data or explicit numerical coordinates.2)Different model backbones: Uses VLMs rather than text-only LLMs.3)Different training pipelines: Combines visual representation design with SFT and GRPO optimization of VLMs.4)Different inference mechanisms: Generates solutions through visual perception and multi-view inference.5)Different application scenarios: Targets planning problems naturally represented as images and difficult to describe in text.

Thank you for your response. I decided to maintain my score.

Thank you for your response. We deeply appreciate the time and effort you have devoted to reviewing our work. If you have any additional questions or need further clarification, please let us know. We will do our best to address them promptly. Thank you again for your thoughtful feedback.

Reviewer_QTE6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper moves beyond traditional LLM methods for solving combinatorial optimization (CO) problems using textual descriptions of the problems to the use of VLMs with deliberately constructed visual inputs to represent routing, graph optimization, and scheduling problems. They develop an iterative, generative process for optimizing the appearance and structure of their visual prompts. With the new input structure they finetune Qwen2.5-VL-7B-Instruct with LoRA using a two-step pipeline of (1) supervised learning on expert solutions and (2) a reinforcement learning objective function that is designed to optimize both constraint following and optimality. Multiple semantics-preserving views of the same problem can be used, allowing for the selection of the most feasible candidate. They provide stronger or comparable performance to other prompting methods, with a main benefit of consuming substantially less running time and inference tokens, demonstrating that VLMs have the necessary reasoning capabilities to approach these problems

Critical strengths
  • This paper provides evaluation over Routing (V-TSP/V-CVRP), Graph optimization (V-MIS/V-MVC), and Scheduling (V-JSSP/V-PFSP) problems, totalling six optimization problems, providing a broad evaluation of combinatorial optimization problems.
  • The individual components (development of optimal visual prompting, Supervised/reinforcement learning, and inference time optimization) are isolated with different ablation studies (optimal visual prompting in the main body, others in Appendix E) to analyze the impact of each component.
Critical weaknesses
  • Section 3, line 144: “instances with infeasible solutions contribute a gap of ∞.” However, Table 1 reports non-infinite average optimality gaps for methods with feasibility rates below 100%, which would theoretically be driven to infinity. It is unclear how infeasible solutions are handled in these averages, which makes it difficult to parse the comparisons to other methodologies that support the claims of the paper.
  • While the trend in the current paper’s results is strong, the current implementation of the methodology only uses one base model (Qwen2.5-VL-7B-Instruct) with fine-tuning applied. It would be more informative to see how a wider range of models, or potentially a more recent model (although I acknowledge that January 2025 is not necessarily old) would respond to the same techniques.
Questions for authors
  • How are infeasible solutions handled when calculating the average optimality gaps in Table 1? Section 3 states that infeasible instances contribute a gap of ∞, yet methods with feasibility rates below 100% still have finite average gaps. Were infeasible instances excluded, capped, or assigned another value? Without clarification, how can the reported comparisons and the paper’s conclusions be considered valid?

  • How generalizable are the results when the methodology is evaluated using only one base model, Qwen2.5-VL-7B-Instruct, with fine-tuning? Could the observed improvements be specific to this model? Would the conclusions remain consistent across models of different sizes, architectures, and generations?

3 comments

Q1: Thank you for raising this issue. We compute the average optimality gap over instances with feasible solutions:

Gap(B)=Σ[(O(q,x^)−O(q,x∗))/∣O(q,x∗)∣]/NGap(B)=Σ[(O(q,x̂)-O(q,x*))/|O(q,x*)|]/N

Here, the sum is taken only over instances with feasible solutions, N is the number of such instances, O(q,x̂) is the objective value of the generated solution, and O(q,x*) is that of the reference solution.

We have revised the definition in Section 2 and highlighted the corresponding passage in blue.

Q2: We primarily train our solver on Qwen2.5-VL-7B-Instruct to enable a fair comparison with the text-based solver built on Qwen2.5-7B-Instruct. Due to space limitations in the main text, we previously reported evaluations of multiple backbone models under the same framework in Appendix E.6. The results are shown in the table below:

Qwen3-VL-8B

ProblemSFT: FeasibilitySFT: Gap (%)Full: FeasibilityFull: Gap (%)
TSP78%8.57 ± 5.3100%6.51 ± 4.7
CVRP65%13.52 ± 8.3100%8.63 ± 6.1
MIS65%0.84 ± 2.5100%2.03 ± 4.0
MVC69%1.85 ± 3.2100%2.16 ± 3.2
PFSP100%3.36 ± 2.8100%1.98 ± 2.4
JSSP73%15.78 ± 10.3100%9.03 ± 8.0

InternVL3-8B

ProblemSFT: FeasibilitySFT: Gap (%)Full: FeasibilityFull: Gap (%)
TSP75%8.06 ± 3.7100%6.03 ± 3.9
CVRP65%15.07 ± 11.6100%9.38 ± 6.8
MIS70%1.39 ± 2.292%1.95 ± 4.1
MVC73%1.56 ± 3.3100%1.83 ± 2.5
PFSP100%4.39 ± 3.6100%2.24 ± 2.9
JSSP66%15.93 ± 13.596%9.17 ± 8.8

We have presented these results and provided a detailed discussion in experimental section 5.6, “Generalization across VLM Backbones.” We hope these answers help address your concerns, and would appreciate your reconsideration of the rating. We are happy to clarify any remaining questions.

I look forward to further discussion with you and ensure that all your questions and concerns have been fully addressed. Below, I also summarize our responses and the additional results provided during the rebuttal period.

1. Clarifications on Q1 and Q2, backbone evaluations, and motivation

We have addressed your questions Q1 and Q2 and reported evaluations of multiple backbone models under the same framework. We have also revised the Introduction to clarify the motivation for our work and provided detailed explanations of its motivation and potential application scenarios in our responses.

2. Additional experiments on the contribution of visual prompts

We have provided additional experimental results to address your concerns about the contribution of visual prompts.

ProblemWithout visual prompts: Gap (%)Without visual prompts: FeasibilityWith visual prompts: Gap (%)With visual prompts: Feasibility
CVRP16.54 ± 11.883%9.07 ± 6.3100%
TSP6.28 ± 3.9100%5.97 ± 4.1100%
MIS2.23 ± 4.2100%2.08 ± 4.3100%
MVC2.05 ± 3.4100%1.98 ± 3.1100%
PFSP2.15 ± 3.0100%1.85 ± 2.5100%
JSSP12.63 ± 11.0100%10.93 ± 8.9100%

3. Additional out-of-distribution evaluations

We have also provided new out-of-distribution (OOD) evaluation results on clustered and mixed distributions to further assess the generalization of our visual solver.

ProblemMethodOOD (Clustered): FeasibilityOOD (Clustered): Gap (%)OOD (Mixed): FeasibilityOOD (Mixed): Gap (%)
V-TSPOurs (BoN)100%5.83100%6.34
V-TSPOurs (VA)100%5.57100%6.02
V-CVRPOurs (BoN)100%10.71100%9.85
V-CVRPOurs (VA)98%10.66100%9.37

4. Comparisons with the text-based solver

We have further clarified the differences between visual and text-based solvers and discussed the trade-offs between solution quality and inference efficiency. The following tables summarize the inference times and input token counts.

Inference time

MethodCVRPTSPMISMVCPFSPJSSP
Text-based solver11.7 s11.3 s4.0 s7.1 s3.1 s20.7 s
Ours (SFT+RL+BoN)9.4 s8.6 s3.8 s6.3 s2.5 s15.1 s

Input token counts: CVRP

Input modality20 nodes40 nodes60 nodes80 nodes100 nodes
Text1,1472,2713,3924,5145,640
Image1,0561,0561,0561,0561,056

Input token counts: MIS

Input modality20 nodes40 nodes60 nodes80 nodes100 nodes
Text5821,4852,4824,0505,836
Image7741,0981,2451,4421,722

5. Further clarifications and appendix updates

We have resubmitted the revised manuscript, having corrected errors, added further details, and provided a more thorough explanation of the visual evolution process in Appendix C.2.

Thank you again for your time and thoughtful feedback. We welcome any further questions and would be happy to address any remaining concerns.

With only one day remaining in the rebuttal discussion period, I look forward to further discussion with you. Our responses to Q1 and Q2 provide detailed explanations regarding feasibility rates and experimental results across different backbone models. We have also resubmitted the revised manuscript. We hope this evidence resolves your main concerns and would be grateful if you could consider raising your rating accordingly. If any concern remains, we would welcome the opportunity to address it.

Reviewer_7u26OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper fine-tunes Vision-Language Models for solving combinatorial optimization problems (COPs) in an end-to-end manner. The overall framework consists of several key components, such as visual instance rendering, visual prompt optimization, supervised learning and reinforcement learning. Specifically, it convert structured COP instances into visual representations and employ an LLM-guided evolutionary strategy to optimize the visual designs. The VLM is first trained on solutions generated by specialized solvers through supervised fine-tuning, followed by reinforcement learning with rewards based on constraint satisfaction and solution quality. At inference time, the authors further introduce visual augmentation to generate diverse candidate solutions. Experiments are conducted on six COPs across routing, graph optimization, and scheduling. The results show that the proposed approach achieves high feasibility rates and generally outperforms general-purpose VLMs and several traditional heuristics, although it still falls behind its text-based counterpart in solution quality.

Critical strengths

The paper testifies that end-to-end COP solving is possible by VLMs, while end-to-end COP solving itself represents an interesting problem setting.

There is a relatively broad empirical evaluation, which evaluates the method across different dimensions, such as experiments on different backbones, different stages, unified solve and OOD distributions.

Critical weaknesses

W1. A major concern is that the paper does not clearly establish why CO problems should be solved in the visual domain. All problem instances are originally available as structured numerical data and are artificially rendered into images. This introduces an additional perception step, potentially losing precise information, without demonstrating a clear benefit.

W2. The paper provides limited insight into what the visual modality actually contributes to optimization. The proposed visual prompts encode handcrafted problem-specific features, but it is unclear whether the improvements come from visual reasoning or simply from providing useful heuristic information.

W3. The proposed framework closely follows Jiang et al. (NeurIPS 2025), which already introduced end-to-end CO solving through supervised fine-tuning, reinforcement learning, and Best-of-N inference. The present work largely replaces textual instance representations with visual inputs while retaining the same training pipeline, reward design, and evaluation tasks. The main additions are evolutionary visual prompt optimization and multi-view augmentation, but their independent contributions are not convincingly established.

W4. The proposed VLM-based solver consistently underperforms the existing text-based end-to-end LLM solver across all six COPs. This raises concerns about the actual benefits of introducing visual representations for CO solving. Moreover, the baseline comparisons are insufficient, as the paper mainly considers general-purpose VLMs and relatively simple traditional heuristics, while lacking comparisons with stronger NCO methods or an image-to-structured-data pipeline combined with specialized solvers.

Questions for authors

What is the actual advantage of solving combinatorial optimization problems directly in the visual domain, rather than recovering their structured representations and applying existing optimization methods?

What specific benefits come from the visual representation itself rather than the additional heuristic information?

13 comments

W1: Thank you for your suggestions and questions. We understand your concerns about VLMs as end-to-end solvers. This work explores a new perspective by using VLMs as end-to-end CO solvers that generate solutions directly from images, without requiring structured numerical inputs or textual descriptions of problem instances. We aim to demonstrate their ability to produce high-quality CO solutions and their potential for broader visual planning tasks, such as urban route planning and indoor navigation. This framework may offer a visual approach to solving problems in other domains. To clarify this motivation, we have clarified this motivation in the revised Introduction and highlighted the corresponding passages in blue.

W2:We develop a purely visual CO solver that takes images as input, accompanied only by brief image descriptions and problem objectives, without structured numerical instance data. Designing effective visual representations is challenging, as different designs can substantially affect performance. Since evolving complete problem images from scratch is difficult, we initialize with manually designed images and refine them through self-evolving visual prompt optimization, reducing reliance on expert design. We have added experimental explanations in Section 5.5 and and presented the evolution process of the original images in Appendix C, “Visual Self-Evolution Training.”

W3:Thank you for this insightful question. We apologize for not clearly explaining how our approach differs from text-based solvers. First, Jiang et al. (NeurIPS 2025) demonstrate the effectiveness of a text-based solving approach that relies on natural-language datasets, whereas our method uses image-based datasets and therefore operates in a different modality. Second, their approach uses text-only LLMs, while ours builds on VLMs, requiring a different training setup. Our pipeline spans visual representation design, image-based input, supervised fine-tuning of VLMs, GRPO-based visual optimization, and multi-view inference. This setting introduces additional challenges compared with text-based solving, as the model must perceive and interpret visual information to generate solutions. In practical scenarios, many problems are naturally presented as images, and their spatial or structural information can be difficult to express in text. Our goal is to provide an approach to visual planning and validate the effectiveness of this pipeline. For example, our CVRP images do not provide explicit numerical coordinates. We have also revised the Introduction to clarify the advantages of visual solvers.

W4: Although our visual solver may underperform text-based solvers in solution quality, it offers advantages in token consumption and inference time. This performance gap may stem from limitations in the VLM’s reasoning capabilities and our visual representation design. We expect further advances in model capabilities and fully self-evolving visual representations to improve the performance of our visual solver. We also compare our method with specialized NCO solvers on uniformly distributed instances with 100 nodes, using LKH3 as the reference. The optimality gaps are reported below: TSP:

MethodOptimality gap (%)
LKH30.00
AM4.26
POMO1.49
POMO × 80.63
Ours7.06

CVRP:

MethodOptimality gap (%)
LKH30.00
AM (greedy)6.07
POMO3.10
POMO × 81.05
Ours9.38
The key distinction is that these NCO solvers rely on problem-specific network architectures, often requiring architectural redesign to address different problems. In contrast, our visual solver takes images as input and generates solutions across different problems through a fully data-driven approach, reducing the need for expert knowledge. Moreover, these NCO solvers rely on structured datasets and lack the ability to solve problems directly from images.

Q1: We have clarified the differences between our approach and other methods, along with its advantages, in our responses to W1, W3, and W4 and in the Introduction of the revised manuscript. We hope these clarifications address your concerns.

Q2: In the experimental section, we compare solution performance using the original and evolved images and analyze how image evolution reduces constraint violations across different problems. Both the original and evolved images are provided in Appendix C, “Visual Self-Evolution Training.”

ProblemConstraint violationSFT w/oSFT w/ΔFull w/oFull w/Δ
TSPMissing nodes1110−1000
TSPDuplicate nodes2017−3000
CVRPCapacity exceeded4329−14130−13
MISEdge conflicts4540−5000
MVCUncovered edges3925−14000
PFSPMissing or duplicate jobs000000
JSSPPrecedence violated2116−5000

I thank the authors for the response. My concerns have been partially addressed.

The authors mentioned that the movation of introducing visual modality is "demonstrate their (VLMs) ability to produce high-quality CO solutions and their potential for broader visual planning tasks, such as urban route planning and indoor navigation. This framework may offer a visual approach to solving problems in other domains." However, I didn't see the application to specific "visual planning tasks" and "solving problems in other domains". The evidence that the trained model can solve similar real-world problems (e.g., urban route planning) is also not given. Revising claim or introdcution cannot address the concern about the motivation.

Prior work has demonstrated that LLMs can solve COPs in an end-to-end manner, so it is not suprising that VLMs can also achieve that. Given that the performance of VLMS is even weaker than LLMs, the performance is also not convincing enough.

Given that the motivation is not sufficiently strong, I incline to maintain my evaluation.

Thank you for the reviewers' response. We apologize that our previous response did not fully address your concerns. We would like to further clarify the fundamental differences between our VLM-based solver and text-based solvers.

First, the input modalities are fundamentally different. Text-based solvers take natural language descriptions as input, whereas our VLM-based solver directly takes visual images as input. These are different types of solvers.

Second, text-based solvers still rely on explicit numerical information, whereas our visual solver operates directly on graphical representations of problem instances without requiring explicit numerical data, such as node coordinates. This distinction also leads to different application scenarios and potential use cases.

Third, although our solver may still slightly underperform text-based solvers in terms of solution quality, it offers greater efficiency in both token consumption and inference time. As illustrated in Fig. 3, the token consumption of text-based solvers grows approximately geometrically with increasing problem size, whereas that of our visual solver grows linearly or even sublinearly. This makes our approach particularly promising for solving large-scale optimization problems, where the token overhead of text-based approaches can become a significant bottleneck.

Furthermore, our multi-view visual input demonstrates strong generalization and robustness. As shown in our experiments, our solver maintains relatively stable performance under different image transformations, with only slight variations in feasibility rates, and some transformations can even improve its performance. In contrast, text-based solvers can be considerably more sensitive to variations in textual prompts, potentially resulting in substantial performance fluctuations. This highlights an additional advantage of our visual approach in terms of robustness to input representations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.