acceptodds
Desk rejected at ICLR 2027

IRest-VL: Decoupled Multimodal Diagnosis and Programmatic Restoration for Infrared Images

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Infrared images can suffer from composite degradations, in which multiple noise and blur primitives act in specific orders. Existing end-to-end methods address diagnosis or restoration alone while agents that perform both require multi-round planning and heavyweight tools with coarse or fixed parameters, making them ill-suited to lightweight infrared applications. We introduce IRest-VL, a unified multimodal large language model (MLLM) that diagnoses types, severities, and order while generating a lightweight, standalone restoration program. Building it poses two obstacles: efficiently constructing code supervision in a large mixed discrete–continuous space of tools, program orders, and parameters, and avoiding task interference and the weak compositional generalization that would arise if diagnosis and code were serialized into a single autoregressive target. We address them with experience-accumulated Bayesian optimization (EABO) and a decoupled structured-generation architecture that models degradation through local attributes and directed transitions, rather than serializing diagnosis into restoration-code generation. EABO constructs supervision 27.3 faster than iterative MLLM search, without requiring MLLM calls, while yielding higher downstream restoration SSIM. On synthetic composites applied to real infrared backgrounds, structured diagnosis raises exact diagnosis accuracy on held-out out-of-distribution (OOD) compositions from 7.11% under the direct autoregressive (AR) diagnosis-and-restoration baseline to 28.11%, while maintaining comparable restoration quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Infrared Image Restoration, Multimodal Large Language Models, Programmatic Restoration, Bayesian Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dcuoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

IRest-VL jointly performs composite infrared degradation diagnosis and generates executable restoration programs. It constructs program supervision through experience-accumulated Bayesian optimization, which warm-starts TPE with previously successful configurations. A shared multimodal backbone supports two decoupled outputs: structured degradation attributes and transitions, and restoration code generated without a serialized diagnosis prefix. Experiments compare output designs across five backbones and evaluate diagnosis, restoration, supervision-construction efficiency, and program execution.

Critical strengths
  1. The task formulation is useful and clearly motivated. Combining degradation diagnosis with explicit, parameterized restoration programs provides interpretable outputs and allows generated programs to execute without repeated MLLM calls.

  2. The decoupled design has consistent empirical support. Matched comparisons across five backbones show improved aggregate OOD diagnosis and shorter outputs while largely maintaining restoration quality.

Critical weaknesses
  1. Compositional generalization remains very limited for longer pipelines. Almost all OOD diagnosis gains come from two-degradation cases. For three-degradation OOD inputs, structured models achieve only 0–0.13% exact accuracy.

  2. Adding diagnosis leaves restoration quality nearly unchanged. A structured diagnosis-only baseline and separate ablations of the prediction head and valid-path decoder would better isolate the contribution.

  3. Evaluation is limited to controlled synthetic degradations. Cross-dataset reuse retains the same controlled degradation pipeline. It does not establish robustness to unknown real-sensor artifacts or changing acquisition conditions.

Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer dcuo

Thank you for your insightful comments and helpful suggestions.

W1: We thank the reviewer for highlighting this issue. It is worth noting that full diagnosis accuracy is a strict conjunctive metric. For L=3, all three primitive types, three severities, two directed transitions, and the complete order must match the reference; any local error makes the full prediction incorrect. Attribute and transition F1 instead give credit for partially correct local predictions.

For IRest-VL-Qwen3.5-0.8B on OOD L=3, attribute F1 is .7260 and transition F1 is .3805, whereas exact full accuracy is .0013. Thus, local degradation evidence has not disappeared, but errors compound when the complete path is reconstructed. The valid-path score sums local log probabilities, equivalently multiplying their probabilities, so exact joint agreement becomes increasingly difficult as the pipeline grows.

The low OOD full accuracy at L=3 does not mean that the proposed method is ineffective. First, IRest-VL achieves higher attribute and transition F1 than Direct AR for both L=2 and L=3. Second, its full diagnosis accuracy is also higher than Direct AR at L=2. Third, although both methods obtain nearly zero full accuracy at L=3, IRest-VL still achieves higher attribute and transition F1. This comparison shows that exact full diagnosis at L=3 is difficult for both approaches rather than being a failure specific to IRest-VL. Appendix H.1/Table 14 reports the complete length-specific results.

W2: Following your suggestions, we added all three analyses.

W2.1 Diagnosis-only branch. We have added a row to Table 6 that removes the restoration-code generation head from IRest-VL. Diagnosis only obtains ID/OOD full accuracy of .6186/.2512, whereas diagnosis plus restoration obtains .6193/.2811, indicating a modest OOD benefit rather than task interference after decoupling.

IRest-VL settingID/OOD Full ↑ID/OOD SSIM ↑
Restoration only--/--.8575/.7954
Diagnosis only.6186/.2512--/--
Diagnosis + restoration.6193/.2811.8575/.7951

W2.2 Diagnosis-head architecture. We have added a new diagnosis-head ablation in Appendix H.4.1/Table 19. Under the same backbone, training data, and decoding protocol, it compares MLP, MSA+MLP, 5Query, 25Query, and MSA+5Query. MLP applies masked-average pooling followed by the shared MLP and task-specific prediction heads, without the MSA block. MSA+MLP adds MSA before pooling. In 5Query, one learnable query per degradation type extracts type-specific features; 25Query adds one query for each directed transition; MSA+5Query first applies multi-head self-attention to the visual tokens.

First, among the three heads without MSA (i.e., MLP, 5Query, 25Query), 5Query has the strongest overall diagnosis: it leads full accuracy and transition F1 on both ID and OOD splits while remaining competitive on attribute F1; MLP and 25Query each surpass it on attribute F1 in only one split. Second, adding MSA improves both corresponding heads, showing that contextualizing the visual tokens before prediction is beneficial. Third, once MSA is included, MSA+MLP and MSA+5Query are broadly comparable and trade off small gains across diagnosis and restoration, suggesting limited additional benefit from the richer query structure after contextualization. We therefore use MSA+MLP as the default configuration.

Diagnosis headID/OOD F1_aID/OOD F1_tID/OOD FullID/OOD SSIM
MLP.8582/.7194.5414/.5216.6060/.2452.8573/.7927
5Query.8512/.7495.5920/.5329.6133/.2705.8567/.7929
25Query.8537/.7511.5561/.5041.5953/.2478.8568/.7947
MSA+MLP.8713/.7547.5660/.5601.6193/.2811.8575/.7951
MSA+5Query.8659/.7548.5904/.6059.6239/.2704.8572/.7917
AuthorsOpenReview ↗

Response to Reviewer dcuo (following)

W2.3 Valid-path decoding. We have added a new ablation in Appendix H.4.2/Table 20 that compares the complete decoder with two alternatives. “Allow repeated primitives” removes the non-repetition constraint while retaining a maximum length of 5. “Fixed-start greedy” first uses independent attribute argmax to determine the set of present primitives and assigns each primitive its most likely severity, without adding or removing primitives through global scoring. It then selects the node with the largest sum of outgoing transition probabilities as the single starting point and repeatedly chooses the highest-probability unvisited successor until all primitives are covered.

Allowing repeated primitives reduces ID/OOD full accuracy from .6193/.2811 to .1003/.0000, showing that the non-repetition constraint is important for producing structurally valid paths. Fixed-start greedy decoding obtains .5309/.1608. Although it performs better than the repeated-primitive variant, it remains substantially below complete valid-path decoding. These results show that both the non-repetition constraint and global enumeration with joint attribute-transition scoring contribute to OOD full diagnosis.

DecoderID/OOD F1aF1_aID/OOD F1tF1_tID/OOD Full
Full valid path.8713/.7547.5660/.5601.6193/.2811
Allow repeated primitives.8179/.7110.5519/.5145.1003/.0000
Fixed-start greedy.8172/.7015.4983/.5336.5309/.1608

W3: We additionally collect 100 real noisy infrared images: 44 from JFAD, 5 from the TNO, and 51 from the TRNeRF. Each image contains stripe noise, random noise, blur, or a combination of these degradations. We compare the methods using MUSIQ and CLIP-IQA.

MethodMUSIQ ↑CLIP-IQA ↑
Original image26.4093.3506
Guided filter (best single tool)22.2888.3640
GLM-5.227.0529.3464
Qwen3.8-Max27.2750.3376
GPT-5.6-sol29.2612.3587
Claude Sonnet 527.9741.3463
Direct AR Qwen3.5-0.8B28.3252.3607
IRest-VL-Qwen3.5-0.8B28.0313.3720

On these real images, IRest-VL obtains the highest CLIP-IQA (.3720), while GPT-5.6-sol obtains the highest MUSIQ (29.2612). The two no-reference metrics favor different methods, but IRest-VL is among the strongest on both and outperforms the guided filter on both metrics. Direct AR and IRest-VL have similar restoration scores because the decoupled architecture primarily targets OOD diagnosis generalization and diagnosis--restoration interference. Appendix K/Fig. 7 provides qualitative examples in which IRest-VL identifies plausible degradation components and adapts the selected lightweight tool and its arguments across images. This added evaluation provides evidence on observed real artifacts, while the limited public data do not cover all degradation combinations and application orders.

With the deadline approaching, we just wanted to gently ask if our response has resolved your concerns or if there are any remaining questions. We would be glad to provide further clarifications if needed.

Reviewer_HfxUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the composite degradation of multiple noises and blurs in infrared images, which are superimposed in a specific sequence. It proposes IRest-VL: a multimodal large model that simultaneously diagnoses the type, severity, and application sequence of the degradation, and generates a standalone lightweight repair program. The program consists of 8 classic operators or lightweight network tools, with image adaptive parameters. After generation, it does not require calling the large model again.

Critical strengths
  1. The problem setting has practical significance, and the output is both interpretable and deployable. Unlike end-to-end networks that only output repaired images, or MLLM intelligent agents that require multiple rounds of planning and heavy tools, IRest-VL outputs explicit tools, execution sequences, and numerical parameters. Once the program is generated, it can run independently of the large model, with an execution time of approximately 0.04 seconds, which is very suitable for the edge deployment scenarios of infrared systems.

  2. The motivation for decoupling design is clear, and the OOD diagnosis improvement is consistent across multiple backbones. The author did not directly present the architecture but instead used data to illustrate the serialization diagnosis and the problems in the code: removing any task would make the other task better, indicating that they interfere with each other; only 22.43% of the optimal search programs exactly reversed the degraded order, indicating that the diagnostic prefix imposes unreliable order priors on code generation. Based on this, the proposed decoupling solution has improved the accuracy of complete OOD diagnosis on five different backbones, while reducing the number of output tokens and inference latency by 25% to 58%, and eliminating the measurable loss in SSIM repair.

Critical weaknesses
  1. The empirical gain of the core contribution is relatively weak. Compared to the ordinary BO, EABO only increases the optimal program SSIM from 0.8442 to approximately 0.8505; the so-called 27.3-fold acceleration is the comparison object of an already expensive LLM iterative search; compared with MCTS, it only leads slightly. In terms of the method, it is essentially a warm start with retrieval. Its novelty is limited. The decoupled diagnostic head is actually a classifier on visual tokens, but the paper only compares it with autoregressive text diagnosis, without comparing it with a regular CNN or ViT multi-label classifier with the same legal path decoding. Therefore, it is impossible to determine whether the OOD improvement comes from the multimodal large model itself, or merely from the decompositional modeling.

  2. This is not the template for ICLR 2027.

Questions for authors

See Critical Weaknesses

5 comments
AuthorsOpenReview ↗

Response to HfxU

Thank you for your insightful comments and helpful suggestions.

W1: We answer the first weakness in four aspects.

W1.1 Significance of EABO's SSIM improvement: We respectfully believe that this result may have been somewhat misunderstood. Below, we provide three points to clarify it.

First, the aggregate (original Table 17) includes L=0 identity samples, for which both methods obtain 1.0 and no search improvement is possible. Excluding L=0, EABO improves mean best-program SSIM from .8269 to .8340 (+.0071). The gain is concentrated on composite degradations: +.0103 at L=2 and +.0058 at L=3.

Second, infrared denoising studies commonly report SSIM gains on the order of .001--.01 at a noise level of 25 [1--3]. In our setting, the average PSNR values for L=1, L=2, and L=3 are approximately 28, 26, and 25 dB, respectively. An SSIM gain of +.0071 is therefore not negligible.

Third, we additionally compare initialization counts of 0 (Vanilla BO) and 7 on the 100x dataset. Each source image corresponds to 100 degradation instances, increasing both the search-space coverage and sample diversity, while the trial budget remains 40. Vanilla BO therefore has greater difficulty finding high-quality programs within the budget. Excluding L=0, EABO improves .8292 to .8408 (+.0116); at L=2 and L=3, the gains are +.0176 and +.0116. This result shows that reusing historical programs becomes more valuable when constructing larger and more diverse supervision sets.

ScaleSearchL=0L=1L=2L=3Avg. w/ L=0Avg. w/o L=0
2xVanilla BO1.0000.8979.8059.7769.8442.8269
2xEABO1.0000.9029.8162.7827.8505.8340
2xGain.0000+.0050+.0103+.0058+.0063+.0071
100xVanilla BO1.0000.8982.8111.7781.8463.8292
100xEABO1.0000.9040.8287.7897.8567.8408
100xGain.0000+.0058+.0176+.0116+.0104+.0116

These scale- and length-specific results have also been added to Appendix H.4.3/Table 21.

[1] Jiang, Nanhe, et al. "An Infrared Image Denoising Method Based on Wavelet Dual-Branch Diffusion Model." IEEE Transactions on Geoscience and Remote Sensing 63 (2025): 1-13.

[2] Liu, Jinyuan, et al. "Enhancing Infrared Vision: Progressive Prompt Fusion Network and Benchmark." Advances in Neural Information Processing Systems 38 (2026): 96850-96875.

[3] Li, Pu, et al. "Expandable, Compressible, Mineable: Open-World Thermal Image Restoration." arXiv preprint arXiv:2605.16967 (2026).

W1.2 EABO vs LLM and MCTS: We understand the reviewer's assessment, but the comparison results do not fully support the characterization that EABO only slightly outperforms MCTS. Compared with iterative MLLM search, EABO is substantially faster, achieves a similar best-program SSIM, and incurs no token cost.

As reported in Table 7, EABO reduces construction time relative to MCTS from 2.6058 to 1.5863 s/sample (39%) and raises best-program SSIM from .8384 to .8505 (+.0121). As discussed in W1.1, an SSIM gain of this magnitude is not negligible. EABO therefore achieves a better accuracy-efficiency balance than MCTS by improving search and downstream restoration quality while requiring less time and no MLLM calls.

AuthorsOpenReview ↗

Response to HfxU (following)

W1.3 Novelty and contribution: We thank the reviewer for the rigorous assessment of methodological novelty. We clarify the contribution at the problem-setting and method levels.

Problem setting and analysis. (1) Existing restoration methods typically address either degradation diagnosis or image restoration, rather than both in a unified model. We formulate a unified perception-to-action task in which a single MLLM infers the ordered combination and severities of composite degradations and generates a parameterized, executable restoration program. The program is lightweight and standalone, making the formulation relevant to edge deployment. (2) Constructing supervision for this task requires search in a mixed discrete--continuous program space. Existing iterative MLLM search and MCTS provide an unsatisfactory speed--quality trade-off in this space, motivating an efficient strategy that reuses verified experience across samples. (3) We identify two model-level obstacles in direct autoregressive diagnosis-and-code generation: task interference and weak compositional diagnosis on unseen degradation pipelines. Together, these challenges define a reproducible setting for studying structured MLLM output and compositional generalization.

Method contribution. (1) EABO reuses cross-sample, execution-verified programs for mixed discrete-continuous restoration-program search. To our knowledge, cross-sample experience reuse remains relatively limited in BO and has not been systematically studied for restoration-program search. Although the retrieval rule is simple, it formalizes severity similarity and historical program quality to warm-start the current search without additional MLLM calls. (2) The model decomposes ordered-sequence prediction into primitive-attribute multilabel classification and directed-transition binary classification, then reconstructs a legal path through log-probability-based valid-path decoding. The novelty lies in this decomposition and reconstruction design for infrared composite-degradation diagnosis rather than in simply replacing one classifier with another.

W1.4 Comparison to CNN or ViT: The core contribution of this work is to unify composite-degradation diagnosis and executable restoration-program generation. The structured diagnosis head is an auxiliary branch in the decoupled architecture, designed to avoid interference between diagnosis and code generation rather than to propose a standalone diagnosis classifier. Comparing only diagnosis accuracy with a CNN or ViT would instead evaluate diagnosis-head architectures and would not provide an end-to-end comparison on the unified task, because these classifiers cannot generate executable restoration programs.

The effectiveness of decomposition is evaluated through a controlled comparison. Direct AR and IRest-VL share the same MLLM backbone, pretrained initialization, input, training data, program supervision, and optimization settings; the only change is whether structured decomposition is used. Table 2 shows consistent OOD improvements across different backbones. These results can support that the OOD improvement comes from the decomposed modeling design.

W2: We rechecked the source against the official style files and confirmed that the manuscript uses the ICLR 2027 template. If the reviewer could specify the issue more clearly, we would correct it immediately.

With the deadline approaching, we just wanted to gently ask if our response has resolved your concerns or if there are any remaining questions. We would be glad to provide further clarifications if needed.

Reviewer_XEXhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose IRest-VL, where they aim to solve the compositie infrared image restoration task by utilizing multimodal LLM to plan and execute a restoration program. They employ MLLM to diagnose the input imgae under 5 degradation types with 3 levels of severity, and restore the image via executing a short Python program that can utilize 8 restoration tools. They train their framework through EABO, and aim to maximize SSIM against the clean image. The authors evaluate their method on synthetic degradation images.

Critical strengths
  • Controlled experimental evaluations with noticeable improvements over the baseline methods.
  • Proposed method is straightforward and easy to understand, with clear motivations.
  • Interpretable output which is an executable and standalone program. This design can be run without further MLLM inference, and can contribute to saving computation.
Critical weaknesses
  • All experiments use synthetic degradations applied to 1,503 HM-TIR images, without any single qualitative or quantitative evaluations on real-world IR image examples.
  • Since the authors claim that "IR images suffer from composite degradations" from the very start of the manuscript, this discrepancy undermines the very motivation and the necessity of the proposed method.
  • OOD experiments employ 8 unseen type-sequence templates, constructed using the same synthetic degradation premitives used at training stage. This seems to be a very weak and easy setting, with high overlap with the training setting.
  • How does the proposed method deal with ambiguous diagnosis where degradation orders are indistinguishable via observation?
  • In Table 13, why does structured diagnosis completely fail on OOD L=3 settings?
Questions for authors

Please refer to the above weaknesses section.

7 comments
AuthorsOpenReview ↗

Response to Reviewer XEXh

Thank you for your insightful comments and helpful suggestions.

W1: We here add two complementary evaluations.

W1.1 Cross-dataset evaluation on LLVIP. The LLVIP dataset is split into 10,822/1,203/578 images for training/validation/test. We generate a 2x synthetic training set and train for six epochs. The results with Qwen3.5-0.8B are reported below and the complete results are included in Appendix H.3.2/Table 18.

MethodID F1_a/F1_t/FullID SSIMOOD F1_a/F1_t/FullOOD SSIM
Direct AR Diagnosis and Restoration.7548/.4883/.4862.8628.5600/.2757/.0007.8133
IRest-VL.7921/.5089/.4844.8647.6590/.3800/.1194.8133

As shown in the table, IRest-VL substantially improves OOD diagnosis while maintaining comparable ID diagnosis and ID/OOD restoration. These results demonstrate that the proposed method remains effective on a different dataset.

W1.2 Evaluation on real noisy infrared images. We additionally collect 100 real noisy infrared images: 44 from JFAD, 5 from the TNO, and 51 from the TRNeRF. Each image contains stripe noise, random noise, blur, or a combination of these degradations. We compare the methods using MUSIQ and CLIP-IQA.

MethodMUSIQ ↑CLIP-IQA ↑
Original image26.4093.3506
Guided filter (best single tool)22.2888.3640
GLM-5.227.0529.3464
Qwen3.8-Max27.2750.3376
GPT-5.6-sol29.2612.3587
Claude Sonnet 527.9741.3463
Direct AR Qwen3.5-0.8B28.3252.3607
IRest-VL-Qwen3.5-0.8B28.0313.3720

On these real images, IRest-VL obtains the highest CLIP-IQA (.3720), while GPT-5.6-sol obtains the highest MUSIQ (29.2612). The two no-reference metrics favor different methods, but IRest-VL is among the strongest on both and outperforms the guided filter on both metrics. Direct AR and IRest-VL have similar restoration scores because the decoupled architecture primarily targets OOD diagnosis generalization and diagnosis--restoration interference. Appendix K/Fig. 7 provides qualitative examples in which IRest-VL identifies plausible degradation components and adapts the selected lightweight tool and its arguments across images. This added evaluation provides evidence on observed real artifacts, while the limited public data do not cover all degradation combinations and application orders.

W2: We understand this concern as a request for evaluation on real infrared images. We therefore conducted the real-image experiment described in W1.2.

W3: We agree that the current split is a controlled composition-generalization test rather than a strong domain-shift benchmark. The eight unseen OOD type-sequence templates are constructed from the same degradation primitives used during training.

This setting is nevertheless nontrivial. Degradation order changes image statistics: blur followed by stripe noise is visually different from stripe noise followed by blur, so learning one composition does not guarantee generalization to its reordered counterpart. The consistent ID-to-OOD drops across multiple backbones in Table 2 confirm that this controlled OOD setting remains challenging for current models. We agree that introducing unseen degradation types would provide a stronger and more difficult test. The current work first addresses this seemingly simple but empirically difficult composition-generalization problem, and future work will consider more challenging generalization settings.

AuthorsOpenReview ↗

Response to Reviewer XEXh (following)

W4: We agree. Recovering degradation order from one image is an inverse problem, and visually similar observations can correspond to different physical generation orders. The diagnosis head predicts attribute and directed-transition probabilities; valid-path decoding returns the highest-scoring legal path under those probabilities. It therefore produces the most probable path, not a guarantee of a uniquely identifiable physical order. The same probabilistic formulation can be extended to return top-k paths or a candidate set. Importantly, the restoration-code branch does not receive diagnosis as a serialized prefix. Order uncertainty is therefore not directly propagated into code generation, and the program branch can still choose tools, order, and parameters from image evidence.

W5: We thank the reviewer for highlighting this issue. It is worth noting that full diagnosis accuracy is a strict conjunctive metric. For L=3, all three primitive types, three severities, two directed transitions, and the complete order must match the reference; any local error makes the full prediction incorrect. Attribute and transition F1 instead give credit for partially correct local predictions.

For IRest-VL-Qwen3.5-0.8B on OOD L=3, attribute F1 is .7260 and transition F1 is .3805, whereas exact full accuracy is .0013. Thus, local degradation evidence has not disappeared, but errors compound when the complete path is reconstructed. The valid-path score sums local log probabilities, equivalently multiplying their probabilities, so exact joint agreement becomes increasingly difficult as the pipeline grows.

The low OOD full accuracy at L=3 does not mean that the proposed method is ineffective. First, IRest-VL achieves higher attribute and transition F1 than Direct AR for both L=2 and L=3. Second, its full diagnosis accuracy is also higher than Direct AR at L=2. Third, although both methods obtain nearly zero full accuracy at L=3, IRest-VL still achieves higher attribute and transition F1. This comparison shows that exact full diagnosis at L=3 is difficult for both approaches rather than being a failure specific to IRest-VL. Appendix H.1/Table 14 reports the complete length-specific results.

With the deadline approaching, we just wanted to gently ask if our response has resolved your concerns or if there are any remaining questions. We would be glad to provide further clarifications if needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.