acceptodds
Withdrawn from ICLR 2027

Micro-Agent: Mitigating Catastrophic Forgetting in Microscopic Spatial Prediction and Semantic QA through Two-Stage Adaptive Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) have emerged as powerful foundations for robotic perception systems, yet their adaptation to specialized tasks presents a fundamental challenge: catastrophic forgetting of pre-trained capabilities. This phenomenon is particularly critical in autonomous micro-imaging systems, where precise spatial prediction must coexist with general visual understanding and semantic reasoning. In this paper, we present Micro-Agent, a two-stage optimization framework that enables VLM specialization for spatial prediction tasks while preserving foundational knowledge representations. Our approach addresses the stability-plasticity dilemma through a novel combination of Low-Rank Adaptation (LoRA) and a Mixture-of-Experts (MoE) correction strategy. In the first stage, LoRA-based fine-tuning adapts the pre-trained Qwen-VL model for distance and direction prediction while maintaining parameter-space integrity. The second stage introduces a hybrid expert ensemble comprising the LoRA-adapted model, four handcrafted sharpness features (Entropy, Log-Brenner, Log-Variance, Log-Energy), and a specialized 3-layer CNN with 16 channels, with outputs integrated through a learned fusion MLP. Experimental evaluation on micro-positioning tasks demonstrates that our framework reduces mean positional error from (Stage 1 only) to while improving directional accuracy from to . Critically, image description quality remains comparable to pre-trained levels ( vs ), confirming effective mitigation of catastrophic forgetting. The proposed architecture offers a principled approach for developing specialized vision-language agents that retain broad perceptual capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI for Science, Vision-Language Models, Catastrophic Forgetting, Low-Rank Adaptation, Mixture-of-Experts, Autonomous Microscopy, Parameter-Efficient Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_z8rMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a two-stage optimization framework to address catastrophic forgetting when adapting vision-language models to micro-positioning tasks. First, Qwen3-VL-2B is fine-tuned using LoRA to predict the distance to optimal focus and movement direction while preserving general semantic capabilities. A Mixture-of-Experts (MoE) correction strategy then fuses representations from the LoRA-adapted model, handcrafted sharpness features, and a lightweight CNN. The experiments show that the method substantially reduces positional MAE to 2.21 µm and improves directional accuracy to 100% while retaining general image description capabilities.

Critical strengths

1、Micro-positioning requires both precise spatial prediction and the preservation of visual-semantic capabilities, making the problem practically relevant.

2、Both positional error and directional accuracy improve on the small-scale test set.

Critical weaknesses

1、Comparisons with classical autofocus methods and more direct regression models are missing.

2、The illustrations show clear signs of AI generation and contain watermarks, such as the star-shaped mark in the lower-right corner. The figures lack the rigor and professionalism expected of scientific illustrations. I recommend replacing them with properly prepared scientific vector graphics.

3、The paper lacks an in-depth analysis and discussion of its core design choices. The quality of the writing and the depth and precision of the scholarly discussion do not yet meet the standard expected for acceptance at a leading conference such as ICLR.

4、The foundational dataset contains only 140 image pairs, and the test set consists of just 14 samples. The reported MAE of 2.21 µm and directional accuracy of 100% are therefore based on too few test samples to adequately support the conclusions.

Questions for authors

See Weaknesses

Reviewer_nR2yOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Micro-Agent, a two-stage framework for adapting vision-language models to microscopic spatial prediction while mitigating catastrophic forgetting. In the first stage, LoRA is applied to Qwen3-VL-2B for distance and direction prediction. The second stage introduces a Mixture-of-Experts module that combines the LoRA-adapted VLM with handcrafted sharpness features and a lightweight CNN through an MLP-based fusion mechanism. Experimental results report a mean absolute error of 2.21 µm and 100% directional accuracy. The authors also evaluate image description quality to assess the preservation of pre-trained capabilities.

Critical strengths
  1. The paper addresses the challenge of adapting vision-language models to microscopic spatial prediction while preserving their pre-trained capabilities. This problem is relevant to autonomous microscopy applications.

  2. The proposed framework combines a LoRA-adapted VLM with handcrafted sharpness features and a lightweight CNN. The experimental results show preliminary improvements in spatial prediction over the considered baselines.

Critical weaknesses
  1. Limited methodological novelty. The core methodology primarily combines two existing techniques: LoRA-based fine-tuning to preserve pre-trained model parameters and MLP-based fusion of representations extracted from different feature extractors.

  2. Questionable formulation of MoE architecture. According to Section 3.3, all three experts are activated simultaneously, and their features are projected into a shared embedding space and directly fed into a fusion MLP. The proposed architecture lacks an explicit routing mechanism or conditional expert activation.

  3. Insufficient ablation studies. The paper lacks comparisons with simpler fusion strategies, such as direct feature concatenation and linear fusion. Without these baselines, it is difficult to establish the advantages of the proposed fusion architecture over conventional alternatives.

  4. Insufficient experimental evidence. The experimental dataset is too small to adequately support the paper's claims. According to Section 4.1, the entire dataset consists of only 140 original image pairs, of which merely 14 are reserved for testing. The authors are encouraged to expand the test set to provide more reliable evidence of the proposed method's effectiveness and generalization ability.

  5. Inadequate evaluation of catastrophic forgetting. According to Section 4.3, the authors evaluate image description quality using GPT-5 on only 20 images selected from the same microscopy dataset. This evaluation is based on an extremely small sample size and lacks clearly defined scoring criteria. Moreover, the evaluation images come from the same domain as the training data, making it difficult to reliably assess catastrophic forgetting.

Questions for authors
  1. What is the main methodological novelty beyond combining LoRA fine-tuning with MLP-based feature fusion?

  2. How does the proposed MoE architecture differ from conventional feature fusion without explicit expert routing or conditional activation?

  3. Could the authors provide comparisons with simpler fusion strategies, such as direct feature concatenation or linear fusion?

  4. Could the authors expand the test set to provide more reliable evidence of the method's effectiveness and generalization ability?

  5. Could the authors evaluate catastrophic forgetting on broader vision-language benchmarks with clearly defined evaluation criteria?

Reviewer_MrwVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Micro-Agent, a method that trains a VLM to autonomously move a microscope, based on queries such as natural language instructions and directions to move the microscope stage. Full fine-tuning the model for focusing wipes out the model’s general knowledge, and the paper proposes a two-stage training pipeline to address the problem. The paper tests the proposed method on image pairs from a private dataset to validate the method.

Critical strengths

None

Critical weaknesses
  1. The evaluation has serious flaws. 14 frame pairs are used for testing, which cannot show the real-world performance of the proposed method.
  2. The presentation is also flawed. I find Figure 1 questionable and does not add any real substance to the paper. The paper also claims stage 2 as an Mixture of Expert, which typically consists of routing and gating mechanisms. The proposed stage 2 is more akin to a feature fusion technique, which is widely used in vision research.
Questions for authors

Please see above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.