acceptodds
Withdrawn from ICLR 2027

Remember the Lab: Adaptive Visual Abstraction for Long-Horizon Laboratory Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern laboratory research involves long-horizon experimental processes, where states, procedures, and outcomes continuously evolve across repeated experiments. With the emergence of multimodal foundation models, AI assistants have shown great potential for laboratory research. However, effective assistance in dynamic laboratory environments requires the ability to maintain and utilize knowledge accumulated throughout experimental processes. Existing multimodal systems mainly focus on current observations, limiting their ability to reason over evolving laboratory processes across long temporal horizons. We introduce RA-Lab, a multimodal memory-augmented agent framework for long-horizon laboratory video understanding. RA-Lab constructs persistent experimental memories through two key components: Adaptive Visual Abstraction, which transforms raw observations into experience-level visual memories, and Targeted Multimodal Retrieval, which retrieves relevant historical experiences based on the current observation and query. The retrieved memories are combined with current observations to support memory-augmented laboratory reasoning. To evaluate long-horizon laboratory understanding, we introduce LabVQA, a benchmark for laboratory video reasoning over experimental procedures, states, and temporal relationships. Extensive experiments demonstrate that RA-Lab effectively constructs and utilizes multimodal experimental memories, improving long-horizon laboratory reasoning compared with existing multimodal models and memory-augmented agents.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Laboratory Video Understanding, Visual Abstraction, Visual Generation Model, Multimodal Memor

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_c8miOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a memory-augmented framework for laboratory video understanding (RA-Lab). It divides videos into segments and uses an MLLM to combine segments corresponding to the same step into experiences. Each experience is then compressed through 'visual abstraction', where an MLLM decides what information to preserve and how to organize it. Based on this, an image-generation model produces an abstract visual memory. Each experience has associated images and structured text descriptions that are stored in memory and retrieved to support reasoning required to answer queries. To evaluate this, the paper proposes a benchmark (LabVQA). The proposed method is compared against prior streaming models, open-source and closed-source MLLMs, and other memory-augmented multimodal agents. RA-Lab shows improved performance on most categories. The paper also ablates the proposed memory retrieval against text-only memory, raw-frame memory, and no-memory settings.

Critical strengths

I find the laboratory setting interesting, especially since losing visual details through text-based memory compression is a well-known problem. The idea of preserving these details through visual abstraction is a useful direction. The ablations in Table 1 and Figure 4 are informative and help show how the memory representation and abstraction choices affect performance.

Critical weaknesses
  1. A proper benchmark statistic is missing in the main paper, and the details given in the main paper are unclear and inconsistent with the appendix. Since the main QA evaluation uses only one benchmark (LabVQA), which is proposed in the paper itself, the main paper should provide a clear benchmark statistics and report the number of videos, QA pairs, and distribution of video durations. The main paper describes 80 videos from three sources: FineBio, BioVL2, and EgoExoLearn. However, Table 7 in the appendix reports approximately 50 videos from only FineBio and BioVL2, without explaining what happened to EgoExoLearn. It also mentions 1050 queries, but is this the set used throughout the main QA experiments? (Not mentioned in the main paper, and unclear). What is the average duration of the videos for the 1050 queries? What is the distribution of video duration in the final benchmark? This is important to understand how much long-term memory retrieval is actually needed.

  2. The method section explains some central components only at a very high level, making it difficult to understand how to reproduce them. For example, Section 3.2 introduces A_plan, but does not explain how the model is prompted to generate the abstraction plan. Section 4 mentions that GPT-5.5 is used, but how is it instructed to decide what to preserve and how to organize it? What does the resulting plan look like, and how is it converted into image-generation instructions? This is one of the main components of the method, but the main paper does not provide a concrete prompt or clearly point to one.

  3. It is unclear how the image-generation step preserves fine visual details without modifying or fabricating information. In laboratory understanding, details such as instrument readings, object counts, and text on tubes can directly affect the answer. The method generates or edits images, so how is it ensured that these details remain faithful to the original observations? The paper acknowledges that abstraction can lose information, but changing or inventing information is another concern. This is particularly relevant in a scientific setting, where small changes in details can alter the interpretation of an experiment. There is no clear verification step or evaluation showing whether such details are preserved.

  4. The evaluation does not sufficiently establish whether the proposed memory pipeline offers an advantage over stronger long-video models. The paper evaluates Qwen2.5-VL-7B and Gemini 1.5 Pro, but does not explain why newer alternatives, such as Qwen3-VL and newer Gemini Pro or Flash models, are not considered. This matters because the proposed method itself uses frontier models for understanding and image generation. A central question here is what information we lose when compressing the observations, and whether a strong model given the video directly can retain and retrieve more of those details. Comparisons with stronger models, given comparable access to historical evidence, would help establish when the proposed abstraction is useful.

Questions for authors

Please address the questions raised in the weaknesses above, particularly the benchmark inconsistencies, the implementation of abstraction planning, the faithfulness of generated visual abstractions, and why comparisons with stronger long-video models are not included.

Reviewer_NDPkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RA-Lab, a memory-augmented framework for understanding long laboratory videos. The main idea is to convert continuous video observations into step-level memories that contain both structured descriptions and compact visual summaries. An adaptive abstraction agent decides which visual information to preserve, and an image generation model creates a summary image for each experimental step. When answering a question, the system retrieves relevant historical memories and combines them with the current observation.

The authors also introduce LabVQA, a benchmark for evaluating laboratory video understanding across five tasks, including historical recall, state evolution, temporal-spatial reasoning, experience comparison, and experience-grounded reasoning. Experiments show that RA-Lab improves performance over several existing video models and memory-augmented agents. The results are promising, particularly for recalling previous experimental steps and tracking changes in experimental states. However, some aspects of the method and evaluation need further clarification.

Critical strengths
  1. The paper addresses a meaningful problem.

Understanding laboratory procedures often requires remembering what happened earlier, which materials were used, and how an experimental state changed over time. Existing video models can struggle with these questions. RA-Lab addresses this limitation by explicitly maintaining historical experimental memories rather than relying entirely on the current observation. This is a reasonable and useful direction.

  1. The visual memory representation is interesting.

Instead of storing only text or retaining many redundant video frames, the method combines structured descriptions with a compact visual summary. Table 1 shows that the proposed representation achieves 57.7% accuracy, compared with 49.3% for text-only memory and 52.1% for text plus four raw frames. This provides some evidence that visual abstraction can help the model recover information that is difficult to preserve in text alone.

  1. The experiments cover several relevant tasks.

The evaluation includes both LabVQA and downstream tasks such as protocol alignment and issue identification. RA-Lab improves the results of both GPT-5.5 and Qwen2.5-VL-7B in the latter setting. These experiments suggest that the proposed memory representation may be useful beyond ordinary video question answering.

Critical weaknesses
  1. The technical novelty is not sufficiently clear.

The framework combines several existing components: video segmentation, protocol-guided step identification, structured memory construction, similarity-based retrieval, and image generation. The main new idea appears to be using an adaptive plan to guide the generation of a compact visual memory. However, most of this process relies on prompting GPT-5.5 and using GPT-Image-2, rather than introducing a new learning algorithm or retrieval mechanism.

  1. The reliability of generated visual memories is not adequately evaluated.

The method relies on an image generation model to summarize laboratory observations. This introduces a potential problem: the generated image may change or omit important details, even when it looks visually plausible. In laboratory settings, small differences in reagent labels, liquid levels, tube positions, or experimental states can matter. The paper evaluates the generated memories mainly through downstream question-answering accuracy. This does not directly tell us whether the generated images faithfully preserve the original observations. A more convincing evaluation would measure whether key objects, actions, spatial relationships, and state changes are preserved, and how often the abstraction introduces incorrect visual information. Without this analysis, it is difficult to know when the proposed memory can be trusted.

  1. The benchmark is relatively limited, and its construction needs clearer reporting.

LabVQA contains 1,050 questions from approximately 50 laboratory videos, with an average history length of 5.2 experimental steps. While this is a useful starting point, it provides limited evidence about performance over substantially longer histories or across different laboratory environments and experimental procedures. Multiple questions from the same video may also be highly related, so the number of questions alone does not necessarily reflect the diversity of the evaluation. There are also some details that need clarification. Section 4 describes an 80-video benchmark, whereas Appendix A.8 describes LabVQA as using approximately 50 videos from FineBio and BioVL2. The paper should clearly distinguish the data used for LabVQA from the data used for the other tasks. In addition, although the authors state that the questions are manually verified, more information about the verification process and the evaluation of open-ended answers would improve confidence in the reported results.

  1. The baseline comparisons do not fully isolate the contribution of RA-Lab.

RA-Lab uses GPT-5.5 for both memory construction and reasoning, together with GPT-Image-2 for visual abstraction. The baselines use different models and processing pipelines, making it difficult to determine how much of the improvement comes from the memory design itself. The comparison with WorldMM is particularly relevant because the paper also uses GPT-5.5 as its base model. However, RA-Lab does not outperform WorldMM on every task: its experience-comparison accuracy is 43.9%, compared with 48.0% for WorldMM. This suggests that the proposed approach may be better suited to some types of laboratory reasoning than others. A more controlled comparison using the same backbone and comparable input settings would make the conclusions more convincing.

  1. The retrieval component is not evaluated independently enough.

Targeted Multimodal Retrieval is one of the two main components of RA-Lab, but the paper provides limited direct evidence about its effectiveness. The retrieval method uses embedding similarity to select the top-K memories, yet the authors do not report standard retrieval metrics or clearly show how often the correct historical experience is retrieved. Table 1 is useful for studying the memory representation, but it directly provides the ground-truth relevant experience to the reasoning model. This removes retrieval errors from the experiment, so it cannot demonstrate that the proposed retrieval mechanism works reliably in realistic settings. I would like to see a comparison with text-only retrieval, raw-frame retrieval, and alternative retrieval strategies, together with an analysis of how retrieval quality affects the final answer.

Questions for authors
  1. Contribution of adaptive visual abstraction: Can the authors compare the proposed method with a simple visual montage and a stronger structured-text memory baseline, while keeping the underlying models fixed? This would help clarify whether adaptive image generation is the main source of the improvement.

  2. Faithfulness of visual memories: How often does the image generation model omit or alter important experimental details? Can the authors provide a quantitative evaluation of object, action, spatial-relation, and state-transition preservation?

  3. Benchmark construction: Can the authors clarify the relationship between the 80-video benchmark described in Section 4 and the approximately 50 videos used for LabVQA in Appendix A.8? How are open-ended answers evaluated, and how is consistency between evaluators ensured?

  4. Retrieval effectiveness: What are the retrieval recall and precision of the proposed method? How does its performance compare with text-only retrieval and raw-frame-based retrieval when the correct historical experience is not provided in advance?

  5. Efficiency: What are the total costs of memory construction, including model calls for experience extraction, abstraction planning, and image generation? How do latency and cost scale with video length and the number of stored experiences?

Reviewer_QqPyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RA-Lab, an agentic framework for long-horizon lab video understanding, and LabVQA, a benchmark for lab video reasoning. The RA-Lab framework contains two key components: “Adaptive Visual Abstraction" builds experience-level visual memories, and “Adaptive Visual Abstraction" recalls such memories. RA-Lab achieves 5.6% higher QA accuracy compared to the best baseline Text + 4 Raw Frames setting.

Critical strengths
  • The paper formulates long-horizon lab video understanding as a multimodal agentic memory problem and proposes two key components “Adaptive Visual Abstraction" and “Adaptive Visual Abstraction" as solutions.
  • The paper also introduces a benchmark LabVQA for lab video reasoning.
Critical weaknesses
  • In Table 1, the authors report RA-Lab to achieve 5.6% higher QA accuracy compared to the best baseline Text + 4 Raw Frames setting but do not provide standard deviation. Since the framework depends on stochastic processes such as image generation, more trials are also needed to determine the statistical significance of the reported results depending on the benchmark size. LabVQA QAs are fully generated, which may introduce biases into the evaluation and raises further questions about whether the improvements come from the proposed method or biases in the benchmark construction process.
  • It remains unclear whether the improvements arise from the generated image representation itself, or it could also be achieved by more textual semantic information obtained during “Adaptive Visual Abstraction".
Questions for authors
  • How many videos and QAs are included in the LabVQA benchmark? Section 4 states that the benchmark contains 80 videos but Appendix A.8 states that it’s ~50 videos and 1050 QAs.
  • Have the authors considered using a more semantically enriched prompting method as a baseline to be compared to RA-Lab?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.