acceptodds
Under review as a conference paper at ICLR 2027

OphMem: Leveraging Cross-Case Experience to Mitigate Visual Ambiguity in Ophthalmic Surgical Video Understanding

Abstract

Ophthalmic surgical video understanding can assist clinicians in analyzing surgical workflows, identifying abnormalities, and supporting postoperative assessment. However, a key challenge lies in visual ambiguity, where visually similar observations may correspond to different surgical states. Existing methods mainly rely on intra-case contextual information, which does not always provide reliable discriminative cues. To address this limitation, we propose OphMem, a framework that continually accumulates and reuses cross-case procedural experience. OphMem consists of two core modules: Experience-Conditioned Agentic Reasoning, which retrieves applicable experience to guide video inspection and tool use, and Persistent Experience Memory, which transforms completed reasoning trajectories into medically grounded reusable experience and consolidates them into an evolving experience bank. To further assess visual ambiguity, we construct a dedicated benchmark with three tasks: History-Conditioned Step Disambiguation, Ambiguous Complication Verification, and Ophthalmic Temporal VQA, together with a sequential experience protocol. Across the three tasks, OphMem achieves a mean relative gain of  8% in primary accuracy over various benchmarks.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Surgical video understanding, Vision-language models, Agentic memory, Visual ambiguity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZasgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces OphMem, an agent-memory framework for resolving visually ambiguous events in ophthalmic surgical videos. During a question, the agent repeatedly retrieves textual procedural strategies from a persistent bank and uses them to guide video inspection and tool calls. After answering, good tool-action steps are abstracted into reusable experience cards, medical claims are checked against a knowledge source, and candidates are added, merged, skipped, or dropped. The bank remains fixed within a question and evolves between questions.

Critical strengths
  1. This paper targets an underexplored form of visual ambiguity. Repeated instruments and anatomy can make local surgical views genuinely difficult to distinguish. Retrieving prior inspection strategies is a plausible way to decide which earlier evidence should be examined.

  2. The benchmark construction and sequential protocol are reasonable. The authors build three complementary tasks from existing case-separated annotations, document the visual-pair mining rules, preserve causal cutoffs, use exact deterministic scoring, and compare structured memory with raw-case and no-memory variants.

Critical weaknesses
  1. The system seems not reproducible. Essential details—including the backbone, retrieval method, prompts, tool budget, medical knowledge source, and memory-update implementation—are missing. The definitions of “successful” actions and medically supported claims are also unclear.

  2. The evaluation seems not to isolate the benefit of cross-case memory. Results use only one fixed question order, without repeated runs, order permutations, uncertainty estimates, or irrelevant/frozen-memory controls. Baseline compute is not consistently matched, and the automatically mined “ambiguous” pairs lack clinician validation.

Questions for authors
  1. Do the authors have a frozen-bank control using only acquisition/development experience, or a control that retrieves an equal amount of shuffled/irrelevant experience?

  2. Did ophthalmologists review the mined evaluation pairs for visual ambiguity, label validity, and necessity of historical evidence?

Reviewer_FXenOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

OphMem draws on previous questions to help interpret ophthalmic surgical videos. It retrieves inspection strategies to guide video tool use, then builds reusable experiences from the completed trajectory, checks their medical claims, and updates memory for later questions. The evaluation includes 1,179 questions from 238 cases across step disambiguation, complication verification, and temporal VQA. The authors report gains over the compared models and memory baselines.

Critical strengths
  1. The paper addresses a relevant problem in surgical video understanding: similar local views can correspond to different surgical states. Using earlier procedural evidence to distinguish them is a useful direction.

  2. The memory stores inspection strategies rather than just past answers. This is a useful design choice: it aims to help the model decide what evidence to check in a new case.

  3. The benchmark is built from existing expert annotations, with explicit rules linking questions and answers to their source labels and temporal intervals. This gives readers a way to trace how the reference answers were obtained.

  4. The appendix provides substantial detail on question construction, accessible video evidence, answer normalization, and scoring. The three tasks cover step identity, complication presence, and temporal event relations, offering several settings in which to study experience reuse.

Critical weaknesses
  1. The reported accuracies cannot be reconciled with the stated task sizes and scoring formula. Baseline coverage also needs clarification.

Table 1 lists 442 evaluation questions for Step Disambiguation and 237 for Complication Verification. Appendix A.5 defines Pair Accuracy as the proportion of questions for which both ordered answers are correct. However, OphMem’s reported scores of 67.74% and 30.58% cannot be obtained from integer counts under these denominators. For 442 questions, the neighboring attainable scores are 299/442 = 67.65% and 300/442 = 67.87%. For 237 questions, they are 72/237 = 30.38% and 73/237 = 30.80%. Rounding alone cannot explain either result.

The comparison protocol is also unclear. Sec. 4.1 says that memory baselines share the backbone, tools, question order, and inference budget, while Table 2 notes that baseline scores retain their “original benchmark coverage.” It is unclear whether all methods were scored on the same question IDs.

  1. The experiments do not separate cross-case transfer from adaptation within the same case.

The paper interprets the results as evidence that inspection strategies transfer beyond their source cases. However, a case can contribute multiple questions, Task 3 produces five questions per case, and memory updates after every completed question. Later questions may therefore benefit from experience produced by earlier questions about the same case. Separating cases across acquisition, development, and evaluation pools does not exclude this effect within the evaluation stream.

A comparison that excludes experiences originating from the current case, or delays updates until all questions from that case are completed, would help establish the cross-case contribution. Figure 3 alone does not resolve this: changes in accuracy along a fixed stream can also reflect changes in question difficulty or composition.

  1. The ablations support parts of the overall configuration, but do not establish the proposed inspection and grounding mechanisms.

The agentic inspection loop has a reasonable motivation, but its benefit over simpler retrieval-based video understanding remains unclear. No Memory and Raw Case Memory mainly vary memory access or representation. They do not directly compare adaptive inspection with retrieving the same experiences once and answering from fixed video samples. A comparison using the same backbone, memory, and matched frame and model-call budgets would help isolate the value of dynamic inspection. Figure 5 illustrates one plausible sequence, but does not establish its contribution across the benchmark.

The visual-ambiguity claim also needs a more direct test. Task 1 selects similar clips using color, thumbnail, and edge descriptors. Similarity under these descriptors does not establish that local video evidence is insufficient. Local-only and controlled history-removal comparisons would help determine whether the gains actually depend on the historical evidence highlighted by the method.

Removing Medical Grounding lowers performance in Table 3, but no direct assessment shows whether grounding correctly identifies unsupported medical claims. Likewise, “w/o Experience Evolution” is not defined precisely enough. Since the bank starts empty, completely disabling updates would leave it empty; if this variant instead retains additions but disables consolidation, that distinction needs to be specified.

Questions for authors

Please see the weaknesses above.

Reviewer_TPiPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OphMem stores procedural experience from completed ophthalmic-video questions and retrieves it to guide later video inspection. It abstracts tool actions, checks medical claims, and consolidates strategies between questions. The evaluation covers step disambiguation, complication verification, and temporal VQA, with 1,179 questions from 238 cases

Critical strengths
  1. Storing inspection strategies fits the ambiguity the benchmark targets. In Sections 3.2.2–3.3, entries specify when and how to inspect evidence, with answers and case identifiers removed. Figure 5 makes the mechanism concrete: checking earlier IOL implantation helps distinguish two removal steps with similar local appearances. The stored strategy directs the search; observations from the current video determine the answer. This is a sensible way to reuse procedural knowledge in this setting.

  2. I put most weight on the step-disambiguation result in Table 2: 67.74% Pair Accuracy versus ReasoningBank's 61.54%. Temporal VQA also improves, from the strongest baseline's 50.40% to 54.60%. Memory baselines share the backbone, tools, stream, and inference budget. Table 3 strengthens the case for the memory design: Raw Case Memory reaches only 52.10% and 42.60% on these tasks. The complication gain is smaller, at 30.58% versus 29.03% Pair Accuracy, so the evidence is uneven across tasks.

Critical weaknesses
  • The main evaluation does not establish how much improvement comes from cross-case transfer. Sections 3.1 and 4.1 update memory after every question, while Table 1 contains multiple questions per case. Full-stream scores therefore mix possible assistance from earlier questions involving the same case with transfer from other cases. Removing case identifiers and stored answers does not separate these effects. This is not evidence of leakage, but it leaves the paper’s central cross-case claim unresolved.

  • Medical grounding has a large effect without a sufficiently concrete account of its information source. Section 3.3.2 defines checking against medical knowledge KK, but does not identify what supplies that knowledge or how support is decided operationally. Removing grounding lowers Temporal VQA accuracy from 54.60% to 39.20%, below the 50.40% No Memory result. Given that effect, the resource and checking procedure are necessary for understanding which part of the system produces the benefit.

  • The experience-evolution ablation is difficult to interpret. Section 3.1 starts with an empty bank, and Section 3.3 describes its subsequent updates. Table 3 nevertheless presents “w/o Experience Evolution” as a distinct condition from No Memory, with Step Pair Accuracy of 42.84% versus 54.75%. The main text does not make clear which writes and curation operations remain active. This matters because the comparison is used to attribute a substantial benefit to evolving experience; its interpretation depends on the exact intervention.

Questions for authors
  • Can an evaluation question retrieve memories written from earlier questions involving any of its source cases? If so, do existing results separate performance on previously encountered cases from performance on genuinely new cases?

  • What supplies KK, and what procedure determines whether a medical claim is supported? Is the same knowledge access available to the relevant comparison systems?

  • In “w/o Experience Evolution,” how is the bank initialized, and which ADD/MERGE/SKIP/DROP operations remain enabled? What distinguishes this condition operationally from No Memory?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.