acceptodds
Under review as a conference paper at ICLR 2027

A Video Question Answering Dataset for Validating Understanding of Scientific Goal-Directed Cognition

Abstract

In scientific research, understanding experiments in the laboratory calls for not only physical scene understanding, but also for inferring how actions relate to a scientist's goals, assessments, and evolving plans. We introduce LabThought, a dataset of naturalistic egocentric video from in-situ wet lab research in organic chemistry, biochemistry, and microfluidics. LabThought also includes concurrent think-aloud narration during experiments and post-experiment interviews, providing unusually direct evidence about the reasoning and intentions underlying behavior in the laboratory. We annotate these transcripts to highlight statements related to different components of goal-directed cognition; specifically assessments of the experimental state, rationales behind decisions, and intentions. Using these annotated transcripts, we construct a video question-answering evaluation in which questions are grounded in the expressed cognitive states. Initial experiments with multimodal language models show substantially different performance across question types and context conditions: concrete intentions are generally easier than questions about rationales and assessments, while providing explicit action descriptions improves performance. These results suggest that LabThought provides a useful setting for studying how well video models connect observed laboratory activity to the underlying goal-directed cognition.

Primary area:
datasets and benchmarks
Keywords:
video understanding, scientific reasoning, model evaluation, multimodal models, benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3u3kOpenReview ↗
Rating
2(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3(+1)
Soundness
2(+1)
Read review
Paper summary

The paper introduces LabThoughts, egocentric video of five unscripted wet-lab experiments by four PhD students, recorded with concurrent think-aloud narration and follow-up interviews. The authors label transcript lines as assessment, rationale or intent, and use Gemini 3.1 Pro to turn them into three-way multiple-choice questions about the experimenter's reasoning and about when goal-relevant events occurred. Three models (two Gemini, one Qwen) are evaluated with video only, video plus the stated goal, and video plus goal and a text action list. The best model is only modestly above chance (42.5% vs. 33.3%), intent questions are reported as easier than rationale and assessment questions, and adding the goal and action list gives the largest gain.

Critical strengths

Idea is interesting, Existing lab-video resources follow given protocols and do not record the experimenter's own running account. This gives near-direct, time-aligned evidence about cognitive states that are normally unobservable, which is a valuable resource for the community. This topoic is important. I believe scaling this paper would be a strong contribution.

Critical weaknesses

Experiments is weak. Cannot support it as top conference paper.

The way the questions are built shapes the headline accuracies, so they cannot be read as a measure of how recoverable goal-directed cognition is from video. Each of the following choices is reasonable on its own, but together they push accuracy down and blur the differences between cognition types:

  • The filter selects against the main evaluated model. A question is kept only if Gemini 3.1 Pro, given no experiment context, answers it wrongly in at least two of three runs (Sec. 3.3). The remaining questions are therefore ones on which this model's prior knowledge points to a wrong option, and the same model is then the main model evaluated (Table 3). This would explain the below-chance results: rationale accuracy is below chance for all three models (Table 3: 29.1, 16.5, 26.0), and Gemini 3.1 Flash-Lite, which may share the filtering model's priors, is below chance overall (23.6). A below-chance score means the options mislead the model, not only that the video is uninformative.
  • A notable share of answer keys is ambiguous. In the authors' check of 90 questions, 15 (16.7%) had a distractor not clearly worse than the correct answer (Sec. 3.3). The paper argues this only limits very high accuracy, but at 40–60% accuracy, ambiguity of this size directly affects the per-category differences used for RQ1 and RQ3.

The evaluation is too small to support conclusions about models or cognition types. Only three models are evaluated, two of them Gemini, with one answer per question (Sec. 4); the authors report negligible variance on repeated questions in pilot tests, but no intervals for the reported numbers. With three options, chance is 33.3% and the best overall result is 9.2 points above it (Table 3), and the per-category differences used for RQ1 and RQ3 are of similar size, so they are hard to separate from noise. A wider set of open models and more options per question (e.g. four, which lowers chance to 25%) would make the comparisons interpretable.

The paper says only that the transcripts "are annotated by the authors" (Sec. 3.1): it does not say how many authors labelled, whether any line was labelled twice, or how often labellers agreed. The quality check of the generated questions was likewise done by a single author (Sec. 3.3). Reporting the number of annotators and inter-annotator agreement on a double-labelled subset, per label, would show whether the category boundaries are stable enough to compare accuracies across them.

Questions for authors
  1. Filtering (clarification and additional results). How many questions did the context-free filter remove, and what is each model's accuracy on the questions before filtering, in the layout of Table 3?

  2. Ambiguous answer keys (additional results). How many questions is each cell of Tables 3 and 5 based on, and how do these results change when the 15 flagged questions are removed?

  3. How many authors labelled the transcripts, was any part labelled by more than one of them, and what was the agreement (e.g. Cohen's κ) per label? A check of a larger sample by a second annotator, reported per category, would show whether ambiguity is concentrated in rationale questions. If the ordering across cognition types holds on a cleaned subset, the RQ3 conclusion would be much stronger.

  4. Could you provide results on more models?

2 comments

We thank the reviewer for recognizing our effort to collect not only the recordings but also the real-time cognitive states, and for acknowledging the importance of having this kind of data available to the research community.

We only kept questions that Gemini 3.1 Pro could not answer with its prior knowledge, a below-chance score means the options mislead the model, not only that the video is uninformative.

We are testing the model’s understanding of the scientific experiment given the experiment context, not just the general scientific knowledge it has without the video. Regardless of the misleading options, a model with good scientific understanding could have navigated around the wrong answers. Thus, the below-chance results reveal the models’ lack of scientific experiment understanding.

How many questions did the context-free filter remove, and what is each model's accuracy on the questions before filtering?

We did not record the number of questions removed by the context-free filtering step; we treated it as an intermediate part of the question-construction pipeline rather than tracking it as a reportable statistic. We only recorded results on the final question set.

How many questions is each cell of Tables 3 and 5 based on, and how do these results change when the 15 flagged questions are removed?

The 15 flagged questions were corrected rather than removed. We only recorded experimental results on the final question set and did not retain intermediate results from before the correction.

How many authors labelled the transcripts, was any part labelled by more than one of them?

The transcripts are labeled by one author so we do not have inter-annotator agreement statistics. However, three label categories are defined by substantially distinct criteria (assessment evaluates the current state, rationale explains the reasons behind a decision, and intent expresses the short-term subgoal behind future actions), which we expect to result in low disagreement.

Could you provide results on more models?

Given our institution’s AI policy, the evaluated models are the ones we are permitted to access. However, our pipeline and experiments can easily be tested and replicated on other models. We agree that extending to other models would be beneficial, and we see this as a natural direction for follow-up work, potentially in collaboration with other institutions.

Thank you for the response. It confirms that a single author labelled the transcripts and that the 15 flagged questions were corrected. The following concerns remain:

  1. Below chance. The response argues that a model that understands the experiment should navigate around misleading options. But the kept questions are exactly those Gemini 3.1 Pro answered wrongly without experiment information. Could you report the accuracy of the two non-Gemini models on the final set given only the question and options, ideally with a human baseline, so that readers can tell how much of the below-chance result comes from understanding the video and how much from the filter.
  2. The released vqa.json files contain only the blind-filtered (hard) subset — 574 of the 1,251 generated items; see data/README.md. Is it the number of questions removed by the context-free filtering step? Could you report each model's accuracy on all 1,251 questions before filtering?
  3. Could you give the number of questions per cell of Tables 3 and 5, and clarify which question set Table 10 uses?
  4. More models. Since the paper already runs Qwen3-VL-32B locally (Table 3 ), could you add another open-weight model, which the institutional API policy would not restrict?
Reviewer_5cuCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LabThought, which introduces approximately five hours of egocentric video from five authentic laboratory experiments conducted by four PhD researchers. Concurrent think-aloud narration is annotated as assessments, rationales, or intents and used to construct 574 multiple-choice questions. Evaluations of three multimodal models examine cognitive-state inference and temporal localization under different context conditions. Providing action descriptions substantially improves accuracy, suggesting potential limitations in recognizing or integrating experimental activity.

Critical strengths
  1. Distinctive and valuable data collection. Recording researchers conducting their ongoing experiments, together with concurrent narration and follow-up interviews, provides richer evidence about experimental decision-making than action labels alone.

  2. Well-motivated evaluation direction. Connecting observed actions to assessments, rationales, and intentions addresses an important challenge for laboratory assistance. Including temporal dependencies extends the evaluation beyond isolated action recognition.

  3. Useful initial analyses and transparent documentation. The context comparisons and qualitative examples offer insights into model behavior. The appendix documents generation prompts, and the paper openly reports ambiguity identified during its question-quality audit.

Critical weaknesses
  1. Transcript grounding does not establish visual answerability or cognitive dependency. An answer explicitly stated by the experimenter may depend on private knowledge or preferences unavailable from video. Conversely, some questions may be answerable through procedural knowledge without inferring the experimenter’s cognition. Without expert performance under the same inputs, low model accuracy cannot clearly distinguish reasoning failures from underdetermined questions.

  2. Question and annotation reliability require stronger validation. In the human audit, 15 of 90 questions had distractors that were not clearly worse than the target answer. This substantial ambiguity can affect current model comparisons, not merely impose a ceiling on future systems. The paper also lacks inter-annotator agreement for the cognitive labels and independent validation of answer uniqueness.

  3. The action-scaffolding experiment has a potential information-leakage confound. Appendix B constructs action descriptions using the same narration that supplies the target answers, supplemented by video processing and the experimental goal. These descriptions may convey privileged semantic information beyond visually recoverable actions. Consequently, the improvement from 42.4% to 61.3% does not isolate action recognition as the bottleneck.

  4. Benchmark construction and evaluation are coupled to the same model. Gemini 3.1 Pro generates questions and distractors, filters questions using its own context-free answers, and is then evaluated on the retained set. This may introduce model-specific selection effects. Context-free filtering alone also does not establish that the remaining questions require video. Independent text-only baselines and alternative filtering models are needed.

Questions for authors
  1. How accurately can independent domain experts answer the questions using exactly the model-visible inputs? Can they identify items that require information available only in the narration?

  2. Were the ambiguous items discovered in the audit corrected or removed? Can the authors provide agreement statistics and results on an independently validated subset?

  3. How much improvement remains when action descriptions are generated exclusively from silent video? How well does the action text alone perform without video, and was narration audio removed from all evaluation videos?

  4. What are the text-only results for each evaluated model on the final question set? Are the conclusions stable when a different model performs context-free filtering?

2 comments

We thank the reviewer for recognizing the value of our dataset, which includes video recordings, narration, follow-up interview transcripts, and annotations of researchers’ assessments, rationales, and intentions.

How accurately can independent domain experts answer the questions using exactly the model-visible inputs?

Our collaborators are highly confident that an independent expert in the relevant domain would be able to answer the questions correctly using the model-visible inputs. However, due to demands on the experts’ time, we are not able to formally test this with independent domain experts.

Were the ambiguous items discovered in the audit corrected or removed? Can the authors provide agreement statistics and results on an independently validated subset?

The ambiguous items discovered in the audit were corrected. We don’t have the annotation resources to do agreement statistics for multiple experts.

How much improvement remains when action descriptions are generated exclusively from silent video? How well does the action text alone perform without video, and was narration audio removed from all evaluation videos?

In our experiments, the action descriptions are generated exclusively from silent video, the narration audio was removed from all videos. The experiment results for action-text-only are coming under way.

Are the conclusions stable when a different model performs context-free filtering? We did not test context-free filtering with a different model; given the time constraints of the rebuttal period, we were not able to run it. We agree this is a meaningful question and will flag it as a limitation and priority for follow-up work.

How well does the action text alone perform without video?

Here are our results for Qwen3-VL-32B when adding two more modes: Blind and Actions-text-only.

ModeOverallAssessmentRationaleIntent
Blind, no video, no text35.129.425.037.5
Actions text only, no video47.355.945.765.0
Video only39.229.429.133.3
+ Goal text41.437.326.043.3
+ Goal + Actions text57.976.562.283.3

Action text only is more accurate than blind and scores slightly higher than video only, but below video and additional contexts. This suggests that video or action text alone is less helpful than a combined context (including video and supporting text) for models to infer the experimenter’s intentions, assessments, etc. Importantly, this means that state of the art models do not appear ready to assist with lab experiments using only raw video, but would benefit from access to additional context that makes goals and actions explicit.

Reviewer_huZhOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a video task setting for a human/MLLM model watching someone work (in egocentric view) for an extremely long time, while constantly generating explicit text-based characterizations of model states inside a naive goal-directed cognitive model, specifically, goal-state assessment ("assessment"), reasoning/planning ("rationale"), and intent formation ("intent"). The authors then use Gemini 3.1 Pro to generate a VQA dataset based on these cognitive states, serving as the benchmark for MLLMs.

Critical strengths
  1. I agree with the potential significance of the proposed setting, i.e., A. a persistent cognitive assistant and B. lab sciences domains, could have meaningful impacts on lab sciences, but the authors should clarify C. at the current stage, what implications are supported by what evidence? and D. what are the potential harm (e.g., could this mean that lab sciences become driven by resource-intensive AI control loop, where scientists become commoditized agents serving as the physical enablement of AI-driven science)? Since a key contribution is the setting, it is necessary to discuss both its positive and negative implications.

  2. I am also convinced by the importance of earlier cognitive AI research that motivated this work. This paper serves as an operationalization of abstract cognitive frameworks for human cognition expressed in natural language. However, a deeper study of cognitive systems should not stay at the abstract level, instead requiring a systems view - what is the cognitive model(s) you are investigating or assuming in your construction. You do have an implicit model ("Goal-Directed Cognition") in your Figure 2, but it requires more serious study because you are necessarily modeling human cognition.

Critical weaknesses
  1. Writing: P1 and introduction at large lack citations. While I appreciate early cognitive models, are there truly no recent (2023-2026) works on goal-directed or cognitive agents? As written, the introduction makes clear the domain-level motivation, but not motivation informed by current research.

  2. Writing (high): Figure 2 is too simple. What you need is a pipeline figure, including the lifecycle of your agent, including different levels of understanding across the full length of the task, including the outputs at different stages. I still don't understand the mechanism of the overall method after reading the manuscript several times. This new Figure 2 could possibly subsume Figure 3. There is a risk that this new Figure 2 may partially overlap with Figure 1, but it is critical that you need a figure to clearly demonstrate the overall pipeline that's not just four block arrows.

  3. Writing. Your core contribution is cognitive grounding, which should carry much more weight in 3.3 "Cognitively Grounded VQA Construction" and be connected to the new Figure 2. The other aspects can be mentioned in a combined paragraph and be pushed to the appendix for details.

  4. Soundness (high). There is a lack of experiments, especial newer models and the diversity of model families. One proprietary model family (Google's Gemini 3.1 and Gemini 3.1 Flash Lite) is not enough. The latest GPT and/or Fable/Opus/Sonnet is required unless there is a serious funding constraint (which would be understandable). Using only one open-source model family (and an outdated one at that), i.e., Qwen3-VL-32B is a significant soundness weakness. You should at least evaluate Qwen 3.8-27B (available in BF16 if your A6000s are Ada) and another model family (Gemma 4 31B, Muse Glimmer-30B, Nemotron-3.5-Bolt-74B-A9B which takes 32GB VRAM despite the 74B in its name)

  5. Soundness (highest). In Section 4, Evaluation settings, you listed video only, video + goal, and video + goal + actions. This is not enough. In fact, based on your findings on action recognition being a bottleneck, you need 4. video + actions, 5. actions + goals, 6. actions only, and 7. goals only. These would enable significant findings into:

    5a. At what point does planning, a type of semantic reasoning, break down? Cognitive planning is mostly semantic (earlier works like the ones you cite - Simon & Newell 1971, Laird 2012, among many others were rules-based logical production systems), so this enables you to ablate the "visual reasoning" (which psychologists have argued is mostly semantic on translated visual tokens) from the semantic reasoning.

    5b. Do actions outweigh the goals?

    5c. Additional study design: if you precompute actions with a good ActivityNet model instead of dynamically with whatever VLM you apply, you ablate away your "action recognition is a bottleneck finding". How does that affect your core studies? I'd expect you to be able to discover counter-intuitive findings.

  6. Soundness (high). For any model, your pipeline uses a 0.2fps video sampling rate, or one frame every 5 seconds. Missing 5 seconds at a time can be a significant problem and can be a cause of your finding that "Action recognition as a potential bottleneck." Humans can take fewer than 5 seconds to execute egocentric movements (see most of ActivityNet), let alone when sampled at inopportune points mid movement at a 5-second interval. Ad infinitum, if you were to sample a frame every one hour, there is no such thing as action recognition at all.

    You attributed this low sampling rate to a short context window. This is not a good reason. There are many, many techniques in video understanding to compress frames while oxymoronically increasing information. Most notably, you should apply egocentric video action segmentation (micro) and egocentric video scene/episode segmentation (macro) to ablate the action recognition step to address this implementation soundness concern as well as the larger soundness concern with study design/finding. This is potentially time-consuming to fully address within a rebuttal period, but I would like to see sincere and significant steps taken towards it.

  7. Writing (medium). In Section 5: Results, headings like "Overall accuracy (RQ1)" are not actual research questions. You should consider moving the RQ1/2/3 prose from Section 4 to their respective paragraphs in Section 5.

  8. Soundness and writing (medium). The "Implications for laboratory assistance" entry is necessary, but as written, it's not sufficiently supported by the evidence. You could potentially reframe it as "in order for lab application to be feasible, we would need to see xxx effectiveness and reduce [e.g., harmful hallucination metric H] by xxx." This could be a more convincing discussion at this stage of the work, before future work can fully convince the larger community.

  9. Soundness (high): since you use Gemini 3.1 Pro to (line 285) to generate the VQAs, one could suspect Gemini 3.1 Pro to also give the highest scores simply by alignment. This is potentially unsound and can be partially addressed by evaluation on other model families.

  10. Why not have a human VQA baseline (not the four PhD providing the cognitive labels)? Even a small subset?

Questions for authors
  1. Is there an A. temporal and B. hierarchical component to the question/answers themselves? Or are they flat regardless of where the subject is in their workflow?

  2. How does your model handle the temporal and hierarchical aspects of cognition in the videos, or do you just modify your model context over time?

  3. How do you judge model QA success?

  4. Why do actions contribute so much to assessment effectiveness? Is it because assessment depends heavily on actions (text-based) that the model otherwise cannot access? Why does intent correlate so significantly to goals and goals + actions? How do these results inform the assessment + rationale + intent A. cognitive modeling and B. metrics/study design? What are some possible alternative cognitive models and metrics/study designs?

1 comment

We thank the reviewer for recognizing the importance of cognitively-inspired AI systems and the potential significance of the proposed setting of cognitive assistant in the lab sciences domain.

Temporal component to the question answer

For questions / answers based on the experimenter’s statements of assessment or intent, there is a temporal component insofar as the questions / answers include timestamps. The context presented for the question includes the whole experiment video, regardless of where the subject was in their workflow when making the statement.

How do the models handle the temporal and hierarchical aspects of cognition in the videos?

Our VQA design handles the temporal aspects of cognition through timestamps when needed, which grounds the questions or answers at specific timestamps. For instance, QA pairs focused on intent require the model to select a timestamp that depicts the experimenter preparing for a following step at the given timestamp. Because this temporal intent-based structure is very naturally captured in our data, we are able to represent in this way in the VQA; while hierarchical and other temporal aspects of cognition are certainly present, they are not so clearly present in the data (i.e., the experimenter’s spoken thoughts and the experiment video do not make hierarchical cognition explicit) and so we do not capture it in the VQA.

How do we define QA success?

Model QA success is judged based on accuracy, but as discussed in Section 3.3, the presented VQA set is unsuitable for distinguishing between two models with high accuracy. Human validation exists for a subset of 90 questions, which provides the best metric for success.

Why do actions contribute so much to assessment effectiveness?

We suppose that actions contribute so much to VQA accuracy because (a) action recognition is a prerequisite to accurate question answering, as you mention; and (b) both action recognition and the cognitive VQA task are relatively difficult for the model, and the model struggles to perform both simultaneously. It would be interesting to conduct a scaffolding experiment, in which the model must first identify actions and then complete the VQA task, but running these scaffolding experiments is unfortunately not possible within our current budget constraints.

Why does intent correlate so significantly to goals and goals + actions?

We believe intent-based questions receive such a strong boost from the addition of a “goal” context because the end-goal of the experiment directly informs what kind of intentions and plans an experimenter has. Prior work suggests that inferring goals (even relatively simple ones) from video alone is very difficult for models (e.g., Veerabadran et al., “Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agents” NeurIPS 2025), so providing a “goal” context likely supplements that task.

How do these results inform the cognitive modeling and study design of the paper?

Finally, the study design and cognitive modeling design are not based on the results. Instead, the study design (i.e., think aloud experimentation) is inspired by the think-aloud protocol in psychology literature (e.g., Van Someren, Barnard, and Sandberg. (1994.) "The think aloud method: a practical approach to modeling cognitive processes."). The cognitive modeling design (i.e., the designation of assessments + rationales + intents from experimenter statements) is inspired by Simon and Newell’s model of goal-directed cognition (see Section 2), and the specific labeling schematic is informed by the authors’ observations of recurring themes in the experimenters’ spoken thoughts and what kinds of questions they support. Many alternate cognitive models exist, but many we considered would not be well-supported by the think-aloud data. A more top-down approach would be selecting a cognitive model and instructing experimenters to speak thoughts specifically related to that model, but we opted to not take this approach because (a) we were concerned about increased cognitive load on the experimenters, and (b) we wanted to adhere more closely to the well-studied think-aloud protocol, which emphasizes unstructured thought expression.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.