- Rating
- 2(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 2(+1)
Read review
Paper summary
The paper introduces LabThoughts, egocentric video of five unscripted wet-lab experiments by four PhD students, recorded with concurrent think-aloud narration and follow-up interviews. The authors label transcript lines as assessment, rationale or intent, and use Gemini 3.1 Pro to turn them into three-way multiple-choice questions about the experimenter's reasoning and about when goal-relevant events occurred. Three models (two Gemini, one Qwen) are evaluated with video only, video plus the stated goal, and video plus goal and a text action list. The best model is only modestly above chance (42.5% vs. 33.3%), intent questions are reported as easier than rationale and assessment questions, and adding the goal and action list gives the largest gain.
Critical strengths
Idea is interesting, Existing lab-video resources follow given protocols and do not record the experimenter's own running account. This gives near-direct, time-aligned evidence about cognitive states that are normally unobservable, which is a valuable resource for the community. This topoic is important. I believe scaling this paper would be a strong contribution.
Critical weaknesses
Experiments is weak. Cannot support it as top conference paper.
The way the questions are built shapes the headline accuracies, so they cannot be read as a measure of how recoverable goal-directed cognition is from video. Each of the following choices is reasonable on its own, but together they push accuracy down and blur the differences between cognition types:
- The filter selects against the main evaluated model. A question is kept only if Gemini 3.1 Pro, given no experiment context, answers it wrongly in at least two of three runs (Sec. 3.3). The remaining questions are therefore ones on which this model's prior knowledge points to a wrong option, and the same model is then the main model evaluated (Table 3). This would explain the below-chance results: rationale accuracy is below chance for all three models (Table 3: 29.1, 16.5, 26.0), and Gemini 3.1 Flash-Lite, which may share the filtering model's priors, is below chance overall (23.6). A below-chance score means the options mislead the model, not only that the video is uninformative.
- A notable share of answer keys is ambiguous. In the authors' check of 90 questions, 15 (16.7%) had a distractor not clearly worse than the correct answer (Sec. 3.3). The paper argues this only limits very high accuracy, but at 40–60% accuracy, ambiguity of this size directly affects the per-category differences used for RQ1 and RQ3.
The evaluation is too small to support conclusions about models or cognition types. Only three models are evaluated, two of them Gemini, with one answer per question (Sec. 4); the authors report negligible variance on repeated questions in pilot tests, but no intervals for the reported numbers. With three options, chance is 33.3% and the best overall result is 9.2 points above it (Table 3), and the per-category differences used for RQ1 and RQ3 are of similar size, so they are hard to separate from noise. A wider set of open models and more options per question (e.g. four, which lowers chance to 25%) would make the comparisons interpretable.
The paper says only that the transcripts "are annotated by the authors" (Sec. 3.1): it does not say how many authors labelled, whether any line was labelled twice, or how often labellers agreed. The quality check of the generated questions was likewise done by a single author (Sec. 3.3). Reporting the number of annotators and inter-annotator agreement on a double-labelled subset, per label, would show whether the category boundaries are stable enough to compare accuracies across them.
Questions for authors
-
Filtering (clarification and additional results). How many questions did the context-free filter remove, and what is each model's accuracy on the questions before filtering, in the layout of Table 3?
-
Ambiguous answer keys (additional results). How many questions is each cell of Tables 3 and 5 based on, and how do these results change when the 15 flagged questions are removed?
-
How many authors labelled the transcripts, was any part labelled by more than one of them, and what was the agreement (e.g. Cohen's κ) per label? A check of a larger sample by a second annotator, reported per category, would show whether ambiguity is concentrated in rationale questions. If the ordering across cognition types holds on a cleaned subset, the RQ3 conclusion would be much stronger.
-
Could you provide results on more models?
2 comments
Thank you for the response. It confirms that a single author labelled the transcripts and that the 15 flagged questions were corrected. The following concerns remain:
- Below chance. The response argues that a model that understands the experiment should navigate around misleading options. But the kept questions are exactly those Gemini 3.1 Pro answered wrongly without experiment information. Could you report the accuracy of the two non-Gemini models on the final set given only the question and options, ideally with a human baseline, so that readers can tell how much of the below-chance result comes from understanding the video and how much from the filter.
- The released vqa.json files contain only the blind-filtered (hard) subset — 574 of the 1,251 generated items; see data/README.md. Is it the number of questions removed by the context-free filtering step? Could you report each model's accuracy on all 1,251 questions before filtering?
- Could you give the number of questions per cell of Tables 3 and 5, and clarify which question set Table 10 uses?
- More models. Since the paper already runs Qwen3-VL-32B locally (Table 3 ), could you add another open-weight model, which the institutional API policy would not restrict?
We thank the reviewer for recognizing our effort to collect not only the recordings but also the real-time cognitive states, and for acknowledging the importance of having this kind of data available to the research community.
We only kept questions that Gemini 3.1 Pro could not answer with its prior knowledge, a below-chance score means the options mislead the model, not only that the video is uninformative.
We are testing the model’s understanding of the scientific experiment given the experiment context, not just the general scientific knowledge it has without the video. Regardless of the misleading options, a model with good scientific understanding could have navigated around the wrong answers. Thus, the below-chance results reveal the models’ lack of scientific experiment understanding.
How many questions did the context-free filter remove, and what is each model's accuracy on the questions before filtering?
We did not record the number of questions removed by the context-free filtering step; we treated it as an intermediate part of the question-construction pipeline rather than tracking it as a reportable statistic. We only recorded results on the final question set.
How many questions is each cell of Tables 3 and 5 based on, and how do these results change when the 15 flagged questions are removed?
The 15 flagged questions were corrected rather than removed. We only recorded experimental results on the final question set and did not retain intermediate results from before the correction.
How many authors labelled the transcripts, was any part labelled by more than one of them?
The transcripts are labeled by one author so we do not have inter-annotator agreement statistics. However, three label categories are defined by substantially distinct criteria (assessment evaluates the current state, rationale explains the reasons behind a decision, and intent expresses the short-term subgoal behind future actions), which we expect to result in low disagreement.
Could you provide results on more models?
Given our institution’s AI policy, the evaluated models are the ones we are permitted to access. However, our pipeline and experiments can easily be tested and replicated on other models. We agree that extending to other models would be beneficial, and we see this as a natural direction for follow-up work, potentially in collaboration with other institutions.