- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper presents a benchmark that measures the ability of a pair of small weights (< 8B) models to reason on a rendered image of geometric objects: a VLM describes the image into text, to which an LLM decides on the action to take.
Some of the reported key findings: 1) full parameter training outperforms LoRA on their proposed benchmark; 2) the VLM is the bottleneck of the overall performance; 3) some of the proposed benchmark tasks are saturated by the models tested.
Due to a very heavy usage of AI for writing, the motivation / experimental setup / results are all very hard to parse. The manuscript uses a lot of non-conventional keywords (e.g., “a cascade of small models” (line 50) or “cascade stages are educated on corpora” (line 21)) or metrics (“five-year-old criterion” (line 304)) that obscure the content. None of these keywords are sufficiently described / defined, and the reader has to infer their meaning from the context. My best understanding of “cascade” is a pair of models where the output of the first model becomes the input to the second model; “educated” means being finetuned; “five-year-old criterion” means how long tshe LLM action trajectory is compared to the gold trajectory.
Figures and tables are also very difficult to parse. See Figure 1 for example. To my best understanding, the difference in (b) and (c) is that (c) seems to be a rendered image after an LLM attempts to use the action “reach+grasp” on the blue block in (b). However, I have no clue as to what the other parts of the figure is trying to convey.
At this stage of the manuscript, it is unclear what the exact contribution of this paper is.
Critical strengths
N/A
Critical weaknesses
- (Major) The main research question of this paper is obscure
-
As of right now, the paper progresses from 1) building a synthetic benchmark -> 2) training some chosen pairs of (VLM, LLM) on in-distribution benchmark data and evaluating the resulting setup on the proposed benchmark -> 3) analyzing which proposed benchmark tasks have been saturated by the trained models.
-
The paper only evaluates models trained on this synthetic dataset. This makes it hard to decouple the usefulness of the benchmark from the effect training dynamics. Evaluating existing models would help validate the quality / calibration of the benchmark.
-
The same goes for the “audit” part of the paper. If the models are all finetuned on a synthetic dataset in-distribution, it is not difficult to make them saturate on the benchmark.
- (Major) The main motivation of this paper is unclear
-
The paper says this benchmark is “for a cascade of small models” (line 50). To my best knowledge, a cascade refers to a (VLM, LLM) pair, where a VLM first describes the input image, and the LLM reasons on that output text. However, it is never described why the VLM is not being directly evaluated on solving the task directly from the input image. Why is there a need to evaluate this pair, instead of a single model? The paper suggests that “decomposing a system into perception and planning modules is an old idea” (lines 101-102), but the VLMs are already decomposed into such modules. Internally, there is a visual encoder that perceives the image input, and there are transformer decoder layers that do that planning.
-
The paragraph titled “Educational agents and preschool mathematics” under the related work section is very confusing. The paper suggests that “Whether a model can be taught a competence it does not acquire zero-shot” (line 89-90) is somehow relevant to early-childhood education, then cites a paper that uses an interactive AI to assist children learn. Here are some detailed questions.
- Machine learning is all about models learning to solve the task it was trained on. This concept is not new.
- It is unclear how this concept is relevant to early-childhood education.
- It is also unclear how the proposed experimental setup is relevant to an AI agent that assists children learn, nor “children’s over-trust of conversational agents” (line 99).
- (Major) The experimental setup is unclear.
-
The 19 tasks are never fully described. Table 2 in the Appendix only lists the names of the tasks, not what they are. What exactly is the input / output structure? What is the model expected to do?
-
The paper refers to “answer modality” (lines 136-137) and classifies each task as either “say” or “act” (Table 2, page 14). It seems like the authors are classifying a task as a “say” modality if the final required action is “say” from a choice of 8 (lines 264-265). Note that this is not a conventional definition of a modality. Since the second stage of the benchmark passes through an LLM, the only modality relevant here is “text.”
-
The ablations are run (Table 6) without a clear purpose. The (VLM, LLM) pairs are chosen seemingly arbitrarily, and full parameter finetuning and LoRA finetuning are also mix-and-match ed without a clear explanation. A good experimental design would ablate one design choice at a time for a particular purpose.
Questions for authors
-
Can the authors clarify the motivation and the main research question of the paper?
-
Can the authors clarify the experimental setup?