- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PEVA-Bench, a benchmark designed to evaluate whether multimodal models can act like professional assessors: given a complete photographic record of a classroom (averaging 96 photos) and a written rubric criterion, the model must determine whether the criterion is satisfied while citing supporting evidence images and visual facts. The dataset comprises 12,418 photos collected from 129 classrooms across 43 kindergartens in Shenzhen. The authors adapt 50 photo-verifiable criteria from the ECQRS-EC scale, yielding 6,450 annotated instances; photo-based labels achieve a 91.0% agreement rate () with independent on-site assessments. In addition, the authors construct a 10-classroom diagnostic subset that provides expert evidence images, minimal sufficient subsets, and expert-verified facts as oracle inputs, alongside 12 pairs of lenient-versus-strict criteria.
There are four main empirical findings: (1) among the 15 evaluated models, GPT-5.5 performs best, achieving 89.12% accuracy; (2) when provided with oracle expert facts, small models perform comparably to GPT-5.5, with Qwen3.5-4B reaching a Macro-F1 of 91.42 compared to GPT-5.5's 90.46; (3) 80% of GPT-5.5's errors involve rubric application, 98% of which stem from misclassifying Unsatisfied instances as Satisfied; and (4) on boundary cases defined by the lenient–strict criterion pairs, no model exceeds 42% accuracy. Finally, the authors propose the Perceive-Aggregate-Judge (PAJ) pipeline, which improves performance across 11 out of 13 evaluated models.
Critical strengths
New dataset contribution: This benchmark uniquely combines complete multi-view photographic records of real-world environments, a professional assessment scale, and external validation against independent on-site evaluations, which is a rare combination in existing benchmarks.
Original diagnostic design: Two diagnostic mechanisms stand out: first, a hierarchical oracle ladder progressing from the full photographic record to expert evidence images, minimal sufficient subsets, and finally expert-verified facts, paired with size-matched random-image controls; second, lenient-strict criterion pairs that hold the photographic record fixed while varying only the strictness of the rubric. To the best of my knowledge, the latter has no direct precedent in vision benchmarks, with the closest analogue in the text domain being RuleArena (ACL 2025).
Comprehensive evaluation and clear presentation: The evaluation spans 15 models, including frontier commercial models and open-weight models ranging from 4B to 35B parameters, and provides fine-grained breakdowns across domains, categories, and error types. The finding that strong models exhibit a pronounced bias toward predicting Satisfied carries an important cautionary message for self-assessment applications. Furthermore, ethical considerations are handled carefully, with IRB approval, face/privacy blurring, and controlled data release.
Critical weaknesses
Frontier models have reached the label-noise ceiling, so the conclusion that "judgment capability bottlenecks the strongest models" may partly reflect inherent label ambiguity. On the diagnostic subset, the agreement rate between two independent human reviewers is only 90.8% (), whereas GPT-5.5 achieves 91.72% on the same subset under the Full Images condition and reaches 94.63% under the Oracle Images condition. Across the full dataset, GPT-5.5 achieves 89.12% accuracy, while the agreement rate between photo-based labels and on-site evaluations is 91.0%. Consequently, a substantial fraction of GPT-5.5's 702 errors may fall into genuinely ambiguous borderline cases, for instance, whether sorting shells, pinecones, and stones by type qualifies as sorting by "scientific attributes." Furthermore, error attribution is performed within the same annotation pipeline after inspecting the model's cited images, and the paper does not report inter-annotator agreement for this error attribution step. I recommend: (1) having independent experts blindly adjudicate a sample of model "errors" and reporting the proportion of reference labels that get overturned; (2) reporting human expert accuracy relative to the adjudicated labels; and (3) reporting the direct agreement rate between model predictions and on-site assessments.
Oracle Facts are authored with reference to the ground-truth labels, introducing label leakage. Appendix B.3 notes that experts select evidence images and write factual descriptions based on both the criterion and its reference label. Because these facts have already undergone relevance filtering and are phrased using vocabulary aligned with the rubric, the judgment task largely degenerates into straightforward textual entailment. This has two major consequences: first, it is entirely expected that Qwen3.5-4B performs comparably to GPT-5.5 under this setting, and both sit near the ~91% label-noise ceiling; second, the performance gain attributed to "seeing" is confounded with the judgment skill of knowing which visual facts are relevant to the rubric in the first place. Moreover, using text oracles to disentangle perception from reasoning is already a well-established paradigm, e.g., Prism (NeurIPS 2024), Caption This, Reason That (NeurIPS 2025), and Bad Seeing or Bad Thinking (ICML 2026). While the paper's incremental value lies in its multi-step oracle ladder and specialized domain, establishing that "seeing and judging are distinct capabilities" as a clean empirical insight requires eliminating this label leakage first.
Limited contribution of the proposed PAJ pipeline. PAJ follows the well-known perceive-then-judge paradigm, closely resembling Socratic Models, Prism, LLoVi (EMNLP 2024; segment-wise captioning followed by aggregation), and checklist decomposition in TICKing All the Boxes. Specifically: (1) both the observation checklist and the 8-category spatial zone mapping rely on hand-crafted domain priors, making the claim that PAJ operates "without any expert input" inaccurate; (2) PAJ actually degrades GPT-5.5's Macro-F1 from 90.76 to 89.16, failing to resolve the paper's own diagnosis of frontier-model bottlenecks; (3) a large portion of the gains on smaller models comes simply from atomic criterion decomposition, e.g., Qwen3.5-9B jumps from 66.96 to 82.71 from decomposition alone; and (4) inference cost and token overhead are not reported.
Small sample sizes, inconsistent evaluation splits across models, and missing uncertainty estimates. The diagnostic subset spans only 10 classrooms (447 instances), with strong intra-classroom correlation across instances. In the lenient-strict pair experiments, commercial models are evaluated only on these 10 classrooms (120 pairs in total, containing roughly 45 boundary pairs, meaning GPT-5.5's 31.11% corresponds to roughly , with a wide 95% confidence interval of $\approx $[18%, 46%]), whereas open-weight models are evaluated across all 129 classrooms. Thus, the headline claim that "no model exceeds 42%" conflates two different evaluation sets. In addition, the paper lacks confidence intervals clustered at the classroom or kindergarten level, as well as multi-run variance estimates for commercial APIs.
These issues do not negate the value of the new dataset. I currently lean toward a weak reject, though I will consider raising my score if the rebuttal can provide sufficient supplementary evidence and a convincing explanation.
Questions for authors
Q1. When a sample of GPT-5.5's error instances is blindly re-adjudicated by independent domain experts, what proportion of the original reference labels are overturned/modified?
Q2. If the Oracle Facts are authored by annotators who are blinded to the ground-truth reference labels, what is the performance gap between smaller open-weight models and GPT-5.5? Furthermore, under this blind Oracle Facts condition, what is the directional error breakdown for frontier models, i.e., do they still predominantly misclassify Unsatisfied as Satisfied?
Q3. When evaluating commercial models across all 129 classrooms on the full set of lenient–strict criterion pairs, what is the boundary accuracy and its corresponding confidence interval?
Q4. What is the root cause of the anomalous evaluation results observed for the Qwen3.5 series in the Books & Reading domain, and what are their exact invalid output/parsing failure rates?
Q5. Does this work share any overlap in participating kindergartens, classrooms, or on-site assessment records with the AIED 2026 study on ECQRS-EC-based LLM kindergarten evaluation?
5 comments
Official Response to Reviewer GvJZ [2/2]
W3. PAJ's contribution, expert input, and inference cost
PAJ is a baseline for testing the benchmark's diagnostic value. It builds on Socratic Models and Prism, with related approaches in LLoVi and TICKing All the Boxes. The checklist, area mappings and atomic conditions are expert-designed; PAJ generates its own classroom facts without instance-specific expert evidence or facts.
On the original subset, PAJ outperforms Atomic Criterion on 12/13 models with identical atomic conditions. The mean gain across all 13 is 3.85 Macro-F1 points, reflecting perception and aggregation beyond criterion decomposition. With unchanged submission prompts, PAJ also improves 12/13 models on ten additional classrooms (500 instances) under the same instrument, averaging 8.31 points across all 13. Eight paired classroom-bootstrap 95% CIs exclude zero.
On the original 447 instances, GPT-5.5's Unsatisfied→Satisfied errors fall from 31 to 14. Satisfied→Unsatisfied errors rise from six to 32; Macro-F1 changes by −1.61 points. Costs below cover 96 photographs and 50 criteria per classroom; tokens are input / output in thousands.
| Protocol | Requests | Image encodings | GPT-5.5 tokens (K) | Qwen3.8-27B tokens (K) |
|---|---|---|---|---|
| Full Images (per domain) | 6 | 576 | 543.5 / 11.6 | 458.0 / 9.1 |
| Full Images (per criterion) | 50 | 4,800 | 4,508.3 / 29.2 | 3,796.9 / 13.2 |
| PAJ | 242 | 96 | 2,623.2 / 243.1 | 821.7 / 79.7 |
PAJ reduces image encodings but increases requests and output tokens. Domain-level Full Images uses fewer input and output tokens than PAJ. Section 4.5 clarifies expert input; Appendix D.8 adds the comparisons and costs (Tables D14–D15).
W4 & Q3. Sample sizes, evaluation splits, and uncertainty estimates
All 15 models now use the same 129 classrooms and 12 loose–strict criterion pairs, with maximum boundary accuracy 37.72%. Both judgments must be correct; labels and predictions for negatively worded criteria are reversed before scoring.
| Model | Loose Acc. (%) | Boundary Acc. (%) | 95% CI |
|---|---|---|---|
| GPT-5.5 | 84.30 | 30.65 | [25.7, 35.6] |
| GPT-5.6 Sol | 84.69 | 26.33 | [21.8, 30.9] |
| Gemini 3.1 Pro | 81.65 | 28.29 | [23.8, 32.8] |
| Claude Sonnet 5 | 80.30 | 37.52 | [32.0, 43.0] |
| Qwen3.8-Max | 84.43 | 30.65 | [25.7, 35.8] |
We resample whole classrooms to compute confidence intervals. GPT-5.5's full-benchmark accuracy is 89.12%, with 95% CIs [88.2, 90.0] by classroom and [87.95, 90.22] by kindergarten (43 kindergartens). Differences and intervention gains use paired classroom resampling.
Each model below was run twice on 100 instances; SD denotes sample SD of accuracy.
| Model | Agreement (%) | Run 1 Acc. (%) | Run 2 Acc. (%) | Accuracy SD (points) |
|---|---|---|---|---|
| GPT-5.5 | 97.00 | 90.00 | 91.00 | 0.71 |
| Gemini 3.1 Pro | 94.00 | 86.00 | 86.00 | 0.00 |
| Qwen3.8-Max | 97.00 | 85.00 | 86.00 | 0.71 |
Overall decision agreement is 288/300 (96.00%). Section 4.4 and Table D9 now use the common evaluation set; Appendix D.1 adds clustered CIs and repeated-run results.
Q4. Books & Reading: anomalous results and parsing failure rates
Invalid-output/parsing-failure rates are 0.00% in Books and Reading and all other domains for Qwen3.5-4B, 9B, 27B and 35B. For 4B, counts are 0/1,032 and 0/5,418, respectively. Among the 76 diagnostic Books and Reading instances, 46.05% of reference labels are Satisfied. With domain-level requests, the 9B, 27B and 35B models predict Satisfied in 94.74–100.00% of cases. The table compares accuracy (%):
| Protocol | Qwen3.5-9B Acc. (%) | Qwen3.5-27B Acc. (%) | Qwen3.5-35B Acc. (%) |
|---|---|---|---|
| Full Images (per domain) | 46.05 | 51.32 | 40.79 |
| Full Images (per criterion) | 61.84 | 71.05 | 69.74 |
| Atomic Criterion | 85.53 | 80.26 | 89.47 |
Qwen3.5-27B states that C007's condition (more than half the books damaged) does not hold, yet predicts Satisfied. For C011/C012, models accept strict count requirements despite stating that the books cannot be counted. These cases and protocol comparisons point to criterion interpretation and request format. Appendix D.2 (Table D2) now includes these results.
Q5. Overlap with the cited study
PEVA-Bench shares 40 of its 43 kindergartens and 102 of its 129 classrooms with the cited AIED 2026 study. Existing onsite ECQRS-EC ratings for these 102 classrooms were reused for external validation.
The AIED study evaluates verbal interactions from audio transcripts; PEVA-Bench assesses the physical environment using 50 photo-verifiable criteria. Trained collectors followed a standardized photographic coverage protocol. Photographic labels were assigned independently without access to onsite ratings (Appendix B.1). Appendix B.5 now documents the overlap and rating reuse.
Gentle Reminder and Summary of Key Additions
Thank you again for your thoughtful review and constructive suggestions, which have helped us strengthen the paper. We would like to briefly highlight three additions in our response:
- W1/Q1: Experts outside the original annotation team independently assessed 60 sampled GPT-5.5 errors, without seeing the reference labels or model outputs. Their review changed the reference label in only 3 cases.
- W2/Q2: With full images, Qwen3.5-4B trails GPT-5.5 by 16.99 Macro-F1 points. With facts written by annotators who did not see the reference labels (Blind Facts), the gap narrows to 3.94 points on the same 135 instances.
- W4/Q3: All 15 models are now evaluated on the same 12 loose–strict criterion pairs across all 129 classrooms, with confidence intervals clustered by classroom. The highest boundary accuracy is 37.72%.
As the discussion period ends on November 18 (AoE), we would be very grateful if you could take a look at our responses. If they have addressed your concerns, we hope you might consider updating your evaluation accordingly. We are also happy to clarify anything that remains unclear.
Official Response to Reviewer GvJZ [1/2]
Thank you for the detailed review and helpful suggestions. We have added an independent label audit and label-blind diagnostic controls.
Independent review changed reference labels for 3/60 sampled GPT-5.5 errors (5.00%; 95% CI [1.0, 13.9]). We sampled these errors and 30 correct instances outside the diagnostic classrooms, stratified by domain and model correctness. Two experts outside the original annotation team reviewed full photographic records without labels or model outputs; a third adjudicated disagreements. Labels were retained for the other 57 errors and all 30 correct instances. The Figure 4 “scientific property” label was also retained: GPT-5.5 substitutes “scientific property or theme” and accepts sorting by kind.
GPT-5.5 agrees with onsite ratings on 80.53% of the benchmark, versus 91.04% for photographic labels. Onsite ratings support the photographic reference in 690/702 model errors (98.29%). Together with the blind audit, this supports the reference labels for most errors examined.
On the original 500 diagnostic instances, the two reviewers' first independent, pre-adjudication judgments average 95.00% accuracy against adjudicated labels. Their 90.80% agreement (454/500) does not set an upper bound on model accuracy. Model results use the 447 instances with expert evidence sets: GPT-5.5 reaches 91.72% with Full Images and 94.63% with Oracle Images.
A second annotator independently reviewed 100 errors. The first and second annotators assigned criterion application to 82% and 79%, respectively, with Cohen's κ = 0.65 (95% CI [0.43, 0.83]). Appendices B.4–B.5 now report the label audit, human accuracy and onsite comparison; Appendix C.4 adds error-attribution agreement.
Blind Facts narrows Qwen3.5-4B's gap to GPT-5.5 on 135 matched instances. The Macro-F1 difference (Qwen3.5-4B minus GPT-5.5) changes from −16.99 with Full Images to −3.94 with Blind Facts (95% CI [−7.0, −0.9]). Annotators outside the original team used the criterion and complete photographic records, without reference labels or original evidence annotations. They retained the original factual fields and source-image links. The table reports Macro-F1 (%) on the same instances, covering the 15 unique criteria in the loose–strict pairs.
Both fact conditions provide criterion-relevant observations, so gains reflect evidence selection and visual interpretation. Across these 135 instances, commercial-model errors under Blind Facts predominantly follow Satisfied→Unsatisfied:
Shared Facts and Shared Blind Facts merge each pair's expert or label-blind facts, respectively. Both criterion requests receive identical text. Boundary accuracy (%) on the 45 diagnostic pairs is:
Each label-blind human reviewer correctly judged 40/45 pairs in their first independent assessment from Shared Facts. Across 13 models, Shared Blind Facts gives 39.32% mean boundary accuracy. Predicting Satisfied for both criteria accounts for 76.06% of errors on these pairs, where the loose requirement is met and the strict one unmet. Criterion-application errors thus persist with identical factual input.
Text oracles are established in Prism, Caption This, Reason That, and Bad Seeing or Bad Thinking; RuleArena studies rule application in text. PEVA-Bench combines complete records, professional criteria, onsite validation, and oracle and loose–strict diagnostics. Sections 4.3–4.4 and Tables D12 and D16 now report these controls; Section 2 discusses the related work.