acceptodds
Under review as a conference paper at ICLR 2027

Seeing Is Not Judging: Evidence-Grounded Evaluation of Multimodal Models for Professional Preschool Environment Assessment

Abstract

Can multimodal models assess a real environment the way a trained professional does? Professional assessment asks a model to decide, from the complete visual record of a site, whether each written criterion of a rating scale holds, and to justify every decision with evidence. This requires finding the few relevant views among about a hundred photographs and applying the conditions of quantity, accessibility, and relation that each criterion states. Existing benchmarks either supply the relevant views or leave the criterion implicit, and so cannot tell whether a model failed to *see* the evidence or failed to *judge* it. We introduce **PEVA-Bench** (Preschool Environment Visual Assessment), a benchmark built on a professional rating scale: 6,450 expert decisions on 50 criteria over the complete photographic records of 129 classrooms, whose labels agree with independent onsite ratings on 91.0% of these decisions. Expert evidence and facts that can replace a model's own, together with criterion pairs that differ only in strictness, separate seeing from judging. Across fifteen models, the two fail in different ways. First, **Seeing separates models**: with expert facts, Qwen3.5-4B reaches 91.42 Macro-F1 alongside GPT-5.5's 90.46. Second, **Judging limits the strongest**: 80% of GPT-5.5's incorrect assessments involve criterion application, and 98% of these assign Satisfied to an Unsatisfied reference; no model tested tells a looser criterion from its stricter version in more than 42% of the cases that require it. PAJ (Perceive, Aggregate, Judge) structures visual evidence and criterion application, improving eleven of the thirteen models tested, most of all those limited by seeing, by up to 21 Macro-F1 points. *Seeing the evidence and applying the criterion emerge as distinct capabilities in professional assessment.* Code is available at [this anonymous repository](https://anonymous.4open.science/r/PEVA-Bench-68DE).

Primary area:
datasets and benchmarks
Keywords:
multimodal large language models, benchmark, multi-image reasoning, evidence-grounded evaluation, preschool environment assessment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GvJZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PEVA-Bench, a benchmark designed to evaluate whether multimodal models can act like professional assessors: given a complete photographic record of a classroom (averaging 96 photos) and a written rubric criterion, the model must determine whether the criterion is satisfied while citing supporting evidence images and visual facts. The dataset comprises 12,418 photos collected from 129 classrooms across 43 kindergartens in Shenzhen. The authors adapt 50 photo-verifiable criteria from the ECQRS-EC scale, yielding 6,450 annotated instances; photo-based labels achieve a 91.0% agreement rate (κ=0.82\kappa = 0.82) with independent on-site assessments. In addition, the authors construct a 10-classroom diagnostic subset that provides expert evidence images, minimal sufficient subsets, and expert-verified facts as oracle inputs, alongside 12 pairs of lenient-versus-strict criteria.

There are four main empirical findings: (1) among the 15 evaluated models, GPT-5.5 performs best, achieving 89.12% accuracy; (2) when provided with oracle expert facts, small models perform comparably to GPT-5.5, with Qwen3.5-4B reaching a Macro-F1 of 91.42 compared to GPT-5.5's 90.46; (3) 80% of GPT-5.5's errors involve rubric application, 98% of which stem from misclassifying Unsatisfied instances as Satisfied; and (4) on boundary cases defined by the lenient–strict criterion pairs, no model exceeds 42% accuracy. Finally, the authors propose the Perceive-Aggregate-Judge (PAJ) pipeline, which improves performance across 11 out of 13 evaluated models.

Critical strengths

New dataset contribution: This benchmark uniquely combines complete multi-view photographic records of real-world environments, a professional assessment scale, and external validation against independent on-site evaluations, which is a rare combination in existing benchmarks.

Original diagnostic design: Two diagnostic mechanisms stand out: first, a hierarchical oracle ladder progressing from the full photographic record to expert evidence images, minimal sufficient subsets, and finally expert-verified facts, paired with size-matched random-image controls; second, lenient-strict criterion pairs that hold the photographic record fixed while varying only the strictness of the rubric. To the best of my knowledge, the latter has no direct precedent in vision benchmarks, with the closest analogue in the text domain being RuleArena (ACL 2025).

Comprehensive evaluation and clear presentation: The evaluation spans 15 models, including frontier commercial models and open-weight models ranging from 4B to 35B parameters, and provides fine-grained breakdowns across domains, categories, and error types. The finding that strong models exhibit a pronounced bias toward predicting Satisfied carries an important cautionary message for self-assessment applications. Furthermore, ethical considerations are handled carefully, with IRB approval, face/privacy blurring, and controlled data release.

Critical weaknesses

Frontier models have reached the label-noise ceiling, so the conclusion that "judgment capability bottlenecks the strongest models" may partly reflect inherent label ambiguity. On the diagnostic subset, the agreement rate between two independent human reviewers is only 90.8% (κ=0.816\kappa = 0.816), whereas GPT-5.5 achieves 91.72% on the same subset under the Full Images condition and reaches 94.63% under the Oracle Images condition. Across the full dataset, GPT-5.5 achieves 89.12% accuracy, while the agreement rate between photo-based labels and on-site evaluations is 91.0%. Consequently, a substantial fraction of GPT-5.5's 702 errors may fall into genuinely ambiguous borderline cases, for instance, whether sorting shells, pinecones, and stones by type qualifies as sorting by "scientific attributes." Furthermore, error attribution is performed within the same annotation pipeline after inspecting the model's cited images, and the paper does not report inter-annotator agreement for this error attribution step. I recommend: (1) having independent experts blindly adjudicate a sample of model "errors" and reporting the proportion of reference labels that get overturned; (2) reporting human expert accuracy relative to the adjudicated labels; and (3) reporting the direct agreement rate between model predictions and on-site assessments.

Oracle Facts are authored with reference to the ground-truth labels, introducing label leakage. Appendix B.3 notes that experts select evidence images and write factual descriptions based on both the criterion and its reference label. Because these facts have already undergone relevance filtering and are phrased using vocabulary aligned with the rubric, the judgment task largely degenerates into straightforward textual entailment. This has two major consequences: first, it is entirely expected that Qwen3.5-4B performs comparably to GPT-5.5 under this setting, and both sit near the ~91% label-noise ceiling; second, the performance gain attributed to "seeing" is confounded with the judgment skill of knowing which visual facts are relevant to the rubric in the first place. Moreover, using text oracles to disentangle perception from reasoning is already a well-established paradigm, e.g., Prism (NeurIPS 2024), Caption This, Reason That (NeurIPS 2025), and Bad Seeing or Bad Thinking (ICML 2026). While the paper's incremental value lies in its multi-step oracle ladder and specialized domain, establishing that "seeing and judging are distinct capabilities" as a clean empirical insight requires eliminating this label leakage first.

Limited contribution of the proposed PAJ pipeline. PAJ follows the well-known perceive-then-judge paradigm, closely resembling Socratic Models, Prism, LLoVi (EMNLP 2024; segment-wise captioning followed by aggregation), and checklist decomposition in TICKing All the Boxes. Specifically: (1) both the observation checklist and the 8-category spatial zone mapping rely on hand-crafted domain priors, making the claim that PAJ operates "without any expert input" inaccurate; (2) PAJ actually degrades GPT-5.5's Macro-F1 from 90.76 to 89.16, failing to resolve the paper's own diagnosis of frontier-model bottlenecks; (3) a large portion of the gains on smaller models comes simply from atomic criterion decomposition, e.g., Qwen3.5-9B jumps from 66.96 to 82.71 from decomposition alone; and (4) inference cost and token overhead are not reported.

Small sample sizes, inconsistent evaluation splits across models, and missing uncertainty estimates. The diagnostic subset spans only 10 classrooms (447 instances), with strong intra-classroom correlation across instances. In the lenient-strict pair experiments, commercial models are evaluated only on these 10 classrooms (120 pairs in total, containing roughly 45 boundary pairs, meaning GPT-5.5's 31.11% corresponds to roughly 14/4514/45, with a wide 95% confidence interval of $\approx $[18%, 46%]), whereas open-weight models are evaluated across all 129 classrooms. Thus, the headline claim that "no model exceeds 42%" conflates two different evaluation sets. In addition, the paper lacks confidence intervals clustered at the classroom or kindergarten level, as well as multi-run variance estimates for commercial APIs.

These issues do not negate the value of the new dataset. I currently lean toward a weak reject, though I will consider raising my score if the rebuttal can provide sufficient supplementary evidence and a convincing explanation.

Questions for authors

Q1. When a sample of GPT-5.5's error instances is blindly re-adjudicated by independent domain experts, what proportion of the original reference labels are overturned/modified?

Q2. If the Oracle Facts are authored by annotators who are blinded to the ground-truth reference labels, what is the performance gap between smaller open-weight models and GPT-5.5? Furthermore, under this blind Oracle Facts condition, what is the directional error breakdown for frontier models, i.e., do they still predominantly misclassify Unsatisfied as Satisfied?

Q3. When evaluating commercial models across all 129 classrooms on the full set of lenient–strict criterion pairs, what is the boundary accuracy and its corresponding confidence interval?

Q4. What is the root cause of the anomalous evaluation results observed for the Qwen3.5 series in the Books & Reading domain, and what are their exact invalid output/parsing failure rates?

Q5. Does this work share any overlap in participating kindergartens, classrooms, or on-site assessment records with the AIED 2026 study on ECQRS-EC-based LLM kindergarten evaluation?

5 comments
AuthorsOpenReview ↗

Official Response to Reviewer GvJZ [1/2]

Thank you for the detailed review and helpful suggestions. We have added an independent label audit and label-blind diagnostic controls.

W1 & Q1. Label ambiguity, human expert accuracy, and error attribution

Independent review changed reference labels for 3/60 sampled GPT-5.5 errors (5.00%; 95% CI [1.0, 13.9]). We sampled these errors and 30 correct instances outside the diagnostic classrooms, stratified by domain and model correctness. Two experts outside the original annotation team reviewed full photographic records without labels or model outputs; a third adjudicated disagreements. Labels were retained for the other 57 errors and all 30 correct instances. The Figure 4 “scientific property” label was also retained: GPT-5.5 substitutes “scientific property or theme” and accepts sorting by kind.

GPT-5.5 agrees with onsite ratings on 80.53% of the benchmark, versus 91.04% for photographic labels. Onsite ratings support the photographic reference in 690/702 model errors (98.29%). Together with the blind audit, this supports the reference labels for most errors examined.

On the original 500 diagnostic instances, the two reviewers' first independent, pre-adjudication judgments average 95.00% accuracy against adjudicated labels. Their 90.80% agreement (454/500) does not set an upper bound on model accuracy. Model results use the 447 instances with expert evidence sets: GPT-5.5 reaches 91.72% with Full Images and 94.63% with Oracle Images.

Human reviewerCorrect / totalAccuracy (%)
A473/50094.60
B477/50095.40

A second annotator independently reviewed 100 errors. The first and second annotators assigned criterion application to 82% and 79%, respectively, with Cohen's κ = 0.65 (95% CI [0.43, 0.83]). Appendices B.4–B.5 now report the label audit, human accuracy and onsite comparison; Appendix C.4 adds error-attribution agreement.

W2 & Q2. Label leakage and blind Oracle Facts

Blind Facts narrows Qwen3.5-4B's gap to GPT-5.5 on 135 matched instances. The Macro-F1 difference (Qwen3.5-4B minus GPT-5.5) changes from −16.99 with Full Images to −3.94 with Blind Facts (95% CI [−7.0, −0.9]). Annotators outside the original team used the criterion and complete photographic records, without reference labels or original evidence annotations. They retained the original factual fields and source-image links. The table reports Macro-F1 (%) on the same instances, covering the 15 unique criteria in the loose–strict pairs.

ModelFull ImagesOracle FactsBlind Facts
GPT-5.582.0582.9090.93
Gemini 3.1 Pro86.9290.2390.20
Qwen3.8-Max83.6888.0490.23
Qwen3.5-4B65.0688.5387.00
Qwen3.5-9B66.3886.4574.81
Qwen3.5-27B73.7188.7391.63
Qwen3.5-35B66.9385.8182.83
Qwen3.8-27B74.4187.2791.63
Gemma4-E4B60.7387.3188.73
Gemma4-12B70.2390.1590.00
Gemma4-26B71.4188.6493.83
Gemma4-31B82.1788.0090.89
GLM-4.6V-Flash55.5879.9387.39

Both fact conditions provide criterion-relevant observations, so gains reflect evidence selection and visual interpretation. Across these 135 instances, commercial-model errors under Blind Facts predominantly follow Satisfied→Unsatisfied:

ModelUnsatisfied→SatisfiedSatisfied→Unsatisfied
GPT-5.539
Gemini 3.1 Pro310
Qwen3.8-Max211

Shared Facts and Shared Blind Facts merge each pair's expert or label-blind facts, respectively. Both criterion requests receive identical text. Boundary accuracy (%) on the 45 diagnostic pairs is:

Model / reviewerShared FactsShared Blind Facts
GPT-5.535.5631.11
Gemini 3.1 Pro26.6742.22
Qwen3.8-Max46.6744.44
Human reviewers (first round, each)88.89—

Each label-blind human reviewer correctly judged 40/45 pairs in their first independent assessment from Shared Facts. Across 13 models, Shared Blind Facts gives 39.32% mean boundary accuracy. Predicting Satisfied for both criteria accounts for 76.06% of errors on these pairs, where the loose requirement is met and the strict one unmet. Criterion-application errors thus persist with identical factual input.

Text oracles are established in Prism, Caption This, Reason That, and Bad Seeing or Bad Thinking; RuleArena studies rule application in text. PEVA-Bench combines complete records, professional criteria, onsite validation, and oracle and loose–strict diagnostics. Sections 4.3–4.4 and Tables D12 and D16 now report these controls; Section 2 discusses the related work.

AuthorsOpenReview ↗

Official Response to Reviewer GvJZ [2/2]

W3. PAJ's contribution, expert input, and inference cost

PAJ is a baseline for testing the benchmark's diagnostic value. It builds on Socratic Models and Prism, with related approaches in LLoVi and TICKing All the Boxes. The checklist, area mappings and atomic conditions are expert-designed; PAJ generates its own classroom facts without instance-specific expert evidence or facts.

On the original subset, PAJ outperforms Atomic Criterion on 12/13 models with identical atomic conditions. The mean gain across all 13 is 3.85 Macro-F1 points, reflecting perception and aggregation beyond criterion decomposition. With unchanged submission prompts, PAJ also improves 12/13 models on ten additional classrooms (500 instances) under the same instrument, averaging 8.31 points across all 13. Eight paired classroom-bootstrap 95% CIs exclude zero.

On the original 447 instances, GPT-5.5's Unsatisfied→Satisfied errors fall from 31 to 14. Satisfied→Unsatisfied errors rise from six to 32; Macro-F1 changes by −1.61 points. Costs below cover 96 photographs and 50 criteria per classroom; tokens are input / output in thousands.

ProtocolRequestsImage encodingsGPT-5.5 tokens (K)Qwen3.8-27B tokens (K)
Full Images (per domain)6576543.5 / 11.6458.0 / 9.1
Full Images (per criterion)504,8004,508.3 / 29.23,796.9 / 13.2
PAJ242962,623.2 / 243.1821.7 / 79.7

PAJ reduces image encodings but increases requests and output tokens. Domain-level Full Images uses fewer input and output tokens than PAJ. Section 4.5 clarifies expert input; Appendix D.8 adds the comparisons and costs (Tables D14–D15).

W4 & Q3. Sample sizes, evaluation splits, and uncertainty estimates

All 15 models now use the same 129 classrooms and 12 loose–strict criterion pairs, with maximum boundary accuracy 37.72%. Both judgments must be correct; labels and predictions for negatively worded criteria are reversed before scoring.

ModelLoose Acc. (%)Boundary Acc. (%)95% CI
GPT-5.584.3030.65[25.7, 35.6]
GPT-5.6 Sol84.6926.33[21.8, 30.9]
Gemini 3.1 Pro81.6528.29[23.8, 32.8]
Claude Sonnet 580.3037.52[32.0, 43.0]
Qwen3.8-Max84.4330.65[25.7, 35.8]

We resample whole classrooms to compute confidence intervals. GPT-5.5's full-benchmark accuracy is 89.12%, with 95% CIs [88.2, 90.0] by classroom and [87.95, 90.22] by kindergarten (43 kindergartens). Differences and intervention gains use paired classroom resampling.

Each model below was run twice on 100 instances; SD denotes sample SD of accuracy.

ModelAgreement (%)Run 1 Acc. (%)Run 2 Acc. (%)Accuracy SD (points)
GPT-5.597.0090.0091.000.71
Gemini 3.1 Pro94.0086.0086.000.00
Qwen3.8-Max97.0085.0086.000.71

Overall decision agreement is 288/300 (96.00%). Section 4.4 and Table D9 now use the common evaluation set; Appendix D.1 adds clustered CIs and repeated-run results.

Q4. Books & Reading: anomalous results and parsing failure rates

Invalid-output/parsing-failure rates are 0.00% in Books and Reading and all other domains for Qwen3.5-4B, 9B, 27B and 35B. For 4B, counts are 0/1,032 and 0/5,418, respectively. Among the 76 diagnostic Books and Reading instances, 46.05% of reference labels are Satisfied. With domain-level requests, the 9B, 27B and 35B models predict Satisfied in 94.74–100.00% of cases. The table compares accuracy (%):

ProtocolQwen3.5-9B Acc. (%)Qwen3.5-27B Acc. (%)Qwen3.5-35B Acc. (%)
Full Images (per domain)46.0551.3240.79
Full Images (per criterion)61.8471.0569.74
Atomic Criterion85.5380.2689.47

Qwen3.5-27B states that C007's condition (more than half the books damaged) does not hold, yet predicts Satisfied. For C011/C012, models accept strict count requirements despite stating that the books cannot be counted. These cases and protocol comparisons point to criterion interpretation and request format. Appendix D.2 (Table D2) now includes these results.

Q5. Overlap with the cited study

PEVA-Bench shares 40 of its 43 kindergartens and 102 of its 129 classrooms with the cited AIED 2026 study. Existing onsite ECQRS-EC ratings for these 102 classrooms were reused for external validation.

The AIED study evaluates verbal interactions from audio transcripts; PEVA-Bench assesses the physical environment using 50 photo-verifiable criteria. Trained collectors followed a standardized photographic coverage protocol. Photographic labels were assigned independently without access to onsite ratings (Appendix B.1). Appendix B.5 now documents the overlap and rating reuse.

AuthorsOpenReview ↗

Gentle Reminder and Summary of Key Additions

Thank you again for your thoughtful review and constructive suggestions, which have helped us strengthen the paper. We would like to briefly highlight three additions in our response:

  • W1/Q1: Experts outside the original annotation team independently assessed 60 sampled GPT-5.5 errors, without seeing the reference labels or model outputs. Their review changed the reference label in only 3 cases.
  • W2/Q2: With full images, Qwen3.5-4B trails GPT-5.5 by 16.99 Macro-F1 points. With facts written by annotators who did not see the reference labels (Blind Facts), the gap narrows to 3.94 points on the same 135 instances.
  • W4/Q3: All 15 models are now evaluated on the same 12 loose–strict criterion pairs across all 129 classrooms, with confidence intervals clustered by classroom. The highest boundary accuracy is 37.72%.

As the discussion period ends on November 18 (AoE), we would be very grateful if you could take a look at our responses. If they have addressed your concerns, we hope you might consider updating your evaluation accordingly. We are also happy to clarify anything that remains unclear.

Reviewer_brBqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PEVA-Bench, a benchmark for evaluating multimodal models on professional preschool environment assessment. The key contribution is a diagnostic framework that distinguishes Seeing from Judging. Through oracle evidence experiments, error analysis, and loose–strict criterion pairs, the authors find that weaker models are primarily limited by visual perception, while stronger models frequently relax assessment criteria despite recognizing relevant evidence. Based on these findings, the paper introduces PAJ (Perceive, Aggregate, Judge), a structured evaluation pipeline that improves performance for most tested models.

Critical strengths

Strength

  1. Well-constructed benchmark with external validation. PEVA-Bench uses real-world classroom photographs and professionally defined assessment criteria.
  2. The oracle evidence experiments and loose–strict criterion pairs offer a useful way to investigate different sources of model errors. The finding that stronger models systematically relax assessment criteria is particularly interesting.
  3. The paper evaluates 15 models across different scales and families, providing detailed error analyses, controlled comparisons, and ablations.
Critical weaknesses

Weakness

  1. The paper reports that 216 instances require multiple complementary images for assessment. However, it would be helpful to provide a more detailed distribution of the minimum sufficient evidence-set sizes. For example, what is the maximum number of images required, and what proportion of instances require two, three, or more images?
  2. The Oracle Facts setting provides expert-annotated, criterion-relevant visual facts, which appear to contain much of the information needed for the final judgment. However, models still make incorrect predictions under this setting. I am curious about the main causes of these remaining errors. A more detailed error breakdown or several representative failure cases would help clarify the limitations of Judging.
  3. PAJ mainly combines established techniques, including per-image description, evidence aggregation, and atomic criterion decomposition. While the improvements are encouraging, the methodological novelty is relatively incremental. Additionally, the paper does not sufficiently analyze the inference cost introduced by the multi-stage pipeline, especially given that PAJ does not consistently improve stronger models.
  4. It would be interesting to introduce an additional oracle condition in which relevant objects or regions in the Oracle Images are highlighted with bounding boxes (e.g., "Oracle Images with Region Annotations"). Such experiments, if feasible, could help further distinguish failures in visual localization from those in visual understanding and criterion application.
Questions for authors

See weaknesses above. If the authors can adequately address the above weaknesses, I would be inclined to support acceptance.

3 comments
AuthorsOpenReview ↗

Official Response to Reviewer brBq

Thank you for your suggestions on the diagnostic analysis. We provide the additional controls and clarify PAJ's contribution and inference cost.

W1. Distribution of minimum sufficient evidence-set sizes

Across the 447 diagnostic instances with expert evidence sets, the minimum sufficient evidence-set size has a mean of 1.89 images and a maximum of seven:

Minimum imagesInstancesProportion (%)
123151.68
211024.6
35913.2
4235.1
≥5245.4

The ≥5 group contains 22 cases requiring five images, one requiring six, and one requiring seven. Appendix B.2 now reports the full distribution.

W2. Causes of remaining errors under Oracle Facts

Literal reading was the most common behavior in the audited cases. Models required explicit confirmation of conditions that human reviewers judged satisfied from the same facts. We examined all 79 Oracle Facts errors from GPT-5.5 and Qwen3.5-9B. Two reviewers independently judged the facts without seeing reference labels. At least one matched the reference judgment in 75 cases; neither did so in the remaining four. The table classifies those 75 cases:

Model behaviorGPT-5.5Qwen3.5-9BTotal
Literal reading332659
Relaxation4610
Evidence integration156

In C010, GPT-5.5 requires explicit confirmation of a clear entrance; both reviewers find that the facts establish children's access to the reading area. C025 requires books in the mathematics area. Qwen3.5-9B acknowledges that no books are shown but counts teaching cards and worksheets as books. Both reviewers judge C025 Unsatisfied; the model predicts Satisfied even with evidence images. Evidence integration errors involve miscounting or incorrectly combining observations.

On the 447 instances, the 13 models make 477 Oracle Facts errors: 108 Unsatisfied→Satisfied (22.64%) and 369 Satisfied→Unsatisfied (77.36%). Under Full Images, 58.21% of errors are Unsatisfied→Satisfied. Facts therefore change the dominant error direction. Labels denote the truth of the criterion statement.

Oracle Images + Facts reduces the 13-model error total from 477 to 343 (28.09%). For GPT-5.5, it corrects 36 of 41 errors and introduces six, leaving 11. This supports using images and facts together. Section 4.3 and Appendix D.10.1 now include the analysis and examples.

W3. PAJ's methodological novelty and inference cost

PAJ is a baseline for testing whether PEVA-Bench's diagnostics guide improvements. With atomic conditions fixed, PAJ outperforms Atomic Criterion on 12/13 models, with a mean gain of 3.85 Macro-F1 points across all 13. This comparison measures the benefit of per-image perception and aggregation beyond criterion decomposition.

With unchanged submission prompts, PAJ improves Macro-F1 for 12/13 models on ten additional classrooms (500 instances). The mean gain across all 13 is 8.31 points; eight paired classroom-bootstrap 95% CIs exclude zero. For GPT-5.5 on the original 447 instances, Unsatisfied→Satisfied errors fall from 31 to 14, but Satisfied→Unsatisfied errors rise from six to 32, giving a net Macro-F1 change of −1.61 points.

Costs are per classroom with 96 photographs and 50 criteria. Tokens are input / output in thousands; image encodings count photographs processed across requests.

ProtocolRequestsImage encodingsGPT-5.5 tokens (K)Qwen3.8-27B tokens (K)
Full Images (per domain)6576543.5 / 11.6458.0 / 9.1
Full Images (per criterion)504,8004,508.3 / 29.23,796.9 / 13.2
PAJ242962,623.2 / 243.1821.7 / 79.7

PAJ reuses each photograph's facts, reducing image encodings relative to single-criterion assessment while increasing requests and output tokens. Domain-level Full Images uses fewer input and output tokens than PAJ. Appendix D.8 now includes the component analysis, replication, and costs (Tables D14–D15).

W4. Oracle Images with Region Annotations

Region annotations showed no consistent improvement across model groups on 80 instances, stratified by domain, reference label, and single-/multi-view evidence. Annotators boxed relevant regions using the criterion and evidence images, without reference labels. All 13 models received the same instances; the table aggregates their errors.

InputCommercial models (3)Open-weight models (10)
Oracle Images859
Images + Regions561
Oracle Images + Facts646

Relative to Oracle Images, errors change by −37.5% [−83.3, 0.0] for commercial models and +3.4% [−18.2, 29.6] for open-weight models. Both 95% CIs include zero. Section 4.3 and Appendix D.10.2 (Table D13) now report this control.

Thank you for the detailed responses and additional experiments. Your clarifications and new results have addressed most of my previous concerns. I believe PEVA-Bench provides potential practical value in educational environment assessment. Overall, I am inclined to raise my score to 3.

We appreciate your positive assessment and recognition of PEVA-Bench’s practical potential. Thank you for your time and thoughtful suggestions throughout the review process.

Reviewer_aDK6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PEVA-Bench for evidence grounded assessment of preschool environments. The benchmark has 6.450 classroom-criterion decisions covering 129 classrooms and 50 professional criteria, with photographic labels validated against independent onsite ratings. A diagnostic subset provides expert evidence images and visual facts, while loose-strict criterion pairs examine sensitivity to assessment requirements. The paper evaluates 15 multimodal models and introduces PAJ, a perception, aggregation, and judgement pipeline motivated by the diagnostic results.

Critical strengths
  • The benchmark construction and validation are respectfully substantial. The complete classroom records, criterion-level labels, and source-linked evidence provide considerably more information than final-answer annotations only. The comparison with independent onsite rating is especially valuable, with 91.0% agreement providing an external check on the photographic labels. I appreciate the effort to establish that the benchmark reflects a meaningful professional assessment tasks.

  • The paper goes beyond reporting model scores. Oracle evidence conditions and criterion pairs provide useful diagnostic views, and PAJ makes an effort to act on those things. The comparisons with atomic-criterion prompting and unfiltered per-image facts also help explain which parts of the pipeline contribute.

Critical weaknesses
  • The benchmark's value for developing transferable methods remains under-validated. PAJ is a useful initial demonstration, but it uses an observation checklist, area mappings, and atomic conditions tied to the same assessment instrument. Its reported gains are evaluated on 477 instances from the ten-classroom diagnostic subset, rather than on separately identified evaluation set. I could not find a clear separation between the records used to develop the diagnosis and prompts and those used to evaluate the resulting intervention. This makes it difficult to assess whether the benchmark supports reusable improvements or mainly refinements for this fixed collection of criteria. I am not asking for a new model or a second application domain, but a frozen intervention evaluated on additional classroom or held-out criteria would strengthen the contribution. I appreciate the benchmark's richness and careful annotation. So just to be clear: my concern is mostly whether the current experiments sufficiently establish its diagnostic interpretation and its usefulness for developing improvements that generalize beyond the analyzed subset.

  • The interpretation of boundary failures needs a stronger control. The loss-strict pairs hold the photographs fixed, but the models still perform perception separately in each request if I understand it correctly. A stricter requirement can demand more accurate counting counting, category recognition, or accessibility assessment, so a boundary error does not necessarily identify a failure of criterion application. The oracle experiments and annotated error analysis are useful complementary evidence, but testing both criteria against the same expert fact representation would more directly give the proposed separation.

Questions for authors

Q1: Which classrooms and model outputs were used to develop PAJ's observation checklist, area mappings, and atomic criterion formulation? Were these choices frozen before evaluation?

Q2: How does loose-strict boundary accuracy change when both criteria receive the same expert facts, sufficient to decide either criterion? Does PAJ improve boundary accuracy specifically, rather than only overall classification performance?

1 comment
AuthorsOpenReview ↗

Official Response to Reviewer aDK6

Thank you for recognizing the benchmark's annotation quality and external validation. We have evaluated PAJ on additional classrooms and examined boundary errors with the factual input held fixed.

W1 & Q1. Transferable methods and PAJ development

PAJ's observation checklist, area mappings, atomic conditions, and prompts were fixed before PAJ evaluation and used unchanged on the additional classrooms. Domain experts derived these components from ECQRS-EC and assessment experience, without tailoring them to individual classrooms or model outputs. Analysis of the original ten diagnostic classrooms informed the three-stage design; PAJ evaluation on these classrooms covered 447 instances with expert evidence sets.

The unchanged PAJ baseline improves Macro-F1 for 12/13 models on ten additional classrooms. The mean gain is 8.31 points across all 13 models and 10.83 across the ten open-weight models. These classrooms were sampled from the 33 kindergartens outside the diagnostic subset, following the original sampling rule. We evaluated Full Images and PAJ on all 50 criteria per classroom (500 instances), using one criterion per judgment request. The table reports ΔMacro-F1 (PAJ minus Full Images) with paired classroom-bootstrap 95% confidence intervals; eight intervals exclude zero.

ModelΔMacro-F1 (points)95% CI
GPT-5.5−1.87[−3.8, 0.1]
Gemini 3.1 Pro+1.60[−1.7, 4.9]
Qwen3.8-Max+0.03[−3.5, 3.6]
Qwen3.5-4B+21.53[16.2, 26.8]
Qwen3.5-9B+31.67[25.7, 37.3]
Qwen3.5-27B+4.63[1.3, 8.2]
Qwen3.5-35B+19.38[14.5, 24.1]
Qwen3.8-27B+1.36[−2.6, 5.2]
Gemma4-E4B+7.65[2.4, 13.5]
Gemma4-12B+8.80[5.1, 12.8]
Gemma4-26B+4.50[0.3, 8.4]
Gemma4-31B+4.05[−2.5, 10.5]
GLM-4.6V-Flash+4.75[0.4, 9.3]

Diagnostic headroom is the difference in Macro-F1 between Oracle Facts and Full Images on the original subset. Greater headroom is associated with larger PAJ gains on the additional classrooms: Spearman's ρ is 0.87 across all models and 0.72 among open-weight models. Together, the replication and correlation support using the benchmark's diagnostics to guide improvements on additional classrooms under the same assessment instrument. Section 4.5 and Appendix D.8 now describe PAJ's development and replication with unchanged prompts; Table D14 reports the replication results.

W2 & Q2. Boundary accuracy with the same expert facts

On the 45 diagnostic boundary pairs, mean boundary accuracy across 13 models rises from 33.33% with Full Images to 38.63% with Shared Facts; PAJ reaches 42.39%. For Shared Facts, we merged each pair's expert facts and supplied the same text in separate requests for the two criteria. Shared Blind Facts repeats this procedure with facts written without access to reference labels or original expert evidence annotations. A pair is correct when the model judges the loose requirement as met and the strict requirement as unmet. Labels and predictions for negatively worded criteria are reversed before scoring. Gains below are relative to Full Images, with paired classroom-bootstrap 95% CIs.

InputBoundary Acc. (%)Gain (points) [95% CI]
Full Images33.33—
Shared Facts38.63+5.30 [0.7, 9.5]
Shared Blind Facts39.32+5.98 [2.2, 10.4]
PAJ42.39+9.06 [3.0, 14.5]

Two human reviewers, blinded to the labels, each correctly judged 40/45 pairs (88.89%) in their first independent assessment from Shared Facts alone. The model mean was 38.63% with the same input. Predicting Satisfied for both criteria accounts for 80.22% of Shared Facts boundary errors and 76.06% under Shared Blind Facts. These results indicate difficulty applying different criteria to the same factual record.

PAJ also improves boundary accuracy on 52 boundary pairs in the additional classrooms, from 29.14% to 41.12%, a gain of 11.98 points [9.3, 14.8]. We have added the shared-fact controls and PAJ's boundary results to Appendix D.9, with the main findings summarized in Sections 4.4–4.5.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

Dear Area Chair and Reviewers,

Thank you all for your time and thoughtful feedback. We summarize our responses below.

Strengths noted by reviewers. All three reviewers (aDK6, brBq, GvJZ) valued PEVA-Bench's careful construction and its validation against independent onsite ratings. They also found its oracle conditions and loose–strict criterion pairs useful for separating sources of model error. Reviewer GvJZ added that, to their knowledge, the criterion pairs have no direct precedent in vision benchmarks.

New results. Following the reviewers' suggestions, we added experiments and analyses that further support the paper's main findings:

  1. Label reliability (GvJZ). In a blind audit, experts outside the original annotation team re-examined a sample of 60 GPT-5.5 errors without seeing the reference labels or model outputs. They changed only 3 labels. Onsite ratings also agree with the photographic labels on 690 of GPT-5.5's 702 errors. Reviewer GvJZ confirmed that the blind audit alleviates their concerns about label quality.

  2. Separating seeing from judging (aDK6, brBq, GvJZ). Label-blind facts narrow Qwen3.5-4B's Macro-F1 gap to GPT-5.5 from 16.99 points with full images to 3.94. This places most of the gap in selecting and interpreting visual evidence. When both criteria in a pair receive identical facts, two label-blind human reviewers each judge 40 of 45 pairs correctly (88.89%). The 13 models average only 38.63% (39.32% with label-blind facts). Criterion-application errors thus persist with identical factual input. We further classify the remaining Oracle Facts errors and add a region-annotation control on Oracle Images (brBq).

  3. PAJ: generalization and cost (aDK6, brBq, GvJZ). PAJ is a baseline for testing whether the benchmark's diagnostics can guide improvements. With the submission prompts unchanged, it improves Macro-F1 for 12 of 13 models on ten additional classrooms (mean +8.31 points). Boundary accuracy on these classrooms rises from 29.14% to 41.12%. With identical atomic conditions, PAJ outperforms Atomic Criterion on 12 of 13 models, so its gains go beyond criterion decomposition. We also report its inference cost: PAJ encodes each photograph only once per classroom.

  4. Common evaluation set and uncertainty (GvJZ). All 15 models are now evaluated on the same 12 loose–strict criterion pairs across all 129 classrooms, and no model exceeds 37.72% boundary accuracy. We add 95% CIs clustered by classroom and by kindergarten. Repeated runs of three commercial models agree on 96.00% of decisions. Reviewer GvJZ confirmed that the common-set evaluation alleviates their concerns about statistical reliability.

In addition, we report the full distribution of minimum evidence-set sizes (brBq) and analyze the Qwen3.5 results in Books and Reading (GvJZ).

Revisions. The revised PDF adds these results to Sections 2 and 4.3–4.5 and Appendices B.2, B.4–B.5, C.4, D.1–D.2 and D.8–D.10.

We are grateful for the reviewers' suggestions, which have made the paper's evidence much stronger.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.