acceptodds
Under review as a conference paper at ICLR 2027

ScaffoldBench: A Dynamic Benchmark for Evaluating Pedagogical Scaffolding and ZPD Alignment in Large Language Models

Abstract

A language model can solve a problem while failing to help a student solve it. This mismatch exposes two distinct requirements for tutoring: inferring the learner's cognitive state and generating assistance that preserves the learner's remaining reasoning. We introduce ScaffoldBench, a benchmark that separates cognitive objective recognition, structured state diagnosis, and constrained instructional generation. It comprises 2,000 expert-verified synthetic student traces built from physics and reading problems, with explicit knowledge chains, cognitive state vectors, and error categories. We formalize the relationship between local and global diagnostic accuracy, derive the leakage-imposed ceiling on scaffolding fidelity, and distinguish observed Oracle gains from the isolated value of diagnostic information. Evaluation of 19 models reveals a diagnosis–intervention mismatch: high diagnostic accuracy can persist under mismatched student traces, while reasoning-oriented models exhibit substantial answer leakage even with explicit state information. On physics tasks, their mean leakage rate is 91.0% under baseline prompting and 78.9% under the augmented Oracle protocol. Educational models respond much more strongly to this protocol, reaching 1.0% leakage and 4.89/5 scaffolding fidelity on reading tasks. These results establish cognitive diagnosis, trace dependence, and response restraint as separate axes of tutor evaluation, motivating models that infer learner states and control what they reveal.

Primary area:
datasets and benchmarks
Keywords:
large language models, benchmarks, pedagogical alignment, cognitive diagnosis, constrained generation, intelligent tutoring, answer leakage

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MMxROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ScaffoldBench evaluates three separate abilities: identifying a problem's Bloom objective, reconstructing a synthetic student's knowledge-state vector, and producing targeted guidance without revealing the answer. Experts specify knowledge chains and state/error profiles, Gemini generates traces, and experts verify the resulting 2,000 Physics/Reading examples. Nineteen model configurations are evaluated using objective accuracy, state-vector/breakpoint accuracy, and a leakage-gated scaffolding score. The paper reports different rankings across recognition, diagnosis, and intervention, with human checks of the response rubric and a small MathDial transfer evaluation. Oracle supplies both diagnostic information and stronger tutoring instructions; the paper correctly treats its gain as their combined effect rather than the isolated value of diagnosis. These are fixed-trace response evaluations, not measurements of longitudinal student learning.

Critical strengths
  1. A useful decomposition with substantial annotation effort. Sections 3.1–3.4 separate objective recognition, structured diagnosis, and scaffolding instead of equating correct answers with effective tutoring. Expert construction/verification and Appendix H's full-dataset Bloom annotation provide concrete operational targets. This makes the benchmark more diagnostic than a single holistic tutoring score.
  2. Meaningful checks and restrained interpretation. Human validation of the response rubric, Appendix M's trace shuffling, and Appendix Q's MathDial study probe different validity questions. The explicit recognition that Oracle changes both context and instructions, that shuffling is not a causal mechanism test, and that no human learning gain is measured substantially improves the credibility of the reported descriptive findings.
Critical weaknesses
  1. The diagnostic leaderboard lacks a fully specified, reproducible scoring contract. Section 3.5, lines 176–179, explicitly states that historical handling of profiles without a state-1 breakpoint and malformed predictions is undocumented and that scores were retained without recomputation. These cases affect the definition and denominator of BP-Acc and can affect cross-model comparisons, especially with 100-token diagnostic caps (Section 3.4). An idealized equation with a null breakpoint does not identify what produced Table 3. The paper needs the actual historical rule, relevant case counts, and evidence that the reported rankings are not artifacts of parsing or null-state handling. This is a central soundness concern for a benchmark, not merely a missing implementation convenience.
  2. The benchmark does not yet separate student-state inference from question/profile priors sufficiently. On Reading, Qwen3-0.6B achieves 81% BP-Acc and 66% VEM in Figure 5, and still reaches 76% and 62% with mismatched traces in Appendix M, Table 17. This does not prove cheating or a particular shortcut, but it means that much of the high measured performance can survive removal of the intended student evidence. Question-only/profile-prior controls are acknowledged as needed but not supplied. Together with the absence of a predicted-state tutoring cascade and the bundled Oracle intervention, this limits the evidence for evaluating practically useful ZPD alignment rather than reconstructing dataset regularities and following stronger tutoring instructions. A narrower descriptive interpretation is supportable; the intended diagnostic construct remains insufficiently validated.
Questions for authors
  1. Clarification: Can you recover the exact historical handling of no-state-1 profiles, malformed/truncated outputs, and missing predictions, and give their counts for Table 3? Were they counted as errors, omitted, or assigned a default breakpoint?
  2. Clarification using existing data: What are the breakpoint/state-profile distributions by domain and chain length? Are question-only or profile-prior results already available that contextualize the high shuffled-trace scores?
  3. Clarification: How many examples per model, task, and domain produced Tables 2–5, and do all configurations share the same evaluated IDs? Please distinguish the 2,000-item corpus from any evaluation subsampling.
  4. Scope clarification: Which conclusions are intended to concern state diagnosis beyond dataset priors, and which only concern compliance with the supplied Oracle instructions? A new longitudinal learning study is a future-work suggestion, not a rebuttal requirement.
Reviewer_9VkWOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces an education-focused benchmark for LLMs called ScaffoldBench. The idea of ScaffoldBench is to not only evaluate scaffolding but to also have the model predict information related to the concepts of Bloom's taxonomy and the ZPD. The authors manually and automatically create the benchmark based on background knowledge from MMLU and RACE and evaluate several LLMs on their data.

Critical strengths
  • I think that benchmarks are still one of the main bottlenecks of AI in education, so any progress can have significant impact. Bloom's taxonomy and ZPD are well-known concepts but benchmarks targeting them are still expected to be helpful.
Critical weaknesses
  • The paper lacks clarity overall which is both due to the writing and lack of details in the main paper. For example, it is still unclear after reading the entire paper what about the benchmark is "dynamic". It is also unclear how the paper relates to prior works, as the discussion of related works is very short and also the extended version in the appendix is only a listing of prior works and does not really describe how they are related or what gap exactly is filled wrt. them. Another example is Blooms taxonomy, which is a core concept in the work but not really described, it is just said that it produces a 6 point scale. Various related works are also missed, for example, it is claimed that "[t]he results motivate separating state estimation from response control" but similar ideas have already been pursued e.g. by Wang et al. 2024 and Daheim et al. 2024.

  • There are various incorrect and ill-formatted references that raise significant concerns. Below is a list based on a quick manual research of suspicious references:

Submitted Manuscript - Potential citation according to my quick research

  1. "et al. Marquez. Simulating human users with large language models. arXiv, 2025" - Marquez-Carpintero et al. "Simulating Students with Large Language Models: A Review of Architecture, Mechanisms, and Role Modelling in Education with Generative AI" arXiv 2025
  2. "et al. Wang. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv, 2024" - Wang et al. "RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models", ACL Findings 2024
  3. "et al. Huber. Large language models and bloom’s taxonomy: Evaluating cognitive levels. arXiv, 2025." - Huber & Nikolaus "LLMs meet Bloom’s Taxonomy: A Cognitive View on Large Language Model Evaluations", COLING 2025
  4. "et al. Jiang. Eduguardbench: Benchmarking safety of large language models in education. In Conference Proceedings, 2026" - Jiang et al. "EduGuardBench: A Holistic Benchmark for Evaluating the Pedagogical Fidelity and Adversarial Safety of LLMs as Simulated Teachers", AAAI 2026
  5. "Yu Dan et al. Educhat: A large-scale language model-based chatbot system for education. In ACL, 2023." - I could not find any ACL publication under this name on the ACLAnthology but could find the following arXiv article: Dan et al., "EduChat: A Large-Scale Language Model-based Chatbot System for Intelligent Education", arXiv 2023
  6. "Maarten Sap et al. Neural theory of mind: Modeling human social reasoning with neural networks. In ACL, 2022" - Sap et al. "Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs", EMNLP 2022
Questions for authors
  • Fig. 1 is in a sense incomplete, and so is the introduction which says that " a targeted question can address the misconception without completing the task" - the idea is that the student will be able to solve the problem at some point after the scaffolding question, so in that sense the task is eventually completed. Could you please clarify this?

  • Is decomposing 2k knowledge chains (Sec. 3.2) not a very cognitively exhausting task? How did you ensure that annotators still had good enough quality?

  • How do you explain that the 14B Qwen model is better than the 72B Qwen model in Tab4? Overall, the results seem a bit over the place, would you agree?

Suggestions:

  • It does not seem necessary to spell out, for example, the definition of accuracy and many of these seemingly unnecessary details could be replaced by a more thorough discussion of related works and concepts, as noted above.

References

Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakes (Wang et al., NAACL 2024)

Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors (Daheim et al., EMNLP 2024)

Reviewer_iGuyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ScaffoldBench aims to measure whether LLMs can tutor rather than merely solve problems. Starting from 1,000 MMLU (physics) and 1,000 RACE (reading) items, two experts decompose each problem into a 3–5-step knowledge chain. They then assign a ZPD state vector over {0, 1, 2} (unavailable, support-requiring breakpoint, mastered) from four predefined profile families (nine patterns), and an error code from a seven-class taxonomy. Gemini 3 generates a synthetic student trace for each item, consisting of an erroneous solution, a verbalized thinking process and a student query at one of three clarity levels. Experts verify the trace against the assigned labels and regenerate it up to three times on failure. The benchmark defines three tasks. RQ1 classifies the problem's Bloom level (two-shot; accuracy and distance-weighted accuracy). RQ2 predicts the ZPD vector from the problem, chain and trace, either without (Blind) or with (Goal-Aware) the Bloom label (breakpoint accuracy and vector exact match). RQ3 generates a tutoring response under a generic Baseline prompt or an Oracle prompt that adds the gold chain, state, Bloom level and adaptive-teaching instructions. RQ3 is scored with a GLM-5 judge as a leakage-gated quality score (SFS). Across 19 models grouped as reasoning-oriented, general and educational, the authors find that rankings differ across the three tasks. Reasoning-oriented models leak answers most often at baseline, supplying the Bloom objective has model-dependent effects on diagnosis, and educational models score best under Oracle on Reading. A trace-shuffling check and a 100-turn MathDial study are included as validity probes.

Critical strengths

1. The motivating distinction between solving a problem and preserving the student's reasoning opportunity is important. Separating objective recognition, state diagnosis and response control into distinct tasks is a reasonable decomposition.

2. Construction relies heavily on human work. Knowledge chains, state vectors and Bloom labels are assigned by experts for all 2,000 items, with reported agreement statistics (Cohen's κ = 0.82 for trace verification, Fleiss' κ = 0.85 for Bloom labels).

3. The trace-shuffling sanity check is a valuable addition. It reveals that small models keep high diagnostic scores with mismatched traces, which is exactly the kind of shortcut a benchmark paper should probe.

4. The paper is careful about the limits of its claims. It repeatedly states that Oracle mixes information with instructions, that group comparisons are not causal, and that response scores are not learning outcomes.

Critical weaknesses

1. The RQ2 diagnosis task appears solvable from priors, which undermines its validity. The gold vectors come from only nine fixed patterns. Qwen3-4B tops Blind diagnosis (74.5% BP, 71.0% VEM) despite 11.5% Bloom accuracy, and still reaches 0.66/0.59 on Reading with shuffled traces. The paper reports no trivial baselines (always predict the most frequent pattern for a given chain length, or the most frequent breakpoint position). Without them, the RQ2 leaderboard cannot be interpreted. If a constant predictor approaches the top scores, the task needs to be redesigned, for example with balanced patterns per chain length.

2. The simulated traces contain explicit state cues, which reduces diagnosis to spotting those cues. The generation prompt requires the monologue to express confidence ("I know…") on state-2 steps and uncertainty ("I'm not sure…") on the state-1 step, and verification rejects traces that do not. RQ2 inputs include this monologue. Real students rarely verbalize their state this cleanly. Please report RQ2 without T_thought (solution and query only) to show how much of the task depends on these cues.

3. The headline finding that "reasoning models leak more" is likely confounded by decoding settings. RQ1 and RQ2 cap outputs at 50 and 100 tokens, and RQ3 at 500 tokens. For DeepSeek-R1-distill and QwQ models, which emit long <think> blocks, these caps will often cut off the reasoning before any final answer. In RQ3 the judge may also see the chain of thought, which typically contains the worked solution. The paper must state whether thinking content was stripped before judging, whether truncated outputs were scored, and how the hybrid Qwen3 models and Grok-4-fast were configured (thinking on or off). Classifying Grok-4-fast and the Qwen3 family as "non-reasoning" is questionable.

4. The design of RQ3 cannot separate capability from instruction. The Baseline prompt asks the model to "respond naturally" and does not mention withholding the answer, and Level-3 queries explicitly ask for it. Answering is then arguably correct behavior for a helpful assistant, so baseline ALR largely measures default compliance. The MathDial result (0% leakage for both models once a withholding instruction is added) supports this. A 2×2 design is needed: {pedagogical instruction: yes/no} × {gold state: yes/no}. It is cheap, and without it the main RQ3 conclusions remain uninterpretable, as the paper itself partly acknowledges.

5. The paper admits that parts of its evaluation cannot be reproduced. It states that the "historical scorer's handling of [profiles without a breakpoint] and malformed predictions is not documented; the reported scores are retained without recomputation", that the number of discarded seeds "is not specified", that error-category counts sum to 1,600 rather than 2,000 without explanation, and that the reported S_q values are inconsistent with SFS and ALR (e.g., 1.16 vs 1.03 for DS-Llama-8B). For a benchmark paper, whose main contribution is a reliable measurement instrument, this is a serious problem. All scores should be recomputed with a documented scorer before publication.

6. The design and validation of the SFS metric are partly circular. The rubric already assigns quality 1 to leaking responses, and SFS multiplies by (1 − L) again, so leakage is penalized twice. The reported correlation with human ratings (r = 0.93) uses a human rubric that also scores leakage as 1. Because leakage is binary and very common (often >70%), much of the correlation comes from shared leakage handling. Please report the judge–human agreement on S_q restricted to non-leaking responses, which is where the pedagogical-quality distinctions actually lie.

7. The RQ1 Bloom recognition task is weakly specified. The confusion matrices show no Evaluate or Create items in MMLU and no Create items in RACE, so the task is effectively an imbalanced 4–5-class problem. Accuracy without a majority baseline or macro-F1 is not informative, and the two-shot prompt shows only Remember and Apply examples, which biases predictions. The link between RQ1 and the tutoring tasks is also not established: Bloom labels help or hurt diagnosis inconsistently (Table 4), and no analysis shows that correct objective recognition predicts better scaffolding.

8. One evaluated model cannot be reproduced. "Edu-Reasoner" is an anonymized, proprietary model of about 397B parameters in limited beta, and it is the best-performing scaffolding model. If it is affiliated with the authors, this creates a conflict and verification problem. At minimum, results with and without it should be stated so that the conclusions about the "educational" group do not rest on an unverifiable system (the group has only two members).

Questions for authors

1. Why is the benchmark called "dynamic"? The traces are fixed and each item is evaluated in a single turn.

2. Does the GLM-5 judge receive the gold ZPD state and error code when scoring responses? If it does only in the Oracle condition, judge information differs between conditions.

3. How were the nine patterns distributed over chain lengths 3, 4 and 5? For a given length, what is the accuracy of the majority pattern on BP and VEM?

4. In Fig. 7, the "Oracle" response names "pet-prepared food", which looks like naming the correct option. Under the strict rubric (Table 15), was this response scored as leakage?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.