- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
ScaffoldBench evaluates three separate abilities: identifying a problem's Bloom objective, reconstructing a synthetic student's knowledge-state vector, and producing targeted guidance without revealing the answer. Experts specify knowledge chains and state/error profiles, Gemini generates traces, and experts verify the resulting 2,000 Physics/Reading examples. Nineteen model configurations are evaluated using objective accuracy, state-vector/breakpoint accuracy, and a leakage-gated scaffolding score. The paper reports different rankings across recognition, diagnosis, and intervention, with human checks of the response rubric and a small MathDial transfer evaluation. Oracle supplies both diagnostic information and stronger tutoring instructions; the paper correctly treats its gain as their combined effect rather than the isolated value of diagnosis. These are fixed-trace response evaluations, not measurements of longitudinal student learning.
Critical strengths
- A useful decomposition with substantial annotation effort. Sections 3.1–3.4 separate objective recognition, structured diagnosis, and scaffolding instead of equating correct answers with effective tutoring. Expert construction/verification and Appendix H's full-dataset Bloom annotation provide concrete operational targets. This makes the benchmark more diagnostic than a single holistic tutoring score.
- Meaningful checks and restrained interpretation. Human validation of the response rubric, Appendix M's trace shuffling, and Appendix Q's MathDial study probe different validity questions. The explicit recognition that Oracle changes both context and instructions, that shuffling is not a causal mechanism test, and that no human learning gain is measured substantially improves the credibility of the reported descriptive findings.
Critical weaknesses
- The diagnostic leaderboard lacks a fully specified, reproducible scoring contract. Section 3.5, lines 176–179, explicitly states that historical handling of profiles without a state-1 breakpoint and malformed predictions is undocumented and that scores were retained without recomputation. These cases affect the definition and denominator of BP-Acc and can affect cross-model comparisons, especially with 100-token diagnostic caps (Section 3.4). An idealized equation with a null breakpoint does not identify what produced Table 3. The paper needs the actual historical rule, relevant case counts, and evidence that the reported rankings are not artifacts of parsing or null-state handling. This is a central soundness concern for a benchmark, not merely a missing implementation convenience.
- The benchmark does not yet separate student-state inference from question/profile priors sufficiently. On Reading, Qwen3-0.6B achieves 81% BP-Acc and 66% VEM in Figure 5, and still reaches 76% and 62% with mismatched traces in Appendix M, Table 17. This does not prove cheating or a particular shortcut, but it means that much of the high measured performance can survive removal of the intended student evidence. Question-only/profile-prior controls are acknowledged as needed but not supplied. Together with the absence of a predicted-state tutoring cascade and the bundled Oracle intervention, this limits the evidence for evaluating practically useful ZPD alignment rather than reconstructing dataset regularities and following stronger tutoring instructions. A narrower descriptive interpretation is supportable; the intended diagnostic construct remains insufficiently validated.
Questions for authors
- Clarification: Can you recover the exact historical handling of no-state-1 profiles, malformed/truncated outputs, and missing predictions, and give their counts for Table 3? Were they counted as errors, omitted, or assigned a default breakpoint?
- Clarification using existing data: What are the breakpoint/state-profile distributions by domain and chain length? Are question-only or profile-prior results already available that contextualize the high shuffled-trace scores?
- Clarification: How many examples per model, task, and domain produced Tables 2–5, and do all configurations share the same evaluated IDs? Please distinguish the 2,000-item corpus from any evaluation subsampling.
- Scope clarification: Which conclusions are intended to concern state diagnosis beyond dataset priors, and which only concern compliance with the supplied Oracle instructions? A new longitudinal learning study is a future-work suggestion, not a rebuttal requirement.