- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
CourseProject-Bench evaluates agents on 30 university course projects across five disciplines within reconstructed instructional workspaces. The benchmark emphasizes locating, understanding, and applying knowledge distributed across course materials. A five-stage Process Correctness framework assesses task understanding, knowledge access, conceptual understanding, knowledge operationalization, and artifact production using deterministic checks and an automated judge. Experiments compare eight primary agent configurations and additional harness variants, revealing performance gaps and providing stage-level analyses of failures in knowledge acquisition and application during project completion.
Critical strengths
- The benchmark addresses a setting where agents must independently locate and apply instructional knowledge within substantial workspaces. Authentic projects and heterogeneous deliverables provide a useful testbed for developing agents that learn from available resources while completing complex tasks.
- The process-oriented evaluation combines execution trajectories and artifacts to examine failures beyond final deliverable completeness. Explicit verification obligations and deterministic checks provide concrete assessment criteria, while stage-level results offer useful directions for investigating agent weaknesses.
Critical weaknesses
- Human review verifies rubric construction, but agreement between automated scores and independent human judgments is not reported. This limits confidence in semantic evaluations, particularly judgments of conceptual understanding across different disciplines.
- Course dependence is assessed through annotation and manual screening, without performance comparisons that isolate the contribution of instructional materials. The benchmark’s distinction between knowledge acquisition and general reasoning or implementation difficulty therefore remains incompletely validated.
Questions for authors
- What evidence supports the distinction between acquiring course-specific knowledge and applying knowledge already available to the model?
- Were automated scoring decisions reviewed by domain-qualified annotators, and what types of disagreement were observed?
- How does L3 scoring distinguish incorrect understanding from explanations that are correct but incomplete?
2 comments
Thank you to the authors for their detailed response. I appreciate their clarifications and have decided to maintain my original score.
Response to Reviewer 71MF
We would like to sincerely thank the reviewer for carefully reading our paper and providing insightful comments to improve this paper further. We address the reviewer's comments and present additional experimental results below.
Were automated scoring decisions reviewed by domain-qualified annotators, and what types of disagreement were observed?
Response. We conducted an independent human audit of the automated judge’s obligation-level verdicts. We randomly sampled two courses from each of the five disciplinary groups and two model submissions per selected course. This yielded 20 submissions covering 10 of 30 courses and all five disciplinary groups. For each disciplinary group, we recruited two graduate-student annotators whose fields of study fell within that group, for a total of ten annotators. Each annotator was blinded to both the automated judge’s results and the other annotator’s results. We report agreement rates and Cohen’s κ between the automated judge and each annotator, as well as between the two annotators, separately for each of the five disciplines. Rubric-level judgments are derived from each annotator’s obligation-level judgments: a rubric passes only if all associated obligations pass.
The following table is the results in the 4 stages of L1, L2, L3, L4 and L5.
The audit shows high judge–human agreement, including for L3, supporting the reliability of the automated verdicts on the audited submissions. Disagreements were most frequent in L4 and L5, involving explanatory sufficiency, the applicability of conditional requirements, and the treatment of contradictory evidence.
Course dependence is assessed through annotation and manual screening, without performance comparisons that isolate the contribution of instructional materials.
$\color{green}{\mathbf{Response}}. We conducted an instructional-material withholding ablation across all 30 projects for the two best-performing models. The Withheld setting removes separable instructional support, including lecture notes, slides, videos, and student exemplars, while preserving task instructions and necessary inputs. We retain the Full-setting judging protocol and report scores on L1 and L3–L5, excluding L2 to avoid directly penalizing agents for not accessing withheld sources.
Withholding instructional materials reduces the aggregate score by 16.71 percentage points for GPT-5.6-sol and 13.83 points for DeepSeek-v4-pro. Relative to Full, withholding instructional materials reduces L3/L4 scores significantly by 43.9%/44.9% for GPT-5.6-sol and 36.6%/41.3% for DeepSeek-v4-pro. These results provide empirical support for instructional-material dependence: even when task instructions and necessary inputs remain available, the models’ existing capabilities do not fully substitute for the instructional support.
How does L3 scoring distinguish incorrect understanding from explanations that are correct but incomplete?
Response. L3 assesses both conceptual correctness and coverage of task-required content. Incorrect explanations contain conceptual errors; correct but incomplete explanations omit required assumptions, conditions, or mechanisms. Both can cause a binary rubric to fail. We will clarify the definition and distinction in Section 4.1.