acceptodds
Under review as a conference paper at ICLR 2027

CourseProject-Bench: Evaluating AI Agents in Authentic Course Environments

Abstract

AI agents are increasingly evaluated on long-horizon tasks involving tools, files, and complex environments, yet existing benchmarks typically rely on agents' pretraining knowledge or provide the necessary domain knowledge directly. Therefore, we introduce CourseProject-Bench, a benchmark of 30 university course projects spanning five disciplinary domains, each paired with a reconstructed hierarchical course workspace. These projects require Active Knowledge Acquisition: agents must access relevant instructional materials from a broader course environment and understand the conceptual or procedural knowledge they contain before applying it to the project. To evaluate these open-ended tasks, we introduce Process Correctness, a five-stage framework covering Task Understanding, Knowledge Access, Conceptual Understanding, Knowledge Operationalization, and Artifact Production. Across eight frontier agent configurations, the strongest achieves only 60.7% Process Correctness. Stage-level analysis reveals that Active Knowledge Acquisition remains challenging, with conceptual understanding emerging as a major bottleneck. We further find that model capability has a substantially larger effect than harness choice, while performance also varies across disciplines and artifact demands. CourseProject-Bench provides a diagnostic testbed for studying agents that must learn from complex environments and apply acquired knowledge to long-horizon tasks.

Primary area:
datasets and benchmarks
Keywords:
agent evaluation, benchmark, course projects, knowledge acquisition, knowledge operationalization, long-horizon tasks, educational AI, source-grounded evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_71MFOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

CourseProject-Bench evaluates agents on 30 university course projects across five disciplines within reconstructed instructional workspaces. The benchmark emphasizes locating, understanding, and applying knowledge distributed across course materials. A five-stage Process Correctness framework assesses task understanding, knowledge access, conceptual understanding, knowledge operationalization, and artifact production using deterministic checks and an automated judge. Experiments compare eight primary agent configurations and additional harness variants, revealing performance gaps and providing stage-level analyses of failures in knowledge acquisition and application during project completion.

Critical strengths
  1. The benchmark addresses a setting where agents must independently locate and apply instructional knowledge within substantial workspaces. Authentic projects and heterogeneous deliverables provide a useful testbed for developing agents that learn from available resources while completing complex tasks. 
  2. The process-oriented evaluation combines execution trajectories and artifacts to examine failures beyond final deliverable completeness. Explicit verification obligations and deterministic checks provide concrete assessment criteria, while stage-level results offer useful directions for investigating agent weaknesses.
Critical weaknesses
  1. Human review verifies rubric construction, but agreement between automated scores and independent human judgments is not reported. This limits confidence in semantic evaluations, particularly judgments of conceptual understanding across different disciplines. 
  2. Course dependence is assessed through annotation and manual screening, without performance comparisons that isolate the contribution of instructional materials. The benchmark’s distinction between knowledge acquisition and general reasoning or implementation difficulty therefore remains incompletely validated.
Questions for authors
  1. What evidence supports the distinction between acquiring course-specific knowledge and applying knowledge already available to the model? 
  2. Were automated scoring decisions reviewed by domain-qualified annotators, and what types of disagreement were observed? 
  3. How does L3 scoring distinguish incorrect understanding from explanations that are correct but incomplete?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 71MF

We would like to sincerely thank the reviewer for carefully reading our paper and providing insightful comments to improve this paper further. We address the reviewer's comments and present additional experimental results below.

Weakness 1 (Q2)

Were automated scoring decisions reviewed by domain-qualified annotators, and what types of disagreement were observed?

Response.\color{green}{\mathbf{Response}}. We conducted an independent human audit of the automated judge’s obligation-level verdicts. We randomly sampled two courses from each of the five disciplinary groups and two model submissions per selected course. This yielded 20 submissions covering 10 of 30 courses and all five disciplinary groups. For each disciplinary group, we recruited two graduate-student annotators whose fields of study fell within that group, for a total of ten annotators. Each annotator was blinded to both the automated judge’s results and the other annotator’s results. We report agreement rates and Cohen’s κ between the automated judge and each annotator, as well as between the two annotators, separately for each of the five disciplines. Rubric-level judgments are derived from each annotator’s obligation-level judgments: a rubric passes only if all associated obligations pass.

Discipline / Level (Sample Num)Automated judge vs Annotator 1Automated judge vs Annotator 2Annotator 1 vs Annotator 2
Computing — Obligation (276)92.03% (0.784)88.04% (0.688)93.84% (0.848)
Computing — Rubric (40)90.00% (0.792)87.50% (0.738)97.50% (0.946)
Engineering — Obligation (338)91.72% (0.806)90.83% (0.781)95.56% (0.900)
Engineering — Rubric (38)89.47% (0.769)86.84% (0.708)97.37% (0.934)
Natural Sciences — Obligation (464)89.44% (0.754)89.87% (0.763)91.38% (0.803)
Natural Sciences — Rubric (72)90.28% (0.794)87.50% (0.736)97.22% (0.940)
Social Sciences — Obligation (304)89.47% (0.784)90.13% (0.799)91.45% (0.820)
Social Sciences — Rubric (40)90.00% (0.655)90.00% (0.688)95.00% (0.828)
Humanities & Arts — Obligation (232)95.26% (0.878)94.83% (0.871)92.67% (0.821)
Humanities & Arts — Rubric (34)91.18% (0.825)88.24% (0.767)85.29% (0.700)

The following table is the results in the 4 stages of L1, L2, L3, L4 and L5.

Stage (obligation level)Automated judge vs Annotator 1Automated judge vs Annotator 2Annotator 1 vs Annotator 2
L195.65% (0.677)94.93% (0.640)96.38% (0.763)
L395.00% (0.897)92.81% (0.852)94.69% (0.892)
L491.46% (0.812)89.75% (0.774)93.63% (0.865)
L587.30% (0.729)88.87% (0.759)89.84% (0.785)

The audit shows high judge–human agreement, including for L3, supporting the reliability of the automated verdicts on the audited submissions. Disagreements were most frequent in L4 and L5, involving explanatory sufficiency, the applicability of conditional requirements, and the treatment of contradictory evidence.

Weakness 2 (Q1)

Course dependence is assessed through annotation and manual screening, without performance comparisons that isolate the contribution of instructional materials.

$\color{green}{\mathbf{Response}}. We conducted an instructional-material withholding ablation across all 30 projects for the two best-performing models. The Withheld setting removes separable instructional support, including lecture notes, slides, videos, and student exemplars, while preserving task instructions and necessary inputs. We retain the Full-setting judging protocol and report scores on L1 and L3–L5, excluding L2 to avoid directly penalizing agents for not accessing withheld sources.

ModelProject NumFullWithheldChange
GPT-5.6-sol3056.7240.01−16.71
DeepSeek-v4-pro3057.5743.74−13.83

Withholding instructional materials reduces the aggregate score by 16.71 percentage points for GPT-5.6-sol and 13.83 points for DeepSeek-v4-pro. Relative to Full, withholding instructional materials reduces L3/L4 scores significantly by 43.9%/44.9% for GPT-5.6-sol and 36.6%/41.3% for DeepSeek-v4-pro. These results provide empirical support for instructional-material dependence: even when task instructions and necessary inputs remain available, the models’ existing capabilities do not fully substitute for the instructional support.

Question 3

How does L3 scoring distinguish incorrect understanding from explanations that are correct but incomplete?

Response.\color{green}{\mathbf{Response}}. L3 assesses both conceptual correctness and coverage of task-required content. Incorrect explanations contain conceptual errors; correct but incomplete explanations omit required assumptions, conditions, or mechanisms. Both can cause a binary rubric to fail. We will clarify the definition and distinction in Section 4.1.

Thank you to the authors for their detailed response. I appreciate their clarifications and have decided to maintain my original score.

Reviewer_z3DpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CourseProject-Bench, a benchmark based on university course projects, and evaluates multiple frontier models on it.

Critical strengths

The paper introduces an interesting dataset of authentic university course projects and presents a substantial experimental study, including evaluations of multiple models, controlled harness comparisons, and discipline-level analyses. These experiments provide useful insights into how agent performance varies across models, execution frameworks, and task domains.

Critical weaknesses
  1. Unclear distinction from existing benchmarks. TheAgentCompany and Workspace-Bench, discussed in the related work, also involve discovering and applying workspace-specific knowledge. It is unclear whether the course setting itself constitutes a new type of task for agents.

  2. Unverified dependence on course materials. The filtering procedure does not establish whether models already possess the required knowledge. An ablation removing instructional materials while preserving task instructions and necessary data is needed to substantiate the claim of active knowledge acquisition.

Questions for authors
  1. For an LLM, completing a course project may involve searching for and applying knowledge, much like other workspace tasks. What fundamentally distinguishes these tasks from those in existing benchmarks such as TheAgentCompany and Workspace-Bench?

  2. Much of the knowledge in public university courses may already be covered by state-of-the-art models’ pretraining. Could you provide an ablation with and without instructional materials, preserving task instructions and necessary data, to establish whether these materials are actually needed for successful completion?

  3. Given that Table 4 assesses L3 from the submitted report, how are these judgments distinguished from L5 soundness checks? For the L1 example in Table 4, would an otherwise correct submission lose points simply because the model did not explicitly state the key constraints before implementation?

  4. Could process-based evaluation penalize reasoning steps, such as planning or checking constraints, that the model performed internally but did not explicitly document? Could you compare process-based and artifact-only scores, and audit whether the additional deductions reflect genuine task failures or merely missing trace evidence?

3 comments
AuthorsOpenReview ↗

Response to Reviewer z3Dp

We would like to sincerely thank the reviewer for carefully reading our paper and providing insightful comments to improve this paper further. We address the reviewer's comments and present additional experimental results below.

Weakness 1 (Q1)

Unclear distinction from existing benchmarks.

Response.\color{green}{\mathbf{Response}}. While information search is common in workspace tasks, our distinctive focus is Active Knowledge Acquisition: independently locating and understanding conceptual domain knowledge\textbf{domain knowledge} within a broader instructional workspace, before project completion. We make this requirement central to benchmark construction\textbf{benchmark construction} and empirical validation.\textbf{empirical validation.} During construction, we exclude projects judged solvable from task specifications or general background knowledge alone. We then test this intended dependence by withholding instructional materials while preserving task instructions and necessary inputs. Across 30 projects, withholding these materials reduces scores on L1 and L3–L5 by 16.71 points for GPT-5.6-sol and 13.83 points for DeepSeek-v4-pro (more details in the next response). These results provide empirical support for instructional-material dependence, showing that the models’ existing capabilities do not fully substitute for the instructional support.

Weakness 2 (Q2)

Unverified dependence on course materials.

Response.\color{green}{\mathbf{Response}}. We conducted an instructional-material withholding ablation across all 30 projects for the two best-performing models. The Withheld setting removes separable instructional support, including lecture notes, slides, videos, and student exemplars, while preserving task instructions and necessary inputs. We retain the Full-setting judging protocol and report scores on L1 and L3–L5, excluding L2 to avoid directly penalizing agents for not accessing withheld sources.

ModelProject NumFullWithheldChange
GPT-5.6-sol3056.7240.01−16.71
DeepSeek-v4-pro3057.5743.74−13.83

Withholding instructional materials reduces the aggregate score by 16.71 percentage points for GPT-5.6-sol and 13.83 points for DeepSeek-v4-pro. Relative to Full, withholding instructional materials reduces L3/L4 scores significantly by 43.9%/44.9% for GPT-5.6-sol and 36.6%/41.3% for DeepSeek-v4-pro. These results provide empirical support for instructional-material dependence: even when task instructions and necessary inputs remain available, the models’ existing capabilities do not fully substitute for the instructional support.

Question 3

How are these judgments distinguished from L5 soundness checks? Would an otherwise correct submission lose points simply because the model did not explicitly state the key constraints before implementation?

Response.\color{green}{\mathbf{Response}}. L3 and L5 soundness have different evaluation targets, even when both use the submitted report as evidence. L3 assesses whether domain concepts, mechanisms, and methodological assumptions are correctly explained. L5 soundness assesses whether the submitted solution is substantively valid: whether calculations and derivations are correct, results are reproducible, and conclusions are supported by evidence.

An otherwise correct submission can lose L1 points if its observable pre-implementation planning omits or misstates a required constraint. This is intentional: L1 assesses demonstrated task understanding and planning completeness separately from final-artifact correctness.

Question 4

Could process-based evaluation penalize reasoning steps, that the model performed internally but did not explicitly document? Could you compare process-based and artifact-only scores, and audit whether the additional deductions reflect genuine task failures or merely missing trace evidence?

L1 and L2 only assess observable task understanding/planning and source access, respectively, separately from final-artifact correctness. For example, L1 is based on concrete planning records generated from tested agents.

We compare L1–L5 scores with artifact-only L3–L5 scores, whose criteria rely exclusively on final submissions. As shown in the table, removing L1/L2 preserves all eight models’ rankings and lowers every model’s score.

ModelProcess-based score (L1–L5)Artifact-only score (L3–L5)Change
DeepSeek-v4-pro60.7156.62−4.09
GPT-5.6-sol59.9055.36−4.55
GLM-5.355.8951.54−4.34
Kimi-K2.7-Code48.3843.94−4.44
Qwen3.7-Max45.6439.50−6.14
Claude Opus 540.0433.65−6.38
Claude Sonnet 529.7323.15−6.59
Grok-4.622.0616.16−5.90

Deductions from L1 generally concern specific errors or required-content omissions within existing records. However, an omission does not establish that the corresponding reasoning was absent internally. We therefore interpret L1/L2 as observable process measures.

Thank you for the detailed rebuttal and the additional experiments. I appreciate the authors' efforts to address my concerns. However, I remain unconvinced that Active Knowledge Acquisition constitutes a sufficiently distinct capability from those already evaluated in existing agent benchmarks. I also retain some concerns about the validity of the process-based evaluation, particularly whether missing observable evidence necessarily reflects a failure in the underlying capability.

Overall, I consider this paper to be borderline between a 2 and a 3. Although the additional experiments help address some of my concerns, I ultimately decided to maintain my original score of 2.

AuthorsOpenReview ↗

Response to Reviewer z3Dp

Thank you for clarifying your remaining concerns. We would like to offer a few further clarifications in response to your comments.

We will sharpen our contribution as a benchmark specifically constructed around independently locating and understanding instructional domain knowledge, with material dependence empirically tested through withholding ablations. Specifically, Workspace-Bench defines Workspace Learning as “the ability to connect tasks with relevant data and understand lineage and logical relationships across numerous workspace files.” TheAgentCompany introduces a benchmark that “estimates the ability of AI agents to perform tasks encountered in everyday workplaces.” These stated objectives emphasize cross-file relationships and professional task execution. Our distinctive focus is making dependence on instructional concepts and methods an explicit project-selection criterion, supported by requirement-to-material tracing and empirical withholding tests.

As for the second concern, almost all agent–project runs generated concrete planning records for L1 evaluation, and deductions generally concern specific errors or required-content omissions within these records. L1 is a stage that is more easily overlooked than final outcomes. However, such understanding and planning provides insight into an agent’s task-related reasoning, making its behavior more interpretable. Planning quality may also influence subsequent decisions and artifact quality. To examine L1’s diagnostic relevance, we analyzed its association with artifact-only performance across the 240 runs.

Mean L3–L5 score: L1-passed groupMean L3–L5 score: L1-failed groupDifference (pp)Correlation coefficient (r)
49.5328.91+20.610.409

L1 pass status was moderately positively correlated with higher L3–L5 scores (r = 0.409), although these results do not establish causality. These findings support the value of evaluating and analyzing L1 as a distinct stage of agent performance.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to Reviewer ZrSq (Weakness 1 and Q1)

We thank the reviewer for recognizing the promise of our benchmark setting and for providing valuable and insightful feedback. We address the reviewer's comments and present additional experimental results below.

Weakness 1 (Q1) Part One

Have the judge’s actual rubric verdicts and stage-level diagnoses been independently checked by human graders with relevant subject knowledge?

Response.\color{green}{\mathbf{Response}}. We conducted an independent human audit of the automated judge’s obligation-level verdicts. We randomly sampled two courses from each of the five disciplinary groups and two model submissions per selected course. This yielded 20 submissions covering 10 of 30 courses and all five disciplinary groups. For each disciplinary group, we recruited two graduate-student annotators whose fields of study fell within that group, for a total of ten annotators. Each annotator was blinded to both the automated judge’s results and the other annotator’s results. We report agreement rates and Cohen’s κ between the automated judge and each annotator, as well as between the two annotators, separately for each of the five disciplines. Rubric-level judgments are derived from each annotator’s obligation-level judgments: a rubric passes only if all associated obligations pass.

Discipline / Level (Sample Num)Automated judge vs Annotator 1Automated judge vs Annotator 2Annotator 1 vs Annotator 2
Computing — Obligation (276)92.03% (0.784)88.04% (0.688)93.84% (0.848)
Computing — Rubric (40)90.00% (0.792)87.50% (0.738)97.50% (0.946)
Engineering — Obligation (338)91.72% (0.806)90.83% (0.781)95.56% (0.900)
Engineering — Rubric (38)89.47% (0.769)86.84% (0.708)97.37% (0.934)
Natural Sciences — Obligation (464)89.44% (0.754)89.87% (0.763)91.38% (0.803)
Natural Sciences — Rubric (72)90.28% (0.794)87.50% (0.736)97.22% (0.940)
Social Sciences — Obligation (304)89.47% (0.784)90.13% (0.799)91.45% (0.820)
Social Sciences — Rubric (40)90.00% (0.655)90.00% (0.688)95.00% (0.828)
Humanities & Arts — Obligation (232)95.26% (0.878)94.83% (0.871)92.67% (0.821)
Humanities & Arts — Rubric (34)91.18% (0.825)88.24% (0.767)85.29% (0.700)

The following table is the results in the 4 stages of L1, L2, L3, L4 and L5.

Stage (obligation level)Automated judge vs Annotator 1Automated judge vs Annotator 2Annotator 1 vs Annotator 2
L195.65% (0.677)94.93% (0.640)96.38% (0.763)
L395.00% (0.897)92.81% (0.852)94.69% (0.892)
L491.46% (0.812)89.75% (0.774)93.63% (0.865)
L587.30% (0.729)88.87% (0.759)89.84% (0.785)

The audit shows high judge–human agreement, including for L3, supporting the reliability of the automated verdicts on the audited submissions. Disagreements were most frequent in L4 and L5, involving explanatory sufficiency, the applicability of conditional requirements, and the treatment of contradictory evidence.

Weakness 1 (Q1) Part Two

Does the conceptual-understanding bottleneck remain when reporting obligation-level results or otherwise varying how obligations are grouped into rubrics?

Response.\color{green}{\mathbf{Response}}. We thank the reviewer for suggesting this sensitivity analysis. As shown in the table, obligation-level scoring reduces the L2–L3 gap from 45.33 to 15.46 percentage points for DeepSeek-v4-pro and from 55.22 to 19.60 points for GPT-5.6-sol. Still, L3 conceptual understanding remains the larger bottleneck within the defined Activate Knowledge Acquisition (L2 plus L3) for both configurations.

ModelAggregation methodL2 (%)L3 (%)L2–L3 gap (pp)
DeepSeek-v4-proRubric-level AND100.0054.6745.33
DeepSeek-v4-proObligation-level100.0084.5415.46
GPT-5.6-solRubric-level AND100.0044.7855.22
GPT-5.6-solObligation-level100.0080.4019.60

We also clarify the rationale for retaining AND aggregation in the formal score. Obligations are generated after a rubric criterion is defined to help the automated judge verify that established requirement. They are verification conditions, rather than separately weighted grading items whose successes can compensate for failures elsewhere. Their purpose is to prevent partial satisfaction or superficially plausible evidence from being mistaken for complete satisfaction. Obligations under the same rubric may also include complementary positive-verification and counterevidence checks for the same requirement. Consequently, satisfying only a subset does not establish that the parent criterion has been fully met. We therefore retain AND aggregation for the formal criterion-level score and will report obligation-level scores as a complementary sensitivity analysis.

AuthorsOpenReview ↗

Response to Reviewer ZrSq (Weakness 2 and Q2)

We thank the reviewer for recognizing the promise of our benchmark setting and for providing valuable and insightful feedback. We address the reviewer's comments and present additional experimental results below.

Weakness 2 (Q2)

How do the strongest agents perform on the source-dependent project requirements when instructional materials are withheld, while retaining the specification and required input data? Does directly providing the relevant sources change the failure pattern?

Response.\color{green}{\mathbf{Response}}. Thank you for appreciating the filtering procedure in Section 3.1, including the exclusion of “concept pointers.” We agree that additional withheld experiment should be conducted to support the instructional content dependence. We conducted an instructional-material withholding ablation across all 30 projects for the two best-performing models. The Withheld setting removes separable instructional support, including lecture notes, slides, videos, and student exemplars, while preserving task instructions and necessary inputs. We retain the Full-setting judging protocol and report scores on L1 and L3–L5, excluding L2 to avoid directly penalizing agents for not accessing withheld sources.

ModelProject NumFullWithheldChange
GPT-5.6-sol3056.7240.01−16.71
DeepSeek-v4-pro3057.5743.74−13.83

Withholding instructional materials reduces the aggregate score by 16.71 percentage points for GPT-5.6-sol and 13.83 points for DeepSeek-v4-pro. Relative to Full, withholding instructional materials reduces L3/L4 scores significantly by 43.9%/44.9% for GPT-5.6-sol and 36.6%/41.3% for DeepSeek-v4-pro. These results provide empirical support for instructional-material dependence: even when task instructions and necessary inputs remain available, the models’ existing capabilities do not fully substitute for the instructional support.

We evaluated GPT-5.6-sol on 30 paired projects under an oracle-source condition. Starting from the Withheld inputs, we added preselected, task-relevant instructional materials and organized the files in a flat directory, removing the original course hierarchy. We report scores over L1 and L3–L5, excluding L2.

MetricFull (%)Oracle (%)Oracle − Full (pp)
Overall (L1, L3–L5)56.7256.06−0.66
L173.3373.330.00
L344.7844.780.00
L450.4449.72−0.72
L563.2862.36−0.92

Oracle improves over Withheld (40.01) by 16.05 percentage points and achieves a score close to Full. Together with the 100% L2 score under Full, these results suggest that locating relevant sources was not the main performance bottleneck for this configuration in the evaluated runs.

AuthorsOpenReview ↗

Response to Reviewer ZrSq

We sincerely thank the reviewer for their time and thoughtful feedback. We have made every effort to address all questions and concerns, and would be happy to provide any further clarification or additional information.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.