acceptodds
Under review as a conference paper at ICLR 2027

BanduraBench: The Mechanism Mirage in Theory-Grounded Educational LLM Reasoning

Abstract

Large language models are often evaluated by whether they produce the correct final answer, but a correct decision does not necessarily imply that the model identified the right evidence or recovered the mechanism supporting it. We introduce BanduraBench, a 4,530-item benchmark grounded in Bandura's Social Cognitive Theory of Self-Efficacy for evaluating the gap between decision accuracy and mechanism recovery in educational reasoning. BanduraBench combines foundational construct evaluation with contextualized instructor- and researcher-facing reasoning tasks constructed from an expert-reviewed ontology, human-verified seeds, gold-first blueprints, and human-adjudicated admissible pathways. Models are evaluated on explicit, externally specified mechanism components—Bandura source, task-specific online-learning self-efficacy, course-design interpretation, and causal-claim strength. Across 15 LLMs, DesignReasoning answer accuracy is nearly saturated at 96.6% on average (95.3-98.2%), while normalized four-component recovery averages only 65.5% and ranges from 45.1-84.7%. Strict four-component recovery averages 21.4%, showing that much of the strict-score reduction follows from conjunctive accumulation of component errors. BanduraBench therefore provides a conjunction-aware framework for evaluating whether similar educational decisions are supported by similarly coherent theory-grounded representations.

Primary area:
datasets and benchmarks
Keywords:
Educational AI, Self-Efficacy Reasoning, Reasoning Evaluation, Benchmarking, Mechanism Recovery, Theory-Grounded Evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RfewOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BanduraBench, a 4,530-item benchmark designed to evaluate theory-grounded educational reasoning in large language models. Based on Bandura's Social Cognitive Theory of Self-Efficacy, the benchmark distinguishes between final decision accuracy and the recovery of theoretically specified mechanism components. The authors evaluate 15 LLMs and report near-saturated decision accuracy (96.6% on average), while component-normalized mechanism recovery averages 65.5% and strict four-component recovery averages 21.4%.

The paper addresses an interesting distinction between producing appropriate educational decisions and providing theoretically grounded explanations. However, I have substantial concerns about the construct validity, diagnostic value, and educational significance of the proposed benchmark.

Critical strengths
  1. The paper addresses a potentially important problem: correct educational decisions do not necessarily imply theoretically well-grounded explanations.

  2. The benchmark construction is systematic, incorporating expert-reviewed theoretical structures, gold-first generation, human adjudication, and multiple admissible reasoning pathways.

  3. The evaluation covers 15 LLMs and includes component-level analyses and permutation-based controls to distinguish accumulated component errors from additional joint-recovery effects.

Critical weaknesses
  1. Insufficient construct validity of the proposed benchmark

My primary concern is whether the proposed benchmark genuinely measures educational reasoning competence or primarily evaluates agreement with a predefined, theory-specific ontology.

The benchmark assesses whether models reproduce particular theoretical components derived from Bandura's theory. However, an educationally appropriate decision may be supported by alternative theoretical interpretations or reasoning frameworks.

Although the authors account for multiple admissible pathways within their ontology, this does not establish that agreement with the predefined theoretical representation is a valid measure of broader educational reasoning competence.

In particular, the paper does not establish whether models achieving higher mechanism-recovery scores actually provide more appropriate, useful, or effective educational recommendations.

  1. Near-saturation of decision accuracy and limited evidence of a distinctive reasoning failure

Table 2 reports final decision accuracy between 95.3% and 98.2% across all 15 models. This suggests that the decision-level tasks provide limited discrimination among contemporary LLMs.

The authors address this limitation by emphasizing structured mechanism recovery. However, the interpretation of strict four-component recovery also requires caution.

The observed average 4-Exact accuracy is 21.4%, while the stratified permutation null expectation is 19.5%. As the authors appropriately acknowledge, most of the reduction in exact recovery can be explained by the accumulation of component-level errors.

Consequently, it remains unclear whether the benchmark reveals a distinctive failure of integrated theoretical reasoning beyond imperfect classification of individual predefined components.

The paper would benefit from a clearer demonstration of what new reasoning capability or failure mode the benchmark identifies beyond existing component-level evaluations.

  1. Unclear educational significance of mechanism-recovery performance

The practical educational implications of the reported results are insufficiently established.

The paper reports a mean Pedagogical Action Score of 95.1%, despite considerably lower mechanism-recovery scores. This raises an important question: to what extent does recovering the predefined theoretical mechanism matter when models already produce educationally appropriate actions?

I agree that theoretically grounded explanations can be valuable for transparency and accountability. However, the paper does not empirically demonstrate that higher mechanism-recovery scores are associated with higher-quality educational decisions, better instructional recommendations, or improved learner outcomes.

Even an analysis relating mechanism-recovery scores to independent expert assessments of pedagogical response quality would substantially strengthen the educational contribution.

  1. Limited evidence of generalizability beyond the constructed theoretical scenarios

The benchmark is developed using a relatively limited collection of human-verified seeds and predefined theoretical pathways, which are expanded into a much larger set of LLM-generated scenarios.

Although the construction process includes extensive validation, the number of generated items does not necessarily establish diversity in the underlying reasoning problems.

It remains unclear how well the reported results would generalize to independently developed educational cases, different theoretical frameworks, or authentic instructional situations.

Furthermore, the absence of an independent human-solver baseline makes it difficult to interpret the absolute difficulty of the mechanism-recovery tasks. Expert agreement on benchmark labels is not equivalent to demonstrating that independent experts can recover the intended mechanisms from the model-facing scenarios.

Questions for authors
  1. Construct validity: What evidence supports interpreting recovery of the predefined Bandura-based components as educational reasoning competence rather than theory-specific classification accuracy? How should educationally appropriate decisions supported by alternative theoretical interpretations be evaluated?

  2. Educational relevance: Is there an empirical relationship between component-level mechanism recovery and independently assessed pedagogical response quality? Could the authors examine this relationship using the existing pedagogical-action evaluations, ideally while controlling for final-answer correctness?

  3. Benchmark difficulty: Given the near-ceiling decision accuracy and the small observed-minus-null difference in strict joint recovery, what distinctive reasoning failure does BanduraBench reveal that cannot be explained by component-level prediction errors?

  4. Generalizability and human performance: How sensitive are the conclusions to evaluation on unseen seed families or theoretical pathways? How would independent educational experts perform on the same tasks without access to the reference mechanisms?

5 comments
AuthorsOpenReview ↗

Response to Q1: Construct Validity and Alternative Theoretical Interpretations

We thank the reviewer for raising this construct-validity concern. We clarify that BanduraBench targets a specific, educationally consequential capability on interpreting learner evidence through Bandura's self-efficacy framework and assessing consistency between an instructional decision and its stated theoretical mechanism.

1. Theoretical Motivation and Construct

Self-efficacy is learners' beliefs about their capability to perform particular tasks and is associated with engagement, self-regulation, persistence, and academic performance (Bandura, 1977; Usher & Pajares, 2008; Richardson et al., 2012). Bandura's four sources of efficacy information offer a psychologically grounded vocabulary for interpreting such evidence. In educational scenarios, identifying the relevant source is only one step: the model must also relate learner evidence to task-specific self-efficacy components and instructional conditions. We therefore view this as a specific dimension of educational reasoning.

2. Beyond Label Matching: Complementary Validity Evidence

BanduraBench operationalizes these relationships in contextualized DesignReasoning tasks and scores four complementary mechanism components. Reference targets were fixed through gold-first construction, expert-reviewed pathways, human-verified seeds, and adjudication. The 2,606-item test set includes 648 items admitting multiple complete, defensible pathways within the reviewed framework. Component recovery thus tests evidence-sensitive theoretical attribution rather than only final-action selection.

Our additional analyses demonstrate that the decision–mechanism discrepancy persists under catalog-free and free-form elicitation, while mechanism recovery is positively associated with rated pedagogical quality beyond final-decision correctness (Q2 Response). Furthermore, independent educational experts substantially outperform LLMs in recovering the specified mechanisms on identical items (Q4 Response). Together, we believe that these findings provide complementary evidence of the construct's interpretability, robustness, and educational relevance.

3. Evaluating Alternative Theoretical Mechanisms

Importantly, we view BanduraBench as a theory-anchored, extensible evaluation framework. Its gold-first pathway architecture offers a practical basis for evaluating alternative mechanisms at two levels.

First, within Bandura's framework: When a response expresses a defensible Bandura-grounded interpretation that differs from the primary reference, its complete mechanism can be evaluated against additional expert-approved admissible pathways. Our existing accepted-path evaluation demonstrates this capability within the reviewed ontology, accommodating multiple theoretically defensible interpretations without requiring agreement with a single reference pathway.

Second, across different educational theories: When a response invokes a genuinely different educational theory (e.g., self-regulated learning), the same learner evidence and educational scenario could support a parallel, expert-validated, theory-tagged evidence-mechanism-action pathway. Rather than forcing alternative constructs into Bandura-specific categories, an extended BanduraBench could evaluate the response against the corresponding theory's mechanisms while separately assessing pedagogical appropriateness through shared evidence-grounding, instructional relevance, and justification criteria.

This approach would distinguish theoretical mechanism fidelity, pedagogical appropriateness, and Bandura-specific recovery as complementary evaluation dimensions. Consequently, an educationally defensible recommendation supported by an alternative theory could receive appropriate credit without being classified as recovering a Bandura mechanism. The current benchmark implements accepted-path evaluation within Bandura's framework and extending this architecture to additional theories would require independently reviewed theoretical pathways and corresponding validation.

4. Conclusion

In summary, BanduraBench introduces a test bed for self-efficacy-grounded educational mechanism recovery and decision-mechanism consistency. Its contribution lies in making an established psychological dimension of educational reasoning explicitly measurable, while its pathway-based representation provides a foundation for future comparisons of alternative educational mechanisms.

AuthorsOpenReview ↗

Response to Q2: Educational Relevance of Mechanism Recovery

We thank the reviewer for suggesting this empirical test. We evaluated whether recovering more BanduraBench mechanism components is associated with stronger pedagogical-action quality among responses that already select the correct instructional decision. This directly separates mechanism recovery from final-answer success.

1. Evaluation Design

We joined the frozen DesignReasoning predictions for all 15 LLMs with the existing, human-calibrated automated Pedagogical Action Score (PAS) evaluations. Mechanism recovery is the number of correctly recovered components (0–4). To avoid defining pedagogical quality in terms of the predictor itself, we removed the PAS mechanism-to-repair alignment dimension (R) and averaged only applicable remaining action-quality dimensions.

The matched evaluation contains 15,720 responses on 1,048 PAS-eligible items

2. Descriptive Relationship

Among correct-decision responses, PAS excluding R increases monotonically with the number of recovered components:

Correct components01234
Mean PAS excluding R91.82%94.06%95.77%97.28%98.10%

This monotonic pattern motivates the item- and model-adjusted test below.

3. Correctness-Controlled Association

We estimated response-level associations using item, model, and assigned-evaluator fixed effects, with two-way clustering by item and model. The primary analysis conditions on correct final decisions, so final-answer correctness does not vary within its estimation sample. We also estimated an all-response specification that explicitly controls for decision correctness.

AnalysisPAS change per additional correct component95% CIpp-value
Correct-decision responses (N=15,548N=15{,}548)+0.58 pp[+0.15, +1.02] pp0.0125
All responses (N=15,720N=15{,}720), adjusting for decision correctness+0.66 pp[+0.22, +1.10] pp0.0061

The positive association persists after accounting for item-specific difficulty and model-level differences. It remains positive in every leave-one-model-out analysis (+0.40 to +0.64 pp per additional component). The restricted-sample and correctness-adjusted specifications also agree in direction and magnitude.

4. Interpretation

These results provide empirical evidence that recovering the specified educational mechanisms is positively associated with pedagogical-action quality beyond merely choosing the correct final answer. The evaluation framework additionally includes independent human assessment of pedagogical response quality, that we believe could strengthen the evidence for the educational relevance of the measured construct.

Our correctness-controlled analysis excludes explicit mechanism-to-repair alignment from the pedagogical-quality outcome, reducing direct overlap with component-recovery scoring. Together, we were able to determine that these findings provide convergent evidence that mechanism recovery captures educationally relevant information beyond reference-label agreement.

AuthorsOpenReview ↗

Response to Q3: Benchmark Difficulty and Distinctive Reasoning Failure

We thank the reviewer for highlighting the distinction between low strict joint recovery and a genuinely additional composition-specific failure. BanduraBench's distinctive finding is the failure of near-correct educational decisions to reliably certify recovery of the evidence-grounded mechanisms supporting those decisions. The conjunction-aware analysis is important precisely because it separates this decision–mechanism inconsistency from an unwarranted claim of an extra composition penalty.

1. What the Null Models Explain

Across 15 LLMs, mean final-decision accuracy is 96.6%, component-normalized recovery is 65.5%, and strict four-component recovery (4-Exact) is 21.4%.

However, 4-Exact requires four simultaneous correct predictions and is mechanically sensitive to component accuracy. The product-of-marginals expectation is 19.4%, and the stronger stratified permutation null gives 19.5%, leaving a mean observed-minus-null difference of +1.84 percentage points, not a large negative residual.

Thus, we do not interpret 21.4% as evidence of a separate, substantial higher-order composition deficit. Much of the reduction from per-component recovery to 4-Exact is explained by conjunction.

2. The Distinctive Failure: Decision–Mechanism Inconsistency

The benchmark reveals a different and practically important limitation: models often select the reference instructional action without accurately attributing the learner evidence, self-efficacy source, task-specific confidence construct, and course-design condition that support the benchmark's theoretical interpretation.

Decision accuracy varies little across models (95.3–98.2%), whereas component-normalized recovery ranges from 45.1% to 84.7% and 4-Exact from 5.8% to 50.4%.

Consequently, answer-only evaluation substantially compresses differences in models' ability to provide the specified evidence-grounded account, even when no additional composition penalty is present. This is an observable decision–mechanism consistency failure.

3. Component Errors Are the Diagnosis, Not an Alternative Explanation to Discard

Mechanism errors are neither uniform nor interchangeable:

Mechanism componentMean recovery
Bandura source67.0%
OLSE indicator66.9%
Course-design cluster54.2%
Causal-claim strength73.7%

Omitting course-design attribution increases joint-exact recovery by 16.1 pp on average, compared with 8.1 pp for OLSE, 7.1 pp for causal strength, and 4.6 pp for Bandura source.

The theoretical significance is that a model can get an instructional choice right yet misidentify which learner experience matters, what capability belief is implicated, or which design/causal inference is warranted.

BanduraBench makes those failures measurable instead of treating all correct decisions as equivalent. The component-level errors explain much of the low joint recovery; their presence despite near-ceiling decision accuracy is itself the distinctive finding.

4. Difficulty-Stratified Interpretation

At annotated depth 1, mean component recovery / 4-Exact is 71.1% / 27.0%; at depth 5, it is 57.0% / 13.4%.

Intermediate depths are non-monotonic, and depth covaries with task family and component mix. We therefore use these profiles to describe where attribution becomes less reliable, not to assert that higher reasoning depth independently causes a new composition failure.

Conclusion

BanduraBench identifies and localizes a capability missed by answer accuracy: recovery of the specified psychological and pedagogical mechanisms associated with a selected educational decision.

The stratified null is a strength of this claim because it prevents us from mislabeling ordinary conjunction effects as a novel reasoning deficit. Our contribution is not an unexplained excess composition penalty, but the identification and localization of mechanism-recovery failures concealed by near-ceiling final-decision accuracy. We will make this distinction explicit in the revised manuscript.

Reviewer_Fx85OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Large‑language models may produce correct final answers without mastering the educational‑theoretical reasoning mechanisms underlying those answers — a phenomenon termed the “mechanism mirage”. Grounded in Bandura’s Social‑Cognitive Theory of Self‑Efficacy, this benchmark distinguishes decision‑making accuracy from the ability to recover theoretical mechanisms. It comprises 4,530 samples and has been evaluated in a zero‑shot setting across 15 closed‑ and open‑weight large‑language models.

Critical strengths

Correct answer with incorrect reasoning chain is important for futher LLM studies.

Critical weaknesses

Except educational fields, questions with great scores are good enough for application, which means the inner correctness might not be important or even worse: human-believed correctness might be in-adequate, how to justify/verify such assumption should be considered.

Questions for authors

how to justify/verify such assumption above?

2 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Relevant Justification for the Provided Assumption

We thank Reviewer Fx85 for raising this important concern. We agree that strong final-answer performance is essential for practical applications. However, final-answer correctness alone is insufficient to establish reliable reasoning, particularly in educational settings where appropriate interventions depend on changing learner characteristics and contextual conditions.

1. Correct Answers Do Not Necessarily Imply Robust Reasoning

Prior research demonstrates that correct answers do not guarantee robust reasoning under controlled variations:

  • GSM-Plus (ACL 2024) [1]: Li et al. demonstrate that LLMs can fail on previously solved mathematical problems when question targets or contextual statements are modified.
  • GSM-Symbolic (ICLR 2025) [2]: Mirzadeh et al. show that performance can deteriorate under controlled variations of otherwise similar mathematical problems.

These findings suggest that success on an individual question does not guarantee reliable generalization to related situations. This distinction is particularly relevant in education, where changes in a learner's circumstances may require different diagnoses or interventions.

2. Answer Correctness Is Not Equivalent to Pedagogical Competence

Educational AI research provides complementary evidence that problem-solving accuracy does not necessarily translate into effective instructional behavior:

  • MathDial (EMNLP 2023) [3]: Macina et al. show that capable problem-solving models can nevertheless provide incorrect tutoring feedback or reveal solutions prematurely.
  • MathTutorBench (EMNLP 2025) [4]: Macina et al. further demonstrate that subject-matter proficiency does not necessarily translate into effective teaching.
  • KMP-Bench (AAAI 2026) [5]: Shi et al. show that models performing well on objectively verifiable tasks can struggle with nuanced pedagogical principles.
  • Bastani et al. (PNAS 2025) [6]: A randomized educational field experiment finds that AI assistance can improve immediate practice performance while negatively affecting subsequent independent performance when appropriate safeguards are absent.

Collectively, these findings motivate evaluating educational AI beyond immediate response correctness, particularly when assessing its suitability for pedagogical decision-making.

3. What BanduraBench Contributes Beyond Final-Answer Accuracy

BanduraBench addresses a complementary evaluation dimension by examining whether models recover prespecified, theory-grounded mechanisms underlying educational decisions.

Our results reveal a substantial discrepancy:

  • Final-decision accuracy: 96.6%96.6\% (average)
  • Component-normalized mechanism recovery: 65.5%65.5\% (average)

Despite near-ceiling final-decision accuracy, models exhibit substantially weaker recovery of the underlying theoretical components.

This discrepancy is precisely why mechanism-level evaluation matters: final-answer accuracy alone does not reveal whether models recover the theoretically relevant constructs and relationships associated with their decisions.

BanduraBench therefore provides a complementary diagnostic assessment of theory-grounded mechanism recovery, rather than replacing final-answer accuracy or claiming that component recovery alone establishes pedagogical effectiveness.

References

[1] Q. Li, L. Cui, X. Zhao, L. Kong, and W. Bi, "GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers," ACL, 2024, pp. 2961–2984.

[2] I. Mirzadeh et al., "GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models," ICLR, 2025.

[3] J. Macina et al., "MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems," Findings of EMNLP, 2023, pp. 5602–5621.

[4] J. Macina et al., "MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors," EMNLP, 2025, pp. 204–221.

[5] W. Shi et al., "From Solver to Tutor: Evaluating the Pedagogical Intelligence of LLMs with KMP-Bench," AAAI, vol. 40, no. 39, 2026, pp. 32965–32973.

[6] H. Bastani, O. Bastani, A. Sungu, H. Ge, Ö. Kabakcı, and R. Mariman, "Generative AI without guardrails can harm learning: Evidence from high school mathematics," PNAS, vol. 122, no. 26, e2422633122, 2025.

Dear Reviewer Fx85,

As the discussion period approaches its conclusion, we sincerely appreciate your thoughtful feedback and recognition of the importance of evaluating reasoning beyond final-answer correctness. We hope that our response and additional empirical analysis have helped clarify the educational relevance of theory-grounded mechanism recovery.

Thank you again for your valuable insights. We would be grateful if you would consider whether these clarifications and additional findings warrant an updated assessment of our submission.

Best regards,
The Authors

Reviewer_PDrJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a theory-grounded benchmark for evaluating educational reasoning in large language models, with two components: one for foundational construct knowledge and one for contextualized design reasoning.

The benchmark is built gold-first: expert-reviewed theoretical pathways, directed acyclic graph mechanism representations, human-verified seeds, controlled context variation, human adjudication, and shortcut audits are fixed before model-facing items are generated.

Across fifteen large language models, final-decision accuracy is very high, averaging 96.6 percent, while component-normalized recovery of four structured mechanism components averages 65.5 percent and strict joint recovery of all four components averages 21.4 percent.

The paper uses conjunction-aware null models and shows that most of the drop from component-level recovery to strict joint recovery is explained by conjunction rather than by an additional composition penalty.

Critical strengths

The construction rigor is the strongest reason to support acceptance. The benchmark fixes theoretical targets before surface realization, uses expert-reviewed ontology and human adjudication, controls context through a versioned context matrix, and audits for answer-position, option-length, option-only, and overlap shortcuts. This is consequential because the central claim depends on items being theory-grounded and not solvable through superficial artifacts.

The evaluation design usefully separates decision correctness from observable mechanism recovery and adds conjunction-aware null models. This is consequential because it prevents the low strict joint recovery score from being overinterpreted as a separate reasoning failure and makes the benchmark more diagnostic than a standard accuracy benchmark.

Critical weaknesses

The main weakness is construct and scope validity. The benchmark is tied to one theoretical framework, namely Bandura social cognitive theory with online-learning self-efficacy and Quality Matters constructs, and the design-reasoning evaluation supplies fixed closed-set catalogs and a structured output schema. This may measure label mapping, schema compliance, and familiarity with the supplied vocabulary more than open-ended educational reasoning, and it limits claims about educational reasoning in general.

The second weakness is the absence of an independent human expert solver baseline on the model-facing items. The authors explicitly note that their human-reference statistics are not a human ceiling or solver baseline. Without such a baseline, it is difficult to judge whether the observed component recovery is low in absolute terms or whether item difficulty and format are appropriate. This matters because the paper’s central interpretation depends on the gap between near-ceiling decisions and much lower mechanism recovery being meaningful.

Questions for authors

Can you provide an independent human expert baseline on a sample of model-facing design-reasoning items, without access to gold pathways, to establish item answerability and a human reference ceiling?

Can you report an ablation comparing the current structured, catalog-supplied output format with a free-form or catalog-free variant, to show how much of the component-recovery gap is due to schema and vocabulary scaffolding rather than reasoning?

How sensitive are the main conclusions to the choice of model-based judges and to the split-judge production allocation, given that the held-out judge audit shows only moderate chance-corrected agreement?

4 comments
AuthorsOpenReview ↗

Response to Q1: Independent Human Expert Baseline

We thank the reviewer for suggesting an independent human expert baseline. To address this concern, we conducted an additional independent, blinded human recoverability study involving three qualified educational experts and 100 BanduraBench items, balanced across all the task families.

1. Study Design and Blinding Protocol

Each expert independently evaluated the original model-facing scenarios, questions, and answer options using the public theoretical construct catalogs.

Crucially, experts had no access to:

  • Protected gold answers or reference mechanisms/pathways;
  • Benchmark construction metadata or model predictions;
  • One another's annotations.

Each expert selected the final instructional decision and independently identified four theoretical components: Bandura source, online-learning self-efficacy (OLSE) indicator, course-design cluster, and causal-claim strength.

This yielded 300 independent expert–item annotations, covering both unique-pathway and multi-admissible cases.

2. Independent Human Expert Performance

Metric (%)Expert 1Expert 2Expert 3Mean
Decision accuracy98.099.099.098.67
Component-normalized recovery95.395.096.395.5
Primary 4-Exact82.081.085.082.7
Adjudicated accepted 4-Exact87.089.092.089.3

3. Evidence of Item Answerability and Mechanism Recoverability

The blinded study provides direct empirical evidence that the benchmark's theoretical targets are recoverable by qualified experts without privileged access to gold references.

Three findings support this conclusion:

  1. High item answerability: Experts achieved 98.67% mean final-decision accuracy, demonstrating that the model-facing scenarios are highly answerable by qualified human evaluators.

  2. Strong mechanism recoverability: Experts achieved 95.5% component-normalized recovery and 82.7% strict four-component (4-Exact) recovery against the original primary references. Furthermore, all three experts independently agreed on the complete four-component tuple for 88/100 items, indicating substantial consistency in theoretical interpretation.

  3. Accepted-pathway validation: Under the original accepted-pathway references, expert 4-Exact recovery was 84.0%, increasing to 89.3% following adjudication. Original expert annotations remained unchanged, and no items were excluded.

We aim to present the independent, blinded human study to establish high item answerability and substantial theoretical mechanism recoverability without access to gold pathways. It provides an empirical human-performance reference demonstrating that BanduraBench's theoretical targets are accessible to qualified experts, while revealing a substantial gap between human and LLM mechanism recovery.

AuthorsOpenReview ↗

Response to Q2: Ablation of Catalog and Output-Format Scaffolding

We thank the reviewer for this important suggestion. To investigate whether the observed decision–mechanism recovery gap arises from catalog-supplied vocabulary and structured output requirements, we conducted a controlled ablation across four representative LLMs on 100 matched items per model, balanced across task families.

1. Ablation Design

We evaluated three elicitation conditions:

  • A - Original: Catalog-supplied structured output using the original mechanism specification.
  • B - Catalog-free JSON: No supplied construct catalogs; models independently describe the four theoretical components in structured natural-language fields.
  • C - Catalog-free prose: No supplied catalogs or fixed component fields; models provide free-form explanations addressing the same theoretical dimensions.

Catalog-free responses were mapped to the original four-component ontology through gold-blind semantic review. Responses lacking sufficient evidence to identify a specific construct were marked unrecovered. All conditions were scored against frozen gold references.

2. Quantitative Ablation Results

Table 1. Four-model macro-averaged results (100 matched items per model).

MetricA: OriginalB: Catalog-free JSONC: Catalog-free prose
Final-decision accuracy97.75%97.75%97.00%
Component-normalized recovery67.38%57.25%53.81%
Primary 4-Exact recovery23.00%10.00%8.75%
Accepted-path 4-Exact23.75%10.25%9.25%

Key finding: Final-decision accuracy remains nearly unchanged across conditions (97.0–97.75%), whereas component recovery remains substantially lower under every elicitation format.

Even under unrestricted prose elicitation (C), models achieve only 53.81% component-normalized recovery and 8.75% strict four-component recovery, despite 97.00% final-decision accuracy.

Thus, the decision–mechanism discrepancy persists when the original catalog and structured-output requirements are removed.

3. Quantifying the Effect of Scaffolding

The original interface improves explicit theoretical mechanism recoverability. Relative to condition A:

  • A to B (removing catalog support and canonical component elicitation): Component recovery decreases by approximately 10.1 percentage points (95% paired-bootstrap CI for B - A: [−12.50,−7.69][-12.50,-7.69] pp).
  • A to C (catalog-free, unrestricted prose): Recovery decreases by approximately 13.6 points (95% CI for C - A: [−16.19,−10.88][-16.19,-10.88] pp).
  • B to C (structured fields versus free-form prose, without catalogs): Recovery decreases by an additional 3.44 points (95% CI for C - B: [−5.62,−1.25][-5.62,-1.25] pp).

These comparisons show that catalog and formatting scaffolds facilitate explicit, consistently recoverable theoretical representations. The B-C comparison additionally isolates the effect of structured versus free-form elicitation within catalog-free conditions.

Importantly, removing these scaffolds widens rather than eliminates the observed decision–mechanism recovery gap.

4. Implications for BanduraBench

The matched ablation establishes two complementary findings:

  1. Scaffolding affects measured mechanism recovery: Supplying theoretical catalogs and requiring structured responses improves the explicit recoverability of theoretical components.
  2. Scaffolding does not explain away the central discrepancy: Near-ceiling final-decision accuracy coexists with substantially weaker mechanism recovery across all three elicitation conditions.

Overall, we aim to utilize this ablation to strengthen BanduraBench's central motivation: final-answer correctness alone is insufficient to characterize models' explicit recovery of theoretically grounded educational mechanisms.

AuthorsOpenReview ↗

Response to Q3: Robustness to Model-Based Judges and Split-Judge Allocation

Thank you for raising this important question. We distinguish between the primary deterministic mechanism-recovery results and the complementary LLM-judged rationale evaluations. To directly assess judge sensitivity, we conducted an additional matched-response cross-judge robustness experiment involving all 15 evaluated models.

1. New Matched-Response Cross-Judge Experiment

To investigate the reviewer's concern regarding judge agreement, we conducted an additional experiment using 300 identical target-model responses, covering 20 shared benchmark items (two per task family) across all 15 models.

Each response was evaluated by both Sol and Terra using the frozen 16-dimensional rubric. We paired the archived production judgment with a new evaluation from the alternative judge, holding the target response and task context fixed.

All 300 matched pairs were successfully evaluated, yielding 1,920 applicable ordinal-score comparisons.

Table 1. Cross-judge agreement on identical model responses.

Agreement measureResult
Matched response pairs300
Applicable rubric-score pairs1,920
Exact ordinal agreement82.14%
Within-one-point agreement99.38%
Quadratic-weighted κ\kappa0.804
Strict Pass agreement94.33%
Strict Pass Cohen's κ\kappa0.873

The judges agreed exactly on 82.14% of applicable ordinal scores, while 99.38% differed by no more than one point. The quadratic-weighted agreement coefficient was κ=0.804\kappa=0.804.

For the principal rationale-level pass criterion, the judges agreed on 283/300 Strict Pass decisions, corresponding to 94.33% agreement and Cohen's κ=0.873\kappa=0.873.

These results provide additional evidence of strong Sol–Terra consistency on matched responses.. The new experiment specifically evaluates the two production judges on identical responses.

2. Stability of Rationale-Based Conclusions

We independently recomputed the Theory Reasoning Score (TRS), Pedagogical Action Score (PAS), and Strict Pass under each judge, using paired comparisons and bootstrap confidence intervals clustered by benchmark item.

Table 2. Cross-judge stability of rationale outcomes.

Robustness measureResult
TRS model-ranking correlation (Spearman ρ\rho)0.911
Strict Pass model-ranking correlation (Spearman ρ\rho)0.932
Sol Strict Pass rate66.33%
Terra Strict Pass rate66.00%
Paired Strict Pass difference (Sol − Terra)+0.33 pp
95% item-clustered bootstrap CI[−3.00,+3.33][-3.00,+3.33] pp

The strong rank correlations for TRS (ρ=0.911\rho=0.911) and Strict Pass (ρ=0.932\rho=0.932) indicate that model-level comparisons remain highly consistent between the two judges.

Moreover, the substantive PAS-versus-TRS pattern persists under both judges:

  • Sol: PAS = 97.08%; TRS = 85.10%.
  • Terra: PAS = 95.21%; TRS = 86.77%.

Thus, both judges independently produce higher pedagogical-action scores than theory-reasoning scores, supporting the interpretation that producing appropriate pedagogical actions does not necessarily entail complete explicit theory-grounded reasoning.

The overall Strict Pass rates also differ by only 0.33 percentage points, with a paired 95% bootstrap confidence interval of [−3.00,+3.33][-3.00,+3.33] pp.

3. Split-Judge Production Allocation

In the original evaluation, Sol and Terra each assessed 1,277 disjoint responses per target model, collectively covering all 2,554 rationale-eligible items.

Across the 15 models, the observed mean absolute differences between the two production subsets were:

MetricMean absolute subset difference
TRS1.43 pp
PAS2.78 pp
Strict Pass2.57 pp

Because these production subsets were disjoint, the observed differences reflect both possible judge effects and item-subset composition. They therefore provide a descriptive production-allocation check, not an isolated estimate of judge disagreement.

The new matched-response experiment directly addresses this limitation by scoring the same responses under both judges, removing item-subset composition differences from the paired comparison.

4. Implications for the Main Conclusions

We believe that the additional evidence supports two conclusions:

  1. Primary findings are judge-independent. The headline decision–mechanism recovery gap and conjunction-aware analyses rely on deterministic scoring against frozen references, not LLM judgments or split-judge allocation.

  2. Complementary rationale findings are consistent across the two production judges. The matched-response experiment demonstrates strong ordinal and Strict Pass agreement, highly correlated model rankings, and preservation of the PAS-versus-TRS pattern.

Overall, we hope that the new experiment strengthens the robustness evidence for the supplementary rationale evaluations while confirming that BanduraBench's principal findings and contributions

Dear Reviewer PDrJ,

As the discussion period approaches its conclusion, we sincerely appreciate your thoughtful feedback and positive assessment of our benchmark construction, evaluation framework, and conjunction-aware analyses. We hope that our additional experiments and detailed responses have addressed your concerns regarding human recoverability, elicitation sensitivity, and judge robustness.

Thank you again for your valuable suggestions. We would be grateful if you would consider whether these additional findings warrant an updated assessment of our submission.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal

We thank all three reviewers for their constructive feedback and careful assessment of BanduraBench. We appreciate their recognition of the benchmark's systematic construction, expert-reviewed theoretical grounding, human adjudication, and conjunction-aware evaluation. In response to the concerns raised, we conducted five additional validation analyses and clarified the methodological contributions and scope of the benchmark. We summarize these developments below.

1. Construct validity, theoretical scope, and methodological novelty

BanduraBench evaluates observable decision–mechanism consistency within a specified Bandura-grounded framework, rather than general educational competence. Its methodology integrates gold-first construction, expert-reviewed targets, DAG-based admissible pathways, controlled contextual variation, and conjunction-aware evaluation. Human-approved alternative pathways accommodate defensible interpretations within the framework. Future work could extend this architecture to other social cognitive theories through independently reviewed, theory-tagged pathways and corresponding validation, enabling systematic investigation of cross-theory generalization.

2. Independent human baseline and answerability

Three educational experts in a blind study independently evaluated 100 model-facing items without access to protected references or model predictions. They achieved 98.67% decision accuracy, 95.5% component recovery, and 82.7% primary 4-Exact. Adjudicated accepted-path 4-Exact reached 89.3%, reported separately. These results support the recoverability of the specified targets on sampled items.

3. Catalog and output-format sensitivity

Across four LLMs and 100 matched items per model, decision accuracy remained 97.00–97.75%, while component recovery was 67.38% (original structured), 57.25% (catalog-free JSON), and 53.81% (catalog-free prose). The discrepancy persisted across elicitation conditions, although catalog-free responses required semantic mapping.

4. Judge and scoring robustness

Two production judges independently evaluated the same 300 model responses, achieving 82.14% exact ordinal agreement (weighted κ=0.804\kappa=0.804) and 94.33% Strict Pass agreement (κ=0.873\kappa=0.873). The principal decision and mechanism-recovery metrics remain independent of these judges because they use deterministic scoring against frozen references.

5. Educational relevance beyond correct decisions

Among 15,548 correct-decision responses, each additional recovered component was associated with a +0.58 percentage-point increase in human-calibrated, automated pedagogical-action quality (95% CI: [+0.15, +1.02]), adjusting for item, model, and evaluator effects. The mechanism-to-repair dimension was excluded to reduce scoring overlap. This observational association supports diagnostic relevance but does not establish improved learner outcomes.

6. Generalizability and interpretation of joint recovery

Across 416 test items from source-seed families and pathways absent from development and validation, the decision–component gap persisted at 24.17 percentage points (95% CI: [18.90, 29.08]), remaining positive across all 15 models. This demonstrates within-framework robustness rather than external classroom or cross-theory generalization.

BanduraBench identifies an observable failure mode in which models select the benchmark-preferred educational decision while misattributing its theoretical basis. Despite 96.6% mean decision accuracy, component recovery averages 65.5%, with course-design attribution weakest (54.2%). The benchmark therefore localizes theoretical attribution errors that final-answer correctness conceals.

Mean strict 4-Exact (21.4%) is close to the stratified permutation expectation (19.5%), indicating that most joint-recovery loss follows from accumulated component errors. Hence, we clarify that the distinctive contribution is therefore a conjunction-aware diagnosis of observable decision–mechanism inconsistency,

Summary

We believe that these analyses could strengthen evidence for benchmark answerability, elicitation robustness, scoring reliability, and educational relevance while clarifying the scope of the central finding. The revised manuscript incorporates the additional results, with full details in Appendix I. We appreciate the reviewers' suggestions in strengthening the evaluation and interpretation of BanduraBench.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.