acceptodds
Under review as a conference paper at ICLR 2027

AIMPACT: An Evidence-Grounded Multi-Agent Framework for Automated Presentation Assessment

Abstract

Automated presentation assessment is essential for scalable presentation training and feedback in higher education and workplace settings. Despite recent advances in Multimodal Large Language Models (MLLMs), presentation assessment remains constrained by the direct mapping of heterogeneous multimodal evidence to final assessments, which couples evidence reliability with rubric-based judgment. To address this problem, we introduce AIMPACT, an evidence-grounded multi-agent framework for automated presentation assessment. Specifically, we first propose an Evidence-to-Judgment Multi-Agent Framework (EJMA) to organize presentation assessment into a two-level hierarchy. At the first level, multimodal observations are converted into a traceable evidence memory and coordinate seven dimension-specific specialist agents through a reliability-guided collaboration graph, forming an evidence-grounded council prior. On this basis, the second level introduces Rubric-Conditioned Multi-Expert Preference Optimization (RC-MEPO), which first learns expert corrections to the council prior and then performs on-policy preference optimization against the confidence-weighted expert score distribution, where controlled rubric perturbations further enforce assessment consistency with the provided scoring rubric. In addition, we construct a Multi-Expert Presentation Assessment Dataset (MEPA) that comprises 35 graded classroom presentations with synchronized audio, video, and slides. Five expert instructors independently assessed each presentation across eight rubric dimensions, resulting in 1,400 individual expert judgments. Extensive objective and subjective evaluations show that our method achieves the strongest presentation-level expert alignment among comparable-scale MLLMs and remains competitive with substantially larger proprietary models. Zero-shot evaluations on out-of-domain datasets and additional analyses further demonstrate its ability to generalize to unseen presentation data and different scoring standards.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Automated Presentation Assessment, Multi-Agent Systems, Multimodal Presentation Assessment Datasets

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JvuzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript addresses the challenges of multimodal evidence reliability and scoring in automated speech assessment and proposes an agent-based evaluation framework, AIMPACT. The framework generates preliminary scores through multimodal evidence extraction, temporal evidence memory, and reliability-guided multi-agent collaboration. The authors further introduce RC-MEPO, which combines supervised fine-tuning with online preference optimization to align model-generated scores with the score distributions of multiple human experts and improve adaptation to different evaluation criteria. Experimental results show that the proposed framework outperforms existing multimodal large language models.

Critical strengths
  1. The research problem has clear practical relevance.

  2. The proposed AIMPACT framework is reasonably designed and comprehensive.

  3. The experimental results provide strong evidence for the effectiveness of AIMPACT.

Critical weaknesses
  1. The authors should further verify whether the performance gains arise from structured multimodal features and expert supervision, or from multi-agent collaboration.

  2. The novelty of RC-MEPO appears limited, as it mainly lies in the combination of different reward designs.

  3. During SFT, the authors introduce 30% expert correction, but the manuscript lacks a detailed explanation or analysis of this design choice.

  4. MEPA contains only 35 independent speech samples. The dataset size is limited, and evaluations across multiple dimensions should not be treated as independent samples.

  5. The authors are encouraged to provide additional experimental analysis of computational overhead.

Questions for authors

The manuscript excessively compresses the vertical spacing around tables. Please refer to the weaknesses above for the remaining concerns.

Reviewer_C72fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AIMPACT, an evidence-grounded multi-agent framework designed to automate the assessment of multimodal presentations. The framework utilizes a two-level architecture: the first level is an Evidence-to-Judgment Multi-Agent Framework (EJMA) that converts multimodal inputs into a traceable temporal evidence memory and coordinates 7 specialist agents through a reliability-guided collaboration graph to form an evidence-grounded council prior. The second level is Rubric-Conditioned Multi-Expert Preference Optimization (RC-MEPO), which applies supervised fine-tuning and policy preference optimization to align the model's output with a confidence-weighted expert score distribution rather than a single mean score. To support this, the authors introduce the MEPA dataset, which contains 35 classroom presentations independently rated by 5 experts across eight rubric dimensions.

Critical strengths
  1. Decoupling Evidence Collection from Subjective Scoring: Existing multimodal large language models (MLLMs) typically map heterogeneous evidence directly to final scores, whereas the proposed EJMA framework first constructs an objective and traceable evidence memory before handing it over to specialist agents for subjective scoring. This design effectively reduces logical confusion during the evaluation process and cleanly separates the reliability of evidence acquisition from the scoring decision itself.
  2. Innovative Expert-Distribution Alignment (EDA) Mechanism: During the preference optimization (PO) stage, the authors do not simply average multiple experts' scores into a single scalar. Instead, they construct a confidence-weighted expert score distribution and optimize the model output to fit this distribution. This design highly aligns with the characteristics of subjective evaluation in the real world, reasonably preserving valid disagreements among human raters.
Critical weaknesses
  1. Severe Unfairness in Baseline Comparisons: In the main experiment (Table 1), the authors compare the AIMPACT system—which was fine-tuned on the training folds of the target domain (MEPA)—against zero-shot general-purpose large language models (such as Qwen2.5, Gemini 3.1 Pro). The authors explicitly acknowledge in the appendix that the general-purpose baselines were completely frozen and received no MEPA labels for task adaptation. Directly comparing a system that has been fully optimized (SFT+PO) on target data against zero-shot models fails to objectively prove that the proposed multi-agent architecture itself is superior to other models. 2.Extremely Small Dataset Size Threatens Generalization Claims: The MEPA dataset contains a total of only 35 presentations (under 5-fold cross-validation, only 7 are used for testing per fold). Furthermore, the Cross-Course evaluation set used for cross-domain generalization testing contains only 12 presentations. Performing complex fine-tuning and preference optimization on such a microscopic sample size is highly prone to severe overfitting, which significantly weakens the paper's claims regarding robust generalization capabilities.
  2. Model Over-Engineering and Diminishing Marginal Returns: The ablation study (Table 3) shows that adding the Multi-Agent Collaboration Graph (MACG) on top of Multimodal Evidence Extraction (MEE) and Evidence Reliability (ER) yields only a marginal improvement in the Spearman correlation (increasing slightly from 0.704 to 0.722). This indicates that the high computational and inference costs introduced by the massive 7-agent multi-round collaboration mechanism may lack sufficient performance gains to justify its inclusion.
Questions for authors
  1. To verify the true effectiveness of the Multi-Agent Collaboration Graph (MACG) and the preference optimization pipeline (RC-MEPO), could you provide a fair baseline comparison? For example, directly fine-tuning the exact same Qwen3-8B backbone model on the MEPA dataset using standard supervised fine-tuning (SFT) without the complex multi-agent architecture?
  2. Given that the MEPA dataset has only 35 unique presentation samples (with 28 used for training per fold), how did you ensure that the model did not severely overfit to the specific characteristics or topics of this tiny group of speakers during the SFT and PO training stages of RC-MEPO?
  3. As shown in Table 3, the ranking correlation improvement brought by the MACG module is very limited. Considering the increased inference latency and computational overhead introduced by multi-agent multi-round deliberation, how do you justify the necessity of MACG for real-world application deployment?
1 comment

Thank you for the additional experiments and detailed clarifications. The fine-tuned baselines and same-backbone comparisons substantially address my concern about comparison fairness, and the accuracy-latency analysis better justifies MACG for offline assessment. However, despite the reassuring overfitting analyses, the small number of independent presentations still limits the evidence for generalization, so I maintain my recommendation.

Reviewer_egVXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AIMPACT, a framework for automated demonstration evaluation that decouples the extraction of multimodal evidence from the scoring process based on predefined evaluation criteria. It combines reliability-guided collaboration among specialist agents with expert-distribution alignment and rubric-conditioned preference optimization. Furthermore, the authors introduce the MEPA dataset, which comprises 35 demonstrations evaluated by five experts across eight dimensions.

Critical strengths
  • The separation of evidence and judgment. The proposed framework organizes multimodal observations into traceable evidence records, providing a useful structure for distinguishing unreliable observations from scoring errors.
  • The MEPA dataset is meaningful as it conrains multi-expert annotations. MEPA retains each expert’s individual scores, confidence ratings, and textual feedback, allowing researchers to go beyond a single consensus score and explore disagreements among experts.
Critical weaknesses
  • The justification for cross-dimensional collaboration is insufficient. MACG uses score disagreement and evidence reliability to trigger collaboration among specialists assessing different dimensions. However, differences in scores across dimensions — such as pronunciation, fluency, and prosody — may reflect genuine variations in a speaker’s actual proficiency rather than errors in assessment. It remains unclear whether such collaboration corrects unreliable judgments or merely forces an unjustified convergence of scores. Similar concerns arise regarding the inclusion of discrepancies across dimensions into the "residual uncertainty" metric.
  • The primary comparative experiments lack strong baseline models operating under equivalent supervisory conditions. While AIMPACT benefits from task-specific training, structured evidence, and multiple inference samples, the general-purpose baseline models remain static (i.e., they are not fine-tuned for the task). Although this distinction is noted in the appendix, the experimental results do not demonstrate whether the proposed mechanism truly outperforms simpler approaches under comparable conditions.
Questions for authors
  1. Could high evidence reliability coexist with insufficient coverage when missing observations are excluded from the average?
  2. Could you clarify the implementation of the distributional metrics? Appendix C.3 defines Brier score as the mean squared error across five probabilities, whose upper bound is 0.4, whereas Table 6 reports values above this bound.
  3. How does RC-MEPO compare with direct supervised learning of expert score distributions, and does ER-only masking explain the corruption robustness gains?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.