- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Multimodal-CL-Bench to evaluate whether multimodal language models can acquire case-specific knowledge from context and apply it in subsequent tasks. The benchmark contains 344 contexts, 1,080 tasks, and 7,309 rubric criteria, organized into four capability categories and 20 subcategories. Cases combine expert construction and controlled synthesis. Evaluation uses strict all-criteria task success judged by GPT-5.2.
Critical strengths
- The benchmark targets a practically relevant capability: applying newly supplied local rules to heterogeneous evidence over multiple tasks. Its four-category taxonomy, combination of expert and synthetic cases, and detailed rubrics provide a substantial resource for studying failures that conventional perception-focused benchmarks may miss. The construction procedures explicitly address distractors, exceptions, and context-dependent rules, which are useful design features for this goal.
- The paper provides more than a single leaderboard. Appendix G performs a paired multimodal-evidence ablation on 551 tasks, with average success decreasing from 6.15% to 2.52% when visual evidence is removed. Appendix I extends the evaluation to two Kimi models, reporting 12.87% task success for K2.6 and 22.31% for K3. These analyses provide useful evidence that the inputs matter and that the benchmark can distinguish improvements, rather than placing all models at an identical performance floor.
Critical weaknesses
All-or-nothing scoring jointly evaluates semantic correctness, completeness, and format compliance, while rubric counts vary substantially across categories (Table 1). Consequently, low pass rates and category differences may partly reflect the number and strictness of requirements, complicating the capability-level conclusions drawn from these results.
Questions for authors
- What are the exact aggregation formulas and success counts for Table 2 and Figure 6? Are scores micro-averaged over tasks, macro-averaged over contexts, or based on a case-level pass criterion? Please reconcile these values and recompute comparisons using one consistent unit.
- What is human judge agreement on a stratified sample, particularly for responses rejected by a single criterion?
6 comments
3. Author review and single-requirement failures
We have an existing targeted author review of 100 responses, covering unanimous passes, unanimous failures, and disagreements among the three judges. The review examines whether responses actually satisfy explicit task requirements and whether partial or implicit satisfaction is incorrectly accepted as complete satisfaction. Disagreement examples and their grading rationales are included in the anonymous supplement.
For responses failing because of a single rubric, we focus on the substantive role of that requirement. Violating a case-local rule, citing evidence that does not support the conclusion, or omitting a necessary state update can prevent correct task completion even when all other requirements are satisfied. Conversely, synonymous wording, formatting differences not explicitly constrained, or harmless additions that do not change the evidential basis should not automatically cause failure. All-or-nothing scoring should therefore enforce explicit task requirements rather than mechanically match wording.
To make these boundary cases inspectable, we provide per-requirement satisfaction, counts of single-requirement failures, and representative responses with grading rationales. The existing review does not separately report human agreement for this stratum. We distinguish the available qualitative review from stratum-specific quantitative statistics and do not substitute three-judge agreement for judge–human agreement.
Materials: Single-requirement failure records and inspection guide and Three-judge judgments and review materials.
Thanks for quick reply, My questions have been answered with the calculation formulas and human evaluation. I hope you can correct the numerical results and rankings in the manuscript based on the clarifications before the discussion deadline. I recommand you to show these changes and additional discussions in paper using different color. It will be clearer for AC and us to check! After the corrections, I will consider raising my score.
Thank you for recognizing the value of local-rule tasks, evidence ablation, and distinguishing model improvements. Your questions concern the interpretation of complete-task success and the reproducibility of aggregation.
1. Why retain all-or-nothing scoring?
We adopt all-or-nothing scoring based on the characteristics and evaluation objectives of context-learning tasks. Beyond answering conventional knowledge questions, context learning requires models to understand new knowledge in the supplied context and accurately integrate and apply it to complex tasks with multiple constraints.
First, our primary objective is to measure whether a model can solve a task completely and reliably, rather than merely satisfy some requirements. In practical applications, task constraints are often interdependent: omitting or misapplying a key rule can make the final result incorrect. Simply averaging satisfaction across criteria may therefore overstate the model's ability to solve the task.
Second, all-or-nothing scoring provides a stricter test of the integrated application of contextual knowledge. A model succeeds only when it satisfies all predefined grading criteria. This avoids assigning a high task score solely for partial understanding or locally correct reasoning and better reflects the completeness and reliability required by complex tasks.
This setting is also consistent with CL-Bench (Dou et al., 2026), which uses binary task-level scoring: a task is considered solved only when the response satisfies all associated rubrics. We follow this evaluation principle to distinguish complete task success from partial requirement satisfaction. See the official CL-Bench evaluation description.
Rubric-level satisfaction complements this metric by showing partial correctness, while task success stratified by rubric count examines how requirement count relates to performance. Neither replaces complete task success. Rubric-level statistics are restricted to records with equal rubric and judgment-entry counts; model coverage differs, so these are not rankings on a common subset. Results are provided in Rubric-level diagnostics.
2. Aggregation formulas and success counts
Thank you for highlighting the need to clarify the aggregation scope. The original Overall values in Table 2 cover the first-turn tasks of 344 cases, whereas Figure 6 covers all tasks within each source track. We will consistently use task-level micro-averaging: the number of successful tasks divided by the corresponding total number of tasks. The denominators are 1,080 for the full benchmark, 325 for the expert-collected subset, and 755 for the synthetic subset. Per-model success counts and reproducible aggregation records are provided in the supplement.
For any task set S, the metric is Am(S)=∑t∈Ssm,t/∣S∣, where sm,t is the stored binary task score and missing scores count as end-to-end failures. Figure 6 uses the source-specific denominators of 325 and 755, rather than averaging case-level scores. Full-benchmark success counts are:
Reproduction retains stored scores. One Gemini-2.5 record has a score/status conflict; status normalization gives 78/1,080, and the supplement reports this sensitivity result separately. Full reproduction and source-track records are in Task-level scores and reproduction materials.
Our core observations remain under consistent aggregation: complete task success remains challenging, and the gains from Thinking are not consistent across models. The Thinking variant improves Opus by 1.94 percentage points, while the gain for Sonnet is only 0.09 percentage points. These results indicate that additional reasoning does not automatically translate into more reliable learning and application of contextual rules. We will clarify the original statement as “Thinking does not consistently improve performance” and update the numerical results and rankings accordingly.