- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DeliverBench, a benchmark comprising 129 tasks that require multimodal agents to turns a batch of heterogeneous material into a structured artifact that downstream software consumes. The authors claim that batch-processing tasks are more challenging than long-horizon interactive tasks, a phenomenon they term “batch-horizon inversion.” However, this conclusion may be an artifact of the benchmark's limited task coverage rather than a generalizable phenomenon.
Critical strengths
-
The authors conduct extensive experiments across 26 models and provide human performance results as a reference.
-
The proposed tasks align well with real-world needs for batch processing heterogeneous data.
Critical weaknesses
-
The claimed batch-horizon inversion may result from limited and imbalanced task coverage, with 62 batch-processing tasks but only 16 long-horizon interactive tasks.
-
Why is a threshold of 0.999 used for Pass Rate? Would relaxing it to 0.95 or 0.90 change the claimed batch-horizon inversion?
-
Could using only one judge model introduce evaluation bias, such as preferences toward models from the same family?
-
Relying solely on an LLM judge to evaluate deliverables requires human validation to demonstrate the reliability of LLM-as-a-judge in this setting.
Questions for authors
See Weaknesses.