- Rating
- 3
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper’s main contributions is (according to the authors) the first agentic framework for coral reef ecological analysis. Additionally, Coral-Bench is introduced, a multiple-choice question-answer for visual understanding across 23 tasks. This dataset is used to evaluate the CoralAgent framework. The benchmark contribution is rather limited, consisting of only 1.9k manually verified questions over ~2.5k, compared to for instance CoralVQA, covering 12.8k images. The benchmark appears to suggest improved task accuracy over GPT baseline.
Critical strengths
It is encouraging to see agentic AI applied to an ecological domain, as much of the existing research focuses on more conventional general-purpose task automation. However, it would be valuable to expand on the generalizability of the proposed framework to other ecological tasks (e.g., bird monitoring [1]) to better understand its broader applicability and limitations. The literature review provides limited coverage of existing agentic AI solutions addressing similar tasks (i.e., RS-Agent and Earth-Agent). A more comprehensive discussion of related agentic architectures, their evaluation methodologies, and reported limitations would help better position the proposed approach within existing research and clarify its novelty and contributions.
- Kryklyvets, Yevheniia, et al. "MAviS: A multimodal conversational assistant for avian species." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
Critical weaknesses
My main concern is the evaluation. Currently the benchmark is conducted only once per question; recent work [1,2,3] rise importance of VLM assessment in terms of reliability and response consistency. This is particularly problematic for multiple-choice questions because a correct answer does not necessarily demonstrate reliable reasoning or understanding. Table 2 (Appendix A.1.3) shows that a substantial portion of the benchmark relies on a one-image, one-question setup. Also, the benchmark reports accuracy per task (e.g., coral cover task has 135 questions while bleached coral recognition has 300 questions); and a single correct response does not necessarily indicate reliable visual understanding, particularly in multiple-choice settings where guessing is possible. Consider other metrics like Pass@K, paraphrase consistency, option-order robustness.
- Gu, Yu, et al. "The illusion of readiness in health AI." arXiv preprint arXiv:2509.18234 (2025).
- Sun, Yingying, Henry Levesque, and Lu Lin. "Stability Without Sensitivity: Evaluative Description and Directional Judgment in Vision–Language Models." Design and Artificial Intelligence (2026): 100082.
- Cheng, Zerui, et al. "Position: Benchmarking is Broken-Don't Let AI be its Own Judge." Advances in Neural Information Processing Systems 38 (2026).
Questions for authors
1.Do the authors have any insight into how stable are the reported results across repeated independent evaluations of the same image–question pairs? I am curious if the conclusions remain consistent across multiple runs.
2.How sensitive are model predictions to changes in question phrasing or the ordering of multiple-choice options? I encourage the authors to consider this evaluation.
3.To what extent do correct predictions reflect actual visual understanding rather than guessing or reliance on linguistic priors? Could an image-blind baseline be established?
4.Given the varying number of questions per task, are the observed performance differences statistically significant? Could you report confidence intervals?
5.Why is accuracy the primary evaluation metric? Could you provide additional metrics?
3 comments
Q2: Sensitivity to question phrasing and option order
How sensitive are model predictions to changes in question phrasing or the ordering of multiple-choice options? I encourage the authors to consider this evaluation.
Response:
We thank the reviewer for this suggestion.
Option ordering. By design, CoralAgent is insensitive to option ordering on quantitative tasks. For example, on Pielou’s evenness index calculation, CoralAgent first computes the index value from the sampled evidence and then selects the option closest to that value. To verify this empirically, we randomly shuffled the option order four times on three tasks:
Table Q2: Accuracy (%) of CoralAgent under four random shuffles of the answer options. Mean ± SD is computed over the four shuffles.
| Task | Shuffle 1 | Shuffle 2 | Shuffle 3 | Shuffle 4 | Mean ± SD |
|---|---|---|---|---|---|
| Pielou’s evenness index | 60.27 | 60.27 | 61.64 | 60.27 | 60.61 ± 0.68 |
| Coral richness assessment | 55.56 | 55.56 | 56.79 | 56.79 | 56.18 ± 0.71 |
| Bleached coral cover estimation | 56.25 | 57.81 | 56.25 | 56.25 | 56.64 ± 0.78 |
Across shuffles, accuracy varies by at most one question per task, which is consistent with the variation observed across repeated runs.
Question phrasing. We agree this is a valuable evaluation. We will conduct experiments with paraphrased questions to assess the sensitivity of CoralAgent and the baselines to question phrasing, and we will report these results in the revised version.
Q3: Image-blind baseline
To what extent do correct predictions reflect actual visual understanding rather than guessing or reliance on linguistic priors? Could an image-blind baseline be established?
Response:
We thank the reviewer for this suggestion. We established an image-blind baseline in which GPT-5.4 receives only the question and options, without images:
Table Q3: Accuracy (%) of the image-blind GPT-5.4 baseline (question and options only, no images), GPT-5.4 with images, and CoralAgent.
| Task | Image-blind GPT-5.4 | GPT-5.4 (with images) | CoralAgent |
|---|---|---|---|
| Coral cover estimation | 14.07 | 20.00 | 72.59 |
| Family-level coral recognition | 26.53 | 68.37 | 80.61 |
| Genus-level coral recognition | 22.07 | 42.34 | 61.71 |
| Species-level coral recognition | 32.00 | 52.00 | 60.00 |
| Coral composition analysis | 14.81 | 27.16 | 43.21 |
| Dominant genus recognition | 34.57 | 56.79 | 65.43 |
| Shannon diversity index | 12.33 | 23.29 | 52.05 |
Three observations follow:
(1) Linguistic priors provide little advantage. Image-blind accuracy is close to or below chance level (25%) on most tasks. Although GPT-5.4 was used to generate the distractors, it cannot recover the correct answers from the question and option text alone. The slightly higher scores on species-level recognition and dominant genus recognition likely reflect prior knowledge of common coral taxa, such as genera that frequently dominate reef communities. (2) CoralAgent’s correct predictions depend on visual evidence. CoralAgent exceeds the image-blind baseline by 28.0–58.5 points on every task. Its answers are also traceable to visual evidence, since each prediction is derived from recorded masks and point-level labels. (3) Direct prompting makes limited use of visual information on quantitative tasks. With images, GPT-5.4 improves only modestly over the image-blind baseline on coral cover (+5.9) and Shannon diversity (+11.0). This indicates that direct prompting fails to extract quantitative information from images, which is the limitation that CoralAgent addresses.
We will report image-blind results for all tasks in the revised version.
Q4: Statistical significance and confidence intervals
Given the varying number of questions per task, are the observed performance differences statistically significant? Could you report confidence intervals?
Response:
We thank the reviewer for this suggestion. We report the accuracy of GPT-5.4 and CoralAgent with 95% Wilson confidence intervals:
Table Q4: Accuracy (%) of GPT-5.4 and CoralAgent with 95% Wilson confidence intervals.
| Task | GPT [95% CI] | CoralAgent [95% CI] |
|---|---|---|
| Coral cover estimation | 20.00 [14.12, 27.54] | 72.59 [64.52, 79.41] |
| Bleached coral cover estimation | 14.06 [7.58, 24.62] | 56.25 [44.09, 67.71] |
| Bleached coral recognition | 39.33 [33.97, 44.96] | 59.00 [53.35, 64.42] |
| Genus-level coral recognition | 42.34 [36.03, 48.92] | 61.71 [55.17, 67.86] |
| Coral richness assessment | 19.75 [12.54, 29.70] | 55.56 [44.73, 65.88] |
| Shannon diversity index | 23.29 [15.08, 34.17] | 52.05 [40.78, 63.12] |
| Simpson diversity index | 19.18 [11.78, 29.66] | 61.64 [50.17, 71.95] |
On all seven tasks, the 95% confidence intervals of CoralAgent and GPT-5.4 do not overlap: CoralAgent’s lower bound exceeds GPT-5.4’s upper bound in every case, including tasks with as few as 64 questions. Non-overlapping 95% intervals imply that the differences are statistically significant.
Q5: Choice of evaluation metric
Why is accuracy the primary evaluation metric? Could you provide additional metrics?
We thank the reviewer for this question.
Why accuracy. Coral-Bench covers 23 heterogeneous tasks. Their answers include taxonomic labels, numerical values, site or year labels, directional judgments and percentage-point differences. Casting all tasks in a multiple-choice format, and scoring them by accuracy, provides a single metric that is comparable across tasks and applies equally to all methods, including VLMs that cannot reliably produce well-formed numerical outputs. For quantitative tasks, the options were designed by marine scientists around the reference values, so that selecting the correct option requires an estimate within an ecologically meaningful range.
Additional metrics. For quantitative tasks, we additionally compare the numerical estimates with the expert-derived reference values. CoralAgent consistently reduces numerical error relative to GPT-5.4:
Table Q5: Coverage and numerical error (MAE/RMSE) of CoralAgent and GPT-5.4 relative to expert-derived reference values.
| Task | CoralAgent MAE | CoralAgent RMSE | GPT MAE | GPT RMSE |
|---|---|---|---|---|
| Simpson diversity index | 0.152 | 0.209 | 0.195 | 0.245 |
| Pielou’s evenness index | 0.160 | 0.242 | 0.282 | 0.361 |
| Bleached coral cover estimation | 0.117 | 0.153 | 0.140 | 0.176 |
| Coral richness assessment | 0.650 | 0.975 | 1.741 | 2.169 |
CoralAgent reduces MAE by 16–63% across the four tasks, with the largest reductions on coral richness (1.741 → 0.650 genera, i.e., an average error below one genus) and Pielou’s evenness (0.282 → 0.160). In the revised version, we will report MAE and RMSE for all quantitative tasks and macro-F1 for recognition tasks.
W1: Reliability and response consistency of the evaluation
Response:
We thank the reviewer for raising the importance of reliability and response consistency, and we agree that a single correct multiple-choice answer does not by itself demonstrate reliable understanding. We address this in three ways.
First, we added analyses in our responses below: repeated independent runs (Q1), option-order robustness (Q2), an image-blind baseline (Q3), confidence intervals (Q4) and numerical error metrics (Q5).
Second, CoralAgent’s answers do not rely on option selection alone. Each answer is derived from recorded intermediate evidence (segmentation masks, point-level labels and the resulting computation), so its correctness can be inspected beyond the final choice. Its numerical estimates also show lower error than GPT-5.4 (Q5).
Third, regarding the one-image, one-question setup, many quantitative questions require multiple dependent steps on the same image (e.g., identifying genera, computing composition, then deriving a diversity index), so a correct answer requires consistent analysis across these steps rather than a single guess.
In the revised version, we will report Pass@K and paraphrase consistency for all tasks, repeated-run results for the baselines, and discuss the works the reviewer cited [1,2,3].
Q1: Stability across repeated runs
Response:
We thank the reviewer for this question. We evaluated CoralAgent three times on the same image–question pairs for four tasks:
Table Q1: Accuracy (%) of CoralAgent across three independent runs. Mean ± SD is computed over the three runs.
The results are stable: across runs, accuracy varies by at most one question per task (SD ≤ 0.90). The conclusions also remain consistent, since even the lowest of the three runs exceeds the best baseline on every task by 10.95 to 25.00 points. We will report repeated-run results for all tasks in the revised version.