acceptodds
Under review as a conference paper at ICLR 2027

Focus on What Really Matters in Low-Altitude Governance: A Management-Centric Multi-Modal Benchmark with Implicitly Coordinated Vision-Language Reasoning Framework

Abstract

Low-altitude vision systems are becoming a critical infrastructure for smart city governance. However, existing object-centric perception paradigms and loosely coupled vision–language pipelines are still difficult to support management-oriented anomaly understanding required in real-world urban governance. To bridge this gap, we introduce GovLA-10K, the first management-oriented multi-modal benchmark for low-altitude intelligence, along with GovLA-Reasoner, a unified vision-language reasoning framework tailored for governance-aware aerial perception. Unlike existing studies that aim to exhaustively annotate all visible objects, GovLA-10K is deliberately designed around functionally salient targets that directly correspond to practical management needs, and further provides actionable management suggestions grounded in these observations. To effectively coordinate the fine-grained visual grounding with high-level contextual language reasoning, GovLA-Reasoner introduces an efficient Spatially-aware Grounding Adapter (SGA) that implicitly coordinates discriminative representation sharing between the visual detector and the large language model (LLM). Different from existing adapters that primarily focus on global embedding alignment, our SGA is specifically designed to compress and aggregate multi-stream grounding-aware representations, thereby preserving fine-grained spatial cues while enabling their effective integration into the language reasoning process. Extensive experiments indicate that our GovLA-Reasoner effectively improves performance while avoiding the need of fine-tuning for any task-specific individual components. We believe our work offers a new perspective and foundation for future studies on management-aware low-altitude vision–language systems.

Primary area:
datasets and benchmarks
Keywords:
Low-altitude Intelligence, Management-oriented Benchmark, Vision–language Reasoning, Adapter Fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fWAkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GovLA-10K, a management-oriented aerial benchmark containing 10,572 images, bounding-box annotations for nine categories, and captions comprising scene descriptions and management recommendations. It also proposes GovLA-Reasoner, which connects a task-adapted grounding detector with a frozen language model through a Spatially-aware Grounding Adapter (SGA). SGA compresses and integrates image, query, and decoder features to support caption generation. Experiments on GovLA-10K include captioning comparisons, expert assessment, adapter efficiency measurements, and architectural ablations.

Critical strengths
  1. The benchmark addresses a practically relevant problem. GovLA-10K combines aerial localization, contextual descriptions, and management recommendations, extending evaluation beyond conventional object detection. Expert review of the generated annotations is a useful quality-control step.
  2. The framework is coherent and supported by initial experimental evidence. Tables 2–4 examine adapter cost, fusion depth, and feature-stream contributions, while Table 1 supplements automatic metrics with expert assessment. Freezing the detector and LLM during Stage II provides a practical adaptation strategy.
Critical weaknesses
  1. The evaluation does not adequately establish the reliability of management-risk assessment. BLEU, METEOR, ROUGE-L, and CIDEr measure agreement with reference captions, while the expert evaluation in Appendix C does not specify a detailed rubric separating factual grounding, risk-identification accuracy, and recommendation appropriateness. Moreover, Appendix A lists construction workers, fencing, and scaffolding as “violation categories” without explaining how compliant or ambiguous cases are distinguished, leaving open whether the benchmark rewards category-triggered recommendations rather than contextual risk reasoning.

  2. The adapter comparisons are too limited to establish SGA’s architectural advantage. Table 2 compares SGA only with LLaVA-Projector (2023) and Q-Former (2023), without evaluating more recent, task-relevant adapters or token-resampling mechanisms. Since these baselines are also marked as lacking multi-stream integration, the experiment does not clearly separate the benefit of SGA’s design from access to additional grounding features.

  3. The claimed benefit of implicit coordination is not directly validated. Although Figure 4 contrasts SGA with an explicit detector–VLM pipeline, the experiments do not identify a controlled comparison using the same detector predictions, images, and structured prompts. Tables 3–4 provide quantitative ablations, but Appendix E’s full-model qualitative comparison does not show how SGA’s compression and fusion preserve spatial evidence or reduce the claimed information loss and error propagation.

Questions for authors
  1. What instructions and scoring criteria were provided to the three experts? Did they separately assess unsupported risk allegations, missed management concerns, and inappropriate recommendations, or primarily evaluate overall caption quality?

  2. Does GovLA-10K contain compliant scenes involving the same object categories, and how are visually uncertain cases annotated? Please explain how the annotation process avoids equating category presence with a confirmed violation.

  3. Do the Table 2 baselines receive the same three feature streams as SGA? Please clarify how input information, output token counts, and training settings are controlled when attributing the gains to the adapter architecture.

  4. Is any reported baseline the explicit detector–VLM pipeline illustrated in Figure 4, using the same detector predictions? If so, please identify its results and explain what existing evidence supports reduced error propagation.

2 comments
AuthorsOpenReview ↗

Response to Reviewer fWAk

Appreciation

We sincerely thank Reviewer fWAk for the constructive comments on evaluation reliability, adapter comparison fairness, and coordination mechanisms, as well as the valuable suggestions for improving the rigor of our experimental validation. Our point-by-point responses are provided below.

R1: Clarification on Shared Reviewer Concerns

Please refer to our common response titled "Clarifications on Common Concerns Raised by All Three Reviewers" at the top of this page, where we provide detailed clarifications on VLM comparisons and coordination mechanisms, adapter comparison fairness, expert evaluation reliability, and annotation quality and data independence.

R2: Clarification on Output Token Consistency

We further clarify that the output token counts of all adapters are strictly controlled to ensure a fair architectural comparison. Specifically, all adapters receive the same three feature streams and produce an identical number of visual tokens, corresponding to the spatial resolution of the original 512 × 512 image downsampled by a factor of 64, i.e., (512/64) × (512/64) = 64 tokens. This standardized output configuration eliminates token-count differences as a confounding factor and allows the comparison to focus on each adapter's ability to integrate heterogeneous grounding features. Moreover, since the explicit detector–VLM pipeline directly uses the original image as visual input, this design maximizes the consistency of the visual representation budget between explicit and implicit coordination frameworks, facilitating a fair comparison of their effectiveness.

R3: Clarification on Visual Evidence and Error Propagation

Furthermore, the qualitative comparison in Appendix E provides visual evidence of the limitations of explicit coordination. Specifically, although the explicit detector–VLM pipeline accurately identifies and specifies the spatial locations of regions of interest, the VLM still introduces descriptions and considerations of moving vehicles that are not directly relevant to the primary governance concern in the illustrated scene. This suggests that explicitly providing grounding information does not necessarily guarantee that the subsequent language generation remains focused on the intended regions, potentially introducing irrelevant information during cross-stage coordination. In contrast, our implicit feature integration enables more focused interpretation of governance-relevant visual evidence, as illustrated by the corresponding results. These observations qualitatively support our motivation to mitigate information distortion and error propagation through implicit coordination.

R4: Clarification on Violation-Oriented Data Selection

We emphasize that GovLA-10K specifically targets scenes containing identifiable governance violations, rather than arbitrary images containing governance-related objects. During the first annotation stage, we explicitly screen the collected images and retain only those exhibiting visually verifiable violations. Images containing merely relevant objects without sufficient evidence of violations, as well as ambiguous cases, are excluded. Therefore, the presence of an object category is never treated as sufficient evidence of a violation. This selection process ensures that the benchmark focuses on identifiable violation scenarios, supporting the evaluation of scene descriptions and contextually appropriate management recommendations. Accordingly, GovLA-10K primarily addresses violation-oriented interpretation and recommendation generation rather than binary compliance classification.

Expectation

We will incorporate all the above clarifications and implementation details into the revised manuscript to improve the clarity and completeness of our work. We sincerely hope that our responses have adequately addressed your concerns, and we would greatly appreciate it if you would consider raising your evaluation score accordingly. Thank you again for your valuable feedback and consideration.

Thank you very much for the author's response. However, based on the author's current reply, I still find it difficult to determine whether the model truly adjusts its risk assessment and disposal recommendations based on specific visual evidence. Therefore, I will maintain my original rating.

Reviewer_AhD2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies management-oriented understanding of low-altitude UAV imagery. It introduces GovLA-10K, a benchmark containing 10,572 images across nine categories relevant to urban governance, with bounding box annotations, scene descriptions, and management recommendations. The annotation pipeline combines manual localization, detector-assisted verification, caption generation conditioned on regulations, and expert review. The paper also proposes GovLA-Reasoner, which connects a grounding model to a frozen Qwen3-4B language model through a Spatially-aware Grounding Adapter (SGA). After training the detector for the target task, the captioning stage updates only SGA. The adapter compresses query and decoder features, integrates them with image features, and projects the resulting representations into the language model. Validation includes comparisons with detectors and VLMs, adapter efficiency comparisons, component ablations, and an evaluation of 100 test examples by three domain experts. The reported results favor GovLA-Reasoner on automatic captioning metrics and expert ratings. SGA also achieves a better balance between performance and computational cost than the evaluated Q-Former configuration at the adapter level.

Critical strengths

The dataset provides a useful connection between visual localization and language supervision for urban management. It goes beyond conventional object annotations by pairing localized visual evidence with scene descriptions and management recommendations. Its annotation workflow includes manual bounding boxes, detector-assisted reinspection, caption generation informed by regulations, and expert review. This is a substantive resource contribution because it provides a concrete basis for studying how aerial perception can support actionable scene understanding, rather than evaluating object recognition alone.

SGA offers a promising balance between captioning performance and adapter efficiency. In Table 2, SGA achieves a CIDEr-D score of 20.31 compared with 19.62 for Q-Former, while requiring fewer parameters and FLOPs: 17.582M parameters and 36.002G FLOPs versus 21.257M parameters and 43.520G FLOPs. Table 4 additionally shows that removing any of the three feature streams reduces the reported captioning performance. These results support the usefulness of combining grounding representations with global visual context and make the framework a promising baseline for adapting a frozen language model to this task. However, the efficiency evidence applies to the adapter rather than the complete system.

Critical weaknesses

The benchmark does not yet convincingly establish that it measures management reasoning rather than recommendation generation conditioned on object categories. Although the dataset is described as selecting abnormal or risky targets, Appendix A treats all nine categories, including construction workers and scaffolding, as “violation categories” that trigger management analysis. The presence of these objects alone does not establish an unsafe or noncompliant situation. The paper does not sufficiently specify how benign and problematic instances of the same category are distinguished, nor does it report an evaluation specifically testing this distinction. Furthermore, the reference captions originate from a single VLM followed by expert review, and the reported human evaluation does not provide a detailed rubric or separate measurements of unsupported risk claims and inappropriate recommendations. The existing results therefore do not rule out the possibility that models primarily learn category-specific advice and reference caption conventions. This is consequential because the central benchmark contribution depends on measuring management understanding grounded in the particular scene, rather than merely generating plausible advice for recognized object categories.

The experimental controls are insufficient to attribute the gains to the proposed implicit coordination mechanism. Appendix C states that the models are evaluated under consistent settings, but does not adequately specify how each VLM baseline is adapted to GovLA-10K or whether it receives equivalent grounding and regulatory information. More importantly, the paper does not clearly report a matched comparison against the explicit pipeline that converts detector outputs into VLM prompts, despite this pipeline being central to the motivation. The adapter comparison also leaves input equivalence unclear: Table 2 marks Q-Former as lacking integration of multiple feature streams, while SGA receives three streams. Consequently, the relatively small improvement over Q-Former cannot be uniquely attributed to the proposed spatially aware design. Without matched inputs, supervision, and a direct assessment under the same detector failures, the results establish the usefulness of the overall configuration more clearly than they establish the claimed reduction in information loss and error accumulation. This weakens the evidence for the central methodological contribution.

Questions for authors

Clarification of benchmark labels. What operational annotation criteria distinguish a construction worker, scaffold, or material pile requiring intervention from a benign instance of the same category? Does GovLA-10K contain such benign instances, and how are they annotated and evaluated? Please clarify whether the benchmark explicitly measures false alarms or unnecessary management recommendations.

Clarification of baseline supervision and inputs. Were the VLM baselines in Table 1 trained on the same GovLA-10K caption data, and which parameters were updated for each model? Did they receive the same category instructions, detector outputs, and regulatory information? Does any reported baseline implement the explicit pipeline in Figure 4 using the same Stage I detector as GovLA-Reasoner?

Clarification and validation of spatial awareness. In Equations (1) to (3), the query and decoder streams are compressed using separately parameterized learnable tokens and then concatenated with image features along the channel dimension. How is spatial correspondence between these token sequences established or preserved? Does SGA retain an advantage over a Q-Former variant given the same three feature streams and comparable token and parameter budgets?

Clarification of independence between training and testing data. Were crops from the same original image, flight sequence, or physical site assigned to the same split? Please also clarify whether the detector used for annotation verification was kept separate from the final evaluation model, and whether the final Stage I detector was trained exclusively on the training split.

Clarification of expert evaluation. What are the precise scoring criteria for SD, MR, and Overall Score, and is Overall Score independently assessed or derived from the other scores? Were model identities hidden from the experts, and were recommendations checked against the original visual evidence and applicable regulations? Please report agreement among the evaluators and uncertainty estimates for the differences between methods.

Validation of the claim about error accumulation. Can the authors provide a controlled comparison between implicit feature integration and explicit prompting under the same detector failures, such as missed objects and false positives? In particular, can the evaluation separate improvements due to stronger grounding information from improvements specifically due to avoiding prompt conversion and reinterpretation?

I am open to revising my score if the authors can provide satisfactory clarifications that address the concerns raised above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer AhD2

Appreciation

We sincerely thank reviewer AhD2 for the constructive comments on benchmark validity, experimental fairness, spatial awareness, annotation quality, and evaluation reliability, as well as the willingness to reconsider the assessment based on our clarifications. Our point-by-point responses are provided below.

R1: Clarification on Shared Reviewer Concerns

Please refer to our common response titled "Clarifications on Common Concerns Raised by All Three Reviewers" at the top of this page, where we provide detailed clarifications on VLM comparisons and coordination mechanisms, adapter comparison fairness, expert evaluation reliability, and annotation quality and data independence.

R2:Clarification on Detector Utilization

Although both the annotation verification and final evaluation employ MMGroundingDINO, the detectors used in these two stages are independent. For both the explicit and implicit coordination pipelines illustrated in Fig. 4, the Stage I detectors are trained exclusively on the training set, without accessing any test data. This ensures the independence of annotation verification and model evaluation, preventing data leakage during training and evaluation.

R3: Clarification on Violation-Oriented Data Selection

We emphasize that GovLA-10K specifically targets scenes containing identifiable governance violations, rather than arbitrary images containing governance-related objects. During the first annotation stage, we explicitly screen the collected images and retain only those exhibiting visually verifiable violations. Images containing merely relevant objects without sufficient evidence of violations, as well as ambiguous cases, are excluded. Therefore, the presence of an object category is never treated as sufficient evidence of a violation. This selection process ensures that the benchmark focuses on identifiable violation scenarios, supporting the evaluation of scene descriptions and contextually appropriate management recommendations. Accordingly, GovLA-10K primarily addresses violation-oriented interpretation and recommendation generation rather than binary compliance classification.

R4: Clarification on Spatial Awareness and Output Token Consistency

We further clarify that all adapters receive the same three feature streams and produce an identical number of output visual tokens under consistent training configurations. Specifically, the output token count is fixed according to the spatial resolution of the original 512 × 512 image downsampled by a factor of 64, i.e., (512/64) × (512/64) = 64 tokens. This standardized configuration eliminates differences in token counts as a confounding factor, allowing the comparison to focus on each adapter's ability to integrate heterogeneous grounding representations. In SGA, the separately parameterized compression operations adaptively extract complementary information from the query and decoder streams, which is subsequently integrated with spatial image features. This design aims to preserve spatially relevant grounding information while reducing redundant representations. The direct comparisons in Table 2 and the component ablations in Tables 3 and 4 jointly support the effectiveness of this architecture.

R5: Clarification on Visual Evidence and Error Propagation

The qualitative comparison in Appendix E provides visual evidence of the limitations of explicit coordination. Specifically, although the explicit detector–VLM pipeline accurately identifies and specifies the spatial locations of regions of interest, the VLM still introduces descriptions and considerations of moving vehicles that are not directly relevant to the primary governance concern in the illustrated scene. This suggests that explicitly providing grounding information does not necessarily guarantee that subsequent language generation remains focused on the intended regions, potentially introducing irrelevant information during cross-stage coordination. In contrast, our implicit feature integration enables more focused interpretation of governance-relevant visual evidence, as illustrated by the corresponding results. These observations qualitatively support our motivation to mitigate information distortion and error propagation through implicit coordination.

Expectation

We will incorporate all the above clarifications and implementation details into the revised manuscript to improve the clarity and completeness of our work. We sincerely hope that our responses have adequately addressed your concerns, and we would greatly appreciate it if you would consider raising your evaluation score accordingly. Thank you again for your valuable feedback and consideration.

Thanks for the detailed response, I will get back to you soon.

by the way, could you change the visibility into eveyone so that there may be more constructive suggestions.

AuthorsOpenReview ↗

Response to Reviewer AhD2

Thank you for your kind response.

We have updated the visibility of all our responses to the three reviewers to Everyone to facilitate further discussion and constructive suggestions.

Your valuable feedback is greatly appreciated and will help us further improve the quality of our manuscript. We look forward to your further comments and sincerely hope that you will consider raising your score based on our clarifications and revisions.

Thank you again for your valuable time and consideration.

Reviewer_XaVVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work combines a benchmark and a methodological framework. GovLA-10K is a management-oriented perception dataset for low-altitude flight, consisting of 10572 UAV images distributed across 9 categories of functional significance, with captions that combine scene description and management recommendations. The bounding boxes are annotated manually, and cases where the agreement with the detector is below an IoU of 0.5 undergo human review. The captions are generated by Qwen3VL-235B using prompts that incorporate regulatory constraints, and are then validated by experts. GovLA-Reasoner feeds multi-stream grounding features into a frozen 4B LLM via a learnable adapter (SGA), which itself is trained only with cross-entropy on the captions. On the test partition of GovLA-10K, a substantial improvement in caption quality over 9 publicly available VLMs is reported, along with a comparison of adapter costs and an ablation of SGA.

Critical strengths

The shift from exhaustive annotations to functionally important and regulatorily relevant targets fills a practical gap between generic aerial-photo datasets and the needs of urban governance. A two-stage process was chosen for this purpose: manual annotation of bounding boxes with a check of disagreements with the detector, then caption generation with prompts under regulatory constraints, where scene description and management recommendation are separated. Even if the results of the methodological framework were weak, the dataset part would remain useful to the community. The cost reporting is also transparent: Table 2 shows parameter counts, FLOPs, and peak memory compared with the Q-Former and the LLaVA projector. During training, the detector is trained in Stage I, then the detector and LLM are frozen during adapter training in Stage II; the depth ablation in Table 3 and the feature-stream ablation in Table 4 separately validate the choices of adapter depth and multi-stream design.

Critical weaknesses

1.The main comparison does not separate adapter design from domain adaptation. SGA is trained on the 7,500 captions from GovLA-10K for 25 epochs, but it is unknown whether the 9 VLMs in Table 1 received comparable domain adaptation. Appendix C merely mentions "consistent settings". If they were evaluated without prior training, the comparison measures domain adaptation rather than adapter design. Table 2 offers a useful adapter comparison, but access to feature streams is not matched and the training budget is not documented as matched. The explicit pipeline that the paper criticizes in Sections 1.2 and 3.2, in which bounding boxes are serialized in the prompt, appears only in the dataset's annotation step and is not evaluated as a matched baseline in the experiments.

2.The evaluation is partly circular. The reference captions are generated by Qwen3VL-235B, and BLEU/CIDEr reward precisely agreement with that distribution; a model that has learned to imitate the reference generator can thus be favored, regardless of the factual correctness and governance relevance of its output. The manual evaluation (100 randomly drawn test samples, 3 domain experts, a scale from 1 to 10) provides a supplement of evidence, but the detailed evaluation criteria, blinding, agreement rates, and uncertainties are not reported, and neither factual correctness nor well-grounded validity of the recommendations is measured.

Questions for authors

1.Please clarify how each baseline in Table 1 is implemented. Is the input the original image or a grounding input based on the detector, and is this a zero-shot evaluation or after additional training? In the zero-shot case, there should at least be a strong fine-tuned baseline with a training budget matched on the same captions.

2.Please add the ablation that the motivation suggests: compare implicit conditioning via SGA with an explicit pipeline built on the same detector and a comparable VLM architecture, with the same training and the same evaluation.

3.What proportion of the generated captions required substantial manual corrections, and what is the inter-annotator agreement rate on the reviewed subset? Also, what is the status of the licenses for the collected images, and has overlap in crops been examined to detect leakage between training and test?

4.Compared with aerial-photo datasets that already annotate violations or anomalies: which tasks does GovLA-10K make it possible to solve that these do not?

1 comment
AuthorsOpenReview ↗

Response to Reviewer XaVV

Appreciation

We sincerely thank Reviewer XaVV for the constructive comments on experimental fairness, evaluation reliability, and dataset quality, as well as the recognition of GovLA-10K's practical significance. Our point-by-point responses are provided below.

R1: Clarification on Shared Reviewer Concerns

Please refer to our common response titled "Clarifications on Common Concerns Raised by All Three Reviewers" at the top of this page, where we provide detailed clarifications on VLM comparisons and coordination mechanisms, adapter comparison fairness, expert evaluation reliability, and annotation quality and data independence.

R2: Clarification on the Unique Value of GovLA-10K

Existing studies, such as AerialWaste, primarily focus on waste detection, while A2Seek emphasizes anomaly localization and explanation. In contrast, to the best of our knowledge, GovLA-10K is the first benchmark to explicitly investigate visually grounded management decision support in low-altitude governance scenarios, going beyond conventional anomaly understanding by associating governance-relevant violations with detailed scene descriptions and actionable management recommendations across nine categories. For example, rather than merely identifying an overflowing trash bin, GovLA-10K enables models to connect visible waste accumulation with sanitation concerns and appropriate management actions. These annotations provide direct supervision for learning the relationship between visual evidence and management decisions, thereby extending aerial anomaly understanding toward visually grounded management decision support.

Expectation

We will incorporate all the above clarifications and implementation details into the revised manuscript to improve the clarity and completeness of our work. We sincerely hope that our responses have adequately addressed your concerns, and we would greatly appreciate it if you would consider raising your evaluation score accordingly. Thank you again for your valuable feedback and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Clarifications on Common Concerns Raised by All Three Reviewers

R1: Clarification on VLM Comparisons and Coordination Mechanisms

We agree that directly evaluating VLMs on original images in a zero-shot manner would be unfair, as these models lack task-specific adaptation to our governance scenarios. Therefore, all nine VLM baselines in Table 1 follow the explicit coordination pipeline illustrated in the left panel of Fig. 4. Specifically, we employ the same task-adapted detector to accurately localize regions of interest, and then provide its detection results together with the original image to each VLM for caption generation. This ensures consistent grounding information and substantially mitigates the domain adaptation gap. Importantly, the LLM components of all methods, including ours, remain frozen. The primary distinction thus lies in the coordination mechanism: explicit prompting versus our implicit feature integration. Notably, GovLA-Reasoner achieves superior performance using only a 4B LLM, outperforming several VLMs with up to 8B parameters. Therefore, Table 1 provides a meaningful framework-level comparison between explicit and implicit coordination under consistent grounding conditions.

R2: Clarification on the Fairness of Adapter Comparisons

All adapters in Table 2 receive the same three feature streams and follow identical training configurations, ensuring a fair comparison. The checkmarks and crosses indicate whether each adapter is specifically designed for heterogeneous multi-stream integration, rather than whether it receives these features. Existing adapters primarily employ general-purpose feature projection or aggregation mechanisms, whereas our SGA explicitly considers the complementary characteristics of different grounding representations through dedicated compression and integration designs. Our primary objective is precisely to demonstrate the effectiveness of this task-oriented multi-stream architecture. The direct comparisons in Table 2, together with the component and feature-stream ablations in Tables 3 and 4, consistently validate this design.

R3: Clarification on the Reliability of Expert Evaluation

We agree that conventional captioning metrics cannot fully capture factual accuracy or governance relevance. Therefore, we additionally conduct a blinded evaluation involving three domain experts and 100 randomly selected test samples. Each expert independently evaluates anonymized captions against the original images and reference descriptions using three criteria on a 10-point scale:

ScoreScene Description(SD)Management Recommendation(MR)Overall Quality
9–10Accurate and complete, without factual errorsAccurate violation assessment and well-grounded, actionable recommendationsExcellent accuracy, coherence, and practical value
7–8Accurate overall, with minor omissionsAppropriate recommendations with minor limitationsGood quality with minor deficiencies
5–6Generally correct, but with noticeable errors or omissionsPartially appropriate but insufficiently specificAcceptable but requires revision
3–4Multiple factual errors or missing key informationIncorrect risk assessment or inappropriate recommendationsSubstantial errors and limited usefulness
1–2Severely inconsistent with visual evidenceSerious misjudgment or irrelevant management actionsUnreliable and practically unusable

SD assesses factual consistency in object attributes, quantities, and spatial relationships. MR evaluates the correctness of violation interpretation and the appropriateness of management actions, including unsupported claims and missed risks. Overall Quality is independently scored by each expert rather than calculated from SD and MR, considering accuracy, completeness, coherence, and practical usefulness. This protocol complements automatic metrics with direct assessments of factual grounding and governance relevance.

R4: Clarification on Annotation Quality and Data Independence

Regarding annotation quality, approximately 10% of generated captions required substantial revisions, while another 25% underwent minor adjustments, based on available records. Exact statistics are unavailable because some intermediate records were not retained due to computational and storage constraints. Rather than independently rewriting captions, annotators followed a standardized verification protocol using predefined templates, systematically correcting object counts, appearances, spatial distributions, and other factual attributes. This process helps maintain annotation accuracy and descriptive consistency. Since annotation primarily involved verification and correction rather than independent caption generation, conventional inter-annotator agreement statistics are not directly applicable. Image crops are non-overlapping, preventing direct crop-level overlap between training and test sets. We will release the dataset under an appropriate license consistent with the original image sources' usage requirements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.