- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a structured semantic evidence framework for uncertainty management in American Sign Language (ASL) communication. Instead of relying on sentence-level recognition confidence, the method uses Top-K recognition hypotheses and dialogue context to predict whether individual semantic slots are determined, uncertain, missing, or not required. The authors introduce a slot-specific cross-attention model and a human-annotated benchmark for evaluating semantic evidence and communication safety. On unseen signers, the proposed model achieves 0.660 evidence-state macro-F1 and 0.927 repair AUROC, outperforming several input ablations and a zero-shot GPT-5.5 baseline. Additional experiments examine recognition quality, contextual information, and downstream repair decisions.
Critical strengths
-
A practical formulation of uncertainty beyond sentence-level confidence. The paper addresses an important problem: a recognition system may be confident in its prediction while missing meaning-critical information such as negation or time. Representing uncertainty at the semantic-slot level is useful because it identifies what information is unresolved rather than simply flagging an entire utterance as unreliable.
-
Evidence that recognition hypotheses and dialogue context provide complementary information. The ablation results show that combining recognition hypotheses with dialogue context improves evidence-state macro-F1 from 0.502 and 0.521 to 0.660. The signer-disjoint evaluation also provides some evidence that the method generalizes beyond the training signers.
Critical weaknesses
-
The paper does not demonstrate that structured semantic evidence improves downstream communication. The central motivation is to make ASL communication safer by identifying and repairing uncertain meanings. However, the main results only demonstrate improvements in slot-level classification. English meaning recovery is evaluated separately using GPT-5.5, and the proposed structured representation is not shown to improve translation or communication safety over simpler alternatives. More importantly, the repair policy achieves only 7.35% agreement with human actions, and no actual post-repair recovery is evaluated. As a result, the paper does not establish whether its central contribution provides a meaningful benefit beyond predicting semantic-slot labels.
-
Limited methodological novelty and insufficient comparison with closely related approaches. The proposed architecture largely combines established components: slot-specific attention from dialogue state tracking (Yang et al., 2023), semantic uncertainty over multiple hypotheses (Kuhn et al., 2023; Farquhar et al., 2024), and supervised classification of semantic evidence states. While their combination is reasonable, it is unclear what fundamentally new modeling capability is introduced. The baselines mainly remove input components or compare against zero-shot GPT-5.5. Without comparisons to simpler supervised slot classifiers or existing structured uncertainty methods, the experiments do not establish whether the proposed cross-attention architecture is necessary.
References mentioned
- Yang et al. (2023). Dialogue State Tracking with Sparse Local Slot Attention. NLP4ConvAI.
- Kuhn et al. (2023). Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation.
- Farquhar et al. (2024). Detecting Hallucinations in Large Language Models Using Semantic Entropy. Nature.
Questions for authors
-
Can the authors demonstrate that the proposed structured semantic evidence improves downstream release or repair decisions compared with simpler slot-level confidence estimates? This would directly test the paper's main motivation.
-
How does the proposed model compare against a standard supervised slot classifier using the same Top-K hypotheses and dialogue context, but without the proposed slot-specific cross-attention mechanism? This comparison is important for establishing the architectural contribution.
2 comments
Dear Reviewer,
Thank you again for your thoughtful review.
As the author–reviewer discussion period is drawing to a close and the deadline is only a few hours away, we would greatly appreciate your further feedback regarding whether our rebuttal has adequately addressed your concerns.
If there are still any remaining issues or questions, please give us a chance to clarify them before the discussion period ends.
Thanks a lot for your valuable time and hard work.
Rebuttal by Authors
We appreciate the reviewer's focused questions regarding downstream benefit and the necessity of slot-specific cross-attention.
1. Downstream benefit over simpler confidence estimates. The complete structured system clearly outperforms a scalar-confidence policy. To isolate the slots themselves, we also compared it with an otherwise identical system that has the same Top-K and dialogue context but no slots; the structured condition passes only per-slot metadata, no slot content. 95% cluster-bootstrap CIs over reference sentences.
2. Matched supervised classifier without cross-attention. We added the requested supervised baseline. It receives exactly the same Top-K glosses, probabilities, partner context, and slot identity, but replaces slot-specific cross-attention with probability-weighted mean pooling and a shared MLP.
All five seed-wise differences are positive, isolating the architecture from input information. *Ours and shared pooling are retrained in one pipeline (5 seeds); Table 1 reports .660 ± .028 (ablations −.139 / −.158) from the original GPU run.
3. Novelty: semantic evidence sufficiency under uncertain perception.
Beyond prediction uncertainty. Uncertainty estimation measures disagreement among predictions (Kuhn et al., 2023; Farquhar et al., 2024), but agreement does not imply semantic completeness. A signer may mean "I do not want to cancel," while every Top-K hypothesis reads WANT CANCEL APPOINTMENT: semantic entropy is low, yet the negation is lost. We model not only what the recognizer predicts, but whether its predictions sufficiently support the information the turn requires.
Formulation. For each slot, the model jointly decides (i) whether the interaction requires it and (ii) whether the hypotheses support it: Determined, Uncertain (competing evidence), Missing (required but unsupported, even when hypotheses agree), or NotRequired. Unlike slot tracking (Yang et al., 2023), we assess the sufficiency of imperfect perceptual evidence rather than extract values from clean text.
Architecture. Slot-specific queries attend separately to the hypotheses and the context, with gated fusion, so each slot compares what the interaction requires against what the recognizer supports (shared pooling instead: .681 → .578; removing hypotheses / context: −.130 / −.125 State F1).
Comparison and Ablation. Each state maps to an action: retain what is Determined, clarify what is Uncertain, request what is Missing, ignore what is NotRequired; #1 shows this improves release and targeted clarification.
The anonymized repository includes the matched baseline code, five-seed results, per-sample outputs, all downstream analyses, and verification artifacts.