Who Taught the Model to Sign? Fine-Grained Data Attribution for Optimizing Sign Language Production

Gloss-to-Pose (G2P) generation translates a discrete gloss sequence into a continuous sign language pose sequence, where each gloss represents a basic semantic unit. Recent generative models have improved overall G2P quality, yet a model may still generate poor poses for only a few glosses while producing the rest of the sequence well. However, existing methods mainly rely on sequence-level supervision, making it difficult to provide targeted supervision for an individual gloss. We study this problem from the perspective of training data. We refer to each occurrence of a gloss in a training sample as a training occurrence. Different occurrences of the same gloss may provide different supervision due to variations in sentence context and pose quality. This raises a key question: which training occurrences provide more useful supervision for generating a particular gloss, and how can this information be used to improve G2P training? To address this problem, we introduce SignTrak, an occurrence-level data attribution and optimization framework for G2P. First, to enable occurrence-level attribution under sequence-level supervision, SignTrak uses gradient sensitivity to allocate temporal generation losses to individual gloss occurrences, yielding occurrence-level losses without requiring explicit pose segmentation. Based on these losses, we apply gradient-based data attribution to estimate how each training occurrence influences each target occurrence of the same gloss. Second, to use the estimated occurrence-level influence to improve G2P training, we introduce attribution-guided optimization, which aggregates each training occurrence's influence over multiple target occurrences of the same gloss and converts the aggregated influence into its training weight for fine-tuning. Extensive experiments demonstrate improvements in both local pose generation and overall G2P performance.