acceptodds
Under review as a conference paper at ICLR 2027

PhonoNoise: Reshaping Diffusion Geometry via Phonology-Aware Non-isotropic Diffusion for Sign Language Production

Abstract

Sign Language Production (SLP) aims to generate semantically faithful and articulatorily coherent sign motion from spoken-language text. Since the hands carry much of a sign's lexical content, errors in handshape, inter-hand configuration, and movement directly impair intelligibility, yet existing methods still often produce distorted fingers, stretched hand bones, and inaccurate trajectories. We attribute this partly to the forward process of diffusion-based methods, which corrupts all joints with isotropic noise and thus leaves hand-joint dependencies to be learned implicitly from data. Motivated by sign phonology, where handshape and inter-hand arrangement describe spatial hand relations and movement describes how the hands travel over time, we propose a phonology-aware diffusion framework with two components. PhonoNoise derives a phonology-induced covariance from intra-hand topology and bilateral joint correspondence for non-isotropic hand diffusion. PhonoContext extracts movement context from noisy wrist trajectories and injects it into the denoiser. Neither requires phonological annotations. Experiments on PHOENIX-2014T and CSL-Daily show state-of-the-art back-translation performance, with BLEU-4 scores of 17.11 and 6.21, respectively, while substantially reducing limb stretching. Code and more visualizations will be released on our project page.

Primary area:
generative models
Keywords:
Sign Language Production, Human Motion Generation, Text to Motion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_drL1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a phonology-aware diffusion to better capture the correlations of hand motions in sign language production. Specifically, the authors highlight the issue of diffusion models to add independent noise to each body joint by introducing a PhonoNoise correlated noise in the hand region with covariance derived from the hand kinematic graph. The authors additionally introduce a PhonoContext component to model the temporal dimension of the hand wrist motions. The authors show significant performance improvements on several benchmarks.

Critical strengths
  • The paper is well written and the motivation is clear and sound.
  • The proposed approach achieves state-of-the-art results, outperforming recent baselines.
  • The proposed approach is the first one to introduce structured noise in SLP
  • The proposed approach is easy to be extended to any diffusion based SLP model.
Critical weaknesses

Novelty

One of the major concerns regarding this work is the novelty of the proposed PhonoNoise. Specifically, the graph derived covariance, that is the major contribution of this work, has been already introduced in SkeletonDiffusion (Curreli et al., 2025). PhonoContext, despite intuitive and well motivated, brings only minimal performance improvements and hence cannot be seen as a major contribution of this work.

Experiments

a) It is interesting why the authors focused their PhonoNoise only on the hand regions. A natural ablation would be to introduce a whole body (or upper body) kinematic graph which would capture also the relations between arm motions as well. Could the authors clarify this choice?

b) As it is common on SLP papers, a human user study would validate the quality of the generated motions and it is a notable limitation of the current work.

c) The paper does not include any video qualitative experiments. Given that the proposed work tackles a motion generation problem, a supplementary video would be required to quantify the performance improvements of the proposed framework and would validate the contributions of each component.

d) Could the authors clarify the contribution of each adjacency matrix in the covariance? Does the main benefit derive from the A_shape or the A_arr?

e) It is interesting that the BLUE scores are always better compared to the GT and it raises concerns about the quality of the evaluators.

f) It is interesting the baseline model in Tab 3 can achieve sota performance outperforming all previous works. Could the authors clarify if the baseline model contains any specific adjustments? If not it seems that the adjustments of the proposed work compared to the MDM baseline introduce only minimal improvements. Could the authors clarify?

Phonology Concept (Minor)

I think the phonology concept is slightly misleading as it is not directly related to any phonological attributes.

Questions for authors

I would appreciate if the authors could clarify my concerns regarding the a) novelty b) the whole body kinematic graph and c) the low BLUE score of the GT sequences.

Reviewer_KtQfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses text-to-pose sign language production with an MDM-based diffusion model. It proposes two hand-focused components. PhonoNoise builds a covariance from intra-hand kinematic adjacency (“handshape”) and left–right homologous joint correspondence (“arrangement”), and uses it for non-isotropic forward diffusion on the hand joints, together with a covariance-whitened L1 reconstruction loss; body and face keep isotropic diffusion. PhonoContext extracts movement features from noisy wrist trajectories with a timestep-conditioned TCN and injects them into the denoiser through a zero-initialized gated residual. Neither component requires phonological annotations. The method is validated on PHOENIX-2014T and CSL-Daily using back-translation (BLEU-4 17.11 and 6.21), DTW-MJE, and hand bone-stretch metrics, with ablations on each component, the γ_t schedule, movement representation, and the hand loss.

Critical strengths
  1. Moving structural priors into the forward corruption process, rather than only into the architecture or conditioning, is a clean idea that is new for SLP. It requires no extra annotations, which makes it easy to adopt.
  2. PhonoNoise yields a large and consistent reduction in hand bone stretching (Str-Mean 9.07 → 5.38 on PHOENIX-2014T and 17.64 → 1.90 on CSL-Daily). Distorted hands are a well-known failure mode in SLP, so this is a meaningful practical improvement.
Critical weaknesses
  1. The SOTA claim is mostly explained by the baseline, and the reported numbers are inconsistent. The baseline without PN or PC already reaches B4 15.77 on PHOENIX-2014T, above the prior best (13.31) and GT (12.81), while PN+PC add only 1.34. Moreover, the “Baseline” row in Table 3 is identical to the “Euclidean MSE” row in Table 7, which the paper describes as the full model with non-isotropic corruption rather than the isotropic baseline. In addition, the “full model with beta-coupled γ_t” in Table 4 (B4 16.68) differs from the Full model in Table 3 (B4 17.11), which suggests that run-to-run variance is comparable to the PhonoContext gain. No seeds or standard deviations are reported. Together, these issues make it unclear how much of the improvement is attributable to the proposed components.
  2. The source of the gain is confounded, and the “phonology” framing is overstated. PhonoNoise changes both the forward noise and the loss. Table 7 shows that whitened MSE collapses (B4 6.68) and Euclidean L1 brings no gain, so the key ablation is missing: isotropic noise combined with a whitened L1 loss. Without it, the benefit cannot be attributed to the noise geometry itself. Furthermore, the “phonological” structure is hand kinematic adjacency plus mirror correspondence between hands. This is a modest extension of SkeletonDiffusion’s skeleton-derived covariance, and, as the authors acknowledge, it does not show that phonology itself is responsible. The mirror coupling is also questionable for asymmetric two-handed signs.
Questions for authors
  1. Can you provide an ablation with isotropic forward noise and the whitened L1 loss, to separate the effect of the noise geometry from the loss? And a variant using only intra-hand adjacency, without the bilateral correspondence, to test whether the “arrangement” term contributes?
  2. On CSL-Daily, were all baselines re-evaluated with your own Joint-SLT back-translator and the same MSRSLP lift3d data? If the numbers were taken from the original papers, they may not be comparable.
Reviewer_8usmOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper tackles text-to-3D sign language production, where standard isotropic diffusion models tend to produce distorted hand shapes and broken bilateral hand coordination because each joint receives independent noise. The authors propose PhonoNoise, a phonology-aware non-isotropic diffusion framework that embeds sign language phonological priors of hand topology and left-right hand homologous correspondence into the forward diffusion’s noise covariance matrix only for hand joints, paired with a PhonoContext module that extracts wrist temporal motion features and injects them into the denoiser via gated residuals; a Mahalanobis whitened L1 loss is further designed to match the non-isotropic noise distribution

Critical strengths

First, the paper presents a novel paradigm to inject sign phonology prior into the noise geometry of forward diffusion, rather than only changing network or loss designs. This new angle addresses the core issue of hand distortion in sign language production and brings valuable insight for motion diffusion research. Second, the method is validated on two cross-lingual sign language datasets with consistent performance gains, and thorough ablation experiments are provided to verify each design choice, proving the robustness of the proposed framework.

Critical weaknesses
  1. The comparison model data in the table is unreliable, and the comparison methods are all borrowed from other papers. The number of key points in these models is inconsistent, and the SLT models being evaluated are also different. The table is not credible. Among them, the PT, G2P-DDM, Sign-IDD, and FGDM models have already made their codes public, and the authors need to test using the same dataset and SLT methods.
  2. The proposed method performing much better than GT in SLT model evaluation is unreasonable. It is recommended to use the latest SLT models for evaluation.
  3. It's hard to see the advantages of the proposed method from Figure 1.
  4. How does PhonoContext obtain motion perception from noise? The initial noise should not contain any motion information. This needs an explanation from the authors.
Questions for authors
  1. The comparison model data is incorrect.
  2. The explanation of the PC model is insufficient and needs a more detailed explanation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.