acceptodds
Under review as a conference paper at ICLR 2027

SignDiT: Hand-Faithful Pose-Guided Video Generation for Sign Language

Abstract

Pose-guided video generation can produce plausible human motion while corrupting the small hand articulations that determine meaning in sign language. This is not merely a resolution problem: conditioning must allocate geometric information across sources, adapt their reliability over space and time, and preserve discrete linguistic contrasts. We introduce SignDiT, a two-stage framework for these decisions. Stage 1 assigns image-plane structure to a 2D keypoint path and projection-ambiguous surface orientation, relative depth, and occlusion order to a second path, then modulates both with a part-conditioned continuous reliability field. Stage 2, Sign-DPO, changes one phonological factor at a time to construct counterfactual preference pairs and localizes credit to the affected region and frames. In self-reenactment and cross-identity evaluation on How2Sign, CSL-Daily, and Phoenix, SignDiT is best or tied best on 32 of 36 fidelity measures, reduces cross-identity FID by 25.1–41.2%, comes within 1.83 BLEU-4 points of real-video back-translation, and exceeds the strongest generated baseline by 3.22–3.79 points. The resulting design treats pose control as structured inference rather than feature concatenation.

Primary area:
generative models
Keywords:
video generation, diffusion transformer, pose-guided animation, sign language, hand motion, preference optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WA3JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses pose-guided sign language video generation, where a reference image and a driving signing video are used to synthesize a new signer while preserving hand articulation. Built on Wan2.2-Animate-14B, SignDiT has two stages. Stage 1 replaces the pose path with two complementary conditioning streams: a 2D DWPose skeleton path for image-plane motion, and a “fiber” path with surface normals, root-relative depth, and occlusion order derived from SMPL-X fits. Both streams are modulated by a SIREN-based part × time reliability field driven by estimator diagnostics (keypoint confidence, fitting residual, motion). Stage 2 (Sign-DPO) builds counterfactual negatives by transplanting a single phonological factor (handshape, location, movement, or orientation) from ASL minimal pairs, filters pairs with a sign recognizer, and applies a factorized DPO loss localized to the edited region and frames. Experiments on How2Sign, CSL-Daily, and Phoenix report gains in FID/FVD, hand metrics, and back-translation BLEU-4, supported by a study with eight fluent ASL signers and component ablations on How2Sign.

Critical strengths
  1. The paper targets a real and important failure mode: generic human-animation models corrupt hand details that carry lexical meaning in sign language. Sign-DPO, which constructs phonologically controlled hard negatives and localizes credit to the edited region, is a well-motivated way to inject linguistic correctness into training, and it improves minimal-pair accuracy (0.63 → 0.79) and BLEU-4 in the ablations.
  2. The evaluation is broad for this area. It covers three sign languages, visual, hand-geometry, and semantic metrics, and an IRB-approved blinded study with fluent ASL signers, which is rare and valuable for sign language generation.
Critical weaknesses
  1. Limited practicality beyond video-driven retargeting. At inference, SignDiT requires 2D keypoints, SMPL-X fits, and estimator diagnostics extracted from a real driving video, so the evaluated task is signer retargeting or anonymization rather than sign language production from text. Using it in an end-to-end SLP pipeline would require an upstream text-to-pose model that generates accurate SMPL-X sequences, including hands. In that setting, the 2D keypoints would simply be projections of the same 3D source. The central argument that the two paths carry complementary, independently estimated signals would then no longer hold, and the reliability field, whose inputs are detector confidence and fitting residuals, would not be well defined. The word-library application also relies on retrieving pose clips from real videos. The paper does not test the method with generated or projected poses as input, which leaves its usefulness for SLP unclear. The cost is also high: a 14B backbone, per-frame DWPose and SMPL-X fitting, and about 2,500 H100 GPU-hours of training.
  2. Several key results may be inflated by the evaluation protocol. (a) Palm angular error and HH-Occ. are computed by refitting SMPL-X/MANO on generated videos, while SignDiT is explicitly conditioned on normals and occlusion order from the same fitting pipeline. The large margins on these metrics are therefore partly built into the design. (b) Sign-DPO pairs are constructed only from ASL, yet the BLEU-4 gain is much larger on Phoenix/DGS (+3.56) than on How2Sign/ASL (+1.42), and the generated Phoenix videos score above real video. This suggests the back-translation judges partly reward visual clarity or resolution rather than sign correctness. (c) Self-reenactment test signers appear in training. (d) Automatic metrics come from a single seed with no confidence intervals, and run-to-run variance is estimated from a single repeat (0.15 BLEU-4), which is comparable to some reported component gains (e.g., SIREN vs. no gate: +0.21 BLEU-4).
Questions for authors
  1. Can you evaluate SignDiT when driven by the output of a text-to-pose model, or at least with the 2D path computed from projected SMPL-X joints instead of DWPose? How does the reliability field operate when estimator confidence and fitting residuals are unavailable or trivial, as with generated poses?
  2. Can you report palm and occlusion metrics with an evaluator independent of the SMPL-X/MANO pipeline used for conditioning, and explain why ASL-only Sign-DPO yields a larger BLEU-4 gain on DGS than on ASL?
  3. The last question is also the one I am most concerned about. How do you view its application scenarios? Although your approach has increased the variety of output types, it is actually very difficult to use in practice. It is equivalent to a sign language re-renderer or Pose2Video. For instance, if the types of your input become very diverse, it will affect the real-time performance of the entire process.
3 comments
AuthorsOpenReview ↗

Response to Reviewer WA3J (Part 1/3)

Palm and occlusion metrics share the SMPL-X pipeline with our conditioning.

We agree that these metrics share estimators with our conditioning, so we re-evaluated every method on How2Sign with two tools that are not used to build the conditioning: WiLoR and ViTPose-H WholeBody. Each tool is run on both the generated and the real videos. WiLoR is a separate MANO-based hand reconstruction, so this tests whether the results hold under a different reconstruction pipeline.

  • Palm*: palm normals from WiLoR hand reconstructions.
  • HH-Occ.*: front hand from the WiLoR camera-space depth of the two hands, on frames where the hand boxes overlap in the real video.
  • H-PCK*: H-PCK has the same issue, because it used DWPose, our 2D estimator. We recomputed it with ViTPose-H WholeBody.
  • Errors are computed on frames where both hands are detected for all methods; detection rates are 0.967 for SignDiT and 0.862–0.923 for the baselines.

How2Sign, self-reenactment:

MethodH-PCK* ↑Palm* ↓HH-Occ.* ↑
SignMimic0.36329.80.751
SignRefine†0.50824.20.779
Wan2.2-Animate0.42629.90.753
SCAIL-20.45415.40.830
SignDiT0.5539.80.949

SignDiT retains the best self-reenactment results on all three hand metrics. Under WiLoR, absolute palm error is higher for the two methods conditioned on SMPL-X (SignDiT 5.1° → 9.8°, SCAIL-2 8.6° → 15.4°), and the ranking of the five methods is unchanged. On real video, the SMPL-X and WiLoR palm normals differ by 9.0°, which shows how much the estimated orientation depends on the reconstruction pipeline. Cross-identity results against the driver's real video are in Table 17: SignDiT has the best H-PCK* and HH-Occ.*, and its Palm* is 21.3° versus 22.0° for SCAIL-2 (Appendix J.2).

The paper also measures meaning directly, with back-translation BLEU-4 (Table 2), minimal-pair accuracy (Table 3), and the blinded fluent-signer study. These rely on translation and recognition models or on human judgment. In the human study, participants were told to judge sign correctness first, and they preferred SignDiT in 94/85/78/76% of comparisons.

ASL-only Sign-DPO improves DGS BLEU-4 by more points than ASL.

[A] Sign-DPO improves all three languages. The gains over Stage 1 are +1.42 / +1.42 / +3.56 BLEU-4 on How2Sign / CSL-Daily / Phoenix. Each language has its own judge and score scale (real video scores 13.28 / 28.79 / 27.43), so we also report the gain relative to Stage 1: 14.2% / 5.5% / 14.8%. In relative terms the ASL and DGS gains are similar and the CSL gain is smaller. The controls below test where these gains come from.

[B] The advantage persists after controlling for resolution and compression. After downsampling the generated videos to Phoenix's 210×260 and re-compressing them with H.264 at the source bitrate, SignDiT scores 27.09 (27.64 before), below the 27.43 of real video, and its margin over SCAIL-2 remains 3.45 BLEU-4 (3.79 before). All four baselines also output 512×896 video and score 18.63–23.85 on Phoenix, so a higher output resolution alone does not give a high score. Before degradation, SignDiT is +0.21 above real video, with a paired-bootstrap 95% interval of [−0.91, +1.33], so the two are not significantly different under this protocol.

[C] Phonological negatives outperform the degraded-hand control. Table 11 includes a preference stage trained with degraded-hand negatives, which targets generic hand quality. We evaluated this control on all three languages (BLEU-4, same Stage-1 checkpoint):

Stage 2How2SignPhoenixCSL-Daily
None (Stage 1)10.0324.0825.71
Degraded-hand negatives10.5825.2926.33
Sign-DPO11.4527.6427.13

The degraded-hand control recovers less than half of the Sign-DPO gain in every language (0.55 of 1.42, 1.21 of 3.56, and 0.62 of 1.42 BLEU-4) and reaches 0.67 minimal-pair accuracy, compared with 0.79 for Sign-DPO. Minimal-pair accuracy is measured on gloss contrasts unseen in Stage 2, by a judge that is never used in training. These results support two conclusions: the ASL-derived supervision transfers to CSL and DGS, and in each language it helps more than a generic hand-quality objective. The transfer is consistent with handshape, location, movement and orientation being manual parameters shared across sign languages.

AuthorsOpenReview ↗

Response to Reviewer WA3J (Part 2/3)

W1 / Q1 / Q3. Use beyond video-driven retargeting; generated poses; application scenarios.

Scope. SignDiT is a pose-to-video renderer for offline sign-content production, the same rendering task as all four baselines. Its direct uses are signer anonymization, re-rendering legacy low-resolution corpora such as Phoenix at 512×896 (we will release these videos), and identity-consistent composition from a gloss library. In the library application, isolated-sign pose clips are retrieved, canonicalized, joined with synthetic transitions and rendered in one pass (Table 4: 9.0 BLEU-4, 4.3/5 naturalness from fluent signers).

Text-generated poses. We evaluated the submitted renderer on SMPL-X sequences generated by SOKE from How2Sign test sentence texts. The SOKE generator weights are not released, so we trained the generator with the official code and configuration and the released tokenizer. The 2D path is drawn from the projected joints of these sequences and the diagnostics are set to constants. Without retraining the renderer, SignDiT reaches 383 FVD and 7.82 BLEU-4. Reference images are the same as in the self-reenactment manifest. This is the full text-to-pose-to-video pipeline, so the error of the upstream pose generator is included. Even so, FVD is lower than that of every baseline (468–913) and BLEU-4 is within the baseline range (6.41–8.23), although the baselines are driven by video-fitted poses (Section 4.5, Appendix J.3).

Projected poses and missing diagnostics. We also ran the projected-pose test you proposed using video-fitted SMPL-X as the single source: the 2D path is drawn from projected joints instead of DWPose, and confidence and residual are set to constants. The table shows each setting separately:

InputsFVD ↓H-PCK ↑BLEU-4 ↑
Submitted (DWPose + estimated diagnostics)2720.5211.45
(i) 2D path from projected SMPL-X joints2960.48011.07
(ii) Constant reliability inputs2930.51311.28
(iii) Both (single-source inputs)3440.47810.68
(iv) Text-generated SMPL-X sequence (SOKE; Zuo et al., ICCV 2025)383–7.82
Best baseline value per metric4680.498.23

With video-fitted SMPL-X as the single source and constant diagnostics (row (iii)), SignDiT loses 0.77 BLEU-4 and still beats every baseline on FVD and BLEU-4 (the baselines keep their usual estimated poses). H-PCK is 0.478, below SignRefine (0.492) and above the other three baselines (0.384–0.471). The H-PCK target is the driver's DWPose keypoints, which this configuration no longer receives as input.

H-PCK is omitted for row (iv) because SOKE-generated motion has no frame-aligned real video to compare with. In that row, BLEU-4 (7.82) is above SignMimic (6.41) and SCAIL-2 (7.12) and 0.41 below the best baseline; all baselines use video-fitted poses.

Single-source conditioning. When both paths come from one SMPL-X sequence, they still carry different information: the skeleton raster gives image-plane location and motion, and the normal, depth and order maps add 3D cues. Rows (iii) and (iv) measure this case. The reliability field still receives the motion input, while the other two inputs become constants, so its weights still vary with body part and time.

Cost. SignDiT is an offline renderer and does not target real-time use. Conditioning is added to the existing latent tokens (Eq. 5), so the DiT processes the same number of tokens as Wan2.2-Animate. On one H100, sampling a 77-frame clip with 25 steps takes 1.04× the time of Wan2.2-Animate. Preprocessing a driving video takes 0.043 s per frame for DWPose and 0.66 s per frame for SMPL-X fitting. If an SMPL-X sequence is supplied directly, these two preprocessing steps are skipped. Pair generation and the verifier are used only during training.

AuthorsOpenReview ↗

Response to Reviewer WA3J (Part 3/3)

W2(c). Self-reenactment test signers appear in training.

We use the official clip-disjoint splits, which share signers between training and test, for every method (Appendix A). Self-reenactment therefore tests unseen clips from signers that appear in training. Table 1(b) separately evaluates rendering onto 200 unseen synthetic reference identities, using drivers from those same test splits; SignDiT leads on FID, ArcFace, H-PCK and HH-Occ. on all three datasets. Section 4.1 now states this difference between driver signers and reference identities.

W2(d). Single seed, no confidence intervals.

We added two analyses.

Paired bootstrap over test clips (10,000 resamples, How2Sign, 95% intervals of the difference):

ContrastDifference95% CI
SignDiT − strongest baseline, BLEU-4+3.22[+2.45, +3.85]
Sign-DPO − Stage 1, BLEU-4+1.42[+0.88, +1.93]
Fiber maps − full render, BLEU-4+0.84[+0.31, +1.37]
SIREN − no gate, BLEU-4+0.21[−0.22, +0.68]
SIREN − no gate, hand flicker−0.009[−0.0107, −0.0071]
SIREN − no gate, FVD−32[−44.9, −17.6]

Generation seeds. With the SignDiT checkpoint fixed, three sampling seeds on How2Sign self-reenactment give standard deviations of 0.09 BLEU-4, 4.0 FVD and 0.003 H-PCK, all smaller than our margins over the strongest baselines. The bootstrap covers the uncertainty from the choice of test clips, and the seeds cover the variation in generation.

The reliability field reduces hand flicker (0.021 → 0.012) and FVD (333 → 301), and both intervals exclude zero. Its BLEU-4 gain (+0.21) has an interval that includes zero, so Section 4.4 now justifies this component by temporal stability and FVD. Appendix J.4 reports the intervals and seed statistics (Tables 19 and 20).

Reviewer_1WUuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SignDiT for pose-guided sign language video generation. The method combines 2D and 3D pose information, a reliability field, and phonology-guided DPO to improve hand motion accuracy and semantic consistency.

Critical strengths
  1. The method considers the fine-grained hand movements and linguistic characteristics of sign language rather than simply adapting a general video generation model.
  2. Using phonological counterfactual pairs for DPO is an interesting idea, and the motivation is reasonable.
  3. The results are strong across multiple datasets, and the generated hand movements appear accurate in the qualitative examples.
Critical weaknesses
  1. Sign-DPO uses isolated sign datasets (WLASL and MS-ASL) to construct preference pairs, but the main experiments are conducted on continuous sign language datasets. The generalization from isolated to continuous signing needs further discussion, especially regarding co-articulation.
  2. Transplanting local motions to construct counterfactual samples may introduce temporal discontinuities. The authors apply filtering constraints, but it is not clear how well these constraints preserve motion continuity.
  3. The presentation could be improved. Some labels in Figure 2 are not clearly explained or consistently used in the main text, making the framework harder to follow.
Questions for authors

See the weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1WUu

W1. Preference pairs come from isolated signs, the benchmarks are continuous.

All three main benchmarks contain continuous signing, and Sign-DPO improves each over the same Stage-1 checkpoint: full-sentence BLEU-4 rises by +1.42 / +1.42 / +3.56 on How2Sign / CSL-Daily / Phoenix (Table 12), How2Sign FVD improves from 301 to 272, and fluent signers reported understanding 85% of generated continuous sentences.

Co-articulation. The model learns co-articulation in Stage 1, which is trained only on continuous signing. Stage 2 starts from that model and applies the preference loss only to the edited hand region and frames. It runs for 5k updates at a learning rate of 1e-5, and DPO compares the trained model with the frozen Stage-1 model. We checked the effect on continuous signing in two ways:

  • Sign-to-sign boundaries. On the 50 composed sequences of Table 4, boundary flicker is 0.0113 before Stage 2 and 0.010 after, and BLEU-4 is 8.55 before and 9.0 after. Isolated-sign supervision therefore does not degrade the joins between signs.
  • Sentence length. When the How2Sign test sentences are split into terciles by length, the BLEU-4 gain of Sign-DPO is +1.22 / +1.47 / +1.10 for short / medium / long sentences, so the gain holds at every sentence length in natural continuous signing.

This discussion is now in Sections 3.6 and 4.4 and Appendix J.5 (Table 21). The Limitations list mining Sign-DPO contrasts that arise only through co-articulation as future work.

W2. Transplanting local motion may introduce temporal discontinuities.

Before the transplant, the two signs are temporally aligned and scale-normalized; afterwards the full condition sequence is re-rendered. A candidate is accepted only if its joint-limit or temporal-jump violation rate is at most 0.06, together with the reprojection, coverage and visual-quality criteria (Appendix A); 7,601 of 15,327 candidates pass.

We now measure how well this filter works, using the peak wrist and fingertip acceleration within ±4 frames of the edit boundary, normalized by hand size:

SequencesMedian, relative to real clips95th percentile, relative to real clips
Unedited real isolated signs1.001.00
Accepted counterfactual conditions1.051.06
Candidates rejected by the kinematic rule2.164.86

Accepted edits are close to real clips at both the median and the 95th percentile, while rejected candidates are higher. In the generated videos, hand flicker on How2Sign is 0.012 before Stage 2 and 0.0119 after it. The FVD results in the submission agree: on the same pair pool, vanilla Diffusion-DPO worsens FVD (301 → 310), whereas Sign-DPO improves it (301 → 272).

W3. Presentation; labels in Figure 2.

We revised the labels and caption of Figure 2 and aligned its terminology with the text:

  • "SRF" appeared in the figure, its caption and Table 4 without a definition. It is now introduced in Section 3.3 (SIREN reliability field) and used consistently.
  • The tags C1/C2/C3 were only defined in Section 4.4. They are now defined in the caption and in Section 3.1.
  • Panel (a) was titled "Reconstruction training" but showed Signer A driving Signer B, which is cross-identity inference. The panel is now titled "Stage 1: dual-path conditioning" and states that it shows cross-identity inference, whereas training reconstructs a clip from a reference frame of the same video.
  • Figure symbols now match the equations: B(p)B(p), Z2DZ_{2\mathrm D}, ZϕZ_\phi, C~2D\widetilde C^{2\mathrm D}, C~ϕ\widetilde C^{\phi}, wk2D(τ)w_k^{2\mathrm D}(\tau), wkϕ(τ)w_k^{\phi}(\tau).
  • Names are unified with the text: "2D path", "fiber path", "verifier VV" (a frozen sign-recognition model), "factorized Sign-DPO", "2D and fiber condition adapters"; the four factor abbreviations (HS, LOC, MOV, ORI) are spelled out in the caption.
  • The caption now walks through both panels step by step, and Appendix J.1 adds a notation table (Table 15) that maps every symbol to its label in the figure.

We also checked the whole paper for consistent abbreviations and component names.

Thanks for the response. I have read the revised manuscript, but I still believe the writing and presentation need further improvement. In particular, the framework figure appears unpolished, with several unnecessary lines that do not convey meaningful information. Therefore, I will maintain my original score.

AuthorsOpenReview ↗

Figure 2 redrawn in the updated PDF

Thank you for reading the revision and for the follow-up. We agree that the framework figure needed more care, and we have redrawn Figure 2 in the updated PDF (page 4).

  • We removed the dashed connector lines that carried no information about the method: the frozen reference, face and text paths in panel (a), the "shared part regions" line between the two panels, and the connector into the Stage-1 checkpoint.
  • We re-routed panel (b): the three outputs of the phonological intervention now leave through a single trunk, the mask and the selected factor reach the Sign-DPO loss as straight lines, and every input to the loss is labeled.
  • We enlarged the small labels throughout the figure.

We also hope that our earlier responses resolved your two technical concerns, on generalization from isolated to continuous signing and on motion continuity of the counterfactual samples. If any other part of the presentation is unclear, we would be glad to fix it before the discussion closes.

Thank you for the revisions and clarifications. I will maintain my original score.

Reviewer_b7RiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SignDiT adapts Wan2.2-Animate for pose-guided sign-language video generation. Stage 1 conditions the model on a 2D skeleton plus SMPL-X normals, depth, and hand occlusion order, with a SIREN-based gate that weights each path per body part over time. Stage 2, Sign-DPO, creates negatives by transplanting one phonological parameter from a minimal-pair sign and applies DPO only to the edited region and frames. On How2Sign, CSL-Daily, and Phoenix, SignDiT outperforms four baselines on most fidelity metrics, approaches real-video back-translation BLEU, and is preferred by fluent ASL signers in a blinded study.

Critical strengths

The evaluation is thorough and checks meaning, not only appearance: language-matched back-translation, minimal-pair tests, and a fluent-signer study, with baselines fine-tuned under the same budget. Gains are large and consistent across three sign languages.

Sign-DPO is a well-motivated way to build hard negatives, and Table 11 shows clear improvements over vanilla, random-mask, and degraded-hand DPO variants.

Critical weaknesses

Part of the evaluation is entangled with the model's own tools. Palm error and occlusion accuracy are measured by refitting SMPL-X, the same estimator that produces the conditioning maps, so they reward agreement with SMPL-X rather than with the true hand.

The judge for minimal-pair accuracy is not named, and it is unclear whether the test pairs overlap with the Sign-DPO training contrasts, or whether Table 2 uses self-reenactment or cross-identity clips.

The mechanistic claims outrun the evidence. Rasterized normal and depth maps contain the same projected silhouette as a full render, and Table 9 suggests that most of the gain comes from the occlusion-order channel rather than from removing duplicated information. ASL-only Sign-DPO also improves DGS BLEU more than ASL BLEU (+3.56 vs. +1.42), which points to a generic hand-quality effect rather than phonological learning.

Questions for authors

Palm error and occlusion accuracy are measured by refitting SMPL-X, which is the same estimator that produces the conditioning maps. Doesn’t this mainly reward agreement with SMPL-X rather than with the true hand configuration, and would the conclusions hold if these metrics were computed with an independent hand estimator?

3 comments
AuthorsOpenReview ↗

Response to Reviewer b7Ri (Part 1/2)

Palm and occlusion metrics share the SMPL-X pipeline with our conditioning.

We agree that these metrics share estimators with our conditioning, so we re-evaluated every method on How2Sign with two tools that are not used to build the conditioning: WiLoR and ViTPose-H WholeBody. Each tool is run on both the generated and the real videos. WiLoR is a separate MANO-based hand reconstruction, so this tests whether the results hold under a different reconstruction pipeline.

  • Palm*: palm normals from WiLoR hand reconstructions.
  • HH-Occ.*: front hand from the WiLoR camera-space depth of the two hands, on frames where the hand boxes overlap in the real video.
  • H-PCK*: H-PCK has the same issue, because it used DWPose, our 2D estimator. We recomputed it with ViTPose-H WholeBody.
  • Errors are computed on frames where both hands are detected for all methods; detection rates are 0.967 for SignDiT and 0.862–0.923 for the baselines.

How2Sign, self-reenactment:

MethodH-PCK* ↑Palm* ↓HH-Occ.* ↑
SignMimic0.36329.80.751
SignRefine†0.50824.20.779
Wan2.2-Animate0.42629.90.753
SCAIL-20.45415.40.830
SignDiT0.5539.80.949

SignDiT retains the best self-reenactment results on all three hand metrics. Under WiLoR, absolute palm error is higher for the two methods conditioned on SMPL-X (SignDiT 5.1° → 9.8°, SCAIL-2 8.6° → 15.4°), and the ranking of the five methods is unchanged. On real video, the SMPL-X and WiLoR palm normals differ by 9.0°, which shows how much the estimated orientation depends on the reconstruction pipeline. Cross-identity results against the driver's real video are in Table 17: SignDiT has the best H-PCK* and HH-Occ.*, and its Palm* is 21.3° versus 22.0° for SCAIL-2 (Appendix J.2).

The paper also measures meaning directly, with back-translation BLEU-4 (Table 2), minimal-pair accuracy (Table 3), and the blinded fluent-signer study. These rely on translation and recognition models or on human judgment. In the human study, participants were told to judge sign correctness first, and they preferred SignDiT in 94/85/78/76% of comparisons.

ASL-only Sign-DPO improves DGS BLEU-4 by more points than ASL.

[A] Sign-DPO improves all three languages. The gains over Stage 1 are +1.42 / +1.42 / +3.56 BLEU-4 on How2Sign / CSL-Daily / Phoenix. Each language has its own judge and score scale (real video scores 13.28 / 28.79 / 27.43), so we also report the gain relative to Stage 1: 14.2% / 5.5% / 14.8%. In relative terms the ASL and DGS gains are similar and the CSL gain is smaller. The controls below test where these gains come from.

[C] Phonological negatives outperform the degraded-hand control. Table 11 includes a preference stage trained with degraded-hand negatives, which targets generic hand quality. We evaluated this control on all three languages (BLEU-4, same Stage-1 checkpoint):

Stage 2How2SignPhoenixCSL-Daily
None (Stage 1)10.0324.0825.71
Degraded-hand negatives10.5825.2926.33
Sign-DPO11.4527.6427.13

The degraded-hand control recovers less than half of the Sign-DPO gain in every language (0.55 of 1.42, 1.21 of 3.56, and 0.62 of 1.42 BLEU-4) and reaches 0.67 minimal-pair accuracy, compared with 0.79 for Sign-DPO. Minimal-pair accuracy is measured on gloss contrasts unseen in Stage 2, by a judge that is never used in training. These results support two conclusions: the ASL-derived supervision transfers to CSL and DGS, and in each language it helps more than a generic hand-quality objective. The transfer is consistent with handshape, location, movement and orientation being manual parameters shared across sign languages. After downsampling the generated videos to Phoenix's 210×260 and re-compressing them with H.264 at the source bitrate, SignDiT scores 27.09 (27.64 before), below the 27.43 of real video, and its margin over SCAIL-2 remains 3.45 BLEU-4 (3.79 before).

AuthorsOpenReview ↗

Response to Reviewer b7Ri (Part 2/2)

W2. Minimal-pair judge, train/test overlap, and the protocol of Table 2.

Section 4.1, the caption of Table 2, and Appendix J.7 now give the judge, how a pair is scored, the separation between training and test contrasts, and the protocol of Table 2:

  • Judge and scoring. Minimal-pair accuracy is computed with the frozen SIGNET judge (Asasi et al., 2026), separate from the training verifier (Uni-Sign) and never used in training. Each test pair contains two generated videos with distinct target glosses. A pair counts as correct only if SIGNET assigns both videos to their respective target glosses; the reported score is the fraction of correct pairs.
  • Overlap. None. The gloss contrasts of the test pairs are disjoint from the 915 contrasts used in Stage 2, so the reported accuracy (0.63 → 0.79) is measured on contrasts unseen during preference training.
  • Table 2. It uses self-reenactment clips: the same manifest as Table 1(a) and the ablations.

W3(a). Silhouette in normal/depth maps; gain attributed to the occlusion-order channel.

Silhouette. We agree: the normal and depth maps retain the projected outline, and the paper no longer attributes the gain to removing duplicated information. What the fiber path does is give orientation, depth and order their own channels and leave out shading, texture and a separate mask channel. Sections 1 and 3.2 now say this directly.

Which channel matters. In Table 9 each channel carries its own cue, and occlusion order is not the largest contributor to BLEU-4 or FVD:

Removed from fiber mapsFVD ↓Palm ↓Depth ↓HH-Occ. ↑BLEU-4 ↑
nothing30570.090.919.88
occlusion order34090.100.648.62
normal380190.100.859.21
relative depth355100.200.808.47

Removing depth costs the most BLEU-4, removing normals costs the most FVD and palm accuracy, and removing occlusion order costs occlusion accuracy.

What Table 9 supports. Fiber maps beat a full render with the same adapters and token budget, and removing any one channel hurts, as shown above. The "render + normal + depth" variant differs from fiber maps in more than one channel, so that comparison reflects the whole change of representation and not a single factor. Sections 1, 3.2 and 4.4 now state what the data support: giving orientation, depth and order their own channels improves rendering, and each channel contributes to the reported metrics.

Thank you for the detailed rebuttal and additional experiments. I will maintain my score of 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Additions and Revisions

We thank the reviewers for their careful and constructive reading. Each reviewer has an individual reply below. The main additions are:

  1. Hand metrics with separate evaluation pipelines (WA3J, b7Ri). On How2Sign self-reenactment, palm error and occlusion order are recomputed with WiLoR against the real video. SignDiT keeps the lowest palm error (9.8° vs. 15.4° for the strongest baseline) and the highest occlusion accuracy (0.949 vs. 0.830).
  2. Source of the Sign-DPO gain (WA3J, b7Ri). The degraded-hand preference control recovers less than half of the Sign-DPO gain in all three languages. After controlling for Phoenix resolution and compression, SignDiT scores 27.09, below real video (27.43), and its margin over SCAIL-2 remains 3.45 BLEU-4 (3.79 before).
  3. Text-generated and single-source poses (WA3J). SOKE generates SMPL-X sequences from How2Sign test sentence texts; the same SignDiT renderer achieves 383 FVD and 7.82 BLEU-4 with projected 2D joints and constant diagnostics, without renderer retraining; the baselines, driven by video-fitted poses, score 468–913 FVD and 6.41–8.23 BLEU-4. The video-fitted single-source control achieves 344 FVD and 10.68 BLEU-4. Section 4.5 and Appendix J.3 report both settings.
  4. Statistics (WA3J). We add paired-bootstrap intervals for the key How2Sign comparisons and for SignDiT versus real video on Phoenix, and results over three generation seeds for the fixed SignDiT checkpoint on How2Sign self-reenactment.
  5. Sign-DPO on continuous signing (1WUu). We add analyses of sign-to-sign boundaries, sentence length, and the motion continuity of the preference pairs.
  6. Presentation (1WUu, b7Ri). Figure 2 has revised labels and notation aligned to the text; the minimal-pair judge, train/test contrast separation and protocol of Table 2 are documented; the claim about the representation is narrowed to giving orientation, depth and order their own channels.

All changes are marked in blue in the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.