- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses pose-guided sign language video generation, where a reference image and a driving signing video are used to synthesize a new signer while preserving hand articulation. Built on Wan2.2-Animate-14B, SignDiT has two stages. Stage 1 replaces the pose path with two complementary conditioning streams: a 2D DWPose skeleton path for image-plane motion, and a “fiber” path with surface normals, root-relative depth, and occlusion order derived from SMPL-X fits. Both streams are modulated by a SIREN-based part × time reliability field driven by estimator diagnostics (keypoint confidence, fitting residual, motion). Stage 2 (Sign-DPO) builds counterfactual negatives by transplanting a single phonological factor (handshape, location, movement, or orientation) from ASL minimal pairs, filters pairs with a sign recognizer, and applies a factorized DPO loss localized to the edited region and frames. Experiments on How2Sign, CSL-Daily, and Phoenix report gains in FID/FVD, hand metrics, and back-translation BLEU-4, supported by a study with eight fluent ASL signers and component ablations on How2Sign.
Critical strengths
- The paper targets a real and important failure mode: generic human-animation models corrupt hand details that carry lexical meaning in sign language. Sign-DPO, which constructs phonologically controlled hard negatives and localizes credit to the edited region, is a well-motivated way to inject linguistic correctness into training, and it improves minimal-pair accuracy (0.63 → 0.79) and BLEU-4 in the ablations.
- The evaluation is broad for this area. It covers three sign languages, visual, hand-geometry, and semantic metrics, and an IRB-approved blinded study with fluent ASL signers, which is rare and valuable for sign language generation.
Critical weaknesses
- Limited practicality beyond video-driven retargeting. At inference, SignDiT requires 2D keypoints, SMPL-X fits, and estimator diagnostics extracted from a real driving video, so the evaluated task is signer retargeting or anonymization rather than sign language production from text. Using it in an end-to-end SLP pipeline would require an upstream text-to-pose model that generates accurate SMPL-X sequences, including hands. In that setting, the 2D keypoints would simply be projections of the same 3D source. The central argument that the two paths carry complementary, independently estimated signals would then no longer hold, and the reliability field, whose inputs are detector confidence and fitting residuals, would not be well defined. The word-library application also relies on retrieving pose clips from real videos. The paper does not test the method with generated or projected poses as input, which leaves its usefulness for SLP unclear. The cost is also high: a 14B backbone, per-frame DWPose and SMPL-X fitting, and about 2,500 H100 GPU-hours of training.
- Several key results may be inflated by the evaluation protocol. (a) Palm angular error and HH-Occ. are computed by refitting SMPL-X/MANO on generated videos, while SignDiT is explicitly conditioned on normals and occlusion order from the same fitting pipeline. The large margins on these metrics are therefore partly built into the design. (b) Sign-DPO pairs are constructed only from ASL, yet the BLEU-4 gain is much larger on Phoenix/DGS (+3.56) than on How2Sign/ASL (+1.42), and the generated Phoenix videos score above real video. This suggests the back-translation judges partly reward visual clarity or resolution rather than sign correctness. (c) Self-reenactment test signers appear in training. (d) Automatic metrics come from a single seed with no confidence intervals, and run-to-run variance is estimated from a single repeat (0.15 BLEU-4), which is comparable to some reported component gains (e.g., SIREN vs. no gate: +0.21 BLEU-4).
Questions for authors
- Can you evaluate SignDiT when driven by the output of a text-to-pose model, or at least with the 2D path computed from projected SMPL-X joints instead of DWPose? How does the reliability field operate when estimator confidence and fitting residuals are unavailable or trivial, as with generated poses?
- Can you report palm and occlusion metrics with an evaluator independent of the SMPL-X/MANO pipeline used for conditioning, and explain why ASL-only Sign-DPO yields a larger BLEU-4 gain on DGS than on ASL?
- The last question is also the one I am most concerned about. How do you view its application scenarios? Although your approach has increased the variety of output types, it is actually very difficult to use in practice. It is equivalent to a sign language re-renderer or Pose2Video. For instance, if the types of your input become very diverse, it will affect the real-time performance of the entire process.
3 comments
Response to Reviewer WA3J (Part 2/3)
W1 / Q1 / Q3. Use beyond video-driven retargeting; generated poses; application scenarios.
Scope. SignDiT is a pose-to-video renderer for offline sign-content production, the same rendering task as all four baselines. Its direct uses are signer anonymization, re-rendering legacy low-resolution corpora such as Phoenix at 512×896 (we will release these videos), and identity-consistent composition from a gloss library. In the library application, isolated-sign pose clips are retrieved, canonicalized, joined with synthetic transitions and rendered in one pass (Table 4: 9.0 BLEU-4, 4.3/5 naturalness from fluent signers).
Text-generated poses. We evaluated the submitted renderer on SMPL-X sequences generated by SOKE from How2Sign test sentence texts. The SOKE generator weights are not released, so we trained the generator with the official code and configuration and the released tokenizer. The 2D path is drawn from the projected joints of these sequences and the diagnostics are set to constants. Without retraining the renderer, SignDiT reaches 383 FVD and 7.82 BLEU-4. Reference images are the same as in the self-reenactment manifest. This is the full text-to-pose-to-video pipeline, so the error of the upstream pose generator is included. Even so, FVD is lower than that of every baseline (468–913) and BLEU-4 is within the baseline range (6.41–8.23), although the baselines are driven by video-fitted poses (Section 4.5, Appendix J.3).
Projected poses and missing diagnostics. We also ran the projected-pose test you proposed using video-fitted SMPL-X as the single source: the 2D path is drawn from projected joints instead of DWPose, and confidence and residual are set to constants. The table shows each setting separately:
| Inputs | FVD ↓ | H-PCK ↑ | BLEU-4 ↑ |
|---|---|---|---|
| Submitted (DWPose + estimated diagnostics) | 272 | 0.52 | 11.45 |
| (i) 2D path from projected SMPL-X joints | 296 | 0.480 | 11.07 |
| (ii) Constant reliability inputs | 293 | 0.513 | 11.28 |
| (iii) Both (single-source inputs) | 344 | 0.478 | 10.68 |
| (iv) Text-generated SMPL-X sequence (SOKE; Zuo et al., ICCV 2025) | 383 | – | 7.82 |
| Best baseline value per metric | 468 | 0.49 | 8.23 |
With video-fitted SMPL-X as the single source and constant diagnostics (row (iii)), SignDiT loses 0.77 BLEU-4 and still beats every baseline on FVD and BLEU-4 (the baselines keep their usual estimated poses). H-PCK is 0.478, below SignRefine (0.492) and above the other three baselines (0.384–0.471). The H-PCK target is the driver's DWPose keypoints, which this configuration no longer receives as input.
H-PCK is omitted for row (iv) because SOKE-generated motion has no frame-aligned real video to compare with. In that row, BLEU-4 (7.82) is above SignMimic (6.41) and SCAIL-2 (7.12) and 0.41 below the best baseline; all baselines use video-fitted poses.
Single-source conditioning. When both paths come from one SMPL-X sequence, they still carry different information: the skeleton raster gives image-plane location and motion, and the normal, depth and order maps add 3D cues. Rows (iii) and (iv) measure this case. The reliability field still receives the motion input, while the other two inputs become constants, so its weights still vary with body part and time.
Cost. SignDiT is an offline renderer and does not target real-time use. Conditioning is added to the existing latent tokens (Eq. 5), so the DiT processes the same number of tokens as Wan2.2-Animate. On one H100, sampling a 77-frame clip with 25 steps takes 1.04× the time of Wan2.2-Animate. Preprocessing a driving video takes 0.043 s per frame for DWPose and 0.66 s per frame for SMPL-X fitting. If an SMPL-X sequence is supplied directly, these two preprocessing steps are skipped. Pair generation and the verifier are used only during training.
Response to Reviewer WA3J (Part 3/3)
W2(c). Self-reenactment test signers appear in training.
We use the official clip-disjoint splits, which share signers between training and test, for every method (Appendix A). Self-reenactment therefore tests unseen clips from signers that appear in training. Table 1(b) separately evaluates rendering onto 200 unseen synthetic reference identities, using drivers from those same test splits; SignDiT leads on FID, ArcFace, H-PCK and HH-Occ. on all three datasets. Section 4.1 now states this difference between driver signers and reference identities.
W2(d). Single seed, no confidence intervals.
We added two analyses.
Paired bootstrap over test clips (10,000 resamples, How2Sign, 95% intervals of the difference):
| Contrast | Difference | 95% CI |
|---|---|---|
| SignDiT − strongest baseline, BLEU-4 | +3.22 | [+2.45, +3.85] |
| Sign-DPO − Stage 1, BLEU-4 | +1.42 | [+0.88, +1.93] |
| Fiber maps − full render, BLEU-4 | +0.84 | [+0.31, +1.37] |
| SIREN − no gate, BLEU-4 | +0.21 | [−0.22, +0.68] |
| SIREN − no gate, hand flicker | −0.009 | [−0.0107, −0.0071] |
| SIREN − no gate, FVD | −32 | [−44.9, −17.6] |
Generation seeds. With the SignDiT checkpoint fixed, three sampling seeds on How2Sign self-reenactment give standard deviations of 0.09 BLEU-4, 4.0 FVD and 0.003 H-PCK, all smaller than our margins over the strongest baselines. The bootstrap covers the uncertainty from the choice of test clips, and the seeds cover the variation in generation.
The reliability field reduces hand flicker (0.021 → 0.012) and FVD (333 → 301), and both intervals exclude zero. Its BLEU-4 gain (+0.21) has an interval that includes zero, so Section 4.4 now justifies this component by temporal stability and FVD. Appendix J.4 reports the intervals and seed statistics (Tables 19 and 20).
Response to Reviewer WA3J (Part 1/3)
Palm and occlusion metrics share the SMPL-X pipeline with our conditioning.
We agree that these metrics share estimators with our conditioning, so we re-evaluated every method on How2Sign with two tools that are not used to build the conditioning: WiLoR and ViTPose-H WholeBody. Each tool is run on both the generated and the real videos. WiLoR is a separate MANO-based hand reconstruction, so this tests whether the results hold under a different reconstruction pipeline.
How2Sign, self-reenactment:
SignDiT retains the best self-reenactment results on all three hand metrics. Under WiLoR, absolute palm error is higher for the two methods conditioned on SMPL-X (SignDiT 5.1° → 9.8°, SCAIL-2 8.6° → 15.4°), and the ranking of the five methods is unchanged. On real video, the SMPL-X and WiLoR palm normals differ by 9.0°, which shows how much the estimated orientation depends on the reconstruction pipeline. Cross-identity results against the driver's real video are in Table 17: SignDiT has the best H-PCK* and HH-Occ.*, and its Palm* is 21.3° versus 22.0° for SCAIL-2 (Appendix J.2).
The paper also measures meaning directly, with back-translation BLEU-4 (Table 2), minimal-pair accuracy (Table 3), and the blinded fluent-signer study. These rely on translation and recognition models or on human judgment. In the human study, participants were told to judge sign correctness first, and they preferred SignDiT in 94/85/78/76% of comparisons.
ASL-only Sign-DPO improves DGS BLEU-4 by more points than ASL.
[A] Sign-DPO improves all three languages. The gains over Stage 1 are +1.42 / +1.42 / +3.56 BLEU-4 on How2Sign / CSL-Daily / Phoenix. Each language has its own judge and score scale (real video scores 13.28 / 28.79 / 27.43), so we also report the gain relative to Stage 1: 14.2% / 5.5% / 14.8%. In relative terms the ASL and DGS gains are similar and the CSL gain is smaller. The controls below test where these gains come from.
[B] The advantage persists after controlling for resolution and compression. After downsampling the generated videos to Phoenix's 210×260 and re-compressing them with H.264 at the source bitrate, SignDiT scores 27.09 (27.64 before), below the 27.43 of real video, and its margin over SCAIL-2 remains 3.45 BLEU-4 (3.79 before). All four baselines also output 512×896 video and score 18.63–23.85 on Phoenix, so a higher output resolution alone does not give a high score. Before degradation, SignDiT is +0.21 above real video, with a paired-bootstrap 95% interval of [−0.91, +1.33], so the two are not significantly different under this protocol.
[C] Phonological negatives outperform the degraded-hand control. Table 11 includes a preference stage trained with degraded-hand negatives, which targets generic hand quality. We evaluated this control on all three languages (BLEU-4, same Stage-1 checkpoint):
The degraded-hand control recovers less than half of the Sign-DPO gain in every language (0.55 of 1.42, 1.21 of 3.56, and 0.62 of 1.42 BLEU-4) and reaches 0.67 minimal-pair accuracy, compared with 0.79 for Sign-DPO. Minimal-pair accuracy is measured on gloss contrasts unseen in Stage 2, by a judge that is never used in training. These results support two conclusions: the ASL-derived supervision transfers to CSL and DGS, and in each language it helps more than a generic hand-quality objective. The transfer is consistent with handshape, location, movement and orientation being manual parameters shared across sign languages.