- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents PoseDLLM, a discrete-to-continuous diffusion framework for multimodal 3D human pose modeling. The main idea is to use a masked discrete diffusion language model for multimodal semantic reasoning and coarse pose prediction, exploiting bidirectional attention and iterative remasking to better handle coupled pose constraints, followed by a pretrained continuous diffusion pose prior for geometric refinement. The framework supports pose estimation, text-to-pose generation, pose captioning, pose-difference captioning, and language-guided pose editing under a shared instruction-following interface. Experiments on Human3.6M, PoseScript, PoseFix, and Posepart show substantial improvements over autoregressive pose-language baselines, particularly for text-conditioned generation and pose editing.
Critical strengths
- The paper presents an interesting use of masked discrete diffusion language models for multimodal 3D pose modeling.
- The discrete semantic reasoning and continuous geometric refinement stages are conceptually complementary and form a simple, coherent coarse-to-fine framework.
- The writing is easy to follow, and the experimental results look good.
Critical weaknesses
- The contribution of masked diffusion itself is not fully disentangled from the pretrained backbone capabilities. PoseDLLM adopts LLaDA-V-8B, which already exhibits stronger general reasoning performance than the LLaVA-based AR baselines. Although the remasking ablation supports iterative decoding, it remains unclear how much of the improvement over AR models comes from the decoding paradigm rather than the stronger pretrained backbone.
- The retrieval evaluation raises a fairness concern. Each method uses its own learned pose encoder to compute R@K, making the results dependent on both the generated poses and the quality of the learned representations. While this protocol may be appropriate for evaluating representation capabilities, I do not think following PoseLLaVA is sufficient justification for using these results to support claims of superior pose generation and semantic grounding. The paper should clearly distinguish retrieval and representation performance from pose generation accuracy.
- I noticed several inconsistencies between the main paper and the appendix that raise concerns about reproducibility. Specifically, the LoRA rank differs (64 vs. 16), the pose encoder is described as a two-layer MLP versus a six-layer Transformer, and the pose regression objective differs in both formulation (axis-angle L2 vs. rotation-matrix MSE) and loss weight (100 vs. 10).
Minor problems:
- Several citations use \citet where \citep would be more appropriate, particularly when references are not grammatical components of the sentence.
- In Table 1, the best result of MPJPE seems to be the HRM2.0?
- Also in Table 2, some of the best and second-best values are not properly highlighted.
Questions for authors
- Could the authors provide a controlled AR baseline with matched model capacity, pose decoder, and training settings to directly verify the benefit of bidirectional iterative decoding?
- Could the authors reevaluate R@K using a shared frozen pose encoder and identical retrieval galleries to isolate pose generation quality from representation differences?
- Could the authors clarify which architecture, loss formulation, and hyperparameters were actually used for the reported experiments?
5 comments
Rebuttal Table 4. One-shot and iterative decoding on PoseFix-SMPL
| Strategy | MPJPE↓ (mm) | R@1↑ (%) |
|---|---|---|
| One-shot | 76.5 | 71 |
| Random unmasking | 96.4 | 37 |
| Confidence-ranked | 70.5 | 76 |
3. Matched causal/masked adaptation. PoseLLaVA is the primary method-level AR baseline, with UniPose as an additional comparison; these are not fully matched training controls. Our additional causal/masked adaptation experiment fixes LLaDA initialization, model capacity, pose modules, training data, and training budget. Both arms share diffusion pretraining and complete-template readout, so this does not independently establish pretraining-paradigm or iterative-decoding advantages. The results include seed variation and adverse outcomes; independent inference controls likewise show no universal iteration advantage.
4. PoseDLLM2D representation-pathway comparison. Appendix B/Tables 10 and 12 in the submitted paper use the same LLaDA-V backbone to generate 2D coordinate text followed by 3D fitting. PoseFix-SMPL MPJPE is 134.2 mm, compared with 70.5 mm for the main continuous-readout coarse output and 53.4 mm for the full system. This informs the choice of the main 3D interface, while differing supervision and fitting prevent it from substituting for a matched AR training control.
Question 2 (also addressing W2): R@K reevaluation with a shared frozen encoder and identical galleries
1. Geometric mapping shared by all methods. Thank you for the suggestion on evaluation fairness. The shared evaluator is a fixed SMPL geometric mapping, not a pretrained neural pose encoder. All methods use SMPL_NEUTRAL.pkl with β=0 and the fixed J_regressor_h36m.npy to obtain 17 joints, subtracting each pose’s root position. These assets are not trained or updated for individual methods. We will correct the method-specific embedding description in Appendix E and Eqs. (14)–(15).
2. Shared galleries and distance ranking. With F denoting this mapping, candidates in the same task-specific gallery are ranked by the mean Euclidean joint distance below. R@K is the percentage retrieving the correct target within K=1/5/10. Ranking uses neither cosine similarity nor Procrustes alignment.
Rebuttal Table 5. Shared geometric retrieval protocol across methods
| Evaluation component | Specification shared by all methods |
|---|---|
| Shared evaluator | Neutral SMPL mesh-to-H36M17 joint mapping |
| Fixed assets | SMPL_NEUTRAL.pkl + J_regressor_h36m.npy |
| Shape / coordinates | β=0; subtract each pose’s root joint |
| Gallery | Identical candidate pool within each task |
| Ranking | Mean Euclidean error over 17 joints; ascending MPJPE |
| Learned retrieval encoder | None; no method-specific or shared neural embedding checkpoint |
| R@K | Correct target in the top K; K=1,5,10 |
3. Shared neural encoders versus geometric retrieval. A shared frozen neural encoder would define a different common feature space; this clarifies the existing geometric protocol rather than claiming a new encoder-based reevaluation. We will distinguish geometric error, target retrieval, and instruction satisfaction: R@K alone does not establish complete semantic correctness and must be interpreted alongside task results and Appendix H’s failure analysis.
Question 3 (also addressing W3): Architecture, loss, and hyperparameters used for the reported results
1. Module architecture and branch configurations. Thank you for identifying these inconsistencies. We will align the main text with the explicit definitions in Appendix D. The pose encoder is a six-layer Transformer (latent dimension 512, four heads, FFN width 1024); the pose projector and decoder are each two-layer MLPs. The decoder outputs 24×6D rotations, converted to rotation matrices for supervision and axis-angle for SMPL. We will correct the main-text description of the encoder as a two-layer MLP.
Rebuttal Table 6. Training configurations of PoseDLLM and PoseDLLM2D
| Setting | PoseDLLM main 3D | PoseDLLM2D |
|---|---|---|
| LoRA rank / alpha | 16 / 32 | 64 / 16 |
| Learning rate | 2×10⁻⁵ | 1×10⁻⁵ |
| Gradient accumulation | 8 | 4 |
| Precision | fp16 | bf16 |
| Output | Continuous pose decoder | Textual 2D coordinates + 3D fitting |
2. Supervision space and loss weights. Rank 16 belongs to the main 3D branch and rank 64 to PoseDLLM2D; this should not be generalized to identical settings for every task adapter. The training objective is specified in D.4 / Eq. (12) as:
Root and the remaining 23 joints use separate mean-reduced rotation-matrix MSE terms. The appendix specifies λ₁=1, λ₂=10, wᵣ=2, and wᵦ=1, with pose supervision disabled for language-only outputs. The stage-2/3 launch scripts also default to 10 and 2/1, whereas the model class defaults to root/body weights of 1/1 and permits environment overrides. We will document the run configuration rather than substitute class defaults for training settings, and correct the main-text axis-angle L2 formulation and weight of 100 to the appendix’s explicit definition; these losses are not mathematically equivalent.
3. Refinement steps and corresponding results. Finally, K=90 is the uniform default of 3×30 steps; per-task values appear in Table 17 in the submitted paper and the uniform/tuned comparison in Table 7 in the submitted paper. We will associate each result with its configuration rather than describe the default as the setting used for every task.
Minor 1–3. Thank you for noting these issues. In the revised manuscript, we will correct the use of citet/citep, the HMR2.0 spelling and per-metric best-result highlighting in Table 1, and the best/second-best highlighting and corresponding text in Table 2 of the submitted paper.
We again thank the reviewer for the time and specific suggestions devoted to improving this work. We hope these explanations address the main concerns and welcome further feedback so that we can clarify any remaining questions.
Response to Reviewer gKS6
We sincerely thank the reviewer for the positive assessment of using masked dLLMs for multimodal pose modeling, the complementary two-stage design, and the experimental results. We also appreciate the specific suggestions on controlled comparisons, evaluation fairness, and reproducibility. Below, we address each concern with the relevant experiments and protocol details, and explain how we will improve the revised manuscript.
Question 1 (also addressing W1): Matched AR controls and attribution of diffusion-decoding gains
We appreciate the suggestion on attribution and matched controls. PoseLLaVA is our primary AR baseline: PoseDLLM inherits its pose encoder, projector, and decoder architecture and initialization, and adapts a masked diffusion language backbone through this established pose interface. UniPose provides an additional AR-backbone task baseline. Our goal is to transfer existing text-dLLM editing and conditional-completion capabilities to multimodal pose tasks, rather than pretrain a dLLM or ARM from scratch; general reasoning rankings do not replace pose-task comparisons.
1. Shared pose interface with PoseLLaVA, followed by task comparisons. As specified in Appendix D.1–D.2, the systems share the pose input format, encoder architecture, projector dimensions, and decoder output dimensions. Both jointly predict 24×6D rotations from a 4096-dimensional contextual representation. Rebuttal Table 1 places these components side by side.
Rebuttal Table 1. Pose interfaces of PoseLLaVA and PoseDLLM
PoseLLaVA therefore provides the most direct comparison using the same type of pose interface. Shared architecture and inherited initialization do not imply identical fine-tuned weights, pretraining, or training settings. Rebuttal Table 2 summarizes estimation, generation, and SMPL-input editing from Tables 6 and 8 in the submitted paper, retaining coarse and full outputs to distinguish the first stage from continuous refinement.
Rebuttal Table 2. Pose estimation, generation, and SMPL-input editing (MPJPE / PA-MPJPE, mm↓)
SMPL-input editing directly tests the conditional modification central to our work. The source pose supplies the body state and the text specifies the change, without image encoding. Coarse editing MPJPE is 70.5/61.0 mm, close to PoseLLaVA’s 71.7/61.6 mm and below UniPose’s 161.6/172.5 mm. The full system reaches 53.4/53.9 mm, reducing MPJPE by 18.3/7.7 mm relative to PoseLLaVA. Coarse generation also achieves lower MPJPE than either baseline, at 108.8 mm. These results support effective adaptation of a masked model to pose generation and editing through an established interface; removing image encoding does not control all remaining backbone differences.
Image-conditioned editing also includes the same primary baselines. Table 8 in the submitted paper reports:
Rebuttal Table 3. Image-conditioned pose editing (MPJPE / PA-MPJPE, mm↓)
The full system has lower MPJPE than these baselines on both image-editing tasks. Coarse predictions do not consistently lead, however, and UniPose retains lower PA-MPJPE on Posepart-Image. The stage-wise results show the contribution of refinement to the complete system, so we state editing advantages by task, metric, and configuration.
2. Controls for refinement and decoding strategy. Appendix F / Table 18 in the submitted paper applies the same DPoser-X prior to PoseLLaVA and UniPose. With task-best settings, PoseLLaVA obtains 71.9/61.7 mm MPJPE on PoseFix/Posepart-SMPL, and UniPose obtains 161.3/172.3 mm, compared with PoseDLLM full at 53.4/53.9 mm. This comparison uses a common hyperparameter search space and shows that, under the reported settings, adding the prior to existing baselines does not reproduce the full system’s editing results. It does not by itself establish a diffusion-specific causal interaction.
To examine decoding within the same model, Appendix G / Table 19 in the submitted paper compares one-shot, random, and confidence-ranked strategies on PoseFix-SMPL without continuous refinement: