acceptodds
Under review as a conference paper at ICLR 2027

Bridging Semantic Reasoning and Geometric Refinement with Diffusion Language Models for Multimodal 3D Human Pose

Abstract

Multimodal 3D human pose modeling must jointly interpret images, language, and poses while producing anatomically coherent 3D configurations. Existing autoregressive models struggle to preserve coupled joint constraints under causal decoding, whereas continuous pose diffusion priors improve geometric plausibility but lack language grounding. Masked discrete diffusion language models (dLLM) offer a promising alternative through bidirectional context and iterative revision, which are naturally suited to infilling and constrained modification. Motivated by this property, we propose PoseDLLM, a unified discrete-to-continuous diffusion framework that decomposes pose prediction into two complementary stages. DLLM serves as the semantic engine, leveraging bidirectional attention and holistic parallel denoising to handle multimodal conditions across all task configurations. A continuous diffusion prior then refines the coarse prediction toward the biomechanically plausible pose manifold via test-time optimization. Extensive experiments on Human3.6M, PoseScript, PoseFix, and Posepart benchmarks demonstrate that PoseDLLM consistently outperforms autoregressive LLM baselines across all task configurations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Pose Modeling, Discrete Diffusion Language Models, Diffusion-based Pose Priors

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gKS6OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents PoseDLLM, a discrete-to-continuous diffusion framework for multimodal 3D human pose modeling. The main idea is to use a masked discrete diffusion language model for multimodal semantic reasoning and coarse pose prediction, exploiting bidirectional attention and iterative remasking to better handle coupled pose constraints, followed by a pretrained continuous diffusion pose prior for geometric refinement. The framework supports pose estimation, text-to-pose generation, pose captioning, pose-difference captioning, and language-guided pose editing under a shared instruction-following interface. Experiments on Human3.6M, PoseScript, PoseFix, and Posepart show substantial improvements over autoregressive pose-language baselines, particularly for text-conditioned generation and pose editing.

Critical strengths
  1. The paper presents an interesting use of masked discrete diffusion language models for multimodal 3D pose modeling.
  2. The discrete semantic reasoning and continuous geometric refinement stages are conceptually complementary and form a simple, coherent coarse-to-fine framework.
  3. The writing is easy to follow, and the experimental results look good.
Critical weaknesses
  1. The contribution of masked diffusion itself is not fully disentangled from the pretrained backbone capabilities. PoseDLLM adopts LLaDA-V-8B, which already exhibits stronger general reasoning performance than the LLaVA-based AR baselines. Although the remasking ablation supports iterative decoding, it remains unclear how much of the improvement over AR models comes from the decoding paradigm rather than the stronger pretrained backbone.
  2. The retrieval evaluation raises a fairness concern. Each method uses its own learned pose encoder to compute R@K, making the results dependent on both the generated poses and the quality of the learned representations. While this protocol may be appropriate for evaluating representation capabilities, I do not think following PoseLLaVA is sufficient justification for using these results to support claims of superior pose generation and semantic grounding. The paper should clearly distinguish retrieval and representation performance from pose generation accuracy.
  3. I noticed several inconsistencies between the main paper and the appendix that raise concerns about reproducibility. Specifically, the LoRA rank differs (64 vs. 16), the pose encoder is described as a two-layer MLP versus a six-layer Transformer, and the pose regression objective differs in both formulation (axis-angle L2 vs. rotation-matrix MSE) and loss weight (100 vs. 10).

Minor problems:

  1. Several citations use \citet where \citep would be more appropriate, particularly when references are not grammatical components of the sentence.
  2. In Table 1, the best result of MPJPE seems to be the HRM2.0?
  3. Also in Table 2, some of the best and second-best values are not properly highlighted.
Questions for authors
  1. Could the authors provide a controlled AR baseline with matched model capacity, pose decoder, and training settings to directly verify the benefit of bidirectional iterative decoding?
  2. Could the authors reevaluate R@K using a shared frozen pose encoder and identical retrieval galleries to isolate pose generation quality from representation differences?
  3. Could the authors clarify which architecture, loss formulation, and hyperparameters were actually used for the reported experiments?
5 comments

Response to Reviewer gKS6

We sincerely thank the reviewer for the positive assessment of using masked dLLMs for multimodal pose modeling, the complementary two-stage design, and the experimental results. We also appreciate the specific suggestions on controlled comparisons, evaluation fairness, and reproducibility. Below, we address each concern with the relevant experiments and protocol details, and explain how we will improve the revised manuscript.


Question 1 (also addressing W1): Matched AR controls and attribution of diffusion-decoding gains

We appreciate the suggestion on attribution and matched controls. PoseLLaVA is our primary AR baseline: PoseDLLM inherits its pose encoder, projector, and decoder architecture and initialization, and adapts a masked diffusion language backbone through this established pose interface. UniPose provides an additional AR-backbone task baseline. Our goal is to transfer existing text-dLLM editing and conditional-completion capabilities to multimodal pose tasks, rather than pretrain a dLLM or ARM from scratch; general reasoning rankings do not replace pose-task comparisons.

1. Shared pose interface with PoseLLaVA, followed by task comparisons. As specified in Appendix D.1–D.2, the systems share the pose input format, encoder architecture, projector dimensions, and decoder output dimensions. Both jointly predict 24×6D rotations from a 4096-dimensional contextual representation. Rebuttal Table 1 places these components side by side.

Rebuttal Table 1. Pose interfaces of PoseLLaVA and PoseDLLM

ComponentPoseLLaVAPoseDLLM
Pose input72 axis-angle → 24×6D72 axis-angle → 24×6D
Pose encoder6-layer Transformer; 512; 4 heads; FFN 10246-layer Transformer; 512; 4 heads; FFN 1024
Projector512 → 4096 → 4096512 → 4096 → 4096
Pose decoder4096 → 4096 → 1444096 → 4096 → 144

PoseLLaVA therefore provides the most direct comparison using the same type of pose interface. Shared architecture and inherited initialization do not imply identical fine-tuned weights, pretraining, or training settings. Rebuttal Table 2 summarizes estimation, generation, and SMPL-input editing from Tables 6 and 8 in the submitted paper, retaining coarse and full outputs to distinguish the first stage from continuous refinement.

Rebuttal Table 2. Pose estimation, generation, and SMPL-input editing (MPJPE / PA-MPJPE, mm↓)

MethodEstimationGenerationPoseFix-SMPLPosepart-SMPL
PoseLLaVA71.7 / 51.2169.7 / 111.071.7 / 52.161.6 / 42.3
UniPose69.2 / 41.8148.4 / 93.2161.6 / 97.7172.5 / 95.2
PoseDLLM, coarse87.8 / 44.3108.8 / 71.670.5 / 50.861.0 / 43.1
PoseDLLM, full52.7 / 37.2101.3 / 66.453.4 / 39.353.9 / 37.2

SMPL-input editing directly tests the conditional modification central to our work. The source pose supplies the body state and the text specifies the change, without image encoding. Coarse editing MPJPE is 70.5/61.0 mm, close to PoseLLaVA’s 71.7/61.6 mm and below UniPose’s 161.6/172.5 mm. The full system reaches 53.4/53.9 mm, reducing MPJPE by 18.3/7.7 mm relative to PoseLLaVA. Coarse generation also achieves lower MPJPE than either baseline, at 108.8 mm. These results support effective adaptation of a masked model to pose generation and editing through an established interface; removing image encoding does not control all remaining backbone differences.

Image-conditioned editing also includes the same primary baselines. Table 8 in the submitted paper reports:

Rebuttal Table 3. Image-conditioned pose editing (MPJPE / PA-MPJPE, mm↓)

MethodPoseFix-ImagePosepart-Image
PoseLLaVA88.8 / 63.2189.4 / 112.1
UniPose109.6 / 70.0110.4 / 67.0
PoseDLLM, coarse103.2 / 74.5121.8 / 88.3
PoseDLLM, full86.1 / 62.8106.1 / 79.1

The full system has lower MPJPE than these baselines on both image-editing tasks. Coarse predictions do not consistently lead, however, and UniPose retains lower PA-MPJPE on Posepart-Image. The stage-wise results show the contribution of refinement to the complete system, so we state editing advantages by task, metric, and configuration.

2. Controls for refinement and decoding strategy. Appendix F / Table 18 in the submitted paper applies the same DPoser-X prior to PoseLLaVA and UniPose. With task-best settings, PoseLLaVA obtains 71.9/61.7 mm MPJPE on PoseFix/Posepart-SMPL, and UniPose obtains 161.3/172.3 mm, compared with PoseDLLM full at 53.4/53.9 mm. This comparison uses a common hyperparameter search space and shows that, under the reported settings, adding the prior to existing baselines does not reproduce the full system’s editing results. It does not by itself establish a diffusion-specific causal interaction.

To examine decoding within the same model, Appendix G / Table 19 in the submitted paper compares one-shot, random, and confidence-ranked strategies on PoseFix-SMPL without continuous refinement:

Rebuttal Table 4. One-shot and iterative decoding on PoseFix-SMPL

StrategyMPJPE↓ (mm)R@1↑ (%)
One-shot76.571
Random unmasking96.437
Confidence-ranked70.576

3. Matched causal/masked adaptation. PoseLLaVA is the primary method-level AR baseline, with UniPose as an additional comparison; these are not fully matched training controls. Our additional causal/masked adaptation experiment fixes LLaDA initialization, model capacity, pose modules, training data, and training budget. Both arms share diffusion pretraining and complete-template readout, so this does not independently establish pretraining-paradigm or iterative-decoding advantages. The results include seed variation and adverse outcomes; independent inference controls likewise show no universal iteration advantage.

4. PoseDLLM2D representation-pathway comparison. Appendix B/Tables 10 and 12 in the submitted paper use the same LLaDA-V backbone to generate 2D coordinate text followed by 3D fitting. PoseFix-SMPL MPJPE is 134.2 mm, compared with 70.5 mm for the main continuous-readout coarse output and 53.4 mm for the full system. This informs the choice of the main 3D interface, while differing supervision and fitting prevent it from substituting for a matched AR training control.


Question 2 (also addressing W2): R@K reevaluation with a shared frozen encoder and identical galleries

1. Geometric mapping shared by all methods. Thank you for the suggestion on evaluation fairness. The shared evaluator is a fixed SMPL geometric mapping, not a pretrained neural pose encoder. All methods use SMPL_NEUTRAL.pkl with β=0 and the fixed J_regressor_h36m.npy to obtain 17 joints, subtracting each pose’s root position. These assets are not trained or updated for individual methods. We will correct the method-specific embedding description in Appendix E and Eqs. (14)–(15).

2. Shared galleries and distance ranking. With F denoting this mapping, candidates in the same task-specific gallery are ranked by the mean Euclidean joint distance below. R@K is the percentage retrieving the correct target within K=1/5/10. Ranking uses neither cosine similarity nor Procrustes alignment.

d(i,j)=117∑k=117∥F(θ^i)k−F(θj∗)k∥2.d(i,j)=\frac{1}{17}\sum_{k=1}^{17}\left\lVert F(\widehat{\theta}_i)_k-F(\theta_j^{*})_k\right\rVert_2.

Rebuttal Table 5. Shared geometric retrieval protocol across methods

Evaluation componentSpecification shared by all methods
Shared evaluatorNeutral SMPL mesh-to-H36M17 joint mapping
Fixed assetsSMPL_NEUTRAL.pkl + J_regressor_h36m.npy
Shape / coordinatesβ=0; subtract each pose’s root joint
GalleryIdentical candidate pool within each task
RankingMean Euclidean error over 17 joints; ascending MPJPE
Learned retrieval encoderNone; no method-specific or shared neural embedding checkpoint
R@KCorrect target in the top K; K=1,5,10

3. Shared neural encoders versus geometric retrieval. A shared frozen neural encoder would define a different common feature space; this clarifies the existing geometric protocol rather than claiming a new encoder-based reevaluation. We will distinguish geometric error, target retrieval, and instruction satisfaction: R@K alone does not establish complete semantic correctness and must be interpreted alongside task results and Appendix H’s failure analysis.

Question 3 (also addressing W3): Architecture, loss, and hyperparameters used for the reported results

1. Module architecture and branch configurations. Thank you for identifying these inconsistencies. We will align the main text with the explicit definitions in Appendix D. The pose encoder is a six-layer Transformer (latent dimension 512, four heads, FFN width 1024); the pose projector and decoder are each two-layer MLPs. The decoder outputs 24×6D rotations, converted to rotation matrices for supervision and axis-angle for SMPL. We will correct the main-text description of the encoder as a two-layer MLP.

Rebuttal Table 6. Training configurations of PoseDLLM and PoseDLLM2D

SettingPoseDLLM main 3DPoseDLLM2D
LoRA rank / alpha16 / 3264 / 16
Learning rate2×10⁻⁵1×10⁻⁵
Gradient accumulation84
Precisionfp16bf16
OutputContinuous pose decoderTextual 2D coordinates + 3D fitting

2. Supervision space and loss weights. Rank 16 belongs to the main 3D branch and rank 64 to PoseDLLM2D; this should not be generalized to identical settings for every task adapter. The training objective is specified in D.4 / Eq. (12) as:

L=λ1LdLLM+λ2(wrLroot+wbLbody).\mathcal{L}=\lambda_1\mathcal{L}_{\mathrm{dLLM}}+\lambda_2\bigl(w_r\mathcal{L}_{\mathrm{root}}+w_b\mathcal{L}_{\mathrm{body}}\bigr).

Root and the remaining 23 joints use separate mean-reduced rotation-matrix MSE terms. The appendix specifies λ₁=1, λ₂=10, wᵣ=2, and wᵦ=1, with pose supervision disabled for language-only outputs. The stage-2/3 launch scripts also default to 10 and 2/1, whereas the model class defaults to root/body weights of 1/1 and permits environment overrides. We will document the run configuration rather than substitute class defaults for training settings, and correct the main-text axis-angle L2 formulation and weight of 100 to the appendix’s explicit definition; these losses are not mathematically equivalent.

3. Refinement steps and corresponding results. Finally, K=90 is the uniform default of 3×30 steps; per-task values appear in Table 17 in the submitted paper and the uniform/tuned comparison in Table 7 in the submitted paper. We will associate each result with its configuration rather than describe the default as the setting used for every task.

Minor 1–3. Thank you for noting these issues. In the revised manuscript, we will correct the use of citet/citep, the HMR2.0 spelling and per-metric best-result highlighting in Table 1, and the best/second-best highlighting and corresponding text in Table 2 of the submitted paper.


We again thank the reviewer for the time and specific suggestions devoted to improving this work. We hope these explanations address the main concerns and welcome further feedback so that we can clarify any remaining questions.

Reviewer_ccJmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PoseDLLM, a two-stage framework for multimodal 3D human pose modeling. A masked discrete diffusion language model (dLLM) first produces a coarse SMPL pose from images, text, or an input pose. A frozen DPoser-X prior then refines that pose with a joint-space task loss, a pose-space semantic anchor, and a diffusion prior regularizer. The paper evaluates the model on Human3.6M, PoseScript, PoseFix, and Posepart, covering pose estimation, text-to-pose generation, captioning, and language-guided adjustment. The reported results are strong on several MPJPE and retrieval metrics, especially for SMPL-input adjustment.

Critical strengths
  • The coarse-to-fine decomposition is easy to follow. The dLLM handles multimodal semantic conditioning, while DPoser-X provides a separate continuous pose refinement stage.
  • The model is tested across several pose tasks and input modalities under one instruction-following interface. This gives the proposed design a reasonably broad empirical scope.
  • The ablations distinguish the discrete stage from continuous refinement, and the paper includes failure cases that expose weaknesses in body-relative spatial reasoning and occluded-limb editing.
Critical weaknesses
  1. The main theoretical interpretation is unsupported. The paper describes the DPoser-X denoiser as projecting a coarse pose onto the biomechanically plausible pose manifold (Sections 3.4 and D.8), but Eq. (7) does not establish such a projection. Even with an optimal denoiser, the expression in Eq. (7) gives a posterior mean in general. For a simple counterexample, let the pose manifold be the unit circle with a uniform prior and isotropic Gaussian corruption. The posterior mean lies strictly inside the circle for any finite noise level, so it is not a projection onto the manifold. The semantic anchor in Eq. (8) has a related problem: an axis-angle penalty controls parameter distance, not semantic properties such as left/right identity or the direction of an edit. A change from +ϵ+\epsilon to −ϵ-\epsilon flips a sign-defined semantic attribute while incurring an arbitrarily small anchor cost.

  2. The refinement objective does not provide a direct signal for correcting semantic errors. The joint-space self-consistency loss in Eq. (9) matches the refined joints to those of the model's own coarse prediction, and the pose-space semantic anchor keeps the parameters close to that prediction. Appendix D.8 states that the refinement receives no image, text, or ground-truth signal. If the coarse prediction assigns a limb to the wrong side or misreads an edit direction, the objective gives DPoser-X no image or instruction signal for identifying the intended pose. The paper describes this stage as correcting geometric artifacts while preserving semantic intent, so its objective does not establish that it can recover from semantic errors in the coarse prediction. The paper attributes the Human3.6M MPJPE reduction from 87.8 to 52.7 mm to correcting local rotation errors and axis-angle discontinuities under the joint-preservation constraint. It does not isolate this mechanism.

  3. The method description is internally inconsistent and prevents reproduction. Main-text Eq. (6) uses an axis-angle regression loss with λ2=100\lambda_2=100, whereas Appendix D.4 describes a weighted rotation-matrix loss with λ2=10\lambda_2=10, wr=2w_r=2, and wb=1w_b=1. The main text calls both the pose encoder and decoder two-layer MLPs, while Appendix D.2 specifies a six-layer Transformer pose encoder. The stated LoRA rank is 64 in the main text but 16 in Table 15.

  4. The R@K evaluation is not comparable across methods. In Appendix E, each method embeds predictions and gallery poses with its own pose encoder. These embedding spaces are learned separately, so a high R@K can reflect the geometry of a method's encoder rather than the quality of the predicted pose. In the extreme, an encoder could memorize gallery identities and map each method output close to its paired target, producing high R@1 without improving MPJPE.

Questions for authors

See weaknesses

4 comments

Response to Reviewer ccJm

We sincerely thank the reviewer for the careful and constructive assessment, and for recognizing the clear coarse-to-fine design, the breadth of tasks and modalities, and the ablation and failure analyses. Your comments on the theoretical interpretation, implementation consistency, and retrieval protocol help us present the method more precisely. Below, we respond point by point using the formulation and existing experiments, and specify the corrections and clarifications we will make.


Comment 1 (W1): Validity of the projection interpretation and semantic-preservation guarantees

1. Correcting the projection interpretation. Thank you for the precise distinction concerning projection. The continuous stage is defined by Eqs. (8)–(10), without requiring an exact nearest-point projection. Equation (7) supplies a stop-gradient target for prior regularization, combined with joint self-consistency and a pose anchor. We will replace “projection” with “prior-guided continuous refinement” to match the implemented objective.

2. Role and semantic limits of the anchor. The anchor references the first-stage prediction conditioned on the image, instruction, and/or source pose—the predicted post-edit target for editing. Joint self-consistency penalizes 3D displacement, the anchor penalizes parameter changes, and the prior supplies pose-distribution constraints. Parameter proximity does not guarantee left/right identity or edit direction. We will therefore replace “ensures semantic preservation” with “encourages consistency with the condition-dependent coarse prediction.”

Rebuttal Table 1. Roles of the refinement objective terms

TermRoleScope
Prior regularizerSupplies learned pose constraintsDenoised estimate supplies a regularization signal
Joint self-consistencyPenalizes joint-position changes from coarse3D joint space
Pose anchorPenalizes parameter deviations from the conditional predictionPose-parameter space

Comment 2 (W2): Correcting semantic errors without image or instruction input

1. Refinement inputs and optimized variables. We appreciate the request to clarify the information available to refinement and its limits. The continuous stage is image-blind by design. As described in Section 3.1 and Appendix D.8, the first stage translates multimodal conditions into a target pose, and the second stage uses the human-pose prior to refine its continuous geometry. The coarse pose supplies both the initialization and the persistent reference for joint self-consistency and the pose anchor. Refinement therefore remains tied to the predicted target for the current sample, through a common pose interface supporting image-based estimation, text-to-pose generation, and language-guided editing.

Self-consistency permits constrained geometric adjustment. Both proximity terms are soft penalties with finite weights. At the coarse initialization their gradients are zero, while the prior term can still provide a nonzero gradient, allowing local joint adjustments while penalizing deviations from the prediction. Only the 21 body joints are optimized; global orientation, shape, and camera remain fixed. No additional 2D detection or reprojection signal is introduced for HMR.

2. Stage-wise and AR-plus-prior evidence. Tables 4/9 in the submitted paper report the following coarse/full results:

Rebuttal Table 2. Performance before and after refinement across pose tasks (MPJPE / PA-MPJPE, mm↓)

TaskCoarse MPJPE / PA-MPJPE↓Full MPJPE / PA-MPJPE↓
Human3.6M87.8 / 44.352.7 / 37.2
PoseScript108.8 / 71.6101.3 / 66.4
PoseFix, SMPL70.5 / 50.853.4 / 39.3
Posepart, SMPL61.0 / 43.153.9 / 37.2

Appendix F / Table 18 in the submitted paper further applies the same DPoser-X prior to AR outputs. The table below shows MPJPE for three of these tasks; “task-best” denotes per-task hyperparameter selection within the common search space specified in the paper:

Rebuttal Table 3. Applying the same DPoser-X prior to AR baselines and PoseDLLM (MPJPE, mm↓)

Method / settingEstimation↓PoseFix-SMPL↓Posepart-SMPL↓
PoseLLaVA, baseline71.771.761.6
PoseLLaVA + DPoser-X, task-best70.571.961.7
UniPose, baseline69.2161.6172.5
UniPose + DPoser-X, task-best69.8161.3172.3
PoseDLLM, coarse87.870.561.0
PoseDLLM, full52.753.453.9

These results from the submission show that adding the same prior to existing AR outputs does not reproduce the refinement gains reported for PoseDLLM. They support studying the interaction between conditional prediction and geometric refinement, rather than attributing the complete system’s results simply to adding a generic prior. This control uses the same prior and a common search space, not identical fixed hyperparameters for every method; it does not separately isolate backbone effects or specific error types.

3. Attribution of gains and limits of semantic correction. For HMR, we will explain the improvement in terms of the implemented objective: the pose prior refines the conditional prediction under joint-space and parameter-space proximity constraints. The reported metrics do not isolate axis-angle discontinuities or a particular class of local rotation errors as the cause of the full 35.1 mm reduction, and we will revise that overly specific attribution.

The limitation concerning semantic misinterpretation is already discussed in Appendix H.2 / Figure 6. When the coarse prediction misinterprets a body-relative direction, an unconditional prior cannot recover the intended direction from pose regularization alone. This failure mode is consistent with the division of labor between the two stages. We will cross-reference this existing analysis more directly in the main text to distinguish geometric refinement from reinterpreting the instruction.

Comment 3 (W3): Inconsistent architecture, loss, and LoRA specifications

1. Pose modules and output representation. Thank you for identifying these inconsistencies. We will correct the conflicting descriptions in the main text and distinguish the pose modules, the main 3D branch, and PoseDLLM2D as specified in Appendix D. The pose interface follows PoseLLaVA, with the encoder, projector, and decoder defined as follows:

Rebuttal Table 4. Pose encoder, projector, and decoder configurations

ModuleConfigurationReference
Pose encoder / PoseTower6-layer Transformer; latent 512, 4 heads, FFN 1024; input 24×6D rotationsD.2
Pose projector2-layer MLP mapping pose features to the language hidden spaceD.2
Pose decoder2-layer MLP producing 144 outputs (24×6D rotations)D.2 / Eq. (11)

The decoder’s 6D outputs are converted to rotation matrices for supervision and to axis-angle parameters for SMPL. We will correct the main-text description that also calls the encoder a two-layer MLP.

2. LoRA configurations of the 3D and 2D branches. The main 3D branch uses LoRA rank 16; rank 64 belongs to PoseDLLM2D. Table 15 in the submitted paper and Appendix D.5 specify the respective configurations:

Rebuttal Table 5. Training configurations of PoseDLLM and PoseDLLM2D

SettingPoseDLLM, main 3D branchPoseDLLM2D
LoRA rank / alpha16 / 3264 / 16
Learning rate2×10⁻⁵1×10⁻⁵
Gradient accumulation84
Precisionfp16bf16
OutputContinuous pose decoderTextual 2D coordinates + 3D fitting

3. Supervision loss and weights. We will reconcile the training objective with the rotation-matrix supervision specified in Appendix D.4 / Eq. (12):

L=λ1LdLLM+λ2(wrLroot+wbLbody).\mathcal{L} = \lambda_1 \mathcal{L}_{\mathrm{dLLM}} + \lambda_2 \left(w_r \mathcal{L}_{\mathrm{root}} + w_b \mathcal{L}_{\mathrm{body}}\right).

The root and the remaining 23 joints are supervised using separate mean-reduced rotation-matrix MSE terms. The appendix specifies a language-loss weight of 1, a pose-loss weight of 10, and root/body weights of 2/1. Pose supervision is disabled for language-only output tasks. We will correct the main-text axis-angle L2 formulation and pose-loss weight of 100 accordingly, and distinguish the output representation from the supervision space. The axis-angle L2 and rotation-matrix MSE losses are not mathematically equivalent.

The corresponding stage-2/stage-3 launch scripts also default to a pose-loss weight of 10 and root/body weights of 2/1. The model class itself defaults to root/body weights of 1/1, which the launch environment overrides; reproducing a run therefore requires its launch configuration as well as the model-class defaults.

4. Default and task-specific refinement settings. Finally, K=90K=90 is the uniform default of 3×30 steps; per-task refinement settings are given in Table 17 in the submitted paper, and the uniform/tuned comparison appears in Table 7 in the submitted paper. We will explicitly associate each set of results with its configuration and align the descriptions across the main text, appendix, and tables.

Comment 4 (W4): Comparability of R@K with method-specific encoders

1. Shared geometric mapping. Thank you for the suggestion on evaluation fairness. The shared evaluator is a fixed SMPL geometric mapping, not a pretrained neural pose encoder. All methods use SMPL_NEUTRAL.pkl with β=0 and the fixed J_regressor_h36m.npy to obtain 17 joints, subtracting each pose’s root position. These assets are not trained or updated for individual methods. We will correct the method-specific embedding description in Appendix E and Eqs. (14)–(15).

2. Distance ranking and the definition of R@K. With F denoting this mapping, candidates in the same task-specific gallery are ranked by the mean Euclidean joint distance below. R@K is the percentage retrieving the correct target within K=1/5/10. Ranking uses neither cosine similarity nor Procrustes alignment.

d(i,j)=117∑k=117∥F(θ^i)k−F(θj∗)k∥2.d(i,j)=\frac{1}{17}\sum_{k=1}^{17}\left\lVert F(\widehat{\theta}_i)_k-F(\theta_j^{*})_k\right\rVert_2.

Rebuttal Table 6. Shared geometric retrieval protocol across methods

Evaluation componentSpecification shared by all methods
Shared evaluatorNeutral SMPL mesh-to-H36M17 joint mapping
Fixed assetsSMPL_NEUTRAL.pkl + J_regressor_h36m.npy
Shape / coordinatesβ=0; subtract each pose’s root joint
GalleryIdentical candidate pool within each task
RankingMean Euclidean error over 17 joints; ascending MPJPE
Learned retrieval encoderNone; no method-specific or shared neural embedding checkpoint
R@KCorrect target in the top K; K=1,5,10

3. Distinction from a shared neural encoder. A shared frozen neural encoder would define a different common feature space; this clarifies the existing geometric protocol rather than claiming a new encoder-based reevaluation. We will distinguish geometric error, target retrieval, and instruction satisfaction: R@K alone does not establish complete semantic correctness and must be interpreted alongside task results and Appendix H’s failure analysis.


We again thank the reviewer for the time and specific suggestions devoted to improving this work. We hope these explanations address the main concerns and welcome further feedback so that we can clarify any remaining questions.

Reviewer_L266OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PoseDLLM, a two-stage framework for multimodal 3D human pose modeling. The first stage uses the diffusion language model LLaDA-V to process images, text, or source poses and regress SMPL parameters from the hidden state of a dedicated <POSE> token. The second stage performs test-time optimization using a frozen DPoser-X pose prior. The framework supports pose estimation, text-to-pose generation, pose captioning, pose-difference captioning, and language-guided pose adjustment. Experiments cover Human3.6M, PoseScript, PoseFix, and Posepart, with stage-wise ablations, decoding-strategy comparisons, and experiments applying the same refinement procedure to autoregressive baselines.

Critical strengths

1.The framework achieves strong results across multiple pose tasks. The full model performs particularly well on text-to-pose generation and pose adjustment with SMPL inputs. For example, MPJPE on PoseScript decreases from UniPose’s 148.4 mm to 101.3 mm, while on PoseFix with SMPL inputs it decreases from PoseLLaVA’s 71.7 mm to 53.4 mm. These results demonstrate the practical value of the combined framework.

2.The experiments examine the contributions of both stages and the effect of additional optimization. Beyond removing the continuous refinement stage, Appendix F applies the same DPoser-X refinement to autoregressive baselines, and Appendix G compares different decoding strategies. These experiments help characterize the sources of the full system’s performance.

Critical weaknesses

1.The proposed mechanism linking diffusion-based language generation to improved pose reasoning is insufficiently supported. Figures 1 and 2 show pose outputs consisting only of a fixed response template and <POSE>. The entire pose is jointly regressed through an MLP from a single hidden representation; individual joints do not participate in discrete diffusion. Meanwhile, the input instructions specifying joint requirements remain fully visible. In other words, if the model repeatedly completes a fixed prefix such as “Sure, the SMPL pose of the person is,” rather than sample-specific pose descriptions, it remains unclear how changing the mask-prediction procedure for these words helps the <POSE> representation coordinate joint constraints, or how this corresponds to the claimed mechanism of progressively refining instruction-targeted joints. Consequently, the explanation in Section 3.3 lacks a clear connection to the described implementation. For autoregressive methods that also jointly regress the pose from a single token, that token can already attend to the complete input instruction; causal text generation does not directly imply sequential decisions over body joints. Although Appendix G demonstrates benefits from iterative decoding, it does not isolate the contribution of completing the fixed template. Comparisons against autoregressive models also do not disentangle the decoding paradigm from backbone differences. The evidence therefore remains insufficient to establish the claimed mechanism for coordinating multiple joint constraints.

2.The methodological contribution of the continuous refinement stage is limited, and its role is insufficiently validated. Introducing a refiner based on a division between semantic prediction and geometric correction is reasonably motivated, but this stage primarily applies the existing DPoser-X prior as post-processing. The refiner receives no language conditioning and preserves semantics only indirectly by remaining close to the initial prediction. Existing ablations mainly report overall prediction errors and retrieval metrics, without analyzing satisfaction of specific instructions, or the stability of unedited body parts before and after refinement. The additional methodological contribution of this stage is therefore not sufficiently substantiated.

3.Without continuous refinement, PoseDLLM underperforms the strongest autoregressive baselines on pose estimation and image-conditioned pose adjustment and editing. Although Appendix C attributes this limitation to the visual backbone, the effectiveness of the proposed discrete diffusion stage in providing better pose initialization remains task-dependent.

  1. The claimed synergy between discrete diffusion and continuous refinement is not fully substantiated. Appendix F shows that DPoser-X substantially improves PoseDLLM but provides little benefit to autoregressive baselines. Although the authors attribute this difference to diffusion-based decoding, they lack direct analysis of the initial pose errors and why these predictions are more amenable to geometric refinement.
Questions for authors

1.In tasks that output poses, does the response contain sample-specific pose descriptions, or only a fixed template and <POSE>? If it is a fixed template, which remasked response tokens correspond to the “stable body parts” and “instruction-targeted joints” discussed in Section 3.3? Is the <POSE> hidden state explicitly carried across denoising iterations? 2.With the same model and input conditions, how does a single forward pass with the complete fixed response template provided in advance compare with the full iterative diffusion procedure when reading out the <POSE> representation? This comparison would help establish whether iteratively generating the auxiliary text independently benefits pose prediction. 3.Could the authors provide before-and-after refinement comparisons examining the satisfaction of specific instructions and changes to unedited body parts? For example, changes in MM-Dist could be reported to assess text–pose alignment before and after refinement. This would help establish whether refinement improves geometric plausibility while preserving the intended semantics. 4.Have the authors considered or tested other human pose priors, such as VPoser, Pose-NDF, or NRDF, as alternatives to DPoser-X? Since the reported experiments use only DPoser-X, please explain the specific rationale for this choice and whether the framework depends on a diffusion-based pose prior or could also accommodate other geometric refinement methods.

3 comments

Response to Reviewer L266

We thank the reviewer for the careful assessment and for recognizing the practical value of the framework in pose generation, SMPL-input editing, and the stage-wise experiments. Your specific questions about the representation and refinement mechanisms have helped us identify where the explanation needs clarification and correction. Below, we address each concern using the submitted results and supplementary analyses, and specify the corresponding revisions.


Question 1 (also addressing W1): Fixed templates, token-to-joint correspondence, and pose-state persistence

1. Following PoseLLaVA’s established pose interface. The main 3D branch uses a fixed response format, but predicts a continuous pose from a 4096-dimensional representation conditioned on the image, instruction, and source pose. Our primary interface-level reference is PoseLLaVA: PoseDLLM follows its established pose encoder–projector–decoder interface and initialization, and both systems jointly predict 24 rotations. There is no token-to-joint assignment. Committed tokens persist across rounds, while hidden states are recomputed; neither the previous hidden vector nor the decoded pose is explicitly fed back. Inference protects a preset pose suffix and updates auxiliary response positions. The inspected legacy path reads the position preceding the predicted pose marker under shifted-label indexing, without final re-encoding. We will correct the stable-joints-first/targeted-joints-later account in Section 3.3. Shared architecture and inherited initialization do not imply identical fine-tuned weights.

We appreciate the reviewer’s important distinction that an AR joint pose readout can also access the complete input instruction; causal text generation does not imply sequential joint decisions. We will distinguish the conditional representation, training adaptation, and inference benefits, rather than use representation dimensionality alone as evidence for iteration. The interface comparison explains how both models encode and decode pose information.

Rebuttal Table 1. Pose interfaces of PoseLLaVA and PoseDLLM

ComponentPoseLLaVAPoseDLLM
Pose input72 axis-angle → 24×6D72 axis-angle → 24×6D
Pose encoder6-layer Transformer; 512; 4 heads; FFN 10246-layer Transformer; 512; 4 heads; FFN 1024
Projector512 → 4096 → 4096512 → 4096 → 4096
Pose decoder4096 → 4096 → 1444096 → 4096 → 144

PoseLLaVA is the most direct interface-level baseline because PoseDLLM follows its established pose interface; UniPose is an additional AR task baseline. The results from Tables 3, 6, and 8 in the submitted paper below support adaptation to multimodal pose tasks, without isolating pretraining differences or establishing joint-wise diffusion. Our research question is whether text-dLLM editing and conditional-completion capabilities transfer to this setting.

Rebuttal Table 2. Pose generation and SMPL-input editing (MPJPE / PA-MPJPE, mm↓)

MethodGenerationPoseFix-SMPLPosepart-SMPL
PoseLLaVA169.7 / 111.071.7 / 52.161.6 / 42.3
UniPose148.4 / 93.2161.6 / 97.7172.5 / 95.2
PoseDLLM, coarse108.8 / 71.670.5 / 50.861.0 / 43.1
PoseDLLM, full101.3 / 66.453.4 / 39.353.9 / 37.2

2. PoseDLLM2D as a complementary representation pathway. Appendix B uses the same LLaDA-V backbone to generate textual 2D joint coordinates, followed by DPoser-based 3D fitting. This bypasses the main 3D branch’s dedicated pose readout and continuous regression MLP, showing that we also examine expressing pose content directly through discrete generation. Rebuttal Table 4 under Q2 gives the submitted results. Because supervision and fitting also differ, this comparison informs the representation choice rather than independently establishing joint-wise updates or iterative-decoding benefits.

Question 2: Single-pass complete-template inference versus iterative decoding

1. One-shot versus iterative decoding. Appendix G/Table 19 in the submitted paper compares one-shot and iterative decoding on PoseFix-SMPL without continuous refinement. One-shot predicts all masked response positions in one forward pass using the same pose decoder, whereas confidence-ranked decoding performs subsequent iterations. Confidence-ranked decoding reduces MPJPE by 6.0 mm and increases R@1 by five percentage points over one-shot; random ordering shows that multiple rounds alone do not ensure improvement. This isolates the decoding-strategy comparison within the reported model; one-shot still predicts masked positions, rather than receiving a complete prefilled template.

Rebuttal Table 3. One-shot and iterative decoding on PoseFix-SMPL

StrategyMPJPE (mm ↓)R@1 (% ↑)
One-shot76.571
Random unmasking96.437
Confidence-ranked70.576

2. PoseDLLM2D as a representation-pathway comparison. PoseDLLM2D uses the same LLaDA-V backbone to generate textual 2D joint coordinates, followed by DPoser-based 3D fitting. It bypasses the main 3D branch’s dedicated pose readout and continuous regression MLP, so it provides a complementary comparison between discrete coordinate generation and continuous pose readout. The submitted results are summarized in Rebuttal Table 4; because supervision and fitting also differ, this comparison informs representation choice rather than isolating iteration alone.

Rebuttal Table 4. 2D coordinate-text and continuous 3D pose outputs (MPJPE / PA-MPJPE, mm↓)

PathwayEstimation MP / PA↓PoseFix-SMPL MP / PA↓Posepart-SMPL MP / PA↓
PoseDLLM2D: 2D coordinate text + 3D fitting428.3 / 273.2134.2 / 86.394.9 / 74.1
PoseDLLM: continuous readout, coarse87.8 / 44.370.5 / 50.861.0 / 43.1
PoseDLLM: continuous readout + refinement, full52.7 / 37.253.4 / 39.353.9 / 37.2

Question 3 (also addressing W2): Instruction satisfaction and stability of unedited parts after refinement

1. Paired evaluation and metrics. We fix inputs, instructions, and decoding, comparing coarse predictions with actively refined outputs on an evaluation subset of retained Posepart-SMPL examples. MM-Dist(pair) uses a frozen PoseFix pose-pair/text evaluator; local constraints check extractable knee/elbow requirements, and displacement measures pelvis-relative movement of text-unmentioned limbs from coarse. Refinement lowers target-pose error with a small observed change in mean text matching, but local constraint satisfaction decreases and displacement remains. This supports geometric refinement without establishing universal semantic preservation. We will add this paired analysis to the revised manuscript. This evaluation uses 32 confidence-ranked forward passes and a calibration-selected 90-step refiner, averaged over three optimization seeds. Local diagnostics use examples with extractable constraints or available limb masks; “—” marks the displacement reference. The two PoseFix conditions select the no-update setting.

2. Local diagnostics and scope of the conclusion. We appreciate the suggestion to examine specific instructions and changes outside the intended edit. The submission’s aggregate geometric and retrieval metrics do not answer this on their own. The paired analysis therefore retains both improvements and adverse local changes. We will present it as additional evidence about refinement, without interpreting a small mean MM-Dist change as preservation of every instruction.

Rebuttal Table 5. Geometry and local changes before and after refinement on Posepart-SMPL

MetricStage 1: coarseStage 2: refined
MPJPE (mm ↓)86.1185.43
PA-MPJPE (mm ↓)67.3263.07
MM-Dist(pair) (↓)1.3902591.390270
Extracted-constraint pass rate (%)65.5263.79
Unmentioned-limb displacement from coarse (mm)—31.76

Question 4: Rationale for DPoser-X and alternative pose priors

Prior selection and replaceability. The framework accommodates non-diffusion priors. DPoser-X supplies a frozen prior for continuous optimization; Tables 2–3 in the DPoser-X paper also compare VPoser, Pose-NDF, and NRDF.

Prior comparison on identical inputs. We additionally provide identical initial poses to four priors on common evaluation subsets, selecting settings on a separate calibration split. Without prior retraining, DPoser-X achieves the lowest mean MPJPE and PA-MPJPE among the compared checkpoints in all six cohorts below. This conclusion is specific to these constructed inputs and evaluation settings. Initial poses are constructed by perturbing GT local rotations and global orientation to approximate the submission’s coarse error scales; cohort labels refer to these scales. Results average three perturbation seeds. Pose-NDF uses a locally trained CMU-only checkpoint, for which calibration selected the no-update setting.

Rebuttal Table 6. Refinement with different priors from identical initial poses (MPJPE / PA-MPJPE, mm↓)

Error-scale cohortInitial poseDPoser-XVPoserNRDFPose-NDF
HMR-scale84.70 / 44.8175.76 / 35.5881.25 / 41.8481.77 / 41.2684.70 / 44.81
PoseScript-scale107.33 / 71.2290.72 / 60.28104.26 / 69.0699.89 / 65.43107.33 / 71.22
PoseFix Image-scale101.88 / 73.0478.56 / 59.4293.25 / 66.7084.86 / 62.06101.88 / 73.04
PoseFix SMPL-scale69.60 / 49.7656.66 / 41.1763.88 / 46.0063.34 / 46.7769.60 / 49.76
PosePart Image-scale121.97 / 83.3694.92 / 65.04115.58 / 78.82102.87 / 69.87121.97 / 83.36
PosePart SMPL-scale60.92 / 40.8155.03 / 37.3657.68 / 38.5257.62 / 38.4960.92 / 40.81

Additional comment (W3): Task-dependent initialization quality and evidence for two-stage synergy

Initialization quality across tasks. We will qualify advantages by task and stage: Tables 6/8 in the submitted paper report coarse HMR MPJPE of 87.8 mm versus PoseLLaVA’s 71.7 and UniPose’s 69.2, while coarse SMPL editing is closer to PoseLLaVA. Table 18 in the submitted paper uses a common prior-search space, not identical hyperparameters, and does not isolate diffusion-specific synergy. Independent retained-checkpoint decoding and refinement controls also include adverse outcomes. We will describe functional complementarity without attributing gains solely to the visual encoder or diffusion pretraining.

AR-plus-prior controls and attribution limits. In Table 18 in the submitted paper, adding the same prior gives task-best PoseFix/Posepart-SMPL MPJPE of 71.9/61.7 mm for PoseLLaVA and 161.3/172.3 mm for UniPose, compared with 53.4/53.9 mm for PoseDLLM full. These results describe differences between the reported systems and settings. We appreciate the request to distinguish initialization quality from optimization gains and will qualify the “better initialization” and “synergy” wording accordingly.


We again thank the reviewer for the time and specific suggestions devoted to improving this work. We hope these explanations address the main concerns and welcome further feedback so that we can clarify any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.