acceptodds
Under review as a conference paper at ICLR 2027

HumanSkill-4D: Training-Free Language Querying of Animatable Gaussian Avatars in Space and Time

Abstract

We introduce avatar referring segmentation, which uses a language query to select a region of an animatable Gaussian avatar and, when the query specifies a pose or motion, the matching intervals in its driving pose sequence. Such queries can support local editing and motion inspection of digital humans. Adapting existing language Gaussian splatting methods to avatars poses two challenges: adjacent parts and garments are hard to separate and may be referred to indirectly (region ambiguity), while fine-grained motions can look alike in RGB (motion ambiguity). To address these challenges, we propose HumanSkill-4D, a training-free framework with a multimodal large language model (MLLM). For region ambiguity, it builds a semantic memory linking the avatar’s parts and garments to their persistent Gaussian regions and appearance descriptions, from which the MLLM selects target categories, even when referred to by appearance, relation, or function. For motion ambiguity, it stores joint trajectories in a motion memory, and the MLLM composes geometric operators into a motion program that a deterministic executor runs on this memory to find matching intervals. On three human-annotated benchmarks, HumanSkill-4D outperforms prior methods in 3D and 4D referring segmentation and also in part segmentation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian splatting, human avatars, language grounding, referring segmentation, spatiotemporal grounding, motion localization, training-free, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_drd9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HumanSkill-4D, a training-free framework for language-based querying of animatable Gaussian avatars. A frozen MLLM interprets language queries and composes geometric operators to identify target regions and motion intervals. Detailed experiments demonstrate improvements in 3D and 4D referring segmentation over existing methods.

Critical strengths
  1. Clear and well-motivated methodology. The decomposition of 4D language querying into spatial and temporal grounding is intuitive and effective. The overall framework is clearly structured and easy to follow.

  2. Detailed and strong experimental results. The method gets substantial improvements over existing baselines on both 3D and 4D referring segmentation tasks. The ablation studies and experiments support the proposed design.

Critical weaknesses
  1. Limited generalization across avatar reconstruction models. The proposed framework is intended to be training-free and reusable. However, all experiments are conducted on avatars reconstructed using 3DGS-Avatar. Evaluating more frontier 3D avatar reconstruction models, such as LHM++, would help establish whether the method generalizes beyond the specific avatar representation used in the paper.

  2. Potentially unfair baseline comparisons. HumanSkill-4D relies on 3DGS-Avatar, which already uses SMPL-based geometry and pose-driven animation. However, the main 4D baselines rely primarily on RGB/video features, while HumanSkill-4D directly exploits the available SMPL joint trajectories for temporal grounding. The substantial performance gains may therefore largely reflect the use of existing pose information rather than the proposed methodology.

  3. Limited robustness evaluation. Obtaining accurate SMPL trajectories can be challenging in real-world scenarios, especially under occlusions. In practical scenarios, estimated poses may contain noise, temporal jitter, or tracking errors. Since the proposed method heavily relies on SMPL poses for temporal grounding, errors in pose estimation may affect its performance. Therefore, an evaluation of the framework's robustness to inaccurate SMPL estimates is needed.

Questions for authors
  1. The evaluation contains only 40 queries, primarily involving fine-grained motions supported by the predefined operators. Could the authors provide more failure cases and discuss the limitations of the framework?
8 comments
AuthorsOpenReview ↗

Response to Reviewer drd9 (1/4)

Thank you for your insightful review. We appreciate your recognition of our well-motivated and intuitive design and of our detailed and strong experimental results.


W1. Limited generalization across avatar reconstruction models. The proposed framework is intended to be training-free and reusable. However, all experiments are conducted on avatars reconstructed using 3DGS-Avatar. Evaluating more frontier 3D avatar reconstruction models, such as LHM++, would help establish whether the method generalizes beyond the specific avatar representation used in the paper.

Answer: Thank you for this suggestion. As you advised, we have added results with LHM++ [1] and two further avatar reconstruction models, SeqAvatar [2] and GauHuman [3], in place of 3DGS-Avatar. For each model, we only rebuild the semantic memory on its Gaussians and keep the target categories and motion programs unchanged.

Reconstruction model3D mIoU4D vIoU4D tIoU
3DGS-Avatar (default)82.447.363.1
SeqAvatar79.445.463.1
GauHuman73.342.463.1
LHM++75.834.663.1
  • Regions. The region-related scores (3D mIoU and 4D vIoU) depend on how closely each avatar aligns with the real images on which the GT masks are annotated. SeqAvatar and GauHuman, like 3DGS-Avatar, are optimized on the real images of each subject, so they align closely and stay close to 3DGS-Avatar in 4D vIoU. In contrast, LHM++ predicts the avatar in a feed-forward manner without optimization on the subject, so its avatar is slightly displaced from the real images. Its regions are then shifted from the GT masks (see the close-up in Fig. B of the revised paper), which lowers its scores, most clearly in 4D vIoU.
  • Intervals. Every model gives the same tIoU, as the motion programs run on the same SMPL poses.

Overall, even the lowest results stay far above the best compared methods in the main paper, LangSplat† (61.9 mIoU in Tab. 2) and OpenHuman4D (7.0 vIoU in Tab. 3). Therefore, these results show that HumanSkill-4D, as a training-free and reusable framework, generalizes beyond the specific avatar representation used in the paper.

Location in the revised paper: We have added these results to Supp. D.3 (L1341–1362) with Tab. G (p. 25) and Fig. B (p. 26).

Ref:
[1] Qiu et al. LHM++: An Efficient Large Human Reconstruction Model for Pose-free Images to 3D. arXiv:2506.13766, 2025.
[2] Xu et al. Sequential Gaussian Avatars with Hierarchical Motion Context. In ICCV, 2025.
[3] Hu et al. GauHuman: Articulated Gaussian Splatting from Monocular Human Videos. In CVPR, 2024.

AuthorsOpenReview ↗

Response to Reviewer drd9 (2/4)

W2. Potentially unfair baseline comparisons. HumanSkill-4D relies on 3DGS-Avatar, which already uses SMPL-based geometry and pose-driven animation. However, the main 4D baselines rely primarily on RGB/video features, while HumanSkill-4D directly exploits the available SMPL joint trajectories for temporal grounding. The substantial performance gains may therefore largely reflect the use of existing pose information rather than the proposed methodology.

Answer: Thank you for raising this concern. We would respectfully note that the main 4D baselines are the currently available methods closest to our task, language-based 4D Gaussian referring segmentation. Still, to clarify that the performance gains come from the proposed methodology rather than from the use of existing pose information, we have added two kinds of baselines that use the same SMPL joint trajectories but replace our temporal grounding. First, we use the intervals that skeleton-based temporal grounding models (MG-MotionLLM [1], MoChat-R [2], and TMR [3]) predict for each query from these SMPL trajectories. Second, we simply provide the SMPL trajectories to naive MLLMs (GPT-6 Sol and Qwen3.8-27B) in their prompts and let them predict the intervals for each query by themselves. In both cases, the regions are selected as in HumanSkill-4D, so only the temporal grounding differs from ours.

InputMLLMTemporal groundingvIoUtIoU
SMPLQwen3.8-27BMG-MotionLLM10.312.4
SMPLQwen3.8-27BMoChat-R14.018.2
SMPLQwen3.8-27BTMR13.919.6
SMPLGPT-6 SolPrompt21.728.7
SMPLQwen3.8-27BPrompt16.221.4
SMPLQwen3.8-27BHumanSkill-4D47.363.1

All of these baselines use the same SMPL input, yet they fall far below HumanSkill-4D, whether our temporal grounding is replaced with the skeleton-based models or with the naive MLLMs, even the commercial GPT-6 Sol. In contrast, once HumanSkill-4D is applied, the same Qwen3.8-27B nearly triples both scores and far exceeds the commercial MLLM. Therefore, the substantial performance gains largely reflect the proposed methodology, namely how it uses the existing pose information, rather than the use of this information itself.

Location in the revised paper: We have added these two comparisons to Sec. 5.4 (L472–501) with Tabs. 6 and 7 (p. 10).

Ref:
[1] Wu et al. MG-MotionLLM: A Unified Framework for Motion Comprehension and Generation across Multiple Granularities. In CVPR, 2025.
[2] Mo et al. MoChat: Joints-Grouped Spatio-Temporal Grounding Multimodal Large Language Model for Multi-Turn Motion Comprehension and Description. IEEE JBHI, 2026.
[3] Petrovich et al. TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis. In ICCV, 2023.

AuthorsOpenReview ↗

Response to Reviewer drd9 (3/4)

W3. Limited robustness evaluation. Obtaining accurate SMPL trajectories can be challenging in real-world scenarios, especially under occlusions. In practical scenarios, estimated poses may contain noise, temporal jitter, or tracking errors. Since the proposed method heavily relies on SMPL poses for temporal grounding, errors in pose estimation may affect its performance. Therefore, an evaluation of the framework's robustness to inaccurate SMPL estimates is needed.

Answer: Thank you for raising this point. Following your suggestion, we have evaluated the robustness of our HumanSkill-4D to more realistic SMPL estimation errors. Specifically, we replace the provided SMPL trajectories with those estimated directly from single-view RGB videos using two recent pose estimators, GVHMR [1] and Human3R [2]. Unlike the provided SMPL trajectories, these estimated trajectories naturally contain pose inaccuracies, temporal jitter, and potential tracking errors, reflecting the practical challenges highlighted in your concern.

MethodvIoUtIoU
4-LEGS1.814.1
4D LangSplat5.819.7
OpenHuman4D7.024.5
HumanSkill-4D with SMPL estimated by Human3R32.945.9
HumanSkill-4D with SMPL estimated by GVHMR37.751.0
HumanSkill-4D with provided SMPL (GT-like)47.363.1

While using estimated SMPL trajectories incurs slight performance degradation, HumanSkill-4D maintains strong temporal grounding performance despite realistic pose estimation errors. Notably, even with these imperfect trajectories, our method substantially outperforms the compared methods. These results highlight the practical effectiveness of HumanSkill-4D even when accurate SMPL trajectories are unavailable.

Location in the revised paper: We have added this evaluation to Sec. 5.4 (L502–512) with Tab. 8 (p. 10). The compared methods in the table above are from Tab. 3 (p. 7).

Ref:
[1] Shen et al. World-Grounded Human Motion Recovery via Gravity-View Coordinates. In SIGGRAPH Asia, 2024.
[2] Chen et al. Human3R: Everyone Everywhere All at Once. In ICLR, 2026.

Reviewer_ozjkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents HumanSkill-4D, a framework for language-based referring segmentation of animatable Gaussian avatars without additional task-specific training. It builds a semantic memory by fusing multi-view human parsing labels onto canonical Gaussians and associating categories with appearance descriptions, avoiding the need to learn a language field. A frozen MLLM selects target categories and compiles pose or motion conditions into executable programs composed of geometric operators adapted from PoseScript and MotionScript. These programs operate on SMPL joint trajectories to identify matching temporal intervals, which are combined with the spatial selection. The authors introduce three benchmarks based on existing human datasets and report improvements over the evaluated baselines in part segmentation and 3D/4D referring segmentation.

Critical strengths
  1. The framework is training-free.
  2. The method achieves substantial improvements over the evaluated baselines across part segmentation and 3D/4D referring segmentation, supported by both the benchmarks and supplementary video.
Critical weaknesses

The framework combines pretrained human parsing, semantic memory, and MLLM-based category selection with geometric primitives adapted from PoseScript/MotionScript and MLLM-based program compilation. While this integration is useful, the specific contribution of the proposed motion-program design remains insufficiently established. In particular, 4-LEGS and 4D LangSplat derive temporal relevance from RGB videos, whereas HumanSkill-4D explicitly evaluates conditions on driving joint trajectories; OpenHuman4D does not predict query-conditioned intervals. These comparisons therefore do not disentangle the benefit of structured motion information from that of the registry and compiler. Although model-choice ablations provide indirect evidence about compilation reliability, the category-selection and Gaussian-threshold ablations hold temporal predictions fixed. A baseline using the same driving poses, together with controlled ablations of the registry and temporal composition, would help establish the methodological contribution beyond the integration of existing components.

Questions for authors
  1. Would it be possible to include a skeleton-based temporal grounding baseline using the same driving poses, evaluated with tIoU?
  2. The referring-segmentation evaluation covers only a small set of query instances (Appendix F). A broader evaluation across language variations and motion conditions would strengthen the evidence for robustness. Could the authors add a query-generalization evaluation with new paraphrases and operator combinations absent from the in-context examples, keeping the prompt and registry fixed?
6 comments
AuthorsOpenReview ↗

Response to Reviewer ozjk (1/3)

Thank you for the detailed and constructive review. We appreciate your recognition of our training-free framework and its substantial improvements over the evaluated baselines.


W1-1. In particular, 4-LEGS and 4D LangSplat derive temporal relevance from RGB videos, whereas HumanSkill-4D explicitly evaluates conditions on driving joint trajectories; OpenHuman4D does not predict query-conditioned intervals. These comparisons therefore do not disentangle the benefit of structured motion information from that of the registry and compiler.

Answer: Thank you for raising this point. We would respectfully note that we compared with 4-LEGS and 4D LangSplat because they are the currently available methods closest to our task, language-based 4D Gaussian referring segmentation. We also compared with OpenHuman4D because, to our knowledge, it is the only open-vocabulary 4D segmentation method that specifically targets humans. We believe these comparisons show the benefit that HumanSkill-4D gains by addressing the two challenges that human avatars pose, region ambiguity (adjacent parts and garments are hard to separate) and motion ambiguity (fine-grained motions can look alike in RGB). Still, we agree that these comparisons alone are not sufficient to disentangle the benefit of structured motion information from that of the registry and compiler. Therefore, as you suggested, we have examined baselines using the same driving poses and controlled ablations of the registry and temporal composition, both of which we present below in our response to W1-2 and Q1.

AuthorsOpenReview ↗

Response to Reviewer ozjk (2/3)

W1-2. While this integration is useful, the specific contribution of the proposed motion-program design remains insufficiently established. […] A baseline using the same driving poses, together with controlled ablations of the registry and temporal composition, would help establish the methodological contribution beyond the integration of existing components.

Q1. Would it be possible to include a skeleton-based temporal grounding baseline using the same driving poses, evaluated with tIoU?

Answer: Thank you for these suggestions. Yes, we have added the baselines and the ablations you suggested, evaluated with both vIoU and tIoU. In all of them, our region selection is kept, so only the temporal predictions change.

  • Baselines using the same driving poses (a–e). We use the intervals predicted by three publicly available skeleton-based temporal grounding models (a–c: MG-MotionLLM [1], MoChat-R [2], and TMR [3]). Alternatively, we simply provide these driving poses to an MLLM in its prompt and let it predict the intervals by itself, using Qwen3.8-27B (d) or the commercial GPT-6 Sol (e).
  • Controlled ablations (f–h). We remove one part at a time from the motion programs. For the registry, we remove its state templates (f) or its change templates (g), and for temporal composition, we remove the temporal relations (h).
Driving posesState templatesChange templatesTemporal relationsTemporal groundingvIoUtIoU
(a)✓×××MG-MotionLLM10.312.4
(b)✓×××MoChat-R14.018.2
(c)✓×××TMR13.919.6
(d)✓×××MLLM prompt (Qwen3.8-27B)16.221.4
(e)✓×××MLLM prompt (GPT-6 Sol)21.728.7
(f)✓×✓✓Motion program (ablated)31.639.4
(g)✓✓×✓Motion program (ablated)16.725.1
(h)✓✓✓×Motion program (ablated)37.651.2
(i)✓✓✓✓Motion program (HumanSkill-4D, Qwen3.8-27B)47.363.1

Even with the same driving poses, all baselines (a–e) fall far below HumanSkill-4D (i). The skeleton-based models (a–c), often trained on coarse motion descriptions, return a single interval per query, whereas our queries require the exact frames of a fine-grained condition on a specific body part, and such a condition can hold more than once in a clip. Likewise, an MLLM reading the driving poses directly in its prompt remains limited, whether it is Qwen3.8-27B (d) or the commercial GPT-6 Sol (e). In contrast, with the same Qwen3.8-27B as (d), our proposed motion program (i) measures the condition at every frame and returns every matching interval, far exceeding both (d) and (e). Therefore, the improvement comes not from the driving poses alone but from the proposed motion-program design.

The ablations (f–h) further show that every part of this design contributes, as removing any single one lowers both scores, by at least 9.7 vIoU and 11.9 tIoU. Each part plays a role that the others cannot replace: within the registry, state templates identify the frames in which a pose holds and change templates identify the intervals in which a movement occurs, while temporal composition combines the outputs of these templates over time through temporal relations. Thus, the motion program reaches its full performance only when the registry and temporal composition are combined. Together with the baselines above, these ablations establish the specific contribution of the proposed motion-program design to temporal grounding of 4D queries.

Location in the revised paper: We have added the skeleton-based baselines to Sec. 5.4 (L472–482) with Tab. 6 (p. 10), the MLLM-prompt baselines to Sec. 5.4 (L483–501) with Tab. 7 (p. 10), and the ablations to Supp. D.2 (L1309–1338) with Tab. F (p. 25).

Ref:
[1] Wu et al. MG-MotionLLM: A Unified Framework for Motion Comprehension and Generation across Multiple Granularities. In CVPR, 2025.
[2] Mo et al. MoChat: Joints-Grouped Spatio-Temporal Grounding Multimodal Large Language Model for Multi-Turn Motion Comprehension and Description. IEEE JBHI, 2026.
[3] Petrovich et al. TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis. In ICCV, 2023.

AuthorsOpenReview ↗

Response to Reviewer ozjk (3/3)

Q2. The referring-segmentation evaluation covers only a small set of query instances (Appendix F). A broader evaluation across language variations and motion conditions would strengthen the evidence for robustness. Could the authors add a query-generalization evaluation with new paraphrases and operator combinations absent from the in-context examples, keeping the prompt and registry fixed?

Answer: Thank you for proposing this evaluation. As a reference for scale, we would respectfully note that the compared language-based 4D Gaussian referring methods use 15 (4-LEGS) and 9 (4D LangSplat) queries in their 4D evaluation. Nevertheless, we agree that a broader evaluation is valuable, so we have added this query-generalization evaluation with new paraphrases and operator combinations absent from the in-context examples, keeping the prompt and registry fixed:

  • New queries. We add 14 queries that require the MLLM to compose new operator combinations, absent from the in-context examples, and evaluate every method on the resulting 54 queries (Original in the table below), which are listed in Supp. F.
  • New paraphrases. We use a commercial LLM (GPT-6) to write two paraphrases of every 4D query (108 in total; Paraphrased in the table below), which keeps their wording independent of the authors, and we only verify that the paraphrases keep the annotated meaning. For example, "Select the face and neck while the head tilts backward" becomes "While the head tips back, highlight the face together with the neck." The paraphrases follow each query in Supp. F.
  • No in-context examples. We remove only the five in-context examples from the MLLM's prompt, so that no operator combination appears in an example, and keep its instructions and the registry unchanged.
MethodOriginal vIoUOriginal tIoUParaphrased vIoUParaphrased tIoU
HumanSkill-4D47.363.145.460.7
w/o in-context examples46.862.145.860.8

Even with the queries that require new operator combinations, HumanSkill-4D maintains its performance, scoring slightly higher than the 45.0 vIoU and 62.2 tIoU of the submission version, which used 40 queries. On the paraphrases, its scores stay close to those on the original queries. In our framework, the MLLM interprets the intended meaning of each query in terms of our categories and operators, so the wording of a query has little effect. Furthermore, HumanSkill-4D remains robust without the in-context examples, with only slight changes in its scores, as the examples merely illustrate the output format. These results strengthen the evidence for the robustness of our design across language variations and motion conditions.

Location in the revised paper: We have added this evaluation to Sec. 5.4 (L514–524) with Tab. 9 (p. 10), and all 54 queries with their paraphrases to Supp. F (L1782–1788) with Tabs. I and J (pp. 34–38). As every method is now evaluated on the 54 queries, we have also updated the 4D results in Tab. 3 (p. 7), Tab. 4 (p. 8) with its description (L413–415), Tab. 5 (p. 9), and Supp. Tab. E (p. 25), and the benchmark size in Supp. B.2 (L1095–1096) and Supp. C.1 (L1151 and L1168).

Reviewer_pBcHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents HumanSkill-4D, a training-free framework for selecting body or garment regions and motion intervals in animatable Gaussian avatars through natural-language queries. It maps human parsing results to persistent Gaussian regions and uses an LLM to translate motion conditions into programs executed on joint trajectories. Experiments on three benchmarks derived from ZJU-MoCap and HuMMan report substantial improvements over existing language-based segmentation methods.

Critical strengths
  1. The framework makes effective use of information available in animatable avatars. Persistent Gaussian identities support consistent region selection across frames, while joint trajectories allow motion conditions to be checked through explicit geometric rules. This provides an understandable connection between a language query and its spatial and temporal outputs.

  2. The reported improvements are substantial, and the paper includes useful controls. Spatial segmentation reaches 82.4 mIoU, compared with 61.9 for the parser-assisted LangSplat baseline, while the proposed method achieves 45.0 vIoU on the 4D evaluation. Parser-assisted baselines, language-model substitutions, and preparation-time measurements strengthen the empirical study beyond a comparison of final accuracy alone.

Critical weaknesses
  1. The practical value of the proposed task is unclear. The paper reports selection accuracy but does not demonstrate how joint region-and-time selection helps users perform a concrete editing or motion-inspection task.
  2. The temporal comparisons do not clearly isolate the benefit of the proposed method. HumanSkill-4D uses joint trajectories to compute motion intervals, whereas the main temporal baselines rely on RGB observations. The paper therefore leaves open whether the large improvement comes primarily from access to explicit motion information or from the proposed program formulation.
Questions for authors

How does the method compare with a simple GPT-based baseline that selects target regions and motion intervals using the same candidate masks and joint trajectories?

4 comments
AuthorsOpenReview ↗

Response to Reviewer pBcH (1/2)

We sincerely thank you for your thoughtful review. We appreciate your recognition of our effective use of the information in animatable avatars, the understandable connection between a query and its outputs, and the substantial improvements with useful controls.


W1. The practical value of the proposed task is unclear. The paper reports selection accuracy but does not demonstrate how joint region-and-time selection helps users perform a concrete editing or motion-inspection task.

Answer: Thank you for your comment on the practical value. To demonstrate how joint region-and-time selection helps users perform a concrete editing or motion-inspection task, we have added an example of editing many regions of a Gaussian avatar in Fig. J (p. 32) of the revised paper, assuming a user who customizes a character for a game or AR/VR. We would kindly ask you to view this figure alongside the three points below, for which we use the 3D editors RL3DEdit [1] and ViP3DE [2]:

  • (a) Editing with 3D queries. A user chooses the region to edit with a 3D query, and the edit is applied only to its selected Gaussians. Therefore, the user can accurately change each region as desired (e.g., the hair, shirt, gloves, and shoes) while the rest of the avatar is preserved, and can continue to edit other regions, which makes dense editing possible.
  • (b) Comparison with giving all edits at once. To show the benefit of this dense editing, we give all edits to the same editors in one instruction for the whole avatar. Here, the editors fail to apply the edits accurately (e.g., the skin tone also changes and the glove colors are swapped), because they have to decide by themselves where each edit applies.
  • (c) Extension with 4D queries. A user also wants to add a visual effect only when a specific motion occurs. With a 4D query, the motion is inspected at every frame, and the effect appears on the selected Gaussians only when the queried motion is detected (e.g., a lightning impact appears on the blue glove whenever "the right hand rises above the head").

Together, these points show that joint region-and-time selection helps users edit a Gaussian avatar densely, accurately, and at the right moments, with language queries that find where each edit applies and inspect when the queried motion occurs.

Location in the revised paper: We have added this example to Supp. D.4 (L1428–1446) and Fig. J (p. 32), with a reference to it in the Introduction (L060).

Ref:
[1] Wang et al. Edit in 2D, Verify in 3D: Reinforcement Learning for Multi-view Consistent Scene Editing. In ECCV, 2026.
[2] Chen et al. Fast Multi-view Consistent 3D Editing with Video Priors. In AAAI, 2026.

AuthorsOpenReview ↗

Response to Reviewer pBcH (2/2)

W2. The temporal comparisons do not clearly isolate the benefit of the proposed method. HumanSkill-4D uses joint trajectories to compute motion intervals, whereas the main temporal baselines rely on RGB observations. The paper therefore leaves open whether the large improvement comes primarily from access to explicit motion information or from the proposed program formulation.

Q1. How does the method compare with a simple GPT-based baseline that selects target regions and motion intervals using the same candidate masks and joint trajectories?

Answer: Thank you for raising this point. Following your suggestion, to isolate the benefit of the proposed method, we have compared HumanSkill-4D with a simple GPT-based baseline (Direct Prompt) that selects the target regions and motion intervals using the same candidate masks and joint trajectories. In Direct Prompt, the MLLM reads the full joint trajectories in its prompt and predicts the intervals by itself, whereas in HumanSkill-4D the same MLLM composes a motion program that an executor runs on the joint trajectories. To ensure that this comparison does not depend on a particular model, we use four MLLMs, three commercial ones (GPT-6 Sol, GPT-6 Luna, and Claude Opus 5) and an open-source one (Qwen3.8-27B).

MLLMMethodvIoUtIoUTokens (k)
GPT-6 SolDirect Prompt21.728.745.0–218.7
HumanSkill-4D46.762.516.4–16.7
GPT-6 LunaDirect Prompt15.620.145.0–218.7
HumanSkill-4D44.759.816.4–16.7
Claude Opus 5Direct Prompt15.819.849.1–233.1
HumanSkill-4D45.460.925.5–26.0
Qwen3.8-27BDirect Prompt16.221.479.5–388.1
HumanSkill-4D47.363.121.2–21.5

Although Direct Prompt reads the same full joint trajectories that HumanSkill-4D uses, it falls far below HumanSkill-4D with every MLLM, by at least 25.0 vIoU and 33.8 tIoU. These results show that the large improvement does not come primarily from access to explicit motion information but from the proposed program formulation, which also keeps the performance consistent across all four MLLMs. In addition to the low preparation cost in Tab. 5, the proposed program formulation also reduces the input tokens per query (Tokens), as the MLLM composes the motion program without directly reading the joint trajectories, whereas the tokens of Direct Prompt grow with the number of frames.

Location in the revised paper: We have added this comparison to Sec. 5.4 (L483–501) with Tab. 7 (p. 10).

Note: The scores of HumanSkill-4D with Qwen3.8-27B differ slightly from those in the submission version (45.0 vIoU and 62.2 tIoU), since every method is now evaluated on 54 queries, extended from 40 to cover more cases as requested by Reviewer ozjk (#Q2).

AuthorsOpenReview ↗

Follow-up to Reviewer pBcH

Dear Reviewer pBcH,

Thank you again for your thoughtful review.

As the discussion period closes soon (Nov 18), we would like to gently ask whether our responses and the revised paper have addressed your concerns. If so, we would be grateful if you could kindly reflect this in your assessment.

If anything remains unclear, we would be glad to clarify further.

Best regards,
The Authors

AuthorsOpenReview ↗

Final Follow-up to Reviewer pBcH

Dear Reviewer pBcH,

We apologize for writing again. As about one day remains in the discussion period, we briefly summarize our responses to your requests below, and would truly appreciate it if you could take a look.

  • W1 (practical example): We have added an avatar-editing example that shows how joint region-and-time selection helps users (Fig. J).
  • W2, Q1 (comparison with a GPT-based baseline): HumanSkill-4D outperforms the simple GPT-based baseline using the same candidate masks and joint trajectories (Tab. 7).

Thank you for your time.

Best regards, The Authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

Summary of Reviews and Responses (1/2)

We sincerely thank Reviewers pBcH, ozjk, and drd9 for their thoughtful and constructive feedback. Below, we summarize the strengths they highlighted and our responses to their key concerns. All changes are marked in red in the revised paper. Please see the supplementary video for qualitative results and the newly added Fig. J (p. 32) for a practical example of avatar editing.

Strengths highlighted by the reviewers

  • Well-motivated and intuitive methodology that decomposes 4D language querying into spatial and temporal grounding [drd9].
  • Effective use of the information in animatable avatars, selecting regions on persistent Gaussians and checking motions with explicit geometric rules on joint trajectories [pBcH].
  • Training-free framework [ozjk].
  • Substantial improvements in 3D and 4D referring segmentation [pBcH, ozjk, drd9] and part segmentation [ozjk], supported by the benchmarks and the supplementary video [ozjk].
  • Useful controls and ablations that support the design beyond final accuracy [pBcH, drd9].
  • Clear presentation that is easy to follow [ozjk, drd9].

Summary of the reviewers' key concerns

  • Source of the improvement [pBcH, ozjk, drd9]. The reviewers questioned whether the improvement comes simply from the joint trajectories. We have given the same joint trajectories to baselines without our proposed design for temporal grounding, and all of them fall far below HumanSkill-4D, which shows that the improvement comes from this design, not simply from the joint trajectories.
  • Additional ablations [ozjk]. The reviewer requested ablations of the registry and temporal composition. We have added these ablations, and each of them lowers the scores.
  • Practical value [pBcH]. The reviewer asked how joint region-and-time selection helps users in a concrete task. We have added a practical example of editing a Gaussian avatar, assuming a user who customizes a character for a game (Fig. J, p. 32).
  • Generalization across avatar reconstruction models [drd9]. The reviewer asked whether the method depends on 3DGS-Avatar. We have evaluated three other models, on which HumanSkill-4D remains effective.
  • Robustness to estimated poses [drd9]. The reviewer asked whether inaccurate SMPL estimates would undermine the method in real-world scenarios. To reflect such scenarios, we have used poses estimated from a single RGB view, with which HumanSkill-4D remains far above the compared methods. We have also added results with SAM 3D Body, including under occlusion.
  • Query generalization and evaluation scale [ozjk, drd9]. The reviewers questioned the dependence on the query wording and the in-context examples, and noted the small number of queries. We have clarified the number of queries in comparable studies and extended our benchmark. We have also evaluated HumanSkill-4D on paraphrases of every 4D query and without the in-context examples, and it remains robust in all cases.
  • Failure cases and limitations [drd9]. The reviewer requested more failure cases and a discussion of the limitations. We have clarified the scope with two representative failure cases and a discussion of each limitation with its cause in our design.
AuthorsOpenReview ↗

Summary of Reviews and Responses (2/2)

Main paper

  • Sec. 5.4 (L471–524): We have added a new subsection, "Further Analysis of 4D Referring Segmentation."
  • L472–482, Tab. 6 (p. 10): We have added skeleton-based temporal grounding baselines on the same driving poses to examine whether they can replace our proposed design for temporal grounding. [ozjk #Q1, drd9 #W2]
  • L483–501, Tab. 7 (p. 10): We have separated the contribution of our proposed design for temporal grounding from access to the joint trajectories with a simple MLLM-based baseline, including commercial MLLMs. [pBcH #W2 and #Q1, ozjk #W1, drd9 #W2]
  • L502–512, Tab. 8 (p. 10): We have checked whether HumanSkill-4D remains effective with inaccurate SMPL estimates. [drd9 #W3]
  • L514–524, Tab. 9 (p. 10): We have evaluated HumanSkill-4D with paraphrased queries and without in-context examples, to check whether it depends on the wording of a query or on the examples. [ozjk #Q2, drd9 #Q1]
  • Tab. 3 (p. 7), L413–415 with Tab. 4 (p. 8), and Tab. 5 (p. 9): We have extended the 4D benchmark and reported all 4D results on it. [ozjk #Q2, drd9 #Q1]
  • L377: We have placed vIoU before tIoU, following Tab. 3.
  • L060: We have added a reference to the practical example (Supp. Fig. J) in the Introduction. [pBcH #W1]
  • L538–539: We have added a reference to the failure cases in the Conclusion. [drd9 #Q1]
  • L544–548: We have updated the AI use statement. [ozjk #Q2]
  • References (pp. 11–15): We have added the references for the new experiments.

Supplementary material

  • L1095–1096 (Supp. B.2), L1151 and L1168 (Supp. C.1): We have updated the size of the 4D benchmark. [ozjk #Q2, drd9 #Q1]
  • Tab. E (p. 25, Supp. D.1): We have reported the 4D results of the smaller MLLMs on the extended benchmark. [ozjk #Q2, drd9 #Q1]
  • L1309–1338, Tab. F (p. 25, Supp. D.2): We have ablated the registry and temporal composition to show what each part of our proposed design contributes. [ozjk #W1]
  • L1341–1362, Tab. G (p. 25), Fig. B (p. 26, Supp. D.3): We have evaluated HumanSkill-4D on three other avatar reconstruction models to show that it does not depend on 3DGS-Avatar. [drd9 #W1]
  • L1428–1446, Fig. J (p. 32, Supp. D.4): We have presented a practical example of editing a Gaussian avatar. [pBcH #W1]
  • L1730–1740 (Supp. E): We have stated each limitation together with its cause in our design. [drd9 #Q1]
  • L1742–1750, Fig. K (p. 33, Supp. E): We have added two representative failure cases. [drd9 #Q1]
  • L1782–1788, Tabs. I and J (pp. 34–38, Supp. F): We have added the new queries and the two paraphrases of every 4D query. [ozjk #Q2, drd9 #Q1]
  • L2106–2123, Tab. N and Fig. L (p. 40, Supp. H): We have evaluated HumanSkill-4D with poses estimated from videos in which a body part is occluded. [drd9 #W3]
AuthorsOpenReview ↗

A Brief Guide to Our Responses

Dear Reviewers and Area Chair,

Thank you all for the time and care you have devoted to our paper. We know this is a busy period, and we are sincerely grateful for your efforts.

As the discussion period is coming to a close, we hope the brief guide below makes it easier to find our response to each concern:

Reviewer pBcH

  • W1 (practical value of the task): a practical avatar-editing example (Fig. J).
  • W2, Q1 (joint trajectories vs. the proposed formulation): a comparison with a simple GPT-based baseline using the same candidate masks and joint trajectories (Tab. 7).

Reviewer ozjk

  • W1, Q1 (structured motion information vs. the registry and compiler): baselines using the same driving poses (Tabs. 6 and 7) and ablations of the registry and temporal composition (Tab. F), which together isolate the contribution of the motion-program design.
  • Q2 (evaluation scale and query generalization): an extended benchmark with paraphrases and operator combinations absent from the in-context examples (Tab. 9; queries in Tabs. I and J).

Reviewer drd9

  • W1 (generalization across reconstruction models): results on three other models, including LHM++ (Tab. G).
  • W2 (fairness of the baseline comparisons): baselines using the same SMPL joint trajectories (Tabs. 6 and 7).
  • W3 (robustness to inaccurate SMPL estimates): results with poses estimated from single-view RGB videos (Tab. 8), plus additional results with SAM 3D Body and under occlusion (Tab. N and Fig. L).
  • Q1 (failure cases and limitations): two failure cases (Fig. K) and a revised discussion of the limitations (Supp. E).

We remain available throughout the discussion period and would be happy to clarify anything further. Thank you once again for your valuable feedback.

Best regards,
The Authors

AuthorsOpenReview ↗

Summary of Author-Reviewer Discussion for AC and Reviewers

Dear Area Chair and Reviewers,

What we propose. We introduce avatar referring segmentation, where a language query selects the Gaussians of an animatable avatar that it names, at the moments it describes. Our HumanSkill-4D lets a frozen MLLM select the region from a semantic memory of parser labels on the Gaussians, and compose predefined geometric operators into a program that measures the motion on the poses. By this design, it is training-free, takes minutes instead of the hours of RGB-based methods, extends to diverse motions, and outperforms prior methods in both 3D and 4D.

What we addressed during the rebuttal.

  • Source of the improvement. To examine whether the improvement comes simply from the driving poses, we have given the same poses to skeleton-based temporal grounding models and to MLLMs, including commercial ones, and all of them fall far below HumanSkill-4D. We have also ablated the registry and the temporal relations.
  • Generalization and robustness. We have evaluated three other avatar models, poses estimated from single-view videos even under occlusion, an extended benchmark, paraphrased queries, and prompts without in-context examples. HumanSkill-4D remains effective in all of them.
  • Practical value. We have added an example of editing a Gaussian avatar, in which each edit stays on its selected part and a visual effect appears only when the queried motion occurs, whereas editors given all edits at once fail to apply them accurately.
  • Failure cases and limitations. We have added two representative failure cases and stated each limitation.

Where the discussion stands. After our rebuttal, Reviewer ozjk wrote that we have "largely resolved" the concerns and deferred the final assessment to the Area Chair. Reviewer drd9 wrote that "some concerns have been addressed", and we then asked about the remaining ones and added further experiments, but received no further reply. For Reviewer pBcH, we provided every experiment requested with meaningful results, yet have not received any response. We hope that the Area Chair can take this into consideration in the final recommendation.

Thank you for the time and effort you have devoted to our paper.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.