acceptodds
Under review as a conference paper at ICLR 2027

SketchHMR: Generative Projection Guidance for Human Mesh Recovery from Freehand Sketches

Abstract

Reconstructing a 3D human mesh from a freehand sketch is complicated by the inherent sparsity and style variance of hand-drawn strokes, which offer insufficient cues regarding body surface, joint ordering, occlusion, and depth. To bypass the ambiguity of single-stage regressors that map raw strokes directly to parametric body dimensions, we propose SketchHMR, a two-stage paradigm that decouples sketch interpretation from mesh estimation via an intermediate, camera-aligned generated SMPL projection. The first stage, the core component, GeoInk, combines a spatially aligned sketch latent with DINOv2 visual features and input-anchored guidance to translate the sketch into a structured body observation. The second stage iteratively estimates SMPL pose, shape, and weak-perspective camera from the generated projection. To reduce the domain gap between clean rendered projections and generated projections, we pretrain the regressor on clean inputs and subsequently adapt it using paired clean and generated projections. On the Sketch3D dataset, SketchHMR achieves 96.60 mm MPJPE, 66.10 mm PA-MPJPE, and 115.90 mm PVE. Reducing flow sampling from 25 to 10 steps also decreases translation latency by 45.2% without degrading reconstruction accuracy. SketchHMR provides a modular and interpretable approach to sketch-based human mesh recovery by making sketch interpretation an explicit intermediate stage.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Mesh Recovery, Sketch-Based 3D Reconstruction, Generative Intermediate Representations, SMPL Parametric Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rvgpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SketchHMR, a two-stage framework for recovering a 3D human mesh from a freehand sketch. The first stage uses conditional latent flow matching to generate a camera-aligned grayscale SMPL projection. The second stage predicts SMPL pose, shape, and camera parameters using an iterative regressor adapted with paired clean and generated projections. Experiments on Sketch3D evaluate reconstruction accuracy and the effects of the projection bridge and mixed-domain adaptation.

Critical strengths
  1. Clear presentation. The paper clearly explains the motivation, two-stage pipeline, and training procedure. The explicit intermediate projection also makes the reconstruction process easier to inspect.
  2. Well-organized experiments and modest computational requirements. The experiments examine the projection bridge, mixed-domain adaptation, sampling configuration, and regressor backbone. Training GeoInk on a single RTX 4090 demonstrates a relatively accessible computational setup.
Critical weaknesses
  1. Limited methodological novelty. The framework primarily combines established components: conditional latent flow matching, pretrained visual encoders, iterative SMPL regression, and mixed-domain adaptation. Although the combination is reasonable, the paper does not sufficiently establish a distinctive methodological contribution beyond this integration.

  2. Missing comparisons with alternative two-stage pipelines. A natural alternative is to generate a human image conditioned on the sketch and then recover SMPL parameters using an existing human mesh recovery model. The paper does not compare against such a pipeline, leaving unclear whether the proposed grayscale projection provides advantages over a more conventional generated-image representation.

  3. Insufficient justification of the second-stage regressor. Given the extensive literature on human mesh recovery, the authors should compare their ResNet-34-based iterative regressor with established HMR models fine-tuned on the same clean and generated projections. Such comparisons would clarify whether the current regressor is an effective design choice and whether the gains mainly arise from the intermediate representation and training protocol rather than the regression architecture.

Questions for authors

Same as weakness.

3 comments

We thank the reviewer for the constructive feedback and for recognizing the clarity of our framework and experimental analysis.

1. Methodological Novelty

The main contribution of SketchHMR is introducing an explicit, camera-aligned SMPL projection as an intermediate geometric representation between freehand sketches and human mesh recovery.

GeoInk combines spatial sketch latents with DINOv2 semantic features and input-anchored guidance to preserve body structure during projection generation. Unlike direct regression, this intermediate representation makes body geometry observable and allows sketch interpretation errors to be distinguished from downstream regression errors.

We further introduce paired clean/generated projection adaptation to address the domain gap between rendered and generated projections. As shown in Table 2, the projection bridge reduces MPJPE from 130.06 mm to 111.76 mm, with subsequent adaptation further reducing it to 96.60 mm under the original evaluation protocol.

2. Alternative Two-Stage Frameworks

We appreciate the suggestion to consider sketch-to-RGB-to-HMR pipelines.

Our grayscale SMPL projection is specifically designed to preserve geometry-related information, including body silhouette, articulation, and relative limb configuration, while minimizing appearance-related factors such as clothing, texture, and illumination.

This provides a structured, geometry-oriented observation for SMPL regression. We agree that comparing it with alternative intermediate representations would further strengthen the evaluation.

3. Choice of HMR Regressor

We adopt a conventional iterative SMPL regressor to focus on the contribution of the intermediate representation without introducing additional architectural complexity.

Our backbone ablation (Table 4) shows that ResNet-34 achieves 96.60 mm MPJPE, compared with 102.62 mm for ResNet-18 and 100.66 mm for ResNet-50, supporting its selection within the evaluated backbone family.

The proposed intermediate representation is not inherently tied to ResNet-34, and evaluating additional HMR backends would be a useful extension.

We thank the reviewer again for the valuable suggestions, which will help us further clarify the contributions and design motivations of SketchHMR.

Thank you for the clarification. While the response provides additional motivation for the design choices, my main concerns remain unresolved. The discussion of novelty largely reiterates the existing components and ablation results, without sufficiently establishing a distinctive methodological contribution beyond their integration. The stated advantages of grayscale SMPL projections over generated RGB images remain unverified without the suggested alternative two-stage comparison. Moreover, the ResNet backbone ablation does not substitute for comparisons with established HMR models trained under the same clean/generated projection protocol. Therefore, the response does not provide sufficient additional evidence to change my assessment, and I maintain my original rating of 2 (Weak Reject).

Clarification: I write this review by myself.

Reviewer_aNwCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SketchHMR to recover 3D human meshes from sparse and ambiguous freehand sketches. It first generates a 2D body projection, then estimates SMPL parameters from that projection. The regressor is trained on both clean renders and generated projections to handle generation errors. Experiments on Sketch3D show lower reconstruction errors than the evaluated baselines.

Critical strengths

1.Recovering human meshes from freehand sketches is a useful task. The motivation for separating sketch interpretation from mesh estimation is clear, given the ambiguity of sparse and stylized strokes.

2.The intermediate projection makes reconstruction errors easier to inspect. The ablation also supports the benefit of mixed-domain adaptation, with lower errors when both variants are evaluated on the same generated projections.

Critical weaknesses

1.Most sketches shown in the paper look quite clean and regular. The first two examples in Figure 3 look closer to casual hand drawings, but there are too few of them. Please show more similar cases from Sketch3D.

2.All results come from Sketch3D. Please add OOD or in-the-wild tests with different users and drawing styles, and compare the method with the baselines through both visual examples and quantitative results.

3.My third concern is again about the input sketches. I think they look more like body outlines than sketches people would actually draw. This greatly reduces the unique challenges of sketch input, particularly sparsity and abstraction, making the task easier and the results less impressive. I suggest collecting real hand-drawn sketches paired with 3D models and using them to develop the method. Collecting these sketches is certainly expensive, so I also recommend using sketch generation methods such as CLIPasso [1], CLIPascene [2],MSFormer [3], LBS [4], and SwiftSketch [5] to produce sketches closer to human drawing styles (3D model → rendered image → sketch). Although I think this is a complete piece of work, incorporating training sketches that better reflect how people actually draw would make me more convinced of the contribution.

  • [1] Yael Vinker et al. CLIPasso: Semantically-Aware Object Sketching. ACM Transactions on Graphics (SIGGRAPH), 2022.
  • [2] Yael Vinker et al. CLIPascene: Scene Sketching with Different Types and Levels of Abstraction. ICCV, 2023.
  • [3] Zhichao Liao et al. Freehand Sketch Generation from Mechanical Components. ACM Multimedia, 2024.
  • [4] Hyundo Lee et al. Learning Geometry-Aware Representations by Sketching. CVPR, 2023.
  • [5] Ellie Arar et al. SwiftSketch: A Diffusion Model for Image-to-Vector Sketch Generation. SIGGRAPH, 2025.

4.It is unclear how much the intermediate projection itself contributes to the improvement. GeoInk also uses DINOv2 features, so the gains over direct regression may partly come from stronger visual features. Please add a direct sketch-to-mesh baseline using the same DINOv2 features and a comparable training budget.

Questions for authors

Please address the concerns above with additional qualitative and quantitative evidence in the final version.

3 comments

We sincerely thank the reviewer for the constructive comments and valuable suggestions.

1. Sketch Realism and Diversity

We would like to clarify that the sketches in Sketch3D are human-drawn rather than automatically generated. Their visual quality varies with individual drawing proficiency and style. Some exhibit clean contours and accurate proportions, while others contain irregular strokes or simplified structures. Even well-drawn sketches retain ambiguities in depth, occlusion, and body-part relationships, which our framework aims to address.

2. Generalization Across Drawing Styles

Sketch3D contains human-drawn sketches with varying visual characteristics. We agree that evaluation on independently collected sketches from unseen artists would further strengthen the assessment of cross-style generalization. We consider this an important direction for future work.

3. Contribution of DINOv2 Features

Following the reviewer's suggestion, we conducted an additional direct sketch-to-SMPL regression experiment using the same frozen DINOv2-B/14 spatial features as GeoInk, followed by a trainable spatial adapter and iterative SMPL regressor, without intermediate projection generation.

Under the same evaluation protocol, the DINOv2-based direct baseline achieves 102.15 mm MPJPE, compared with 96.60 mm for SketchHMR, representing a 5.4% reduction.

This result provides additional evidence that the performance of our framework is not solely attributable to pretrained DINOv2 features. We have incorporated this baseline into Table 2 and added the corresponding analysis in Section 4.3.

We thank the reviewer again for the insightful suggestions.

Dear Reviewer,

Thank you again for your constructive feedback. We have provided detailed responses to your concerns and incorporated an additional DINOv2-based direct regression baseline into the revised manuscript.

As the discussion period is approaching its end, we would greatly appreciate your feedback on whether our additional experiments and clarifications have addressed your main concerns.

Thank you for your time and consideration.

Reviewer_aNwCOpenReview ↗

Response to the Rebuttal

  • I appreciate the authors’ response and the additional baseline experiment. However, evaluation on unseen users and drawing styles is important for establishing the method’s practical applicability, rather than merely an optional extension. Deferring this evaluation to future work does not resolve my concern. With all reported results limited to Sketch3D, the evidence for real-world generalization remains insufficient.

  • Regarding the additional DINOv2-based direct regression experiment, please provide a detailed description of the dataset preparation and training settings.Please also include side-by-side qualitative comparisons using the same input sketches, alongside the quantitative results. These details and visual comparisons should be incorporated into the revised manuscript to improve reproducibility and provide stronger evidence for the contribution.

Reviewer_ExVpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SketchHMR, a two-stage framework for recovering 3D human meshes from freehand sketches. The first stage, GeoInk, uses a flow-matching generative model to translate input sketches into camera-aligned SMPL projections, providing a structured intermediate representation of human geometry. The second stage estimates SMPL pose, shape, and camera parameters from these generated projections using an iterative regressor. Experiments on Sketch3D demonstrate improved reconstruction accuracy over existing methods.

Critical strengths
  • Well-motivated problem formulation.

Recovering 3D human geometry from sparse and stylized freehand sketches is challenging due to missing depth, occlusion, and ambiguous body-part relationships. Introducing an intermediate SMPL projection to bridge sketch interpretation and mesh recovery is an intuitive and reasonable approach.

  • Effective two-stage design.

The proposed two-stage framework effectively decouples sketch interpretation from human mesh recovery through an intermediate SMPL projection. This design provides a more structured representation for downstream regression while improving interpretability by making intermediate errors observable. The strong experimental results further demonstrate the effectiveness of this approach.

Critical weaknesses
  • Unclear source of performance improvements.

While the proposed framework achieves substantial improvements, it is unclear whether the gains primarily come from the two-stage design or the use of a generative model. Comparisons with simpler intermediate representations or non-generative projection predictors would help isolate the contribution of each component.

  • Limited evaluation of generalization.

All experiments are conducted on Sketch3D using a random train/validation split. Given the significant variation in freehand sketch styles, evaluation on unseen drawing styles or different datasets would strengthen the generalization claims.

  • Underutilization of the generative model.

Although GeoInk is a generative model capable of producing multiple plausible projections, the framework uses only a single generated projection for downstream mesh regression. This effectively treats the generative model as a deterministic predictor, without fully exploiting its ability to represent multiple plausible interpretations of ambiguous sketches.

Questions for authors

See above

3 comments

We sincerely thank the reviewer for the constructive feedback and for recognizing the effectiveness and interpretability of our two-stage framework.

1. Source of Performance Improvements

The key motivation of SketchHMR is to introduce an explicit geometric intermediate representation that separates sketch interpretation from SMPL parameter regression.

As shown in Table 2, introducing the projection bridge reduces MPJPE from 130.06 mm to 111.76 mm compared with the original direct HMR baseline. Subsequent mixed-domain adaptation further reduces MPJPE to 96.60 mm.

These results demonstrate the effectiveness of the complete framework under our original evaluation protocol. Importantly, the two-stage formulation also makes intermediate body geometry directly observable, enabling projection-generation errors to be distinguished from downstream regression errors.

We agree that comparisons with deterministic projection predictors and simpler intermediate representations would help further isolate the contribution of generative modeling.

2. Generalization Across Sketch Styles

GeoInk is designed to handle sparse and ambiguous sketch information by combining spatially aligned sketch latents with DINOv2 semantic guidance. The resulting camera-aligned SMPL projection preserves body configuration and relative limb relationships for downstream reconstruction.

Our experiments demonstrate the effectiveness of this formulation on Sketch3D. We appreciate the suggestion to further evaluate generalization across independent artists and more diverse drawing styles.

3. Multiple Plausible Projections

Our current framework focuses on generating a structured geometric observation for downstream mesh recovery. The explicit intermediate projection makes the inferred body configuration directly inspectable.

We agree that exploiting multiple generated projections could further address the inherent ambiguity of freehand sketches. Extending GeoInk to multi-hypothesis generation and uncertainty-aware reconstruction is a valuable direction for future work.

We thank the reviewer again for the insightful suggestions.

Dear Reviewer,

Thank you again for your valuable comments. We have provided detailed responses regarding the source of performance improvements, generalization, and the use of a single generated projection.

As the discussion period is approaching its end, we would sincerely appreciate any further feedback on whether our explanations have adequately addressed your concerns.

Thank you for your time and consideration.

Thanks for the detailed response. After reading it along with other reviews, I decided to maintain my original rating. I shared the same concerns as Reviewer aNwC -- the current version of the paper is only evaluated on Sketch3D and lacks evidence for generalization on OOD data.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.