MeshPad: Interactive Sketch-Conditioned Artist-Designed Mesh Generation and Editing

Haoxuan Li1, Ziya Erkoc1, Lei Li1, Daniele Sirigatti2, Vladislav Rosov2, Angela Dai1, Matthias Nießner1
1Technical University of Munich, 2AUDI AG
Paper arXiv Video Code (Coming Soon)

MeshPad enables interactive mesh creation and editing with sketches. We decompose this complex task into two sketch-conditioned operations: addition and deletion. Top: our method allows for a user to create and modify artist-designed triangle meshes by simply drawing and editing 2D sketches, achieving intuitive and interactive 3D modeling. Bottom (left): our method generates a variety of complex yet compact meshes. Bottom (right): our interactive user interface allows users to iteratively edit the mesh, with each edit step taking a few seconds (video speed up by 3x).

Abstract

We introduce MeshPad, a generative approach that creates 3D meshes from sketch inputs.

Building on recent advances in artist-designed triangle mesh generation, our approach addresses the need for interactive mesh creation. To this end, we focus on enabling consistent edits by decomposing editing into ‘deletion’ of regions of a mesh, followed by ‘addition’ of new mesh geometry. Both operations are invoked by simple user edits of a sketch image, facilitating an iterative content creation process and enabling the construction of complex 3D meshes. Our approach is based on a triangle sequence-based mesh representation, exploiting a large Transformer model for mesh triangle addition and deletion. In order to perform edits interactively, we introduce a vertex-aligned speculative prediction strategy on top of our additive mesh generator. This speculator predicts multiple output tokens corresponding to a vertex, thus significantly reducing the computational cost of inference and accelerating the editing process, making it possible to execute each editing step in only a few seconds.

Comprehensive experiments demonstrate that MeshPad outperforms state-of-the-art sketch-conditioned mesh generation methods, achieving more than 22% mesh quality improvement in Chamfer distance, and being preferred by 90% of participants in perceptual evaluations.

Video

Method Overview

Method Overview

We decompose mesh creation and editing into addition and deletion operations (left and right, respectively). ADD: in mesh addition, a transformer generates new mesh regions corresponding to newly added strokes (in red) in the input sketch. The generated mesh triangles are then merged with the existing mesh. DEL: in mesh deletion, we show a deletion operation applied to the mesh addition output. We erase sketch strokes to remove corresponding mesh regions, and overlay the erased strokes in red alongside untouched regions to provide more context to the deletion network. We then predict which mesh vertices correspond to regions to be deleted, and prune the mesh accordingly. After each addition or deletion operation, we automatically generate an updated sketch corresponding to the current output mesh, to enable further sketch-based editing.

Mesh Generation and Editing

With predefined addition and deletion operations, MeshPad is capable of sketch-based mesh generation (adding to empty mesh) and partial editing (first delete then add).

Input Sketch

Input Sketch

Generated Mesh

Edited Sketch

Edited Sketch

Edited Mesh

Input Sketch

Input Sketch

Generated Mesh

Edited Sketch

Edited Sketch

Edited Mesh

Interactive Mesh Creation

Applying mesh addition and deletion iteratively enables interactive mesh creation. This approach enables intuitive fine-grained control over the mesh structure and integrates seamlessly human supervision into the mesh generation process.

Below are some examples of interactive mesh creation using MeshPad. Use the slider to view intermediate results.

BibTeX

@article{li2025meshpad,
  title={MeshPad: Interactive Sketch-Conditioned Artist-designed Mesh Generation and Editing},
  author={Li, Haoxuan and Erkoc, Ziya and Li, Lei and Sirigatti, Daniele and Rozov, Vladyslav and Dai, Angela and Nie{\ss}ner, Matthias},
  journal={arXiv preprint arXiv:2503.01425},
  year={2025}
}