JustShape: Exploring Co-Speech Gestures for Multimodal LLM-Powered 3D Parametric Modeling
Abstract
Parametric modeling is a prevailing 3D modeling approach in design, architecture, and engineering. The emergence of multimodal large language models (LLMs) brings a new opportunity to lower the entry barriers to this powerful tool. However, describing 3D geometries through natural language can be fuzzy and challenging. We introduce co-speech gesture, a natural and expressive interaction modality to complement text prompts for LLM-empowered generative parametric modeling. We first conducted an elicitation study to explore and categorize co-speech gesture expressions. Based on the findings, we designed a multimodal fusion pipeline that parametrizes gestures and synthesizes them with speech. This approach reduces language ambiguity by translating implicit user intentions into explicit parametric attributes, thus lifting the model generation performance. We conducted a two-session user study testing and comparing it with traditional language and sketch inputs. This work streamlines the parametric modeling workflow and explores novel multimodal interaction paradigms for LLM-empowered design and creation.
Shape Attributes
JustShape turns free-form "talk + gesture" into a compact, editable representation of shape: six Shape Attributes—Profile (p), Position (a), Direction (d), Scale (s), Trajectory (γ(t)), and Angle (φ). This attribute layer is our key contribution for making multimodal input computable and controllable: it captures both high-level intent (what to make) and low-level parameters (how it changes) in a shared format that supports consistent interpretation, revision, and reuse across modeling steps.
CAD Operations
Built on the attribute layer, JustShape executes real CAD modeling by mapping attributes to five core parametric operations—Extrude, Revolve, Sweep, Loft, and Fillet. Each operation is driven by a minimal set of attributes (e.g., extrude = p + d + s, revolve = p + a + d + φ), enabling the system to translate natural interaction into executable parametric commands and to keep designs iteratively editable (users can adjust attributes and re-run operations without rebuilding from scratch).
Gesture Design Space
To reliably capture these attributes in AR, we contribute a gesture design space grounded in hand landmarks: users trace a profile (p), stretch to set scale (s), move to define a trajectory (γ(t)), and twist to specify rotation (φ). By linking recurring co-speech gestures to explicit geometric variables, JustShape makes gesturing not just expressive but precise, enabling low-friction, on-the-fly specification of parametric intent that is robust enough to drive CAD operations and fast enough for rapid exploration.
BibTeX
@inproceedings{Duan2026JustShape,
title={JustShape: Exploring Co-Speech Gestures for Multimodal LLM-Powered 3D Parametric Modeling},
author={Duan, Runlin and Chen, Yuzhao and Hu, Yichen and Liu, Ziyi and Zhu, Chenfei and Hu, Xiyun and Ma, Dizhi and Wang, Xinyi and Ramani, Karthik},
booktitle={Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems},
year={2026},
pages={},
organization={ACM},
address={Barcelona, Spain}
}