ECCV 2026
PIAvatar: Physically Interactive Avatars
via Deformation Gradient Decoupling
Sang-Hun Han1,
Min-Gyu Park2,
Jisu Shin1,
Seunghyun Shin1,
Jin-Hwi Park3, and
Hae-Gon Jeon4†
1GIST, 2KETI, 3Chung-Ang University, 4Yonsei University
†Corresponding author
We decouple the deformation gradient update from the kinematic velocity, enabling bidirectional and non-rigid avatar interactions.
Abstract
3D human avatars have shown impressive visual fidelity driven by pose-conditioned models, yet they still lack the physical ability required for interactions with each other and environments. Although recent studies have made various attempts to incorporate physical characteristics into 3D avatars, they only exhibit limited physical deformations, often leading to constrained interaction behaviors. To resolve this issue, we present PIAvatar, a framework to simultaneously enable physically aware interactions between avatar-avatar and avatar-environment, and a non-rigid deformable human body simulation. In this work, our key insight is to decouple kinematic velocity from deformation gradient. When external forces act on avatars, the kinematic velocity induces stress which hinders the avatar's ability to achieve a desired pose. In addition, we integrate a skeletal framework within the avatar. It allows estimating its poses and real-time tracking in a closed form, even during non-rigid physical interactions. Our approach is implemented within a conventional Material Point Method framework to ensure physically consistent dynamics. We lastly evaluate the method on both human-object and human-human interaction scenarios to assess its behavior under diverse interaction settings.
Method
PIAvatar is built on the Material Point Method (MPM), a particle–grid continuum framework that natively supports non-rigid deformation and bidirectional momentum transfer between an avatar and its surroundings. Driving such a simulation with user-defined motion, however, raises two challenges: (i) MPM's stress-based constitutive model generates restorative forces that counteract the applied kinematic velocity, preventing the avatar from reaching its target pose; and (ii) once the body is non-rigidly deformed, its pose can no longer be read off directly. PIAvatar resolves both in closed form. Velocity Decoupling explicitly excludes the user-defined kinematic velocity from the deformation gradient, so the avatar follows the intended motion without spurious stress while still reacting to genuine external forces. A skeletal structure embedded in the avatar then recovers the pose through a closed-form solution (Kabsch → LBS), enabling robust real-time tracking even under non-rigid physical interaction — without any per-frame optimization or learning.
Motivation
Why decoupling is necessary. In standard MPM, each particle's stress σp is derived solely from its deformation gradient Fp, and Fp encodes every deformation — including the purely kinematic motion imposed to drive a target pose. Consequently, the kinematic velocity that should simply pose the avatar (a) is accumulated into Fp (b) and mistaken for elastic strain, generating restorative stress σp (c). This pose-induced stress (⚠) pushes back against the intended motion, so the avatar cannot reach its target pose and physical interactions are distorted. PIAvatar disentangles Fp into kinematic and elastic parts and removes the kinematic velocity from the stress computation, so that stress arises only from genuine physical contact.
Qualitative Results
Our simulator reproduces the secondary soft-tissue jiggling (e.g., of the belly) that simple LBS-based avatars cannot capture.
Assigning cloth-like material properties lets the avatar wear loose garments that swing and wrinkle freely, driven by body motion and external contact.
Distinct material properties can be assigned to different body regions (e.g., hair, clothing, feet) within a single avatar, so each part deforms according to its own physical characteristics.
Forces transmitted through the avatar are visualized as changes in the deformation gradient Fp, propagating outward from the contact point.
In standard MPM, strong contact leaves a permanent dent after the ball passes (a), and single-field contact drags the belly along with the ball so it stretches instead of releasing (b). Our shape-preservation term combined with multi-field contact recovers the body shape and lets the ball detach cleanly (c).
With single-field MPM contact, the two legs fuse together once they touch and lose their individual shape (a). Our multi-field contact keeps them as distinct bodies, preserving each leg's shape while interactions proceed naturally (b).
VLM-Driven Physical Parameter Optimization
As an exploratory application, we automatically set physical parameters using a Vision-Language Model (VLM) instead of manual tuning. Since VLMs do not directly understand abstract physical quantities, we provide a text prompt that explains the physics behind the observable cues in the rendered image (e.g., a heavier ball produces a larger belly deformation). Guided by this prior knowledge, the VLM scores each rendered simulation, and Bayesian optimization uses these scores as rewards to automatically converge to physically appropriate parameters. A user study further confirms that human perception of heavy/light and stiff/soft agrees with the optimized parameters, supporting the reliability of this VLM-based reward.
BibTeX
@article{han2026piavatar,
title={PIAvatar: Physically Interactive Avatars via Deformation Gradient Decoupling},
author={Han, Sang-Hun and Park, Min-Gyu and Shin, Jisu and Shin, Seunghyun and Park, Jin-Hwi and Jeon, Hae-Gon},
year={2026}
}