Human Motion to 3D Object Generation

In this research, we focus on generating interacting objects by predicting contact points from short human motion data represented as point clouds. The key idea is to analyze the point cloud data of human motion, identify the points in contact with objects, and use this information to generate the corresponding interacting objects. This approach allows us to model realistic interactions between humans and objects in a 3D space.

Network Architecture

Our proposed InteractFormer predicts which objects a motion is in contact with by analyzing sequential point cloud data. Using a P4D block, the model interprets point cloud motion information in four dimensions, enabling accurate prediction of contact points.

In the overall network architecture, the contact points and motion information generated by the InteractFormer are passed as conditions to a diffusion model. This allows the diffusion model to generate realistic and contextually accurate interactions between the motion and the objects.

Quantitative Results

Our InteractFormer consistently outperforms the baseline models across all evaluation metrics, demonstrating its effectiveness in accurately predicting contact points. As shown in the above table, our model achieves a significant improvement in F1 score, Accuracy, and Contact-Accuracy for both the PROX and HUMANISE datasets.

For evaluation of generating interacting objects, we adopted standard point cloud comparison metrics used in previous research, LSDM, including Chamfer Distance (CD), Earth Mover’s Distance (EMD), and F1 Score, which measure the similarity between the predicted and ground truth point clouds.