Function-Preserving Data Generation for Zero-Shot Real2Sim2Real Manipulation

Anonymous Authors
Anonymous Affiliation

Abstract

Robotic data generation is a promising paradigm for scaling robot learning without collecting large-scale real-world data. However, generating geometrically diverse yet physically valid data for contact-rich tasks remains challenging, especially when success depends on precise geometric interfaces. Standard shape augmentation methods often distort task-critical interfaces, resulting in invalid contact relationships, e.g., fit mismatches or interpenetration, rendering downstream interactions infeasible. To address these limitations, we propose a function-preserving Real-to-Sim-to-Real framework that generates synthetic demonstrations from reconstructed assets without teleoperated source trajectories. Our method augments task-relevant object geometries through constraint-guided mesh deformation, together with physically consistent transfer of task poses and collision proxies. Visual domain randomization is further applied during simulation rollouts, enabling robust zero-shot policy deployment without real-world fine-tuning. Extensive experiments in both real-world and simulation settings demonstrate that our method enables robust generalization across unseen object geometries and diverse visual conditions in contact-rich and long-horizon tasks. Our method provides a practical path toward scalable robot learning for contact-rich tasks via shape deformation.

Project Video

Core Contributions

No Teleoperated Source Demonstration
Function-Preserving Shape Augmentation
Zero-Shot Deployment
(No Real-World Data or Fine-Tuning)

Zero-Shot Real2Sim2Real

Real-World Policy Deployment

Our method can generalize to unseen geometries and visual perturbations across contact-rich and long-horizon tasks. All policies are trained exclusively on synthetic datasets without any fine-tuning.

Task 01 / Basic Grasping

Object Pick-Up

Task 02 / Contact-Rich Manipulation

Wrench-Based Screw Fastening

Task 03 / Contact-Rich Manipulation

Assembly

Task 04 / Articulated Object Manipulation

Single-Gear Extraction

Task 05 / Long-Horizon Execution

Sequential Extraction of Three Gears

Simulation Environment / Data Generation

Simulation Results

Our FPSA preserves task-relevant object function during shape variation, enabling reliable data generation for contact-rich manipulation. In contrast, axis-aligned scaling can distort functional geometry and contact relationships, frequently producing unsuccessful demonstrations on contact-sensitive tasks.

Methodology

Method overview

Method Overview. Overview of our function-preserving Real2Sim2Real pipeline. Geometry-aware Real2Sim reconstructs and aligns the background mesh \(M_{\mathrm{sce}}\) and object meshes \(M_{\mathrm{obj}}\). Starting from \(M_{\mathrm{obj}}\), FPSA generates diverse, functionally valid geometries \(M_d\) by applying slippage-preserving reshaping for stretching and as-rigid-as-possible (ARAP) deformation for bending, while preserving annotated task-critical interfaces, mesh topology, and vertex correspondences. These correspondences enable the consistent transfer of task poses \(G_d\) and collision proxies \(C_d\), yielding augmented asset bundles \((M_d, G_d, C_d)\) that maintain task-critical contact relations. The augmented assets are then used across parallel, domain-randomized simulation environments to generate a synthetic dataset \(\mathbb{D}\) for policy training. Finally, the policy \(\pi_{\theta}\) enables zero-shot deployment across unseen object shapes and visual perturbations.