Function-Preserving Data Generation for Zero-Shot Real2Sim2Real Manipulation

Anonymous Authors
Anonymous Affiliation

Abstract

Robotic data generation is a promising paradigm for scaling robot learning without collecting large-scale real-world data. However, generating geometrically diverse yet physically valid data for contact-rich tasks remains challenging, especially when success depends on precise geometric interfaces. Standard shape augmentation methods often distort task-critical interfaces, resulting in invalid contact relationships, e.g., fit mismatches or interpenetration, rendering downstream interactions infeasible. To address these limitations, we propose a function-preserving Real-to-Sim-to-Real framework that generates synthetic demonstrations from reconstructed assets without teleoperated source trajectories. Our method augments task-relevant object geometries through constraint-guided mesh deformation, together with physically consistent transfer of task poses and collision proxies. Visual domain randomization is further applied during simulation rollouts, enabling robust zero-shot policy deployment without real-world fine-tuning. Extensive experiments in both real-world and simulation settings demonstrate that our method enables robust generalization across unseen object geometries and diverse visual conditions in contact-rich and long-horizon tasks. Our method provides a practical path toward scalable robot learning for contact-rich tasks via shape deformation.

Project Video

Core Contributions

No Teleoperated Source Demonstration
Function-Preserving Shape Augmentation
Zero-Shot Deployment
(No Real-World Data or Fine-Tuning)

Zero-Shot Real2Sim2Real

Real-World Policy Deployment

Our method can generalize to unseen geometries and visual perturbations across contact-rich and long-horizon tasks. All policy are trained exclusively on synthetic dataset without any fine-tuning

Task 01 / Basic Grasping

Object Pick-Up

Source bracket and bracket test-shape variations used for object pick-up Task Suite Overview
Source + test shapes
Agent view Base view only
source_shape
Agent view Base view only
test_shape_uniform_shink
Agent view Base view only
test_shape_x_expand
Agent view Base view only
test_shape_x_shrink
Agent view Base view only
test_shape_y_expand
Agent view Base view only
test_shape_y_shrink
Agent view Base view only
test_shape_lightning

Task 02 / Contact-Rich Manipulation

Wrench-Based Screw Fastening

Source wrench and wrench test-shape variations used for screw fastening Task Suite Overview
Source + test shapes
Agent view
Wrist
source_shape
Agent view
Wrist
test_shape_y_nz
Agent view
Wrist
test_shape_ny
Agent view
Wrist
test_shape_ny_nz
Agent view
Wrist
test_shape_ny_z
Agent view
Wrist
test_shape_nz
Agent view
Wrist
test_shape_nz
Agent view
Wrist
test_shape_y_z
Agent view
Wrist
test_shape_origin_lightning_disturbance
Agent view
Wrist
test_shape_y_hand_disturbance
Agent view
Wrist
test_shape_y_nz_hand_disturbance

Task 03 / Contact-Rich Manipulation

Assembly

Source assembly part and assembly test-shape variations Task Suite Overview
Source + test shapes
Agent view
Wrist
source_shape
Agent view
Wrist
test_shape_long
Agent view
Wrist
test_shape_minus_y_long
Agent view
Wrist
test_shape_minus_y_short
Agent view
Wrist
test_shape_short
Agent view
Wrist
test_shape_y_longer
Agent view
Wrist
test_shape_y_short
Agent view
Wrist
test_shape_long_lighting

Task 04 / Contact-Rich Manipulation

Single-Gear Extraction

Source gear and gear test-shape variations used for single-gear extraction Task Suite Overview
Source + test shapes
Agent view
Wrist
source_shape
Agent view
Wrist
test_shape_large
Agent view
Wrist
test_shape_largest
Agent view
Wrist
test_shape_small
Agent view
Wrist
test_shape_smaller
Agent view
Wrist
test_shape_smallest
Agent view
Wrist
test_shape_lightingAndHand
Agent view
Wrist
test_shape_lightingAndHand
Agent view
Wrist
test_shape_smaller_lighting
Agent view
Wrist
test_shape_smallest_lighting

Task 05 / Long-Horizon Execution

Sequential Extraction of Three Gears

Gear test-shape variations used for sequential three-gear extraction Task Suite Overview
Source + test shapes
Agent view
Wrist
Long-horizon Gear Extraction

Simulation Environment / Data Generation

Simulation Results

Our FPSA preserves task-relevant object function during shape variation, enabling reliable data generation for contact-rich manipulation. In contrast, axis-aligned scaling can distort functional geometry and contact relationships, frequently producing unsuccessful demonstrations on contact-sensitive tasks.

Methodology

Method overview

Method Overview. Overview of our function-preserving Real2Sim2Real pipeline. Geometry-aware Real2Sim reconstructs and aligns the background mesh \(M_{\mathrm{sce}}\) and object meshes \(M_{\mathrm{obj}}\). Starting from \(M_{\mathrm{obj}}\), FPSA generates diverse, functionally valid geometries \(M_d\) by applying slippage-preserving reshaping for stretching and as-rigid-as-possible (ARAP) deformation for bending, while preserving annotated task-critical interfaces, mesh topology, and vertex correspondences. These correspondences enable the consistent transfer of task poses \(G_d\) and collision proxies \(C_d\), yielding augmented asset bundles \((M_d, G_d, C_d)\) that maintain task-critical contact relations. The augmented assets are then used across parallel, domain-randomized simulation environments to generate a synthetic dataset \(\mathbb{D}\) for policy training. Finally, the policy \(\pi_{\theta}\) enables zero-shot deployment across unseen object shapes and visual perturbations.