SemGeo-Gen: Unsupervised Generation of Approximate
Cross-Instance Semantic-Geometric Correspondences

1Lift semantic features to 3D

Each object is rendered from calibrated virtual cameras, dense DINOv2 features are extracted in image space and lifted back to the visible 3D points, and multi-view observations are aggregated into one descriptor per point. Semantic compatibility is the cosine similarity of the descriptors after a joint 64-D PCA:

\[ s_{\mathrm{sem}}(x_i,x_j)=\frac{z_i^\top z_j}{\|z_i\|_2\,\|z_j\|_2} > \tau_{\mathrm{sem}}, \qquad \tau_{\mathrm{sem}}=0.85. \]

2Semantic-coverage pair filtering optional

A pair is kept only if, in both directions, a large enough fraction of points has a semantically compatible point in the other object:

\[ \rho_{\mathrm{sem}}(\mathcal{S}\!\rightarrow\!\mathcal{T})=\frac{1}{|\mathcal{X}_S|}\sum_{x_i^S\in\mathcal{X}_S}\mathbf{1}\Big[\max_{x_j^T\in\mathcal{X}_T} s_{\mathrm{sem}}(x_i^S,x_j^T)>\tau_{\mathrm{sem}}\Big] \;\ge\; \gamma=0.6 . \]

3Continuous piecewise-affine (CPA) registration

The source is deformed onto the target by a control grid over the normalized 3D domain, decomposed into tetrahedra by Delaunay triangulation. Each vertex has a learnable displacement \(u_v\), and points move by barycentric interpolation, so the deformation is affine inside each tetrahedron and continuous across cells. It is fit by gradient descent on a Chamfer loss with smoothness regularization:

\[ \min_{T_{\mathrm{CPA}}}\; \mathcal{L}_{\mathrm{Chamfer}}\big(T_{\mathrm{CPA}}(\mathcal{X}_S),\mathcal{X}_T\big) + \lambda_{\mathrm{smooth}}\,\frac{1}{|\mathcal{E}|}\sum_{(v_a,v_b)\in\mathcal{E}}\|u_{v_a}-u_{v_b}\|_2^2, \qquad \lambda_{\mathrm{smooth}}=0.3, \] \[ \mathcal{L}_{\mathrm{Chamfer}}(\widehat{\mathcal{X}}_S,\mathcal{X}_T)=\frac{1}{|\widehat{\mathcal{X}}_S|}\sum_{\hat{x}\in\widehat{\mathcal{X}}_S}\min_{x\in\mathcal{X}_T}\|\hat{x}-x\|_2^2+\frac{1}{|\mathcal{X}_T|}\sum_{x\in\mathcal{X}_T}\min_{\hat{x}\in\widehat{\mathcal{X}}_S}\|x-\hat{x}\|_2^2, \]

where \(\mathcal{E}\) are the edges between neighboring vertices of the tetrahedral control grid.

4Match, then prune

Each deformed source point \(\hat{x}_i^S=T_{\mathrm{CPA}}(x_i^S)\) is assigned its nearest target point. The stored match links the original source point to that target point, and is kept only if it is also semantically compatible:

\[ a(i)=\arg\min_{j}\big\|\hat{x}_i^S-x_j^T\big\|_2, \qquad \mathcal{C}_{\mathcal{S},\mathcal{T}}=\Big\{\big(x_i^S,x_{a(i)}^T\big)\;:\;s_{\mathrm{sem}}\big(x_i^S,x_{a(i)}^T\big)>\tau_{\mathrm{sem}}\Big\}. \]

5Project to 2D, scale combinatorially

Rendering with calibrated cameras \(\pi_{c_S},\pi_{c_T}\) turns each 3D match into 3D–2D (one endpoint projected) or 2D–2D (both projected) supervision:

\[ \big(u_i^S,u_j^T\big)=\big(\pi_{c_S}(x_i^S),\,\pi_{c_T}(x_j^T)\big). \]

A category with \(N\) objects yields up to \(\binom{N}{2}\) cross-instance pairs, and each pair can be rendered from many views.