Gaussian Heritage

The creation of digital replicas of physical objects has interesting applications for the preservation and diffusion of tangible cultural heritage. However, existing methods are often slow, expensive, and require expert knowledge. We propose a pipeline to generate a 3D replica of a scene using only RGB images (e.g., photos of a museum), and then extract a model for each item of interest (e.g., pieces in the exhibit). Our approach leverages advancements in novel view synthesis and Gaussian Splatting, modified for efficient 3D segmentation. This method does not require manual annotation, and the visual inputs can be collected using a standard smartphone, making it affordable and easy to deploy. We provide an overview of the method and baseline evaluation of the accuracy of object segmentation. The code is available as a Docker container at this GitHub link.

Given a set of images, we:

• a) use a web interface to upload them to a local server, where they are processed to generate:
• b) 2D instance segmentation masks and
• c) a sparse 3D model.
• d) With these, we train a model capturing appearances and 3D segmentation of the scene, from which we then
• e) extract a model of each object.

The model we trained for 3D segmentation is based on Contrastive Gaussian Clustering.

Sample view extracted from the a) “Family” scene of Tanks and Temples dataset, using the labels b) “man statue” and c) “mother and baby statue”.

@article{dahaghin2024gaussian,
  title={Gaussian Heritage: 3D Digitization of Cultural Heritage with Integrated Object Segmentation},
  author={Dahaghin, Mahtab and Castillo, Myrna and Riahidehkordi, Kourosh and Toso, Matteo and Del Bue, Alessio},
  journal={arXiv preprint arXiv:2409.19039},
  year={2024}
}