python

import open3d as o3d
from open3d.web_visualizer import draw

3D Gaussian Splatting

Open3D supports basic operations with 3D Gaussian Splats. You can read and write Gaussian Splats to and from the ply and splat file formats. In addition, we have partial rendering support, i.e. you can visualize a Gaussian Splat as a point cloud with color. In the future we will add complete rendering support as well as editing support. This tutorial will guide you through the process of loading, visualizing, and manipulating 3D Gaussian splats using Open3D.

To begin with, let's download some sample 3DGS data. We will use a cropped version of the garden scene from the Mip NeRF 360 dataset. Lets download the data to get started:

python

!curl -O https://github.com/isl-org/open3d_downloads/releases/download/3dgs-1/mipnerf360_garden_crop_table.ply
!curl -O https://github.com/isl-org/open3d_downloads/releases/download/3dgs-1/mipnerf360_garden_crop_table.splat

Reading, writing and converting Gaussian Splats

Now we can read these files into Open3D. The data format for the 3DGS representation is the following:

pcd.point[‘positions’] (N,3) – (x,y,z) for each splat.
pcd.point[‘opacity’] – (N,1) opacity
pcd.point[‘rot’] (N, 4) - quaternion rotation of Gaussian.
pcd.point[‘scale'] (N, 3) - x, y, z scales for Gaussian.
pcd.point[‘f_dc’] (N, 3) – DC components for RGB colors.
pcd.point[‘f_rest’] (N, Nc, 3) – SH coeffs for RGB colors. (Nc = 3, 8 or 15)

f_rest will not be present for the .splat format files. opacity, f_dc and rot (quaternions) are also quantized to 8 bit to reduce storage space in this format.

python

ply_3dgs = o3d.t.io.read_point_cloud('mipnerf360_garden_crop_table.ply')
print(ply_3dgs)

python

splat_3dgs = o3d.t.io.read_point_cloud('mipnerf360_garden_crop_table.splat')
print(splat_3dgs)

We can convert .splat files to .ply and vice versa.

python

# Compress .ply format 3DGS by saving to .splat format
o3d.t.io.write_point_cloud('mipnerf360_garden_crop_table_from_ply.splat', ply_3dgs)
# Convert .splat format 3DGS to .ply format for wider compatibility
o3d.t.io.write_point_cloud('mipnerf360_garden_crop_table_from_splat.ply', splat_3dgs)

Visualizing Gaussian Splats

We can visualize Gaussian Splats similar to Point Clouds. Full 3DGS visualization support is in progress and at the moment, we can only visualize the colors in the Gaussian centers as colored points.

python

# Visualize splat format 3DGS as a point cloud (no view-dependent effects)
draw(splat_3dgs)

python

# Visualize full ply format 3DGS as a point cloud.
draw(ply_3dgs)