Open3D renders partial view point clouds of meshes.

Introduction#

As shown in the figure, we have a mesh model (along with its pose in the world coordinate system) and we want to render colored point cloud data under a specified camera pose.

My application involves registering a predicted mesh with a known pose to the observed point cloud with scale. I believe that by first rendering the corresponding point cloud of the mesh and then registering it, even in the presence of occlusion, the accuracy will be higher.

References#

How to capture depth image from a point cloud? · Issue #1152 · isl-org/Open3D · GitHub - Rendering known point clouds using the capture_depth_float_buffer function in open3d. However, it does not address how to reproject back to the point cloud, and I have verified that the crucial steps come later.
3d - How do I generate a partial view of a mesh as a point cloud in Python? - Stack Overflow - Similar question, using the capture_screen_image and capture_depth_image functions in open3d. This solution loses scale completely because it saves a discretized depth image (0~255). It also found that the camera parameters of these two functions are not consistent with those used in projecting to the point cloud using create_from_rgbd_image, resulting in deformation.
I used capture_depth_float_buffer combined with create_from_rgbd_image for reprojection testing and found that there are also scale and deformation issues.

Proposed Solution#

Use open3d's capture_depth_point_cloud for direct rendering.

def render_mesh_depth(mesh, depth):
    center_mesh = np.mean(np.array(mesh.vertices), axis=0)
    depth_data_center = np.mean(depth_data, axis=0)
    mesh.vertices = o3d.utility.Vector3dVector(np.array(mesh.vertices) - center_mesh + depth_data_center)

    # Set camera pose
    camera_pose = np.array([
        [1, 0, 0, 0],
        [0, 1, 0, 0],
        [0, 0, 1, 0], 
        [0, 0, 0, 1]
    ])

    # Create renderer and rendering options
    vis = o3d.visualization.Visualizer()
    vis.create_window(visible=False)  # Create an invisible rendering window
    vis.add_geometry(mesh)

    # Set camera parameters and viewport
    ctr = vis.get_view_control()
    cam_params = ctr.convert_to_pinhole_camera_parameters()
    cam_params.extrinsic = camera_pose
    ctr.convert_from_pinhole_camera_parameters(cam_params)

    # Render depth image
    vis.capture_depth_point_cloud("o3d_tmp.ply", do_render=True, convert_to_world_coordinate=True)
    vis.destroy_window()

    # Generate point cloud from depth image
    pcd = o3d.io.read_point_cloud("o3d_tmp.ply")
    os.remove("o3d_tmp.ply")
    return pcd

During the process, I also encountered a bug in open3d Visualizer.get_view_control() gives a copy. · Issue #6009 · isl-org/Open3D · GitHub. The view_control obtained from the get_view_control function in open3d version 0.17.0 does not work. It works properly after downgrading to version 0.16.0.

Next Steps#

Currently, it is necessary to save the file first, then read it, and then delete it. Using capture_depth_float_buffer does not have this issue but has scale and deformation problems. It is speculated that the inconsistency in camera parameters between rendering and reprojection is the cause. You can refer to the source code of VisualizerRender.cpp for a solution.