Paper Info

• Qingjie Meng, Wenjia Bai, Declan O'Regan, Daniel Rueckert
• Based their conference paper: Mesh-based 3D Motion Tracking in Cardiac MRI using Deep Learning, MICCAI 2022
  • Also with Tianrui Liu
  • Journal Version: MulViMotion, TMI 2022

Table Of Content

• Introduction
  • a Brief Look at Mesh Data
  • Conference Paper overview
  • Journal Paper overview
• Related Work
  • Mesh Reconstruction
  • Motion Tracking
• Methods
  • Mesh Reconstruction module
  • Motion Tracking module
  • Mesh-to-image rasterizer
• Experiments

.stl file

• .stl means stereolithography, a technique or process for creating three-dimensional objects.

.stl file

• Mesh data is composed of vertex (3d point) and facet (triple index of vertex).

MulViMotion

DeepMesh

Mesh Reconstruction

• Multi-view based
  • Multi-View 3D Reconstruction[34]
• Learning based
  • the marching cube algorithm[6]
  • statistical shape modeling[45]
  • predict point clouds (dense 3d model with points) directly[43]

DeepMesh directly predicts the 3D surface mesh of the heart at the ED frame by deforming a cardiac template according to the input 2D multi-view cine CMR images.

Motion Tracking

• Conventional methods
  • using FFD registration[2][6][8]
  • continuous spatiotemporal B-spline kernels[20]
• Deep learning-based methods: U-Net or GNN
  • joint deep learning network for simultaneous cardiac segmentation and motion estimation[5]
  • [27] utilizes a modified U-Net to generate flow maps between the ED frame and any other frame.
  • [12] focused on 3D motion tracking by fully combining multiple anatomical views.

DeepMesh focuses on estimating 3D motion in mesh space using multi-view CMR including SAX and LAX.

Mesh reconstruction module

• Input: ED frame SAX and LAX (2CH+4CH) CMR
• Estimate a deformation
• Bi-linear Grid Sampling:
• Output:
  • apply to template on each vertex:
  • F (facets) remain unchanged
  • 22, 043 vertices and 43, 840 faces
• Mesh-to-image rasterizer: see later.

Mesh reconstruction module: deep net

Mesh reconstruction module: loss function

1. Regularization (Huber) loss on deformation

2. Smooth loss: regularize adjacent vertices

3. Surface loss: between reconstructed mesh and the ground truth mesh

4. Shape loss: weighted Hausdorff distance, see later.

Motion tracking module

• Input: Multi-frame SAX and LAX (2CH+4CH) CMR
• Estimate a deformation
• Bi-linear Grid Sampling:
• Output:
  • apply to template on each vertex:
  • F (facets) remain unchanged

Motion tracking module: deep net

Motion tracking module: loss function

1. Regularization (Huber) loss on deformation

2. Smooth loss: regularize adjacent vertices

3. Simliarirty loss: inverse transform.

4. Shape loss: weighted Hausdorff distance, see later.

Differentiable Mesh-to-image rasterizer

Experiments: Mesh to plane

Experiments: Motion Tracking

Quantitatively evaluated the performance on the ES frame (have gt).

Experiments: Motion Tracking across cardiac cycle

Ablation Study: Mesh reconstruction

Compared with ground truth 2D myocardium contours.

Multi-view CMR helps a lot.

Ablation Study: Motion Tracking

Experiments: Loss Ablation viz

THANKS