1.

Introduction

Coronary artery calcification (CAC) is associated with major adverse cardiovascular events.^1–3 To date, the clinical impact of arterial calcification on local plaque vulnerability remains unclear,⁴ but the extent of CAC is associated with worse outcomes in the general population and in patients undergoing revascularization.^5,6 In addition, the presence of calcifications leads to under-expansion during percutaneous coronary intervention (PCI).⁷ The previous study demonstrated that severely calcified plaques undergoing PCI were associated with higher procedural complications and lower success rates.⁸ Hence, detection of calcified region is important for PCI treatment. For example, in the heavily calcified lesions, it is hard to cross or dilate a coronary stenosis with PCI devices such as balloons or stents.⁹ However, rotational atherectomy facilitates delivery or expansion for the treatment of complex calcified lesions. Furthermore, other available treatment strategies such as cutting and scoring balloons improve vessel compliance by creating discrete incisions instead of producing small particles to ablate the heavily calcified plaque for rotablation.⁴ For above reasons, a cardiologist pays more attention to the calcified lesion when making a clinical decision. Therefore, accurate identification and quantification of calcified lesion is crucial when treating patients with advanced coronary atherosclerosis.

The quantitative analysis of calcified lesions is a challenging task for a number of reasons. The heterogeneous appearance of the lesions including the large variability in location, size, shape, and frequency makes it difficult to conduct effective and quantitative analysis.¹⁰ An imaging system containing artifacts such as speckle, motion artifacts makes the image interpretation challenging to a novice reader. Manual detection is the current approach, but is tedious, expensive, time-consuming, impractical in larger studies, and it introduces interobserver variability. Moreover, the lesion is intrinsically a 3D structure and a single slice measurement is unable to characterize the volumetric nature of the lesion. To decide a particular tissue component, a multiple image slices need to be considered. Moreover, the level of expert knowledge and experience is also an important factor that affects the accuracy. Hence, in clinical routine often only qualitative, visual inspection, or at best crude measures such as approximate lesion volume are used.^11–13 Therefore, development of an accurate, automatic plaque classification method is needed to preferentially provide a feedback on the existence of calcification deposits on the pullback level.

Both intravascular optical coherence tomography (OCT) and intravascular ultrasound (IVUS) can be used to detect a calcified plaques.^14,15 OCT has been proven to be a primary choice because of its high resolution (10 to $15\mu \mathrm{m}$) in comparison with IVUS.¹⁶ Although IVUS has been widely used to estimate coronary calcification during PCI to date,¹⁷ it is not able to precisely visualize microcalcification. Moreover, IVUS cannot penetrate heavily calcified plaques in the assessment of coronary calcification. In this study, we developed a 3D convolutional neural network (CNN) for automatic classification of calcification lesion on the OCT pullback level. Full fine-tuning, a pretrained ResNet-2D network allowed us to transfer natural image features to OCT images. Furthermore, we also compared different input slices to evaluate the ResNet-3D network performance. Finally, a majority voting to integrate all the ResNet-3D networks was used to further improve the classification results.

2.

Related Works

Various studies have assessed the efficacy of the quantification and characterization of calcified plaques using machine learning based on OCT images.^18–20 Most of these researches have emphasized segmentation of the calcified plaques using the traditional machine learning methods on a single OCT slice. The atherosclerotic plaque segmentation has not been adequately and fully performed because it needs to manually select the slices with a calcified deposit in advance. For this reason, plaque segmentation should be the next important step, and the feedback on the existence and characteristics of calcium deposits on the pullback level should be the priority.²¹ Furthermore, deep learning has once again become a state-of-the art learning algorithm in the image classification. Compared to the traditional machine learning, deep networks naturally integrate low-, mid-, and high-level image features.²² Machine learning focuses on mining prior knowledge in the data and converting it to corresponding regularization constraints or artificial design features. Deep learning is weak on the prior knowledge of the data. It is expected to learn the data through stacking of hierarchical expressions. The intrinsic regularity of this makes this kind of separability or discriminative feature as much as possible to weaken the complex design of the classifier and achieve the purpose of simplicity, novelty, and universality. Recently, there has been increasingly more attention on deep learning based on the intravascular OCT images for detection and characterization atherosclerotic plaque. Abdolmanafi et al.^23–25 studied the CNN application in the OCT slices and confirmed the feasibility and effectiveness of deep learning in detection and characterization of atherosclerotic plaque. They successfully classified the coronary artery layers in pediatric patients using the powerful feature representation abilities of CNN models. In addition, they also compared three common deep learning models (e.g., Alexnet, VGG-19, and Inception-v3) for tissue characterization of intracoronary pathological formations. The results demonstrated that deep learning models are robust for automatic interpretation of the OCT slices. Furthermore, they designed a diagnostic model about coronary artery lesions using multisteps strategies based on the above studies. These studies demonstrated successful applications of the pretrained CNN networks on OCT dataset. However, it had shown that a custom network could obtain a more effective and robust result in the characterization of OCT slices. Kolluru et al.²⁶ classified three tissue types (fibrocalcific, fibrolipidic, or others) by A-line using the customized CNN and fully connected artificial neural network. In addition, the hybrid model such as the restructure pretrained model based on the specific task could improve the network performance. Gessert et al.²¹ executed a multipath network architecture based on the ResNet network and achieved an accuracy of 91.7%, a sensitivity of 90.9%, and a specificity of 92.4%. Lee et al.²⁷ developed a fully automated semantic segmentation model of atherosclerotic plaques for OCT images. The sensitivities/specificities against manually annotated clinical dataset were 87.4%/89.5% and 85.1%/94.2% for pixelwise classification of lipid and calcified plaque, respectively. They also used hybrid features, including the deep features, handcrafted features and lumen morphological features to characterize atherosclerotic plaque in OCT image. All the above studies were based on a single OCT slice, which reduced the training time but limited the further performance improvement. Even though single OCT slice provided the local and global information of the plaque, considering the fact that a coronary lesion is intrinsically a 3D structure, 3D CNN may be a better approach.

One important aspect of CNNs is the “transferability” of knowledge embedded in the pretrained CNNs. Recent research conducted by Tajbakhsh et al.²⁸ consistently demonstrated that fine-tuning was better than trained from scratch for deep networks. Moreover, layerwise fine-tuning is an alternative optimal choice to reach the better performance for the limited medical image dataset. Several studies have demonstrated that a pretrained CNN could be adapted to the medical image classification.^29–31 For instance, Christodoulidis et al.³⁰ fine-tuned different CNN layers for classification of interstitial lung diseases. Their studies suggested that the ensemble learn could help for the unbalance medical images. In another study, two pretrained networks, VGG 16 and Inception V4, were employed to class the Alzheimer’s disease from MRI images.³¹ The labeled MRI data were used to train the fully connected layers while keeping the rest network layer fixed. Different from the previous approaches, Miyagawa et al.²⁹ transferred the knowledge from other tasks based on OCT images to the classification of vascular bifurcation, and this method yielded promising results.

3.

Methods

Figure 1 shows the flowchart of the methodology. A trained expert reviewed each pullback independently and labeled each slice with calcified plaque and noncalcified plaque. According the expert consensus, calcium components are well-delineated heterogeneous regions in the arterial wall with less signal attenuation,³² as can be seen in Fig. 2.

Fig. 1

Flowchart of the methodology. 2D, two dimension; 3D: three dimension.

Fig. 2

The polar representations of OCT images. (a) Preprocessed calcified image, (b) preprocessed noncalcified image. (a1, b1) gray image, (a2, b2) RGB image, (a3, b3) augmentation image with flip along the depth axis, and (a4, b4) augmentation image with cycle translation along the depth axis.

3.3.

Convolutional Neural Networks

3.3.1.

Network architectures

We employed one of state-of-the-art architectures from the natural image domain, 50-layer ResNet-2D as an initial network model in the experiment. The 50-layer ResNet-2D has an obvious hierarchical structure and powerful output with representational ability. In addition, a large amount of downsampling was used, which significantly improved the gradient propagation efficiency. Batch normalization and global average pooling for regularization speeded up the training of the network. Furthermore, the network utilized the bottlenecks to reduce network parameters. Here, a $1\times 1$ convolution first downsampled and then upsampled the tensor along the feature map dimension. Then, the normal $3\times 3$ filter embedded on the lower input/output dimension. The 2D bottleneck is shown in Fig. 3(b). To realize the ResNet-3D architecture, we directly expended the 2D bottleneck to the 3D bottleneck [Fig. 3(c)]. Both main architectures were the same, whereas the convolution kernel of the bottleneck part was different. The basic bottleneck block in the Resnet-3D consisted of three convolutional layers, and the kernel sizes of the first and the third convolutional layers were $1\times 1\times 1$, whereas it was $3\times 3\times 3$ for the second layer.

Fig. 3

The network architectures that were employed with our dataset. In each ResNet input and output block, the number of output feature maps was given. /2 denotes spatial downsampling with a stride of two. $pr$ denotes a retraining point for transfer learning. Left of the retraining point, weights are freezed, right of the retraining point, weights are retrained.

3.3.2.

Loss function

It is important to choose an appropriate loss function on the pullback level. With the rise of deep learning, cross entropy loss function becomes a recognized effective loss function, especially in the field of image classification.³⁴ Assume that $X_1,\dots ,X_{N1}$ is a set of $N1$ training samples and $y_1,\dots ,y_n\in \{1,\dots ,M\}$ ($M$ is the class number) are the corresponding class labels. Cross entropy loss function is defined as

Eq. (3)

$$L_i=\sum _{j=1}^M{y}_{i,j}\log (p_{i,j}),$$

where $y_{i,j}$ is the number of the class, and $y_{i,j}$ is 1 if the sample $X_i$ belongs to class $y_i$ ($j=y_i$) and 0 otherwise. $p_{i,j}$ is the class probabilities that is normalized by the softmax function. Cross entropy loss with softmax function is commonly used as the output layer, which could provide a classification probability distribution.

3.3.3.

Training

The ResNet-2D network was trained on the ImageNet dataset, consisting of 1.2 million training images, with 1000 classes of objects. It is unwise to directly apply the pretrained ResNet-2D network to the OCT datasets because of the significant differences between nature images and OCT images. Therefore, we replaced image mean of the pretrained network with OCT images before the retraining started using the OCT dataset in our ResNet-2D models. Then, we simultaneously set the fully connected layer to have the same size as the number of classes in the OCT dataset. Accordingly, we also replaced the final layer with a new classification output layer where the classes were default to automation. To further investigate the effects of different transfer learning strategies, we accurately froze different weights layers. Compared to the ResNet-2D models, the training of ResNet-3D models is intuitively more difficult because of more parameters and expensive computational time. With the availability of the trained ResNet-2D model, it is a natural choice to utilize the trained ResNet-2D model parameters to initialize the ResNet-3D models. However, the sizes of both convolution kernels often do not match. In order to solve the problem, we assume $H\in {\mathbb{R}}^{c_{\text{in}}\times c_{\text{out}}\times 3\times 3}$ is the trained ResNet-2D network convolutional filter where $C_{\text{in}}$ and $C_{\text{out}}$ are the channel numbers of the input and output, respectively. According to the assumption, we implemented non-zeros padding and zeros padding to pad the 3D convolution filter $D\in {\mathbb{R}}^{c_{\text{in}}\times c_{\text{out}}\times 3\times 3\times 3}$ as

Eq. (4)

$$D_{\text{non-zeros padding}}=\{\begin{array}{l}{D}_{(0)}=H_{C_{\text{in}}\times C_{\text{out}}\times 3\times 3}\\ D_{(1)}=H_{C_{\text{in}}\times C_{\text{out}}\times 3\times 3}\\ D_{(2)}=H_{C_{\text{in}}\times C_{\text{out}}\times 3\times 3}\end{array},$$

Eq. (5)

$$D_{\text{zeros padding}}=\{\begin{array}{c}{D}_{(0)}=0_{C_{\text{in}}\times C_{\text{out}}\times 3\times 3}\\ D_{(1)}=H_{C_{\text{in}}\times C_{\text{out}}\times 3\times 3}\\ D_{(2)}=0_{C_{\text{in}}\times C_{\text{out}}\times 3\times 3}\end{array},$$

where the subscript of $D$ is the depth of ResNet-3D convolution. By design, the initialized ResNet-3D model is based on a trained ResNet-2D model, which may achieve at least the same performance as the ResNet-2D model.

Training with original imbalance dataset makes it impossible for deep network to favor minority class through entire pullbacks. To overcome the difficulty, a two-phase training method was implemented. In the first phase training, the labels of the dataset were rigorously equiprobable by oversampling. In the second phase, the original and imbalance data were used to fine-tune the output layer and keep the kernels of all others fixed.

In the study, we used the Adam algorithm³⁵ with a starting learning rate of ${10}^{-4}$. To find the optimal schedule, we reduced the learning rate by a factor of two when the validation error saturated. A batch size of Batchnorm = 64 was used for all models. In addition, early stop of the training could also happen when the classification accuracy on the validation set did not improve. All analyses were conducted using Matlab R2019b and related toolboxes (MathWork^®, Natick, Massachusetts): Image Processing™ and Parallel Computing™ toolboxes and Deep Learning™ toolboxes. We also implemented our models using GPU to speed up the training.

4.

Results

The ResNet-2D classification results of fine-tuning parameters at different levels are shown in Table 1. The $pr=1$ represents fine-tuning all parameters before this node (as can be seen in Fig. 3). To keep consistent for all experiment conditions, no data augmentation is shown in Table 1. The result shows that the higher the $F1$-score, the more the fine-tuning parameter layers. This result is consistent with a previous study.²⁸

Table 1

Fine tuning of different weight layers. pr denotes the point of weight freezing in the network (see Fig. 3).

To investigate the effect of imbalance dataset, we not only compared the one-phase and two-phase training but also set two other contrastive experiments, with and without data augmentation. The comparison results are shown in Table 2, which was based on the fine-tuning parameter $pr=4$. In both methods, the data augmentation and two-phase training improved the classification results. Comparatively, it was found that the two-phase training method was slightly better than the data augmentation method for the performance improvement.

Table 2

Results for plaque classification using the ResNet-2D model. One-phase represents using the original data distribution as model inputs.

As described in the previous section, we initialized ResNet-3D model parameters by the trained ResNet-2D model (superscript a in Table 2). Here, we compared ResNet-2D and ResNet-3D models convergence using cross-entropy loss on the validation set (Fig. 4). As can be seen, both models converged quickly, which indicated that the model parameters were suitable. In addition, a considerable gap was observed between the ResNet-2D model and the ResNet-3D model, which implied that the ResNet-3D model performed better than the trained ResNet-2D model. Although the loss curves fluctuated greatly when the epoch was less than $10$, it became stable when the epoch was more than 15.

Fig. 4

Comparisons of trainings with transfer learning between ResNet-2D and ResNet-3D models in terms of cross-entropy loss (means) on the OCT pullback dataset. Note that the metrics were better when the loss was smaller.

The impact of the number of input slices on the ResNet-3D model performance was investigated. The classification results of different methods are shown in Table 3. For both non-zeros padding and zeros padding, a similar phenomenon was observed in terms of the $F1$-score performance. The five slices achieved the best result followed by the three slices and seven slices. Moreover, majority voting obtained the best results not only for the zeros padding but also for the non-zeros padding, respectively. The performance for zeros padding was better than non-zeros padding with the same number of input slices.

Table 3

Results for the ResNet-3D model, which was initialized by trained ResNet-2D model with non-zeros padding and zeros padding.

Figure 5 shows a 3D visualization of the classification results of ResNet-2D model (superscript a in Table 2) and ResNet-3D models (superscript a and b in Table 3) on the pullback level. Compared to the ResNet-2D, the misclassification rate of the ResNet-3D dramatically decreased. In addition, the misclassification results of ResNte-3D models mainly concentrated in the consecutive slices.

Fig. 5

Qualitative results of a pullback using different models. (a) The original pullback with ground-truth labels; (b) predicted results of ResNet-2D model along the pullback; (c) predicted results of ResNet-3D model of non-zeros padding along the pullback; (d) predicted results of ResNet-3D model of zeros padding along the pullback. The red color indicates regions without calcified plaque and the green color indicates regions of calcified plaque.

The recently published state-of-the-art method by Gessert et al.²¹ analyzed the different transfer learning strategies and network architecture and demonstrated that two-path architecture could provide a more effective feature differentiation. In addition, their study also demonstrated that $pr=3$ was the optimal option compared to the other stages. Table 4 shows that the comparison of the ResNet-3D model with the two-path model on the OCT data. The advantages of our method are clearly demonstrated.

Table 4

Comparison of our implemented method with the method by Gessert et al.21 on our dataset.

5.

Discussion

This study presented ResNet-3D CNN models for an automatic classification of plaque calcification from OCT pullback dataset. The benefits of the transfer learning strategies included a decrease in the computational cost as well as the number of trainable parameters. We proposed an efficient solution for processing large image dataset by adopting the trained parameters based on the ResNet-2D model as the initialization parameters of the ResNet-3D models, alleviating one of the main computational limitations of a 3D CNNs. Furthermore, we also considered the majority voting as a postprocessing step to refine the network’s output, a method that has also shown its advantage for processing 2D images.³⁸ It is important to note that the entire process was fully automatic, and no manual intervention and complicated image preprocessing algorithm were involved. In addition, our work could be expanded into a detailed classification system for the clinical classification of plaque morphology in coronary disease, and it may lead to a number of possibilities for future online clinical support system.

The results in Table 1 show that there is a difference in classification performance between both scenarios that full fine-tune network achieves a better classification performance. These results are consistent with the previous research²⁸ that suggests that if the target domains are significantly different from source domains, full fine-tune should be performed. In addition, the classification performance of the two-phase method is better than the data augmentation, whereas the time required for the two-phase training is more than data augmentation. In the future, more optimization experiments should be executed to compare them. The results in Table 3 show that the classification performance of zero padding was generally better than the non-zeros padding. The possible reason may be that the adjacent slices provide the complementary information in the ResNet-3D models. In addition, we considered 3, 5, and 7 consequent slices as the ResNet-3D models input. The results suggest that it is better to refer to the five consecutive slices when we label the OCT images. Moreover, it would also be helpful to combine complementary methods through ensemble learning for a best performance.^39,40 A majority voting improved the classification result up to 10% for ResNet-2D model, 3% and 5% for non-zeros padding and zeros padding for ResNet-3D. In future research, more research could be extended to investigate the impact of the number of slices on network performance.

The generic nature of our methods allows its straightforward application to different lesion classification tasks without major adaptations. To our knowledge, ResNet-3D network combined with majority voting achieved the highest classification accuracy on an OCT pullback. Gessert et al.²¹ reported that the best classification accuracy, sensitivity, and specificity were 92%, 91%, and 93%, respectively. Table 4 shows that the ResNet-3D model outperforms their two-path model. It is important to note that their work focused only on images excluding stents, whereas our study has shown capable of classifying an entire OCT pullback. Moreover, the work described in this article may be extended to other applications for future research. For example, we can start with three 2D networks trained for lateral and coronal sections, and then initialize the 3D network. Also, we could consider 3D voxels segmentation and use the spatial information between adjacent slices to segment OCT voxels.

Although neural networks seem promising for OCT image classification, further work is required to make the inference process more interpretable. This would improve our understanding when the network fails, an important aspect in biomedical applications. Although the ResNet-3D network needed one-hour training time, an optimal and simple network may further speed up the convergence.

6.

Conclusion

A Resnet-2D model based on each OCT slice was implemented to meet the challenges of automated classification of plaque calcification in coronary artery. Then, we established an automatic classification model on the OCT pullback level by expanding the ResNst-2D model to the ResNet-3D model. We compared non-zeros padding and zeros padding methods to convert the 2D convolutional filter to 3D. The result demonstrated that zeros padding was more suitable for the ResNet-3D model. We further implemented a majority voting algorithm by incorporating a contextual slice integration scheme for accurate lesion classification. Our study demonstrated that ResNet-3D models with effective training mechanisms could be employed to solve complicated medical image classification problems, even with a limited training dataset. Further investigations can be focused on integrating supervised information into our networks to further enhance the discrimination capability and explore more applications.