This PDF file contains the front matter associated with SPIE Proceedings Volume 12465, including the Title Page, Copyright information, Table of Contents, and Conference Committee listings.

Claims of deep learning’s superhuman performance on medical tasks are typically predicated on evaluations in which the training data and test data sets are exchangeably (statistically identical), differentiated only by the random partitioning of a larger population from which both are sampled. However, this assumption is always violated in practice, sometimes with disastrous effects. Subtle differences in how images are captured or represented can bring an otherwise superhuman system down to unacceptably poor levels of accuracy. In this talk I will discuss the fundamental hardness of these problems—absent further assumptions on the nature of the shifts in distribution, no principled technique can exist for estimating the accuracy of models, let alone for adapting them to new environments. I will also discuss the vast landscape of (i) principled methods (and corresponding assumptions) for developing robust and adaptive predictive models and (ii) heuristic methods that have shown some promise on benchmarks but lack theory to guide their application.

Normative modelling is a method for understanding the underlying heterogeneity within brain disorders like Alzheimer Disease (AD), by quantifying how each patient deviates from the expected normative pattern that has been learned from a healthy control distribution. Existing deep learning based normative models have been applied on only single modality Magnetic Resonance Imaging (MRI) neuroimaging data. However, these do not take into account the complementary information offered by multimodal MRI, which is essential for understanding a multifactorial disease like AD. To address this limitation, we propose a multi-modal variational autoencoder (mmVAE) based normative modelling framework that can capture the joint distribution between different modalities to identify abnormal brain volume deviations due to AD. Our multi-modal framework takes as input Freesurfer processed brain region volumes from T1-weighted (cortical and subcortical) and T2-weighed (hippocampal) scans of cognitively normal participants to learn the morphological characteristics of the healthy brain. The estimated normative model is then applied on AD patients to quantify the deviation in brain volumes and identify abnormal brain pattern deviations due to the progressive stages of AD. We compared our proposed mmVAE with a baseline unimodal VAE having a single encoder and decoder and the two modalities concatenated as unimodal input. Our experimental results show that deviation maps generated by mmVAE are more sensitive to disease staging within AD, have a better correlation with patient cognition and result in higher number of brain regions with statistically significant deviations compared to the unimodal baseline model.

Automatic detection of abnormalities to assist radiologists in acute and screening scenarios has become a particular focus in medical imaging research. Various approaches have been proposed for the detection of anomalies in magnetic resonance (MR) data, but very little work has been done for computed tomography (CT). As far as we know, there is no satisfactory approach for anomaly detection in CT brain images. We present a novel unsupervised deep learning approach to generate a normal representation (without anomalies) of CT head scans that we use to discriminate between healthy and abnormal images. In the first step, we train a GAN with 1000 healthy CT scans to generate normal head images. Subsequently, we attach an encoder to the generator and train the auto encoder network to reconstruct healthy anatomy from new input images. The auto encoder is pre-trained with generated images using a perceptual loss function. When applied to abnormal scans, the reconstructed healthy output is then used to detect anomalies by computing the Mean Squared Error between input and output image. We evaluate our slice-wise anomaly detection on 250 test images including hemorrhages and tumors. Our approach achieves an area under receiver operating characteristic curve (AUC) of 0.90 with 85.8% sensitivity and 85.5% precision without requiring large training data sets or labeled anomaly data. Therefore, our method discriminates between normal and abnormal CT scans with good accuracy.

Pediatric diffuse midline glioma (DMG) is a rare but fatal pediatric brain tumor. Tumor MRI features, extracted from segmented DMG, have shown promise for predicting DMG progression and overall survival. The data and knowledge accumulated from the more common adult brain tumors cannot be directly applied to DMG due to different tumor locations and appearances. The purpose of this work is to develop a transfer learning-based approach to automatically preprocess and segment sub-regions of DMG from multisequence MRIs. We retrospectively collected T1, contrastenhanced T1, T2 and T2 FLAIR images of 45 children diagnosed with DMG. MRI images at two timepoints were considered: at diagnosis and after completion of radiation therapy (RT). This generated a DMG dataset of 82 cases. Manual segmentation of two labels were created: the enhancing region (ER) and the whole tumor (WT). We modified the SegResNet model developed by NVIDIA and pre-trained it on BraTS 2021 challenge dataset, which contains 1,251 subjects with adult glioblastoma multiforme. DMG data was automatically preprocessed to have the same resolution and format as the input data in the BraTS challenge. A 5-fold cross-validation was performed using the preprocessed DMG data to finetune and validate the model. The proposed method resulted in mean Dice scores of 0.831 and 0.840 for the ER and WT segmentations, respectively. The method produced decent segmentation results for a small dataset. We demonstrated transfer learning from adult brain tumors to rare pediatric brain tumors was feasible and would improve segmentation results.

Schwannomas and meningiomas account for large proportion of primary spinal tumors and need surgical procedures. Although preoperative discrimination of schwannomas and meningiomas is crucial, differentiation between the two is challenging based on magnetic resonance imaging. The two have not only different patterns of magnetic resonance imaging but also different types of epidemiology. TabNet was recently invented as a deep neural network for tabular data and achieved state-of-the-art results on several datasets. As TabNet is a deep neural network, we can simultaneously train TabNet and a convolutional neural network, allowing simultaneous image and tabular data analysis. We aim to build a bi-modal model combining a convolutional neural network and TabNet and evaluate its performance for differentiating between schwannomas and meningiomas based upon integrated magnetic resonance imaging and clinical factors.

Identifying final pathology of FNA-indeterminant nodules before surgical resection could decrease the number of unnecessary surgeries and total cost to patients. This project explores how radiomics (RM) and deep learning (DTLM) models may be combined to improve the potential for clinical interpretability of machine learning models in the task of classifying indeterminant thyroid nodules on ultrasound. Two radiomic and deep learning combination models were created: a simple classifier combination model (SCM) and an interpretability-driven combination model (ICM). SCM provided a nodule malignancy score. ICM merged radiomic and deep learning features through correlation and provided echogenicity-related, composition-related, and shape/margin-related malignancy scores which were averaged to yield an overall nodule malignancy score. Models were trained and tested on a de-identified dataset of 476 grayscale ultrasound images collected under IRB approval containing 222 images from 69 indeterminant nodules with a final pathology of malignant and 254 images from 82 indeterminant nodules with a final pathology of benign. Models were tested using 5- fold cross-validation by nodule over 100 iterations. Receiver-operating characteristic (ROC) analysis was conducted with area under the ROC curve (AUC) serving as the statistic of merit for model performance. Models yielded mean AUC [95%CI] of 0.75 [.67,.83], 0.70 [.62,.78], 0.77 [0.70,0.84], 0.76 [.69,.84] for RM, DTLM, SCM, and ICM respectively. This work failed to demonstrate a statistically significant difference in model performances. However, the ICM presents a novel method for combining radiomics and deep learning features focused on improving interpretability for clinical implementation in the task of indeterminant thyroid nodule classification.

Background: ComBat is a promising harmonization method for biomedical imaging data acquired on multiple scanners, but it cannot harmonize by multiple batch effects. In this work, we generalize the ComBat model to incorporate multiple batch effects and modify the estimation algorithm to estimate the corresponding corrections. We evaluate our method, which we refer to as MultiComBat, in simulated data and demonstrate effective harmonization. Methods: In MultiComBat, the standard ComBat model is generalized to incorporate multiple batch variables by taking the sum and product of location and scale parameters across all batch memberships for a sample. The estimation algorithm for MultiComBat frames the method of moments estimation for the hyperparameters of the prior distributions for the location and scale parameters as a system of equations solvable with least squares. The hyperparameters are then used in posterior estimation for the location and scale correction factors needed for final data adjustment. We evaluate the MultiComBat approach by adding simulated batch effects for multiple batch variables to simulated radiomic features. Results: MultiComBat reduced the number of features with significant distribution differences due to multiple batch variables and resulted in more visually similar feature distributions, indicating successful correction of multiple batch effects. Conclusions: Our findings suggest that MultiComBat can be used to harmonize by multiple batch effects and is superior to standard ComBat. Larger studies are warranted to fully validate these findings.

Non-small cell lung cancer (NSCLC) accounts for approximately 85% of lung cancer patients. Recurrence rate for NSCLC patients is 30%-50% with a significant risk of mortality. Predicting recurrence can lead to personalized target therapy. Prediction models using radiomic features extracted from CT images have been developed but have not shown optimal performance. Gabor filters are linear filters that improve the performance of texture classification. This work shows how Gabor features improve performance in models used to predict recurrence in NSCLC patients.

Alterations in brain network connectivity play an important role in the pathogenesis of schizophrenia. We investigate whether large-scale Kernelized Granger Causality (lsKGC) can capture such alterations using restingstate fMRI data. Our method utilizes dimension reduction combined with the augmentation of source timeseries in a predictive time-series model for estimating directed causal relationships among fMRI time-series. As a multivariate approach, lsKGC identifies the relationship of the underlying dynamic system in the presence of all other time-series. Here, we examine the ability of lsKGC to accurately identify schizophrenia patients from fMRI data using a subset of 31 subjects from the Centers of Biomedical Research Excellence (COBRE) data repository. We use brain connections estimated by lsKGC as features for classification. After feature extraction, we perform feature selection by Kendall’s tau rank correlation coefficient followed by classification using a support vector machine. For reference, we compare our results with cross-correlation, typically used in the literature as a standard measure of functional connectivity, and several other standard methods. Using 100 different training/test data splits with 10-fold cross-validation we obtain mean/std f1-scores of 84.87% ± 19.78% and mean Area Under the receiver operating characteristic Curve (AUC) values of 93.00% ± 16.61% across all tested numbers of features for lsKGC, which is significantly better than the results obtained with cross-correlation (AUC=53.25% ± 29.29%, f1-score=45.03% ± 30.82%), and multiple other competing methods, including partial correlation, tangent, precision, and covariance methods. Our results suggest the applicability of lsKGC as a potential imaging biomarker for schizophrenia.

Anterior cruciate ligament (ACL) is one of the most common injuries associated with sports. Knee osseous morphology can play a role in increased knee instability. Our hypothesis is that the morphological features of the knee, as seen in knee osseous morphology, can contribute to increased knee instability and, thus, increase the likelihood of ACL tear. To test this relationship, it is necessary to segment the femur and tibia bones and extract relevant imaging features. However, manual annotation of 3D medical images, such as on magnetic resonance imaging (MRI) scans, can be a time-consuming and challenging task. In this work, we propose an automated pipeline for creating pseudo-masks of the femur and tibia bones in knee MRI. Our approach involves unsupervised segmentation and deep learning models to classify ACL integrity (intact or torn). Our results demonstrate a high agreement between the automated pseudo-masks and a radiologist’s manual segmentation, which also leads to comparable AUC values for the ACL integrity classification.

The coronavirus disease 2019 (COVID-19) pandemic had a major impact on global health and was associated with millions of deaths worldwide. During the pandemic, imaging characteristics of chest X-ray (CXR) and chest computed tomography (CT) played an important role in the screening, diagnosis and monitoring the disease progression. Various studies suggested that quantitative image analysis methods including artificial intelligence and radiomics can greatly boost the value of imaging in the management of COVID-19. However, few studies have explored the use of longitudinal multi-modal medical images with varying visit intervals for outcome prediction in COVID-19 patients. This study aims to explore the potential of longitudinal multimodal radiomics in predicting the outcome of COVID-19 patients by integrating both CXR and CT images with variable visit intervals through deep learning. 2274 patients who underwent CXR and/or CT scans during disease progression were selected for this study. Of these, 946 patients were treated at the University of Pennsylvania Health System (UPHS) and the remaining 1328 patients were acquired at Stony Brook University (SBU) and curated by the Medical Imaging and Data Resource Center (MIDRC). 532 radiomic features were extracted with the Cancer Imaging Phenomics Toolkit (CaPTk) from the lung regions in CXR and CT images at all visits. We employed two commonly used deep learning algorithms to analyze the longitudinal multimodal features, and evaluated the prediction results based on the area under the receiver operating characteristic curve (AUC). Our models achieved testing AUC scores of 0.816 and 0.836, respectively, for the prediction of mortality.

This paper introduces the improved method for the COVID-19 classification based on computed tomography (CT) volumes using a combination of a complex-architecture convolutional neural network (CNN) and orthogonal ensemble networks (OEN). The novel coronavirus disease reported in 2019 (COVID-19) is still spreading worldwide. Early and accurate diagnosis of COVID-19 is required in such a situation, and the CT scan is an essential examination. Various computer-aided diagnosis (CAD) methods have been developed to assist and accelerate doctors’ diagnoses. Although one of the effective methods is ensemble learning, existing methods combine some major models which do not specialize in COVID-19. In this study, we attempted to improve the performance of a CNN for the COVID-19 classification based on chest CT volumes. The CNN model specializes in feature extraction from anisotropic chest CT volumes. We adopt the OEN, an ensemble learning method considering inter-model diversity, to boost its feature extraction ability. For the experiment, We used chest CT volumes of 1283 cases acquired in multiple medical institutions in Japan. The classification result on 257 test cases indicated that the combination could improve the classification performance.

COVID-19 still affects a large population worldwide with possible post-traumatic sequelae requiring long-term patient follow-up for the most severe cases. The lung is the primary target of severe acute respiratory syndrome coronavirus 2 (SARS- CoV-2) infection. In particular, the virus affects the entire pulmonary vascular tree from large vessels to capillaries probably leading to an abnormal vascular remodeling. In this study we investigated two modalities for assessing this remodeling, SPECT perfusion scintigraphy and computed tomography, the latter enabling the computation of vascular remodeling patterns. We analyzed on a cohort of 30 patients the relationship between vascular remodeling and perfusion defects in the peripheral lung area, which is a predominant focus of the COVID-19 infectious patterns. We found that such relationship exists, demonstrated by moderate significant correlations between SPECT and CT measures. In addition, a vascular remodeling index derived from the z-score normalized peripheral CT images showed a moderate significant correlation with the diffusing capacity of the lung for carbon monoxide (DLCO) measures. Altogether these results point CT scan as a good tool for a standardized, quantitative, and easy-to-use routine characterization and follow-up of COVID19-induced vascular remodeling. An extensive validation of these results will be carried out in the near future on a larger cohort.

The COVID-19 pandemic has made a dramatic impact on human life, medical systems, and financial resources. Due to the disease’s pervasive nature, many different and interdisciplinary fields of research pivoted to study the disease. For example, deep learning (DL) techniques were employed early to assess patient diagnosis and prognosis from chest radiographs (CXRs) and computed tomography (CT) scans. While the use of artificial intelligence (AI) in the medical sector has displayed promising results, DL may suffer from lack of reproducibility and generalizability. In this study, the robustness of a pre-trained DL model utilizing the DenseNet-121 architecture was evaluated by using a larger collection of CXRs from the same institution that provided the original model with its test and training datasets. The current test set contained a larger span of dates, incorporated different strains of the virus, and included different immunization statuses. Considering differences in these factors, model performance between the original and current test sets was evaluated using area under the receiver operating characteristic curve (ROC AUC) [95% CI]. Statistical comparisons were performed using the Delong, KolmogorovSmirnov, and Wilcoxon rank-sum tests. Uniform manifold approximation and projection (UMAP) was used to help visualize whether underlying causes were responsible for differences in performance between test sets. In the task of classifying between COVID-positive and COVID-negative patients, the DL model achieved an AUC of 0.67 [0.65, 0.70], compared with the original performance of 0.76 [0.73, 0.79]. The results of this study suggest that underlying biases or overfitting may hinder performance when generalizing the model.

We present a new approach based on Bayesian neural networks (BNNs) for severity assessment of lung diseases using chest X-rays (CXRs). In contrast to reqular NNs, our model can provide uncertainty of the prediction for an input CXR which is crucial for clinical implementation of machine learning-assisted tools in radiology. With no loss of generality, we apply this method for severity assessment of COVID-19 pneumonia using multi-reader datasets from the USA and Korea. Our results show that the BNN can classify COVID-19 pneumonia with performance comparable to human experts while providing prediction uncertainty. We also compare the uncertainty of the model over different severity classes with inter-reader variability among the radiologists.

Deep learning models have achieved great success for the automated analysis of chest x-rays. However, many such models lack generalizability, i.e., a model trained in one dataset often performs poorly in a different dataset. One possible reason of such performance drop is the difference in the distribution of data from different institutions. In this context, utilization of data from multiple institutions to train a deep learning model may be helpful towards including a wider variety of data during training. This can improve the generalizability of the trained model. However, such an approach do not to preserve data privacy. To deal with the aforementioned limitation, federated learning may be useful. Federated learning allows multiple institutions to develop a machine learning model utilizing data from all institutions without sharing the data. Thus, federated learning approaches help in preserving data privacy. Although there has been a significant advancement in federated learning, such methods are rare in the context of chest x-ray diagnosis. Furthermore, most of such models do not utilize chest x-ray datasets from multiple institutions. In this work, we design a federated learning framework for chest x-ray diagnosis using datasets from multiple institutions. Our model shows improved generalizability in chest x-ray diagnosis across several publicly available large-scale chest x-ray datasets.

Most of the existing CAD frameworks for digital breast tomosynthesis (DBT) are single-view only, while radiologists typically utilize information from multiple screening views to better detect breast cancer lesions. Previously, we developed the Retina-Match framework for lesion detection that performed ipsilateral matching between CC and MLO views of the same breast. In this work, we improve that framework in both sampling strategy and feature extraction processes. We proposed a “hard negative” sampling strategy to train on more difficult ipsilateral lesion pairs to increase the robustness of the matching model. We introduced a grid-attention (GA) module to apply spatial attention mechanism for ipsilateral patch similarity learning. A screening DBT dataset with 4182 cases including 1498 (36%) cancers were used for training and testing. Case specificity improved by 15% at 96% sensitivity and average False Positive numbers per detection image (FPPI) decreased from 0.6 to 0.5 at 95% case sensitivity. These experiments indicated that proper ipsilateral matching result is the key to improve performance of multi-view lesion detection framework.

The purpose of this study was to develop a loss function that can drive a given CNN to achieve high sensitivity (or recall) for identifying women with a high-risk of having breast cancer. The cross-entropy (CE) loss function is widely used to optimize a CNN for natural scene classification due to its stability. However, CE loss treats each class equally, thus, it may not be suitable to train the CNN to have high sensitivity performance. Therefore, we hypothesized that a loss function based on the Fβ-measure, the weighted harmonic mean of precision and recall, can improve the sensitivity of the resulting CNN model by giving more weight to the recall metric. To do so, we combined CE loss with the Fβ-measure to implement a task-oriented loss function for achieving high sensitivity performance. In this preliminary work, we used a screening mammogram dataset of 2000 scans (1000 recalled lesions;1000 normal). We extracted recalled lesion patches using radiologists’ annotations and normal patches from the center of the breast. We fine-tuned the DenseNet121 network using the image patch dataset with a data split ratio of 0.8:0.1:0.1 for training, validation, and testing. We conducted ROC analysis to evaluate the performance of our proposed model. In the test set, the model with the task-oriented loss function achieved an AUC of 0.90 compared to CE loss (AUC=0.88) alone. The ROC curve of the proposed loss function achieved (a sensitivity of 53% at 98% specificity level) higher sensitivity than the CE loss alone (a sensitivity of 41% at 98% specificity level) for a high specificity area.

High breast density (BD) is recognized as an independent risk factor for breast cancer development, in addition to negatively impacting the sensitivity of mammography. Although BD is normally assessed with the BI-RADS reporting system, this evaluation is qualitative and has been shown to vary considerably across readers. In this pilot study, we present a deep learning (DL) method to quantify BD from a standard two-view (cranio-caudal, and medio-lateral-oblique) mammography exam. With the aim of developing a method based on an objective ground truth, the DL model was trained and validated using 88 simulated mammograms from an equal number of distinct 3D digital breast phantoms for which BD is known. The phantoms had been previously generated through segmentation and simulated mechanical compression of patient dedicated breast CT images, allowing for the exact calculation of BD in each case. Different data augmentations were applied prior to simulation, to increase the dataset size, yielding a total of 528 cases. These were divided, randomly and on a patient level, into training (N=360), validation (N=60), and test sets (N=108). The DL model performance was tested by stratifying the breasts into four different density ranges: 1-15%, 15-25%, 25-60%, and <60%. The median absolute errors and interquartile ranges (IQR), in percentage points, were 3.3 (IQR: 3.5), 3.4 (IQR: 2.5), 3.5 (IQR: 3.9), and 14.8 (IQR: 8.4), respectively. Although preliminary, these results show the potential of the proposed approach for accurate BD quantification, which is based, as opposed to most previously proposed approaches, on an objective ground truth.

Radiotherapy (RT) is a standard treatment component for early-stage breast cancer. Despite the improved tumor control, RT-induced toxicity can negatively impact patient’s quality of life. This study aims to develop a quantitative toxicity assessment technique using breast ultrasound (BUS) radiomics. We conducted a longitudinal imaging study with 30 patients receiving RT for breast cancer. Each patient received BUS scans at five time points from 1-2 weeks before RT to one year post RT. BUS scans were performed bilaterally to account for patient variations. Our radiomics analysis start with an automatic breast surface detection, and then a region of interest uses the skin surface as the upper bound and includes both cutaneous and subcutaneous breast tissues. Radiomics of both histogram and second-order textural features are extracted. In total, we extract 163 radiomic features from both irradiated and contralateral breasts and used their relative differences to characterize the breast toxicity, and p values are calculated with the pairwise t-test. To eliminate redundant features, we used the correlation coefficient and variance inflation factors (VIFs) to screen out collinearity. Overall, 23 out of 163 radiomic features were selected as significantly different (p < 0.05) between the irradiated and contralateral breast, with the greatest difference (57%) at 3 months post RT. This study has demonstrated that BUSbased radiomic features can be utilized to quantitatively evaluate and monitor radiation-induced toxicity.

Autism Spectrum Disorder (ASD) is a complex neurodevelopmental disorder that impacts how people communicate, interact, behave and learn. Neuroimaging techniques including MRI have been used to not only characterize biomarkers of ASD but also identify individuals with ASD based on analyzing neural structural and functional features, which may assist the precise diagnosis of ASD especially at an early age. Most existing neuroimaging methods for ASD identification focus on a single type of neural measure such as brain functional connections, but ignore the influence of other neural features such as regional activities. There is an increasing need for computational models to use complementary information from multi-modal data for identifying mental disorders. In this work, we propose a framework of graph convolution networks (GCNs) based on maximum entropy weighted independent set pooling (MEWISPool), called MEWISPool-GCN, which not only learns the functional connections and regional activities of the entire brain network, but also integrates non-imaging data such as demographics. Specifically, the graph structure of brain imaging is first downsampled by the MEWISPool method. This structure-adaptive pooling method considers the input graph structure as a noisy communication channel to maximize the mutual information between the input nodes and the pooled nodes. Then, a population graph is constructed in order to further recalibrate the distribution of extracted features using the non-imaging phenotypic information. The feature vectors obtained after pooling are embedded into the nodes of the population graph; and the similarities between non-imaging data are used as edges connecting the nodes. GCN is then employed to learn node embeddings. In the experiment of ASD identification using ABIDE-I dataset, MEWISPool-GCN achieved an accuracy of 87.68% and AUC of 92.89%, which outperformed other related classification methods.

Magnetic resonance imaging (MRI) has potential benefits in understanding fetal and placental complications in pregnancy. An accurate segmentation of the uterine cavity and placenta can help facilitate fast and automated analyses of placenta accreta spectrum and other pregnancy complications. In this study, we trained a deep neural network for fully automatic segmentation of the uterine cavity and placenta from MR images of pregnant women with and without placental abnormalities. The two datasets were axial MRI data of 241 pregnant women, among whom, 101 patients also had sagittal MRI data. Our trained model was able to perform fully automatic 3D segmentation of MR image volumes and achieved an average Dice similarity coefficient (DSC) of 92% for uterine cavity and of 82% for placenta on the sagittal dataset and an average DSC of 87% for uterine cavity and of 82% for placenta on the axial dataset. Use of our automatic segmentation method is the first step in designing an analyticstool for to assess the risk of pregnant women with placenta accreta spectrum.

Intravenous contrast enhancement phase information is important for computer-aided diagnosis of CT scans because the visual appearance of the scans varies substantially among the different phases. Although phase information could help to refine training data curation for downstream tasks, it is seldom included in the process of data augmentation for training a deep learning model. Unfortunately, in the current clinical settings, phase information is either unavailable or unreliable in most PACS systems. This motivates us to develop a method to automatically classify multiphase CT scans. In this study, a residual network (ResNet34) was utilized to classify five CT phases commonly used in the clinical environment: non-contrast, arterial, portal venous, nephrographic, and delayed contrast phases. A dataset of 395 multiphase CT scans was weakly labeled using keywords. The weakly-labeled dataset was split into 316 training, and 79 test CT scans. We compared the ResNet34 with two other popular classification models, VGG19 and DenseNet121. ResNet34 achieved the highest accuracy of 99%, while the accuracy of VGG19 and DenseNet121 were 97% and 95%, respectively. In addition, ResNet34 had fewer parameters to train in comparison with two other models, which could reduce the inference time to 35 seconds per scan and enhance generalizability of the model. High accuracy of multiphase classification suggests a potential way to improve data curation based on CT contrast enhancement phase. This would be useful to improve deep learning models by enhancing dataset curation and providing more realistic augmented data.

Assessment of prostate cancer aggressiveness is important because the effectiveness of treatment vary depending on the aggressiveness. The use of multi-parametric MR imaging prior to biopsy is recommended for accurate prostate cancer aggressiveness assessment but suffers from similar visual appearance of tumors between adjacent grades. To improve the predictive performance of prostate cancer aggressiveness, this study proposes a deep regression model involving size-normalized patch generation and multiple losses. First, we generate two types of input patches such as tumor-centered patch and size-normalized patch to effectively learn the characteristics of small tumors. Second, we propose a multiple loss functions consisting of triplet loss, mean squared error, and cross-entropy ordinal loss to increase the ability to discriminate between tumors with similar visual appearance and different aggressiveness. As a result, the proposed model trained with the size-normalized ADC map showed the highest performance with an accuracy of 78.85%, specificity of 89.66%, and AUC of 0.77. The ensemble model of tumor-centered T2w image and size-normalized ADC map improved sensitivity by 8.69% and showed the best performance with accuracy of 78.85%.

According to the Centers for Disease Control and Prevention (CDC) more than 932,000 people in the US have died since 1999 from a drug overdose. Just about 75% of drug overdose deaths in 2020 involved Opioid, which suggests that the US is in an Opioid overdose epidemic. Identifying individuals likely to develop Opioid use disorder (OUD) can help public health in planning effective prevention, intervention, drug overdose and recovery policies. Further, a better understanding of prediction of overdose leading to the neurobiology of OUD may lead to new therapeutics. In recent years, very limited work has been done using statistical analysis of functional magnetic resonance imaging (fMRI) methods to analyze the neurobiology of Opioid addictions in humans. In this work, for the first time in the literature, we propose a machine learning (ML) framework to predict OUD users utilizing clinical fMRI-BOLD (Blood oxygen level dependent) signal from OUD users and healthy controls (HC). We first obtain the features and validate these with those extracted from selected brain subcortical areas identified in our previous statistical analysis of the fMRI-BOLD signal discriminating OUD subjects from that of the HC. The selected features from three representative brain areas such as default mode network (DMN), salience network (SN), and executive control network (ECN) for both OUD participants and HC subjects are then processed for OUD and HC subjects’ prediction. Our leave one out cross validated results with sixty-nine OUD and HC cases show 88.40% prediction accuracies. These results suggest that the proposed techniques may be utilized to gain a greater understanding of the neurobiology of OUD leading to novel therapeutic development.

To assess a Smart Imagery Framing and Truthing (SIFT) system in automatically labeling and annotating chest X-ray (CXR) images with multiple diseases as an assist to radiologists on multi-disease CXRs. SIFT system was developed by integrating a convolutional neural network based-augmented MaskR-CNN and a multi-layer perceptron neural network. It is trained with images containing 307,415 ROIs representing 69 different abnormalities and 67,071 normal CXRs. SIFT automatically labels ROIs with a specific type of abnormality, annotates fine-grained boundary, gives confidence score, and recommends other possible types of abnormality. An independent set of 178 CXRs containing 272 ROIs depicting five different abnormalities including pulmonary tuberculosis, pulmonary nodule, pneumonia, COVID-19, and fibrogenesis was used to evaluate radiologists’ performance based on three radiologists in a double-blinded study. The radiologist first manually annotated each ROI without SIFT. Two weeks later, the radiologist annotated the same ROIs with SIFT aid to generate final results. Evaluation of consistency, efficiency and accuracy for radiologists with and without SIFT was conducted. After using SIFT, radiologists accept 93% SIFT annotated area, and variation across annotated area reduce by 28.23%. Inter-observer variation improves by 25.27% on averaged IOU. The consensus true positive rate increases by 5.00% (p=0.16), and false positive rate decreases by 27.70% (p<0.001). The radiologist’s time to annotate these cases decreases by 42.30%. Performance in labelling abnormalities statistically remains the same. Independent observer study showed that SIFT is a promising step toward improving the consistency and efficiency of annotation, which is important for improving clinical X-ray diagnostic and monitoring efficiency.

Colorectal cancer (CRC) is one of the leading causes of cancer-related deaths with rising incidence. Since the survival rate of CRC is correlated with the cancer stage at diagnosis, timely detection and adequate treatment strategies are of utmost importance. Technical innovations such as machine learning (ML) and its application in endoscopy show promising results, but the trust of medical doctors in ML is lacking and the ‘black box’ nature complicates the understanding of such systems in clinical practice. In contrast to CT and MRI, image quality is a limiting factor in especially endoscopic imaging, as it is very operator dependent. However, the influence of image quality on convolutional (deep) neural networks (CNNs) is insufficiently studied in relation to clinical practice and the usage of medical image data for computer-aided detection and diagnosis (CADx) systems. This paper explores the influence of degraded image quality on the performance of CNNs applied to colorectal polyp (CRP) characterization. Five commonly used CNN architectures, from simple to more complex, are employed with a custom classification head for common CRP characterization. To degrade the quality of images, distortions such as noise, blur, and contrast changes are imposed on the data and their influence on the performance degradation is studied for the mentioned CNN architectures. A large prospectively collected in vivo data set, gathered from four Dutch, both academic and community, hospitals is employed. Results for CRP characterization show that promising CNN-based methods are rather susceptible to noise and blur distortions but reasonably resilient to changes in contrast. This implies that image quality needs monitoring and control prior to directly using image data in CNN models, in order to gain trustworthy use of deep learning (DL) models in a clinical setting. We propose that incorporating an image quality indicator in CADx systems will lead to better acceptance of such systems, and is necessary for the safe implementation of DL applications in clinical practice.

Coronary computed tomography angiography (CCTA) is a major clinical imaging technique used to diagnose cardiovascular diseases. To improve diagnostic accuracy, it is necessary to determine the most optimal reconstruction phase that has the best image quality with little or no motion artifacts. The end-systolic phase and end-diastolic phase are the two commonly used phases for image reconstruction, but they are not always optimal in terms of motion-based image quality. In this paper we propose a deep learning method to automatically select an optimal phase based on a set of 2D axial phase image reconstructions. We select the right coronary artery (RCA) as our main vessel of interest to analyze reconstruction quality. Two deep convolutional neural networks are developed to perform efficient heart region segmentation and RCA localization without manually performing patient-specific image segmentation. We also demonstrate how to calculate image entropy as a figure of merit to evaluate RCA reconstruction quality. Results based on real clinical data with a heart rate of 74 beats per minute (bpm) have shown our proposed algorithm can efficiently localize the RCA for arbitrary cardiac phase, and accurately determine the optimal RCA reconstruction with minimal motion artifacts.

Inspired by the potential of artificial intelligence (AI) to improve access to expert-level medical image interpretation, several organizations began developing deep learning-based AI systems around 2015. Today, these AI-based tools are finally being deployed at scale in certain parts of the world, often bringing screening to populations lacking easy access to timely diagnosis. The path to translating AI research into a useful clinical tool has gone through several unforeseen challenges along the way. In this talk, we share some lessons contrasting a priori expectations (“myths”) with synthesized learnings of what truly transpired (“reality”), to help others who wish to develop and deploy medical AI tools

Deep learning (DL)-based medical imaging and image segmentation algorithms achieve impressive performance on many benchmarks. Yet the efficacy of deep learning methods for future clinical applications may become questionable due to the lack of ability to reason with uncertainty and interpret probable areas of failures in prediction decisions. Therefore, it is desired that such a deep learning model for segmentation classification is able to reliably predict its confidence measure and map back to the original imaging cases to interpret the prediction decisions. In this work, uncertainty estimation for multiorgan segmentation task is evaluated to interpret the predictive modeling in DL solutions. We use the state-of-the-art nnU-Net to perform segmentation of 15 abdominal organs (spleen, right kidney, left kidney, gallbladder, esophagus, liver, stomach, aorta, inferior vena cava, pancreas, right adrenal gland, left adrenal gland, duodenum, bladder, prostate/uterus) using 200 patient cases for the Multimodality Abdominal Multi-Organ Segmentation Challenge 2022. Further, the softmax probabilities from different variants of nnU-Net are used to compute the knowledge uncertainty in the deep learning framework. Knowledge uncertainty from ensemble of DL models is utilized to quantify and visualize class activation map for two example segmented organs. The preliminary result of our model shows that class activation maps may be used to interpret the prediction decision made by the DL model used in this study.

Glioma is the most common form of brain tumor with a high degree of heterogeneity in imaging characteristics, treatment-response, and survival rate. An important factor causing this heterogeneity is the mutation of isocitrate dehydrogenase (IDH) enzyme. The current clinical gold-standard for identifying IDH mutation status involves invasive procedures that involve risk, may fail to capture intra-tumoral spatial heterogeneity or can be inaccessible in low-resource settings. In this study, we propose a deep learning-based method to non-invasively and preoperatively determine IDH status of high- and low-grade gliomas by leveraging their phenotypical characteristics from volumetric MRI scans. For this purpose, we propose a 3D Mask R-CNN-based approach to simultaneously detect and segment glioma as well as classify its IDH status - thus obviating the requirement of any separate tumor segmentation step. The network can operate on routinely acquired MRI sequences and is agnostic to glioma grade. It was trained on patient-cases from publicly available datasets (n = 223) and tested on two hold-out datasets acquired from The Cancer Genome Atlas (TCGA; n = 62) and Washington University School of Medicine (WUSM; n = 261). The model achieved areas under the receiver operating characteristic of 0.83 and 0.87, and areas under the precision-recall curves of 0.78 and 0.79, on the TCGA and WUSM sets, respectively. The model can be used to perform a pre-operative ‘virtual biopsy’ of gliomas, thus facilitating treatment planning, potentially leading to better overall survival.

An automated quality control (QC) system is essential to ensure streamlined head computed tomography (CT) scan interpretations that do not affect subsequent image analysis. Such a system is advantageous compared to current human QC protocols, which are subjective and time-consuming. In this work, we aim to develop a deep learning-based framework to classify a scan to be of usable or unusable quality. Supervised deep learning models have been highly effective in classification tasks, but they are highly complex and require large, annotated data for effective training. Additional challenges with QC datasets include - 1) class-imbalance – usable cases far exceed the unusable ones and 2) weak-labels – scan level labels may not match slice level labels. The proposed framework utilizes these weak labels to augment a standard anomaly detection technique. Specifically, we proposed a hybrid model that consists of a variational autoencoder (VAE) and a Siamese Neural Network (SNN). While the VAE is trained to learn how usable scans appear and reconstruct an input scan, the SNN compares how similar this input scan is to its reconstruction and flags the ones that are less similar than a threshold. The proposed method is more suited to capture the differences in non-linear feature structure between the two classes of data than typical anomaly detection methods that depend on intensity-based metrics like root mean square error (RMSE). Comparison with state-of-the-art anomaly detection methods using multiple classification metrics establishes superiority of the proposed framework in flagging inferior quality scans for review by radiologists, thus reducing their workload and establishing a reliable and consistent dataflow.

Early detection of breast cancer is important to reduce morbidity and mortality. Access to breast imaging is limited in low- and middle-income countries compared to high-income countries. This contributes to advancestage breast cancer presentation with poor survival. Pocket-sized portable ultrasound device, also known as point-of-care ultrasound (POCUS), aided by decision support using deep learning-based algorithms for lesion classification could be a cost-effective way to enable access to breast imaging in low-resource settings. A previous study, where using convolutional neural networks (CNN) to classify breast cancer in conventional ultrasound (US) images, showed promising results. The aim of the present study is to classify POCUS breast images. A POCUS data set containing 1100 breast images was collected. To increase the size of the data set, a CycleConsistent Adversarial Network (CycleGAN) was trained on US images to generate synthetic POCUS images. A CNN was implemented, trained, validated and tested on POCUS images. To improve performance, the CNN was trained with different combinations of data consisting of POCUS images, US images, CycleGAN-generated POCUS images and spatial augmentation. The best result was achieved by a CNN trained on a combination of POCUS images and CycleGAN-generated POCUS images and augmentation. This combination achieved a 95% confidence interval for AUC between 93.5% – 96.6%.

Retinopathy refers to pathologies of the retina that can ultimately result in vision impairment and blindness. Optical Coherence Tomography (OCT) is a technique to image these diseases, aiding in the early detection of retinal damage, which may mitigate the risk of vision loss. In this work, we propose an end-to-end Graph Neural Network (GNN) pipeline that can extract deep graph-based features for multi-class retinopathy classification for the first time. To our knowledge, this is also the first work applying Vision-GNN for OCT image analysis. We trained and tested the proposed GNN on a public OCT retina dataset divided into four categories (Normal, Choroidal Neovascularization (CNV), Diabetic Macular Edema (DME), and Drusen). Using our method, we achieve an average accuracy of 99.07% over four classes proving the effectiveness of a deep learning classifier for OCT images with graph-based features. This work lays the foundation to apply GNNs for OCT imaging to aid the early detection of retinal damage.

The main application of optical coherence tomography (OCT) is in the field of ophthalmology, where it is used for diagnosis of various eye diseases. The automatic segmentation of the individual retinal layers as well as pathological structures in OCT scans is helpful for clinical examination and treatment planning. Current methods often do not consider the strict arrangement of retinal layers. Although graph-based methods are suitable for correcting topology errors, their applicability is costly and complex, especially in the presence of pathologies. In this work, a segmentation method is proposed that utilizes additional shape information of the retinal layers to provide improved topology preservation while maintaining simple applicability. For this purpose, a U-Netbased network architecture is extended to a multi-task approach to allow regression of the shape information of the retinal layers in addition to pixel-wise classification. This introduces spatial regularization and allows the generation of plausible segmentations. A consistency term ensures agreement between the classification and regression task, which also allows for semi-supervised training. In a comprehensive evaluation, the performance of the proposed multi-task approach is investigated, using OCT image data from patients with diabetic macular edema. The results demonstrate that the integration of shape information improves the preservation of the topology of the retinal layers. Moreover, the use of a semi-supervised training scheme via a consistency term improves the robustness and refines the fluid delineation of the proposed method.

The performance of a segmentation network optimized on data from a specific type of OCT sensor will decrease when applied to data from a different sensor. In this work, we deal with the research question of adapting models to data from an unlabeled new sensor with new properties in an unsupervised way. This challenge is known as unsupervised domain adaptation and can alleviate the need for costly manual annotation by radiologists. We show that one can strongly improve a model’s result that was trained in a supervised way on the source OCT sensor domain on the target sensor domain. We do this by aligning the source and target domain distributions in the feature space through a semantic clustering method. Apart from the unsupervised domain adaptation, we improved even the supervised training compared to the results in the RETOUCH challenge by employing a sophisticated training strategy. The RETOUCH challenge contains three different types of OCT scanners and provides annotations for the task of disease-related fluid classes.

Universal lesion detection and tagging (ULDT) in CT studies is critical for tumor burden assessment and tracking the progression of lesion status (growth/shrinkage) over time. However, a lack of fully annotated data hinders the development of effective ULDT approaches. Prior work used the DeepLesion dataset (4,427 patients, 10,594 studies, 32,120 CT slices, 32,735 lesions, 8 body part labels) for algorithmic development, but this dataset is not completely annotated and contains class imbalances. To address these issues, in this work, we developed a self-training pipeline for ULDT. A VFNet model was trained on a limited 11.5% subset of DeepLesion (bounding boxes + tags) to detect and classify lesions in CT studies. Then, it identified and incorporated novel lesion candidates from a larger unseen data subset into its training set, and self-trained itself over multiple rounds. Multiple self-training experiments were conducted with different threshold policies to select predicted lesions with higher quality and cover the class imbalances. We discovered that direct self-training improved the sensitivities of over-represented lesion classes at the expense of under-represented classes. However, upsampling the lesions mined during self-training along with a variable threshold policy yielded a 6.5% increase in sensitivity at 4 FP in contrast to self-training without class balancing (72% vs 78.5%) and a 11.7% increase compared to the same self-training policy without upsampling (66.8% vs 78.5%). Furthermore, we show that our results either improved or maintained the sensitivity at 4FP for all 8 lesion classes.

Early detection of ocular abnormalities is important in preventing retinal damage that can cause blindness. As there is a high possibility for a patient to suffer from more than one ocular abnormality, diagnosis of multiple abnormalities (multi-label detection) is essential considering its efficiency. Multi-label detection of ocular abnormalities is still challenging due to the presence of rare ocular abnormalities. Only a limited number of data on rare ocular abnormalities are available which usually makes these abnormalities be ignored in multi-label detection. The aim of this study is to detect multi-label ocular abnormalities from color fundus images for both frequent and rare cases. The proposed method addresses this challenge by combining the visual features extracted from a color fundus image and the label co-occurrence dependencies extracted from linguistic features. The label co-occurrence approach has not been used so far for multi-label detection in medical applications as it can significantly reduce detection accuracy because of uncorrelated image and label features. However, this study shows that the label co-occurrence approach can increase the performance of multi-label detection of ocular abnormalities by tackling the miss-representation of the correlation between image and label features using semantic dictionary learning, taking into consideration the presence of labels that belongs to out-of-vocabulary (OOV) words as it has irrelevant label features.

Diagnosis of thoracic aortic aneurysm typically involves measuring the diameters at various locations on the aorta from computed tomography angiograms (CTAs). Human measurement is time-consuming and suffers from inter and intra-user variability, motivating the need for automated, repeatable measurement software. This work presents a convolutional neural network (CNN)-based algorithm for fully automated aortic measurements. We employ the CNN to perform aortic segmentation and localization of key landmarks jointly, which performs better than individual models for each task. The segmentation mask and landmarks are subsequently used to obtain the centerline and cross-sectional diameters of the aorta using a combination of image processing techniques. We gather a dataset of CTAs from patients with ongoing imaging surveillance of thoracic aortic aneurysm and demonstrate the performance of our algorithm by quantitative comparisons against measurements from human raters. We observe that for most locations, the mean absolute error between human and computer-generated measurements is less than 1 mm, which is at or lower than the level of variability in human measurements. Furthermore, we showcase the behavior of our method through various visual examples, discuss its limitations and propose possible improvements.

Purpose: This work aims to identify areas with sub-retinal-pigment-epithelium (sub-RPE) accumulations on 2- dimensional (2D) color-fundus-photographs (CFPs) in patients with age-related macular degeneration (AMD) using the definitions in spectral-domain-optical-coherence-tomography (SD-OCT) imaging. Detecting and quantifying areas of RPE elevations (most notably drusen) in CFPs will aid in objective evaluation of AMDseverity-scores as well as patient selection and monitoring in clinical trials. Methods: A retinal-layersegmentation algorithm for SD-OCTs was used to automatically identify areas with RPE elevations and build the ground-truth 2D binary maps for training. CFP was registered to the enface projection images of SD-OCT to overlay OCT-defined drusen areas on CFP images. A 2D-UNet segmentation network was trained using bilateral stereo CFP pairs in a Siamese architecture that share OCT-defined drusen areas as ground-truth. Results: Dataset consists of AMD patients with 127 train and 23 test eyes. Dice-similarity-coefficient for the predictions on CFPs was found to be 0.70±0.13 (mean±std), and overall accuracy was 0.73. 89% of test eyes exhibited drusen area prediction error <1mm² compared to reading-center measures. Conclusion: Our work demonstrates the potential of using 2D CFP images to predict areas of sub-RPE elevations as defined in 3D-SDOCT imaging. Qualitative evaluation of the mismatch between the two imaging modalities shows regions with complementary features in a subset of the cases making it challenging to achieve optimal segmentation. However, the results show clinically useful performance in CFPs that can be used to quantify accumulations in the sub-RPE space which are the key pathologic biomarkers of AMD relevant to patient selection and trial outcome measure designs.

Deep learning methods are the state-of-the-art for medical imaging segmentation tasks. Still, numerous segmentation algorithms based on heuristic-based methods have been proposed with exceptional results. To validate segmentation algorithms, manual annotations are typically considered as ground truth. However, manual annotations often suffer from inter/intra-operator variability and can also be occasionally inaccurate, especially when considering time-consuming and precise tasks. A sample case is the manual delineation of the lumen-intima (LI) and media-adventitia (MA) borders for intima-media thickness (IMT) measurement in B-mode ultrasound images. In this work, a novel hybrid learning paradigm which combines manual segmentations with the automatic segmentation of a dynamic programming technique for ground truth determination is presented. A profile consensus strategy is proposed to construct the hybrid ground truth. Two open-source datasets (n=2576) were employed for training four deep learning networks using the hybrid learning paradigm and three single source training targets as a comparison. The pipeline was fixed across the four tests and included a Faster R-CNN detection network to locate the carotid artery and then subsequent division into patches which were segmented using a UNet. The validation of the results was performed on an external test set comparing the predictions of the four different models to the annotations of three independent manual operators. The hybrid learning paradigm showed the best overall segmentation results (Dice=0.907±0.037, p<0.001) and demonstrated an exceptional correlation between the mean of three operators and the automatic measure (ICC(2,1)=0.958), demonstrating how the incorporation of heuristic-based segmentation methods within the learning paradigm of a deep neural network can enhance and improve final segmentation performance results.

Ascites generally manifests in the advance stage of ovarian cancer, and often mimicked by abdominal fluids such as urine in bladder. Segmentation of ascites in the pelvic region becomes increasingly challenging when the bladder is filled with urine. Anatomical location is utilized in this work to distinguish ascites from bladder. The location information is computed from a body part regressor and concatenated with contrast-enhanced computed tomography (CT) data through an embedding layer. A 3D residual U-Net is trained on the concatenated data to segment ascites and identify bladder simultaneously. 112 CT scans were used in this study; 55 of them were used for training, 20 for validation, and the remaining 37 for testing. Dice coefficient score and Jacard index are two metrics to evaluate ascites and bladder segmentation. In comparison with 3D residual U-Net, the addition of anatomical location information and two-class segmentation improved the average dice scores of ascites and urine segmentation from 0.44±0.23 to 0.51±0.16 and 0.36±0.15 to 0.40±0.13 respectively. The average volume errors of ascites and urine were reduced from 1.81±3.09 to 0.93±1.85 liters and 0.6±0.81 to 0.56±0.76 liters, respectively. These results suggested that anatomical location information and two-class segmentation are the key features to improve ascites segmentation by differentiating bladder filled with urine regions.

Segmentation of medical images with known ground truth is useful for investigating properties of performance metrics and comparing different approaches of combining multiple manual segmentations to establish a reference standard, thereby informing selection of performance metrics and truthing methods. For medical images, however, segmentation ground truth is typically not available. One way of synthesizing segmentation errors is to use regular geometric objects as ground truth, but they lack the complexity and variability of real anatomical objects. To address this problem, we developed a medical image segmentation synthesis (MISS)-tool. The MISS-tool emulates segmentations by adjusting truth masks of anatomical objects extracted from real medical images. We categorized six types of segmentation errors and developed contour transformation tools with a set of user-adjustable parameters to modify the defined truth contours to emulate different types of segmentation errors, thereby generating synthetic segmentations. In a simulation study, we synthesized multiple segmentations to emulate algorithms or observers with pre-defined sets of segmentation errors (e.g., under/over-segmentation) using 220 lung nodule cases from the LIDC lung computed tomography dataset. We verified that the synthetic segmentation results manifest the type of errors that are consistent with our pre-configured setting. Our tool is useful for synthesizing a range of segmentation errors within a clinical segmentation task.

Progression of thoracic aortic calcification (TAC) has been shown to be associated with hard cardiovascular events including stroke and all-cause mortality as well as coronary events. In this study, we propose an automated detection method of TACs of non-contrast CT images using mediastinal anatomical label map. This method consists of two steps: (1) the construction of a mediastinal anatomical label map, and (2) the detection of TACs using the intensity and the mediastinal anatomical label map. The proposed method was applied to two non-contrast CT image datasets: 24 cases of chronic thromboembolic pulmonary hypertension (CTEPH) and 100 non-CTEPH cases of low-dose CT screening. The method was compared with two-dimensional U-Nets and the Swin UNETR. The results showed that the method achieved significantly higher F1 score of 0.937 than other methods for the non-CTEPH case dataset (p-value < 0.05, pairwise Wilcoxon signed rank test with Bonferroni correction).

In this work, we fuse imaging features from Chest X-Ray (CXR) scans and audio features from dictations of a radiologist to improve thoracic disease classification. Recent deep learning-based disease classification methods mostly use imaging modalities. Dictation audio from a radiologist contains rich auxiliary diseaserelated contextual information. The main hypothesis of this proposed work is that leveraging complementary imaging and audio representations improves disease classification. We use shifting window (Swin) transformer architectures as encoders for both visual and audio modalities and finally fuse the feature representations using cross-correlational feature multiplication fusion strategy. This fused feature representation is fed to a classification head for downstream disease classification. We experimentally show that the proposed fused model outperforms the individual modality models for multi-class thoracic disease classification that includes normal, pneumonia, and congestive heart failure cases. We report F1-score of 0.5415 and 0.5353 for shifting window transformer base and small architectures respectively, for fused modalities, while the corresponding baselines are reported at 0.5046 and 0.5076 for the audio modality and 0.4676 and 0.5261 for the imaging modality, respectively.

Deep learning (DL) algorithms can be used to automate paranasal anomaly detection from Magnetic Resonance Imaging (MRI). However, previous works relied on supervised learning techniques to distinguish between normal and abnormal samples. This method limits the type of anomalies that can be classified as the anomalies need to be present in the training data. Further, many data points from normal and anomaly class are needed for the model to achieve satisfactory classification performance. However, experienced clinicians can segregate between normal samples (healthy maxillary sinus) and anomalous samples (anomalous maxillary sinus) after looking at a few normal samples. We mimic the clinicians ability by learning the distribution of healthy maxillary sinuses using a 3D convolutional auto-encoder (cAE) and its variant, a 3D variational autoencoder (VAE) architecture and evaluate cAE and VAE for this task. Concretely, we pose the paranasal anomaly detection as an unsupervised anomaly detection problem. Thereby, we are able to reduce the labelling effort of the clinicians as we only use healthy samples during training. Additionally, we can classify any type of anomaly that differs from the training distribution. We train our 3D cAE and VAE to learn a latent representation of healthy maxillary sinus volumes using L1 reconstruction loss. During inference, we use the reconstruction error to classify between normal and anomalous maxillary sinuses. We extract sub-volumes from larger head and neck MRIs and analyse the effect of different fields of view on the detection performance. Finally, we report which anomalies are easiest and hardest to classify using our approach. Our results demonstrate the feasibility of unsupervised detection of paranasal anomalies from MRIs with an AUPRC of 85% and 80% for cAE and VAE, respectively.

Patients with oropharyngeal cancer (OPC) treated with chemoradiation experience weight loss and tumor shrinkage. As a result, many of these patients will require a replan during radiation treatment. We aimed to develop a machine learning model to predict the need for a replan in patients with OPC (n=315). A total of 78 patients (25%) required a replan. The dataset was split into independent training (n=220) and testing (n=95) datasets. Tumor volumes and organs at risk (OARs) were contoured on planning CT images prior to treatment. PyRadiomics was used to compute radiomic features from the primary tumor, nodal volumes, and parotid glands. Clinical and dose features extracted from the OARs were collected and those significantly associated with the need for a replan in the training dataset were used in a baseline model. Feature selection was applied to select the optimal radiomic features. Classifiers were built using the non-correlated selected radiomic, clinical, and dose features on the training dataset and performance was assessed in the testing dataset. Three clinical and one dose feature were incorporated into the baseline model, as well as into the combined models. Eight predictive radiomic features were selected. The baseline model achieved an AUC of 0.66 [95% CI: 0.51-0.79] in the testing dataset. The Naïve Bayes was the top-performing radiomics model and achieved an AUC of 0.80 [95% CI: 0.69-0.90] in the testing dataset, outperforming the baseline model (p=0.005). This model could assist physicians in identifying patients who may benefit from a replan, improving the replanning workflow.

Osteoporosis is a complex multifactorial skeletal disease and has become a major socioeconomic issue, causing tremendous hospitalization and rehabilitation costs. In addition to age, premature menopause or the use of oral corticosteroids, vertebral fractures are regarded as main risk factors for developing osteoporosis and associated fractures. In this work, we adapt an existing automated pipeline to classify fracture grades of individual vertebrae and the spine as a whole: First, vertebral body centers were identified on CT images by a hierarchical neural network. Next, the fracture grades of individual vertebrae were processed by a multi-head, feed-forward convolutional neural network. The sum of the classified grades was then evaluated according to the German radiology guidelines. A hyperparameter search on validation data showed the most promising results for an output configuration based on three sequentially applied binary classification outputs trained using binary cross-entropy: Grade 0–1 vs 2–3, and 0 vs 1–3, and 0–2 vs 3. In a cross-validation setting on 159 low-dose CT images, our pipeline accurately classified patients to have sum-scores ≥ 2 with sensitivities and specificities of 90 % ± 5.0 % and 87 % ± 2.7 %, respectively. As our method was based on classifying individual vertebrae, we were able to provide both the fracture position and severity to enhance transparency, interpretability and usability.

Robust localization of lymph nodes (LNs) in multiparametric MRI (mpMRI) is critical for the assessment of lymphadenopathy. Radiologists routinely measure the size of LN to distinguish benign from malignant nodes, which would require subsequent cancer staging. Sizing is a cumbersome task compounded by the diverse appearances of LNs in mpMRI, which renders their measurement difficult. Furthermore, smaller and potentially metastatic LNs could be missed during a busy clinical day. To alleviate these imaging and workflow problems, we propose a pipeline to universally detect both benign and metastatic nodes in the body for their ensuing measurement. The recently proposed VFNet neural network was employed to identify LN in T2 fat suppressed and diffusion weighted imaging (DWI) sequences acquired by various scanners with a variety of exam protocols. We also use a selective augmentation technique known as Intra-Label LISA (ILL) to diversify the input data samples the model sees during training, such that it improves its robustness during the evaluation phase. We achieved a sensitivity of ∼83% with ILL vs. ∼80% without ILL at 4 FP/vol. Compared with current LN detection approaches evaluated on mpMRI, we show a sensitivity improvement of ∼9% at 4 FP/vol.

Complete pulmonary fissures are assessed on computed tomography (CT) and are required for emphysema patients to be successfully treated with endobronchial valve (EBV) therapy. We propose a deep learning (DL) pipeline that uses a patch-based approach to quantitatively assess fissure completeness on CT and evaluate it in a clinical trial cohort. From the EBV for emphysema palliation trial (VENT), 130 CT scans were used in this study. The DL model utilizes nnU-Net as a backbone for the automatic pre- and post-processing of CT images and configuration of a 3D U-Net to segment patches of fissure and non-fissure. Five-fold cross validation is applied for training and inferences are obtained using a sliding window approach. Average symmetric surface distance (ASSD) and surface dice coefficient (SDC) at a threshold of 2mm evaluates segmentation performance. A fissure integrity score (FIS) is calculated as the percentage of complete fissure voxels along the surface of the assumed interlobar region using pulmonary lobar segmentations. A predicted-FIS (p-FIS) is quantified from the CNN output and is compared to the reference-FIS (r-FIS) as complete (FIS≥90%), partial (10%≤ FIS< 90%) or absent (FIS< 10%). A mean(±SD) SDC of 0.95(±0.037) is achieved for the left oblique fissure (LOF); 0.84(±0.144) for the right horizontal fissure (RHF), and 0.94(±0.098) for the right oblique fissure (ROF). Concordance rate of p-FIS and r-FIS is 86.4%, 88.6%, and 86.4% for the LOF, RHF, and ROF, respectively. A DL pipeline using a patch-based approach has potential to segment interlobar fissures from CT to quantitatively assess fissure completeness.

Colorectal polyps (CRPs) are potential precursors of colorectal cancer (CRC), one of the most common types of cancer worldwide. Computer-Aided Diagnosis (CADx) systems can play a crucial role as a second opinion for endoscopists in characterizing CRPs and contribute to the diagnostic performance of colonoscopies. Despite their potential, deep neural network-based systems often tend to overestimate the confidence about their decisions and provide predictive probabilities that are poorly related to their classification accuracy. Quantifying uncertainty of such supportive systems is crucial for optimal clinical workflow integration and physician’s acceptance. Thus, a trustworthy CADx system is expected to provide accurate and well-calibrated classification confidence. Transfer learning from either natural image datasets, such as ImageNet, or other datasets with similar modalities, has been widely used for improving the accuracy of deep learning-based systems in medical image classification. In this paper, we study the impact of domain-specific pretraining on the calibration and the overall performance of a CADx system for CRP characterization. We evaluate our hypothesis on a fully deterministic and a hybrid Bayesian version of each approach using a generic ResNet50 architecture. Experimental results demonstrate the effectiveness of domain-specific pretraining in achieving a higher overall characterization AUC. Additionally, the in-domain and out-of-domain pretrained models portray similar calibration error rates, however, their corresponding hybrid Bayesian models offer higher robustness with improved calibration performance. A hybrid Bayesian version of a domain-specific pretraining approach has shown to significantly improve the accuracy and reliability of CADx systems used for CRP characterization and similar positive effects may be expected for other medical imaging applications.

Assessing the generalizability of deep learning algorithms based on the size and diversity of the training data is not trivial. This study uses the mapping of samples in the image data space to the decision regions in the prediction space to understand how different subgroups in the data impact the neural network learning process and affect model generalizability. Using vicinal distribution-based linear interpolation, a plane of the decision region space spanned by the random ‘triplet’ of three images can be constructed. Analyzing these decision regions for many random triplets can provide insight into the relationships between distinct subgroups. In this study, a contrastive self-supervised approach is used to develop a ‘base’ classification model trained on a large chest x-ray (CXR) dataset. The base model is fine-tuned on COVID-19 CXR data to predict image acquisition technology (computed radiography (CR) or digital radiography (DX) and patient sex (male (M) or female (F)). Decision region analysis shows that the model’s image acquisition technology decision space is dominated by CR, regardless of the acquisition technology for the base images. Similarly, the Female class dominates the decision space. This study shows that decision region analysis has the potential to provide insights into subgroup diversity, sources of imbalances in the data, and model generalizability.

Uncertainty quantification in medical images has become an essential addition to segmentation models for practical application in the real world. Although there are valuable developments in accurate uncertainty quantification methods using 2D images and slices of 3D volumes, in clinical practice, the complete 3D volumes (such as CT and MRI scans) are used to evaluate and plan the medical procedure. As a result, the existing 2D methods miss the rich 3D spatial information when resolving the uncertainty. A popular approach for quantifying the ambiguity in the data is to learn a distribution over the possible hypotheses. In recent work, this ambiguity has been modeled to be strictly Gaussian. Normalizing Flows (NFs) are capable of modelling more complex distributions and thus, better fit the embedding space of the data. To this end, we have developed a 3D probabilistic segmentation framework augmented with NFs, to enable capturing the distributions of various complexity. To test the proposed approach, we evaluate the model on the LIDC-IDRI dataset for lung nodule segmentation and quantify the aleatoric uncertainty introduced by the multi-annotator setting and inherent ambiguity in the CT data. Following this approach, we are the first to present a 3D Squared Generalized Energy Distance (D2 GED) of 0.401 and a high 0.468 Hungarian-matched 3D IoU. The obtained results reveal the value in capturing the 3D uncertainty, using a flexible posterior distribution augmented with a Normalizing Flow. Finally, we present the aleatoric uncertainty in a visual manner with the aim to provide clinicians with additional insight into data ambiguity and facilitating more informed decision-making.

Pulmonary emphysema is a form of Chronic Obstructive Pulmonary Disease (COPD) and a chronic lung condition that results in a breakdown of alveoli walls. Quantitative Computed Tomography (CT) is increasingly used to assess the presence or progression of emphysema. CT quantifications are affected by the acquisition protocols and scanner makes and models. This variability is a major concern for cross-sectional and longitudinal disease characterizations with largescale, multi-institutional datasets. Therefore, CT images need to be harmonized to reflect the patient condition and not the attributes of the imaging systems. The purpose of this study was to develop a physics-based harmonization framework that transforms CT images to a reference quality index (iso resolution and noise conditions) enabling robust emphysema quantifications across varied CT conditions. The harmonizer was developed using a virtual imaging trial (VIT) platform taking advantage of ground truth knowledge and control over CT parameters in VIT. The developed harmonizer was applied to clinical data from the COPDGene dataset to demonstrate its clinical utility. The established imaging biomarkers of “LAA-950” and “Perc15” were selected for emphysema quantifications. Results demonstrated that the harmonizer improved the quantification performance by reducing the bias in “LAA-950" from 7.03 (𝐶𝐼: [6.38, 7.68]) to 0.14 (𝐶𝐼: [0.08, 0.20]) after matching for kernel and from 2.48 (𝐶𝐼: [2.21, 2.76]) to −0.34 (𝐶𝐼: [−0.48, −0.20]) after matching for noise settings on the COPDGene dataset. This developed harmonization framework provides robust emphysema quantifications, enabling objective disease characterizations in large-scale, multi-center, and longitudinal studies.

Accurate polyp segmentation from colonoscopy is essential for the early detection of colorectal cancer. However, the variety of polyps manifested in images and the blurry boundary between a polyp and its surrounding mucosa make segmentation challenging. Hence, it is crucial to accurately identify regional boundaries of polyps in colonoscopy. In this paper, we propose Gated Semantic Boundary Network (Gate-SBNet), which is a novel twostream CNN architecture based on an encoder-decoder framework to segment polyps in colonoscopy images. One stream of Gate-SBNet uses a pre-trained ConvNeXt-B model from the image classification task as the semantic encoder to obtain multi-level semantic features from colonoscopy images. Another branch uses the Semantic Boundary Learning Module (SBLM) to learn boundary features based on multi-level semantic features, which process information in parallel with the semantic encoder. By introducing the Gate Convolution Layer (GCLs) into the SBLM module, the semantic information is converted more accurately to boundary information. Therefore, only boundary-related information will be processed by the SBLM. Then, we merge semantic and boundary features as input to an Unet model to obtain the final segmentation result. Our proposed approach was evaluated on five benchmark datasets: Kvasir, CVC-ClinicDB, CVC-ColonDB, CVC-300, and ETIS-LaribPolypDB. Experiments have demonstrated that it is an efficient architecture and capable of making accurate predictions about object boundaries and significantly improving the performance of finding thin and small objects.

Colorectal cancer is among the most prevalent cause of cancer-related mortality worldwide. Detection and removal of polyps at an early stage can help reduce mortality and even help in spreading over adjacent organs. Early polyp detection could save the lives of millions of patients over the world as well as reduce the clinical burden. However, the detection polyp rate varies significantly among endoscopists. There is numerous deep learning-based method proposed, however, most of the studies improve accuracy. Here, we propose a novel architecture, Residual Upsampling Network (RUPNet) for colon polyp segmentation that can process in realtime and show high recall and precision. The proposed architecture, RUPNet, is an encoder-decoder network that consists of three encoders, three decoder blocks, and some additional upsampling blocks at the end of the network. With an image size of 512 × 512, the proposed method achieves an excellent real-time operation speed of 152.60 frames per second with an average dice coefficient of 0.7658, mean intersection of union of 0.6553, sensitivity of 0.8049, precision of 0.7995, and F2-score of 0.9361. The results suggest that RUPNet can give real-time feedback while retaining high accuracy indicating a good benchmark for early polyp detection.

Due to the superior soft tissue contrast in magnetic resonance imaging (MRI), MRI may be well suited for renal mass characterization (e.g., benign vs. malignant). Though renal mass detection and characterization using deeplearning (DL) methods have been extensively studied for CT images, those same tasks are yet to be investigated on MR images. Existing algorithms for renal mass characterization require manual segmentation, therefore development of algorithms to localize and detect renal masses is important fully automatically. In this study, we developed a DL-based fully automated renal mass detection model on T2- weighted (T2W) images. In a cascaded approach, we initially segmented kidneys as a region-of-interest (ROI) using 2D U-Net model, then renal masses were detected on segmented kidneys using 2D U-Net convolutional neural network (CNN) model. We trained our model on randomly selected 80% of dataset using 5-fold cross-validation technique and evaluated on remaining 20% test cases for renal mass detection. Our T2W MRI dataset contained 108 patients with malignant (renal cell carcinoma- clear cell, papillary and chromophobe) and benign (fat poor angiomyolipoma-fpAML, oncocytomas) renal masses. The U-Net model for renal mass detection generated Dice similarity coefficient (DSC) of 90.00 ± 6.00 % (mean ± standard deviation). When localized kidneys evaluated on U-Net renal mass detection model yielded a sensitivity/recall, and specificity of 76.49% and 86.55%, respectively. Thus, our proposed fully automated cascaded approach has potential to be used as the first step in renal mass characterization study on T2W MRI images.

To cope with the growing prevalence of colorectal cancer (CRC), screening programs for polyp detection and removal have proven their usefulness. Colonoscopy is considered the best-performing procedure for CRC screening. To ease the examination, deep learning based methods for automatic polyp detection have been developed for conventional white-light imaging (WLI). Compared with WLI, narrow-band imaging (NBI) can improve polyp classification during colonoscopy but requires special equipment. We propose a CycleGAN-based framework to convert images captured with regular WLI to synthetic NBI (SNBI) as a pre-processing method for improving object detection on WLI when NBI is unavailable. This paper first shows that better results for polyp detection can be achieved on NBI compared to a relatively similar dataset of WLI. Secondly, experimental results demonstrate that our proposed modality translation can achieve improved polyp detection on SNBI images generated from WLI compared to the original WLI. This is because our WLI-to-SNBI translation model can enhance the observation of polyp surface patterns in the generated SNBI images.

Machine learning models have been widely used in lung cancer computer-aided diagnosis (CAD) studies. However, the heterogeneity in the visual appearance of lung nodules as well as lack of consideration of hidden subgroups in the data are significant obstacles to generating accurate CAD outcomes across all nodule instances. Previous lung cancer CAD models aim to achieve Empirical Risk Minimization (ERM), which leads to a high overall accuracy but often fails at predicting certain subgroups caused by the lung cancer heterogeneity. In this study, we aim to discover hidden lung nodule subgroups and enhance the malignancy classification performance of the worst-performance subgroup when compared to traditional ERM methods. We experiment with three different stratification methods for lung nodule subgroup discovery: spiculation-based, clustering-based, and malignancybased. A high overlap between subgroup labels generated from the clustering-based approach and labels obtained from radiologists’ semantic annotations indicates our discovered subgroups are semantically meaningful. We successfully improved the worst malignancy classification performance lung nodule subgroup through utilizing Group Distributionally Robust Optimization (gDRO) when compared to ERM as a baseline. Our study creates a framework for augmenting lung nodule malignancy classification under domain shift situations caused by the disease heterogeneity and underscores the necessity of addressing hidden stratification for future CAD schemes.

Computer-Aided Diagnosis (CADx) systems for characterization of Narrow-Band Imaging (NBI) videos of suspected lesions in Barrett’s Esophagus (BE) can assist endoscopists during endoscopic surveillance. The real clinical value and application of such CADx systems lies in real-time analysis of endoscopic videos inside the endoscopy suite, placing demands on robustness in decision making and insightful classification matching with the clinical opinions. In this paper, we propose a lightweight int8-based quantized neural network architecture supplemented with an efficient stability function on the output for real-time classification of NBI videos. The proposed int8-architecture has low-memory footprint (4.8 MB), enabling operation on a range of edge devices and even existing endoscopy equipment. Moreover, the stability function ensures robust inclusion of temporal information from the video to provide a continuously stable video classification. The algorithm is trained, validated and tested with a total of 3,799 images and 284 videos of in total 598 patients, collected from 7 international centers. Several stability functions are experimented with, some of them being clinically inspired by weighing low-confidence predictions. For the detection of early BE neoplasia, the proposed algorithm achieves a performance of 92.8% accuracy, 95.7% sensitivity, and 91.4% specificity, while only 5.6% of the videos are without a final video classification. This work shows a robust, lightweight and effective deep learning-based CADx system for accurate automated real-time endoscopic video analysis, suited for embedding in endoscopy clinical practice.

Distinguishing the types of liver tumors is important for determining treatment strategy. Dynamic contrast-enhanced computed tomography (DCE-CT), which captures images at multiple timings after injecting contrast medium, provides essential characteristics to distinguish tumor types without biopsy. However, recognizing such characteristics takes a lot of time for radiologists because it requires to distinguish ambiguous image features comparing multiple images. Although several studies have proposed systems that classify tumor types from DCE-CT images, these systems usually output only classification results without the basis of classification such as the tumor characteristics. In this study, we propose a novel liver tumor characterization system that analyzes multi-phase DCE-CT images to help radiologists classify tumor types. We defined a list of eight essential tumor characteristics that radiologists commonly find to distinguish tumor types such as hepatocellular carcinomas (HCC), hemangiomas, and metastases. To deal with variable number of input images, we propose three deep neural network classification models that can take both two- and three-phase DCE-CT images as input. Using a dataset consisting of 3,318 tumors with labeled characteristics, each model was trained to classify the eight characteristics and validated. Evaluation results showed high discrimination performance exceeding 91% in ROC-AUC on average.

Convolutional neural networks (CNNs) are a popular choice for medical image segmentation. However, they may be challenged by the large inter-subject variation in organ shapes and sizes due to CNNs typically employing convolutions with fixed-sized local receptive fields. To address this limitation, we proposed multi-scale aggregated residual convolution (MARC) and iterative multi-scale aggregated residual convolution (iMARC) to capture finer and richer features at various scales. Our goal is to improve single convolutions’ representation capabilities. This is achieved by employing convolutions with varying-sized receptive fields, combining multiple convolutions into a deeper one, and dividing single convolutions into a set of channel-independent sub-convolutions. These implementations result in an increase in their depth, width, and cardinality. The proposed MARC and iMARC can be easily integrated into general CNN architectures and trained end-to-end. To evaluate the improvements of MARC and iMARC on CNNs’ segmentation capabilities, we integrated MARC and iMARC into a standard 2D U-Net architecture for pancreas segmentation on abdominal computed tomography (CT) images. The results showed that our proposed MARC and iMARC enhanced the representation capabilities of single convolutions, resulting in improved segmentation performance with lower computational complexity.

Finding small lesions is very challenging due to lack of noticeable features, severe class imbalance, as well as the size itself. One approach to improve small lesion segmentation is to reduce the region of interest and inspect it at a higher sensitivity rather than performing it for the entire region. It is usually implemented as sequential or joint segmentation of organ and lesion, which requires additional supervision on organ segmentation. Instead, we propose to utilize an intensity distribution of a target lesion at no additional labeling cost to effectively separate regions where the lesions are possibly located from the background. It is incorporated into network training as an auxiliary task. We applied the proposed method to segmentation of small bowel carcinoid tumors in CT scans. We observed improvements for all metrics (33.5% → 38.2%, 41.3% → 47.8%, 30.0% → 35.9% for the global, per case, and per tumor Dice scores, respectively.) compared to the baseline method, which proves the validity of our idea. Our method can be one option for explicitly incorporating intensity distribution information of a target in network training.

Deep convolutional neural networks (CNNs) are widely used in medical image segmentation, but they tend to ignore smaller foreground classes in the training process and thus degrade segmentation accuracy, which is known as the class imbalance problem. Central venous catheter (CVC) segmentation suffers from such problems, leading to low accuracy. The purpose of this study is to address the class imbalance problem in CNN training for segmenting the right internal jugular lines (RIJLs), the most common type of CVCs, in chest X-ray (CXR) images. We applied an inverse class frequency weight to the standard Dice loss to formulate a class frequency weighted Dice loss (CFDL) function to train the CNNs. A U-net based segmentation model was constructed with multichannel pre-processing, including normalization, denoising, and histogram equalization, and post-processing including thresholding segmentation candidates and interpolating discontinued line segmentations. The segmentation model was trained on CXR images with the Dice loss and the CFDL respectively. A separate test set of images were used to evaluate the CNN output performances with the Dice Similarity Coefficient (DSC). Between the Dice loss and the CFDL, the CFDL-trained CNN generated segmentation results with a mean DSC of 0.581 on the separate test set, which indicates a statistically significant difference (p=0.001) from the Dice loss-trained CNN outputs with a mean DSC of 0.562. The inverse class frequency weighted Dice loss function improved the RIJL segmentation with a U-net to the state-of-the-art performance.

Purpose: Convolutional Neural Networks (CNN) are frequently used for organ segmentation in medical imaging. Many CNNs, however, struggle for acceptance into clinical practice because of mis-segmentations that are obvious to radiologists or other experts. We propose using a Cognitive AI framework that applies anatomical knowledge and machine reasoning to check and improve CNN segmentations and avoid obvious mis-segmentations. Methods: We used the open-source SimpleMind framework that allows post-processing and machine reasoning to be applied to CNN segmentation results. Within this framework, a 3D-UNet CNN was trained on 212 contrast-enhanced kidney CT scans. From an anatomical knowledge base, SimpleMind derived the following Cognitive AI post-processing steps: (1) identification of the abdomen, (2) segmentation of the spine by searching for bone in the posterior region of the abdomen, (3) CNN output separation into right and left kidneys using the spine as reference, (4) refinement of individual kidney segmentations using thresholds of volume, HU, and voxel count, which are designed to eliminate disconnected false positives, (5) morphological opening to reduce connected false positives bordering the kidneys’ segmentations. On a test set of 53 scans, using reference annotations, Dice Coefficient (DC), Hausdorff Distance (HD) and Average Symmetric Surface Distances (ASSD) were computed and compared for the CNN and post-processed outputs. Results: Post-processing with Cognitive AI reduced the kidney segmentation HD from 46.4±55.8 mm to 30.7±17.6 mm, with the decrease in variance being statistically significant (p = 0.0296). DC and ASSD metrics were also improved. Conclusions: This initial work demonstrates that the incorporation of anatomical knowledge using Cognitive AI techniques can improve the segmentation accuracy of CNNs. The CNN provides very good overall kidney segmentation performance, and in cases where segmentation errors occur the post processing was able to improve performance.

Deep learning models are widely studied for radiotherapy toxicity prediction; however, one of the major challenges is that they are complex models and difficult to understand. To aid in the creation of optimal dose treatment plans, it is critical to understand the mechanism and reasoning behind the network’s prediction, as well as the specific anatomical regions involved in toxicity. In this work, we propose a convolutional neural network to predict the toxicity after pelvic radiotherapy that is able to explain the network’s prediction. The proposed model analyses the dose treatment plan using multiple instance learning and convolutional encores. A dataset of 315 patients was included in the study, and experiments with both quantitative and qualitative approaches were conducted to assess the network’s performance.

Tuberculosis (TB) is still considered a leading cause of death and a substantial threat to global child health. Both TB infection and disease are curable using antibiotics. However, most children who die of TB are never diagnosed or treated. In clinical practice, experienced physicians assess TB by examining chest X-rays (CXR). Pediatric CXR has specific challenges compared to adult CXR, which makes TB diagnosis in children more difficult. Computer-aided diagnosis systems supported by Artificial Intelligence have shown performance comparable to experienced radiologist TB readings, which could ease mass TB screening and reduce clinical burden. We propose a multi-view deep learning-based solution which, by following a proposed template, aims to automatically regionalize and extract lung and mediastinal regions of interest from pediatric CXR images where key TB findings may be present. Experimental results have shown accurate region extraction, which can be used for further analysis to confirm TB finding presence and severity assessment.

The presence of plaques in the coronary arteries is a major risk to the patients’ life. In particular, non-calcified plaques pose a great challenge, as they are harder to detect and more likely to rupture than calcified plaques. While current deep learning techniques allow precise segmentation of real-life images, the performance in medical images is still low. This is caused mostly by blurriness and ambiguous voxel intensities of unrelated parts that fall on the same value range. In this paper, we propose a novel methodology for segmenting calcified and noncalcified plaques in CCTA-CPR scans of coronary arteries. The input slices are masked so only the voxels within the wall vessel are considered for segmentation, thus, reducing ambiguity. This mask can be automatically generated via a deep learning-based vessel detector, that provides not only the contour of the outer artery wall, but also the inner contour. For evaluation, we utilized a dataset in which each voxel is carefully annotated as one of five classes: background, lumen, artery wall, calcified plaque, or non-calcified plaque. We also provide an exhaustive evaluation by applying different types of masks, in order to validate the potential of vessel masking for plaque segmentation. Our methodology results in a prominent boost in segmentation performance, in both quantitative and qualitative evaluation, achieving accurate plaque shapes even for the challenging non-calcified plaques. Furthermore, when using highly accurate masks, difficult cases such as stenosis become segmentable. We believe our findings can lead the future research for high-performance plaque segmentation.

Heart disease is a common group of circulatory diseases, but it seriously affects people’s life and health. Among many heart diseases, ventricular tachycardia (VT) is the most common cause of death. The current clinical treatment plan is to treat ventricular tachycardia by finding the origin of ventricular activation and then ablating it with an invasive catheter. However, the longer the catheter remains in the body, the more dangerous it is for the patient, so it is of interest to achieve noninvasive assisted localization of the origin site. To solve this problem, we propose an end-to-end neural network structure, Densefomer (DenseNet-Transformer Network), in a data-driven manner. This simple and effective model can localize origin of ventricular activation using only 12-lead electrocardiogram (ECG). Densefomer introduces Transformer and DenseNet to obtain global and local feature information, respectively. Our model finally achieved a localization error of 10.79 mm and a classification accuracy of 58.16% on real clinical data.

Purpose: Hematoma expansion (HE) for patients with intracerebral hemorrhage (ICH) has been shown to be a predictor of clinical neurological deterioration in ICH patients. As of now, there is no diagnosis which may indicate HE at the time of presentation. In this study, a Random Forest-based machine learning model with clinical data from ICH patients was developed and used as input to predict HE. Materials and Methods: 200 ICH patients with known hematoma evolution, were enrolled in this study. Data included brain volume, and hematoma volume based on non-contrast CT (NCCT) measurements; and the following patient specific clinical variables: age, sex, Glasgow Coma Scale score (GCS), ICH score, NIH Stroke Scale (NIHSS) and time from onset of ICH to initial NCCT. Random Forest machine learning model was developed to predict HE using 104/26 subjects training/testing split. Grid search strategy tuned the classifier parameters and a 5-fold cross-validation approach was used during training. The performance of model was evaluated by sensitivity, specificity, and Area Under the Curve (AUC). Results: The developed Random Forest model was able to predict HE with sensitivity of 0.846, specificity of 0.769, AUC of 0.807. Hematoma volume and time from onset of ICH to initial NCCT were the most important features, followed by NIHSS and brain volume. Conclusion: A Random Forest-based machine learning model with multiple clinical data from ICH patients as input performed well in predicting HE. Brain volume may be a new predictor of hematoma expansion.

Lower-grade gliomas (LGG) can eventually progress to glioblastoma (GBM) and death. This study aimed to construct the radiomics model for predicting survival outcomes in glioma patients, compared the radiomics model with clinical and gene status models, and evaluated the combined models. Two direction approach was used: build models on GBM patients, and then verified in LGG patients and vice versa The Cancer Genome Atlas (TCGA) data set includes 102 patients in TCGA-GBM collection, and 107 patients in TCGA-LGG. The GBM data were divided randomly into seventy percent and thirty percent for training and testing. The LGG data set was used to validate. From the initial 704 MRI-based radiomics features of training set, we chose 17 optimal MRI-based radiomics signatures after consecutive selection steps to build the radiomics score of each patient as a representative of radiomics model. The iAUCs of combined models in training, testing, and validation sets were respectively 0.804 (95% CI, 0.741-0.866), 0.878 (95% CI, 0.802-0.955), and 0.802 (95% CI, 0.669-0.935), and those of radiomics models were 0.798 (95%CI, 0.743-0.852), 0.867 (95% CI, 0.736-0.999), and 0.717 (95% CI, 0.549-0.884). Applied the same consecutive selection steps, we chose 8 MRI-based radiomics signatures from LGG training set. The iAUCs of combined models in training, testing, and validation sets were respectively 0.842 (95% CI, 0.697_0.988), 0.894 (95% CI, 0.615_0.945), and 0.618 (95% CI, 0.526_ 0.710), and those of radiomics models were 0.780 (95%CI, 0.674_ 0.949), 0.832 (95% CI, 0.738_0.925), and 0.525 (95% CI, 0.506_0.545). In conclusion, the radiomics model can independently predict the overall survival of glioma patients, and the combined model integrating radiomics, clinical, and gene status models improved this ability.

When a CAD-AI network is created and employed, both safety and effectiveness need to be guaranteed for all subgroups of the target population. We present a novel toolbox for automatic slicing and performance assessment in a generic and modular approach, helping to find the subpopulations where cautiousness is warranted, and the model may need improvement. In a first step slices are generated and saved for further analysis inspired by the existing ‘Slice Finder’ algorithm. Depending on the type of AI task (classification, object detection, segmentation, instance segmentation...) multiple metrics are evaluated. Both labeled (specificity, sensitivity...), unlabeled (outlier score, confidence score...) and user-defined metrics can be included. Optionally, the confidence interval (CI) is calculated. In a last step, the metric values and CI are used to rank the slices to quickly find the slices of interest. Custom ranking methods can be added, keeping the full process from slice generation up to and including visualization modular and customizable. We illustrate the toolbox with a dermatology classification and object detection use-case. First a single model is evaluated down to crosses of three slices where slices of interest are detected on degree three which would be difficult to find if not automated. Additionally, the usage of unlabeled metrics such as outlier score is illustrated to automatically find slices of interest.

Although many AI-based scientific works regarding chest X-ray (CXR) interpretation focused on COVID-19 diagnosis, fewer papers focused on other relevant tasks, like severity estimation, deterioration, and prognosis. The same holds for explainable decisions to estimate COVID-19 prognosis as well. The international hackathon launched during Dubai Expo 2020, aimed at designing machine learning solutions to help physicians formulate COVID-19 patients’ prognosis, was the occasion to develop a machine learning model capable of predicting such prognoses and justifying them through interpretable explanations. The large hackathon dataset comprised subjects characterized by their CXR and numerous clinical features collected during triage. To calculate the prognostic value, our model considered both patients’ CXRs and clinical features. After automatic pre-processing to improve their quality, CXRs were processed by a Deep Learning model to estimate the lung compromise degree, which has been considered as an additional clinical feature. Original clinical parameters suffered from missing values that were adequately handled. We trained and evaluated multiple models to find the best one and fine-tune it before the inference process. Finally, we produced novel explanations, both visual and numerical, to justify the model predictions. Ultimately, our model processes a CXR and several clinical data to estimate a patient’s prognosis related to the COVID-19 disease. It proved to be accurate and was ranked second in the final rankings with 75%, 73.9%, and 74.4% in sensitivity, specificity, and balanced accuracy, respectively. In terms of model explainability, it was ranked first since it was agreed to be the most interpretable by health professionals.

Magnetic Resonance imaging (MRI) is a non-invasive modality for diagnosing prostate carcinoma (PCa) and deep learning has gained increasing interest in MR images. We propose a novel 3D Capsule Network to perform low grade vs high grade PCa classification. The proposed network utilizes Efficient CapsNet as backbone and consists of three main components, 3D convolutional blocks, depth-wise separable 3D convolution, and self-attention routing. The network employs convolutional blocks to extract high level features, which will form primary capsules via depth-wise separable convolution operations. A self-attention mechanism is used to route primary capsules to higher level capsules and finally a PCa grade is assigned. The proposed 3D Capsule Network was trained and tested using a public dataset that involves 529 patients diagnosed with PCa. A baseline 3D CNN method was also experimented for comparison. Our Capsule Network achieved 85% accuracy and 0.87 AUC, while the baseline CNN achieved 80% accuracy and 0.84 AUC. The superior performance of Capsule Network demonstrates its feasibility for PCa grade classification from prostate MRI and shows its potential in assisting clinical decision-making.

Clinical decision-making in oncology involves multi-modal data such as radiology and clinical factors. In recent years, several computer-aided multi-modal decision systems have been developed to predict the recurrence of hepatocellular carcinoma (HCC) after hepatectomy, but these models simply concatenate features naively at the feature level, which would create redundancy and hinder model performance. Particularly, we found that integrating deep multi-modal models based on tensor fusion, which can better handle relevant information between different modalities and reduce model redundancy to obtain better results. This finding suggests the presence of independent, complementary prognostic information between radiology and clinical modalities. Hence we propose a multi-modal learning network that uses a tensor-based fusion approach with clinical parameters, radiomics features and corresponding complex interrelationships between pathological data to predict postoperative early recurrence of single hepatocellular carcinoma. In order to maximize the information gathered from each modality, we introduced a multi-modal fusion loss function based on orthogonal loss to assist multi-modal training. With IRB’s approval, we collected 176 cases with radiomics (MRI) and pathological features diagnosed by experienced clinicians, and established an approach based on tensor fusion that takes pathological findings and postoperative early recurrence as ground truth. Training with 140 cases and tested with 36 cases, the proposed network achieved the AUC of 0.883, which showed great potential in predicting postoperative early HCC recurrence. The ablation experimental results also showed that the auxiliary loss function had a statistically significant improvement in model performance (p<0.01).

This study aimed to explore topological prediction models for relapses, i.e., locoregional relapse (LRR) and distant metastasis (DM), of stage I patients with non-small cell lung cancer (NSCLC) prior to stereotactic ablative radiotherapy (SABR). Pretreatment planning computed tomography (CT) images of 125 primary NSCLC patients with SABR were divided into training (n = 65) and test datasets (n = 60). The signatures on lung tumor heterogeneity in CT images were constructed with conventional wavelet features (WFs), invariant topological features (original: TFs, inverted: iTFs), and their combined features (TWFs, iTWFs). Invariant topological features are related to intrinsic holes or heterogeneity in the tumors. The predictability of the tumor volumes which may be associated with lung cancer prognosis was also delved into with the image signatures. The patients were stratified into high-risk and low-risk groups using a radiomics score calculated from the signature. The predictability was evaluated using a p-value (log-rank test) between Kaplan–Meier (KM) curves of high-risk and low-risk groups, a concordance index (c-index), and a multiplication of negative logarithm of p-value and c-index (nLPC), which was considered a comprehensive evaluation index. For the test dataset, the iTFs and WFs combined with the tumor volumes had statistically significant differences of p-values (< 0.05) of the KM curves and higher nLPCs for the relapses. The signatures derived from inverted topology-based features and wavelet features combined with tumor volumes showed the potential of improving the high-risk and low-risk stratification for the relapses.

Accurate prediction of individual breast cancer risk paves the way for personalised prevention and early detection. Whilst well-established clinical risk factors can be used to stratify the population into risk groups, the addition of genetic information and breast density has been shown to improve prediction. Machine learning enabled automatic risk prediction provides key advantages over existing methods such as the ability to extract more complex information from mammograms. However, this is a challenging area of research, partly due to the lack of data within the field, therefore there is scope for novel approaches. Our method uses Multiple Instance Learning in tandem with attention in order to make accurate, short-term risk predictions from full-sized mammograms taken prior to the detection of cancer. This approach ensures small features like calcifications are not lost in a downsizing process and the whole mammogram is analysed effectively. An attention pooling mechanism is designed to highlight patches of increased importance and improve performance. Additionally, this increases the interpretability of our model as important patches can be shown in a saliency map. We also use transfer learning in order to utilise a rich source of screen-detected cancers and evaluate whether a model trained to detect cancers in mammograms allows us also to predict risk in priors. Our model achieves an AUC of 0.635 (0.600,0.669) in cancer-free screening mammograms of women who went on to a screen-detected or interval cancer between 5 and 55 months and an AUC of 0.804 (0.777,0.830) in screen-detected cancers.

Embedded processing architectures are often integrated into devices to develop novel functions in a cost-effective medical system. In order to integrate neural networks in medical equipment, these models require specialized optimizations for preparing their integration in a high-efficiency and power-constrained environment. In this paper, we research the feasibility of quantized networks with limited memory for the detection of Barrett’s neoplasia. An Efficientnet-lite1+Deeplabv3 architecture is proposed, which is trained using a quantizationaware training scheme, in order to achieve an 8-bit integer-based model. The performance of the quantized model is comparable with float32 precision models. We show that the quantized model with only 5-MB memory is capable of reaching the same performance scores with 95% Area Under the Curve (AUC), compared to a fullprecision U-Net architecture, which is 10× larger. We have also optimized the segmentation head for efficiency and reduced the output to a resolution of 32×32 pixels. The results show that this resolution captures sufficient segmentation detail to reach a DICE score of 66.51%, which is comparable to the full floating-point model. The proposed lightweight approach also makes the model quite energy-efficient, since it can be real-time executed on a 2-Watt Coral Edge TPU. The obtained low power consumption of the lightweight Barrett’s esophagus neoplasia detection and segmentation system enables the direct integration into standard endoscopic equipment.

Brain metastases are the most common malignant form of tumors and occur in 10%-30% of adult patients with systematic cancer. With recent advances in treatment options, there is an increasing evidence that automated detection and segmentation from MRI can assist clinicians for diagnosis and therapy planning. In this study, we investigate the impact of data domain on self-supervised learning (SSL) for pretraining a deep learning network to detect and segment brain metastases on 3D post-contrast T1-weighted images. We performed pretraining a 3D patch-based U-Net using the Model Genesis framework on three subject cohorts that have different data domain. The pretrained networks were then finetuned on brain MR scans from patients with metastases as a downstream task dataset. We analyzed the impact of data domain on SSL by examining validation metric evolution, FROC analyses and testing performance of early-trained models and best-validated models. Our results suggested that, in the early stage of finetuning for the target task, SSL is crucial for faster training convergence and similar data domain on SSL could be helpful to attain improved detection and segmentation performance earlier. However, we observed that the importance of data domain similarity for SSL progressively diminished as training continued with sufficient amount of iterations in our relatively large data regime. After training convergence, the best-validated models pretrained with SSL provided enhanced detection performance over the model without pretraining regardless of data domain.

Diagnosis of breast cancer is often achieved through expert radiologist examination of medical images such as mammograms. Computer-aided diagnosis (CADx) methods can be useful tools in the medical field with applications such as aiding radiologists in making diagnosis decisions. However, such CADx systems require a sufficient amount of data to train on, in conjunction with efficient machine learning techniques. Our Spatially Localized Ensembles Sparse Analysis using Deep Features (DF-SLESA) machine learning model uses local information of features extracted from deep neural networks to learn and classify breast imaging patterns based on sparse approximations. We have also developed a new technique of patch sampling for learning sparse approximations and making classification decisions that we denote as PatchSample decomposition. The PatchSample method differs from our previous approach, our BlockBoost method, in that larger dictionaries are constructed that hold not just spatial-specific information, but a larger collective of visual information from all locations in the region of interest (ROI). Of note is that we trained and tested our method on a merged dataset of mammograms obtained from two sources. Experimental results have reached up to 67.80% classification accuracy (ACC) and 73.21% area under the ROC curve (AUC) using PatchSample decomposition on a merged dataset consisting of the MLO view regions of interest of the MIAS and CBIS-DDSM datasets.

Parkinson’s disease (PD) is the second most common neurodegenerative disease affecting 2-3% of the population over 65 years of age. Considerable research has investigated the benefit of using neuroimaging to improve PD diagnosis. However, it is challenging for medical experts to manually identify the subtle differences associated with PD in such complex data. It has been shown that machine learning models can achieve human-like accuracies for many computer-aided diagnosis applications. However, model performance usually depends on the amount and diversity of training data available, whereas most Parkinson’s disease classification models were trained on rather small datasets. Training data size and diversity can be increased by curating multi-site datasets. However, this may also increase biological and non-biological variances due to differences in participant cohorts, scanners, and data acquisition protocols. Thus, data harmonization is important to reduce those variances and enable the models to focus primarily on the patterns associated with PD. This work compares intensity harmonization techniques on 1796 MRI scans from twelve studies. Our results show that a histogram matching approach does not improve classification accuracy (78%) compared to the model trained on unharmonized data (baseline). However, it reduces the disparity between sensitivity and specificity from 81% and 73% to 77% and 79%, respectively. Moreover, combining histogram matching and least squares mean tissue intensity harmonization methods outperform the baseline model (accuracy of 74% compared to 67%) for an independent test set. Finally, our analysis considering sex (male, female) and groups (PD, healthy) shows that models trained on harmonized data exhibited reduced performance disparities between groups, which may be interpreted as a form of bias mitigation.

Sub-solid lung nodules present unique diagnostic challenges. Pre-surgical characterization of the degree of invasiveness may be inaccurate even with tissue sampling. We hypothesize that the high-throughput information captured by radiomic descriptors on CT images can characterize the invasiveness of sub-solid nodules. In this study, we analyzed immediate pre-surgical CT scans of 72 nodules from 68 patients. Region of interest (ROI) segmentation on the scans was performed by our fellowship trained chest radiologists using the ITK-SNAP software. Feature extraction on ROIs was performed using the CaPTk toolkit. Scan parameter heterogeneity can affect radiomic features. To overcome this, image parameters including scanner manufacturer and voxel spacing parameters were harmonized at each time point using a nested ComBat harmonization technique. Clinical variables of ethnicity, BMI, smoking status and pathological category were protected during harmonization, to prevent the removal of biological variables of interest. Features with negative Mean Decrease in Accuracy (MDA) metric (non-optimal prognostic value) were dropped. Dimensionality of the feature set was reduced using the first radiomic principal component (PC) as a representative feature. Multiple logistic regression analysis using radiomic PC and clinical factors revealed only radiomic PC to be a significant predictor of nodule invasiveness (p < 0.05). A model containing clinical variables gave an accuracy of 73% (AUC=0.58) in identifying invasive sub-solid nodules. The accuracy increased to 93% (AUC-0.88) with the addition of radiomic PC. We further wanted to investigate if the change in the radiomic descriptors of the nodule properties over time, can improve the diagnosis of nodule invasiveness. Thus, from our original set of patients, we identified a subset of 40 nodules from 37 patients, with a total of 2 CT scans (immediate pre-surgical scan and 1 additional time point) and a subset of 34 nodules from 29 patients, with a total of 3 CT scans (immediate pre-surgical scan and 2 additional time points), each scan separated by at least a 12-month from each other. The features were harmonized at each time point. Delta radiomic features were computed (features from immediate pre-surgical scan- previous time point). Dimensionality of the delta radiomics feature set was reduced using the first radiomic principal component (PC) as a representative feature. In the case of longitudinal analysis using two scans, the model containing clinical variables gave an accuracy of 70% (AUC-0.55) in the diagnosis of nodule invasiveness. The accuracy increased to 82% (AUC- 0.65) upon the addition of delta radiomic PC (immediate pre-surgicalfirst time point). In the case of longitudinal analysis using three scans, the model containing clinical variables gave an accuracy of 68% (AUC-0.54) in the diagnosis of nodule invasiveness. The accuracy increased to 79% (AUC- 0.63) upon the addition of the first delta radiomic PC (immediate pre-surgical-first time point) and the second delta radiomic PC (immediate pre-surgical-second time point). Thus, the representative radiomic PCs, describing the change in the properties of the nodules over time, were found to be significant predictors of nodule invasiveness and augmented the performance of standard prognostic clinical factors. Thus, the ability of radiomic descriptors to predict nodule invasiveness could potentially be utilized to facilitate management of sub-solid nodules identified on chest CT imaging.

Machine learning has been used to classify melanoma from 2D dermoscopic images, with diagnosis accuracy comparable to that of dermatologists. However, there are few approaches for reconstructing the Breslow depth of the melanoma, which is a crucial metric for skin cancer diagnosis and stage identification. To estimate the depth of tissue lesions, we propose a multiplexed Diffused Optical Imaging Generative Adversarial Network (mDOI-GAN) solution. mDOI-GAN is composed of two main parts: the reconstruction and the projection sections. The reconstruction section estimates the optimal structure of 3D tissue volume depending on the presented 2D reflectance images. The projection section simulates the light-tissue interaction, generating 2D diffusion reflectance images corresponding to the given composition of 3D tissue volume. These two sections work together to form an optimization loop. The network utilizes structurally illuminated 2D reflectance images captured from a low-cost and contact-free Diffuse Optical Imaging acquisition platform as inputs. By iteratively minimizing the distance between experimental and synthetic 2D reflectance images, we will obtain a global optimal 3D reconstruction of tissue volume. We train the network with a customized 3D synthetic dermatology dataset derived from the HAM10000 dataset. Our design has the advantage of introducing the physics modeling of steady-state diffuse optical imaging and 3D spatial probability density function into the deep learning network. Our model is posed to be more flexible, interpretable, and predictable than the current end-to-end, black-box neural network benchmarks.

The goal of personalized medicine is to tailor disease prevention, diagnosis and treatment to each individual, while considering their genes, environment and lifestyles. This emerging field recognizes the limitations of the one-size-fits-all approach to treating the “average patient”, which ignores the numerous, sometimes subtle differences between patients. Variations include the fact that diseases manifest differently amongst human beings. To address a more robust diagnosis across patients, researchers are exploring the topic of disease subtypes discovery or the discovery of different characterizations of the same diseases. From a machine learning perspective, this problem translates into a clustering (unsupervised grouping of data) task in a supervised setting. In other words, the superclasses are known but there exist various unknown substructures in the data within each superclass that improve the discrimination of the original classes. We apply the idea of subtype discovery to the diagnosis of lung cancer nodules by attempting to discover different types of malignant and benign nodules in imaging data. Early detection/diagnosis could improve patient survival rates for this type of cancer, which is the leading cause of cancer death in the U.S. Our method 1) finds robust—homogeneous by design and differientiable—subtypes of lung nodules through iterative K-means clustering that help classify them and 2) leaves some data unclustered. This set of unclustered or “hard” data represents images that cannot confidently be assigned to any subtypes and may require more resources (e.g., time or radiologists) to diagnose. Our approach is applied to the Lung Image Database Consortium (LIDC) data set. We hypothesize that our subtypes classification will outperform the classification of the original classes and produce quantitatively and qualitatively more meaningful representations of the diseases when compared not only to the original classes but also to subtypes produces by simply overclustering the data (i.e., producing more clusters than necessary to capture original classes or minimizing the clustering loss function without checking the content of clusters). We improve the performance by 11% over the original classification and provide a detail evaluation of our newly discovered sub-types.

Mathematical morphology is a technique frequently used in image processing, it has several applications such as segmentation, filtering, compression, edge detection, and feature extraction. Considering this last application here is presented a morphological and convolutional neural network (MCNN) that takes advantage of the different types of morphological operations – erosion, dilation, opening, and closing – by including them in a single layer. Three independent neural networks are used to learn information per channel, Random Forest is used at the end, fed with the three outputs of the NNs. The classification performance of the method was compared against three common CNN architectures: ResNet-18, ShuffleNet-V2, and MobileNet-V2. Two training approaches were used: training from scratch and using transfer learning. A glaucoma classification was conducted using the ORIGA dataset. The MCNN method obtained an AUC of 0.704 (0.672, 0.743 95% CI) with a performance similar to the other CNN methods when trained using transfer learning.

Although deep learning models have been widely used in medical imaging research field to perform lesion segmentation and classification tasks, several challenges remain to optimally apply deep learning models and improve model performance. The objective of this study is to investigate a new novel joint model and assess model performance improvement as increase of training dataset size. Specifically, we select and modify a novel J-Net as a joint model, which includes a two-way CNN architecture that combines a U-net model with an image classification model. A skin cancer dataset with 1200 images along with the annotated lesion masks and ground truth of “mild” and “severe” status is used. From this dataset, 11 subsets are randomly generated from 200 to 1200 images with an incremental rate of 100. Each subset is then divided into training, validation and testing groups using a ratio of 70:20:10, respectively. The performance of the new joint model is compared with two independent models to separately perform lesion segmentation and classification. The study results show when training the models using data subsets of 200 to 1200 images, accuracy levels increase from 0.80 to 0.92, or 0.86 to 0.95 in lesion segmentation. The lesion classification increases from 0.80 to 0.90, or 0.82 to 0.93 using two single models and one joint J-Net model, respectively. Thus, this study demonstrates that applying this new JNet joint model enables to achieve higher lesion segmentation and classification accuracy than two single models. Additionally, model performance also increases as increase of training dataset size.

Our purpose is to develop a breast lesion simulator and remover using Cycle-GAN that can simulate plausible breast lesions in normal mammograms and remove existing lesions from mammograms. We curated 10310 screening mammograms from 4832 women (2416 normal/healthy and 2416 with recalled lesions). We divided our dataset into development and testing with a ratio of 8:2. We segmented 400 by 400 pixel patches, containing lesions from recalled cases and the same sized patches from the center of a normal breast mammogram. CycleGAN consists of two GANs and each GAN works as a lesion simulator and lesion remover when we train it on recalled lesions and normal patches. We used the development set to optimize the Cycle-GAN. Once trained, we applied Cycle-GAN on our test set to create lesion-simulated-normal controls and lesion-removed-recalled cases. After that, we trained ResNet18 as our CADe algorithm and tested it on our test set under four conditions: 1) original images (ResNet18_Control), 2) original lesion cases + lesion simulated normal (ResNet18_LSimul), 3) lesion removed + original normal controls (ResNet18_LRemov), and 4) lesion removed + lesion simulated normal (ResNet18_{LSimul+LRemov}). The test AUCs of ResNet18_Control, ResNet18_LSimul, ResNet18_LRemov, and ResNet18_{LSimul+LRemov} were 0.92, 0.79, 0.79, and 0.6, and their pairwise differences were statistically significant (p-value ¡ 0.0001), except the difference between ResNet18_LSimul and ResNet_18LRemov. Cycle-GAN can add plausible lesions to normal mammograms and remove existing lesions from mammograms such that the resulting images confused the computer-aided detection algorithm trained on the original mammograms.

Myocardial Infarction (MI), commonly known as heart attack, is the irreversible death of the Myocardium’s tissue due to oxygen deprivation for an extended period of time. LGE-MRI scans are considered the defacto in the diagnosis and prognosis of MI. Still, they require manually segmenting the Myocardium and the infarcted tissue, which is a complex and time-consuming task. Hence, an automatic segmentation method of the Myocardium tissue is highly desirable. CNNs (Convolutional Neural Networks) are used extensively in cardiac tissue segmentation in general and for solving this problem particularly. EMIDEC (automatic Evaluation of Myocardial Infarction from Delayed Enhancement Cardiac MRI) challenge in MICCAI 2020 provides a good overview of the different CNN-based architectures used for MI segmentation. Still, they required complex pipelines to achieve state-of-the-art results and the research followed the challenge tends to go in the same direction of building more complex pipelines revolving around the famous CNN-UNet either 2D or 3D. In this paper, we present a different direction by presenting a simple 2D novel architecture based on Self-Attention Transformers offering a possible alternative to the CNN-UNet as a building block for bigger systems. We introduce NesT-UNet a novel segmentation architecture based on the NesT architecture as an encoder and we also introduce a novel decoder inspired by the same architecture. NesT achieved state-of-the-art results on ImageNet and CIFAR classification tasks with minimal training compared to other transformer networks. 2D NesT-UNet produced results comparable to the state-of-the-art on the EMIDEC dataset using a simple training process and an extra self-supervised pre-training step to improve the network’s performance. we also present a novel loss function for false positives reduction.

The development of Artificial Intelligence (AI) for detection and characterization of Pancreatic Ductal Adenocarcinoma (PDAC) is a challenging task, since PDAC data is scarce compared to data of other types of cancer. However, due to the high mortality rate of the disease, early detection is crucial. For this reason, recent work has focused on exploiting indirect pathological features, e.g. dilated bile ducts due to tumor involvement, as an additional input for supportive algorithms. However, the presented methods require manual annotations of several structures in a CT volume, which is a cumbersome task and not feasible in clinical practice. Therefore, this work investigates the automated segmentation of bile ducts to facilitate improved tumor detection by such methods. Using a coarse-to-fine segmentation architecture, the pancreas, pancreatic duct and the common bile duct are segmented from 3D CT-scans. The resulting yet individual segmentations form a primary stage, of which the outputs are supplied as input to a secondary pre-trained U-Net-based PDAC detection algorithm, to ultimately detect tumors. We evaluate the performance of the proposed primary segmentation and secondary detection models on a publicly available test set in terms of mean Dice Similarity Coefficient (DSC). The pancreas, common bile duct and pancreatic duct are segmented with a mean DSC of 0.86, 0.69 and 0.57, respectively. With these segmentations as input, a tumor detection sensitivity of 100% is maintained for the tumor detection model. This continuously high detection sensitivity for tumor detection is comparable to the tumor detection score achieved by using manually annotated structures. This study highlights the benefit of primarily segmenting relevant structures, to use as input for a secondary model for final PDAC detection.

Pulmonary nodules are the principal lung cancer indicator, whose malignancy is mainly related to their size, morphological and textural features. Computational deep representations are today the most common tool to characterize lung nodules but remain limited to capturing nodule variability. In consequence, nodule malignancy classification from CT observations remains an open problem. This work introduces a multi-head attention network that takes advantage of volumetric nodule observations and robustly represents textural and geometrical patterns, learned from a discriminative task. The proposed approach starts by computing 3D convolutions, exploiting textural patterns of volumetric nodules. Such convolutional representation is enriched from a multi-scale projection using receptive field blocks, followed by multiple volumetric attentions that exploit non-local nodule relationships. These attentions are fused to enhance the representation and achieve more robust malignancy discrimination. The proposed approach was validated on the public LIDC-IDRI dataset, achieving a 91.82% in F1-score, 91.19% in sensitivity, and 92.43% in AUC for binary classification. The reported results outperform the state-of-the-art strategy with 3D nodule representations.

We are developing deep learning models for the segmentation of mouse tibia in MRI scans by utilizing three U-Net architectures: Attention, Inception, and basic U-Net, on a data set of 32 mice with 158 MRI scans. The data set was split into training (23 mice, 108 scans), validation (3 mice, 17 scans), and test (6 mice, 33 scans) sets. Two expert annotators (EA1 and EA2) provided manual 3D segmentations of the tibia on the MRI scans. EA1 provided outlines on all MRI scans, which were used as the reference for the training, validation, and testing of U-net models. EA2 provided outlines on the validation and test set, which were used for the assessment of inter-observer reference variability. The model performance was evaluated based on the average Jaccard index (%AJI), average volume intersection ratio (%AVI), average volume error (%AVE), and average Hausdorff distance (AHD, mm). For the test set, the %AJI with reference to EA1 was 83.45 ± 5.11 for the Attention U-Net, 83.05 ± 6.21 for the Inception U-Net, and 83.38 ± 5.36 for the basic U-Net. The %AJI was 80.70 ± 2.91 for EA1 versus EA2 and 79.70 ± 6.28 for Attention U-Net versus EA2. The variability between the U-Net models and EA1 and EA2 references was similar to the variability between EA1 and EA2. All 3 U-Net architectures achieved similar performances with the Attention U-Net performing marginally better.

Obstructive sleep apnea (OSA) is a prevalent disease affecting 10 to 15% of Americans and nearly one billion people worldwide. It leads to multiple symptoms including daytime sleepiness; snoring, choking, or gasping during sleep; fatigue; headaches; non-restorative sleep; and insomnia due to frequent arousals. Although polysomnography (PSG) is the gold standard for OSA diagnosis, it is expensive, not universally available, and time-consuming, so many patients go undiagnosed due to lack of access to the test. Given the incomplete access and high cost of PSG, many studies are seeking alternative diagnosis approaches based on different data modalities. Here, we propose a machine learning model to predict OSA severity from 2D frontal view craniofacial images. In a cross-validation study of 280 patients, our method achieves an average AUC of 0.780. In comparison, the craniofacial analysis model proposed by a recent study only achieves 0.638 AUC on our dataset. The proposed model also outperforms the widely used STOP-BANG OSA screening questionnaire, which achieves an AUC of 0.52 on our dataset. Our findings indicate that deep learning has the potential to significantly reduce the cost of OSA diagnosis.

Dual-energy computed tomography (DECT) enables to generate a series of virtual monoenergetic images (VMIs). Using VMIs of a desired energy level (5 – 45 keV) can enhance the lesion-to-background and voxel-to-voxel within lesion contrast, because that the lesion material composition may vary from voxel to voxel. However, there are also strong correlation of the voxel values among different energy channels. This correlation may result in redundant information for the VMIs based lesion pathology differentiation. Therefore, we transformed the VMIs in the Karhunen–Loève domain to reduce the correlation. In the new domain, the leading three principal components accounts for more than 99% information and then were used to form a new descriptor for the differentiation task. Two pathological proven datasets were used for the evaluation. Experimental results showed that the VMIs can improved the AUC (area under the receiver operating characteristic curve) value from 0.862 and 0.647 to 0.912 and 0.830 comparing to using the conventional CT.

Recent advancement of spectral computed tomography (SpCT) technologies by either multi-energy spectral data acquisition with energy-integration detector or single-energy spectral data acquisition with photon counting detector has enabled the reconstruction of virtual monochromatic images (VMIs) at any energy values within and outside the energy spectral ranges of current CTs’ X-ray tubes, resulting in the possibility of not only visualizing the tissue contrast variation characteristics along the X-ray energy dimension, but also quantifying the variation characteristics by machine learning (ML) for prediction of lesion malignancy or computer-aided diagnosis (CADx). This study explored the energy spectral information of SpCT, i.e., the contrast variation characteristics along the X-ray energy dimension, for ML-CADx of lesion type of colorectal polyps. Particularly, the tissue contrast variation patterns, called energy spectral features, along the Xray energy dimension in the VMIs is investigated. A figure of merit (FOM) for the task of ML-CADx is proposed, which ranks the series of VMIs along the X-ray energy dimension by inputting each VMI into a single channel deep learning (DL) pipeline and generating a corresponding a score of AUC (area under the curve of receiver operating characteristics). Then the FOM selects different numbers of the most highly ranked VMIs as the inputs to a multi-channel DL pipeline to generate the corresponding of AUC scores until all VMIs are selected. It is hypothesized that the AUC scores from the multi-channel DL pipeline will increase to reach the highest score and then drop along the ranking order, because all VMIs have the same anatomic structure and, therefore, the strong data redundancy. The FOM reaches the highest AUC score by minimizing the redundancy. We tested the hypothesis by comparing the proposed FOM-rank ML-CADx with the widely used Karhunen-Loève (KL) transform-based ranking method where the principal components are ordered automatically by the KL transform. The lesion data include the CT images of colorectal polyps and the pathological reports after they were resected. The proposed FOM-rank method outperformed the KL-based ranking method with an optimal gain of 4.7%, showing its effectiveness in prediction of lesion malignancy.

Coronary plaque risk classification in images acquired with photon-counting-detector (PCD) CT was performed using a radiomics-based machine learning (ML) model. With IRB approval, 17 coronary CTA patients were scanned on a PCD-CT (NAEOTOM Alpha, Siemens Healthineers) with median CTDIvol of 4.56 mGy. Four types of images: 120-kV PCD-CT image, virtual monoenergetic images (VMIs) at 50-keV and 100-keV, and iodine maps were reconstructed using an iterative reconstruction algorithm, a vascular kernel (Bv40) and 0.6-mm/0.4-mm slice thickness/increment. Atherosclerotic plaques were segmented using semi-automatic software (Research Frontier, Siemens). Segmentation confirmation and risk stratification (low- vs high-risk) were performed by a board-certified cardiac radiologist. A total of 1674 radiomic features were extracted from each image using PyRadiomics (v2.2.0b1). For each feature, a t-test was performed between low- and high-risk plaques (p<0.05 considered significant). Feature reduction was performed with a clustering algorithm and 6 non-redundant features were input into a linear support vector machine (SVM). A leave-one-out cross-validation strategy was adopted and the area under the ROC curve (AUC) was computed. Twelve low-risk and 5 high-risk plaques were identified by the radiologist. A total of 80, 66, 183, and 48 out of 1674 features in 120-kV, 50-keV, 100-keV, and iodine map images were statistically significant. The SVM classified 16/17 plaques correctly in the 120-kV PCD-CT and 50-keV VMI images. The AUC was 0.967, 0.967, 0.917, and 0.833 in 120-kV, 50-keV, 100-keV, and iodine map images, respectively. A ML model using coronary PCD-CTA images at 120-kV and 50-keV best automatically differentiated low- and high-risk coronary plaques.

The identification and management of Multiple Sclerosis (MS) patients who are at risk of disease progression and/or conversion to Secondary Progressive MS (SPMS) is a significant unmet clinical need. This study aimed to develop machine and deep learning (ML/DL) models to predict disability progression and/or conversion to SPMS at least 6 months before progression, determined by Expanded Disability Status Scale (EDSS) score. Three traditional ML algorithms were trained on brain parcellation volume measurements derived from T1 weighted magnetic resonance images (MRI) and convolutional neural networks (CNNs) were trained directly on MRI images. The results showed that the three ML models performed slightly better than a random classifier, with the Support Vector Classifier achieving the best results in terms of area under the receiver operating characteristic curve (AUROC) with a score of 0.62. The CNN approach yielded the most promising results, with an AUROC score of 0.75. This suggests that the ability to identify MS patients at high risk of future disability progression directly from MRI scans is a promising research direction. Further development of these algorithms could enable timely interventions, potentially preventing or delaying the onset of disability progression and/or SPMS.

Architectural distortion (AD) is one of the important breast abnormal signs in digital breast tomosynthesis (DBT). It is hard to be detected due to its subtle appearance and similar intensity with surrounding tissue. To assist radiologists to detect ADs, a single-view based computer-aided detection model in DBT was developed by us previously. In this study, considering the fact that radiologists always use information from craniocaudal (CC) and mediolateral oblique (MLO) views of DBT simultaneously for better diagnosis of each breast in clinic, we further develop a multi-view based AD detection model in DBT that combines the information from the two views. In this model, AD candidates in each view are detected by our previous AD detection model. Anatomical position priors of AD candidates in the two views are considered through establishing a 3D anatomical coordinate system. A multi-view based classifier is trained to fuse information from the two views and distinguish the true AD candidates. A dataset of 196 CC-MLO DBT pairs were collected with IRB approval, 101 of them contained ADs and the remaining were negative pairs. Ten-fold cross-validation showed that after involving our proposed multi-view method, the sensitivities of AD detection at 1, 2, 3 and 4 false positive predictions per DBT pairs improved from 0.66, 0.73, 0.77 and 0.79 to 0.69, 0.77, 0.78, and 0.83, respectively. The results showed that the multi-view based model achieved better detection performance than single-view based model. This model has potential to assist radiologists in detection of ADs in DBT.

Stroke is a major cause of death and permanent disability. Magnetic resonance imaging (MRI) is often the modality for evaluating lesion extension, affected brain, area and classification between hemorrhagic or ischemic, which are critical for treatment and rehabilitation decisions. MRI manual evaluation and lesion delineation is time-consuming and subject to inter- and intra-observer variation. Although promising, convolutional neural network (CNN) approaches face challenges in dealing with small-size lesions, irregular morphology, and idiosyncrasy. The best-published segmentation approaches based on the ATLAS dataset and CNNs are either fully 3D, or use up to eight anatomically specific 2D CNNs, incurring high computation costs and limited deployment feasibility. We developed a more straightforward segmentation method using only three CNNs. First, an Attention UNet and an Attention ResUNet are trained only on a lesion-wise balanced sample of slice patches. By working on the patch level, these networks learn to segment lesions’ texture and shape irregularities. Then, another Attention ResUNet uses the previous two CNNs output patches reassembled and stacked along with the original slice. By having the broader slice context available, this second step combines the first step segmentations, handling disagreements while keeping the segmentation coherence on the slice level. We validated the method on 239 exams from the multicenter ATLAS dataset, using a 5-fold cross-validation. In the test set of 45,171 slices, we observed a mean slice Dice coefficient of 0.8070±0.05, a state-of-the-art result in this dataset, showing generalization capacity on different centers and acquisition conditions.

The Corpus Callosum is the major interhemispheric commisure and, because of its highly organized fibers, it is often studied using diffusion tensor images (DTI). A firstnecessary step for CC studies is its segmentation, preferably automated. Since most available softwares are not able to perform CC volumetric segmentation, and the only ones that do it, are based on T1-weighted images and not DTI, this work presents the extension of an open-source software, called inCCsight, incorporating a DTI-based CC volumetric segmentation method into it. The software is open-source and offers the possibility of incorporating customized plots and integrating other segmentation and/or parcellation methods by the user.

Radiomics has shown predictive utility in kidney function decline for patients with autosomal dominant polycystic kidney disease (ADPKD), but a limiting factor in the clinical use of radiomics is the standardization of pre-processing parameters, which may be disease specific. Currently, there is a need to identify texture-based differences of riskstratified Mayo Imaging Classification (MIC) groups in ADPKD and the optimal pre-processing parameters for feature extraction. A cohort of 128 age- and gender-matched low/intermediate (MIC classes 1A-1B) and high-risk (MIC classes 1C-1E) patients and their respective T2-weighted fat saturated MRI representative coronal images were used to classify MIC risk using radiomic features extracted from (1) the non-cystic kidney parenchyma and (2) the entire kidney. Graylevel discretization across 8, 16, 32, 64, 128, and 256 gray levels using (1) fixed bin size and (2) fixed bin number methods were used for feature extraction with up-sampling (1.0×1.0 mm²) and down-sampling (2.0×2.0 mm²) pixel resampling. Feature selection using least absolute shrinkage operator (LASSO) combined relevant features into a logistic regression model to classify risk-stratified MIC classes. The non-cystic kidney classification yielded area under the receiver operating characteristic curve (AUC) values that ranged from 0.68-0.84, and the entire kidney texture classification yielded AUC values that ranged from 0.83-0.88. The non-cystic kidney parenchyma AUC values had a decreasing trend with increasing gray levels and were sensitive across pre-processing methods more so than features extracted from the entire kidney. Results suggest there are texture-based differences among risk-stratified MIC classes in both the non-cystic and entire kidney parenchyma that may help better identify patients who are at risk for end-stage kidney disease.

We introduce a simple physics-based model of RA-950 emphysema scoring. Our model assumes that the lung is strictly composed of healthy tissue and emphysematous tissue, each described by a single attenuation value and contaminated with Gaussian noise. We show that when combined with curve-fitting, the model can accurately capture change in RA-950 score with respect to image noise and subject breath hold, and accurately compute “true” RA-950 scores (relative to a clinical reference scan) in a cohort of 16 patients. To validate the model, noise realizations of 10 lung screening subjects and 6 COPD patients were created using various combinations of reconstruction parameters and simulated reduced dose acquisitions. Least-squares curve fitting software was utilized to determine the amount of emphysema and the attenuation value of healthy lung tissue for each subject using the model. The derived model provided accurate emphysema scores (difference between model value and clinical reference of < 0.02) in all cases except one. Upon radiologist review of this case, the score derived from our model was deemed more appropriate than RA-950 from the clinical reference scan. The R-squared values were < 0.9 in all cases except one, and < 0.95 in 12 of 16 cases. The case with low R² value was also reviewed by a radiologist and found to have substantial other disease that violated key model assumptions. The model appears to be robust to breath hold, image noise, and amount of emphysema present, factors that have been found to confound other approaches (such as denoising) to improving emphysema scoring.

Segmentation of the renal parenchyma responsible for a renal function is necessary for surgical planning and decisionmaking of renal partial nephrectomy (RPN) by identifying the correlation between the renal parenchyma volume and renal function after RPN on abdominal magnetic resonance (MR) images without radiation exposure. This paper proposes a cascaded self-adaptive framework that uses local context-aware mix-up regularization on abdominal MR images acquired from multiple devices. The proposed renal parenchyma segmentation network consists of two stages: kidney bounding volume extraction and renal parenchyma segmentation. Before kidney bounding volume extraction, self-adaptive normalization is performed using nnU-Net as the backbone network to reduce differences in signal intensity and pixel spacing among MR images of different intensity ranges acquired from multiple MR devices. In the kidney bounding volume extraction stage, the renal parenchyma area is segmented using 3D U-Net with low-resolution data down-sampled twice from the original to efficiently localize the kidney in the abdomen. Bounding volume is generated to focus on the renal parenchyma area during the renal parenchyma segmentation stage by cropping to the volume-of-interest region using the segmentation results up-sampled to the original resolution. In the segmentation stage, the renal parenchyma is segmented using 3D U-Net with mix-up augmented bounding volume to improve the regularization performance of the model. The average F1-score of our method was 92.27%, which was 3.07%p and 0.32%p higher than the segmentation method using original 3D cascaded nnU-Net and 3D cascaded nnU-Net with kidney bounding volume extraction, respectively.

Deep learning models can be applied successfully in real-work problems; however, training most of these models requires massive data. Recent methods use language and vision, but unfortunately, they rely on datasets that are not usually publicly available. Here we pave the way for further research in the multimodal language-vision domain for radiology. In this paper, we train a representation learning method that uses local and global representations of the language and vision through an attention mechanism and based on the publicly available Indiana University Radiology Report (IU-RR) dataset. Furthermore, we use the learned representations to diagnose five lung pathologies: atelectasis, cardiomegaly, edema, pleural effusion, and consolidation. Finally, we use both supervised and zero-shot classifications to extensively analyze the performance of the representation learning on the IU-RR dataset. Average Area Under the Curve (AUC) is used to evaluate the accuracy of the classifiers for classifying the five lung pathologies. The average AUC for classifying the five lung pathologies on the IU-RR test set ranged from 0.85 to 0.87 using the different training datasets, namely CheXpert and CheXphoto. These results compare favorably to other studies using UI-RR. Extensive experiments confirm consistent results for classifying lung pathologies using the multimodal global local representations of language and vision information.

We are developing a decision support system for treatment response assessment of bladder cancer in CT urography (CTU). Accurate segmentation of bladder cancer is a critical and challenging task. We previously developed a bladder cancer segmentation method using a deep learning convolutional neural network and level set (DL-CNN+LS) approach. In this study we investigated the application of a U-Net based deep learning (U-Net) model for bladder cancer segmentation. Our new U-Net method did not require the second-stage level set refinement, greatly simplifying the overall segmentation pipeline. The proposed U-Net model utilized a user-defined box to direct the attention of the U-Net to the lesion region by masking out the structured background outside the box. We trained and evaluated the performance of the U-Net segmentations by using hand-drawn 3D contours from a radiologist as reference standard. The segmentation accuracy was evaluated by the average volume intersection ratio (AVI), average percent volume error (AVE), average absolute volume error (AAVE), average minimum distance (AMD), and the Jaccard index (JI). On the validation set, the cropped U-Net achieved values of AVI = 65.5±20.2%, AVE = 3.7±44.4%, AAVE = 30.9±30.8%, AMD = 2.9±1.5 mm, and JI = 51.4±16.8%. The previous DL-CNN+LS approach achieved values of AVI = 36.2±28.4%, AVE = 51.1±39.3%, AAVE = 52.0±38.0%, AMD = 4.1±1.7 mm, and JI = 32.9±27.2%. On the independent test set, the cropped U-Net achieved values of AVI = 66.0±22.8%, AVE = -3.7±39.4%, AAVE = 28.1±27.3%, AMD = 4.5±3.3 mm, and JI = 49.2±20.0%. The DL-CNN+LS achieved values of AVI = 31.2±24.3%, AVE = 59.9±30.9%, AAVE = 60.8±29.0%, AMD = 5.5±2.2 mm, and JI = 27.9±20.8%, respectively. The results demonstrated that the U-Net model could achieve a higher accuracy than the previous DL-CNN+LS model while reducing the complexity of the segmentation pipeline.

We are developing deep-learning convolutional neural network (DL-CNN) and radiomics models to assist physicians in treatment response assessment in CT urography of bladder cancer to neoadjuvant chemotherapy (NAC). We collected a total of 264 pre- and post-treatment lesion pairs of 227 patients from University of Michigan hospital with IRB approval. The data were split into 3 sets by case: a training set including 35 complete responders (CRs) (T0 stage after treatment) and 113 non-complete responders (NCRs) (< T0 stage after treatment), a validation set including 5 CRs and 5 NCRs, and an independent test set including 19 CRs and 87 NCRs. The training set was used to train the models to classify CRs and NCRs and the selection of optimal models was guided by the validation set. The selected models were deployed on the test set to generate the likelihood score of CR of each pair. The classifying performance was evaluated by the area under ROC curve (AUC). Hybrid ROIs extracted from the lesions in the pre- and post- treatment scan pairs were used as input to the DL-CNN model. The optimal DL-CNN model achieved an AUC of 0.75 ± 0.06 on the test set. For the radiomics model, the random forest classifier was applied to the features extracted from the pre- and post-treatment lesions. The optimal radiomics model achieved an AUC of 0.76 ± 0.05. A combined DL-CNN and radiomics model increased the AUC to 0.77 ± 0.06. The results indicated the feasibility of using the DL-CNN model and radiomics model for assessing treatment response of bladder cancer.

This paper proposes an automated classification method of COVID-19 chest CT volumes using improved 3D MLP-Mixer. Novel coronavirus disease 2019 (COVID-19) spreads over the world, causing a large number of infected patients and deaths. Sudden increase in the number of COVID-19 patients causes a manpower shortage in medical institutions. Computer-aided diagnosis (CAD) system provides quick and quantitative diagnosis results. CAD system for COVID-19 enables efficient diagnosis workflow and contributes to reduce such manpower shortage. In image-based diagnosis of viral pneumonia cases including COVID-19, both local and global image features are important because viral pneumonia cause many ground glass opacities and consolidations in large areas in the lung. This paper proposes an automated classification method of chest CT volumes for COVID19 diagnosis assistance. MLP-Mixer is a recent method of image classification using Vision Transformer-like architecture. It performs classification using both local and global image features. To classify 3D CT volumes, we developed a hybrid classification model that consists of both a 3D convolutional neural network (CNN) and a 3D version of the MLP-Mixer. Classification accuracy of the proposed method was evaluated using a dataset that contains 1205 CT volumes and obtained 79.5% of classification accuracy. The accuracy was higher than that of conventional 3D CNN models consists of 3D CNN layers and simple MLP layers.

Worldwide, about 4-5 million deaths (9 percent) occur due to stroke. Stroke is the leading cause for neurological dysfunction resulting in considerable morbidity and mortality and holds the 6th position in the list of causes for reduced DALYS (Disability Adjusted Life Years). It’s categorized into two types: ischemic and hemorrhagic. Till date manual segmentation remains the gold standard in segmenting the stroke region of interest for every scan. This is a time-consuming process and inefficient in accurately identifying the ROIs. This work explores the hypothesis to develop an accurate automated stroke segmentation model. The 2D U-NET architecture used in segmenting the ROIs is based on the format to include the lesion size, lesion volume and their locations in the left and right hemisphere of the brain, and cortical and subcortical brain regions. The cohort consisted of single and multiple ROIs and were further bifurcated based on the volumes of these lesions. Based on this information we optimized our network into 5 different models keeping in mind all the information about the lesion size and volume. Eventually the average dice score obtained was highest for the network that included the combination of smaller and larger lesion, and Gd-T1w MRI scans containing the stroke lesion but no information of its size. The dice score for this model was 0.822 on the testing set. This result shows that considering the lesion sizes and volumes can help in obtaining a precise and automated segmentation model and can be the base work to further localize and classify the lesions.

Classification of Breast Imaging Reporting and Data System (BI-RADS) breast density categories generally reflects the amount of dense/fibroglandular tissue in the breast. Studies have consistently shown that breast with higher density has a higher risk of developing breast cancer compared to breast with lower density. In this paper, we propose a novel end-to-end method, namely, Medical Knowledge-guided Deep Learning (MKDL), for breast mammogram density classification. The principle behind MKDL lies in the fact that many breast image density classification tasks are partly or largely based on certain pre-known image features, such as image contrast and brightness. These pre-known features can be computationally represented and then leveraged as prior knowledge to facilitate more effective model learning and thus boost the classification performance. We designed specific knowledge-based transformations for breast density classification and showed that our model outperformed several state-of-the-art models.

COVID-19 has spread around the world since 2019. Approximately 6.5% of COVID-19 a risk of developing severe disease with high mortality rate. To reduce the mortality rate and provide appropriate treatment, this research established an integrated models with to predict the clinical outcome of COVID-19 patients with clinical, deep learning and radiomics features. To obtain the optimal feature combination for prediction, 9 clinical features combination was selected from all available clinical factors after using LASSO, 18 deep learning features from U-Net architecture, and9radiomics features from segmentation result. A total of 213 COVID-19 patients and 335 non-COVID-19 patients from5hospitals were enrolled and used as training and test sample in this research. The proposed model obtained an accuracy, precision, recall, specificity, F1-score and ROC curve of 0.971, 0.943, 0.937, 0.974, 0.941 and 0.979, respectively, which exceeds the related work using only clinical, deep learning or radiomics factors.

Radiologists routinely perform the tedious task of lesion localization, classification, and size measurement in computed tomography (CT) studies. Universal lesion detection and tagging (ULDT) can simultaneously help alleviate the cumbersome nature of lesion measurement and enable tumor burden assessment. Previous ULDT approaches utilize the publicly available DeepLesion dataset, however it does not provide the full volumetric (3D) extent of lesions and also displays a severe class imbalance. In this work, we propose a self-training pipeline to detect 3D lesions and tag them according to the body part they occur in. We used a significantly limited 30% subset of DeepLesion to train a VFNet model for 2D lesion detection and tagging. Next, the 2D lesion context was expanded into 3D, and the mined 3D lesion proposals were integrated back into the baseline training data in order to retrain the model over multiple rounds. Through the self-training procedure, our VFNet model learned from its own predictions, detected lesions in 3D, and tagged them. Our results indicated that our VFNet model achieved an average sensitivity of 46.9% at [0.125:8] false positives (FP) with a limited 30% data subset in comparison to the 46.8% of an existing approach that used the entire DeepLesion dataset. To our knowledge, we are the first to jointly detect lesions in 3D and tag them according to the body part label.

Dental panoramic radiographs are often obtained at dental clinic visits for diagnosis and recording purposes. Automated filing of dental charts can help dentists in reducing their workload and improving diagnostic efficiency. The purpose of this study is to develop a system that prerecords a dental chart by recognizing teeth with their numbers and restoration history on dental panoramic radiographs. The proposed system uses YOLO which detects 16 types of teeth and restoration conditions simultaneously. Based on the detected tooth types, they were further classified into 32 types and combined with the tooth conditions by post-processing. We tested our method on 870 panoramic images obtained at 10 different facilities by 5-fold cross validation. The proposed method obtained 0.99 recall and precision for recognition of 32 tooth types and 0.90 recall and 0.90 precision on determining the tooth condition. It has the potential to assist prefiling the dental charts for efficient dental care.

We studied the feasibility of developing a machine learning model to predict the survival of patients with metastatic urothelial cancer after immunotherapy. CT scans of 363 metastatic tumors in 49 patients undergoing immunotherapy were collected at every treatment time point. 1040 temporal triplets of metastatic cancers were formed. At every time point, a radiologist measured the tumor diameter. The patient survival data was collected from clinical records. Using the tumor diameters at prior time points as inputs, we built a model to predict patient survival after immunotherapy using artificial neural networks (PSNN). The PSNN used 3 prior time points to predict patient survival at a future time point: PS(t₄)=PSNN(d(t₁), d(t₂), d(t₃)). Specifically, PSNN was trained to predict patient survival at 4 years from the beginning of treatment (t₄=4) using 3 prior time points within 3 years from the beginning of treatment (0<t₁<t₂<t₃<3). We split the dataset into training (53 tumors, 13 patients, 335 triplets) and independent test (310 lesions, 36 patients, 705 triplets) sets. The final patient-based survival prediction scores were obtained by averaging PSNN scores of all triplets for a given patient. Area under the ROC curve (AUC) and Kaplan-Meier analysis were used for performance evaluation. The training and test AUCs for survival prediction at 4 years were 0.77±0.13 and 0.73±0.09, respectively. Using a decision threshold determined by the training set, the test set was stratified into two subgroups of longer and shorted survival. Median survival time for the 2 test subgroups estimated by the PSNN was 5 and 2 years, respectively (p=0.025). The PSNN shows promise for predicting patient survival after immunotherapy.

Pulmonary vessel segmentation from CT images is essential to diagnosis and treatment of lung diseases, particularly in treatment planning and clinical outcome evaluation. The main challenge for pulmonary vessel segmentation is complicated structures of the vascular trees and their similar intensity values with other tissues like the tracheal wall and lung nodules. This paper presents a novel relation extractor U-shaped network combining convolution and self-attention mechanism in an encoder-decoder mode. Particularly, we employ convolution in the shallow layers to extract local information of vessels in a short range and apply self-attention in the deep layers to capture long-range contextual relationship between ancestors and descendants of the vascular tree. We evaluate our proposed method on 50 computer tomography volumes, with the experimental results showing that our method can improve the average coefficient dice and recall to 85.60 and 86.04 respectively.

Optical coherence tomography (OCT) is a non-invasive imaging modality that suitable for accessing retinal diseases. Since the thickness and shape of the retinal layer are diagnostic indicators for many ophthalmic diseases, segmentation of the retinal layer in OCT images is a critical step. Automated segmentation of oct images has made many efforts but there are still some challenges, such as lack of context information, ambiguous boundaries and inconsistent prediction of retinal lesion regions. In this work, we propose a new framework of Densely Encoded Attention Networks (DEAN) that combines dense encoders with position attention in an U-architecture for retinal layers segmentation. Since the spatial position of each layer in OCT image is relatively fixed, we use convolution in dense connections to obtain diverse feature maps in the encoder and employ position attention to improve the spatial information of learning targets. Moreover, up-sampling and skip connections in the decoder are to restore resolution by the position index saved during down-sampling, while supplementing the corresponding pixels is to guide the network capturing the global context information. This method is evaluated on two public datasets, and the results demonstrate that our method is an effective strategy on improving the performance of segmenting the retinal layers.

Over 795,000 Americans suffer a stroke every year, leading to a death every 3.5 minutes. Approximately 87% of all strokes are Acute Ischemic Strokes (AIS), i.e., an abrupt interruption in cerebral circulation due to a blocked artery. Early prediction of AIS final outcomes (AIS lesions) is crucial to effective treatment planning for AIS patients. Due to its speed, availability, and lack of contraindications, Computed Tomography Perfusion (CTP) is preferred over other imaging modalities with higher resolution (e.g., MRI), for AIS lesion prediction. However, the low contrast of baseline CTP images makes it difficult to determine AIS lesions precisely, while follow-up MRI images do. Therefore, this paper proposes a method of synthesizing follow-up MRI images from baseline CTP scans by a Temporal Generative Adversarial Network (TGAN) — which encodes baseline CTP frames with a series of encoders, followed by a decoder that forecasts the high-resolution follow-up MRIs. It also uses a discriminator that competes with the generator to identify whether its input MRI is real or fake. Furthermore, our TGAN includes a segmentor that can identify AIS lesions in those synthesized MRI images. The generator, discriminator, and segmentor in TGAN each use MultiRes U-Nets, an extension of the original U-Net architecture, which can robustly segment objects of various scales and shapes. Our experiments with Leave-One-person-Out Cross-Validation (LOOCV) obtained an average dice coefficient of 56.73%, with a significant p<0.05. In comparison to traditional methods using CTP perfusion parameters, we found that our novel method was more accurate in predicting AIS lesions.