Purpose: Computer-assisted skill assessment has traditionally been focused on general metrics related to tool motion and
usage time. While these metrics are important for an overall evaluation of skill, they do not address critical errors made
during the procedure. This study examines the effectiveness of utilizing object detection to quantify the critical error of
making multiple needle insertion attempts in central venous catheterization. Methods: 6860 images were annotated with
ground truth bounding boxes around the syringe attached to the needle. The images were registered using the location of
the phantom, and the bounding boxes from the training set were used to identify the regions where the needle was most
likely inserting the phantom. A Faster region-based convolutional neural network was trained to identify the syringe and
produce the bounding box location for images in the test set. A needle insertion attempt began when the location of the
predicted bounding box fell within the identified insertion region. To evaluate this method, we compared the computed
number of insertions to the number of insertions identified by human reviewers. Results: The object detection network
had an overall mean average precision (mAP) of 0.71. This tracking method computed an average of 4.40 insertion attempts
per recording compared to a reviewer count of 1.39 attempts per recording. Conclusions: The difference in the number of
insertion attempts identified by the computer and reviewers decreases with an increasing mAP, making this method suitable
for detecting multiple needle insertions using an object detection network with a high accuracy.
Purpose: Computer-assisted surgical skill assessment methods have traditionally relied on tracking tool motion with physical sensors. These tracking systems can be expensive, bulky, and impede tool function. Recent advances in object detection networks have made it possible to quantify tool motion using only a camera. These advances open the door for a low-cost alternative to current physical tracking systems for surgical skill assessment. This study determines the feasibility of using metrics computed with object detection by comparing them to widely accepted metrics computed using traditional tracking methods in central venous catheterization. Methods: Both video and tracking data were recorded from participants performing central venous catheterization on a venous access phantom. A Faster Region-Based Convolutional Neural Network was trained to recognize the ultrasound probe and syringe on the video data. Tracking-based metrics were computed using the Perk Tutor extension of 3D Slicer. The path length and usage time for each tool were then computed using both the video and tracking data. The metrics from object detection and tracking were compared using Spearman rank correlation. Results: The path lengths had a rank correlation coefficient of 0.22 for the syringe (p<0.03) and 0.35 (p<0.001) for the ultrasound probe. For the usage times, the correlation coefficient was 0.37 (p<0.001) for the syringe and 0.34 (p<0.001) for the ultrasound probe. Conclusions: The video-based metrics correlated significantly with the tracked metrics, suggesting that object detection could be a feasible skill assessment method for central venous catheterization.
Purpose: As medical schools move toward competency-based medical education, they seek methods of quantifying trainee skill without human expert supervision. This study evaluates the efficacy of using object detection to track performance metrics in ultrasound-guided interventions, specifically central venous catheterization. While several studies have explored methods to automate the evaluation of these interventions, they typically rely on expensive, bulky markers. Therefore, a webcam-based approach is desirable. Methods: We used the Faster Region-Based Convolutional Neural Network object detection network developed by Ren et al. to track the two-dimensional path length and the usage time of seven tools used in central venous catheterization. Object detection relies solely on webcam imagery. Video data were collected from recordings of 20 central venous catheterization trials by four different medical students. Each recording was separated into individual frames, annotated, and inputted to the object detection network. Mean average precision was calculated for each fold and each tool. Results: The average mean average precision was 0.66. Between trials one and five, the average reduction in tool usage time was 52%, and the average reduction in 2D path length was 29%. Conclusions: The neural network was able to identify each tool with considerable accuracy. Furthermore, the neural network successfully computed differences in performance metrics that emerge as trainees gain experience. Faster Region-Based Convolutional Neural Network is an effective method to assess trainee skill in ultrasound-guided interventions
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.