1.

INTRODUCTION

This manuscript expands upon a Keynote Presentation on this topic at the SPIE Medical Imaging Conference, San Diego, CA, in February 2024.[1] The keynote address began with a pseudo-scientific, audience participation survey with three questions.

The fundamental issue exposed in this survey, and discussed further in this manuscript, explores the challenges in developing technology in academia and in translating academic technologies beyond the walls of its institution of origin. To illustrate these challenges, an example project will be presented that translated technology broadly through various separate and distinct licensing mechanisms: Academic, open-source, and commercial licensing. The challenges and benefits of each of these approaches will be discussed.

The case study example will focus on the development and dissemination of a software platform for managing the workflow and imaging assessments for oncology clinical trials, first developed at a single NCI-designated Comprehensive Cancer Center, then distributed through academic licensing to other cancer centers, and then licensed commercially through a company. The imaging informatics platform went through several generations, and its latest imaging viewer is built upon an open-source web-based imaging framework also developed by our group and integrated into the clinical trials informatics platform.

The fundamental problem in clinical trials imaging is that there are many stakeholders, and until recently, there has been no technology platform to link them together in a standardized and harmonized transparent workflow. Pharma companies, government agencies, or consortia are typically the funders of clinical trials. Each participating trial site is expected to manage their local “site reads”, while Contract Research Organizations (CROs), imaging CROs (iCROs), or imaging core labs (academic or commercial) may be contracted and tasked with managing the “central read” imaging assessments and workflow. While some trials may include a blinded independent central review (BICR) in an effort to avoid bias of reads at the participating trial sites, studies have shown a high discordance rate between site and central reviews.[2] This means that patients who do not meet the trial criteria may be enrolled and treated at sites so that the wrong patients are then sent for central BICR assessment. Because the cost to enroll each patient is so high, even conservative estimates of 10% enrollment errors can cost the sponsor more than the total trial radiology costs combined.[3] Thus, a solution that addresses clinical trials imaging assessment and workflow challenges that begin at the site, that can standardize and harmonize compliant reads at the point of patient care as well as across all sites in a trial, and can seamlessly connect sites, CROs, and sponsors is sorely needed by academia and the life sciences industry alike.

Unfortunately, sites are given the impossible task of trying to manage dozens or hundreds of clinical trials, each of which is described only in a lengthy set of documents and whose imaging criteria (the set of lesion selections, measurements, calculations, and rules that are applied to arrive at an overall tumor/patient response) vary for each trial. While the most commonly used oncology clinical trial criteria is RECIST 1.1,[4] there are over 30 established imaging criteria and each clinical trial typically has modifications making each trial a unique ‘snowflake’. Radiologists at the sites are asked to provide imaging assessments without adequate tools, noting that radiology picture archive and communications systems (PACS) do not manage these research workflows nor provide response criteria training or guidance. Often, radiologists are asked to provide these imaging assessments without clear communication of the protocol specifics for each patient on each trial, all while trying to manage an overwhelming clinical workload and pressure to publish, get grant funding, and attend conferences (see Figure 1). As a result, there is a 25-30% error rate in site reads of imaging assessments.[5] In addition, local radiologists are often unable to provide the quality, timeliness, and service needed by the oncology clinical trials teams, who require the imaging assessment prior to the patients’ office visits when the clinical trial treatment decisions are made. With these delays, oncology teams are sometimes left to perform their own imaging assessments. Unfortunately, because the treating teams manage the trial patients’ care and are thus prone to bias, studies have shown an overestimation of treatment effect response rates when oncology study teams perform their own site imaging assessments.[6]

Figure 1.

Clinical trial imaging assessment workflow challenges from the radiologist’s perspective at a cancer center.

The problem is a systems problem, not an image analysis problem. Image analysis techniques that use machine learning or artificial intelligence to help automate tumor measurements are only a tiny piece of the issue. For example, making a bi-dimensional measurement on a tumor, as is required for most oncology criteria, requires three clicks if done manually. Using machine learning to reduce the annotation to one-click might save 3 seconds, and if there are a maximum of five lesions in a standard RECIST 1.1 protocol, that might save 15 seconds total, although it may end up costing more time for semi-automated methods if the segmentation is not accurate and the radiologist has to make adjustments. On the other hand, the radiologist ends up spending a great deal of time determining the protocol specifics, fielding emails and phone calls from the study team, being interrupted in their clinical workflow, responding to re-review requests if there are questions, filling out paper forms and spreadsheets, etc. The study teams also spend extensive time interacting with the radiologists and manually transferring these data from the radiology forms onto their source documents, continuously updating them to stay prepared for audits, managing the billing, interpreting the imaging protocols for new trials, etc. The study teams may spend eight hours dealing with imaging assessments for every hour of radiology time performing the assessments (see Figure 2).

Figure 2.

Clinical trials imaging assessment workflow challenges from a system-wide communication level for oncology study teams and radiologists at a cancer center. The email with the counter at the bottom right indicates the number of emails that may be sent to track down a single imaging assessment for one patient.

A sponsor may have dozens of sites on a trial, and each site must independently interpret the lengthy imaging protocol and determine how to implement it, which affects site activation, protocol compliance, and variability across sites. Each site is left on their own to figure out what software to use to make measurements, what measurements to make, and there is typically no oversight, quality control, or compliant tools to ensure it is done correctly and on time for the office visit.

These confounding factors can materially impact early signals of efficacy, especially in early phase and co-operative group trials, because there is no standardization or harmonization across sites for the site reads, and CRO services may not be used. This has led to high rates of protocol violations, observed site-central discordance, delays and errors in determining the effectiveness of new therapeutic agents, and higher costs of drug discovery (See Figure 3).

Figure 3.

The challenges described in Figures 1 and 2 within each site in a trial (left) are multiplied many times over for sponsors, CROs, and imaging CROs that work across many sites for each trial.

2.

METHODS

Tumor Imaging Metrics Core Mission: In 2004, we created a new Tumor Imaging Metrics Core (TIMC) to provide protocol-specific imaging assessments for patients on oncology clinical trials across the five teaching hospitals of an NCI-designated Comprehensive Cancer Center. Requirements included supporting the needs of oncology investigators and study teams who could be located at any of the five participating hospitals with patients who could be enrolled and imaged at any of the sites. The hospitals had separate radiology departments and PACS archives, and the results needed to be available to the trials’ staff in time for the patients’ office visits, compliant with the specifics of each protocol, and with traceability to support trial audits. The core was required to become self-sustaining through “chargeback” fees, billed per imaging assessment time point to each trial budget, and the system needed to track and manage all the work performed to report how much to bill to each clinical trial budget and how much to pay radiologists for each image assessment. It also needed to provide a mechanism for communication between study staff and imaging teams and enable requests for modifications if there were questions from the study team requiring re-reviews.

TIMC Funding Strategy: Funding was needed to hire three initial staff and set up infrastructure to 1) develop a web-based platform to manage the workflow, communication, and imaging assessments, 2) develop the IT infrastructure to host the servers, and 3) develop the standard operating procedures, compliance processes, and documentation, and perform preliminary imaging assessments. Participating radiologists were enrolled to perform final reviews. By having an image analyst do a preliminary assessment with a board-certified radiologist performing the final review, we were able to reduce cost by making the most efficient use of radiologist time and effort.[7] Initial funding was acquired from several sources, including our National Cancer Institute (NCI) Cancer Center Support Grant (CCSG) as a new “core in development”, a grant from the Children’s Tumor Foundation, a departmental research grant, and support from lab revenues. However, the TIMC imaging assessment “chargeback” fees became the predominant source of core funding as the core grew.

TIMC First Generation Academic Software Development: To meet the above requirements, we investigated available software, but we did not find any that met our needs, so we built a comprehensive platform to manage imaging assessments and workflow informatics across multi-site clinical trials. The web-based system we initially developed was a workflow management system (WMS). We initially used a 3^rd-party thick-client commercial imaging platform to perform the imaging assessments and then manually uploaded the annotated images and measurement results. We felt it was too much development effort to build both the WMS and a custom image viewer at the outset. After several years, as our service grew, we integrated a custom image viewer into the WMS, which we built upon a widely used open-source thick-client imaging workstation. This was the first generation of our integrated workflow informatics and image assessment platform.

TIMC Second Generation Software and Academic Licensing: In 2012, we began receiving requests from other cancer centers indicating that they had heard about our platform solution and asked if they could use our system to address their similar clinical trials imaging assessment problems. Providing software services and support to outside institutions required us to harden the software into a more commercial-grade application, and so we re-invested our chargeback revenues to build the second generation of our system (v2). We hired commercial software developers and a user experience engineer to build the next-generation platform from scratch. Outside counsel was employed to develop an “Academic Software License Agreement,” as our institution did not have the experience nor a template for this type of software licensing between academic institutions. The non-royalty-bearing academic license agreement included usage and support fees to cover our costs of hosting, developing, and supporting the platform for other cancer centers and provided an increasing revenue stream to make continuous improvements. We did not have sales or marketing, and collaborations under this academic software licensing model developed slowly, primarily by word of mouth or chance interactions.

Web-Viewer Development and Open-Source Licensing: In 2015, we believed that web-browser technology had advanced to the point where we could move from our open-source thick-client workstation-based viewer to a zero-footprint web-viewer and make the entire platform a hosted software as a service (SaaS) solution. Whereas desktop imaging applications like our first-generation viewer are limited to specific operating systems, can conflict with other installed software, and are challenging to support and update across users and sites, a cloud-hosted web-viewer can be accessed from any browser on any computer, requires no local software installation or support, and can be updated centrally without having to manage version controls across installation sites. Again, we searched for a commercial-grade open-source web-viewer platform and did not find any that would meet our requirements. Together with like-minded collaborators, we formed the Open Health Imaging Foundation (OHIF) and applied for a grant from the National Cancer Institute (NCI) Informatics Technology for Cancer Research (ITCR) program and received an Advanced Development grant (2015-2020) to build the OHIF Viewer. When the original grant ended, we funded the continued development of OHIF through a combination of grants from the Chan Zuckerberg Initiative (CZI) Essential Open-Source Software for Science (EOSS) program (2021-2024), and an NCI ITCR sustainment grant (2023-2028) as well as funding from an academic-industry partnership project and departmental support. The OHIF Viewer [8] provides a framework for display and measurements on medical images that is vendor-neutral, extensible, zero-footprint running entirely in the web browser, and is available for free under the commercially permissive, non-viral MIT open-source software license. The OHIF Viewer was integrated with the WMS, enhanced with proprietary integrations and functionality specifically for the clinical trials use case, and ultimately replaced the first generation thick-client workstation-based viewer.

Integrated Cloud-Hosted Platform and Commercial Licensing: In 2020, we released the new fully-cloud-hosted platform with the integrated web-viewer and began migrating existing sites onto the new viewer. The following year, we began to draw attention from potential industry partners who expressed interest in obtaining a commercial license to the integrated clinical trials imaging informatics platform. Finally, in 2022, we executed an exclusive commercialization license to a company, Yunu, Inc. (www.yunu.io), which has launched our academic endeavor forward as a true commercial platform, with expanded resources, scope, compliance profile, and functionality.

3.

RESULTS

In this section, we describe usage of TIMC, OHIF, and Yunu.

TIMC was able to gain rapid adoption by utilizing the clinical trials imaging informatics platform to provide timely, protocol-compliant, multi-modality (CT, MRI, PET, etc.) imaging assessment services. The core service improves imaging assessment turnaround time and accessibility, reduces study team preparation time for data locks and audits, ensures protocol compliance, improves reliability and accuracy of tumor metrics, streamlines communication across oncology and radiology, and ensures financial tracking and compliance As a result, TIMC usage grew rapidly since 2005, when we began performing services for our first clinical trial, and is currently performing imaging assessments for about 1,000 time points per month.

The OHIF Viewer and its underlying Cornerstone libraries (together the OHIF Framework), have gained a global presence of developers and users. OHIF and Cornerstone usage statistics include:

Yunu has accelerated innovation for the cloud-hosted clinical trials imaging informatics platform, initially developed by TIMC with its integrated OHIF-based web-viewer, and has evolved the system toward increased collaboration, efficiency, functionality, and accuracy. Yunu is currently used to manage over 4,200 active clinical trials (including trials from over 400 life sciences companies) by over 2,600 active users across 22 organizations, including 13 (of the roughly 60) NCI-designated cancer centers that participate actively in oncology clinical trials of solid tumors. Yunu has obtained Series A financing from institutional investors, which has enabled accelerated software development and implementation of commercial-grade business, compliance, and software processes. These investments have unlocked opportunities in life sciences through the ability of the platform to support imaging CROs, full-service CROs, Pharma, and clinical trial technology partners. Yunu enables all participating stakeholders to set up and manage imaging clinical trials in a more operationally efficient, extensible, and harmonized way. Yunu can also incorporate meaningful forms of imaging AI and radiomic analyses and apply these to imaging data from prospective or retrospective clinical trials.

4.

CONCLUSIONS

In this section, challenges and benefits of the various licensing approaches will be discussed.

There are many hurdles in building and maintaining academic software. First and foremost, it is crucial to identify broad and significant gaps, look for off-the-shelf solutions, and deeply investigate the specific needs of potential users before taking on a software development effort. Once a project has been defined and justified, one must acquire funding for developers, project management, and user support. Sources available in academia include grants, departmental funds, chargebacks, collaboration, or philanthropy. If widespread use is a goal, the project must implement solid development processes and provide an intuitive user experience that supports multiple projects, users, and high-value use cases. Finally, to create a lasting project with maximum impact, it is necessary to develop a sustainable financial model and licensing plan. Benefits of academic software development and of dissemination through academic licensing include the availability of support from institutional resources (legal, IT, compute infrastructure, insurance, facilities, grant management, etc.) and support from grant sponsors like the National Institutes of Health (NIH). There is also an easier pace of work in academia, allowing for organic growth, where a project can be self-funded and sustained during the formative years without undue commercial pressure to expand. This is a significant advantage and success driver for technology translation because it provides more time for incubation, experimentation, and evolution.

Developing open-source software and making it freely available can be emotionally gratifying in serving the greater good, and it also has its own set of tangible benefits and challenges that need to be considered. Funding for open-source software is competitive and limited, but there are viable paths such as grants, collaborations, and institutional support. Since open-source software is free to use, it is hard to support financially. In addition, developing commercial-grade open-source software that can be widely used and deployed is complex and requires an operation akin to a software company inside an academic institution. Realizing this goal requires identifying and supporting expert developers who can perform excellent product management, software architecture, software development, and UI/UX engineering and can implement solid development processes and testing procedures. Which innovations to make open-source and which others to keep proprietary or license out exclusively is also a difficult choice involving thoughtful consideration of the stand-alone commercial value of any proprietary components involved. For example, an open-source project might include functionality of common interest (not proprietary “special sauce”). Once something is released as open source, everyone has unlimited rights to use it however they want in perpetuity and for free. The consumers of open-source software are often contributors to its development, creating a rapid prototyping environment and a shared interest in jointly hardening the software. Open-source software licensing provides an excellent way for a large group of users and use cases to share the effort and cost of maintaining foundational building blocks that would otherwise be developed separately at greater cost and likely at lower quality.

Commercial licensing also has many challenges, both in the process of licensing software from academia to a company and in creating commercial success and sustainability. Firstly, commercial licensing out from academia is complex. At many institutions, out-licensing of software may be unfamiliar (ours was used to licensing patents from drug development but not for software). There are many steps to license out software and/or intellectual property, with a host of seemingly insurmountable hurdles to be tackled one at a time. It is important to work with the institutional resources in coordinating conflicts-of-interest review, licensing, equity allocations, financing, transfer of employees, and existing academic licensing contracts (if any). The institution will review the agreements for conflicts of interest for staff that wish to participate in the company financially. Related to financial incentives, it is quite difficult for an academic investigator to negotiate with the industry partner to make sure their institution and other internal participating parties involved are all getting a fair deal. We hired an independent consultant with company and intellectual property transactional experience to negotiate the transfer mechanics and equity participation for the team. A software license and/or intellectual property agreement defining the institutional benefit, such as royalties, cash, and/or equity, must be negotiated with the company. Typically, the investigator who is most knowledgeable on the matters must be recused from these negotiations due to financial conflicts. An end-user agreement for the institution to become a customer of the company may be needed if the institution will continue to use derivative versions of software developed further by the company that licensed the technology, and any end-user agreements in place with other institutions need to be assigned to the company. A services agreement between the institution and the company may be needed if the institution will provide services to the company or through the company to other entities. Each of these steps is time-consuming and can be frustrating to the company negotiating a license, which is trying to move at a more energetic pace than academic institutions are typically accustomed to. There may be a culture gap between the entities, and some key players may not feel making this move is right for them. On the positive side, once these steps are accomplished and the technology is launched commercially, the product development and customer base growth can have a faster pace of action and activity than is typically achievable within academia. In addition, those who have invested a significant amount of time and reputational capital in the development of the licensed technology often find a positive market response to their innovation one of the greatest rewards.

However, successful commercial licensing is not the end but merely the beginning of a new chapter with its own set of challenges, opportunities, and requirements for the new company to succeed and sustain. Sadly, the large majority of new companies do not survive as a direct result of how post-license activities are managed. A successful company requires not only a product that fills an unmet need but also must employ a wide range of commercial skills to attract both customers and institutional investment. These include an experienced executive team, branding, marketing, sales and business development, efficient operations, regulatory and compliance, legal, product management, development, and testing. There is more pressure to grow than in academia as each round of investment expects to see growth and progress since the prior round, and then there are other global factors that impact the investment markets and comparable valuations that must be met for a company to attract continued investment as well. A small company must balance resources to make both the product and the company itself a saleable product worthy of investment. This process continues until the company reaches profitability and can self-sustain, or it sells its assets along with the license to another entity.

In conclusion, the keys to success in developing and deploying software, methods, or products from academia start with assessing the need: Listen to your clients; they will tell you what you need to do. Secondly, it is essential to provide exemplary customer service, and a product is only a means to deliver a service that someone needs. A team approach is important as a successful project requires a broad range of skills, and as it grows, more resources are needed. These resources can be found through open-source communities, grants, and private companies. There is no one-size-fits-all solution. In each case, the operations must be run with solid and realistic business planning, operational expertise, outside advice, and careful financial management. All of these are critical to sustaining a project over the long term, regardless of the path chosen.

In terms of the question of academic vs. commercial vs. open-source licensing, in our latest endeavor, we have found that it is possible to do all three successfully. Building a clinical trials imaging informatics platform required close collaboration with academic collaborators, and the academic licensing model proved invaluable. The open-source model then provided an outstanding opportunity to build a foundation that was useful for our clinical trials platform and that has also benefited the imaging community at large. Finally, to take the platform to market with full vigor and focus, a commercial license provided the best path for value creation for our institution while also ensuring the rapid and continued development of the platform we rely on heavily.