Since stepping into the role as Editor in Chief of the Journal of Medical Imaging (JMI), I have encountered amazingly creative uses of generative AI (“GenAI,” including large language models (LLMs), AI imaging, and multimodal models) that change the way I think about information, probability, and learnable functions. We are clearly in the midst of a transformation of both scientific and creative processes ( Is it art?). At my home institution, we have wholeheartedly embraced this discussion, updated our honor code, and partnered with Coursera to offer dozens of GenAI classes beyond our physical campus. Yet, as my daughter starts her first year of college, she is reviewing her syllabi to find exactly the opposite stance, where GenAI is broadly prohibited.

In JMI, I have received papers obviously written by GenAI, and I subscribe to retraction watch reports highlighting what happens when academics lose focus on the ultimate importance of integrity. On a weekly basis, JMI receives submissions that do not appear to have not followed the long-established scientific process (which waste substantial staff and editor time for no public benefit). Yet, it is always easiest to point fingers “over-there” and imagine that the line between right and wrong is crystal clear. Unfortunately, with new technologies, this can be a difficult distinction. For SPIE and JMI, an absolute imperative in the use of GenAI is disclosure.

To provide content for the discussion, I partnered with Elias Levy (co-author and the person who did the vast majority of the work for this editorial) to construct a “ChatGP-me”: a chatbot based on my published writings. Then, Elias worked with our lab members in the Medical-image Analysis and Statistical Interpretation (MASI) Lab to evaluate how the GenAI technologies available in 2024 (without specific coding) could impact scientific processes. A synopsis of interaction with Elias’s system is included in the Supplementary Material. Briefly, we explore two AI tools and aim to demonstrate what happens when using custom LLMs in five research lab workflows.

The first of these tools is NVIDIA’s ChatRTX which allows for secure training and tuning of LLMs locally on relatively affordable graphics cards. ChatRTX can be set up to run on a local area network for use within an organization or deployed on the web for public use. There are various models available for use with the software, including Gemma, Llama2, and Mistral, each with their own strengths and weaknesses. In ChatRTX, tuning is done through the addition of a “dataset” folder and it takes roughly 5 minutes to fine-tune the model with a 1 GB dataset (on an NVIDIA RTX 4090 card). The cost for using this software is almost entirely upfront, with the main cost being the device to run the software itself.

The other tool we looked at is Vanderbilt’s Amplify GenAI. The stated mission of the Amplify GenAI team is “to help make enterprise GenAI more open, cost-effective, and accessible.” The costs for this software are based on token usage rather than a per-user periodical subscription payment model, with data being stored in the cloud. Amplify GenAI offers a range of pre-trained models including OpenAI’s GPT-4o, Anthropic, Google, etc., with a variety of additional features such as assistant creation, prompt templates, workspaces, and sharing within organizations.

Both tools use retrieval augmented generation (RAG) technology, are open-source, and were designed with data privacy in mind. The data going into these models and the prompting used in chats have a very strong effect on the quality of responses. It is also important to acknowledge that at no point is AI meant to replace scientific researchers, but rather to help researchers complete their work with a higher minimum level of quality and do it more efficiently. Through this, we want to help develop a more consistent and safe experience for researchers who wish to use AI in their workflows.

In Quarter 3 2024, Amplify was further along the development process and yielded higher quality responses, so we used it to qualitatively analyze the effectiveness of using custom LLMs in the following lab workflows:

In Amplify GenAI, an “assistant” is a customizable bot that will respond from a chosen pre-trained model in the platform. Amplify GenAI allows for instructions to help the model understand the content of prompts and how to respond to them. It also allows for the addition of data sources, such as internal documents that models can reference outside of their training. This enables the models to give meaningful information that is relevant to the researcher. These assistants can also be easily shared within an organization using the platform.

In all examples, we used approximately 1 GB of data consisting of 302 of my (BL) refereed journal articles. We decided to only use our own articles for several reasons. First, we wanted to avoid any possible copyright concerns as we only used author-copies of manuscripts in processes internal to our lab. Second, we wanted to be able to ask AI to be critical of our papers for a use case like AI-assisted review, and we felt that it would be overly harsh to ask AI to critique the work of others if we were not willing to looking at our own work through AI’s critical lens.

While the integration of AI tools in scientific research offers substantial benefits, it is important to acknowledge and address the ethical implications and potential risks associated with their use, especially in academic environments.